Re: [Nouveau] Rewriting Intel PCI bridge prefetch base address bits solves nvidia graphics issues

[Peter Wu <peter@lekensteyn.nl>]

On Tue, Aug 28, 2018 at 10:23:24AM +0800, Daniel Drake wrote:
> On Fri, Aug 24, 2018 at 11:42 PM, Peter Wu <peter@lekensteyn.nl> wrote:
> > Are these systems also affected through runtime power management? For
> > example:
> >
> >     modprobe nouveau    # should enable runtime PM
> >     sleep 6             # wait for runtime suspend to kick in
> >     lspci -s1:          # runtime resume by reading PCI config space
> >
> > On laptops from about 2015-2016 with a GTX 9xxM this sequence results in
> > hangs on various laptops
> > (https://bugzilla.kernel.org/show_bug.cgi?id=156341).
> 
> This works fine here. I'm facing a different issue.

Just to be sure, after "sleep", do both devices report "suspended" in
/sys/bus/pci/devices/0000:00:1c.0/power/runtime_status
/sys/bus/pci/devices/0000:01:00.0/power/runtime_status

and was this reproduced with a recent mainline kernel with no special
cmdline options? The endlessm kernel on Github seems to have quite some
patches, one of them explicitly disable runtime PM:
https://github.com/endlessm/linux/commit/8b128b50cd6725eee2ae9025a1510a221d9b42f2

> >> After a lot of experimentation I found a workaround: during resume,
> >> set the value of PCI_PREF_BASE_UPPER32 to 0 on the parent PCI bridge.
> >> Easily done in drivers/pci/quirks.c. Now all nvidia stuff works fine.
> >
> > I am curious, how did you discover this? While this could work, perhaps
> > there are alternative workarounds/fixes?
> 
> Based on the observation that the following procedure works fine (note
> the addition of step 3):
> 
> 1. Boot
> 2. Suspend/resume
> 3. echo rescan > /sys/bus/pci/devices/0000:00:1c.0/rescan
> 4. Load nouveau driver
> 5. Start X
> 
> I worked through the rescan codepath until I had isolated the specific
> code which magically makes things work (in pci_bridge_check_ranges).
> 
> Having found that, step 3 in the above test procedure can be replaced
> with a simple:
>    setpci -s 00:1c.0 0x28.l=0
> 
> > When you say "parent PCI" bridge, is that actually the device you see in
> > "lspci -tv"? On a Dell XPS 9560, the GPU is under a different device:
> >
> >   -[0000:00]-+-00.0  Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers
> >              +-01.0-[01]----00.0  NVIDIA Corporation GP107M [GeForce GTX 1050 Mobile]
> >
> >  00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05)
> 
> Yes, it's the parent bridge shown by lspci. The address of this varies
> from system to system.

Could you share some details:
- acpidump
- lspci -nnxxxxvvv
- BIOS version (from /sys/class/dmi/id/)
- kernel version (mainline?)

Perhaps there is some magic in the ACPI suspend or resume path that
causes this.

> >> 1. Is the Intel PCI bridge misbehaving here? Why does writing the same
> >> value of PCI_PREF_BASE_UPPER32 make any difference at all?
> >
> > At what point in the suspend code path did you insert this write? It is
> > possible that the write somehow acted as a fence/memory barrier?
> 
> static void quirk_pref_base_upper32(struct pci_dev *dev)
> {
>        u32 pref_base_upper32;
>        pci_read_config_dword(dev, PCI_PREF_BASE_UPPER32, &pref_base_upper32);
>        pci_write_config_dword(dev, PCI_PREF_BASE_UPPER32, pref_base_upper32);
> }
> DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL,  0x9d10, quirk_pref_base_upper32);
> 
> I don't think it's acting as a barrier. I tried changing this code to
> rewrite other registers such as PCI_PREF_MEMORY_BASE and that makes
> the bug come back.
> 
> >> 2. Who is responsible for saving and restoring PCI bridge
> >> configuration during suspend and resume? Linux? ACPI? BIOS?
> >
> > Not sure about PCI bridges, but at least for the PCI Express Capability
> > registers, it is in control of the OS when control is granted via the
> > ACPI _OSC method.
> 
> I guess you are referring to pci_save_pcie_state(). I can't see
> anything equivalent for the bridge registers.

Yes that would be the function, called via pci_save_state.

> > I recently compared PCI configuration space access and ACPI method
> > invocation using QEMU + VFIO with Linux 4.18, Windows 7 and Windows 10
> > (1803). There were differences like disabling MSI/interrupts before
> > suspend, setting the Enable Clock Power Management bit in PCI Express
> > Link Control and more, but applying these changes were so far not really
> > successful.
> 
> Interesting. Do you know any way that I could spy on Windows' accesses
> to the PCI bridge registers?
> Looking at at https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
> I suspect VFIO would not help me here.
> It says:
>     Note: If they are grouped with other devices in this manner, pci
> root ports and bridges should neither be bound to vfio at boot, nor be
> added to the VM.

Only non-bridge devices can be passed to a guest, but perhaps logging
access to the emulated bridge is already sufficient. The Prefetchable
Base Upper 32 Bits register is at offset 0x28.

In a trace where the Nvidia device is disabled/enabled via Device
Manager, I see writes on the enable path:

    2571@1535108904.593107:rp_write_config (ioh3420, @0x28, 0x0, len=0x4)

For Linux, I only see one write at startup, none on runtime resume.
I did not test system sleep/resume. (disable/enable is arguably a bit
different from system s/r, you may want to do additional testing here.)

Full log for WIndows 10 and Linux:
https://github.com/Lekensteyn/acpi-stuff/blob/master/d3test/XPS9560/slogs/win10-rp-enable-disable.txt#L3418
https://github.com/Lekensteyn/acpi-stuff/blob/master/d3test/XPS9560/slogs/linux-rp.txt
lspci for the emulated bridge:
https://github.com/Lekensteyn/acpi-stuff/blob/master/d3test/XPS9560/lspci-vm-vfio.txt#L359
The rp_*_config trace points are non-standard and require patches:
https://github.com/Lekensteyn/acpi-stuff/blob/master/d3test/patches/qemu-trace.diff
-- 
Kind regards,
Peter Wu
https://lekensteyn.nl