From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lekensteyn.nl ([178.21.112.251]:38361 "EHLO lekensteyn.nl" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726513AbeHXToT (ORCPT ); Fri, 24 Aug 2018 15:44:19 -0400 Date: Fri, 24 Aug 2018 17:42:12 +0200 From: Peter Wu To: Daniel Drake Cc: linux-pci@vger.kernel.org, nouveau@lists.freedesktop.org, Linux PM , Endless Linux Upstreaming Team Subject: Re: [Nouveau] Rewriting Intel PCI bridge prefetch base address bits solves nvidia graphics issues Message-ID: <20180824154212.GB8601@al> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii In-Reply-To: Sender: linux-pci-owner@vger.kernel.org List-ID: Hi Daniel, On Fri, Aug 24, 2018 at 11:31:54AM +0800, Daniel Drake wrote: > Hi, > > We are facing a suspend/resume problem with many different Asus laptop > models (30+ products) with Intel chipsets (multiple generations) and > nvidia GPUs (several different ones). Reproducers include: Are these systems also affected through runtime power management? For example: modprobe nouveau # should enable runtime PM sleep 6 # wait for runtime suspend to kick in lspci -s1: # runtime resume by reading PCI config space On laptops from about 2015-2016 with a GTX 9xxM this sequence results in hangs on various laptops (https://bugzilla.kernel.org/show_bug.cgi?id=156341). I wonder if you are experiencing the same issue. Do you have a list of affected models, an acpidump, the output of "lspci -nnvvvxxxx" and the corresponding BIOS version (e.g. from /sys/class/dmi/id/)? > After a lot of experimentation I found a workaround: during resume, > set the value of PCI_PREF_BASE_UPPER32 to 0 on the parent PCI bridge. > Easily done in drivers/pci/quirks.c. Now all nvidia stuff works fine. I am curious, how did you discover this? While this could work, perhaps there are alternative workarounds/fixes? When you say "parent PCI" bridge, is that actually the device you see in "lspci -tv"? On a Dell XPS 9560, the GPU is under a different device: -[0000:00]-+-00.0 Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers +-01.0-[01]----00.0 NVIDIA Corporation GP107M [GeForce GTX 1050 Mobile] 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05) Under 00:1c.0, there is a wireless adapter. > As an example of an affected product, take the Asus X542UQ (Intel > KabyLake i7-7500U with Nvidia GeForce 940MX). The PCI bridge is: > > 00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI > Express Root Port [8086:9d10] (rev f1) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0, IRQ 120 > Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 > I/O behind bridge: 0000e000-0000efff > Memory behind bridge: ee000000-ef0fffff > Prefetchable memory behind bridge: 00000000d0000000-00000000e1ffffff > Capabilities: [40] Express Root Port (Slot+), MSI 00 > Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit- > Capabilities: [90] Subsystem: ASUSTeK Computer Inc. Sunrise > Point-LP PCI Express Root Port [1043:1a00] > Capabilities: [a0] Power Management version 3 > Capabilities: [100] Advanced Error Reporting > Capabilities: [140] Access Control Services > Capabilities: [200] L1 PM Substates > Capabilities: [220] #19 > Kernel driver in use: pcieport > > The really weird thing here is that the workaround register > PCI_PREF_BASE_UPPER32 already appears to have value 0, as shown above > and also verified during resume. But simply writing value 0 again > definitely results in all the problems going away. > > 1. Is the Intel PCI bridge misbehaving here? Why does writing the same > value of PCI_PREF_BASE_UPPER32 make any difference at all? At what point in the suspend code path did you insert this write? It is possible that the write somehow acted as a fence/memory barrier? > 2. Who is responsible for saving and restoring PCI bridge > configuration during suspend and resume? Linux? ACPI? BIOS? Not sure about PCI bridges, but at least for the PCI Express Capability registers, it is in control of the OS when control is granted via the ACPI _OSC method. > I could not see any Linux code to save and restore these registers. > Likewise I didn't find anything in the ACPI DSDT/SSDT - neither on the > affected products, nor on a similar product that does not suffer this > nvidia issue. Linux does put the PCI bridge into D3 power state during > suspend, and upon resume the lower 32 bits of the prefetch address are > still set to the same value, so through some means this info is not > being lost. > > > 3. Any other suggestions, hints or experiments I could do to help move > forward on this issue? > > My goal is to add a workaround to Linux (perhaps as a pci quirk) for > existing devices, but also we are in conversation with Asus engineers > and if we can come up with a concrete diagnosis, we should be able to > have them fix this at the BIOS level in future products. As Windows is probably not affected by this issue, a change must be possible to make Linux more compatible with Windows. Though I am not sure what change is needed. I recently compared PCI configuration space access and ACPI method invocation using QEMU + VFIO with Linux 4.18, Windows 7 and Windows 10 (1803). There were differences like disabling MSI/interrupts before suspend, setting the Enable Clock Power Management bit in PCI Express Link Control and more, but applying these changes were so far not really successful. Some supporting files for that investigation are here: https://github.com/Lekensteyn/acpi-stuff/tree/master/d3test Karol noticed that by not setting the State in PMCSR to D3 for the Nvidia GPU during runtime suspend, then the device would successfully resume. However, based on traces using VFIO-PCI, it does not seem a good solution as Windows does not behave like that. -- Kind regards, Peter Wu https://lekensteyn.nl From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter Wu Subject: Re: Rewriting Intel PCI bridge prefetch base address bits solves nvidia graphics issues Date: Fri, 24 Aug 2018 17:42:12 +0200 Message-ID: <20180824154212.GB8601@al> References: Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: nouveau-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org Sender: "Nouveau" To: Daniel Drake Cc: linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Linux PM , Endless Linux Upstreaming Team , nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org List-Id: linux-pm@vger.kernel.org SGkgRGFuaWVsLAoKT24gRnJpLCBBdWcgMjQsIDIwMTggYXQgMTE6MzE6NTRBTSArMDgwMCwgRGFu aWVsIERyYWtlIHdyb3RlOgo+IEhpLAo+IAo+IFdlIGFyZSBmYWNpbmcgYSBzdXNwZW5kL3Jlc3Vt ZSBwcm9ibGVtIHdpdGggbWFueSBkaWZmZXJlbnQgQXN1cyBsYXB0b3AKPiBtb2RlbHMgKDMwKyBw cm9kdWN0cykgd2l0aCBJbnRlbCBjaGlwc2V0cyAobXVsdGlwbGUgZ2VuZXJhdGlvbnMpIGFuZAo+ IG52aWRpYSBHUFVzIChzZXZlcmFsIGRpZmZlcmVudCBvbmVzKS4gUmVwcm9kdWNlcnMgaW5jbHVk ZToKCkFyZSB0aGVzZSBzeXN0ZW1zIGFsc28gYWZmZWN0ZWQgdGhyb3VnaCBydW50aW1lIHBvd2Vy IG1hbmFnZW1lbnQ/IEZvcgpleGFtcGxlOgoKICAgIG1vZHByb2JlIG5vdXZlYXUgICAgIyBzaG91 bGQgZW5hYmxlIHJ1bnRpbWUgUE0KICAgIHNsZWVwIDYgICAgICAgICAgICAgIyB3YWl0IGZvciBy dW50aW1lIHN1c3BlbmQgdG8ga2ljayBpbgogICAgbHNwY2kgLXMxOiAgICAgICAgICAjIHJ1bnRp bWUgcmVzdW1lIGJ5IHJlYWRpbmcgUENJIGNvbmZpZyBzcGFjZQoKT24gbGFwdG9wcyBmcm9tIGFi b3V0IDIwMTUtMjAxNiB3aXRoIGEgR1RYIDl4eE0gdGhpcyBzZXF1ZW5jZSByZXN1bHRzIGluCmhh bmdzIG9uIHZhcmlvdXMgbGFwdG9wcwooaHR0cHM6Ly9idWd6aWxsYS5rZXJuZWwub3JnL3Nob3df YnVnLmNnaT9pZD0xNTYzNDEpLgoKSSB3b25kZXIgaWYgeW91IGFyZSBleHBlcmllbmNpbmcgdGhl IHNhbWUgaXNzdWUuIERvIHlvdSBoYXZlIGEgbGlzdCBvZgphZmZlY3RlZCBtb2RlbHMsIGFuIGFj cGlkdW1wLCB0aGUgb3V0cHV0IG9mICJsc3BjaSAtbm52dnZ4eHh4IiBhbmQgdGhlCmNvcnJlc3Bv bmRpbmcgQklPUyB2ZXJzaW9uIChlLmcuIGZyb20gL3N5cy9jbGFzcy9kbWkvaWQvKT8KCj4gQWZ0 ZXIgYSBsb3Qgb2YgZXhwZXJpbWVudGF0aW9uIEkgZm91bmQgYSB3b3JrYXJvdW5kOiBkdXJpbmcg cmVzdW1lLAo+IHNldCB0aGUgdmFsdWUgb2YgUENJX1BSRUZfQkFTRV9VUFBFUjMyIHRvIDAgb24g dGhlIHBhcmVudCBQQ0kgYnJpZGdlLgo+IEVhc2lseSBkb25lIGluIGRyaXZlcnMvcGNpL3F1aXJr cy5jLiBOb3cgYWxsIG52aWRpYSBzdHVmZiB3b3JrcyBmaW5lLgoKSSBhbSBjdXJpb3VzLCBob3cg ZGlkIHlvdSBkaXNjb3ZlciB0aGlzPyBXaGlsZSB0aGlzIGNvdWxkIHdvcmssIHBlcmhhcHMKdGhl cmUgYXJlIGFsdGVybmF0aXZlIHdvcmthcm91bmRzL2ZpeGVzPwoKV2hlbiB5b3Ugc2F5ICJwYXJl bnQgUENJIiBicmlkZ2UsIGlzIHRoYXQgYWN0dWFsbHkgdGhlIGRldmljZSB5b3Ugc2VlIGluCiJs c3BjaSAtdHYiPyBPbiBhIERlbGwgWFBTIDk1NjAsIHRoZSBHUFUgaXMgdW5kZXIgYSBkaWZmZXJl bnQgZGV2aWNlOgoKICAtWzAwMDA6MDBdLSstMDAuMCAgSW50ZWwgQ29ycG9yYXRpb24gWGVvbiBF My0xMjAwIHY2Lzd0aCBHZW4gQ29yZSBQcm9jZXNzb3IgSG9zdCBCcmlkZ2UvRFJBTSBSZWdpc3Rl cnMKICAgICAgICAgICAgICstMDEuMC1bMDFdLS0tLTAwLjAgIE5WSURJQSBDb3Jwb3JhdGlvbiBH UDEwN00gW0dlRm9yY2UgR1RYIDEwNTAgTW9iaWxlXQoKIDAwOjAxLjAgUENJIGJyaWRnZSBbMDYw NF06IEludGVsIENvcnBvcmF0aW9uIFhlb24gRTMtMTIwMCB2NS9FMy0xNTAwIHY1LzZ0aCBHZW4g Q29yZSBQcm9jZXNzb3IgUENJZSBDb250cm9sbGVyICh4MTYpIFs4MDg2OjE5MDFdIChyZXYgMDUp CgpVbmRlciAwMDoxYy4wLCB0aGVyZSBpcyBhIHdpcmVsZXNzIGFkYXB0ZXIuCgo+IEFzIGFuIGV4 YW1wbGUgb2YgYW4gYWZmZWN0ZWQgcHJvZHVjdCwgdGFrZSB0aGUgQXN1cyBYNTQyVVEgKEludGVs Cj4gS2FieUxha2UgaTctNzUwMFUgd2l0aCBOdmlkaWEgR2VGb3JjZSA5NDBNWCkuIFRoZSBQQ0kg YnJpZGdlIGlzOgo+IAo+IDAwOjFjLjAgUENJIGJyaWRnZSBbMDYwNF06IEludGVsIENvcnBvcmF0 aW9uIFN1bnJpc2UgUG9pbnQtTFAgUENJCj4gRXhwcmVzcyBSb290IFBvcnQgWzgwODY6OWQxMF0g KHJldiBmMSkgKHByb2ctaWYgMDAgW05vcm1hbCBkZWNvZGVdKQo+ICAgICBGbGFnczogYnVzIG1h c3RlciwgZmFzdCBkZXZzZWwsIGxhdGVuY3kgMCwgSVJRIDEyMAo+ICAgICBCdXM6IHByaW1hcnk9 MDAsIHNlY29uZGFyeT0wMSwgc3Vib3JkaW5hdGU9MDEsIHNlYy1sYXRlbmN5PTAKPiAgICAgSS9P IGJlaGluZCBicmlkZ2U6IDAwMDBlMDAwLTAwMDBlZmZmCj4gICAgIE1lbW9yeSBiZWhpbmQgYnJp ZGdlOiBlZTAwMDAwMC1lZjBmZmZmZgo+ICAgICBQcmVmZXRjaGFibGUgbWVtb3J5IGJlaGluZCBi cmlkZ2U6IDAwMDAwMDAwZDAwMDAwMDAtMDAwMDAwMDBlMWZmZmZmZgo+ICAgICBDYXBhYmlsaXRp ZXM6IFs0MF0gRXhwcmVzcyBSb290IFBvcnQgKFNsb3QrKSwgTVNJIDAwCj4gICAgIENhcGFiaWxp dGllczogWzgwXSBNU0k6IEVuYWJsZSsgQ291bnQ9MS8xIE1hc2thYmxlLSA2NGJpdC0KPiAgICAg Q2FwYWJpbGl0aWVzOiBbOTBdIFN1YnN5c3RlbTogQVNVU1RlSyBDb21wdXRlciBJbmMuIFN1bnJp c2UKPiBQb2ludC1MUCBQQ0kgRXhwcmVzcyBSb290IFBvcnQgWzEwNDM6MWEwMF0KPiAgICAgQ2Fw YWJpbGl0aWVzOiBbYTBdIFBvd2VyIE1hbmFnZW1lbnQgdmVyc2lvbiAzCj4gICAgIENhcGFiaWxp dGllczogWzEwMF0gQWR2YW5jZWQgRXJyb3IgUmVwb3J0aW5nCj4gICAgIENhcGFiaWxpdGllczog WzE0MF0gQWNjZXNzIENvbnRyb2wgU2VydmljZXMKPiAgICAgQ2FwYWJpbGl0aWVzOiBbMjAwXSBM MSBQTSBTdWJzdGF0ZXMKPiAgICAgQ2FwYWJpbGl0aWVzOiBbMjIwXSAjMTkKPiAgICAgS2VybmVs IGRyaXZlciBpbiB1c2U6IHBjaWVwb3J0Cj4gCj4gVGhlIHJlYWxseSB3ZWlyZCB0aGluZyBoZXJl IGlzIHRoYXQgdGhlIHdvcmthcm91bmQgcmVnaXN0ZXIKPiBQQ0lfUFJFRl9CQVNFX1VQUEVSMzIg YWxyZWFkeSBhcHBlYXJzIHRvIGhhdmUgdmFsdWUgMCwgYXMgc2hvd24gYWJvdmUKPiBhbmQgYWxz byB2ZXJpZmllZCBkdXJpbmcgcmVzdW1lLiBCdXQgc2ltcGx5IHdyaXRpbmcgdmFsdWUgMCBhZ2Fp bgo+IGRlZmluaXRlbHkgcmVzdWx0cyBpbiBhbGwgdGhlIHByb2JsZW1zIGdvaW5nIGF3YXkuCj4g Cj4gMS4gSXMgdGhlIEludGVsIFBDSSBicmlkZ2UgbWlzYmVoYXZpbmcgaGVyZT8gV2h5IGRvZXMg d3JpdGluZyB0aGUgc2FtZQo+IHZhbHVlIG9mIFBDSV9QUkVGX0JBU0VfVVBQRVIzMiBtYWtlIGFu eSBkaWZmZXJlbmNlIGF0IGFsbD8KCkF0IHdoYXQgcG9pbnQgaW4gdGhlIHN1c3BlbmQgY29kZSBw YXRoIGRpZCB5b3UgaW5zZXJ0IHRoaXMgd3JpdGU/IEl0IGlzCnBvc3NpYmxlIHRoYXQgdGhlIHdy aXRlIHNvbWVob3cgYWN0ZWQgYXMgYSBmZW5jZS9tZW1vcnkgYmFycmllcj8KCj4gMi4gV2hvIGlz IHJlc3BvbnNpYmxlIGZvciBzYXZpbmcgYW5kIHJlc3RvcmluZyBQQ0kgYnJpZGdlCj4gY29uZmln dXJhdGlvbiBkdXJpbmcgc3VzcGVuZCBhbmQgcmVzdW1lPyBMaW51eD8gQUNQST8gQklPUz8KCk5v dCBzdXJlIGFib3V0IFBDSSBicmlkZ2VzLCBidXQgYXQgbGVhc3QgZm9yIHRoZSBQQ0kgRXhwcmVz cyBDYXBhYmlsaXR5CnJlZ2lzdGVycywgaXQgaXMgaW4gY29udHJvbCBvZiB0aGUgT1Mgd2hlbiBj b250cm9sIGlzIGdyYW50ZWQgdmlhIHRoZQpBQ1BJIF9PU0MgbWV0aG9kLgoKPiBJIGNvdWxkIG5v dCBzZWUgYW55IExpbnV4IGNvZGUgdG8gc2F2ZSBhbmQgcmVzdG9yZSB0aGVzZSByZWdpc3RlcnMu Cj4gTGlrZXdpc2UgSSBkaWRuJ3QgZmluZCBhbnl0aGluZyBpbiB0aGUgQUNQSSBEU0RUL1NTRFQg LSBuZWl0aGVyIG9uIHRoZQo+IGFmZmVjdGVkIHByb2R1Y3RzLCBub3Igb24gYSBzaW1pbGFyIHBy b2R1Y3QgdGhhdCBkb2VzIG5vdCBzdWZmZXIgdGhpcwo+IG52aWRpYSBpc3N1ZS4gTGludXggZG9l cyBwdXQgdGhlIFBDSSBicmlkZ2UgaW50byBEMyBwb3dlciBzdGF0ZSBkdXJpbmcKPiBzdXNwZW5k LCBhbmQgdXBvbiByZXN1bWUgdGhlIGxvd2VyIDMyIGJpdHMgb2YgdGhlIHByZWZldGNoIGFkZHJl c3MgYXJlCj4gc3RpbGwgc2V0IHRvIHRoZSBzYW1lIHZhbHVlLCBzbyB0aHJvdWdoIHNvbWUgbWVh bnMgdGhpcyBpbmZvIGlzIG5vdAo+IGJlaW5nIGxvc3QuCj4gCj4gCj4gMy4gQW55IG90aGVyIHN1 Z2dlc3Rpb25zLCBoaW50cyBvciBleHBlcmltZW50cyBJIGNvdWxkIGRvIHRvIGhlbHAgbW92ZQo+ IGZvcndhcmQgb24gdGhpcyBpc3N1ZT8KPiAKPiBNeSBnb2FsIGlzIHRvIGFkZCBhIHdvcmthcm91 bmQgdG8gTGludXggKHBlcmhhcHMgYXMgYSBwY2kgcXVpcmspIGZvcgo+IGV4aXN0aW5nIGRldmlj ZXMsIGJ1dCBhbHNvIHdlIGFyZSBpbiBjb252ZXJzYXRpb24gd2l0aCBBc3VzIGVuZ2luZWVycwo+ IGFuZCBpZiB3ZSBjYW4gY29tZSB1cCB3aXRoIGEgY29uY3JldGUgZGlhZ25vc2lzLCB3ZSBzaG91 bGQgYmUgYWJsZSB0bwo+IGhhdmUgdGhlbSBmaXggdGhpcyBhdCB0aGUgQklPUyBsZXZlbCBpbiBm dXR1cmUgcHJvZHVjdHMuCgpBcyBXaW5kb3dzIGlzIHByb2JhYmx5IG5vdCBhZmZlY3RlZCBieSB0 aGlzIGlzc3VlLCBhIGNoYW5nZSBtdXN0IGJlCnBvc3NpYmxlIHRvIG1ha2UgTGludXggbW9yZSBj b21wYXRpYmxlIHdpdGggV2luZG93cy4gVGhvdWdoIEkgYW0gbm90CnN1cmUgd2hhdCBjaGFuZ2Ug aXMgbmVlZGVkLgoKSSByZWNlbnRseSBjb21wYXJlZCBQQ0kgY29uZmlndXJhdGlvbiBzcGFjZSBh Y2Nlc3MgYW5kIEFDUEkgbWV0aG9kCmludm9jYXRpb24gdXNpbmcgUUVNVSArIFZGSU8gd2l0aCBM aW51eCA0LjE4LCBXaW5kb3dzIDcgYW5kIFdpbmRvd3MgMTAKKDE4MDMpLiBUaGVyZSB3ZXJlIGRp ZmZlcmVuY2VzIGxpa2UgZGlzYWJsaW5nIE1TSS9pbnRlcnJ1cHRzIGJlZm9yZQpzdXNwZW5kLCBz ZXR0aW5nIHRoZSBFbmFibGUgQ2xvY2sgUG93ZXIgTWFuYWdlbWVudCBiaXQgaW4gUENJIEV4cHJl c3MKTGluayBDb250cm9sIGFuZCBtb3JlLCBidXQgYXBwbHlpbmcgdGhlc2UgY2hhbmdlcyB3ZXJl IHNvIGZhciBub3QgcmVhbGx5CnN1Y2Nlc3NmdWwuCgpTb21lIHN1cHBvcnRpbmcgZmlsZXMgZm9y IHRoYXQgaW52ZXN0aWdhdGlvbiBhcmUgaGVyZToKaHR0cHM6Ly9naXRodWIuY29tL0xla2Vuc3Rl eW4vYWNwaS1zdHVmZi90cmVlL21hc3Rlci9kM3Rlc3QKCkthcm9sIG5vdGljZWQgdGhhdCBieSBu b3Qgc2V0dGluZyB0aGUgU3RhdGUgaW4gUE1DU1IgdG8gRDMgZm9yIHRoZQpOdmlkaWEgR1BVIGR1 cmluZyBydW50aW1lIHN1c3BlbmQsIHRoZW4gdGhlIGRldmljZSB3b3VsZCBzdWNjZXNzZnVsbHkK cmVzdW1lLiBIb3dldmVyLCBiYXNlZCBvbiB0cmFjZXMgdXNpbmcgVkZJTy1QQ0ksIGl0IGRvZXMg bm90IHNlZW0gYSBnb29kCnNvbHV0aW9uIGFzIFdpbmRvd3MgZG9lcyBub3QgYmVoYXZlIGxpa2Ug dGhhdC4KLS0gCktpbmQgcmVnYXJkcywKUGV0ZXIgV3UKaHR0cHM6Ly9sZWtlbnN0ZXluLm5sCl9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCk5vdXZlYXUgbWFp bGluZyBsaXN0Ck5vdXZlYXVAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8vbGlzdHMuZnJl ZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vbm91dmVhdQo=