[PATCH v6 0/1] Fix suspend issues with AMD root ports

[PATCH v6 0/1] Fix suspend issues with AMD root ports

[Mario Limonciello]

Problems have been reported on AMD laptops with suspend/resume
where particular root ports are put into D3 and then the system is unable
to resume properly.

The issue boils down to the currently selected kernel policy for root port
behavior at suspend time:
0) If the machine is from 2015 or later
1) If a PCIe root port is power manageable by the platform then platform
   will be used to determine the power state of the root port at suspend.
2) If the PCIe root is not power manageable by the platform then the kernel
   will check if it was configured to wakeup. 
3) If it was, then it will be put into the deepest state that supports
   wakeup from PME.
4) If it wasn't, then it will be put into D3hot.

Earlier more general attempts [1] to fix this have been made in the past to
change the policy so that only roots power manageable by the platform would
be put into the appropriate power state advertised by the platform
firmware.

It was hyptothesized[2] that this is a general issue that affects non-AMD
hardware as well, so I spent time to try to match it against the PCI or
ACPI specs but I can't find anything that says how the OSPM /should behave/
in this case.  It seems that this is up to OSPM to decide.

A concern was also raised how this would affect other existing hardware
that may not have root ports that are power manageable by the platform [3].
I suspect that they *should* be fine as Intel reviewed and acked the
original patch, but I have no way to prove this.

This patch is a more targeted approach to workaround the issue only for the
affected AMD hardware instead of generally changing the policy.

Link: https://lore.kernel.org/linux-pci/20230530163947.230418-2-mario.limonciello@amd.com/ [1]
Link: https://lore.kernel.org/linux-pci/ZHp45fNGuPUWPnO7@bhelgaas/ [2]
Link: https://lore.kernel.org/linux-pci/20230607080154.GA7545@wunner.de/ [3]

Mario Limonciello (1):
  PCI: Avoid putting some root ports into D3 on some Ryzen chips

 drivers/pci/quirks.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

-- 
2.34.1

[PATCH v6 1/1] PCI: Avoid putting some root ports into D3 on some Ryzen chips

[Mario Limonciello]

commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
sets the policy that all PCIe ports are allowed to use D3.  When
the system is suspended if the port is not power manageable by the
platform and won't be used for wakeup via a PME this sets up the
policy for these ports to go into D3hot.

This policy generally makes sense from an OSPM perspective but it leads
to problems with wakeup from suspend on laptops with AMD chips:

- On family 19h model 44h (PCI 0x14b9) this manifests as a missing wakeup
  interrupt.
- On family 19h model 74h (PCI 0x14eb) this manifests as a system hang.

Add a quirk for the PCI device ID used by the problematic root port on
both chips to ensure that these root ports are not put into D3hot at
suspend.

Cc: stable@vger.kernel.org # 6.1+
Reported-by: Iain Lane <iain@orangesquash.org.uk>
Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121
Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
---
 drivers/pci/quirks.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 321156ca273d5..e0346073e5855 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -3867,6 +3867,22 @@ static void quirk_apple_poweroff_thunderbolt(struct pci_dev *dev)
 DECLARE_PCI_FIXUP_SUSPEND_LATE(PCI_VENDOR_ID_INTEL,
 			       PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C,
 			       quirk_apple_poweroff_thunderbolt);
+
+/*
+ * Putting PCIe root ports on Ryzen SoCs with USB4 controllers into D3hot
+ * may cause problems when the system attempts wake up from s2idle.
+ *
+ * On family 19h model 44h (PCI 0x14b9) this manifests as a missing wakeup
+ * interrupt.
+ * On family 19h model 74h (PCI 0x14eb) this manifests as a system hang.
+ */
+static void quirk_ryzen_rp_d3(struct pci_dev *pdev)
+{
+	if (!acpi_pci_power_manageable(pdev))
+		pdev->dev_flags |= PCI_DEV_FLAGS_NO_D3;
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14b9, quirk_ryzen_rp_d3);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14eb, quirk_ryzen_rp_d3);
 #endif
 
 /*
-- 
2.34.1

Re: [PATCH v6 1/1] PCI: Avoid putting some root ports into D3 on some Ryzen chips

[Bjorn Helgaas]

On Sat, Jul 08, 2023 at 04:44:57PM -0500, Mario Limonciello wrote:
> commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
> sets the policy that all PCIe ports are allowed to use D3.  When
> the system is suspended if the port is not power manageable by the
> platform and won't be used for wakeup via a PME this sets up the
> policy for these ports to go into D3hot.
> 
> This policy generally makes sense from an OSPM perspective but it leads
> to problems with wakeup from suspend on laptops with AMD chips:
> 
> - On family 19h model 44h (PCI 0x14b9) this manifests as a missing wakeup
>   interrupt.
> - On family 19h model 74h (PCI 0x14eb) this manifests as a system hang.
> 
> Add a quirk for the PCI device ID used by the problematic root port on
> both chips to ensure that these root ports are not put into D3hot at
> suspend.

What is problematic about these root ports?  Is this a hardware
erratum?  Some corner of the ACPI spec that allows undefined behavior?

Does AMD have any guidance about generic ways to use D3, or does AMD
expect to add quirks piecemeal as problems are discovered?  How does
Windows handle all this?

Adding quirks as we discover random devices that don't behave
correctly for reasons unknown is not very sustainable.

Bjorn

> Cc: stable@vger.kernel.org # 6.1+
> Reported-by: Iain Lane <iain@orangesquash.org.uk>
> Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121
> Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
> ---
>  drivers/pci/quirks.c | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> index 321156ca273d5..e0346073e5855 100644
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -3867,6 +3867,22 @@ static void quirk_apple_poweroff_thunderbolt(struct pci_dev *dev)
>  DECLARE_PCI_FIXUP_SUSPEND_LATE(PCI_VENDOR_ID_INTEL,
>  			       PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C,
>  			       quirk_apple_poweroff_thunderbolt);
> +
> +/*
> + * Putting PCIe root ports on Ryzen SoCs with USB4 controllers into D3hot
> + * may cause problems when the system attempts wake up from s2idle.
> + *
> + * On family 19h model 44h (PCI 0x14b9) this manifests as a missing wakeup
> + * interrupt.
> + * On family 19h model 74h (PCI 0x14eb) this manifests as a system hang.
> + */
> +static void quirk_ryzen_rp_d3(struct pci_dev *pdev)
> +{
> +	if (!acpi_pci_power_manageable(pdev))
> +		pdev->dev_flags |= PCI_DEV_FLAGS_NO_D3;
> +}
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14b9, quirk_ryzen_rp_d3);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14eb, quirk_ryzen_rp_d3);
>  #endif
>  
>  /*
> -- 
> 2.34.1
> 

Re: [PATCH v6 1/1] PCI: Avoid putting some root ports into D3 on some Ryzen chips

[Limonciello, Mario]

On 7/10/2023 14:32, Bjorn Helgaas wrote:
> On Sat, Jul 08, 2023 at 04:44:57PM -0500, Mario Limonciello wrote:
>> commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
>> sets the policy that all PCIe ports are allowed to use D3.  When
>> the system is suspended if the port is not power manageable by the
>> platform and won't be used for wakeup via a PME this sets up the
>> policy for these ports to go into D3hot.
>>
>> This policy generally makes sense from an OSPM perspective but it leads
>> to problems with wakeup from suspend on laptops with AMD chips:
>>
>> - On family 19h model 44h (PCI 0x14b9) this manifests as a missing wakeup
>>    interrupt.
>> - On family 19h model 74h (PCI 0x14eb) this manifests as a system hang.
>>
>> Add a quirk for the PCI device ID used by the problematic root port on
>> both chips to ensure that these root ports are not put into D3hot at
>> suspend.
> 
> What is problematic about these root ports?  Is this a hardware
> erratum?  

I mentioned some of this in earlier versions; but the problem is deeper
in the platform firmware and only happens when the root ports are in 
D3hot across an s2idle cycle.

When looked at independently they can be moved in and out of D3hot no 
problem.

 From the perspective of hardware designers they say this is a software 
bug that Linux puts PCI RP into D3hot over Modern Standby when they 
don't specify for this to be done.  I don't expect any erratum for it.

> Some corner of the ACPI spec that allows undefined behavior?
> 

These root ports are not reported as power manageable by ACPI.

As I mentioned in the cover letter how the OSPM handles this case is 
outside of any spec AFAICT.

> Does AMD have any guidance about generic ways to use D3, or does AMD
> expect to add quirks piecemeal as problems are discovered?  How does
> Windows handle all this?
> 

Windows doesn't put root ports into D3hot over suspend unless they are 
power managed by ACPI (as described in section 7.3 of the ACPI spec).

https://uefi.org/htmlspecs/ACPI_Spec_6_4_html/07_Power_and_Performance_Mgmt/device-power-management-objects.html

So on Windows these ports are all in D0 and none of the issues happen.

Linux if PCI devices aren't power managed by ACPI will either follow 
deepest state it can wake from PME or fall back to D3hot.

> Adding quirks as we discover random devices that don't behave
> correctly for reasons unknown is not very sustainable.

I don't disagree but in v5 I tried to align this to the Windows behavior 
and it wasn't accepted.

If you still think v5 is the better approach I'm fine to try to do that 
again and rewrite the commit message.

> 
> Bjorn
> 
>> Cc: stable@vger.kernel.org # 6.1+
>> Reported-by: Iain Lane <iain@orangesquash.org.uk>
>> Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121
>> Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
>> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
>> ---
>>   drivers/pci/quirks.c | 16 ++++++++++++++++
>>   1 file changed, 16 insertions(+)
>>
>> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
>> index 321156ca273d5..e0346073e5855 100644
>> --- a/drivers/pci/quirks.c
>> +++ b/drivers/pci/quirks.c
>> @@ -3867,6 +3867,22 @@ static void quirk_apple_poweroff_thunderbolt(struct pci_dev *dev)
>>   DECLARE_PCI_FIXUP_SUSPEND_LATE(PCI_VENDOR_ID_INTEL,
>>   			       PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C,
>>   			       quirk_apple_poweroff_thunderbolt);
>> +
>> +/*
>> + * Putting PCIe root ports on Ryzen SoCs with USB4 controllers into D3hot
>> + * may cause problems when the system attempts wake up from s2idle.
>> + *
>> + * On family 19h model 44h (PCI 0x14b9) this manifests as a missing wakeup
>> + * interrupt.
>> + * On family 19h model 74h (PCI 0x14eb) this manifests as a system hang.
>> + */
>> +static void quirk_ryzen_rp_d3(struct pci_dev *pdev)
>> +{
>> +	if (!acpi_pci_power_manageable(pdev))
>> +		pdev->dev_flags |= PCI_DEV_FLAGS_NO_D3;
>> +}
>> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14b9, quirk_ryzen_rp_d3);
>> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14eb, quirk_ryzen_rp_d3);
>>   #endif
>>   
>>   /*
>> -- 
>> 2.34.1
>>

Re: [PATCH v6 1/1] PCI: Avoid putting some root ports into D3 on some Ryzen chips

[Bjorn Helgaas]

[+cc Mika, author of 9d26d3a8f1b0]

On Mon, Jul 10, 2023 at 02:44:23PM -0500, Limonciello, Mario wrote:
> On 7/10/2023 14:32, Bjorn Helgaas wrote:
> > On Sat, Jul 08, 2023 at 04:44:57PM -0500, Mario Limonciello wrote:
> > > commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
> > > sets the policy that all PCIe ports are allowed to use D3.  When
> > > the system is suspended if the port is not power manageable by the
> > > platform and won't be used for wakeup via a PME this sets up the
> > > policy for these ports to go into D3hot.
> > > 
> > > This policy generally makes sense from an OSPM perspective but it leads
> > > to problems with wakeup from suspend on laptops with AMD chips:
> > > 
> > > - On family 19h model 44h (PCI 0x14b9) this manifests as a missing wakeup
> > >    interrupt.
> > > - On family 19h model 74h (PCI 0x14eb) this manifests as a system hang.
> > > 
> > > Add a quirk for the PCI device ID used by the problematic root port on
> > > both chips to ensure that these root ports are not put into D3hot at
> > > suspend.
> > 
> > What is problematic about these root ports?  Is this a hardware
> > erratum?
> 
> I mentioned some of this in earlier versions; but the problem is deeper
> in the platform firmware and only happens when the root ports are in D3hot
> across an s2idle cycle.
> 
> When looked at independently they can be moved in and out of D3hot no
> problem.
> 
> From the perspective of hardware designers they say this is a software bug
> that Linux puts PCI RP into D3hot over Modern Standby when they don't
> specify for this to be done.  I don't expect any erratum for it.

It sounds like there's someplace the hardware designers specify how
this should work?  Where is that?  "Modern Standby" doesn't mean
anything to me, but maybe there's some spec that covers it?

> > Some corner of the ACPI spec that allows undefined behavior?
> 
> These root ports are not reported as power manageable by ACPI.
> 
> As I mentioned in the cover letter how the OSPM handles this case is outside
> of any spec AFAICT.

If it's truly outside the spec, then I don't think Linux should use D3
until we have some standardized way to tell whether it's safe.

> > Does AMD have any guidance about generic ways to use D3, or does AMD
> > expect to add quirks piecemeal as problems are discovered?  How does
> > Windows handle all this?
> 
> Windows doesn't put root ports into D3hot over suspend unless they are power
> managed by ACPI (as described in section 7.3 of the ACPI spec).
> 
> https://uefi.org/htmlspecs/ACPI_Spec_6_4_html/07_Power_and_Performance_Mgmt/device-power-management-objects.html
> 
> So on Windows these ports are all in D0 and none of the issues happen.

Maybe this is the clue we need.  My eyes glaze over when reading that
section, but if we can come up with a commit log that starts with a
sentence from that section and connects the dots all the way to the
platform_pci_power_manageable() implementation, that might be a good
argument that 9d26d3a8f1b0 was a little too aggressive.

> Linux if PCI devices aren't power managed by ACPI will either follow deepest
> state it can wake from PME or fall back to D3hot.
> 
> > Adding quirks as we discover random devices that don't behave
> > correctly for reasons unknown is not very sustainable.
> 
> I don't disagree but in v5 I tried to align this to the Windows behavior and
> it wasn't accepted.

I like the fact that v5 ([1] for anybody following along at home) is
very generic:

  @@ bool pci_bridge_d3_possible(struct pci_dev *bridge)
  ...
  +       if (pci_pcie_type(bridge) == PCI_EXP_TYPE_ROOT_PORT &&
  +           !platform_pci_power_manageable(bridge))
  +               return false;

My objection was that we didn't have a clear connection to any specs,
so even though the code is perfectly generic, the commit log mentioned
specific cases (USB keyboard/mouse on xHCI device connected to USB-C
on AMD USB4 router).

But maybe we *could* make a convincing generic commit log.  I guess
we'd also have to explain the PCI_EXP_TYPE_ROOT_PORT part of the
patch.

Bjorn

[1] https://lore.kernel.org/r/20230530163947.230418-2-mario.limonciello@amd.com

> > > Cc: stable@vger.kernel.org # 6.1+
> > > Reported-by: Iain Lane <iain@orangesquash.org.uk>
> > > Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121
> > > Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
> > > Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
> > > ---
> > >   drivers/pci/quirks.c | 16 ++++++++++++++++
> > >   1 file changed, 16 insertions(+)
> > > 
> > > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> > > index 321156ca273d5..e0346073e5855 100644
> > > --- a/drivers/pci/quirks.c
> > > +++ b/drivers/pci/quirks.c
> > > @@ -3867,6 +3867,22 @@ static void quirk_apple_poweroff_thunderbolt(struct pci_dev *dev)
> > >   DECLARE_PCI_FIXUP_SUSPEND_LATE(PCI_VENDOR_ID_INTEL,
> > >   			       PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C,
> > >   			       quirk_apple_poweroff_thunderbolt);
> > > +
> > > +/*
> > > + * Putting PCIe root ports on Ryzen SoCs with USB4 controllers into D3hot
> > > + * may cause problems when the system attempts wake up from s2idle.
> > > + *
> > > + * On family 19h model 44h (PCI 0x14b9) this manifests as a missing wakeup
> > > + * interrupt.
> > > + * On family 19h model 74h (PCI 0x14eb) this manifests as a system hang.
> > > + */
> > > +static void quirk_ryzen_rp_d3(struct pci_dev *pdev)
> > > +{
> > > +	if (!acpi_pci_power_manageable(pdev))
> > > +		pdev->dev_flags |= PCI_DEV_FLAGS_NO_D3;
> > > +}
> > > +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14b9, quirk_ryzen_rp_d3);
> > > +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14eb, quirk_ryzen_rp_d3);
> > >   #endif
> > >   /*
> > > -- 
> > > 2.34.1
> > > 
> 

Re: [PATCH v6 1/1] PCI: Avoid putting some root ports into D3 on some Ryzen chips

[Limonciello, Mario]

On 7/10/2023 3:33 PM, Bjorn Helgaas wrote:
>
> It sounds like there's someplace the hardware designers specify how
> this should work?  Where is that?  "Modern Standby" doesn't mean
> anything to me, but maybe there's some spec that covers it?

https://learn.microsoft.com/en-us/windows-hardware/design/device-experiences/modern-standby

It quickly devolves into Microsoft specific stuff though and
I can't find anything interesting to our specific issue.
> Maybe this is the clue we need.  My eyes glaze over when reading that
> section, but if we can come up with a commit log that starts with a
> sentence from that section and connects the dots all the way to the
> platform_pci_power_manageable() implementation, that might be a good
> argument that 9d26d3a8f1b0 was a little too aggressive.
Yeah.
> I like the fact that v5 ([1] for anybody following along at home) is
> very generic:
>
>    @@ bool pci_bridge_d3_possible(struct pci_dev *bridge)
>    ...
>    +       if (pci_pcie_type(bridge) == PCI_EXP_TYPE_ROOT_PORT &&
>    +           !platform_pci_power_manageable(bridge))
>    +               return false;
>
> My objection was that we didn't have a clear connection to any specs,
> so even though the code is perfectly generic, the commit log mentioned
> specific cases (USB keyboard/mouse on xHCI device connected to USB-C
> on AMD USB4 router).
>
> But maybe we *could* make a convincing generic commit log.  I guess
> we'd also have to explain the PCI_EXP_TYPE_ROOT_PORT part of the
> patch.
OK, I'll take a stab at rewriting the v5 commit message to be more
generic as you suggested as a v7.

We might be able to drop the PCI_EXP_TYPE_ROOT_PORT part well but I
would be more worried about regressions from this.