[PATCH v2] PCI: Don't assume root ports from > 2015 are power manageable

[PATCH v2] PCI: Don't assume root ports from > 2015 are power manageable

[Mario Limonciello]

Using an XHCI device to wakeup the system from s2idle fails when
that XHCI device is connected to a USB-C port for an AMD USB4
router.

Due to commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during
suspend") all root port go into D3 during s2idle.
When the root ports are in D3 over s2idle it's not possible for the
platform firmware to properly identify the wakeup source.

Comparing registers between Linux and Windows 11 this behavior to put root
ports into D3 at suspend is unique to Linux.  On an affected system
Windows does not put the root ports into D3 over Modern Standby.

Windows doesn't put the root ports into D3 because root ports are not
power manageable; they're missing _PRW and _S0W.

Linux shouldn't be assuming they support D3 just because they're newer
than 2015, the ports should also be deemed power manageable.
Add an extra check for this to ensure D3 isn't selected for such machines.

Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
Reported-by: Iain Lane <iain@orangesquash.org.uk>
Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
---
 drivers/pci/pci.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 5ede93222bc1..3fe27aef09e6 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -3010,6 +3010,9 @@ bool pci_bridge_d3_possible(struct pci_dev *bridge)
 		if (dmi_check_system(bridge_d3_blacklist))
 			return false;
 
+		if (!platform_pci_power_manageable(bridge))
+			return false;
+
 		/*
 		 * It should be safe to put PCIe ports from 2015 or newer
 		 * to D3.
-- 
2.34.1

Re: [PATCH v2] PCI: Don't assume root ports from > 2015 are power manageable

[Mika Westerberg]

Hi Mario,

On Wed, May 17, 2023 at 10:08:27AM -0500, Mario Limonciello wrote:
> Using an XHCI device to wakeup the system from s2idle fails when
> that XHCI device is connected to a USB-C port for an AMD USB4
> router.
> 
> Due to commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during
> suspend") all root port go into D3 during s2idle.
> When the root ports are in D3 over s2idle it's not possible for the
> platform firmware to properly identify the wakeup source.
> 
> Comparing registers between Linux and Windows 11 this behavior to put root
> ports into D3 at suspend is unique to Linux.  On an affected system
> Windows does not put the root ports into D3 over Modern Standby.
> 
> Windows doesn't put the root ports into D3 because root ports are not
> power manageable; they're missing _PRW and _S0W.
> 
> Linux shouldn't be assuming they support D3 just because they're newer
> than 2015, the ports should also be deemed power manageable.
> Add an extra check for this to ensure D3 isn't selected for such machines.
> 
> Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
> Reported-by: Iain Lane <iain@orangesquash.org.uk>
> Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121
> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
> ---
>  drivers/pci/pci.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index 5ede93222bc1..3fe27aef09e6 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -3010,6 +3010,9 @@ bool pci_bridge_d3_possible(struct pci_dev *bridge)
>  		if (dmi_check_system(bridge_d3_blacklist))
>  			return false;
>  
> +		if (!platform_pci_power_manageable(bridge))
> +			return false;
> +

We already call platform_pci_bridge_d3() few lines up. That function
should know whether "platform" supports D3 for the bridges, and I think
it actually calls acpi_device_power_manageable() that platform_pci_power_manageable()
ends up checking too.

Re: [PATCH v2] PCI: Don't assume root ports from > 2015 are power manageable

[Mario Limonciello]

On 5/22/23 00:55, Mika Westerberg wrote:
> Hi Mario,
> 
> On Wed, May 17, 2023 at 10:08:27AM -0500, Mario Limonciello wrote:
>> Using an XHCI device to wakeup the system from s2idle fails when
>> that XHCI device is connected to a USB-C port for an AMD USB4
>> router.
>>
>> Due to commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during
>> suspend") all root port go into D3 during s2idle.
>> When the root ports are in D3 over s2idle it's not possible for the
>> platform firmware to properly identify the wakeup source.
>>
>> Comparing registers between Linux and Windows 11 this behavior to put root
>> ports into D3 at suspend is unique to Linux.  On an affected system
>> Windows does not put the root ports into D3 over Modern Standby.
>>
>> Windows doesn't put the root ports into D3 because root ports are not
>> power manageable; they're missing _PRW and _S0W.
>>
>> Linux shouldn't be assuming they support D3 just because they're newer
>> than 2015, the ports should also be deemed power manageable.
>> Add an extra check for this to ensure D3 isn't selected for such machines.
>>
>> Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
>> Reported-by: Iain Lane <iain@orangesquash.org.uk>
>> Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121
>> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
>> ---
>>   drivers/pci/pci.c | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
>> index 5ede93222bc1..3fe27aef09e6 100644
>> --- a/drivers/pci/pci.c
>> +++ b/drivers/pci/pci.c
>> @@ -3010,6 +3010,9 @@ bool pci_bridge_d3_possible(struct pci_dev *bridge)
>>   		if (dmi_check_system(bridge_d3_blacklist))
>>   			return false;
>>   
>> +		if (!platform_pci_power_manageable(bridge))
>> +			return false;
>> +
> 
> We already call platform_pci_bridge_d3() few lines up. That function
> should know whether "platform" supports D3 for the bridges, and I think
> it actually calls acpi_device_power_manageable() that platform_pci_power_manageable()
> ends up checking too.

It does, but it doesn't end up returning false if it doesn't support it. 
  It only returns true if it does.

I've tested this patch on two different failing platforms and it works 
now on both.

Re: [PATCH v2] PCI: Don't assume root ports from > 2015 are power manageable

[Mika Westerberg]

On Mon, May 22, 2023 at 06:28:25AM -0500, Mario Limonciello wrote:
> On 5/22/23 00:55, Mika Westerberg wrote:
> > Hi Mario,
> > 
> > On Wed, May 17, 2023 at 10:08:27AM -0500, Mario Limonciello wrote:
> > > Using an XHCI device to wakeup the system from s2idle fails when
> > > that XHCI device is connected to a USB-C port for an AMD USB4
> > > router.
> > > 
> > > Due to commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during
> > > suspend") all root port go into D3 during s2idle.
> > > When the root ports are in D3 over s2idle it's not possible for the
> > > platform firmware to properly identify the wakeup source.
> > > 
> > > Comparing registers between Linux and Windows 11 this behavior to put root
> > > ports into D3 at suspend is unique to Linux.  On an affected system
> > > Windows does not put the root ports into D3 over Modern Standby.
> > > 
> > > Windows doesn't put the root ports into D3 because root ports are not
> > > power manageable; they're missing _PRW and _S0W.
> > > 
> > > Linux shouldn't be assuming they support D3 just because they're newer
> > > than 2015, the ports should also be deemed power manageable.
> > > Add an extra check for this to ensure D3 isn't selected for such machines.
> > > 
> > > Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
> > > Reported-by: Iain Lane <iain@orangesquash.org.uk>
> > > Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121
> > > Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
> > > ---
> > >   drivers/pci/pci.c | 3 +++
> > >   1 file changed, 3 insertions(+)
> > > 
> > > diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> > > index 5ede93222bc1..3fe27aef09e6 100644
> > > --- a/drivers/pci/pci.c
> > > +++ b/drivers/pci/pci.c
> > > @@ -3010,6 +3010,9 @@ bool pci_bridge_d3_possible(struct pci_dev *bridge)
> > >   		if (dmi_check_system(bridge_d3_blacklist))
> > >   			return false;
> > > +		if (!platform_pci_power_manageable(bridge))
> > > +			return false;
> > > +
> > 
> > We already call platform_pci_bridge_d3() few lines up. That function
> > should know whether "platform" supports D3 for the bridges, and I think
> > it actually calls acpi_device_power_manageable() that platform_pci_power_manageable()
> > ends up checking too.
> 
> It does, but it doesn't end up returning false if it doesn't support it.  It
> only returns true if it does.

Good point!

> I've tested this patch on two different failing platforms and it works now
> on both.

Okay LGTM then,

Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>

Re: [PATCH v2] PCI: Don't assume root ports from > 2015 are power manageable

[Bjorn Helgaas]

[+cc Rafael, Lukas, linux-pm]

On Wed, May 17, 2023 at 10:08:27AM -0500, Mario Limonciello wrote:
> Using an XHCI device to wakeup the system from s2idle fails when
> that XHCI device is connected to a USB-C port for an AMD USB4
> router.

Are XHCI, USB-C, and the AMD USB4 router just examples?  I assume the
same issue could happen with non-XHCI and non-AMD devices, too?

I assume the problem has something to do with PME_Support and some
device being put in a power state where it cannot generate or forward
PME messages?  I think the PCIe protocol details would be helpful
here.

> Due to commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during
> suspend") all root port go into D3 during s2idle.
> When the root ports are in D3 over s2idle it's not possible for the
> platform firmware to properly identify the wakeup source.

I'm not sure what the critical device is here.  9d26d3a8f1b0 and the
current pci_bridge_d3_possible() are not specific to Root Ports.  The
PCIe protocol details would probably clear this up.

> Comparing registers between Linux and Windows 11 this behavior to put root
> ports into D3 at suspend is unique to Linux.  On an affected system
> Windows does not put the root ports into D3 over Modern Standby.
> 
> Windows doesn't put the root ports into D3 because root ports are not
> power manageable; they're missing _PRW and _S0W.

platform_pci_power_manageable() tests adev->flags.power_manageable,
which is set by acpi_bus_get_power_flags() when a device has _PS0 or
_PR0.

So I don't know what's relevant out of _PRW, _S0W, _PS0, _PR0, but
this sentence doesn't seem to match the code.

> Linux shouldn't be assuming they support D3 just because they're newer
> than 2015, the ports should also be deemed power manageable.
> Add an extra check for this to ensure D3 isn't selected for such machines.

Is this talking about D3hot or D3cold or both?  If we can make this
explicit, it will help me out.  It's probably obvious to power
experts, but I'm not one.

> Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
> Reported-by: Iain Lane <iain@orangesquash.org.uk>
> Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121
> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
> ---
>  drivers/pci/pci.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index 5ede93222bc1..3fe27aef09e6 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -3010,6 +3010,9 @@ bool pci_bridge_d3_possible(struct pci_dev *bridge)
>  		if (dmi_check_system(bridge_d3_blacklist))
>  			return false;
>  
> +		if (!platform_pci_power_manageable(bridge))
> +			return false;
> +
>  		/*
>  		 * It should be safe to put PCIe ports from 2015 or newer
>  		 * to D3.
> -- 
> 2.34.1
> 

Re: [PATCH v2] PCI: Don't assume root ports from > 2015 are power manageable

[Limonciello, Mario]

On 5/23/2023 3:35 PM, Bjorn Helgaas wrote:
> [+cc Rafael, Lukas, linux-pm]
>
> On Wed, May 17, 2023 at 10:08:27AM -0500, Mario Limonciello wrote:
>> Using an XHCI device to wakeup the system from s2idle fails when
>> that XHCI device is connected to a USB-C port for an AMD USB4
>> router.
> Are XHCI, USB-C, and the AMD USB4 router just examples?

They're very specific to this case.  If XHCI
keyboard/mouse is connected to a type-C port that is
not connected to AMD USB4 router this issue doesn't occur.

> I assume the
> same issue could happen with non-XHCI and non-AMD devices, too?
Based on what's wrong with Linux and fixed by this patch,
yes this *can* happen to any vendor that the root port doesn't
support waking from for D3 but Linux uses it anyway.
>
> I assume the problem has something to do with PME_Support and some
> device being put in a power state where it cannot generate or forward
> PME messages?  I think the PCIe protocol details would be helpful
> here.

No; it's specific to an internal sequence in the SoC.

If the problematic root port is in D3 during s0i3 this
problematic sequence happens.  If the root port is in D0
then it doesn't.

 From discussion with others in AMD that's at least one
reason why the firmware doesn't advertise any power management
on this root port and why Linux shouldn't be using it.

>> Comparing registers between Linux and Windows 11 this behavior to put root
>> ports into D3 at suspend is unique to Linux.  On an affected system
>> Windows does not put the root ports into D3 over Modern Standby.
>>
>> Windows doesn't put the root ports into D3 because root ports are not
>> power manageable; they're missing _PRW and _S0W.
> platform_pci_power_manageable() tests adev->flags.power_manageable,
> which is set by acpi_bus_get_power_flags() when a device has _PS0 or
> _PR0.
>
> So I don't know what's relevant out of _PRW, _S0W, _PS0, _PR0, but
> this sentence doesn't seem to match the code.

The firmware doesn't have _PS0 or _PR0 either for this root
port.

>> Linux shouldn't be assuming they support D3 just because they're newer
>> than 2015, the ports should also be deemed power manageable.
>> Add an extra check for this to ensure D3 isn't selected for such machines.
> Is this talking about D3hot or D3cold or both?  If we can make this
> explicit, it will help me out.  It's probably obvious to power
> experts, but I'm not one.
Both.
>> Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
>> Reported-by: Iain Lane <iain@orangesquash.org.uk>
>> Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121
>> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
>> ---
>>   drivers/pci/pci.c | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
>> index 5ede93222bc1..3fe27aef09e6 100644
>> --- a/drivers/pci/pci.c
>> +++ b/drivers/pci/pci.c
>> @@ -3010,6 +3010,9 @@ bool pci_bridge_d3_possible(struct pci_dev *bridge)
>>   		if (dmi_check_system(bridge_d3_blacklist))
>>   			return false;
>>   
>> +		if (!platform_pci_power_manageable(bridge))
>> +			return false;
>> +
>>   		/*
>>   		 * It should be safe to put PCIe ports from 2015 or newer
>>   		 * to D3.
>> -- 
>> 2.34.1
Something that this patch makes me wonder is if the original
heuristic was actually correct.

Did the PCIe ports from "older" machine have everything needed
to let them go to D3?

Or would this change also let the heuristic be dropped?

Re: [PATCH v2] PCI: Don't assume root ports from > 2015 are power manageable

[Rafael J. Wysocki]

On Tue, May 23, 2023 at 11:30 PM Limonciello, Mario
<mario.limonciello@amd.com> wrote:
>
>
> On 5/23/2023 3:35 PM, Bjorn Helgaas wrote:
> > [+cc Rafael, Lukas, linux-pm]
> >
> > On Wed, May 17, 2023 at 10:08:27AM -0500, Mario Limonciello wrote:
> >> Using an XHCI device to wakeup the system from s2idle fails when
> >> that XHCI device is connected to a USB-C port for an AMD USB4
> >> router.
> > Are XHCI, USB-C, and the AMD USB4 router just examples?
>
> They're very specific to this case.  If XHCI
> keyboard/mouse is connected to a type-C port that is
> not connected to AMD USB4 router this issue doesn't occur.
>
> > I assume the
> > same issue could happen with non-XHCI and non-AMD devices, too?
> Based on what's wrong with Linux and fixed by this patch,
> yes this *can* happen to any vendor that the root port doesn't
> support waking from for D3 but Linux uses it anyway.
> >
> > I assume the problem has something to do with PME_Support and some
> > device being put in a power state where it cannot generate or forward
> > PME messages?  I think the PCIe protocol details would be helpful
> > here.
>
> No; it's specific to an internal sequence in the SoC.
>
> If the problematic root port is in D3 during s0i3 this
> problematic sequence happens.  If the root port is in D0
> then it doesn't.
>
>  From discussion with others in AMD that's at least one
> reason why the firmware doesn't advertise any power management
> on this root port and why Linux shouldn't be using it.
>
> >> Comparing registers between Linux and Windows 11 this behavior to put root
> >> ports into D3 at suspend is unique to Linux.  On an affected system
> >> Windows does not put the root ports into D3 over Modern Standby.
> >>
> >> Windows doesn't put the root ports into D3 because root ports are not
> >> power manageable; they're missing _PRW and _S0W.
> > platform_pci_power_manageable() tests adev->flags.power_manageable,
> > which is set by acpi_bus_get_power_flags() when a device has _PS0 or
> > _PR0.
> >
> > So I don't know what's relevant out of _PRW, _S0W, _PS0, _PR0, but
> > this sentence doesn't seem to match the code.
>
> The firmware doesn't have _PS0 or _PR0 either for this root
> port.
>
> >> Linux shouldn't be assuming they support D3 just because they're newer
> >> than 2015, the ports should also be deemed power manageable.
> >> Add an extra check for this to ensure D3 isn't selected for such machines.
> > Is this talking about D3hot or D3cold or both?  If we can make this
> > explicit, it will help me out.  It's probably obvious to power
> > experts, but I'm not one.
> Both.
> >> Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
> >> Reported-by: Iain Lane <iain@orangesquash.org.uk>
> >> Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121
> >> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
> >> ---
> >>   drivers/pci/pci.c | 3 +++
> >>   1 file changed, 3 insertions(+)
> >>
> >> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> >> index 5ede93222bc1..3fe27aef09e6 100644
> >> --- a/drivers/pci/pci.c
> >> +++ b/drivers/pci/pci.c
> >> @@ -3010,6 +3010,9 @@ bool pci_bridge_d3_possible(struct pci_dev *bridge)
> >>              if (dmi_check_system(bridge_d3_blacklist))
> >>                      return false;
> >>
> >> +            if (!platform_pci_power_manageable(bridge))
> >> +                    return false;
> >> +

This goes too far, because it causes all ports, not just root ports.

If the intention is to address a problem with a root port, then the
change in behavior should be limited to root ports.

And yes, you can argue that for root ports, specifically, power
management without firmware support is rather pointless if not
harmful.

> >>              /*
> >>               * It should be safe to put PCIe ports from 2015 or newer
> >>               * to D3.
> >> --
> >> 2.34.1
> Something that this patch makes me wonder is if the original
> heuristic was actually correct.

Do you mean the cutoff date?

> Did the PCIe ports from "older" machine have everything needed
> to let them go to D3?
>
> Or would this change also let the heuristic be dropped?

For root ports - probably.  In general - not at all.