* Re: [PATCH] PCI: Apply mandatory recovery delay on return from D3cold
2026-07-10 20:16 ` [PATCH] PCI: Apply mandatory recovery delay on return from D3cold Bjorn Helgaas
@ 2026-07-10 20:19 ` Mario Limonciello
2026-07-13 3:22 ` Mario Limonciello
1 sibling, 0 replies; 3+ messages in thread
From: Mario Limonciello @ 2026-07-10 20:19 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: Bjorn Helgaas, linux-pci, linux-usb, linux-kernel, mrh, stern,
hannes, jase_harley, superveridical, david.c.hubbard, bugzilla,
michal.pecio, Rafael J. Wysocki, linux-pm
On 7/10/26 15:16, Bjorn Helgaas wrote:
> [+cc Rafael, linux-pm]
>
> On Wed, Jul 08, 2026 at 10:26:50AM -0500, Mario Limonciello wrote:
>> Per the "PCI Bus Power Management Interface Specification", rev. 1.2,
>> sec. 5.4, there is a minimum recovery time between programming a function
>> from D3 to D0 and accessing it. The spec does not limit this to D3hot; it
>> applies to the D3cold to D0 transition as well.
>>
>> pci_power_up() only honors this delay on the D3hot branch. When a device
>> returns from D3cold, platform_pci_set_power_state() has already restored
>> main power before PCI_PM_CTRL is read, so the state read from the register
>> is D0 and the transition delay block is skipped by the
>>
>> if (state == PCI_D0)
>> goto end;
>>
>> early return. The register value is masked with PCI_PM_CTRL_STATE_MASK
>> and cannot represent D3cold, so only dev->current_state still reflects the
>> D3cold origin at this point.
>>
>> Apply the delay based on dev->current_state, ahead of the early return, so
>> it takes effect on the D3cold to D0 path before the device is accessed.
>> Use the device's d3cold_delay, which the platform may tune via _DSM and
>> quirks may raise, rather than the D3hot delay.
>>
>> To keep the existing D3hot callers unchanged, pci_dev_d3_sleep() now takes
>> the delay in milliseconds and a pci_dev_d3hot_sleep() wrapper supplies the
>> D3hot delay as before.
>>
>> Reported-by: mrh@frame.work
>> Closes: https://bugzilla.kernel.org/show_bug.cgi?id=221073
>> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
>> ---
>> Cc: mrh@frame.work
>> Cc: stern@rowland.harvard.edu
>> Cc: hannes@vonhaugwitz.com
>> Cc: jase_harley@protonmail.com
>> Cc: superveridical@gmail.com
>> Cc: david.c.hubbard@gmail.com
>> Cc: bugzilla@logical.ink
>> Cc: michal.pecio@gmail.com
>> ---
>> drivers/pci/pci.c | 26 ++++++++++++++++++++------
>> 1 file changed, 20 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
>> index 77b17b13ee615..e09cfb28fe61c 100644
>> --- a/drivers/pci/pci.c
>> +++ b/drivers/pci/pci.c
>> @@ -81,9 +81,8 @@ struct pci_pme_device {
>> */
>> #define PCIE_RESET_READY_POLL_MS 60000 /* msec */
>>
>> -static void pci_dev_d3_sleep(struct pci_dev *dev)
>> +static void pci_dev_d3_sleep(unsigned int delay_ms)
>> {
>> - unsigned int delay_ms = max(dev->d3hot_delay, pci_pm_d3hot_delay);
>> unsigned int upper;
>>
>> if (delay_ms) {
>> @@ -94,6 +93,11 @@ static void pci_dev_d3_sleep(struct pci_dev *dev)
>> }
>> }
>>
>> +static void pci_dev_d3hot_sleep(struct pci_dev *dev)
>> +{
>> + pci_dev_d3_sleep(max(dev->d3hot_delay, pci_pm_d3hot_delay));
>> +}
>> +
>> bool pci_reset_supported(struct pci_dev *dev)
>> {
>> return dev->reset_methods[0] != 0;
>> @@ -1333,6 +1337,16 @@ int pci_power_up(struct pci_dev *dev)
>> need_restore = (state == PCI_D3hot || dev->current_state >= PCI_D3hot) &&
>> !(pmcsr & PCI_PM_CTRL_NO_SOFT_RESET);
>>
>> + /*
>> + * A device returning from D3cold has already been powered back on by
>> + * platform_pci_set_power_state() above, so PCI_PM_CTRL now reads back
>> + * as D0 and the transition delays below are skipped. PCI PM 1.2 still
>> + * requires a minimum recovery time on the D3 to D0 transition, so apply
>> + * the device's D3cold recovery delay here before it is accessed.
>> + */
>> + if (dev->current_state == PCI_D3cold)
>> + pci_dev_d3_sleep(dev->d3cold_delay);
>
> My understanding is that the "mandatory transition delays" below only
> cover a transition caused by the write to PCI_PM_CTRL, which would be
> from D1, D2, or D3hot to D0.
>
> If the device started in D3cold, platform_pci_set_power_state(PCI_D0)
> should transition it to D0uninitialized *and* take care of any
> required delays [1]. The device should be Configuration-Ready upon
> return (and we've already read PCI_PM_CTRL above, and the read
> returned something other than PCI_ERROR_RESPONSE).
FWIW - I was wondering if we shouldn't be reading PCI_PM_CTRL until the
mandatory delay too.
>
> But evidently adding more delay does make a difference, even though
> the config read of PCI_PM_CTRL seemed successful. I guess it's
> conceivable that platform AML doesn't wait quite long enough, although
> it does seem to affect more than one platform (AMD Strix Halo,
> Framework Desktop/AMD Ryzen AI Max 300, Lenovo ThinkPad T14 Gen 6 AMD
> Ryzen AI 7 Pro 350), and I *assume* the issue doesn't happen under
> Windows?
I don't want to make any rash decisions until we have more testing
evidence than one person testing 4 cycles. I would like more of the
people who reported this to confirm the patches really help and it
wasn't just luck in those 4 cycles for that 1 person.
The part that still doesn't sit well with me is this behavior being tied
to the VRAM size. I don't feel these should be connected.
>
> [1] https://lore.kernel.org/linux-pci/CAJZ5v0iZN5NtUztqe=MxCRcXdBaaqzZ749OqSUkadwwBy0ugUQ@mail.gmail.com/
>
>> if (state == PCI_D0)
>> goto end;
>>
>> @@ -1344,7 +1358,7 @@ int pci_power_up(struct pci_dev *dev)
>>
>> /* Mandatory transition delays; see PCI PM 1.2. */
>> if (state == PCI_D3hot) {
>> - pci_dev_d3_sleep(dev);
>> + pci_dev_d3hot_sleep(dev);
>> if (!(pmcsr & PCI_PM_CTRL_NO_SOFT_RESET)) {
>> ret = pci_dev_wait(dev, "power up D3hot->D0uninitialized",
>> PCIE_RESET_READY_POLL_MS);
>> @@ -1514,7 +1528,7 @@ static int pci_set_low_power_state(struct pci_dev *dev, pci_power_t state, bool
>>
>> /* Mandatory power management transition delays; see PCI PM 1.2. */
>> if (state == PCI_D3hot)
>> - pci_dev_d3_sleep(dev);
>> + pci_dev_d3hot_sleep(dev);
>> else if (state == PCI_D2)
>> udelay(PCI_PM_D2_DELAY);
>>
>> @@ -4511,12 +4525,12 @@ static int pci_pm_reset(struct pci_dev *dev, bool probe)
>> csr &= ~PCI_PM_CTRL_STATE_MASK;
>> csr |= PCI_D3hot;
>> pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, csr);
>> - pci_dev_d3_sleep(dev);
>> + pci_dev_d3hot_sleep(dev);
>>
>> csr &= ~PCI_PM_CTRL_STATE_MASK;
>> csr |= PCI_D0;
>> pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, csr);
>> - pci_dev_d3_sleep(dev);
>> + pci_dev_d3hot_sleep(dev);
>>
>> ret = pci_dev_wait(dev, "PM D3hot->D0", PCIE_RESET_READY_POLL_MS);
>> pci_dev_reset_iommu_done(dev);
>> --
>> 2.43.0
>>
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [PATCH] PCI: Apply mandatory recovery delay on return from D3cold
2026-07-10 20:16 ` [PATCH] PCI: Apply mandatory recovery delay on return from D3cold Bjorn Helgaas
2026-07-10 20:19 ` Mario Limonciello
@ 2026-07-13 3:22 ` Mario Limonciello
1 sibling, 0 replies; 3+ messages in thread
From: Mario Limonciello @ 2026-07-13 3:22 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: Bjorn Helgaas, linux-pci, linux-usb, linux-kernel, mrh, stern,
hannes, jase_harley, superveridical, david.c.hubbard, bugzilla,
michal.pecio, Rafael J. Wysocki, linux-pm
On 7/10/26 15:16, Bjorn Helgaas wrote:
> [+cc Rafael, linux-pm]
>
> On Wed, Jul 08, 2026 at 10:26:50AM -0500, Mario Limonciello wrote:
>> Per the "PCI Bus Power Management Interface Specification", rev. 1.2,
>> sec. 5.4, there is a minimum recovery time between programming a function
>> from D3 to D0 and accessing it. The spec does not limit this to D3hot; it
>> applies to the D3cold to D0 transition as well.
>>
>> pci_power_up() only honors this delay on the D3hot branch. When a device
>> returns from D3cold, platform_pci_set_power_state() has already restored
>> main power before PCI_PM_CTRL is read, so the state read from the register
>> is D0 and the transition delay block is skipped by the
>>
>> if (state == PCI_D0)
>> goto end;
>>
>> early return. The register value is masked with PCI_PM_CTRL_STATE_MASK
>> and cannot represent D3cold, so only dev->current_state still reflects the
>> D3cold origin at this point.
>>
>> Apply the delay based on dev->current_state, ahead of the early return, so
>> it takes effect on the D3cold to D0 path before the device is accessed.
>> Use the device's d3cold_delay, which the platform may tune via _DSM and
>> quirks may raise, rather than the D3hot delay.
>>
>> To keep the existing D3hot callers unchanged, pci_dev_d3_sleep() now takes
>> the delay in milliseconds and a pci_dev_d3hot_sleep() wrapper supplies the
>> D3hot delay as before.
>>
>> Reported-by: mrh@frame.work
>> Closes: https://bugzilla.kernel.org/show_bug.cgi?id=221073
>> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
>> ---
>> Cc: mrh@frame.work
>> Cc: stern@rowland.harvard.edu
>> Cc: hannes@vonhaugwitz.com
>> Cc: jase_harley@protonmail.com
>> Cc: superveridical@gmail.com
>> Cc: david.c.hubbard@gmail.com
>> Cc: bugzilla@logical.ink
>> Cc: michal.pecio@gmail.com
>> ---
>> drivers/pci/pci.c | 26 ++++++++++++++++++++------
>> 1 file changed, 20 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
>> index 77b17b13ee615..e09cfb28fe61c 100644
>> --- a/drivers/pci/pci.c
>> +++ b/drivers/pci/pci.c
>> @@ -81,9 +81,8 @@ struct pci_pme_device {
>> */
>> #define PCIE_RESET_READY_POLL_MS 60000 /* msec */
>>
>> -static void pci_dev_d3_sleep(struct pci_dev *dev)
>> +static void pci_dev_d3_sleep(unsigned int delay_ms)
>> {
>> - unsigned int delay_ms = max(dev->d3hot_delay, pci_pm_d3hot_delay);
>> unsigned int upper;
>>
>> if (delay_ms) {
>> @@ -94,6 +93,11 @@ static void pci_dev_d3_sleep(struct pci_dev *dev)
>> }
>> }
>>
>> +static void pci_dev_d3hot_sleep(struct pci_dev *dev)
>> +{
>> + pci_dev_d3_sleep(max(dev->d3hot_delay, pci_pm_d3hot_delay));
>> +}
>> +
>> bool pci_reset_supported(struct pci_dev *dev)
>> {
>> return dev->reset_methods[0] != 0;
>> @@ -1333,6 +1337,16 @@ int pci_power_up(struct pci_dev *dev)
>> need_restore = (state == PCI_D3hot || dev->current_state >= PCI_D3hot) &&
>> !(pmcsr & PCI_PM_CTRL_NO_SOFT_RESET);
>>
>> + /*
>> + * A device returning from D3cold has already been powered back on by
>> + * platform_pci_set_power_state() above, so PCI_PM_CTRL now reads back
>> + * as D0 and the transition delays below are skipped. PCI PM 1.2 still
>> + * requires a minimum recovery time on the D3 to D0 transition, so apply
>> + * the device's D3cold recovery delay here before it is accessed.
>> + */
>> + if (dev->current_state == PCI_D3cold)
>> + pci_dev_d3_sleep(dev->d3cold_delay);
>
> My understanding is that the "mandatory transition delays" below only
> cover a transition caused by the write to PCI_PM_CTRL, which would be
> from D1, D2, or D3hot to D0.
>
> If the device started in D3cold, platform_pci_set_power_state(PCI_D0)
> should transition it to D0uninitialized *and* take care of any
> required delays [1]. The device should be Configuration-Ready upon
> return (and we've already read PCI_PM_CTRL above, and the read
> returned something other than PCI_ERROR_RESPONSE).
>
> But evidently adding more delay does make a difference, even though
> the config read of PCI_PM_CTRL seemed successful. I guess it's
> conceivable that platform AML doesn't wait quite long enough, although
> it does seem to affect more than one platform (AMD Strix Halo,
> Framework Desktop/AMD Ryzen AI Max 300, Lenovo ThinkPad T14 Gen 6 AMD
> Ryzen AI 7 Pro 350), and I *assume* the issue doesn't happen under
> Windows?
Based on some more testing that occurred this patch doesn't help. Back
to the drawing board.
>
> [1] https://lore.kernel.org/linux-pci/CAJZ5v0iZN5NtUztqe=MxCRcXdBaaqzZ749OqSUkadwwBy0ugUQ@mail.gmail.com/
>
>> if (state == PCI_D0)
>> goto end;
>>
>> @@ -1344,7 +1358,7 @@ int pci_power_up(struct pci_dev *dev)
>>
>> /* Mandatory transition delays; see PCI PM 1.2. */
>> if (state == PCI_D3hot) {
>> - pci_dev_d3_sleep(dev);
>> + pci_dev_d3hot_sleep(dev);
>> if (!(pmcsr & PCI_PM_CTRL_NO_SOFT_RESET)) {
>> ret = pci_dev_wait(dev, "power up D3hot->D0uninitialized",
>> PCIE_RESET_READY_POLL_MS);
>> @@ -1514,7 +1528,7 @@ static int pci_set_low_power_state(struct pci_dev *dev, pci_power_t state, bool
>>
>> /* Mandatory power management transition delays; see PCI PM 1.2. */
>> if (state == PCI_D3hot)
>> - pci_dev_d3_sleep(dev);
>> + pci_dev_d3hot_sleep(dev);
>> else if (state == PCI_D2)
>> udelay(PCI_PM_D2_DELAY);
>>
>> @@ -4511,12 +4525,12 @@ static int pci_pm_reset(struct pci_dev *dev, bool probe)
>> csr &= ~PCI_PM_CTRL_STATE_MASK;
>> csr |= PCI_D3hot;
>> pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, csr);
>> - pci_dev_d3_sleep(dev);
>> + pci_dev_d3hot_sleep(dev);
>>
>> csr &= ~PCI_PM_CTRL_STATE_MASK;
>> csr |= PCI_D0;
>> pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, csr);
>> - pci_dev_d3_sleep(dev);
>> + pci_dev_d3hot_sleep(dev);
>>
>> ret = pci_dev_wait(dev, "PM D3hot->D0", PCIE_RESET_READY_POLL_MS);
>> pci_dev_reset_iommu_done(dev);
>> --
>> 2.43.0
>>
^ permalink raw reply [flat|nested] 3+ messages in thread