Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Jon Hunter]

Hi Manivannan,

On 22/09/2025 17:16, Manivannan Sadhasivam via B4 Relay wrote:
> From: Manivannan Sadhasivam <manivannan.sadhasivam@oss.qualcomm.com>
> 
> So far, the PCI subsystem has honored the ASPM and Clock PM states set by
> the BIOS (through LNKCTL) during device initialization, if it relies on the
> default state selected using:
> 
> * Kconfig: CONFIG_PCIEASPM_DEFAULT=y, or
> * cmdline: "pcie_aspm=off", or
> * FADT: ACPI_FADT_NO_ASPM
> 
> This was done conservatively to avoid issues with the buggy devices that
> advertise ASPM capabilities, but behave erratically if the ASPM states are
> enabled. So the PCI subsystem ended up trusting the BIOS to enable only the
> ASPM states that were known to work for the devices.
> 
> But this turned out to be a problem for devicetree platforms, especially
> the ARM based devicetree platforms powering Embedded and *some* Compute
> devices as they tend to run without any standard BIOS. So the ASPM states
> on these platforms were left disabled during boot and the PCI subsystem
> never bothered to enable them, unless the user has forcefully enabled the
> ASPM states through Kconfig, cmdline, and sysfs or the device drivers
> themselves, enabling the ASPM states through pci_enable_link_state() APIs.
> 
> This caused runtime power issues on those platforms. So a couple of
> approaches were tried to mitigate this BIOS dependency without user
> intervention by enabling the ASPM states in the PCI controller drivers
> after device enumeration, and overriding the ASPM/Clock PM states
> by the PCI controller drivers through an API before enumeration.
> 
> But it has been concluded that none of these mitigations should really be
> required and the PCI subsystem should enable the ASPM states advertised by
> the devices without relying on BIOS or the PCI controller drivers. If any
> device is found to be misbehaving after enabling ASPM states that they
> advertised, then those devices should be quirked to disable the problematic
> ASPM/Clock PM states.
> 
> In an effort to do so, start by overriding the ASPM and Clock PM states set
> by the BIOS for devicetree platforms first. Separate helper functions are
> introduced to override the BIOS set states by enabling all of them if
> of_have_populated_dt() returns true. To aid debugging, print the overridden
> ASPM and Clock PM states as well.
> 
> In the future, these helpers could be extended to allow other platforms
> like VMD, newer ACPI systems with a cutoff year etc... to follow the path.
> 
> Link: https://lore.kernel.org/linux-pci/20250828204345.GA958461@bhelgaas
> Suggested-by: Bjorn Helgaas <helgaas@kernel.org>
> Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@oss.qualcomm.com>
> Link: https://patch.msgid.link/20250916-pci-dt-aspm-v1-1-778fe907c9ad@oss.qualcomm.com
> ---
>   drivers/pci/pcie/aspm.c | 42 ++++++++++++++++++++++++++++++++++++++++--
>   1 file changed, 40 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c
> index 919a05b9764791c3cc469c9ada62ba5b2c405118..cda31150aec1b67b6a48b60569222ea3d1c3d41f 100644
> --- a/drivers/pci/pcie/aspm.c
> +++ b/drivers/pci/pcie/aspm.c
> @@ -15,6 +15,7 @@
>   #include <linux/math.h>
>   #include <linux/module.h>
>   #include <linux/moduleparam.h>
> +#include <linux/of.h>
>   #include <linux/pci.h>
>   #include <linux/pci_regs.h>
>   #include <linux/errno.h>
> @@ -235,13 +236,15 @@ struct pcie_link_state {
>   	u32 aspm_support:7;		/* Supported ASPM state */
>   	u32 aspm_enabled:7;		/* Enabled ASPM state */
>   	u32 aspm_capable:7;		/* Capable ASPM state with latency */
> -	u32 aspm_default:7;		/* Default ASPM state by BIOS */
> +	u32 aspm_default:7;		/* Default ASPM state by BIOS or
> +					   override */
>   	u32 aspm_disable:7;		/* Disabled ASPM state */
>   
>   	/* Clock PM state */
>   	u32 clkpm_capable:1;		/* Clock PM capable? */
>   	u32 clkpm_enabled:1;		/* Current Clock PM state */
> -	u32 clkpm_default:1;		/* Default Clock PM state by BIOS */
> +	u32 clkpm_default:1;		/* Default Clock PM state by BIOS or
> +					   override */
>   	u32 clkpm_disable:1;		/* Clock PM disabled */
>   };
>   
> @@ -373,6 +376,18 @@ static void pcie_set_clkpm(struct pcie_link_state *link, int enable)
>   	pcie_set_clkpm_nocheck(link, enable);
>   }
>   
> +static void pcie_clkpm_override_default_link_state(struct pcie_link_state *link,
> +						   int enabled)
> +{
> +	struct pci_dev *pdev = link->downstream;
> +
> +	/* Override the BIOS disabled Clock PM state for devicetree platforms */
> +	if (of_have_populated_dt() && !enabled) {
> +		link->clkpm_default = 1;
> +		pci_info(pdev, "Clock PM state overridden: ClockPM+\n");
> +	}
> +}
> +
>   static void pcie_clkpm_cap_init(struct pcie_link_state *link, int blacklist)
>   {
>   	int capable = 1, enabled = 1;
> @@ -395,6 +410,7 @@ static void pcie_clkpm_cap_init(struct pcie_link_state *link, int blacklist)
>   	}
>   	link->clkpm_enabled = enabled;
>   	link->clkpm_default = enabled;
> +	pcie_clkpm_override_default_link_state(link, enabled);
>   	link->clkpm_capable = capable;
>   	link->clkpm_disable = blacklist ? 1 : 0;
>   }
> @@ -788,6 +804,26 @@ static void aspm_l1ss_init(struct pcie_link_state *link)
>   		aspm_calc_l12_info(link, parent_l1ss_cap, child_l1ss_cap);
>   }
>   
> +static void pcie_aspm_override_default_link_state(struct pcie_link_state *link)
> +{
> +	struct pci_dev *pdev = link->downstream;
> +	u32 override;
> +
> +	/* Override the BIOS disabled ASPM states for devicetree platforms */
> +	if (of_have_populated_dt()) {
> +		link->aspm_default = PCIE_LINK_STATE_ASPM_ALL;
> +		override = link->aspm_default & ~link->aspm_enabled;
> +		if (override)
> +			pci_info(pdev, "ASPM states overridden: %s%s%s%s%s%s\n",
> +				 (override & PCIE_LINK_STATE_L0S) ? "L0s+, " : "",
> +				 (override & PCIE_LINK_STATE_L1) ? "L1+, " : "",
> +				 (override & PCIE_LINK_STATE_L1_1) ? "L1.1+, " : "",
> +				 (override & PCIE_LINK_STATE_L1_2) ? "L1.2+, " : "",
> +				 (override & PCIE_LINK_STATE_L1_1_PCIPM) ? "L1.1 PCI-PM+, " : "",
> +				 (override & PCIE_LINK_STATE_L1_2_PCIPM) ? "L1.2 PCI-PM+" : "");
> +	}
> +}
> +
>   static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist)
>   {
>   	struct pci_dev *child = link->downstream, *parent = link->pdev;
> @@ -868,6 +904,8 @@ static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist)
>   	/* Save default state */
>   	link->aspm_default = link->aspm_enabled;
>   
> +	pcie_aspm_override_default_link_state(link);
> +
>   	/* Setup initial capable state. Will be updated later */
>   	link->aspm_capable = link->aspm_support;


Since this commit was added in Linux v6.18, I have been observing a 
suspend test failures on some of our boards. The suspend test suspends 
the devices for 20 secs and before this change the board would resume in 
about ~27 secs (including the 20 sec sleep). After this change the board 
would take over 80 secs to resume and this triggered a failure.

Looking at the logs, I can see it is the NVMe device on the board that 
is having an issue, and I see the reset failing ...

  [  945.754939] r8169 0007:01:00.0 enP7p1s0: Link is Up - 1Gbps/Full -
   flow control rx/tx
  [ 1002.467432] nvme nvme0: I/O tag 12 (400c) opcode 0x9 (Admin Cmd) QID
   0 timeout, reset controller
  [ 1002.493713] nvme nvme0: 12/0/0 default/read/poll queues
  [ 1003.050448] nvme nvme0: ctrl state 1 is not RESETTING
  [ 1003.050481] OOM killer enabled.
  [ 1003.054035] nvme nvme0: Disabling device after reset failure: -19

 From the above timestamps the delay is coming from the NVMe. I see this 
issue on several boards with different NVMe devices and I can workaround 
this by disabling ASPM L0/L1 for these devices ...

  DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5011, quirk_disable_aspm_l0s_l1);
  DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5036, quirk_disable_aspm_l0s_l1);
  DECLARE_PCI_FIXUP_HEADER(0x1b4b, 0x1322, quirk_disable_aspm_l0s_l1);
  DECLARE_PCI_FIXUP_HEADER(0xc0a9, 0x540a, quirk_disable_aspm_l0s_l1);

I am curious if you have seen any similar issues?

Other PCIe devices seem to be OK (like the realtek r8169) but just the 
NVMe is having issues. So I am trying to figure out the best way to 
resolve this?

Thanks
Jon

-- 
nvpublic

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Manivannan Sadhasivam]

On Thu, Jan 22, 2026 at 12:12:42PM +0000, Jon Hunter wrote:
> Hi Manivannan,
> 
> On 22/09/2025 17:16, Manivannan Sadhasivam via B4 Relay wrote:
> > From: Manivannan Sadhasivam <manivannan.sadhasivam@oss.qualcomm.com>
> > 
> > So far, the PCI subsystem has honored the ASPM and Clock PM states set by
> > the BIOS (through LNKCTL) during device initialization, if it relies on the
> > default state selected using:
> > 
> > * Kconfig: CONFIG_PCIEASPM_DEFAULT=y, or
> > * cmdline: "pcie_aspm=off", or
> > * FADT: ACPI_FADT_NO_ASPM
> > 
> > This was done conservatively to avoid issues with the buggy devices that
> > advertise ASPM capabilities, but behave erratically if the ASPM states are
> > enabled. So the PCI subsystem ended up trusting the BIOS to enable only the
> > ASPM states that were known to work for the devices.
> > 
> > But this turned out to be a problem for devicetree platforms, especially
> > the ARM based devicetree platforms powering Embedded and *some* Compute
> > devices as they tend to run without any standard BIOS. So the ASPM states
> > on these platforms were left disabled during boot and the PCI subsystem
> > never bothered to enable them, unless the user has forcefully enabled the
> > ASPM states through Kconfig, cmdline, and sysfs or the device drivers
> > themselves, enabling the ASPM states through pci_enable_link_state() APIs.
> > 
> > This caused runtime power issues on those platforms. So a couple of
> > approaches were tried to mitigate this BIOS dependency without user
> > intervention by enabling the ASPM states in the PCI controller drivers
> > after device enumeration, and overriding the ASPM/Clock PM states
> > by the PCI controller drivers through an API before enumeration.
> > 
> > But it has been concluded that none of these mitigations should really be
> > required and the PCI subsystem should enable the ASPM states advertised by
> > the devices without relying on BIOS or the PCI controller drivers. If any
> > device is found to be misbehaving after enabling ASPM states that they
> > advertised, then those devices should be quirked to disable the problematic
> > ASPM/Clock PM states.
> > 
> > In an effort to do so, start by overriding the ASPM and Clock PM states set
> > by the BIOS for devicetree platforms first. Separate helper functions are
> > introduced to override the BIOS set states by enabling all of them if
> > of_have_populated_dt() returns true. To aid debugging, print the overridden
> > ASPM and Clock PM states as well.
> > 
> > In the future, these helpers could be extended to allow other platforms
> > like VMD, newer ACPI systems with a cutoff year etc... to follow the path.
> > 
> > Link: https://lore.kernel.org/linux-pci/20250828204345.GA958461@bhelgaas
> > Suggested-by: Bjorn Helgaas <helgaas@kernel.org>
> > Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@oss.qualcomm.com>
> > Link: https://patch.msgid.link/20250916-pci-dt-aspm-v1-1-778fe907c9ad@oss.qualcomm.com
> > ---
> >   drivers/pci/pcie/aspm.c | 42 ++++++++++++++++++++++++++++++++++++++++--
> >   1 file changed, 40 insertions(+), 2 deletions(-)
> > 
> > diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c
> > index 919a05b9764791c3cc469c9ada62ba5b2c405118..cda31150aec1b67b6a48b60569222ea3d1c3d41f 100644
> > --- a/drivers/pci/pcie/aspm.c
> > +++ b/drivers/pci/pcie/aspm.c
> > @@ -15,6 +15,7 @@
> >   #include <linux/math.h>
> >   #include <linux/module.h>
> >   #include <linux/moduleparam.h>
> > +#include <linux/of.h>
> >   #include <linux/pci.h>
> >   #include <linux/pci_regs.h>
> >   #include <linux/errno.h>
> > @@ -235,13 +236,15 @@ struct pcie_link_state {
> >   	u32 aspm_support:7;		/* Supported ASPM state */
> >   	u32 aspm_enabled:7;		/* Enabled ASPM state */
> >   	u32 aspm_capable:7;		/* Capable ASPM state with latency */
> > -	u32 aspm_default:7;		/* Default ASPM state by BIOS */
> > +	u32 aspm_default:7;		/* Default ASPM state by BIOS or
> > +					   override */
> >   	u32 aspm_disable:7;		/* Disabled ASPM state */
> >   	/* Clock PM state */
> >   	u32 clkpm_capable:1;		/* Clock PM capable? */
> >   	u32 clkpm_enabled:1;		/* Current Clock PM state */
> > -	u32 clkpm_default:1;		/* Default Clock PM state by BIOS */
> > +	u32 clkpm_default:1;		/* Default Clock PM state by BIOS or
> > +					   override */
> >   	u32 clkpm_disable:1;		/* Clock PM disabled */
> >   };
> > @@ -373,6 +376,18 @@ static void pcie_set_clkpm(struct pcie_link_state *link, int enable)
> >   	pcie_set_clkpm_nocheck(link, enable);
> >   }
> > +static void pcie_clkpm_override_default_link_state(struct pcie_link_state *link,
> > +						   int enabled)
> > +{
> > +	struct pci_dev *pdev = link->downstream;
> > +
> > +	/* Override the BIOS disabled Clock PM state for devicetree platforms */
> > +	if (of_have_populated_dt() && !enabled) {
> > +		link->clkpm_default = 1;
> > +		pci_info(pdev, "Clock PM state overridden: ClockPM+\n");
> > +	}
> > +}
> > +
> >   static void pcie_clkpm_cap_init(struct pcie_link_state *link, int blacklist)
> >   {
> >   	int capable = 1, enabled = 1;
> > @@ -395,6 +410,7 @@ static void pcie_clkpm_cap_init(struct pcie_link_state *link, int blacklist)
> >   	}
> >   	link->clkpm_enabled = enabled;
> >   	link->clkpm_default = enabled;
> > +	pcie_clkpm_override_default_link_state(link, enabled);
> >   	link->clkpm_capable = capable;
> >   	link->clkpm_disable = blacklist ? 1 : 0;
> >   }
> > @@ -788,6 +804,26 @@ static void aspm_l1ss_init(struct pcie_link_state *link)
> >   		aspm_calc_l12_info(link, parent_l1ss_cap, child_l1ss_cap);
> >   }
> > +static void pcie_aspm_override_default_link_state(struct pcie_link_state *link)
> > +{
> > +	struct pci_dev *pdev = link->downstream;
> > +	u32 override;
> > +
> > +	/* Override the BIOS disabled ASPM states for devicetree platforms */
> > +	if (of_have_populated_dt()) {
> > +		link->aspm_default = PCIE_LINK_STATE_ASPM_ALL;
> > +		override = link->aspm_default & ~link->aspm_enabled;
> > +		if (override)
> > +			pci_info(pdev, "ASPM states overridden: %s%s%s%s%s%s\n",
> > +				 (override & PCIE_LINK_STATE_L0S) ? "L0s+, " : "",
> > +				 (override & PCIE_LINK_STATE_L1) ? "L1+, " : "",
> > +				 (override & PCIE_LINK_STATE_L1_1) ? "L1.1+, " : "",
> > +				 (override & PCIE_LINK_STATE_L1_2) ? "L1.2+, " : "",
> > +				 (override & PCIE_LINK_STATE_L1_1_PCIPM) ? "L1.1 PCI-PM+, " : "",
> > +				 (override & PCIE_LINK_STATE_L1_2_PCIPM) ? "L1.2 PCI-PM+" : "");
> > +	}
> > +}
> > +
> >   static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist)
> >   {
> >   	struct pci_dev *child = link->downstream, *parent = link->pdev;
> > @@ -868,6 +904,8 @@ static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist)
> >   	/* Save default state */
> >   	link->aspm_default = link->aspm_enabled;
> > +	pcie_aspm_override_default_link_state(link);
> > +
> >   	/* Setup initial capable state. Will be updated later */
> >   	link->aspm_capable = link->aspm_support;
> 
> 
> Since this commit was added in Linux v6.18, I have been observing a suspend
> test failures on some of our boards. The suspend test suspends the devices
> for 20 secs and before this change the board would resume in about ~27 secs
> (including the 20 sec sleep). After this change the board would take over 80
> secs to resume and this triggered a failure.
> 
> Looking at the logs, I can see it is the NVMe device on the board that is
> having an issue, and I see the reset failing ...
> 
>  [  945.754939] r8169 0007:01:00.0 enP7p1s0: Link is Up - 1Gbps/Full -
>   flow control rx/tx
>  [ 1002.467432] nvme nvme0: I/O tag 12 (400c) opcode 0x9 (Admin Cmd) QID
>   0 timeout, reset controller
>  [ 1002.493713] nvme nvme0: 12/0/0 default/read/poll queues
>  [ 1003.050448] nvme nvme0: ctrl state 1 is not RESETTING
>  [ 1003.050481] OOM killer enabled.
>  [ 1003.054035] nvme nvme0: Disabling device after reset failure: -19
> 
> From the above timestamps the delay is coming from the NVMe. I see this
> issue on several boards with different NVMe devices and I can workaround
> this by disabling ASPM L0/L1 for these devices ...
> 
>  DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5011, quirk_disable_aspm_l0s_l1);
>  DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5036, quirk_disable_aspm_l0s_l1);
>  DECLARE_PCI_FIXUP_HEADER(0x1b4b, 0x1322, quirk_disable_aspm_l0s_l1);
>  DECLARE_PCI_FIXUP_HEADER(0xc0a9, 0x540a, quirk_disable_aspm_l0s_l1);
> 
> I am curious if you have seen any similar issues?
> 

Marek reported a similar issue on ARM Juno board [1] on which one of the switch
downstream port failed to come up while *entering* system suspend. But I was
clueless as to why the device fails to function only while entering system
suspend and not during runtime. I suspect something is going wrong in the
suspend path.

In your case, looks like the device is failing while resuming from suspend. Did
you see any error log during suspend as well?

> Other PCIe devices seem to be OK (like the realtek r8169) but just the NVMe
> is having issues. So I am trying to figure out the best way to resolve this?
> 

First let's try to isolate the issue to L0s or L1. Can you try disabling L0s
first, then L1?

I will also inspect the suspend/resume path in the meantime.

- Mani

[1] https://lore.kernel.org/linux-pci/cae5cb24-a8b0-4088-bacd-14368f32bdc5@samsung.com/

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Jon Hunter]

On 22/01/2026 13:17, Manivannan Sadhasivam wrote:

...

>> Since this commit was added in Linux v6.18, I have been observing a suspend
>> test failures on some of our boards. The suspend test suspends the devices
>> for 20 secs and before this change the board would resume in about ~27 secs
>> (including the 20 sec sleep). After this change the board would take over 80
>> secs to resume and this triggered a failure.
>>
>> Looking at the logs, I can see it is the NVMe device on the board that is
>> having an issue, and I see the reset failing ...
>>
>>   [  945.754939] r8169 0007:01:00.0 enP7p1s0: Link is Up - 1Gbps/Full -
>>    flow control rx/tx
>>   [ 1002.467432] nvme nvme0: I/O tag 12 (400c) opcode 0x9 (Admin Cmd) QID
>>    0 timeout, reset controller
>>   [ 1002.493713] nvme nvme0: 12/0/0 default/read/poll queues
>>   [ 1003.050448] nvme nvme0: ctrl state 1 is not RESETTING
>>   [ 1003.050481] OOM killer enabled.
>>   [ 1003.054035] nvme nvme0: Disabling device after reset failure: -19
>>
>>  From the above timestamps the delay is coming from the NVMe. I see this
>> issue on several boards with different NVMe devices and I can workaround
>> this by disabling ASPM L0/L1 for these devices ...
>>
>>   DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5011, quirk_disable_aspm_l0s_l1);
>>   DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5036, quirk_disable_aspm_l0s_l1);
>>   DECLARE_PCI_FIXUP_HEADER(0x1b4b, 0x1322, quirk_disable_aspm_l0s_l1);
>>   DECLARE_PCI_FIXUP_HEADER(0xc0a9, 0x540a, quirk_disable_aspm_l0s_l1);
>>
>> I am curious if you have seen any similar issues?
>>
> 
> Marek reported a similar issue on ARM Juno board [1] on which one of the switch
> downstream port failed to come up while *entering* system suspend. But I was
> clueless as to why the device fails to function only while entering system
> suspend and not during runtime. I suspect something is going wrong in the
> suspend path.
> 
> In your case, looks like the device is failing while resuming from suspend. Did
> you see any error log during suspend as well?

I don't see any errors on entering suspend, just resuming from suspend. 
One other thing that I notice, on resuming in a good case I see ...

  tegra194-pcie 141e0000.pcie: Link didn't transition to L2 state

In a bad case I see ...

  tegra194-pcie 141e0000.pcie: Link didn't transition to L2 state
  tegra194-pcie 14160000.pcie: Link didn't transition to L2 state
  tegra194-pcie 14160000.pcie: Link didn't go to detect state

It appears that this is related because ...

  tegra194-pcie 14160000.pcie: PCI host bridge to bus 0004:00
  ...
  nvme nvme0: pci function 0004:01:00.0

>> Other PCIe devices seem to be OK (like the realtek r8169) but just the NVMe
>> is having issues. So I am trying to figure out the best way to resolve this?
>>
> 
> First let's try to isolate the issue to L0s or L1. Can you try disabling L0s
> first, then L1?

Yes I will try this today.

Jon

-- 
nvpublic

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Manivannan Sadhasivam]

On Thu, Jan 22, 2026 at 01:43:41PM +0000, Jon Hunter wrote:
> 
> On 22/01/2026 13:17, Manivannan Sadhasivam wrote:
> 
> ...
> 
> > > Since this commit was added in Linux v6.18, I have been observing a suspend
> > > test failures on some of our boards. The suspend test suspends the devices
> > > for 20 secs and before this change the board would resume in about ~27 secs
> > > (including the 20 sec sleep). After this change the board would take over 80
> > > secs to resume and this triggered a failure.
> > > 
> > > Looking at the logs, I can see it is the NVMe device on the board that is
> > > having an issue, and I see the reset failing ...
> > > 
> > >   [  945.754939] r8169 0007:01:00.0 enP7p1s0: Link is Up - 1Gbps/Full -
> > >    flow control rx/tx
> > >   [ 1002.467432] nvme nvme0: I/O tag 12 (400c) opcode 0x9 (Admin Cmd) QID
> > >    0 timeout, reset controller
> > >   [ 1002.493713] nvme nvme0: 12/0/0 default/read/poll queues
> > >   [ 1003.050448] nvme nvme0: ctrl state 1 is not RESETTING
> > >   [ 1003.050481] OOM killer enabled.
> > >   [ 1003.054035] nvme nvme0: Disabling device after reset failure: -19
> > > 
> > >  From the above timestamps the delay is coming from the NVMe. I see this
> > > issue on several boards with different NVMe devices and I can workaround
> > > this by disabling ASPM L0/L1 for these devices ...
> > > 
> > >   DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5011, quirk_disable_aspm_l0s_l1);
> > >   DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5036, quirk_disable_aspm_l0s_l1);
> > >   DECLARE_PCI_FIXUP_HEADER(0x1b4b, 0x1322, quirk_disable_aspm_l0s_l1);
> > >   DECLARE_PCI_FIXUP_HEADER(0xc0a9, 0x540a, quirk_disable_aspm_l0s_l1);
> > > 
> > > I am curious if you have seen any similar issues?
> > > 
> > 
> > Marek reported a similar issue on ARM Juno board [1] on which one of the switch
> > downstream port failed to come up while *entering* system suspend. But I was
> > clueless as to why the device fails to function only while entering system
> > suspend and not during runtime. I suspect something is going wrong in the
> > suspend path.
> > 
> > In your case, looks like the device is failing while resuming from suspend. Did
> > you see any error log during suspend as well?
> 
> I don't see any errors on entering suspend, just resuming from suspend. One
> other thing that I notice, on resuming in a good case I see ...
> 
>  tegra194-pcie 141e0000.pcie: Link didn't transition to L2 state
> 
> In a bad case I see ...
> 
>  tegra194-pcie 141e0000.pcie: Link didn't transition to L2 state
>  tegra194-pcie 14160000.pcie: Link didn't transition to L2 state
>  tegra194-pcie 14160000.pcie: Link didn't go to detect state
> 

But this error print is coming from:

	tegra_pcie_dw_suspend_noirq()
		\__ tegra_pcie_dw_pme_turnoff()

This function broadcasts PME_Turn_Off message and expects the PME_TO_Ack
response from the device. But in both working and non-working cases, response is
not being received. Then the driver *deasserts* PERST# without asserting it
before (which is weird) and then restarts LTSSM expecting the LTSSM to be in
Detect state. And this one is not happening in non-working case. I don't know
why, maybe the device got crashed?

Can you also try skipping the suspend/resume handlers to isolate the issue with
PME_Turn_Off?

diff --git drivers/pci/controller/dwc/pcie-tegra194.c drivers/pci/controller/dwc/pcie-tegra194.c
index 0ddeef70726d..a5bb122a9477 100644
--- drivers/pci/controller/dwc/pcie-tegra194.c
+++ drivers/pci/controller/dwc/pcie-tegra194.c
@@ -2490,7 +2490,7 @@ static struct platform_driver tegra_pcie_dw_driver = {
        .shutdown = tegra_pcie_dw_shutdown,
        .driver = {
                .name   = "tegra194-pcie",
-               .pm = &tegra_pcie_dw_pm_ops,
+//             .pm = &tegra_pcie_dw_pm_ops,
                .of_match_table = tegra_pcie_dw_of_match,
        },
 };

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Bjorn Helgaas]

[+cc NVMe folks]

On Thu, Jan 22, 2026 at 12:12:42PM +0000, Jon Hunter wrote:
> ...

> Since this commit was added in Linux v6.18, I have been observing a suspend
> test failures on some of our boards. The suspend test suspends the devices
> for 20 secs and before this change the board would resume in about ~27 secs
> (including the 20 sec sleep). After this change the board would take over 80
> secs to resume and this triggered a failure.
> 
> Looking at the logs, I can see it is the NVMe device on the board that is
> having an issue, and I see the reset failing ...
> 
>  [  945.754939] r8169 0007:01:00.0 enP7p1s0: Link is Up - 1Gbps/Full -
>   flow control rx/tx
>  [ 1002.467432] nvme nvme0: I/O tag 12 (400c) opcode 0x9 (Admin Cmd) QID
>   0 timeout, reset controller
>  [ 1002.493713] nvme nvme0: 12/0/0 default/read/poll queues
>  [ 1003.050448] nvme nvme0: ctrl state 1 is not RESETTING
>  [ 1003.050481] OOM killer enabled.
>  [ 1003.054035] nvme nvme0: Disabling device after reset failure: -19
> 
> From the above timestamps the delay is coming from the NVMe. I see this
> issue on several boards with different NVMe devices and I can workaround
> this by disabling ASPM L0/L1 for these devices ...
> 
>  DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5011, quirk_disable_aspm_l0s_l1);
>  DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5036, quirk_disable_aspm_l0s_l1);
>  DECLARE_PCI_FIXUP_HEADER(0x1b4b, 0x1322, quirk_disable_aspm_l0s_l1);
>  DECLARE_PCI_FIXUP_HEADER(0xc0a9, 0x540a, quirk_disable_aspm_l0s_l1);
> 
> I am curious if you have seen any similar issues?
> 
> Other PCIe devices seem to be OK (like the realtek r8169) but just
> the NVMe is having issues. So I am trying to figure out the best way
> to resolve this?

For context, "this commit" refers to f3ac2ff14834, modified by
df5192d9bb0e:

  f3ac2ff14834 ("PCI/ASPM: Enable all ClockPM and ASPM states for devicetree platforms")
  df5192d9bb0e ("PCI/ASPM: Enable only L0s and L1 for devicetree platforms")

The fact that this suspend issue only affects NVMe reminds me of the
code in dw_pcie_suspend_noirq() [1] that bails out early if L1 is
enabled because of some NVMe expectation:

  dw_pcie_suspend_noirq()
  {
    ...
    /*
     * If L1SS is supported, then do not put the link into L2 as some
     * devices such as NVMe expect low resume latency.
     */
    if (dw_pcie_readw_dbi(pci, offset + PCI_EXP_LNKCTL) & PCI_EXP_LNKCTL_ASPM_L1)
      return 0;
    ...

That suggests there's some NVMe/ASPM interaction that the PCI core
doesn't understand yet.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-designware-host.c?id=v6.18#n1146

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Manivannan Sadhasivam]

On Thu, Jan 22, 2026 at 09:29:03AM -0600, Bjorn Helgaas wrote:
> [+cc NVMe folks]
> 
> On Thu, Jan 22, 2026 at 12:12:42PM +0000, Jon Hunter wrote:
> > ...
> 
> > Since this commit was added in Linux v6.18, I have been observing a suspend
> > test failures on some of our boards. The suspend test suspends the devices
> > for 20 secs and before this change the board would resume in about ~27 secs
> > (including the 20 sec sleep). After this change the board would take over 80
> > secs to resume and this triggered a failure.
> > 
> > Looking at the logs, I can see it is the NVMe device on the board that is
> > having an issue, and I see the reset failing ...
> > 
> >  [  945.754939] r8169 0007:01:00.0 enP7p1s0: Link is Up - 1Gbps/Full -
> >   flow control rx/tx
> >  [ 1002.467432] nvme nvme0: I/O tag 12 (400c) opcode 0x9 (Admin Cmd) QID
> >   0 timeout, reset controller
> >  [ 1002.493713] nvme nvme0: 12/0/0 default/read/poll queues
> >  [ 1003.050448] nvme nvme0: ctrl state 1 is not RESETTING
> >  [ 1003.050481] OOM killer enabled.
> >  [ 1003.054035] nvme nvme0: Disabling device after reset failure: -19
> > 
> > From the above timestamps the delay is coming from the NVMe. I see this
> > issue on several boards with different NVMe devices and I can workaround
> > this by disabling ASPM L0/L1 for these devices ...
> > 
> >  DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5011, quirk_disable_aspm_l0s_l1);
> >  DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5036, quirk_disable_aspm_l0s_l1);
> >  DECLARE_PCI_FIXUP_HEADER(0x1b4b, 0x1322, quirk_disable_aspm_l0s_l1);
> >  DECLARE_PCI_FIXUP_HEADER(0xc0a9, 0x540a, quirk_disable_aspm_l0s_l1);
> > 
> > I am curious if you have seen any similar issues?
> > 
> > Other PCIe devices seem to be OK (like the realtek r8169) but just
> > the NVMe is having issues. So I am trying to figure out the best way
> > to resolve this?
> 
> For context, "this commit" refers to f3ac2ff14834, modified by
> df5192d9bb0e:
> 
>   f3ac2ff14834 ("PCI/ASPM: Enable all ClockPM and ASPM states for devicetree platforms")
>   df5192d9bb0e ("PCI/ASPM: Enable only L0s and L1 for devicetree platforms")
> 
> The fact that this suspend issue only affects NVMe reminds me of the
> code in dw_pcie_suspend_noirq() [1] that bails out early if L1 is
> enabled because of some NVMe expectation:
> 
>   dw_pcie_suspend_noirq()
>   {
>     ...
>     /*
>      * If L1SS is supported, then do not put the link into L2 as some
>      * devices such as NVMe expect low resume latency.
>      */
>     if (dw_pcie_readw_dbi(pci, offset + PCI_EXP_LNKCTL) & PCI_EXP_LNKCTL_ASPM_L1)
>       return 0;
>     ...
> 
> That suggests there's some NVMe/ASPM interaction that the PCI core
> doesn't understand yet.
> 

We have this check in place since NVMe driver keeps the device in D0 and expects
the link to be in L1ss on platforms not passing below checks:

        if (pm_suspend_via_firmware() || !ctrl->npss ||
            !pcie_aspm_enabled(pdev) ||
            (ndev->ctrl.quirks & NVME_QUIRK_SIMPLE_SUSPEND))

Since the majority of the DWC platforms do not pass the above checks, we don't
transition the device to D3Cold or link to L2/L3 in dw_pcie_suspend_noirq() if
the link is in L1ss. Though I think we should be checking for D0 state instead
of L1ss here.

I think what is going on here is that since before commits f3ac2ff14834 and
df5192d9bb0e, !pcie_aspm_enabled() check was passing as ASPM was not enabled for
the device (and upstream port) and after those commits, this check is not
passing and the NVMe driver is not shutting down the controller and expects the
link to be in L0/L1ss. But the Tegra controller driver initiates L2/L3
transition, and also turns off the device. So all the NVMe context is lost
during suspend and while resuming, the NVMe driver got confused due to lost
context.

Jon, could you please try the below hack and see if it fixes the issue?

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 0e4caeab739c..4b8d261117f5 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -3723,7 +3723,7 @@ static int nvme_suspend(struct device *dev)
         * state (which may not be possible if the link is up).
         */
        if (pm_suspend_via_firmware() || !ctrl->npss ||
-           !pcie_aspm_enabled(pdev) ||
+           pcie_aspm_enabled(pdev) ||
            (ndev->ctrl.quirks & NVME_QUIRK_SIMPLE_SUSPEND))
                return nvme_disable_prepare_reset(ndev, true);
 
This will confirm whether the issue is due to Tegra controller driver breaking
the NVMe driver assumption or not.

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Jon Hunter]

On 22/01/2026 17:01, Manivannan Sadhasivam wrote:
> On Thu, Jan 22, 2026 at 09:29:03AM -0600, Bjorn Helgaas wrote:
>> [+cc NVMe folks]
>>
>> On Thu, Jan 22, 2026 at 12:12:42PM +0000, Jon Hunter wrote:
>>> ...
>>
>>> Since this commit was added in Linux v6.18, I have been observing a suspend
>>> test failures on some of our boards. The suspend test suspends the devices
>>> for 20 secs and before this change the board would resume in about ~27 secs
>>> (including the 20 sec sleep). After this change the board would take over 80
>>> secs to resume and this triggered a failure.
>>>
>>> Looking at the logs, I can see it is the NVMe device on the board that is
>>> having an issue, and I see the reset failing ...
>>>
>>>   [  945.754939] r8169 0007:01:00.0 enP7p1s0: Link is Up - 1Gbps/Full -
>>>    flow control rx/tx
>>>   [ 1002.467432] nvme nvme0: I/O tag 12 (400c) opcode 0x9 (Admin Cmd) QID
>>>    0 timeout, reset controller
>>>   [ 1002.493713] nvme nvme0: 12/0/0 default/read/poll queues
>>>   [ 1003.050448] nvme nvme0: ctrl state 1 is not RESETTING
>>>   [ 1003.050481] OOM killer enabled.
>>>   [ 1003.054035] nvme nvme0: Disabling device after reset failure: -19
>>>
>>>  From the above timestamps the delay is coming from the NVMe. I see this
>>> issue on several boards with different NVMe devices and I can workaround
>>> this by disabling ASPM L0/L1 for these devices ...
>>>
>>>   DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5011, quirk_disable_aspm_l0s_l1);
>>>   DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5036, quirk_disable_aspm_l0s_l1);
>>>   DECLARE_PCI_FIXUP_HEADER(0x1b4b, 0x1322, quirk_disable_aspm_l0s_l1);
>>>   DECLARE_PCI_FIXUP_HEADER(0xc0a9, 0x540a, quirk_disable_aspm_l0s_l1);
>>>
>>> I am curious if you have seen any similar issues?
>>>
>>> Other PCIe devices seem to be OK (like the realtek r8169) but just
>>> the NVMe is having issues. So I am trying to figure out the best way
>>> to resolve this?
>>
>> For context, "this commit" refers to f3ac2ff14834, modified by
>> df5192d9bb0e:
>>
>>    f3ac2ff14834 ("PCI/ASPM: Enable all ClockPM and ASPM states for devicetree platforms")
>>    df5192d9bb0e ("PCI/ASPM: Enable only L0s and L1 for devicetree platforms")
>>
>> The fact that this suspend issue only affects NVMe reminds me of the
>> code in dw_pcie_suspend_noirq() [1] that bails out early if L1 is
>> enabled because of some NVMe expectation:
>>
>>    dw_pcie_suspend_noirq()
>>    {
>>      ...
>>      /*
>>       * If L1SS is supported, then do not put the link into L2 as some
>>       * devices such as NVMe expect low resume latency.
>>       */
>>      if (dw_pcie_readw_dbi(pci, offset + PCI_EXP_LNKCTL) & PCI_EXP_LNKCTL_ASPM_L1)
>>        return 0;
>>      ...
>>
>> That suggests there's some NVMe/ASPM interaction that the PCI core
>> doesn't understand yet.
>>
> 
> We have this check in place since NVMe driver keeps the device in D0 and expects
> the link to be in L1ss on platforms not passing below checks:
> 
>          if (pm_suspend_via_firmware() || !ctrl->npss ||
>              !pcie_aspm_enabled(pdev) ||
>              (ndev->ctrl.quirks & NVME_QUIRK_SIMPLE_SUSPEND))
> 
> Since the majority of the DWC platforms do not pass the above checks, we don't
> transition the device to D3Cold or link to L2/L3 in dw_pcie_suspend_noirq() if
> the link is in L1ss. Though I think we should be checking for D0 state instead
> of L1ss here.
> 
> I think what is going on here is that since before commits f3ac2ff14834 and
> df5192d9bb0e, !pcie_aspm_enabled() check was passing as ASPM was not enabled for
> the device (and upstream port) and after those commits, this check is not
> passing and the NVMe driver is not shutting down the controller and expects the
> link to be in L0/L1ss. But the Tegra controller driver initiates L2/L3
> transition, and also turns off the device. So all the NVMe context is lost
> during suspend and while resuming, the NVMe driver got confused due to lost
> context.
> 
> Jon, could you please try the below hack and see if it fixes the issue?
> 
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index 0e4caeab739c..4b8d261117f5 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -3723,7 +3723,7 @@ static int nvme_suspend(struct device *dev)
>           * state (which may not be possible if the link is up).
>           */
>          if (pm_suspend_via_firmware() || !ctrl->npss ||
> -           !pcie_aspm_enabled(pdev) ||
> +           pcie_aspm_enabled(pdev) ||
>              (ndev->ctrl.quirks & NVME_QUIRK_SIMPLE_SUSPEND))
>                  return nvme_disable_prepare_reset(ndev, true);
>   
> This will confirm whether the issue is due to Tegra controller driver breaking
> the NVMe driver assumption or not.

Yes that appears to be working! I will test some more boards to confirm.

Cheers
Jon

-- 
nvpublic

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Jon Hunter]

On 22/01/2026 19:14, Jon Hunter wrote:

...

>> I think what is going on here is that since before commits 
>> f3ac2ff14834 and
>> df5192d9bb0e, !pcie_aspm_enabled() check was passing as ASPM was not 
>> enabled for
>> the device (and upstream port) and after those commits, this check is not
>> passing and the NVMe driver is not shutting down the controller and 
>> expects the
>> link to be in L0/L1ss. But the Tegra controller driver initiates L2/L3
>> transition, and also turns off the device. So all the NVMe context is 
>> lost
>> during suspend and while resuming, the NVMe driver got confused due to 
>> lost
>> context.
>>
>> Jon, could you please try the below hack and see if it fixes the issue?
>>
>> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
>> index 0e4caeab739c..4b8d261117f5 100644
>> --- a/drivers/nvme/host/pci.c
>> +++ b/drivers/nvme/host/pci.c
>> @@ -3723,7 +3723,7 @@ static int nvme_suspend(struct device *dev)
>>           * state (which may not be possible if the link is up).
>>           */
>>          if (pm_suspend_via_firmware() || !ctrl->npss ||
>> -           !pcie_aspm_enabled(pdev) ||
>> +           pcie_aspm_enabled(pdev) ||
>>              (ndev->ctrl.quirks & NVME_QUIRK_SIMPLE_SUSPEND))
>>                  return nvme_disable_prepare_reset(ndev, true);
>> This will confirm whether the issue is due to Tegra controller driver 
>> breaking
>> the NVMe driver assumption or not.
> 
> Yes that appears to be working! I will test some more boards to confirm.

So yes with the above all boards appear to be working fine.

How is this usually coordinated between the NVMe driver and Host 
controller driver? It is not clear to me exactly where the problem is 
and if the NVMe is not shutting down, then what should be preventing the 
Host controller from shutting down.

Thanks
Jon

-- 
nvpublic

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Manivannan Sadhasivam]

+ Krishna

On Fri, Jan 23, 2026 at 10:55:28AM +0000, Jon Hunter wrote:
> 
> On 22/01/2026 19:14, Jon Hunter wrote:
> 
> ...
> 
> > > I think what is going on here is that since before commits
> > > f3ac2ff14834 and
> > > df5192d9bb0e, !pcie_aspm_enabled() check was passing as ASPM was not
> > > enabled for
> > > the device (and upstream port) and after those commits, this check is not
> > > passing and the NVMe driver is not shutting down the controller and
> > > expects the
> > > link to be in L0/L1ss. But the Tegra controller driver initiates L2/L3
> > > transition, and also turns off the device. So all the NVMe context
> > > is lost
> > > during suspend and while resuming, the NVMe driver got confused due
> > > to lost
> > > context.
> > > 
> > > Jon, could you please try the below hack and see if it fixes the issue?
> > > 
> > > diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> > > index 0e4caeab739c..4b8d261117f5 100644
> > > --- a/drivers/nvme/host/pci.c
> > > +++ b/drivers/nvme/host/pci.c
> > > @@ -3723,7 +3723,7 @@ static int nvme_suspend(struct device *dev)
> > >           * state (which may not be possible if the link is up).
> > >           */
> > >          if (pm_suspend_via_firmware() || !ctrl->npss ||
> > > -           !pcie_aspm_enabled(pdev) ||
> > > +           pcie_aspm_enabled(pdev) ||
> > >              (ndev->ctrl.quirks & NVME_QUIRK_SIMPLE_SUSPEND))
> > >                  return nvme_disable_prepare_reset(ndev, true);
> > > This will confirm whether the issue is due to Tegra controller
> > > driver breaking
> > > the NVMe driver assumption or not.
> > 
> > Yes that appears to be working! I will test some more boards to confirm.
> 
> So yes with the above all boards appear to be working fine.
> 
> How is this usually coordinated between the NVMe driver and Host controller
> driver? It is not clear to me exactly where the problem is and if the NVMe
> is not shutting down, then what should be preventing the Host controller
> from shutting down.
> 

Well if the NVMe driver is not shutting down the device, then it expects the
device to be in APST (NVMe low power state if supported) state and retain all
the context across the suspend/resume cycle.

But if the host controller powers down the device, then during resume, the
device will start afresh and would've lost all the context (like queue info
etc..). So when the NVMe driver resumes, it would expect the device to retain
the context and try to use the device as such. But it won't work as the device
will be in an unconfigured state and you'll see failures as you reported.

Apparently, most host controller drivers never cared about it because either
they were not tested with NVMe or they haven't enabled ASPM before. So the NVMe
driver ended up shutting down the controller during suspend. But since we
started enabling ASPM by default since 6.18, this issue is being uncovered.

So to properly fix it, we need the controller drivers to perform below checks
for all devices under the Root bus(ses) before initiating D3Cold:

1. Check if the device state is D3Hot. If it is not D3Hot, then the device is
expected to stay in the current D-state by the client driver, so D3Cold should
not be initiated.

2. Check if the device is wakeup capable. If it is, then check if it can support
wakeup from D3Cold (with WAKE#).

Only if both conditions are satisfied for all the devices under the Root busses,
then the host controller driver should initiate D3Cold sequence.

Krishna is going to post a patch that performs the above checks for
pcie-designware-host driver. But since the above checks are platform agnostic,
we should introduce a helper and resuse it across other controllers as well.

Hope this clarifies.

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Jon Hunter]

On 23/01/2026 13:56, Manivannan Sadhasivam wrote:
> + Krishna
> 
> On Fri, Jan 23, 2026 at 10:55:28AM +0000, Jon Hunter wrote:
>>
>> On 22/01/2026 19:14, Jon Hunter wrote:
>>
>> ...
>>
>>>> I think what is going on here is that since before commits
>>>> f3ac2ff14834 and
>>>> df5192d9bb0e, !pcie_aspm_enabled() check was passing as ASPM was not
>>>> enabled for
>>>> the device (and upstream port) and after those commits, this check is not
>>>> passing and the NVMe driver is not shutting down the controller and
>>>> expects the
>>>> link to be in L0/L1ss. But the Tegra controller driver initiates L2/L3
>>>> transition, and also turns off the device. So all the NVMe context
>>>> is lost
>>>> during suspend and while resuming, the NVMe driver got confused due
>>>> to lost
>>>> context.
>>>>
>>>> Jon, could you please try the below hack and see if it fixes the issue?
>>>>
>>>> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
>>>> index 0e4caeab739c..4b8d261117f5 100644
>>>> --- a/drivers/nvme/host/pci.c
>>>> +++ b/drivers/nvme/host/pci.c
>>>> @@ -3723,7 +3723,7 @@ static int nvme_suspend(struct device *dev)
>>>>            * state (which may not be possible if the link is up).
>>>>            */
>>>>           if (pm_suspend_via_firmware() || !ctrl->npss ||
>>>> -           !pcie_aspm_enabled(pdev) ||
>>>> +           pcie_aspm_enabled(pdev) ||
>>>>               (ndev->ctrl.quirks & NVME_QUIRK_SIMPLE_SUSPEND))
>>>>                   return nvme_disable_prepare_reset(ndev, true);
>>>> This will confirm whether the issue is due to Tegra controller
>>>> driver breaking
>>>> the NVMe driver assumption or not.
>>>
>>> Yes that appears to be working! I will test some more boards to confirm.
>>
>> So yes with the above all boards appear to be working fine.
>>
>> How is this usually coordinated between the NVMe driver and Host controller
>> driver? It is not clear to me exactly where the problem is and if the NVMe
>> is not shutting down, then what should be preventing the Host controller
>> from shutting down.
>>
> 
> Well if the NVMe driver is not shutting down the device, then it expects the
> device to be in APST (NVMe low power state if supported) state and retain all
> the context across the suspend/resume cycle.
> 
> But if the host controller powers down the device, then during resume, the
> device will start afresh and would've lost all the context (like queue info
> etc..). So when the NVMe driver resumes, it would expect the device to retain
> the context and try to use the device as such. But it won't work as the device
> will be in an unconfigured state and you'll see failures as you reported.
> 
> Apparently, most host controller drivers never cared about it because either
> they were not tested with NVMe or they haven't enabled ASPM before. So the NVMe
> driver ended up shutting down the controller during suspend. But since we
> started enabling ASPM by default since 6.18, this issue is being uncovered.
> 
> So to properly fix it, we need the controller drivers to perform below checks
> for all devices under the Root bus(ses) before initiating D3Cold:
> 
> 1. Check if the device state is D3Hot. If it is not D3Hot, then the device is
> expected to stay in the current D-state by the client driver, so D3Cold should
> not be initiated.
> 
> 2. Check if the device is wakeup capable. If it is, then check if it can support
> wakeup from D3Cold (with WAKE#).
> 
> Only if both conditions are satisfied for all the devices under the Root busses,
> then the host controller driver should initiate D3Cold sequence.
> 
> Krishna is going to post a patch that performs the above checks for
> pcie-designware-host driver. But since the above checks are platform agnostic,
> we should introduce a helper and resuse it across other controllers as well.
> 
> Hope this clarifies.

Yes it does. I am happy to test any patches for this.

Jon
-- 
nvpublic

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Jon Hunter]

Hi Mani, Krishna,

On 23/01/2026 13:56, Manivannan Sadhasivam wrote:

...

> So to properly fix it, we need the controller drivers to perform below checks
> for all devices under the Root bus(ses) before initiating D3Cold:
> 
> 1. Check if the device state is D3Hot. If it is not D3Hot, then the device is
> expected to stay in the current D-state by the client driver, so D3Cold should
> not be initiated.
> 
> 2. Check if the device is wakeup capable. If it is, then check if it can support
> wakeup from D3Cold (with WAKE#).
> 
> Only if both conditions are satisfied for all the devices under the Root busses,
> then the host controller driver should initiate D3Cold sequence.
> 
> Krishna is going to post a patch that performs the above checks for
> pcie-designware-host driver. But since the above checks are platform agnostic,
> we should introduce a helper and resuse it across other controllers as well.


Do you have a rough idea of when you will be posting patches for this?

Thanks
Jon

-- 
nvpublic

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Manivannan Sadhasivam]

On Mon, Feb 16, 2026 at 02:03:41PM +0000, Jon Hunter wrote:
> Hi Mani, Krishna,
> 
> On 23/01/2026 13:56, Manivannan Sadhasivam wrote:
> 
> ...
> 
> > So to properly fix it, we need the controller drivers to perform below checks
> > for all devices under the Root bus(ses) before initiating D3Cold:
> > 
> > 1. Check if the device state is D3Hot. If it is not D3Hot, then the device is
> > expected to stay in the current D-state by the client driver, so D3Cold should
> > not be initiated.
> > 
> > 2. Check if the device is wakeup capable. If it is, then check if it can support
> > wakeup from D3Cold (with WAKE#).
> > 
> > Only if both conditions are satisfied for all the devices under the Root busses,
> > then the host controller driver should initiate D3Cold sequence.
> > 
> > Krishna is going to post a patch that performs the above checks for
> > pcie-designware-host driver. But since the above checks are platform agnostic,
> > we should introduce a helper and resuse it across other controllers as well.
> 
> 
> Do you have a rough idea of when you will be posting patches for this?
> 

Krishna posted the series a couple of weeks before but forgot to CC you:
https://lore.kernel.org/linux-pci/20260128-d3cold-v1-0-dd8f3f0ce824@oss.qualcomm.com/

You are expected to use the helper pci_host_common_can_enter_d3cold() in the
suspend path.

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Jon Hunter]

On 16/02/2026 14:18, Manivannan Sadhasivam wrote:
> On Mon, Feb 16, 2026 at 02:03:41PM +0000, Jon Hunter wrote:
>> Hi Mani, Krishna,
>>
>> On 23/01/2026 13:56, Manivannan Sadhasivam wrote:
>>
>> ...
>>
>>> So to properly fix it, we need the controller drivers to perform below checks
>>> for all devices under the Root bus(ses) before initiating D3Cold:
>>>
>>> 1. Check if the device state is D3Hot. If it is not D3Hot, then the device is
>>> expected to stay in the current D-state by the client driver, so D3Cold should
>>> not be initiated.
>>>
>>> 2. Check if the device is wakeup capable. If it is, then check if it can support
>>> wakeup from D3Cold (with WAKE#).
>>>
>>> Only if both conditions are satisfied for all the devices under the Root busses,
>>> then the host controller driver should initiate D3Cold sequence.
>>>
>>> Krishna is going to post a patch that performs the above checks for
>>> pcie-designware-host driver. But since the above checks are platform agnostic,
>>> we should introduce a helper and resuse it across other controllers as well.
>>
>>
>> Do you have a rough idea of when you will be posting patches for this?
>>
> 
> Krishna posted the series a couple of weeks before but forgot to CC you:
> https://lore.kernel.org/linux-pci/20260128-d3cold-v1-0-dd8f3f0ce824@oss.qualcomm.com/
> 
> You are expected to use the helper pci_host_common_can_enter_d3cold() in the
> suspend path.

Thanks! I will take a look.

Jon

-- 
nvpublic

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Jon Hunter]

Hi Mani,

On 16/02/2026 14:35, Jon Hunter wrote:

...

>> Krishna posted the series a couple of weeks before but forgot to CC you:
>> https://lore.kernel.org/linux-pci/20260128-d3cold-v1-0- 
>> dd8f3f0ce824@oss.qualcomm.com/
>>
>> You are expected to use the helper pci_host_common_can_enter_d3cold() 
>> in the
>> suspend path.


I have been playing around with this, but so far I have not got anything
to work. Right now I have just made the following change (note that this
is based upon Manikanta's fixes series [0]) ...

diff --git a/drivers/pci/controller/dwc/pcie-tegra194.c b/drivers/pci/controller/dwc/pcie-tegra194.c
index 9883d14f7f97..9f88e4c1db08 100644
--- a/drivers/pci/controller/dwc/pcie-tegra194.c
+++ b/drivers/pci/controller/dwc/pcie-tegra194.c
@@ -2311,6 +2311,7 @@ static int tegra_pcie_dw_suspend_late(struct device *dev)
  static int tegra_pcie_dw_suspend_noirq(struct device *dev)
  {
         struct tegra_pcie_dw *pcie = dev_get_drvdata(dev);
+       struct dw_pcie *pci = &pcie->pci;
  
         if (pcie->of_data->mode == DW_PCIE_EP_TYPE)
                 return 0;
@@ -2318,6 +2319,9 @@ static int tegra_pcie_dw_suspend_noirq(struct device *dev)
         if (!pcie->link_state)
                 return 0;
  
+       if (!pci_host_common_can_enter_d3cold(pci->pp.bridge))
+               return 0;
+
         tegra_pcie_dw_pme_turnoff(pcie);
         tegra_pcie_unconfig_controller(pcie);


At first I was thinking that is we are not actually suspending the
controller we can skip the configuration of the controller in the
resume. However, if we skip configuring the controller in the resume
then the device does not resume at all. So right now I have the
above, but clearly this is not sufficient. The device resumes but
the NVMe is not working ...

  nvme nvme0: ctrl state 1 is not RESETTING
  nvme nvme0: Disabling device after reset failure: -19
  nvme nvme0: Ignoring bogus Namespace Identifiers
  Aborting journal on device nvme0n1p1-8.
  nvme0n1: detected capacity change from 0 to 976773168
  EXT4-fs error (device nvme0n1p1): __ext4_find_entry:1613: inode #18622533: comm (t-helper): reading directory lblock 0
  Buffer I/O error on dev nvme0n1p1, logical block 60850176, lost sync page write
  Buffer I/O error on dev nvme0n1p1, logical block 0, lost sync page write
  JBD2: I/O error when updating journal superblock for nvme0n1p1-8.
  EXT4-fs (nvme0n1p1): I/O error while writing superblock
  EXT4-fs error (device nvme0n1p1): ext4_journal_check_start:86: comm rs:main Q:Reg: Detected aborted journal
  Buffer I/O error on dev nvme0n1p1, logical block 0, lost sync page write
  EXT4-fs (nvme0n1p1): I/O error while writing superblock
  EXT4-fs (nvme0n1p1): Remounting filesystem read-only
  EXT4-fs (nvme0n1p1): shut down requested (2)

Is the above what you were thinking? Anything else I am missing?

Jon


[0] https://lore.kernel.org/linux-tegra/20260208180746.2024338-1-mmaddireddy@nvidia.com/
-- 
nvpublic

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Jon Hunter]

Hi Mani, Bjorn,

On 19/02/2026 17:42, Jon Hunter wrote:
> Hi Mani,
> 
> On 16/02/2026 14:35, Jon Hunter wrote:
> 
> ...
> 
>>> Krishna posted the series a couple of weeks before but forgot to CC you:
>>> https://lore.kernel.org/linux-pci/20260128-d3cold-v1-0- 
>>> dd8f3f0ce824@oss.qualcomm.com/
>>>
>>> You are expected to use the helper pci_host_common_can_enter_d3cold() 
>>> in the
>>> suspend path.
> 
> 
> I have been playing around with this, but so far I have not got anything
> to work. Right now I have just made the following change (note that this
> is based upon Manikanta's fixes series [0]) ...
> 
> diff --git a/drivers/pci/controller/dwc/pcie-tegra194.c b/drivers/pci/ 
> controller/dwc/pcie-tegra194.c
> index 9883d14f7f97..9f88e4c1db08 100644
> --- a/drivers/pci/controller/dwc/pcie-tegra194.c
> +++ b/drivers/pci/controller/dwc/pcie-tegra194.c
> @@ -2311,6 +2311,7 @@ static int tegra_pcie_dw_suspend_late(struct 
> device *dev)
>   static int tegra_pcie_dw_suspend_noirq(struct device *dev)
>   {
>          struct tegra_pcie_dw *pcie = dev_get_drvdata(dev);
> +       struct dw_pcie *pci = &pcie->pci;
> 
>          if (pcie->of_data->mode == DW_PCIE_EP_TYPE)
>                  return 0;
> @@ -2318,6 +2319,9 @@ static int tegra_pcie_dw_suspend_noirq(struct 
> device *dev)
>          if (!pcie->link_state)
>                  return 0;
> 
> +       if (!pci_host_common_can_enter_d3cold(pci->pp.bridge))
> +               return 0;
> +
>          tegra_pcie_dw_pme_turnoff(pcie);
>          tegra_pcie_unconfig_controller(pcie);
> 
> 
> At first I was thinking that is we are not actually suspending the
> controller we can skip the configuration of the controller in the
> resume. However, if we skip configuring the controller in the resume
> then the device does not resume at all. So right now I have the
> above, but clearly this is not sufficient. The device resumes but
> the NVMe is not working ...
> 
>   nvme nvme0: ctrl state 1 is not RESETTING
>   nvme nvme0: Disabling device after reset failure: -19
>   nvme nvme0: Ignoring bogus Namespace Identifiers
>   Aborting journal on device nvme0n1p1-8.
>   nvme0n1: detected capacity change from 0 to 976773168
>   EXT4-fs error (device nvme0n1p1): __ext4_find_entry:1613: inode 
> #18622533: comm (t-helper): reading directory lblock 0
>   Buffer I/O error on dev nvme0n1p1, logical block 60850176, lost sync 
> page write
>   Buffer I/O error on dev nvme0n1p1, logical block 0, lost sync page write
>   JBD2: I/O error when updating journal superblock for nvme0n1p1-8.
>   EXT4-fs (nvme0n1p1): I/O error while writing superblock
>   EXT4-fs error (device nvme0n1p1): ext4_journal_check_start:86: comm 
> rs:main Q:Reg: Detected aborted journal
>   Buffer I/O error on dev nvme0n1p1, logical block 0, lost sync page write
>   EXT4-fs (nvme0n1p1): I/O error while writing superblock
>   EXT4-fs (nvme0n1p1): Remounting filesystem read-only
>   EXT4-fs (nvme0n1p1): shut down requested (2)
> 
> Is the above what you were thinking? Anything else I am missing?

So NVMe is still broken for us and I admit, I don't fully understand the 
issue. However, it seems to me that this change is not working for all 
device-tree platforms as intended. So for now, would it be acceptable to 
add a callback function for drivers such as the Tegra194 PCIe driver to 
opt out of this? This would at least allow NVMe to work as it was before.

Thanks
Jon

-- 
nvpublic

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Manivannan Sadhasivam]

On Thu, Feb 26, 2026 at 10:34:18AM +0000, Jon Hunter wrote:
> Hi Mani, Bjorn,
> 
> On 19/02/2026 17:42, Jon Hunter wrote:
> > Hi Mani,
> > 
> > On 16/02/2026 14:35, Jon Hunter wrote:
> > 
> > ...
> > 
> > > > Krishna posted the series a couple of weeks before but forgot to CC you:
> > > > https://lore.kernel.org/linux-pci/20260128-d3cold-v1-0-
> > > > dd8f3f0ce824@oss.qualcomm.com/
> > > > 
> > > > You are expected to use the helper
> > > > pci_host_common_can_enter_d3cold() in the
> > > > suspend path.
> > 
> > 
> > I have been playing around with this, but so far I have not got anything
> > to work. Right now I have just made the following change (note that this
> > is based upon Manikanta's fixes series [0]) ...
> > 
> > diff --git a/drivers/pci/controller/dwc/pcie-tegra194.c b/drivers/pci/
> > controller/dwc/pcie-tegra194.c
> > index 9883d14f7f97..9f88e4c1db08 100644
> > --- a/drivers/pci/controller/dwc/pcie-tegra194.c
> > +++ b/drivers/pci/controller/dwc/pcie-tegra194.c
> > @@ -2311,6 +2311,7 @@ static int tegra_pcie_dw_suspend_late(struct
> > device *dev)
> >   static int tegra_pcie_dw_suspend_noirq(struct device *dev)
> >   {
> >          struct tegra_pcie_dw *pcie = dev_get_drvdata(dev);
> > +       struct dw_pcie *pci = &pcie->pci;
> > 
> >          if (pcie->of_data->mode == DW_PCIE_EP_TYPE)
> >                  return 0;
> > @@ -2318,6 +2319,9 @@ static int tegra_pcie_dw_suspend_noirq(struct
> > device *dev)
> >          if (!pcie->link_state)
> >                  return 0;
> > 
> > +       if (!pci_host_common_can_enter_d3cold(pci->pp.bridge))
> > +               return 0;
> > +
> >          tegra_pcie_dw_pme_turnoff(pcie);
> >          tegra_pcie_unconfig_controller(pcie);
> > 
> > 
> > At first I was thinking that is we are not actually suspending the
> > controller we can skip the configuration of the controller in the
> > resume. However, if we skip configuring the controller in the resume
> > then the device does not resume at all. So right now I have the
> > above, but clearly this is not sufficient. The device resumes but
> > the NVMe is not working ...
> > 
> >   nvme nvme0: ctrl state 1 is not RESETTING
> >   nvme nvme0: Disabling device after reset failure: -19
> >   nvme nvme0: Ignoring bogus Namespace Identifiers
> >   Aborting journal on device nvme0n1p1-8.
> >   nvme0n1: detected capacity change from 0 to 976773168
> >   EXT4-fs error (device nvme0n1p1): __ext4_find_entry:1613: inode
> > #18622533: comm (t-helper): reading directory lblock 0
> >   Buffer I/O error on dev nvme0n1p1, logical block 60850176, lost sync
> > page write
> >   Buffer I/O error on dev nvme0n1p1, logical block 0, lost sync page write
> >   JBD2: I/O error when updating journal superblock for nvme0n1p1-8.
> >   EXT4-fs (nvme0n1p1): I/O error while writing superblock
> >   EXT4-fs error (device nvme0n1p1): ext4_journal_check_start:86: comm
> > rs:main Q:Reg: Detected aborted journal
> >   Buffer I/O error on dev nvme0n1p1, logical block 0, lost sync page write
> >   EXT4-fs (nvme0n1p1): I/O error while writing superblock
> >   EXT4-fs (nvme0n1p1): Remounting filesystem read-only
> >   EXT4-fs (nvme0n1p1): shut down requested (2)
> > 
> > Is the above what you were thinking? Anything else I am missing?
> 
> So NVMe is still broken for us and I admit, I don't fully understand the
> issue. However, it seems to me that this change is not working for all
> device-tree platforms as intended. So for now, would it be acceptable to add
> a callback function for drivers such as the Tegra194 PCIe driver to opt out
> of this? This would at least allow NVMe to work as it was before.
> 

Since we know that ASPM is the issue on your platform and the failure also
confirms that ASPM was never enabled before, I'd suggest disabling ASPM for the
Root Port as a workaround:

```
diff --git a/drivers/pci/controller/dwc/pcie-tegra194.c b/drivers/pci/controller/dwc/pcie-tegra194.c
index 06571d806ab3..f504b4ffbcb6 100644
--- a/drivers/pci/controller/dwc/pcie-tegra194.c
+++ b/drivers/pci/controller/dwc/pcie-tegra194.c
@@ -2499,6 +2499,13 @@ module_platform_driver(tegra_pcie_dw_driver);
 
 MODULE_DEVICE_TABLE(of, tegra_pcie_dw_of_match);
 
+static void tegra_pcie_quirk_disable_aspm(struct pci_dev *dev)
+{
+       pcie_aspm_remove_cap(dev, PCI_EXP_LNKCAP_ASPM_L1 |
+                                 PCI_EXP_LNKCAP_ASPM_L0S);
+}
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID, tegra_pcie_quirk_disable_aspm);
+
 MODULE_AUTHOR("Vidya Sagar <vidyas@nvidia.com>");
 MODULE_DESCRIPTION("NVIDIA PCIe host controller driver");
 MODULE_LICENSE("GPL v2");
```

You can use specific Root Port IDs or PCI_ANY_ID depending on the impact. We can
also work on fixing the actual issue parallelly.

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Jon Hunter]

On 26/02/2026 11:08, Manivannan Sadhasivam wrote:

...

> Since we know that ASPM is the issue on your platform and the failure also
> confirms that ASPM was never enabled before, I'd suggest disabling ASPM for the
> Root Port as a workaround:
> 
> ```
> diff --git a/drivers/pci/controller/dwc/pcie-tegra194.c b/drivers/pci/controller/dwc/pcie-tegra194.c
> index 06571d806ab3..f504b4ffbcb6 100644
> --- a/drivers/pci/controller/dwc/pcie-tegra194.c
> +++ b/drivers/pci/controller/dwc/pcie-tegra194.c
> @@ -2499,6 +2499,13 @@ module_platform_driver(tegra_pcie_dw_driver);
>   
>   MODULE_DEVICE_TABLE(of, tegra_pcie_dw_of_match);
>   
> +static void tegra_pcie_quirk_disable_aspm(struct pci_dev *dev)
> +{
> +       pcie_aspm_remove_cap(dev, PCI_EXP_LNKCAP_ASPM_L1 |
> +                                 PCI_EXP_LNKCAP_ASPM_L0S);
> +}
> +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID, tegra_pcie_quirk_disable_aspm);
> +
>   MODULE_AUTHOR("Vidya Sagar <vidyas@nvidia.com>");
>   MODULE_DESCRIPTION("NVIDIA PCIe host controller driver");
>   MODULE_LICENSE("GPL v2");
> ```
> 
> You can use specific Root Port IDs or PCI_ANY_ID depending on the impact. We can
> also work on fixing the actual issue parallelly.

Thanks. By default we are building the PCIe driver for Tegra as a module 
and so I am not sure we can use DECLARE_PCI_FIXUP_EARLY() right?

I was just thinking that in pcie_aspm_override_default_link_state() we 
just need a callback to specify the default ASPM override state?

Cheers
Jon

-- 
nvpublic

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Manivannan Sadhasivam]

On Thu, Feb 26, 2026 at 04:55:34PM +0000, Jon Hunter wrote:
> 
> On 26/02/2026 11:08, Manivannan Sadhasivam wrote:
> 
> ...
> 
> > Since we know that ASPM is the issue on your platform and the failure also
> > confirms that ASPM was never enabled before, I'd suggest disabling ASPM for the
> > Root Port as a workaround:
> > 
> > ```
> > diff --git a/drivers/pci/controller/dwc/pcie-tegra194.c b/drivers/pci/controller/dwc/pcie-tegra194.c
> > index 06571d806ab3..f504b4ffbcb6 100644
> > --- a/drivers/pci/controller/dwc/pcie-tegra194.c
> > +++ b/drivers/pci/controller/dwc/pcie-tegra194.c
> > @@ -2499,6 +2499,13 @@ module_platform_driver(tegra_pcie_dw_driver);
> >   MODULE_DEVICE_TABLE(of, tegra_pcie_dw_of_match);
> > +static void tegra_pcie_quirk_disable_aspm(struct pci_dev *dev)
> > +{
> > +       pcie_aspm_remove_cap(dev, PCI_EXP_LNKCAP_ASPM_L1 |
> > +                                 PCI_EXP_LNKCAP_ASPM_L0S);
> > +}
> > +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID, tegra_pcie_quirk_disable_aspm);
> > +
> >   MODULE_AUTHOR("Vidya Sagar <vidyas@nvidia.com>");
> >   MODULE_DESCRIPTION("NVIDIA PCIe host controller driver");
> >   MODULE_LICENSE("GPL v2");
> > ```
> > 
> > You can use specific Root Port IDs or PCI_ANY_ID depending on the impact. We can
> > also work on fixing the actual issue parallelly.
> 
> Thanks. By default we are building the PCIe driver for Tegra as a module and
> so I am not sure we can use DECLARE_PCI_FIXUP_EARLY() right?
> 

Ah, yes. We cannot use any of these DECLARE_PCI_FIXUP*() in a module anyway :/

> I was just thinking that in pcie_aspm_override_default_link_state() we just
> need a callback to specify the default ASPM override state?
> 

That looks like a dirty hack. Moreover, your platform works perfectly fine with
ASPM. So you should not be turning it off due to one driver behaving
erratically.

As I mentioned in [1], you should try to advertise the
PM_SUSPEND_FLAG_FW_SUSPEND flag for the platform as per [2].

FYI, this is a known issue that is plauging us for so long.
PM_SUSPEND_FLAG_FW_SUSPEND is the cleanest solution we came up with. But
unfortunately, we cannot use it across all of our Qcom SoCs due to firmware not
advertising S2RAM in some of them.

- Mani

[1] https://lore.kernel.org/linux-pci/kkly3z4durpagtenadvmzdpojlctachgfgi2fdapt6zthdl2gx@n2qhmlud2zb7/
[2] https://lore.kernel.org/all/20251231162126.7728-1-manivannan.sadhasivam@oss.qualcomm.com/

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Manivannan Sadhasivam]

On Thu, Feb 19, 2026 at 05:42:37PM +0000, Jon Hunter wrote:
> Hi Mani,
> 
> On 16/02/2026 14:35, Jon Hunter wrote:
> 
> ...
> 
> > > Krishna posted the series a couple of weeks before but forgot to CC you:
> > > https://lore.kernel.org/linux-pci/20260128-d3cold-v1-0-
> > > dd8f3f0ce824@oss.qualcomm.com/
> > > 
> > > You are expected to use the helper
> > > pci_host_common_can_enter_d3cold() in the
> > > suspend path.
> 
> 
> I have been playing around with this, but so far I have not got anything
> to work. Right now I have just made the following change (note that this
> is based upon Manikanta's fixes series [0]) ...
> 
> diff --git a/drivers/pci/controller/dwc/pcie-tegra194.c b/drivers/pci/controller/dwc/pcie-tegra194.c
> index 9883d14f7f97..9f88e4c1db08 100644
> --- a/drivers/pci/controller/dwc/pcie-tegra194.c
> +++ b/drivers/pci/controller/dwc/pcie-tegra194.c
> @@ -2311,6 +2311,7 @@ static int tegra_pcie_dw_suspend_late(struct device *dev)
>  static int tegra_pcie_dw_suspend_noirq(struct device *dev)
>  {
>         struct tegra_pcie_dw *pcie = dev_get_drvdata(dev);
> +       struct dw_pcie *pci = &pcie->pci;
>         if (pcie->of_data->mode == DW_PCIE_EP_TYPE)
>                 return 0;
> @@ -2318,6 +2319,9 @@ static int tegra_pcie_dw_suspend_noirq(struct device *dev)
>         if (!pcie->link_state)
>                 return 0;
> +       if (!pci_host_common_can_enter_d3cold(pci->pp.bridge))
> +               return 0;
> +
>         tegra_pcie_dw_pme_turnoff(pcie);
>         tegra_pcie_unconfig_controller(pcie);
> 
> 
> At first I was thinking that is we are not actually suspending the
> controller we can skip the configuration of the controller in the
> resume. However, if we skip configuring the controller in the resume
> then the device does not resume at all.

Device mean the 'host' here?

> So right now I have the
> above, but clearly this is not sufficient. The device resumes but
> the NVMe is not working ...
> 
>  nvme nvme0: ctrl state 1 is not RESETTING
>  nvme nvme0: Disabling device after reset failure: -19
>  nvme nvme0: Ignoring bogus Namespace Identifiers
>  Aborting journal on device nvme0n1p1-8.
>  nvme0n1: detected capacity change from 0 to 976773168
>  EXT4-fs error (device nvme0n1p1): __ext4_find_entry:1613: inode #18622533: comm (t-helper): reading directory lblock 0
>  Buffer I/O error on dev nvme0n1p1, logical block 60850176, lost sync page write
>  Buffer I/O error on dev nvme0n1p1, logical block 0, lost sync page write
>  JBD2: I/O error when updating journal superblock for nvme0n1p1-8.
>  EXT4-fs (nvme0n1p1): I/O error while writing superblock
>  EXT4-fs error (device nvme0n1p1): ext4_journal_check_start:86: comm rs:main Q:Reg: Detected aborted journal
>  Buffer I/O error on dev nvme0n1p1, logical block 0, lost sync page write
>  EXT4-fs (nvme0n1p1): I/O error while writing superblock
>  EXT4-fs (nvme0n1p1): Remounting filesystem read-only
>  EXT4-fs (nvme0n1p1): shut down requested (2)
> 
> Is the above what you were thinking? Anything else I am missing?
> 

I can't certainly know what is going wrong. If controller driver suspend is
skipped, then ideally the controller and the NVMe device should stay powered ON
during suspend. But if the platform pulls the plug at the end of suspend
(firmware, gdsc or some other entity), then all the context would be lost and
that might explain the failure because both the controller driver and NVMe
driver would expect the RC and NVMe to be active.

You can try commenting out the whole PM callbacks:
		// .pm = &tegra_pcie_dw_pm_ops

If the host itself doesn't resume, then it confirms that some other entity is
pulling the plug (which is common in ARM platforms). In that case, we have to
let the NVMe driver know about it so that it can shutdown the controller.

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Jon Hunter]

On 26/02/2026 11:16, Manivannan Sadhasivam wrote:

...

> I can't certainly know what is going wrong. If controller driver suspend is
> skipped, then ideally the controller and the NVMe device should stay powered ON
> during suspend. But if the platform pulls the plug at the end of suspend
> (firmware, gdsc or some other entity), then all the context would be lost and
> that might explain the failure because both the controller driver and NVMe
> driver would expect the RC and NVMe to be active.
> 
> You can try commenting out the whole PM callbacks:
> 		// .pm = &tegra_pcie_dw_pm_ops
> 
> If the host itself doesn't resume, then it confirms that some other entity is
> pulling the plug (which is common in ARM platforms). In that case, we have to
> let the NVMe driver know about it so that it can shutdown the controller.

For Tegra, we enter a deep low power state known as SC7 on suspend which 
does involve firmware. Nonetheless I tried for fun, but this breaks 
suspend completely.

Jon

-- 
nvpublic

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Manivannan Sadhasivam]

On Thu, Feb 26, 2026 at 04:52:57PM +0000, Jon Hunter wrote:
> 
> On 26/02/2026 11:16, Manivannan Sadhasivam wrote:
> 
> ...
> 
> > I can't certainly know what is going wrong. If controller driver suspend is
> > skipped, then ideally the controller and the NVMe device should stay powered ON
> > during suspend. But if the platform pulls the plug at the end of suspend
> > (firmware, gdsc or some other entity), then all the context would be lost and
> > that might explain the failure because both the controller driver and NVMe
> > driver would expect the RC and NVMe to be active.
> > 
> > You can try commenting out the whole PM callbacks:
> > 		// .pm = &tegra_pcie_dw_pm_ops
> > 
> > If the host itself doesn't resume, then it confirms that some other entity is
> > pulling the plug (which is common in ARM platforms). In that case, we have to
> > let the NVMe driver know about it so that it can shutdown the controller.
> 
> For Tegra, we enter a deep low power state known as SC7 on suspend which
> does involve firmware. Nonetheless I tried for fun, but this breaks suspend
> completely.
> 

Ah, this explains the problem. We also have a similar problem on our Qcom Auto
boards where the firmware completely shuts down the SoC and puts the DRAM in
self refresh mode. So NVMe driver never resumes properly. We tried multiple ways
to address this issue in the NVMe driver, but the NVMe maintainers rejected
every single one of them and asking for some API in the PCI or PM core to tell
the NVMe driver when to shutdown the device during suspend. But this turned out
to be not so trivial.

Another way to workaround this issue would be by calling
pm_set_suspend_via_firmware() from the driver that controls the entity doing
power management of the SoC (firmware). In your case, it is
drivers/soc/tegra/pmc.c?

In that case, you can use this patch as a reference:
https://lore.kernel.org/all/20251231162126.7728-1-manivannan.sadhasivam@oss.qualcomm.com

When pm_set_suspend_via_firmware() is set, NVMe driver assumes that the firmware
might pull the plug during suspend, so it shutdowns the controller completely
and brings it back from reset during resume.

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Jon Hunter]

On 03/03/2026 16:17, Manivannan Sadhasivam wrote:

...

>> For Tegra, we enter a deep low power state known as SC7 on suspend which
>> does involve firmware. Nonetheless I tried for fun, but this breaks suspend
>> completely.
>>
> 
> Ah, this explains the problem. We also have a similar problem on our Qcom Auto
> boards where the firmware completely shuts down the SoC and puts the DRAM in
> self refresh mode. So NVMe driver never resumes properly. We tried multiple ways
> to address this issue in the NVMe driver, but the NVMe maintainers rejected
> every single one of them and asking for some API in the PCI or PM core to tell
> the NVMe driver when to shutdown the device during suspend. But this turned out
> to be not so trivial.
> 
> Another way to workaround this issue would be by calling
> pm_set_suspend_via_firmware() from the driver that controls the entity doing
> power management of the SoC (firmware). In your case, it is
> drivers/soc/tegra/pmc.c?

Actually for newer devices it is PSCI and so ...

> In that case, you can use this patch as a reference:
> https://lore.kernel.org/all/20251231162126.7728-1-manivannan.sadhasivam@oss.qualcomm.com

This change fixes the problem as implemented. What is the status of the 
above? Any plans to get this merged?

Jon

-- 
nvpublic

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Manivannan Sadhasivam]

On Fri, Mar 06, 2026 at 04:03:35PM +0000, Jon Hunter wrote:
> 
> On 03/03/2026 16:17, Manivannan Sadhasivam wrote:
> 
> ...
> 
> > > For Tegra, we enter a deep low power state known as SC7 on suspend which
> > > does involve firmware. Nonetheless I tried for fun, but this breaks suspend
> > > completely.
> > > 
> > 
> > Ah, this explains the problem. We also have a similar problem on our Qcom Auto
> > boards where the firmware completely shuts down the SoC and puts the DRAM in
> > self refresh mode. So NVMe driver never resumes properly. We tried multiple ways
> > to address this issue in the NVMe driver, but the NVMe maintainers rejected
> > every single one of them and asking for some API in the PCI or PM core to tell
> > the NVMe driver when to shutdown the device during suspend. But this turned out
> > to be not so trivial.
> > 
> > Another way to workaround this issue would be by calling
> > pm_set_suspend_via_firmware() from the driver that controls the entity doing
> > power management of the SoC (firmware). In your case, it is
> > drivers/soc/tegra/pmc.c?
> 
> Actually for newer devices it is PSCI and so ...
> 
> > In that case, you can use this patch as a reference:
> > https://lore.kernel.org/all/20251231162126.7728-1-manivannan.sadhasivam@oss.qualcomm.com
> 
> This change fixes the problem as implemented. What is the status of the
> above? Any plans to get this merged?
> 

It didn't get any love so far. It'd help if you can give your tags to that
patch.

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Claudiu Beznea]

Hi,

On 1/22/26 19:01, Manivannan Sadhasivam wrote:
> On Thu, Jan 22, 2026 at 09:29:03AM -0600, Bjorn Helgaas wrote:
>> [+cc NVMe folks]
>>
>> On Thu, Jan 22, 2026 at 12:12:42PM +0000, Jon Hunter wrote:
>>> ...
>>
>>> Since this commit was added in Linux v6.18, I have been observing a suspend
>>> test failures on some of our boards. The suspend test suspends the devices
>>> for 20 secs and before this change the board would resume in about ~27 secs
>>> (including the 20 sec sleep). After this change the board would take over 80
>>> secs to resume and this triggered a failure.
>>>
>>> Looking at the logs, I can see it is the NVMe device on the board that is
>>> having an issue, and I see the reset failing ...
>>>
>>>   [  945.754939] r8169 0007:01:00.0 enP7p1s0: Link is Up - 1Gbps/Full -
>>>    flow control rx/tx
>>>   [ 1002.467432] nvme nvme0: I/O tag 12 (400c) opcode 0x9 (Admin Cmd) QID
>>>    0 timeout, reset controller
>>>   [ 1002.493713] nvme nvme0: 12/0/0 default/read/poll queues
>>>   [ 1003.050448] nvme nvme0: ctrl state 1 is not RESETTING
>>>   [ 1003.050481] OOM killer enabled.
>>>   [ 1003.054035] nvme nvme0: Disabling device after reset failure: -19
>>>
>>>  From the above timestamps the delay is coming from the NVMe. I see this
>>> issue on several boards with different NVMe devices and I can workaround
>>> this by disabling ASPM L0/L1 for these devices ...
>>>
>>>   DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5011, quirk_disable_aspm_l0s_l1);
>>>   DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5036, quirk_disable_aspm_l0s_l1);
>>>   DECLARE_PCI_FIXUP_HEADER(0x1b4b, 0x1322, quirk_disable_aspm_l0s_l1);
>>>   DECLARE_PCI_FIXUP_HEADER(0xc0a9, 0x540a, quirk_disable_aspm_l0s_l1);
>>>
>>> I am curious if you have seen any similar issues?
>>>
>>> Other PCIe devices seem to be OK (like the realtek r8169) but just
>>> the NVMe is having issues. So I am trying to figure out the best way
>>> to resolve this?
>>
>> For context, "this commit" refers to f3ac2ff14834, modified by
>> df5192d9bb0e:
>>
>>    f3ac2ff14834 ("PCI/ASPM: Enable all ClockPM and ASPM states for devicetree platforms")
>>    df5192d9bb0e ("PCI/ASPM: Enable only L0s and L1 for devicetree platforms")
>>
>> The fact that this suspend issue only affects NVMe reminds me of the
>> code in dw_pcie_suspend_noirq() [1] that bails out early if L1 is
>> enabled because of some NVMe expectation:
>>
>>    dw_pcie_suspend_noirq()
>>    {
>>      ...
>>      /*
>>       * If L1SS is supported, then do not put the link into L2 as some
>>       * devices such as NVMe expect low resume latency.
>>       */
>>      if (dw_pcie_readw_dbi(pci, offset + PCI_EXP_LNKCTL) & PCI_EXP_LNKCTL_ASPM_L1)
>>        return 0;
>>      ...
>>
>> That suggests there's some NVMe/ASPM interaction that the PCI core
>> doesn't understand yet.
>>
> 
> We have this check in place since NVMe driver keeps the device in D0 and expects
> the link to be in L1ss on platforms not passing below checks:
> 
>          if (pm_suspend_via_firmware() || !ctrl->npss ||
>              !pcie_aspm_enabled(pdev) ||
>              (ndev->ctrl.quirks & NVME_QUIRK_SIMPLE_SUSPEND))
> 

We noticed a similar issue with the Renesas RZ/G3S host driver and NVMe devices. 
We currently have 2 SoCs where we identified this problem (RZ/G3S and RZ/G3E), 
both present on SoM modules, and the SoM modules could be connected to the same 
carrier board where the PCIe signals are routed and connectors exists. On the 
carrier board we have 2 connectors were we can attach NVMe devices, one M.2 Key 
B and one PCIe x4 connector 
(https://www.amphenol-cs.com/product/10061913111plf.html).

The issue described in this thread is reproducible for us only after suspend and 
only for the NVMe device connected to the the PCIe x4 connector. The device is 
working correctly just after boot. On suspend, power to the most SoC components 
(including PCIe) is lost but the endpoints remains powered.

The issue is not reproducible if the following command is executed before 
suspend: echo performance > /sys/module/pcie_aspm/parameters/policy

The difference we identified in terms of signals connected from the SoC to the 
on board connectors relies in the CLKREQ#. This signal is only connected to the 
PCIe x4 slot.

On RZ/G3E the CLKREQ# is configured as a individual GPIO pin. On RZ/G3S it is 
muxed by the pin controller with PCIe function. We tried on RZ/G3E to not 
configure the CLKREQ# pin at all and with this the NVMe connected on the PCIe x4 
slot started to work even after suspend. We cannot reproduce the same behavior 
on RZ/G3S.

Initially, we considered we might have to update the existing code to do 
specific configuration for the boards were CLKREQ# is not connected (through 
supports-clkreq DT property that some controllers are using).

Currently, the manual is unclear on how to control CLKREQ#.

Apart from the suggestions mentioned in [1], could you please let me know if you 
have any others?

Thank you,
Claudiu

[1] 
https://lore.kernel.org/all/unc5zefwndgcv7wufaezz3gkg3qtaymkjlmymhyqdqwzn3wybl@ow2rhbyt772h/

Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms

[Manivannan Sadhasivam]

On Mon, Feb 16, 2026 at 07:19:46PM +0200, Claudiu Beznea wrote:
> Hi,
> 
> On 1/22/26 19:01, Manivannan Sadhasivam wrote:
> > On Thu, Jan 22, 2026 at 09:29:03AM -0600, Bjorn Helgaas wrote:
> > > [+cc NVMe folks]
> > > 
> > > On Thu, Jan 22, 2026 at 12:12:42PM +0000, Jon Hunter wrote:
> > > > ...
> > > 
> > > > Since this commit was added in Linux v6.18, I have been observing a suspend
> > > > test failures on some of our boards. The suspend test suspends the devices
> > > > for 20 secs and before this change the board would resume in about ~27 secs
> > > > (including the 20 sec sleep). After this change the board would take over 80
> > > > secs to resume and this triggered a failure.
> > > > 
> > > > Looking at the logs, I can see it is the NVMe device on the board that is
> > > > having an issue, and I see the reset failing ...
> > > > 
> > > >   [  945.754939] r8169 0007:01:00.0 enP7p1s0: Link is Up - 1Gbps/Full -
> > > >    flow control rx/tx
> > > >   [ 1002.467432] nvme nvme0: I/O tag 12 (400c) opcode 0x9 (Admin Cmd) QID
> > > >    0 timeout, reset controller
> > > >   [ 1002.493713] nvme nvme0: 12/0/0 default/read/poll queues
> > > >   [ 1003.050448] nvme nvme0: ctrl state 1 is not RESETTING
> > > >   [ 1003.050481] OOM killer enabled.
> > > >   [ 1003.054035] nvme nvme0: Disabling device after reset failure: -19
> > > > 
> > > >  From the above timestamps the delay is coming from the NVMe. I see this
> > > > issue on several boards with different NVMe devices and I can workaround
> > > > this by disabling ASPM L0/L1 for these devices ...
> > > > 
> > > >   DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5011, quirk_disable_aspm_l0s_l1);
> > > >   DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5036, quirk_disable_aspm_l0s_l1);
> > > >   DECLARE_PCI_FIXUP_HEADER(0x1b4b, 0x1322, quirk_disable_aspm_l0s_l1);
> > > >   DECLARE_PCI_FIXUP_HEADER(0xc0a9, 0x540a, quirk_disable_aspm_l0s_l1);
> > > > 
> > > > I am curious if you have seen any similar issues?
> > > > 
> > > > Other PCIe devices seem to be OK (like the realtek r8169) but just
> > > > the NVMe is having issues. So I am trying to figure out the best way
> > > > to resolve this?
> > > 
> > > For context, "this commit" refers to f3ac2ff14834, modified by
> > > df5192d9bb0e:
> > > 
> > >    f3ac2ff14834 ("PCI/ASPM: Enable all ClockPM and ASPM states for devicetree platforms")
> > >    df5192d9bb0e ("PCI/ASPM: Enable only L0s and L1 for devicetree platforms")
> > > 
> > > The fact that this suspend issue only affects NVMe reminds me of the
> > > code in dw_pcie_suspend_noirq() [1] that bails out early if L1 is
> > > enabled because of some NVMe expectation:
> > > 
> > >    dw_pcie_suspend_noirq()
> > >    {
> > >      ...
> > >      /*
> > >       * If L1SS is supported, then do not put the link into L2 as some
> > >       * devices such as NVMe expect low resume latency.
> > >       */
> > >      if (dw_pcie_readw_dbi(pci, offset + PCI_EXP_LNKCTL) & PCI_EXP_LNKCTL_ASPM_L1)
> > >        return 0;
> > >      ...
> > > 
> > > That suggests there's some NVMe/ASPM interaction that the PCI core
> > > doesn't understand yet.
> > > 
> > 
> > We have this check in place since NVMe driver keeps the device in D0 and expects
> > the link to be in L1ss on platforms not passing below checks:
> > 
> >          if (pm_suspend_via_firmware() || !ctrl->npss ||
> >              !pcie_aspm_enabled(pdev) ||
> >              (ndev->ctrl.quirks & NVME_QUIRK_SIMPLE_SUSPEND))
> > 
> 
> We noticed a similar issue with the Renesas RZ/G3S host driver and NVMe
> devices. We currently have 2 SoCs where we identified this problem (RZ/G3S
> and RZ/G3E), both present on SoM modules, and the SoM modules could be
> connected to the same carrier board where the PCIe signals are routed and
> connectors exists. On the carrier board we have 2 connectors were we can
> attach NVMe devices, one M.2 Key B and one PCIe x4 connector
> (https://www.amphenol-cs.com/product/10061913111plf.html).
> 
> The issue described in this thread is reproducible for us only after suspend
> and only for the NVMe device connected to the the PCIe x4 connector. The
> device is working correctly just after boot. On suspend, power to the most
> SoC components (including PCIe) is lost but the endpoints remains powered.
> 
> The issue is not reproducible if the following command is executed before
> suspend: echo performance > /sys/module/pcie_aspm/parameters/policy
> 
> The difference we identified in terms of signals connected from the SoC to
> the on board connectors relies in the CLKREQ#. This signal is only connected
> to the PCIe x4 slot.
> 
> On RZ/G3E the CLKREQ# is configured as a individual GPIO pin. On RZ/G3S it
> is muxed by the pin controller with PCIe function. We tried on RZ/G3E to not
> configure the CLKREQ# pin at all and with this the NVMe connected on the
> PCIe x4 slot started to work even after suspend. We cannot reproduce the
> same behavior on RZ/G3S.
> 
> Initially, we considered we might have to update the existing code to do
> specific configuration for the boards were CLKREQ# is not connected (through
> supports-clkreq DT property that some controllers are using).
> 
> Currently, the manual is unclear on how to control CLKREQ#.
> 
> Apart from the suggestions mentioned in [1], could you please let me know if
> you have any others?
> 

If you do not know how to control CLKREQ# and it is broken, then disable L1 PM
Substates in the Root Port L1 PM Susbstates Capabilities register during
controller driver probe.

- Mani

-- 
மணிவண்ணன் சதாசிவம்