[bug report] nvme-pci: NVMe lost after S3 resume, CSTS reads all-ones for a few 100 ms

[bug report] nvme-pci: NVMe lost after S3 resume, CSTS reads all-ones for a few 100 ms

[Moritz Hiller]

Hi,

on my desktop the root NVMe drive sporadically does not come back after
S3 resume (roughly 5-10% of resumes in automated runs):

  nvme nvme0: Disabling device after reset failure: -19

The single readl(CSTS) in nvme_pci_enable() returns all-ones about
200 ms after resume. The controller is not dead, it just needs a few
hundred ms more. A small wait loop (diff below) fixes it here, tested
on CachyOS kernels and on vanilla v7.4-rc2. I am not asking for the
diff to be merged as is, I would like to know where this should be
fixed properly.

System:
- ASUS ROG STRIX X570-F GAMING (BIOS 5044), Ryzen 7 5800X, S3 (deep)
- Samsung 9100 PRO 2TB [144d:a810], FW 0B2QNXH7, at 0000:01:00.0,
  directly behind root port 0000:00:01.1 [1022:1483]

Observations:
- pci_pm_resume_noirq takes ~0.3 ms for the SSD, config space answers.
- At the failing CSTS read config space is intact, the link is up
  (16 GT/s x4), the device is in D0, no AER errors:

  vendor=144d cmd=0006 (saved 0002) bar0=fcf00004 (saved fcf00004)
  pmcsr=0008 lnksta=1044 rp_lnksta=3044

- sysfs remove + rescan brings the drive back after ~3 s.
- ASPM is off on this link, pcie_aspm=off and
  nvme_core.default_ps_max_latency_us=0 make no difference.
- Unpatched, 2 of 15 and 2 of 24 cycles failed in automated runs
  (90 s sleep, RTC wake). With the wait, no cycle failed (30/30 on
  CachyOS, 15/15 on v7.4-rc2); the controller answered after
  200-400 ms whenever the wait triggered.

My reading: pci_power_up() finds PMCSR already in D0 and does not
wait, and pci_bridge_wait_for_secondary_bus() only runs after D3cold
and only checks config space. So nothing waits for MMIO to come back,
and nvme_pci_enable() gives up on the first all-ones read.

The diff polls CSTS every 100 ms for up to 10 s, only if
NVME_CC_SHN_NORMAL is still set (the was_suspend test in
nvme_reset_work()) and while pci_device_is_present(). Probe and
runtime resets are unchanged. Known downsides:
- The 10 s bound is arbitrary.
- It also triggers on thaw/restore after hibernation (same bit set).
- The wait sleeps with shutdown_lock held, as nvme_wait_ready() already
  does in the same call chain.
- A controller that answers config space but is really dead after
  resume is declared dead 10 s later than today.
- A wait that times out is not logged.

Should this go into nvme-pci, the PCI core, or a quirk for this board
or drive? I can provide lspci -vvv, full dmesg or test patches.

The problem has existed on this machine for about a year, possibly
also with the previous SSD in this slot. It was hard to pin down: when
it happens the root filesystem is gone, so nothing gets written to
disk. I only got the first log of a failure in August 2026 via
netconsole, and every failure captured since then looks the same.

Disclosure: I am not a kernel developer. Log analysis, code reading
and the diff were done with an LLM assistant (Claude); I ran all tests
on my hardware. I can send patch with Signed-off-by if useful.

Thanks,
Moritz Hiller

---8<--- proof of concept, against v7.3.4 ---8<---

--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -7,6 +7,7 @@
 #include <linux/acpi.h>
 #include <linux/async.h>
 #include <linux/blkdev.h>
+#include <linux/delay.h>
 #include <linux/blk-mq-dma.h>
 #include <linux/blk-integrity.h>
 #include <linux/dmi.h>
@@ -3206,6 +3207,38 @@
 	return true;
 }
 
+/*
+ * A controller that was shut down for suspend may have lost power while the
+ * PCI core still treats it as D3hot, so nothing waited for it to come back.
+ * Config space can already answer while MMIO still reads all-ones. Give such
+ * a controller some time instead of declaring it dead on the first read.
+ */
+#define NVME_RESUME_MMIO_TIMEOUT_MS	10000
+#define NVME_RESUME_MMIO_POLL_MS	100
+
+static bool nvme_pci_wait_mmio_after_suspend(struct nvme_dev *dev)
+{
+	struct pci_dev *pdev = to_pci_dev(dev->dev);
+	unsigned int waited = 0;
+
+	/* Same test nvme_reset_work() uses to detect a resume. */
+	if (!(dev->ctrl.ctrl_config & NVME_CC_SHN_NORMAL))
+		return false;
+
+	while (waited < NVME_RESUME_MMIO_TIMEOUT_MS &&
+	       pci_device_is_present(pdev)) {
+		msleep(NVME_RESUME_MMIO_POLL_MS);
+		waited += NVME_RESUME_MMIO_POLL_MS;
+		if (readl(dev->bar + NVME_REG_CSTS) != -1) {
+			dev_warn(dev->ctrl.device,
+				 "CSTS read all-ones after resume, controller ready after %u ms\n",
+				 waited);
+			return true;
+		}
+	}
+	return false;
+}
+
 static int nvme_pci_enable(struct nvme_dev *dev)
 {
 	int result = -ENOMEM;
@@ -3217,7 +3250,8 @@
 
 	pci_set_master(pdev);
 
-	if (readl(dev->bar + NVME_REG_CSTS) == -1) {
+	if (readl(dev->bar + NVME_REG_CSTS) == -1 &&
+	    !nvme_pci_wait_mmio_after_suspend(dev)) {
 		dev_dbg(dev->ctrl.device, "reading CSTS register failed\n");
 		result = -ENODEV;
 		goto disable;

Re: [bug report] nvme-pci: NVMe lost after S3 resume, CSTS reads all-ones for a few 100 ms

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [High] nvme-pci: missing Signed-off-by trailer and RFC email commit
  message
--

commit ba0016eb2b4fe54e7c1345a47b1af60d7461f91e
Author: Moritz Hiller <m.hiller@pc-heini.de>

> nvme-pci: NVMe lost after S3 resume, CSTS reads all-ones for a few 100 ms
>
> Hi,
>
> on my desktop the root NVMe drive sporadically does not come back after
> S3 resume (roughly 5-10% of resumes in automated runs):
>
>   nvme nvme0: Disabling device after reset failure: -19
>
> The single readl(CSTS) in nvme_pci_enable() returns all-ones about
> 200 ms after resume. The controller is not dead, it just needs a few
> hundred ms more. A small wait loop (diff below) fixes it here, tested
> on CachyOS kernels and on vanilla v7.4-rc2. I am not asking for the
> diff to be merged as is, I would like to know where this should be
> fixed properly.
>
> System:
> - ASUS ROG STRIX X570-F GAMING (BIOS 5044), Ryzen 7 5800X, S3 (deep)
> - Samsung 9100 PRO 2TB [144d:a810], FW 0B2QNXH7, at 0000:01:00.0,
>   directly behind root port 0000:00:01.1 [1022:1483]
>
> Observations:
> - pci_pm_resume_noirq takes ~0.3 ms for the SSD, config space answers.
> - At the failing CSTS read config space is intact, the link is up
>   (16 GT/s x4), the device is in D0, no AER errors:
>
>   vendor=144d cmd=0006 (saved 0002) bar0=fcf00004 (saved fcf00004)
>   pmcsr=0008 lnksta=1044 rp_lnksta=3044
>
> - sysfs remove + rescan brings the drive back after ~3 s.
> - ASPM is off on this link, pcie_aspm=off and
>   nvme_core.default_ps_max_latency_us=0 make no difference.
> - Unpatched, 2 of 15 and 2 of 24 cycles failed in automated runs
>   (90 s sleep, RTC wake). With the wait, no cycle failed (30/30 on
>   CachyOS, 15/15 on v7.4-rc2); the controller answered after
>   200-400 ms whenever the wait triggered.
>
> My reading: pci_power_up() finds PMCSR already in D0 and does not
> wait, and pci_bridge_wait_for_secondary_bus() only runs after D3cold
> and only checks config space. So nothing waits for MMIO to come back,
> and nvme_pci_enable() gives up on the first all-ones read.
>
> The diff polls CSTS every 100 ms for up to 10 s, only if
> NVME_CC_SHN_NORMAL is still set (the was_suspend test in
> nvme_reset_work()) and while pci_device_is_present(). Probe and
> runtime resets are unchanged. Known downsides:
> - The 10 s bound is arbitrary.
> - It also triggers on thaw/restore after hibernation (same bit set).
> - The wait sleeps with shutdown_lock held, as nvme_wait_ready() already
>   does in the same call chain.
> - A controller that answers config space but is really dead after
>   resume is declared dead 10 s later than today.
> - A wait that times out is not logged.
>
> Should this go into nvme-pci, the PCI core, or a quirk for this board
> or drive? I can provide lspci -vvv, full dmesg or test patches.
>
> The problem has existed on this machine for about a year, possibly
> also with the previous SSD in this slot. It was hard to pin down: when
> it happens the root filesystem is gone, so nothing gets written to
> disk. I only got the first log of a failure in August 2026 via
> netconsole, and every failure captured since then looks the same.
>
> Disclosure: I am not a kernel developer. Log analysis, code reading
> and the diff were done with an LLM assistant (Claude); I ran all tests
> on my hardware. I can send patch with Signed-off-by if useful.
>
> Thanks,
> Moritz Hiller
>
> ---8<--- proof of concept, against v7.3.4 ---8<---

[Severity: High]
This commit is missing a Signed-off-by trailer certifying the Developer
Certificate of Origin.  The commit message is also formatted as a mailing
list bug report and inquiry rather than a standard kernel commit changelog,
with conversational notes that should be moved below the "---" separator if
submitted as a patch.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/598ee2b4-32aa-4ad3-99c7-db7f9a7e7736@pc-heini.de?part=1