* [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds
[not found] <20261106082612.1840817-1-ziyou.dev@foxmail.com>
@ 2026-11-06 8:56 ` Ziyou Wang (Lenovo)
2026-11-06 9:05 ` sashiko-bot
2026-11-06 15:21 ` Bjorn Helgaas
0 siblings, 2 replies; 5+ messages in thread
From: Ziyou Wang (Lenovo) @ 2026-11-06 8:56 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: Lukas Wunner, Kuppuswamy Sathyanarayanan, linux-pci, linux-kernel,
stable, Ziyou Wang (Lenovo)
Some NVMe devices need more than 5 seconds to recover after DPC
containment, but pci_dpc_recovered() waits only 4 seconds. If the device
hasn't recovered by then, pciehp reports "device recovery failed" and
the device is re-enumerated.
Move PCIE_RESET_READY_POLL_MS definition from pci.c to pci.h so it can be
used in dpc.c. Use PCIE_RESET_READY_POLL_MS (60 seconds) as the timeout
so pciehp waits as long as pci_dev_wait() does, which is sufficient for
all known devices.
Tested with INNOGRIT NVMe SSD [1dbe:5668] with ACS Fatal error
injection:
Upstream kernel (7.3.0-master+, 4s timeout):
[123.018691] nvme: not ready 1023ms after DPC; waiting
[124.106690] nvme: not ready 2047ms after DPC; waiting
[125.642703] pciehp: Slot(64): Link Down
[126.218696] AER: subordinate device reset failed
[126.218700] AER: device recovery failed
Device re-enumerated as nvme1
With this patch (60s timeout):
[74.894811] nvme: not ready 1023ms after DPC; waiting
[75.982813] nvme: not ready 2047ms after DPC; waiting
[78.094810] nvme: ready 4095ms after DPC
[78.094816] nvme: restart after slot reset
[78.111620] AER: device recovery successful
Device works normally
Fixes: a97396c6eb13 ("PCI: pciehp: Ignore Link Down/Up caused by DPC")
Cc: stable@vger.kernel.org
Signed-off-by: Ziyou Wang (Lenovo) <ziyou.dev@foxmail.com>
---
drivers/pci/pci.c | 1 -
drivers/pci/pci.h | 10 ++++++++++
drivers/pci/pcie/dpc.c | 11 ++++++-----
3 files changed, 16 insertions(+), 6 deletions(-)
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 8cc6a89130b5..08f08f25e597 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -80,7 +80,6 @@ struct pci_pme_device {
* limit, but 60 sec ought to be enough for any device to become
* responsive.
*/
-#define PCIE_RESET_READY_POLL_MS 60000 /* msec */
static void pci_dev_d3_sleep(struct pci_dev *dev)
{
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index ba3c3fddddc2..1e8ddcc013ff 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -61,6 +61,16 @@ struct pcie_tlp_log;
*/
#define PCIE_RESET_CONFIG_WAIT_MS 100
+/*
+ * Time to wait for a device to become ready after a reset.
+ *
+ * Devices may extend the 1 sec period through Request Retry Status
+ * completions (PCIe r6.0 sec 2.3.1). The spec does not provide an upper
+ * limit, but 60 sec ought to be enough for any device to become
+ * responsive.
+ */
+#define PCIE_RESET_READY_POLL_MS 60000 /* msec */
+
/* Parameters for the waiting for link up routine */
#define PCIE_LINK_WAIT_MAX_RETRIES 10
#define PCIE_LINK_WAIT_SLEEP_MS 90
diff --git a/drivers/pci/pcie/dpc.c b/drivers/pci/pcie/dpc.c
index 2b779bd1d861..df1c469fa24c 100644
--- a/drivers/pci/pcie/dpc.c
+++ b/drivers/pci/pcie/dpc.c
@@ -116,12 +116,13 @@ bool pci_dpc_recovered(struct pci_dev *pdev)
return false;
/*
- * Need a timeout in case DPC never completes due to failure of
- * dpc_wait_rp_inactive(). The spec doesn't mandate a time limit,
- * but reports indicate that DPC completes within 4 seconds.
+ * Wait for DPC recovery completion. Use PCIE_RESET_READY_POLL_MS
+ * (60s) to match pci_dev_wait() timeout. Some devices need >5 seconds
+ * to recover after DPC containment. The 4 second timeout is too short.
*/
- wait_event_timeout(dpc_completed_waitqueue, dpc_completed(pdev),
- msecs_to_jiffies(4000));
+ wait_event_timeout(dpc_completed_waitqueue,
+ dpc_completed(pdev),
+ msecs_to_jiffies(PCIE_RESET_READY_POLL_MS));
return test_and_clear_bit(PCI_DPC_RECOVERED, &pdev->priv_flags);
}
--
2.53.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds
2026-11-06 8:56 ` [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds Ziyou Wang (Lenovo)
@ 2026-11-06 9:05 ` sashiko-bot
2026-11-06 15:21 ` Bjorn Helgaas
1 sibling, 0 replies; 5+ messages in thread
From: sashiko-bot @ 2026-11-06 9:05 UTC (permalink / raw)
To: Ziyou Wang (Lenovo); +Cc: linux-pci
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] PCI: orphaned PCIE_RESET_READY_POLL_MS comment left in
drivers/pci/pci.c
--
commit b65aa35af6ea60927f9e516408b33a731c59921c
Author: Ziyou Wang (Lenovo) <ziyou.dev@foxmail.com>
PCI/DPC: Increase recovery timeout to 60 seconds
This commit moves PCIE_RESET_READY_POLL_MS from drivers/pci/pci.c to
drivers/pci/pci.h and uses it as the DPC recovery timeout in
pci_dpc_recovered().
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index d6590805faeb2..dae2e74e64777 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -82,7 +82,6 @@ struct pci_pme_device {
> * limit, but 60 sec ought to be enough for any device to become
> * responsive.
> */
> -#define PCIE_RESET_READY_POLL_MS 60000 /* msec */
>
> static void pci_dev_d3_sleep(struct pci_dev *dev)
> {
[Severity: Low]
This isn't a bug, but moving PCIE_RESET_READY_POLL_MS to drivers/pci/pci.h
leaves its original comment block orphaned above pci_dev_d3_sleep(). Should
the comment be removed here as well since it was copied to drivers/pci/pci.h?
--
Sashiko AI review · https://sashiko.dev/#/patchset/tencent_BB887B4DB8D2131AF5026893E0FE567D6D05@qq.com?part=1
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds
2026-11-06 8:56 ` [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds Ziyou Wang (Lenovo)
2026-11-06 9:05 ` sashiko-bot
@ 2026-11-06 15:21 ` Bjorn Helgaas
2026-11-19 3:30 ` Ziyou Wang (Lenovo)
1 sibling, 1 reply; 5+ messages in thread
From: Bjorn Helgaas @ 2026-11-06 15:21 UTC (permalink / raw)
To: Ziyou Wang (Lenovo)
Cc: Bjorn Helgaas, Lukas Wunner, Kuppuswamy Sathyanarayanan,
linux-pci, linux-kernel, stable
On Fri, Nov 06, 2026 at 04:56:55PM +0800, Ziyou Wang (Lenovo) wrote:
> Some NVMe devices need more than 5 seconds to recover after DPC
> containment, but pci_dpc_recovered() waits only 4 seconds. If the device
> hasn't recovered by then, pciehp reports "device recovery failed" and
> the device is re-enumerated.
>
> Move PCIE_RESET_READY_POLL_MS definition from pci.c to pci.h so it can be
> used in dpc.c. Use PCIE_RESET_READY_POLL_MS (60 seconds) as the timeout
> so pciehp waits as long as pci_dev_wait() does, which is sufficient for
> all known devices.
>
> Tested with INNOGRIT NVMe SSD [1dbe:5668] with ACS Fatal error
> injection:
>
> Upstream kernel (7.3.0-master+, 4s timeout):
> [123.018691] nvme: not ready 1023ms after DPC; waiting
> [124.106690] nvme: not ready 2047ms after DPC; waiting
Was there a "not ready ... after DPC; giving up" message here?
pci_dev_wait() should log *something* about how it resolved this
situation.
> [125.642703] pciehp: Slot(64): Link Down
> [126.218696] AER: subordinate device reset failed
> [126.218700] AER: device recovery failed
> Device re-enumerated as nvme1
>
> With this patch (60s timeout):
> [74.894811] nvme: not ready 1023ms after DPC; waiting
> [75.982813] nvme: not ready 2047ms after DPC; waiting
> [78.094810] nvme: ready 4095ms after DPC
> [78.094816] nvme: restart after slot reset
> [78.111620] AER: device recovery successful
> Device works normally
>
> Fixes: a97396c6eb13 ("PCI: pciehp: Ignore Link Down/Up caused by DPC")
> Cc: stable@vger.kernel.org
> Signed-off-by: Ziyou Wang (Lenovo) <ziyou.dev@foxmail.com>
> ---
30 minutes after v1 is too soon to post a v2. A couple days is a
reasonable interval.
v2 should include a note here (after the "---") about what changed
since v1 and ideally a lore link to the v1 posting.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds
2026-11-06 15:21 ` Bjorn Helgaas
@ 2026-11-19 3:30 ` Ziyou Wang (Lenovo)
2026-11-19 9:01 ` Lukas Wunner
0 siblings, 1 reply; 5+ messages in thread
From: Ziyou Wang (Lenovo) @ 2026-11-19 3:30 UTC (permalink / raw)
To: helgaas
Cc: bhelgaas, lukas, sathyanarayanan.kuppuswamy, linux-pci,
linux-kernel, stable, ziyou.dev
Hi Bjorn,
Thanks for your review!
On Fri, Nov 06, 2026 at 09:21:43AM -0600, Bjorn Helgaas wrote:
> Was there a "not ready ... after DPC; giving up" message here?
> pci_dev_wait() should log *something* about how it resolved this
> situation.
No. Here is the complete log from the DPC event to the re-enumeration:
[121.693046] pcieport 0000:00:03.1: EDR: EDR event received
[121.695743] pcieport 0000:00:03.1: DPC: containment event, status:0x1f01: unmasked uncorrectable error detected
[121.695825] pcieport 0000:00:03.1: PCIe Bus Error: severity=Uncorrectable (Fatal)
[121.695829] pcieport 0000:00:03.1: device [1022:1108] error status/mask=00200000/00000000
[121.695831] pcieport 0000:00:03.1: [21] ACSViol | Receiver | Transaction Layer (First)
[121.695834] pcieport 0000:00:03.1: AER: TLP Header: 0x60002080 0x010042ff 0x00000017 0x7e325c00
[121.695841] nvme nvme0: frozen state error detected, reset controller
[121.721168] nvme0n1: I/O Cmd(0x2) @ LBA 67600640, 256 blocks, I/O Error (sct 0x3 / sc 0x71)
[121.721182] I/O error, dev nvme0n1, sector 67600640 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[121.721193] nvme0n1: I/O Cmd(0x2) @ LBA 67600384, 256 blocks, I/O Error (sct 0x3 / sc 0x71)
[121.721196] I/O error, dev nvme0n1, sector 67600384 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[123.018691] nvme 0000:01:00.0: not ready 1023ms after DPC; waiting
[124.106690] nvme 0000:01:00.0: not ready 2047ms after DPC; waiting
[125.642703] pcieport 0000:00:03.1: pciehp: Slot(64): Link Down
[125.647840] Buffer I/O error on dev nvme0n1, logical block 8450048, async page read
[125.647846] Buffer I/O error on dev nvme0n1, logical block 8450049, async page read
[125.647847] Buffer I/O error on dev nvme0n1, logical block 8450050, async page read
[125.647848] Buffer I/O error on dev nvme0n1, logical block 8450051, async page read
[125.647850] Buffer I/O error on dev nvme0n1, logical block 8450052, async page read
[125.647850] Buffer I/O error on dev nvme0n1, logical block 8450053, async page read
[125.647851] Buffer I/O error on dev nvme0n1, logical block 8450054, async page read
[125.647852] Buffer I/O error on dev nvme0n1, logical block 8450055, async page read
[125.647853] Buffer I/O error on dev nvme0n1, logical block 8450056, async page read
[125.647854] Buffer I/O error on dev nvme0n1, logical block 8450057, async page read
[125.849608] pcieport 0000:00:03.1: pciehp: Slot(64): Card present
[125.849614] pcieport 0000:00:03.1: pciehp: Slot(64): Link Up
[126.218696] pcieport 0000:00:03.1: AER: subordinate device reset failed
[126.218700] pcieport 0000:00:03.1: AER: device recovery failed
[126.705745] pci 0000:01:00.0: [1dbe:5668] type 00 class 0x010802 PCIe Endpoint
[126.705786] pci 0000:01:00.0: BAR 0 [mem 0x00000000-0x00003fff 64bit]
[126.705793] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff pref]
[126.705801] pci 0000:01:00.0: Max Payload Size set to 512 (was 128, max 512)
[126.705987] pci 0000:01:00.0: _HPX attempts Link Control setting (AND 0xffff OR 0x0040)
[126.706073] pci 0000:01:00.0: supports D1 D2
[126.706074] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[126.706432] pci 0000:01:00.0: Adding to iommu group 16
[126.706481] pci 0000:01:00.0: ROM [mem 0xde300000-0xde30ffff pref]: assigned
[126.706483] pci 0000:01:00.0: BAR 0 [mem 0xde310000-0xde313fff 64bit]: assigned
[126.706491] pcieport 0000:00:03.1: PCI bridge to [bus 01]
[126.706494] pcieport 0000:00:03.1: bridge window [io 0x1000-0x1fff]
[126.706498] pcieport 0000:00:03.1: bridge window [mem 0xde300000-0xde3fffff]
[126.706500] pcieport 0000:00:03.1: bridge window [mem 0x10021600000-0x100217fffff 64bit pref]
[126.709685] nvme nvme1: pci function 0000:01:00.0
[126.709692] nvme 0000:01:00.0: enabling device (0140 -> 0142)
[126.712068] nvme nvme1: D3 entry latency set to 10 seconds
[126.720936] nvme nvme1: 129/0/0 default/read/poll queues
[126.727879] nvme0n2: p1 p2 p3
pci_dev_wait() never reached its timeout. The 4s pci_dpc_recovered()
timeout expired first, so pciehp removed the device and marked it
disconnected. When pci_dev_wait() woke up, it took the
pci_dev_is_disconnected() path, which only logs at debug level and
returns -ENOTTY. That is why the -ENOTTY surfaced as "AER:
subordinate device reset failed" instead.
The device needs a little over 4s to become ready ("ready 4095ms
after DPC" with this patch), so the 4s timeout is losing that race.
One thing worth noting: before e8df1d194c39b ("PCI: Don't treat ~0 as
ready in pci_dev_wait() with RRS SV"), this failure was masked on this
system. The RRS SV path only checked for the 0x0001 RRS Vendor ID, so
a ~0 read from the not-yet-ready device was treated as "ready",
pci_bridge_wait_for_secondary_bus() returned early, and the DPC
"recovery" appeared to succeed within the 4s window, even though the
device was not actually usable (nvme then failed with -19 after the
"successful" recovery). With e8df1d194c39b, pci_dev_wait() correctly
waits, which re-exposed this 4s race.
Best regards,
Ziyou Wang
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds
2026-11-19 3:30 ` Ziyou Wang (Lenovo)
@ 2026-11-19 9:01 ` Lukas Wunner
0 siblings, 0 replies; 5+ messages in thread
From: Lukas Wunner @ 2026-11-19 9:01 UTC (permalink / raw)
To: Ziyou Wang (Lenovo)
Cc: helgaas, bhelgaas, sathyanarayanan.kuppuswamy, linux-pci,
linux-kernel, stable
On Thu, Nov 19, 2026 at 11:30:36AM +0800, Ziyou Wang (Lenovo) wrote:
> [121.693046] pcieport 0000:00:03.1: EDR: EDR event received
> [121.695743] pcieport 0000:00:03.1: DPC: containment event, status:0x1f01: unmasked uncorrectable error detected
> [121.695825] pcieport 0000:00:03.1: PCIe Bus Error: severity=Uncorrectable (Fatal)
> [121.695829] pcieport 0000:00:03.1: device [1022:1108] error status/mask=00200000/00000000
> [121.695831] pcieport 0000:00:03.1: [21] ACSViol | Receiver | Transaction Layer (First)
> [121.695834] pcieport 0000:00:03.1: AER: TLP Header: 0x60002080 0x010042ff 0x00000017 0x7e325c00
[...]
> [123.018691] nvme 0000:01:00.0: not ready 1023ms after DPC; waiting
I'm wondering why the ACS Violation Error is raised in the first place.
The TLP causing the DPC event is a Memory Write Request from the NVMe
drive to memory address 0x000000177e325c00. The Requester ID 0x100
seems to match the bus address 0000:01:00.0.
What are the ACS settings at the Root Port and why is it signaling an
ACS violation?
TLP decoded with tlp-tool (https://github.com/mmpg-x86/tlp-tool):
+----------+-------------+----------------------+
| TLP Type | MemWriteReq | 4DW with Data Header |
+----------+-------------+----------------------+
+------------+--------+--------+-------+
| Field Name | Offset | Length | Value |
| | (bits) | (bits) | |
+------------+--------+--------+-------+
| Fmt | 0 | 3 | 3 |
| Type | 3 | 5 | 0 |
| T9 | 8 | 1 | 0 |
| TC | 9 | 3 | 0 |
| T8 | 12 | 1 | 0 |
| Attr_b2 | 13 | 1 | 0 |
| LN | 14 | 1 | 0 |
| TH | 15 | 1 | 0 |
| Td | 16 | 1 | 0 |
| Ep | 17 | 1 | 0 |
| Attr | 18 | 2 | 2 |
| AT | 20 | 2 | 0 |
| Length | 22 | 10 | 128 |
+------------+--------+--------+-------+
+-----------------+----------------------+
| TLP: | 4DW with Data Header |
+-----------------+----------------------+
| Req ID | 0x100 |
| Tag | 0x42 |
| Last DW BE | 0xF |
| First DW BE | 0xF |
| Addr High (DW2) | 0x00000017 |
| Addr Low (DW3) | 0x7E325C00 |
+-----------------+----------------------+
Thanks,
Lukas
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-11-19 9:01 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20261106082612.1840817-1-ziyou.dev@foxmail.com>
2026-11-06 8:56 ` [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds Ziyou Wang (Lenovo)
2026-11-06 9:05 ` sashiko-bot
2026-11-06 15:21 ` Bjorn Helgaas
2026-11-19 3:30 ` Ziyou Wang (Lenovo)
2026-11-19 9:01 ` Lukas Wunner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox