* [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds [not found] <20261106082612.1840817-1-ziyou.dev@foxmail.com> @ 2026-11-06 8:56 ` Ziyou Wang (Lenovo) 2026-11-06 15:21 ` Bjorn Helgaas 0 siblings, 1 reply; 4+ messages in thread From: Ziyou Wang (Lenovo) @ 2026-11-06 8:56 UTC (permalink / raw) To: Bjorn Helgaas Cc: Lukas Wunner, Kuppuswamy Sathyanarayanan, linux-pci, linux-kernel, stable, Ziyou Wang (Lenovo) Some NVMe devices need more than 5 seconds to recover after DPC containment, but pci_dpc_recovered() waits only 4 seconds. If the device hasn't recovered by then, pciehp reports "device recovery failed" and the device is re-enumerated. Move PCIE_RESET_READY_POLL_MS definition from pci.c to pci.h so it can be used in dpc.c. Use PCIE_RESET_READY_POLL_MS (60 seconds) as the timeout so pciehp waits as long as pci_dev_wait() does, which is sufficient for all known devices. Tested with INNOGRIT NVMe SSD [1dbe:5668] with ACS Fatal error injection: Upstream kernel (7.3.0-master+, 4s timeout): [123.018691] nvme: not ready 1023ms after DPC; waiting [124.106690] nvme: not ready 2047ms after DPC; waiting [125.642703] pciehp: Slot(64): Link Down [126.218696] AER: subordinate device reset failed [126.218700] AER: device recovery failed Device re-enumerated as nvme1 With this patch (60s timeout): [74.894811] nvme: not ready 1023ms after DPC; waiting [75.982813] nvme: not ready 2047ms after DPC; waiting [78.094810] nvme: ready 4095ms after DPC [78.094816] nvme: restart after slot reset [78.111620] AER: device recovery successful Device works normally Fixes: a97396c6eb13 ("PCI: pciehp: Ignore Link Down/Up caused by DPC") Cc: stable@vger.kernel.org Signed-off-by: Ziyou Wang (Lenovo) <ziyou.dev@foxmail.com> --- drivers/pci/pci.c | 1 - drivers/pci/pci.h | 10 ++++++++++ drivers/pci/pcie/dpc.c | 11 ++++++----- 3 files changed, 16 insertions(+), 6 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 8cc6a89130b5..08f08f25e597 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -80,7 +80,6 @@ struct pci_pme_device { * limit, but 60 sec ought to be enough for any device to become * responsive. */ -#define PCIE_RESET_READY_POLL_MS 60000 /* msec */ static void pci_dev_d3_sleep(struct pci_dev *dev) { diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index ba3c3fddddc2..1e8ddcc013ff 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -61,6 +61,16 @@ struct pcie_tlp_log; */ #define PCIE_RESET_CONFIG_WAIT_MS 100 +/* + * Time to wait for a device to become ready after a reset. + * + * Devices may extend the 1 sec period through Request Retry Status + * completions (PCIe r6.0 sec 2.3.1). The spec does not provide an upper + * limit, but 60 sec ought to be enough for any device to become + * responsive. + */ +#define PCIE_RESET_READY_POLL_MS 60000 /* msec */ + /* Parameters for the waiting for link up routine */ #define PCIE_LINK_WAIT_MAX_RETRIES 10 #define PCIE_LINK_WAIT_SLEEP_MS 90 diff --git a/drivers/pci/pcie/dpc.c b/drivers/pci/pcie/dpc.c index 2b779bd1d861..df1c469fa24c 100644 --- a/drivers/pci/pcie/dpc.c +++ b/drivers/pci/pcie/dpc.c @@ -116,12 +116,13 @@ bool pci_dpc_recovered(struct pci_dev *pdev) return false; /* - * Need a timeout in case DPC never completes due to failure of - * dpc_wait_rp_inactive(). The spec doesn't mandate a time limit, - * but reports indicate that DPC completes within 4 seconds. + * Wait for DPC recovery completion. Use PCIE_RESET_READY_POLL_MS + * (60s) to match pci_dev_wait() timeout. Some devices need >5 seconds + * to recover after DPC containment. The 4 second timeout is too short. */ - wait_event_timeout(dpc_completed_waitqueue, dpc_completed(pdev), - msecs_to_jiffies(4000)); + wait_event_timeout(dpc_completed_waitqueue, + dpc_completed(pdev), + msecs_to_jiffies(PCIE_RESET_READY_POLL_MS)); return test_and_clear_bit(PCI_DPC_RECOVERED, &pdev->priv_flags); } -- 2.53.0 ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds 2026-11-06 8:56 ` [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds Ziyou Wang (Lenovo) @ 2026-11-06 15:21 ` Bjorn Helgaas 2026-11-19 3:30 ` Ziyou Wang (Lenovo) 0 siblings, 1 reply; 4+ messages in thread From: Bjorn Helgaas @ 2026-11-06 15:21 UTC (permalink / raw) To: Ziyou Wang (Lenovo) Cc: Bjorn Helgaas, Lukas Wunner, Kuppuswamy Sathyanarayanan, linux-pci, linux-kernel, stable On Fri, Nov 06, 2026 at 04:56:55PM +0800, Ziyou Wang (Lenovo) wrote: > Some NVMe devices need more than 5 seconds to recover after DPC > containment, but pci_dpc_recovered() waits only 4 seconds. If the device > hasn't recovered by then, pciehp reports "device recovery failed" and > the device is re-enumerated. > > Move PCIE_RESET_READY_POLL_MS definition from pci.c to pci.h so it can be > used in dpc.c. Use PCIE_RESET_READY_POLL_MS (60 seconds) as the timeout > so pciehp waits as long as pci_dev_wait() does, which is sufficient for > all known devices. > > Tested with INNOGRIT NVMe SSD [1dbe:5668] with ACS Fatal error > injection: > > Upstream kernel (7.3.0-master+, 4s timeout): > [123.018691] nvme: not ready 1023ms after DPC; waiting > [124.106690] nvme: not ready 2047ms after DPC; waiting Was there a "not ready ... after DPC; giving up" message here? pci_dev_wait() should log *something* about how it resolved this situation. > [125.642703] pciehp: Slot(64): Link Down > [126.218696] AER: subordinate device reset failed > [126.218700] AER: device recovery failed > Device re-enumerated as nvme1 > > With this patch (60s timeout): > [74.894811] nvme: not ready 1023ms after DPC; waiting > [75.982813] nvme: not ready 2047ms after DPC; waiting > [78.094810] nvme: ready 4095ms after DPC > [78.094816] nvme: restart after slot reset > [78.111620] AER: device recovery successful > Device works normally > > Fixes: a97396c6eb13 ("PCI: pciehp: Ignore Link Down/Up caused by DPC") > Cc: stable@vger.kernel.org > Signed-off-by: Ziyou Wang (Lenovo) <ziyou.dev@foxmail.com> > --- 30 minutes after v1 is too soon to post a v2. A couple days is a reasonable interval. v2 should include a note here (after the "---") about what changed since v1 and ideally a lore link to the v1 posting. ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds 2026-11-06 15:21 ` Bjorn Helgaas @ 2026-11-19 3:30 ` Ziyou Wang (Lenovo) 2026-11-19 9:01 ` Lukas Wunner 0 siblings, 1 reply; 4+ messages in thread From: Ziyou Wang (Lenovo) @ 2026-11-19 3:30 UTC (permalink / raw) To: helgaas Cc: bhelgaas, lukas, sathyanarayanan.kuppuswamy, linux-pci, linux-kernel, stable, ziyou.dev Hi Bjorn, Thanks for your review! On Fri, Nov 06, 2026 at 09:21:43AM -0600, Bjorn Helgaas wrote: > Was there a "not ready ... after DPC; giving up" message here? > pci_dev_wait() should log *something* about how it resolved this > situation. No. Here is the complete log from the DPC event to the re-enumeration: [121.693046] pcieport 0000:00:03.1: EDR: EDR event received [121.695743] pcieport 0000:00:03.1: DPC: containment event, status:0x1f01: unmasked uncorrectable error detected [121.695825] pcieport 0000:00:03.1: PCIe Bus Error: severity=Uncorrectable (Fatal) [121.695829] pcieport 0000:00:03.1: device [1022:1108] error status/mask=00200000/00000000 [121.695831] pcieport 0000:00:03.1: [21] ACSViol | Receiver | Transaction Layer (First) [121.695834] pcieport 0000:00:03.1: AER: TLP Header: 0x60002080 0x010042ff 0x00000017 0x7e325c00 [121.695841] nvme nvme0: frozen state error detected, reset controller [121.721168] nvme0n1: I/O Cmd(0x2) @ LBA 67600640, 256 blocks, I/O Error (sct 0x3 / sc 0x71) [121.721182] I/O error, dev nvme0n1, sector 67600640 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [121.721193] nvme0n1: I/O Cmd(0x2) @ LBA 67600384, 256 blocks, I/O Error (sct 0x3 / sc 0x71) [121.721196] I/O error, dev nvme0n1, sector 67600384 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [123.018691] nvme 0000:01:00.0: not ready 1023ms after DPC; waiting [124.106690] nvme 0000:01:00.0: not ready 2047ms after DPC; waiting [125.642703] pcieport 0000:00:03.1: pciehp: Slot(64): Link Down [125.647840] Buffer I/O error on dev nvme0n1, logical block 8450048, async page read [125.647846] Buffer I/O error on dev nvme0n1, logical block 8450049, async page read [125.647847] Buffer I/O error on dev nvme0n1, logical block 8450050, async page read [125.647848] Buffer I/O error on dev nvme0n1, logical block 8450051, async page read [125.647850] Buffer I/O error on dev nvme0n1, logical block 8450052, async page read [125.647850] Buffer I/O error on dev nvme0n1, logical block 8450053, async page read [125.647851] Buffer I/O error on dev nvme0n1, logical block 8450054, async page read [125.647852] Buffer I/O error on dev nvme0n1, logical block 8450055, async page read [125.647853] Buffer I/O error on dev nvme0n1, logical block 8450056, async page read [125.647854] Buffer I/O error on dev nvme0n1, logical block 8450057, async page read [125.849608] pcieport 0000:00:03.1: pciehp: Slot(64): Card present [125.849614] pcieport 0000:00:03.1: pciehp: Slot(64): Link Up [126.218696] pcieport 0000:00:03.1: AER: subordinate device reset failed [126.218700] pcieport 0000:00:03.1: AER: device recovery failed [126.705745] pci 0000:01:00.0: [1dbe:5668] type 00 class 0x010802 PCIe Endpoint [126.705786] pci 0000:01:00.0: BAR 0 [mem 0x00000000-0x00003fff 64bit] [126.705793] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff pref] [126.705801] pci 0000:01:00.0: Max Payload Size set to 512 (was 128, max 512) [126.705987] pci 0000:01:00.0: _HPX attempts Link Control setting (AND 0xffff OR 0x0040) [126.706073] pci 0000:01:00.0: supports D1 D2 [126.706074] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold [126.706432] pci 0000:01:00.0: Adding to iommu group 16 [126.706481] pci 0000:01:00.0: ROM [mem 0xde300000-0xde30ffff pref]: assigned [126.706483] pci 0000:01:00.0: BAR 0 [mem 0xde310000-0xde313fff 64bit]: assigned [126.706491] pcieport 0000:00:03.1: PCI bridge to [bus 01] [126.706494] pcieport 0000:00:03.1: bridge window [io 0x1000-0x1fff] [126.706498] pcieport 0000:00:03.1: bridge window [mem 0xde300000-0xde3fffff] [126.706500] pcieport 0000:00:03.1: bridge window [mem 0x10021600000-0x100217fffff 64bit pref] [126.709685] nvme nvme1: pci function 0000:01:00.0 [126.709692] nvme 0000:01:00.0: enabling device (0140 -> 0142) [126.712068] nvme nvme1: D3 entry latency set to 10 seconds [126.720936] nvme nvme1: 129/0/0 default/read/poll queues [126.727879] nvme0n2: p1 p2 p3 pci_dev_wait() never reached its timeout. The 4s pci_dpc_recovered() timeout expired first, so pciehp removed the device and marked it disconnected. When pci_dev_wait() woke up, it took the pci_dev_is_disconnected() path, which only logs at debug level and returns -ENOTTY. That is why the -ENOTTY surfaced as "AER: subordinate device reset failed" instead. The device needs a little over 4s to become ready ("ready 4095ms after DPC" with this patch), so the 4s timeout is losing that race. One thing worth noting: before e8df1d194c39b ("PCI: Don't treat ~0 as ready in pci_dev_wait() with RRS SV"), this failure was masked on this system. The RRS SV path only checked for the 0x0001 RRS Vendor ID, so a ~0 read from the not-yet-ready device was treated as "ready", pci_bridge_wait_for_secondary_bus() returned early, and the DPC "recovery" appeared to succeed within the 4s window, even though the device was not actually usable (nvme then failed with -19 after the "successful" recovery). With e8df1d194c39b, pci_dev_wait() correctly waits, which re-exposed this 4s race. Best regards, Ziyou Wang ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds 2026-11-19 3:30 ` Ziyou Wang (Lenovo) @ 2026-11-19 9:01 ` Lukas Wunner 0 siblings, 0 replies; 4+ messages in thread From: Lukas Wunner @ 2026-11-19 9:01 UTC (permalink / raw) To: Ziyou Wang (Lenovo) Cc: helgaas, bhelgaas, sathyanarayanan.kuppuswamy, linux-pci, linux-kernel, stable On Thu, Nov 19, 2026 at 11:30:36AM +0800, Ziyou Wang (Lenovo) wrote: > [121.693046] pcieport 0000:00:03.1: EDR: EDR event received > [121.695743] pcieport 0000:00:03.1: DPC: containment event, status:0x1f01: unmasked uncorrectable error detected > [121.695825] pcieport 0000:00:03.1: PCIe Bus Error: severity=Uncorrectable (Fatal) > [121.695829] pcieport 0000:00:03.1: device [1022:1108] error status/mask=00200000/00000000 > [121.695831] pcieport 0000:00:03.1: [21] ACSViol | Receiver | Transaction Layer (First) > [121.695834] pcieport 0000:00:03.1: AER: TLP Header: 0x60002080 0x010042ff 0x00000017 0x7e325c00 [...] > [123.018691] nvme 0000:01:00.0: not ready 1023ms after DPC; waiting I'm wondering why the ACS Violation Error is raised in the first place. The TLP causing the DPC event is a Memory Write Request from the NVMe drive to memory address 0x000000177e325c00. The Requester ID 0x100 seems to match the bus address 0000:01:00.0. What are the ACS settings at the Root Port and why is it signaling an ACS violation? TLP decoded with tlp-tool (https://github.com/mmpg-x86/tlp-tool): +----------+-------------+----------------------+ | TLP Type | MemWriteReq | 4DW with Data Header | +----------+-------------+----------------------+ +------------+--------+--------+-------+ | Field Name | Offset | Length | Value | | | (bits) | (bits) | | +------------+--------+--------+-------+ | Fmt | 0 | 3 | 3 | | Type | 3 | 5 | 0 | | T9 | 8 | 1 | 0 | | TC | 9 | 3 | 0 | | T8 | 12 | 1 | 0 | | Attr_b2 | 13 | 1 | 0 | | LN | 14 | 1 | 0 | | TH | 15 | 1 | 0 | | Td | 16 | 1 | 0 | | Ep | 17 | 1 | 0 | | Attr | 18 | 2 | 2 | | AT | 20 | 2 | 0 | | Length | 22 | 10 | 128 | +------------+--------+--------+-------+ +-----------------+----------------------+ | TLP: | 4DW with Data Header | +-----------------+----------------------+ | Req ID | 0x100 | | Tag | 0x42 | | Last DW BE | 0xF | | First DW BE | 0xF | | Addr High (DW2) | 0x00000017 | | Addr Low (DW3) | 0x7E325C00 | +-----------------+----------------------+ Thanks, Lukas ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-11-19 9:01 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20261106082612.1840817-1-ziyou.dev@foxmail.com>
2026-11-06 8:56 ` [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds Ziyou Wang (Lenovo)
2026-11-06 15:21 ` Bjorn Helgaas
2026-11-19 3:30 ` Ziyou Wang (Lenovo)
2026-11-19 9:01 ` Lukas Wunner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox