* [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds
[not found] <20261106082612.1840817-1-ziyou.dev@foxmail.com>
@ 2026-11-06 8:56 ` Ziyou Wang (Lenovo)
2026-11-06 9:05 ` sashiko-bot
2026-11-06 15:21 ` Bjorn Helgaas
0 siblings, 2 replies; 6+ messages in thread
From: Ziyou Wang (Lenovo) @ 2026-11-06 8:56 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: Lukas Wunner, Kuppuswamy Sathyanarayanan, linux-pci, linux-kernel,
stable, Ziyou Wang (Lenovo)
Some NVMe devices need more than 5 seconds to recover after DPC
containment, but pci_dpc_recovered() waits only 4 seconds. If the device
hasn't recovered by then, pciehp reports "device recovery failed" and
the device is re-enumerated.
Move PCIE_RESET_READY_POLL_MS definition from pci.c to pci.h so it can be
used in dpc.c. Use PCIE_RESET_READY_POLL_MS (60 seconds) as the timeout
so pciehp waits as long as pci_dev_wait() does, which is sufficient for
all known devices.
Tested with INNOGRIT NVMe SSD [1dbe:5668] with ACS Fatal error
injection:
Upstream kernel (7.3.0-master+, 4s timeout):
[123.018691] nvme: not ready 1023ms after DPC; waiting
[124.106690] nvme: not ready 2047ms after DPC; waiting
[125.642703] pciehp: Slot(64): Link Down
[126.218696] AER: subordinate device reset failed
[126.218700] AER: device recovery failed
Device re-enumerated as nvme1
With this patch (60s timeout):
[74.894811] nvme: not ready 1023ms after DPC; waiting
[75.982813] nvme: not ready 2047ms after DPC; waiting
[78.094810] nvme: ready 4095ms after DPC
[78.094816] nvme: restart after slot reset
[78.111620] AER: device recovery successful
Device works normally
Fixes: a97396c6eb13 ("PCI: pciehp: Ignore Link Down/Up caused by DPC")
Cc: stable@vger.kernel.org
Signed-off-by: Ziyou Wang (Lenovo) <ziyou.dev@foxmail.com>
---
drivers/pci/pci.c | 1 -
drivers/pci/pci.h | 10 ++++++++++
drivers/pci/pcie/dpc.c | 11 ++++++-----
3 files changed, 16 insertions(+), 6 deletions(-)
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 8cc6a89130b5..08f08f25e597 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -80,7 +80,6 @@ struct pci_pme_device {
* limit, but 60 sec ought to be enough for any device to become
* responsive.
*/
-#define PCIE_RESET_READY_POLL_MS 60000 /* msec */
static void pci_dev_d3_sleep(struct pci_dev *dev)
{
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index ba3c3fddddc2..1e8ddcc013ff 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -61,6 +61,16 @@ struct pcie_tlp_log;
*/
#define PCIE_RESET_CONFIG_WAIT_MS 100
+/*
+ * Time to wait for a device to become ready after a reset.
+ *
+ * Devices may extend the 1 sec period through Request Retry Status
+ * completions (PCIe r6.0 sec 2.3.1). The spec does not provide an upper
+ * limit, but 60 sec ought to be enough for any device to become
+ * responsive.
+ */
+#define PCIE_RESET_READY_POLL_MS 60000 /* msec */
+
/* Parameters for the waiting for link up routine */
#define PCIE_LINK_WAIT_MAX_RETRIES 10
#define PCIE_LINK_WAIT_SLEEP_MS 90
diff --git a/drivers/pci/pcie/dpc.c b/drivers/pci/pcie/dpc.c
index 2b779bd1d861..df1c469fa24c 100644
--- a/drivers/pci/pcie/dpc.c
+++ b/drivers/pci/pcie/dpc.c
@@ -116,12 +116,13 @@ bool pci_dpc_recovered(struct pci_dev *pdev)
return false;
/*
- * Need a timeout in case DPC never completes due to failure of
- * dpc_wait_rp_inactive(). The spec doesn't mandate a time limit,
- * but reports indicate that DPC completes within 4 seconds.
+ * Wait for DPC recovery completion. Use PCIE_RESET_READY_POLL_MS
+ * (60s) to match pci_dev_wait() timeout. Some devices need >5 seconds
+ * to recover after DPC containment. The 4 second timeout is too short.
*/
- wait_event_timeout(dpc_completed_waitqueue, dpc_completed(pdev),
- msecs_to_jiffies(4000));
+ wait_event_timeout(dpc_completed_waitqueue,
+ dpc_completed(pdev),
+ msecs_to_jiffies(PCIE_RESET_READY_POLL_MS));
return test_and_clear_bit(PCI_DPC_RECOVERED, &pdev->priv_flags);
}
--
2.53.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds
2026-11-06 8:56 ` [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds Ziyou Wang (Lenovo)
@ 2026-11-06 9:05 ` sashiko-bot
2026-11-06 15:21 ` Bjorn Helgaas
1 sibling, 0 replies; 6+ messages in thread
From: sashiko-bot @ 2026-11-06 9:05 UTC (permalink / raw)
To: Ziyou Wang (Lenovo); +Cc: linux-pci
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] PCI: orphaned PCIE_RESET_READY_POLL_MS comment left in
drivers/pci/pci.c
--
commit b65aa35af6ea60927f9e516408b33a731c59921c
Author: Ziyou Wang (Lenovo) <ziyou.dev@foxmail.com>
PCI/DPC: Increase recovery timeout to 60 seconds
This commit moves PCIE_RESET_READY_POLL_MS from drivers/pci/pci.c to
drivers/pci/pci.h and uses it as the DPC recovery timeout in
pci_dpc_recovered().
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index d6590805faeb2..dae2e74e64777 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -82,7 +82,6 @@ struct pci_pme_device {
> * limit, but 60 sec ought to be enough for any device to become
> * responsive.
> */
> -#define PCIE_RESET_READY_POLL_MS 60000 /* msec */
>
> static void pci_dev_d3_sleep(struct pci_dev *dev)
> {
[Severity: Low]
This isn't a bug, but moving PCIE_RESET_READY_POLL_MS to drivers/pci/pci.h
leaves its original comment block orphaned above pci_dev_d3_sleep(). Should
the comment be removed here as well since it was copied to drivers/pci/pci.h?
--
Sashiko AI review · https://sashiko.dev/#/patchset/tencent_BB887B4DB8D2131AF5026893E0FE567D6D05@qq.com?part=1
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds
2026-11-06 8:56 ` [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds Ziyou Wang (Lenovo)
2026-11-06 9:05 ` sashiko-bot
@ 2026-11-06 15:21 ` Bjorn Helgaas
2026-11-19 3:30 ` Ziyou Wang (Lenovo)
1 sibling, 1 reply; 6+ messages in thread
From: Bjorn Helgaas @ 2026-11-06 15:21 UTC (permalink / raw)
To: Ziyou Wang (Lenovo)
Cc: Bjorn Helgaas, Lukas Wunner, Kuppuswamy Sathyanarayanan,
linux-pci, linux-kernel, stable
On Fri, Nov 06, 2026 at 04:56:55PM +0800, Ziyou Wang (Lenovo) wrote:
> Some NVMe devices need more than 5 seconds to recover after DPC
> containment, but pci_dpc_recovered() waits only 4 seconds. If the device
> hasn't recovered by then, pciehp reports "device recovery failed" and
> the device is re-enumerated.
>
> Move PCIE_RESET_READY_POLL_MS definition from pci.c to pci.h so it can be
> used in dpc.c. Use PCIE_RESET_READY_POLL_MS (60 seconds) as the timeout
> so pciehp waits as long as pci_dev_wait() does, which is sufficient for
> all known devices.
>
> Tested with INNOGRIT NVMe SSD [1dbe:5668] with ACS Fatal error
> injection:
>
> Upstream kernel (7.3.0-master+, 4s timeout):
> [123.018691] nvme: not ready 1023ms after DPC; waiting
> [124.106690] nvme: not ready 2047ms after DPC; waiting
Was there a "not ready ... after DPC; giving up" message here?
pci_dev_wait() should log *something* about how it resolved this
situation.
> [125.642703] pciehp: Slot(64): Link Down
> [126.218696] AER: subordinate device reset failed
> [126.218700] AER: device recovery failed
> Device re-enumerated as nvme1
>
> With this patch (60s timeout):
> [74.894811] nvme: not ready 1023ms after DPC; waiting
> [75.982813] nvme: not ready 2047ms after DPC; waiting
> [78.094810] nvme: ready 4095ms after DPC
> [78.094816] nvme: restart after slot reset
> [78.111620] AER: device recovery successful
> Device works normally
>
> Fixes: a97396c6eb13 ("PCI: pciehp: Ignore Link Down/Up caused by DPC")
> Cc: stable@vger.kernel.org
> Signed-off-by: Ziyou Wang (Lenovo) <ziyou.dev@foxmail.com>
> ---
30 minutes after v1 is too soon to post a v2. A couple days is a
reasonable interval.
v2 should include a note here (after the "---") about what changed
since v1 and ideally a lore link to the v1 posting.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds
2026-11-06 15:21 ` Bjorn Helgaas
@ 2026-11-19 3:30 ` Ziyou Wang (Lenovo)
2026-11-19 9:01 ` Lukas Wunner
0 siblings, 1 reply; 6+ messages in thread
From: Ziyou Wang (Lenovo) @ 2026-11-19 3:30 UTC (permalink / raw)
To: helgaas
Cc: bhelgaas, lukas, sathyanarayanan.kuppuswamy, linux-pci,
linux-kernel, stable, ziyou.dev
Hi Bjorn,
Thanks for your review!
On Fri, Nov 06, 2026 at 09:21:43AM -0600, Bjorn Helgaas wrote:
> Was there a "not ready ... after DPC; giving up" message here?
> pci_dev_wait() should log *something* about how it resolved this
> situation.
No. Here is the complete log from the DPC event to the re-enumeration:
[121.693046] pcieport 0000:00:03.1: EDR: EDR event received
[121.695743] pcieport 0000:00:03.1: DPC: containment event, status:0x1f01: unmasked uncorrectable error detected
[121.695825] pcieport 0000:00:03.1: PCIe Bus Error: severity=Uncorrectable (Fatal)
[121.695829] pcieport 0000:00:03.1: device [1022:1108] error status/mask=00200000/00000000
[121.695831] pcieport 0000:00:03.1: [21] ACSViol | Receiver | Transaction Layer (First)
[121.695834] pcieport 0000:00:03.1: AER: TLP Header: 0x60002080 0x010042ff 0x00000017 0x7e325c00
[121.695841] nvme nvme0: frozen state error detected, reset controller
[121.721168] nvme0n1: I/O Cmd(0x2) @ LBA 67600640, 256 blocks, I/O Error (sct 0x3 / sc 0x71)
[121.721182] I/O error, dev nvme0n1, sector 67600640 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[121.721193] nvme0n1: I/O Cmd(0x2) @ LBA 67600384, 256 blocks, I/O Error (sct 0x3 / sc 0x71)
[121.721196] I/O error, dev nvme0n1, sector 67600384 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[123.018691] nvme 0000:01:00.0: not ready 1023ms after DPC; waiting
[124.106690] nvme 0000:01:00.0: not ready 2047ms after DPC; waiting
[125.642703] pcieport 0000:00:03.1: pciehp: Slot(64): Link Down
[125.647840] Buffer I/O error on dev nvme0n1, logical block 8450048, async page read
[125.647846] Buffer I/O error on dev nvme0n1, logical block 8450049, async page read
[125.647847] Buffer I/O error on dev nvme0n1, logical block 8450050, async page read
[125.647848] Buffer I/O error on dev nvme0n1, logical block 8450051, async page read
[125.647850] Buffer I/O error on dev nvme0n1, logical block 8450052, async page read
[125.647850] Buffer I/O error on dev nvme0n1, logical block 8450053, async page read
[125.647851] Buffer I/O error on dev nvme0n1, logical block 8450054, async page read
[125.647852] Buffer I/O error on dev nvme0n1, logical block 8450055, async page read
[125.647853] Buffer I/O error on dev nvme0n1, logical block 8450056, async page read
[125.647854] Buffer I/O error on dev nvme0n1, logical block 8450057, async page read
[125.849608] pcieport 0000:00:03.1: pciehp: Slot(64): Card present
[125.849614] pcieport 0000:00:03.1: pciehp: Slot(64): Link Up
[126.218696] pcieport 0000:00:03.1: AER: subordinate device reset failed
[126.218700] pcieport 0000:00:03.1: AER: device recovery failed
[126.705745] pci 0000:01:00.0: [1dbe:5668] type 00 class 0x010802 PCIe Endpoint
[126.705786] pci 0000:01:00.0: BAR 0 [mem 0x00000000-0x00003fff 64bit]
[126.705793] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff pref]
[126.705801] pci 0000:01:00.0: Max Payload Size set to 512 (was 128, max 512)
[126.705987] pci 0000:01:00.0: _HPX attempts Link Control setting (AND 0xffff OR 0x0040)
[126.706073] pci 0000:01:00.0: supports D1 D2
[126.706074] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[126.706432] pci 0000:01:00.0: Adding to iommu group 16
[126.706481] pci 0000:01:00.0: ROM [mem 0xde300000-0xde30ffff pref]: assigned
[126.706483] pci 0000:01:00.0: BAR 0 [mem 0xde310000-0xde313fff 64bit]: assigned
[126.706491] pcieport 0000:00:03.1: PCI bridge to [bus 01]
[126.706494] pcieport 0000:00:03.1: bridge window [io 0x1000-0x1fff]
[126.706498] pcieport 0000:00:03.1: bridge window [mem 0xde300000-0xde3fffff]
[126.706500] pcieport 0000:00:03.1: bridge window [mem 0x10021600000-0x100217fffff 64bit pref]
[126.709685] nvme nvme1: pci function 0000:01:00.0
[126.709692] nvme 0000:01:00.0: enabling device (0140 -> 0142)
[126.712068] nvme nvme1: D3 entry latency set to 10 seconds
[126.720936] nvme nvme1: 129/0/0 default/read/poll queues
[126.727879] nvme0n2: p1 p2 p3
pci_dev_wait() never reached its timeout. The 4s pci_dpc_recovered()
timeout expired first, so pciehp removed the device and marked it
disconnected. When pci_dev_wait() woke up, it took the
pci_dev_is_disconnected() path, which only logs at debug level and
returns -ENOTTY. That is why the -ENOTTY surfaced as "AER:
subordinate device reset failed" instead.
The device needs a little over 4s to become ready ("ready 4095ms
after DPC" with this patch), so the 4s timeout is losing that race.
One thing worth noting: before e8df1d194c39b ("PCI: Don't treat ~0 as
ready in pci_dev_wait() with RRS SV"), this failure was masked on this
system. The RRS SV path only checked for the 0x0001 RRS Vendor ID, so
a ~0 read from the not-yet-ready device was treated as "ready",
pci_bridge_wait_for_secondary_bus() returned early, and the DPC
"recovery" appeared to succeed within the 4s window, even though the
device was not actually usable (nvme then failed with -19 after the
"successful" recovery). With e8df1d194c39b, pci_dev_wait() correctly
waits, which re-exposed this 4s race.
Best regards,
Ziyou Wang
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds
2026-11-19 3:30 ` Ziyou Wang (Lenovo)
@ 2026-11-19 9:01 ` Lukas Wunner
2026-11-20 3:22 ` Ziyou Wang (Lenovo)
0 siblings, 1 reply; 6+ messages in thread
From: Lukas Wunner @ 2026-11-19 9:01 UTC (permalink / raw)
To: Ziyou Wang (Lenovo)
Cc: helgaas, bhelgaas, sathyanarayanan.kuppuswamy, linux-pci,
linux-kernel, stable
On Thu, Nov 19, 2026 at 11:30:36AM +0800, Ziyou Wang (Lenovo) wrote:
> [121.693046] pcieport 0000:00:03.1: EDR: EDR event received
> [121.695743] pcieport 0000:00:03.1: DPC: containment event, status:0x1f01: unmasked uncorrectable error detected
> [121.695825] pcieport 0000:00:03.1: PCIe Bus Error: severity=Uncorrectable (Fatal)
> [121.695829] pcieport 0000:00:03.1: device [1022:1108] error status/mask=00200000/00000000
> [121.695831] pcieport 0000:00:03.1: [21] ACSViol | Receiver | Transaction Layer (First)
> [121.695834] pcieport 0000:00:03.1: AER: TLP Header: 0x60002080 0x010042ff 0x00000017 0x7e325c00
[...]
> [123.018691] nvme 0000:01:00.0: not ready 1023ms after DPC; waiting
I'm wondering why the ACS Violation Error is raised in the first place.
The TLP causing the DPC event is a Memory Write Request from the NVMe
drive to memory address 0x000000177e325c00. The Requester ID 0x100
seems to match the bus address 0000:01:00.0.
What are the ACS settings at the Root Port and why is it signaling an
ACS violation?
TLP decoded with tlp-tool (https://github.com/mmpg-x86/tlp-tool):
+----------+-------------+----------------------+
| TLP Type | MemWriteReq | 4DW with Data Header |
+----------+-------------+----------------------+
+------------+--------+--------+-------+
| Field Name | Offset | Length | Value |
| | (bits) | (bits) | |
+------------+--------+--------+-------+
| Fmt | 0 | 3 | 3 |
| Type | 3 | 5 | 0 |
| T9 | 8 | 1 | 0 |
| TC | 9 | 3 | 0 |
| T8 | 12 | 1 | 0 |
| Attr_b2 | 13 | 1 | 0 |
| LN | 14 | 1 | 0 |
| TH | 15 | 1 | 0 |
| Td | 16 | 1 | 0 |
| Ep | 17 | 1 | 0 |
| Attr | 18 | 2 | 2 |
| AT | 20 | 2 | 0 |
| Length | 22 | 10 | 128 |
+------------+--------+--------+-------+
+-----------------+----------------------+
| TLP: | 4DW with Data Header |
+-----------------+----------------------+
| Req ID | 0x100 |
| Tag | 0x42 |
| Last DW BE | 0xF |
| First DW BE | 0xF |
| Addr High (DW2) | 0x00000017 |
| Addr Low (DW3) | 0x7E325C00 |
+-----------------+----------------------+
Thanks,
Lukas
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds
2026-11-19 9:01 ` Lukas Wunner
@ 2026-11-20 3:22 ` Ziyou Wang (Lenovo)
0 siblings, 0 replies; 6+ messages in thread
From: Ziyou Wang (Lenovo) @ 2026-11-20 3:22 UTC (permalink / raw)
To: lukas
Cc: helgaas, bhelgaas, sathyanarayanan.kuppuswamy, linux-pci,
linux-kernel, stable
Hi Lukas,
The ACS Violation in that log is not organic, it is injected on purpose.
The platform is AMD and we use AMD's RAS Error Injection Test tool to
inject an "ACS fatal" error at the Root Port. The test steps are:
1. Shorten the machine check polling interval:
echo 1 > /sys/devices/system/machinecheck/machinecheck0/check_interval
2. List the PCIe slot info to find the socket/die/NBIO, PCIe core and
port that the NVMe drive is attached to:
./cmdline_amdraserrorinjectiontest --listpcieslotinfo
3. Start I/O traffic on the NVMe drive:
dd if=/dev/nvmeXnY of=/dev/null bs=64k status=progress conv=noerror,sync
4. Inject the ACS fatal error at that Root Port:
./cmdline_amdraserrorinjectiontest --pcieerror \
--sockdienbio <socket:die:nbio> --errortype 4 \
--pciecore <core> --pcieport <port> --enablestatusregisterpolling 1
This raises ACSViol in the Root Port's AER Uncorrectable Error Status.
The BIOS has AER and ACS enabled and AER reporting is set to firmware
first, so firmware signals the EDR event and DPC containment follows.
The TLP you decoded is a regular in-flight DMA write from the NVMe drive
that happened to be captured in the AER Header Log (there was read
traffic on the drive at injection time). Its Requester ID is valid and
Source Validation is enabled, so nothing in it violates any ACS rule.
The ACSViol comes from the injector, not from real ACS logic.
The ACS settings on the Root Port (same platform, 1022:1108):
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
The goal of the test was to produce a DPC containment event on an NVMe
device that needs about 4 seconds to become config-ready again, in order
to exercise the recovery path.
Thanks,
Ziyou
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-11-20 3:22 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20261106082612.1840817-1-ziyou.dev@foxmail.com>
2026-11-06 8:56 ` [PATCH v2] PCI/DPC: Increase recovery timeout to 60 seconds Ziyou Wang (Lenovo)
2026-11-06 9:05 ` sashiko-bot
2026-11-06 15:21 ` Bjorn Helgaas
2026-11-19 3:30 ` Ziyou Wang (Lenovo)
2026-11-19 9:01 ` Lukas Wunner
2026-11-20 3:22 ` Ziyou Wang (Lenovo)
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.