[PATCH] PCI/DPC: Extend DPC recovery timeout

[PATCH] PCI/DPC: Extend DPC recovery timeout

[Andy Xu]

From: Hongbo Yao <andy.xu@hj-micro.com>

Extend the DPC recovery timeout from 4 seconds to 7 seconds to
support Mellanox ConnectX series network adapters.

My environment:
  - Platform: arm64 N2 based server
  - Endpoint1: Mellanox Technologies MT27800 Family [ConnectX-5]
  - Endpoint2: Mellanox Technologies MT2910 Family [ConnectX-7]

With the original 4s timeout, hotplug would still be triggered:

[ 81.012463] pcieport 0004:00:00.0: DPC: containment event, status:0x1f01 source:0x0000
[ 81.014536] pcieport 0004:00:00.0: DPC: unmasked uncorrectable error detected
[ 81.029598] pcieport 0004:00:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Receiver ID)
[ 81.040830] pcieport 0004:00:00.0: device [0823:0110] error status/mask=00008000/04d40000
[ 81.049870] pcieport 0004:00:00.0: [ 0] ERCR (First)
[ 81.053520] pcieport 0004:00:00.0: AER: TLP Header: 60008010 010000ff 00001000 9c4c0000
[ 81.065793] mlx5_core 0004:01:00.0: mlx5_pci_err_detected Device state = 1 health sensors: 1 pci_status: 1. Enter, pci channel state = 2
[ 81.076183] mlx5_core 0004:01:00.0: mlx5_error_sw_reset:231:(pid 1618): start
[ 81.083307] mlx5_core 0004:01:00.0: mlx5_error_sw_reset:252:(pid 1618): PCI channel offline, stop waiting for NIC IFC
[ 81.077428] mlx5_core 0004:01:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), neovfs(0), active vports(0)
[ 81.486693] mlx5_core 0004:01:00.0: mlx5_wait_for_pages:786:(pid 1618): Skipping wait for vf pages stage
[ 81.496965] mlx5_core 0004:01:00.0: mlx5_wait_for_pages:786:(pid 1618): Skipping wait for vf pages stage
[ 82.395040] mlx5_core 0004:01:00.1: print_health:819:(pid 0): Fatal error detected
[ 82.395493] mlx5_core 0004:01:00.1: print_health_info:423:(pid 0): PCI slot 1 is unavailable
[ 83.431094] mlx5_core 0004:01:00.0: mlx5_pci_err_detected Device state = 2 pci_status: 0. Exit, result = 3, need reset
[ 83.442100] mlx5_core 0004:01:00.1: mlx5_pci_err_detected Device state = 2 health sensors: 1 pci_status: 1. Enter, pci channel state = 2
[ 83.441801] mlx5_core 0004:01:00.0: mlx5_crdump_collect:50:(pid 2239): crdump: failed to lock gw status -13
[ 83.454050] mlx5_core 0004:01:00.1: mlx5_error_sw_reset:231:(pid 1618): start
[ 83.454050] mlx5_core 0004:01:00.1: mlx5_error_sw_reset:252:(pid 1618): PCI channel offline, stop waiting for NIC IFC
[ 83.849429] mlx5_core 0004:01:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), neovfs(0), active vports(0)
[ 83.858892] mlx5_core 0004:01:00.1: mlx5_wait_for_pages:786:(pid 1618): Skipping wait for vf pages stage
[ 83.869464] mlx5_core 0004:01:00.1: mlx5_wait_for_pages:786:(pid 1618): Skipping wait for vf pages stage
[ 85.201433] pcieport 0004:00:00.0: pciehp: Slot(41): Link Down
[ 85.815016] mlx5_core 0004:01:00.1: mlx5_health_try_recover:335:(pid 2239): handling bad device here
[ 85.824164] mlx5_core 0004:01:00.1: mlx5_error_sw_reset:231:(pid 2239): start
[ 85.831283] mlx5_core 0004:01:00.1: mlx5_error_sw_reset:252:(pid 2239): PCI channel offline, stop waiting for NIC IFC
[ 85.841899] mlx5_core 0004:01:00.1: mlx5_unload_one_dev_locked:1612:(pid 2239): mlx5_unload_one_dev_locked: interface is down, NOP
[ 85.853799] mlx5_core 0004:01:00.1: mlx5_health_wait_pci_up:325:(pid 2239): PCI channel offline, stop waiting for PCI
[ 85.863494] mlx5_core 0004:01:00.1: mlx5_health_try_recover:338:(pid 2239): health recovery flow aborted, PCI reads still not working
[ 85.873231] mlx5_core 0004:01:00.1: mlx5_pci_err_detected Device state = 2 pci_status: 0. Exit, result = 3, need reset
[ 85.879899] mlx5_core 0004:01:00.1: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), neovfs(0), active vports(0)
[ 85.921428] mlx5_core 0004:01:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), neovfs(0), active vports(0)
[ 85.930491] mlx5_core 0004:01:00.1: mlx5_wait_for_pages:786:(pid 1617): Skipping wait for vf pages stage
[ 85.940849] mlx5_core 0004:01:00.1: mlx5_wait_for_pages:786:(pid 1617): Skipping wait for vf pages stage
[ 85.949971] mlx5_core 0004:01:00.1: mlx5_uninit_one:1528:(pid 1617): mlx5_uninit_one: interface is down, NOP
[ 85.959944] mlx5_core 0004:01:00.1: E-Switch: cleanup
[ 86.035541] mlx5_core 0004:01:00.0: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), neovfs(0), active vports(0)
[ 86.077568] mlx5_core 0004:01:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), neovfs(0), active vports(0)
[ 86.071727] mlx5_core 0004:01:00.0: mlx5_wait_for_pages:786:(pid 1617): Skipping wait for vf pages stage
[ 86.096577] mlx5_core 0004:01:00.0: mlx5_wait_for_pages:786:(pid 1617): Skipping wait for vf pages stage
[ 86.106909] mlx5_core 0004:01:00.0: mlx5_uninit_one:1528:(pid 1617): mlx5_uninit_one: interface is down, NOP
[ 86.115940] pcieport 0004:00:00.0: AER: subordinate device reset failed
[ 86.122557] pcieport 0004:00:00.0: AER: device recovery failed
[ 86.128571] mlx5_core 0004:01:00.0: E-Switch: cleanup

I added some prints and found that:
 - ConnectX-5 requires >5s for full recovery
 - ConnectX-7 requires >6s for full recovery

Setting timeout to 7s covers both devices with safety margin.

Signed-off-by: Hongbo Yao <andy.xu@hj-micro.com>
---
 drivers/pci/pcie/dpc.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/pcie/dpc.c b/drivers/pci/pcie/dpc.c
index fc18349614d7..35a37fd86dcd 100644
--- a/drivers/pci/pcie/dpc.c
+++ b/drivers/pci/pcie/dpc.c
@@ -118,10 +118,10 @@ bool pci_dpc_recovered(struct pci_dev *pdev)
 	/*
 	 * Need a timeout in case DPC never completes due to failure of
 	 * dpc_wait_rp_inactive().  The spec doesn't mandate a time limit,
-	 * but reports indicate that DPC completes within 4 seconds.
+	 * but reports indicate that DPC completes within 7 seconds.
 	 */
 	wait_event_timeout(dpc_completed_waitqueue, dpc_completed(pdev),
-			   msecs_to_jiffies(4000));
+			   msecs_to_jiffies(7000));
 
 	return test_and_clear_bit(PCI_DPC_RECOVERED, &pdev->priv_flags);
 }
-- 
2.43.0

Re: [PATCH] PCI/DPC: Extend DPC recovery timeout

[Sathyanarayanan Kuppuswamy]

On 7/7/25 3:30 AM, Andy Xu wrote:
> From: Hongbo Yao <andy.xu@hj-micro.com>
>
> Extend the DPC recovery timeout from 4 seconds to 7 seconds to
> support Mellanox ConnectX series network adapters.
>
> My environment:
>    - Platform: arm64 N2 based server
>    - Endpoint1: Mellanox Technologies MT27800 Family [ConnectX-5]
>    - Endpoint2: Mellanox Technologies MT2910 Family [ConnectX-7]
>
> With the original 4s timeout, hotplug would still be triggered:
>
> [ 81.012463] pcieport 0004:00:00.0: DPC: containment event, status:0x1f01 source:0x0000
> [ 81.014536] pcieport 0004:00:00.0: DPC: unmasked uncorrectable error detected
> [ 81.029598] pcieport 0004:00:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Receiver ID)
> [ 81.040830] pcieport 0004:00:00.0: device [0823:0110] error status/mask=00008000/04d40000
> [ 81.049870] pcieport 0004:00:00.0: [ 0] ERCR (First)
> [ 81.053520] pcieport 0004:00:00.0: AER: TLP Header: 60008010 010000ff 00001000 9c4c0000
> [ 81.065793] mlx5_core 0004:01:00.0: mlx5_pci_err_detected Device state = 1 health sensors: 1 pci_status: 1. Enter, pci channel state = 2
> [ 81.076183] mlx5_core 0004:01:00.0: mlx5_error_sw_reset:231:(pid 1618): start
> [ 81.083307] mlx5_core 0004:01:00.0: mlx5_error_sw_reset:252:(pid 1618): PCI channel offline, stop waiting for NIC IFC
> [ 81.077428] mlx5_core 0004:01:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), neovfs(0), active vports(0)
> [ 81.486693] mlx5_core 0004:01:00.0: mlx5_wait_for_pages:786:(pid 1618): Skipping wait for vf pages stage
> [ 81.496965] mlx5_core 0004:01:00.0: mlx5_wait_for_pages:786:(pid 1618): Skipping wait for vf pages stage
> [ 82.395040] mlx5_core 0004:01:00.1: print_health:819:(pid 0): Fatal error detected
> [ 82.395493] mlx5_core 0004:01:00.1: print_health_info:423:(pid 0): PCI slot 1 is unavailable
> [ 83.431094] mlx5_core 0004:01:00.0: mlx5_pci_err_detected Device state = 2 pci_status: 0. Exit, result = 3, need reset
> [ 83.442100] mlx5_core 0004:01:00.1: mlx5_pci_err_detected Device state = 2 health sensors: 1 pci_status: 1. Enter, pci channel state = 2
> [ 83.441801] mlx5_core 0004:01:00.0: mlx5_crdump_collect:50:(pid 2239): crdump: failed to lock gw status -13
> [ 83.454050] mlx5_core 0004:01:00.1: mlx5_error_sw_reset:231:(pid 1618): start
> [ 83.454050] mlx5_core 0004:01:00.1: mlx5_error_sw_reset:252:(pid 1618): PCI channel offline, stop waiting for NIC IFC
> [ 83.849429] mlx5_core 0004:01:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), neovfs(0), active vports(0)
> [ 83.858892] mlx5_core 0004:01:00.1: mlx5_wait_for_pages:786:(pid 1618): Skipping wait for vf pages stage
> [ 83.869464] mlx5_core 0004:01:00.1: mlx5_wait_for_pages:786:(pid 1618): Skipping wait for vf pages stage
> [ 85.201433] pcieport 0004:00:00.0: pciehp: Slot(41): Link Down
> [ 85.815016] mlx5_core 0004:01:00.1: mlx5_health_try_recover:335:(pid 2239): handling bad device here
> [ 85.824164] mlx5_core 0004:01:00.1: mlx5_error_sw_reset:231:(pid 2239): start
> [ 85.831283] mlx5_core 0004:01:00.1: mlx5_error_sw_reset:252:(pid 2239): PCI channel offline, stop waiting for NIC IFC
> [ 85.841899] mlx5_core 0004:01:00.1: mlx5_unload_one_dev_locked:1612:(pid 2239): mlx5_unload_one_dev_locked: interface is down, NOP
> [ 85.853799] mlx5_core 0004:01:00.1: mlx5_health_wait_pci_up:325:(pid 2239): PCI channel offline, stop waiting for PCI
> [ 85.863494] mlx5_core 0004:01:00.1: mlx5_health_try_recover:338:(pid 2239): health recovery flow aborted, PCI reads still not working
> [ 85.873231] mlx5_core 0004:01:00.1: mlx5_pci_err_detected Device state = 2 pci_status: 0. Exit, result = 3, need reset
> [ 85.879899] mlx5_core 0004:01:00.1: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), neovfs(0), active vports(0)
> [ 85.921428] mlx5_core 0004:01:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), neovfs(0), active vports(0)
> [ 85.930491] mlx5_core 0004:01:00.1: mlx5_wait_for_pages:786:(pid 1617): Skipping wait for vf pages stage
> [ 85.940849] mlx5_core 0004:01:00.1: mlx5_wait_for_pages:786:(pid 1617): Skipping wait for vf pages stage
> [ 85.949971] mlx5_core 0004:01:00.1: mlx5_uninit_one:1528:(pid 1617): mlx5_uninit_one: interface is down, NOP
> [ 85.959944] mlx5_core 0004:01:00.1: E-Switch: cleanup
> [ 86.035541] mlx5_core 0004:01:00.0: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), neovfs(0), active vports(0)
> [ 86.077568] mlx5_core 0004:01:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), neovfs(0), active vports(0)
> [ 86.071727] mlx5_core 0004:01:00.0: mlx5_wait_for_pages:786:(pid 1617): Skipping wait for vf pages stage
> [ 86.096577] mlx5_core 0004:01:00.0: mlx5_wait_for_pages:786:(pid 1617): Skipping wait for vf pages stage
> [ 86.106909] mlx5_core 0004:01:00.0: mlx5_uninit_one:1528:(pid 1617): mlx5_uninit_one: interface is down, NOP
> [ 86.115940] pcieport 0004:00:00.0: AER: subordinate device reset failed
> [ 86.122557] pcieport 0004:00:00.0: AER: device recovery failed
> [ 86.128571] mlx5_core 0004:01:00.0: E-Switch: cleanup
>
> I added some prints and found that:
>   - ConnectX-5 requires >5s for full recovery
>   - ConnectX-7 requires >6s for full recovery
>
> Setting timeout to 7s covers both devices with safety margin.


Instead of updating the recovery time, can you check why your device recovery takes
such a long time and how to fix it from the device end?


> Signed-off-by: Hongbo Yao <andy.xu@hj-micro.com>
> ---
>   drivers/pci/pcie/dpc.c | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/pci/pcie/dpc.c b/drivers/pci/pcie/dpc.c
> index fc18349614d7..35a37fd86dcd 100644
> --- a/drivers/pci/pcie/dpc.c
> +++ b/drivers/pci/pcie/dpc.c
> @@ -118,10 +118,10 @@ bool pci_dpc_recovered(struct pci_dev *pdev)
>   	/*
>   	 * Need a timeout in case DPC never completes due to failure of
>   	 * dpc_wait_rp_inactive().  The spec doesn't mandate a time limit,
> -	 * but reports indicate that DPC completes within 4 seconds.
> +	 * but reports indicate that DPC completes within 7 seconds.
>   	 */
>   	wait_event_timeout(dpc_completed_waitqueue, dpc_completed(pdev),
> -			   msecs_to_jiffies(4000));
> +			   msecs_to_jiffies(7000));
>   
>   	return test_and_clear_bit(PCI_DPC_RECOVERED, &pdev->priv_flags);
>   }

-- 
Sathyanarayanan Kuppuswamy
Linux Kernel Developer

Re: [PATCH] PCI/DPC: Extend DPC recovery timeout

[Hongbo Yao]

在 2025/7/8 1:04, Sathyanarayanan Kuppuswamy 写道:
> 
> On 7/7/25 3:30 AM, Andy Xu wrote:
>> From: Hongbo Yao <andy.xu@hj-micro.com>
>>
>> Extend the DPC recovery timeout from 4 seconds to 7 seconds to
>> support Mellanox ConnectX series network adapters.
>>
>> My environment:
>>    - Platform: arm64 N2 based server
>>    - Endpoint1: Mellanox Technologies MT27800 Family [ConnectX-5]
>>    - Endpoint2: Mellanox Technologies MT2910 Family [ConnectX-7]
>>
>> With the original 4s timeout, hotplug would still be triggered:
>>
>> [ 81.012463] pcieport 0004:00:00.0: DPC: containment event,
>> status:0x1f01 source:0x0000
>> [ 81.014536] pcieport 0004:00:00.0: DPC: unmasked uncorrectable error
>> detected
>> [ 81.029598] pcieport 0004:00:00.0: PCIe Bus Error:
>> severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Receiver ID)
>> [ 81.040830] pcieport 0004:00:00.0: device [0823:0110] error status/
>> mask=00008000/04d40000
>> [ 81.049870] pcieport 0004:00:00.0: [ 0] ERCR (First)
>> [ 81.053520] pcieport 0004:00:00.0: AER: TLP Header: 60008010 010000ff
>> 00001000 9c4c0000
>> [ 81.065793] mlx5_core 0004:01:00.0: mlx5_pci_err_detected Device
>> state = 1 health sensors: 1 pci_status: 1. Enter, pci channel state = 2
>> [ 81.076183] mlx5_core 0004:01:00.0: mlx5_error_sw_reset:231:(pid
>> 1618): start
>> [ 81.083307] mlx5_core 0004:01:00.0: mlx5_error_sw_reset:252:(pid
>> 1618): PCI channel offline, stop waiting for NIC IFC
>> [ 81.077428] mlx5_core 0004:01:00.0: E-Switch: Disable: mode(LEGACY),
>> nvfs(0), neovfs(0), active vports(0)
>> [ 81.486693] mlx5_core 0004:01:00.0: mlx5_wait_for_pages:786:(pid
>> 1618): Skipping wait for vf pages stage
>> [ 81.496965] mlx5_core 0004:01:00.0: mlx5_wait_for_pages:786:(pid
>> 1618): Skipping wait for vf pages stage
>> [ 82.395040] mlx5_core 0004:01:00.1: print_health:819:(pid 0): Fatal
>> error detected
>> [ 82.395493] mlx5_core 0004:01:00.1: print_health_info:423:(pid 0):
>> PCI slot 1 is unavailable
>> [ 83.431094] mlx5_core 0004:01:00.0: mlx5_pci_err_detected Device
>> state = 2 pci_status: 0. Exit, result = 3, need reset
>> [ 83.442100] mlx5_core 0004:01:00.1: mlx5_pci_err_detected Device
>> state = 2 health sensors: 1 pci_status: 1. Enter, pci channel state = 2
>> [ 83.441801] mlx5_core 0004:01:00.0: mlx5_crdump_collect:50:(pid
>> 2239): crdump: failed to lock gw status -13
>> [ 83.454050] mlx5_core 0004:01:00.1: mlx5_error_sw_reset:231:(pid
>> 1618): start
>> [ 83.454050] mlx5_core 0004:01:00.1: mlx5_error_sw_reset:252:(pid
>> 1618): PCI channel offline, stop waiting for NIC IFC
>> [ 83.849429] mlx5_core 0004:01:00.1: E-Switch: Disable: mode(LEGACY),
>> nvfs(0), neovfs(0), active vports(0)
>> [ 83.858892] mlx5_core 0004:01:00.1: mlx5_wait_for_pages:786:(pid
>> 1618): Skipping wait for vf pages stage
>> [ 83.869464] mlx5_core 0004:01:00.1: mlx5_wait_for_pages:786:(pid
>> 1618): Skipping wait for vf pages stage
>> [ 85.201433] pcieport 0004:00:00.0: pciehp: Slot(41): Link Down
>> [ 85.815016] mlx5_core 0004:01:00.1: mlx5_health_try_recover:335:(pid
>> 2239): handling bad device here
>> [ 85.824164] mlx5_core 0004:01:00.1: mlx5_error_sw_reset:231:(pid
>> 2239): start
>> [ 85.831283] mlx5_core 0004:01:00.1: mlx5_error_sw_reset:252:(pid
>> 2239): PCI channel offline, stop waiting for NIC IFC
>> [ 85.841899] mlx5_core 0004:01:00.1: mlx5_unload_one_dev_locked:1612:
>> (pid 2239): mlx5_unload_one_dev_locked: interface is down, NOP
>> [ 85.853799] mlx5_core 0004:01:00.1: mlx5_health_wait_pci_up:325:(pid
>> 2239): PCI channel offline, stop waiting for PCI
>> [ 85.863494] mlx5_core 0004:01:00.1: mlx5_health_try_recover:338:(pid
>> 2239): health recovery flow aborted, PCI reads still not working
>> [ 85.873231] mlx5_core 0004:01:00.1: mlx5_pci_err_detected Device
>> state = 2 pci_status: 0. Exit, result = 3, need reset
>> [ 85.879899] mlx5_core 0004:01:00.1: E-Switch: Unload vfs:
>> mode(LEGACY), nvfs(0), neovfs(0), active vports(0)
>> [ 85.921428] mlx5_core 0004:01:00.1: E-Switch: Disable: mode(LEGACY),
>> nvfs(0), neovfs(0), active vports(0)
>> [ 85.930491] mlx5_core 0004:01:00.1: mlx5_wait_for_pages:786:(pid
>> 1617): Skipping wait for vf pages stage
>> [ 85.940849] mlx5_core 0004:01:00.1: mlx5_wait_for_pages:786:(pid
>> 1617): Skipping wait for vf pages stage
>> [ 85.949971] mlx5_core 0004:01:00.1: mlx5_uninit_one:1528:(pid 1617):
>> mlx5_uninit_one: interface is down, NOP
>> [ 85.959944] mlx5_core 0004:01:00.1: E-Switch: cleanup
>> [ 86.035541] mlx5_core 0004:01:00.0: E-Switch: Unload vfs:
>> mode(LEGACY), nvfs(0), neovfs(0), active vports(0)
>> [ 86.077568] mlx5_core 0004:01:00.0: E-Switch: Disable: mode(LEGACY),
>> nvfs(0), neovfs(0), active vports(0)
>> [ 86.071727] mlx5_core 0004:01:00.0: mlx5_wait_for_pages:786:(pid
>> 1617): Skipping wait for vf pages stage
>> [ 86.096577] mlx5_core 0004:01:00.0: mlx5_wait_for_pages:786:(pid
>> 1617): Skipping wait for vf pages stage
>> [ 86.106909] mlx5_core 0004:01:00.0: mlx5_uninit_one:1528:(pid 1617):
>> mlx5_uninit_one: interface is down, NOP
>> [ 86.115940] pcieport 0004:00:00.0: AER: subordinate device reset failed
>> [ 86.122557] pcieport 0004:00:00.0: AER: device recovery failed
>> [ 86.128571] mlx5_core 0004:01:00.0: E-Switch: cleanup
>>
>> I added some prints and found that:
>>   - ConnectX-5 requires >5s for full recovery
>>   - ConnectX-7 requires >6s for full recovery
>>
>> Setting timeout to 7s covers both devices with safety margin.
> 
> 
> Instead of updating the recovery time, can you check why your device
> recovery takes
> such a long time and how to fix it from the device end?
> 
Hi, Sathyanarayanan.

Thanks for the valuable feedback and suggestions.

I fully agree that ideally the root cause should be addressed on the
device side to reduce the DPC recovery latency, and that waiting longer
in the kernel is not a perfect solution.

However, the current 4 seconds timeout in pci_dpc_recovered() is indeed
an empirical value rather than a hard requirement from the PCIe
specification. In real-world scenarios, like with Mellanox ConnectX-5/7
adapters, we've observed that full DPC recovery can take more than 5-6
seconds, which leads to premature hotplug processing and device removal.

To improve robustness and maintain flexibility, I’m considering
introducing a module parameter to allow tuning the DPC recovery timeout
dynamically. Would you like me to prepare and submit such a patch for
review?


Best regards,
Hongbo Yao


> 
>> Signed-off-by: Hongbo Yao <andy.xu@hj-micro.com>
>> ---
>>   drivers/pci/pcie/dpc.c | 4 ++--
>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/pci/pcie/dpc.c b/drivers/pci/pcie/dpc.c
>> index fc18349614d7..35a37fd86dcd 100644
>> --- a/drivers/pci/pcie/dpc.c
>> +++ b/drivers/pci/pcie/dpc.c
>> @@ -118,10 +118,10 @@ bool pci_dpc_recovered(struct pci_dev *pdev)
>>       /*
>>        * Need a timeout in case DPC never completes due to failure of
>>        * dpc_wait_rp_inactive().  The spec doesn't mandate a time limit,
>> -     * but reports indicate that DPC completes within 4 seconds.
>> +     * but reports indicate that DPC completes within 7 seconds.
>>        */
>>       wait_event_timeout(dpc_completed_waitqueue, dpc_completed(pdev),
>> -               msecs_to_jiffies(4000));
>> +               msecs_to_jiffies(7000));
>>         return test_and_clear_bit(PCI_DPC_RECOVERED, &pdev->priv_flags);
>>   }
> 

Re: [PATCH] PCI/DPC: Extend DPC recovery timeout

[Lukas Wunner]

On Fri, Jul 11, 2025 at 11:20:15AM +0800, Hongbo Yao wrote:
> 2025/7/8 1:04, Sathyanarayanan Kuppuswamy:
> > On 7/7/25 3:30 AM, Andy Xu wrote:
> > > Setting timeout to 7s covers both devices with safety margin.
> > 
> > Instead of updating the recovery time, can you check why your device
> > recovery takes
> > such a long time and how to fix it from the device end?
> 
> I fully agree that ideally the root cause should be addressed on the
> device side to reduce the DPC recovery latency, and that waiting longer
> in the kernel is not a perfect solution.
> 
> However, the current 4 seconds timeout in pci_dpc_recovered() is indeed
> an empirical value rather than a hard requirement from the PCIe
> specification. In real-world scenarios, like with Mellanox ConnectX-5/7
> adapters, we've observed that full DPC recovery can take more than 5-6
> seconds, which leads to premature hotplug processing and device removal.

I think Sathya's point was:  Have you made an effort to talk to the
vendor and ask them to root-cause and fix the issue e.g. with a firmware
update.

> To improve robustness and maintain flexibility, I???m considering
> introducing a module parameter to allow tuning the DPC recovery timeout
> dynamically. Would you like me to prepare and submit such a patch for
> review?

We try to avoid adding new module parameters.  Things should just work
out of the box without the user having to adjust the kernel command
line for their system.

So the solution is indeed to either adjust the delay for everyone
(as you've done) or introduce an unsigned int to struct pci_dev
which can be assigned the delay after reset for the device to be
responsive.

For comparison, we're allowing up to 60 sec for devices to become
available after a Fundamental Reset or Conventional Reset
(PCIE_RESET_READY_POLL_MS).  That's how long we're waiting in
dpc_reset_link() -> pci_bridge_wait_for_secondary_bus() and
we're not consistent with that when we wait only 4 sec in
pci_dpc_recovered().

I think the reason is that we weren't really sure whether this approach
to synchronize hotplug with DPC works well and how to choose delays.
But we've had this for a few years now and it seems to have worked nicely
for people.  I think this is the first report where it's not been
working out of the box.

Thanks,

Lukas

Re: [PATCH] PCI/DPC: Extend DPC recovery timeout

[Bjorn Helgaas]

On Fri, Jul 11, 2025 at 06:13:01AM +0200, Lukas Wunner wrote:
> On Fri, Jul 11, 2025 at 11:20:15AM +0800, Hongbo Yao wrote:
> > 2025/7/8 1:04, Sathyanarayanan Kuppuswamy:
> > > On 7/7/25 3:30 AM, Andy Xu wrote:
> > > > Setting timeout to 7s covers both devices with safety margin.
> > > 
> > > Instead of updating the recovery time, can you check why your device
> > > recovery takes
> > > such a long time and how to fix it from the device end?
> > 
> > I fully agree that ideally the root cause should be addressed on the
> > device side to reduce the DPC recovery latency, and that waiting longer
> > in the kernel is not a perfect solution.
> > 
> > However, the current 4 seconds timeout in pci_dpc_recovered() is indeed
> > an empirical value rather than a hard requirement from the PCIe
> > specification. In real-world scenarios, like with Mellanox ConnectX-5/7
> > adapters, we've observed that full DPC recovery can take more than 5-6
> > seconds, which leads to premature hotplug processing and device removal.
> 
> I think Sathya's point was:  Have you made an effort to talk to the
> vendor and ask them to root-cause and fix the issue e.g. with a firmware
> update.

Would definitely be great, but unless we have a number in the spec to
point to, they might just shrug and ask what the requirement is.

> > To improve robustness and maintain flexibility, I???m considering
> > introducing a module parameter to allow tuning the DPC recovery timeout
> > dynamically. Would you like me to prepare and submit such a patch for
> > review?
> 
> We try to avoid adding new module parameters.  Things should just work
> out of the box without the user having to adjust the kernel command
> line for their system.
> 
> So the solution is indeed to either adjust the delay for everyone
> (as you've done) or introduce an unsigned int to struct pci_dev
> which can be assigned the delay after reset for the device to be
> responsive.
> 
> For comparison, we're allowing up to 60 sec for devices to become
> available after a Fundamental Reset or Conventional Reset
> (PCIE_RESET_READY_POLL_MS).  That's how long we're waiting in
> dpc_reset_link() -> pci_bridge_wait_for_secondary_bus() and
> we're not consistent with that when we wait only 4 sec in
> pci_dpc_recovered().
> 
> I think the reason is that we weren't really sure whether this approach
> to synchronize hotplug with DPC works well and how to choose delays.
> But we've had this for a few years now and it seems to have worked nicely
> for people.  I think this is the first report where it's not been
> working out of the box.

Why would we wait less than PCIE_RESET_READY_POLL_MS?  DPC disables
the link, so that's basically a reset for the device.  Seems like we
should allow as much time as we do for any other kind of reset.

Bjorn

Re: [PATCH] PCI/DPC: Extend DPC recovery timeout

[Keith Busch]

On Wed, Aug 06, 2025 at 04:34:09PM -0500, Bjorn Helgaas wrote:
> > > However, the current 4 seconds timeout in pci_dpc_recovered() is indeed
> > > an empirical value rather than a hard requirement from the PCIe
> > > specification. In real-world scenarios, like with Mellanox ConnectX-5/7
> > > adapters, we've observed that full DPC recovery can take more than 5-6
> > > seconds, which leads to premature hotplug processing and device removal.
> > 
> > I think Sathya's point was:  Have you made an effort to talk to the
> > vendor and ask them to root-cause and fix the issue e.g. with a firmware
> > update.
> 
> Would definitely be great, but unless we have a number in the spec to
> point to, they might just shrug and ask what the requirement is.

I agree, and I have similar problems with other arbitrary kernel timing
decicsions. Specifically RRL where there's no spec defined number yet my
patch to modify it has not received much consideration.

  https://lore.kernel.org/linux-pci/20250218165444.2406119-1-kbusch@meta.com/

Re: [PATCH] PCI/DPC: Extend DPC recovery timeout

[Ethan Zhao]

On 8/7/2025 5:52 AM, Keith Busch wrote:
> On Wed, Aug 06, 2025 at 04:34:09PM -0500, Bjorn Helgaas wrote:
>>>> However, the current 4 seconds timeout in pci_dpc_recovered() is indeed
>>>> an empirical value rather than a hard requirement from the PCIe
>>>> specification. In real-world scenarios, like with Mellanox ConnectX-5/7
>>>> adapters, we've observed that full DPC recovery can take more than 5-6
>>>> seconds, which leads to premature hotplug processing and device removal.
>>>
>>> I think Sathya's point was:  Have you made an effort to talk to the
>>> vendor and ask them to root-cause and fix the issue e.g. with a firmware
>>> update.
>>
>> Would definitely be great, but unless we have a number in the spec to
>> point to, they might just shrug and ask what the requirement is.
> 
> I agree, and I have similar problems with other arbitrary kernel timing
> decicsions. Specifically RRL where there's no spec defined number yet my
> patch to modify it has not received much consideration.
> 
>    https://lore.kernel.org/linux-pci/20250218165444.2406119-1-kbusch@meta.com/
> 
At least, with this patch, have a workaround in hand to make some device 
work.

Thanks,
Ethan

Re: [PATCH] PCI/DPC: Extend DPC recovery timeout

[Ethan Zhao]

On 7/11/2025 11:20 AM, Hongbo Yao wrote:
> 
> 
> 在 2025/7/8 1:04, Sathyanarayanan Kuppuswamy 写道:
>>
>> On 7/7/25 3:30 AM, Andy Xu wrote:
>>> From: Hongbo Yao <andy.xu@hj-micro.com>
>>>
>>> Extend the DPC recovery timeout from 4 seconds to 7 seconds to
>>> support Mellanox ConnectX series network adapters.
>>>
>>> My environment:
>>>     - Platform: arm64 N2 based server
>>>     - Endpoint1: Mellanox Technologies MT27800 Family [ConnectX-5]
>>>     - Endpoint2: Mellanox Technologies MT2910 Family [ConnectX-7]
>>>
>>> With the original 4s timeout, hotplug would still be triggered:
>>>
>>> [ 81.012463] pcieport 0004:00:00.0: DPC: containment event,
>>> status:0x1f01 source:0x0000
>>> [ 81.014536] pcieport 0004:00:00.0: DPC: unmasked uncorrectable error
>>> detected
>>> [ 81.029598] pcieport 0004:00:00.0: PCIe Bus Error:
>>> severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Receiver ID)
>>> [ 81.040830] pcieport 0004:00:00.0: device [0823:0110] error status/
>>> mask=00008000/04d40000
>>> [ 81.049870] pcieport 0004:00:00.0: [ 0] ERCR (First)
>>> [ 81.053520] pcieport 0004:00:00.0: AER: TLP Header: 60008010 010000ff
>>> 00001000 9c4c0000
>>> [ 81.065793] mlx5_core 0004:01:00.0: mlx5_pci_err_detected Device
>>> state = 1 health sensors: 1 pci_status: 1. Enter, pci channel state = 2
>>> [ 81.076183] mlx5_core 0004:01:00.0: mlx5_error_sw_reset:231:(pid
>>> 1618): start
>>> [ 81.083307] mlx5_core 0004:01:00.0: mlx5_error_sw_reset:252:(pid
>>> 1618): PCI channel offline, stop waiting for NIC IFC
>>> [ 81.077428] mlx5_core 0004:01:00.0: E-Switch: Disable: mode(LEGACY),
>>> nvfs(0), neovfs(0), active vports(0)
>>> [ 81.486693] mlx5_core 0004:01:00.0: mlx5_wait_for_pages:786:(pid
>>> 1618): Skipping wait for vf pages stage
>>> [ 81.496965] mlx5_core 0004:01:00.0: mlx5_wait_for_pages:786:(pid
>>> 1618): Skipping wait for vf pages stage
>>> [ 82.395040] mlx5_core 0004:01:00.1: print_health:819:(pid 0): Fatal
>>> error detected
>>> [ 82.395493] mlx5_core 0004:01:00.1: print_health_info:423:(pid 0):
>>> PCI slot 1 is unavailable
>>> [ 83.431094] mlx5_core 0004:01:00.0: mlx5_pci_err_detected Device
>>> state = 2 pci_status: 0. Exit, result = 3, need reset
>>> [ 83.442100] mlx5_core 0004:01:00.1: mlx5_pci_err_detected Device
>>> state = 2 health sensors: 1 pci_status: 1. Enter, pci channel state = 2
>>> [ 83.441801] mlx5_core 0004:01:00.0: mlx5_crdump_collect:50:(pid
>>> 2239): crdump: failed to lock gw status -13
>>> [ 83.454050] mlx5_core 0004:01:00.1: mlx5_error_sw_reset:231:(pid
>>> 1618): start
>>> [ 83.454050] mlx5_core 0004:01:00.1: mlx5_error_sw_reset:252:(pid
>>> 1618): PCI channel offline, stop waiting for NIC IFC
>>> [ 83.849429] mlx5_core 0004:01:00.1: E-Switch: Disable: mode(LEGACY),
>>> nvfs(0), neovfs(0), active vports(0)
>>> [ 83.858892] mlx5_core 0004:01:00.1: mlx5_wait_for_pages:786:(pid
>>> 1618): Skipping wait for vf pages stage
>>> [ 83.869464] mlx5_core 0004:01:00.1: mlx5_wait_for_pages:786:(pid
>>> 1618): Skipping wait for vf pages stage
>>> [ 85.201433] pcieport 0004:00:00.0: pciehp: Slot(41): Link Down
>>> [ 85.815016] mlx5_core 0004:01:00.1: mlx5_health_try_recover:335:(pid
>>> 2239): handling bad device here
>>> [ 85.824164] mlx5_core 0004:01:00.1: mlx5_error_sw_reset:231:(pid
>>> 2239): start
>>> [ 85.831283] mlx5_core 0004:01:00.1: mlx5_error_sw_reset:252:(pid
>>> 2239): PCI channel offline, stop waiting for NIC IFC
>>> [ 85.841899] mlx5_core 0004:01:00.1: mlx5_unload_one_dev_locked:1612:
>>> (pid 2239): mlx5_unload_one_dev_locked: interface is down, NOP
>>> [ 85.853799] mlx5_core 0004:01:00.1: mlx5_health_wait_pci_up:325:(pid
>>> 2239): PCI channel offline, stop waiting for PCI
>>> [ 85.863494] mlx5_core 0004:01:00.1: mlx5_health_try_recover:338:(pid
>>> 2239): health recovery flow aborted, PCI reads still not working
>>> [ 85.873231] mlx5_core 0004:01:00.1: mlx5_pci_err_detected Device
>>> state = 2 pci_status: 0. Exit, result = 3, need reset
>>> [ 85.879899] mlx5_core 0004:01:00.1: E-Switch: Unload vfs:
>>> mode(LEGACY), nvfs(0), neovfs(0), active vports(0)
>>> [ 85.921428] mlx5_core 0004:01:00.1: E-Switch: Disable: mode(LEGACY),
>>> nvfs(0), neovfs(0), active vports(0)
>>> [ 85.930491] mlx5_core 0004:01:00.1: mlx5_wait_for_pages:786:(pid
>>> 1617): Skipping wait for vf pages stage
>>> [ 85.940849] mlx5_core 0004:01:00.1: mlx5_wait_for_pages:786:(pid
>>> 1617): Skipping wait for vf pages stage
>>> [ 85.949971] mlx5_core 0004:01:00.1: mlx5_uninit_one:1528:(pid 1617):
>>> mlx5_uninit_one: interface is down, NOP
>>> [ 85.959944] mlx5_core 0004:01:00.1: E-Switch: cleanup
>>> [ 86.035541] mlx5_core 0004:01:00.0: E-Switch: Unload vfs:
>>> mode(LEGACY), nvfs(0), neovfs(0), active vports(0)
>>> [ 86.077568] mlx5_core 0004:01:00.0: E-Switch: Disable: mode(LEGACY),
>>> nvfs(0), neovfs(0), active vports(0)
>>> [ 86.071727] mlx5_core 0004:01:00.0: mlx5_wait_for_pages:786:(pid
>>> 1617): Skipping wait for vf pages stage
>>> [ 86.096577] mlx5_core 0004:01:00.0: mlx5_wait_for_pages:786:(pid
>>> 1617): Skipping wait for vf pages stage
>>> [ 86.106909] mlx5_core 0004:01:00.0: mlx5_uninit_one:1528:(pid 1617):
>>> mlx5_uninit_one: interface is down, NOP
>>> [ 86.115940] pcieport 0004:00:00.0: AER: subordinate device reset failed
>>> [ 86.122557] pcieport 0004:00:00.0: AER: device recovery failed
>>> [ 86.128571] mlx5_core 0004:01:00.0: E-Switch: cleanup
>>>
>>> I added some prints and found that:
>>>    - ConnectX-5 requires >5s for full recovery
>>>    - ConnectX-7 requires >6s for full recovery
>>>
>>> Setting timeout to 7s covers both devices with safety margin.
>>
>>
>> Instead of updating the recovery time, can you check why your device
>> recovery takes
>> such a long time and how to fix it from the device end?
>>
> Hi, Sathyanarayanan.
> 
> Thanks for the valuable feedback and suggestions.
> 
> I fully agree that ideally the root cause should be addressed on the
> device side to reduce the DPC recovery latency, and that waiting longer
> in the kernel is not a perfect solution.
> 
> However, the current 4 seconds timeout in pci_dpc_recovered() is indeed
> an empirical value rather than a hard requirement from the PCIe
> specification. In real-world scenarios, like with Mellanox ConnectX-5/7
> adapters, we've observed that full DPC recovery can take more than 5-6
> seconds, which leads to premature hotplug processing and device removal.
> 
> To improve robustness and maintain flexibility, I’m considering
> introducing a module parameter to allow tuning the DPC recovery timeout
> dynamically. Would you like me to prepare and submit such a patch for
> review?
> 
What if another device just needs 7.1 seconds to recover ? revise the
timeout again ?  no spec says 4 seconds is mandated. have a kernel 
parameter to override it's default value is one choice to workaround.

Ask FW guys to fix ? what justification we have ?

Thanks,
Ethan
> 
> Best regards,
> Hongbo Yao
> 
> 
>>
>>> Signed-off-by: Hongbo Yao <andy.xu@hj-micro.com>
>>> ---
>>>    drivers/pci/pcie/dpc.c | 4 ++--
>>>    1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/drivers/pci/pcie/dpc.c b/drivers/pci/pcie/dpc.c
>>> index fc18349614d7..35a37fd86dcd 100644
>>> --- a/drivers/pci/pcie/dpc.c
>>> +++ b/drivers/pci/pcie/dpc.c
>>> @@ -118,10 +118,10 @@ bool pci_dpc_recovered(struct pci_dev *pdev)
>>>        /*
>>>         * Need a timeout in case DPC never completes due to failure of
>>>         * dpc_wait_rp_inactive().  The spec doesn't mandate a time limit,
>>> -     * but reports indicate that DPC completes within 4 seconds.
>>> +     * but reports indicate that DPC completes within 7 seconds.
>>>         */
>>>        wait_event_timeout(dpc_completed_waitqueue, dpc_completed(pdev),
>>> -               msecs_to_jiffies(4000));
>>> +               msecs_to_jiffies(7000));
>>>          return test_and_clear_bit(PCI_DPC_RECOVERED, &pdev->priv_flags);
>>>    }
>>
> 
>