Nvme-rdma: Long IO HANG (~4.8h) during link failure with dm-multipath (Kernel 6.6)

Nvme-rdma: Long IO HANG (~4.8h) during link failure with dm-multipath (Kernel 6.6)

[shimiaofeng]

[-- Attachment #1.1: Type: text/plain, Size: 3987 bytes --]

Hi all,
We encountered a severe I/O hang issue during NVMe-oF (RoCEv2) link failure when native multipath is disabled and dm-multipath is used instead.
Based on our test results, this issue is highly likely located within the reconnect/backoff mechanism of the vendor's out-of-tree driver (NVIDIA DOCA-Host 3.3.0). The attachment contains the logs captured when the fault occurs.
We look forward to the vendor analyzing this driver-side issue, but we would also like to seek advice from community experts regarding potential optimization suggestions for the block layer and NVMe-oF (NOF) collaboration mechanisms to guard against such behavior.

Environment

  *   Kernel: 6.6.0 (openEuler 24.03 LTS SP4 baseline)
  *   Hardware: Intel Xeon Gold 5220R / Mellanox ConnectX-5 (MT27800, FW: 26.01-1.0.0)
  *   Storage: Huawei OceanStor Dorado (NVMe over RoCEv2)
  *   Configuration: nvme_core.multipath=N, dm-multipath enabled.
  *   Fabric Parameters: reconnect_delay = 10, ctrl_loss_tmo = 600 (or ctrl_loss_tmo = 10 for testing)

Problem Description & Observations
When we manually inject a link fault on one of the paths and restart multipathd, the multipathd process gets stuck in the D state (the path detection I/O does not return). Application I/O hangs for a very long time due to a requeue ping-pong loop.

  1.  The Requeue Ping-Pong: After the link goes down, the NVMe controller transitions to the NVME_CTRL_CONNECTING state. During this time, nvme_fail_nonready_command() returns BLK_STS_RESOURCE. Upon receiving this, the block layer (blk-mq) immediately requeues the request. This causes the I/O to endlessly "ping-pong" between the block layer and nvme-core.
  2.  Abnormal Retry Interval: Based on ctrl_loss_tmo / reconnect_delay = 600 / 10, the subsystem is expected to retry 60 times. However, the actual measured interval between two consecutive retries is stretched to 288s - 290s, completely ignoring the configured reconnect_delay = 10. The total hang time reaches 17,340 seconds (~4.8 hours).
  3.  Single Retry Test: To isolate the issue, we set ctrl_loss_tmo = 10 (which triggers only 1 retry before tearing down the controller). Even in this case, the I/O still HANGS for exactly ~290 seconds. This indicates that the problem is not cumulative retry multiplication, but rather that the very first reconnect attempt or the single reconnect worker itself is being blocked/delayed internally for ~290 seconds by the underlying driver stack.

In-box Driver Contrast
If we switch back to the kernel in-box (mainline upstream) mlx5 driver, this issue DOES NOT occur. The NVMe controller status updates rapidly upon link failure, and failover finishes within seconds. This further confirms that the 290-second blocking behavior is specific to the DOCA driver stack.

Technical Consultation
The most appropriate solution is for the vendor to fix this issue within their driver. However, if the vendor driver cannot technically resolve it, are there any potential optimization mechanisms within the block layer or the multipath subsystem to mitigate this?
For example:

  1.  Block Layer: During the NVME_CTRL_CONNECTING state, if requests continuously receive BLK_STS_RESOURCE, should the block layer introduce an exponential backoff mechanism or a maximum retry threshold? Blindly requeuing these requests seems to create an infinite loop that leaves the system vulnerable to worker starvation or long hangs if the underlying driver blocks.
  2.  Multipath Subsystem: The multipath subsystem's path detection mechanism for NVMe devices could be optimized. I believe that path-checking I/Os (such as those sent by multipathd to verify link sanity) should not be allowed to retry indefinitely under these conditions, as it completely stalls the failover process.

Any insights or architectural suggestions on how the block layer can better handle or guard against such non-responsive driver behavior would be greatly appreciated.

Thanks.

[-- Attachment #1.2: Type: text/html, Size: 22124 bytes --]

[-- Attachment #2: log.txt --]
[-- Type: text/plain, Size: 6970 bytes --]

Sun Jun  7 18:54:35 2026 nvme nvme2: unreachable (7): status -110 id 00000000fc110286
Sun Jun  7 18:54:35 2026 nvme nvme2: CM error event 7
Sun Jun  7 18:54:35 2026 nvme nvme2: rdma connection establishment failed (-104)
Sun Jun  7 18:54:35 2026 nvme nvme2: Failed reconnect attempt 1/60
Sun Jun  7 18:54:35 2026 nvme nvme2: Reconnecting in 10 seconds...
Sun Jun  7 18:54:45 2026 nvme nvme2: address resolved (0): status 0 id 000000006c41792a
Sun Jun  7 18:54:45 2026 nvme nvme2: route resolved (2): status 0 id 000000006c41792a
Sun Jun  7 18:55:20 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 18:55:55 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 18:56:12 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 18:56:30 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 18:56:47 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 18:57:04 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 18:57:22 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 18:57:39 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 18:57:57 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 18:58:14 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 18:58:31 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 18:58:49 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 18:59:06 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 18:59:13 2026 fsnotify_insert_event: 2897 callbacks suppressed
Sun Jun  7 18:59:13 2026 pid=2379609 cmd:python3 q_len=16385 max_events=16384
Sun Jun  7 18:59:24 2026 Warning: at fsnotify_insert_event, overflow event or group->max_events reached
Sun Jun  7 18:59:24 2026 nvme nvme2: unreachable (7): status -110 id 000000006c41792a
Sun Jun  7 18:59:24 2026 nvme nvme2: CM error event 7
Sun Jun  7 18:59:24 2026 nvme nvme2: rdma connection establishment failed (-104)
Sun Jun  7 18:59:24 2026 nvme nvme2: Failed reconnect attempt 2/60
Sun Jun  7 18:59:24 2026 nvme nvme2: Reconnecting in 10 seconds...
Sun Jun  7 18:59:34 2026 nvme nvme2: address resolved (0): status 0 id 0000000075fad7b1
Sun Jun  7 18:59:34 2026 nvme nvme2: route resolved (2): status 0 id 0000000075fad7b1
Sun Jun  7 18:59:34 2026 infiniband mlx5_0: calc cq_size:747:(pid 818943): wqe size 256
Sun Jun  7 18:59:34 2026 infiniband mlx5_0: create qp:3328:(pid 818943): QP type 2, ib qpn 0xC8, mlx qpn 0xC8, rcqn 0x4cc, scqn 0x4cc, ece 0x0
Sun Jun  7 18:59:51 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 18:59:51 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:00:08 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:00:26 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:00:44 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:01:01 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:01:03 2026 (kworker/1:2,2234172,1):o2net_scan_del_work:4781 o2net: delete the seq num which is over 2 hours
Sun Jun  7 19:01:18 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:01:36 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:01:53 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:02:11 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:02:28 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:02:45 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:03:03 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:03:20 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:03:38 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:03:55 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:04:12 2026 nvme nvme2: unreachable (7): status -110 id 0000000075fad7b1
Sun Jun  7 19:04:12 2026 nvme nvme2: CM error event 7
Sun Jun  7 19:04:12 2026 nvme nvme2: rdma connection establishment failed (-104)
Sun Jun  7 19:04:12 2026 nvme nvme2: Failed reconnect attempt 3/60
Sun Jun  7 19:04:12 2026 nvme nvme2: Reconnecting in 10 seconds...
Sun Jun  7 19:04:13 2026 fsnotify_insert_event: 2897 callbacks suppressed
Sun Jun  7 19:04:13 2026 pid=2460821 cmd:python3 q_len=16385 max_events=16384
Sun Jun  7 19:04:23 2026 Warning: at fsnotify_insert_event, overflow event or group->max_events reached
Sun Jun  7 19:04:23 2026 nvme nvme2: address resolved (0): status 0 id 0000000075fad7b1
Sun Jun  7 19:04:23 2026 nvme nvme2: route resolved (2): status 0 id 0000000075fad7b1
Sun Jun  7 19:04:23 2026 infiniband mlx5_0: calc cq_size:747:(pid 818943): wqe size 256
Sun Jun  7 19:04:23 2026 infiniband mlx5_0: create qp:3328:(pid 818943): QP type 2, ib qpn 0xC8, mlx qpn 0xC8, rcqn 0x4cc, scqn 0x4cc, ece 0x0
Sun Jun  7 19:04:40 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:04:57 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:05:15 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:05:15 2026 nvme nvme2: established (9): status 0 id 0000000075fad7b1
Sun Jun  7 19:05:15 2026 infiniband mlx5_0: poll_soft_wc:600:(pid 393): polled software generated completion on CQ 0x401
Sun Jun  7 19:05:15 2026 nvme nvme2: queue size 128 x ctrl sqsize 64, clamping down
Sun Jun  7 19:05:15 2026 nvme nvme2: creating 8 I/O queues

Nvme-rdma: Long IO HANG (~4.8h) during link failure with dm-multipath (Kernel 6.6)

[shimiaofeng]

Hi all,
We encountered a severe I/O hang issue during NVMe-oF (RoCEv2) link failure when native multipath is disabled and dm-multipath is used instead.
Based on our test results, this issue is highly likely located within the reconnect/backoff mechanism of the vendor's out-of-tree driver (NVIDIA DOCA-Host 3.3.0). The attachment contains the logs captured when the fault occurs.
We look forward to the vendor analyzing this driver-side issue, but we would also like to seek advice from community experts regarding potential optimization suggestions for the block layer and NVMe-oF (NOF) collaboration mechanisms to guard against such behavior.

Environment
*	Kernel: 6.6.0 (openEuler 24.03 LTS SP4 baseline)
*	Hardware: Intel Xeon Gold 5220R / Mellanox ConnectX-5 (MT27800, FW: 26.01-1.0.0)
*	Storage: Huawei OceanStor Dorado (NVMe over RoCEv2)
*	Configuration: nvme_core.multipath=N, dm-multipath enabled.
*	Fabric Parameters: reconnect_delay = 10, ctrl_loss_tmo = 600 (or ctrl_loss_tmo = 10 for testing)

Problem Description & Observations
When we manually inject a link fault on one of the paths and restart multipathd, the multipathd process gets stuck in the D state (the path detection I/O does not return). Application I/O hangs for a very long time due to a requeue ping-pong loop.
1.	The Requeue Ping-Pong: After the link goes down, the NVMe controller transitions to the NVME_CTRL_CONNECTING state. During this time, nvme_fail_nonready_command() returns BLK_STS_RESOURCE. Upon receiving this, the block layer (blk-mq) immediately requeues the request. This causes the I/O to endlessly "ping-pong" between the block layer and nvme-core.
2.	Abnormal Retry Interval: Based on ctrl_loss_tmo / reconnect_delay = 600 / 10, the subsystem is expected to retry 60 times. However, the actual measured interval between two consecutive retries is stretched to 288s - 290s, completely ignoring the configured reconnect_delay = 10. The total hang time reaches 17,340 seconds (~4.8 hours).
3.	Single Retry Test: To isolate the issue, we set ctrl_loss_tmo = 10 (which triggers only 1 retry before tearing down the controller). Even in this case, the I/O still HANGS for exactly ~290 seconds. This indicates that the problem is not cumulative retry multiplication, but rather that the very first reconnect attempt or the single reconnect worker itself is being blocked/delayed internally for ~290 seconds by the underlying driver stack.

In-box Driver Contrast
If we switch back to the kernel in-box (mainline upstream) mlx5 driver, this issue DOES NOT occur. The NVMe controller status updates rapidly upon link failure, and failover finishes within seconds. This further confirms that the 290-second blocking behavior is specific to the DOCA driver stack.

Technical Consultation
The most appropriate solution is for the vendor to fix this issue within their driver. However, if the vendor driver cannot technically resolve it, are there any potential optimization mechanisms within the block layer or the multipath subsystem to mitigate this?
For example:
1.	Block Layer: During the NVME_CTRL_CONNECTING state, if requests continuously receive BLK_STS_RESOURCE, should the block layer introduce an exponential backoff mechanism or a maximum retry threshold? Blindly requeuing these requests seems to create an infinite loop that leaves the system vulnerable to worker starvation or long hangs if the underlying driver blocks.
2.	Multipath Subsystem: The multipath subsystem's path detection mechanism for NVMe devices could be optimized. I believe that path-checking I/Os (such as those sent by multipathd to verify link sanity) should not be allowed to retry indefinitely under these conditions, as it completely stalls the failover process.

Any insights or architectural suggestions on how the block layer can better handle or guard against such non-responsive driver behavior would be greatly appreciated.

Thanks.