Nvme-rdma: Long IO HANG (~4.8h) during link failure with dm-multipath (Kernel 6.6)

Nvme-rdma: Long IO HANG (~4.8h) during link failure with dm-multipath (Kernel 6.6)

[shimiaofeng]

Hi all,
We encountered a severe I/O hang issue during NVMe-oF (RoCEv2) link failure when native multipath is disabled and dm-multipath is used instead.
Based on our test results, this issue is highly likely located within the reconnect/backoff mechanism of the vendor's out-of-tree driver (NVIDIA DOCA-Host 3.3.0). The attachment contains the logs captured when the fault occurs.
We look forward to the vendor analyzing this driver-side issue, but we would also like to seek advice from community experts regarding potential optimization suggestions for the block layer and NVMe-oF (NOF) collaboration mechanisms to guard against such behavior.

Environment
*	Kernel: 6.6.0 (openEuler 24.03 LTS SP4 baseline)
*	Hardware: Intel Xeon Gold 5220R / Mellanox ConnectX-5 (MT27800, FW: 26.01-1.0.0)
*	Storage: Huawei OceanStor Dorado (NVMe over RoCEv2)
*	Configuration: nvme_core.multipath=N, dm-multipath enabled.
*	Fabric Parameters: reconnect_delay = 10, ctrl_loss_tmo = 600 (or ctrl_loss_tmo = 10 for testing)

Problem Description & Observations
When we manually inject a link fault on one of the paths and restart multipathd, the multipathd process gets stuck in the D state (the path detection I/O does not return). Application I/O hangs for a very long time due to a requeue ping-pong loop.
1.	The Requeue Ping-Pong: After the link goes down, the NVMe controller transitions to the NVME_CTRL_CONNECTING state. During this time, nvme_fail_nonready_command() returns BLK_STS_RESOURCE. Upon receiving this, the block layer (blk-mq) immediately requeues the request. This causes the I/O to endlessly "ping-pong" between the block layer and nvme-core.
2.	Abnormal Retry Interval: Based on ctrl_loss_tmo / reconnect_delay = 600 / 10, the subsystem is expected to retry 60 times. However, the actual measured interval between two consecutive retries is stretched to 288s - 290s, completely ignoring the configured reconnect_delay = 10. The total hang time reaches 17,340 seconds (~4.8 hours).
3.	Single Retry Test: To isolate the issue, we set ctrl_loss_tmo = 10 (which triggers only 1 retry before tearing down the controller). Even in this case, the I/O still HANGS for exactly ~290 seconds. This indicates that the problem is not cumulative retry multiplication, but rather that the very first reconnect attempt or the single reconnect worker itself is being blocked/delayed internally for ~290 seconds by the underlying driver stack.

In-box Driver Contrast
If we switch back to the kernel in-box (mainline upstream) mlx5 driver, this issue DOES NOT occur. The NVMe controller status updates rapidly upon link failure, and failover finishes within seconds. This further confirms that the 290-second blocking behavior is specific to the DOCA driver stack.

Technical Consultation
The most appropriate solution is for the vendor to fix this issue within their driver. However, if the vendor driver cannot technically resolve it, are there any potential optimization mechanisms within the block layer or the multipath subsystem to mitigate this?
For example:
1.	Block Layer: During the NVME_CTRL_CONNECTING state, if requests continuously receive BLK_STS_RESOURCE, should the block layer introduce an exponential backoff mechanism or a maximum retry threshold? Blindly requeuing these requests seems to create an infinite loop that leaves the system vulnerable to worker starvation or long hangs if the underlying driver blocks.
2.	Multipath Subsystem: The multipath subsystem's path detection mechanism for NVMe devices could be optimized. I believe that path-checking I/Os (such as those sent by multipathd to verify link sanity) should not be allowed to retry indefinitely under these conditions, as it completely stalls the failover process.

Any insights or architectural suggestions on how the block layer can better handle or guard against such non-responsive driver behavior would be greatly appreciated.

Thanks.