[PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down

[PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down

[Jian Wen]

mlx5r_umr_post_send_wait() will get stuck when the pcie link is down
as the call trace[1].

When pciehp detects the link is down it calls
pci_dev_set_disconnected() before mlx5_ib_dereg_mr(). Thus we can use
mlx5_cmd_is_down() to detect PCIe Surprise Link Down in
mlx5r_umr_post_send().

[1]
pcieport 0000:b9:01.0: pciehp: Slot(2-4): Link Down
pcieport 0000:b9:01.0: pciehp: Slot(2-4): Card not present
mlx5_core 0000:bb:00.0: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
mlx5_core 0000:bb:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
mlx5_core 0000:bb:00.0: poll_health:1083:(pid 0): Fatal error 1 detected
mlx5_core 0000:bb:00.0: print_health_info:491:(pid 0): PCI slot is unavailable
INFO: task irq/105-pciehp:1246 blocked for more than 122 seconds.
      Tainted: G           OE      6.8.0-54-generic #56-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:irq/105-pciehp  state:D stack:0     pid:1246  tgid:1246  ppid:2      flags:0x00004000
Call Trace:
 <TASK>
 __schedule+0x27c/0x6b0
 schedule+0x33/0x110
 schedule_timeout+0x157/0x170
 wait_for_completion+0x88/0x150
 mlx5r_umr_post_send_wait+0x15f/0x2d0 [mlx5_ib]
 ? __pfx_mlx5r_umr_done+0x10/0x10 [mlx5_ib]
 mlx5r_umr_revoke_mr+0x98/0xc0 [mlx5_ib]
 __mlx5_ib_dereg_mr+0x24a/0x740 [mlx5_ib]
 ? wait_for_completion+0x114/0x150
 mlx5_ib_dereg_mr+0x21/0xc0 [mlx5_ib]
 ? rdma_restrack_del+0x59/0x160 [ib_core]
 ib_dereg_mr_user+0x41/0xc0 [ib_core]
 uverbs_free_mr+0x15/0x30 [ib_uverbs]
 destroy_hw_idr_uobject+0x21/0x60 [ib_uverbs]
 uverbs_destroy_uobject+0x38/0x1d0 [ib_uverbs]
 __uverbs_cleanup_ufile+0xcf/0x150 [ib_uverbs]
 uverbs_destroy_ufile_hw+0x3f/0x100 [ib_uverbs]
 ib_uverbs_remove_one+0x147/0x1c0 [ib_uverbs]
 remove_client_context+0x95/0x100 [ib_core]
 disable_device+0x8f/0x180 [ib_core]
 __ib_unregister_device+0x108/0x170 [ib_core]
 ? __pfx_mlx5_ib_stage_ib_reg_cleanup+0x10/0x10 [mlx5_ib]
 ib_unreister_device+0x26/0x40 [ib_core]
 mlx5_ib_stage_ib_reg_cleanup+0xe/0x20 [mlx5_ib]
 mlx5r_remove+0x52/0xb0 [mlx5_ib]
 auxiliary_bus_remove+0x1c/0x40
 device_remove+0x40/0x80
 device_release_driver_internal+0x20b/0x270
 device_release_driver+0x12/0x20
 bus_remove_device+0xcb/0x140
 device_del+0x161/0x3e0
 ? is_ib_enabled+0x52/0x90 [mlx5_core]
 mlx5_rescan_drivers_locked+0xfe/0x350 [mlx5_core]
 mlx5_unregister_device+0x38/0x60 [mlx5_core]
 mlx5_uninit_one+0x39/0x160 [mlx5_core]
 remove_one+0x55/0x100 [mlx5_core]
 pci_device_remove+0x3e/0xb0
 device_remove+0x40/0x80
 device_release_driver_internal+0x20b/0x270
 device_release_driver+0x12/0x20
 pci_stop_bus_device+0x7a/0xb0
 pci_stop_and_remove_bus_device+0x12/0x30
 pciehp_unconfigure_device+0x98/0x170
 pciehp_disable_slot+0x69/0x130
 pciehp_handle_presence_or_link_change+0x71/0x220
 pciehp_ist+0x22e/0x260
 ? __pfx_irq_thread_fn+0x10/0x10
 irq_thread_fn+0x21/0x70
 irq_thread+0xf8/0x1c0
 ? __pfx_irq_thread_dtor+0x10/0x10
 ? __pfx_irq_thread+0x10/0x10
 kthread+0xef/0x120
 ? __pfx_kthread+0x10/0x10
 ret_from_fork+0x44/0x70
 ? __pfx_kthread+0x10/0x10
 ret_from_fork_asm+0x1b/0x30
 </TASK>

Fixes: 6f0689fdf19e ("RDMA/mlx5: Introduce mlx5_umr_post_send_wait()")
Signed-off-by: Jian Wen <wenjian1@xiaomi.com>
---
 drivers/infiniband/hw/mlx5/umr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/infiniband/hw/mlx5/umr.c b/drivers/infiniband/hw/mlx5/umr.c
index 4e562e0dd9e1..f3700ef6ea07 100644
--- a/drivers/infiniband/hw/mlx5/umr.c
+++ b/drivers/infiniband/hw/mlx5/umr.c
@@ -254,7 +254,7 @@ static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey, struct ib_cqe *cqe,
 	unsigned int idx;
 	int size, err;
 
-	if (unlikely(mdev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR))
+	if (unlikely(mlx5_cmd_is_down(mdev)))
 		return -EIO;
 
 	spin_lock_irqsave(&qp->sq.lock, flags);
-- 
2.43.0

Re: [PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down

[Leon Romanovsky]

On Thu, Oct 09, 2025 at 10:23:20PM +0800, Jian Wen wrote:
> mlx5r_umr_post_send_wait() will get stuck when the pcie link is down
> as the call trace[1].
> 
> When pciehp detects the link is down it calls
> pci_dev_set_disconnected() before mlx5_ib_dereg_mr(). Thus we can use
> mlx5_cmd_is_down() to detect PCIe Surprise Link Down in
> mlx5r_umr_post_send().
> 
> [1]
> pcieport 0000:b9:01.0: pciehp: Slot(2-4): Link Down
> pcieport 0000:b9:01.0: pciehp: Slot(2-4): Card not present
> mlx5_core 0000:bb:00.0: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
> mlx5_core 0000:bb:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
> mlx5_core 0000:bb:00.0: poll_health:1083:(pid 0): Fatal error 1 detected
> mlx5_core 0000:bb:00.0: print_health_info:491:(pid 0): PCI slot is unavailable
> INFO: task irq/105-pciehp:1246 blocked for more than 122 seconds.
>       Tainted: G           OE      6.8.0-54-generic #56-Ubuntu
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> task:irq/105-pciehp  state:D stack:0     pid:1246  tgid:1246  ppid:2      flags:0x00004000
> Call Trace:
>  <TASK>
>  __schedule+0x27c/0x6b0
>  schedule+0x33/0x110
>  schedule_timeout+0x157/0x170
>  wait_for_completion+0x88/0x150
>  mlx5r_umr_post_send_wait+0x15f/0x2d0 [mlx5_ib]
>  ? __pfx_mlx5r_umr_done+0x10/0x10 [mlx5_ib]
>  mlx5r_umr_revoke_mr+0x98/0xc0 [mlx5_ib]
>  __mlx5_ib_dereg_mr+0x24a/0x740 [mlx5_ib]
>  ? wait_for_completion+0x114/0x150
>  mlx5_ib_dereg_mr+0x21/0xc0 [mlx5_ib]
>  ? rdma_restrack_del+0x59/0x160 [ib_core]
>  ib_dereg_mr_user+0x41/0xc0 [ib_core]
>  uverbs_free_mr+0x15/0x30 [ib_uverbs]
>  destroy_hw_idr_uobject+0x21/0x60 [ib_uverbs]
>  uverbs_destroy_uobject+0x38/0x1d0 [ib_uverbs]
>  __uverbs_cleanup_ufile+0xcf/0x150 [ib_uverbs]
>  uverbs_destroy_ufile_hw+0x3f/0x100 [ib_uverbs]
>  ib_uverbs_remove_one+0x147/0x1c0 [ib_uverbs]
>  remove_client_context+0x95/0x100 [ib_core]
>  disable_device+0x8f/0x180 [ib_core]
>  __ib_unregister_device+0x108/0x170 [ib_core]
>  ? __pfx_mlx5_ib_stage_ib_reg_cleanup+0x10/0x10 [mlx5_ib]
>  ib_unreister_device+0x26/0x40 [ib_core]
>  mlx5_ib_stage_ib_reg_cleanup+0xe/0x20 [mlx5_ib]
>  mlx5r_remove+0x52/0xb0 [mlx5_ib]
>  auxiliary_bus_remove+0x1c/0x40
>  device_remove+0x40/0x80
>  device_release_driver_internal+0x20b/0x270
>  device_release_driver+0x12/0x20
>  bus_remove_device+0xcb/0x140
>  device_del+0x161/0x3e0
>  ? is_ib_enabled+0x52/0x90 [mlx5_core]
>  mlx5_rescan_drivers_locked+0xfe/0x350 [mlx5_core]
>  mlx5_unregister_device+0x38/0x60 [mlx5_core]
>  mlx5_uninit_one+0x39/0x160 [mlx5_core]
>  remove_one+0x55/0x100 [mlx5_core]
>  pci_device_remove+0x3e/0xb0
>  device_remove+0x40/0x80
>  device_release_driver_internal+0x20b/0x270
>  device_release_driver+0x12/0x20
>  pci_stop_bus_device+0x7a/0xb0
>  pci_stop_and_remove_bus_device+0x12/0x30
>  pciehp_unconfigure_device+0x98/0x170
>  pciehp_disable_slot+0x69/0x130
>  pciehp_handle_presence_or_link_change+0x71/0x220
>  pciehp_ist+0x22e/0x260
>  ? __pfx_irq_thread_fn+0x10/0x10
>  irq_thread_fn+0x21/0x70
>  irq_thread+0xf8/0x1c0
>  ? __pfx_irq_thread_dtor+0x10/0x10
>  ? __pfx_irq_thread+0x10/0x10
>  kthread+0xef/0x120
>  ? __pfx_kthread+0x10/0x10
>  ret_from_fork+0x44/0x70
>  ? __pfx_kthread+0x10/0x10
>  ret_from_fork_asm+0x1b/0x30
>  </TASK>
> 
> Fixes: 6f0689fdf19e ("RDMA/mlx5: Introduce mlx5_umr_post_send_wait()")
> Signed-off-by: Jian Wen <wenjian1@xiaomi.com>
> ---
>  drivers/infiniband/hw/mlx5/umr.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/infiniband/hw/mlx5/umr.c b/drivers/infiniband/hw/mlx5/umr.c
> index 4e562e0dd9e1..f3700ef6ea07 100644
> --- a/drivers/infiniband/hw/mlx5/umr.c
> +++ b/drivers/infiniband/hw/mlx5/umr.c
> @@ -254,7 +254,7 @@ static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey, struct ib_cqe *cqe,
>  	unsigned int idx;
>  	int size, err;
>  
> -	if (unlikely(mdev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR))
> +	if (unlikely(mlx5_cmd_is_down(mdev)))

mlx5_cmd_is_down() is defined in mlx5_core.ko module and doesn't have EXPORT_SYMBOL.
Did you success to compile this patch?

Thanks

>  		return -EIO;
>  
>  	spin_lock_irqsave(&qp->sq.lock, flags);
> -- 
> 2.43.0
> 
> 

Re: [PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down

[Jason Gunthorpe]

On Thu, Oct 09, 2025 at 10:23:20PM +0800, Jian Wen wrote:
> --- a/drivers/infiniband/hw/mlx5/umr.c
> +++ b/drivers/infiniband/hw/mlx5/umr.c
> @@ -254,7 +254,7 @@ static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey, struct ib_cqe *cqe,
>  	unsigned int idx;
>  	int size, err;
>  
> -	if (unlikely(mdev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR))
> +	if (unlikely(mlx5_cmd_is_down(mdev)))
>  		return -EIO;

I feel like this is just changing the race around..

The removal flow for the device is different if the HW is working than
if it isn't.

If it isn't working then the removal should disable and cancel all the
UMRs, using the umrc->lock. Otherwise there will be dead threads
floating around. It should also be setting
MLX5_DEVICE_STATE_INTERNAL_ERROR way at the start of removal.

So IDK, maybe check mlx5_cmd_is_down() and trigger the flow to
activate INTERNAL_ERROR befor doing anything else?

Jason

Re: [PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down

[kernel test robot]

Hi Jian,

kernel test robot noticed the following build errors:

[auto build test ERROR on rdma/for-next]
[also build test ERROR on linus/master v6.17 next-20251010]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Jian-Wen/RDMA-mlx5-Use-mlx5_cmd_is_down-to-detect-PCIe-Surprise-Link-Down/20251009-224616
base:   https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git for-next
patch link:    https://lore.kernel.org/r/20251009142326.3794769-1-wenjian1%40xiaomi.com
patch subject: [PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down
config: s390-defconfig (https://download.01.org/0day-ci/archive/20251012/202510120715.P7oFOBOQ-lkp@intel.com/config)
compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 39f292ffa13d7ca0d1edff27ac8fd55024bb4d19)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251012/202510120715.P7oFOBOQ-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202510120715.P7oFOBOQ-lkp@intel.com/

All errors (new ones prefixed by >>, old ones prefixed by <<):

>> ERROR: modpost: "mlx5_cmd_is_down" [drivers/infiniband/hw/mlx5/mlx5_ib.ko] undefined!

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down

[Maher Sanalla]

On 09/10/2025 21:02, Jason Gunthorpe wrote:
> External email: Use caution opening links or attachments
> 
> 
> On Thu, Oct 09, 2025 at 10:23:20PM +0800, Jian Wen wrote:
>> --- a/drivers/infiniband/hw/mlx5/umr.c
>> +++ b/drivers/infiniband/hw/mlx5/umr.c
>> @@ -254,7 +254,7 @@ static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey, struct ib_cqe *cqe,
>>        unsigned int idx;
>>        int size, err;
>>
>> -     if (unlikely(mdev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR))
>> +     if (unlikely(mlx5_cmd_is_down(mdev)))
>>                return -EIO;
> 
> I feel like this is just changing the race around..
> 
> The removal flow for the device is different if the HW is working than
> if it isn't.
> 
> If it isn't working then the removal should disable and cancel all the
> UMRs, using the umrc->lock. Otherwise there will be dead threads
> floating around. It should also be setting
> MLX5_DEVICE_STATE_INTERNAL_ERROR way at the start of removal.
> 
> So IDK, maybe check mlx5_cmd_is_down() and trigger the flow to
> activate INTERNAL_ERROR befor doing anything else?
> 
> Jason
> 

I agree with Jason concern. While this may serve as a workaround for 
certain cases, the race condition still exists. We’re aware of this 
issue and are working on a proper fix. The intended solution is that 
when the device enters an error state, the UMR state should transition 
directly to error as well, preventing any new UMRs from being posted.

Maher

Re: [PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down

[Maher Sanalla]

Hey Jian,

Yes. This is the fix we went with ultimately:
[1/1] RDMA/mlx5: Fix UMR hang in LAG error state unload
       https://git.kernel.org/rdma/rdma/c/ebc2164a4cd431

On 22/01/2026 11:36, Jian Wen wrote:
> *External email: Use caution opening links or attachments*
> 
> 
> Hi Maher,
> 
> Has the fix released yet?
> 
> Thanks.
> 
> 
> Maher Sanalla <msanalla@nvidia.com <mailto:msanalla@nvidia.com>>于2025年 
> 10月16日 周四21:32写道：
> 
> 
> 
>     On 09/10/2025 21:02, Jason Gunthorpe wrote:
>      > External email: Use caution opening links or attachments
>      >
>      >
>      > On Thu, Oct 09, 2025 at 10:23:20PM +0800, Jian Wen wrote:
>      >> --- a/drivers/infiniband/hw/mlx5/umr.c
>      >> +++ b/drivers/infiniband/hw/mlx5/umr.c
>      >> @@ -254,7 +254,7 @@ static int mlx5r_umr_post_send(struct ib_qp
>     *ibqp, u32 mkey, struct ib_cqe *cqe,
>      >>        unsigned int idx;
>      >>        int size, err;
>      >>
>      >> -     if (unlikely(mdev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR))
>      >> +     if (unlikely(mlx5_cmd_is_down(mdev)))
>      >>                return -EIO;
>      >
>      > I feel like this is just changing the race around..
>      >
>      > The removal flow for the device is different if the HW is working
>     than
>      > if it isn't.
>      >
>      > If it isn't working then the removal should disable and cancel
>     all the
>      > UMRs, using the umrc->lock. Otherwise there will be dead threads
>      > floating around. It should also be setting
>      > MLX5_DEVICE_STATE_INTERNAL_ERROR way at the start of removal.
>      >
>      > So IDK, maybe check mlx5_cmd_is_down() and trigger the flow to
>      > activate INTERNAL_ERROR befor doing anything else?
>      >
>      > Jason
>      >
> 
>     I agree with Jason concern. While this may serve as a workaround for
>     certain cases, the race condition still exists. We’re aware of this
>     issue and are working on a proper fix. The intended solution is that
>     when the device enters an error state, the UMR state should transition
>     directly to error as well, preventing any new UMRs from being posted.
> 
>     Maher
>