* [PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down
@ 2025-10-09 14:23 Jian Wen
2025-10-09 17:49 ` Leon Romanovsky
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Jian Wen @ 2025-10-09 14:23 UTC (permalink / raw)
To: jgg, leonro; +Cc: Jian Wen, linux-rdma, wenjianhn
mlx5r_umr_post_send_wait() will get stuck when the pcie link is down
as the call trace[1].
When pciehp detects the link is down it calls
pci_dev_set_disconnected() before mlx5_ib_dereg_mr(). Thus we can use
mlx5_cmd_is_down() to detect PCIe Surprise Link Down in
mlx5r_umr_post_send().
[1]
pcieport 0000:b9:01.0: pciehp: Slot(2-4): Link Down
pcieport 0000:b9:01.0: pciehp: Slot(2-4): Card not present
mlx5_core 0000:bb:00.0: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
mlx5_core 0000:bb:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
mlx5_core 0000:bb:00.0: poll_health:1083:(pid 0): Fatal error 1 detected
mlx5_core 0000:bb:00.0: print_health_info:491:(pid 0): PCI slot is unavailable
INFO: task irq/105-pciehp:1246 blocked for more than 122 seconds.
Tainted: G OE 6.8.0-54-generic #56-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:irq/105-pciehp state:D stack:0 pid:1246 tgid:1246 ppid:2 flags:0x00004000
Call Trace:
<TASK>
__schedule+0x27c/0x6b0
schedule+0x33/0x110
schedule_timeout+0x157/0x170
wait_for_completion+0x88/0x150
mlx5r_umr_post_send_wait+0x15f/0x2d0 [mlx5_ib]
? __pfx_mlx5r_umr_done+0x10/0x10 [mlx5_ib]
mlx5r_umr_revoke_mr+0x98/0xc0 [mlx5_ib]
__mlx5_ib_dereg_mr+0x24a/0x740 [mlx5_ib]
? wait_for_completion+0x114/0x150
mlx5_ib_dereg_mr+0x21/0xc0 [mlx5_ib]
? rdma_restrack_del+0x59/0x160 [ib_core]
ib_dereg_mr_user+0x41/0xc0 [ib_core]
uverbs_free_mr+0x15/0x30 [ib_uverbs]
destroy_hw_idr_uobject+0x21/0x60 [ib_uverbs]
uverbs_destroy_uobject+0x38/0x1d0 [ib_uverbs]
__uverbs_cleanup_ufile+0xcf/0x150 [ib_uverbs]
uverbs_destroy_ufile_hw+0x3f/0x100 [ib_uverbs]
ib_uverbs_remove_one+0x147/0x1c0 [ib_uverbs]
remove_client_context+0x95/0x100 [ib_core]
disable_device+0x8f/0x180 [ib_core]
__ib_unregister_device+0x108/0x170 [ib_core]
? __pfx_mlx5_ib_stage_ib_reg_cleanup+0x10/0x10 [mlx5_ib]
ib_unreister_device+0x26/0x40 [ib_core]
mlx5_ib_stage_ib_reg_cleanup+0xe/0x20 [mlx5_ib]
mlx5r_remove+0x52/0xb0 [mlx5_ib]
auxiliary_bus_remove+0x1c/0x40
device_remove+0x40/0x80
device_release_driver_internal+0x20b/0x270
device_release_driver+0x12/0x20
bus_remove_device+0xcb/0x140
device_del+0x161/0x3e0
? is_ib_enabled+0x52/0x90 [mlx5_core]
mlx5_rescan_drivers_locked+0xfe/0x350 [mlx5_core]
mlx5_unregister_device+0x38/0x60 [mlx5_core]
mlx5_uninit_one+0x39/0x160 [mlx5_core]
remove_one+0x55/0x100 [mlx5_core]
pci_device_remove+0x3e/0xb0
device_remove+0x40/0x80
device_release_driver_internal+0x20b/0x270
device_release_driver+0x12/0x20
pci_stop_bus_device+0x7a/0xb0
pci_stop_and_remove_bus_device+0x12/0x30
pciehp_unconfigure_device+0x98/0x170
pciehp_disable_slot+0x69/0x130
pciehp_handle_presence_or_link_change+0x71/0x220
pciehp_ist+0x22e/0x260
? __pfx_irq_thread_fn+0x10/0x10
irq_thread_fn+0x21/0x70
irq_thread+0xf8/0x1c0
? __pfx_irq_thread_dtor+0x10/0x10
? __pfx_irq_thread+0x10/0x10
kthread+0xef/0x120
? __pfx_kthread+0x10/0x10
ret_from_fork+0x44/0x70
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1b/0x30
</TASK>
Fixes: 6f0689fdf19e ("RDMA/mlx5: Introduce mlx5_umr_post_send_wait()")
Signed-off-by: Jian Wen <wenjian1@xiaomi.com>
---
drivers/infiniband/hw/mlx5/umr.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/infiniband/hw/mlx5/umr.c b/drivers/infiniband/hw/mlx5/umr.c
index 4e562e0dd9e1..f3700ef6ea07 100644
--- a/drivers/infiniband/hw/mlx5/umr.c
+++ b/drivers/infiniband/hw/mlx5/umr.c
@@ -254,7 +254,7 @@ static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey, struct ib_cqe *cqe,
unsigned int idx;
int size, err;
- if (unlikely(mdev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR))
+ if (unlikely(mlx5_cmd_is_down(mdev)))
return -EIO;
spin_lock_irqsave(&qp->sq.lock, flags);
--
2.43.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down
2025-10-09 14:23 [PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down Jian Wen
@ 2025-10-09 17:49 ` Leon Romanovsky
2025-10-09 18:02 ` Jason Gunthorpe
2025-10-11 23:35 ` kernel test robot
2 siblings, 0 replies; 6+ messages in thread
From: Leon Romanovsky @ 2025-10-09 17:49 UTC (permalink / raw)
To: Jian Wen; +Cc: jgg, Jian Wen, linux-rdma
On Thu, Oct 09, 2025 at 10:23:20PM +0800, Jian Wen wrote:
> mlx5r_umr_post_send_wait() will get stuck when the pcie link is down
> as the call trace[1].
>
> When pciehp detects the link is down it calls
> pci_dev_set_disconnected() before mlx5_ib_dereg_mr(). Thus we can use
> mlx5_cmd_is_down() to detect PCIe Surprise Link Down in
> mlx5r_umr_post_send().
>
> [1]
> pcieport 0000:b9:01.0: pciehp: Slot(2-4): Link Down
> pcieport 0000:b9:01.0: pciehp: Slot(2-4): Card not present
> mlx5_core 0000:bb:00.0: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
> mlx5_core 0000:bb:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
> mlx5_core 0000:bb:00.0: poll_health:1083:(pid 0): Fatal error 1 detected
> mlx5_core 0000:bb:00.0: print_health_info:491:(pid 0): PCI slot is unavailable
> INFO: task irq/105-pciehp:1246 blocked for more than 122 seconds.
> Tainted: G OE 6.8.0-54-generic #56-Ubuntu
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> task:irq/105-pciehp state:D stack:0 pid:1246 tgid:1246 ppid:2 flags:0x00004000
> Call Trace:
> <TASK>
> __schedule+0x27c/0x6b0
> schedule+0x33/0x110
> schedule_timeout+0x157/0x170
> wait_for_completion+0x88/0x150
> mlx5r_umr_post_send_wait+0x15f/0x2d0 [mlx5_ib]
> ? __pfx_mlx5r_umr_done+0x10/0x10 [mlx5_ib]
> mlx5r_umr_revoke_mr+0x98/0xc0 [mlx5_ib]
> __mlx5_ib_dereg_mr+0x24a/0x740 [mlx5_ib]
> ? wait_for_completion+0x114/0x150
> mlx5_ib_dereg_mr+0x21/0xc0 [mlx5_ib]
> ? rdma_restrack_del+0x59/0x160 [ib_core]
> ib_dereg_mr_user+0x41/0xc0 [ib_core]
> uverbs_free_mr+0x15/0x30 [ib_uverbs]
> destroy_hw_idr_uobject+0x21/0x60 [ib_uverbs]
> uverbs_destroy_uobject+0x38/0x1d0 [ib_uverbs]
> __uverbs_cleanup_ufile+0xcf/0x150 [ib_uverbs]
> uverbs_destroy_ufile_hw+0x3f/0x100 [ib_uverbs]
> ib_uverbs_remove_one+0x147/0x1c0 [ib_uverbs]
> remove_client_context+0x95/0x100 [ib_core]
> disable_device+0x8f/0x180 [ib_core]
> __ib_unregister_device+0x108/0x170 [ib_core]
> ? __pfx_mlx5_ib_stage_ib_reg_cleanup+0x10/0x10 [mlx5_ib]
> ib_unreister_device+0x26/0x40 [ib_core]
> mlx5_ib_stage_ib_reg_cleanup+0xe/0x20 [mlx5_ib]
> mlx5r_remove+0x52/0xb0 [mlx5_ib]
> auxiliary_bus_remove+0x1c/0x40
> device_remove+0x40/0x80
> device_release_driver_internal+0x20b/0x270
> device_release_driver+0x12/0x20
> bus_remove_device+0xcb/0x140
> device_del+0x161/0x3e0
> ? is_ib_enabled+0x52/0x90 [mlx5_core]
> mlx5_rescan_drivers_locked+0xfe/0x350 [mlx5_core]
> mlx5_unregister_device+0x38/0x60 [mlx5_core]
> mlx5_uninit_one+0x39/0x160 [mlx5_core]
> remove_one+0x55/0x100 [mlx5_core]
> pci_device_remove+0x3e/0xb0
> device_remove+0x40/0x80
> device_release_driver_internal+0x20b/0x270
> device_release_driver+0x12/0x20
> pci_stop_bus_device+0x7a/0xb0
> pci_stop_and_remove_bus_device+0x12/0x30
> pciehp_unconfigure_device+0x98/0x170
> pciehp_disable_slot+0x69/0x130
> pciehp_handle_presence_or_link_change+0x71/0x220
> pciehp_ist+0x22e/0x260
> ? __pfx_irq_thread_fn+0x10/0x10
> irq_thread_fn+0x21/0x70
> irq_thread+0xf8/0x1c0
> ? __pfx_irq_thread_dtor+0x10/0x10
> ? __pfx_irq_thread+0x10/0x10
> kthread+0xef/0x120
> ? __pfx_kthread+0x10/0x10
> ret_from_fork+0x44/0x70
> ? __pfx_kthread+0x10/0x10
> ret_from_fork_asm+0x1b/0x30
> </TASK>
>
> Fixes: 6f0689fdf19e ("RDMA/mlx5: Introduce mlx5_umr_post_send_wait()")
> Signed-off-by: Jian Wen <wenjian1@xiaomi.com>
> ---
> drivers/infiniband/hw/mlx5/umr.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/infiniband/hw/mlx5/umr.c b/drivers/infiniband/hw/mlx5/umr.c
> index 4e562e0dd9e1..f3700ef6ea07 100644
> --- a/drivers/infiniband/hw/mlx5/umr.c
> +++ b/drivers/infiniband/hw/mlx5/umr.c
> @@ -254,7 +254,7 @@ static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey, struct ib_cqe *cqe,
> unsigned int idx;
> int size, err;
>
> - if (unlikely(mdev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR))
> + if (unlikely(mlx5_cmd_is_down(mdev)))
mlx5_cmd_is_down() is defined in mlx5_core.ko module and doesn't have EXPORT_SYMBOL.
Did you success to compile this patch?
Thanks
> return -EIO;
>
> spin_lock_irqsave(&qp->sq.lock, flags);
> --
> 2.43.0
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down
2025-10-09 14:23 [PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down Jian Wen
2025-10-09 17:49 ` Leon Romanovsky
@ 2025-10-09 18:02 ` Jason Gunthorpe
2025-10-16 13:32 ` Maher Sanalla
2025-10-11 23:35 ` kernel test robot
2 siblings, 1 reply; 6+ messages in thread
From: Jason Gunthorpe @ 2025-10-09 18:02 UTC (permalink / raw)
To: Jian Wen; +Cc: leonro, Jian Wen, linux-rdma
On Thu, Oct 09, 2025 at 10:23:20PM +0800, Jian Wen wrote:
> --- a/drivers/infiniband/hw/mlx5/umr.c
> +++ b/drivers/infiniband/hw/mlx5/umr.c
> @@ -254,7 +254,7 @@ static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey, struct ib_cqe *cqe,
> unsigned int idx;
> int size, err;
>
> - if (unlikely(mdev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR))
> + if (unlikely(mlx5_cmd_is_down(mdev)))
> return -EIO;
I feel like this is just changing the race around..
The removal flow for the device is different if the HW is working than
if it isn't.
If it isn't working then the removal should disable and cancel all the
UMRs, using the umrc->lock. Otherwise there will be dead threads
floating around. It should also be setting
MLX5_DEVICE_STATE_INTERNAL_ERROR way at the start of removal.
So IDK, maybe check mlx5_cmd_is_down() and trigger the flow to
activate INTERNAL_ERROR befor doing anything else?
Jason
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down
2025-10-09 18:02 ` Jason Gunthorpe
@ 2025-10-16 13:32 ` Maher Sanalla
[not found] ` <CAMXzGWKbt1+U6O2EUuzFRbbUc7Qx7Xy+2+1WCk5_OASejJN=JQ@mail.gmail.com>
0 siblings, 1 reply; 6+ messages in thread
From: Maher Sanalla @ 2025-10-16 13:32 UTC (permalink / raw)
To: Jason Gunthorpe, Jian Wen; +Cc: leonro, Jian Wen, linux-rdma
On 09/10/2025 21:02, Jason Gunthorpe wrote:
> External email: Use caution opening links or attachments
>
>
> On Thu, Oct 09, 2025 at 10:23:20PM +0800, Jian Wen wrote:
>> --- a/drivers/infiniband/hw/mlx5/umr.c
>> +++ b/drivers/infiniband/hw/mlx5/umr.c
>> @@ -254,7 +254,7 @@ static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey, struct ib_cqe *cqe,
>> unsigned int idx;
>> int size, err;
>>
>> - if (unlikely(mdev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR))
>> + if (unlikely(mlx5_cmd_is_down(mdev)))
>> return -EIO;
>
> I feel like this is just changing the race around..
>
> The removal flow for the device is different if the HW is working than
> if it isn't.
>
> If it isn't working then the removal should disable and cancel all the
> UMRs, using the umrc->lock. Otherwise there will be dead threads
> floating around. It should also be setting
> MLX5_DEVICE_STATE_INTERNAL_ERROR way at the start of removal.
>
> So IDK, maybe check mlx5_cmd_is_down() and trigger the flow to
> activate INTERNAL_ERROR befor doing anything else?
>
> Jason
>
I agree with Jason concern. While this may serve as a workaround for
certain cases, the race condition still exists. We’re aware of this
issue and are working on a proper fix. The intended solution is that
when the device enters an error state, the UMR state should transition
directly to error as well, preventing any new UMRs from being posted.
Maher
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down
2025-10-09 14:23 [PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down Jian Wen
2025-10-09 17:49 ` Leon Romanovsky
2025-10-09 18:02 ` Jason Gunthorpe
@ 2025-10-11 23:35 ` kernel test robot
2 siblings, 0 replies; 6+ messages in thread
From: kernel test robot @ 2025-10-11 23:35 UTC (permalink / raw)
To: Jian Wen, jgg, leonro
Cc: llvm, oe-kbuild-all, Jian Wen, linux-rdma, wenjianhn
Hi Jian,
kernel test robot noticed the following build errors:
[auto build test ERROR on rdma/for-next]
[also build test ERROR on linus/master v6.17 next-20251010]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Jian-Wen/RDMA-mlx5-Use-mlx5_cmd_is_down-to-detect-PCIe-Surprise-Link-Down/20251009-224616
base: https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git for-next
patch link: https://lore.kernel.org/r/20251009142326.3794769-1-wenjian1%40xiaomi.com
patch subject: [PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down
config: s390-defconfig (https://download.01.org/0day-ci/archive/20251012/202510120715.P7oFOBOQ-lkp@intel.com/config)
compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 39f292ffa13d7ca0d1edff27ac8fd55024bb4d19)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251012/202510120715.P7oFOBOQ-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202510120715.P7oFOBOQ-lkp@intel.com/
All errors (new ones prefixed by >>, old ones prefixed by <<):
>> ERROR: modpost: "mlx5_cmd_is_down" [drivers/infiniband/hw/mlx5/mlx5_ib.ko] undefined!
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-01-26 9:32 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-10-09 14:23 [PATCH] RDMA/mlx5: Use mlx5_cmd_is_down to detect PCIe Surprise Link Down Jian Wen
2025-10-09 17:49 ` Leon Romanovsky
2025-10-09 18:02 ` Jason Gunthorpe
2025-10-16 13:32 ` Maher Sanalla
[not found] ` <CAMXzGWKbt1+U6O2EUuzFRbbUc7Qx7Xy+2+1WCk5_OASejJN=JQ@mail.gmail.com>
2026-01-26 9:32 ` Maher Sanalla
2025-10-11 23:35 ` kernel test robot
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.