Re: [PATCH rdma-next v3] RDMA/mlx5: quiesce CQ polling before device shutdown on reboot

[Leon Romanovsky <leon@kernel.org>]

On Thu, Jul 16, 2026 at 05:03:37PM +0800, Chenguang Zhao wrote:
> Hi, Leon
> reboot -f skips orderly shutdown and goes directly to:
> 
> kernel_restart_prepare() -> device_shutdown() -> mlx5 shutdown
> Upper layers may still hold live CQs, while ib-comp-wq keeps
> polling — a use-after-free race.

The point is that this flow is neither RDMA- nor mlx5-specific, and it
works as expected. mlx5 shutdown() stops the FW/HW, while the kernel
stops and tears down the running threads.

> 
> Normal reboot usually works because userspace has already
> torn down RDMA and called ib_free_cq().

There is no difference, from the kernel's perspective, between a
normal and a forced reboot, except that the former bypasses
userspace shutdown.

Thanks

> 
> Thanks
> 
> 在 2026/7/16 16:42, Leon Romanovsky 写道:
> > On Wed, Jul 15, 2026 at 04:23:07PM +0800, Chenguang Zhao wrote:
> >> From: Chenguang Zhao <zhaochenguang@kylinos.cn>
> >>
> >> On reboot -f with NFS over RDMA, mlx5 shutdown can tear the device
> >> down while ib-comp-wq still polls live CQs, leading to UAF in
> >> wr_cqe->done().
> >>
> >> Mark the device shutting down before teardown, flush completion
> >> workqueues so in-flight pollers observe the flag, skip SYS_ERROR
> >> completion delivery, and make poll/arm CQ a no-op under the CQ lock
> >> while shutting down.
> >>
> >> Signed-off-by: Chenguang Zhao <zhaochenguang@kylinos.cn>
> >> ---
> >> changelog:
> >>  - Fix the race on MLX5_INTERFACE_STATE_SHUTTING_DOWN: set the
> >>    flag, then flush ib-comp / mlx5_ib event workqueues via an
> >>    mlx5_ib quiesce hook before fast_unload/teardown.
> >>  - Check shutting-down under cq->lock in mlx5_ib_poll_cq/arm_cq.
> >>  - Export ib_comp_wq and ib_comp_unbound_wq so modular mlx5_ib
> >>    can flush them.
> >>
> >> v2:
> >>  https://lore.kernel.org/all/20260714075558.1420384-1-chenguang.zhao@linux.dev/
> >>
> >> v1:
> >>  https://lore.kernel.org/all/20260702073422.279820-1-chenguang.zhao@linux.dev/
> >>
> >>  drivers/infiniband/core/device.c              |  2 ++
> >>  drivers/infiniband/hw/mlx5/cq.c               | 11 ++++++++++
> >>  drivers/infiniband/hw/mlx5/main.c             | 20 +++++++++++++++++++
> >>  .../net/ethernet/mellanox/mlx5/core/health.c  |  3 +++
> >>  .../net/ethernet/mellanox/mlx5/core/main.c    | 10 ++++++++++
> >>  .../mellanox/mlx5/core/sf/dev/driver.c        |  3 +++
> >>  include/linux/mlx5/driver.h                   | 11 ++++++++++
> >>  7 files changed, 60 insertions(+)
> > <...>
> >
> >> +static void mlx5_ib_shutdown_quiesce(void)
> >> +{
> >> +	flush_workqueue(ib_comp_wq);
> >> +	flush_workqueue(ib_comp_unbound_wq);
> >> +	flush_workqueue(mlx5_ib_event_wq);
> >> +}
> > These workqueues are shared by all IB drivers and the core. Drivers
> > must not flush or destroy them.
> >
> > Why this is not FW issue?
> >
> > Thanks