Linux RDMA and InfiniBand development
 help / color / mirror / Atom feed
From: sashiko-bot@kernel.org
To: "Yishai Hadas" <yishaih@nvidia.com>
Cc: linux-rdma@vger.kernel.org
Subject: Re: [PATCH V2 rdma-next 01/21] net/mlx5: Add mlx5_trigger_health_work_force()
Date: Wed, 18 Nov 2026 17:13:40 +0000	[thread overview]
Message-ID: <sashiko-outbox-221602@kernel.org> (raw)
In-Reply-To: <20261117170914.27410-2-yishaih@nvidia.com>

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] net/mlx5: uncanceled force_fatal_report_work tears down
  recovered device
--

commit 8880445e0e3559ece304416ceb136dde00d937a0
Author: Yishai Hadas <yishaih@nvidia.com>

net/mlx5: Add mlx5_trigger_health_work_force()

This commit adds mlx5_trigger_health_work_force() to queue a dedicated
health work item that stops the firmware and forces the device into the
internal error state even when the health sensors report no fault.

> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c
> index aeeb136f5ebc7..e76351860b154 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c
> @@ -627,18 +627,11 @@ mlx5_fw_fatal_reporter_dump(struct devlink_health_reporter *reporter,
>  	return err;
>  }
>  
> -static void mlx5_fw_fatal_reporter_err_work(struct work_struct *work)
> +static void mlx5_fw_fatal_reporter_err(struct mlx5_core_dev *dev, bool forced)
>  {
>  	struct mlx5_fw_reporter_ctx fw_reporter_ctx;
> -	struct mlx5_core_health *health;
> -	struct mlx5_core_dev *dev;
> -	struct devlink *devlink;
> -	struct mlx5_priv *priv;
> -
> -	health = container_of(work, struct mlx5_core_health, fatal_report_work);
> -	priv = container_of(health, struct mlx5_priv, health);
> -	dev = container_of(priv, struct mlx5_core_dev, priv);
> -	devlink = priv_to_devlink(dev);
> +	struct mlx5_core_health *health = &dev->priv.health;
> +	struct devlink *devlink = priv_to_devlink(dev);
>  
>  	mutex_lock(&dev->intf_state_mutex);
>  	if (test_bit(MLX5_DROP_HEALTH_WORK, &health->flags)) {
> @@ -646,8 +639,39 @@ static void mlx5_fw_fatal_reporter_err_work(struct work_struct *work)
>  		mutex_unlock(&dev->intf_state_mutex);
>  		return;
>  	}
> +	if (forced) {
> +		int err;
> +
> +		/*
> +		 * Once the interface is down, the unload already removed the
> +		 * consumers that need the error flow. Once the cmd interface
> +		 * is down, the path that brought it down owns the flush and
> +		 * the recovery.
> +		 */
> +		if (mlx5_cmd_is_down(dev) ||
> +		    !test_bit(MLX5_INTERFACE_STATE_UP, &dev->intf_state)) {
> +			mutex_unlock(&dev->intf_state_mutex);
> +			return;
> +		}
> +		/*
> +		 * The firmware might still be running, so stop it first. Do it
> +		 * under intf_state_mutex, so the cmd interface is valid.
> +		 */
> +		mlx5_stop_health_poll(dev, false);
> +		err = mlx5_cmd_fast_teardown_hca(dev);
> +		if (err)
> +			err = mlx5_cmd_force_teardown_hca(dev);
> +		if (err)
> +			mlx5_core_err(dev, "Failed to stop the device before forcing the error state, err %d\n",
> +				      err);
> +	}

[Severity: Medium]
Can a queued force_fatal_report_work run right after an in-progress
recovery completes on health->wq and tear down the newly recovered device?

Because force_fatal_report_work is a separate work_struct on the
single-threaded health->wq and neither mlx5_unload_one_devl_locked() nor
mlx5_function_teardown() cancels it, a forced trigger that races with a
fatal sensor error queues force_fatal_report_work behind fatal_report_work:

CPU0 (health->wq)                 CPU1
poll_health()
  queue_work(fatal_report_work)
                                  mlx5_trigger_health_work_force()
                                    queue_work(force_fatal_report_work)
mlx5_fw_fatal_reporter_err_work()
  devlink_health_report()
    ...
      mlx5_health_try_recover()
        mlx5_unload_one_devl_locked()
        mlx5_recover_device()
          mlx5_load_one_devl_locked()
            /* restores CMDIF_STATE_UP and INTERFACE_STATE_UP */
mlx5_fw_fatal_reporter_force_err_work()
  /* mlx5_cmd_is_down() is false, INTERFACE_STATE_UP is set */
  mlx5_cmd_fast_teardown_hca()    <-- tears down recovered HCA
  enter_error_state(dev, true)
  devlink_health_report()         <-- returns -ECANCELED (grace period)
  mlx5_unload_one(dev, false)

When fatal_report_work finishes, the device is back up, so the
mlx5_cmd_is_down() and MLX5_INTERFACE_STATE_UP checks pass.
force_fatal_report_work then stops the healthy HCA, forces
MLX5_DEVICE_STATE_INTERNAL_ERROR, hits -ECANCELED in
devlink_health_report() because the reporter is still within the
post-recovery graceful period, and unloads the driver.  The same thing
happens if mlx5_trigger_health_work_force() is called during devlink
reload while mlx5_unload_one_devl_locked() is detaching consumers before
mlx5_load_one_devl_locked() brings the interface back up.

Cancelling force_fatal_report_work during teardown (or draining it before
bringing the interface back up) would prevent stale forced work from
acting on the new device instance.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261117170914.27410-1-yishaih@nvidia.com?part=1

  reply	other threads:[~2026-11-18 17:13 UTC|newest]

Thread overview: 43+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-17 17:08 [PATCH V2 rdma-next 00/21] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
2026-11-17 17:08 ` [PATCH V2 rdma-next 01/21] net/mlx5: Add mlx5_trigger_health_work_force() Yishai Hadas
2026-11-18 17:13   ` sashiko-bot [this message]
2026-11-17 17:08 ` [PATCH V2 rdma-next 02/21] net/mlx5: Derive adversarial mode from CC guest memory encryption Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 03/21] net/mlx5: Add capability filtering in adversarial mode Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 04/21] net/mlx5: Catch too low max number of EQs Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 05/21] net/mlx5: Add macros for reading log-based caps Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 06/21] RDMA/mlx5: Reject QP transport types not enabled by FW capability Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 07/21] RDMA/mlx5: Require IB_QPT_DRIVER for the DC vendor QP types Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 08/21] RDMA/core: Suppress kverbs_provider for adversarial_mode devices Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 09/21] RDMA/mlx5: Set ib_device::adversarial_mode from mlx5_core Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 10/21] RDMA/mlx5: Remove duplicate mlx5_ib_umr_context struct Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 11/21] RDMA/mlx5: Add shadow FIFO infrastructure for UMR completions Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 12/21] RDMA/mlx5: Verify UMR CQ polling against a hostile NIC Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 13/21] RDMA/mlx5: Keep the UMR QP failed once a forged CQE is detected Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 14/21] RDMA/core: Set the CQ comp_handler after its poll context is set up Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 15/21] RDMA/mlx5: Ignore completion events for a kernel CQ that is not ready Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 16/21] RDMA/mlx5: Validate EQE port and GSI port before indexing devr.ports[] Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 17/21] RDMA/mlx5: Validate firmware RQ/SQ state before sqrq_trans[] lookup Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 18/21] RDMA/mlx5: Clamp log_max_cq_sz shift in CQ size checks Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 19/21] RDMA/mlx5: Clamp log_max_* shifts in QP and WQ limit checks Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 20/21] RDMA/mlx5: Clamp log_max_* shifts in query_device and ucontext paths Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 21/21] RDMA/core: Report and enforce adversarial mode via GET_CONTEXT Yishai Hadas
2026-11-18 17:13   ` sashiko-bot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=sashiko-outbox-221602@kernel.org \
    --to=sashiko-bot@kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=sashiko-reviews@lists.linux.dev \
    --cc=yishaih@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox