From: sashiko-bot@kernel.org
To: "Yishai Hadas" <yishaih@nvidia.com>
Cc: linux-rdma@vger.kernel.org
Subject: Re: [PATCH V2 rdma-next 01/21] net/mlx5: Add mlx5_trigger_health_work_force()
Date: Wed, 18 Nov 2026 17:13:40 +0000 [thread overview]
Message-ID: <sashiko-outbox-221602@kernel.org> (raw)
In-Reply-To: <20261117170914.27410-2-yishaih@nvidia.com>
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] net/mlx5: uncanceled force_fatal_report_work tears down
recovered device
--
commit 8880445e0e3559ece304416ceb136dde00d937a0
Author: Yishai Hadas <yishaih@nvidia.com>
net/mlx5: Add mlx5_trigger_health_work_force()
This commit adds mlx5_trigger_health_work_force() to queue a dedicated
health work item that stops the firmware and forces the device into the
internal error state even when the health sensors report no fault.
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c
> index aeeb136f5ebc7..e76351860b154 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c
> @@ -627,18 +627,11 @@ mlx5_fw_fatal_reporter_dump(struct devlink_health_reporter *reporter,
> return err;
> }
>
> -static void mlx5_fw_fatal_reporter_err_work(struct work_struct *work)
> +static void mlx5_fw_fatal_reporter_err(struct mlx5_core_dev *dev, bool forced)
> {
> struct mlx5_fw_reporter_ctx fw_reporter_ctx;
> - struct mlx5_core_health *health;
> - struct mlx5_core_dev *dev;
> - struct devlink *devlink;
> - struct mlx5_priv *priv;
> -
> - health = container_of(work, struct mlx5_core_health, fatal_report_work);
> - priv = container_of(health, struct mlx5_priv, health);
> - dev = container_of(priv, struct mlx5_core_dev, priv);
> - devlink = priv_to_devlink(dev);
> + struct mlx5_core_health *health = &dev->priv.health;
> + struct devlink *devlink = priv_to_devlink(dev);
>
> mutex_lock(&dev->intf_state_mutex);
> if (test_bit(MLX5_DROP_HEALTH_WORK, &health->flags)) {
> @@ -646,8 +639,39 @@ static void mlx5_fw_fatal_reporter_err_work(struct work_struct *work)
> mutex_unlock(&dev->intf_state_mutex);
> return;
> }
> + if (forced) {
> + int err;
> +
> + /*
> + * Once the interface is down, the unload already removed the
> + * consumers that need the error flow. Once the cmd interface
> + * is down, the path that brought it down owns the flush and
> + * the recovery.
> + */
> + if (mlx5_cmd_is_down(dev) ||
> + !test_bit(MLX5_INTERFACE_STATE_UP, &dev->intf_state)) {
> + mutex_unlock(&dev->intf_state_mutex);
> + return;
> + }
> + /*
> + * The firmware might still be running, so stop it first. Do it
> + * under intf_state_mutex, so the cmd interface is valid.
> + */
> + mlx5_stop_health_poll(dev, false);
> + err = mlx5_cmd_fast_teardown_hca(dev);
> + if (err)
> + err = mlx5_cmd_force_teardown_hca(dev);
> + if (err)
> + mlx5_core_err(dev, "Failed to stop the device before forcing the error state, err %d\n",
> + err);
> + }
[Severity: Medium]
Can a queued force_fatal_report_work run right after an in-progress
recovery completes on health->wq and tear down the newly recovered device?
Because force_fatal_report_work is a separate work_struct on the
single-threaded health->wq and neither mlx5_unload_one_devl_locked() nor
mlx5_function_teardown() cancels it, a forced trigger that races with a
fatal sensor error queues force_fatal_report_work behind fatal_report_work:
CPU0 (health->wq) CPU1
poll_health()
queue_work(fatal_report_work)
mlx5_trigger_health_work_force()
queue_work(force_fatal_report_work)
mlx5_fw_fatal_reporter_err_work()
devlink_health_report()
...
mlx5_health_try_recover()
mlx5_unload_one_devl_locked()
mlx5_recover_device()
mlx5_load_one_devl_locked()
/* restores CMDIF_STATE_UP and INTERFACE_STATE_UP */
mlx5_fw_fatal_reporter_force_err_work()
/* mlx5_cmd_is_down() is false, INTERFACE_STATE_UP is set */
mlx5_cmd_fast_teardown_hca() <-- tears down recovered HCA
enter_error_state(dev, true)
devlink_health_report() <-- returns -ECANCELED (grace period)
mlx5_unload_one(dev, false)
When fatal_report_work finishes, the device is back up, so the
mlx5_cmd_is_down() and MLX5_INTERFACE_STATE_UP checks pass.
force_fatal_report_work then stops the healthy HCA, forces
MLX5_DEVICE_STATE_INTERNAL_ERROR, hits -ECANCELED in
devlink_health_report() because the reporter is still within the
post-recovery graceful period, and unloads the driver. The same thing
happens if mlx5_trigger_health_work_force() is called during devlink
reload while mlx5_unload_one_devl_locked() is detaching consumers before
mlx5_load_one_devl_locked() brings the interface back up.
Cancelling force_fatal_report_work during teardown (or draining it before
bringing the interface back up) would prevent stale forced work from
acting on the new device instance.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261117170914.27410-1-yishaih@nvidia.com?part=1
next prev parent reply other threads:[~2026-11-18 17:13 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-17 17:08 [PATCH V2 rdma-next 00/21] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
2026-11-17 17:08 ` [PATCH V2 rdma-next 01/21] net/mlx5: Add mlx5_trigger_health_work_force() Yishai Hadas
2026-11-18 17:13 ` sashiko-bot [this message]
2026-11-17 17:08 ` [PATCH V2 rdma-next 02/21] net/mlx5: Derive adversarial mode from CC guest memory encryption Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 03/21] net/mlx5: Add capability filtering in adversarial mode Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 04/21] net/mlx5: Catch too low max number of EQs Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 05/21] net/mlx5: Add macros for reading log-based caps Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 06/21] RDMA/mlx5: Reject QP transport types not enabled by FW capability Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 07/21] RDMA/mlx5: Require IB_QPT_DRIVER for the DC vendor QP types Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 08/21] RDMA/core: Suppress kverbs_provider for adversarial_mode devices Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 09/21] RDMA/mlx5: Set ib_device::adversarial_mode from mlx5_core Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 10/21] RDMA/mlx5: Remove duplicate mlx5_ib_umr_context struct Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 11/21] RDMA/mlx5: Add shadow FIFO infrastructure for UMR completions Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 12/21] RDMA/mlx5: Verify UMR CQ polling against a hostile NIC Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 13/21] RDMA/mlx5: Keep the UMR QP failed once a forged CQE is detected Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 14/21] RDMA/core: Set the CQ comp_handler after its poll context is set up Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 15/21] RDMA/mlx5: Ignore completion events for a kernel CQ that is not ready Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 16/21] RDMA/mlx5: Validate EQE port and GSI port before indexing devr.ports[] Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 17/21] RDMA/mlx5: Validate firmware RQ/SQ state before sqrq_trans[] lookup Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 18/21] RDMA/mlx5: Clamp log_max_cq_sz shift in CQ size checks Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 19/21] RDMA/mlx5: Clamp log_max_* shifts in QP and WQ limit checks Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 20/21] RDMA/mlx5: Clamp log_max_* shifts in query_device and ucontext paths Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 21/21] RDMA/core: Report and enforce adversarial mode via GET_CONTEXT Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=sashiko-outbox-221602@kernel.org \
--to=sashiko-bot@kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=sashiko-reviews@lists.linux.dev \
--cc=yishaih@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox