From: sashiko-bot@kernel.org
To: "Yishai Hadas" <yishaih@nvidia.com>
Cc: linux-rdma@vger.kernel.org
Subject: Re: [PATCH V2 rdma-next 01/21] net/mlx5: Add mlx5_trigger_health_work_force()
Date: Wed, 18 Nov 2026 17:13:40 +0000 [thread overview]
Message-ID: <sashiko-outbox-221602@kernel.org> (raw)
In-Reply-To: <20261117170914.27410-2-yishaih@nvidia.com>
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] net/mlx5: uncanceled force_fatal_report_work tears down
recovered device
--
commit 8880445e0e3559ece304416ceb136dde00d937a0
Author: Yishai Hadas <yishaih@nvidia.com>
net/mlx5: Add mlx5_trigger_health_work_force()
This commit adds mlx5_trigger_health_work_force() to queue a dedicated
health work item that stops the firmware and forces the device into the
internal error state even when the health sensors report no fault.
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c
> index aeeb136f5ebc7..e76351860b154 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c
> @@ -627,18 +627,11 @@ mlx5_fw_fatal_reporter_dump(struct devlink_health_reporter *reporter,
> return err;
> }
>
> -static void mlx5_fw_fatal_reporter_err_work(struct work_struct *work)
> +static void mlx5_fw_fatal_reporter_err(struct mlx5_core_dev *dev, bool forced)
> {
> struct mlx5_fw_reporter_ctx fw_reporter_ctx;
> - struct mlx5_core_health *health;
> - struct mlx5_core_dev *dev;
> - struct devlink *devlink;
> - struct mlx5_priv *priv;
> -
> - health = container_of(work, struct mlx5_core_health, fatal_report_work);
> - priv = container_of(health, struct mlx5_priv, health);
> - dev = container_of(priv, struct mlx5_core_dev, priv);
> - devlink = priv_to_devlink(dev);
> + struct mlx5_core_health *health = &dev->priv.health;
> + struct devlink *devlink = priv_to_devlink(dev);
>
> mutex_lock(&dev->intf_state_mutex);
> if (test_bit(MLX5_DROP_HEALTH_WORK, &health->flags)) {
> @@ -646,8 +639,39 @@ static void mlx5_fw_fatal_reporter_err_work(struct work_struct *work)
> mutex_unlock(&dev->intf_state_mutex);
> return;
> }
> + if (forced) {
> + int err;
> +
> + /*
> + * Once the interface is down, the unload already removed the
> + * consumers that need the error flow. Once the cmd interface
> + * is down, the path that brought it down owns the flush and
> + * the recovery.
> + */
> + if (mlx5_cmd_is_down(dev) ||
> + !test_bit(MLX5_INTERFACE_STATE_UP, &dev->intf_state)) {
> + mutex_unlock(&dev->intf_state_mutex);
> + return;
> + }
> + /*
> + * The firmware might still be running, so stop it first. Do it
> + * under intf_state_mutex, so the cmd interface is valid.
> + */
> + mlx5_stop_health_poll(dev, false);
> + err = mlx5_cmd_fast_teardown_hca(dev);
> + if (err)
> + err = mlx5_cmd_force_teardown_hca(dev);
> + if (err)
> + mlx5_core_err(dev, "Failed to stop the device before forcing the error state, err %d\n",
> + err);
> + }
[Severity: Medium]
Can a queued force_fatal_report_work run right after an in-progress
recovery completes on health->wq and tear down the newly recovered device?
Because force_fatal_report_work is a separate work_struct on the
single-threaded health->wq and neither mlx5_unload_one_devl_locked() nor
mlx5_function_teardown() cancels it, a forced trigger that races with a
fatal sensor error queues force_fatal_report_work behind fatal_report_work:
CPU0 (health->wq) CPU1
poll_health()
queue_work(fatal_report_work)
mlx5_trigger_health_work_force()
queue_work(force_fatal_report_work)
mlx5_fw_fatal_reporter_err_work()
devlink_health_report()
...
mlx5_health_try_recover()
mlx5_unload_one_devl_locked()
mlx5_recover_device()
mlx5_load_one_devl_locked()
/* restores CMDIF_STATE_UP and INTERFACE_STATE_UP */
mlx5_fw_fatal_reporter_force_err_work()
/* mlx5_cmd_is_down() is false, INTERFACE_STATE_UP is set */
mlx5_cmd_fast_teardown_hca() <-- tears down recovered HCA
enter_error_state(dev, true)
devlink_health_report() <-- returns -ECANCELED (grace period)
mlx5_unload_one(dev, false)
When fatal_report_work finishes, the device is back up, so the
mlx5_cmd_is_down() and MLX5_INTERFACE_STATE_UP checks pass.
force_fatal_report_work then stops the healthy HCA, forces
MLX5_DEVICE_STATE_INTERNAL_ERROR, hits -ECANCELED in
devlink_health_report() because the reporter is still within the
post-recovery graceful period, and unloads the driver. The same thing
happens if mlx5_trigger_health_work_force() is called during devlink
reload while mlx5_unload_one_devl_locked() is detaching consumers before
mlx5_load_one_devl_locked() brings the interface back up.
Cancelling force_fatal_report_work during teardown (or draining it before
bringing the interface back up) would prevent stale forced work from
acting on the new device instance.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261117170914.27410-1-yishaih@nvidia.com?part=1
next prev parent reply other threads:[~2026-11-18 17:13 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-17 17:08 [PATCH V2 rdma-next 00/21] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
2026-11-17 17:08 ` [PATCH V2 rdma-next 01/21] net/mlx5: Add mlx5_trigger_health_work_force() Yishai Hadas
2026-11-18 17:13 ` sashiko-bot [this message]
2026-11-19 8:06 ` Yishai Hadas
2026-11-17 17:08 ` [PATCH V2 rdma-next 02/21] net/mlx5: Derive adversarial mode from CC guest memory encryption Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 03/21] net/mlx5: Add capability filtering in adversarial mode Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 04/21] net/mlx5: Catch too low max number of EQs Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-19 7:47 ` Yishai Hadas
2026-11-17 17:08 ` [PATCH V2 rdma-next 05/21] net/mlx5: Add macros for reading log-based caps Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 06/21] RDMA/mlx5: Reject QP transport types not enabled by FW capability Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 07/21] RDMA/mlx5: Require IB_QPT_DRIVER for the DC vendor QP types Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 08/21] RDMA/core: Suppress kverbs_provider for adversarial_mode devices Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 09/21] RDMA/mlx5: Set ib_device::adversarial_mode from mlx5_core Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 10/21] RDMA/mlx5: Remove duplicate mlx5_ib_umr_context struct Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 11/21] RDMA/mlx5: Add shadow FIFO infrastructure for UMR completions Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 12/21] RDMA/mlx5: Verify UMR CQ polling against a hostile NIC Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-19 10:17 ` Yishai Hadas
2026-11-17 17:09 ` [PATCH V2 rdma-next 13/21] RDMA/mlx5: Keep the UMR QP failed once a forged CQE is detected Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 14/21] RDMA/core: Set the CQ comp_handler after its poll context is set up Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 15/21] RDMA/mlx5: Ignore completion events for a kernel CQ that is not ready Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 16/21] RDMA/mlx5: Validate EQE port and GSI port before indexing devr.ports[] Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 17/21] RDMA/mlx5: Validate firmware RQ/SQ state before sqrq_trans[] lookup Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-19 10:47 ` Yishai Hadas
2026-11-17 17:09 ` [PATCH V2 rdma-next 18/21] RDMA/mlx5: Clamp log_max_cq_sz shift in CQ size checks Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 19/21] RDMA/mlx5: Clamp log_max_* shifts in QP and WQ limit checks Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 20/21] RDMA/mlx5: Clamp log_max_* shifts in query_device and ucontext paths Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 21/21] RDMA/core: Report and enforce adversarial mode via GET_CONTEXT Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=sashiko-outbox-221602@kernel.org \
--to=sashiko-bot@kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=sashiko-reviews@lists.linux.dev \
--cc=yishaih@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.