From: sashiko-bot@kernel.org
To: "Yishai Hadas" <yishaih@nvidia.com>
Cc: linux-rdma@vger.kernel.org
Subject: Re: [PATCH V2 rdma-next 12/21] RDMA/mlx5: Verify UMR CQ polling against a hostile NIC
Date: Wed, 18 Nov 2026 17:13:48 +0000 [thread overview]
Message-ID: <sashiko-outbox-221613@kernel.org> (raw)
In-Reply-To: <20261117170914.27410-13-yishaih@nvidia.com>
Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [High] RDMA/mlx5: incomplete umrc->state checks in
mlx5_ib_poll_one_umr()
- [High] RDMA/mlx5: deadlock and lost flush on forged CQE in
mlx5_ib_poll_one_umr()
- [Medium] RDMA/mlx5: CQ re-arm loop in mlx5_ib_arm_cq() when umr_verify
is false
--
commit d8d114091f7d4e4fa6dc6263146eaffadede67be
Author: Yishai Hadas <yishaih@nvidia.com>
RDMA/mlx5: Verify UMR CQ polling against a hostile NIC
This commit adds a dedicated UMR CQ poller for adversarial mode that
validates CQEs against a shadow FIFO of posted UMR requests and triggers
forced health recovery if a forged CQE is detected.
> diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
> index 2581a7693585b..a5197318d96f8 100644
> --- a/drivers/infiniband/hw/mlx5/cq.c
> +++ b/drivers/infiniband/hw/mlx5/cq.c
[ ... ]
> @@ -608,6 +609,190 @@ static int poll_soft_wc(struct mlx5_ib_cq *cq, int num_entries,
> +static int mlx5_ib_poll_one_umr(struct mlx5_ib_cq *cq, struct ib_wc *wc)
> +{
[ ... ]
> + if (opcode == MLX5_CQE_REQ) {
> + wc->status = IB_WC_SUCCESS;
> + } else {
> + /* Copy the CQE, so its syndrome is read once */
> + err_cqe = *(struct mlx5_err_cqe *)cqe64;
> + mlx5_handle_error_cqe(dev, &err_cqe, wc);
> + /*
> + * A real failure during a recovery is forged, the QP
> + * only flushes.
> + */
> + if (wc->status != IB_WC_WR_FLUSH_ERR &&
> + cmpxchg(&umrc->state, MLX5_UMR_STATE_ACTIVE,
> + MLX5_UMR_STATE_RECOVER) != MLX5_UMR_STATE_ACTIVE) {
> + mlx5_ib_err(dev,
> + "UMR error CQE status %d in state %d, possible NIC forgery\n",
> + wc->status, READ_ONCE(umrc->state));
> + goto forged_cqe;
> + }
> + }
[Severity: High]
Should mlx5_ib_poll_one_umr() also verify umrc->state for IB_WC_WR_FLUSH_ERR
and MLX5_CQE_REQ completions?
When wc->status is IB_WC_WR_FLUSH_ERR, umrc->state is not checked. If a
hostile NIC reports MLX5_CQE_SYNDROME_WR_FLUSH_ERR while umrc->state is
MLX5_UMR_STATE_ACTIVE, mlx5_ib_poll_one_umr() accepts the CQE without
changing umrc->state. The waiter in mlx5r_umr_post_send_wait() then sees
IB_WC_WR_FLUSH_ERR, loops around while umrc->state is still
MLX5_UMR_STATE_ACTIVE, and immediately reposts the WQE without sleeping or
running recovery, allowing repeated flush CQEs to trap the thread in an
unkillable infinite loop.
Conversely, once a non-flush error has moved the QP to error and umrc->state
to MLX5_UMR_STATE_RECOVER, subsequent in-flight WQEs can only flush until
mlx5r_umr_recover() resets the QP, but the MLX5_CQE_REQ branch accepts
completions as IB_WC_SUCCESS without checking that umrc->state is
MLX5_UMR_STATE_ACTIVE.
> +
> + /* Pop the shadow FIFO head: validated, dispatch to its waiter */
> + WRITE_ONCE(umrc->pending[head & (MLX5_UMR_MAX_WR - 1)], NULL);
> + umrc->pending_head = head + 1;
> + wc->wr_cqe = &expected->cqe;
> +
> + wc->qp = &qp->ibqp;
> + wq->tail = wq->wqe_head[idx] + 1;
> + /* Advance wq->last_poll, the internal error flush starts from it */
> + handle_atomics(qp, cqe64, wq->last_poll, idx);
> +
> + return 0;
> +
> +forged_cqe:
> + /* mlx5_ib_poll_cq() updates the CI only for a returned completion */
> + mlx5_cq_set_ci(&cq->mcq);
> + /*
> + * The NIC forged a CQE. Fail the UMR path for good, ignore any later
> + * CQE and force the health recovery, as the fatal sensors still read
> + * clean. The internal error flush then completes the waiters. Return
> + * an error, not -EAGAIN, as a CQE was consumed.
> + */
> + umrc->state = MLX5_UMR_STATE_ERR;
> + cq->umr_verify = false;
> + mlx5_trigger_health_work_force(dev->mdev);
> + return -EIO;
> +}
[Severity: High]
Can deferring waiter completion to mlx5_trigger_health_work_force() deadlock
on dev->intf_state_mutex or lose the software flush completion?
During driver teardown, mlx5_uninit_one() and mlx5_unload_one_devl_locked()
hold dev->intf_state_mutex while unregistering mlx5_ib, which deregisters
remaining user MRs via mlx5_ib_dereg_mr() -> mlx5r_umr_revoke_mr() ->
mlx5r_umr_post_send_wait(). If the NIC returns a forged CQE at that point,
stopping CQ polling without completing umrc->pending deadlocks against
force_fatal_report_work:
CPU0 CPU1 (health->wq)
---- -----------------
mlx5_unload_one_devl_locked()
mutex_lock(&dev->intf_state_mutex)
...
mlx5r_umr_post_send_wait()
wait_for_completion(&umr_context.done)
<softirq>
mlx5_ib_poll_one_umr()
mlx5_trigger_health_work_force()
mlx5_fw_fatal_reporter_err()
mutex_lock(&dev->intf_state_mutex)
*** DEADLOCK ***
In mlx5_uninit_one(), mlx5_unregister_device() has also already set
MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV, so mlx5_dev_is_lightweight() is true and
mlx5_trigger_health_work_force() drops the work item outright.
Even during normal operation, the UMR CQ uses IB_POLL_SOFTIRQ, where
IRQ_POLL_F_SCHED stays set on cq->iop until ib_poll_handler() calls
irq_poll_complete() after __ib_process_cq() returns. If
mlx5_ib_handle_internal_error() runs on another CPU after mlx5_ib_poll_cq()
checks mdev->state and before ib_poll_handler() calls irq_poll_complete(),
its irq_poll_sched(&cq->iop) call is a no-op, and mlx5_ib_arm_cq() then
returns 0 without rescheduling cq->iop, leaving waiters hung in
wait_for_completion().
> +static int mlx5_ib_poll_cq_adversarial(struct mlx5_ib_cq *cq, int num_entries,
> + struct ib_wc *wc)
> +{
> + int npolled;
> +
> + if (!cq->umr_verify)
> + return 0;
> +
> + for (npolled = 0; npolled < num_entries; npolled++) {
> + if (mlx5_ib_poll_one_umr(cq, wc + npolled))
> + break;
> + }
> +
> + return npolled;
> +}
[Severity: Medium]
Can returning 0 when cq->umr_verify is false cause an interrupt and softirq
re-arm storm via mlx5_ib_arm_cq()?
When forged_cqe sets cq->umr_verify = false, it consumes only that single
CQE. If the NIC placed another CQE behind it in the UMR CQ (or if a CQE
arrives before umr_verify is enabled or after mlx5_ib_cq_stop_polling()
clears it), mlx5_ib_poll_cq_adversarial() returns 0 without advancing
cq->mcq.cons_index. Because completed < budget, ib_poll_handler() then
calls ib_req_notify_cq() -> mlx5_ib_arm_cq(), which rings the CQ arm
doorbell at the unadvanced cons_index and causes the NIC to fire another
completion EQE immediately in a tight loop.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261117170914.27410-1-yishaih@nvidia.com?part=12
next prev parent reply other threads:[~2026-11-18 17:13 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-17 17:08 [PATCH V2 rdma-next 00/21] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
2026-11-17 17:08 ` [PATCH V2 rdma-next 01/21] net/mlx5: Add mlx5_trigger_health_work_force() Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 02/21] net/mlx5: Derive adversarial mode from CC guest memory encryption Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 03/21] net/mlx5: Add capability filtering in adversarial mode Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 04/21] net/mlx5: Catch too low max number of EQs Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 05/21] net/mlx5: Add macros for reading log-based caps Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 06/21] RDMA/mlx5: Reject QP transport types not enabled by FW capability Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 07/21] RDMA/mlx5: Require IB_QPT_DRIVER for the DC vendor QP types Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 08/21] RDMA/core: Suppress kverbs_provider for adversarial_mode devices Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 09/21] RDMA/mlx5: Set ib_device::adversarial_mode from mlx5_core Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 10/21] RDMA/mlx5: Remove duplicate mlx5_ib_umr_context struct Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 11/21] RDMA/mlx5: Add shadow FIFO infrastructure for UMR completions Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 12/21] RDMA/mlx5: Verify UMR CQ polling against a hostile NIC Yishai Hadas
2026-11-18 17:13 ` sashiko-bot [this message]
2026-11-17 17:09 ` [PATCH V2 rdma-next 13/21] RDMA/mlx5: Keep the UMR QP failed once a forged CQE is detected Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 14/21] RDMA/core: Set the CQ comp_handler after its poll context is set up Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 15/21] RDMA/mlx5: Ignore completion events for a kernel CQ that is not ready Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 16/21] RDMA/mlx5: Validate EQE port and GSI port before indexing devr.ports[] Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 17/21] RDMA/mlx5: Validate firmware RQ/SQ state before sqrq_trans[] lookup Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 18/21] RDMA/mlx5: Clamp log_max_cq_sz shift in CQ size checks Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 19/21] RDMA/mlx5: Clamp log_max_* shifts in QP and WQ limit checks Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 20/21] RDMA/mlx5: Clamp log_max_* shifts in query_device and ucontext paths Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 21/21] RDMA/core: Report and enforce adversarial mode via GET_CONTEXT Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=sashiko-outbox-221613@kernel.org \
--to=sashiko-bot@kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=sashiko-reviews@lists.linux.dev \
--cc=yishaih@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox