Linux RDMA and InfiniBand development
 help / color / mirror / Atom feed
From: sashiko-bot@kernel.org
To: "Yishai Hadas" <yishaih@nvidia.com>
Cc: linux-rdma@vger.kernel.org
Subject: Re: [PATCH V2 rdma-next 12/21] RDMA/mlx5: Verify UMR CQ polling against a hostile NIC
Date: Wed, 18 Nov 2026 17:13:48 +0000	[thread overview]
Message-ID: <sashiko-outbox-221613@kernel.org> (raw)
In-Reply-To: <20261117170914.27410-13-yishaih@nvidia.com>

Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [High] RDMA/mlx5: incomplete umrc->state checks in
  mlx5_ib_poll_one_umr()
- [High] RDMA/mlx5: deadlock and lost flush on forged CQE in
  mlx5_ib_poll_one_umr()
- [Medium] RDMA/mlx5: CQ re-arm loop in mlx5_ib_arm_cq() when umr_verify
  is false
--

commit d8d114091f7d4e4fa6dc6263146eaffadede67be
Author: Yishai Hadas <yishaih@nvidia.com>

RDMA/mlx5: Verify UMR CQ polling against a hostile NIC

This commit adds a dedicated UMR CQ poller for adversarial mode that
validates CQEs against a shadow FIFO of posted UMR requests and triggers
forced health recovery if a forged CQE is detected.

> diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
> index 2581a7693585b..a5197318d96f8 100644
> --- a/drivers/infiniband/hw/mlx5/cq.c
> +++ b/drivers/infiniband/hw/mlx5/cq.c

[ ... ]

> @@ -608,6 +609,190 @@ static int poll_soft_wc(struct mlx5_ib_cq *cq, int num_entries,
> +static int mlx5_ib_poll_one_umr(struct mlx5_ib_cq *cq, struct ib_wc *wc)
> +{

[ ... ]

> +	if (opcode == MLX5_CQE_REQ) {
> +		wc->status = IB_WC_SUCCESS;
> +	} else {
> +		/* Copy the CQE, so its syndrome is read once */
> +		err_cqe = *(struct mlx5_err_cqe *)cqe64;
> +		mlx5_handle_error_cqe(dev, &err_cqe, wc);
> +		/*
> +		 * A real failure during a recovery is forged, the QP
> +		 * only flushes.
> +		 */
> +		if (wc->status != IB_WC_WR_FLUSH_ERR &&
> +		    cmpxchg(&umrc->state, MLX5_UMR_STATE_ACTIVE,
> +			    MLX5_UMR_STATE_RECOVER) != MLX5_UMR_STATE_ACTIVE) {
> +			mlx5_ib_err(dev,
> +				    "UMR error CQE status %d in state %d, possible NIC forgery\n",
> +				    wc->status, READ_ONCE(umrc->state));
> +			goto forged_cqe;
> +		}
> +	}

[Severity: High]
Should mlx5_ib_poll_one_umr() also verify umrc->state for IB_WC_WR_FLUSH_ERR
and MLX5_CQE_REQ completions?

When wc->status is IB_WC_WR_FLUSH_ERR, umrc->state is not checked.  If a
hostile NIC reports MLX5_CQE_SYNDROME_WR_FLUSH_ERR while umrc->state is
MLX5_UMR_STATE_ACTIVE, mlx5_ib_poll_one_umr() accepts the CQE without
changing umrc->state.  The waiter in mlx5r_umr_post_send_wait() then sees
IB_WC_WR_FLUSH_ERR, loops around while umrc->state is still
MLX5_UMR_STATE_ACTIVE, and immediately reposts the WQE without sleeping or
running recovery, allowing repeated flush CQEs to trap the thread in an
unkillable infinite loop.

Conversely, once a non-flush error has moved the QP to error and umrc->state
to MLX5_UMR_STATE_RECOVER, subsequent in-flight WQEs can only flush until
mlx5r_umr_recover() resets the QP, but the MLX5_CQE_REQ branch accepts
completions as IB_WC_SUCCESS without checking that umrc->state is
MLX5_UMR_STATE_ACTIVE.

> +
> +	/* Pop the shadow FIFO head: validated, dispatch to its waiter */
> +	WRITE_ONCE(umrc->pending[head & (MLX5_UMR_MAX_WR - 1)], NULL);
> +	umrc->pending_head = head + 1;
> +	wc->wr_cqe = &expected->cqe;
> +
> +	wc->qp = &qp->ibqp;
> +	wq->tail = wq->wqe_head[idx] + 1;
> +	/* Advance wq->last_poll, the internal error flush starts from it */
> +	handle_atomics(qp, cqe64, wq->last_poll, idx);
> +
> +	return 0;
> +
> +forged_cqe:
> +	/* mlx5_ib_poll_cq() updates the CI only for a returned completion */
> +	mlx5_cq_set_ci(&cq->mcq);
> +	/*
> +	 * The NIC forged a CQE. Fail the UMR path for good, ignore any later
> +	 * CQE and force the health recovery, as the fatal sensors still read
> +	 * clean. The internal error flush then completes the waiters. Return
> +	 * an error, not -EAGAIN, as a CQE was consumed.
> +	 */
> +	umrc->state = MLX5_UMR_STATE_ERR;
> +	cq->umr_verify = false;
> +	mlx5_trigger_health_work_force(dev->mdev);
> +	return -EIO;
> +}

[Severity: High]
Can deferring waiter completion to mlx5_trigger_health_work_force() deadlock
on dev->intf_state_mutex or lose the software flush completion?

During driver teardown, mlx5_uninit_one() and mlx5_unload_one_devl_locked()
hold dev->intf_state_mutex while unregistering mlx5_ib, which deregisters
remaining user MRs via mlx5_ib_dereg_mr() -> mlx5r_umr_revoke_mr() ->
mlx5r_umr_post_send_wait().  If the NIC returns a forged CQE at that point,
stopping CQ polling without completing umrc->pending deadlocks against
force_fatal_report_work:

CPU0                                  CPU1 (health->wq)
----                                  -----------------
mlx5_unload_one_devl_locked()
  mutex_lock(&dev->intf_state_mutex)
  ...
  mlx5r_umr_post_send_wait()
    wait_for_completion(&umr_context.done)
                                      <softirq>
                                      mlx5_ib_poll_one_umr()
                                        mlx5_trigger_health_work_force()

                                      mlx5_fw_fatal_reporter_err()
                                        mutex_lock(&dev->intf_state_mutex)

 *** DEADLOCK ***

In mlx5_uninit_one(), mlx5_unregister_device() has also already set
MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV, so mlx5_dev_is_lightweight() is true and
mlx5_trigger_health_work_force() drops the work item outright.

Even during normal operation, the UMR CQ uses IB_POLL_SOFTIRQ, where
IRQ_POLL_F_SCHED stays set on cq->iop until ib_poll_handler() calls
irq_poll_complete() after __ib_process_cq() returns.  If
mlx5_ib_handle_internal_error() runs on another CPU after mlx5_ib_poll_cq()
checks mdev->state and before ib_poll_handler() calls irq_poll_complete(),
its irq_poll_sched(&cq->iop) call is a no-op, and mlx5_ib_arm_cq() then
returns 0 without rescheduling cq->iop, leaving waiters hung in
wait_for_completion().

> +static int mlx5_ib_poll_cq_adversarial(struct mlx5_ib_cq *cq, int num_entries,
> +				       struct ib_wc *wc)
> +{
> +	int npolled;
> +
> +	if (!cq->umr_verify)
> +		return 0;
> +
> +	for (npolled = 0; npolled < num_entries; npolled++) {
> +		if (mlx5_ib_poll_one_umr(cq, wc + npolled))
> +			break;
> +	}
> +
> +	return npolled;
> +}

[Severity: Medium]
Can returning 0 when cq->umr_verify is false cause an interrupt and softirq
re-arm storm via mlx5_ib_arm_cq()?

When forged_cqe sets cq->umr_verify = false, it consumes only that single
CQE.  If the NIC placed another CQE behind it in the UMR CQ (or if a CQE
arrives before umr_verify is enabled or after mlx5_ib_cq_stop_polling()
clears it), mlx5_ib_poll_cq_adversarial() returns 0 without advancing
cq->mcq.cons_index.  Because completed < budget, ib_poll_handler() then
calls ib_req_notify_cq() -> mlx5_ib_arm_cq(), which rings the CQ arm
doorbell at the unadvanced cons_index and causes the NIC to fire another
completion EQE immediately in a tight loop.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261117170914.27410-1-yishaih@nvidia.com?part=12

  reply	other threads:[~2026-11-18 17:13 UTC|newest]

Thread overview: 43+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-17 17:08 [PATCH V2 rdma-next 00/21] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
2026-11-17 17:08 ` [PATCH V2 rdma-next 01/21] net/mlx5: Add mlx5_trigger_health_work_force() Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 02/21] net/mlx5: Derive adversarial mode from CC guest memory encryption Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 03/21] net/mlx5: Add capability filtering in adversarial mode Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 04/21] net/mlx5: Catch too low max number of EQs Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 05/21] net/mlx5: Add macros for reading log-based caps Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 06/21] RDMA/mlx5: Reject QP transport types not enabled by FW capability Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 07/21] RDMA/mlx5: Require IB_QPT_DRIVER for the DC vendor QP types Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 08/21] RDMA/core: Suppress kverbs_provider for adversarial_mode devices Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 09/21] RDMA/mlx5: Set ib_device::adversarial_mode from mlx5_core Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 10/21] RDMA/mlx5: Remove duplicate mlx5_ib_umr_context struct Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 11/21] RDMA/mlx5: Add shadow FIFO infrastructure for UMR completions Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 12/21] RDMA/mlx5: Verify UMR CQ polling against a hostile NIC Yishai Hadas
2026-11-18 17:13   ` sashiko-bot [this message]
2026-11-17 17:09 ` [PATCH V2 rdma-next 13/21] RDMA/mlx5: Keep the UMR QP failed once a forged CQE is detected Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 14/21] RDMA/core: Set the CQ comp_handler after its poll context is set up Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 15/21] RDMA/mlx5: Ignore completion events for a kernel CQ that is not ready Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 16/21] RDMA/mlx5: Validate EQE port and GSI port before indexing devr.ports[] Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 17/21] RDMA/mlx5: Validate firmware RQ/SQ state before sqrq_trans[] lookup Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 18/21] RDMA/mlx5: Clamp log_max_cq_sz shift in CQ size checks Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 19/21] RDMA/mlx5: Clamp log_max_* shifts in QP and WQ limit checks Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 20/21] RDMA/mlx5: Clamp log_max_* shifts in query_device and ucontext paths Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 21/21] RDMA/core: Report and enforce adversarial mode via GET_CONTEXT Yishai Hadas
2026-11-18 17:13   ` sashiko-bot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=sashiko-outbox-221613@kernel.org \
    --to=sashiko-bot@kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=sashiko-reviews@lists.linux.dev \
    --cc=yishaih@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox