All of lore.kernel.org
 help / color / mirror / Atom feed
From: sashiko-bot@kernel.org
To: "Yishai Hadas" <yishaih@nvidia.com>
Cc: linux-rdma@vger.kernel.org
Subject: Re: [PATCH V2 rdma-next 12/21] RDMA/mlx5: Verify UMR CQ polling against a hostile NIC
Date: Wed, 18 Nov 2026 17:13:48 +0000	[thread overview]
Message-ID: <sashiko-outbox-221613@kernel.org> (raw)
In-Reply-To: <20261117170914.27410-13-yishaih@nvidia.com>

Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [High] RDMA/mlx5: incomplete umrc->state checks in
  mlx5_ib_poll_one_umr()
- [High] RDMA/mlx5: deadlock and lost flush on forged CQE in
  mlx5_ib_poll_one_umr()
- [Medium] RDMA/mlx5: CQ re-arm loop in mlx5_ib_arm_cq() when umr_verify
  is false
--

commit d8d114091f7d4e4fa6dc6263146eaffadede67be
Author: Yishai Hadas <yishaih@nvidia.com>

RDMA/mlx5: Verify UMR CQ polling against a hostile NIC

This commit adds a dedicated UMR CQ poller for adversarial mode that
validates CQEs against a shadow FIFO of posted UMR requests and triggers
forced health recovery if a forged CQE is detected.

> diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
> index 2581a7693585b..a5197318d96f8 100644
> --- a/drivers/infiniband/hw/mlx5/cq.c
> +++ b/drivers/infiniband/hw/mlx5/cq.c

[ ... ]

> @@ -608,6 +609,190 @@ static int poll_soft_wc(struct mlx5_ib_cq *cq, int num_entries,
> +static int mlx5_ib_poll_one_umr(struct mlx5_ib_cq *cq, struct ib_wc *wc)
> +{

[ ... ]

> +	if (opcode == MLX5_CQE_REQ) {
> +		wc->status = IB_WC_SUCCESS;
> +	} else {
> +		/* Copy the CQE, so its syndrome is read once */
> +		err_cqe = *(struct mlx5_err_cqe *)cqe64;
> +		mlx5_handle_error_cqe(dev, &err_cqe, wc);
> +		/*
> +		 * A real failure during a recovery is forged, the QP
> +		 * only flushes.
> +		 */
> +		if (wc->status != IB_WC_WR_FLUSH_ERR &&
> +		    cmpxchg(&umrc->state, MLX5_UMR_STATE_ACTIVE,
> +			    MLX5_UMR_STATE_RECOVER) != MLX5_UMR_STATE_ACTIVE) {
> +			mlx5_ib_err(dev,
> +				    "UMR error CQE status %d in state %d, possible NIC forgery\n",
> +				    wc->status, READ_ONCE(umrc->state));
> +			goto forged_cqe;
> +		}
> +	}

[Severity: High]
Should mlx5_ib_poll_one_umr() also verify umrc->state for IB_WC_WR_FLUSH_ERR
and MLX5_CQE_REQ completions?

When wc->status is IB_WC_WR_FLUSH_ERR, umrc->state is not checked.  If a
hostile NIC reports MLX5_CQE_SYNDROME_WR_FLUSH_ERR while umrc->state is
MLX5_UMR_STATE_ACTIVE, mlx5_ib_poll_one_umr() accepts the CQE without
changing umrc->state.  The waiter in mlx5r_umr_post_send_wait() then sees
IB_WC_WR_FLUSH_ERR, loops around while umrc->state is still
MLX5_UMR_STATE_ACTIVE, and immediately reposts the WQE without sleeping or
running recovery, allowing repeated flush CQEs to trap the thread in an
unkillable infinite loop.

Conversely, once a non-flush error has moved the QP to error and umrc->state
to MLX5_UMR_STATE_RECOVER, subsequent in-flight WQEs can only flush until
mlx5r_umr_recover() resets the QP, but the MLX5_CQE_REQ branch accepts
completions as IB_WC_SUCCESS without checking that umrc->state is
MLX5_UMR_STATE_ACTIVE.

> +
> +	/* Pop the shadow FIFO head: validated, dispatch to its waiter */
> +	WRITE_ONCE(umrc->pending[head & (MLX5_UMR_MAX_WR - 1)], NULL);
> +	umrc->pending_head = head + 1;
> +	wc->wr_cqe = &expected->cqe;
> +
> +	wc->qp = &qp->ibqp;
> +	wq->tail = wq->wqe_head[idx] + 1;
> +	/* Advance wq->last_poll, the internal error flush starts from it */
> +	handle_atomics(qp, cqe64, wq->last_poll, idx);
> +
> +	return 0;
> +
> +forged_cqe:
> +	/* mlx5_ib_poll_cq() updates the CI only for a returned completion */
> +	mlx5_cq_set_ci(&cq->mcq);
> +	/*
> +	 * The NIC forged a CQE. Fail the UMR path for good, ignore any later
> +	 * CQE and force the health recovery, as the fatal sensors still read
> +	 * clean. The internal error flush then completes the waiters. Return
> +	 * an error, not -EAGAIN, as a CQE was consumed.
> +	 */
> +	umrc->state = MLX5_UMR_STATE_ERR;
> +	cq->umr_verify = false;
> +	mlx5_trigger_health_work_force(dev->mdev);
> +	return -EIO;
> +}

[Severity: High]
Can deferring waiter completion to mlx5_trigger_health_work_force() deadlock
on dev->intf_state_mutex or lose the software flush completion?

During driver teardown, mlx5_uninit_one() and mlx5_unload_one_devl_locked()
hold dev->intf_state_mutex while unregistering mlx5_ib, which deregisters
remaining user MRs via mlx5_ib_dereg_mr() -> mlx5r_umr_revoke_mr() ->
mlx5r_umr_post_send_wait().  If the NIC returns a forged CQE at that point,
stopping CQ polling without completing umrc->pending deadlocks against
force_fatal_report_work:

CPU0                                  CPU1 (health->wq)
----                                  -----------------
mlx5_unload_one_devl_locked()
  mutex_lock(&dev->intf_state_mutex)
  ...
  mlx5r_umr_post_send_wait()
    wait_for_completion(&umr_context.done)
                                      <softirq>
                                      mlx5_ib_poll_one_umr()
                                        mlx5_trigger_health_work_force()

                                      mlx5_fw_fatal_reporter_err()
                                        mutex_lock(&dev->intf_state_mutex)

 *** DEADLOCK ***

In mlx5_uninit_one(), mlx5_unregister_device() has also already set
MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV, so mlx5_dev_is_lightweight() is true and
mlx5_trigger_health_work_force() drops the work item outright.

Even during normal operation, the UMR CQ uses IB_POLL_SOFTIRQ, where
IRQ_POLL_F_SCHED stays set on cq->iop until ib_poll_handler() calls
irq_poll_complete() after __ib_process_cq() returns.  If
mlx5_ib_handle_internal_error() runs on another CPU after mlx5_ib_poll_cq()
checks mdev->state and before ib_poll_handler() calls irq_poll_complete(),
its irq_poll_sched(&cq->iop) call is a no-op, and mlx5_ib_arm_cq() then
returns 0 without rescheduling cq->iop, leaving waiters hung in
wait_for_completion().

> +static int mlx5_ib_poll_cq_adversarial(struct mlx5_ib_cq *cq, int num_entries,
> +				       struct ib_wc *wc)
> +{
> +	int npolled;
> +
> +	if (!cq->umr_verify)
> +		return 0;
> +
> +	for (npolled = 0; npolled < num_entries; npolled++) {
> +		if (mlx5_ib_poll_one_umr(cq, wc + npolled))
> +			break;
> +	}
> +
> +	return npolled;
> +}

[Severity: Medium]
Can returning 0 when cq->umr_verify is false cause an interrupt and softirq
re-arm storm via mlx5_ib_arm_cq()?

When forged_cqe sets cq->umr_verify = false, it consumes only that single
CQE.  If the NIC placed another CQE behind it in the UMR CQ (or if a CQE
arrives before umr_verify is enabled or after mlx5_ib_cq_stop_polling()
clears it), mlx5_ib_poll_cq_adversarial() returns 0 without advancing
cq->mcq.cons_index.  Because completed < budget, ib_poll_handler() then
calls ib_req_notify_cq() -> mlx5_ib_arm_cq(), which rings the CQ arm
doorbell at the unadvanced cons_index and causes the NIC to fire another
completion EQE immediately in a tight loop.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261117170914.27410-1-yishaih@nvidia.com?part=12

  reply	other threads:[~2026-11-18 17:13 UTC|newest]

Thread overview: 47+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-17 17:08 [PATCH V2 rdma-next 00/21] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
2026-11-17 17:08 ` [PATCH V2 rdma-next 01/21] net/mlx5: Add mlx5_trigger_health_work_force() Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-19  8:06     ` Yishai Hadas
2026-11-17 17:08 ` [PATCH V2 rdma-next 02/21] net/mlx5: Derive adversarial mode from CC guest memory encryption Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 03/21] net/mlx5: Add capability filtering in adversarial mode Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 04/21] net/mlx5: Catch too low max number of EQs Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-19  7:47     ` Yishai Hadas
2026-11-17 17:08 ` [PATCH V2 rdma-next 05/21] net/mlx5: Add macros for reading log-based caps Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 06/21] RDMA/mlx5: Reject QP transport types not enabled by FW capability Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 07/21] RDMA/mlx5: Require IB_QPT_DRIVER for the DC vendor QP types Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 08/21] RDMA/core: Suppress kverbs_provider for adversarial_mode devices Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 09/21] RDMA/mlx5: Set ib_device::adversarial_mode from mlx5_core Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 10/21] RDMA/mlx5: Remove duplicate mlx5_ib_umr_context struct Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 11/21] RDMA/mlx5: Add shadow FIFO infrastructure for UMR completions Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 12/21] RDMA/mlx5: Verify UMR CQ polling against a hostile NIC Yishai Hadas
2026-11-18 17:13   ` sashiko-bot [this message]
2026-11-19 10:17     ` Yishai Hadas
2026-11-17 17:09 ` [PATCH V2 rdma-next 13/21] RDMA/mlx5: Keep the UMR QP failed once a forged CQE is detected Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 14/21] RDMA/core: Set the CQ comp_handler after its poll context is set up Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 15/21] RDMA/mlx5: Ignore completion events for a kernel CQ that is not ready Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 16/21] RDMA/mlx5: Validate EQE port and GSI port before indexing devr.ports[] Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 17/21] RDMA/mlx5: Validate firmware RQ/SQ state before sqrq_trans[] lookup Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-19 10:47     ` Yishai Hadas
2026-11-17 17:09 ` [PATCH V2 rdma-next 18/21] RDMA/mlx5: Clamp log_max_cq_sz shift in CQ size checks Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 19/21] RDMA/mlx5: Clamp log_max_* shifts in QP and WQ limit checks Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 20/21] RDMA/mlx5: Clamp log_max_* shifts in query_device and ucontext paths Yishai Hadas
2026-11-18 17:13   ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 21/21] RDMA/core: Report and enforce adversarial mode via GET_CONTEXT Yishai Hadas
2026-11-18 17:13   ` sashiko-bot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=sashiko-outbox-221613@kernel.org \
    --to=sashiko-bot@kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=sashiko-reviews@lists.linux.dev \
    --cc=yishaih@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.