From: Yishai Hadas <yishaih@nvidia.com>
To: <sashiko-reviews@lists.linux.dev>
Cc: <linux-rdma@vger.kernel.org>
Subject: Re: [PATCH V2 rdma-next 12/21] RDMA/mlx5: Verify UMR CQ polling against a hostile NIC
Date: Thu, 19 Nov 2026 12:17:40 +0200 [thread overview]
Message-ID: <86bf6a16-41af-4c4f-bec1-877ebac6aef4@nvidia.com> (raw)
In-Reply-To: <sashiko-outbox-221613@kernel.org>
On 18/11/2026 19:13, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 3 potential
> issues to consider:
> - [High] RDMA/mlx5: incomplete umrc->state checks in
> mlx5_ib_poll_one_umr()
> - [High] RDMA/mlx5: deadlock and lost flush on forged CQE in
> mlx5_ib_poll_one_umr()
> - [Medium] RDMA/mlx5: CQ re-arm loop in mlx5_ib_arm_cq() when umr_verify
> is false
> --
>
> commit d8d114091f7d4e4fa6dc6263146eaffadede67be
> Author: Yishai Hadas <yishaih@nvidia.com>
>
> RDMA/mlx5: Verify UMR CQ polling against a hostile NIC
>
> This commit adds a dedicated UMR CQ poller for adversarial mode that
> validates CQEs against a shadow FIFO of posted UMR requests and triggers
> forced health recovery if a forged CQE is detected.
>
>> diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
>> index 2581a7693585b..a5197318d96f8 100644
>> --- a/drivers/infiniband/hw/mlx5/cq.c
>> +++ b/drivers/infiniband/hw/mlx5/cq.c
>
> [ ... ]
>
>> @@ -608,6 +609,190 @@ static int poll_soft_wc(struct mlx5_ib_cq *cq, int num_entries,
>> +static int mlx5_ib_poll_one_umr(struct mlx5_ib_cq *cq, struct ib_wc *wc)
>> +{
>
> [ ... ]
>
>> + if (opcode == MLX5_CQE_REQ) {
>> + wc->status = IB_WC_SUCCESS;
>> + } else {
>> + /* Copy the CQE, so its syndrome is read once */
>> + err_cqe = *(struct mlx5_err_cqe *)cqe64;
>> + mlx5_handle_error_cqe(dev, &err_cqe, wc);
>> + /*
>> + * A real failure during a recovery is forged, the QP
>> + * only flushes.
>> + */
>> + if (wc->status != IB_WC_WR_FLUSH_ERR &&
>> + cmpxchg(&umrc->state, MLX5_UMR_STATE_ACTIVE,
>> + MLX5_UMR_STATE_RECOVER) != MLX5_UMR_STATE_ACTIVE) {
>> + mlx5_ib_err(dev,
>> + "UMR error CQE status %d in state %d, possible NIC forgery\n",
>> + wc->status, READ_ONCE(umrc->state));
>> + goto forged_cqe;
>> + }
>> + }
>
> [Severity: High]
> Should mlx5_ib_poll_one_umr() also verify umrc->state for IB_WC_WR_FLUSH_ERR
> and MLX5_CQE_REQ completions?
>
> When wc->status is IB_WC_WR_FLUSH_ERR, umrc->state is not checked. If a
> hostile NIC reports MLX5_CQE_SYNDROME_WR_FLUSH_ERR while umrc->state is
> MLX5_UMR_STATE_ACTIVE, mlx5_ib_poll_one_umr() accepts the CQE without
> changing umrc->state. The waiter in mlx5r_umr_post_send_wait() then sees
> IB_WC_WR_FLUSH_ERR, loops around while umrc->state is still
> MLX5_UMR_STATE_ACTIVE, and immediately reposts the WQE without sleeping or
> running recovery, allowing repeated flush CQEs to trap the thread in an
> unkillable infinite loop.
Each repost waits in wait_for_completion() for its CQE, so the loop runs
only as fast as the NIC returns flush CQEs. A hostile NIC can cause the
same load by flooding any EQ with events, so this is a DoS, not a CPU
spin. It can also hang the waiter by not returning a CQE at all. This
is out of scope for this series.
>
> Conversely, once a non-flush error has moved the QP to error and umrc->state
> to MLX5_UMR_STATE_RECOVER, subsequent in-flight WQEs can only flush until
> mlx5r_umr_recover() resets the QP, but the MLX5_CQE_REQ branch accepts
> completions as IB_WC_SUCCESS without checking that umrc->state is
> MLX5_UMR_STATE_ACTIVE.
A NIC can report a false success in any state, so checking the state
doesn't add protection.
> [Severity: High]
> Can deferring waiter completion to mlx5_trigger_health_work_force() deadlock
> on dev->intf_state_mutex or lose the software flush completion?
The deadlock needs a forged CQE while a UMR is pending during unload, a
rare timing that the NIC doesn't fully control. The same deadlock can
happen today, when the firmware hits a fatal error while a UMR is
pending during unload, as the fatal report work also takes intf_state_mutex.
The software flush, which completes the pending UMR WRs once the device
is in error, can also be lost in rare cases, e.g. during remove, where
the health work is no longer queued. This can happen today as well,
when the firmware hits a fatal error during remove.
In all these cases the outcome is a hung UMR waiter, which a hostile
NIC can cause anyway by not returning a CQE at all. This is out of
scope for this series.
> [Severity: Medium]
> Can returning 0 when cq->umr_verify is false cause an interrupt and softirq
> re-arm storm via mlx5_ib_arm_cq()?
>
Only a hostile NIC can leave a CQE there, and it's short lived:
- After a forged CQE, the forced health recovery tears down the device.
- A CQE written during UMR init stays in the CQ only until umr_verify is
set, once the UMR QP is ready. The next poll then checks it against
the posted UMR WRs, as for any CQE.
- After polling stops, the CQ is destroyed right after.
Again, this is a DoS, which is out of scope for this series.
Yishai
next prev parent reply other threads:[~2026-11-19 10:18 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-17 17:08 [PATCH V2 rdma-next 00/21] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
2026-11-17 17:08 ` [PATCH V2 rdma-next 01/21] net/mlx5: Add mlx5_trigger_health_work_force() Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-19 8:06 ` Yishai Hadas
2026-11-17 17:08 ` [PATCH V2 rdma-next 02/21] net/mlx5: Derive adversarial mode from CC guest memory encryption Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 03/21] net/mlx5: Add capability filtering in adversarial mode Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 04/21] net/mlx5: Catch too low max number of EQs Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-19 7:47 ` Yishai Hadas
2026-11-17 17:08 ` [PATCH V2 rdma-next 05/21] net/mlx5: Add macros for reading log-based caps Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:08 ` [PATCH V2 rdma-next 06/21] RDMA/mlx5: Reject QP transport types not enabled by FW capability Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 07/21] RDMA/mlx5: Require IB_QPT_DRIVER for the DC vendor QP types Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 08/21] RDMA/core: Suppress kverbs_provider for adversarial_mode devices Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 09/21] RDMA/mlx5: Set ib_device::adversarial_mode from mlx5_core Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 10/21] RDMA/mlx5: Remove duplicate mlx5_ib_umr_context struct Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 11/21] RDMA/mlx5: Add shadow FIFO infrastructure for UMR completions Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 12/21] RDMA/mlx5: Verify UMR CQ polling against a hostile NIC Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-19 10:17 ` Yishai Hadas [this message]
2026-11-17 17:09 ` [PATCH V2 rdma-next 13/21] RDMA/mlx5: Keep the UMR QP failed once a forged CQE is detected Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 14/21] RDMA/core: Set the CQ comp_handler after its poll context is set up Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 15/21] RDMA/mlx5: Ignore completion events for a kernel CQ that is not ready Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 16/21] RDMA/mlx5: Validate EQE port and GSI port before indexing devr.ports[] Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 17/21] RDMA/mlx5: Validate firmware RQ/SQ state before sqrq_trans[] lookup Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-19 10:47 ` Yishai Hadas
2026-11-17 17:09 ` [PATCH V2 rdma-next 18/21] RDMA/mlx5: Clamp log_max_cq_sz shift in CQ size checks Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 19/21] RDMA/mlx5: Clamp log_max_* shifts in QP and WQ limit checks Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 20/21] RDMA/mlx5: Clamp log_max_* shifts in query_device and ucontext paths Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
2026-11-17 17:09 ` [PATCH V2 rdma-next 21/21] RDMA/core: Report and enforce adversarial mode via GET_CONTEXT Yishai Hadas
2026-11-18 17:13 ` sashiko-bot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=86bf6a16-41af-4c4f-bec1-877ebac6aef4@nvidia.com \
--to=yishaih@nvidia.com \
--cc=linux-rdma@vger.kernel.org \
--cc=sashiko-reviews@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox