* [PATCH rdma-next 01/23] net/mlx5: Add mlx5_trigger_health_work_force()
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:24 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 02/23] net/mlx5: Derive adversarial mode from CC guest memory encryption Yishai Hadas
` (21 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
mlx5_trigger_health_work() queues the health recovery, which enters
MLX5_DEVICE_STATE_INTERNAL_ERROR only if
mlx5_health_check_fatal_sensors() reports a fault.
A consumer can detect a fault the sensors cannot see, such as a device
that keeps answering while misbehaving. It then needs the recovery to
run with the error state forced, whatever the sensors say.
Add mlx5_trigger_health_work_force(), which queues a dedicated work that
runs the recovery with the error state forced. It only queues the work,
so it can be called from atomic context.
Before entering the error state the forced work stops the health poller
and asks the firmware to stop with mlx5_cmd_fast_teardown_hca(), as the
device might still be running. The error state is forced whether or not
the teardown succeeds. The teardown runs under intf_state_mutex and only
while the interface is up, so it cannot race an unload.
Export it for mlx5_ib, which will use it in a later patch of this series
when it detects a NIC forging completion queue entries, a fault the
health sensors cannot see.
Reviewed-by: Moshe Shemesh <moshe@nvidia.com>
Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
.../net/ethernet/mellanox/mlx5/core/health.c | 82 ++++++++++++++++---
include/linux/mlx5/driver.h | 2 +
2 files changed, 73 insertions(+), 11 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c
index aeeb136f5ebc..d000ac335fa7 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/health.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c
@@ -627,18 +627,11 @@ mlx5_fw_fatal_reporter_dump(struct devlink_health_reporter *reporter,
return err;
}
-static void mlx5_fw_fatal_reporter_err_work(struct work_struct *work)
+static void mlx5_fw_fatal_reporter_err(struct mlx5_core_dev *dev, bool forced)
{
struct mlx5_fw_reporter_ctx fw_reporter_ctx;
- struct mlx5_core_health *health;
- struct mlx5_core_dev *dev;
- struct devlink *devlink;
- struct mlx5_priv *priv;
-
- health = container_of(work, struct mlx5_core_health, fatal_report_work);
- priv = container_of(health, struct mlx5_priv, health);
- dev = container_of(priv, struct mlx5_core_dev, priv);
- devlink = priv_to_devlink(dev);
+ struct mlx5_core_health *health = &dev->priv.health;
+ struct devlink *devlink = priv_to_devlink(dev);
mutex_lock(&dev->intf_state_mutex);
if (test_bit(MLX5_DROP_HEALTH_WORK, &health->flags)) {
@@ -646,8 +639,26 @@ static void mlx5_fw_fatal_reporter_err_work(struct work_struct *work)
mutex_unlock(&dev->intf_state_mutex);
return;
}
+ /*
+ * The firmware might still be running, so stop it first. Do it under
+ * intf_state_mutex and only while UP, so the cmd interface is valid.
+ */
+ if (forced && test_bit(MLX5_INTERFACE_STATE_UP, &dev->intf_state)) {
+ int err;
+
+ mlx5_stop_health_poll(dev, false);
+ err = mlx5_cmd_fast_teardown_hca(dev);
+ if (err)
+ mlx5_core_err(dev, "Failed to stop the device before forcing the error state, err %d\n",
+ err);
+ }
+
+ /*
+ * Enter the error state before unlocking, or an unload could send
+ * TEARDOWN_HCA to the stopped NIC, fail and skip function disable.
+ */
+ enter_error_state(dev, forced);
mutex_unlock(&dev->intf_state_mutex);
- enter_error_state(dev, false);
if (IS_ERR_OR_NULL(health->fw_fatal_reporter)) {
devl_lock(devlink);
if (mlx5_health_try_recover(dev))
@@ -669,6 +680,33 @@ static void mlx5_fw_fatal_reporter_err_work(struct work_struct *work)
}
}
+static void mlx5_fw_fatal_reporter_err_work(struct work_struct *work)
+{
+ struct mlx5_core_health *health;
+ struct mlx5_core_dev *dev;
+ struct mlx5_priv *priv;
+
+ health = container_of(work, struct mlx5_core_health, fatal_report_work);
+ priv = container_of(health, struct mlx5_priv, health);
+ dev = container_of(priv, struct mlx5_core_dev, priv);
+
+ mlx5_fw_fatal_reporter_err(dev, false);
+}
+
+static void mlx5_fw_fatal_reporter_force_err_work(struct work_struct *work)
+{
+ struct mlx5_core_health *health;
+ struct mlx5_core_dev *dev;
+ struct mlx5_priv *priv;
+
+ health = container_of(work, struct mlx5_core_health,
+ force_fatal_report_work);
+ priv = container_of(health, struct mlx5_priv, health);
+ dev = container_of(priv, struct mlx5_core_dev, priv);
+
+ mlx5_fw_fatal_reporter_err(dev, true);
+}
+
#define MLX5_FW_REPORTER_ECPF_GRACEFUL_PERIOD 180000
#define MLX5_FW_REPORTER_PF_GRACEFUL_PERIOD 60000
#define MLX5_FW_REPORTER_VF_GRACEFUL_PERIOD 30000
@@ -759,6 +797,25 @@ void mlx5_trigger_health_work(struct mlx5_core_dev *dev)
queue_work(health->wq, &health->fatal_report_work);
}
+/**
+ * mlx5_trigger_health_work_force - enter the error state and recover, even
+ * if the health sensors report the device as healthy
+ * @dev: mlx5 core device
+ *
+ * For faults only a consumer can detect, where the firmware keeps answering
+ * and mlx5_health_check_fatal_sensors() sees nothing wrong. Without forcing,
+ * the device would stay in MLX5_DEVICE_STATE_UP and consumers waiting on
+ * outstanding work would never be released.
+ */
+void mlx5_trigger_health_work_force(struct mlx5_core_dev *dev)
+{
+ struct mlx5_core_health *health = &dev->priv.health;
+
+ if (!mlx5_dev_is_lightweight(dev))
+ queue_work(health->wq, &health->force_fatal_report_work);
+}
+EXPORT_SYMBOL_GPL(mlx5_trigger_health_work_force);
+
#define MLX5_MSEC_PER_HOUR (MSEC_PER_SEC * 60 * 60)
static void mlx5_health_log_ts_update(struct work_struct *work)
{
@@ -874,6 +931,7 @@ void mlx5_drain_health_wq(struct mlx5_core_dev *dev)
cancel_delayed_work_sync(&health->update_fw_log_ts_work);
cancel_work_sync(&health->report_work);
cancel_work_sync(&health->fatal_report_work);
+ cancel_work_sync(&health->force_fatal_report_work);
}
void mlx5_health_cleanup(struct mlx5_core_dev *dev)
@@ -911,6 +969,8 @@ int mlx5_health_init(struct mlx5_core_dev *dev)
if (!health->wq)
goto out_err;
INIT_WORK(&health->fatal_report_work, mlx5_fw_fatal_reporter_err_work);
+ INIT_WORK(&health->force_fatal_report_work,
+ mlx5_fw_fatal_reporter_force_err_work);
INIT_WORK(&health->report_work, mlx5_fw_reporter_err_work);
INIT_DELAYED_WORK(&health->update_fw_log_ts_work, mlx5_health_log_ts_update);
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index abb99371ff13..a1275e6aa6c1 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -450,6 +450,7 @@ struct mlx5_core_health {
struct workqueue_struct *wq;
unsigned long flags;
struct work_struct fatal_report_work;
+ struct work_struct force_fatal_report_work;
struct work_struct report_work;
struct devlink_health_reporter *fw_reporter;
struct devlink_health_reporter *fw_fatal_reporter;
@@ -1034,6 +1035,7 @@ void mlx5_stop_health_poll(struct mlx5_core_dev *dev, bool disable_health);
void mlx5_start_health_fw_log_up(struct mlx5_core_dev *dev);
void mlx5_drain_health_wq(struct mlx5_core_dev *dev);
void mlx5_trigger_health_work(struct mlx5_core_dev *dev);
+void mlx5_trigger_health_work_force(struct mlx5_core_dev *dev);
int mlx5_frag_buf_alloc_node(struct mlx5_core_dev *dev, int size,
struct mlx5_frag_buf *buf, int node);
void mlx5_frag_buf_free(struct mlx5_core_dev *dev, struct mlx5_frag_buf *buf);
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 01/23] net/mlx5: Add mlx5_trigger_health_work_force()
2026-11-03 16:39 ` [PATCH rdma-next 01/23] net/mlx5: Add mlx5_trigger_health_work_force() Yishai Hadas
@ 2026-11-07 20:24 ` netdev-bot+sashiko
2026-11-08 16:17 ` Yishai Hadas
0 siblings, 1 reply; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:24 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found 6 potential
issue(s) to consider.
Critical: 0 · High: 2 · Medium: 2 · Low: 2
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [High] The forced path in mlx5_fw_fatal_reporter_err()
(drivers/net/ethernet/mellanox/mlx5/core/health.c) tries only…
- [High] net/mlx5: forced health work recovers device when interface is
already down
- [Medium] Nothing deduplicates force_fatal_report_work against
fatal_report_work or against the device state.
- [Medium] A forced request is dropped with no record or retry if
force_fatal_report_work runs while MLX5_DROP_HEALTH_WORK is set only…
- [Low] The patch changes the existing sensor-triggered recovery path
(fatal_report_work, forced=false) in mlx5_fw_fatal_reporter_err().
- [Low] A forced, consumer-detected fault goes through the same devlink
fw_fatal reporter path as a real firmware-fatal error.
--
net/mlx5: Add mlx5_trigger_health_work_force()
This adds mlx5_trigger_health_work_force(). It queues a new
force_fatal_report_work that runs the fatal health recovery with the error
state forced, whatever the health sensors report. Before forcing the error
state, the forced path stops the health poller and issues a fast teardown
of the HCA under intf_state_mutex. The function is exported for mlx5_ib,
which uses it later in the series.
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c
> index aeeb136f5ebc7..d000ac335fa7a 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c
> @@ -627,18 +627,11 @@ mlx5_fw_fatal_reporter_dump(struct devlink_health_reporter *reporter,
> return err;
> }
>
> -static void mlx5_fw_fatal_reporter_err_work(struct work_struct *work)
> +static void mlx5_fw_fatal_reporter_err(struct mlx5_core_dev *dev, bool forced)
> {
[ ... ]
> mutex_lock(&dev->intf_state_mutex);
> if (test_bit(MLX5_DROP_HEALTH_WORK, &health->flags)) {
> @@ -646,8 +639,26 @@ static void mlx5_fw_fatal_reporter_err_work(struct work_struct *work)
> mutex_unlock(&dev->intf_state_mutex);
> return;
> }
[Severity: Medium]
Can a forced request get lost here? If force_fatal_report_work runs while
MLX5_DROP_HEALTH_WORK is set only for a while, mlx5_fw_fatal_reporter_err()
logs "health works are not permitted at this stage" and returns. The
request is not recorded and not retried.
The FW sync reset flow opens such a window.
mlx5_sync_reset_set_reset_requested() sets the bit without holding
intf_state_mutex:
mlx5_stop_health_poll(dev, true);
On abort or timeout, mlx5_sync_reset_clear_reset_requested() clears it
again:
if (poll_health)
mlx5_start_health_poll(dev);
A dropped sensor fault is caught again by the next poll_health(). A forced
fault cannot be, because the sensors never see it. So dev->state stays
MLX5_DEVICE_STATE_UP. The kernel-doc of mlx5_trigger_health_work_force()
says this API exists to prevent exactly that.
In the mlx5_ib consumer later in this series, umrc->state is already
MLX5_UMR_STATE_ERR at this point, and cq->poll_one is already
mlx5_ib_poll_one_stopped. The UMR waiters in mlx5r_umr_post_send_wait()
depend on the internal error flush to wake them, and they wait with no
timeout:
wait_for_completion(&umr_context.done);
If the forced request is dropped this way, would those waiters hang
forever?
> + /*
> + * The firmware might still be running, so stop it first. Do it under
> + * intf_state_mutex and only while UP, so the cmd interface is valid.
> + */
> + if (forced && test_bit(MLX5_INTERFACE_STATE_UP, &dev->intf_state)) {
> + int err;
> +
> + mlx5_stop_health_poll(dev, false);
> + err = mlx5_cmd_fast_teardown_hca(dev);
> + if (err)
> + mlx5_core_err(dev, "Failed to stop the device before forcing the error state, err %d\n",
> + err);
> + }
[Severity: High]
What happens here when MLX5_INTERFACE_STATE_UP is already clear? The
forced branch skips mlx5_stop_health_poll() and
mlx5_cmd_fast_teardown_hca(), but the function does not bail out. It
still calls enter_error_state(dev, true) and then continues into
mlx5_health_try_recover() or devlink_health_report().
A non-remove unload leaves the device in exactly that state.
mlx5_unload_one_devl_locked() does:
clear_bit(MLX5_INTERFACE_STATE_UP, &dev->intf_state);
mlx5_unload(dev);
mlx5_function_teardown(dev, false);
mlx5_function_disable(dev, false) then calls
mlx5_stop_health_poll(dev, false), so MLX5_DROP_HEALTH_WORK is never
set. mlx5_suspend() and mlx5_devlink_reload_down() take this path.
mlx5_pci_err_detected() takes it too, before it drains the health wq.
None of them cancel force_fatal_report_work.
Suppose a consumer calls mlx5_trigger_health_work_force() just before
or during one of those unloads. The work runs once intf_state_mutex is
released. It finds MLX5_DROP_HEALTH_WORK clear and the interface down.
It then forces MLX5_DEVICE_STATE_INTERNAL_ERROR on a device whose cmd
interface mlx5_cmd_disable() has already shut down. The recovery then
runs:
mlx5_health_try_recover()
mlx5_handle_bad_state()
/* sensors clean, NIC never disabled by fast teardown */
mlx5_disable_device()
mlx5_error_sw_reset()
/* can poll up to PCI_TOGGLE timeout for IFC disabled */
mlx5_recover_device()
mlx5_pci_slot_reset()
mlx5_load_one_devl_locked(dev, true)
Does that reset and reload a device that was just suspended or
unloaded on purpose?
Devlink reload is affected as well. devl_lock is held from reload_down
to reload_up, so the work blocks on devl_lock, either directly or
inside devlink_health_report(), until reload_up has loaded the device
again. The work then forces the error state on the freshly reloaded,
healthy device and recovers it. If the report lands inside the
graceful period, it unloads the device with
mlx5_unload_one(dev, false) instead.
Should the forced path unlock and return when the interface is not UP,
instead of only skipping the teardown? Or should the non-remove unload
paths cancel force_fatal_report_work?
[Severity: High]
What happens here when mlx5_cmd_fast_teardown_hca() fails? It returns
-EOPNOTSUPP at once when MLX5_CAP_GEN(dev, fast_teardown) is 0. It can also
return -EIO, -EACCES or a command error. The failure is only logged, and
enter_error_state(dev, true) runs anyway. The HCA can then still be in
FULL_DRIVER mode with bus mastering enabled when dev->state becomes
MLX5_DEVICE_STATE_INTERNAL_ERROR.
>From that point, mlx5_internal_err_ret_value() fakes success for
TEARDOWN_HCA, DISABLE_HCA and the DESTROY_* commands. MANAGE_PAGES fails
with -ENXIO, so reclaim_pages_cmd() reclaims the firmware pages in
software:
/* No hard feelings, we want our pages back! */
The recovery then runs:
mlx5_health_try_recover()
mlx5_handle_bad_state()
/* only warns "Expected to see disabled NIC but it is full driver" */
mlx5_disable_device()
mlx5_error_sw_reset()
/* sensors are clean: no SW reset, only "NIC IFC still" logged */
mlx5_unload_one_devl_locked()
/* frees EQ/CQ/QP/RQ buffers and ICM pages */
mlx5_recover_device()
mlx5_pci_disable_device() /* skipped for SFs */
Can the still-running HCA DMA into the freed queue buffers and ICM pages
before bus mastering is cleared? On the -ECANCELED branch,
mlx5_unload_one(dev, false) frees everything and the device is never
disabled.
Compare mlx5_try_fast_unload(). It falls back to force teardown and
enters the error state only after a teardown has succeeded:
ret = mlx5_cmd_fast_teardown_hca(dev);
if (!ret)
goto succeed;
ret = mlx5_cmd_force_teardown_hca(dev);
if (!ret)
goto succeed;
...
mlx5_start_health_poll(dev);
return ret;
Does this path need a mlx5_cmd_force_teardown_hca() fallback, or bus
mastering cleared before the error state is forced? The commit message
says the work asks the firmware to stop "as the device might still be
running", but that does not happen when fast teardown is unsupported or
fails.
> +
> + /*
> + * Enter the error state before unlocking, or an unload could send
> + * TEARDOWN_HCA to the stopped NIC, fail and skip function disable.
> + */
> + enter_error_state(dev, forced);
> mutex_unlock(&dev->intf_state_mutex);
> - enter_error_state(dev, false);
[Severity: Low]
This also changes the existing sensor-triggered path (fatal_report_work,
forced == false). enter_error_state(dev, false) runs mlx5_cmd_flush() and
the MLX5_DEV_EVENT_SYS_ERROR notifier chain. It used to run after
mutex_unlock(&dev->intf_state_mutex). It now runs with the mutex held, for
every firmware fatal recovery.
mlx5_enter_error_state() already calls enter_error_state() under the same
mutex, so this doesn't look like a deadlock. However, the new comment and
the commit message explain the move only for the forced case.
Could the commit message say that the non-forced path changes too?
> if (IS_ERR_OR_NULL(health->fw_fatal_reporter)) {
> devl_lock(devlink);
> if (mlx5_health_try_recover(dev))
> @@ -669,6 +680,33 @@ static void mlx5_fw_fatal_reporter_err_work(struct work_struct *work)
> }
> }
>
[ ... ]
> +static void mlx5_fw_fatal_reporter_force_err_work(struct work_struct *work)
> +{
> + struct mlx5_core_health *health;
> + struct mlx5_core_dev *dev;
> + struct mlx5_priv *priv;
> +
> + health = container_of(work, struct mlx5_core_health,
> + force_fatal_report_work);
> + priv = container_of(health, struct mlx5_priv, health);
> + dev = container_of(priv, struct mlx5_core_dev, priv);
> +
> + mlx5_fw_fatal_reporter_err(dev, true);
> +}
[Severity: Low]
With forced == true, mlx5_fw_fatal_reporter_err() still reports through
the fw_fatal reporter as a firmware fatal error:
fw_reporter_ctx.err_synd = health->synd;
fw_reporter_ctx.miss_counter = health->miss_counter;
if (devlink_health_report(health->fw_fatal_reporter,
"FW fatal error reported", &fw_reporter_ctx) == -ECANCELED) {
health->synd and health->miss_counter are firmware sensor values that have
nothing to do with the forced fault, and health->fatal_error is not set.
Will devlink health output show a firmware fatal event with a stale or zero
syndrome, and nothing that points to the consumer that triggered it?
Each forced event also counts toward the fw_fatal reporter's graceful
period. A second forced trigger inside that period (60s on a PF) makes
devlink_health_report() return -ECANCELED, and the driver then calls
mlx5_unload_one(dev, false). Should the commit message describe this?
[ ... ]
> @@ -759,6 +797,25 @@ void mlx5_trigger_health_work(struct mlx5_core_dev *dev)
> queue_work(health->wq, &health->fatal_report_work);
> }
>
[ ... ]
> +void mlx5_trigger_health_work_force(struct mlx5_core_dev *dev)
> +{
> + struct mlx5_core_health *health = &dev->priv.health;
> +
> + if (!mlx5_dev_is_lightweight(dev))
> + queue_work(health->wq, &health->force_fatal_report_work);
> +}
[Severity: Medium]
Does anything deduplicate this against fatal_report_work, or check
dev->state? The two are separate work items on the ordered health->wq, so
queue_work() never merges them. The recovery path (mlx5_disable_device()
and mlx5_recover_device()) does not drain health->wq.
The forced branch in mlx5_fw_fatal_reporter_err() checks only
MLX5_DROP_HEALTH_WORK and MLX5_INTERFACE_STATE_UP. The peer helpers refuse
to run on a device that is already in error:
mlx5_enter_error_state():
if (!err_detected && dev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR)
goto unlock;/* a previous error is still being handled */
mlx5_try_fast_unload():
if (dev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR) {
mlx5_core_dbg(dev, "Device in internal error state, giving up\n");
return -EAGAIN;
}
Suppose poll_health() has already set MLX5_DEVICE_STATE_INTERNAL_ERROR
and queued fatal_report_work while the forced work is pending.
mlx5_internal_err_ret_value() fakes TEARDOWN_HCA as a success.
mlx5_cmd_fast_teardown_hca() then writes the iseg NIC state and polls for
up to the TEARDOWN timeout on a device in firmware fatal state. Later, the
queued fatal_report_work handles the same fault again on the device that
was just recovered.
The other order is also possible. A forced work queued while
fatal_report_work is recovering runs on the freshly loaded, healthy device.
It fast-tears it down and forces the error state.
In both orders, the second devlink_health_report() falls inside the
graceful period and returns -ECANCELED. mlx5_unload_one(dev, false) then
leaves the device unloaded instead of recovered. Is that intended?
The mlx5_ib consumer in this series switches cq->poll_one to
mlx5_ib_poll_one_stopped before it triggers, so a single UMR CQ cannot
trigger twice. A sensor fault and a forged CQE close together would still
be enough, for example when mlx5_ib_poll_cq() passes its mdev->state check
just before poll_health() sets the error state.
> +EXPORT_SYMBOL_GPL(mlx5_trigger_health_work_force);
[ ... ]
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 01/23] net/mlx5: Add mlx5_trigger_health_work_force()
2026-11-07 20:24 ` netdev-bot+sashiko
@ 2026-11-08 16:17 ` Yishai Hadas
0 siblings, 0 replies; 48+ messages in thread
From: Yishai Hadas @ 2026-11-08 16:17 UTC (permalink / raw)
To: netdev-bot+sashiko
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
On 07/11/2026 22:24, netdev-bot+sashiko@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 6 potential
> issue(s) to consider.
>
> Critical: 0 · High: 2 · Medium: 2 · Low: 2
>
> If you believe none of the issues are legitimate, please reply to each
> issue of Medium severity or higher and explain why the AI is wrong.
> If any of them are legitimate and you plan to respin, please let patchwork
> know by including "pw-bot: cr" as a separate line at the end of your reply
> (one such reply per series is enough).
>
> - [High] The forced path in mlx5_fw_fatal_reporter_err()
> (drivers/net/ethernet/mellanox/mlx5/core/health.c) tries only…
> - [High] net/mlx5: forced health work recovers device when interface is
> already down
> - [Medium] Nothing deduplicates force_fatal_report_work against
> fatal_report_work or against the device state.
> - [Medium] A forced request is dropped with no record or retry if
> force_fatal_report_work runs while MLX5_DROP_HEALTH_WORK is set only…
> - [Low] The patch changes the existing sensor-triggered recovery path
> (fatal_report_work, forced=false) in mlx5_fw_fatal_reporter_err().
> - [Low] A forced, consumer-detected fault goes through the same devlink
> fw_fatal reporter path as a real firmware-fatal error.
>
> --
>
> net/mlx5: Add mlx5_trigger_health_work_force()
>
> This adds mlx5_trigger_health_work_force(). It queues a new
> force_fatal_report_work that runs the fatal health recovery with the error
> state forced, whatever the health sensors report. Before forcing the error
> state, the forced path stops the health poller and issues a fast teardown
> of the HCA under intf_state_mutex. The function is exported for mlx5_ib,
> which uses it later in the series.
>
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c
>> index aeeb136f5ebc7..d000ac335fa7a 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c
>> @@ -627,18 +627,11 @@ mlx5_fw_fatal_reporter_dump(struct devlink_health_reporter *reporter,
>> return err;
>> }
>>
>> -static void mlx5_fw_fatal_reporter_err_work(struct work_struct *work)
>> +static void mlx5_fw_fatal_reporter_err(struct mlx5_core_dev *dev, bool forced)
>> {
>
> [ ... ]
>
>> mutex_lock(&dev->intf_state_mutex);
>> if (test_bit(MLX5_DROP_HEALTH_WORK, &health->flags)) {
>> @@ -646,8 +639,26 @@ static void mlx5_fw_fatal_reporter_err_work(struct work_struct *work)
>> mutex_unlock(&dev->intf_state_mutex);
>> return;
>> }
>
> [Severity: Medium]
> Can a forced request get lost here? If force_fatal_report_work runs while
> MLX5_DROP_HEALTH_WORK is set only for a while, mlx5_fw_fatal_reporter_err()
> logs "health works are not permitted at this stage" and returns. The
> request is not recorded and not retried.
>
> The FW sync reset flow opens such a window.
> mlx5_sync_reset_set_reset_requested() sets the bit without holding
> intf_state_mutex:
>
> mlx5_stop_health_poll(dev, true);
>
> On abort or timeout, mlx5_sync_reset_clear_reset_requested() clears it
> again:
>
> if (poll_health)
> mlx5_start_health_poll(dev);
>
> A dropped sensor fault is caught again by the next poll_health(). A forced
> fault cannot be, because the sensors never see it. So dev->state stays
> MLX5_DEVICE_STATE_UP. The kernel-doc of mlx5_trigger_health_work_force()
> says this API exists to prevent exactly that.
>
> In the mlx5_ib consumer later in this series, umrc->state is already
> MLX5_UMR_STATE_ERR at this point, and cq->poll_one is already
> mlx5_ib_poll_one_stopped. The UMR waiters in mlx5r_umr_post_send_wait()
> depend on the internal error flush to wake them, and they wait with no
> timeout:
>
> wait_for_completion(&umr_context.done);
>
> If the forced request is dropped this way, would those waiters hang
> forever?
The same rare hang can already happen today: a UMR waiter waits for its
completion or a flush, and a fatal error while unbind runs, after the
health work is dropped, leaves its WQE unflushed. A forged completion in
this window reaches the same state, so it is a pre-existing hang, i.e. a
DoS, which is out of scope of this series.
>
>> + /*
>> + * The firmware might still be running, so stop it first. Do it under
>> + * intf_state_mutex and only while UP, so the cmd interface is valid.
>> + */
>> + if (forced && test_bit(MLX5_INTERFACE_STATE_UP, &dev->intf_state)) {
>> + int err;
>> +
>> + mlx5_stop_health_poll(dev, false);
>> + err = mlx5_cmd_fast_teardown_hca(dev);
>> + if (err)
>> + mlx5_core_err(dev, "Failed to stop the device before forcing the error state, err %d\n",
>> + err);
>> + }
>
> [Severity: High]
> What happens here when MLX5_INTERFACE_STATE_UP is already clear?
[...]
> Should the forced path unlock and return when the interface is not UP,
> instead of only skipping the teardown? Or should the non-remove unload
> paths cancel force_fatal_report_work?
Yes, v1 will return early. An unload clears UP only after it detached
the consumers, under intf_state_mutex, so once the forced work sees the
interface down there is nothing left to force.
>
> [Severity: High]
> What happens here when mlx5_cmd_fast_teardown_hca() fails?
[...]
> Does this path need a mlx5_cmd_force_teardown_hca() fallback, or bus
> mastering cleared before the error state is forced?
>
v1 will fall back to mlx5_cmd_force_teardown_hca(), as
mlx5_try_fast_unload() does.
> [Severity: Medium]
> Does anything deduplicate this against fatal_report_work, or check
> dev->state?
[...]
In v1 the forced work will do nothing once mlx5_cmd_is_down() is true,
which covers a device already in internal error. The other order needs
a forged completion close to a real fatal error, and at worst ends in a
second recovery or an unload, i.e. a DoS, which is out of scope of this
series.
pw-bot: cr
Yishai
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 02/23] net/mlx5: Derive adversarial mode from CC guest memory encryption
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
2026-11-03 16:39 ` [PATCH rdma-next 01/23] net/mlx5: Add mlx5_trigger_health_work_force() Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:24 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 03/23] net/mlx5: Add capability filtering in adversarial mode Yishai Hadas
` (20 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
Add mlx5_is_adversarial() to report whether the driver treats the NIC
as untrusted. In a confidential computing guest, the hypervisor and any
passthrough device are outside the guest's trust boundary, so the mode
is derived from cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT).
This turns adversarial mode on automatically in such guests, without
any administrator configuration.
Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/net/ethernet/mellanox/mlx5/core/main.c | 7 +++++++
include/linux/mlx5/driver.h | 1 +
2 files changed, 8 insertions(+)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
index 1a45474ce03d..f9e10464a71b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
@@ -31,6 +31,7 @@
*/
#include <linux/highmem.h>
+#include <linux/cc_platform.h>
#include <linux/module.h>
#include <linux/init.h>
#include <linux/errno.h>
@@ -484,6 +485,12 @@ bool mlx5_is_roce_on(struct mlx5_core_dev *dev)
}
EXPORT_SYMBOL(mlx5_is_roce_on);
+bool mlx5_is_adversarial(void)
+{
+ return cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT);
+}
+EXPORT_SYMBOL_GPL(mlx5_is_adversarial);
+
static int handle_hca_cap_2(struct mlx5_core_dev *dev, void *set_ctx)
{
bool do_set = false;
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index a1275e6aa6c1..3d3bc16e581d 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -1328,6 +1328,7 @@ enum {
};
bool mlx5_is_roce_on(struct mlx5_core_dev *dev);
+bool mlx5_is_adversarial(void);
static inline bool mlx5_get_roce_state(struct mlx5_core_dev *dev)
{
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 02/23] net/mlx5: Derive adversarial mode from CC guest memory encryption
2026-11-03 16:39 ` [PATCH rdma-next 02/23] net/mlx5: Derive adversarial mode from CC guest memory encryption Yishai Hadas
@ 2026-11-07 20:24 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:24 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found more severe
issue(s) in other patches of this series. This mail is only an FYI, in case
you want to address the nit picks below when you respin.
Low priority / nit picks for your consideration:
- [Low] mlx5_is_adversarial() in
drivers/net/ethernet/mellanox/mlx5/core/main.c is exported…
- [Low] mlx5_is_adversarial(void) takes no device argument and returns
cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT), so the decision applies to…
--
net/mlx5: Derive adversarial mode from CC guest memory encryption
This patch adds mlx5_is_adversarial(), exported from mlx5_core. It
reports whether the driver should treat the NIC as untrusted. The result
comes from cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT), so it is true in
any confidential computing guest.
> Add mlx5_is_adversarial() to report whether the driver treats the NIC
> as untrusted. In a confidential computing guest, the hypervisor and any
> passthrough device are outside the guest's trust boundary, so the mode
> is derived from cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT).
>
> This turns adversarial mode on automatically in such guests, without
> any administrator configuration.
[Severity: Low]
This isn't a bug, but is the last sentence accurate for this patch on its
own? Nothing calls mlx5_is_adversarial() at this commit, so this patch by
itself does not change behaviour in a CC guest.
The callers arrive later in the series:
"net/mlx5: Add capability filtering in adversarial mode" (caps_filter.c)
"RDMA/mlx5: Derive mlx5_ib_adversarial_mode global from mlx5_core
adversarial API" (drivers/infiniband/hw/mlx5/main.c)
Could the changelog say that the users come in follow-up patches?
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
> index 1a45474ce03db..f9e10464a71b5 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
[ ... ]
> @@ -484,6 +485,12 @@ bool mlx5_is_roce_on(struct mlx5_core_dev *dev)
> }
> EXPORT_SYMBOL(mlx5_is_roce_on);
>
> +bool mlx5_is_adversarial(void)
> +{
> + return cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT);
[Severity: Low]
This isn't a bug, but this check treats every device in a CC guest as
adversarial, and the helper takes no device argument. That matches the
current tree, which has no guest-side way to detect a trusted (T=1)
device.
ib_register_device() in drivers/infiniband/core/device.c uses the same
check and documents the assumption:
/*
* In a CoCo guest every device is currently assumed to be untrusted
* (T=0) and therefore subject to DMA bouncing. Once trusted (T=1)
* device detection is wired up, narrow this check to exclude such
* devices.
*/
if (dma_device && cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
device->cc_dma_bounce = 1;
Would a similar comment here, or a note in the changelog, be worth
adding? Once T=1 detection exists, narrowing this check would also mean
giving mlx5_is_adversarial() a device argument.
> +}
> +EXPORT_SYMBOL_GPL(mlx5_is_adversarial);
[ ... ]
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 03/23] net/mlx5: Add capability filtering in adversarial mode
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
2026-11-03 16:39 ` [PATCH rdma-next 01/23] net/mlx5: Add mlx5_trigger_health_work_force() Yishai Hadas
2026-11-03 16:39 ` [PATCH rdma-next 02/23] net/mlx5: Derive adversarial mode from CC guest memory encryption Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:24 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 04/23] net/mlx5: Add macros for reading log-based caps Yishai Hadas
` (19 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
From: Dragos Tatulea <dtatulea@nvidia.com>
When in adversarial mode, trust only a subset of capabilities
and do some additional validations.
Start off by clearing all firmware capabilities and enabling
them one by one. Some numeric capabilities are validated.
Unexpected values will trigger a driver load failure flow.
Where it applies, the filtering is done on both current and max caps.
Allowed capabilities are simply passed through. Clamped and checked
capabilities also apply to max values.
Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
Reviewed-by: Yishai Hadas <yishaih@nvidia.com>
Signed-off-by: Dragos Tatulea <dtatulea@nvidia.com>
---
.../net/ethernet/mellanox/mlx5/core/Makefile | 2 +-
.../ethernet/mellanox/mlx5/core/caps_filter.c | 381 ++++++++++++++++++
.../net/ethernet/mellanox/mlx5/core/main.c | 8 +-
.../ethernet/mellanox/mlx5/core/mlx5_core.h | 2 +
4 files changed, 388 insertions(+), 5 deletions(-)
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/caps_filter.c
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
index a1871259210d..dc8eabee631f 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile
+++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_MLX5_CORE) += mlx5_core.o
#
# mlx5 core basic
#
-mlx5_core-y := main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \
+mlx5_core-y := main.o caps_filter.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \
health.o mcg.o cq.o alloc.o port.o mr.o pd.o \
transobj.o vport.o sriov.o fs_cmd.o fs_core.o pci_irq.o \
fs_counters.o fs_ft_pool.o rl.o lag/debugfs.o lag/lag.o dev.o events.o wq.o lib/gid.o \
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/caps_filter.c b/drivers/net/ethernet/mellanox/mlx5/core/caps_filter.c
new file mode 100644
index 000000000000..67b260339745
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/caps_filter.c
@@ -0,0 +1,381 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
+
+#include "linux/mlx5/device.h"
+#include <linux/mlx5/driver.h>
+#include <linux/mlx5/qp.h>
+#include "mlx5_core.h"
+
+#define MLX5_HCA_CAP_ALLOW(cap, fld) \
+ MLX5_SET(cap, dst, fld, MLX5_GET(cap, src, fld))
+#define MLX5_HCA_CAP_ALLOW64(cap, fld) \
+ MLX5_SET64(cap, dst, fld, MLX5_GET64(cap, src, fld))
+
+/* Allow a numeric capability range restricted to [lo, hi]. */
+#define MLX5_HCA_CAP_CLAMP(cap, fld, lo, hi) \
+ MLX5_SET(cap, dst, fld, mlx5_cap_clamp(dev, #fld, \
+ MLX5_GET(cap, src, fld), \
+ (lo), (hi)))
+
+/* Check cap range and return error otherwise. */
+#define MLX5_HCA_CAP_CHECK(cap, fld, lo, hi) \
+ mlx5_cap_check(dev, #fld, MLX5_GET(cap, dst, fld), (lo), (hi), false)
+/* Same but zero is also a valid value. */
+#define MLX5_HCA_CAP_CHECK_Z(cap, fld, lo, hi) \
+ mlx5_cap_check(dev, #fld, MLX5_GET(cap, dst, fld), (lo), (hi), true)
+
+static u32 mlx5_cap_clamp(struct mlx5_core_dev *dev, const char *name,
+ u32 val, u32 lo, u32 hi)
+{
+ u32 clamped = clamp_t(u32, val, lo, hi);
+
+ if (clamped != val)
+ mlx5_core_warn(dev, "cap filter: FW reported %s=%u, outside [%u, %u], using %u\n",
+ name, val, lo, hi, clamped);
+
+ return clamped;
+}
+
+static int mlx5_cap_check(struct mlx5_core_dev *dev, const char *name,
+ u32 val, u32 lo, u32 hi, bool zero)
+{
+ if (val >= lo && val <= hi)
+ return 0;
+
+ if (zero && !val)
+ return 0;
+
+ mlx5_core_err(dev, "cap filter: FW reported %s=%u, outside [%u, %u] %s\n",
+ name, val, lo, hi,
+ zero ? "or 0" : "");
+
+ return -EINVAL;
+}
+
+static int mlx5_adversarial_filter_general(struct mlx5_core_dev *dev, void *dst,
+ void *src)
+{
+ int err;
+
+ memset(dst, 0, MLX5_UN_SZ_BYTES(hca_cap_union));
+
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, page_request_disable);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vhca_resource_manager);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, hca_cap_2);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, event_on_vhca_state_teardown_request);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, event_on_vhca_state_in_use);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, event_on_vhca_state_active);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, event_on_vhca_state_allocated);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, event_on_vhca_state_invalid);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vhca_id);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_srq_sz);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_qp_sz);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, event_cap);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_qp);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_srq);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_sgl_for_optimized_performance);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_cq_sz);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, relaxed_ordering_write_umr);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, relaxed_ordering_read_umr);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_cq);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, relaxed_ordering_write);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, relaxed_ordering_read_pci_enabled);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_mkey);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, terminate_scatter_list_mkey);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, fast_teardown);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_eq);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_indirection);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, fixed_buffer_size);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, force_teardown);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, umr_extended_translation_offset);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_klm_list_size);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, qpc_extension);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_ra_req_dc);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vnic_env_cq_overrun);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_ra_res_dc);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, release_all_pages);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, roce_accl);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_ra_req_qp);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_ra_res_qp);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, end_pad);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cc_query_allowed);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cc_modify_allowed);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cache_line_128byte);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, rts2rts_qp_counters_set_id);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vnic_env_int_rq_oob);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, sbcam_reg);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, qcam_reg);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, gid_table_size);
+
+ /* Values outside of [0, 4] range are unexpected. */
+ err = MLX5_HCA_CAP_CHECK(cmd_hca_cap, gid_table_size, 0, 4);
+ if (err)
+ return err;
+
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, out_of_seq_cnt);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vport_counters);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, retransmission_q_counters);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, debug);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, modify_rq_counter_set_id);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, rq_delay_drop);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_qp_cnt);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, pkey_table_size);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vport_group_manager);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vnic_env_queue_counters);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, ets);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, nic_flow_table);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, mcam_reg);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, pcam_reg);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, local_ca_ack_delay);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, enhanced_error_q_counters);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, beacon_led);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, port_type);
+ /* 0 is invalid, 8 is for MLX5_MAX_PORTS */
+ MLX5_HCA_CAP_CLAMP(cmd_hca_cap, num_ports, 1, 8);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_msg);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_tc);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, dcbx);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, general_notification_event);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, stat_rate_support);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, pci_sync_for_fw_update_event);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cqe_version);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, umr_modify_entity_size_disabled);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, umr_modify_atomic_disabled);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, umr_indirect_mkey_disabled);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, umr_fence);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cmdif_checksum);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, sho);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, qos);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, eth_net_offloads);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, roce);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, atomic);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cq_moderation);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cq_period_mode_modify);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cq_period_start_from_cqe);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, apm);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, umr_ptr_rlky);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, qp_packet_based);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, pkv);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, rc);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, uar_4k);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, fl_rc_qp_when_roce_disabled);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_pg_sz);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, bf);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, driver_version);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, mkey_by_name);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_bf_reg_size);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, disciplined_fr_counter);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, qp_error_syndrome);
+ MLX5_HCA_CAP_CLAMP(cmd_hca_cap, num_lag_ports, 0, 1);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_wqe_sz_sq);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, icm_mng_function_id_mode);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, tis_tir_td_order);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_wqe_sz_rq);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, query_host_net_function_num_max);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_qp_mcg);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_mcg);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_transport_domain);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, relaxed_ordering_read);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_pd);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, dp_ordering_ooo_all_ud);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, dp_ordering_ooo_all_uc);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, dp_ordering_ooo_all_xrc);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, dp_ordering_ooo_all_dc);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, dp_ordering_ooo_all_rc);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, pcie_reset_using_hotreset_method);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap,
+ pci_sync_for_fw_update_with_driver_unload);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vnic_env_cnt_steering_fail);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vport_counter_local_loopback);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, q_counter_aggregation);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, q_counter_other_vport);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, nic_receive_steering_discard);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, receive_discard_vport_down);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, transmit_discard_vport_down);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, eq_overrun_count);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, invalid_command_count);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, quota_exceeded_count);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_flow_counter_bulk);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_rq);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_sq);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_tir);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, basic_cyclic_rcv_wqe);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_rqt);
+ MLX5_HCA_CAP_CLAMP(cmd_hca_cap, log_max_rqt_size, 0, 13);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vnic_env_cnt_bar_uar_access);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vnic_env_cnt_odp_page_fault);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, roce_rw_supported);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_wq_sz);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vhca_state);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_vlan_list);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_current_mc_list);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_current_uc_list);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, sq_ts_format);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, rq_ts_format);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, steering_format_version);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, create_qp_start_hint);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, ats);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cross_vhca_rqt);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_uctx);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_num_eqs);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_uar_page_sz);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_geneve_tlv_options);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_geneve_tlv_option_data_len);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_dci_stream_channels);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_dci_errored_streams);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, crossing_vhca_mkey);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, native_port_num);
+ MLX5_HCA_CAP_CLAMP(cmd_hca_cap, num_vhca_ports, 0, 1);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, sw_owner_id);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_num_of_monitor_counters);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, num_ppcnt_monitor_counters);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, num_q_monitor_counters);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, apu);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, uctx_cap);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, num_total_dynamic_vf_msix);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, min_dynamic_vf_msix_table_size);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, data_direct);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_dynamic_vf_msix_table_size);
+ MLX5_HCA_CAP_ALLOW64(cmd_hca_cap, vhca_tunnel_commands);
+
+ MLX5_SET(cmd_hca_cap, dst, num_of_uars_per_page, MLX5_UARS_IN_PAGE);
+
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap, flow_counter_bulk_alloc);
+
+ err = MLX5_HCA_CAP_CHECK(cmd_hca_cap, log_pg_sz,
+ MLX5_ADAPTER_PAGE_SHIFT,
+ MLX5_ADAPTER_PAGE_SHIFT);
+ if (err)
+ return err;
+
+#define LOG_BF_REG_SIZE_MIN ilog2(2 * MLX5_SEND_WQE_BB)
+#define LOG_BF_REG_SIZE_MAX \
+ ilog2((MLX5_ADAPTER_PAGE_SIZE - MLX5_BF_OFFSET) / MLX5_BFREGS_PER_UAR)
+
+ err = MLX5_HCA_CAP_CHECK(cmd_hca_cap, log_bf_reg_size,
+ LOG_BF_REG_SIZE_MIN,
+ LOG_BF_REG_SIZE_MAX);
+ if (err)
+ return err;
+
+#undef LOG_BF_REG_SIZE_MAX
+#undef LOG_BF_REG_SIZE_MIN
+
+ return MLX5_HCA_CAP_CHECK(cmd_hca_cap, min_dynamic_vf_msix_table_size, 0,
+ MLX5_GET(cmd_hca_cap, dst,
+ max_dynamic_vf_msix_table_size));
+}
+
+static int mlx5_adversarial_filter_general_2(struct mlx5_core_dev *dev, void *dst,
+ void *src)
+{
+ int err;
+ u32 val;
+
+ memset(dst, 0, MLX5_UN_SZ_BYTES(hca_cap_union));
+
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, dp_ordering_force);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, query_vuid);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, umr_log_entity_size_5);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, max_reformat_insert_size);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, max_reformat_insert_offset);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, max_reformat_remove_size);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, max_reformat_remove_offset);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, sf_eq_usage);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, max_rqt_vhca_id);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2,
+ min_mkey_log_entity_size_fixed_buffer_valid);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, log_min_mkey_entity_size);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2,
+ min_mkey_log_entity_size_fixed_buffer);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2,
+ max_mkey_log_entity_size_fixed_buffer);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, max_mkey_log_entity_size_mtt);
+ MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, max_num_eqs_24b);
+
+ err = MLX5_HCA_CAP_CHECK_Z(cmd_hca_cap_2, max_mkey_log_entity_size_mtt,
+ MLX5_ADAPTER_PAGE_SHIFT, 63);
+ if (err)
+ return err;
+
+ err = MLX5_HCA_CAP_CHECK_Z(cmd_hca_cap_2,
+ max_mkey_log_entity_size_fixed_buffer,
+ MLX5_ADAPTER_PAGE_SHIFT, 63);
+ if (err)
+ return err;
+
+ val = MLX5_GET(cmd_hca_cap_2, dst, max_mkey_log_entity_size_mtt) ?: 31;
+ err = MLX5_HCA_CAP_CHECK(cmd_hca_cap_2, log_min_mkey_entity_size, 0,
+ val);
+ if (err)
+ return err;
+
+ /* The fixed buffer min is don't-care when its valid bit is clear. */
+ if (!MLX5_GET(cmd_hca_cap_2, dst,
+ min_mkey_log_entity_size_fixed_buffer_valid))
+ return 0;
+
+ val = MLX5_GET(cmd_hca_cap_2, dst,
+ max_mkey_log_entity_size_fixed_buffer) ?: 31;
+ return MLX5_HCA_CAP_CHECK(cmd_hca_cap_2,
+ min_mkey_log_entity_size_fixed_buffer, 0, val);
+}
+
+static int mlx5_adversarial_filter_per_protocol(struct mlx5_core_dev *dev,
+ void *dst, void *src)
+{
+ memset(dst, 0, MLX5_UN_SZ_BYTES(hca_cap_union));
+
+ MLX5_HCA_CAP_ALLOW(per_protocol_networking_offload_caps, csum_cap);
+ MLX5_HCA_CAP_ALLOW(per_protocol_networking_offload_caps, vlan_cap);
+ MLX5_HCA_CAP_ALLOW(per_protocol_networking_offload_caps,
+ self_lb_en_modifiable);
+ MLX5_HCA_CAP_ALLOW(per_protocol_networking_offload_caps, max_lso_cap);
+ MLX5_HCA_CAP_ALLOW(per_protocol_networking_offload_caps,
+ wqe_inline_mode);
+ MLX5_HCA_CAP_ALLOW(per_protocol_networking_offload_caps,
+ rss_ind_tbl_cap);
+ MLX5_HCA_CAP_ALLOW(per_protocol_networking_offload_caps,
+ cqe_checksum_full);
+ MLX5_HCA_CAP_ALLOW(per_protocol_networking_offload_caps,
+ max_vxlan_udp_ports);
+
+ return 0;
+}
+
+#undef MLX5_HCA_CAP_ALLOW
+#undef MLX5_HCA_CAP_ALLOW64
+#undef MLX5_HCA_CAP_CLAMP
+#undef MLX5_HCA_CAP_CHECK
+#undef MLX5_HCA_CAP_CHECK_Z
+
+int mlx5_store_caps(struct mlx5_core_dev *dev, enum mlx5_cap_type cap_type,
+ void *dst, void *src)
+{
+ if (!mlx5_is_adversarial()) {
+ memcpy(dst, src, MLX5_UN_SZ_BYTES(hca_cap_union));
+ return 0;
+ }
+
+ switch (cap_type) {
+ case MLX5_CAP_GENERAL:
+ return mlx5_adversarial_filter_general(dev, dst, src);
+ case MLX5_CAP_GENERAL_2:
+ return mlx5_adversarial_filter_general_2(dev, dst, src);
+ case MLX5_CAP_ETHERNET_OFFLOADS:
+ return mlx5_adversarial_filter_per_protocol(dev, dst, src);
+
+ case MLX5_CAP_ATOMIC:
+ case MLX5_CAP_ROCE:
+ case MLX5_CAP_FLOW_TABLE:
+ case MLX5_CAP_DEV_EVENT:
+ memcpy(dst, src, MLX5_UN_SZ_BYTES(hca_cap_union));
+ break;
+
+ default:
+ memset(dst, 0, MLX5_UN_SZ_BYTES(hca_cap_union));
+ break;
+ }
+
+ return 0;
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
index f9e10464a71b..e76a4486b5a3 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
@@ -325,12 +325,12 @@ int mlx5_core_get_caps_mode(struct mlx5_core_dev *dev, enum mlx5_cap_type cap_ty
switch (cap_mode) {
case HCA_CAP_OPMOD_GET_MAX:
- memcpy(dev->caps.hca[cap_type]->max, hca_caps,
- MLX5_UN_SZ_BYTES(hca_cap_union));
+ err = mlx5_store_caps(dev, cap_type,
+ dev->caps.hca[cap_type]->max, hca_caps);
break;
case HCA_CAP_OPMOD_GET_CUR:
- memcpy(dev->caps.hca[cap_type]->cur, hca_caps,
- MLX5_UN_SZ_BYTES(hca_cap_union));
+ err = mlx5_store_caps(dev, cap_type,
+ dev->caps.hca[cap_type]->cur, hca_caps);
break;
default:
mlx5_core_warn(dev,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
index d84d0424d04e..0084bed669f6 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
@@ -215,6 +215,8 @@ static inline int mlx5_flexible_inlen(struct mlx5_core_dev *dev, size_t fixed,
mlx5_flexible_inlen(dev, fixed, item_size, num_items, __func__, __LINE__)
int mlx5_core_get_caps(struct mlx5_core_dev *dev, enum mlx5_cap_type cap_type);
+int mlx5_store_caps(struct mlx5_core_dev *dev, enum mlx5_cap_type cap_type,
+ void *dst, void *src);
int mlx5_core_get_caps_mode(struct mlx5_core_dev *dev, enum mlx5_cap_type cap_type,
enum mlx5_cap_mode cap_mode);
int mlx5_query_hca_caps(struct mlx5_core_dev *dev);
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 03/23] net/mlx5: Add capability filtering in adversarial mode
2026-11-03 16:39 ` [PATCH rdma-next 03/23] net/mlx5: Add capability filtering in adversarial mode Yishai Hadas
@ 2026-11-07 20:24 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:24 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found 7 potential
issue(s) to consider.
Critical: 0 · High: 3 · Medium: 3 · Low: 1
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [High] The adversarial-mode filter copies cmd_hca_cap.max_tc from
firmware with MLX5_HCA_CAP_ALLOW and never clamps or checks it…
- [High] net/mlx5: unchecked minimum EQ caps cause divide-by-zero in
alloc_cpu_rmap()
- [High] net/mlx5: unclamped log_max_flow_counter_bulk causes OOB heap
read
- [Medium] The commit message says the patch will "trust only a subset of
capabilities" and will "Start off by clearing all firmware…
- [Medium] In adversarial mode, dev->caps.hca[MLX5_CAP_GENERAL]->cur no
longer reflects firmware's current configuration: -…
- [Medium] mlx5_adversarial_filter_general_2() checks
log_min_mkey_entity_size only against (max_mkey_log_entity_size_mtt ?:…
- [Low] The num_ports clamp uses the literal 8 as its upper bound, and
the comment says the bound comes from MLX5_MAX_PORTS (`/* 0 is…
--
net/mlx5: Add capability filtering in adversarial mode
This adds mlx5_store_caps(). In adversarial mode it replaces the raw
memcpy of firmware HCA capabilities with per-field allow-list filtering
for the general, general_2 and ethernet offload capability types. A few
numeric capabilities are clamped or range-checked, and a failed check
aborts driver load. The remaining capability types are either passed
through whole or zeroed.
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/caps_filter.c b/drivers/net/ethernet/mellanox/mlx5/core/caps_filter.c
> new file mode 100644
> index 0000000000000..67b2603397450
> --- /dev/null
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/caps_filter.c
> @@ -0,0 +1,381 @@
[ ... ]
> +static int mlx5_adversarial_filter_general(struct mlx5_core_dev *dev, void *dst,
> + void *src)
> +{
[ ... ]
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, port_type);
> + /* 0 is invalid, 8 is for MLX5_MAX_PORTS */
> + MLX5_HCA_CAP_CLAMP(cmd_hca_cap, num_ports, 1, 8);
[Severity: Low]
This isn't a bug, but the comment says the bound comes from MLX5_MAX_PORTS.
Could the clamp use MLX5_MAX_PORTS directly instead of the literal 8? The
two values agree today.
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_msg);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_tc);
[Severity: High]
Should max_tc be clamped or checked here too? It is a 4 bit field, so
firmware can report up to 15. mlx5_max_tc() turns it into a loop bound:
drivers/net/ethernet/mellanox/mlx5/core/port.c:mlx5_max_tc() {
u8 num_tc = MLX5_CAP_GEN(mdev, max_tc) ? : 8;
return num_tc - 1;
}
That can return up to 14, but the callers assume at most 7 and use it to
index 8-entry arrays.
This path runs at netdev init with no user action. It is gated only by
the qos and ets caps, and both are allow-listed above:
mlx5e_nic_init()
mlx5e_dcbnl_initialize()
mlx5e_ets_init()
ets.ets_cap = mlx5_max_tc(priv->mdev) + 1;
mlx5e_dcbnl_ieee_setets_core()
mlx5e_build_tc_group()
mlx5e_build_tc_tx_bw()
mlx5_set_port_tc_group()
mlx5_set_port_tc_bw_alloc()
mlx5e_dcbnl_ieee_setets_core() declares these on the stack:
u8 tc_tx_bw[IEEE_8021QAZ_MAX_TCS];
u8 tc_group[IEEE_8021QAZ_MAX_TCS];
The build helpers fill them with for (i = 0; i <= max_tc; i++). Does a
max_tc of 9 or more overflow tc_group[] and tc_tx_bw[]?
mlx5_set_port_tc_group() then does:
u32 in[MLX5_ST_SZ_DW(qetc_reg)] = {0};
...
for (i = 0; i <= mlx5_max_tc(mdev); i++) {
MLX5_SET(qetc_reg, in, tc_configuration[i].g, 1);
qetc_reg is 80 bytes, so tc_configuration[14] would land well past the
end of in[].
The same loops appear in mlx5_set_port_tc_bw_alloc(),
mlx5_query_port_ets_rate_limit(), mlx5_modify_port_ets_rate_limit(),
mlx5e_dcbnl_ieee_getets(), mlx5e_dcbnl_ieee_getmaxrate() and
mlx5e_dcbnl_ieee_setmaxrate(). The getets and getmaxrate paths can be
reached through RTM_GETDCB without CAP_NET_ADMIN.
With trusted firmware this behavior predates the patch. num_ports is
clamped here for the same kind of reason, so would it make sense to clamp
max_tc to IEEE_8021QAZ_MAX_TCS as well? No later patch in the series seems
to touch mlx5/core, so the gap would still be there at the end of the
series.
[ ... ]
> +static int mlx5_adversarial_filter_general_2(struct mlx5_core_dev *dev, void *dst,
> + void *src)
> +{
[ ... ]
> + val = MLX5_GET(cmd_hca_cap_2, dst, max_mkey_log_entity_size_mtt) ?: 31;
> + err = MLX5_HCA_CAP_CHECK(cmd_hca_cap_2, log_min_mkey_entity_size, 0,
> + val);
> + if (err)
> + return err;
> +
> + /* The fixed buffer min is don't-care when its valid bit is clear. */
> + if (!MLX5_GET(cmd_hca_cap_2, dst,
> + min_mkey_log_entity_size_fixed_buffer_valid))
> + return 0;
[Severity: Medium]
log_min_mkey_entity_size is only compared against the MTT max here. Is
that enough for the code that consumes it?
In mlx5_ib, get_min_log_entity_size_cap() uses log_min_mkey_entity_size
as the KSM minimum exactly when the fixed buffer valid bit is clear. The
KSM maximum comes from max_mkey_log_entity_size_fixed_buffer. This
combination passes the filter:
max_mkey_log_entity_size_mtt = 0
log_min_mkey_entity_size = 20
max_mkey_log_entity_size_fixed_buffer = 12
min_mkey_log_entity_size_fixed_buffer_valid = 0
mlx5_umem_mkc_find_best_pgsz() then builds GENMASK_ULL(12, 20). That mask
is empty, so KSM mkey registration fails.
There is a second case. The filter accepts log_min_mkey_entity_size up to
31, and mlx5e_mpwrq_min_page_shift() uses it directly. For a regular RQ,
mlx5e_mpwrq_page_shift() hits this whenever the value is above PAGE_SHIFT:
if (WARN_ON_ONCE(!xsk && req_page_shift < min_page_shift))
Should the filter also bound log_min_mkey_entity_size by
max_mkey_log_entity_size_fixed_buffer and by PAGE_SHIFT?
[ ... ]
> + switch (cap_type) {
> + case MLX5_CAP_GENERAL:
> + return mlx5_adversarial_filter_general(dev, dst, src);
> + case MLX5_CAP_GENERAL_2:
> + return mlx5_adversarial_filter_general_2(dev, dst, src);
> + case MLX5_CAP_ETHERNET_OFFLOADS:
> + return mlx5_adversarial_filter_per_protocol(dev, dst, src);
> +
> + case MLX5_CAP_ATOMIC:
> + case MLX5_CAP_ROCE:
> + case MLX5_CAP_FLOW_TABLE:
> + case MLX5_CAP_DEV_EVENT:
> + memcpy(dst, src, MLX5_UN_SZ_BYTES(hca_cap_union));
> + break;
[Severity: Medium]
The commit message says:
When in adversarial mode, trust only a subset of capabilities
and do some additional validations.
Start off by clearing all firmware capabilities and enabling
them one by one.
Is the whole-structure pass-through for MLX5_CAP_ATOMIC, MLX5_CAP_ROCE,
MLX5_CAP_FLOW_TABLE and MLX5_CAP_DEV_EVENT intended? MLX5_CAP_FLOW_TABLE
in particular carries numeric limits that the steering code relies on.
There is a related case. mlx5_adversarial_filter_general() allow-lists
sbcam_reg, qcam_reg, mcam_reg and pcam_reg. In mlx5_query_hca_caps() those
bits gate mlx5_get_pcam_reg(), mlx5_get_mcam_access_reg_group() and
mlx5_get_qcam_reg(). These store firmware data straight into
dev->caps.pcam, dev->caps.mcam and dev->caps.qcam without going through
mlx5_store_caps().
If this is deliberate, could the commit message say which capability
areas are still trusted wholesale?
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
> index f9e10464a71b5..e76a4486b5a31 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
> @@ -325,12 +325,12 @@ int mlx5_core_get_caps_mode(struct mlx5_core_dev *dev, enum mlx5_cap_type cap_ty
>
> switch (cap_mode) {
> case HCA_CAP_OPMOD_GET_MAX:
> - memcpy(dev->caps.hca[cap_type]->max, hca_caps,
> - MLX5_UN_SZ_BYTES(hca_cap_union));
> + err = mlx5_store_caps(dev, cap_type,
> + dev->caps.hca[cap_type]->max, hca_caps);
> break;
> case HCA_CAP_OPMOD_GET_CUR:
> - memcpy(dev->caps.hca[cap_type]->cur, hca_caps,
> - MLX5_UN_SZ_BYTES(hca_cap_union));
> + err = mlx5_store_caps(dev, cap_type,
> + dev->caps.hca[cap_type]->cur, hca_caps);
> break;
[Severity: Medium]
In adversarial mode, dev->caps.hca[MLX5_CAP_GENERAL]->cur now holds the
filtered image:
- it is zeroed, and only allow-listed fields are copied back
- num_of_uars_per_page is forced
- num_ports, num_lag_ports, num_vhca_ports and log_max_rqt_size are
clamped
handle_hca_cap() still uses ->cur as the template for SET_HCA_CAP:
drivers/net/ethernet/mellanox/mlx5/core/main.c:handle_hca_cap() {
...
set_hca_cap = MLX5_ADDR_OF(set_hca_cap_in, set_ctx,
capability);
memcpy(set_hca_cap, dev->caps.hca[MLX5_CAP_GENERAL]->cur,
MLX5_ST_SZ_BYTES(cmd_hca_cap));
...
}
It then writes the whole structure back with set_caps().
Does this program every firmware-writable field to 0 if it is not on the
allow-list and handle_hca_cap() does not set it explicitly? Are the forced
and clamped values also written back to the device?
If so, adversarial mode changes the device configuration, not only what
the driver trusts, and the commit message doesn't mention it. Is that
intended, or should handle_hca_cap() start from the unfiltered firmware
values?
The GENERAL_2 path in handle_hca_cap_2() seems fine here. Its do_set
triggers, sw_vhca_id_valid and lag_per_mp_group, are not allow-listed, so
set_caps() is never reached for GENERAL_2 in adversarial mode.
[ ... ]
> @@ -0,0 +1,381 @@
> +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> +/* Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
> +
> +#include "linux/mlx5/device.h"
> +#include <linux/mlx5/driver.h>
> +#include <linux/mlx5/qp.h>
> +#include "mlx5_core.h"
> +
> +#define MLX5_HCA_CAP_ALLOW(cap, fld) \
> + MLX5_SET(cap, dst, fld, MLX5_GET(cap, src, fld))
> +#define MLX5_HCA_CAP_ALLOW64(cap, fld) \
> + MLX5_SET64(cap, dst, fld, MLX5_GET64(cap, src, fld))
> +
> +/* Allow a numeric capability range restricted to [lo, hi]. */
> +#define MLX5_HCA_CAP_CLAMP(cap, fld, lo, hi) \
> + MLX5_SET(cap, dst, fld, mlx5_cap_clamp(dev, #fld, \
> + MLX5_GET(cap, src, fld), \
> + (lo), (hi)))
> +
> +/* Check cap range and return error otherwise. */
> +#define MLX5_HCA_CAP_CHECK(cap, fld, lo, hi) \
> + mlx5_cap_check(dev, #fld, MLX5_GET(cap, dst, fld), (lo), (hi), false)
> +/* Same but zero is also a valid value. */
> +#define MLX5_HCA_CAP_CHECK_Z(cap, fld, lo, hi) \
> + mlx5_cap_check(dev, #fld, MLX5_GET(cap, dst, fld), (lo), (hi), true)
> +
> +static u32 mlx5_cap_clamp(struct mlx5_core_dev *dev, const char *name,
> + u32 val, u32 lo, u32 hi)
> +{
> + u32 clamped = clamp_t(u32, val, lo, hi);
> +
> + if (clamped != val)
> + mlx5_core_warn(dev, "cap filter: FW reported %s=%u, outside [%u, %u], using %u\n",
> + name, val, lo, hi, clamped);
> +
> + return clamped;
> +}
> +
> +static int mlx5_cap_check(struct mlx5_core_dev *dev, const char *name,
> + u32 val, u32 lo, u32 hi, bool zero)
> +{
> + if (val >= lo && val <= hi)
> + return 0;
> +
> + if (zero && !val)
> + return 0;
> +
> + mlx5_core_err(dev, "cap filter: FW reported %s=%u, outside [%u, %u] %s\n",
> + name, val, lo, hi,
> + zero ? "or 0" : "");
> +
> + return -EINVAL;
> +}
> +
> +static int mlx5_adversarial_filter_general(struct mlx5_core_dev *dev, void *dst,
> + void *src)
> +{
> + int err;
> +
> + memset(dst, 0, MLX5_UN_SZ_BYTES(hca_cap_union));
> +
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, page_request_disable);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vhca_resource_manager);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, hca_cap_2);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, event_on_vhca_state_teardown_request);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, event_on_vhca_state_in_use);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, event_on_vhca_state_active);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, event_on_vhca_state_allocated);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, event_on_vhca_state_invalid);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vhca_id);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_srq_sz);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_qp_sz);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, event_cap);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_qp);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_srq);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_sgl_for_optimized_performance);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_cq_sz);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, relaxed_ordering_write_umr);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, relaxed_ordering_read_umr);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_cq);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, relaxed_ordering_write);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, relaxed_ordering_read_pci_enabled);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_mkey);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, terminate_scatter_list_mkey);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, fast_teardown);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_eq);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_indirection);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, fixed_buffer_size);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, force_teardown);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, umr_extended_translation_offset);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_klm_list_size);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, qpc_extension);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_ra_req_dc);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vnic_env_cq_overrun);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_ra_res_dc);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, release_all_pages);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, roce_accl);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_ra_req_qp);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_ra_res_qp);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, end_pad);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cc_query_allowed);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cc_modify_allowed);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cache_line_128byte);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, rts2rts_qp_counters_set_id);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vnic_env_int_rq_oob);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, sbcam_reg);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, qcam_reg);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, gid_table_size);
> +
> + /* Values outside of [0, 4] range are unexpected. */
> + err = MLX5_HCA_CAP_CHECK(cmd_hca_cap, gid_table_size, 0, 4);
> + if (err)
> + return err;
> +
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, out_of_seq_cnt);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vport_counters);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, retransmission_q_counters);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, debug);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, modify_rq_counter_set_id);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, rq_delay_drop);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_qp_cnt);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, pkey_table_size);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vport_group_manager);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vnic_env_queue_counters);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, ets);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, nic_flow_table);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, mcam_reg);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, pcam_reg);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, local_ca_ack_delay);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, enhanced_error_q_counters);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, beacon_led);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, port_type);
> + /* 0 is invalid, 8 is for MLX5_MAX_PORTS */
> + MLX5_HCA_CAP_CLAMP(cmd_hca_cap, num_ports, 1, 8);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_msg);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_tc);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, dcbx);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, general_notification_event);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, stat_rate_support);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, pci_sync_for_fw_update_event);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cqe_version);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, umr_modify_entity_size_disabled);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, umr_modify_atomic_disabled);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, umr_indirect_mkey_disabled);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, umr_fence);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cmdif_checksum);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, sho);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, qos);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, eth_net_offloads);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, roce);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, atomic);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cq_moderation);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cq_period_mode_modify);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cq_period_start_from_cqe);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, apm);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, umr_ptr_rlky);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, qp_packet_based);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, pkv);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, rc);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, uar_4k);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, fl_rc_qp_when_roce_disabled);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_pg_sz);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, bf);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, driver_version);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, mkey_by_name);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_bf_reg_size);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, disciplined_fr_counter);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, qp_error_syndrome);
> + MLX5_HCA_CAP_CLAMP(cmd_hca_cap, num_lag_ports, 0, 1);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_wqe_sz_sq);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, icm_mng_function_id_mode);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, tis_tir_td_order);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_wqe_sz_rq);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, query_host_net_function_num_max);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_qp_mcg);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_mcg);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_transport_domain);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, relaxed_ordering_read);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_pd);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, dp_ordering_ooo_all_ud);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, dp_ordering_ooo_all_uc);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, dp_ordering_ooo_all_xrc);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, dp_ordering_ooo_all_dc);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, dp_ordering_ooo_all_rc);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, pcie_reset_using_hotreset_method);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap,
> + pci_sync_for_fw_update_with_driver_unload);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vnic_env_cnt_steering_fail);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vport_counter_local_loopback);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, q_counter_aggregation);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, q_counter_other_vport);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, nic_receive_steering_discard);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, receive_discard_vport_down);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, transmit_discard_vport_down);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, eq_overrun_count);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, invalid_command_count);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, quota_exceeded_count);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_flow_counter_bulk);
[Severity: High]
Should log_max_flow_counter_bulk be clamped like log_max_rqt_size?
It is an 8 bit field, so firmware can report up to 255.
get_init_bulk_query_len() uses it directly as a shift count. It
returns min_t(int, MLX5_INIT_COUNTERS_BULK, 1 << the field).
A value of 31 makes the shift yield INT_MIN, so min_t() returns a
negative bulk length. Any value of 32 or more is an undefined shift.
That value is used at init time:
mlx5_init_once()
mlx5_fs_core_alloc()
mlx5_init_fc_stats()
mlx5_fc_stats_bulk_query_buf_realloc()
mlx5_cmd_fc_get_bulk_query_out_len()
mlx5_cmd_fc_get_bulk_query_out_len() multiplies the size_t
MLX5_ST_SZ_BYTES(traffic_counter) by bulk_len and returns the result
as an int. For INT_MIN the low 32 bits of the product are zero. That
leaves out_len as just the 16 byte query_flow_counter_out header. The
16 byte kvzalloc() succeeds, and fc_stats->bulk_query_len is set to
INT_MIN.
Then, once any flow counter exists:
- mlx5_fc_stats_query_all_counters() reads bulk_query_len into a u32
bulk_len of 0x80000000
- mlx5_cmd_fc_bulk_query() is issued with the same 16 byte outlen
- update_counter_cache() indexes flow_statistics[] in bulk_query_out
by counter->id - bulk_base_id
Doesn't that read past the end of the 16 byte allocation for every
counter?
With trusted firmware this predates the patch, but this filter is
where it should be caught. Would clamping to the log of
MLX5_SW_MAX_COUNTERS_BULK be enough here? get_max_bulk_query_len()
does the same shift. If the fix goes into fs_counters.c instead, it
would need to cover both helpers.
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_rq);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_sq);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_tir);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, basic_cyclic_rcv_wqe);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_rqt);
> + MLX5_HCA_CAP_CLAMP(cmd_hca_cap, log_max_rqt_size, 0, 13);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vnic_env_cnt_bar_uar_access);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vnic_env_cnt_odp_page_fault);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, roce_rw_supported);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_wq_sz);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, vhca_state);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_vlan_list);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_current_mc_list);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_current_uc_list);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, sq_ts_format);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, rq_ts_format);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, steering_format_version);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, create_qp_start_hint);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, ats);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, cross_vhca_rqt);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_uctx);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_num_eqs);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_uar_page_sz);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_geneve_tlv_options);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_geneve_tlv_option_data_len);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_dci_stream_channels);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, log_max_dci_errored_streams);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, crossing_vhca_mkey);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, native_port_num);
> + MLX5_HCA_CAP_CLAMP(cmd_hca_cap, num_vhca_ports, 0, 1);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, sw_owner_id);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_num_of_monitor_counters);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, num_ppcnt_monitor_counters);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, num_q_monitor_counters);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, apu);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, uctx_cap);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, num_total_dynamic_vf_msix);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, min_dynamic_vf_msix_table_size);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, data_direct);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, max_dynamic_vf_msix_table_size);
> + MLX5_HCA_CAP_ALLOW64(cmd_hca_cap, vhca_tunnel_commands);
> +
> + MLX5_SET(cmd_hca_cap, dst, num_of_uars_per_page, MLX5_UARS_IN_PAGE);
> +
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap, flow_counter_bulk_alloc);
> +
> + err = MLX5_HCA_CAP_CHECK(cmd_hca_cap, log_pg_sz,
> + MLX5_ADAPTER_PAGE_SHIFT,
> + MLX5_ADAPTER_PAGE_SHIFT);
> + if (err)
> + return err;
> +
> +#define LOG_BF_REG_SIZE_MIN ilog2(2 * MLX5_SEND_WQE_BB)
> +#define LOG_BF_REG_SIZE_MAX \
> + ilog2((MLX5_ADAPTER_PAGE_SIZE - MLX5_BF_OFFSET) / MLX5_BFREGS_PER_UAR)
> +
> + err = MLX5_HCA_CAP_CHECK(cmd_hca_cap, log_bf_reg_size,
> + LOG_BF_REG_SIZE_MIN,
> + LOG_BF_REG_SIZE_MAX);
> + if (err)
> + return err;
> +
> +#undef LOG_BF_REG_SIZE_MAX
> +#undef LOG_BF_REG_SIZE_MIN
> +
> + return MLX5_HCA_CAP_CHECK(cmd_hca_cap, min_dynamic_vf_msix_table_size, 0,
> + MLX5_GET(cmd_hca_cap, dst,
> + max_dynamic_vf_msix_table_size));
> +}
> +
> +static int mlx5_adversarial_filter_general_2(struct mlx5_core_dev *dev, void *dst,
> + void *src)
> +{
> + int err;
> + u32 val;
> +
> + memset(dst, 0, MLX5_UN_SZ_BYTES(hca_cap_union));
> +
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, dp_ordering_force);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, query_vuid);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, umr_log_entity_size_5);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, max_reformat_insert_size);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, max_reformat_insert_offset);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, max_reformat_remove_size);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, max_reformat_remove_offset);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, sf_eq_usage);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, max_rqt_vhca_id);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2,
> + min_mkey_log_entity_size_fixed_buffer_valid);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, log_min_mkey_entity_size);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2,
> + min_mkey_log_entity_size_fixed_buffer);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2,
> + max_mkey_log_entity_size_fixed_buffer);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, max_mkey_log_entity_size_mtt);
> + MLX5_HCA_CAP_ALLOW(cmd_hca_cap_2, max_num_eqs_24b);
> +
> + err = MLX5_HCA_CAP_CHECK_Z(cmd_hca_cap_2, max_mkey_log_entity_size_mtt,
> + MLX5_ADAPTER_PAGE_SHIFT, 63);
> + if (err)
> + return err;
> +
> + err = MLX5_HCA_CAP_CHECK_Z(cmd_hca_cap_2,
> + max_mkey_log_entity_size_fixed_buffer,
> + MLX5_ADAPTER_PAGE_SHIFT, 63);
> + if (err)
> + return err;
> +
> + val = MLX5_GET(cmd_hca_cap_2, dst, max_mkey_log_entity_size_mtt) ?: 31;
> + err = MLX5_HCA_CAP_CHECK(cmd_hca_cap_2, log_min_mkey_entity_size, 0,
> + val);
> + if (err)
> + return err;
> +
> + /* The fixed buffer min is don't-care when its valid bit is clear. */
> + if (!MLX5_GET(cmd_hca_cap_2, dst,
> + min_mkey_log_entity_size_fixed_buffer_valid))
> + return 0;
> +
> + val = MLX5_GET(cmd_hca_cap_2, dst,
> + max_mkey_log_entity_size_fixed_buffer) ?: 31;
> + return MLX5_HCA_CAP_CHECK(cmd_hca_cap_2,
> + min_mkey_log_entity_size_fixed_buffer, 0, val);
> +}
> +
> +static int mlx5_adversarial_filter_per_protocol(struct mlx5_core_dev *dev,
> + void *dst, void *src)
> +{
> + memset(dst, 0, MLX5_UN_SZ_BYTES(hca_cap_union));
> +
> + MLX5_HCA_CAP_ALLOW(per_protocol_networking_offload_caps, csum_cap);
> + MLX5_HCA_CAP_ALLOW(per_protocol_networking_offload_caps, vlan_cap);
> + MLX5_HCA_CAP_ALLOW(per_protocol_networking_offload_caps,
> + self_lb_en_modifiable);
> + MLX5_HCA_CAP_ALLOW(per_protocol_networking_offload_caps, max_lso_cap);
> + MLX5_HCA_CAP_ALLOW(per_protocol_networking_offload_caps,
> + wqe_inline_mode);
> + MLX5_HCA_CAP_ALLOW(per_protocol_networking_offload_caps,
> + rss_ind_tbl_cap);
> + MLX5_HCA_CAP_ALLOW(per_protocol_networking_offload_caps,
> + cqe_checksum_full);
> + MLX5_HCA_CAP_ALLOW(per_protocol_networking_offload_caps,
> + max_vxlan_udp_ports);
> +
> + return 0;
> +}
> +
> +#undef MLX5_HCA_CAP_ALLOW
> +#undef MLX5_HCA_CAP_ALLOW64
> +#undef MLX5_HCA_CAP_CLAMP
> +#undef MLX5_HCA_CAP_CHECK
> +#undef MLX5_HCA_CAP_CHECK_Z
> +
> +int mlx5_store_caps(struct mlx5_core_dev *dev, enum mlx5_cap_type cap_type,
> + void *dst, void *src)
> +{
> + if (!mlx5_is_adversarial()) {
> + memcpy(dst, src, MLX5_UN_SZ_BYTES(hca_cap_union));
> + return 0;
> + }
> +
> + switch (cap_type) {
> + case MLX5_CAP_GENERAL:
> + return mlx5_adversarial_filter_general(dev, dst, src);
> + case MLX5_CAP_GENERAL_2:
> + return mlx5_adversarial_filter_general_2(dev, dst, src);
> + case MLX5_CAP_ETHERNET_OFFLOADS:
> + return mlx5_adversarial_filter_per_protocol(dev, dst, src);
> +
> + case MLX5_CAP_ATOMIC:
> + case MLX5_CAP_ROCE:
> + case MLX5_CAP_FLOW_TABLE:
> + case MLX5_CAP_DEV_EVENT:
> + memcpy(dst, src, MLX5_UN_SZ_BYTES(hca_cap_union));
> + break;
> +
> + default:
> + memset(dst, 0, MLX5_UN_SZ_BYTES(hca_cap_union));
> + break;
> + }
> +
> + return 0;
> +}
[Severity: High]
Should the EQ count be range checked as well? log_max_eq and
max_num_eqs are allow-listed here, and max_num_eqs_24b is allow-listed
in mlx5_adversarial_filter_general_2(). Nothing makes sure the device
reports more EQs than the driver reserves for async EQs.
mlx5_max_eq_cap_get() returns max_num_eqs_24b if it is set, otherwise
max_num_eqs, otherwise 1 << log_max_eq. get_num_eqs() then takes the
minimum of two values:
- the IRQ table's completion vector count
- max_dev_eqs - MLX5_MAX_ASYNC_EQS
MLX5_MAX_ASYNC_EQS is 3 or 4, depending on
CONFIG_INFINIBAND_ON_DEMAND_PAGING.
Suppose firmware reports exactly MLX5_MAX_ASYNC_EQS, for example
max_num_eqs = 4 with ODP enabled. Then eq_table->max_comp_eqs becomes
0, and probe continues into:
mlx5_load()
mlx5_eq_table_create()
alloc_rmap()
alloc_irq_cpu_rmap(eq_table->max_comp_eqs)
alloc_cpu_rmap(0, GFP_KERNEL)
alloc_cpu_rmap() fills the near[] table with:
rmap->near[cpu].index = cpu % size;
Doesn't that divide by zero during driver load? This path is taken
with CONFIG_RFS_ACCEL on non-SF devices and needs no user action.
If firmware reports fewer EQs than MLX5_MAX_ASYNC_EQS, max_comp_eqs
goes negative instead. alloc_cpu_rmap() rejects that value as too
large. But when the rmap is not allocated, what do
mlx5_comp_vectors_max() and the completion EQ code make of a negative
count?
With trusted firmware this predates the patch. The three fields come
from two different cap types, so one MLX5_HCA_CAP_CHECK() in a single
filter function can't cover them all. Would it be simpler to reject
max_dev_eqs <= MLX5_MAX_ASYNC_EQS in get_num_eqs() or
mlx5_eq_table_create()? Or is there a good place in the filter for
this check?
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 04/23] net/mlx5: Add macros for reading log-based caps
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (2 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 03/23] net/mlx5: Add capability filtering in adversarial mode Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:24 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 05/23] RDMA/mlx5: Reject QP transport types not enabled by FW capability Yishai Hadas
` (18 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
From: Dragos Tatulea <dtatulea@nvidia.com>
Add macros that can be used to exponentiate log based capabilities.
When a capability can represent more than the used version (u32 , u64),
value will be clamped to the largest representable power of two.
Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
Reviewed-by: Yishai Hadas <yishaih@nvidia.com>
Signed-off-by: Dragos Tatulea <dtatulea@nvidia.com>
---
include/linux/mlx5/device.h | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h
index 37727b5ab78f..f45454f65d5c 100644
--- a/include/linux/mlx5/device.h
+++ b/include/linux/mlx5/device.h
@@ -38,6 +38,7 @@
#include <linux/mlx5/mlx5_ifc.h>
#include <linux/mlx5/mlx5_ifc_macros.h>
#include <linux/bitfield.h>
+#include <linux/minmax.h>
#if defined(__LITTLE_ENDIAN)
#define MLX5_SET_HOST_ENDIANNESS 0
@@ -47,6 +48,16 @@
#error Host endianness not defined
#endif
+/* Largest shift count that is defined behaviour for a value of type @T. */
+#define __mlx5_max_shift(T) (BITS_PER_TYPE(T) - 1 - is_signed_type(T))
+
+/* Calculate 2^@fld from a log2 cap field. Exponents that don't fit in @T
+ * saturate to the largest representable power of two (not to the type
+ * maximum), avoiding undefined shifts.
+ */
+#define MLX5_GET_LOGSZ_CLAMP(typ, p, fld, T) \
+ ((T)((T)1 << min_t(u32, MLX5_GET(typ, p, fld), __mlx5_max_shift(T))))
+
enum mlx5_inline_modes {
MLX5_INLINE_MODE_NONE,
MLX5_INLINE_MODE_L2,
@@ -1096,6 +1107,15 @@ enum mlx5_qcam_feature_groups {
#define MLX5_CAP_GEN_MAX(mdev, cap) \
MLX5_GET(cmd_hca_cap, mdev->caps.hca[MLX5_CAP_GENERAL]->max, cap)
+/* 2^cap for log2-encoded general caps */
+#define MLX5_CAP_GEN_LOGSZ_U32(mdev, cap) \
+ MLX5_GET_LOGSZ_CLAMP(cmd_hca_cap, \
+ mdev->caps.hca[MLX5_CAP_GENERAL]->cur, cap, u32)
+
+#define MLX5_CAP_GEN_LOGSZ_U64(mdev, cap) \
+ MLX5_GET_LOGSZ_CLAMP(cmd_hca_cap, \
+ mdev->caps.hca[MLX5_CAP_GENERAL]->cur, cap, u64)
+
#define MLX5_CAP_GEN_2(mdev, cap) \
MLX5_GET(cmd_hca_cap_2, mdev->caps.hca[MLX5_CAP_GENERAL_2]->cur, cap)
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 04/23] net/mlx5: Add macros for reading log-based caps
2026-11-03 16:39 ` [PATCH rdma-next 04/23] net/mlx5: Add macros for reading log-based caps Yishai Hadas
@ 2026-11-07 20:24 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:24 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found more severe
issue(s) in other patches of this series. This mail is only an FYI, in case
you want to address the nit picks below when you respin.
Low priority / nit picks for your consideration:
- [Low] The concern says MLX5_GET_LOGSZ_CLAMP(), MLX5_CAP_GEN_LOGSZ_U32()
and MLX5_CAP_GEN_LOGSZ_U64() in include/linux/mlx5/device.h are added…
--
net/mlx5: Add macros for reading log-based caps
This commit adds MLX5_GET_LOGSZ_CLAMP() and the MLX5_CAP_GEN_LOGSZ_U32()
and MLX5_CAP_GEN_LOGSZ_U64() accessors to include/linux/mlx5/device.h.
They compute 2^cap for log2-encoded capability fields, with the shift
clamped to the largest shift that is defined for the result type.
> diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h
> index 37727b5ab78f2..f45454f65d5c0 100644
> --- a/include/linux/mlx5/device.h
> +++ b/include/linux/mlx5/device.h
[ ... ]
> @@ -47,6 +48,16 @@
> #error Host endianness not defined
> #endif
>
> +/* Largest shift count that is defined behaviour for a value of type @T. */
> +#define __mlx5_max_shift(T) (BITS_PER_TYPE(T) - 1 - is_signed_type(T))
> +
> +/* Calculate 2^@fld from a log2 cap field. Exponents that don't fit in @T
> + * saturate to the largest representable power of two (not to the type
> + * maximum), avoiding undefined shifts.
> + */
> +#define MLX5_GET_LOGSZ_CLAMP(typ, p, fld, T) \
> + ((T)((T)1 << min_t(u32, MLX5_GET(typ, p, fld), __mlx5_max_shift(T))))
> +
[ ... ]
> @@ -1096,6 +1107,15 @@ enum mlx5_qcam_feature_groups {
> #define MLX5_CAP_GEN_MAX(mdev, cap) \
> MLX5_GET(cmd_hca_cap, mdev->caps.hca[MLX5_CAP_GENERAL]->max, cap)
>
> +/* 2^cap for log2-encoded general caps */
> +#define MLX5_CAP_GEN_LOGSZ_U32(mdev, cap) \
> + MLX5_GET_LOGSZ_CLAMP(cmd_hca_cap, \
> + mdev->caps.hca[MLX5_CAP_GENERAL]->cur, cap, u32)
> +
> +#define MLX5_CAP_GEN_LOGSZ_U64(mdev, cap) \
> + MLX5_GET_LOGSZ_CLAMP(cmd_hca_cap, \
> + mdev->caps.hca[MLX5_CAP_GENERAL]->cur, cap, u64)
> +
[Severity: Low]
This isn't a bug, but MLX5_GET_LOGSZ_CLAMP(), MLX5_CAP_GEN_LOGSZ_U32() and
MLX5_CAP_GEN_LOGSZ_U64() have no callers in this commit.
Their users show up later in the series, in "RDMA/mlx5: Clamp log_max_cq_sz
shift in CQ size checks", "RDMA/mlx5: Clamp log_max_* shifts in QP and WQ
limit checks" and "RDMA/mlx5: Clamp log_max_* shifts in query_device and
ucontext paths". Those patches convert call sites in cq.c, qp.c, main.c,
mad.c and srq.c under drivers/infiniband/hw/mlx5/. For example, in
mlx5_ib_query_device():
props->max_mr = MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_mkey);
Could the commit message mention these follow-up users and the reason for
the clamping?
The macro itself looks correct. min_t() limits the shift to 31 for u32 and
to 63 for u64.
The next point is a pre-existing issue that this patch did not introduce.
At the end of the series, net/mlx5 core still open-codes some of these
shifts. One is in get_init_bulk_query_len() and get_max_bulk_query_len()
in fs_counters.c:
return min_t(int, MLX5_INIT_COUNTERS_BULK,
(1 << MLX5_CAP_GEN(dev, log_max_flow_counter_bulk)));
Another is mlx5e_mpwrq_max_num_entries() in en/params.c:
u32 klm_limit = min(MLX5E_MAX_RQ_NUM_KSMS,
1 << MLX5_CAP_GEN(mdev, log_max_klm_list_size));
These shifts are exposed to untrusted firmware values by the earlier patch
"net/mlx5: Add capability filtering in adversarial mode". There,
mlx5_adversarial_filter_general() lets both caps through
MLX5_HCA_CAP_ALLOW without checking their range.
Would it make sense to convert these sites to the new helpers too?
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 05/23] RDMA/mlx5: Reject QP transport types not enabled by FW capability
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (3 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 04/23] net/mlx5: Add macros for reading log-based caps Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:24 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 06/23] RDMA/mlx5: Require IB_QPT_DRIVER for the DC vendor QP types Yishai Hadas
` (17 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
check_qp_type() checks the FW capability only for XRC and DC. RC, UC
and UD QPs are created even if the FW does not report the matching
capability bit.
Check the rc, uc and ud capabilities as well, so that QP creation is
refused for a transport the FW does not support.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/hw/mlx5/qp.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index 627dd17804c9..943ee5c55538 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -2824,15 +2824,24 @@ static int check_qp_type(struct mlx5_ib_dev *dev, struct ib_qp_init_attr *attr,
case IB_QPT_XRC_INI:
if (!MLX5_CAP_GEN(dev->mdev, xrc))
goto out;
- fallthrough;
+ break;
case IB_QPT_RC:
+ if (!MLX5_CAP_GEN(dev->mdev, rc))
+ goto out;
+ break;
case IB_QPT_UC:
+ if (!MLX5_CAP_GEN(dev->mdev, uc))
+ goto out;
+ break;
+ case IB_QPT_UD:
+ if (!MLX5_CAP_GEN(dev->mdev, ud))
+ goto out;
+ break;
case IB_QPT_SMI:
case MLX5_IB_QPT_HW_GSI:
case IB_QPT_DRIVER:
case IB_QPT_GSI:
case IB_QPT_RAW_PACKET:
- case IB_QPT_UD:
case MLX5_IB_QPT_REG_UMR:
break;
default:
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 05/23] RDMA/mlx5: Reject QP transport types not enabled by FW capability
2026-11-03 16:39 ` [PATCH rdma-next 05/23] RDMA/mlx5: Reject QP transport types not enabled by FW capability Yishai Hadas
@ 2026-11-07 20:24 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:24 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found more severe
issue(s) in other patches of this series. This mail is only an FYI, in case
you want to address the nit picks below when you respin.
Low priority / nit picks for your consideration:
- [Low] The commit message says the new rc/uc/ud checks in
check_qp_type() (drivers/infiniband/hw/mlx5/qp.c) refuse 'a transport…
--
RDMA/mlx5: Reject QP transport types not enabled by FW capability
This commit makes check_qp_type() in drivers/infiniband/hw/mlx5/qp.c also
check the rc, uc and ud general capability bits. QP creation for RC, UC
and UD now returns -EOPNOTSUPP when the matching bit is clear. Before
this change, only XRC and DC were gated on capabilities.
> Check the rc, uc and ud capabilities as well, so that QP creation is
> refused for a transport the FW does not support.
[Severity: Low]
Could the commit message explain why this is needed and what userspace
will see after it?
As far as I can tell, the new lines in check_qp_type() are the first
readers of the rc, uc and ud bits anywhere in the tree. With trusted FW,
the change seems to only move a CREATE_QP failure from the FW into the
driver.
The visible effect seems to come from the earlier patch "net/mlx5: Add
capability filtering in adversarial mode". In caps_filter.c,
mlx5_adversarial_filter_general() clears cmd_hca_cap and then restores
only an allow-list of fields:
mlx5_adversarial_filter_general() {
...
memset(dst, 0, MLX5_UN_SZ_BYTES(hca_cap_union));
...
MLX5_HCA_CAP_ALLOW(cmd_hca_cap, rc);
...
}
rc is on that list, but ud, uc and xrc are not. In confidential computing
guests running in adversarial mode, every UD and UC QP request would now
fail in check_qp_type():
case IB_QPT_UD:
if (!MLX5_CAP_GEN(dev->mdev, ud))
goto out;
Only RC, RAW_PACKET and the internal special QP types could still be
created. This affects userspace. It also affects kernel ULPs such as
IPoIB until the later patch "RDMA/core: Suppress kverbs_provider for
adversarial_mode devices".
The driver itself clears these bits by policy, so would it be clearer to
describe that as the purpose of this patch? Could the message also say
whether this is a bug fix (with a Fixes: tag) or hardening, so stable
backporters can decide?
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 06/23] RDMA/mlx5: Require IB_QPT_DRIVER for the DC vendor QP types
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (4 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 05/23] RDMA/mlx5: Reject QP transport types not enabled by FW capability Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:24 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 07/23] RDMA/mlx5: Derive mlx5_ib_adversarial_mode global from mlx5_core adversarial API Yishai Hadas
` (16 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
check_qp_type() checks the dct capability only for IB_QPT_DRIVER, but
process_vendor_flags() turns any QP that carries a DC type flag into a
DCI or DCT QP. A QP created as, for example, IB_QPT_RC with a DC flag
thus becomes a DC QP without the capability check.
Accept the DC type flags only on IB_QPT_DRIVER, the type userspace
uses to create DC QPs.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/hw/mlx5/qp.c | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index 943ee5c55538..19cbf6b2b979 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -2965,18 +2965,22 @@ static int process_vendor_flags(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
switch (flags & (MLX5_QP_FLAG_TYPE_DCT | MLX5_QP_FLAG_TYPE_DCI)) {
case MLX5_QP_FLAG_TYPE_DCI:
+ if (qp->type != IB_QPT_DRIVER)
+ return -EINVAL;
qp->type = MLX5_IB_QPT_DCI;
break;
case MLX5_QP_FLAG_TYPE_DCT:
+ if (qp->type != IB_QPT_DRIVER)
+ return -EINVAL;
qp->type = MLX5_IB_QPT_DCT;
break;
+ case 0:
+ /* IB_QPT_DRIVER needs a subtype */
+ if (qp->type == IB_QPT_DRIVER)
+ return -EINVAL;
+ break;
default:
- if (qp->type != IB_QPT_DRIVER)
- break;
- /*
- * It is IB_QPT_DRIVER and or no subtype or
- * wrong subtype were provided.
- */
+ /* Both subtypes */
return -EINVAL;
}
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 06/23] RDMA/mlx5: Require IB_QPT_DRIVER for the DC vendor QP types
2026-11-03 16:39 ` [PATCH rdma-next 06/23] RDMA/mlx5: Require IB_QPT_DRIVER for the DC vendor QP types Yishai Hadas
@ 2026-11-07 20:24 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:24 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found 1 potential
issue(s) to consider.
Critical: 0 · High: 0 · Medium: 1 · Low: 0
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [Medium] The commit "RDMA/mlx5: Require IB_QPT_DRIVER for the DC vendor
QP types" fixes a regression that userspace can trigger, but its…
--
RDMA/mlx5: Require IB_QPT_DRIVER for the DC vendor QP types
This patch changes process_vendor_flags() in drivers/infiniband/hw/mlx5/qp.c
so that it rejects MLX5_QP_FLAG_TYPE_DCI and MLX5_QP_FLAG_TYPE_DCT unless
the QP was requested as IB_QPT_DRIVER. Without this check, an RC/UC/UD QP
that carries a DC flag becomes a DC QP and skips the dct capability check
in check_qp_type().
> Accept the DC type flags only on IB_QPT_DRIVER, the type userspace
> uses to create DC QPs.
>
> Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
[Severity: Medium]
Should this patch carry a Fixes: tag and Cc: stable? Userspace can reach
the behavior it fixes, and nothing in the message tells stable maintainers
to pick it up.
The problem seems to come from commit 37518fa49f76 ("RDMA/mlx5: Process
all vendor flags in one place"). That commit changed the guard around the
process_vendor_flags() call in the create path from:
if (init_attr->qp_type == IB_QPT_DRIVER)
to:
if (udata)
After that, the DCI/DCT type flags were processed for every user QP type.
Commit 7aede1a25f4b ("RDMA/mlx5: Store QP type in the vendor QP
structure") then made the switch in process_vendor_flags() write
qp->type = MLX5_IB_QPT_DCI or MLX5_IB_QPT_DCT directly. It still did not
check which type was requested.
>From then on, an unprivileged verbs user can create an IB_QPT_RC QP with
MLX5_QP_FLAG_TYPE_DCT set in the ucmd flags and get this path:
mlx5_ib_create_qp()
check_qp_type() <- dct cap only checked for IB_QPT_DRIVER
qp->type = IB_QPT_RC;
process_vendor_flags()
qp->type = MLX5_IB_QPT_DCT;
check_qp_attr()
create_qp()
check_qp_attr(), create_qp() and the modify/query/destroy paths then
dispatch on the mlx5 qp->type. Meanwhile ibqp->qp_type still says
IB_QPT_RC.
Since this goes back to the 2020 refactor (v5.8), would something like
the following be appropriate?
Fixes: 37518fa49f76 ("RDMA/mlx5: Process all vendor flags in one place")
Cc: stable@vger.kernel.org
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 07/23] RDMA/mlx5: Derive mlx5_ib_adversarial_mode global from mlx5_core adversarial API
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (5 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 06/23] RDMA/mlx5: Require IB_QPT_DRIVER for the DC vendor QP types Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:24 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 08/23] RDMA/core: Suppress kverbs_provider for adversarial_mode devices Yishai Hadas
` (15 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
Use the mlx5_core mlx5_is_adversarial() exported API to set a
module-level static bool mlx5_ib_adversarial_mode at init time,
instead of a per-device bit field. This avoids propagating the flag
through every mlx5_ib_dev pointer and makes the module-wide policy
clear at the call sites.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/hw/mlx5/main.c | 3 +++
drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 +
2 files changed, 4 insertions(+)
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 65455aedd559..f4f2ad80e76c 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -76,6 +76,7 @@ enum {
MLX5_ATOMIC_SIZE_QP_8BYTES = 1 << 3,
};
+bool mlx5_ib_adversarial_mode;
static struct workqueue_struct *mlx5_ib_event_wq;
static LIST_HEAD(mlx5_ib_unaffiliated_port_list);
static LIST_HEAD(mlx5_ib_dev_list);
@@ -5431,6 +5432,8 @@ static int __init mlx5_ib_init(void)
return -ENOMEM;
}
+ mlx5_ib_adversarial_mode = mlx5_is_adversarial();
+
ret = mlx5_ib_qp_event_init();
if (ret)
goto qp_event_err;
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 4a6ee14ffd7d..0f81b8aa487e 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -1582,6 +1582,7 @@ static inline int get_uars_per_sys_page(struct mlx5_ib_dev *dev, bool lib_suppor
}
extern void *xlt_emergency_page;
+extern bool mlx5_ib_adversarial_mode;
int bfregn_to_uar_index(struct mlx5_ib_dev *dev,
struct mlx5_bfreg_info *bfregi, u32 bfregn,
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 07/23] RDMA/mlx5: Derive mlx5_ib_adversarial_mode global from mlx5_core adversarial API
2026-11-03 16:39 ` [PATCH rdma-next 07/23] RDMA/mlx5: Derive mlx5_ib_adversarial_mode global from mlx5_core adversarial API Yishai Hadas
@ 2026-11-07 20:24 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:24 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found more severe
issue(s) in other patches of this series. This mail is only an FYI, in case
you want to address the nit picks below when you respin.
Low priority / nit picks for your consideration:
- [Low] The commit message does not describe the code correctly.
--
RDMA/mlx5: Derive mlx5_ib_adversarial_mode global from mlx5_core adversarial API
This commit adds a global bool mlx5_ib_adversarial_mode to the mlx5_ib
driver. mlx5_ib_init() sets it from mlx5_is_adversarial(). The flag is
declared extern in mlx5_ib.h so that other files in the driver can read it.
> Use the mlx5_core mlx5_is_adversarial() exported API to set a
> module-level static bool mlx5_ib_adversarial_mode at init time,
> instead of a per-device bit field. This avoids propagating the flag
> through every mlx5_ib_dev pointer and makes the module-wide policy
> clear at the call sites.
[Severity: Low]
Is "static" the right word here? The patch adds a non-static global in
drivers/infiniband/hw/mlx5/main.c:
bool mlx5_ib_adversarial_mode;
and makes it visible to the rest of the driver in
drivers/infiniband/hw/mlx5/mlx5_ib.h:
extern bool mlx5_ib_adversarial_mode;
Later in the series the variable stays non-static, and umr.c and cq.c
read it. That is why it can't be static.
The text "instead of a per-device bit field" also seems to point at
something that isn't in the tree. No adversarial field exists under
drivers/infiniband/ before this patch, and the patch only adds lines. Is
this a leftover from an earlier, unmerged version of the series?
Could the message instead say that the patch adds a driver-wide global,
shared through mlx5_ib.h, that caches mlx5_is_adversarial() in
mlx5_ib_init()?
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 08/23] RDMA/core: Suppress kverbs_provider for adversarial_mode devices
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (6 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 07/23] RDMA/mlx5: Derive mlx5_ib_adversarial_mode global from mlx5_core adversarial API Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-03 19:03 ` Jacob Moroni
2026-11-07 20:24 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 09/23] RDMA/mlx5: Remove duplicate mlx5_ib_umr_context struct Yishai Hadas
` (14 subsequent siblings)
22 siblings, 2 replies; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
In adversarial mode the NIC is not trusted. Kernel ULPs such as MAD,
IB CM, rdma_cm, IPoIB, SRP, iSER and NVMe-oF trust what the device
reports, its completions and the data it writes, so they must not use
such a device.
Add an adversarial_mode field to struct ib_device and clear
kverbs_provider when it is set. add_client_context() then skips every
client but uverbs, the only one that sets no_kverbs_req, so the kernel
ULPs never bind to the device.
The driver still implements the kverbs ops, as the uverbs path shares
them, so kverbs_provider can't be cleared by the existing check of the
mandatory ops.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/core/device.c | 8 ++++++++
include/rdma/ib_verbs.h | 2 ++
2 files changed, 10 insertions(+)
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index 29af9fe860e2..eb6aa567d365 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -312,6 +312,14 @@ static void ib_device_check_mandatory(struct ib_device *device)
break;
}
}
+
+ /*
+ * Kernel ULPs must not bind to an adversarial_mode device. Its kverbs
+ * ops are still present, as uverbs uses them, so clear kverbs_provider
+ * explicitly. uverbs sets no_kverbs_req, so it still binds.
+ */
+ if (device->adversarial_mode)
+ device->kverbs_provider = false;
}
/*
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index d6eaa3fee66e..6209a5166f0d 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -3008,6 +3008,8 @@ struct ib_device {
u16 use_cq_dim:1;
/* CoCo guest with DMA bounce buffering required */
u16 cc_dma_bounce:1;
+ /* Untrusted-NIC, adversarial mode is required */
+ u8 adversarial_mode;
u8 node_type;
u32 phys_port_cnt;
struct ib_device_attr attrs;
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 08/23] RDMA/core: Suppress kverbs_provider for adversarial_mode devices
2026-11-03 16:39 ` [PATCH rdma-next 08/23] RDMA/core: Suppress kverbs_provider for adversarial_mode devices Yishai Hadas
@ 2026-11-03 19:03 ` Jacob Moroni
2026-11-03 19:07 ` Jason Gunthorpe
2026-11-07 20:24 ` netdev-bot+sashiko
1 sibling, 1 reply; 48+ messages in thread
From: Jacob Moroni @ 2026-11-03 19:03 UTC (permalink / raw)
To: Yishai Hadas
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Hi,
Out of curiosity, is the idea that these restrictions will eventually
be lifted once the
ULPs (and I guess the drivers) are hardened? Or is it considered
infeasible? I am
most curious about RDMA CM.
Even if the drivers are hardened (no round-tripping WR IDs thru HW
from WQE->CQE,
sanitizing values from FW, etc.), I think it may be impossible to
protect against certain
bogus payload scenarios such as a NIC reporting a CQE even though it
never wrote the
data (or wrote bogus data), but this sort of risk seems like it would
apply to untrusted
Ethernet devices as well, IIUC. I can imagine it being very difficult
to harden RDMA CM
for this though.
Thanks,
Jake
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [PATCH rdma-next 08/23] RDMA/core: Suppress kverbs_provider for adversarial_mode devices
2026-11-03 19:03 ` Jacob Moroni
@ 2026-11-03 19:07 ` Jason Gunthorpe
0 siblings, 0 replies; 48+ messages in thread
From: Jason Gunthorpe @ 2026-11-03 19:07 UTC (permalink / raw)
To: Jacob Moroni
Cc: Yishai Hadas, leon, linux-rdma, netdev, saeedm, tariqt, mbloch,
kuba, pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe,
maorg
On Tue, Nov 03, 2026 at 02:03:22PM -0500, Jacob Moroni wrote:
> Hi,
>
> Out of curiosity, is the idea that these restrictions will
> eventually be lifted once the ULPs (and I guess the drivers) are
> hardened? Or is it considered infeasible? I am most curious about
> RDMA CM.
I think it depends on people's interest. This is not easy work and I
would insist any attempt be "complete" in that it at least
architecturally defends against a remote compromise.
> Even if the drivers are hardened (no round-tripping WR IDs thru HW
> from WQE->CQE, sanitizing values from FW, etc.), I think it may be
> impossible to protect against certain bogus payload scenarios such
> as a NIC reporting a CQE even though it never wrote the data (or
> wrote bogus data), but this sort of risk seems like it would apply
> to untrusted Ethernet devices as well, IIUC. I can imagine it being
> very difficult to harden RDMA CM for this though.
Yes, the verbs user is exposed to some of the threat as the
completions are no longer reliable. The user can assume the wr_id is
correct, and they only get one completion per wr_id, but the rest of
the completion information can't have any effect on integrity.
Jason
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [PATCH rdma-next 08/23] RDMA/core: Suppress kverbs_provider for adversarial_mode devices
2026-11-03 16:39 ` [PATCH rdma-next 08/23] RDMA/core: Suppress kverbs_provider for adversarial_mode devices Yishai Hadas
2026-11-03 19:03 ` Jacob Moroni
@ 2026-11-07 20:24 ` netdev-bot+sashiko
1 sibling, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:24 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found more severe
issue(s) in other patches of this series. This mail is only an FYI, in case
you want to address the nit picks below when you respin.
Low priority / nit picks for your consideration:
- [Low] At this commit, nothing in the tree writes the new struct
ib_device::adversarial_mode field.
- [Low] Clearing kverbs_provider in ib_device_check_mandatory() for
adversarial_mode devices does more than keep in-kernel ULPs off the…
--
RDMA/core: Suppress kverbs_provider for adversarial_mode devices
This commit adds an adversarial_mode field to struct ib_device. When the
field is set, ib_device_check_mandatory() clears kverbs_provider. After
that, add_client_context() only lets clients with no_kverbs_req (uverbs)
bind to the device, which keeps kernel ULPs off an untrusted NIC.
> diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
> index 29af9fe860e27..eb6aa567d3652 100644
> --- a/drivers/infiniband/core/device.c
> +++ b/drivers/infiniband/core/device.c
> @@ -312,6 +312,14 @@ static void ib_device_check_mandatory(struct ib_device *device)
> break;
> }
> }
> +
> + /*
> + * Kernel ULPs must not bind to an adversarial_mode device. Its kverbs
> + * ops are still present, as uverbs uses them, so clear kverbs_provider
> + * explicitly. uverbs sets no_kverbs_req, so it still binds.
> + */
> + if (device->adversarial_mode)
> + device->kverbs_provider = false;
[Severity: Low]
At this commit, does anything set ib_device->adversarial_mode?
mlx5 has its own module-global flag, set in mlx5_ib_init(), but this
commit does not copy it into the ib_device:
drivers/infiniband/hw/mlx5/main.c:
bool mlx5_ib_adversarial_mode;
...
mlx5_ib_adversarial_mode = mlx5_is_adversarial();
So the field stays zero, this check never fires, and add_client_context()
still binds every kernel client. The commit message says "the kernel ULPs
never bind to the device", which is not true for this commit on its own.
The last patch in the series, "RDMA/mlx5: Advertise adversarial mode via
ib_device::adversarial_mode", adds this line to mlx5_ib_stage_caps_init():
dev->ib_dev.adversarial_mode = mlx5_ib_adversarial_mode;
MLX5_IB_STAGE_CAPS runs before MLX5_IB_STAGE_IB_REG in all mlx5 profiles,
so the gate works once the whole series is applied.
The remaining problem is bisectability. Between "RDMA/mlx5: Block generic
CQ poll path in adversarial mode" and that final patch, kernel ULPs can
still bind in a CoCo guest while mlx5_poll_one() returns -EOPNOTSUPP. In
that window mlx5_ib_poll_cq() reports zero completions to every ULP CQ.
Could the patch that sets ib_dev.adversarial_mode be moved before the one
that blocks the CQ poll path?
[Severity: Low]
This also keeps umad and cma from binding, not just the in-kernel ULPs.
Only uverbs sets no_kverbs_req:
drivers/infiniband/core/uverbs_main.c:
.no_kverbs_req = true,
so add_client_context() returns early for every other client:
if (!device->kverbs_provider && !client->no_kverbs_req)
return 0;
Without umad, no /dev/infiniband/umadN nodes are created for the device.
Without cma, the device never joins cma's device list, so ucma and
librdmacm cannot bind or resolve on it.
This looks intended, since the kernel MAD and CM layers interpret
device-supplied data. Should the commit message say that userspace MAD
and rdma_cm are not available on an adversarial_mode device?
[ ... ]
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 09/23] RDMA/mlx5: Remove duplicate mlx5_ib_umr_context struct
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (7 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 08/23] RDMA/core: Suppress kverbs_provider for adversarial_mode devices Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-03 16:39 ` [PATCH rdma-next 10/23] RDMA/mlx5: Introduce per-CQ poll_one hook in mlx5_ib_cq Yishai Hadas
` (13 subsequent siblings)
22 siblings, 0 replies; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
mlx5_ib_umr_context in mlx5_ib.h and mlx5r_umr_context in umr.h have
identical layouts: an ib_cqe, a status, and a completion. The done
callback used mlx5_ib_umr_context via container_of, while every other
site in umr.c used mlx5r_umr_context.
Remove the duplicate and use mlx5r_umr_context consistently throughout.
No behaviour change.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/hw/mlx5/mlx5_ib.h | 6 ------
drivers/infiniband/hw/mlx5/umr.c | 4 ++--
2 files changed, 2 insertions(+), 8 deletions(-)
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 0f81b8aa487e..980b41800df5 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -749,12 +749,6 @@ struct mlx5_ib_mw {
struct mlx5_ib_mkey mmkey;
};
-struct mlx5_ib_umr_context {
- struct ib_cqe cqe;
- enum ib_wc_status status;
- struct completion done;
-};
-
enum {
MLX5_UMR_STATE_UNINIT,
MLX5_UMR_STATE_ACTIVE,
diff --git a/drivers/infiniband/hw/mlx5/umr.c b/drivers/infiniband/hw/mlx5/umr.c
index 711075e3d9ea..4a3fa79eba3a 100644
--- a/drivers/infiniband/hw/mlx5/umr.c
+++ b/drivers/infiniband/hw/mlx5/umr.c
@@ -339,8 +339,8 @@ static int mlx5r_umr_recover(struct mlx5_ib_dev *dev, u32 mkey,
static void mlx5r_umr_done(struct ib_cq *cq, struct ib_wc *wc)
{
- struct mlx5_ib_umr_context *context =
- container_of(wc->wr_cqe, struct mlx5_ib_umr_context, cqe);
+ struct mlx5r_umr_context *context =
+ container_of(wc->wr_cqe, struct mlx5r_umr_context, cqe);
context->status = wc->status;
complete(&context->done);
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH rdma-next 10/23] RDMA/mlx5: Introduce per-CQ poll_one hook in mlx5_ib_cq
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (8 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 09/23] RDMA/mlx5: Remove duplicate mlx5_ib_umr_context struct Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:24 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 11/23] RDMA/mlx5: Add shadow FIFO infrastructure for UMR completions Yishai Hadas
` (12 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
Add a poll_one function pointer to struct mlx5_ib_cq and initialize it
to mlx5_poll_one in both mlx5_ib_create_cq() (kernel-ULP CQs) and
mlx5_ib_create_user_cq() (uverbs CQs, the path used by both the
modern ioctl-based and legacy write-based CQ creation commands).
mlx5_ib_poll_cq() is updated to dispatch through the pointer instead
of calling mlx5_poll_one() directly.
This is a pure refactor with no behaviour change. The indirection
allows a specialised poll_one implementation to be installed on a
specific CQ, which a follow-up patch uses to verify the UMR CQ against
a hostile NIC.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/hw/mlx5/cq.c | 9 ++++++++-
drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 ++
2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
index 55ffd7d4de51..11c1153747c2 100644
--- a/drivers/infiniband/hw/mlx5/cq.c
+++ b/drivers/infiniband/hw/mlx5/cq.c
@@ -633,7 +633,7 @@ int mlx5_ib_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc)
soft_polled = poll_soft_wc(cq, num_entries, wc, false);
for (npolled = 0; npolled < num_entries - soft_polled; npolled++) {
- if (mlx5_poll_one(cq, &cur_qp, wc + soft_polled + npolled))
+ if (cq->poll_one(cq, &cur_qp, wc + soft_polled + npolled))
break;
}
@@ -1010,6 +1010,8 @@ int mlx5_ib_create_user_cq(struct ib_cq *ibcq,
cq->mcq.comp = mlx5_add_cq_to_tasklet;
cq->mcq.tasklet_ctx.comp = mlx5_ib_cq_comp;
+ /* Set before the CQ is created, see mlx5_ib_create_cq() */
+ cq->poll_one = mlx5_poll_one;
err = mlx5_core_create_cq(dev->mdev, &cq->mcq, cqb, inlen, out, sizeof(out));
if (err)
@@ -1093,6 +1095,11 @@ int mlx5_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
MLX5_SET64(cqc, cqc, dbr_addr, cq->db.dma);
cq->mcq.comp = mlx5_ib_cq_comp;
+ /*
+ * Set before mlx5_core_create_cq(). No real CQE can arrive before a
+ * QP uses the CQ, but a forged one can make mlx5_ib_poll_cq() call it.
+ */
+ cq->poll_one = mlx5_poll_one;
err = mlx5_core_create_cq(dev->mdev, &cq->mcq, cqb, inlen, out,
sizeof(out));
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 980b41800df5..4833f42b3fd1 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -592,6 +592,8 @@ struct mlx5_ib_cq {
enum ib_cq_notify_flags notify_flags;
struct work_struct notify_work;
u16 private_flags; /* Use mlx5_ib_cq_pr_flags */
+ int (*poll_one)(struct mlx5_ib_cq *cq, struct mlx5_ib_qp **cur_qp,
+ struct ib_wc *wc);
};
struct mlx5_ib_wc {
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 10/23] RDMA/mlx5: Introduce per-CQ poll_one hook in mlx5_ib_cq
2026-11-03 16:39 ` [PATCH rdma-next 10/23] RDMA/mlx5: Introduce per-CQ poll_one hook in mlx5_ib_cq Yishai Hadas
@ 2026-11-07 20:24 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:24 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found more severe
issue(s) in other patches of this series. This mail is only an FYI, in case
you want to address the nit picks below when you respin.
Low priority / nit picks for your consideration:
- [Low] The commit message calls this "a pure refactor with no behaviour
change", but it changes the per-CQE fast path.
--
RDMA/mlx5: Introduce per-CQ poll_one hook in mlx5_ib_cq
This adds a poll_one function pointer to struct mlx5_ib_cq. It is
initialized to mlx5_poll_one() in mlx5_ib_create_cq() and
mlx5_ib_create_user_cq(), and mlx5_ib_poll_cq() now dispatches through
it. A follow-up patch uses the hook to install a specialised poll_one for
the UMR CQ.
> diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
> index 55ffd7d4de517..11c1153747c24 100644
> --- a/drivers/infiniband/hw/mlx5/cq.c
> +++ b/drivers/infiniband/hw/mlx5/cq.c
> @@ -633,7 +633,7 @@ int mlx5_ib_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc)
> soft_polled = poll_soft_wc(cq, num_entries, wc, false);
>
> for (npolled = 0; npolled < num_entries - soft_polled; npolled++) {
> - if (mlx5_poll_one(cq, &cur_qp, wc + soft_polled + npolled))
> + if (cq->poll_one(cq, &cur_qp, wc + soft_polled + npolled))
> break;
> }
[Severity: Low]
The commit message says:
This is a pure refactor with no behaviour change.
Does that hold for the per-CQE fast path?
Before this patch, mlx5_poll_one() was static and had one caller, this
loop in mlx5_ib_poll_cq(), so the compiler could inline it.
The patch stores its address in cq->poll_one in mlx5_ib_create_cq() and
mlx5_ib_create_user_cq(). As a result, every polled CQE on every mlx5 CQ
now goes through an indirect call. On kernels with Spectre-v2
mitigations, that call goes through a retpoline thunk.
Only the UMR CQ needs a different implementation, which the later commit
"RDMA/mlx5: Verify UMR CQ polling against a hostile NIC" installs. Even
so, every in-kernel consumer polling an mlx5 CQ (NVMe-oF, iSER, SRP,
NFS/RDMA, SMC-R, RDS, IPoIB) takes the indirect path:
ib_poll_cq()->mlx5_ib_poll_cq()->cq->poll_one()
The end of the series doesn't seem to restore a direct call for the
common case, and there is no INDIRECT_CALL_1() use under
drivers/infiniband/. Would something like this avoid the cost?
if (INDIRECT_CALL_1(cq->poll_one, mlx5_poll_one,
cq, &cur_qp, wc + soft_polled + npolled))
The final mlx5_poll_one() also already has an
unlikely(mlx5_ib_adversarial_mode) check, which a cheap dispatch here
could reuse.
If the indirect call stays, could the commit message mention this
trade-off rather than say there is no behaviour change?
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 11/23] RDMA/mlx5: Add shadow FIFO infrastructure for UMR completions
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (9 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 10/23] RDMA/mlx5: Introduce per-CQ poll_one hook in mlx5_ib_cq Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:24 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 12/23] RDMA/mlx5: Verify UMR CQ polling against a hostile NIC Yishai Hadas
` (11 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
The UMR CQ is polled by the generic path, which finds the completed
request by indexing qp->sq.wrid[] with the CQE wqe_counter. A hostile
NIC can forge it to complete a request other than the one that really
completed, or one that is not pending at all.
The UMR QP is RC and signals every WR, so its completions arrive in
posting order. In adversarial mode, keep a FIFO of the posted UMR
contexts, so a follow-up patch can check each CQE against the oldest
pending request instead of trusting it. Each context also records the
SQ producer counter it was posted at, as its address alone doesn't
identify the post: it is usually on the stack and reused across posts.
Nothing consumes the FIFO yet.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/hw/mlx5/mlx5_ib.h | 10 +++++++
drivers/infiniband/hw/mlx5/umr.c | 42 +++++++++++++++++++++++-----
drivers/infiniband/hw/mlx5/umr.h | 6 ++++
3 files changed, 51 insertions(+), 7 deletions(-)
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 4833f42b3fd1..9cc4b2f99b69 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -758,6 +758,12 @@ enum {
MLX5_UMR_STATE_ERR,
};
+#define MLX5_UMR_MAX_WR 128
+static_assert((MLX5_UMR_MAX_WR & (MLX5_UMR_MAX_WR - 1)) == 0,
+ "MLX5_UMR_MAX_WR must be a power of two for FIFO index masking");
+
+struct mlx5r_umr_context;
+
struct umr_common {
struct ib_pd *pd;
struct ib_cq *cq;
@@ -771,6 +777,10 @@ struct umr_common {
unsigned int state;
/* Protects from repeat UMR QP creation */
struct mutex init_lock;
+ /* Posted UMR contexts in posting order, adversarial_mode only */
+ struct mlx5r_umr_context **pending;
+ u32 pending_head;
+ u32 pending_tail;
};
struct mlx5_ib_port_resources {
diff --git a/drivers/infiniband/hw/mlx5/umr.c b/drivers/infiniband/hw/mlx5/umr.c
index 4a3fa79eba3a..aba79a19d627 100644
--- a/drivers/infiniband/hw/mlx5/umr.c
+++ b/drivers/infiniband/hw/mlx5/umr.c
@@ -98,9 +98,7 @@ static int umr_check_mkey_mask(struct mlx5_ib_dev *dev, u64 mask)
return 0;
}
-enum {
- MAX_UMR_WR = 128,
-};
+#define MAX_UMR_WR MLX5_UMR_MAX_WR
static int mlx5r_umr_qp_rst2rts(struct mlx5_ib_dev *dev, struct ib_qp *qp)
{
@@ -181,6 +179,16 @@ int mlx5r_umr_resource_init(struct mlx5_ib_dev *dev)
if (ret)
goto destroy_qp;
+ if (mlx5_ib_adversarial_mode) {
+ dev->umrc.pending = kcalloc(MAX_UMR_WR,
+ sizeof(*dev->umrc.pending),
+ GFP_KERNEL);
+ if (!dev->umrc.pending) {
+ ret = -ENOMEM;
+ goto destroy_qp;
+ }
+ }
+
dev->umrc.cq = cq;
sema_init(&dev->umrc.sem, MAX_UMR_WR);
@@ -208,6 +216,7 @@ void mlx5r_umr_resource_cleanup(struct mlx5_ib_dev *dev)
/* After device init, UMR cp/qp are not unset during the lifetime. */
ib_destroy_qp(dev->umrc.qp);
ib_free_cq(dev->umrc.cq);
+ kfree(dev->umrc.pending);
}
int mlx5r_umr_init(struct mlx5_ib_dev *dev)
@@ -236,7 +245,8 @@ void mlx5r_umr_cleanup(struct mlx5_ib_dev *dev)
}
-static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey, struct ib_cqe *cqe,
+static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey,
+ struct mlx5r_umr_context *ctx,
struct mlx5r_umr_wqe *wqe, bool with_data)
{
unsigned int wqe_size =
@@ -245,6 +255,7 @@ static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey, struct ib_cqe *cqe,
sizeof(struct mlx5_wqe_data_seg);
struct mlx5_ib_dev *dev = to_mdev(ibqp->device);
struct mlx5_core_dev *mdev = dev->mdev;
+ struct umr_common *umrc = &dev->umrc;
struct mlx5_ib_qp *qp = to_mqp(ibqp);
struct mlx5_wqe_ctrl_seg *ctrl;
union {
@@ -267,13 +278,30 @@ static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey, struct ib_cqe *cqe,
goto out;
qp->sq.wr_data[idx] = MLX5_IB_WR_UMR;
+ if (mlx5_ib_adversarial_mode) {
+ /* mlx5r_finish_wqe() advances cur_post, so record it first */
+ ctx->wqe_ctr = qp->sq.cur_post;
+ }
mlx5r_memcpy_send_wqe(&qp->sq, &cur_edge, &seg, &size, wqe, wqe_size);
- id.ib_cqe = cqe;
+ id.ib_cqe = &ctx->cqe;
mlx5r_finish_wqe(qp, ctrl, seg, size, cur_edge, idx, id.wr_id, 0,
MLX5_FENCE_MODE_INITIATOR_SMALL, MLX5_OPCODE_UMR);
+ /*
+ * Push to the shadow FIFO before the doorbell, so the entry exists
+ * before a CQE for it can arrive. A hostile NIC can post a CQE before
+ * the doorbell, so the UMR CQ poller, on another CPU, can't rely on
+ * the doorbell to see ctx->wqe_ctr and ctx->cqe.done with the entry.
+ * Publish it with a release, which pairs with the poller's acquire.
+ */
+ if (mlx5_ib_adversarial_mode) {
+ smp_store_release(&umrc->pending[umrc->pending_tail & (MAX_UMR_WR - 1)],
+ ctx);
+ umrc->pending_tail++;
+ }
+
mlx5r_ring_db(qp, 1, ctrl);
out:
@@ -303,7 +331,7 @@ static int mlx5r_umr_recover(struct mlx5_ib_dev *dev, u32 mkey,
* This will ensure that all the previous WRs got a completion before
* we set the QP state to RESET.
*/
- err = mlx5r_umr_post_send(umrc->qp, mkey, &umr_context->cqe, wqe,
+ err = mlx5r_umr_post_send(umrc->qp, mkey, umr_context, wqe,
with_data);
if (err) {
mlx5_ib_warn(dev, "UMR recovery post send failed, err %d\n", err);
@@ -380,7 +408,7 @@ static int mlx5r_umr_post_send_wait(struct mlx5_ib_dev *dev, u32 mkey,
continue;
}
- err = mlx5r_umr_post_send(umrc->qp, mkey, &umr_context.cqe, wqe,
+ err = mlx5r_umr_post_send(umrc->qp, mkey, &umr_context, wqe,
with_data);
mutex_unlock(&umrc->lock);
if (err) {
diff --git a/drivers/infiniband/hw/mlx5/umr.h b/drivers/infiniband/hw/mlx5/umr.h
index bda7123781a9..0b245eec521f 100644
--- a/drivers/infiniband/hw/mlx5/umr.h
+++ b/drivers/infiniband/hw/mlx5/umr.h
@@ -84,6 +84,12 @@ struct mlx5r_umr_context {
struct ib_cqe cqe;
enum ib_wc_status status;
struct completion done;
+ /*
+ * SQ producer counter the WQE was posted at, matched against the CQE
+ * wqe_counter in adversarial_mode. Unlike the context address, which
+ * is reused across posts, or the ring slot, it identifies the WQE.
+ */
+ u16 wqe_ctr;
};
struct mlx5r_umr_wqe {
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 11/23] RDMA/mlx5: Add shadow FIFO infrastructure for UMR completions
2026-11-03 16:39 ` [PATCH rdma-next 11/23] RDMA/mlx5: Add shadow FIFO infrastructure for UMR completions Yishai Hadas
@ 2026-11-07 20:24 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:24 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found more severe
issue(s) in other patches of this series. This mail is only an FYI, in case
you want to address the nit picks below when you respin.
Low priority / nit picks for your consideration:
- [Low] The commit message's design premise is wrong.
--
RDMA/mlx5: Add shadow FIFO infrastructure for UMR completions
This commit adds a shadow FIFO of posted UMR contexts in adversarial
mode. A later patch can then check each UMR CQE against the oldest
pending request instead of trusting the CQE wqe_counter. Each context
also records the SQ producer counter it was posted at.
> The UMR QP is RC and signals every WR, so its completions arrive in
> posting order.
[Severity: Low]
Is this description accurate? The UMR QP does not look like an RC QP.
mlx5r_umr_resource_init() in drivers/infiniband/hw/mlx5/umr.c creates it
as a REG_UMR QP:
drivers/infiniband/hw/mlx5/umr.c:mlx5r_umr_resource_init() {
...
init_attr.qp_type = MLX5_IB_QPT_REG_UMR;
...
}
to_mlx5_st() in drivers/infiniband/hw/mlx5/qp.c maps that type to its
own service type, not MLX5_QP_ST_RC:
case MLX5_IB_QPT_REG_UMR: return MLX5_QP_ST_REG_UMR;
mlx5r_umr_qp_rst2rts() also moves the QP to RTS without a remote QPN or
a path, which an RC connection would need.
The shadow FIFO design depends on in-order completion. The consumer added
later in the series, mlx5_ib_poll_one_umr(), treats any CQE whose
wqe_counter differs from the FIFO head as forged.
The in-order conclusion is probably still correct, since mlx5 send queues
complete in order and the generic poll path already relies on that when
it sets wq->tail = wq->wqe_head[idx] + 1. Only the stated reason seems
wrong.
Could the message say that the UMR SQ (a REG_UMR service type QP)
completes in posting order, instead of calling the QP RC?
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 12/23] RDMA/mlx5: Verify UMR CQ polling against a hostile NIC
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (10 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 11/23] RDMA/mlx5: Add shadow FIFO infrastructure for UMR completions Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:24 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 13/23] RDMA/mlx5: Keep the UMR QP failed once a forged CQE is detected Yishai Hadas
` (10 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
The generic poll path trusts the CQE content: its opcode, QPN and
wqe_counter decide which request is completed and which waiter is woken.
A forged CQE can therefore complete a request the driver never posted,
or one that already completed. The driver then writes through a context
it no longer owns, which corrupts memory, as the context is usually on
the waiter's stack.
In adversarial mode, install a dedicated poll_one for the UMR CQ that
does not take the CQE's word for it. The driver keeps a shadow FIFO of
its posted requests, and a CQE is accepted only if its opcode and QPN
match the UMR QP and its full wqe_counter matches the oldest pending
request, one already passed to the doorbell. The context is then taken
from the FIFO, never from the CQE. This cannot prove that the NIC
really executed the request, as it can always report the oldest one as
completed; it ensures that the driver completes only a context it owns,
once and in order.
A CQE that fails validation proves the NIC is forging completions, so
nothing it reports on this QP can be trusted any more. Mark the UMR
path permanently failed, so neither a new post nor a QP recovery can
reopen it, stop interpreting this CQ and force the mlx5_core health
recovery. The forced variant is needed because such a NIC still looks
healthy to the fatal sensors. Recovery moves the device to the internal
error state, and the existing flush then completes the pending requests
in software and wakes their waiters, so this path does not complete
them itself.
On teardown, stop polling the CQ before destroying the UMR QP, so a CQE
for the destroyed QP, or for a reused QPN, is never interpreted.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/hw/mlx5/cq.c | 156 +++++++++++++++++++++++++++
drivers/infiniband/hw/mlx5/mlx5_ib.h | 5 +
drivers/infiniband/hw/mlx5/umr.c | 38 +++++++
3 files changed, 199 insertions(+)
diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
index 11c1153747c2..1a98263ec311 100644
--- a/drivers/infiniband/hw/mlx5/cq.c
+++ b/drivers/infiniband/hw/mlx5/cq.c
@@ -37,6 +37,7 @@
#include "mlx5_ib.h"
#include "srq.h"
#include "qp.h"
+#include "umr.h"
#define UVERBS_MODULE_NAME mlx5_ib
#include <rdma/uverbs_named_ioctl.h>
@@ -608,6 +609,161 @@ static int poll_soft_wc(struct mlx5_ib_cq *cq, int num_entries,
return npolled;
}
+static int mlx5_ib_poll_one_stopped(struct mlx5_ib_cq *cq,
+ struct mlx5_ib_qp **cur_qp, struct ib_wc *wc)
+{
+ return -EAGAIN;
+}
+
+/**
+ * mlx5_ib_cq_stop_polling - stop interpreting CQEs on a CQ
+ * @ibcq: the CQ
+ *
+ * Takes cq->lock, which mlx5_ib_poll_cq() holds while polling, so no poll
+ * is in flight on return and no CQE is looked at again.
+ */
+void mlx5_ib_cq_stop_polling(struct ib_cq *ibcq)
+{
+ struct mlx5_ib_cq *cq = to_mcq(ibcq);
+ unsigned long flags;
+
+ spin_lock_irqsave(&cq->lock, flags);
+ cq->poll_one = mlx5_ib_poll_one_stopped;
+ spin_unlock_irqrestore(&cq->lock, flags);
+}
+
+/*
+ * poll_one of the UMR CQ in adversarial_mode. A CQE is accepted only if it
+ * matches the oldest request in the shadow FIFO, otherwise the device is
+ * treated as compromised.
+ */
+int mlx5_ib_poll_one_umr(struct mlx5_ib_cq *cq, struct mlx5_ib_qp **cur_qp,
+ struct ib_wc *wc)
+{
+ struct mlx5_ib_dev *dev = to_mdev(cq->ibcq.device);
+ struct mlx5r_umr_context *expected;
+ struct umr_common *umrc = &dev->umrc;
+ struct mlx5_err_cqe err_cqe;
+ struct mlx5_cqe64 *cqe64;
+ struct mlx5_ib_wq *wq;
+ struct mlx5_ib_qp *qp;
+ struct ib_qp *ibqp;
+ u16 wqe_ctr;
+ u8 opcode;
+ u32 head;
+ u32 sop_qpn;
+ u32 qpn;
+ u8 sop;
+ int idx;
+ void *cqe;
+
+ cqe = next_cqe_sw(cq);
+ if (!cqe)
+ return -EAGAIN;
+
+ cqe64 = (cq->mcq.cqe_sz == 64) ? cqe : cqe + 64;
+ ++cq->mcq.cons_index;
+ /*
+ * Read each field once, so a NIC rewriting the CQE cannot change it
+ * between validation and use.
+ */
+ rmb();
+ opcode = READ_ONCE(cqe64->op_own) >> 4;
+ sop_qpn = ntohl(READ_ONCE(cqe64->sop_drop_qpn));
+ qpn = sop_qpn & 0xffffff;
+ sop = sop_qpn >> 24;
+ wqe_ctr = be16_to_cpu(READ_ONCE(cqe64->wqe_counter));
+
+ /*
+ * This poll_one is installed before the UMR QP is published, so a CQE
+ * seen while the QP is NULL is forged. Teardown stops polling before
+ * the QP is destroyed.
+ */
+ ibqp = smp_load_acquire(&dev->umrc.qp);
+ if (unlikely(!ibqp)) {
+ mlx5_ib_err(dev, "CQE on the UMR CQ before its QP is published\n");
+ goto forged_cqe;
+ }
+ qp = to_mqp(ibqp);
+ wq = &qp->sq;
+ idx = wqe_ctr & (wq->wqe_cnt - 1);
+
+ if (unlikely((opcode != MLX5_CQE_REQ && opcode != MLX5_CQE_REQ_ERR) ||
+ qpn != umrc->qpn || sop != MLX5_OPCODE_UMR)) {
+ mlx5_ib_err(dev,
+ "Unexpected CQE on UMR CQ: opcode 0x%x sop 0x%x qpn 0x%x\n",
+ opcode, sop, qpn);
+ goto forged_cqe;
+ }
+
+ /* Pairs with smp_store_release() in mlx5r_umr_post_send() */
+ head = umrc->pending_head;
+ expected = smp_load_acquire(&umrc->pending[head & (MLX5_UMR_MAX_WR - 1)]);
+
+ if (unlikely(!expected)) {
+ mlx5_ib_err(dev,
+ "UMR CQE wqe_ctr 0x%x with an empty shadow FIFO, possible NIC forgery\n",
+ wqe_ctr);
+ goto forged_cqe;
+ }
+
+ if (unlikely(wqe_ctr != expected->wqe_ctr)) {
+ mlx5_ib_err(dev,
+ "UMR wqe_ctr 0x%x mismatch with expected context, possible NIC forgery\n",
+ wqe_ctr);
+ goto forged_cqe;
+ }
+
+ /*
+ * A WQE records wq->head when posted, and mlx5r_ring_db() then advances
+ * wq->head. If they are still equal, mlx5r_ring_db() was not reached
+ * yet, so the WQE was never handed to the NIC.
+ */
+ if (unlikely(wq->wqe_head[idx] == READ_ONCE(wq->head))) {
+ mlx5_ib_err(dev,
+ "UMR CQE wqe_ctr 0x%x for an uncommitted WQE, possible NIC forgery\n",
+ wqe_ctr);
+ goto forged_cqe;
+ }
+
+ /* Pop the shadow FIFO head: validated, dispatch to its waiter */
+ WRITE_ONCE(umrc->pending[head & (MLX5_UMR_MAX_WR - 1)], NULL);
+ umrc->pending_head = head + 1;
+ wc->wr_cqe = &expected->cqe;
+
+ wc->qp = &qp->ibqp;
+ wq->tail = wq->wqe_head[idx] + 1;
+ /* Advance wq->last_poll, the internal error flush starts from it */
+ handle_atomics(qp, cqe64, wq->last_poll, idx);
+
+ if (opcode == MLX5_CQE_REQ) {
+ wc->status = IB_WC_SUCCESS;
+ } else {
+ /* Copy the CQE, so its syndrome is read once */
+ err_cqe = *(struct mlx5_err_cqe *)cqe64;
+ mlx5_handle_error_cqe(dev, &err_cqe, wc);
+ /* As in the generic path, a real failure starts the recovery */
+ if (wc->status != IB_WC_WR_FLUSH_ERR)
+ umrc->state = MLX5_UMR_STATE_RECOVER;
+ }
+
+ return 0;
+
+forged_cqe:
+ /* mlx5_ib_poll_cq() updates the CI only for a returned completion */
+ mlx5_cq_set_ci(&cq->mcq);
+ /*
+ * The NIC forged a CQE. Fail the UMR path for good, ignore any later
+ * CQE and force the health recovery, as the fatal sensors still read
+ * clean. The internal error flush then completes the waiters. Return
+ * an error, not -EAGAIN, as a CQE was consumed.
+ */
+ umrc->state = MLX5_UMR_STATE_ERR;
+ cq->poll_one = mlx5_ib_poll_one_stopped;
+ mlx5_trigger_health_work_force(dev->mdev);
+ return -EIO;
+}
+
int mlx5_ib_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc)
{
struct mlx5_ib_cq *cq = to_mcq(ibcq);
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 9cc4b2f99b69..f7c718e3d974 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -768,6 +768,8 @@ struct umr_common {
struct ib_pd *pd;
struct ib_cq *cq;
struct ib_qp *qp;
+ /* Checked against each CQE in adversarial_mode */
+ u32 qpn;
/* Protects from UMR QP overflow
*/
struct semaphore sem;
@@ -1311,6 +1313,9 @@ int mlx5_ib_create_user_cq(struct ib_cq *ibcq,
struct uverbs_attr_bundle *attrs);
int mlx5_ib_destroy_cq(struct ib_cq *cq, struct ib_udata *udata);
int mlx5_ib_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc);
+int mlx5_ib_poll_one_umr(struct mlx5_ib_cq *cq, struct mlx5_ib_qp **cur_qp,
+ struct ib_wc *wc);
+void mlx5_ib_cq_stop_polling(struct ib_cq *ibcq);
int mlx5_ib_pre_destroy_cq(struct ib_cq *cq);
void mlx5_ib_post_destroy_cq(struct ib_cq *cq);
int mlx5_ib_arm_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags);
diff --git a/drivers/infiniband/hw/mlx5/umr.c b/drivers/infiniband/hw/mlx5/umr.c
index aba79a19d627..b0f00d4bc93c 100644
--- a/drivers/infiniband/hw/mlx5/umr.c
+++ b/drivers/infiniband/hw/mlx5/umr.c
@@ -194,6 +194,23 @@ int mlx5r_umr_resource_init(struct mlx5_ib_dev *dev)
sema_init(&dev->umrc.sem, MAX_UMR_WR);
mutex_init(&dev->umrc.lock);
dev->umrc.state = MLX5_UMR_STATE_ACTIVE;
+
+ if (mlx5_ib_adversarial_mode) {
+ struct mlx5_ib_cq *mcq = to_mcq(cq);
+ unsigned long flags;
+
+ dev->umrc.qpn = qp->qp_num;
+ /*
+ * UMRs can be posted as soon as the QP is published below, by
+ * this caller or by a concurrent MR registration, so install
+ * mlx5_ib_poll_one_umr() first. The CQ is already armed, so
+ * swap it under cq->lock, which the poll loop holds.
+ */
+ spin_lock_irqsave(&mcq->lock, flags);
+ mcq->poll_one = mlx5_ib_poll_one_umr;
+ spin_unlock_irqrestore(&mcq->lock, flags);
+ }
+
smp_store_release(&dev->umrc.qp, qp);
mutex_unlock(&dev->umrc.init_lock);
@@ -212,6 +229,14 @@ void mlx5r_umr_resource_cleanup(struct mlx5_ib_dev *dev)
{
if (dev->umrc.state == MLX5_UMR_STATE_UNINIT)
return;
+
+ /*
+ * Stop polling before the QP is destroyed, as its QPN may then be
+ * reused and a forged CQE carrying it must not be interpreted.
+ */
+ if (mlx5_ib_adversarial_mode)
+ mlx5_ib_cq_stop_polling(dev->umrc.cq);
+
mutex_destroy(&dev->umrc.lock);
/* After device init, UMR cp/qp are not unset during the lifetime. */
ib_destroy_qp(dev->umrc.qp);
@@ -319,6 +344,11 @@ static int mlx5r_umr_recover(struct mlx5_ib_dev *dev, u32 mkey,
int err;
mutex_lock(&umrc->lock);
+ if (mlx5_ib_adversarial_mode && umrc->state == MLX5_UMR_STATE_ERR) {
+ /* The NIC forged a CQE, don't revive the QP */
+ mutex_unlock(&umrc->lock);
+ return -EFAULT;
+ }
/* Preventing any further WRs to be sent now */
if (umrc->state != MLX5_UMR_STATE_RECOVER) {
mlx5_ib_warn(dev, "UMR recovery encountered an unexpected state=%d\n",
@@ -425,6 +455,14 @@ static int mlx5r_umr_post_send_wait(struct mlx5_ib_dev *dev, u32 mkey,
if (umr_context.status == IB_WC_WR_FLUSH_ERR)
continue;
+ if (mlx5_ib_adversarial_mode) {
+ /* A forged CQE was detected, don't recover the QP */
+ if (umrc->state == MLX5_UMR_STATE_ERR) {
+ err = -EFAULT;
+ break;
+ }
+ }
+
WARN_ON_ONCE(1);
mlx5_ib_warn(dev,
"reg umr failed (%u). Trying to recover and resubmit the flushed WQEs, mkey = %u\n",
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 12/23] RDMA/mlx5: Verify UMR CQ polling against a hostile NIC
2026-11-03 16:39 ` [PATCH rdma-next 12/23] RDMA/mlx5: Verify UMR CQ polling against a hostile NIC Yishai Hadas
@ 2026-11-07 20:24 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:24 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found 5 potential
issue(s) to consider.
Critical: 1 · High: 0 · Medium: 2 · Low: 2
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [Critical] RDMA/mlx5: unvalidated error CQE syndromes in
mlx5_ib_poll_one_umr()
- [Medium] When a forged CQE is detected, the waiters are only woken if
the forced health work runs, and on the unload and remove paths it…
- [Medium] mlx5_ib_poll_one_umr() treats a CQE as forged if the top byte
of sop_drop_qpn is not MLX5_OPCODE_UMR.
- [Low] After the forged-CQE path sets cq->poll_one =
mlx5_ib_poll_one_stopped, the UMR CQ's consumer index never advances…
- [Low] The kernel-doc of mlx5_ib_cq_stop_polling() says that on return
'no CQE is looked at again'.
--
RDMA/mlx5: Verify UMR CQ polling against a hostile NIC
In adversarial mode, this installs a dedicated poll_one for the UMR CQ.
A CQE is accepted only if its opcode, QPN and wqe_counter match the oldest
entry in a shadow FIFO of posted UMR requests, and the completion context
is then taken from that FIFO. A CQE that fails validation marks the UMR
path permanently failed, stops polling the CQ and forces the mlx5_core
health recovery; teardown also stops polling before the UMR QP is
destroyed.
> diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
> index 11c1153747c24..1a98263ec3111 100644
> --- a/drivers/infiniband/hw/mlx5/cq.c
> +++ b/drivers/infiniband/hw/mlx5/cq.c
[ ... ]
> @@ -608,6 +609,161 @@ static int poll_soft_wc(struct mlx5_ib_cq *cq, int num_entries,
> return npolled;
> }
>
> +static int mlx5_ib_poll_one_stopped(struct mlx5_ib_cq *cq,
> + struct mlx5_ib_qp **cur_qp, struct ib_wc *wc)
> +{
> + return -EAGAIN;
> +}
[Severity: Low]
After the forged_cqe path in mlx5_ib_poll_one_umr() installs this
function, can the UMR CQ get stuck in a re-arm loop?
mlx5_ib_poll_one_stopped() never advances cq->mcq.cons_index. Other UMR
WQEs, up to MAX_UMR_WR of them, can still be outstanding, and their real
CQEs land past that index. ib_poll_handler() sees 0 completions and
re-arms the CQ:
drivers/infiniband/core/cq.c:ib_poll_handler() {
...
if (completed < budget) {
irq_poll_complete(&cq->iop);
if (ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0) {
...
}
mlx5_ib_arm_cq() then arms it with the stale consumer index:
drivers/infiniband/hw/mlx5/cq.c:mlx5_ib_arm_cq() {
...
mlx5_cq_arm(&cq->mcq,
(flags & IB_CQ_SOLICITED_MASK) == IB_CQ_SOLICITED ?
MLX5_CQ_DB_REQ_NOT_SOL : MLX5_CQ_DB_REQ_NOT,
uar_page, to_mcq(ibcq)->mcq.cons_index);
...
}
With mlx5 arm semantics, an armed CQ that still holds unconsumed CQEs
raises an event right away. Each re-arm would then produce another EQE,
another interrupt and another irq_poll pass that does nothing. Does this
continue until the forced health work tears the device down?
In the unload windows described further below, that work never runs, so
the loop would not stop.
> +
> +/**
> + * mlx5_ib_cq_stop_polling - stop interpreting CQEs on a CQ
> + * @ibcq: the CQ
> + *
> + * Takes cq->lock, which mlx5_ib_poll_cq() holds while polling, so no poll
> + * is in flight on return and no CQE is looked at again.
> + */
[Severity: Low]
Is "no CQE is looked at again" accurate? The commit message makes a
similar claim:
On teardown, stop polling the CQ before destroying the UMR QP, so a CQE
for the destroyed QP, or for a reused QPN, is never interpreted.
mlx5r_umr_resource_cleanup() calls mlx5_ib_cq_stop_polling() and then
ib_destroy_qp(dev->umrc.qp). For a kernel QP that path reaches:
ib_destroy_qp()
destroy_qp_common()
__mlx5_ib_cq_clean(recv_cq, base->mqp.qpn, ...)
__mlx5_ib_cq_clean() scans every software-owned CQE on the UMR CQ,
starting at cons_index. It reads each QPN and compacts the remaining
entries, without going through cq->poll_one.
The same thing happens in mlx5r_umr_recover() during the RESET
transition:
mlx5r_umr_recover()
ib_modify_qp(RESET)
__mlx5_ib_modify_qp()
mlx5_ib_cq_clean(recv_cq, base->mqp.qpn, ...)
On that path, any leftover CQE carrying the UMR QPN is silently dropped.
It is never checked against the shadow FIFO and never reaches the
forged_cqe handling.
There is no memory-safety impact here, because __mlx5_ib_cq_clean() is
bounded by cq->ibcq.cqe and dispatches no wr_cqe. Could the comment and
the commit message be changed to describe only what the code actually
guarantees?
> +void mlx5_ib_cq_stop_polling(struct ib_cq *ibcq)
> +{
[ ... ]
> +int mlx5_ib_poll_one_umr(struct mlx5_ib_cq *cq, struct mlx5_ib_qp **cur_qp,
> + struct ib_wc *wc)
> +{
[ ... ]
> + qp = to_mqp(ibqp);
> + wq = &qp->sq;
> + idx = wqe_ctr & (wq->wqe_cnt - 1);
> +
> + if (unlikely((opcode != MLX5_CQE_REQ && opcode != MLX5_CQE_REQ_ERR) ||
> + qpn != umrc->qpn || sop != MLX5_OPCODE_UMR)) {
> + mlx5_ib_err(dev,
> + "Unexpected CQE on UMR CQ: opcode 0x%x sop 0x%x qpn 0x%x\n",
> + opcode, sop, qpn);
> + goto forged_cqe;
> + }
[Severity: Medium]
Could this check flag a legitimate UMR error recovery as forgery?
The sop check also runs on MLX5_CQE_REQ_ERR CQEs, including WR_FLUSH_ERR
ones, before the syndrome is examined. In the error CQE layout, this byte
is the top byte of s_wqe_opcode_qpn in struct mlx5_err_cqe.
Elsewhere in the tree, handle_good_req() reads it only for successful REQ
CQEs. Nothing reads s_wqe_opcode for error or flush CQEs.
The device may not report the UMR opcode in flush CQEs for WQEs it never
executed. Examples are the WQEs queued behind a failed UMR and the
barrier WR posted by mlx5r_umr_recover().
In that case, an honest NIC that hits a recoverable error such as
LOCAL_PROT_ERR would get the UMR path disabled for good, with -EFAULT for
every later UMR, plus a forced health recovery.
This depends on hardware behavior that nothing in the tree confirms or
rules out. Would it be safer to apply the sop check only to MLX5_CQE_REQ,
and rely on the full wqe_counter match against the shadow FIFO for error
CQEs?
[ ... ]
> +forged_cqe:
> + /* mlx5_ib_poll_cq() updates the CI only for a returned completion */
> + mlx5_cq_set_ci(&cq->mcq);
> + /*
> + * The NIC forged a CQE. Fail the UMR path for good, ignore any later
> + * CQE and force the health recovery, as the fatal sensors still read
> + * clean. The internal error flush then completes the waiters. Return
> + * an error, not -EAGAIN, as a CQE was consumed.
> + */
> + umrc->state = MLX5_UMR_STATE_ERR;
> + cq->poll_one = mlx5_ib_poll_one_stopped;
> + mlx5_trigger_health_work_force(dev->mdev);
> + return -EIO;
> +}
[Severity: Medium]
What wakes the pending UMR waiters if the forced health work cannot run?
The commit message says:
Recovery moves the device to the internal error state, and the existing
flush then completes the pending requests in software and wakes their
waiters, so this path does not complete them itself.
The forced work ends up in mlx5_fw_fatal_reporter_err() in
drivers/net/ethernet/mellanox/mlx5/core/health.c:
mlx5_fw_fatal_reporter_err() {
...
mutex_lock(&dev->intf_state_mutex);
if (test_bit(MLX5_DROP_HEALTH_WORK, &health->flags)) {
mlx5_core_err(dev, "health works are not permitted at this stage\n");
mutex_unlock(&dev->intf_state_mutex);
return;
}
...
}
Both checks stop it during teardown:
- remove_one() calls mlx5_drain_health_wq(), which sets
MLX5_DROP_HEALTH_WORK, before it calls mlx5_uninit_one().
- mlx5_uninit_one() and mlx5_unload_one_devl_locked() hold
intf_state_mutex while mlx5_unregister_device() or
mlx5_detach_device() deletes the mlx5_ib aux device. mlx5r_driver has
no .suspend, so the device is always deleted.
In that window, ib_unregister_device() destroys user MRs, and pooled or
cacheable mkeys are revoked through UMR:
mlx5r_handle_mkey_cleanup()
mlx5_umr_revoke_mr_with_lock()
mlx5r_umr_revoke_mr()
mlx5r_umr_post_send_wait()
wait_for_completion(&umr_context.done);
Suppose a forged CQE is detected during this window. The UMR CQ is never
polled again and the device never enters the internal error state. Is
anything left that can complete this uninterruptible wait?
In the intf_state_mutex case, isn't this a deadlock?
- Unload holds intf_state_mutex and the devlink lock while it waits for
the UMR.
- The UMR waits for the forced health work.
- The forced health work waits for intf_state_mutex.
[ ... ]
> @@ -608,6 +609,161 @@ static int poll_soft_wc(struct mlx5_ib_cq *cq, int num_entries,
> return npolled;
> }
>
> +static int mlx5_ib_poll_one_stopped(struct mlx5_ib_cq *cq,
> + struct mlx5_ib_qp **cur_qp, struct ib_wc *wc)
> +{
> + return -EAGAIN;
> +}
> +
> +/**
> + * mlx5_ib_cq_stop_polling - stop interpreting CQEs on a CQ
> + * @ibcq: the CQ
> + *
> + * Takes cq->lock, which mlx5_ib_poll_cq() holds while polling, so no poll
> + * is in flight on return and no CQE is looked at again.
> + */
> +void mlx5_ib_cq_stop_polling(struct ib_cq *ibcq)
> +{
> + struct mlx5_ib_cq *cq = to_mcq(ibcq);
> + unsigned long flags;
> +
> + spin_lock_irqsave(&cq->lock, flags);
> + cq->poll_one = mlx5_ib_poll_one_stopped;
> + spin_unlock_irqrestore(&cq->lock, flags);
> +}
> +
> +/*
> + * poll_one of the UMR CQ in adversarial_mode. A CQE is accepted only if it
> + * matches the oldest request in the shadow FIFO, otherwise the device is
> + * treated as compromised.
> + */
> +int mlx5_ib_poll_one_umr(struct mlx5_ib_cq *cq, struct mlx5_ib_qp **cur_qp,
> + struct ib_wc *wc)
> +{
> + struct mlx5_ib_dev *dev = to_mdev(cq->ibcq.device);
> + struct mlx5r_umr_context *expected;
> + struct umr_common *umrc = &dev->umrc;
> + struct mlx5_err_cqe err_cqe;
> + struct mlx5_cqe64 *cqe64;
> + struct mlx5_ib_wq *wq;
> + struct mlx5_ib_qp *qp;
> + struct ib_qp *ibqp;
> + u16 wqe_ctr;
> + u8 opcode;
> + u32 head;
> + u32 sop_qpn;
> + u32 qpn;
> + u8 sop;
> + int idx;
> + void *cqe;
> +
> + cqe = next_cqe_sw(cq);
> + if (!cqe)
> + return -EAGAIN;
> +
> + cqe64 = (cq->mcq.cqe_sz == 64) ? cqe : cqe + 64;
> + ++cq->mcq.cons_index;
> + /*
> + * Read each field once, so a NIC rewriting the CQE cannot change it
> + * between validation and use.
> + */
> + rmb();
> + opcode = READ_ONCE(cqe64->op_own) >> 4;
> + sop_qpn = ntohl(READ_ONCE(cqe64->sop_drop_qpn));
> + qpn = sop_qpn & 0xffffff;
> + sop = sop_qpn >> 24;
> + wqe_ctr = be16_to_cpu(READ_ONCE(cqe64->wqe_counter));
> +
> + /*
> + * This poll_one is installed before the UMR QP is published, so a CQE
> + * seen while the QP is NULL is forged. Teardown stops polling before
> + * the QP is destroyed.
> + */
> + ibqp = smp_load_acquire(&dev->umrc.qp);
> + if (unlikely(!ibqp)) {
> + mlx5_ib_err(dev, "CQE on the UMR CQ before its QP is published\n");
> + goto forged_cqe;
> + }
> + qp = to_mqp(ibqp);
> + wq = &qp->sq;
> + idx = wqe_ctr & (wq->wqe_cnt - 1);
> +
> + if (unlikely((opcode != MLX5_CQE_REQ && opcode != MLX5_CQE_REQ_ERR) ||
> + qpn != umrc->qpn || sop != MLX5_OPCODE_UMR)) {
> + mlx5_ib_err(dev,
> + "Unexpected CQE on UMR CQ: opcode 0x%x sop 0x%x qpn 0x%x\n",
> + opcode, sop, qpn);
> + goto forged_cqe;
> + }
> +
> + /* Pairs with smp_store_release() in mlx5r_umr_post_send() */
> + head = umrc->pending_head;
> + expected = smp_load_acquire(&umrc->pending[head & (MLX5_UMR_MAX_WR - 1)]);
> +
> + if (unlikely(!expected)) {
> + mlx5_ib_err(dev,
> + "UMR CQE wqe_ctr 0x%x with an empty shadow FIFO, possible NIC forgery\n",
> + wqe_ctr);
> + goto forged_cqe;
> + }
> +
> + if (unlikely(wqe_ctr != expected->wqe_ctr)) {
> + mlx5_ib_err(dev,
> + "UMR wqe_ctr 0x%x mismatch with expected context, possible NIC forgery\n",
> + wqe_ctr);
> + goto forged_cqe;
> + }
> +
> + /*
> + * A WQE records wq->head when posted, and mlx5r_ring_db() then advances
> + * wq->head. If they are still equal, mlx5r_ring_db() was not reached
> + * yet, so the WQE was never handed to the NIC.
> + */
> + if (unlikely(wq->wqe_head[idx] == READ_ONCE(wq->head))) {
> + mlx5_ib_err(dev,
> + "UMR CQE wqe_ctr 0x%x for an uncommitted WQE, possible NIC forgery\n",
> + wqe_ctr);
> + goto forged_cqe;
> + }
> +
> + /* Pop the shadow FIFO head: validated, dispatch to its waiter */
> + WRITE_ONCE(umrc->pending[head & (MLX5_UMR_MAX_WR - 1)], NULL);
> + umrc->pending_head = head + 1;
> + wc->wr_cqe = &expected->cqe;
> +
> + wc->qp = &qp->ibqp;
> + wq->tail = wq->wqe_head[idx] + 1;
> + /* Advance wq->last_poll, the internal error flush starts from it */
> + handle_atomics(qp, cqe64, wq->last_poll, idx);
> +
> + if (opcode == MLX5_CQE_REQ) {
> + wc->status = IB_WC_SUCCESS;
> + } else {
> + /* Copy the CQE, so its syndrome is read once */
> + err_cqe = *(struct mlx5_err_cqe *)cqe64;
> + mlx5_handle_error_cqe(dev, &err_cqe, wc);
> + /* As in the generic path, a real failure starts the recovery */
> + if (wc->status != IB_WC_WR_FLUSH_ERR)
> + umrc->state = MLX5_UMR_STATE_RECOVER;
> + }
> +
> + return 0;
> +
> +forged_cqe:
> + /* mlx5_ib_poll_cq() updates the CI only for a returned completion */
> + mlx5_cq_set_ci(&cq->mcq);
> + /*
> + * The NIC forged a CQE. Fail the UMR path for good, ignore any later
> + * CQE and force the health recovery, as the fatal sensors still read
> + * clean. The internal error flush then completes the waiters. Return
> + * an error, not -EAGAIN, as a CQE was consumed.
> + */
> + umrc->state = MLX5_UMR_STATE_ERR;
> + cq->poll_one = mlx5_ib_poll_one_stopped;
> + mlx5_trigger_health_work_force(dev->mdev);
> + return -EIO;
> +}
> +
> int mlx5_ib_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc)
> {
> struct mlx5_ib_cq *cq = to_mcq(ibcq);
[Severity: Critical]
Can a hostile NIC push the UMR path into an invalid state with error
CQEs that still pass the shadow FIFO match?
The CQE is checked against the oldest pending context. Its syndrome is
then taken at face value, whatever umrc->state is. Two cases look
reachable by the NIC this patch defends against.
The first case is a WR_FLUSH_ERR CQE while umrc->state is
MLX5_UMR_STATE_ACTIVE. An honest QP only flushes after an earlier error
CQE has moved the state to MLX5_UMR_STATE_RECOVER. Here the flush is
accepted and the state is left alone. mlx5r_umr_post_send_wait() sees
IB_WC_WR_FLUSH_ERR, takes the continue, still finds the state ACTIVE
and reposts the same WQE. If the NIC flushes every repost, the caller
never returns and keeps its umrc->sem slot.
The second case is a non-flush error for more than one pending WQE.
Each matching CQE sets MLX5_UMR_STATE_RECOVER and wakes its waiter with
a non-flush status, so several waiters call mlx5r_umr_recover().
mlx5r_umr_recover() drops umrc->lock before it posts the barrier WR and
resets the QP. The recoveries therefore run concurrently, and each one
posts its own barrier.
When the first barrier completes, ib_modify_qp(IB_QPS_RESET) reaches
__mlx5_ib_modify_qp(), which zeroes qp->sq.head, sq.tail, sq.cur_post
and sq.last_poll. It clears neither umrc->pending nor qp->sq.wqe_head[],
so the second barrier is still at the head of the shadow FIFO.
Suppose the NIC now completes that barrier with its old wqe_ctr. The
uncommitted-WQE check compares a stale wq->wqe_head[idx] with the reset
wq->head and passes. wq->tail becomes wq->wqe_head[idx] + 1 while
wq->head is 0.
A later forged CQE then forces the health recovery. Once the device is
in the internal error state, mlx5_ib_poll_cq() completes the SQ in
software. sw_comp() computes wq->head - wq->tail, which has wrapped,
and walks wq->wrid[]. The entries it dispatches point at
mlx5r_umr_context structures on the stacks of callers that have
already returned. mlx5r_umr_done() then writes the status and completes
the done field in that freed stack memory.
Should a flush seen in MLX5_UMR_STATE_ACTIVE take the forged_cqe path
instead? The same question applies to a second non-flush error once the
state is already MLX5_UMR_STATE_RECOVER.
Separately, should mlx5r_umr_recover() be serialized? And should the
shadow FIFO be reset together with the SQ counters on the RESET
transition?
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 13/23] RDMA/mlx5: Keep the UMR QP failed once a forged CQE is detected
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (11 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 12/23] RDMA/mlx5: Verify UMR CQ polling against a hostile NIC Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:24 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 14/23] RDMA/mlx5: Block generic CQ poll path in adversarial mode Yishai Hadas
` (9 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
In adversarial mode MLX5_UMR_STATE_ERR is a final state: the poller sets
it when it rejects a forged CQE, after which hardware polling is stopped,
and it stops mlx5r_umr_post_send_wait() from posting again.
mlx5r_umr_recover() runs concurrently with that and moves the state with
plain stores: it enters RECOVER from whatever state the path is in and
ends with an unconditional ACTIVE. An ERR set while a recovery is in
flight is therefore overwritten, which opens the UMR path again for a
NIC that was just caught lying, with its poller already stopped.
In adversarial mode move the state with cmpxchg() in recovery, ACTIVE to
RECOVER on entry and RECOVER to ACTIVE at the end. If ERR was set in the
meantime the transition leaves it in place and recovery returns
-EFAULT. Outside adversarial mode the transitions are unchanged.
This keeps new UMRs away from the NIC until the forced health recovery
moves the device to MLX5_DEVICE_STATE_INTERNAL_ERROR, after which posts
fail with -EIO anyway.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/hw/mlx5/umr.c | 27 +++++++++++++++++++++++++--
1 file changed, 25 insertions(+), 2 deletions(-)
diff --git a/drivers/infiniband/hw/mlx5/umr.c b/drivers/infiniband/hw/mlx5/umr.c
index b0f00d4bc93c..086b397b830c 100644
--- a/drivers/infiniband/hw/mlx5/umr.c
+++ b/drivers/infiniband/hw/mlx5/umr.c
@@ -335,6 +335,23 @@ static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey,
return err;
}
+/*
+ * Move the UMR state from @from to @to. In adversarial_mode ERR is a final
+ * state: it can be set concurrently by the CQ poller, so the move is atomic
+ * and does not replace it. Returns false if ERR was found, and true
+ * otherwise. Outside adversarial_mode this is the plain store it always was.
+ */
+static bool mlx5r_umr_move_state(struct umr_common *umrc, unsigned int from,
+ unsigned int to)
+{
+ if (!mlx5_ib_adversarial_mode) {
+ umrc->state = to;
+ return true;
+ }
+
+ return cmpxchg(&umrc->state, from, to) != MLX5_UMR_STATE_ERR;
+}
+
static int mlx5r_umr_recover(struct mlx5_ib_dev *dev, u32 mkey,
struct mlx5r_umr_context *umr_context,
struct mlx5r_umr_wqe *wqe, bool with_data)
@@ -353,7 +370,11 @@ static int mlx5r_umr_recover(struct mlx5_ib_dev *dev, u32 mkey,
if (umrc->state != MLX5_UMR_STATE_RECOVER) {
mlx5_ib_warn(dev, "UMR recovery encountered an unexpected state=%d\n",
umrc->state);
- umrc->state = MLX5_UMR_STATE_RECOVER;
+ if (!mlx5r_umr_move_state(umrc, MLX5_UMR_STATE_ACTIVE,
+ MLX5_UMR_STATE_RECOVER)) {
+ mutex_unlock(&umrc->lock);
+ return -EFAULT;
+ }
}
mutex_unlock(&umrc->lock);
@@ -387,7 +408,9 @@ static int mlx5r_umr_recover(struct mlx5_ib_dev *dev, u32 mkey,
goto err;
}
- umrc->state = MLX5_UMR_STATE_ACTIVE;
+ if (!mlx5r_umr_move_state(umrc, MLX5_UMR_STATE_RECOVER,
+ MLX5_UMR_STATE_ACTIVE))
+ return -EFAULT;
return 0;
err:
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 13/23] RDMA/mlx5: Keep the UMR QP failed once a forged CQE is detected
2026-11-03 16:39 ` [PATCH rdma-next 13/23] RDMA/mlx5: Keep the UMR QP failed once a forged CQE is detected Yishai Hadas
@ 2026-11-07 20:24 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:24 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found 2 potential
issue(s) to consider.
Critical: 0 · High: 0 · Medium: 2 · Low: 0
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [Medium] In adversarial mode, mlx5r_umr_recover()
(drivers/infiniband/hw/mlx5/umr.c) checks for MLX5_UMR_STATE_ERR at…
- [Medium] The new comment on mlx5r_umr_move_state() (umr.c:338-343) and
the commit message both say that in adversarial mode ERR is a final…
--
RDMA/mlx5: Keep the UMR QP failed once a forged CQE is detected
In adversarial mode, this commit changes mlx5r_umr_recover() to move the
UMR state with cmpxchg(): ACTIVE to RECOVER on entry and RECOVER to
ACTIVE at the end. The goal is to stop recovery from overwriting an
MLX5_UMR_STATE_ERR that the CQ poller sets on a forged CQE. Outside
adversarial mode the transitions are unchanged.
> diff --git a/drivers/infiniband/hw/mlx5/umr.c b/drivers/infiniband/hw/mlx5/umr.c
> index b0f00d4bc93ca..086b397b830c6 100644
> --- a/drivers/infiniband/hw/mlx5/umr.c
> +++ b/drivers/infiniband/hw/mlx5/umr.c
> @@ -335,6 +335,23 @@ static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey,
> return err;
> }
>
> +/*
> + * Move the UMR state from @from to @to. In adversarial_mode ERR is a final
> + * state: it can be set concurrently by the CQ poller, so the move is atomic
> + * and does not replace it. Returns false if ERR was found, and true
> + * otherwise. Outside adversarial_mode this is the plain store it always was.
> + */
[Severity: Medium]
Is ERR really final here? Only the transitions in mlx5r_umr_recover()
were converted to cmpxchg(). mlx5_ib_poll_one_umr() in cq.c still does a
plain store for every validated non-flush error CQE:
drivers/infiniband/hw/mlx5/cq.c:mlx5_ib_poll_one_umr() {
...
if (wc->status != IB_WC_WR_FLUSH_ERR)
umrc->state = MLX5_UMR_STATE_RECOVER;
...
}
That store can overwrite the ERR written by the err: label in
mlx5r_umr_recover(), and that label does not stop the poller:
err:
umrc->state = MLX5_UMR_STATE_ERR;
return err;
Could a hostile NIC reopen the UMR path with a sequence like this?
1. Waiters A and B each post a UMR while the state is ACTIVE.
2. The NIC returns validated non-flush error CQEs for both. A and B enter
mlx5r_umr_recover(), find the state already RECOVER, skip the entry
cmpxchg, and post barriers WQE3 and WQE4.
3. The NIC completes WQE3. A moves the QP to RESET, the NIC fails
RST2INIT, and A's err: label stores ERR.
4. The NIC returns a non-flush error CQE for WQE4. It passes the opcode,
QPN, sop, wqe_ctr (FIFO head) and wqe_head checks, so the poller
stores RECOVER over ERR.
5. B's RESET and RST2RTS succeed, and B's cmpxchg(RECOVER->ACTIVE) also
succeeds, so the UMR path is ACTIVE again.
The generic store of RECOVER in mlx5_poll_one() has the same problem at
this commit. The later commit "RDMA/mlx5: Block generic CQ poll path in
adversarial mode" closes that path by making mlx5_poll_one() return
-EOPNOTSUPP in adversarial mode before the store is reached.
Would a cmpxchg(ACTIVE->RECOVER) in mlx5_ib_poll_one_umr() fix this? If
not, should the comment say that only an ERR set on a forged CQE is
final?
[ ... ]
> @@ -353,7 +370,11 @@ static int mlx5r_umr_recover(struct mlx5_ib_dev *dev, u32 mkey,
> if (umrc->state != MLX5_UMR_STATE_RECOVER) {
> mlx5_ib_warn(dev, "UMR recovery encountered an unexpected state=%d\n",
> umrc->state);
> - umrc->state = MLX5_UMR_STATE_RECOVER;
> + if (!mlx5r_umr_move_state(umrc, MLX5_UMR_STATE_ACTIVE,
> + MLX5_UMR_STATE_RECOVER)) {
> + mutex_unlock(&umrc->lock);
> + return -EFAULT;
> + }
> }
> mutex_unlock(&umrc->lock);
[Severity: Medium]
What happens if the poller detects a forged CQE after this
mutex_unlock()?
mlx5_ib_poll_one_umr() never takes umrc->lock. It sets ERR under cq->lock
and only queues the forced health work:
umrc->state = MLX5_UMR_STATE_ERR;
cq->poll_one = mlx5_ib_poll_one_stopped;
mlx5_trigger_health_work_force(dev->mdev);
After the unlock, mlx5r_umr_recover() does not look at umrc->state again
until the final cmpxchg, so recovery continues:
mlx5r_umr_recover()
mlx5r_umr_post_send()
only checks mdev->state, writes the barrier WQE and rings the doorbell
wait_for_completion()
ib_modify_qp(umrc->qp, &attr, IB_QP_STATE) /* RESET */
__mlx5_ib_modify_qp()->mlx5_ib_cq_clean() sweeps the stopped UMR CQ
mlx5r_umr_qp_rst2rts()
RST2INIT/INIT2RTR/RTR2RTS put the hardware UMR QP back in RTS
mlx5r_umr_move_state(umrc, MLX5_UMR_STATE_RECOVER, MLX5_UMR_STATE_ACTIVE)
first point where ERR is seen
The NIC controls CQE timing. It can complete the barrier with a valid CQE
and then forge one while recovery is in the RESET/RTS steps. All of this
can happen before the forced health work moves the device to
MLX5_DEVICE_STATE_INTERNAL_ERROR.
umrc->state stays ERR, so mlx5r_umr_post_send_wait() still refuses new
UMRs. Even so, in this window the subject line "Keep the UMR QP failed",
the existing comment "don't revive the QP" in mlx5r_umr_recover(), and
this part of the commit message don't seem fully met:
This keeps new UMRs away from the NIC until the forced health recovery
moves the device to MLX5_DEVICE_STATE_INTERNAL_ERROR
In adversarial mode, should ERR be checked again at these points?
- before the barrier post, perhaps inside mlx5r_umr_post_send() under
qp->sq.lock
- after wait_for_completion()
- before each ib_modify_qp() step
[ ... ]
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 14/23] RDMA/mlx5: Block generic CQ poll path in adversarial mode
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (12 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 13/23] RDMA/mlx5: Keep the UMR QP failed once a forged CQE is detected Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:24 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 15/23] RDMA/core: Set the CQ comp_handler after its poll context is set up Yishai Hadas
` (8 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
mlx5_poll_one() trusts the CQE content, so it must not run on a CQE
from an untrusted NIC. In adversarial mode no CQ needs it: the UMR CQ
has its own poll_one that validates the CQE, kernel ULPs don't bind to
the device, and user CQs are polled by userspace. Refuse it there, so
an unexpected caller, such as the uverbs POLL_CQ command, finds the CQ
empty and no CQE is interpreted.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/hw/mlx5/cq.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
index 1a98263ec311..5cc86c3f93f3 100644
--- a/drivers/infiniband/hw/mlx5/cq.c
+++ b/drivers/infiniband/hw/mlx5/cq.c
@@ -464,6 +464,15 @@ static int mlx5_poll_one(struct mlx5_ib_cq *cq,
void *cqe;
int idx;
+ /*
+ * This path trusts the CQE, so refuse it in adversarial_mode. Only an
+ * unexpected caller gets here, e.g. the uverbs POLL_CQ command or an
+ * event on the UMR CQ before its own poll_one is installed. Nothing is
+ * consumed, so such a CQE is left for that poll_one.
+ */
+ if (unlikely(mlx5_ib_adversarial_mode))
+ return -EOPNOTSUPP;
+
repoll:
cqe = next_cqe_sw(cq);
if (!cqe)
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 14/23] RDMA/mlx5: Block generic CQ poll path in adversarial mode
2026-11-03 16:39 ` [PATCH rdma-next 14/23] RDMA/mlx5: Block generic CQ poll path in adversarial mode Yishai Hadas
@ 2026-11-07 20:24 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:24 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found more severe
issue(s) in other patches of this series. This mail is only an FYI, in case
you want to address the nit picks below when you respin.
Low priority / nit picks for your consideration:
- [Low] The commit message and the new comment in mlx5_poll_one()
(drivers/infiniband/hw/mlx5/cq.c) give the wrong example of the…
--
RDMA/mlx5: Block generic CQ poll path in adversarial mode
This commit makes mlx5_poll_one() return -EOPNOTSUPP when
mlx5_ib_adversarial_mode is set. The generic CQ poller trusts CQE
content, so with this change it never interprets a CQE from an untrusted
NIC. The UMR CQ is expected to use its own validating poll_one instead.
> diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
> index 1a98263ec3111..5cc86c3f93f3e 100644
> --- a/drivers/infiniband/hw/mlx5/cq.c
> +++ b/drivers/infiniband/hw/mlx5/cq.c
> @@ -464,6 +464,15 @@ static int mlx5_poll_one(struct mlx5_ib_cq *cq,
> void *cqe;
> int idx;
>
> + /*
> + * This path trusts the CQE, so refuse it in adversarial_mode. Only an
> + * unexpected caller gets here, e.g. the uverbs POLL_CQ command or an
> + * event on the UMR CQ before its own poll_one is installed. Nothing is
> + * consumed, so such a CQE is left for that poll_one.
> + */
[Severity: Low]
Can the uverbs POLL_CQ command actually reach mlx5_poll_one() on mlx5?
The commit message uses the same example:
"Refuse it there, so an unexpected caller, such as the uverbs POLL_CQ
command, finds the CQ empty and no CQE is interpreted."
_ib_alloc_device() builds the default uverbs_cmd_mask without
IB_USER_VERBS_CMD_POLL_CQ. mlx5 never adds that bit; only bnxt_re and
rdmavt set it. As a result, uapi_create_write() disables the write
method:
drivers/infiniband/core/uverbs_uapi.c:uapi_create_write() {
...
method_elm->disabled = !(ibdev->uverbs_cmd_mask &
BIT_ULL(def->write.command_num));
...
}
uapi_finalize_disable() then removes the method from the uapi tree. So
POLL_CQ doesn't appear to be reachable on mlx5.
The case that can be reached seems to be the UMR CQ during setup.
mlx5r_umr_resource_init() arms the CQ here:
cq = ib_alloc_cq(&dev->ib_dev, NULL, 128, 0, IB_POLL_SOFTIRQ);
mlx5_ib_create_cq() sets cq->poll_one = mlx5_poll_one.
mlx5_ib_poll_one_umr is only installed later, after ib_create_qp() and
mlx5r_umr_qp_rst2rts().
If a completion event with a forged CQE arrives in that window, the
softirq poller calls mlx5_poll_one(). The earlier "Verify UMR CQ polling
against a hostile NIC" patch leaves this window open, and it looks like
the case this patch really closes.
The code comment mentions this window, but the commit message doesn't.
Could the commit message describe the UMR setup window as the case being
fixed? Could both the commit message and the comment drop the POLL_CQ
example?
The commit message also says:
"kernel ULPs don't bind to the device"
ib_device::adversarial_mode is only set by the later "RDMA/mlx5:
Advertise adversarial mode via ib_device::adversarial_mode" patch. Is
that statement accurate at this commit?
> + if (unlikely(mlx5_ib_adversarial_mode))
> + return -EOPNOTSUPP;
> +
> repoll:
> cqe = next_cqe_sw(cq);
> if (!cqe)
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 15/23] RDMA/core: Set the CQ comp_handler after its poll context is set up
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (13 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 14/23] RDMA/mlx5: Block generic CQ poll path in adversarial mode Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:24 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 16/23] RDMA/mlx5: Ignore completion events for a CQ without a handler Yishai Hadas
` (7 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
__ib_alloc_cq() sets comp_handler before it initializes the poll context
the handler schedules, and for the workqueue contexts it arms the CQ
before setting comp_wq. The CQ is reachable from the completion
interrupt once ops.create_cq() returns, so a completion event raised in
that window would schedule an uninitialized irq_poll or work, or queue
work on a NULL workqueue. A conforming device raises no event for a new,
empty CQ, but nothing in the core relies on that.
Initialize the poll context first, then set comp_handler, and arm the CQ
only after that.
Fixes: 14d3a3b2498e ("IB: add a proper completion queue abstraction")
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/core/cq.c | 19 +++++++++++++------
1 file changed, 13 insertions(+), 6 deletions(-)
diff --git a/drivers/infiniband/core/cq.c b/drivers/infiniband/core/cq.c
index 12304c9a9403..42e4d272f5ad 100644
--- a/drivers/infiniband/core/cq.c
+++ b/drivers/infiniband/core/cq.c
@@ -217,6 +217,7 @@ struct ib_cq *__ib_alloc_cq(struct ib_device *dev, void *private, int nr_cqe,
.cqe = nr_cqe,
.comp_vector = comp_vector,
};
+ ib_comp_handler comp_handler;
struct ib_cq *cq;
int ret = -ENOMEM;
@@ -248,19 +249,16 @@ struct ib_cq *__ib_alloc_cq(struct ib_device *dev, void *private, int nr_cqe,
switch (cq->poll_ctx) {
case IB_POLL_DIRECT:
- cq->comp_handler = ib_cq_completion_direct;
+ comp_handler = ib_cq_completion_direct;
break;
case IB_POLL_SOFTIRQ:
- cq->comp_handler = ib_cq_completion_softirq;
-
+ comp_handler = ib_cq_completion_softirq;
irq_poll_init(&cq->iop, IB_POLL_BUDGET_IRQ, ib_poll_handler);
- ib_req_notify_cq(cq, IB_CQ_NEXT_COMP);
break;
case IB_POLL_WORKQUEUE:
case IB_POLL_UNBOUND_WORKQUEUE:
- cq->comp_handler = ib_cq_completion_workqueue;
+ comp_handler = ib_cq_completion_workqueue;
INIT_WORK(&cq->work, ib_cq_poll_work);
- ib_req_notify_cq(cq, IB_CQ_NEXT_COMP);
cq->comp_wq = (cq->poll_ctx == IB_POLL_WORKQUEUE) ?
ib_comp_wq : ib_comp_unbound_wq;
break;
@@ -269,6 +267,15 @@ struct ib_cq *__ib_alloc_cq(struct ib_device *dev, void *private, int nr_cqe,
goto out_destroy_cq;
}
+ /*
+ * The CQ is reachable from the completion interrupt once created. Set
+ * the handler only after the poll context it schedules is set up, and
+ * arm the CQ only after that.
+ */
+ cq->comp_handler = comp_handler;
+ if (cq->poll_ctx != IB_POLL_DIRECT)
+ ib_req_notify_cq(cq, IB_CQ_NEXT_COMP);
+
rdma_restrack_add(&cq->res);
trace_cq_alloc(cq, nr_cqe, comp_vector, poll_ctx);
return cq;
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 15/23] RDMA/core: Set the CQ comp_handler after its poll context is set up
2026-11-03 16:39 ` [PATCH rdma-next 15/23] RDMA/core: Set the CQ comp_handler after its poll context is set up Yishai Hadas
@ 2026-11-07 20:24 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:24 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found 2 potential
issue(s) to consider.
Critical: 0 · High: 1 · Medium: 1 · Low: 0
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [High] The patch relies on program order alone to publish the poll
context before the handler, and that ordering does not hold for…
- [Medium] __ib_alloc_cq() (drivers/infiniband/core/cq.c) still stores
cq->comp_handler after dev->ops.create_cq() has returned (create_cq at…
--
RDMA/core: Set the CQ comp_handler after its poll context is set up
This commit reorders __ib_alloc_cq(). The irq_poll or work context and
comp_wq are now initialized before cq->comp_handler is stored, and the CQ
is armed only after the handler is set. The goal is that a completion
event raised right after ops.create_cq() returns no longer finds an
uninitialized poll context.
> diff --git a/drivers/infiniband/core/cq.c b/drivers/infiniband/core/cq.c
> index 12304c9a9403e..42e4d272f5ade 100644
> --- a/drivers/infiniband/core/cq.c
> +++ b/drivers/infiniband/core/cq.c
[ ... ]
> @@ -269,6 +267,15 @@ struct ib_cq *__ib_alloc_cq(struct ib_device *dev, void *private, int nr_cqe,
> goto out_destroy_cq;
> }
>
> + /*
> + * The CQ is reachable from the completion interrupt once created. Set
> + * the handler only after the poll context it schedules is set up, and
> + * arm the CQ only after that.
> + */
> + cq->comp_handler = comp_handler;
[Severity: Medium]
This store still runs after dev->ops.create_cq() has returned. By then, can
the driver's EQ dispatch already reach the CQ while comp_handler is still
NULL from rdma_zalloc_drv_obj()?
The driver publishes the CQN during create_cq. For example,
mlx5_core_create_cq()->mlx5_eq_add_cq() inserts it into the radix tree,
efa stores it in dev->cqs_xa, and mthca adds it to cq_table.
Several drivers then call the handler without a NULL check. For example,
in drivers/infiniband/hw/mlx4/cq.c:
static void mlx4_ib_cq_comp(struct mlx4_cq *cq)
{
struct ib_cq *ibcq = &to_mibcq(cq)->ibcq;
ibcq->comp_handler(ibcq, ibcq->cq_context);
}
efa_process_comp_eqe() and mthca_cq_completion() do the same, and so does
mlx5_ib_cq_comp() at this commit.
So an early event from a non-conforming device would now be a NULL
function pointer call in irq or tasklet context, instead of scheduling an
uninitialized irq_poll or work. The patch also moves the store later than
before, past rdma_dim_init() (which can do a GFP_KERNEL allocation) and the
whole poll context switch, so the window gets a bit longer.
The commit message says:
A conforming device raises no event for a new,
empty CQ, but nothing in the core relies on that.
Doesn't the core now rely on every driver NULL-checking comp_handler?
The later commit "RDMA/mlx5: Ignore completion events for a CQ without a
handler" adds that check to mlx5_ib_cq_comp(), which closes the window for
mlx5. mlx4, efa and mthca still call the handler unconditionally at the end
of the series.
__ib_create_cq() in verbs.c, and the uverbs paths, set the handler before
calling into the driver:
cq->device = device;
cq->comp_handler = comp_handler;
...
ret = device->ops.create_cq(cq, cq_attr, NULL);
Could __ib_alloc_cq() do the same? That would mean running irq_poll_init(),
INIT_WORK(), the comp_wq store and the comp_handler store before
ops.create_cq(), and keeping only ib_req_notify_cq() for afterwards. The
driver's own publication of the CQ would then provide the ordering.
[Severity: High]
Is program order enough to make the poll context visible before the
handler? The irq_poll_init() stores (memset, INIT_LIST_HEAD, weight, poll),
INIT_WORK() and the cq->comp_wq store are all plain stores. This
comp_handler store is plain too, with no smp_store_release() or smp_wmb().
The reader side has no acquire ordering either:
mlx5 EQ irq
mlx5_cq_tasklet_cb()
mlx5_ib_cq_comp()
ibcq->comp_handler() <- plain load, indirect call
ib_cq_completion_softirq()
irq_poll_sched(&cq->iop)
or
ib_cq_completion_workqueue()
queue_work(cq->comp_wq, &cq->work)
An event that arrives before the CQ is armed gets no ordering from the
doorbell write in ib_req_notify_cq().
On weakly ordered CPUs such as arm64 and POWER, can another CPU see the new
comp_handler before the iop, work or comp_wq stores? In the workqueue
cases, the compiler is also free to reorder the adjacent plain stores to
comp_wq and comp_handler.
If that happens, three things could go wrong:
- The reader schedules an iop whose memset and INIT_LIST_HEAD land after
its own test_and_set_bit() and list_add_tail(). That corrupts the per-cpu
irq_poll list.
- irq_poll_softirq() calls a stale NULL iop->poll.
- queue_work() is called with a NULL workqueue.
These are the failures the commit message describes.
Adversarial mode is enabled through cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT).
That is true for arm64 realm/pKVM guests and pseries secure guests. The mlx5
UMR CQ allocated in mlx5r_umr_resource_init() with IB_POLL_SOFTIRQ goes
through this path. The plain NULL check that the later commit adds to
mlx5_ib_cq_comp() does not provide acquire ordering.
Would this need smp_store_release() on comp_handler, paired with an acquire
load in the drivers? The other option is to do all of this setup before
ops.create_cq(). Then the driver's publication, such as radix_tree_insert()
under table->lock in mlx5_eq_add_cq(), would order the stores for readers
that look up the CQ.
> + if (cq->poll_ctx != IB_POLL_DIRECT)
> + ib_req_notify_cq(cq, IB_CQ_NEXT_COMP);
[ ... ]
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 16/23] RDMA/mlx5: Ignore completion events for a CQ without a handler
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (14 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 15/23] RDMA/core: Set the CQ comp_handler after its poll context is set up Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:25 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 17/23] RDMA/mlx5: Validate EQE port and GSI port before indexing devr.ports[] Yishai Hadas
` (6 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
mlx5_ib_cq_comp() calls the CQ comp_handler unconditionally, but
devr->c0, created by mlx5_ib_dev_res_cq_init() on the first SRQ create,
has none, as it is never armed.
A trusted NIC raises no completion event for a CQ that is not armed, but
a hostile NIC in adversarial mode can do so with any CQ number it
returned, which ends in a NULL call in interrupt context.
Skip the call when no handler is set.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/hw/mlx5/cq.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
index 5cc86c3f93f3..f955eb762051 100644
--- a/drivers/infiniband/hw/mlx5/cq.c
+++ b/drivers/infiniband/hw/mlx5/cq.c
@@ -46,7 +46,12 @@ static void mlx5_ib_cq_comp(struct mlx5_core_cq *cq, struct mlx5_eqe *eqe)
{
struct ib_cq *ibcq = &to_mibcq(cq)->ibcq;
- ibcq->comp_handler(ibcq, ibcq->cq_context);
+ /*
+ * A CQ created without a handler (devr->c0) is never armed, but a
+ * hostile NIC can still raise a completion event for it.
+ */
+ if (ibcq->comp_handler)
+ ibcq->comp_handler(ibcq, ibcq->cq_context);
}
static void mlx5_ib_cq_event(struct mlx5_core_cq *mcq, enum mlx5_event type)
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 16/23] RDMA/mlx5: Ignore completion events for a CQ without a handler
2026-11-03 16:39 ` [PATCH rdma-next 16/23] RDMA/mlx5: Ignore completion events for a CQ without a handler Yishai Hadas
@ 2026-11-07 20:25 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:25 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found 1 potential
issue(s) to consider.
Critical: 0 · High: 1 · Medium: 0 · Low: 0
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [High] The fix leaves a memory-ordering race on comp_handler.
--
RDMA/mlx5: Ignore completion events for a CQ without a handler
This patch makes mlx5_ib_cq_comp() skip the comp_handler call when no
handler is set. The case it targets is devr->c0, which is never armed, but
a hostile NIC in adversarial mode can still raise a completion event for
it.
> diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
> index 5cc86c3f93f3e..f955eb762051d 100644
> --- a/drivers/infiniband/hw/mlx5/cq.c
> +++ b/drivers/infiniband/hw/mlx5/cq.c
> @@ -46,7 +46,12 @@ static void mlx5_ib_cq_comp(struct mlx5_core_cq *cq, struct mlx5_eqe *eqe)
> {
> struct ib_cq *ibcq = &to_mibcq(cq)->ibcq;
>
> - ibcq->comp_handler(ibcq, ibcq->cq_context);
> + /*
> + * A CQ created without a handler (devr->c0) is never armed, but a
> + * hostile NIC can still raise a completion event for it.
> + */
> + if (ibcq->comp_handler)
> + ibcq->comp_handler(ibcq, ibcq->cq_context);
[Severity: High]
For devr->c0 this check is enough, because comp_handler stays NULL for
that CQ. Is it also enough for the CQs set up by __ib_alloc_cq()?
The companion commit "RDMA/core: Set the CQ comp_handler after its poll
context is set up" relies on this NULL check as a "poll context ready"
gate. Neither side of that gate has any ordering, though.
On the writer side, the CQ can be reached from the EQ interrupt once
create_cq() puts it in eq->cq_table.tree. The handler is published later
with a plain store:
drivers/infiniband/core/cq.c:__ib_alloc_cq() {
...
ret = dev->ops.create_cq(cq, &cq_attr, NULL);
...
case IB_POLL_SOFTIRQ:
comp_handler = ib_cq_completion_softirq;
irq_poll_init(&cq->iop, IB_POLL_BUDGET_IRQ, ib_poll_handler);
break;
case IB_POLL_WORKQUEUE:
case IB_POLL_UNBOUND_WORKQUEUE:
comp_handler = ib_cq_completion_workqueue;
INIT_WORK(&cq->work, ib_cq_poll_work);
cq->comp_wq = ...
...
cq->comp_handler = comp_handler;
...
}
On the reader side, mlx5_eq_comp_int() gets the CQ with mlx5_eq_cq_get(),
which is an RCU radix_tree_lookup() plus a refcount increment. That orders
against the tree insertion, but not against the stores made after it.
mlx5_ib_cq_comp() then loads comp_handler twice with plain loads.
Adversarial mode is enabled when cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)
is true. That includes arm64 Realm and pKVM guests, and those CPUs are
weakly ordered. Take the UMR CQ, which mlx5r_umr_resource_init() allocates
with IB_POLL_SOFTIRQ.
Could another CPU see comp_handler == ib_cq_completion_softirq before the
memset(), INIT_LIST_HEAD(), weight and poll stores from irq_poll_init()
are visible? If so, irq_poll_sched() would link iop onto the per-CPU list.
The delayed irq_poll_init() stores could then overwrite iop->state and
iop->list while iop is still on that list.
Possible results are per-CPU list corruption, irq_poll_softirq() spinning
on a stale entry, or a NULL iop->poll call. For the workqueue cases, the
compiler is also free to move the plain comp_handler store ahead of the
INIT_WORK() and cq->comp_wq stores. That could lead to queue_work() with a
NULL workqueue or an uninitialized work item.
The wmb() in the later ib_req_notify_cq() only helps if the device waits
for the arm doorbell, and a hostile device does not wait.
Would it make sense to publish the handler with:
smp_store_release(&cq->comp_handler, comp_handler);
in __ib_alloc_cq(), and read it once in mlx5_ib_cq_comp() with:
ib_comp_handler h = smp_load_acquire(&ibcq->comp_handler);
if (h)
h(ibcq, ibcq->cq_context);
That also removes the double fetch. KCSAN would likely report the current
plain check against the plain store as a data race. Later patches in the
series do not change this code.
> }
>
> static void mlx5_ib_cq_event(struct mlx5_core_cq *mcq, enum mlx5_event type)
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 17/23] RDMA/mlx5: Validate EQE port and GSI port before indexing devr.ports[]
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (15 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 16/23] RDMA/mlx5: Ignore completion events for a CQ without a handler Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:25 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 18/23] RDMA/mlx5: Validate firmware RQ/SQ state before sqrq_trans[] lookup Yishai Hadas
` (5 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
devr.ports[] has 2 entries, but it is indexed by port numbers taken
from the NIC, which adversarial mode does not trust.
handle_port_change() uses the EQE port before any validation, so port 0
wraps the index to UINT_MAX and a port above 2 overflows the array.
mlx5_ib_create_gsi() gets a port already checked against phys_port_cnt,
but that comes from the num_ports and num_vhca_ports FW caps, which may
report more than 2 ports.
Validate the port and bound it by ARRAY_SIZE() in both paths.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/hw/mlx5/gsi.c | 4 ++++
drivers/infiniband/hw/mlx5/main.c | 4 ++++
2 files changed, 8 insertions(+)
diff --git a/drivers/infiniband/hw/mlx5/gsi.c b/drivers/infiniband/hw/mlx5/gsi.c
index b2e2a219639d..7ce390a2820f 100644
--- a/drivers/infiniband/hw/mlx5/gsi.c
+++ b/drivers/infiniband/hw/mlx5/gsi.c
@@ -95,6 +95,10 @@ int mlx5_ib_create_gsi(struct ib_pd *pd, struct mlx5_ib_qp *mqp,
int num_qps = 0;
int ret;
+ if (!rdma_is_port_valid(&dev->ib_dev, port_num) ||
+ port_num > ARRAY_SIZE(dev->devr.ports))
+ return -EINVAL;
+
if (mlx5_ib_deth_sqpn_cap(dev)) {
if (MLX5_CAP_GEN(dev->mdev,
port_type) == MLX5_CAP_PORT_TYPE_IB)
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index f4f2ad80e76c..c29102260989 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -3063,6 +3063,10 @@ static int handle_port_change(struct mlx5_ib_dev *ibdev, struct mlx5_eqe *eqe,
{
u32 port = (eqe->data.port.port >> 4) & 0xf;
+ if (!rdma_is_port_valid(&ibdev->ib_dev, port) ||
+ port > ARRAY_SIZE(ibdev->devr.ports))
+ return -EINVAL;
+
ibev->element.port_num = port;
switch (eqe->sub_type) {
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 17/23] RDMA/mlx5: Validate EQE port and GSI port before indexing devr.ports[]
2026-11-03 16:39 ` [PATCH rdma-next 17/23] RDMA/mlx5: Validate EQE port and GSI port before indexing devr.ports[] Yishai Hadas
@ 2026-11-07 20:25 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:25 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found more severe
issue(s) in other patches of this series. This mail is only an FYI, in case
you want to address the nit picks below when you respin.
Low priority / nit picks for your consideration:
- [Low] This patch makes the existing invalid-port check and warning in
mlx5_ib_handle_event() (drivers/infiniband/hw/mlx5/main.c)…
--
RDMA/mlx5: Validate EQE port and GSI port before indexing devr.ports[]
This commit adds port validation to handle_port_change() and
mlx5_ib_create_gsi(). The port number from the NIC is checked with
rdma_is_port_valid() and limited to ARRAY_SIZE(devr.ports) before it is
used to index devr.ports[].
> diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
> index f4f2ad80e76c4..c29102260989f 100644
> --- a/drivers/infiniband/hw/mlx5/main.c
> +++ b/drivers/infiniband/hw/mlx5/main.c
> @@ -3063,6 +3063,10 @@ static int handle_port_change(struct mlx5_ib_dev *ibdev, struct mlx5_eqe *eqe,
> {
> u32 port = (eqe->data.port.port >> 4) & 0xf;
>
> + if (!rdma_is_port_valid(&ibdev->ib_dev, port) ||
> + port > ARRAY_SIZE(ibdev->devr.ports))
> + return -EINVAL;
> +
[Severity: Low]
Does this early return make the port check and warning in
mlx5_ib_handle_event() unreachable?
mlx5_ib_handle_event() treats any non-zero return from
handle_port_change() as a reason to skip the event:
drivers/infiniband/hw/mlx5/main.c:mlx5_ib_handle_event() {
...
case MLX5_EVENT_TYPE_PORT_CHANGE:
if (handle_port_change(ibdev, work->param, &ibev))
goto out;
break;
case MLX5_EVENT_TYPE_GENERAL_EVENT:
handle_general_event(ibdev, work->param, &ibev);
fallthrough;
default:
goto out;
}
ibev.device = &ibdev->ib_dev;
if (!rdma_is_port_valid(&ibdev->ib_dev, ibev.element.port_num)) {
mlx5_ib_warn(ibdev, "warning: event on port %d\n", ibev.element.port_num);
goto out;
}
...
}
The only way to reach the rdma_is_port_valid() check is a successful
handle_port_change(). After this patch, a successful return always means
the port is valid.
Before the patch, a port change EQE with port 0 or a port above
phys_port_cnt went through handle_port_change() and returned 0 if its
sub_type was not PKEY (LID, GUID, CLIENT_REREG, or ACTIVE/DOWN on IB).
The caller's check then printed the "warning: event on port %d" message.
After the patch, the same event is dropped with no log message.
Could the warning be moved into the new early-return path in
handle_port_change()? Or, if the silent drop is intended, could the
caller's check be removed, since it is now dead code?
> +
> ibev->element.port_num = port;
>
> switch (eqe->sub_type) {
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 18/23] RDMA/mlx5: Validate firmware RQ/SQ state before sqrq_trans[] lookup
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (16 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 17/23] RDMA/mlx5: Validate EQE port and GSI port before indexing devr.ports[] Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:25 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 19/23] RDMA/mlx5: Clamp log_max_cq_sz shift in CQ size checks Yishai Hadas
` (4 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
sqrq_state_to_qp_state() indexes the 5x5 sqrq_trans[] table with the RQ
and SQ states returned by QUERY_RQ and QUERY_SQ. The state fields are 4
bits wide, so an untrusted NIC can return a state above 4 and the lookup
reads past the table.
Reject out of range states with -EIO before the lookup.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/hw/mlx5/qp.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index 19cbf6b2b979..05e8e75d734f 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -5081,6 +5081,9 @@ static int sqrq_state_to_qp_state(u8 sq_state, u8 rq_state,
},
};
+ if (sq_state >= MLX5_SQ_NUM_STATE || rq_state >= MLX5_RQ_NUM_STATE)
+ return -EIO;
+
*qp_state = sqrq_trans[rq_state][sq_state];
if (*qp_state == MLX5_QP_STATE_BAD) {
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 18/23] RDMA/mlx5: Validate firmware RQ/SQ state before sqrq_trans[] lookup
2026-11-03 16:39 ` [PATCH rdma-next 18/23] RDMA/mlx5: Validate firmware RQ/SQ state before sqrq_trans[] lookup Yishai Hadas
@ 2026-11-07 20:25 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:25 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found 1 potential
issue(s) to consider.
Critical: 0 · High: 0 · Medium: 1 · Low: 0
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [Medium] The new check in sqrq_state_to_qp_state()
(drivers/infiniband/hw/mlx5/qp.c:5084) bounds the firmware SQ/RQ state…
--
RDMA/mlx5: Validate firmware RQ/SQ state before sqrq_trans[] lookup
This commit adds a bounds check to sqrq_state_to_qp_state() in
drivers/infiniband/hw/mlx5/qp.c. Any SQ or RQ state returned by QUERY_SQ
or QUERY_RQ that is at or above the sqrq_trans[] table dimension is now
rejected with -EIO, which prevents an out-of-bounds read of the 5x5
table.
> diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
> index 19cbf6b2b979d..05e8e75d734fb 100644
> --- a/drivers/infiniband/hw/mlx5/qp.c
> +++ b/drivers/infiniband/hw/mlx5/qp.c
> @@ -5081,6 +5081,9 @@ static int sqrq_state_to_qp_state(u8 sq_state, u8 rq_state,
> },
> };
>
> + if (sq_state >= MLX5_SQ_NUM_STATE || rq_state >= MLX5_RQ_NUM_STATE)
> + return -EIO;
> +
> *qp_state = sqrq_trans[rq_state][sq_state];
[Severity: Medium]
Should this check compare against the hardware state encodings instead of
the table size?
The hardware only defines three values in include/linux/mlx5/mlx5_ifc.h.
The RQC values are the same:
MLX5_SQC_STATE_RST = 0x0,
MLX5_SQC_STATE_RDY = 0x1,
MLX5_SQC_STATE_ERR = 0x3,
The table dimension comes from a driver-only sentinel in
include/linux/mlx5/qp.h:
MLX5_SQ_STATE_NA = MLX5_SQC_STATE_ERR + 1,
MLX5_SQ_NUM_STATE = MLX5_SQ_STATE_NA + 1,
That means firmware values 2 and 4 still get past the new check.
query_raw_packet_qp_rq_state() passes the raw 4-bit field through
unchanged, and mlx5_core_query_sq_state() does the same for the SQ:
*rq_state = MLX5_GET(rqc, rqc, state);
query_raw_packet_qp_state() sets the NA sentinel by default and only
overwrites it for queues that exist:
u8 sq_state = MLX5_SQ_STATE_NA;
u8 rq_state = MLX5_RQ_STATE_NA;
if (qp->sq.wqe_cnt) {
err = query_raw_packet_qp_sq_state(dev, sq, &sq_state);
...
Take an SQ-only raw packet QP where firmware returns SQ state 4. Doesn't
that value alias MLX5_SQ_STATE_NA and land on sqrq_trans[NA][NA]?
[MLX5_SQ_STATE_NA] = MLX5_QP_STATE_BAD,
That entry reaches WARN(1, "Buggy Raw Packet QP state, ...") through
ib_query_qp()->mlx5_ib_query_qp()->query_raw_packet_qp_state(). A legal
SQ value on that QP always maps to MLX5_QP_STATE. The RQ-only case with
RQ state 4 behaves the same way.
Value 2 indexes a row or column that is never initialized, so it reads
as 0, which is IB_QPS_RESET. Would ib_query_qp() then report RESET with
no error and no WARN? On an SQ-only QP, any legal value would have
returned qp->state.
On a panic_on_warn system, a device-triggered WARN here becomes a panic.
Would it close this gap to accept only the RST, RDY and ERR encodings,
and only for the queues that were actually queried?
>
> if (*qp_state == MLX5_QP_STATE_BAD) {
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 19/23] RDMA/mlx5: Clamp log_max_cq_sz shift in CQ size checks
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (17 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 18/23] RDMA/mlx5: Validate firmware RQ/SQ state before sqrq_trans[] lookup Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:25 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 20/23] RDMA/mlx5: Clamp log_max_* shifts in QP and WQ limit checks Yishai Hadas
` (3 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
The CQ create and resize paths check the requested size against
1 << log_max_cq_sz. The cap is 8 bits wide, so an untrusted NIC can
report 31 or more, which makes the shift undefined. A large limit also
lets the rounded up CQ size exceed INT_MAX, and it is then truncated
when passed on as an int.
Take the limit from MLX5_CAP_GEN_LOGSZ_U64(), round up in 64 bits, and
reject sizes above INT_MAX.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/hw/mlx5/cq.c | 39 +++++++++++++++++++++++----------
1 file changed, 28 insertions(+), 11 deletions(-)
diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
index f955eb762051..25344dfba1c1 100644
--- a/drivers/infiniband/hw/mlx5/cq.c
+++ b/drivers/infiniband/hw/mlx5/cq.c
@@ -1117,6 +1117,15 @@ static void notify_soft_wc_handler(struct work_struct *work)
cq->ibcq.comp_handler(&cq->ibcq, cq->ibcq.cq_context);
}
+/*
+ * Next power of 2 above entries. It can be 2^32, which doesn't fit the
+ * unsigned long of roundup_pow_of_two() on 32-bit archs.
+ */
+static u64 cq_roundup_entries(u64 entries)
+{
+ return 1ULL << fls64(entries);
+}
+
int mlx5_ib_create_user_cq(struct ib_cq *ibcq,
const struct ib_cq_init_attr *attr,
struct uverbs_attr_bundle *attrs)
@@ -1124,7 +1133,7 @@ int mlx5_ib_create_user_cq(struct ib_cq *ibcq,
struct ib_udata *udata = &attrs->driver_udata;
struct ib_device *ibdev = ibcq->device;
struct mlx5_ib_create_cq_resp uresp = {};
- int entries = attr->cqe;
+ u64 entries = attr->cqe;
int vector = attr->comp_vector;
struct mlx5_ib_dev *dev = to_mdev(ibdev);
struct mlx5_ib_cq *cq = to_mcq(ibcq);
@@ -1136,15 +1145,17 @@ int mlx5_ib_create_user_cq(struct ib_cq *ibcq,
int cqe_size;
int eqn;
int err;
+ u64 max_cq_entries = MLX5_CAP_GEN_LOGSZ_U64(dev->mdev, log_max_cq_sz);
- if (attr->cqe > (1 << MLX5_CAP_GEN(dev->mdev, log_max_cq_sz)))
+ if (attr->cqe > max_cq_entries)
return -EINVAL;
if (check_cq_create_flags(attr->flags))
return -EOPNOTSUPP;
- entries = roundup_pow_of_two(entries + 1);
- if (entries > (1 << MLX5_CAP_GEN(dev->mdev, log_max_cq_sz)))
+ entries = cq_roundup_entries(entries);
+ /* cq->ibcq.cqe and create_cq_user() take an int */
+ if (entries > max_cq_entries || entries > INT_MAX)
return -EINVAL;
cq->ibcq.cqe = entries - 1;
@@ -1214,7 +1225,7 @@ int mlx5_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
struct uverbs_attr_bundle *attrs)
{
struct ib_device *ibdev = ibcq->device;
- int entries = attr->cqe;
+ u64 entries = attr->cqe;
int vector = attr->comp_vector;
struct mlx5_ib_dev *dev = to_mdev(ibdev);
struct mlx5_ib_cq *cq = to_mcq(ibcq);
@@ -1226,12 +1237,14 @@ int mlx5_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
int cqe_size;
int eqn;
int err;
+ u64 max_cq_entries = MLX5_CAP_GEN_LOGSZ_U64(dev->mdev, log_max_cq_sz);
- if (attr->cqe > (1 << MLX5_CAP_GEN(dev->mdev, log_max_cq_sz)))
+ if (attr->cqe > max_cq_entries)
return -EINVAL;
- entries = roundup_pow_of_two(entries + 1);
- if (entries > (1 << MLX5_CAP_GEN(dev->mdev, log_max_cq_sz)))
+ entries = cq_roundup_entries(entries);
+ /* cq->ibcq.cqe and create_cq_kernel() take an int */
+ if (entries > max_cq_entries || entries > INT_MAX)
return -EINVAL;
cq->ibcq.cqe = entries - 1;
@@ -1532,18 +1545,22 @@ int mlx5_ib_resize_cq(struct ib_cq *ibcq, unsigned int entries,
int inlen;
int cqe_size;
unsigned long flags;
+ u64 entries64 = entries;
+ u64 max_cq_entries = MLX5_CAP_GEN_LOGSZ_U64(dev->mdev, log_max_cq_sz);
if (!MLX5_CAP_GEN(dev->mdev, cq_resize)) {
pr_info("Firmware does not support resize CQ\n");
return -ENOSYS;
}
- if (entries > (1 << MLX5_CAP_GEN(dev->mdev, log_max_cq_sz)))
+ if (entries64 > max_cq_entries)
return -EINVAL;
- entries = roundup_pow_of_two(entries + 1);
- if (entries > (1 << MLX5_CAP_GEN(dev->mdev, log_max_cq_sz)) + 1)
+ entries64 = cq_roundup_entries(entries64);
+ /* resize_user() and resize_kernel() take an int */
+ if (entries64 > max_cq_entries + 1 || entries64 > INT_MAX)
return -EINVAL;
+ entries = entries64;
if (entries == ibcq->cqe + 1)
return 0;
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 19/23] RDMA/mlx5: Clamp log_max_cq_sz shift in CQ size checks
2026-11-03 16:39 ` [PATCH rdma-next 19/23] RDMA/mlx5: Clamp log_max_cq_sz shift in CQ size checks Yishai Hadas
@ 2026-11-07 20:25 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:25 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found 1 potential
issue(s) to consider.
Critical: 0 · High: 1 · Medium: 0 · Low: 0
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [High] The new `entries > INT_MAX` check only keeps the CQ entry count
inside an int.
--
RDMA/mlx5: Clamp log_max_cq_sz shift in CQ size checks
This commit computes the CQ size limit with MLX5_CAP_GEN_LOGSZ_U64(), so
an untrusted log_max_cq_sz no longer causes an undefined shift. The
requested entry count is now rounded up in 64 bits. Sizes above INT_MAX
are rejected in the create and resize paths.
> diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
> index f955eb762051d..25344dfba1c12 100644
> --- a/drivers/infiniband/hw/mlx5/cq.c
> +++ b/drivers/infiniband/hw/mlx5/cq.c
[ ... ]
> @@ -1226,12 +1237,14 @@ int mlx5_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
> int cqe_size;
> int eqn;
> int err;
> + u64 max_cq_entries = MLX5_CAP_GEN_LOGSZ_U64(dev->mdev, log_max_cq_sz);
>
> - if (attr->cqe > (1 << MLX5_CAP_GEN(dev->mdev, log_max_cq_sz)))
> + if (attr->cqe > max_cq_entries)
> return -EINVAL;
>
> - entries = roundup_pow_of_two(entries + 1);
> - if (entries > (1 << MLX5_CAP_GEN(dev->mdev, log_max_cq_sz)))
> + entries = cq_roundup_entries(entries);
> + /* cq->ibcq.cqe and create_cq_kernel() take an int */
> + if (entries > max_cq_entries || entries > INT_MAX)
> return -EINVAL;
[Severity: High]
Is INT_MAX the right bound here? It keeps the entry count inside an int,
but the CQ buffer's byte size is still computed from it in 32-bit
arithmetic.
With a device-reported log_max_cq_sz of 26 or more, entries can now be up
to 2^30. They are then passed to create_cq_kernel() and on to
alloc_cq_frag_buf():
drivers/infiniband/hw/mlx5/cq.c:alloc_cq_frag_buf() {
...
err = mlx5_frag_buf_alloc_node(dev->mdev,
nent * cqe_size,
frag_buf,
dev->mdev->priv.numa_node);
...
}
Both values are powers of two. So nent * cqe_size wraps to exactly 0 for
nent >= 2^26 with 64-byte CQEs. With 128-byte CQEs
(cache_line_size() == 128) it wraps at nent >= 2^25.
With size 0, mlx5_frag_buf_alloc_node() sets npages to 0, and
kcalloc_node(0, ...) returns ZERO_SIZE_PTR. The function then returns
success, and buf->nent is still set to the full count.
init_cq_frag_buf() then does:
for (i = 0; i < buf->nent; i++) {
cqe = mlx5_frag_buf_get_wqe(&buf->fbc, i);
...
}
Won't mlx5_frag_buf_get_wqe() dereference fbc->frags[0].buf through
ZERO_SIZE_PTR on the first iteration and oops?
mlx5_ib_resize_cq() with udata == NULL has the same INT_MAX-only check.
It reaches the same code through resize_kernel()->alloc_cq_frag_buf()->
init_cq_frag_buf().
The user path in mlx5_ib_create_user_cq() has a similar problem.
create_cq_user() passes this as the minimum umem size:
cq->buf.umem = ib_umem_get_cq_buf_or_va(&dev->ib_dev, attrs,
ucmd.buf_addr,
entries * ucmd.cqe_size,
IB_ACCESS_LOCAL_WRITE);
That is a 32-bit unsigned multiply, and it wraps to 0 the same way. On the
UMEM-attribute path, the only length check in ib_umem_get_desc_check() is
umem->length < min_size, so any umem passes. The device is still
programmed with log_cq_size = ilog2(entries). resize_user() already uses
(size_t)ucmd.cqe_size * entries for this.
The commit message says a large limit made the CQ size get "truncated
when passed on as an int". Since create_cq_kernel() still multiplies that
int by cqe_size, can this truncation still happen?
Should the bound account for the CQE size, for example by rejecting
entries > INT_MAX / 128? Another option would be to compute the byte size
in size_t with check_mul_overflow().
>
> cq->ibcq.cqe = entries - 1;
[ ... ]
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 20/23] RDMA/mlx5: Clamp log_max_* shifts in QP and WQ limit checks
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (18 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 19/23] RDMA/mlx5: Clamp log_max_cq_sz shift in CQ size checks Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:25 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 21/23] RDMA/mlx5: Clamp log_max_* shifts in query_device and ucontext paths Yishai Hadas
` (2 subsequent siblings)
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
Several QP and WQ size checks in qp.c compute their limit as
1 << cap, where cap is an 8-bit log_max_* FW capability. An untrusted
NIC can report 31 or more, which makes the shift undefined.
Use MLX5_CAP_GEN_LOGSZ_U32() for these limits, as done for
log_max_cq_sz.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/hw/mlx5/qp.c | 36 +++++++++++++++------------------
1 file changed, 16 insertions(+), 20 deletions(-)
diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index 05e8e75d734f..d784a35675d5 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -447,7 +447,7 @@ static int set_rq_size(struct mlx5_ib_dev *dev, struct ib_qp_cap *cap,
int wq_size;
/* Sanity check RQ size before proceeding */
- if (cap->max_recv_wr > (1 << MLX5_CAP_GEN(dev->mdev, log_max_qp_sz)))
+ if (cap->max_recv_wr > MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_qp_sz))
return -EINVAL;
if (!has_rq) {
@@ -621,11 +621,11 @@ static int calc_sq_size(struct mlx5_ib_dev *dev, struct ib_qp_init_attr *attr,
wq_size = roundup_pow_of_two(attr->cap.max_send_wr * wqe_size);
qp->sq.wqe_cnt = wq_size / MLX5_SEND_WQE_BB;
- if (qp->sq.wqe_cnt > (1 << MLX5_CAP_GEN(dev->mdev, log_max_qp_sz))) {
- mlx5_ib_dbg(dev, "send queue size (%d * %d / %d -> %d) exceeds limits(%d)\n",
+ if (qp->sq.wqe_cnt > MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_qp_sz)) {
+ mlx5_ib_dbg(dev, "send queue size (%d * %d / %d -> %d) exceeds limits(%u)\n",
attr->cap.max_send_wr, wqe_size, MLX5_SEND_WQE_BB,
qp->sq.wqe_cnt,
- 1 << MLX5_CAP_GEN(dev->mdev, log_max_qp_sz));
+ MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_qp_sz));
return -ENOMEM;
}
qp->sq.wqe_shift = ilog2(MLX5_SEND_WQE_BB);
@@ -663,10 +663,10 @@ static int set_user_buf_size(struct mlx5_ib_dev *dev,
qp->sq.wqe_cnt = ucmd->sq_wqe_count;
- if (qp->sq.wqe_cnt > (1 << MLX5_CAP_GEN(dev->mdev, log_max_qp_sz))) {
- mlx5_ib_warn(dev, "wqe_cnt %d, max_wqes %d\n",
+ if (qp->sq.wqe_cnt > MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_qp_sz)) {
+ mlx5_ib_warn(dev, "wqe_cnt %d, max_wqes %u\n",
qp->sq.wqe_cnt,
- 1 << MLX5_CAP_GEN(dev->mdev, log_max_qp_sz));
+ MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_qp_sz));
return -EINVAL;
}
@@ -2155,7 +2155,7 @@ static int create_dci(struct mlx5_ib_dev *dev, struct ib_pd *pd,
ucmd->rq_wqe_count != qp->rq.wqe_cnt)
return -EINVAL;
- if (ucmd->sq_wqe_count > (1 << MLX5_CAP_GEN(mdev, log_max_qp_sz)))
+ if (ucmd->sq_wqe_count > MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_qp_sz))
return -EINVAL;
ts_format = get_qp_ts_format(dev, to_mcq(init_attr->send_cq),
@@ -2322,7 +2322,7 @@ static int create_user_qp(struct mlx5_ib_dev *dev, struct ib_pd *pd,
ucmd->rq_wqe_count != qp->rq.wqe_cnt)
return -EINVAL;
- if (ucmd->sq_wqe_count > (1 << MLX5_CAP_GEN(mdev, log_max_qp_sz)))
+ if (ucmd->sq_wqe_count > MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_qp_sz))
return -EINVAL;
if (init_attr->qp_type != IB_QPT_RAW_PACKET) {
@@ -4809,19 +4809,15 @@ static bool mlx5_ib_modify_qp_allowed(struct mlx5_ib_dev *dev,
static int validate_rd_atomic(struct mlx5_ib_dev *dev, struct ib_qp_attr *attr,
int attr_mask, enum ib_qp_type qp_type)
{
- int log_max_ra_res;
- int log_max_ra_req;
+ u32 log_max_ra_res;
+ u32 log_max_ra_req;
if (qp_type == MLX5_IB_QPT_DCI) {
- log_max_ra_res = 1 << MLX5_CAP_GEN(dev->mdev,
- log_max_ra_res_dc);
- log_max_ra_req = 1 << MLX5_CAP_GEN(dev->mdev,
- log_max_ra_req_dc);
+ log_max_ra_res = MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_ra_res_dc);
+ log_max_ra_req = MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_ra_req_dc);
} else {
- log_max_ra_res = 1 << MLX5_CAP_GEN(dev->mdev,
- log_max_ra_res_qp);
- log_max_ra_req = 1 << MLX5_CAP_GEN(dev->mdev,
- log_max_ra_req_qp);
+ log_max_ra_res = MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_ra_res_qp);
+ log_max_ra_req = MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_ra_req_qp);
}
if (attr_mask & IB_QP_MAX_QP_RD_ATOMIC &&
@@ -5511,7 +5507,7 @@ static int set_user_rq_size(struct mlx5_ib_dev *dev,
struct mlx5_ib_rwq *rwq)
{
/* Sanity check RQ size before proceeding */
- if (wq_init_attr->max_wr > (1 << MLX5_CAP_GEN(dev->mdev, log_max_wq_sz)))
+ if (wq_init_attr->max_wr > MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_wq_sz))
return -EINVAL;
if (!ucmd->rq_wqe_count)
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 20/23] RDMA/mlx5: Clamp log_max_* shifts in QP and WQ limit checks
2026-11-03 16:39 ` [PATCH rdma-next 20/23] RDMA/mlx5: Clamp log_max_* shifts in QP and WQ limit checks Yishai Hadas
@ 2026-11-07 20:25 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:25 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found more severe
issue(s) in other patches of this series. This mail is only an FYI, in case
you want to address the nit picks below when you respin.
Low priority / nit picks for your consideration:
- [Low] The commit message says each limit is computed from "an 8-bit
log_max_* FW capability".
--
RDMA/mlx5: Clamp log_max_* shifts in QP and WQ limit checks
This commit replaces several open-coded 1 << MLX5_CAP_GEN(...) limit
calculations in drivers/infiniband/hw/mlx5/qp.c with
MLX5_CAP_GEN_LOGSZ_U32(). This avoids an undefined shift when a NIC
reports a log_max_* capability of 31 or more. The affected checks are in
set_rq_size(), calc_sq_size(), set_user_buf_size(), create_dci(),
create_user_qp(), validate_rd_atomic() and set_user_rq_size().
> Several QP and WQ size checks in qp.c compute their limit as
> 1 << cap, where cap is an 8-bit log_max_* FW capability. An untrusted
> NIC can report 31 or more, which makes the shift undefined.
[Severity: Low]
This isn't a bug, but is "an 8-bit log_max_* FW capability" accurate for
all the caps this patch touches?
In struct mlx5_ifc_cmd_hca_cap_bits in include/linux/mlx5/mlx5_ifc.h,
only log_max_qp_sz is 8 bits wide:
u8 log_max_qp_sz[0x8];
The rd_atomic caps used by validate_rd_atomic() are 6 bits wide:
u8 log_max_ra_req_dc[0x6];
u8 log_max_ra_res_dc[0x6];
u8 log_max_ra_req_qp[0x6];
u8 log_max_ra_res_qp[0x6];
The cap used by set_user_rq_size() is 5 bits wide, so its maximum value
is exactly 31:
u8 log_max_wq_sz[0x5];
The main point still holds, since every one of these fields can reach 31
and the shift can be undefined.
Could the wording be changed to something like "a log_max_* FW capability
of up to 8 bits"?
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 21/23] RDMA/mlx5: Clamp log_max_* shifts in query_device and ucontext paths
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (19 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 20/23] RDMA/mlx5: Clamp log_max_* shifts in QP and WQ limit checks Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:25 ` netdev-bot+sashiko
2026-11-03 16:39 ` [PATCH rdma-next 22/23] RDMA/core: Report and enforce adversarial mode via GET_CONTEXT Yishai Hadas
2026-11-03 16:39 ` [PATCH rdma-next 23/23] RDMA/mlx5: Advertise adversarial mode via ib_device::adversarial_mode Yishai Hadas
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
The device, port and ucontext attributes and the SRQ size limit are
computed as 1 << cap from 8-bit log_max_* FW capabilities. An untrusted
NIC can report 31 or more, which makes the shift undefined. The same
applies to 1ull << log_pg_sz with a value of 64 or more.
Use MLX5_CAP_GEN_LOGSZ_U32() and MLX5_CAP_GEN_LOGSZ_U64() for these
values. As the limits can now reach 2^31, compute max_qp_wr in 64 bits
and saturate the max_res_rd_atom and max_total_mcast_qp_attach products
at U32_MAX.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/hw/mlx5/mad.c | 2 +-
drivers/infiniband/hw/mlx5/main.c | 63 ++++++++++++++++---------------
drivers/infiniband/hw/mlx5/srq.c | 2 +-
3 files changed, 35 insertions(+), 32 deletions(-)
diff --git a/drivers/infiniband/hw/mlx5/mad.c b/drivers/infiniband/hw/mlx5/mad.c
index 8925331855ff..cf9f6a4d5ccf 100644
--- a/drivers/infiniband/hw/mlx5/mad.c
+++ b/drivers/infiniband/hw/mlx5/mad.c
@@ -629,7 +629,7 @@ int mlx5_query_mad_ifc_port(struct ib_device *ibdev, u32 port,
props->phys_state = out_mad->data[33] >> 4;
props->port_cap_flags = be32_to_cpup((__be32 *)(out_mad->data + 20));
props->gid_tbl_len = out_mad->data[50];
- props->max_msg_sz = 1 << MLX5_CAP_GEN(mdev, log_max_msg);
+ props->max_msg_sz = MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_msg);
props->pkey_tbl_len = dev->pkey_table_len;
props->bad_pkey_cntr = be16_to_cpup((__be16 *)(out_mad->data + 46));
props->qkey_viol_cntr = be16_to_cpup((__be16 *)(out_mad->data + 48));
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index c29102260989..ace4cdc95c9d 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -599,7 +599,7 @@ static int mlx5_query_port_roce(struct ib_device *device, u32 port_num,
props->qkey_viol_cntr = qkey_viol_cntr;
}
props->max_mtu = IB_MTU_4096;
- props->max_msg_sz = 1 << MLX5_CAP_GEN(dev->mdev, log_max_msg);
+ props->max_msg_sz = MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_msg);
props->pkey_tbl_len = 1;
props->state = IB_PORT_DOWN;
props->phys_state = IB_PORT_PHYS_STATE_DISABLED;
@@ -930,7 +930,7 @@ static u32 mlx5_ib_calc_max_sq_overhead(void)
static u32 mlx5_ib_calc_max_qp_wr(struct mlx5_ib_dev *dev)
{
struct mlx5_core_dev *mdev = dev->mdev;
- u32 max_wqe_bb_units = 1 << MLX5_CAP_GEN(mdev, log_max_qp_sz);
+ u64 max_wqe_bb_units = MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_qp_sz);
u32 max_wqe_size;
/* max QP overhead + 1 SGE, no inline, no special features */
max_wqe_size = mlx5_ib_calc_max_sq_overhead() +
@@ -940,7 +940,8 @@ static u32 mlx5_ib_calc_max_qp_wr(struct mlx5_ib_dev *dev)
max_wqe_size = ALIGN(max_wqe_size, MLX5_SEND_WQE_BB);
- return (max_wqe_bb_units * MLX5_SEND_WQE_BB) / max_wqe_size;
+ /* Up to 2^31 units, so the product needs 64 bits */
+ return div_u64(max_wqe_bb_units * MLX5_SEND_WQE_BB, max_wqe_size);
}
static int mlx5_ib_query_device(struct ib_device *ibdev,
@@ -954,7 +955,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
int max_sq_desc;
int max_rq_sg;
int max_sq_sg;
- u64 min_page_size = 1ull << MLX5_CAP_GEN(mdev, log_pg_sz);
+ u64 min_page_size = MLX5_CAP_GEN_LOGSZ_U64(mdev, log_pg_sz);
bool raw_support = !mlx5_core_mp_enabled(mdev);
struct mlx5_ib_query_device_resp resp = {};
size_t resp_len;
@@ -1000,7 +1001,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
if (MLX5_CAP_GEN(mdev, imaicl)) {
props->device_cap_flags |= IB_DEVICE_MEM_WINDOW |
IB_DEVICE_MEM_WINDOW_TYPE_2B;
- props->max_mw = 1 << MLX5_CAP_GEN(mdev, log_max_mkey);
+ props->max_mw = MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_mkey);
/* We support 'Gappy' memory registration too */
props->kernel_cap_flags |= IBK_SG_GAPS_REG;
}
@@ -1033,7 +1034,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
if (offsetofend(typeof(resp), tso_caps) <= uhw_outlen) {
max_tso = MLX5_CAP_ETH(mdev, max_lso_cap);
if (max_tso) {
- resp.tso_caps.max_tso = 1 << max_tso;
+ resp.tso_caps.max_tso = BIT(max_tso);
resp.tso_caps.supported_qpts |=
1 << IB_QPT_RAW_PACKET;
resp.response_length += sizeof(resp.tso_caps);
@@ -1100,7 +1101,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
props->max_mr_size = ~0ull;
props->page_size_cap = ~(min_page_size - 1);
- props->max_qp = 1 << MLX5_CAP_GEN(mdev, log_max_qp);
+ props->max_qp = MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_qp);
props->max_qp_wr = mlx5_ib_calc_max_qp_wr(dev);
max_rq_sg = MLX5_CAP_GEN(mdev, max_wqe_sz_rq) /
sizeof(struct mlx5_wqe_data_seg);
@@ -1111,29 +1112,31 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
props->max_send_sge = max_sq_sg;
props->max_recv_sge = max_rq_sg;
props->max_sge_rd = MLX5_MAX_SGE_RD;
- props->max_cq = 1 << MLX5_CAP_GEN(mdev, log_max_cq);
- props->max_cqe = (1 << MLX5_CAP_GEN(mdev, log_max_cq_sz)) - 1;
- props->max_mr = 1 << MLX5_CAP_GEN(mdev, log_max_mkey);
- props->max_pd = 1 << MLX5_CAP_GEN(mdev, log_max_pd);
- props->max_qp_rd_atom = 1 << MLX5_CAP_GEN(mdev, log_max_ra_req_qp);
- props->max_qp_init_rd_atom = 1 << MLX5_CAP_GEN(mdev, log_max_ra_res_qp);
- props->max_srq = 1 << MLX5_CAP_GEN(mdev, log_max_srq);
- props->max_srq_wr = (1 << MLX5_CAP_GEN(mdev, log_max_srq_sz)) - 1;
+ props->max_cq = MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_cq);
+ props->max_cqe = MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_cq_sz) - 1;
+ props->max_mr = MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_mkey);
+ props->max_pd = MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_pd);
+ props->max_qp_rd_atom = MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_ra_req_qp);
+ props->max_qp_init_rd_atom = MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_ra_res_qp);
+ props->max_srq = MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_srq);
+ props->max_srq_wr = MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_srq_sz) - 1;
props->local_ca_ack_delay = MLX5_CAP_GEN(mdev, local_ca_ack_delay);
- props->max_res_rd_atom = props->max_qp_rd_atom * props->max_qp;
+ props->max_res_rd_atom = min_t(u64, (u64)props->max_qp_rd_atom * props->max_qp,
+ U32_MAX);
props->max_srq_sge = max_rq_sg - 1;
props->max_fast_reg_page_list_len =
- 1 << MLX5_CAP_GEN(mdev, log_max_klm_list_size);
+ MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_klm_list_size);
props->max_pi_fast_reg_page_list_len =
props->max_fast_reg_page_list_len / 2;
props->max_sgl_rd =
MLX5_CAP_GEN(mdev, max_sgl_for_optimized_performance);
get_atomic_caps_qp(dev, props);
props->masked_atomic_cap = IB_ATOMIC_NONE;
- props->max_mcast_grp = 1 << MLX5_CAP_GEN(mdev, log_max_mcg);
+ props->max_mcast_grp = MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_mcg);
props->max_mcast_qp_attach = MLX5_CAP_GEN(mdev, max_qp_mcg);
- props->max_total_mcast_qp_attach = props->max_mcast_qp_attach *
- props->max_mcast_grp;
+ props->max_total_mcast_qp_attach =
+ min_t(u64, (u64)props->max_mcast_qp_attach * props->max_mcast_grp,
+ U32_MAX);
props->max_ah = INT_MAX;
props->hca_core_clock = MLX5_CAP_GEN(mdev, device_frequency_khz);
props->timestamp_mask = 0x7FFFFFFFFFFFFFFFULL;
@@ -1167,19 +1170,19 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
if (mlx5_ib_port_link_layer(ibdev, 1) ==
IB_LINK_LAYER_ETHERNET && raw_support) {
props->rss_caps.max_rwq_indirection_tables =
- 1 << MLX5_CAP_GEN(dev->mdev, log_max_rqt);
+ MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_rqt);
props->rss_caps.max_rwq_indirection_table_size =
- 1 << MLX5_CAP_GEN(dev->mdev, log_max_rqt_size);
+ MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_rqt_size);
props->rss_caps.supported_qpts = 1 << IB_QPT_RAW_PACKET;
props->max_wq_type_rq =
- 1 << MLX5_CAP_GEN(dev->mdev, log_max_rq);
+ MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_rq);
}
if (MLX5_CAP_GEN(mdev, tag_matching)) {
props->tm_caps.max_num_tags =
- (1 << MLX5_CAP_GEN(mdev, log_tag_matching_list_sz)) - 1;
+ MLX5_CAP_GEN_LOGSZ_U32(mdev, log_tag_matching_list_sz) - 1;
props->tm_caps.max_ops =
- 1 << MLX5_CAP_GEN(mdev, log_max_qp_sz);
+ MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_qp_sz);
props->tm_caps.max_sge = MLX5_TM_MAX_SGE;
}
@@ -1503,7 +1506,7 @@ static int mlx5_query_hca_port(struct ib_device *ibdev, u32 port,
props->port_cap_flags &= ~IB_PORT_CM_SUP;
props->gid_tbl_len = mlx5_get_gid_table_len(MLX5_CAP_GEN(mdev, gid_table_size));
- props->max_msg_sz = 1 << MLX5_CAP_GEN(mdev, log_max_msg);
+ props->max_msg_sz = MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_msg);
props->pkey_tbl_len = mlx5_to_sw_pkey_sz(MLX5_CAP_GEN(mdev, pkey_table_size));
props->bad_pkey_cntr = rep->pkey_violation_counter;
props->qkey_viol_cntr = rep->qkey_violation_counter;
@@ -2127,16 +2130,16 @@ static int set_ucontext_resp(struct ib_ucontext *uctx,
MLX5_IB_ALLOC_UCONTEXT_RESP_MASK_DUMP_FILL_MKEY;
}
- resp->qp_tab_size = 1 << MLX5_CAP_GEN(dev->mdev, log_max_qp);
+ resp->qp_tab_size = MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_qp);
if (mlx5_wc_support_get(dev->mdev))
resp->bf_reg_size = 1 << MLX5_CAP_GEN(dev->mdev,
log_bf_reg_size);
resp->cache_line_size = cache_line_size();
resp->max_sq_desc_sz = MLX5_CAP_GEN(dev->mdev, max_wqe_sz_sq);
resp->max_rq_desc_sz = MLX5_CAP_GEN(dev->mdev, max_wqe_sz_rq);
- resp->max_send_wqebb = 1 << MLX5_CAP_GEN(dev->mdev, log_max_qp_sz);
- resp->max_recv_wr = 1 << MLX5_CAP_GEN(dev->mdev, log_max_qp_sz);
- resp->max_srq_recv_wr = 1 << MLX5_CAP_GEN(dev->mdev, log_max_srq_sz);
+ resp->max_send_wqebb = MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_qp_sz);
+ resp->max_recv_wr = MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_qp_sz);
+ resp->max_srq_recv_wr = MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_srq_sz);
resp->cqe_version = context->cqe_version;
resp->log_uar_size = MLX5_CAP_GEN(dev->mdev, uar_4k) ?
MLX5_ADAPTER_PAGE_SHIFT : PAGE_SHIFT;
diff --git a/drivers/infiniband/hw/mlx5/srq.c b/drivers/infiniband/hw/mlx5/srq.c
index 6a4a937d97ce..3fbb0eaf1255 100644
--- a/drivers/infiniband/hw/mlx5/srq.c
+++ b/drivers/infiniband/hw/mlx5/srq.c
@@ -198,7 +198,7 @@ int mlx5_ib_create_srq(struct ib_srq *ib_srq,
size_t buf_size;
int err;
struct mlx5_srq_attr in = {};
- __u32 max_srq_wqes = 1 << MLX5_CAP_GEN(dev->mdev, log_max_srq_sz);
+ __u32 max_srq_wqes = MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_srq_sz);
__u32 max_sge_sz = MLX5_CAP_GEN(dev->mdev, max_wqe_sz_rq) /
sizeof(struct mlx5_wqe_data_seg);
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 21/23] RDMA/mlx5: Clamp log_max_* shifts in query_device and ucontext paths
2026-11-03 16:39 ` [PATCH rdma-next 21/23] RDMA/mlx5: Clamp log_max_* shifts in query_device and ucontext paths Yishai Hadas
@ 2026-11-07 20:25 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:25 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found 3 potential
issue(s) to consider.
Critical: 0 · High: 0 · Medium: 1 · Low: 2
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [Medium] mlx5_ib_create_srq() (drivers/infiniband/hw/mlx5/srq.c) now
takes its SRQ size limit from MLX5_CAP_GEN_LOGSZ_U32(log_max_srq_sz).
- [Low] The commit message does not fully match the diff.
- [Low] When FW reports log_max_cq_sz >= 31, mlx5_ib_query_device() now
advertises props->max_cqe = MLX5_CAP_GEN_LOGSZ_U32(log_max_cq_sz) - 1…
--
RDMA/mlx5: Clamp log_max_* shifts in query_device and ucontext paths
This commit replaces the open-coded 1 << cap computations of FW log_max_*
capabilities with MLX5_CAP_GEN_LOGSZ_U32() and MLX5_CAP_GEN_LOGSZ_U64().
This covers the query_device, query_port, ucontext and SRQ create paths.
It also computes max_qp_wr in 64 bits and saturates the max_res_rd_atom
and max_total_mcast_qp_attach products at U32_MAX.
> The device, port and ucontext attributes and the SRQ size limit are
> computed as 1 << cap from 8-bit log_max_* FW capabilities. An untrusted
> NIC can report 31 or more, which makes the shift undefined. The same
> applies to 1ull << log_pg_sz with a value of 64 or more.
[Severity: Low]
This isn't a bug, but is the "8-bit log_max_*" description accurate? In
mlx5_ifc.h several of the converted fields are narrower, for example:
u8 log_max_msg[0x5];
log_tag_matching_list_sz is also 5 bits wide. log_max_klm_list_size and
log_max_ra_req_qp are 6 bits.
The diff also changes the tso_caps computation in mlx5_ib_query_device():
resp.tso_caps.max_tso = BIT(max_tso);
Here max_tso comes from the 5-bit Ethernet cap MLX5_CAP_ETH(max_lso_cap),
not from a log_max_* general cap, and the commit message doesn't mention
this change. The BIT() conversion itself looks correct.
Could the message be updated to cover this change and to fix the field
width?
> diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
> index c29102260989f..ace4cdc95c9da 100644
> --- a/drivers/infiniband/hw/mlx5/main.c
> +++ b/drivers/infiniband/hw/mlx5/main.c
[ ... ]
> @@ -1111,29 +1112,31 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
> props->max_send_sge = max_sq_sg;
> props->max_recv_sge = max_rq_sg;
> props->max_sge_rd = MLX5_MAX_SGE_RD;
> - props->max_cq = 1 << MLX5_CAP_GEN(mdev, log_max_cq);
> - props->max_cqe = (1 << MLX5_CAP_GEN(mdev, log_max_cq_sz)) - 1;
> + props->max_cq = MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_cq);
> + props->max_cqe = MLX5_CAP_GEN_LOGSZ_U32(mdev, log_max_cq_sz) - 1;
[Severity: Low]
If FW reports log_max_cq_sz >= 31, this advertises max_cqe as 2^31 - 1.
Can the CQ create path actually honour that value?
mlx5_ib_create_user_cq() in cq.c rounds the requested size up and then
checks it against INT_MAX:
if (entries > max_cq_entries || entries > INT_MAX)
return -EINVAL;
cq_roundup_entries() is 1ULL << fls64(cqe), so any cqe in
[2^30, 2^31 - 1] rounds up to 2^31 and gets -EINVAL. That makes the real
maximum 2^30 - 1.
Would a consumer that sizes its CQ from attrs.max_cqe then fail to create
the CQ? For log_max_cq_sz <= 30 the two values agree.
[ ... ]
> diff --git a/drivers/infiniband/hw/mlx5/srq.c b/drivers/infiniband/hw/mlx5/srq.c
> index 6a4a937d97ce9..3fbb0eaf12558 100644
> --- a/drivers/infiniband/hw/mlx5/srq.c
> +++ b/drivers/infiniband/hw/mlx5/srq.c
> @@ -198,7 +198,7 @@ int mlx5_ib_create_srq(struct ib_srq *ib_srq,
> size_t buf_size;
> int err;
> struct mlx5_srq_attr in = {};
> - __u32 max_srq_wqes = 1 << MLX5_CAP_GEN(dev->mdev, log_max_srq_sz);
> + __u32 max_srq_wqes = MLX5_CAP_GEN_LOGSZ_U32(dev->mdev, log_max_srq_sz);
[Severity: Medium]
With log_max_srq_sz >= 31 this limit now saturates at 2^31. Can the rest
of mlx5_ib_create_srq() handle queue sizes that large?
In adversarial mode the caps filter passes log_max_srq_sz through
unchanged (MLX5_HCA_CAP_ALLOW in caps_filter.c). A uverbs user could then
request max_wr = 0x7fffffff, which passes the
init_attr->attr.max_wr >= max_srq_wqes check. After that:
srq->msrq.max = roundup_pow_of_two(init_attr->attr.max_wr + 1);
This stores 2^31 into the int max field of struct mlx5_core_srq, which
becomes INT_MIN.
buf_size = srq->msrq.max * desc_size;
if (buf_size < desc_size)
return -EINVAL;
When msrq.max >= 2^26, this size_t product is a power of two >= 2^31. It
passes the wrap check. However, create_srq_user() and create_srq_kernel()
take int buf_size, so the value is truncated to 0 or INT_MIN.
On the user path the truncated value becomes the min_size passed to
ib_umem_get_desc_check(), so a dmabuf descriptor of any size is accepted.
in.log_size = ilog2(srq->msrq.max) still describes the full queue to FW.
mlx5_cmd_create_srq()->create_srq_cmd()->__set_srq_page_size() then calls
get_pas_size(), which computes:
u32 rq_sz = 1 << (log_srq_size + 4 + log_rq_stride);
Here the shift count is 32 to 36. That is the same kind of undefined shift
this commit removes.
The garbage result feeds the WARN_ON() that compares against
ib_umem_num_dma_blocks() in __set_srq_page_size(). A local user could
make that WARN fire, or create an SRQ whose FW-declared size doesn't match
the pinned buffer.
Finally, init_attr->attr.max_wr = srq->msrq.max - 1 is a signed overflow
when msrq.max is INT_MIN.
The downstream overflow could already be reached with log_max_srq_sz
values of 26 to 30. However, the commit message notes "the limits can now
reach 2^31" and fixes other consumers of 2^31 values.
Should the SRQ path get a tighter bound as well? For example, it could
reject msrq.max > INT_MAX / desc_size, or keep
log_size + 4 + wqe_shift below 32. The sibling patch "RDMA/mlx5: Clamp
log_max_cq_sz shift in CQ size checks" adds a similar INT_MAX bound for
CQs.
> __u32 max_sge_sz = MLX5_CAP_GEN(dev->mdev, max_wqe_sz_rq) /
> sizeof(struct mlx5_wqe_data_seg);
>
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH rdma-next 22/23] RDMA/core: Report and enforce adversarial mode via GET_CONTEXT
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (20 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 21/23] RDMA/mlx5: Clamp log_max_* shifts in query_device and ucontext paths Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-03 16:39 ` [PATCH rdma-next 23/23] RDMA/mlx5: Advertise adversarial mode via ib_device::adversarial_mode Yishai Hadas
22 siblings, 0 replies; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
Userspace must know when a device is in adversarial mode, and an
unaware rdma-core must not get a context on such a device, as it
doesn't harden itself against the NIC.
Report the mode with IB_UVERBS_CORE_SUPPORT_ADVERSARIAL in the
core_support of GET_CONTEXT and QUERY_CONTEXT. Add an optional
UVERBS_ATTR_GET_CONTEXT_FLAGS input, where userspace declares
UVERBS_ATTR_GET_CONTEXT_FLAG_ADVERSARIAL_SUPPORTED. In adversarial
mode, GET_CONTEXT without that flag fails with -EPERM, and so does the
legacy write() GET_CONTEXT, which has no way to pass it.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/core/uverbs_cmd.c | 5 +++++
.../infiniband/core/uverbs_std_types_device.c | 16 ++++++++++++++++
include/uapi/rdma/ib_user_ioctl_cmds.h | 5 +++++
include/uapi/rdma/ib_user_ioctl_verbs.h | 1 +
4 files changed, 27 insertions(+)
diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c
index aca7c6ab55cd..b78207a81091 100644
--- a/drivers/infiniband/core/uverbs_cmd.c
+++ b/drivers/infiniband/core/uverbs_cmd.c
@@ -292,6 +292,11 @@ static int ib_uverbs_get_context(struct uverbs_attr_bundle *attrs)
goto err_ucontext;
}
+ if (ib_dev->adversarial_mode) {
+ ret = -EPERM;
+ goto err_uobj;
+ }
+
resp = (struct ib_uverbs_get_context_resp){
.num_comp_vectors = attrs->ufile->device->num_comp_vectors,
.async_fd = uobj->id,
diff --git a/drivers/infiniband/core/uverbs_std_types_device.c b/drivers/infiniband/core/uverbs_std_types_device.c
index db65032e80b4..72c67f09533e 100644
--- a/drivers/infiniband/core/uverbs_std_types_device.c
+++ b/drivers/infiniband/core/uverbs_std_types_device.c
@@ -248,6 +248,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_GET_CONTEXT)(
u32 num_comp = attrs->ufile->device->num_comp_vectors;
u64 core_support = IB_UVERBS_CORE_SUPPORT_OPTIONAL_MR_ACCESS;
struct ib_device *ib_dev;
+ u32 ctx_flags;
int ret;
ib_dev = srcu_dereference(attrs->ufile->device->ib_dev,
@@ -255,6 +256,15 @@ static int UVERBS_HANDLER(UVERBS_METHOD_GET_CONTEXT)(
if (!ib_dev)
return -EIO;
+ ret = uverbs_get_flags32(&ctx_flags, attrs, UVERBS_ATTR_GET_CONTEXT_FLAGS,
+ UVERBS_ATTR_GET_CONTEXT_FLAG_ADVERSARIAL_SUPPORTED);
+ if (ret)
+ return ret;
+
+ if (ib_dev->adversarial_mode &&
+ !(ctx_flags & UVERBS_ATTR_GET_CONTEXT_FLAG_ADVERSARIAL_SUPPORTED))
+ return -EPERM;
+
ret = uverbs_copy_to(attrs, UVERBS_ATTR_GET_CONTEXT_NUM_COMP_VECTORS,
&num_comp, sizeof(num_comp));
if (IS_UVERBS_COPY_ERR(ret))
@@ -262,6 +272,8 @@ static int UVERBS_HANDLER(UVERBS_METHOD_GET_CONTEXT)(
if (ib_dev->ops.uverbs_robust_udata)
core_support |= IB_UVERBS_CORE_SUPPORT_ROBUST_UDATA;
+ if (ib_dev->adversarial_mode)
+ core_support |= IB_UVERBS_CORE_SUPPORT_ADVERSARIAL;
ret = uverbs_copy_to(attrs, UVERBS_ATTR_GET_CONTEXT_CORE_SUPPORT,
&core_support, sizeof(core_support));
if (IS_UVERBS_COPY_ERR(ret))
@@ -303,6 +315,8 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_CONTEXT)(
if (IS_UVERBS_COPY_ERR(ret))
return ret;
+ if (ib_dev->adversarial_mode)
+ core_support |= IB_UVERBS_CORE_SUPPORT_ADVERSARIAL;
ret = uverbs_copy_to(attrs, UVERBS_ATTR_QUERY_CONTEXT_CORE_SUPPORT,
&core_support, sizeof(core_support));
if (IS_UVERBS_COPY_ERR(ret))
@@ -519,6 +533,8 @@ DECLARE_UVERBS_NAMED_METHOD(
UVERBS_ATTR_MIN_SIZE(sizeof(int)),
UA_OPTIONAL,
UA_ALLOC_AND_COPY),
+ UVERBS_ATTR_FLAGS_IN(UVERBS_ATTR_GET_CONTEXT_FLAGS, u32,
+ UA_OPTIONAL),
UVERBS_ATTR_UHW());
DECLARE_UVERBS_NAMED_METHOD(
diff --git a/include/uapi/rdma/ib_user_ioctl_cmds.h b/include/uapi/rdma/ib_user_ioctl_cmds.h
index 816b3107464f..3e482a4338f9 100644
--- a/include/uapi/rdma/ib_user_ioctl_cmds.h
+++ b/include/uapi/rdma/ib_user_ioctl_cmds.h
@@ -101,10 +101,15 @@ enum uverbs_attrs_query_comp_cntr_caps_attr_ids {
UVERBS_ATTR_QUERY_COMP_CNTR_CAPS_SUPPORTED_QP_ATTACH_OPS,
};
+enum uverbs_attrs_get_context_flags {
+ UVERBS_ATTR_GET_CONTEXT_FLAG_ADVERSARIAL_SUPPORTED = 1 << 0,
+};
+
enum uverbs_attrs_get_context_attr_ids {
UVERBS_ATTR_GET_CONTEXT_NUM_COMP_VECTORS,
UVERBS_ATTR_GET_CONTEXT_CORE_SUPPORT,
UVERBS_ATTR_GET_CONTEXT_FD_ARR,
+ UVERBS_ATTR_GET_CONTEXT_FLAGS,
};
enum uverbs_attrs_query_context_attr_ids {
diff --git a/include/uapi/rdma/ib_user_ioctl_verbs.h b/include/uapi/rdma/ib_user_ioctl_verbs.h
index 21f86cc7bb1f..5861f34f8343 100644
--- a/include/uapi/rdma/ib_user_ioctl_verbs.h
+++ b/include/uapi/rdma/ib_user_ioctl_verbs.h
@@ -47,6 +47,7 @@
enum ib_uverbs_core_support {
IB_UVERBS_CORE_SUPPORT_OPTIONAL_MR_ACCESS = 1 << 0,
IB_UVERBS_CORE_SUPPORT_ROBUST_UDATA = 1 << 1,
+ IB_UVERBS_CORE_SUPPORT_ADVERSARIAL = 1 << 2,
};
enum ib_uverbs_access_flags {
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH rdma-next 23/23] RDMA/mlx5: Advertise adversarial mode via ib_device::adversarial_mode
2026-11-03 16:39 [PATCH rdma-next 00/23] mlx5: Adversarial mode for confidential computing guests Yishai Hadas
` (21 preceding siblings ...)
2026-11-03 16:39 ` [PATCH rdma-next 22/23] RDMA/core: Report and enforce adversarial mode via GET_CONTEXT Yishai Hadas
@ 2026-11-03 16:39 ` Yishai Hadas
2026-11-07 20:25 ` netdev-bot+sashiko
22 siblings, 1 reply; 48+ messages in thread
From: Yishai Hadas @ 2026-11-03 16:39 UTC (permalink / raw)
To: jgg, leon
Cc: linux-rdma, netdev, saeedm, tariqt, mbloch, kuba, pabeni,
edumazet, davem, andrew+netdev, dtatulea, moshe, yishaih, maorg
Set ib_device::adversarial_mode during capability init so that the
uverbs core layer can report adversarial mode to rdma-core and enforce
that only adversarial-mode-aware userspace opens a context on this
device.
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
drivers/infiniband/hw/mlx5/main.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index ace4cdc95c9d..f8af90941b53 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -4778,6 +4778,7 @@ static int mlx5_ib_stage_caps_init(struct mlx5_ib_dev *dev)
}
dev->ib_dev.use_cq_dim = true;
+ dev->ib_dev.adversarial_mode = mlx5_ib_adversarial_mode;
return 0;
--
2.18.1
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH rdma-next 23/23] RDMA/mlx5: Advertise adversarial mode via ib_device::adversarial_mode
2026-11-03 16:39 ` [PATCH rdma-next 23/23] RDMA/mlx5: Advertise adversarial mode via ib_device::adversarial_mode Yishai Hadas
@ 2026-11-07 20:25 ` netdev-bot+sashiko
0 siblings, 0 replies; 48+ messages in thread
From: netdev-bot+sashiko @ 2026-11-07 20:25 UTC (permalink / raw)
To: yishaih
Cc: jgg, leon, linux-rdma, netdev, saeedm, tariqt, mbloch, kuba,
pabeni, edumazet, davem, andrew+netdev, dtatulea, moshe, maorg
Thank you for your contribution! Sashiko AI review found 2 potential
issue(s) to consider.
Critical: 0 · High: 0 · Medium: 1 · Low: 1
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [Medium] The series is not bisectable on confidential-computing (CoCo)
guests.
- [Low] The commit message understates what this one line does.
--
RDMA/mlx5: Advertise adversarial mode via ib_device::adversarial_mode
This commit sets ib_device::adversarial_mode from the
mlx5_ib_adversarial_mode global in mlx5_ib_stage_caps_init(). The uverbs
core can then report adversarial mode to rdma-core and only let
adversarial-mode-aware userspace open a context on the device.
> diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
> index ace4cdc95c9da..f8af90941b53d 100644
> --- a/drivers/infiniband/hw/mlx5/main.c
> +++ b/drivers/infiniband/hw/mlx5/main.c
> @@ -4778,6 +4778,7 @@ static int mlx5_ib_stage_caps_init(struct mlx5_ib_dev *dev)
> }
>
> dev->ib_dev.use_cq_dim = true;
> + dev->ib_dev.adversarial_mode = mlx5_ib_adversarial_mode;
[Severity: Medium]
Can the series still be bisected on CoCo guests when this assignment
comes last?
An earlier patch, "RDMA/mlx5: Derive mlx5_ib_adversarial_mode global from
mlx5_core adversarial API", sets the global in mlx5_ib_init():
mlx5_ib_adversarial_mode = mlx5_is_adversarial();
This comes from cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT), so it is true
on any SEV-SNP/TDX guest.
Next, "RDMA/mlx5: Block generic CQ poll path in adversarial mode" adds
this to mlx5_poll_one():
if (unlikely(mlx5_ib_adversarial_mode))
return -EOPNOTSUPP;
That patch says this is safe because kernel ULPs don't bind to the device.
However, kernel ULPs only stop binding once ib_device_check_mandatory()
sees the core flag:
if (device->adversarial_mode)
device->kverbs_provider = false;
mlx5 doesn't set ib_dev.adversarial_mode until this commit.
That leaves a window from the CQ poll blocking patch up to "RDMA/core:
Report and enforce adversarial mode via GET_CONTEXT". In that window,
kverbs_provider stays true for mlx5 on a CoCo guest.
add_client_context() then binds MAD, CM, rdma_cm, IPoIB, SRP, iSER,
NVMe-oF and the other clients. Their CQs use mlx5_poll_one():
mlx5_ib_poll_cq()
cq->poll_one() == mlx5_poll_one()
returns -EOPNOTSUPP
mlx5_ib_poll_cq() stops at the first non-zero return, so it always reports
0 completions. Wouldn't MAD sends time out, the IPoIB TX ring fill up, and
NVMe-oF/SRP I/O hang in those intermediate commits?
In the same window, legacy rdma-core would still get contexts, and uverbs
POLL_CQ would always come back empty.
Could this assignment move into the patch that enables the driver-side
mode? Another option is to place it right after "RDMA/core: Suppress
kverbs_provider for adversarial_mode devices", before any hardening patch
that changes behaviour. Either way, every commit in the series would keep
working.
[Severity: Low]
This isn't a bug, but the commit message only says:
Set ib_device::adversarial_mode during capability init so that the
uverbs core layer can report adversarial mode to rdma-core and enforce
that only adversarial-mode-aware userspace opens a context on this
device.
In practice, this line turns on the full adversarial policy for every
mlx5 ib_device on a CC_ATTR_GUEST_MEM_ENCRYPT guest. The pf, raw_eth/rep
and plane profiles all run MLX5_IB_STAGE_CAPS.
With the flag set, ib_device_check_mandatory() clears kverbs_provider.
add_client_context() then skips every client that lacks no_kverbs_req:
if (!device->kverbs_provider && !client->no_kverbs_req)
return 0;
Only uverbs has no_kverbs_req. As a result, MAD, ib_umad, IB CM, SA,
rdma_cm, IPoIB, SRP, iSER, NVMe-oF, NFS/RDMA, RDS and SMC-R no longer
bind.
Because cma and SA never bind, librdmacm users going through ucma can't
use the device either, even if they are adversarial-aware.
Unmodified rdma-core also gets -EPERM, both from GET_CONTEXT and from the
legacy write() path.
There is no opt-out, since the decision comes only from
cc_platform_has(). Other patches in the series say this is intended, but
this commit is where it takes effect for mlx5.
Could the commit message say that, once this is applied, kernel ULPs and
rdma_cm-based userspace stop working on mlx5 in CoCo guests?
>
> return 0;
>
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261103163949.1554-1-yishaih%40nvidia.com
^ permalink raw reply [flat|nested] 48+ messages in thread