[PATCH] RDMA/rxe: check the PD when invalidating an MR or MW

[PATCH] RDMA/rxe: check the PD when invalidating an MR or MW

[Yehyeong Lee]

rxe_invalidate_mr() and rxe_invalidate_mw() look the object up by index in
the device-wide pool and then only compare the key, so a QP can invalidate
an MR or MW that belongs to a different protection domain on the same rxe
device.  Both entry points are affected: IB_WR_LOCAL_INV from the requester
and SEND_WITH_INVALIDATE from the responder, where the rkey comes off the
wire.

Everything else that resolves one of these objects already refuses a
foreign PD.  lookup_mr() rejects mr_pd(mr) != pd, rxe_reg_fast_mr() rejects
qp->ibqp.pd != mr->ibmr.pd, and rxe_check_bind_mw() was given the same
check in commit fbbe1d5cd7d4 ("RDMA/rxe: Check PDs for memory window
binds").  Remote access is therefore scoped to a PD while remote
invalidation is not.

A peer on one connection can invalidate a kernel ULP's fast-reg MR on
another, which ends that ULP's I/O.  Reject a foreign PD in both
functions; -EINVAL turns into RESPST_ERR_INVALIDATE_RKEY for the
responder and a failed work request for the requester.

The MR side has behaved this way since the driver was merged, where
do_complete() freed the MR it found by index; rxe_invalidate_mw() arrived
later carrying the same omission.

Fixes: 8700e3e7c485 ("Soft RoCE driver")
Cc: stable@vger.kernel.org
Signed-off-by: Yehyeong Lee <yhlee@isslab.korea.ac.kr>
---
Reproduced on rxe with two protection domains on one device.  PD-A holds the
attacking QP, nvme-rdma holds PD-B and is running I/O.  The RXEINV/RXEPD lines
below are instrumentation I added to observe it and are not part of this patch:
RXEINV prints the two PDs and the MR state inside rxe_invalidate_mr(), and the
attacker takes the target rkey from rxe_reg_fast_mr() rather than guessing it.

Before, a SEND_WITH_INVALIDATE naming nvme-rdma's rkey is accepted:

  RXEPD: hunter: TARGET is a foreign-PD fast-reg MR: rkey=0x29ba mr_pd=00000000185f3e0e (mine: A=00000000d715b639 B=000000001440b63b) iova=0xffff888065174000
  RXEPD: hunter: firing SEND_WITH_INV from QP-A(PD-A=00000000d715b639) at rkey=0x29ba
  rdma_rxe: RXEINV: REMOTE entry rxe_resp.c execute() qp=27 wire_rkey=0x29ba
  rdma_rxe: RXEINV: enter rxe_invalidate_mr qp=27 qp_pd=00000000d715b639 mr_pd=00000000185f3e0e cross_pd=1 key=0x29ba state_before=2
  rdma_rxe: RXEINV: done  rxe_invalidate_mr key=0x29ba state_after=1 (RXE_MR_STATE_FREE=1)
  RXEPD: hunter SEND_WITH_INV -> wc.status=0 (success)

and nvme-rdma's I/O stops:

  nvme nvme0: RECV for CQE 0x00000000448b8c67 failed with status WR flushed (5)
  nvme nvme0: starting error recovery
  nvme nvme0: Reconnecting in 10 seconds...

The controller does not recover, because each reconnect registers a new MR that
is invalidated the same way.

After, the same attack is refused:

  RXEPD: hunter: TARGET is a foreign-PD fast-reg MR: rkey=0x90a0 mr_pd=0000000049ba8cc5 (mine: A=00000000a32c3059 B=000000006c5e8389) iova=0xffff88806bbb0000
  RXEPD: hunter: firing SEND_WITH_INV from QP-A(PD-A=00000000a32c3059) at rkey=0x90a0
  RXEPD: hunter SEND_WITH_INV -> wc.status=5 (WR flushed)

rxe_invalidate_mr() is not reached for a foreign PD, so no cross_pd=1 entry is
traced (4 before, 0 after), nvme-rdma keeps running,

  BCHECK[D] DONE iters=524 clean_p0=524 ioerr=0 CORRUPT_HITS=0 POST_COMPLETION=0

Only the MR path is exercised above.  The rxe_invalidate_mw() hunk is the same
omission in the MW path and is fixed for symmetry; I did not reproduce a
cross-PD MW invalidation.

nvme-rdma's own same-PD invalidations keep working: 3180 of them are traced with the
check in place.

 drivers/infiniband/sw/rxe/rxe_mr.c | 6 ++++++
 drivers/infiniband/sw/rxe/rxe_mw.c | 5 +++++
 2 files changed, 11 insertions(+)

diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c
index 71d9ea4772890..2da5134560730 100644
--- a/drivers/infiniband/sw/rxe/rxe_mr.c
+++ b/drivers/infiniband/sw/rxe/rxe_mr.c
@@ -743,6 +743,12 @@ int rxe_invalidate_mr(struct rxe_qp *qp, u32 key)
 		goto err_drop_ref;
 	}
 
+	if (unlikely(qp->ibqp.pd != mr->ibmr.pd)) {
+		rxe_dbg_mr(mr, "qp->pd and mr->pd don't match\n");
+		ret = -EINVAL;
+		goto err_drop_ref;
+	}
+
 	if (atomic_read(&mr->num_mw) > 0) {
 		rxe_dbg_mr(mr, "Attempt to invalidate an MR while bound to MWs\n");
 		ret = -EINVAL;
diff --git a/drivers/infiniband/sw/rxe/rxe_mw.c b/drivers/infiniband/sw/rxe/rxe_mw.c
index bddb7a2578313..ab2212eabd71e 100644
--- a/drivers/infiniband/sw/rxe/rxe_mw.c
+++ b/drivers/infiniband/sw/rxe/rxe_mw.c
@@ -276,6 +276,11 @@ int rxe_invalidate_mw(struct rxe_qp *qp, u32 rkey)
 		goto err_drop_ref;
 	}
 
+	if (unlikely(qp->pd != rxe_mw_pd(mw))) {
+		ret = -EINVAL;
+		goto err_drop_ref;
+	}
+
 	spin_lock_bh(&mw->lock);
 
 	ret = rxe_check_invalidate_mw(qp, mw);
-- 
2.43.0

Re: [PATCH] RDMA/rxe: check the PD when invalidating an MR or MW

[sashiko-bot]

> rxe_invalidate_mr() and rxe_invalidate_mw() look the object up by index in
> the device-wide pool and then only compare the key, so a QP can invalidate
> an MR or MW that belongs to a different protection domain on the same rxe
> device.  Both entry points are affected: IB_WR_LOCAL_INV from the requester
> and SEND_WITH_INVALIDATE from the responder, where the rkey comes off the
> wire.
> 
> Everything else that resolves one of these objects already refuses a
> foreign PD.  lookup_mr() rejects mr_pd(mr) != pd, rxe_reg_fast_mr() rejects
> qp->ibqp.pd != mr->ibmr.pd, and rxe_check_bind_mw() was given the same
> check in commit fbbe1d5cd7d4 ("RDMA/rxe: Check PDs for memory window
> binds").  Remote access is therefore scoped to a PD while remote
> invalidation is not.
> 
> A peer on one connection can invalidate a kernel ULP's fast-reg MR on
> another, which ends that ULP's I/O.  Reject a foreign PD in both
> functions; -EINVAL turns into RESPST_ERR_INVALIDATE_RKEY for the
> responder and a failed work request for the requester.
> 
> The MR side has behaved this way since the driver was merged, where
> do_complete() freed the MR it found by index; rxe_invalidate_mw() arrived
> later carrying the same omission.
> 
> Fixes: 8700e3e7c485 ("Soft RoCE driver")
> Cc: stable@vger.kernel.org
> Signed-off-by: Yehyeong Lee <yhlee@isslab.korea.ac.kr>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260929031811.1550334-1-yhlee@isslab.korea.ac.kr?part=1