[PATCH v2 0/2] RDMA/rxe: fix ADVISE_MR prefetch on non-ODP MRs

[PATCH v2 0/2] RDMA/rxe: fix ADVISE_MR prefetch on non-ODP MRs

[Norbert Szetei via B4 Relay]

ADVISE_MR prefetch on a plain MR runs to_ib_umem_odp() on a struct
ib_umem, giving a KASAN slab-out-of-bounds read in
ib_umem_odp_map_dma_and_lock() (splat in patch 2).

v1 checked is_odp_mr() after lookup_mr(). Leon asked for lookup_mr() to
do the check itself, off mr->access. That only holds once mr->access
cannot disagree with the umem, so patch 1 stops the two paths that
assign mr->access after registration from touching IB_ACCESS_ON_DEMAND,
and patch 2 passes the flag to lookup_mr() in both prefetch arms.

Patch 1 adds another failure return to rxe_rereg_user_mr(), so it
depends on ae36a5b609ae ("RDMA/rxe: validate access flags before
swapping the MR's PD"), which moved the validation above the
mr->ibmr.pd swap. The series is based on for-rc.

v1 -> v2:
 - patch 1 (new): reject IB_ACCESS_ON_DEMAND in rxe_rereg_user_mr() and
   rxe_reg_fast_mr(), so mr->access keeps agreeing with the umem
 - patch 2: use lookup_mr(..., IB_ACCESS_ON_DEMAND) instead of an
   explicit is_odp_mr() check; a non-ODP MR now fails with -EINVAL from
   lookup_mr() rather than -EOPNOTSUPP
 - dropped Zhu Yanjun's Reviewed-by, the implementation changed

v1: https://lore.kernel.org/all/521D5E74-89E3-43C0-81C7-AC0BE52591E7@doyensec.com/

Signed-off-by: Norbert Szetei <norbert@doyensec.com>
---
Norbert Szetei (2):
      RDMA/rxe: Reject IB_ACCESS_ON_DEMAND changes after MR creation
      RDMA/rxe: Reject prefetch of a non-ODP MR

 drivers/infiniband/sw/rxe/rxe_mr.c    | 6 ++++++
 drivers/infiniband/sw/rxe/rxe_odp.c   | 4 ++--
 drivers/infiniband/sw/rxe/rxe_verbs.c | 6 ++++++
 3 files changed, 14 insertions(+), 2 deletions(-)
---
base-commit: e22a3627b7151754f07f90ea3d1ab6e85f5d93f4
change-id: 20260913-rxe-advise-mr-v2-97b102b7bb6c

Best regards,
--  
Norbert Szetei <norbert@doyensec.com>

[PATCH v2 1/2] RDMA/rxe: Reject IB_ACCESS_ON_DEMAND changes after MR creation

[Norbert Szetei via B4 Relay]

From: Norbert Szetei <norbert@doyensec.com>

Whether an MR is an ODP MR is decided once, at registration time:
rxe_reg_user_mr() picks rxe_odp_mr_init_user() over rxe_mr_init_user()
based on IB_ACCESS_ON_DEMAND, and only the former builds an ib_umem_odp
via ib_umem_odp_get(). The umem cannot change type afterwards, and
is_odp_mr() reads mr->umem->is_odp.

Two paths assign mr->access after that point and can leave it
describing an MR type the umem does not have:

rxe_rereg_user_mr() with IB_MR_REREG_ACCESS overwrites mr->access with
the caller's value, and IB_ACCESS_ON_DEMAND is part of
RXE_ACCESS_SUPPORTED_MR, so userspace can set the flag on a plain MR or
clear it on an ODP MR while the umem stays what it was.

rxe_reg_fast_mr() takes mr->access from the REG_MR work request
unmasked and moves the MR to RXE_MR_STATE_VALID, on an MR that
rxe_mr_init_fast() left with a NULL umem.

Reject IB_ACCESS_ON_DEMAND in both, so mr->access carries the flag only
for an MR that has an ODP umem and the flag can be used to identify one.

Fixes: 544c7f62cf32 ("RDMA/rxe: Implement rereg_user_mr")
Cc: stable@vger.kernel.org
Signed-off-by: Norbert Szetei <norbert@doyensec.com>
---
 drivers/infiniband/sw/rxe/rxe_mr.c    | 6 ++++++
 drivers/infiniband/sw/rxe/rxe_verbs.c | 6 ++++++
 2 files changed, 12 insertions(+)

diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c
index 71d9ea477289..615da4bb9a38 100644
--- a/drivers/infiniband/sw/rxe/rxe_mr.c
+++ b/drivers/infiniband/sw/rxe/rxe_mr.c
@@ -796,6 +796,12 @@ int rxe_reg_fast_mr(struct rxe_qp *qp, struct rxe_send_wqe *wqe)
 		return -EINVAL;
 	}
 
+	/* an MR with no umem is never an ODP MR */
+	if (unlikely(access & IB_ACCESS_ON_DEMAND)) {
+		rxe_dbg_mr(mr, "access = 0x%x requests ODP\n", access);
+		return -EINVAL;
+	}
+
 	mr->access = access;
 	mr->lkey = key;
 	mr->rkey = key;
diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c
index 3864284522eb..46d0810ea2a7 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.c
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
@@ -1337,6 +1337,12 @@ static struct ib_mr *rxe_rereg_user_mr(struct ib_mr *ibmr, int flags,
 		return ERR_PTR(-EOPNOTSUPP);
 	}
 
+	if ((flags & IB_MR_REREG_ACCESS) &&
+	    ((access ^ mr->access) & IB_ACCESS_ON_DEMAND)) {
+		rxe_err_mr(mr, "cannot change IB_ACCESS_ON_DEMAND\n");
+		return ERR_PTR(-EOPNOTSUPP);
+	}
+
 	if (flags & IB_MR_REREG_PD) {
 		rxe_put(old_pd);
 		rxe_get(pd);

-- 
2.55.0

[PATCH v2 2/2] RDMA/rxe: Reject prefetch of a non-ODP MR

[Norbert Szetei via B4 Relay]

From: Norbert Szetei <norbert@doyensec.com>

rxe_ib_advise_mr_prefetch() and rxe_ib_prefetch_sg_list() look up the MR
by lkey and hand it to rxe_odp_do_pagefault_and_lock() without checking
that it is an ODP MR. That path runs to_ib_umem_odp() on mr->umem, and
for a non-ODP MR mr->umem is a plain struct ib_umem from ib_umem_get(),
so the container_of() in to_ib_umem_odp() lands past the end of the
object and ib_umem_odp_map_dma_and_lock() reads its ib_umem_odp fields
out of bounds.

lookup_mr() validates the lkey, PD, access and state but not the MR
type, and IB_UVERBS_ADVISE_MR_ADVICE_PREFETCH is accepted for any MR.

 BUG: KASAN: slab-out-of-bounds in ib_umem_odp_map_dma_and_lock+0x884/0x8a0
 Read of size 8 at addr ffff88810a3ebcf0 by task advi/921
  ib_umem_odp_map_dma_and_lock+0x884/0x8a0
  rxe_ib_advise_mr+0x543/0xad0
  ib_uverbs_handler_UVERBS_METHOD_ADVISE_MR+0x446/0x530
  ib_uverbs_cmd_verbs+0x2b3c/0x3b20
  ib_uverbs_ioctl+0x1e3/0x310
 Allocated by task 921:
  __ib_umem_get_va+0x13e/0xae0
  rxe_mr_init_user+0x2ae/0xb00
  rxe_reg_user_mr+0x337/0x510
 The buggy address belongs to the object at ffff88810a3ebc80
  which belongs to the cache kmalloc-96 of size 96

Ask lookup_mr() for IB_ACCESS_ON_DEMAND in both the synchronous and
the asynchronous prefetch arm. mr->access carries that flag only for
an MR registered as ODP, so the existing
(access & mr->access) != access test rejects a plain MR and the
prefetch fails with -EINVAL.

Fixes: 3576b0df1588 ("RDMA/rxe: Implement synchronous prefetch for ODP MRs")
Cc: stable@vger.kernel.org
Signed-off-by: Norbert Szetei <norbert@doyensec.com>
---
 drivers/infiniband/sw/rxe/rxe_odp.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c
index ab21b620e94c..c375d3efd999 100644
--- a/drivers/infiniband/sw/rxe/rxe_odp.c
+++ b/drivers/infiniband/sw/rxe/rxe_odp.c
@@ -469,7 +469,7 @@ static int rxe_ib_prefetch_sg_list(struct ib_pd *ibpd,
 		struct rxe_mr *mr;
 		struct ib_umem_odp *umem_odp;
 
-		mr = lookup_mr(pd, IB_ACCESS_LOCAL_WRITE,
+		mr = lookup_mr(pd, IB_ACCESS_LOCAL_WRITE | IB_ACCESS_ON_DEMAND,
 			       sg_list[i].lkey, RXE_LOOKUP_LOCAL);
 
 		if (!mr) {
@@ -535,7 +535,7 @@ static int rxe_ib_advise_mr_prefetch(struct ib_pd *ibpd,
 
 	for (i = 0; i < num_sge; ++i) {
 		/* Takes a reference, which will be released in the queued work */
-		mr = lookup_mr(pd, IB_ACCESS_LOCAL_WRITE,
+		mr = lookup_mr(pd, IB_ACCESS_LOCAL_WRITE | IB_ACCESS_ON_DEMAND,
 			       sg_list[i].lkey, RXE_LOOKUP_LOCAL);
 		if (!mr) {
 			mr = ERR_PTR(-EINVAL);

-- 
2.55.0

Re: [PATCH v2 2/2] RDMA/rxe: Reject prefetch of a non-ODP MR

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Prefetch incorrectly requires IB_ACCESS_LOCAL_WRITE unconditionally, breaking read-prefetch on read-only ODP MRs and making the writable check redundant.
--

commit a9cc6b0ff5adbd3d36fe2189b597deea1078f23c
Author: Norbert Szetei <norbert@doyensec.com>

RDMA/rxe: Reject prefetch of a non-ODP MR

This commit modifies the prefetch paths in rxe_odp.c to require
IB_ACCESS_ON_DEMAND when calling lookup_mr. This prevents out-of-bounds
reads that occur when non-ODP MRs are erroneously processed by ODP
pagefault handlers.

Cc: stable@vger.kernel.org

> diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c
> --- a/drivers/infiniband/sw/rxe/rxe_odp.c
> +++ b/drivers/infiniband/sw/rxe/rxe_odp.c
> @@ -469,7 +469,7 @@ static int rxe_ib_prefetch_sg_list(struct ib_pd *ibpd,
>  		struct rxe_mr *mr;
>  		struct ib_umem_odp *umem_odp;
>  
> -		mr = lookup_mr(pd, IB_ACCESS_LOCAL_WRITE,
> +		mr = lookup_mr(pd, IB_ACCESS_LOCAL_WRITE | IB_ACCESS_ON_DEMAND,
>  			       sg_list[i].lkey, RXE_LOOKUP_LOCAL);

[Severity: High]
This isn't a bug introduced by this patch, but does this require local write
access for all synchronous prefetches?

It appears this might improperly break read-prefetch for read-only ODP MRs.
Valid ibv_advise_mr calls for read-only ODP MRs requesting
IB_UVERBS_ADVISE_MR_ADVICE_PREFETCH could fail with -EINVAL.

>  
>  		if (!mr) {
>  			rxe_dbg_pd(pd, "mr with lkey %x not found\n",
>  				   sg_list[i].lkey);
>  			return -EINVAL;
>  		}
>  
>  		if (advice == IB_UVERBS_ADVISE_MR_ADVICE_PREFETCH_WRITE &&
>  		    !mr->umem->writable) {

[Severity: High]
This is also a pre-existing issue, but does lookup_mr already strictly enforce
IB_ACCESS_LOCAL_WRITE?

If so, does that render this missing write permission check completely dead code?

[ ... ]

> @@ -535,7 +535,7 @@ static int rxe_ib_advise_mr_prefetch(struct ib_pd *ibpd,
>  
>  	for (i = 0; i < num_sge; ++i) {
>  		/* Takes a reference, which will be released in the queued work */
> -		mr = lookup_mr(pd, IB_ACCESS_LOCAL_WRITE,
> +		mr = lookup_mr(pd, IB_ACCESS_LOCAL_WRITE | IB_ACCESS_ON_DEMAND,
>  			       sg_list[i].lkey, RXE_LOOKUP_LOCAL);

[Severity: High]
This is a pre-existing issue, but does the asynchronous path also suffer from
the same incorrect IB_ACCESS_LOCAL_WRITE enforcement?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260913-rxe-advise-mr-v2-v2-0-b806c789871c@doyensec.com?part=2

Re: [PATCH v2 1/2] RDMA/rxe: Reject IB_ACCESS_ON_DEMAND changes after MR creation

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [Critical] Arbitrary Kernel Pointer Dereference / Local Privilege Escalation via IB_WR_REG_MR WQE
--

commit fa8758be337c701491b43023c09167d84e64fdc0
Author: Norbert Szetei <norbert@doyensec.com>

RDMA/rxe: Reject IB_ACCESS_ON_DEMAND changes after MR creation

This commit prevents userspace from setting or clearing the
IB_ACCESS_ON_DEMAND flag on memory regions after they are created. This
ensures that the access flag correctly matches the underlying capabilities
of the memory region.

> diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c
> index 71d9ea4772890..615da4bb9a38c 100644
> --- a/drivers/infiniband/sw/rxe/rxe_mr.c
> +++ b/drivers/infiniband/sw/rxe/rxe_mr.c
> @@ -796,6 +796,12 @@ int rxe_reg_fast_mr(struct rxe_qp *qp, struct rxe_send_wqe *wqe)
>  		return -EINVAL;
>  	}
>  
> +	/* an MR with no umem is never an ODP MR */
> +	if (unlikely(access & IB_ACCESS_ON_DEMAND)) {
> +		rxe_dbg_mr(mr, "access = 0x%x requests ODP\n", access);
> +		return -EINVAL;
> +	}
> +

[Severity: Critical]
This is a pre-existing issue, but does writing to mr here allow an arbitrary
kernel memory overwrite?

If an unprivileged local user creates a Reliable Connection Queue Pair and
memory-maps its send queue ring buffer, they can write a crafted WQE with
IB_WR_REG_MR and set wqe->wr.wr.reg.mr to an arbitrary kernel pointer.

The kernel processes this WQE in rxe_requester(), which routes the WQE to
kernel-only local operations without verifying if the Queue Pair belongs to
a privileged context:

rxe_req.c:rxe_requester() {
    ...
    if (wqe->mask & WR_LOCAL_OP_MASK) {
        err = rxe_do_local_ops(qp, wqe);
    ...
}

Since rxe_wr_opcode_info allows WR_LOCAL_OP_MASK for IB_WR_REG_MR,
rxe_do_local_ops() dispatches it to rxe_reg_fast_mr().

In rxe_reg_fast_mr(), the arbitrary pointer from the user is directly cast:

rxe_mr.c:rxe_reg_fast_mr() {
    struct rxe_mr *mr = to_rmr(wqe->wr.wr.reg.mr);
    ...
}

Could this cause the subsequent assignments to mr->access, mr->lkey,
mr->rkey, and mr->state to overwrite arbitrary kernel memory?

>  	mr->access = access;
>  	mr->lkey = key;
>  	mr->rkey = key;

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260913-rxe-advise-mr-v2-v2-0-b806c789871c@doyensec.com?part=1