* [PATCH] RDMA/rxe: fix responder UAF on IB_QP_MAX_DEST_RD_ATOMIC modify_qp
@ 2026-07-08 22:45 Ibrahim Hashimov
2026-07-09 1:22 ` yanjun.zhu
2026-07-09 7:26 ` [PATCH v2] " Ibrahim Hashimov
0 siblings, 2 replies; 5+ messages in thread
From: Ibrahim Hashimov @ 2026-07-08 22:45 UTC (permalink / raw)
To: Zhu Yanjun, Jason Gunthorpe, Leon Romanovsky
Cc: linux-rdma, linux-kernel, stable
rxe_qp_from_attr()'s IB_QP_MAX_DEST_RD_ATOMIC branch frees and
reallocates qp->resp.resources[] (the rd_atomic resource array used by
the responder to track in-flight RDMA READ/ATOMIC/FLUSH requests)
completely outside of the IB_QP_STATE handling above it. Unlike every
other place that tears this array down -- rxe_qp_reset(), reached only
under IB_QP_STATE, always calls rxe_disable_task(&qp->recv_task) /
rxe_disable_task(&qp->send_task) to drain the responder and requester
tasks before touching per-QP state, then re-enables them -- this branch
runs with the responder task (rxe_receiver(), scheduled as recv_task on
the rxe_wq workqueue) fully live and unlocked. A userspace modify_qp()
that sets only IB_QP_MAX_DEST_RD_ATOMIC (no state change, so
__qp_chk_state()/ib_modify_qp_is_ok() never runs and qp->state_lock is
never taken here) can therefore race the responder in two ways:
1. free_rd_atomic_resources() calls kfree(qp->resp.resources) and
alloc_rd_atomic_resources() kzalloc_objs()'s a new array while
rxe_prepare_res()/find_resource() in rxe_resp.c are concurrently
walking &qp->resp.resources[i] with no lock held -- a straight
free-vs-read race on the array itself.
2. free_rd_atomic_resources() only NULLs qp->resp.resources; it never
clears qp->resp.res, the raw pointer *into* that array that
rxe_resp.c caches across a multi-packet RDMA READ/ATOMIC/FLUSH
reply (set at rxe_resp.c read/atomic/flush-reply sites, cleared
only on the normal completion paths). If a modify_qp() races a
resource still referenced by qp->resp.res, the array is freed out
from under the cached pointer and the next reply packet dereferences
it -- independent of the kfree/kzalloc_objs() window in (1).
Reproduced with KASAN: a single process driving one RC QP pair in rxe
loopback, one thread pumping large multi-packet IBV_WR_RDMA_READs
against qpB while a second thread hammers
ibv_modify_qp(qpB, IB_QP_MAX_DEST_RD_ATOMIC), reliably (~11s) produces
BUG: KASAN: slab-use-after-free in rxe_receiver+0x4f78/0x89e0 [rdma_rxe]
Workqueue: rxe_wq do_work [rdma_rxe]
with the freed kmalloc-1k object being the rd_atomic resource array
freed by the modify_qp() thread while the recv_task kworker reads it.
An identical run modifying only IB_QP_MIN_RNR_TIMER (no resource free)
is clean.
Fix both races the same way rxe_qp_reset() already handles tearing down
this exact array: quiesce the responder task around the free/realloc by
calling rxe_disable_task(&qp->recv_task) before free_rd_atomic_resources()
and rxe_enable_task(&qp->recv_task) after alloc_rd_atomic_resources(),
so rxe_receiver() cannot observe the array mid-free/mid-realloc. And
close the still-open window for (2) at the source: have
free_rd_atomic_resources() clear qp->resp.res along with
qp->resp.resources, exactly like the existing completion paths in
rxe_resp.c (check_rkey()/duplicate_request()/RESPST_CLEANUP) already do
when a resource's lifetime ends, so a drained-and-resumed responder
restarts at RESPST_CHK_PSN against the fresh array instead of replaying
a stale reference into the old one.
Only qp->recv_task is drained: qp->resp.resources / qp->resp.res are
touched exclusively by the responder (rxe_resp.c); the requester
(send_task / rxe_sender()) never reads them, so there is no need to
widen this beyond what rxe_qp_reset() would drain for the equivalent
state.
Verified on the same v6.19 KASAN stand: with this fix applied, the
identical differential reproducer drives sustained MAX_DEST_RD_ATOMIC
storms against qpB well past the ~11s pre-fix time-to-first-splat with
zero KASAN reports, versus reliably tripping the slab-use-after-free in
rxe_receiver() described above before the fix.
Fixes: 8700e3e7c485 ("Soft RoCE driver")
Cc: stable@vger.kernel.org
Signed-off-by: Ibrahim Hashimov <security@auditcode.ai>
Assisted-by: AuditCode-AI:2026.07
---
drivers/infiniband/sw/rxe/rxe_qp.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c
index f3dff1aea96a..646957707765 100644
--- a/drivers/infiniband/sw/rxe/rxe_qp.c
+++ b/drivers/infiniband/sw/rxe/rxe_qp.c
@@ -172,6 +172,7 @@ static void free_rd_atomic_resources(struct rxe_qp *qp)
}
kfree(qp->resp.resources);
qp->resp.resources = NULL;
+ qp->resp.res = NULL;
}
}
@@ -709,9 +710,15 @@ int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask,
qp->attr.max_dest_rd_atomic = max_dest_rd_atomic;
+ /*
+ * Not gated by IB_QP_STATE above: quiesce the responder task
+ * the same way rxe_qp_reset() does before touching this
+ * array, so rxe_receiver() can't race the free/realloc.
+ */
+ rxe_disable_task(&qp->recv_task);
free_rd_atomic_resources(qp);
-
err = alloc_rd_atomic_resources(qp, max_dest_rd_atomic);
+ rxe_enable_task(&qp->recv_task);
if (err)
return err;
}
--
2.50.1 (Apple Git-155)
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH] RDMA/rxe: fix responder UAF on IB_QP_MAX_DEST_RD_ATOMIC modify_qp
2026-07-08 22:45 [PATCH] RDMA/rxe: fix responder UAF on IB_QP_MAX_DEST_RD_ATOMIC modify_qp Ibrahim Hashimov
@ 2026-07-09 1:22 ` yanjun.zhu
2026-07-09 7:22 ` Ibrahim Hashimov
2026-07-09 7:26 ` [PATCH v2] " Ibrahim Hashimov
1 sibling, 1 reply; 5+ messages in thread
From: yanjun.zhu @ 2026-07-09 1:22 UTC (permalink / raw)
To: Ibrahim Hashimov, Zhu Yanjun, Jason Gunthorpe, Leon Romanovsky,
Zhu Yanjun
Cc: linux-rdma, linux-kernel, stable
On 7/8/26 3:45 PM, Ibrahim Hashimov wrote:
> rxe_qp_from_attr()'s IB_QP_MAX_DEST_RD_ATOMIC branch frees and
> reallocates qp->resp.resources[] (the rd_atomic resource array used by
> the responder to track in-flight RDMA READ/ATOMIC/FLUSH requests)
> completely outside of the IB_QP_STATE handling above it. Unlike every
> other place that tears this array down -- rxe_qp_reset(), reached only
> under IB_QP_STATE, always calls rxe_disable_task(&qp->recv_task) /
> rxe_disable_task(&qp->send_task) to drain the responder and requester
> tasks before touching per-QP state, then re-enables them -- this branch
> runs with the responder task (rxe_receiver(), scheduled as recv_task on
> the rxe_wq workqueue) fully live and unlocked. A userspace modify_qp()
> that sets only IB_QP_MAX_DEST_RD_ATOMIC (no state change, so
> __qp_chk_state()/ib_modify_qp_is_ok() never runs and qp->state_lock is
> never taken here) can therefore race the responder in two ways:
>
> 1. free_rd_atomic_resources() calls kfree(qp->resp.resources) and
> alloc_rd_atomic_resources() kzalloc_objs()'s a new array while
> rxe_prepare_res()/find_resource() in rxe_resp.c are concurrently
> walking &qp->resp.resources[i] with no lock held -- a straight
> free-vs-read race on the array itself.
>
> 2. free_rd_atomic_resources() only NULLs qp->resp.resources; it never
> clears qp->resp.res, the raw pointer *into* that array that
> rxe_resp.c caches across a multi-packet RDMA READ/ATOMIC/FLUSH
> reply (set at rxe_resp.c read/atomic/flush-reply sites, cleared
> only on the normal completion paths). If a modify_qp() races a
> resource still referenced by qp->resp.res, the array is freed out
> from under the cached pointer and the next reply packet dereferences
> it -- independent of the kfree/kzalloc_objs() window in (1).
>
> Reproduced with KASAN: a single process driving one RC QP pair in rxe
> loopback, one thread pumping large multi-packet IBV_WR_RDMA_READs
> against qpB while a second thread hammers
> ibv_modify_qp(qpB, IB_QP_MAX_DEST_RD_ATOMIC), reliably (~11s) produces
>
> BUG: KASAN: slab-use-after-free in rxe_receiver+0x4f78/0x89e0 [rdma_rxe]
> Workqueue: rxe_wq do_work [rdma_rxe]
Can you share all the stack trace with us? Thanks a lot.
>
> with the freed kmalloc-1k object being the rd_atomic resource array
> freed by the modify_qp() thread while the recv_task kworker reads it.
> An identical run modifying only IB_QP_MIN_RNR_TIMER (no resource free)
> is clean.
>
> Fix both races the same way rxe_qp_reset() already handles tearing down
> this exact array: quiesce the responder task around the free/realloc by
> calling rxe_disable_task(&qp->recv_task) before free_rd_atomic_resources()
> and rxe_enable_task(&qp->recv_task) after alloc_rd_atomic_resources(),
> so rxe_receiver() cannot observe the array mid-free/mid-realloc. And
> close the still-open window for (2) at the source: have
> free_rd_atomic_resources() clear qp->resp.res along with
> qp->resp.resources, exactly like the existing completion paths in
> rxe_resp.c (check_rkey()/duplicate_request()/RESPST_CLEANUP) already do
> when a resource's lifetime ends, so a drained-and-resumed responder
> restarts at RESPST_CHK_PSN against the fresh array instead of replaying
> a stale reference into the old one.
>
> Only qp->recv_task is drained: qp->resp.resources / qp->resp.res are
> touched exclusively by the responder (rxe_resp.c); the requester
> (send_task / rxe_sender()) never reads them, so there is no need to
> widen this beyond what rxe_qp_reset() would drain for the equivalent
> state.
>
> Verified on the same v6.19 KASAN stand: with this fix applied, the
> identical differential reproducer drives sustained MAX_DEST_RD_ATOMIC
> storms against qpB well past the ~11s pre-fix time-to-first-splat with
> zero KASAN reports, versus reliably tripping the slab-use-after-free in
> rxe_receiver() described above before the fix.
>
> Fixes: 8700e3e7c485 ("Soft RoCE driver")
> Cc: stable@vger.kernel.org
> Signed-off-by: Ibrahim Hashimov <security@auditcode.ai>
> Assisted-by: AuditCode-AI:2026.07
> ---
> drivers/infiniband/sw/rxe/rxe_qp.c | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c
> index f3dff1aea96a..646957707765 100644
> --- a/drivers/infiniband/sw/rxe/rxe_qp.c
> +++ b/drivers/infiniband/sw/rxe/rxe_qp.c
> @@ -172,6 +172,7 @@ static void free_rd_atomic_resources(struct rxe_qp *qp)
> }
> kfree(qp->resp.resources);
> qp->resp.resources = NULL;
> + qp->resp.res = NULL;
> }
> }
>
> @@ -709,9 +710,15 @@ int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask,
>
> qp->attr.max_dest_rd_atomic = max_dest_rd_atomic;
>
> + /*
> + * Not gated by IB_QP_STATE above: quiesce the responder task
> + * the same way rxe_qp_reset() does before touching this
> + * array, so rxe_receiver() can't race the free/realloc.
> + */
> + rxe_disable_task(&qp->recv_task);
> free_rd_atomic_resources(qp);
> -
> err = alloc_rd_atomic_resources(qp, max_dest_rd_atomic);
> + rxe_enable_task(&qp->recv_task);
If alloc_rd_atomic_resources fails, that is, qp->resp.resources is NULL.
After rxe_enable_task(&qp->recv_task); qp->resp.resources(NULL) will be
used in resp. This will cause problems.
drivers/infiniband/sw/rxe/rxe_resp.c:656: res =
&qp->resp.resources[qp->resp.res_head];
drivers/infiniband/sw/rxe/rxe_resp.c:1325: struct resp_res *res =
&qp->resp.resources[i];
Zhu Yanjun
> if (err)
> return err;
> }
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH] RDMA/rxe: fix responder UAF on IB_QP_MAX_DEST_RD_ATOMIC modify_qp
2026-07-09 1:22 ` yanjun.zhu
@ 2026-07-09 7:22 ` Ibrahim Hashimov
0 siblings, 0 replies; 5+ messages in thread
From: Ibrahim Hashimov @ 2026-07-09 7:22 UTC (permalink / raw)
To: yanjun.zhu, zyjzyj2000, jgg, leon; +Cc: linux-rdma, linux-kernel
Hi Zhu Yanjun, thanks for the review.
> > BUG: KASAN: slab-use-after-free in rxe_receiver+0x4f78/0x89e0 [rdma_rxe]
> > Workqueue: rxe_wq do_work [rdma_rxe]
> Can you share all the stack trace with us? Thanks a lot.
Sure. The read side -- the recv_task kworker running rxe_receiver() on
the freed rd_atomic resource array -- from the pre-fix v6.19 kernel
(CONFIG_KASAN generic, rxe loopback):
BUG: KASAN: slab-use-after-free in rxe_receiver+0x4f78/0x89e0 [rdma_rxe]
Read of size 4 at addr ffff88800b79f84c by task kworker/u8:2/51
CPU: 0 UID: 0 PID: 51 Comm: kworker/u8:2 Not tainted 6.19.0 #1
Workqueue: rxe_wq do_work [rdma_rxe]
Call Trace:
<TASK>
dump_stack_lvl+0x4d/0x70
print_report+0x170/0x4f3
kasan_report+0xda/0x110
rxe_receiver+0x4f78/0x89e0 [rdma_rxe]
do_work+0x144/0x470 [rdma_rxe]
process_one_work+0x611/0xe80
worker_thread+0x52e/0xdc0
kthread+0x30c/0x630
ret_from_fork+0x2fd/0x3e0
ret_from_fork_asm+0x1a/0x30
</TASK>
The buggy address belongs to the object at ffff88800b79f800
which belongs to the cache kmalloc-1k of size 1024
The buggy address is located 76 bytes inside of
freed 1024-byte region [ffff88800b79f800, ffff88800b79fc00)
The freed kmalloc-1k object is qp->resp.resources[] (the rd_atomic
array); it is freed by the concurrent
ib_modify_qp(IB_QP_MAX_DEST_RD_ATOMIC) thread in
free_rd_atomic_resources() <- rxe_qp_from_attr(), while this recv_task
kworker reads it. This was a kasan_multi_shot run and the per-object
Allocated-by/Freed-by backtraces were lost in the splat storm -- I can
send a single-shot capture that includes those two stacks if it would
help.
> If alloc_rd_atomic_resources fails, that is, qp->resp.resources is NULL.
> After rxe_enable_task(&qp->recv_task); qp->resp.resources(NULL) will be
> used in resp. This will cause problems.
> drivers/infiniband/sw/rxe/rxe_resp.c:656: res = &qp->resp.resources[qp->resp.res_head];
> drivers/infiniband/sw/rxe/rxe_resp.c:1325: struct resp_res *res = &qp->resp.resources[i];
Good catch -- you're right. Re-enabling recv_task before the error
check resumes the responder against a NULL qp->resp.resources on the
ENOMEM path. v2 moves rxe_enable_task() below the error check, so the
responder is re-enabled only once a fresh array has been installed and
stays quiesced on failure:
rxe_disable_task(&qp->recv_task);
free_rd_atomic_resources(qp);
err = alloc_rd_atomic_resources(qp, max_dest_rd_atomic);
if (err)
return err;
rxe_enable_task(&qp->recv_task);
rxe_disable_task()/rxe_enable_task() are state-based (TASK_STATE_DRAINED
/IDLE), not refcounted, so leaving recv_task drained on the failed
modify is safe and recoverable: a subsequent successful modify_qp or a
QP destroy both handle the DRAINED state.
I'll send this as [PATCH v2].
Thanks,
Ibrahim
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH v2] RDMA/rxe: fix responder UAF on IB_QP_MAX_DEST_RD_ATOMIC modify_qp
2026-07-08 22:45 [PATCH] RDMA/rxe: fix responder UAF on IB_QP_MAX_DEST_RD_ATOMIC modify_qp Ibrahim Hashimov
2026-07-09 1:22 ` yanjun.zhu
@ 2026-07-09 7:26 ` Ibrahim Hashimov
2026-07-09 18:24 ` yanjun.zhu
1 sibling, 1 reply; 5+ messages in thread
From: Ibrahim Hashimov @ 2026-07-09 7:26 UTC (permalink / raw)
To: yanjun.zhu, zyjzyj2000, jgg, leon; +Cc: linux-rdma, linux-kernel, stable
rxe_qp_from_attr()'s IB_QP_MAX_DEST_RD_ATOMIC branch frees and
reallocates qp->resp.resources[] (the rd_atomic resource array used by
the responder to track in-flight RDMA READ/ATOMIC/FLUSH requests)
completely outside of the IB_QP_STATE handling above it. Unlike every
other place that tears this array down -- rxe_qp_reset(), reached only
under IB_QP_STATE, always calls rxe_disable_task(&qp->recv_task) /
rxe_disable_task(&qp->send_task) to drain the responder and requester
tasks before touching per-QP state, then re-enables them -- this branch
runs with the responder task (rxe_receiver(), scheduled as recv_task on
the rxe_wq workqueue) fully live and unlocked. A userspace modify_qp()
that sets only IB_QP_MAX_DEST_RD_ATOMIC (no state change, so
__qp_chk_state()/ib_modify_qp_is_ok() never runs and qp->state_lock is
never taken here) can therefore race the responder in two ways:
1. free_rd_atomic_resources() calls kfree(qp->resp.resources) and
alloc_rd_atomic_resources() kzalloc_objs()'s a new array while
rxe_prepare_res()/find_resource() in rxe_resp.c are concurrently
walking &qp->resp.resources[i] with no lock held -- a straight
free-vs-read race on the array itself.
2. free_rd_atomic_resources() only NULLs qp->resp.resources; it never
clears qp->resp.res, the raw pointer *into* that array that
rxe_resp.c caches across a multi-packet RDMA READ/ATOMIC/FLUSH
reply (set at rxe_resp.c read/atomic/flush-reply sites, cleared
only on the normal completion paths). If a modify_qp() races a
resource still referenced by qp->resp.res, the array is freed out
from under the cached pointer and the next reply packet dereferences
it -- independent of the kfree/kzalloc_objs() window in (1).
Reproduced with KASAN: a single process driving one RC QP pair in rxe
loopback, one thread pumping large multi-packet IBV_WR_RDMA_READs
against qpB while a second thread hammers
ibv_modify_qp(qpB, IB_QP_MAX_DEST_RD_ATOMIC), reliably (~11s) produces
BUG: KASAN: slab-use-after-free in rxe_receiver+0x4f78/0x89e0 [rdma_rxe]
Workqueue: rxe_wq do_work [rdma_rxe]
with the freed kmalloc-1k object being the rd_atomic resource array
freed by the modify_qp() thread while the recv_task kworker reads it.
An identical run modifying only IB_QP_MIN_RNR_TIMER (no resource free)
is clean.
Fix both races the same way rxe_qp_reset() already handles tearing down
this exact array: quiesce the responder task around the free/realloc by
calling rxe_disable_task(&qp->recv_task) before free_rd_atomic_resources()
and rxe_enable_task(&qp->recv_task) only after alloc_rd_atomic_resources()
has succeeded, so rxe_receiver() cannot observe the array mid-free/
mid-realloc. On the alloc-failure path the responder is deliberately
left quiesced: qp->resp.resources is NULL at that point and
rxe_prepare_res()/find_resource() would dereference it, so recv_task
must not be re-enabled until a fresh array has been installed. And
close the still-open window for (2) at the source: have
free_rd_atomic_resources() clear qp->resp.res along with
qp->resp.resources, exactly like the existing completion paths in
rxe_resp.c (check_rkey()/duplicate_request()/RESPST_CLEANUP) already do
when a resource's lifetime ends, so a drained-and-resumed responder
restarts at RESPST_CHK_PSN against the fresh array instead of replaying
a stale reference into the old one.
Only qp->recv_task is drained: qp->resp.resources / qp->resp.res are
touched exclusively by the responder (rxe_resp.c); the requester
(send_task / rxe_sender()) never reads them, so there is no need to
widen this beyond what rxe_qp_reset() would drain for the equivalent
state.
Verified on the same v6.19 KASAN stand: with this fix applied, the
identical differential reproducer drives sustained MAX_DEST_RD_ATOMIC
storms against qpB well past the ~11s pre-fix time-to-first-splat with
zero KASAN reports, versus reliably tripping the slab-use-after-free in
rxe_receiver() described above before the fix.
Fixes: 8700e3e7c485 ("Soft RoCE driver")
Cc: stable@vger.kernel.org
Signed-off-by: Ibrahim Hashimov <security@auditcode.ai>
Assisted-by: AuditCode-AI:2026.07
---
v2: address Zhu Yanjun's review of v1
(https://lore.kernel.org/linux-rdma/20260708224550.1281-1-security@auditcode.ai/):
only re-enable recv_task after alloc_rd_atomic_resources() succeeds, so
the responder is not resumed against a NULL qp->resp.resources on the
ENOMEM path (rxe_prepare_res()/find_resource() would dereference it).
No change to the successful path; fix description updated accordingly.
drivers/infiniband/sw/rxe/rxe_qp.c | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c
index f3dff1aea96a..e39fb144cbbb 100644
--- a/drivers/infiniband/sw/rxe/rxe_qp.c
+++ b/drivers/infiniband/sw/rxe/rxe_qp.c
@@ -172,6 +172,7 @@ static void free_rd_atomic_resources(struct rxe_qp *qp)
}
kfree(qp->resp.resources);
qp->resp.resources = NULL;
+ qp->resp.res = NULL;
}
}
@@ -709,11 +710,24 @@ int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask,
qp->attr.max_dest_rd_atomic = max_dest_rd_atomic;
+ /*
+ * This branch is not gated by IB_QP_STATE, so the responder
+ * task is live here. Quiesce it the way rxe_qp_reset() does
+ * before swapping the rd_atomic resource array, so
+ * rxe_receiver() cannot race the free/realloc.
+ */
+ rxe_disable_task(&qp->recv_task);
free_rd_atomic_resources(qp);
-
err = alloc_rd_atomic_resources(qp, max_dest_rd_atomic);
+ /*
+ * On failure the responder stays quiesced: qp->resp.resources
+ * is NULL now, and rxe_prepare_res()/find_resource() would
+ * dereference it, so do not re-enable recv_task until a fresh
+ * array has been installed.
+ */
if (err)
return err;
+ rxe_enable_task(&qp->recv_task);
}
if (mask & IB_QP_EN_SQD_ASYNC_NOTIFY)
--
2.50.1 (Apple Git-155)
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH v2] RDMA/rxe: fix responder UAF on IB_QP_MAX_DEST_RD_ATOMIC modify_qp
2026-07-09 7:26 ` [PATCH v2] " Ibrahim Hashimov
@ 2026-07-09 18:24 ` yanjun.zhu
0 siblings, 0 replies; 5+ messages in thread
From: yanjun.zhu @ 2026-07-09 18:24 UTC (permalink / raw)
To: Ibrahim Hashimov, zyjzyj2000, jgg, leon, Zhu Yanjun
Cc: linux-rdma, linux-kernel, stable
On 7/9/26 12:26 AM, Ibrahim Hashimov wrote:
> rxe_qp_from_attr()'s IB_QP_MAX_DEST_RD_ATOMIC branch frees and
> reallocates qp->resp.resources[] (the rd_atomic resource array used by
> the responder to track in-flight RDMA READ/ATOMIC/FLUSH requests)
> completely outside of the IB_QP_STATE handling above it. Unlike every
> other place that tears this array down -- rxe_qp_reset(), reached only
> under IB_QP_STATE, always calls rxe_disable_task(&qp->recv_task) /
> rxe_disable_task(&qp->send_task) to drain the responder and requester
> tasks before touching per-QP state, then re-enables them -- this branch
> runs with the responder task (rxe_receiver(), scheduled as recv_task on
> the rxe_wq workqueue) fully live and unlocked. A userspace modify_qp()
> that sets only IB_QP_MAX_DEST_RD_ATOMIC (no state change, so
> __qp_chk_state()/ib_modify_qp_is_ok() never runs and qp->state_lock is
> never taken here) can therefore race the responder in two ways:
>
> 1. free_rd_atomic_resources() calls kfree(qp->resp.resources) and
> alloc_rd_atomic_resources() kzalloc_objs()'s a new array while
> rxe_prepare_res()/find_resource() in rxe_resp.c are concurrently
> walking &qp->resp.resources[i] with no lock held -- a straight
> free-vs-read race on the array itself.
>
> 2. free_rd_atomic_resources() only NULLs qp->resp.resources; it never
> clears qp->resp.res, the raw pointer *into* that array that
> rxe_resp.c caches across a multi-packet RDMA READ/ATOMIC/FLUSH
> reply (set at rxe_resp.c read/atomic/flush-reply sites, cleared
> only on the normal completion paths). If a modify_qp() races a
> resource still referenced by qp->resp.res, the array is freed out
> from under the cached pointer and the next reply packet dereferences
> it -- independent of the kfree/kzalloc_objs() window in (1).
>
> Reproduced with KASAN: a single process driving one RC QP pair in rxe
> loopback, one thread pumping large multi-packet IBV_WR_RDMA_READs
> against qpB while a second thread hammers
> ibv_modify_qp(qpB, IB_QP_MAX_DEST_RD_ATOMIC), reliably (~11s) produces
>
> BUG: KASAN: slab-use-after-free in rxe_receiver+0x4f78/0x89e0 [rdma_rxe]
> Workqueue: rxe_wq do_work [rdma_rxe]
>
> with the freed kmalloc-1k object being the rd_atomic resource array
> freed by the modify_qp() thread while the recv_task kworker reads it.
> An identical run modifying only IB_QP_MIN_RNR_TIMER (no resource free)
> is clean.
>
> Fix both races the same way rxe_qp_reset() already handles tearing down
> this exact array: quiesce the responder task around the free/realloc by
> calling rxe_disable_task(&qp->recv_task) before free_rd_atomic_resources()
> and rxe_enable_task(&qp->recv_task) only after alloc_rd_atomic_resources()
> has succeeded, so rxe_receiver() cannot observe the array mid-free/
> mid-realloc. On the alloc-failure path the responder is deliberately
> left quiesced: qp->resp.resources is NULL at that point and
> rxe_prepare_res()/find_resource() would dereference it, so recv_task
> must not be re-enabled until a fresh array has been installed. And
> close the still-open window for (2) at the source: have
> free_rd_atomic_resources() clear qp->resp.res along with
> qp->resp.resources, exactly like the existing completion paths in
> rxe_resp.c (check_rkey()/duplicate_request()/RESPST_CLEANUP) already do
> when a resource's lifetime ends, so a drained-and-resumed responder
> restarts at RESPST_CHK_PSN against the fresh array instead of replaying
> a stale reference into the old one.
>
> Only qp->recv_task is drained: qp->resp.resources / qp->resp.res are
> touched exclusively by the responder (rxe_resp.c); the requester
> (send_task / rxe_sender()) never reads them, so there is no need to
> widen this beyond what rxe_qp_reset() would drain for the equivalent
> state.
>
> Verified on the same v6.19 KASAN stand: with this fix applied, the
> identical differential reproducer drives sustained MAX_DEST_RD_ATOMIC
> storms against qpB well past the ~11s pre-fix time-to-first-splat with
> zero KASAN reports, versus reliably tripping the slab-use-after-free in
> rxe_receiver() described above before the fix.
>
> Fixes: 8700e3e7c485 ("Soft RoCE driver")
> Cc: stable@vger.kernel.org
> Signed-off-by: Ibrahim Hashimov <security@auditcode.ai>
> Assisted-by: AuditCode-AI:2026.07
> ---
> v2: address Zhu Yanjun's review of v1
> (https://lore.kernel.org/linux-rdma/20260708224550.1281-1-security@auditcode.ai/):
> only re-enable recv_task after alloc_rd_atomic_resources() succeeds, so
> the responder is not resumed against a NULL qp->resp.resources on the
> ENOMEM path (rxe_prepare_res()/find_resource() would dereference it).
> No change to the successful path; fix description updated accordingly.
>
> drivers/infiniband/sw/rxe/rxe_qp.c | 16 +++++++++++++++-
> 1 file changed, 15 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c
> index f3dff1aea96a..e39fb144cbbb 100644
> --- a/drivers/infiniband/sw/rxe/rxe_qp.c
> +++ b/drivers/infiniband/sw/rxe/rxe_qp.c
> @@ -172,6 +172,7 @@ static void free_rd_atomic_resources(struct rxe_qp *qp)
> }
> kfree(qp->resp.resources);
> qp->resp.resources = NULL;
> + qp->resp.res = NULL;
> }
> }
>
> @@ -709,11 +710,24 @@ int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask,
>
> qp->attr.max_dest_rd_atomic = max_dest_rd_atomic;
>
> + /*
> + * This branch is not gated by IB_QP_STATE, so the responder
> + * task is live here. Quiesce it the way rxe_qp_reset() does
> + * before swapping the rd_atomic resource array, so
> + * rxe_receiver() cannot race the free/realloc.
> + */
> + rxe_disable_task(&qp->recv_task);
> free_rd_atomic_resources(qp);
> -
> err = alloc_rd_atomic_resources(qp, max_dest_rd_atomic);
> + /*
> + * On failure the responder stays quiesced: qp->resp.resources
> + * is NULL now, and rxe_prepare_res()/find_resource() would
> + * dereference it, so do not re-enable recv_task until a fresh
> + * array has been installed.
> + */
> if (err)
> return err;
> + rxe_enable_task(&qp->recv_task);
Thanks a lot. I am fine with this commit. Please Leon and Jason comment
on this commit.
Reviewed-by: Zhu Yanjun <yanjun.zhu@linux.dev>
Zhu Yanjun
> }
>
> if (mask & IB_QP_EN_SQD_ASYNC_NOTIFY)
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-07-09 18:24 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-08 22:45 [PATCH] RDMA/rxe: fix responder UAF on IB_QP_MAX_DEST_RD_ATOMIC modify_qp Ibrahim Hashimov
2026-07-09 1:22 ` yanjun.zhu
2026-07-09 7:22 ` Ibrahim Hashimov
2026-07-09 7:26 ` [PATCH v2] " Ibrahim Hashimov
2026-07-09 18:24 ` yanjun.zhu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox