* [PATCH] RDMA/rxe: Fix race condition in QP timer handlers
@ 2026-01-20 7:44 Li Zhijian
2026-01-21 5:04 ` Zhu Yanjun
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Li Zhijian @ 2026-01-20 7:44 UTC (permalink / raw)
To: linux-rdma; +Cc: linux-kernel, zyjzyj2000, jgg, leon, Li Zhijian
I encontered the following warning:
WARNING: drivers/infiniband/sw/rxe/rxe_task.c:249 at rxe_sched_task+0x1c8/0x238 [rdma_rxe], CPU#0: swapper/0/0
...
libsha1 [last unloaded: ip6_udp_tunnel]
CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G C 6.19.0-rc5-64k-v8+ #37 PREEMPT
Tainted: [C]=CRAP
Hardware name: Raspberry Pi 4 Model B Rev 1.2
Call trace:
rxe_sched_task+0x1c8/0x238 [rdma_rxe] (P)
retransmit_timer+0x130/0x188 [rdma_rxe]
call_timer_fn+0x68/0x4d0
__run_timers+0x630/0x888
...
WARNING: drivers/infiniband/sw/rxe/rxe_task.c:38 at rxe_sched_task+0x1c0/0x238 [rdma_rxe], CPU#0: swapper/0/0
...
WARNING: drivers/infiniband/sw/rxe/rxe_task.c:111 at do_work+0x488/0x5c8 [rdma_rxe], CPU#3: kworker/u17:4/93400
...
refcount_t: underflow; use-after-free.
WARNING: lib/refcount.c:28 at refcount_warn_saturate+0x138/0x1a0, CPU#3: kworker/u17:4/93400
The issue is caused by a race condition between retransmit_timer() and
rxe_destroy_qp, leading to the Queue Pair's (QP) reference count dropping
to zero during timer handler execution.
It seems this warning is harmless because rxe_qp_do_cleanup() will flush
all pending timers and requests.
Example of flow causing the issue:
CPU0 CPU1
retransmit_timer() {
spin_lock_irqsave
rxe_destroy_qp()
__rxe_cleanup()
__rxe_put() // qp->ref_count decrease to 0
rxe_qp_do_cleanup() {
if (qp->valid) {
rxe_sched_task() {
WARN_ON(rxe_read(task->qp) <= 0);
}
}
spin_unlock_irqrestore
}
spin_lock_irqsave
qp->valid = 0
spin_unlock_irqrestore
}
Ensure the QP's reference count is maintained and its validity is checked
within the timer callbacks by adding calls to rxe_get(qp) and corresponding
rxe_put(qp) after use.
Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
---
drivers/infiniband/sw/rxe/rxe_comp.c | 3 +++
drivers/infiniband/sw/rxe/rxe_req.c | 3 +++
2 files changed, 6 insertions(+)
diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c
index a5b2b62f596b..1390e861bd1d 100644
--- a/drivers/infiniband/sw/rxe/rxe_comp.c
+++ b/drivers/infiniband/sw/rxe/rxe_comp.c
@@ -119,12 +119,15 @@ void retransmit_timer(struct timer_list *t)
rxe_dbg_qp(qp, "retransmit timer fired\n");
+ if (!rxe_get(qp))
+ return;
spin_lock_irqsave(&qp->state_lock, flags);
if (qp->valid) {
qp->comp.timeout = 1;
rxe_sched_task(&qp->send_task);
}
spin_unlock_irqrestore(&qp->state_lock, flags);
+ rxe_put(qp);
}
void rxe_comp_queue_pkt(struct rxe_qp *qp, struct sk_buff *skb)
diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c
index 373b03f223be..12d03f390b09 100644
--- a/drivers/infiniband/sw/rxe/rxe_req.c
+++ b/drivers/infiniband/sw/rxe/rxe_req.c
@@ -102,6 +102,8 @@ void rnr_nak_timer(struct timer_list *t)
rxe_dbg_qp(qp, "nak timer fired\n");
+ if (!rxe_get(qp))
+ return;
spin_lock_irqsave(&qp->state_lock, flags);
if (qp->valid) {
/* request a send queue retry */
@@ -110,6 +112,7 @@ void rnr_nak_timer(struct timer_list *t)
rxe_sched_task(&qp->send_task);
}
spin_unlock_irqrestore(&qp->state_lock, flags);
+ rxe_put(qp);
}
static void req_check_sq_drain_done(struct rxe_qp *qp)
--
2.41.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH] RDMA/rxe: Fix race condition in QP timer handlers
2026-01-20 7:44 [PATCH] RDMA/rxe: Fix race condition in QP timer handlers Li Zhijian
@ 2026-01-21 5:04 ` Zhu Yanjun
2026-01-25 14:08 ` Leon Romanovsky
2026-01-28 10:25 ` Leon Romanovsky
2 siblings, 0 replies; 7+ messages in thread
From: Zhu Yanjun @ 2026-01-21 5:04 UTC (permalink / raw)
To: Li Zhijian, linux-rdma; +Cc: linux-kernel, zyjzyj2000, jgg, leon
在 2026/1/19 23:44, Li Zhijian 写道:
> I encontered the following warning:
> WARNING: drivers/infiniband/sw/rxe/rxe_task.c:249 at rxe_sched_task+0x1c8/0x238 [rdma_rxe], CPU#0: swapper/0/0
> ...
> libsha1 [last unloaded: ip6_udp_tunnel]
> CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G C 6.19.0-rc5-64k-v8+ #37 PREEMPT
> Tainted: [C]=CRAP
> Hardware name: Raspberry Pi 4 Model B Rev 1.2
> Call trace:
> rxe_sched_task+0x1c8/0x238 [rdma_rxe] (P)
> retransmit_timer+0x130/0x188 [rdma_rxe]
> call_timer_fn+0x68/0x4d0
> __run_timers+0x630/0x888
> ...
> WARNING: drivers/infiniband/sw/rxe/rxe_task.c:38 at rxe_sched_task+0x1c0/0x238 [rdma_rxe], CPU#0: swapper/0/0
> ...
> WARNING: drivers/infiniband/sw/rxe/rxe_task.c:111 at do_work+0x488/0x5c8 [rdma_rxe], CPU#3: kworker/u17:4/93400
> ...
> refcount_t: underflow; use-after-free.
> WARNING: lib/refcount.c:28 at refcount_warn_saturate+0x138/0x1a0, CPU#3: kworker/u17:4/93400
>
> The issue is caused by a race condition between retransmit_timer() and
> rxe_destroy_qp, leading to the Queue Pair's (QP) reference count dropping
> to zero during timer handler execution.
>
> It seems this warning is harmless because rxe_qp_do_cleanup() will flush
> all pending timers and requests.
>
> Example of flow causing the issue:
>
> CPU0 CPU1
> retransmit_timer() {
> spin_lock_irqsave
> rxe_destroy_qp()
> __rxe_cleanup()
> __rxe_put() // qp->ref_count decrease to 0
In __rxe_cleanup, __rxe_put decrease qp->ref_count to 0.
Then in the timer functions retransmit_timer and rnr_nak_timer will
check qp and resend the packets. IMO, it may be a solution to use the
function rxe_get to check if ref_count is 0 or not.
I am fine with it.
Reviewed-by: Zhu Yanjun <yanjun.zhu@linux.dev>
Thanks,
Zhu Yanjun
> rxe_qp_do_cleanup() {
> if (qp->valid) {
> rxe_sched_task() {
> WARN_ON(rxe_read(task->qp) <= 0);
> }
> }
> spin_unlock_irqrestore
> }
> spin_lock_irqsave
> qp->valid = 0
> spin_unlock_irqrestore
> }
>
> Ensure the QP's reference count is maintained and its validity is checked
> within the timer callbacks by adding calls to rxe_get(qp) and corresponding
> rxe_put(qp) after use.
>
> Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
> ---
> drivers/infiniband/sw/rxe/rxe_comp.c | 3 +++
> drivers/infiniband/sw/rxe/rxe_req.c | 3 +++
> 2 files changed, 6 insertions(+)
>
> diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c
> index a5b2b62f596b..1390e861bd1d 100644
> --- a/drivers/infiniband/sw/rxe/rxe_comp.c
> +++ b/drivers/infiniband/sw/rxe/rxe_comp.c
> @@ -119,12 +119,15 @@ void retransmit_timer(struct timer_list *t)
>
> rxe_dbg_qp(qp, "retransmit timer fired\n");
>
> + if (!rxe_get(qp))
> + return;
> spin_lock_irqsave(&qp->state_lock, flags);
> if (qp->valid) {
> qp->comp.timeout = 1;
> rxe_sched_task(&qp->send_task);
> }
> spin_unlock_irqrestore(&qp->state_lock, flags);
> + rxe_put(qp);
> }
>
> void rxe_comp_queue_pkt(struct rxe_qp *qp, struct sk_buff *skb)
> diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c
> index 373b03f223be..12d03f390b09 100644
> --- a/drivers/infiniband/sw/rxe/rxe_req.c
> +++ b/drivers/infiniband/sw/rxe/rxe_req.c
> @@ -102,6 +102,8 @@ void rnr_nak_timer(struct timer_list *t)
>
> rxe_dbg_qp(qp, "nak timer fired\n");
>
> + if (!rxe_get(qp))
> + return;
> spin_lock_irqsave(&qp->state_lock, flags);
> if (qp->valid) {
> /* request a send queue retry */
> @@ -110,6 +112,7 @@ void rnr_nak_timer(struct timer_list *t)
> rxe_sched_task(&qp->send_task);
> }
> spin_unlock_irqrestore(&qp->state_lock, flags);
> + rxe_put(qp);
> }
>
> static void req_check_sq_drain_done(struct rxe_qp *qp)
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH] RDMA/rxe: Fix race condition in QP timer handlers
2026-01-20 7:44 [PATCH] RDMA/rxe: Fix race condition in QP timer handlers Li Zhijian
2026-01-21 5:04 ` Zhu Yanjun
@ 2026-01-25 14:08 ` Leon Romanovsky
2026-01-25 21:24 ` Zhu Yanjun
2026-01-27 9:27 ` Zhijian Li (Fujitsu)
2026-01-28 10:25 ` Leon Romanovsky
2 siblings, 2 replies; 7+ messages in thread
From: Leon Romanovsky @ 2026-01-25 14:08 UTC (permalink / raw)
To: Li Zhijian; +Cc: linux-rdma, linux-kernel, zyjzyj2000, jgg
On Tue, Jan 20, 2026 at 03:44:37PM +0800, Li Zhijian wrote:
> I encontered the following warning:
> WARNING: drivers/infiniband/sw/rxe/rxe_task.c:249 at rxe_sched_task+0x1c8/0x238 [rdma_rxe], CPU#0: swapper/0/0
> ...
> libsha1 [last unloaded: ip6_udp_tunnel]
> CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G C 6.19.0-rc5-64k-v8+ #37 PREEMPT
> Tainted: [C]=CRAP
> Hardware name: Raspberry Pi 4 Model B Rev 1.2
> Call trace:
> rxe_sched_task+0x1c8/0x238 [rdma_rxe] (P)
> retransmit_timer+0x130/0x188 [rdma_rxe]
> call_timer_fn+0x68/0x4d0
> __run_timers+0x630/0x888
> ...
> WARNING: drivers/infiniband/sw/rxe/rxe_task.c:38 at rxe_sched_task+0x1c0/0x238 [rdma_rxe], CPU#0: swapper/0/0
> ...
> WARNING: drivers/infiniband/sw/rxe/rxe_task.c:111 at do_work+0x488/0x5c8 [rdma_rxe], CPU#3: kworker/u17:4/93400
> ...
> refcount_t: underflow; use-after-free.
> WARNING: lib/refcount.c:28 at refcount_warn_saturate+0x138/0x1a0, CPU#3: kworker/u17:4/93400
>
> The issue is caused by a race condition between retransmit_timer() and
> rxe_destroy_qp, leading to the Queue Pair's (QP) reference count dropping
> to zero during timer handler execution.
>
> It seems this warning is harmless because rxe_qp_do_cleanup() will flush
> all pending timers and requests.
>
> Example of flow causing the issue:
>
> CPU0 CPU1
> retransmit_timer() {
> spin_lock_irqsave
> rxe_destroy_qp()
> __rxe_cleanup()
> __rxe_put() // qp->ref_count decrease to 0
> rxe_qp_do_cleanup() {
> if (qp->valid) {
> rxe_sched_task() {
> WARN_ON(rxe_read(task->qp) <= 0);
> }
> }
> spin_unlock_irqrestore
> }
> spin_lock_irqsave
> qp->valid = 0
> spin_unlock_irqrestore
> }
>
> Ensure the QP's reference count is maintained and its validity is checked
> within the timer callbacks by adding calls to rxe_get(qp) and corresponding
> rxe_put(qp) after use.
>
> Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
Fixes line?
Thanks
> ---
> drivers/infiniband/sw/rxe/rxe_comp.c | 3 +++
> drivers/infiniband/sw/rxe/rxe_req.c | 3 +++
> 2 files changed, 6 insertions(+)
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH] RDMA/rxe: Fix race condition in QP timer handlers
2026-01-25 14:08 ` Leon Romanovsky
@ 2026-01-25 21:24 ` Zhu Yanjun
2026-01-27 9:27 ` Zhijian Li (Fujitsu)
1 sibling, 0 replies; 7+ messages in thread
From: Zhu Yanjun @ 2026-01-25 21:24 UTC (permalink / raw)
To: Leon Romanovsky, Li Zhijian; +Cc: linux-rdma, linux-kernel, zyjzyj2000, jgg
在 2026/1/25 6:08, Leon Romanovsky 写道:
> On Tue, Jan 20, 2026 at 03:44:37PM +0800, Li Zhijian wrote:
>> I encontered the following warning:
>> WARNING: drivers/infiniband/sw/rxe/rxe_task.c:249 at rxe_sched_task+0x1c8/0x238 [rdma_rxe], CPU#0: swapper/0/0
>> ...
>> libsha1 [last unloaded: ip6_udp_tunnel]
>> CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G C 6.19.0-rc5-64k-v8+ #37 PREEMPT
>> Tainted: [C]=CRAP
>> Hardware name: Raspberry Pi 4 Model B Rev 1.2
>> Call trace:
>> rxe_sched_task+0x1c8/0x238 [rdma_rxe] (P)
>> retransmit_timer+0x130/0x188 [rdma_rxe]
>> call_timer_fn+0x68/0x4d0
>> __run_timers+0x630/0x888
>> ...
>> WARNING: drivers/infiniband/sw/rxe/rxe_task.c:38 at rxe_sched_task+0x1c0/0x238 [rdma_rxe], CPU#0: swapper/0/0
>> ...
>> WARNING: drivers/infiniband/sw/rxe/rxe_task.c:111 at do_work+0x488/0x5c8 [rdma_rxe], CPU#3: kworker/u17:4/93400
>> ...
>> refcount_t: underflow; use-after-free.
>> WARNING: lib/refcount.c:28 at refcount_warn_saturate+0x138/0x1a0, CPU#3: kworker/u17:4/93400
>>
>> The issue is caused by a race condition between retransmit_timer() and
>> rxe_destroy_qp, leading to the Queue Pair's (QP) reference count dropping
>> to zero during timer handler execution.
>>
>> It seems this warning is harmless because rxe_qp_do_cleanup() will flush
>> all pending timers and requests.
>>
>> Example of flow causing the issue:
>>
>> CPU0 CPU1
>> retransmit_timer() {
>> spin_lock_irqsave
>> rxe_destroy_qp()
>> __rxe_cleanup()
>> __rxe_put() // qp->ref_count decrease to 0
>> rxe_qp_do_cleanup() {
>> if (qp->valid) {
>> rxe_sched_task() {
>> WARN_ON(rxe_read(task->qp) <= 0);
>> }
>> }
>> spin_unlock_irqrestore
>> }
>> spin_lock_irqsave
>> qp->valid = 0
>> spin_unlock_irqrestore
>> }
>>
>> Ensure the QP's reference count is maintained and its validity is checked
>> within the timer callbacks by adding calls to rxe_get(qp) and corresponding
>> rxe_put(qp) after use.
>>
>> Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
>
> Fixes line?
The Fixes line should be the following?
Fixes: 8700e3e7c485 ("Soft RoCE driver")
Best Regards,
Zhu Yanjun
>
> Thanks
>
>> ---
>> drivers/infiniband/sw/rxe/rxe_comp.c | 3 +++
>> drivers/infiniband/sw/rxe/rxe_req.c | 3 +++
>> 2 files changed, 6 insertions(+)
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH] RDMA/rxe: Fix race condition in QP timer handlers
2026-01-25 14:08 ` Leon Romanovsky
2026-01-25 21:24 ` Zhu Yanjun
@ 2026-01-27 9:27 ` Zhijian Li (Fujitsu)
2026-01-28 10:02 ` Leon Romanovsky
1 sibling, 1 reply; 7+ messages in thread
From: Zhijian Li (Fujitsu) @ 2026-01-27 9:27 UTC (permalink / raw)
To: Leon Romanovsky
Cc: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org,
zyjzyj2000@gmail.com, jgg@ziepe.ca
On 25/01/2026 22:08, Leon Romanovsky wrote:
>> Ensure the QP's reference count is maintained and its validity is checked
>> within the timer callbacks by adding calls to rxe_get(qp) and corresponding
>> rxe_put(qp) after use.
>>
>> Signed-off-by: Li Zhijian<lizhijian@fujitsu.com>
> Fixes line?
I believe the following `Fixes` tag is appropriate, as this commit introduced
the WARN_ON that now triggers:
Fixes: d94671632572 ("RDMA/rxe: Rewrite rxe_task.c")
However, I'm not entirely certain if this race condition also existed
before this commit, as it involved a significant rewrite.
Thanks
Zhijian
>
> Thanks
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH] RDMA/rxe: Fix race condition in QP timer handlers
2026-01-27 9:27 ` Zhijian Li (Fujitsu)
@ 2026-01-28 10:02 ` Leon Romanovsky
0 siblings, 0 replies; 7+ messages in thread
From: Leon Romanovsky @ 2026-01-28 10:02 UTC (permalink / raw)
To: Zhijian Li (Fujitsu)
Cc: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org,
zyjzyj2000@gmail.com, jgg@ziepe.ca
On Tue, Jan 27, 2026 at 09:27:27AM +0000, Zhijian Li (Fujitsu) wrote:
>
>
> On 25/01/2026 22:08, Leon Romanovsky wrote:
> >> Ensure the QP's reference count is maintained and its validity is checked
> >> within the timer callbacks by adding calls to rxe_get(qp) and corresponding
> >> rxe_put(qp) after use.
> >>
> >> Signed-off-by: Li Zhijian<lizhijian@fujitsu.com>
> > Fixes line?
>
> I believe the following `Fixes` tag is appropriate, as this commit introduced
> the WARN_ON that now triggers:
>
> Fixes: d94671632572 ("RDMA/rxe: Rewrite rxe_task.c")
>
> However, I'm not entirely certain if this race condition also existed
> before this commit, as it involved a significant rewrite.
Let's include the most recent fix. It is unlikely that anyone is still
running such an old version of RXE.
Thanks
>
> Thanks
> Zhijian
>
> >
> > Thanks
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] RDMA/rxe: Fix race condition in QP timer handlers
2026-01-20 7:44 [PATCH] RDMA/rxe: Fix race condition in QP timer handlers Li Zhijian
2026-01-21 5:04 ` Zhu Yanjun
2026-01-25 14:08 ` Leon Romanovsky
@ 2026-01-28 10:25 ` Leon Romanovsky
2 siblings, 0 replies; 7+ messages in thread
From: Leon Romanovsky @ 2026-01-28 10:25 UTC (permalink / raw)
To: linux-rdma, Li Zhijian; +Cc: linux-kernel, zyjzyj2000, jgg
On Tue, 20 Jan 2026 15:44:37 +0800, Li Zhijian wrote:
> I encontered the following warning:
> WARNING: drivers/infiniband/sw/rxe/rxe_task.c:249 at rxe_sched_task+0x1c8/0x238 [rdma_rxe], CPU#0: swapper/0/0
> ...
> libsha1 [last unloaded: ip6_udp_tunnel]
> CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G C 6.19.0-rc5-64k-v8+ #37 PREEMPT
> Tainted: [C]=CRAP
> Hardware name: Raspberry Pi 4 Model B Rev 1.2
> Call trace:
> rxe_sched_task+0x1c8/0x238 [rdma_rxe] (P)
> retransmit_timer+0x130/0x188 [rdma_rxe]
> call_timer_fn+0x68/0x4d0
> __run_timers+0x630/0x888
> ...
> WARNING: drivers/infiniband/sw/rxe/rxe_task.c:38 at rxe_sched_task+0x1c0/0x238 [rdma_rxe], CPU#0: swapper/0/0
> ...
> WARNING: drivers/infiniband/sw/rxe/rxe_task.c:111 at do_work+0x488/0x5c8 [rdma_rxe], CPU#3: kworker/u17:4/93400
> ...
> refcount_t: underflow; use-after-free.
> WARNING: lib/refcount.c:28 at refcount_warn_saturate+0x138/0x1a0, CPU#3: kworker/u17:4/93400
>
> [...]
Applied, thanks!
[1/1] RDMA/rxe: Fix race condition in QP timer handlers
https://git.kernel.org/rdma/rdma/c/87bf646921430e
Best regards,
--
Leon Romanovsky <leon@kernel.org>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2026-01-28 10:25 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-01-20 7:44 [PATCH] RDMA/rxe: Fix race condition in QP timer handlers Li Zhijian
2026-01-21 5:04 ` Zhu Yanjun
2026-01-25 14:08 ` Leon Romanovsky
2026-01-25 21:24 ` Zhu Yanjun
2026-01-27 9:27 ` Zhijian Li (Fujitsu)
2026-01-28 10:02 ` Leon Romanovsky
2026-01-28 10:25 ` Leon Romanovsky
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox