[PATCH for-rc] RDMA/rxe: don't re-execute the current packet after the QP moved to error state

[PATCH for-rc] RDMA/rxe: don't re-execute the current packet after the QP moved to error state

[Allison Henderson]

When do_complete() finds the QP in the error state it returns
RESPST_CHK_RESOURCE.  Before commit 49dc9c1f0c7e ("RDMA/rxe: Cleanup
reset state handling in rxe_resp.c") this was the flush loop:
check_resource() had an error-state branch that fetched each remaining
recv WQE and completed it with IB_WC_WR_FLUSH_ERR, without touching
the current packet.  That commit removed the error-state branch from
check_resource() (draining is now done at rxe_receiver() entry) but
kept the do_complete() error-state return.

As a result, when a QP moves to the error state while a packet is
being completed - e.g. an rdma_cm disconnect racing with receive
processing - the responder state machine loops back into the request
processing chain with the already-completed packet still in hand:
check_resource() fetches a fresh recv WQE, execute()/send_data_in()
copies the same packet payload again, do_complete() posts another
IB_WC_SUCCESS CQE (qp->resp.status is still 0), and control returns
to the error-state check.  The loop re-executes the same packet once
per posted recv WQE (observed: ~1000 duplicate IB_WC_SUCCESS
completions of one SEND, one per ~8us, matching the RQ occupancy)
until the RQ is exhausted, after which qp->resp.wqe is NULL and
send_data_in() dereferences it:

  BUG: kernel NULL pointer dereference, address: 0000000000000014
  Workqueue: rxe_wq do_work
  RIP: copy_data+0x29/0x1f0
  Call Trace:
   send_data_in+0x25/0x50
   rxe_receiver+0xf36/0x1dd0

The duplicate completions are indistinguishable from real receives to
the ULP.  During an rds stress test, the message was accepted as new and
delivered the same datagram to user space hundreds of times, corrupting
the stream; any ULP that relies on RC exactly-once delivery is affected.

A live packet reaching the error-state check in do_complete() has
been executed and completed exactly once and must be consumed, not
re-processed.  Return RESPST_CLEANUP for it (dequeue and free); keep
returning RESPST_CHK_RESOURCE for the pkt == NULL case.

Fixes: 49dc9c1f0c7e ("RDMA/rxe: Cleanup reset state handling in rxe_resp.c")
Assisted-by: Claude-Code:claude-fable-5
Signed-off-by: Allison Henderson <achender@kernel.org>
---
 drivers/infiniband/sw/rxe/rxe_resp.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
index d8cbdfa70cdbd..02b16e2b49b8f 100644
--- a/drivers/infiniband/sw/rxe/rxe_resp.c
+++ b/drivers/infiniband/sw/rxe/rxe_resp.c
@@ -1217,7 +1217,14 @@ static enum resp_states do_complete(struct rxe_qp *qp,
 	spin_lock_irqsave(&qp->state_lock, flags);
 	if (unlikely(qp_state(qp) == IB_QPS_ERR)) {
 		spin_unlock_irqrestore(&qp->state_lock, flags);
-		return RESPST_CHK_RESOURCE;
+		/* The packet was executed and completed before the QP
+		 * moved to ERROR; it must be consumed exactly once.
+		 * Re-entering the request chain with the stale packet
+		 * would copy it into every remaining recv WQE as a new
+		 * completion.  Remaining WQEs are flushed by the drain
+		 * path at rxe_receiver() entry.
+		 */
+		return pkt ? RESPST_CLEANUP : RESPST_CHK_RESOURCE;
 	}
 	spin_unlock_irqrestore(&qp->state_lock, flags);
 
-- 
2.25.1

Re: [PATCH for-rc] RDMA/rxe: don't re-execute the current packet after the QP moved to error state

[Zhu Yanjun]

在 2026/7/11 9:54, Allison Henderson 写道:
> When do_complete() finds the QP in the error state it returns
> RESPST_CHK_RESOURCE.  Before commit 49dc9c1f0c7e ("RDMA/rxe: Cleanup
> reset state handling in rxe_resp.c") this was the flush loop:
> check_resource() had an error-state branch that fetched each remaining
> recv WQE and completed it with IB_WC_WR_FLUSH_ERR, without touching
> the current packet.  That commit removed the error-state branch from
> check_resource() (draining is now done at rxe_receiver() entry) but
> kept the do_complete() error-state return.
> 
> As a result, when a QP moves to the error state while a packet is
> being completed - e.g. an rdma_cm disconnect racing with receive
> processing - the responder state machine loops back into the request
> processing chain with the already-completed packet still in hand:
> check_resource() fetches a fresh recv WQE, execute()/send_data_in()
> copies the same packet payload again, do_complete() posts another
> IB_WC_SUCCESS CQE (qp->resp.status is still 0), and control returns
> to the error-state check.  The loop re-executes the same packet once
> per posted recv WQE (observed: ~1000 duplicate IB_WC_SUCCESS
> completions of one SEND, one per ~8us, matching the RQ occupancy)
> until the RQ is exhausted, after which qp->resp.wqe is NULL and
> send_data_in() dereferences it:
> 
>    BUG: kernel NULL pointer dereference, address: 0000000000000014
>    Workqueue: rxe_wq do_work
>    RIP: copy_data+0x29/0x1f0
>    Call Trace:
>     send_data_in+0x25/0x50
>     rxe_receiver+0xf36/0x1dd0
> 
> The duplicate completions are indistinguishable from real receives to
> the ULP.  During an rds stress test, the message was accepted as new and
> delivered the same datagram to user space hundreds of times, corrupting
> the stream; any ULP that relies on RC exactly-once delivery is affected.
> 
> A live packet reaching the error-state check in do_complete() has
> been executed and completed exactly once and must be consumed, not
> re-processed.  Return RESPST_CLEANUP for it (dequeue and free); keep
> returning RESPST_CHK_RESOURCE for the pkt == NULL case.
> 
> Fixes: 49dc9c1f0c7e ("RDMA/rxe: Cleanup reset state handling in rxe_resp.c")
> Assisted-by: Claude-Code:claude-fable-5
> Signed-off-by: Allison Henderson <achender@kernel.org>

The fix is correct and strictly adheres to the InfiniBand RC service
guarantees. When a QP transitions to the error state while processing
a live packet, returning RESPST_CLEANUP ensures the executed packet
is properly consumed rather than improperly re-injected into the request
chain. This accurately prevents both data stream corruption from
duplicate completions and the kernel panic caused by RQ exhaustion.

Thanks a lot.
Reviewed-by: Zhu Yanjun <yanjun.zhu@linux.dev>

Zhu Yanjun

> ---
>   drivers/infiniband/sw/rxe/rxe_resp.c | 9 ++++++++-
>   1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
> index d8cbdfa70cdbd..02b16e2b49b8f 100644
> --- a/drivers/infiniband/sw/rxe/rxe_resp.c
> +++ b/drivers/infiniband/sw/rxe/rxe_resp.c
> @@ -1217,7 +1217,14 @@ static enum resp_states do_complete(struct rxe_qp *qp,
>   	spin_lock_irqsave(&qp->state_lock, flags);
>   	if (unlikely(qp_state(qp) == IB_QPS_ERR)) {
>   		spin_unlock_irqrestore(&qp->state_lock, flags);
> -		return RESPST_CHK_RESOURCE;
> +		/* The packet was executed and completed before the QP
> +		 * moved to ERROR; it must be consumed exactly once.
> +		 * Re-entering the request chain with the stale packet
> +		 * would copy it into every remaining recv WQE as a new
> +		 * completion.  Remaining WQEs are flushed by the drain
> +		 * path at rxe_receiver() entry.
> +		 */
> +		return pkt ? RESPST_CLEANUP : RESPST_CHK_RESOURCE;
>   	}
>   	spin_unlock_irqrestore(&qp->state_lock, flags);
>   

Re: [PATCH for-rc] RDMA/rxe: don't re-execute the current packet after the QP moved to error state

[Leon Romanovsky]

On Sat, 11 Jul 2026 09:54:19 -0700, Allison Henderson wrote:
> When do_complete() finds the QP in the error state it returns
> RESPST_CHK_RESOURCE.  Before commit 49dc9c1f0c7e ("RDMA/rxe: Cleanup
> reset state handling in rxe_resp.c") this was the flush loop:
> check_resource() had an error-state branch that fetched each remaining
> recv WQE and completed it with IB_WC_WR_FLUSH_ERR, without touching
> the current packet.  That commit removed the error-state branch from
> check_resource() (draining is now done at rxe_receiver() entry) but
> kept the do_complete() error-state return.
> 
> [...]

Applied, thanks!

[1/1] RDMA/rxe: don't re-execute the current packet after the QP moved to error state
      https://git.kernel.org/rdma/rdma/c/15ae32c4a3551c

Best regards,
-- 
Leon Romanovsky <leon@kernel.org>