[PATCH rdma-next 0/4] RDMA/irdma: Fix SRQ completion handling

[PATCH rdma-next 0/4] RDMA/irdma: Fix SRQ completion handling

[Jacob Moroni]

SRQ support was introduced along with GEN3 HW support, but the kernel
implementation contained several bugs which rendered it unusable for
kernel consumers.

- The QP handle was being loaded from the incorrect offset, which
  would cause a crash/hang upon receiving the first receive completion
  for an SRQ-associated QP.
- Similar to above, the CQ scrubbing logic was scrubbing the wrong
  CQE offset, which would leave stale CQEs in the CQ for a QP that
  has been destroyed.
- When SRQ CQEs were scrubbed, the SRQ slots weren't being released.
- A NULL dereference could occur during CQ poll if scrubbed CQEs
  happen to be in the ring at the time of polling, due to the NULL
  check being located after the pointer was dereferenced.

SRQ functionality was verified using nvmet-rdma with use_srq=1 with
sequential+parallel I/O, disconnects under load, etc.

Jacob Moroni (4):
  RDMA/irdma: Fix QP handle assignment for SRQ completions
  RDMA/irdma: Check QP handle before decoding CQE errors
  RDMA/irdma: Fix CQ scrub for SRQ associated QPs
  RDMA/irdma: Release SRQ slots during CQ scrub

 drivers/infiniband/hw/irdma/uk.c   | 64 ++++++++++++++++++++----------
 drivers/infiniband/hw/irdma/user.h |  2 +-
 2 files changed, 44 insertions(+), 22 deletions(-)


base-commit: e50525a486300e5ff8014c9985b8477b2fca0368
-- 
2.56.0.455.g6c5a503b38-goog

[PATCH rdma-next 1/4] RDMA/irdma: Fix QP handle assignment for SRQ completions

[Jacob Moroni]

The QP handle comes from a different location in the CQE depending
on whether it's an SRQ or normal completion. For normal SQ/RQ CQEs,
it comes from offset 8, but for SRQs, it comes from offset 40.

This was correctly handled when the QP handle was initially loaded
in irdma_uk_cq_poll_cmpl(), but there was a leftover re-load of
the QP handle later on in that function which unconditionally
loaded it from offset 8. This would result in a bogus QP handle
being used when processing SRQ completions.

Fix this by removing the extra QP handle load.

While at it, load the QP handle into the u64 comp_ctx and cast
it rather than loading directly into the QP pointer since this
could overflow on 32 bit systems.

Fixes: 563e1feb5f6e ("RDMA/irdma: Add SRQ support")
Signed-off-by: Jacob Moroni <jmoroni@google.com>
---
 drivers/infiniband/hw/irdma/uk.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/drivers/infiniband/hw/irdma/uk.c b/drivers/infiniband/hw/irdma/uk.c
index a34883fe9983..250267ef5727 100644
--- a/drivers/infiniband/hw/irdma/uk.c
+++ b/drivers/infiniband/hw/irdma/uk.c
@@ -1220,11 +1220,11 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq,
 	is_srq = (u8)FIELD_GET(IRDMA_CQ_SRQ, qword3);
 	info->error = (bool)FIELD_GET(IRDMA_CQ_ERROR, qword3);
 	info->ipv4 = (bool)FIELD_GET(IRDMACQ_IPV4, qword3);
-	get_64bit_val(cqe, 8, &comp_ctx);
 	if (is_srq)
-		get_64bit_val(cqe, 40, (u64 *)&qp);
+		get_64bit_val(cqe, 40, &comp_ctx);
 	else
-		qp = (struct irdma_qp_uk *)(unsigned long)comp_ctx;
+		get_64bit_val(cqe, 8, &comp_ctx);
+	qp = (struct irdma_qp_uk *)(unsigned long)comp_ctx;
 	if (info->error) {
 		info->major_err = FIELD_GET(IRDMA_CQ_MAJERR, qword3);
 		info->minor_err = FIELD_GET(IRDMA_CQ_MINERR, qword3);
@@ -1268,10 +1268,7 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq,
 	info->qp_id = (u32)FIELD_GET(IRDMACQ_QPID, qword2);
 	info->ud_src_qpn = (u32)FIELD_GET(IRDMACQ_UDSRCQPN, qword2);
 
-	get_64bit_val(cqe, 8, &comp_ctx);
-
 	info->solicited_event = (bool)FIELD_GET(IRDMACQ_SOEVENT, qword3);
-	qp = (struct irdma_qp_uk *)(unsigned long)comp_ctx;
 	if (!qp || qp->destroy_pending) {
 		ret_code = -EFAULT;
 		goto exit;
-- 
2.56.0.455.g6c5a503b38-goog

[PATCH rdma-next 2/4] RDMA/irdma: Check QP handle before decoding CQE errors

[Jacob Moroni]

When a QP is destroyed, any pending completions for it are scrubbed
from the associated CQ ring(s) by setting the QP handle value to NULL.

When completions are polled, these scrubbed CQEs are identified and
skipped by checking for a NULL QP handle, but this check was
located after the QP handle could already be dereferenced.

Fix this by moving the NULL check earlier, right after the handle
is obtained.

Fixes: 42f1d099093b ("RDMA/irdma: Extend CQE Error and Flush Handling for GEN3 Devices")
Signed-off-by: Jacob Moroni <jmoroni@google.com>
---
 drivers/infiniband/hw/irdma/uk.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/infiniband/hw/irdma/uk.c b/drivers/infiniband/hw/irdma/uk.c
index 250267ef5727..800b51003e18 100644
--- a/drivers/infiniband/hw/irdma/uk.c
+++ b/drivers/infiniband/hw/irdma/uk.c
@@ -1225,6 +1225,10 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq,
 	else
 		get_64bit_val(cqe, 8, &comp_ctx);
 	qp = (struct irdma_qp_uk *)(unsigned long)comp_ctx;
+	if (!qp || qp->destroy_pending) {
+		ret_code = -EFAULT;
+		goto exit;
+	}
 	if (info->error) {
 		info->major_err = FIELD_GET(IRDMA_CQ_MAJERR, qword3);
 		info->minor_err = FIELD_GET(IRDMA_CQ_MINERR, qword3);
@@ -1269,10 +1273,6 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq,
 	info->ud_src_qpn = (u32)FIELD_GET(IRDMACQ_UDSRCQPN, qword2);
 
 	info->solicited_event = (bool)FIELD_GET(IRDMACQ_SOEVENT, qword3);
-	if (!qp || qp->destroy_pending) {
-		ret_code = -EFAULT;
-		goto exit;
-	}
 	wqe_idx = (u32)FIELD_GET(IRDMA_CQ_WQEIDX, qword3);
 	info->qp_handle = (irdma_qp_handle)(unsigned long)qp;
 	info->op_type = (u8)FIELD_GET(IRDMACQ_OP, qword3);
-- 
2.56.0.455.g6c5a503b38-goog

[PATCH rdma-next 3/4] RDMA/irdma: Fix CQ scrub for SRQ associated QPs

[Jacob Moroni]

When a QP is destroyed, any pending completions for it are
scrubbed by setting the QP handle to NULL so that the CQE
is skipped during poll. However, completions for SRQ WQEs
use a different offset (40 rather than 8) for the QP handle
but the driver did not account for this.

Fix this by checking whether the CQE being scrubbed is for
an SRQ WQE and if so, clearing the correct offset.

Fixes: 563e1feb5f6e ("RDMA/irdma: Add SRQ support")
Signed-off-by: Jacob Moroni <jmoroni@google.com>
---
 drivers/infiniband/hw/irdma/uk.c   | 16 +++++++++++-----
 drivers/infiniband/hw/irdma/user.h |  2 +-
 2 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/drivers/infiniband/hw/irdma/uk.c b/drivers/infiniband/hw/irdma/uk.c
index 800b51003e18..05f512b01525 100644
--- a/drivers/infiniband/hw/irdma/uk.c
+++ b/drivers/infiniband/hw/irdma/uk.c
@@ -1764,10 +1764,10 @@ void irdma_uk_cq_init(struct irdma_cq_uk *cq,
 
 /**
  * irdma_uk_clean_cq - clean cq entries
- * @q: completion context
+ * @qp: QP
  * @cq: cq to clean
  */
-void irdma_uk_clean_cq(void *q, struct irdma_cq_uk *cq)
+void irdma_uk_clean_cq(struct irdma_qp_uk *qp, struct irdma_cq_uk *cq)
 {
 	__le64 *cqe;
 	u64 qword3, comp_ctx;
@@ -1790,9 +1790,15 @@ void irdma_uk_clean_cq(void *q, struct irdma_cq_uk *cq)
 		/* Ensure CQE contents are read after valid bit is checked */
 		dma_rmb();
 
-		get_64bit_val(cqe, 8, &comp_ctx);
-		if ((void *)(unsigned long)comp_ctx == q)
-			set_64bit_val(cqe, 8, 0);
+		if (cq->avoid_mem_cflct && FIELD_GET(IRDMA_CQ_SRQ, qword3)) {
+			get_64bit_val(cqe, 40, &comp_ctx);
+			if ((struct irdma_qp_uk *)(unsigned long)comp_ctx == qp)
+				set_64bit_val(cqe, 40, 0);
+		} else {
+			get_64bit_val(cqe, 8, &comp_ctx);
+			if ((struct irdma_qp_uk *)(unsigned long)comp_ctx == qp)
+				set_64bit_val(cqe, 8, 0);
+		}
 
 		cq_head = (cq_head + 1) % cq->cq_ring.size;
 		if (!cq_head)
diff --git a/drivers/infiniband/hw/irdma/user.h b/drivers/infiniband/hw/irdma/user.h
index 4dd3776a4cdd..c7f115b28498 100644
--- a/drivers/infiniband/hw/irdma/user.h
+++ b/drivers/infiniband/hw/irdma/user.h
@@ -581,7 +581,7 @@ __le64 *irdma_qp_get_next_send_wqe(struct irdma_qp_uk *qp, u32 *wqe_idx,
 				   struct irdma_post_sq_info *info);
 __le64 *irdma_srq_get_next_recv_wqe(struct irdma_srq_uk *srq, u32 *wqe_idx);
 __le64 *irdma_qp_get_next_recv_wqe(struct irdma_qp_uk *qp, u32 *wqe_idx);
-void irdma_uk_clean_cq(void *q, struct irdma_cq_uk *cq);
+void irdma_uk_clean_cq(struct irdma_qp_uk *qp, struct irdma_cq_uk *cq);
 int irdma_nop(struct irdma_qp_uk *qp, u64 wr_id, bool signaled, bool post_sq);
 int irdma_fragcnt_to_quanta_sq(u32 frag_cnt, u16 *quanta);
 int irdma_fragcnt_to_wqesize_rq(u32 frag_cnt, u16 *wqe_size);
-- 
2.56.0.455.g6c5a503b38-goog

[PATCH rdma-next 4/4] RDMA/irdma: Release SRQ slots during CQ scrub

[Jacob Moroni]

When the CQ is scrubbed during QP destroy, any unpolled SRQ
completions are invalidated, which causes those SRQ slots to
be leaked because the SRQ tail only advances when completions
are polled.

Fix this by releasing the SRQ slots whenever an SRQ CQE is scrubbed
during QP destroy as well as during the small window prior to CQ
scrub where the QP is marked as "destroy_pending".

In practice, the leak only matters if a consumer posts new buffers
to replace the ones tied to the discarded completions, but releasing
the slots keeps the driver's SRQ accounting correct.

Fixes: 563e1feb5f6e ("RDMA/irdma: Add SRQ support")
Signed-off-by: Jacob Moroni <jmoroni@google.com>
---
 drivers/infiniband/hw/irdma/uk.c | 33 +++++++++++++++++++++++++-------
 1 file changed, 26 insertions(+), 7 deletions(-)

diff --git a/drivers/infiniband/hw/irdma/uk.c b/drivers/infiniband/hw/irdma/uk.c
index 05f512b01525..afd1f5ba0c28 100644
--- a/drivers/infiniband/hw/irdma/uk.c
+++ b/drivers/infiniband/hw/irdma/uk.c
@@ -1130,6 +1130,19 @@ bool irdma_uk_cq_empty(struct irdma_cq_uk *cq)
 	return polarity != cq->polarity;
 }
 
+/**
+ * irdma_srq_release_wqe() - Release an SRQ WQE.
+ * @srq: SRQ
+ */
+static void irdma_srq_release_wqe(struct irdma_srq_uk *srq)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(srq->lock, flags);
+	IRDMA_RING_MOVE_TAIL(srq->srq_ring);
+	spin_unlock_irqrestore(srq->lock, flags);
+}
+
 /**
  * irdma_uk_cq_poll_cmpl - get cq completion info
  * @cq: hw cq
@@ -1226,6 +1239,12 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq,
 		get_64bit_val(cqe, 8, &comp_ctx);
 	qp = (struct irdma_qp_uk *)(unsigned long)comp_ctx;
 	if (!qp || qp->destroy_pending) {
+		/* destroy_pending is set prior to locking and scrubbing the CQ
+		 * so this can potentially be reached.
+		 */
+		if (qp && is_srq && info->q_type == IRDMA_CQE_QTYPE_RQ &&
+		    qp->srq_uk)
+			irdma_srq_release_wqe(qp->srq_uk);
 		ret_code = -EFAULT;
 		goto exit;
 	}
@@ -1278,8 +1297,6 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq,
 	info->op_type = (u8)FIELD_GET(IRDMACQ_OP, qword3);
 
 	if (info->q_type == IRDMA_CQE_QTYPE_RQ && is_srq) {
-		unsigned long flags;
-
 		srq = qp->srq_uk;
 
 		get_64bit_val(cqe, 8, &info->wr_id);
@@ -1292,11 +1309,9 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq,
 		} else {
 			info->stag_invalid_set = false;
 		}
-		spin_lock_irqsave(srq->lock, flags);
-		IRDMA_RING_MOVE_TAIL(srq->srq_ring);
-		spin_unlock_irqrestore(srq->lock, flags);
-		pring = &srq->srq_ring;
 
+		irdma_srq_release_wqe(srq);
+		pring = &srq->srq_ring;
 	} else if (info->q_type == IRDMA_CQE_QTYPE_RQ && !is_srq) {
 		u32 array_idx;
 
@@ -1792,8 +1807,12 @@ void irdma_uk_clean_cq(struct irdma_qp_uk *qp, struct irdma_cq_uk *cq)
 
 		if (cq->avoid_mem_cflct && FIELD_GET(IRDMA_CQ_SRQ, qword3)) {
 			get_64bit_val(cqe, 40, &comp_ctx);
-			if ((struct irdma_qp_uk *)(unsigned long)comp_ctx == qp)
+			if ((struct irdma_qp_uk *)(unsigned long)comp_ctx == qp) {
+				if (qp->srq_uk &&
+				    FIELD_GET(IRDMA_CQ_SQ, qword3) == IRDMA_CQE_QTYPE_RQ)
+					irdma_srq_release_wqe(qp->srq_uk);
 				set_64bit_val(cqe, 40, 0);
+			}
 		} else {
 			get_64bit_val(cqe, 8, &comp_ctx);
 			if ((struct irdma_qp_uk *)(unsigned long)comp_ctx == qp)
-- 
2.56.0.455.g6c5a503b38-goog

Re: [PATCH rdma-next 1/4] RDMA/irdma: Fix QP handle assignment for SRQ completions

[sashiko-bot]

> The QP handle comes from a different location in the CQE depending
> on whether it's an SRQ or normal completion. For normal SQ/RQ CQEs,
> it comes from offset 8, but for SRQs, it comes from offset 40.
> 
> This was correctly handled when the QP handle was initially loaded
> in irdma_uk_cq_poll_cmpl(), but there was a leftover re-load of
> the QP handle later on in that function which unconditionally
> loaded it from offset 8. This would result in a bogus QP handle
> being used when processing SRQ completions.
> 
> Fix this by removing the extra QP handle load.
> 
> While at it, load the QP handle into the u64 comp_ctx and cast
> it rather than loading directly into the QP pointer since this
> could overflow on 32 bit systems.
> 
> Fixes: 563e1feb5f6e ("RDMA/irdma: Add SRQ support")
> Signed-off-by: Jacob Moroni <jmoroni@google.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106221201.2266301-1-jmoroni@google.com?part=1

Re: [PATCH rdma-next 2/4] RDMA/irdma: Check QP handle before decoding CQE errors

[sashiko-bot]

> When a QP is destroyed, any pending completions for it are scrubbed
> from the associated CQ ring(s) by setting the QP handle value to NULL.
> 
> When completions are polled, these scrubbed CQEs are identified and
> skipped by checking for a NULL QP handle, but this check was
> located after the QP handle could already be dereferenced.
> 
> Fix this by moving the NULL check earlier, right after the handle
> is obtained.
> 
> Fixes: 42f1d099093b ("RDMA/irdma: Extend CQE Error and Flush Handling for GEN3 Devices")
> Signed-off-by: Jacob Moroni <jmoroni@google.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106221201.2266301-1-jmoroni@google.com?part=2

Re: [PATCH rdma-next 4/4] RDMA/irdma: Release SRQ slots during CQ scrub

[sashiko-bot]

> When the CQ is scrubbed during QP destroy, any unpolled SRQ
> completions are invalidated, which causes those SRQ slots to
> be leaked because the SRQ tail only advances when completions
> are polled.
> 
> Fix this by releasing the SRQ slots whenever an SRQ CQE is scrubbed
> during QP destroy as well as during the small window prior to CQ
> scrub where the QP is marked as "destroy_pending".
> 
> In practice, the leak only matters if a consumer posts new buffers
> to replace the ones tied to the discarded completions, but releasing
> the slots keeps the driver's SRQ accounting correct.
> 
> Fixes: 563e1feb5f6e ("RDMA/irdma: Add SRQ support")
> Signed-off-by: Jacob Moroni <jmoroni@google.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106221201.2266301-1-jmoroni@google.com?part=4

Re: [PATCH rdma-next 3/4] RDMA/irdma: Fix CQ scrub for SRQ associated QPs

[sashiko-bot]

> When a QP is destroyed, any pending completions for it are
> scrubbed by setting the QP handle to NULL so that the CQE
> is skipped during poll. However, completions for SRQ WQEs
> use a different offset (40 rather than 8) for the QP handle
> but the driver did not account for this.
> 
> Fix this by checking whether the CQE being scrubbed is for
> an SRQ WQE and if so, clearing the correct offset.
> 
> Fixes: 563e1feb5f6e ("RDMA/irdma: Add SRQ support")
> Signed-off-by: Jacob Moroni <jmoroni@google.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106221201.2266301-1-jmoroni@google.com?part=3

Re: [PATCH rdma-next 1/4] RDMA/irdma: Fix QP handle assignment for SRQ completions

[David Hu]

On Fri, Nov 6, 2026 at 5:12 PM Jacob Moroni <jmoroni@google.com> wrote:
>
> The QP handle comes from a different location in the CQE depending
> on whether it's an SRQ or normal completion. For normal SQ/RQ CQEs,
> it comes from offset 8, but for SRQs, it comes from offset 40.
>
> This was correctly handled when the QP handle was initially loaded
> in irdma_uk_cq_poll_cmpl(), but there was a leftover re-load of
> the QP handle later on in that function which unconditionally
> loaded it from offset 8. This would result in a bogus QP handle
> being used when processing SRQ completions.
>
> Fix this by removing the extra QP handle load.
>
> While at it, load the QP handle into the u64 comp_ctx and cast
> it rather than loading directly into the QP pointer since this
> could overflow on 32 bit systems.
>
> Fixes: 563e1feb5f6e ("RDMA/irdma: Add SRQ support")
> Signed-off-by: Jacob Moroni <jmoroni@google.com>
> ---
>  drivers/infiniband/hw/irdma/uk.c | 9 +++------
>  1 file changed, 3 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/infiniband/hw/irdma/uk.c b/drivers/infiniband/hw/irdma/uk.c
> index a34883fe9983..250267ef5727 100644
> --- a/drivers/infiniband/hw/irdma/uk.c
> +++ b/drivers/infiniband/hw/irdma/uk.c
> @@ -1220,11 +1220,11 @@ int irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq,
>         is_srq = (u8)FIELD_GET(IRDMA_CQ_SRQ, qword3);
>         info->error = (bool)FIELD_GET(IRDMA_CQ_ERROR, qword3);
>         info->ipv4 = (bool)FIELD_GET(IRDMACQ_IPV4, qword3);
> -       get_64bit_val(cqe, 8, &comp_ctx);
>         if (is_srq)
> -               get_64bit_val(cqe, 40, (u64 *)&qp);
> +               get_64bit_val(cqe, 40, &comp_ctx);
>         else
> -               qp = (struct irdma_qp_uk *)(unsigned long)comp_ctx;
> +               get_64bit_val(cqe, 8, &comp_ctx);
> +       qp = (struct irdma_qp_uk *)(unsigned long)comp_ctx;
AULT;

Looks good. Curious if `cq->avoid_mem_cflct` is worth checking before
reading offset at 40, similar to that in 3/4

Re: [PATCH rdma-next 1/4] RDMA/irdma: Fix QP handle assignment for SRQ completions

[Jacob Moroni]

Hi,

Thanks for taking a look. It is not really necessary to check avoid_mem_cflct
in this case because SRQs are only supported on GEN3+, which always
use 64B CQEs. The reason I added the check in the clean loop is because
that loop walks every ring slot, so with 32B CQEs it can land on the second
half of an extended CQE, where bit 52 isn't the SRQ bit (it's undefined) and
reading offset 40 would run into the next slot (or past the end of the ring).
The poll path always skips the extension slot, so it doesn't have this problem.

Thanks,
Jake

Re: [PATCH rdma-next 1/4] RDMA/irdma: Fix QP handle assignment for SRQ completions

[David Hu]

On Tue, Nov 10, 2026 at 6:08 PM Jacob Moroni <jmoroni@google.com> wrote:
>
> Hi,
>
> Thanks for taking a look. It is not really necessary to check avoid_mem_cflct
> in this case because SRQs are only supported on GEN3+, which always
> use 64B CQEs. The reason I added the check in the clean loop is because
> that loop walks every ring slot, so with 32B CQEs it can land on the second
> half of an extended CQE, where bit 52 isn't the SRQ bit (it's undefined) and
> reading offset 40 would run into the next slot (or past the end of the ring).
> The poll path always skips the extension slot, so it doesn't have this problem.
>

Thank you, Jacob, for the clarification. That is helpful. Tidying up
the handling between default 32B CQE, 32B extended CQE, 64B CQE, SRQ,
and 64B extended CQE is not an easy task.

Reviewed-by: David Hu <xuehaohu@google.com>