[PATCH rdma-next v4 0/4] Define and use mana queues for CQs and WQs

[PATCH rdma-next v4 0/4] Define and use mana queues for CQs and WQs

[Konstantin Taranov]

From: Konstantin Taranov <kotaranov@microsoft.com>

This patch series aims to reduce code duplication by
introducing a notion of mana ib queues and corresponding helpers
to create and destroy them.

v3->v4:
* Removed debug prints in patches, as asked by Leon

v2->v3:
* [in 4/4] Do not define an additional struct for a raw qp

v1->v2:
* [in 1/4] Added a comment about the ignored return value
* [in 2/4] Replaced RDMA:mana_ib to RDMA/mana_ib in the subject
* [in 4/4] Renamed mana_ib_raw_qp to mana_ib_raw_sq

Konstantin Taranov (4):
  RDMA/mana_ib: Introduce helpers to create and destroy mana queues
  RDMA/mana_ib: Use struct mana_ib_queue for CQs
  RDMA/mana_ib: Use struct mana_ib_queue for WQs
  RDMA/mana_ib: Use struct mana_ib_queue for RAW QPs

 drivers/infiniband/hw/mana/cq.c      | 52 +++-------------
 drivers/infiniband/hw/mana/main.c    | 39 ++++++++++++
 drivers/infiniband/hw/mana/mana_ib.h | 26 ++++----
 drivers/infiniband/hw/mana/qp.c      | 93 +++++++++-------------------
 drivers/infiniband/hw/mana/wq.c      | 33 ++--------
 5 files changed, 96 insertions(+), 147 deletions(-)


base-commit: 96d9cbe2f2ff7abde021bac75eafaceabe9a51fa
-- 
2.43.0

[PATCH rdma-next v4 1/4] RDMA/mana_ib: Introduce helpers to create and destroy mana queues

[Konstantin Taranov]

From: Konstantin Taranov <kotaranov@microsoft.com>

Intoduce helpers to work with mana ib queues (struct mana_ib_queue).
A queue always consists of umem, gdma_region, and id.
A queue can become a WQ or a CQ.

Signed-off-by: Konstantin Taranov <kotaranov@microsoft.com>
Reviewed-by: Long Li <longli@microsoft.com>
---
 drivers/infiniband/hw/mana/main.c    | 39 ++++++++++++++++++++++++++++
 drivers/infiniband/hw/mana/mana_ib.h | 10 +++++++
 2 files changed, 49 insertions(+)

diff --git a/drivers/infiniband/hw/mana/main.c b/drivers/infiniband/hw/mana/main.c
index 71e33feee..2d9d7a519 100644
--- a/drivers/infiniband/hw/mana/main.c
+++ b/drivers/infiniband/hw/mana/main.c
@@ -237,6 +237,45 @@ void mana_ib_dealloc_ucontext(struct ib_ucontext *ibcontext)
 		ibdev_dbg(ibdev, "Failed to destroy doorbell page %d\n", ret);
 }
 
+int mana_ib_create_queue(struct mana_ib_dev *mdev, u64 addr, u32 size,
+			 struct mana_ib_queue *queue)
+{
+	struct ib_umem *umem;
+	int err;
+
+	queue->umem = NULL;
+	queue->id = INVALID_QUEUE_ID;
+	queue->gdma_region = GDMA_INVALID_DMA_REGION;
+
+	umem = ib_umem_get(&mdev->ib_dev, addr, size, IB_ACCESS_LOCAL_WRITE);
+	if (IS_ERR(umem)) {
+		err = PTR_ERR(umem);
+		ibdev_dbg(&mdev->ib_dev, "Failed to get umem, %d\n", err);
+		return err;
+	}
+
+	err = mana_ib_create_zero_offset_dma_region(mdev, umem, &queue->gdma_region);
+	if (err) {
+		ibdev_dbg(&mdev->ib_dev, "Failed to create dma region, %d\n", err);
+		goto free_umem;
+	}
+	queue->umem = umem;
+
+	return 0;
+free_umem:
+	ib_umem_release(umem);
+	return err;
+}
+
+void mana_ib_destroy_queue(struct mana_ib_dev *mdev, struct mana_ib_queue *queue)
+{
+	/* Ignore return code as there is not much we can do about it.
+	 * The error message is printed inside.
+	 */
+	mana_ib_gd_destroy_dma_region(mdev, queue->gdma_region);
+	ib_umem_release(queue->umem);
+}
+
 static int
 mana_ib_gd_first_dma_region(struct mana_ib_dev *dev,
 			    struct gdma_context *gc,
diff --git a/drivers/infiniband/hw/mana/mana_ib.h b/drivers/infiniband/hw/mana/mana_ib.h
index f83390eeb..859fd3bfc 100644
--- a/drivers/infiniband/hw/mana/mana_ib.h
+++ b/drivers/infiniband/hw/mana/mana_ib.h
@@ -45,6 +45,12 @@ struct mana_ib_adapter_caps {
 	u32 max_inline_data_size;
 };
 
+struct mana_ib_queue {
+	struct ib_umem *umem;
+	u64 gdma_region;
+	u64 id;
+};
+
 struct mana_ib_dev {
 	struct ib_device ib_dev;
 	struct gdma_dev *gdma_dev;
@@ -169,6 +175,10 @@ int mana_ib_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
 int mana_ib_gd_destroy_dma_region(struct mana_ib_dev *dev,
 				  mana_handle_t gdma_region);
 
+int mana_ib_create_queue(struct mana_ib_dev *mdev, u64 addr, u32 size,
+			 struct mana_ib_queue *queue);
+void mana_ib_destroy_queue(struct mana_ib_dev *mdev, struct mana_ib_queue *queue);
+
 struct ib_wq *mana_ib_create_wq(struct ib_pd *pd,
 				struct ib_wq_init_attr *init_attr,
 				struct ib_udata *udata);
-- 
2.43.0

[PATCH rdma-next v4 2/4] RDMA/mana_ib: Use struct mana_ib_queue for CQs

[Konstantin Taranov]

From: Konstantin Taranov <kotaranov@microsoft.com>

Use struct mana_ib_queue and its helpers for CQs

Signed-off-by: Konstantin Taranov <kotaranov@microsoft.com>
Reviewed-by: Long Li <longli@microsoft.com>
---
 drivers/infiniband/hw/mana/cq.c      | 52 ++++++----------------------
 drivers/infiniband/hw/mana/mana_ib.h |  4 +--
 drivers/infiniband/hw/mana/qp.c      | 30 ++++++----------
 3 files changed, 22 insertions(+), 64 deletions(-)

diff --git a/drivers/infiniband/hw/mana/cq.c b/drivers/infiniband/hw/mana/cq.c
index 4a71e678d..c9129218f 100644
--- a/drivers/infiniband/hw/mana/cq.c
+++ b/drivers/infiniband/hw/mana/cq.c
@@ -39,37 +39,13 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
 	}
 
 	cq->cqe = attr->cqe;
-	cq->umem = ib_umem_get(ibdev, ucmd.buf_addr, cq->cqe * COMP_ENTRY_SIZE,
-			       IB_ACCESS_LOCAL_WRITE);
-	if (IS_ERR(cq->umem)) {
-		err = PTR_ERR(cq->umem);
-		ibdev_dbg(ibdev, "Failed to get umem for create cq, err %d\n",
-			  err);
-		return err;
-	}
-
-	err = mana_ib_create_zero_offset_dma_region(mdev, cq->umem, &cq->gdma_region);
+	err = mana_ib_create_queue(mdev, ucmd.buf_addr, cq->cqe * COMP_ENTRY_SIZE, &cq->queue);
 	if (err) {
-		ibdev_dbg(ibdev,
-			  "Failed to create dma region for create cq, %d\n",
-			  err);
-		goto err_release_umem;
+		ibdev_dbg(ibdev, "Failed to create queue for create cq, %d\n", err);
+		return err;
 	}
 
-	ibdev_dbg(ibdev,
-		  "create_dma_region ret %d gdma_region 0x%llx\n",
-		  err, cq->gdma_region);
-
-	/*
-	 * The CQ ID is not known at this time. The ID is generated at create_qp
-	 */
-	cq->id = INVALID_QUEUE_ID;
-
 	return 0;
-
-err_release_umem:
-	ib_umem_release(cq->umem);
-	return err;
 }
 
 int mana_ib_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
@@ -78,24 +54,16 @@ int mana_ib_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
 	struct ib_device *ibdev = ibcq->device;
 	struct mana_ib_dev *mdev;
 	struct gdma_context *gc;
-	int err;
 
 	mdev = container_of(ibdev, struct mana_ib_dev, ib_dev);
 	gc = mdev_to_gc(mdev);
 
-	err = mana_ib_gd_destroy_dma_region(mdev, cq->gdma_region);
-	if (err) {
-		ibdev_dbg(ibdev,
-			  "Failed to destroy dma region, %d\n", err);
-		return err;
-	}
-
-	if (cq->id != INVALID_QUEUE_ID) {
-		kfree(gc->cq_table[cq->id]);
-		gc->cq_table[cq->id] = NULL;
+	if (cq->queue.id != INVALID_QUEUE_ID) {
+		kfree(gc->cq_table[cq->queue.id]);
+		gc->cq_table[cq->queue.id] = NULL;
 	}
 
-	ib_umem_release(cq->umem);
+	mana_ib_destroy_queue(mdev, &cq->queue);
 
 	return 0;
 }
@@ -114,7 +82,7 @@ int mana_ib_install_cq_cb(struct mana_ib_dev *mdev, struct mana_ib_cq *cq)
 	struct gdma_queue *gdma_cq;
 
 	/* Create CQ table entry */
-	WARN_ON(gc->cq_table[cq->id]);
+	WARN_ON(gc->cq_table[cq->queue.id]);
 	gdma_cq = kzalloc(sizeof(*gdma_cq), GFP_KERNEL);
 	if (!gdma_cq)
 		return -ENOMEM;
@@ -122,7 +90,7 @@ int mana_ib_install_cq_cb(struct mana_ib_dev *mdev, struct mana_ib_cq *cq)
 	gdma_cq->cq.context = cq;
 	gdma_cq->type = GDMA_CQ;
 	gdma_cq->cq.callback = mana_ib_cq_handler;
-	gdma_cq->id = cq->id;
-	gc->cq_table[cq->id] = gdma_cq;
+	gdma_cq->id = cq->queue.id;
+	gc->cq_table[cq->queue.id] = gdma_cq;
 	return 0;
 }
diff --git a/drivers/infiniband/hw/mana/mana_ib.h b/drivers/infiniband/hw/mana/mana_ib.h
index 859fd3bfc..6acb5c281 100644
--- a/drivers/infiniband/hw/mana/mana_ib.h
+++ b/drivers/infiniband/hw/mana/mana_ib.h
@@ -88,10 +88,8 @@ struct mana_ib_mr {
 
 struct mana_ib_cq {
 	struct ib_cq ibcq;
-	struct ib_umem *umem;
+	struct mana_ib_queue queue;
 	int cqe;
-	u64 gdma_region;
-	u64 id;
 	u32 comp_vector;
 };
 
diff --git a/drivers/infiniband/hw/mana/qp.c b/drivers/infiniband/hw/mana/qp.c
index 6e7627745..adf613b82 100644
--- a/drivers/infiniband/hw/mana/qp.c
+++ b/drivers/infiniband/hw/mana/qp.c
@@ -197,7 +197,7 @@ static int mana_ib_create_qp_rss(struct ib_qp *ibqp, struct ib_pd *pd,
 		wq_spec.gdma_region = wq->gdma_region;
 		wq_spec.queue_size = wq->wq_buf_size;
 
-		cq_spec.gdma_region = cq->gdma_region;
+		cq_spec.gdma_region = cq->queue.gdma_region;
 		cq_spec.queue_size = cq->cqe * COMP_ENTRY_SIZE;
 		cq_spec.modr_ctx_id = 0;
 		eq = &mpc->ac->eqs[cq->comp_vector % gc->max_num_queues];
@@ -213,16 +213,12 @@ static int mana_ib_create_qp_rss(struct ib_qp *ibqp, struct ib_pd *pd,
 
 		/* The GDMA regions are now owned by the WQ object */
 		wq->gdma_region = GDMA_INVALID_DMA_REGION;
-		cq->gdma_region = GDMA_INVALID_DMA_REGION;
+		cq->queue.gdma_region = GDMA_INVALID_DMA_REGION;
 
 		wq->id = wq_spec.queue_index;
-		cq->id = cq_spec.queue_index;
+		cq->queue.id = cq_spec.queue_index;
 
-		ibdev_dbg(&mdev->ib_dev,
-			  "ret %d rx_object 0x%llx wq id %llu cq id %llu\n",
-			  ret, wq->rx_object, wq->id, cq->id);
-
-		resp.entries[i].cqid = cq->id;
+		resp.entries[i].cqid = cq->queue.id;
 		resp.entries[i].wqid = wq->id;
 
 		mana_ind_table[i] = wq->rx_object;
@@ -232,7 +228,7 @@ static int mana_ib_create_qp_rss(struct ib_qp *ibqp, struct ib_pd *pd,
 		if (ret)
 			goto fail;
 
-		gdma_cq_allocated[i] = gc->cq_table[cq->id];
+		gdma_cq_allocated[i] = gc->cq_table[cq->queue.id];
 	}
 	resp.num_entries = i;
 
@@ -264,7 +260,7 @@ static int mana_ib_create_qp_rss(struct ib_qp *ibqp, struct ib_pd *pd,
 		wq = container_of(ibwq, struct mana_ib_wq, ibwq);
 		cq = container_of(ibcq, struct mana_ib_cq, ibcq);
 
-		gc->cq_table[cq->id] = NULL;
+		gc->cq_table[cq->queue.id] = NULL;
 		kfree(gdma_cq_allocated[i]);
 
 		mana_destroy_wq_obj(mpc, GDMA_RQ, wq->rx_object);
@@ -374,7 +370,7 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,
 	wq_spec.gdma_region = qp->sq_gdma_region;
 	wq_spec.queue_size = ucmd.sq_buf_size;
 
-	cq_spec.gdma_region = send_cq->gdma_region;
+	cq_spec.gdma_region = send_cq->queue.gdma_region;
 	cq_spec.queue_size = send_cq->cqe * COMP_ENTRY_SIZE;
 	cq_spec.modr_ctx_id = 0;
 	eq_vec = send_cq->comp_vector % gc->max_num_queues;
@@ -392,22 +388,18 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,
 
 	/* The GDMA regions are now owned by the WQ object */
 	qp->sq_gdma_region = GDMA_INVALID_DMA_REGION;
-	send_cq->gdma_region = GDMA_INVALID_DMA_REGION;
+	send_cq->queue.gdma_region = GDMA_INVALID_DMA_REGION;
 
 	qp->sq_id = wq_spec.queue_index;
-	send_cq->id = cq_spec.queue_index;
+	send_cq->queue.id = cq_spec.queue_index;
 
 	/* Create CQ table entry */
 	err = mana_ib_install_cq_cb(mdev, send_cq);
 	if (err)
 		goto err_destroy_wq_obj;
 
-	ibdev_dbg(&mdev->ib_dev,
-		  "ret %d qp->tx_object 0x%llx sq id %llu cq id %llu\n", err,
-		  qp->tx_object, qp->sq_id, send_cq->id);
-
 	resp.sqid = qp->sq_id;
-	resp.cqid = send_cq->id;
+	resp.cqid = send_cq->queue.id;
 	resp.tx_vp_offset = pd->tx_vp_offset;
 
 	err = ib_copy_to_udata(udata, &resp, sizeof(resp));
@@ -422,7 +414,7 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,
 
 err_release_gdma_cq:
 	kfree(gdma_cq);
-	gc->cq_table[send_cq->id] = NULL;
+	gc->cq_table[send_cq->queue.id] = NULL;
 
 err_destroy_wq_obj:
 	mana_destroy_wq_obj(mpc, GDMA_SQ, qp->tx_object);
-- 
2.43.0

[PATCH rdma-next v4 3/4] RDMA/mana_ib: Use struct mana_ib_queue for WQs

[Konstantin Taranov]

From: Konstantin Taranov <kotaranov@microsoft.com>

Use struct mana_ib_queue and its helpers for WQs

Signed-off-by: Konstantin Taranov <kotaranov@microsoft.com>
Reviewed-by: Long Li <longli@microsoft.com>
---
 drivers/infiniband/hw/mana/mana_ib.h |  4 +---
 drivers/infiniband/hw/mana/qp.c      |  8 +++----
 drivers/infiniband/hw/mana/wq.c      | 33 ++++------------------------
 3 files changed, 9 insertions(+), 36 deletions(-)

diff --git a/drivers/infiniband/hw/mana/mana_ib.h b/drivers/infiniband/hw/mana/mana_ib.h
index 6acb5c281..a8953ee80 100644
--- a/drivers/infiniband/hw/mana/mana_ib.h
+++ b/drivers/infiniband/hw/mana/mana_ib.h
@@ -59,11 +59,9 @@ struct mana_ib_dev {
 
 struct mana_ib_wq {
 	struct ib_wq ibwq;
-	struct ib_umem *umem;
+	struct mana_ib_queue queue;
 	int wqe;
 	u32 wq_buf_size;
-	u64 gdma_region;
-	u64 id;
 	mana_handle_t rx_object;
 };
 
diff --git a/drivers/infiniband/hw/mana/qp.c b/drivers/infiniband/hw/mana/qp.c
index adf613b82..8eccd0e6f 100644
--- a/drivers/infiniband/hw/mana/qp.c
+++ b/drivers/infiniband/hw/mana/qp.c
@@ -194,7 +194,7 @@ static int mana_ib_create_qp_rss(struct ib_qp *ibqp, struct ib_pd *pd,
 		ibcq = ibwq->cq;
 		cq = container_of(ibcq, struct mana_ib_cq, ibcq);
 
-		wq_spec.gdma_region = wq->gdma_region;
+		wq_spec.gdma_region = wq->queue.gdma_region;
 		wq_spec.queue_size = wq->wq_buf_size;
 
 		cq_spec.gdma_region = cq->queue.gdma_region;
@@ -212,14 +212,14 @@ static int mana_ib_create_qp_rss(struct ib_qp *ibqp, struct ib_pd *pd,
 		}
 
 		/* The GDMA regions are now owned by the WQ object */
-		wq->gdma_region = GDMA_INVALID_DMA_REGION;
+		wq->queue.gdma_region = GDMA_INVALID_DMA_REGION;
 		cq->queue.gdma_region = GDMA_INVALID_DMA_REGION;
 
-		wq->id = wq_spec.queue_index;
+		wq->queue.id = wq_spec.queue_index;
 		cq->queue.id = cq_spec.queue_index;
 
 		resp.entries[i].cqid = cq->queue.id;
-		resp.entries[i].wqid = wq->id;
+		resp.entries[i].wqid = wq->queue.id;
 
 		mana_ind_table[i] = wq->rx_object;
 
diff --git a/drivers/infiniband/hw/mana/wq.c b/drivers/infiniband/hw/mana/wq.c
index 7c9c69962..524261904 100644
--- a/drivers/infiniband/hw/mana/wq.c
+++ b/drivers/infiniband/hw/mana/wq.c
@@ -13,7 +13,6 @@ struct ib_wq *mana_ib_create_wq(struct ib_pd *pd,
 		container_of(pd->device, struct mana_ib_dev, ib_dev);
 	struct mana_ib_create_wq ucmd = {};
 	struct mana_ib_wq *wq;
-	struct ib_umem *umem;
 	int err;
 
 	if (udata->inlen < sizeof(ucmd))
@@ -30,41 +29,18 @@ struct ib_wq *mana_ib_create_wq(struct ib_pd *pd,
 	if (!wq)
 		return ERR_PTR(-ENOMEM);
 
-	ibdev_dbg(&mdev->ib_dev, "ucmd wq_buf_addr 0x%llx\n", ucmd.wq_buf_addr);
-
-	umem = ib_umem_get(pd->device, ucmd.wq_buf_addr, ucmd.wq_buf_size,
-			   IB_ACCESS_LOCAL_WRITE);
-	if (IS_ERR(umem)) {
-		err = PTR_ERR(umem);
+	err = mana_ib_create_queue(mdev, ucmd.wq_buf_addr, ucmd.wq_buf_size, &wq->queue);
+	if (err) {
 		ibdev_dbg(&mdev->ib_dev,
-			  "Failed to get umem for create wq, err %d\n", err);
+			  "Failed to create queue for create wq, %d\n", err);
 		goto err_free_wq;
 	}
 
-	wq->umem = umem;
 	wq->wqe = init_attr->max_wr;
 	wq->wq_buf_size = ucmd.wq_buf_size;
 	wq->rx_object = INVALID_MANA_HANDLE;
-
-	err = mana_ib_create_zero_offset_dma_region(mdev, wq->umem, &wq->gdma_region);
-	if (err) {
-		ibdev_dbg(&mdev->ib_dev,
-			  "Failed to create dma region for create wq, %d\n",
-			  err);
-		goto err_release_umem;
-	}
-
-	ibdev_dbg(&mdev->ib_dev,
-		  "create_dma_region ret %d gdma_region 0x%llx\n",
-		  err, wq->gdma_region);
-
-	/* WQ ID is returned at wq_create time, doesn't know the value yet */
-
 	return &wq->ibwq;
 
-err_release_umem:
-	ib_umem_release(umem);
-
 err_free_wq:
 	kfree(wq);
 
@@ -86,8 +62,7 @@ int mana_ib_destroy_wq(struct ib_wq *ibwq, struct ib_udata *udata)
 
 	mdev = container_of(ib_dev, struct mana_ib_dev, ib_dev);
 
-	mana_ib_gd_destroy_dma_region(mdev, wq->gdma_region);
-	ib_umem_release(wq->umem);
+	mana_ib_destroy_queue(mdev, &wq->queue);
 
 	kfree(wq);
 
-- 
2.43.0

[PATCH rdma-next v4 4/4] RDMA/mana_ib: Use struct mana_ib_queue for RAW QPs

[Konstantin Taranov]

From: Konstantin Taranov <kotaranov@microsoft.com>

Use struct mana_ib_queue and its helpers for RAW QPs

Signed-off-by: Konstantin Taranov <kotaranov@microsoft.com>
Reviewed-by: Long Li <longli@microsoft.com>
---
 drivers/infiniband/hw/mana/mana_ib.h |  8 +---
 drivers/infiniband/hw/mana/qp.c      | 55 +++++++---------------------
 2 files changed, 16 insertions(+), 47 deletions(-)

diff --git a/drivers/infiniband/hw/mana/mana_ib.h b/drivers/infiniband/hw/mana/mana_ib.h
index a8953ee80..ceca21cef 100644
--- a/drivers/infiniband/hw/mana/mana_ib.h
+++ b/drivers/infiniband/hw/mana/mana_ib.h
@@ -94,12 +94,8 @@ struct mana_ib_cq {
 struct mana_ib_qp {
 	struct ib_qp ibqp;
 
-	/* Work queue info */
-	struct ib_umem *sq_umem;
-	int sqe;
-	u64 sq_gdma_region;
-	u64 sq_id;
-	mana_handle_t tx_object;
+	mana_handle_t qp_handle;
+	struct mana_ib_queue raw_sq;
 
 	/* The port on the IB device, starting with 1 */
 	u32 port;
diff --git a/drivers/infiniband/hw/mana/qp.c b/drivers/infiniband/hw/mana/qp.c
index 8eccd0e6f..0705e1995 100644
--- a/drivers/infiniband/hw/mana/qp.c
+++ b/drivers/infiniband/hw/mana/qp.c
@@ -293,7 +293,6 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,
 	struct mana_obj_spec cq_spec = {};
 	struct mana_port_context *mpc;
 	struct net_device *ndev;
-	struct ib_umem *umem;
 	struct mana_eq *eq;
 	int eq_vec;
 	u32 port;
@@ -339,35 +338,15 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,
 
 	qp->port = port;
 
-	ibdev_dbg(&mdev->ib_dev, "ucmd sq_buf_addr 0x%llx port %u\n",
-		  ucmd.sq_buf_addr, ucmd.port);
-
-	umem = ib_umem_get(ibpd->device, ucmd.sq_buf_addr, ucmd.sq_buf_size,
-			   IB_ACCESS_LOCAL_WRITE);
-	if (IS_ERR(umem)) {
-		err = PTR_ERR(umem);
-		ibdev_dbg(&mdev->ib_dev,
-			  "Failed to get umem for create qp-raw, err %d\n",
-			  err);
-		goto err_free_vport;
-	}
-	qp->sq_umem = umem;
-
-	err = mana_ib_create_zero_offset_dma_region(mdev, qp->sq_umem,
-						    &qp->sq_gdma_region);
+	err = mana_ib_create_queue(mdev, ucmd.sq_buf_addr, ucmd.sq_buf_size, &qp->raw_sq);
 	if (err) {
 		ibdev_dbg(&mdev->ib_dev,
-			  "Failed to create dma region for create qp-raw, %d\n",
-			  err);
-		goto err_release_umem;
+			  "Failed to create queue for create qp-raw, err %d\n", err);
+		goto err_free_vport;
 	}
 
-	ibdev_dbg(&mdev->ib_dev,
-		  "create_dma_region ret %d gdma_region 0x%llx\n",
-		  err, qp->sq_gdma_region);
-
 	/* Create a WQ on the same port handle used by the Ethernet */
-	wq_spec.gdma_region = qp->sq_gdma_region;
+	wq_spec.gdma_region = qp->raw_sq.gdma_region;
 	wq_spec.queue_size = ucmd.sq_buf_size;
 
 	cq_spec.gdma_region = send_cq->queue.gdma_region;
@@ -378,19 +357,19 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,
 	cq_spec.attached_eq = eq->eq->id;
 
 	err = mana_create_wq_obj(mpc, mpc->port_handle, GDMA_SQ, &wq_spec,
-				 &cq_spec, &qp->tx_object);
+				 &cq_spec, &qp->qp_handle);
 	if (err) {
 		ibdev_dbg(&mdev->ib_dev,
 			  "Failed to create wq for create raw-qp, err %d\n",
 			  err);
-		goto err_destroy_dma_region;
+		goto err_destroy_queue;
 	}
 
 	/* The GDMA regions are now owned by the WQ object */
-	qp->sq_gdma_region = GDMA_INVALID_DMA_REGION;
+	qp->raw_sq.gdma_region = GDMA_INVALID_DMA_REGION;
 	send_cq->queue.gdma_region = GDMA_INVALID_DMA_REGION;
 
-	qp->sq_id = wq_spec.queue_index;
+	qp->raw_sq.id = wq_spec.queue_index;
 	send_cq->queue.id = cq_spec.queue_index;
 
 	/* Create CQ table entry */
@@ -398,7 +377,7 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,
 	if (err)
 		goto err_destroy_wq_obj;
 
-	resp.sqid = qp->sq_id;
+	resp.sqid = qp->raw_sq.id;
 	resp.cqid = send_cq->queue.id;
 	resp.tx_vp_offset = pd->tx_vp_offset;
 
@@ -417,13 +396,10 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,
 	gc->cq_table[send_cq->queue.id] = NULL;
 
 err_destroy_wq_obj:
-	mana_destroy_wq_obj(mpc, GDMA_SQ, qp->tx_object);
+	mana_destroy_wq_obj(mpc, GDMA_SQ, qp->qp_handle);
 
-err_destroy_dma_region:
-	mana_ib_gd_destroy_dma_region(mdev, qp->sq_gdma_region);
-
-err_release_umem:
-	ib_umem_release(umem);
+err_destroy_queue:
+	mana_ib_destroy_queue(mdev, &qp->raw_sq);
 
 err_free_vport:
 	mana_ib_uncfg_vport(mdev, pd, port);
@@ -497,12 +473,9 @@ static int mana_ib_destroy_qp_raw(struct mana_ib_qp *qp, struct ib_udata *udata)
 	mpc = netdev_priv(ndev);
 	pd = container_of(ibpd, struct mana_ib_pd, ibpd);
 
-	mana_destroy_wq_obj(mpc, GDMA_SQ, qp->tx_object);
+	mana_destroy_wq_obj(mpc, GDMA_SQ, qp->qp_handle);
 
-	if (qp->sq_umem) {
-		mana_ib_gd_destroy_dma_region(mdev, qp->sq_gdma_region);
-		ib_umem_release(qp->sq_umem);
-	}
+	mana_ib_destroy_queue(mdev, &qp->raw_sq);
 
 	mana_ib_uncfg_vport(mdev, pd, qp->port);
 
-- 
2.43.0

Re: [PATCH rdma-next v4 0/4] Define and use mana queues for CQs and WQs

[Leon Romanovsky]

On Mon, Apr 08, 2024 at 02:14:02AM -0700, Konstantin Taranov wrote:
> From: Konstantin Taranov <kotaranov@microsoft.com>
> 
> This patch series aims to reduce code duplication by
> introducing a notion of mana ib queues and corresponding helpers
> to create and destroy them.
> 
> v3->v4:
> * Removed debug prints in patches, as asked by Leon
> 
> v2->v3:
> * [in 4/4] Do not define an additional struct for a raw qp
> 
> v1->v2:
> * [in 1/4] Added a comment about the ignored return value
> * [in 2/4] Replaced RDMA:mana_ib to RDMA/mana_ib in the subject
> * [in 4/4] Renamed mana_ib_raw_qp to mana_ib_raw_sq
> 
> Konstantin Taranov (4):
>   RDMA/mana_ib: Introduce helpers to create and destroy mana queues
>   RDMA/mana_ib: Use struct mana_ib_queue for CQs
>   RDMA/mana_ib: Use struct mana_ib_queue for WQs
>   RDMA/mana_ib: Use struct mana_ib_queue for RAW QPs
> 
>  drivers/infiniband/hw/mana/cq.c      | 52 +++-------------
>  drivers/infiniband/hw/mana/main.c    | 39 ++++++++++++
>  drivers/infiniband/hw/mana/mana_ib.h | 26 ++++----
>  drivers/infiniband/hw/mana/qp.c      | 93 +++++++++-------------------
>  drivers/infiniband/hw/mana/wq.c      | 33 ++--------
>  5 files changed, 96 insertions(+), 147 deletions(-)

It doesn't apply.

Grabbing thread from lore.kernel.org/all/1712567646-5247-1-git-send-email-kotaranov@linux.microsoft.com/t.mbox.gz
Checking for newer revisions
Grabbing search results from lore.kernel.org
Analyzing 5 messages in the thread
Looking for additional code-review trailers on lore.kernel.org
Checking attestation on all messages, may take a moment...
Retrieving CI status, may take a moment...
---
  ✓ [PATCH v4 1/4] RDMA/mana_ib: Introduce helpers to create and destroy mana queues
    + Link: https://lore.kernel.org/r/1712567646-5247-2-git-send-email-kotaranov@linux.microsoft.com
    + Signed-off-by: Leon Romanovsky <leon@kernel.org>
  ✓ [PATCH v4 2/4] RDMA/mana_ib: Use struct mana_ib_queue for CQs
    + Link: https://lore.kernel.org/r/1712567646-5247-3-git-send-email-kotaranov@linux.microsoft.com
    + Signed-off-by: Leon Romanovsky <leon@kernel.org>
  ✓ [PATCH v4 3/4] RDMA/mana_ib: Use struct mana_ib_queue for WQs
    + Link: https://lore.kernel.org/r/1712567646-5247-4-git-send-email-kotaranov@linux.microsoft.com
    + Signed-off-by: Leon Romanovsky <leon@kernel.org>
  ✓ [PATCH v4 4/4] RDMA/mana_ib: Use struct mana_ib_queue for RAW QPs
    + Link: https://lore.kernel.org/r/1712567646-5247-5-git-send-email-kotaranov@linux.microsoft.com
    + Signed-off-by: Leon Romanovsky <leon@kernel.org>
  ---
  ✓ Signed: DKIM/linux.microsoft.com
---
Total patches: 4
---
 Base: using specified base-commit 96d9cbe2f2ff7abde021bac75eafaceabe9a51fa
Applying: RDMA/mana_ib: Introduce helpers to create and destroy mana queues
Patch failed at 0001 RDMA/mana_ib: Introduce helpers to create and destroy mana queues
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
error: patch failed: drivers/infiniband/hw/mana/main.c:237
error: drivers/infiniband/hw/mana/main.c: patch does not apply
error: patch failed: drivers/infiniband/hw/mana/mana_ib.h:45
error: drivers/infiniband/hw/mana/mana_ib.h: patch does not apply
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Press any key to continue...

RE: [EXTERNAL] Re: [PATCH rdma-next v4 0/4] Define and use mana queues for CQs and WQs

[Konstantin Taranov]

> From: Leon Romanovsky <leon@kernel.org>
> On Mon, Apr 08, 2024 at 02:14:02AM -0700, Konstantin Taranov wrote:
> > From: Konstantin Taranov <kotaranov@microsoft.com>
> >
> > This patch series aims to reduce code duplication by introducing a
> > notion of mana ib queues and corresponding helpers to create and
> > destroy them.
> >
> > v3->v4:
> > * Removed debug prints in patches, as asked by Leon
> >
> > v2->v3:
> > * [in 4/4] Do not define an additional struct for a raw qp
> >
> > v1->v2:
> > * [in 1/4] Added a comment about the ignored return value
> > * [in 2/4] Replaced RDMA:mana_ib to RDMA/mana_ib in the subject
> > * [in 4/4] Renamed mana_ib_raw_qp to mana_ib_raw_sq
> >
> > Konstantin Taranov (4):
> >   RDMA/mana_ib: Introduce helpers to create and destroy mana queues
> >   RDMA/mana_ib: Use struct mana_ib_queue for CQs
> >   RDMA/mana_ib: Use struct mana_ib_queue for WQs
> >   RDMA/mana_ib: Use struct mana_ib_queue for RAW QPs
> >
> >  drivers/infiniband/hw/mana/cq.c      | 52 +++-------------
> >  drivers/infiniband/hw/mana/main.c    | 39 ++++++++++++
> >  drivers/infiniband/hw/mana/mana_ib.h | 26 ++++----
> >  drivers/infiniband/hw/mana/qp.c      | 93 +++++++++-------------------
> >  drivers/infiniband/hw/mana/wq.c      | 33 ++--------
> >  5 files changed, 96 insertions(+), 147 deletions(-)
> 
> It doesn't apply.
> 

I guess there was some mis-synchronisation between us.
I see that you have already applied the patch 6 days ago:
https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git/log/

I am sorry for sending a newer version after the patch has been applied.
I have not checked this before sending.
I can take care of useless debug prints in a future cleanup patch.

Konstantin

> Grabbing thread from lore.kernel.org/all/1712567646-5247-1-git-send-
> email-kotaranov@linux.microsoft.com/t.mbox.gz
> Checking for newer revisions
> Grabbing search results from lore.kernel.org Analyzing 5 messages in the
> thread Looking for additional code-review trailers on lore.kernel.org Checking
> attestation on all messages, may take a moment...
> Retrieving CI status, may take a moment...
> ---
>   ✓ [PATCH v4 1/4] RDMA/mana_ib: Introduce helpers to create and destroy
> mana queues
>     + Link:
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> kernel.org%2Fr%2F1712567646-5247-2-git-send-email-
> kotaranov%40linux.microsoft.com&data=05%7C02%7Ckotaranov%40micros
> oft.com%7C444cc0903207454a063d08dc57be9f9e%7C72f988bf86f141af9
> 1ab2d7cd011db47%7C1%7C0%7C638481724056487729%7CUnknown%7
> CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWw
> iLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=CaArcA4VvabbCg4pdJbMSLnIZ
> iMxDOREFsip%2FO3Iwg4%3D&reserved=0
>     + Signed-off-by: Leon Romanovsky <leon@kernel.org>
>   ✓ [PATCH v4 2/4] RDMA/mana_ib: Use struct mana_ib_queue for CQs
>     + Link:
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> kernel.org%2Fr%2F1712567646-5247-3-git-send-email-
> kotaranov%40linux.microsoft.com&data=05%7C02%7Ckotaranov%40micros
> oft.com%7C444cc0903207454a063d08dc57be9f9e%7C72f988bf86f141af9
> 1ab2d7cd011db47%7C1%7C0%7C638481724056495700%7CUnknown%7
> CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWw
> iLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=CF%2B2%2FxK%2FbqU4tylJ%2
> BL9k4XOjI6IqtthydKsZ%2Fkxz8nE%3D&reserved=0
>     + Signed-off-by: Leon Romanovsky <leon@kernel.org>
>   ✓ [PATCH v4 3/4] RDMA/mana_ib: Use struct mana_ib_queue for WQs
>     + Link:
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> kernel.org%2Fr%2F1712567646-5247-4-git-send-email-
> kotaranov%40linux.microsoft.com&data=05%7C02%7Ckotaranov%40micros
> oft.com%7C444cc0903207454a063d08dc57be9f9e%7C72f988bf86f141af9
> 1ab2d7cd011db47%7C1%7C0%7C638481724056500642%7CUnknown%7
> CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWw
> iLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=ZlIFtWT0fOdwiuJbqViki%2FTd
> CZaByBbzUy7g4D9HCHE%3D&reserved=0
>     + Signed-off-by: Leon Romanovsky <leon@kernel.org>
>   ✓ [PATCH v4 4/4] RDMA/mana_ib: Use struct mana_ib_queue for RAW QPs
>     + Link:
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> kernel.org%2Fr%2F1712567646-5247-5-git-send-email-
> kotaranov%40linux.microsoft.com&data=05%7C02%7Ckotaranov%40micros
> oft.com%7C444cc0903207454a063d08dc57be9f9e%7C72f988bf86f141af9
> 1ab2d7cd011db47%7C1%7C0%7C638481724056504252%7CUnknown%7
> CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWw
> iLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=bZTxG0kz5cosbtW4QMg2rSkei
> h08Q%2FBOrR%2BEHI4ROrY%3D&reserved=0
>     + Signed-off-by: Leon Romanovsky <leon@kernel.org>
>   ---
>   ✓ Signed: DKIM/linux.microsoft.com
> ---
> Total patches: 4
> ---
>  Base: using specified base-commit
> 96d9cbe2f2ff7abde021bac75eafaceabe9a51fa
> Applying: RDMA/mana_ib: Introduce helpers to create and destroy mana
> queues Patch failed at 0001 RDMA/mana_ib: Introduce helpers to create and
> destroy mana queues When you have resolved this problem, run "git am --
> continue".
> If you prefer to skip this patch, run "git am --skip" instead.
> To restore the original branch and stop patching, run "git am --abort".
> error: patch failed: drivers/infiniband/hw/mana/main.c:237
> error: drivers/infiniband/hw/mana/main.c: patch does not apply
> error: patch failed: drivers/infiniband/hw/mana/mana_ib.h:45
> error: drivers/infiniband/hw/mana/mana_ib.h: patch does not apply
> hint: Use 'git am --show-current-patch=diff' to see the failed patch Press any
> key to continue...

Re: [EXTERNAL] Re: [PATCH rdma-next v4 0/4] Define and use mana queues for CQs and WQs

[Leon Romanovsky]

On Mon, Apr 08, 2024 at 12:50:12PM +0000, Konstantin Taranov wrote:
> > From: Leon Romanovsky <leon@kernel.org>
> > On Mon, Apr 08, 2024 at 02:14:02AM -0700, Konstantin Taranov wrote:
> > > From: Konstantin Taranov <kotaranov@microsoft.com>
> > >
> > > This patch series aims to reduce code duplication by introducing a
> > > notion of mana ib queues and corresponding helpers to create and
> > > destroy them.
> > >
> > > v3->v4:
> > > * Removed debug prints in patches, as asked by Leon
> > >
> > > v2->v3:
> > > * [in 4/4] Do not define an additional struct for a raw qp
> > >
> > > v1->v2:
> > > * [in 1/4] Added a comment about the ignored return value
> > > * [in 2/4] Replaced RDMA:mana_ib to RDMA/mana_ib in the subject
> > > * [in 4/4] Renamed mana_ib_raw_qp to mana_ib_raw_sq
> > >
> > > Konstantin Taranov (4):
> > >   RDMA/mana_ib: Introduce helpers to create and destroy mana queues
> > >   RDMA/mana_ib: Use struct mana_ib_queue for CQs
> > >   RDMA/mana_ib: Use struct mana_ib_queue for WQs
> > >   RDMA/mana_ib: Use struct mana_ib_queue for RAW QPs
> > >
> > >  drivers/infiniband/hw/mana/cq.c      | 52 +++-------------
> > >  drivers/infiniband/hw/mana/main.c    | 39 ++++++++++++
> > >  drivers/infiniband/hw/mana/mana_ib.h | 26 ++++----
> > >  drivers/infiniband/hw/mana/qp.c      | 93 +++++++++-------------------
> > >  drivers/infiniband/hw/mana/wq.c      | 33 ++--------
> > >  5 files changed, 96 insertions(+), 147 deletions(-)
> > 
> > It doesn't apply.
> > 
> 
> I guess there was some mis-synchronisation between us.
> I see that you have already applied the patch 6 days ago:
> https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git/log/
> 
> I am sorry for sending a newer version after the patch has been applied.
> I have not checked this before sending.
> I can take care of useless debug prints in a future cleanup patch.

Please rebase your series, and resend.

Thanks

Re: [PATCH rdma-next v4 0/4] Define and use mana queues for CQs and WQs

[Konstantin Taranov]

> From: Leon Romanovsky <leon@kernel.org>
> On Mon, Apr 08, 2024 at 12:50:12PM +0000, Konstantin Taranov wrote:
> > > From: Leon Romanovsky <leon@kernel.org> On Mon, Apr 08, 2024 at
> > > 02:14:02AM -0700, Konstantin Taranov wrote:
> > > > From: Konstantin Taranov <kotaranov@microsoft.com>
> > > >
> > > > This patch series aims to reduce code duplication by introducing a
> > > > notion of mana ib queues and corresponding helpers to create and
> > > > destroy them.
> > > >
> > > > v3->v4:
> > > > * Removed debug prints in patches, as asked by Leon
> > > >
> > > > v2->v3:
> > > > * [in 4/4] Do not define an additional struct for a raw qp
> > > >
> > > > v1->v2:
> > > > * [in 1/4] Added a comment about the ignored return value
> > > > * [in 2/4] Replaced RDMA:mana_ib to RDMA/mana_ib in the subject
> > > > * [in 4/4] Renamed mana_ib_raw_qp to mana_ib_raw_sq
> > > >
> > > > Konstantin Taranov (4):
> > > >   RDMA/mana_ib: Introduce helpers to create and destroy mana queues
> > > >   RDMA/mana_ib: Use struct mana_ib_queue for CQs
> > > >   RDMA/mana_ib: Use struct mana_ib_queue for WQs
> > > >   RDMA/mana_ib: Use struct mana_ib_queue for RAW QPs
> > > >
> > > >  drivers/infiniband/hw/mana/cq.c      | 52 +++-------------
> > > >  drivers/infiniband/hw/mana/main.c    | 39 ++++++++++++
> > > >  drivers/infiniband/hw/mana/mana_ib.h | 26 ++++----
> > > >  drivers/infiniband/hw/mana/qp.c      | 93 +++++++++-------------------
> > > >  drivers/infiniband/hw/mana/wq.c      | 33 ++--------
> > > >  5 files changed, 96 insertions(+), 147 deletions(-)
> > >
> > > It doesn't apply.
> > >
> >
> > I guess there was some mis-synchronisation between us.
> > I see that you have already applied the patch 6 days ago:
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.
> >
> kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Frdma%2Frdma.git%2
> Flog%
> >
> 2F&data=05%7C02%7Ckotaranov%40microsoft.com%7C09ea6de381194295c
> 4ae08dc
> >
> 57cbe121%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63848178
> 04102717
> >
> 33%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz
> IiLCJBTiI
> >
> 6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=jwGGhmatHqdN4bW
> Xc%2FtyXtubD
> > ZxxCXpnyL26S5lEKd0%3D&reserved=0
> >
> > I am sorry for sending a newer version after the patch has been applied.
> > I have not checked this before sending.
> > I can take care of useless debug prints in a future cleanup patch.
> 
> Please rebase your series, and resend.

Sorry for a confusion. I mean you have already applied this patch series (v3) 6 days ago.
See commits:
46f5be7cd4bceb3a503c544b3dab7b75fe4bb96b
60a7ac0b8bec5df9764b7460ffee91fc981e8a31
688bac28e3dc9eb795ae8ea5aa40cb637e289faa
f10242b3da908dc9d4bfa040e6511a5b86522499

As a result, I cannot rebase. I could send a completely new patch that removes some debug prints.

I might look at a wrong branch. If so, please send the branch that does not have this patch series applied.

Thanks,
Konstantin

> 
> Thanks

Re: [PATCH rdma-next v4 0/4] Define and use mana queues for CQs and WQs

[Leon Romanovsky]

On Mon, Apr 08, 2024 at 01:47:45PM +0000, Konstantin Taranov wrote:
> > From: Leon Romanovsky <leon@kernel.org>
> > On Mon, Apr 08, 2024 at 12:50:12PM +0000, Konstantin Taranov wrote:
> > > > From: Leon Romanovsky <leon@kernel.org> On Mon, Apr 08, 2024 at
> > > > 02:14:02AM -0700, Konstantin Taranov wrote:
> > > > > From: Konstantin Taranov <kotaranov@microsoft.com>
> > > > >
> > > > > This patch series aims to reduce code duplication by introducing a
> > > > > notion of mana ib queues and corresponding helpers to create and
> > > > > destroy them.
> > > > >
> > > > > v3->v4:
> > > > > * Removed debug prints in patches, as asked by Leon
> > > > >
> > > > > v2->v3:
> > > > > * [in 4/4] Do not define an additional struct for a raw qp
> > > > >
> > > > > v1->v2:
> > > > > * [in 1/4] Added a comment about the ignored return value
> > > > > * [in 2/4] Replaced RDMA:mana_ib to RDMA/mana_ib in the subject
> > > > > * [in 4/4] Renamed mana_ib_raw_qp to mana_ib_raw_sq
> > > > >
> > > > > Konstantin Taranov (4):
> > > > >   RDMA/mana_ib: Introduce helpers to create and destroy mana queues
> > > > >   RDMA/mana_ib: Use struct mana_ib_queue for CQs
> > > > >   RDMA/mana_ib: Use struct mana_ib_queue for WQs
> > > > >   RDMA/mana_ib: Use struct mana_ib_queue for RAW QPs
> > > > >
> > > > >  drivers/infiniband/hw/mana/cq.c      | 52 +++-------------
> > > > >  drivers/infiniband/hw/mana/main.c    | 39 ++++++++++++
> > > > >  drivers/infiniband/hw/mana/mana_ib.h | 26 ++++----
> > > > >  drivers/infiniband/hw/mana/qp.c      | 93 +++++++++-------------------
> > > > >  drivers/infiniband/hw/mana/wq.c      | 33 ++--------
> > > > >  5 files changed, 96 insertions(+), 147 deletions(-)
> > > >
> > > > It doesn't apply.
> > > >
> > >
> > > I guess there was some mis-synchronisation between us.
> > > I see that you have already applied the patch 6 days ago:
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.
> > >
> > kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Frdma%2Frdma.git%2
> > Flog%
> > >
> > 2F&data=05%7C02%7Ckotaranov%40microsoft.com%7C09ea6de381194295c
> > 4ae08dc
> > >
> > 57cbe121%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63848178
> > 04102717
> > >
> > 33%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz
> > IiLCJBTiI
> > >
> > 6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=jwGGhmatHqdN4bW
> > Xc%2FtyXtubD
> > > ZxxCXpnyL26S5lEKd0%3D&reserved=0
> > >
> > > I am sorry for sending a newer version after the patch has been applied.
> > > I have not checked this before sending.
> > > I can take care of useless debug prints in a future cleanup patch.
> > 
> > Please rebase your series, and resend.
> 
> Sorry for a confusion. I mean you have already applied this patch series (v3) 6 days ago.
> See commits:
> 46f5be7cd4bceb3a503c544b3dab7b75fe4bb96b
> 60a7ac0b8bec5df9764b7460ffee91fc981e8a31
> 688bac28e3dc9eb795ae8ea5aa40cb637e289faa
> f10242b3da908dc9d4bfa040e6511a5b86522499
> 
> As a result, I cannot rebase. I could send a completely new patch that removes some debug prints.

Yes, sorry for not being clear. Please send a cleanup patch.

Thanks