[PATCH v3 0/5] Reduce RDMA RW API SGE limit

[PATCH v3 0/5] Reduce RDMA RW API SGE limit

[Bart Van Assche]

Hello Doug,

The five patches in this series modify the RDMA RW API slightly. This is 
needed to avoid that the SRP and iSER target drivers submit RDMA 
requests with an SGE list that exceeds the queue pair limits. The 
ib_srpt changes in this series have been tested but the ib_isert changes 
not yet.

The changes compared to v2 of this patch series are:
* For RDMA READs, limit SGE back to dev->attrs.max_sge_rd for iWARP.

Changes compared to v1:
* max_send_sge is now stored in struct ib_qp. This greatly simplifies
   this patch series.
* An unneeded initialization that I had added to rdma_rw_init_one_mr()
   has been left out again.
* Corrected "Fixes" tag in the patch description where needed.

The individual patches in this series are:

0001-IB-core-Make-rdma_rw_ctx_init-initialize-all-used-fi.patch
0002-IB-core-RDMA-RW-API-Do-not-exceed-QP-SGE-send-limit.patch
0003-IB-srpt-Limit-the-number-of-SG-elements-per-work-req.patch
0004-IB-srpt-Simplify-srpt_queue_response.patch
0005-IB-isert-Remove-an-unused-member-variable.patch

Thanks,

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v3 1/5] IB/core: Make rdma_rw_ctx_init() initialize all used fields

[Bart Van Assche]

Some but not all callers of rdma_rw_ctx_init() zero-initialize
struct rdma_rw_ctx. Hence make rdma_rw_ctx_init() initialize all
work request fields that will be read by ib_post_send().

Fixes: a060b5629ab0 ("IB/core: generic RDMA READ/WRITE API")
Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> #v4.7+
Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
Cc: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
Cc: Parav Pandit <pandit.parav-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Cc: Nicholas Bellinger <nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org>
Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 drivers/infiniband/core/rw.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c
index 1eb9b12..1ad2baa 100644
--- a/drivers/infiniband/core/rw.c
+++ b/drivers/infiniband/core/rw.c
@@ -71,6 +71,7 @@ static inline u32 rdma_rw_fr_page_list_len(struct ib_device *dev)
 	return min_t(u32, dev->attrs.max_fast_reg_page_list_len, 256);
 }
 
+/* Caller must have zero-initialized *reg. */
 static int rdma_rw_init_one_mr(struct ib_qp *qp, u8 port_num,
 		struct rdma_rw_reg_ctx *reg, struct scatterlist *sg,
 		u32 sg_cnt, u32 offset)
@@ -114,6 +115,7 @@ static int rdma_rw_init_mr_wrs(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
 		u8 port_num, struct scatterlist *sg, u32 sg_cnt, u32 offset,
 		u64 remote_addr, u32 rkey, enum dma_data_direction dir)
 {
+	struct rdma_rw_reg_ctx *prev = NULL;
 	u32 pages_per_mr = rdma_rw_fr_page_list_len(qp->pd->device);
 	int i, j, ret = 0, count = 0;
 
@@ -125,7 +127,6 @@ static int rdma_rw_init_mr_wrs(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
 	}
 
 	for (i = 0; i < ctx->nr_ops; i++) {
-		struct rdma_rw_reg_ctx *prev = i ? &ctx->reg[i - 1] : NULL;
 		struct rdma_rw_reg_ctx *reg = &ctx->reg[i];
 		u32 nents = min(sg_cnt, pages_per_mr);
 
@@ -162,9 +163,13 @@ static int rdma_rw_init_mr_wrs(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
 		sg_cnt -= nents;
 		for (j = 0; j < nents; j++)
 			sg = sg_next(sg);
+		prev = reg;
 		offset = 0;
 	}
 
+	if (prev)
+		prev->wr.wr.next = NULL;
+
 	ctx->type = RDMA_RW_MR;
 	return count;
 
@@ -205,11 +210,10 @@ static int rdma_rw_init_map_wrs(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
 			rdma_wr->wr.opcode = IB_WR_RDMA_READ;
 		rdma_wr->remote_addr = remote_addr + total_len;
 		rdma_wr->rkey = rkey;
+		rdma_wr->wr.num_sge = nr_sge;
 		rdma_wr->wr.sg_list = sge;
 
 		for (j = 0; j < nr_sge; j++, sg = sg_next(sg)) {
-			rdma_wr->wr.num_sge++;
-
 			sge->addr = ib_sg_dma_address(dev, sg) + offset;
 			sge->length = ib_sg_dma_len(dev, sg) - offset;
 			sge->lkey = qp->pd->local_dma_lkey;
@@ -220,8 +224,8 @@ static int rdma_rw_init_map_wrs(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
 			offset = 0;
 		}
 
-		if (i + 1 < ctx->nr_ops)
-			rdma_wr->wr.next = &ctx->map.wrs[i + 1].wr;
+		rdma_wr->wr.next = i + 1 < ctx->nr_ops ?
+			&ctx->map.wrs[i + 1].wr : NULL;
 	}
 
 	ctx->type = RDMA_RW_MULTI_WR;
-- 
2.9.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 1/5] IB/core: Make rdma_rw_ctx_init() initialize all used fields

[Christoph Hellwig]

Looks fine,

Reviewed-by: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v3 2/5] IB/core, RDMA RW API: Do not exceed QP SGE send limit

[Bart Van Assche]

For IB and RoCE, the SGE limit for a queue pair is typically lower
than what is defined by the HCA limits. For iWARP, the RDMA READ SGE
limit is defined by dev->attrs.max_sge_rd. Modify rdma_rw_max_sge()
accordingly.

Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> #v4.7+
Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
Cc: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
Cc: Parav Pandit <pandit.parav-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Cc: Nicholas Bellinger <nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org>
Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 drivers/infiniband/core/rw.c    | 8 ++++----
 drivers/infiniband/core/verbs.c | 2 ++
 include/rdma/ib_verbs.h         | 1 +
 3 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c
index 1ad2baa..6ecdad6 100644
--- a/drivers/infiniband/core/rw.c
+++ b/drivers/infiniband/core/rw.c
@@ -58,11 +58,11 @@ static inline bool rdma_rw_io_needs_mr(struct ib_device *dev, u8 port_num,
 	return false;
 }
 
-static inline u32 rdma_rw_max_sge(struct ib_device *dev,
+static inline u32 rdma_rw_max_sge(struct ib_device *dev, struct ib_qp *qp,
 		enum dma_data_direction dir)
 {
-	return dir == DMA_TO_DEVICE ?
-		dev->attrs.max_sge : dev->attrs.max_sge_rd;
+	return dir == DMA_TO_DEVICE ? qp->max_send_sge :
+		min_t(u32, qp->max_send_sge, dev->attrs.max_sge_rd);
 }
 
 static inline u32 rdma_rw_fr_page_list_len(struct ib_device *dev)
@@ -186,7 +186,7 @@ static int rdma_rw_init_map_wrs(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
 		u64 remote_addr, u32 rkey, enum dma_data_direction dir)
 {
 	struct ib_device *dev = qp->pd->device;
-	u32 max_sge = rdma_rw_max_sge(dev, dir);
+	u32 max_sge = rdma_rw_max_sge(dev, qp, dir);
 	struct ib_sge *sge;
 	u32 total_len = 0, i, j;
 
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index 6298f54..c7f840e 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -814,6 +814,8 @@ struct ib_qp *ib_create_qp(struct ib_pd *pd,
 		}
 	}
 
+	qp->max_send_sge = qp_init_attr->cap.max_send_sge;
+
 	return qp;
 }
 EXPORT_SYMBOL(ib_create_qp);
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 7e440d4..c44dbf6 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -1449,6 +1449,7 @@ struct ib_qp {
 	void                  (*event_handler)(struct ib_event *, void *);
 	void		       *qp_context;
 	u32			qp_num;
+	u32			max_send_sge;
 	enum ib_qp_type		qp_type;
 };
 
-- 
2.9.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 2/5] IB/core, RDMA RW API: Do not exceed QP SGE send limit

[Christoph Hellwig]

On Tue, Jul 19, 2016 at 09:22:03AM -0700, Bart Van Assche wrote:
> For IB and RoCE, the SGE limit for a queue pair is typically lower
> than what is defined by the HCA limits. For iWARP, the RDMA READ SGE
> limit is defined by dev->attrs.max_sge_rd. Modify rdma_rw_max_sge()
> accordingly.

It's not just iWarp - Melannox IB HCAs also have a lower RDMA READ than
WRITE limit.

> -static inline u32 rdma_rw_max_sge(struct ib_device *dev,
> +static inline u32 rdma_rw_max_sge(struct ib_device *dev, struct ib_qp *qp,
>  		enum dma_data_direction dir)
>  {
> +	return dir == DMA_TO_DEVICE ? qp->max_send_sge :
> +		min_t(u32, qp->max_send_sge, dev->attrs.max_sge_rd);
>  }

I'm a bit worried about this implicit and barely documented assumption
that we'll always have a lower RDMA READ than WRITE / SEND limit.

Can we fine tune the patch to add max_write_sge and max_read_sge
to struct ib_qp instead?  That also makes clear they apply to RDMA READ
and WRITE only.  Please also add a comment to ib_verbs.h to document
the exact semantics of this field(s) while you're at it, e.g. that
they only apply to users of the RDMA R/W API.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v3 3/5] IB/srpt: Limit the number of SG elements per work request

[Bart Van Assche]

Limit the number of SG elements per work request to what the HCA
and the queue pair support.

Fixes: 34693573fde0 ("IB/srpt: Reduce QP buffer size")
Reported-by: Parav Pandit <pandit.parav-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> #v4.7+
Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
Cc: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
Cc: Parav Pandit <pandit.parav-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Cc: Nicholas Bellinger <nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org>
Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 drivers/infiniband/ulp/srpt/ib_srpt.c | 3 ++-
 drivers/infiniband/ulp/srpt/ib_srpt.h | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index 4a41556..9a3b954 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -1601,6 +1601,7 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch)
 	struct ib_qp_init_attr *qp_init;
 	struct srpt_port *sport = ch->sport;
 	struct srpt_device *sdev = sport->sdev;
+	const struct ib_device_attr *attrs = &sdev->device->attrs;
 	u32 srp_sq_size = sport->port_attrib.srp_sq_size;
 	int ret;
 
@@ -1638,7 +1639,7 @@ retry:
 	 */
 	qp_init->cap.max_send_wr = srp_sq_size / 2;
 	qp_init->cap.max_rdma_ctxs = srp_sq_size / 2;
-	qp_init->cap.max_send_sge = SRPT_DEF_SG_PER_WQE;
+	qp_init->cap.max_send_sge = min(attrs->max_sge, SRPT_MAX_SG_PER_WQE);
 	qp_init->port_num = ch->sport->port;
 
 	ch->qp = ib_create_qp(sdev->pd, qp_init);
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.h b/drivers/infiniband/ulp/srpt/ib_srpt.h
index 3890304..d444f8d 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.h
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.h
@@ -106,7 +106,7 @@ enum {
 	SRP_LOGIN_RSP_MULTICHAN_MAINTAINED = 0x2,
 
 	SRPT_DEF_SG_TABLESIZE = 128,
-	SRPT_DEF_SG_PER_WQE = 16,
+	SRPT_MAX_SG_PER_WQE = 16,
 
 	MIN_SRPT_SQ_SIZE = 16,
 	DEF_SRPT_SQ_SIZE = 4096,
-- 
2.9.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 3/5] IB/srpt: Limit the number of SG elements per work request

[Christoph Hellwig]

On Tue, Jul 19, 2016 at 09:22:29AM -0700, Bart Van Assche wrote:
> Limit the number of SG elements per work request to what the HCA
> and the queue pair support.

Can you add a comment on why this limit was chosen while you're at it?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v3 4/5] IB/srpt: Simplify srpt_queue_response()

[Bart Van Assche]

Initialize first_wr to &send_wr. This allows to remove a ternary
operator and an else branch. This patch does not change the behavior
of srpt_queue_response().

Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Reviewed-by: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
Cc: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
Cc: Parav Pandit <pandit.parav-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Cc: Nicholas Bellinger <nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org>
Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 drivers/infiniband/ulp/srpt/ib_srpt.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index 9a3b954..dfa23b0 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -2262,7 +2262,7 @@ static void srpt_queue_response(struct se_cmd *cmd)
 		container_of(cmd, struct srpt_send_ioctx, cmd);
 	struct srpt_rdma_ch *ch = ioctx->ch;
 	struct srpt_device *sdev = ch->sport->sdev;
-	struct ib_send_wr send_wr, *first_wr = NULL, *bad_wr;
+	struct ib_send_wr send_wr, *first_wr = &send_wr, *bad_wr;
 	struct ib_sge sge;
 	enum srpt_command_state state;
 	unsigned long flags;
@@ -2303,11 +2303,8 @@ static void srpt_queue_response(struct se_cmd *cmd)
 			struct srpt_rw_ctx *ctx = &ioctx->rw_ctxs[i];
 
 			first_wr = rdma_rw_ctx_wrs(&ctx->rw, ch->qp,
-					ch->sport->port, NULL,
-					first_wr ? first_wr : &send_wr);
+					ch->sport->port, NULL, first_wr);
 		}
-	} else {
-		first_wr = &send_wr;
 	}
 
 	if (state != SRPT_STATE_MGMT)
-- 
2.9.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v3 5/5] IB/isert: Remove an unused member variable

[Bart Van Assche]

Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
Cc: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
Cc: Parav Pandit <pandit.parav-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Cc: Nicholas Bellinger <nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org>
Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 drivers/infiniband/ulp/isert/ib_isert.c | 2 --
 drivers/infiniband/ulp/isert/ib_isert.h | 1 -
 2 files changed, 3 deletions(-)

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index a990c04..ba6be06 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -137,8 +137,6 @@ isert_create_qp(struct isert_conn *isert_conn,
 	attr.cap.max_recv_wr = ISERT_QP_MAX_RECV_DTOS + 1;
 	attr.cap.max_rdma_ctxs = ISCSI_DEF_XMIT_CMDS_MAX;
 	attr.cap.max_send_sge = device->ib_device->attrs.max_sge;
-	isert_conn->max_sge = min(device->ib_device->attrs.max_sge,
-				  device->ib_device->attrs.max_sge_rd);
 	attr.cap.max_recv_sge = 1;
 	attr.sq_sig_type = IB_SIGNAL_REQ_WR;
 	attr.qp_type = IB_QPT_RC;
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
index e512ba9..fc791ef 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -138,7 +138,6 @@ struct isert_conn {
 	u32			responder_resources;
 	u32			initiator_depth;
 	bool			pi_support;
-	u32			max_sge;
 	struct iser_rx_desc	*login_req_buf;
 	char			*login_rsp_buf;
 	u64			login_req_dma;
-- 
2.9.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 5/5] IB/isert: Remove an unused member variable

[Christoph Hellwig]

Looks fine,

Reviewed-by: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 0/5] Reduce RDMA RW API SGE limit

[Laurence Oberman]

----- Original Message -----
> From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: "Christoph Hellwig" <hch-jcswGhMUV9g@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Steve Wise" <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>,
> "Parav Pandit" <pandit.parav-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Nicholas A. Bellinger"
> <nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Tuesday, July 19, 2016 12:20:45 PM
> Subject: [PATCH v3 0/5] Reduce RDMA RW API SGE limit
> 
> Hello Doug,
> 
> The five patches in this series modify the RDMA RW API slightly. This is
> needed to avoid that the SRP and iSER target drivers submit RDMA
> requests with an SGE list that exceeds the queue pair limits. The
> ib_srpt changes in this series have been tested but the ib_isert changes
> not yet.
> 
> The changes compared to v2 of this patch series are:
> * For RDMA READs, limit SGE back to dev->attrs.max_sge_rd for iWARP.
> 
> Changes compared to v1:
> * max_send_sge is now stored in struct ib_qp. This greatly simplifies
>    this patch series.
> * An unneeded initialization that I had added to rdma_rw_init_one_mr()
>    has been left out again.
> * Corrected "Fixes" tag in the patch description where needed.
> 
> The individual patches in this series are:
> 
> 0001-IB-core-Make-rdma_rw_ctx_init-initialize-all-used-fi.patch
> 0002-IB-core-RDMA-RW-API-Do-not-exceed-QP-SGE-send-limit.patch
> 0003-IB-srpt-Limit-the-number-of-SG-elements-per-work-req.patch
> 0004-IB-srpt-Simplify-srpt_queue_response.patch
> 0005-IB-isert-Remove-an-unused-member-variable.patch
> 
> Thanks,
> 
> Bart.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
Will pull and test these ASAP.

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v3 0/5] Reduce RDMA RW API SGE limit

[Steve Wise]

> > Hello Doug,
> >
> > The five patches in this series modify the RDMA RW API slightly. This is
> > needed to avoid that the SRP and iSER target drivers submit RDMA
> > requests with an SGE list that exceeds the queue pair limits. The
> > ib_srpt changes in this series have been tested but the ib_isert changes
> > not yet.
> >
> > The changes compared to v2 of this patch series are:
> > * For RDMA READs, limit SGE back to dev->attrs.max_sge_rd for iWARP.
> >
> > Changes compared to v1:
> > * max_send_sge is now stored in struct ib_qp. This greatly simplifies
> >    this patch series.
> > * An unneeded initialization that I had added to rdma_rw_init_one_mr()
> >    has been left out again.
> > * Corrected "Fixes" tag in the patch description where needed.
> >
> > The individual patches in this series are:
> >
> > 0001-IB-core-Make-rdma_rw_ctx_init-initialize-all-used-fi.patch
> > 0002-IB-core-RDMA-RW-API-Do-not-exceed-QP-SGE-send-limit.patch
> > 0003-IB-srpt-Limit-the-number-of-SG-elements-per-work-req.patch
> > 0004-IB-srpt-Simplify-srpt_queue_response.patch
> > 0005-IB-isert-Remove-an-unused-member-variable.patch
> >
> > Thanks,
> >
> > Bart.


> Will pull and test these ASAP.
> 

Ditto

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 0/5] Reduce RDMA RW API SGE limit

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Christoph Hellwig" <hch-jcswGhMUV9g@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
> "Steve Wise" <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>, "Parav Pandit" <pandit.parav-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, "Nicholas A. Bellinger"
> <nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Tuesday, July 19, 2016 1:14:45 PM
> Subject: Re: [PATCH v3 0/5] Reduce RDMA RW API SGE limit
> 
> 
> 
> ----- Original Message -----
> > From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > To: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > Cc: "Christoph Hellwig" <hch-jcswGhMUV9g@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
> > "Steve Wise" <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>,
> > "Parav Pandit" <pandit.parav-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, "Laurence Oberman"
> > <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Nicholas A. Bellinger"
> > <nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Sent: Tuesday, July 19, 2016 12:20:45 PM
> > Subject: [PATCH v3 0/5] Reduce RDMA RW API SGE limit
> > 
> > Hello Doug,
> > 
> > The five patches in this series modify the RDMA RW API slightly. This is
> > needed to avoid that the SRP and iSER target drivers submit RDMA
> > requests with an SGE list that exceeds the queue pair limits. The
> > ib_srpt changes in this series have been tested but the ib_isert changes
> > not yet.
> > 
> > The changes compared to v2 of this patch series are:
> > * For RDMA READs, limit SGE back to dev->attrs.max_sge_rd for iWARP.
> > 
> > Changes compared to v1:
> > * max_send_sge is now stored in struct ib_qp. This greatly simplifies
> >    this patch series.
> > * An unneeded initialization that I had added to rdma_rw_init_one_mr()
> >    has been left out again.
> > * Corrected "Fixes" tag in the patch description where needed.
> > 
> > The individual patches in this series are:
> > 
> > 0001-IB-core-Make-rdma_rw_ctx_init-initialize-all-used-fi.patch
> > 0002-IB-core-RDMA-RW-API-Do-not-exceed-QP-SGE-send-limit.patch
> > 0003-IB-srpt-Limit-the-number-of-SG-elements-per-work-req.patch
> > 0004-IB-srpt-Simplify-srpt_queue_response.patch
> > 0005-IB-isert-Remove-an-unused-member-variable.patch
> > 
> > Thanks,
> > 
> > Bart.
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> Will pull and test these ASAP.
> 
> Thanks
> Laurence
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Replying to my own email:

Hello Bart

Testing SRP for these 5 patches here and I think the issues I am seeing may be changes in the latest upstream next kernel and not your code causing this.
'
I applied these to 4.7.0-rc7 and I am seeing some behavior I have not figured out yet.
I can find and map all LUNS via SRP but there are some differences in multipath.
I am still working on figuring some things out because I will likely have to apply these to an earlier base kernel to make sure.

I am not sure yet if something in the  4.7.0-rc7 base changed ALUA behavior, likely that is the case.
I have to go back and check.

On an older kernel with recent patches from you and Leon everything looks fine and is working fine.

4.7.0-rc1.bartleon+ 

360001ff0b035d000000000098d79000a dm-11 DDN     ,SFA14K          
size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=90 status=active
| `- 1:0:0:9  sdt  65:48  active ready running
`-+- policy='round-robin 0' prio=50 status=enabled
  `- 2:0:0:9  sdv  65:80  active ready running
360001ff0b035d000000000078d770008 dm-9 DDN     ,SFA14K          
size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=90 status=active
| `- 2:0:0:7  sds  65:32  active ready running
`-+- policy='round-robin 0' prio=50 status=enabled
  `- 1:0:0:7  sdp  8:240  active ready running


Looks fine here:

# time mkfs -t xfs -f  /dev/mapper/360001ff0b035d000000000078d770008 
meta-data=/dev/mapper/360001ff0b035d000000000078d770008 isize=256    agcount=33, agsize=243265536 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=7784628224, imaxpct=5
         =                       sunit=4096   swidth=4096 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Finishes quickly, as usual.

real	0m1.764s
user	0m0.006s
sys	0m0.168s


[root@jumpclient ~]# sg_rtpg -d /dev/sds
Report target port groups:
  target port group id : 0x0 , Pref=0
    target port group asymmetric access state : 0x01 (active/non optimized)
    T_SUP : 0, O_SUP : 0, LBD_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
    status code : 0x00 (no status available)
    vendor unique status : 0x00
    target port count : 02
    Relative target port ids:
      0x01
      0x02
  target port group id : 0x1 , Pref=0
    target port group asymmetric access state : 0x01 (active/non optimized)
    T_SUP : 0, O_SUP : 0, LBD_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
    status code : 0x00 (no status available)
    vendor unique status : 0x00
    target port count : 02
    Relative target port ids:
      0x03
      0x04
  target port group id : 0x2 , Pref=1
    target port group asymmetric access state : 0x00 (active/optimized)
    T_SUP : 0, O_SUP : 0, LBD_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
    status code : 0x00 (no status available)
    vendor unique status : 0x00
    target port count : 02
    Relative target port ids:
      0x05
      0x06
  target port group id : 0x3 , Pref=1
    target port group asymmetric access state : 0x01 (active/non optimized)
    T_SUP : 0, O_SUP : 0, LBD_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
    status code : 0x00 (no status available)
    vendor unique status : 0x00
    target port count : 02
    Relative target port ids:
      0x07
      0x08


On this new kernel,4.7.0-rc7.bart5+

I see this behaviour:
ALUA is failing to map properly against the SFA14K,and we are coming up active/active and not active optimized/active non-optimized.

Creating an xfs on the LUN take a long time

[root@jumptest1 ~]# time mkfs -t xfs -f  /dev/mapper/360001ff0b035d000000000078d770008 
meta-data=/dev/mapper/360001ff0b035d000000000078d770008 isize=512    agcount=33, agsize=243265536 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0
data     =                       bsize=4096   blocks=7784628224, imaxpct=5
         =                       sunit=4096   swidth=4096 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

real	10m34.395s
user	0m0.010s
sys	0m0.222s


[  321.500021] sd 2:0:0:29: alua: port group 01 state N non-preferred supports tolUsNA

360001ff0b035d000000000098d79000a dm-14 DDN     ,SFA14K          
size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=10 status=active
  |- 1:0:0:9  sdl  8:176  active ready running
  `- 2:0:0:9  sdba 67:64  active ready running
360001ff0b035d000000000078d770008 dm-12 DDN     ,SFA14K          
size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=10 status=active
  |- 1:0:0:7  sdj  8:144  active ready running
  `- 2:0:0:7  sday 67:32  active ready running

Checking with sg_rtpg looks right however ???

Report target port groups:
  target port group id : 0x0 , Pref=0
    target port group asymmetric access state : 0x01 (active/non optimized)
    T_SUP : 0, O_SUP : 0, LBD_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
    status code : 0x00 (no status available)
    vendor unique status : 0x00
    target port count : 02
    Relative target port ids:
      0x01
      0x02
  target port group id : 0x1 , Pref=0
    target port group asymmetric access state : 0x01 (active/non optimized)
    T_SUP : 0, O_SUP : 0, LBD_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
    status code : 0x00 (no status available)
    vendor unique status : 0x00
    target port count : 02
    Relative target port ids:
      0x03
      0x04
  target port group id : 0x2 , Pref=1
    target port group asymmetric access state : 0x00 (active/optimized)
    T_SUP : 0, O_SUP : 0, LBD_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
    status code : 0x00 (no status available)
    vendor unique status : 0x00
    target port count : 02
    Relative target port ids:
      0x05
      0x06
  target port group id : 0x3 , Pref=1
    target port group asymmetric access state : 0x01 (active/non optimized)
    T_SUP : 0, O_SUP : 0, LBD_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
    status code : 0x00 (no status available)
    vendor unique status : 0x00
    target port count : 02
    Relative target port ids:
      0x07
      0x08

Back when I have more.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 0/5] Reduce RDMA RW API SGE limit

[Bart Van Assche]

On 07/20/2016 07:43 AM, Laurence Oberman wrote:
 > [ ... ]
> I am not sure yet if something in the  4.7.0-rc7 base changed ALUA behavior, likely that is the case.
> I have to go back and check.
>
> On an older kernel with recent patches from you and Leon everything looks fine and is working fine.
>
> 4.7.0-rc1.bartleon+
>
> 360001ff0b035d000000000098d79000a dm-11 DDN     ,SFA14K
> size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
> |-+- policy='round-robin 0' prio=90 status=active
> | `- 1:0:0:9  sdt  65:48  active ready running
> `-+- policy='round-robin 0' prio=50 status=enabled
>   `- 2:0:0:9  sdv  65:80  active ready running
> 360001ff0b035d000000000078d770008 dm-9 DDN     ,SFA14K
> size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
> |-+- policy='round-robin 0' prio=90 status=active
> | `- 2:0:0:7  sds  65:32  active ready running
> `-+- policy='round-robin 0' prio=50 status=enabled
>   `- 1:0:0:7  sdp  8:240  active ready running

Hello Laurence,

Can you check your /etc/multipath.conf? In the multipath -ll output 
"hwhandler" should be something like hwhandler='1 alua' instead of '0' 
when communicating with an array that supports ALUA.

Thanks,

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 0/5] Reduce RDMA RW API SGE limit

[Laurence Oberman]

----- Original Message -----
> From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Christoph Hellwig" <hch-jcswGhMUV9g@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
> "Steve Wise" <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>, "Parav Pandit" <pandit.parav-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, "Nicholas A. Bellinger"
> <nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Wednesday, July 20, 2016 11:07:34 AM
> Subject: Re: [PATCH v3 0/5] Reduce RDMA RW API SGE limit
> 
> On 07/20/2016 07:43 AM, Laurence Oberman wrote:
>  > [ ... ]
> > I am not sure yet if something in the  4.7.0-rc7 base changed ALUA
> > behavior, likely that is the case.
> > I have to go back and check.
> >
> > On an older kernel with recent patches from you and Leon everything looks
> > fine and is working fine.
> >
> > 4.7.0-rc1.bartleon+
> >
> > 360001ff0b035d000000000098d79000a dm-11 DDN     ,SFA14K
> > size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
> > |-+- policy='round-robin 0' prio=90 status=active
> > | `- 1:0:0:9  sdt  65:48  active ready running
> > `-+- policy='round-robin 0' prio=50 status=enabled
> >   `- 2:0:0:9  sdv  65:80  active ready running
> > 360001ff0b035d000000000078d770008 dm-9 DDN     ,SFA14K
> > size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
> > |-+- policy='round-robin 0' prio=90 status=active
> > | `- 2:0:0:7  sds  65:32  active ready running
> > `-+- policy='round-robin 0' prio=50 status=enabled
> >   `- 1:0:0:7  sdp  8:240  active ready running
> 
> Hello Laurence,
> 
> Can you check your /etc/multipath.conf? In the multipath -ll output
> "hwhandler" should be something like hwhandler='1 alua' instead of '0'
> when communicating with an array that supports ALUA.
> 
> Thanks,
> 
> Bart.
> 

Hi Bart

I have not changed my multipath.conf and the older kernel works fine.
Its been the same configuration all along.

I have had this all along (came from the vendor) and its also on the other server booting the older kernel.

       device {
        vendor                  "DDN"
        product                 "SFA14K"
        path_grouping_policy    group_by_prio
        prio                    alua
        path_selector           "round-robin 0"
        path_checker            tur
        failback                2
        rr_weight               uniform
        no_path_retry           12
        dev_loss_tmo            10
        fast_io_fail_tmo        5
    }

Older kernel is coming up ALUA

360001ff0b035d000000000078d770008 dm-9 DDN     ,SFA14K          
size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=90 status=active            Active/optimized
| `- 2:0:0:7  sds  65:32  active ready running
`-+- policy='round-robin 0' prio=50 status=enabled          Active/non-optimized
  `- 1:0:0:7  sdp  8:240  active ready running

New kernel with latest patches

360001ff0b035d000000000078d770008 dm-12 DDN     ,SFA14K          
size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=10 status=active
  |- 1:0:0:7  sdj  8:144  active ready running
  `- 2:0:0:7  sday 67:32  active ready running

I dont think its your recent 5 patches, but I will test the latest kernel without them to validate that its not your changes.
Let me go ahead and test the generic 4.7.0-rc7 and confirm.

Thanks!!
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 0/5] Reduce RDMA RW API SGE limit

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Christoph Hellwig" <hch-jcswGhMUV9g@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
> "Steve Wise" <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>, "Parav Pandit" <pandit.parav-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, "Nicholas A. Bellinger"
> <nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Wednesday, July 20, 2016 11:19:36 AM
> Subject: Re: [PATCH v3 0/5] Reduce RDMA RW API SGE limit
> 
> 
> 
> ----- Original Message -----
> > From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Christoph Hellwig" <hch-jcswGhMUV9g@public.gmane.org>,
> > "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
> > "Steve Wise" <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>, "Parav Pandit"
> > <pandit.parav-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, "Nicholas A. Bellinger"
> > <nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Sent: Wednesday, July 20, 2016 11:07:34 AM
> > Subject: Re: [PATCH v3 0/5] Reduce RDMA RW API SGE limit
> > 
> > On 07/20/2016 07:43 AM, Laurence Oberman wrote:
> >  > [ ... ]
> > > I am not sure yet if something in the  4.7.0-rc7 base changed ALUA
> > > behavior, likely that is the case.
> > > I have to go back and check.
> > >
> > > On an older kernel with recent patches from you and Leon everything looks
> > > fine and is working fine.
> > >
> > > 4.7.0-rc1.bartleon+
> > >
> > > 360001ff0b035d000000000098d79000a dm-11 DDN     ,SFA14K
> > > size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
> > > |-+- policy='round-robin 0' prio=90 status=active
> > > | `- 1:0:0:9  sdt  65:48  active ready running
> > > `-+- policy='round-robin 0' prio=50 status=enabled
> > >   `- 2:0:0:9  sdv  65:80  active ready running
> > > 360001ff0b035d000000000078d770008 dm-9 DDN     ,SFA14K
> > > size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
> > > |-+- policy='round-robin 0' prio=90 status=active
> > > | `- 2:0:0:7  sds  65:32  active ready running
> > > `-+- policy='round-robin 0' prio=50 status=enabled
> > >   `- 1:0:0:7  sdp  8:240  active ready running
> > 
> > Hello Laurence,
> > 
> > Can you check your /etc/multipath.conf? In the multipath -ll output
> > "hwhandler" should be something like hwhandler='1 alua' instead of '0'
> > when communicating with an array that supports ALUA.
> > 
> > Thanks,
> > 
> > Bart.
> > 
> 
> Hi Bart
> 
> I have not changed my multipath.conf and the older kernel works fine.
> Its been the same configuration all along.
> 
> I have had this all along (came from the vendor) and its also on the other
> server booting the older kernel.
> 
>        device {
>         vendor                  "DDN"
>         product                 "SFA14K"
>         path_grouping_policy    group_by_prio
>         prio                    alua
>         path_selector           "round-robin 0"
>         path_checker            tur
>         failback                2
>         rr_weight               uniform
>         no_path_retry           12
>         dev_loss_tmo            10
>         fast_io_fail_tmo        5
>     }
> 
> Older kernel is coming up ALUA
> 
> 360001ff0b035d000000000078d770008 dm-9 DDN     ,SFA14K
> size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
> |-+- policy='round-robin 0' prio=90 status=active            Active/optimized
> | `- 2:0:0:7  sds  65:32  active ready running
> `-+- policy='round-robin 0' prio=50 status=enabled
> Active/non-optimized
>   `- 1:0:0:7  sdp  8:240  active ready running
> 
> New kernel with latest patches
> 
> 360001ff0b035d000000000078d770008 dm-12 DDN     ,SFA14K
> size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
> `-+- policy='round-robin 0' prio=10 status=active
>   |- 1:0:0:7  sdj  8:144  active ready running
>   `- 2:0:0:7  sday 67:32  active ready running
> 
> I dont think its your recent 5 patches, but I will test the latest kernel
> without them to validate that its not your changes.
> Let me go ahead and test the generic 4.7.0-rc7 and confirm.
> 
> Thanks!!
> Laurence
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Replying to my own email.

Hi Bart

OK, seems something on the connectivity to this other server has changed without my knowledge.
Booting the older kernel on this node showing the issue also sees this same behavior so something has changed on the back-end here for this server.
Its not the kernel or your patches.
I need to go into the lab and check what has has changed but that's for another day.

So I installed the new kernel with latest 5 patches on the other server and its passed all my tests.
Comes up as normal and passes all SRP tests I know need to be performed.

Looks better :)

360001ff0b035d000000000078d770008 dm-9 DDN     ,SFA14K          
size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=90 status=active
| `- 2:0:0:7  sdq  65:0   active ready running
`-+- policy='round-robin 0' prio=50 status=enabled
  `- 1:0:0:7  sdr  65:16  active ready running


XFS tests are fast

# time mkfs -t xfs -f  /dev/mapper/360001ff0b035d000000000078d770008
meta-data=/dev/mapper/360001ff0b035d000000000078d770008 isize=256    agcount=33, agsize=243265536 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=7784628224, imaxpct=5
         =                       sunit=4096   swidth=4096 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

real	0m1.799s
user	0m0.009s
sys	0m0.179s


For the series:
Tested-by Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v3 0/5] Reduce RDMA RW API SGE limit

[Steve Wise]

> > > Hello Doug,
> > >
> > > The five patches in this series modify the RDMA RW API slightly. This is
> > > needed to avoid that the SRP and iSER target drivers submit RDMA
> > > requests with an SGE list that exceeds the queue pair limits. The
> > > ib_srpt changes in this series have been tested but the ib_isert changes
> > > not yet.
> > >
> > > The changes compared to v2 of this patch series are:
> > > * For RDMA READs, limit SGE back to dev->attrs.max_sge_rd for iWARP.
> > >
> > > Changes compared to v1:
> > > * max_send_sge is now stored in struct ib_qp. This greatly simplifies
> > >    this patch series.
> > > * An unneeded initialization that I had added to rdma_rw_init_one_mr()
> > >    has been left out again.
> > > * Corrected "Fixes" tag in the patch description where needed.
> > >
> > > The individual patches in this series are:
> > >
> > > 0001-IB-core-Make-rdma_rw_ctx_init-initialize-all-used-fi.patch
> > > 0002-IB-core-RDMA-RW-API-Do-not-exceed-QP-SGE-send-limit.patch
> > > 0003-IB-srpt-Limit-the-number-of-SG-elements-per-work-req.patch
> > > 0004-IB-srpt-Simplify-srpt_queue_response.patch
> > > 0005-IB-isert-Remove-an-unused-member-variable.patch
> > >
> > > Thanks,
> > >
> > > Bart.
> 
> 
> > Will pull and test these ASAP.
> >
> 
> Ditto
> 

Hey Bart, these test out ok with iozone and fio testing over iser/iw_cxgb4.

Tested-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 0/5] Reduce RDMA RW API SGE limit

[Sagi Grimberg]

For the series:

Reviewed-by: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html