[PATCH V5 0/5] iSER support for iWARP

[PATCH V5 0/5] iSER support for iWARP

[Steve Wise]

The following series implements support for iWARP transports in the iSER
initiator and target.  This is based on Doug's k.o/for-4.2 branch.

I've tested this on cxgb4 and mlx4 hardware.

Changes since V4:

iser: fixedcompiler warning

isert: back to setting REMOTE_WRITE only for iWARP devices

Changes since V3:

Fixed commit messages based on feedback.

iser: adjust max_sectors

isert: split into 2 patches

isert: always set REMOTE_WRITE for dma mrs

Changes since V2:

The transport independent work is removed from this series and will
be submitted in a subsequent series.  This V3 series now enables iWARP
using existing core services.

Changes since V1:

Introduce and use transport-independent RDMA core services for allocating
DMA MRs and computing fast register access flags.

Correctly set the device max_sge_rd capability in several rdma device
drivers.

isert: use device capability max_sge_rd for the read sge depth.

isert: change max_sge to max_write_sge in struct isert_conn.

---

Sagi Grimberg (1):
      mlx4, mlx5, mthca: Expose max_sge_rd correctly

Steve Wise (4):
      RDMA/isert: Limit read depth based on the device max_sge_rd capability
      RDMA/isert: Set REMOTE_WRITE on DMA MRs to support iWARP devices
      RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
      ipath,qib: Expose max_sge_rd correctly


 drivers/infiniband/hw/ipath/ipath_verbs.c    |    1 
 drivers/infiniband/hw/mlx4/main.c            |    1 
 drivers/infiniband/hw/mlx5/main.c            |    1 
 drivers/infiniband/hw/mthca/mthca_provider.c |    1 
 drivers/infiniband/hw/qib/qib_verbs.c        |    1 
 drivers/infiniband/ulp/iser/iscsi_iser.c     |    9 ++++
 drivers/infiniband/ulp/isert/ib_isert.c      |   53 ++++++++++++++++++++++----
 drivers/infiniband/ulp/isert/ib_isert.h      |    3 +
 8 files changed, 60 insertions(+), 10 deletions(-)

-- 
Steve.

[PATCH V5 1/5] mlx4, mlx5, mthca: Expose max_sge_rd correctly

[Steve Wise]

From: Sagi Grimberg <sagig@mellanox.com>

Applications must not assume that max_sge and max_sge_rd are the same,
Hence expose max_sge_rd correctly as well.

Reported-by: Steve Wise <swise@opengridcomputing.com>
Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---

 drivers/infiniband/hw/mlx4/main.c            |    1 +
 drivers/infiniband/hw/mlx5/main.c            |    1 +
 drivers/infiniband/hw/mthca/mthca_provider.c |    1 +
 3 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index 166da78..81c342f 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -229,6 +229,7 @@ static int mlx4_ib_query_device(struct ib_device *ibdev,
 	props->max_qp_wr	   = dev->dev->caps.max_wqes - MLX4_IB_SQ_MAX_SPARE;
 	props->max_sge		   = min(dev->dev->caps.max_sq_sg,
 					 dev->dev->caps.max_rq_sg);
+	props->max_sge_rd = props->max_sge;
 	props->max_cq		   = dev->dev->quotas.cq;
 	props->max_cqe		   = dev->dev->caps.max_cqes;
 	props->max_mr		   = dev->dev->quotas.mpt;
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index c6cb26e..a2bcd96 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -137,6 +137,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
 	max_sq_sg = (gen->max_sq_desc_sz - sizeof(struct mlx5_wqe_ctrl_seg)) /
 		sizeof(struct mlx5_wqe_data_seg);
 	props->max_sge = min(max_rq_sg, max_sq_sg);
+	props->max_sge_rd = props->max_sge;
 	props->max_cq		   = 1 << gen->log_max_cq;
 	props->max_cqe		   = gen->max_cqes - 1;
 	props->max_mr		   = 1 << gen->log_max_mkey;
diff --git a/drivers/infiniband/hw/mthca/mthca_provider.c b/drivers/infiniband/hw/mthca/mthca_provider.c
index 93ae51d..dc2d48c 100644
--- a/drivers/infiniband/hw/mthca/mthca_provider.c
+++ b/drivers/infiniband/hw/mthca/mthca_provider.c
@@ -97,6 +97,7 @@ static int mthca_query_device(struct ib_device *ibdev, struct ib_device_attr *pr
 	props->max_qp              = mdev->limits.num_qps - mdev->limits.reserved_qps;
 	props->max_qp_wr           = mdev->limits.max_wqes;
 	props->max_sge             = mdev->limits.max_sg;
+	props->max_sge_rd          = props->max_sge;
 	props->max_cq              = mdev->limits.num_cqs - mdev->limits.reserved_cqs;
 	props->max_cqe             = mdev->limits.max_cqes;
 	props->max_mr              = mdev->limits.num_mpts - mdev->limits.reserved_mrws;

[PATCH V5 2/5] ipath,qib: Expose max_sge_rd correctly

[Steve Wise]

Applications must not assume that max_sge and max_sge_rd are the same,
Hence expose max_sge_rd correctly as well.

Signed-off-by: Steve Wise <swise@opengridcomputing.com>
Acked-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
---

 drivers/infiniband/hw/ipath/ipath_verbs.c |    1 +
 drivers/infiniband/hw/qib/qib_verbs.c     |    1 +
 2 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/drivers/infiniband/hw/ipath/ipath_verbs.c b/drivers/infiniband/hw/ipath/ipath_verbs.c
index 48253b8..d958236 100644
--- a/drivers/infiniband/hw/ipath/ipath_verbs.c
+++ b/drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -1521,6 +1521,7 @@ static int ipath_query_device(struct ib_device *ibdev, struct ib_device_attr *pr
 	props->max_qp = ib_ipath_max_qps;
 	props->max_qp_wr = ib_ipath_max_qp_wrs;
 	props->max_sge = ib_ipath_max_sges;
+	props->max_sge_rd = ib_ipath_max_sges;
 	props->max_cq = ib_ipath_max_cqs;
 	props->max_ah = ib_ipath_max_ahs;
 	props->max_cqe = ib_ipath_max_cqes;
diff --git a/drivers/infiniband/hw/qib/qib_verbs.c b/drivers/infiniband/hw/qib/qib_verbs.c
index a05d1a3..bc723b5 100644
--- a/drivers/infiniband/hw/qib/qib_verbs.c
+++ b/drivers/infiniband/hw/qib/qib_verbs.c
@@ -1574,6 +1574,7 @@ static int qib_query_device(struct ib_device *ibdev, struct ib_device_attr *prop
 	props->max_qp = ib_qib_max_qps;
 	props->max_qp_wr = ib_qib_max_qp_wrs;
 	props->max_sge = ib_qib_max_sges;
+	props->max_sge_rd = ib_qib_max_sges;
 	props->max_cq = ib_qib_max_cqs;
 	props->max_ah = ib_qib_max_ahs;
 	props->max_cqe = ib_qib_max_cqes;

[PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

[Steve Wise]

Currently the sg tablesize, which dictates fast register page list
depth to use, does not take into account the limits of the rdma device.
So adjust it once we discover the device fastreg max depth limit.  Also
adjust the max_sectors based on the resulting sg tablesize.

Signed-off-by: Steve Wise <swise@opengridcomputing.com>
---

 drivers/infiniband/ulp/iser/iscsi_iser.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.c b/drivers/infiniband/ulp/iser/iscsi_iser.c
index 6a594aa..de8730d 100644
--- a/drivers/infiniband/ulp/iser/iscsi_iser.c
+++ b/drivers/infiniband/ulp/iser/iscsi_iser.c
@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
 						   SHOST_DIX_GUARD_CRC);
 		}
 
+		/*
+		 * Limit the sg_tablesize and max_sectors based on the device
+		 * max fastreg page list length.
+		 */
+		shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize,
+			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
+		shost->max_sectors = min_t(unsigned int,
+			1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
+
 		if (iscsi_host_add(shost,
 				   ib_conn->device->ib_device->dma_device)) {
 			mutex_unlock(&iser_conn->state_mutex);

Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

[Sagi Grimberg]

On 7/5/2015 8:44 PM, Steve Wise wrote:
> Currently the sg tablesize, which dictates fast register page list
> depth to use, does not take into account the limits of the rdma device.
> So adjust it once we discover the device fastreg max depth limit.  Also
> adjust the max_sectors based on the resulting sg tablesize.
>
> Signed-off-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
> ---
>
>   drivers/infiniband/ulp/iser/iscsi_iser.c |    9 +++++++++
>   1 files changed, 9 insertions(+), 0 deletions(-)
>
> diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.c b/drivers/infiniband/ulp/iser/iscsi_iser.c
> index 6a594aa..de8730d 100644
> --- a/drivers/infiniband/ulp/iser/iscsi_iser.c
> +++ b/drivers/infiniband/ulp/iser/iscsi_iser.c
> @@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
>   						   SHOST_DIX_GUARD_CRC);
>   		}
>
> +		/*
> +		 * Limit the sg_tablesize and max_sectors based on the device
> +		 * max fastreg page list length.
> +		 */
> +		shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize,
> +			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
> +		shost->max_sectors = min_t(unsigned int,
> +			1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
> +

The min statement is meaningless for max_sectors - you do a min between
default sg_tablesize and frpl length - so the maximum sg_tablesize is
128 which is 1024 max_sectors.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

[Steve Wise]

> -----Original Message-----
> From: Sagi Grimberg [mailto:sagig-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org]
> Sent: Monday, July 06, 2015 2:51 AM
> To: Steve Wise; dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
> Cc: infinipath-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org; sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; roid-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org;
> eli-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; target-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
> 
> On 7/5/2015 8:44 PM, Steve Wise wrote:
> > Currently the sg tablesize, which dictates fast register page list
> > depth to use, does not take into account the limits of the rdma device.
> > So adjust it once we discover the device fastreg max depth limit.  Also
> > adjust the max_sectors based on the resulting sg tablesize.
> >
> > Signed-off-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
> > ---
> >
> >   drivers/infiniband/ulp/iser/iscsi_iser.c |    9 +++++++++
> >   1 files changed, 9 insertions(+), 0 deletions(-)
> >
> > diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.c b/drivers/infiniband/ulp/iser/iscsi_iser.c
> > index 6a594aa..de8730d 100644
> > --- a/drivers/infiniband/ulp/iser/iscsi_iser.c
> > +++ b/drivers/infiniband/ulp/iser/iscsi_iser.c
> > @@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
> >   						   SHOST_DIX_GUARD_CRC);
> >   		}
> >
> > +		/*
> > +		 * Limit the sg_tablesize and max_sectors based on the device
> > +		 * max fastreg page list length.
> > +		 */
> > +		shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize,
> > +			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
> > +		shost->max_sectors = min_t(unsigned int,
> > +			1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
> > +
> 
> The min statement is meaningless for max_sectors - you do a min between
> default sg_tablesize and frpl length - so the maximum sg_tablesize is
> 128 which is 1024 max_sectors.

I'm not following.  What if ib_conn->device->dev_attr.max_fast_reg_page_list_len is say, 32?  Then shost->sg_tablesize is set to 32, and max_sectors is set to (32*4K) >> 9 == 256 512B sectors.

  

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

[Sagi Grimberg]

On 7/6/2015 5:35 PM, Steve Wise wrote:
>
>
>> -----Original Message-----
>> From: Sagi Grimberg [mailto:sagig@dev.mellanox.co.il]
>> Sent: Monday, July 06, 2015 2:51 AM
>> To: Steve Wise; dledford@redhat.com
>> Cc: infinipath@intel.com; sagig@mellanox.com; ogerlitz@mellanox.com; roid@mellanox.com; linux-rdma@vger.kernel.org;
>> eli@mellanox.com; target-devel@vger.kernel.org
>> Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
>>
>> On 7/5/2015 8:44 PM, Steve Wise wrote:
>>> Currently the sg tablesize, which dictates fast register page list
>>> depth to use, does not take into account the limits of the rdma device.
>>> So adjust it once we discover the device fastreg max depth limit.  Also
>>> adjust the max_sectors based on the resulting sg tablesize.
>>>
>>> Signed-off-by: Steve Wise <swise@opengridcomputing.com>
>>> ---
>>>
>>>    drivers/infiniband/ulp/iser/iscsi_iser.c |    9 +++++++++
>>>    1 files changed, 9 insertions(+), 0 deletions(-)
>>>
>>> diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.c b/drivers/infiniband/ulp/iser/iscsi_iser.c
>>> index 6a594aa..de8730d 100644
>>> --- a/drivers/infiniband/ulp/iser/iscsi_iser.c
>>> +++ b/drivers/infiniband/ulp/iser/iscsi_iser.c
>>> @@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
>>>    						   SHOST_DIX_GUARD_CRC);
>>>    		}
>>>
>>> +		/*
>>> +		 * Limit the sg_tablesize and max_sectors based on the device
>>> +		 * max fastreg page list length.
>>> +		 */
>>> +		shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize,
>>> +			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
>>> +		shost->max_sectors = min_t(unsigned int,
>>> +			1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
>>> +
>>
>> The min statement is meaningless for max_sectors - you do a min between
>> default sg_tablesize and frpl length - so the maximum sg_tablesize is
>> 128 which is 1024 max_sectors.
>
> I'm not following.  What if ib_conn->device->dev_attr.max_fast_reg_page_list_len is say, 32?
> Then shost->sg_tablesize is set to 32, and max_sectors is set to (32*4K) >> 9 == 256 512B sectors.

Correct - but it cannot exceed 1024 (as it is derived from sg_tablesize
which is maximum 128).

RE: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

[Steve Wise]

> -----Original Message-----
> From: Sagi Grimberg [mailto:sagig@dev.mellanox.co.il]
> Sent: Tuesday, July 07, 2015 1:27 AM
> To: Steve Wise; dledford@redhat.com
> Cc: infinipath@intel.com; sagig@mellanox.com; ogerlitz@mellanox.com; roid@mellanox.com; linux-rdma@vger.kernel.org;
> eli@mellanox.com; target-devel@vger.kernel.org
> Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
> 
> On 7/6/2015 5:35 PM, Steve Wise wrote:
> >
> >
> >> -----Original Message-----
> >> From: Sagi Grimberg [mailto:sagig@dev.mellanox.co.il]
> >> Sent: Monday, July 06, 2015 2:51 AM
> >> To: Steve Wise; dledford@redhat.com
> >> Cc: infinipath@intel.com; sagig@mellanox.com; ogerlitz@mellanox.com; roid@mellanox.com; linux-rdma@vger.kernel.org;
> >> eli@mellanox.com; target-devel@vger.kernel.org
> >> Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
> >>
> >> On 7/5/2015 8:44 PM, Steve Wise wrote:
> >>> Currently the sg tablesize, which dictates fast register page list
> >>> depth to use, does not take into account the limits of the rdma device.
> >>> So adjust it once we discover the device fastreg max depth limit.  Also
> >>> adjust the max_sectors based on the resulting sg tablesize.
> >>>
> >>> Signed-off-by: Steve Wise <swise@opengridcomputing.com>
> >>> ---
> >>>
> >>>    drivers/infiniband/ulp/iser/iscsi_iser.c |    9 +++++++++
> >>>    1 files changed, 9 insertions(+), 0 deletions(-)
> >>>
> >>> diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.c b/drivers/infiniband/ulp/iser/iscsi_iser.c
> >>> index 6a594aa..de8730d 100644
> >>> --- a/drivers/infiniband/ulp/iser/iscsi_iser.c
> >>> +++ b/drivers/infiniband/ulp/iser/iscsi_iser.c
> >>> @@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
> >>>    						   SHOST_DIX_GUARD_CRC);
> >>>    		}
> >>>
> >>> +		/*
> >>> +		 * Limit the sg_tablesize and max_sectors based on the device
> >>> +		 * max fastreg page list length.
> >>> +		 */
> >>> +		shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize,
> >>> +			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
> >>> +		shost->max_sectors = min_t(unsigned int,
> >>> +			1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
> >>> +
> >>
> >> The min statement is meaningless for max_sectors - you do a min between
> >> default sg_tablesize and frpl length - so the maximum sg_tablesize is
> >> 128 which is 1024 max_sectors.
> >
> > I'm not following.  What if ib_conn->device->dev_attr.max_fast_reg_page_list_len is say, 32?
> > Then shost->sg_tablesize is set to 32, and max_sectors is set to (32*4K) >> 9 == 256 512B sectors.
> 
> Correct - but it cannot exceed 1024 (as it is derived from sg_tablesize
> which is maximum 128).

Actually it is initialized to 1024 in iscsi_iser_sht / iscsi_iser.c, so it isn't derived from sg_tables (although it probably should be).  I can remove the min_t() though.

Hey Or, thoughts?

Steve.

Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

[Or Gerlitz]

On 7/7/2015 4:59 PM, Steve Wise wrote:
>>>>> > >>>diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.c b/drivers/infiniband/ulp/iser/iscsi_iser.c
>>>>> > >>>index 6a594aa..de8730d 100644
>>>>> > >>>--- a/drivers/infiniband/ulp/iser/iscsi_iser.c
>>>>> > >>>+++ b/drivers/infiniband/ulp/iser/iscsi_iser.c
>>>>> > >>>@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
>>>>> > >>>    						   SHOST_DIX_GUARD_CRC);
>>>>> > >>>    		}
>>>>> > >>>
>>>>> > >>>+		/*
>>>>> > >>>+		 * Limit the sg_tablesize and max_sectors based on the device
>>>>> > >>>+		 * max fastreg page list length.
>>>>> > >>>+		 */
>>>>> > >>>+		shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize,
>>>>> > >>>+			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
>>>>> > >>>+		shost->max_sectors = min_t(unsigned int,
>>>>> > >>>+			1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
>>>>> > >>>+
>>>> > >>
>>>> > >>The min statement is meaningless for max_sectors - you do a min between
>>>> > >>default sg_tablesize and frpl length - so the maximum sg_tablesize is
>>>> > >>128 which is 1024 max_sectors.
>>> > >
>>> > >I'm not following.  What if ib_conn->device->dev_attr.max_fast_reg_page_list_len is say, 32?
>>> > >Then shost->sg_tablesize is set to 32, and max_sectors is set to (32*4K) >> 9 == 256 512B sectors.
>> >
>> >Correct - but it cannot exceed 1024 (as it is derived from sg_tablesize
>> >which is maximum 128).
> Actually it is initialized to 1024 in iscsi_iser_sht / iscsi_iser.c, so it isn't derived from sg_tables (although it probably should be).  I can remove the min_t() though.
>
> Hey Or, thoughts?

Originally, we've put the double restriction of 128 SG entries AND 1024 
sectors to make sure that whatever SG is up there, it'snot spanning > 
512KB.

Think on SG whose one/some of their element/s is > one page or on 
systems with > 4KB page size or others examples... since your patch 
touched the number of SG entries I was thinking you need to make sure no 
regression was introduced re the max_sectors to be <= 1024

If U2 are @ consensus that this is  the case with the original patch w.o 
further changes, let it be.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

[Steve Wise]

> -----Original Message-----
> From: linux-rdma-owner@vger.kernel.org [mailto:linux-rdma-owner@vger.kernel.org] On Behalf Of Or Gerlitz
> Sent: Tuesday, July 07, 2015 9:32 AM
> To: Steve Wise; 'Sagi Grimberg'
> Cc: dledford@redhat.com; infinipath@intel.com; roid@mellanox.com; linux-rdma@vger.kernel.org; eli@mellanox.com; target-
> devel@vger.kernel.org
> Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
> 
> On 7/7/2015 4:59 PM, Steve Wise wrote:
> >>>>> > >>>diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.c b/drivers/infiniband/ulp/iser/iscsi_iser.c
> >>>>> > >>>index 6a594aa..de8730d 100644
> >>>>> > >>>--- a/drivers/infiniband/ulp/iser/iscsi_iser.c
> >>>>> > >>>+++ b/drivers/infiniband/ulp/iser/iscsi_iser.c
> >>>>> > >>>@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
> >>>>> > >>>    						   SHOST_DIX_GUARD_CRC);
> >>>>> > >>>    		}
> >>>>> > >>>
> >>>>> > >>>+		/*
> >>>>> > >>>+		 * Limit the sg_tablesize and max_sectors based on the device
> >>>>> > >>>+		 * max fastreg page list length.
> >>>>> > >>>+		 */
> >>>>> > >>>+		shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize,
> >>>>> > >>>+			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
> >>>>> > >>>+		shost->max_sectors = min_t(unsigned int,
> >>>>> > >>>+			1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
> >>>>> > >>>+
> >>>> > >>
> >>>> > >>The min statement is meaningless for max_sectors - you do a min between
> >>>> > >>default sg_tablesize and frpl length - so the maximum sg_tablesize is
> >>>> > >>128 which is 1024 max_sectors.
> >>> > >
> >>> > >I'm not following.  What if ib_conn->device->dev_attr.max_fast_reg_page_list_len is say, 32?
> >>> > >Then shost->sg_tablesize is set to 32, and max_sectors is set to (32*4K) >> 9 == 256 512B sectors.
> >> >
> >> >Correct - but it cannot exceed 1024 (as it is derived from sg_tablesize
> >> >which is maximum 128).
> > Actually it is initialized to 1024 in iscsi_iser_sht / iscsi_iser.c, so it isn't derived from sg_tables (although it probably should be).  I can
> remove the min_t() though.
> >
> > Hey Or, thoughts?
> 
> Originally, we've put the double restriction of 128 SG entries AND 1024
> sectors to make sure that whatever SG is up there, it'snot spanning >
> 512KB.
> 
> Think on SG whose one/some of their element/s is > one page or on
> systems with > 4KB page size or others examples... since your patch
> touched the number of SG entries I was thinking you need to make sure no
> regression was introduced re the max_sectors to be <= 1024
> 
> If U2 are @ consensus that this is  the case with the original patch w.o
> further changes, let it be.
> 

I'm not sure... you guys are the iSER experts. :)  But considering a 64K PAGE_SIZE and an adjusted sg_tablesize of say, 32, w/o the min() we get (32 * 65536) >> 9 ==  4096.  So if the requirement is that max_sectors always be <= 1024, then we need the min()...

Steve.

Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

[Sagi Grimberg]

On 7/7/2015 6:41 PM, Steve Wise wrote:
>
>
>> -----Original Message-----
>> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Or Gerlitz
>> Sent: Tuesday, July 07, 2015 9:32 AM
>> To: Steve Wise; 'Sagi Grimberg'
>> Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org; infinipath-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org; roid-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; eli-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; target-
>> devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
>>
>> On 7/7/2015 4:59 PM, Steve Wise wrote:
>>>>>>>>>>> diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.c b/drivers/infiniband/ulp/iser/iscsi_iser.c
>>>>>>>>>>> index 6a594aa..de8730d 100644
>>>>>>>>>>> --- a/drivers/infiniband/ulp/iser/iscsi_iser.c
>>>>>>>>>>> +++ b/drivers/infiniband/ulp/iser/iscsi_iser.c
>>>>>>>>>>> @@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
>>>>>>>>>>>     						   SHOST_DIX_GUARD_CRC);
>>>>>>>>>>>     		}
>>>>>>>>>>>
>>>>>>>>>>> +		/*
>>>>>>>>>>> +		 * Limit the sg_tablesize and max_sectors based on the device
>>>>>>>>>>> +		 * max fastreg page list length.
>>>>>>>>>>> +		 */
>>>>>>>>>>> +		shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize,
>>>>>>>>>>> +			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
>>>>>>>>>>> +		shost->max_sectors = min_t(unsigned int,
>>>>>>>>>>> +			1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
>>>>>>>>>>> +
>>>>>>>>>
>>>>>>>>> The min statement is meaningless for max_sectors - you do a min between
>>>>>>>>> default sg_tablesize and frpl length - so the maximum sg_tablesize is
>>>>>>>>> 128 which is 1024 max_sectors.
>>>>>>>
>>>>>>> I'm not following.  What if ib_conn->device->dev_attr.max_fast_reg_page_list_len is say, 32?
>>>>>>> Then shost->sg_tablesize is set to 32, and max_sectors is set to (32*4K) >> 9 == 256 512B sectors.
>>>>>
>>>>> Correct - but it cannot exceed 1024 (as it is derived from sg_tablesize
>>>>> which is maximum 128).
>>> Actually it is initialized to 1024 in iscsi_iser_sht / iscsi_iser.c, so it isn't derived from sg_tables (although it probably should be).  I can
>> remove the min_t() though.
>>>
>>> Hey Or, thoughts?
>>
>> Originally, we've put the double restriction of 128 SG entries AND 1024
>> sectors to make sure that whatever SG is up there, it'snot spanning >
>> 512KB.
>>
>> Think on SG whose one/some of their element/s is > one page or on
>> systems with > 4KB page size or others examples... since your patch
>> touched the number of SG entries I was thinking you need to make sure no
>> regression was introduced re the max_sectors to be <= 1024
>>
>> If U2 are @ consensus that this is  the case with the original patch w.o
>> further changes, let it be.
>>
>
> I'm not sure... you guys are the iSER experts. :)  But considering a 64K PAGE_SIZE and an adjusted sg_tablesize of say, 32, w/o the min() we get (32 * 65536) >> 9 ==  4096.  So if the requirement is that max_sectors always be <= 1024, then we need the min()...
>

I guess the min is fine given that I have a patchset for 8MB support,
so we'll get it right then.

Sagi.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V5 4/5] RDMA/isert: Set REMOTE_WRITE on DMA MRs to support iWARP devices

[Steve Wise]

iWARP devices require REMOTE_WRITE for MRs used as the destination of
an RDMA READ.  So if the device protocol is iWARP, then set REMOTE_WRITE
when allocating the DMA MR.

Signed-off-by: Steve Wise <swise@opengridcomputing.com>
---

 drivers/infiniband/ulp/isert/ib_isert.c |   22 +++++++++++++++++++++-
 1 files changed, 21 insertions(+), 1 deletions(-)

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 9e7b492..ee40478 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -348,6 +348,17 @@ out_cq:
 	return ret;
 }
 
+static int any_port_is_iwarp(struct isert_device *device)
+{
+	int i;
+
+	for (i = rdma_start_port(device->ib_device);
+	     i <= rdma_end_port(device->ib_device); i++)
+		if (rdma_protocol_iwarp(device->ib_device, i))
+			return 1;
+	return 0;
+}
+
 static int
 isert_create_device_ib_res(struct isert_device *device)
 {
@@ -383,7 +394,16 @@ isert_create_device_ib_res(struct isert_device *device)
 		goto out_cq;
 	}
 
-	device->mr = ib_get_dma_mr(device->pd, IB_ACCESS_LOCAL_WRITE);
+	/*
+	 * IWARP transports need REMOTE_WRITE for MRs used as the target of
+	 * an RDMA_READ.  Since the DMA MR is used for all ports, then if
+	 * any port is running IWARP, add REMOTE_WRITE.
+	 */
+	if (any_port_is_iwarp(device))
+		device->mr = ib_get_dma_mr(device->pd, IB_ACCESS_LOCAL_WRITE |
+						       IB_ACCESS_REMOTE_WRITE);
+	else
+		device->mr = ib_get_dma_mr(device->pd, IB_ACCESS_LOCAL_WRITE);
 	if (IS_ERR(device->mr)) {
 		ret = PTR_ERR(device->mr);
 		isert_err("failed to create dma mr, device %p, ret=%d\n",

Re: [PATCH V5 4/5] RDMA/isert: Set REMOTE_WRITE on DMA MRs to support iWARP devices

[Sagi Grimberg]

On 7/5/2015 8:45 PM, Steve Wise wrote:
> iWARP devices require REMOTE_WRITE for MRs used as the destination of
> an RDMA READ.  So if the device protocol is iWARP, then set REMOTE_WRITE
> when allocating the DMA MR.
>
> Signed-off-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>

Reviewed-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

[Steve Wise]

Use the device's max_sge_rd capability to compute the target's read sge
depth.  Save both the read and write max_sge values in the isert_conn
struct, and use these when creating RDMA_WRITE/READ work requests.

Signed-off-by: Steve Wise <swise@opengridcomputing.com>
---

 drivers/infiniband/ulp/isert/ib_isert.c |   31 +++++++++++++++++++++++--------
 drivers/infiniband/ulp/isert/ib_isert.h |    3 ++-
 2 files changed, 25 insertions(+), 9 deletions(-)

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index ee40478..35015c9 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -163,7 +163,9 @@ isert_create_qp(struct isert_conn *isert_conn,
 	 * outgoing control PDU responses.
 	 */
 	attr.cap.max_send_sge = max(2, device->dev_attr.max_sge - 2);
-	isert_conn->max_sge = attr.cap.max_send_sge;
+	isert_conn->max_write_sge = attr.cap.max_send_sge;
+	isert_conn->max_read_sge = min_t(u32, device->dev_attr.max_sge_rd,
+					 attr.cap.max_send_sge);
 
 	attr.cap.max_recv_sge = 1;
 	attr.sq_sig_type = IB_SIGNAL_REQ_WR;
@@ -2395,7 +2397,7 @@ isert_put_text_rsp(struct iscsi_cmd *cmd, struct iscsi_conn *conn)
 static int
 isert_build_rdma_wr(struct isert_conn *isert_conn, struct isert_cmd *isert_cmd,
 		    struct ib_sge *ib_sge, struct ib_send_wr *send_wr,
-		    u32 data_left, u32 offset)
+		    u32 data_left, u32 offset, u32 max_sge)
 {
 	struct iscsi_cmd *cmd = isert_cmd->iscsi_cmd;
 	struct scatterlist *sg_start, *tmp_sg;
@@ -2406,7 +2408,7 @@ isert_build_rdma_wr(struct isert_conn *isert_conn, struct isert_cmd *isert_cmd,
 
 	sg_off = offset / PAGE_SIZE;
 	sg_start = &cmd->se_cmd.t_data_sg[sg_off];
-	sg_nents = min(cmd->se_cmd.t_data_nents - sg_off, isert_conn->max_sge);
+	sg_nents = min(cmd->se_cmd.t_data_nents - sg_off, max_sge);
 	page_off = offset % PAGE_SIZE;
 
 	send_wr->sg_list = ib_sge;
@@ -2450,8 +2452,9 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	struct isert_data_buf *data = &wr->data;
 	struct ib_send_wr *send_wr;
 	struct ib_sge *ib_sge;
-	u32 offset, data_len, data_left, rdma_write_max, va_offset = 0;
+	u32 offset, data_len, data_left, rdma_max_len, va_offset = 0;
 	int ret = 0, i, ib_sge_cnt;
+	u32 max_sge;
 
 	isert_cmd->tx_desc.isert_cmd = isert_cmd;
 
@@ -2473,7 +2476,12 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	}
 	wr->ib_sge = ib_sge;
 
-	wr->send_wr_num = DIV_ROUND_UP(data->nents, isert_conn->max_sge);
+	if (wr->iser_ib_op == ISER_IB_RDMA_WRITE)
+		max_sge = isert_conn->max_write_sge;
+	else
+		max_sge =  isert_conn->max_read_sge;
+
+	wr->send_wr_num = DIV_ROUND_UP(data->nents, max_sge);
 	wr->send_wr = kzalloc(sizeof(struct ib_send_wr) * wr->send_wr_num,
 				GFP_KERNEL);
 	if (!wr->send_wr) {
@@ -2483,11 +2491,11 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	}
 
 	wr->isert_cmd = isert_cmd;
-	rdma_write_max = isert_conn->max_sge * PAGE_SIZE;
+	rdma_max_len = max_sge * PAGE_SIZE;
 
 	for (i = 0; i < wr->send_wr_num; i++) {
 		send_wr = &isert_cmd->rdma_wr.send_wr[i];
-		data_len = min(data_left, rdma_write_max);
+		data_len = min(data_left, rdma_max_len);
 
 		send_wr->send_flags = 0;
 		if (wr->iser_ib_op == ISER_IB_RDMA_WRITE) {
@@ -2509,7 +2517,7 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 		}
 
 		ib_sge_cnt = isert_build_rdma_wr(isert_conn, isert_cmd, ib_sge,
-					send_wr, data_len, offset);
+					send_wr, data_len, offset, max_sge);
 		ib_sge += ib_sge_cnt;
 
 		offset += data_len;
@@ -2638,6 +2646,13 @@ isert_fast_reg_mr(struct isert_conn *isert_conn,
 	fr_wr.wr.fast_reg.rkey = mr->rkey;
 	fr_wr.wr.fast_reg.access_flags = IB_ACCESS_LOCAL_WRITE;
 
+	/*
+	 * IWARP transports need REMOTE_WRITE for MRs used as the target of
+	 * an RDMA_READ.
+	 */
+	if (rdma_protocol_iwarp(ib_dev, isert_conn->cm_id->port_num))
+		fr_wr.wr.fast_reg.access_flags |= IB_ACCESS_REMOTE_WRITE;
+
 	if (!wr)
 		wr = &fr_wr;
 	else
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
index 9ec23a7..29fde27 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -152,7 +152,8 @@ struct isert_conn {
 	u32			responder_resources;
 	u32			initiator_depth;
 	bool			pi_support;
-	u32			max_sge;
+	u32			max_write_sge;
+	u32			max_read_sge;
 	char			*login_buf;
 	char			*login_req_buf;
 	char			*login_rsp_buf;

Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

[Sagi Grimberg]

On 7/5/2015 8:45 PM, Steve Wise wrote:
> Use the device's max_sge_rd capability to compute the target's read sge
> depth.  Save both the read and write max_sge values in the isert_conn
> struct, and use these when creating RDMA_WRITE/READ work requests.
>
> Signed-off-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>

Reviewed-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

[Christoph Hellwig]

On Sun, Jul 05, 2015 at 12:45:06PM -0500, Steve Wise wrote:
> Use the device's max_sge_rd capability to compute the target's read sge
> depth.  Save both the read and write max_sge values in the isert_conn
> struct, and use these when creating RDMA_WRITE/READ work requests.

Btw, any hance to make the NFS client use these values as well instead
of the current rdma_read_max_sge() hack?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

[Steve Wise]

> -----Original Message-----
> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Christoph Hellwig
> Sent: Tuesday, July 14, 2015 3:27 AM
> To: Steve Wise
> Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org; infinipath-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org; sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; roid-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; linux-
> rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; eli-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; target-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability
> 
> On Sun, Jul 05, 2015 at 12:45:06PM -0500, Steve Wise wrote:
> > Use the device's max_sge_rd capability to compute the target's read sge
> > depth.  Save both the read and write max_sge values in the isert_conn
> > struct, and use these when creating RDMA_WRITE/READ work requests.
> 
> Btw, any hance to make the NFS client use these values as well instead
> of the current rdma_read_max_sge() hack?

Chuck, can you add this to your cleanup list?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

['Christoph Hellwig']

On Tue, Jul 14, 2015 at 09:41:00AM -0500, Steve Wise wrote:
> > Btw, any hance to make the NFS client use these values as well instead
> > of the current rdma_read_max_sge() hack?
> 
> Chuck, can you add this to your cleanup list?

It would be useful to add this to your series so we can get rid of
it for the next merge window instead of introducing a depenency that
would defer it to the next merge window.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

[Steve Wise]

> -----Original Message-----
> From: 'Christoph Hellwig' [mailto:hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org]
> Sent: Tuesday, July 14, 2015 10:42 AM
> To: Steve Wise
> Cc: 'Christoph Hellwig'; Chuck Lever; dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org; infinipath-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org; sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org;
> roid-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; eli-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; target-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability
> 
> On Tue, Jul 14, 2015 at 09:41:00AM -0500, Steve Wise wrote:
> > > Btw, any hance to make the NFS client use these values as well instead
> > > of the current rdma_read_max_sge() hack?
> >
> > Chuck, can you add this to your cleanup list?
> 
> It would be useful to add this to your series so we can get rid of
> it for the next merge window instead of introducing a depenency that
> would defer it to the next merge window.

Ok, I will do this.




--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

[Chuck Lever]

On Jul 14, 2015, at 11:49 AM, Steve Wise <swise@opengridcomputing.com> wrote:

> 
> 
>> -----Original Message-----
>> From: 'Christoph Hellwig' [mailto:hch@infradead.org]
>> Sent: Tuesday, July 14, 2015 10:42 AM
>> To: Steve Wise
>> Cc: 'Christoph Hellwig'; Chuck Lever; dledford@redhat.com; infinipath@intel.com; sagig@mellanox.com; ogerlitz@mellanox.com;
>> roid@mellanox.com; linux-rdma@vger.kernel.org; eli@mellanox.com; target-devel@vger.kernel.org
>> Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability
>> 
>> On Tue, Jul 14, 2015 at 09:41:00AM -0500, Steve Wise wrote:
>>>> Btw, any hance to make the NFS client use these values as well instead
>>>> of the current rdma_read_max_sge() hack?
>>> 
>>> Chuck, can you add this to your cleanup list?
>> 
>> It would be useful to add this to your series so we can get rid of
>> it for the next merge window instead of introducing a depenency that
>> would defer it to the next merge window.
> 
> Ok, I will do this.

Steve, can you review v2 of the nfs-rdma-for-4.3 series I posted
yesterday? Specifically:

  http://git.linux-nfs.org/?p=cel/cel-2.6.git;a=commit;h=6abafb636e03fbb7f93a26796223833581a70190

Which changes the way xprtrdma uses max_sge.

--
Chuck Lever

RE: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

[Steve Wise]

> -----Original Message-----
> From: linux-rdma-owner@vger.kernel.org [mailto:linux-rdma-owner@vger.kernel.org] On Behalf Of Chuck Lever
> Sent: Tuesday, July 14, 2015 1:47 PM
> To: Steve Wise
> Cc: Christoph Hellwig; Doug Ledford; infinipath@intel.com; Sagi Grimberg; Or Gerlitz; roid@mellanox.com; linux-rdma; Eli Cohen;
target-
> devel@vger.kernel.org
> Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability
> 
> 
> On Jul 14, 2015, at 11:49 AM, Steve Wise <swise@opengridcomputing.com> wrote:
> 
> >
> >
> >> -----Original Message-----
> >> From: 'Christoph Hellwig' [mailto:hch@infradead.org]
> >> Sent: Tuesday, July 14, 2015 10:42 AM
> >> To: Steve Wise
> >> Cc: 'Christoph Hellwig'; Chuck Lever; dledford@redhat.com; infinipath@intel.com; sagig@mellanox.com; ogerlitz@mellanox.com;
> >> roid@mellanox.com; linux-rdma@vger.kernel.org; eli@mellanox.com; target-devel@vger.kernel.org
> >> Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability
> >>
> >> On Tue, Jul 14, 2015 at 09:41:00AM -0500, Steve Wise wrote:
> >>>> Btw, any hance to make the NFS client use these values as well instead
> >>>> of the current rdma_read_max_sge() hack?
> >>>
> >>> Chuck, can you add this to your cleanup list?
> >>
> >> It would be useful to add this to your series so we can get rid of
> >> it for the next merge window instead of introducing a depenency that
> >> would defer it to the next merge window.
> >
> > Ok, I will do this.
> 
> Steve, can you review v2 of the nfs-rdma-for-4.3 series I posted
> yesterday? Specifically:
> 
>   http://git.linux-nfs.org/?p=cel/cel-2.6.git;a=commit;h=6abafb636e03fbb7f93a26796223833581a70190
> 
> Which changes the way xprtrdma uses max_sge.
> 

Sure, but xprtrdma doesn't issue reads, so max_sge_rd isn't needed.  The change Christoph wants is actually in svcrdma.  I will
change the server to use max_sge_rd instead of rdma_cap_read_multi_sge().

Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

[Chuck Lever]

On Jul 14, 2015, at 3:11 PM, Steve Wise <swise@opengridcomputing.com> wrote:

> 
> 
>> -----Original Message-----
>> From: linux-rdma-owner@vger.kernel.org [mailto:linux-rdma-owner@vger.kernel.org] On Behalf Of Chuck Lever
>> Sent: Tuesday, July 14, 2015 1:47 PM
>> To: Steve Wise
>> Cc: Christoph Hellwig; Doug Ledford; infinipath@intel.com; Sagi Grimberg; Or Gerlitz; roid@mellanox.com; linux-rdma; Eli Cohen;
> target-
>> devel@vger.kernel.org
>> Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability
>> 
>> 
>> On Jul 14, 2015, at 11:49 AM, Steve Wise <swise@opengridcomputing.com> wrote:
>> 
>>> 
>>> 
>>>> -----Original Message-----
>>>> From: 'Christoph Hellwig' [mailto:hch@infradead.org]
>>>> Sent: Tuesday, July 14, 2015 10:42 AM
>>>> To: Steve Wise
>>>> Cc: 'Christoph Hellwig'; Chuck Lever; dledford@redhat.com; infinipath@intel.com; sagig@mellanox.com; ogerlitz@mellanox.com;
>>>> roid@mellanox.com; linux-rdma@vger.kernel.org; eli@mellanox.com; target-devel@vger.kernel.org
>>>> Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability
>>>> 
>>>> On Tue, Jul 14, 2015 at 09:41:00AM -0500, Steve Wise wrote:
>>>>>> Btw, any hance to make the NFS client use these values as well instead
>>>>>> of the current rdma_read_max_sge() hack?
>>>>> 
>>>>> Chuck, can you add this to your cleanup list?
>>>> 
>>>> It would be useful to add this to your series so we can get rid of
>>>> it for the next merge window instead of introducing a depenency that
>>>> would defer it to the next merge window.
>>> 
>>> Ok, I will do this.
>> 
>> Steve, can you review v2 of the nfs-rdma-for-4.3 series I posted
>> yesterday? Specifically:
>> 
>>  http://git.linux-nfs.org/?p=cel/cel-2.6.git;a=commit;h=6abafb636e03fbb7f93a26796223833581a70190
>> 
>> Which changes the way xprtrdma uses max_sge.
>> 
> 
> Sure, but xprtrdma doesn't issue reads, so max_sge_rd isn't needed.  The change Christoph wants is actually in svcrdma.  I will
> change the server to use max_sge_rd instead of rdma_cap_read_multi_sge().

OK, glad the conflict has been avoided.

--
Chuck Lever

Re: [PATCH V5 0/5] iSER support for iWARP

[Nicholas A. Bellinger]

Hi Steve & Co,

On Sun, 2015-07-05 at 12:44 -0500, Steve Wise wrote:
> The following series implements support for iWARP transports in the iSER
> initiator and target.  This is based on Doug's k.o/for-4.2 branch.
> 
> I've tested this on cxgb4 and mlx4 hardware.
> 
> Changes since V4:
> 
> iser: fixedcompiler warning
> 
> isert: back to setting REMOTE_WRITE only for iWARP devices
> 
> Changes since V3:
> 
> Fixed commit messages based on feedback.
> 
> iser: adjust max_sectors
> 
> isert: split into 2 patches
> 
> isert: always set REMOTE_WRITE for dma mrs
> 
> Changes since V2:
> 
> The transport independent work is removed from this series and will
> be submitted in a subsequent series.  This V3 series now enables iWARP
> using existing core services.
> 
> Changes since V1:
> 
> Introduce and use transport-independent RDMA core services for allocating
> DMA MRs and computing fast register access flags.
> 
> Correctly set the device max_sge_rd capability in several rdma device
> drivers.
> 
> isert: use device capability max_sge_rd for the read sge depth.
> 
> isert: change max_sge to max_write_sge in struct isert_conn.
> 
> ---
> 
> Sagi Grimberg (1):
>       mlx4, mlx5, mthca: Expose max_sge_rd correctly
> 
> Steve Wise (4):
>       RDMA/isert: Limit read depth based on the device max_sge_rd capability
>       RDMA/isert: Set REMOTE_WRITE on DMA MRs to support iWARP devices
>       RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
>       ipath,qib: Expose max_sge_rd correctly
> 
> 
>  drivers/infiniband/hw/ipath/ipath_verbs.c    |    1 
>  drivers/infiniband/hw/mlx4/main.c            |    1 
>  drivers/infiniband/hw/mlx5/main.c            |    1 
>  drivers/infiniband/hw/mthca/mthca_provider.c |    1 
>  drivers/infiniband/hw/qib/qib_verbs.c        |    1 
>  drivers/infiniband/ulp/iser/iscsi_iser.c     |    9 ++++
>  drivers/infiniband/ulp/isert/ib_isert.c      |   53 ++++++++++++++++++++++----
>  drivers/infiniband/ulp/isert/ib_isert.h      |    3 +
>  8 files changed, 60 insertions(+), 10 deletions(-)
> 

Very excited to see iWARP support for iser-initiator + iser-target.  ;)

No objections to taking the iser-target changes through Doug's
infiniband.git.  For the target pieces:

Acked-by: Nicholas Bellinger <nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org>

Also Doug, since this series is sufficiently small enough, and allows a class
of hardware that people are expecting to function with iser to 'just work', I'd
recommend to go ahead and push this series for v4.2-rc code.

Thank you,

--nab


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html