RE: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

Steve Wise
2015-07-07T13:59:39Z

> -----Original Message-----
> From: Sagi Grimberg [mailto:sagig@dev.mellanox.co.il]
> Sent: Tuesday, July 07, 2015 1:27 AM
> To: Steve Wise; dledford@redhat.com
> Cc: infinipath@intel.com; sagig@mellanox.com; ogerlitz@mellanox.com; roid@mellanox.com; linux-rdma@vger.kernel.org;
> eli@mellanox.com; target-devel@vger.kernel.org
> Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
>
> On 7/6/2015 5:35 PM, Steve Wise wrote:
> >
> >
> >> -----Original Message-----
> >> From: Sagi Grimberg [mailto:sagig@dev.mellanox.co.il]
> >> Sent: Monday, July 06, 2015 2:51 AM
> >> To: Steve Wise; dledford@redhat.com
> >> Cc: infinipath@intel.com; sagig@mellanox.com; ogerlitz@mellanox.com; roid@mellanox.com; linux-rdma@vger.kernel.org;
> >> eli@mellanox.com; target-devel@vger.kernel.org
> >> Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
> >>
> >> On 7/5/2015 8:44 PM, Steve Wise wrote:
> >>> Currently the sg tablesize, which dictates fast register page list
> >>> depth to use, does not take into account the limits of the rdma device.
> >>> So adjust it once we discover the device fastreg max depth limit. Also
> >>> adjust the max_sectors based on the resulting sg tablesize.
> >>>
> >>> Signed-off-by: Steve Wise <swise@opengridcomputing.com>
> >>> ---
> >>>
> >>> drivers/infiniband/ulp/iser/iscsi_iser.c | 9 +++++++++
> >>> 1 files changed, 9 insertions(+), 0 deletions(-)
> >>>
> >>> diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.c b/drivers/infiniband/ulp/iser/iscsi_iser.c
> >>> index 6a594aa..de8730d 100644
> >>> --- a/drivers/infiniband/ulp/iser/iscsi_iser.c
> >>> +++ b/drivers/infiniband/ulp/iser/iscsi_iser.c
> >>> @@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
> >>> SHOST_DIX_GUARD_CRC);
> >>> }
> >>>
> >>> + /*
> >>> + * Limit the sg_tablesize and max_sectors based on the device
> >>> + * max fastreg page list length.
> >>> + */
> >>> + shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize,
> >>> + ib_conn->device->dev_attr.max_fast_reg_page_list_len);
> >>> + shost->max_sectors = min_t(unsigned int,
> >>> + 1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
> >>> +
> >>
> >> The min statement is meaningless for max_sectors - you do a min between
> >> default sg_tablesize and frpl length - so the maximum sg_tablesize is
> >> 128 which is 1024 max_sectors.
> >
> > I'm not following. What if ib_conn->device->dev_attr.max_fast_reg_page_list_len is say, 32?
> > Then shost->sg_tablesize is set to 32, and max_sectors is set to (32*4K) >> 9 == 256 512B sectors.
>
> Correct - but it cannot exceed 1024 (as it is derived from sg_tablesize
> which is maximum 128).

Actually it is initialized to 1024 in iscsi_iser_sht / iscsi_iser.c, so it isn't derived from sg_tables (although it probably should be). I can remove the min_t() though.

Hey Or, thoughts?

Steve.