[PATCH 0/6] IB/srp: Second series of patches for kernel v4.7

[PATCH 0/6] IB/srp: Second series of patches for kernel v4.7

[Bart Van Assche]

Hello Doug,

Since the first eight patches of the patch series "[PATCH 0/11] More SRP 
patches for kernel v4.7" have already been applied I'm reposting the 
reworked remaining patches. These patches are:

0001-IB-srp-Print-ib_srp-prefix-once.patch
0002-IB-srp-Fix-a-memory-descriptor-leak-in-an-error-path.patch
0003-IB-srp-Fix-srp_create_target-error-handling.patch
0004-IB-core-Enhance-ib_map_mr_sg.patch
0005-IB-srp-Swap-two-code-blocks-in-srp_add_one.patch
0006-IB-srp-Prevent-mapping-failures.patch
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 1/6] IB/srp: Print "ib_srp: " prefix once

[Bart Van Assche]

pr_debug() already prints prefix PFX. Avoid that PFX is printed
twice if the debug statement in srp_add_target() is enabled.

Fixes: 34aa654ecb8e ("IB/srp: Avoid that I/O hangs due to a cable pull during LUN scanning")
Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # v3.19+
---
 drivers/infiniband/ulp/srp/ib_srp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index caefd1a..e088a49 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -2839,7 +2839,7 @@ static int srp_add_target(struct srp_host *host, struct srp_target_port *target)
 		goto out;
 	}
 
-	pr_debug(PFX "%s: SCSI scan succeeded - detected %d LUNs\n",
+	pr_debug("%s: SCSI scan succeeded - detected %d LUNs\n",
 		 dev_name(&target->scsi_host->shost_gendev),
 		 srp_sdev_count(target->scsi_host));
 
-- 
2.8.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/6] IB/srp: Print "ib_srp: " prefix once

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

On Tue, May 10, 2016 at 05:12:52PM -0700, Bart Van Assche wrote:
> pr_debug() already prints prefix PFX. Avoid that PFX is printed
> twice if the debug statement in srp_add_target() is enabled.
> 
> Fixes: 34aa654ecb8e ("IB/srp: Avoid that I/O hangs due to a cable pull during LUN scanning")
> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # v3.19+

Looks good,
Reviewed-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

RE: [PATCH 1/6] IB/srp: Print "ib_srp: " prefix once

[Steve Wise]

> pr_debug() already prints prefix PFX. Avoid that PFX is printed
> twice if the debug statement in srp_add_target() is enabled.
> 
> Fixes: 34aa654ecb8e ("IB/srp: Avoid that I/O hangs due to a cable pull during
> LUN scanning")
> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # v3.19+

Is this really needed for stable?


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/6] IB/srp: Print "ib_srp: " prefix once

[Bart Van Assche]

On 05/11/2016 06:24 AM, Steve Wise wrote:
>> pr_debug() already prints prefix PFX. Avoid that PFX is printed
>> twice if the debug statement in srp_add_target() is enabled.
>>
>> Fixes: 34aa654ecb8e ("IB/srp: Avoid that I/O hangs due to a cable pull during
>> LUN scanning")
>> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
>> Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
>> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
>> Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # v3.19+
>
> Is this really needed for stable?

Hello Steve,

Since this patch touches a debug statement only I'm fine with not CC-ing 
stable for this patch.

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 2/6] IB/srp: Fix a memory descriptor leak in an error path

[Bart Van Assche]

If an error occurs after srp_fr_pool_get() succeeded and before the
descriptor is stored in srp_map_state (*state->fr.next++ = desc)
then srp_unmap_data() won't free the newly allocated memory
descriptor. Hence free the descriptor explicitly.

Fixes: f7f7aab1a5c0 ("IB/srp: Convert to new registration API")
Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: Sagi Grimberg <sai-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # v4.4+
---
 drivers/infiniband/ulp/srp/ib_srp.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index e088a49..74e3ec8 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -1330,8 +1330,13 @@ static int srp_map_finish_fr(struct srp_map_state *state,
 	ib_update_fast_reg_key(desc->mr, rkey);
 
 	n = ib_map_mr_sg(desc->mr, state->sg, sg_nents, 0, dev->mr_page_size);
-	if (unlikely(n < 0))
+	if (unlikely(n < 0)) {
+		srp_fr_pool_put(ch->fr_pool, &desc, 1);
+		pr_debug("%s: ib_map_mr_sg(%d) returned %d.\n",
+			 dev_name(&req->scmnd->device->sdev_gendev), sg_nents,
+			 n);
 		return n;
+	}
 
 	req->reg_cqe.done = srp_reg_mr_err_done;
 
-- 
2.8.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/6] IB/srp: Fix a memory descriptor leak in an error path

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2037 bytes --]

On Tue, May 10, 2016 at 05:13:21PM -0700, Bart Van Assche wrote:
> If an error occurs after srp_fr_pool_get() succeeded and before the
> descriptor is stored in srp_map_state (*state->fr.next++ = desc)
> then srp_unmap_data() won't free the newly allocated memory
> descriptor. Hence free the descriptor explicitly.
> 
> Fixes: f7f7aab1a5c0 ("IB/srp: Convert to new registration API")
> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: Sagi Grimberg <sai-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
> Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # v4.4+
> ---
>  drivers/infiniband/ulp/srp/ib_srp.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
> index e088a49..74e3ec8 100644
> --- a/drivers/infiniband/ulp/srp/ib_srp.c
> +++ b/drivers/infiniband/ulp/srp/ib_srp.c
> @@ -1330,8 +1330,13 @@ static int srp_map_finish_fr(struct srp_map_state *state,
>  	ib_update_fast_reg_key(desc->mr, rkey);
>  
>  	n = ib_map_mr_sg(desc->mr, state->sg, sg_nents, 0, dev->mr_page_size);
> -	if (unlikely(n < 0))
> +	if (unlikely(n < 0)) {

The ib_map_mr_sg can return 0 which is not error, but still pretty
useless number of mapped SGE. I didn't look on the srp code close enough, but will
the code handle this case correctly?

> +		srp_fr_pool_put(ch->fr_pool, &desc, 1);
> +		pr_debug("%s: ib_map_mr_sg(%d) returned %d.\n",
> +			 dev_name(&req->scmnd->device->sdev_gendev), sg_nents,
> +			 n);
>  		return n;
> +	}
>  
>  	req->reg_cqe.done = srp_reg_mr_err_done;
>  
> -- 
> 2.8.2
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH 2/6] IB/srp: Fix a memory descriptor leak in an error path

[Bart Van Assche]

On 05/11/2016 12:31 AM, Leon Romanovsky wrote:
> On Tue, May 10, 2016 at 05:13:21PM -0700, Bart Van Assche wrote:
>> If an error occurs after srp_fr_pool_get() succeeded and before the
>> descriptor is stored in srp_map_state (*state->fr.next++ = desc)
>> then srp_unmap_data() won't free the newly allocated memory
>> descriptor. Hence free the descriptor explicitly.
>>
>> Fixes: f7f7aab1a5c0 ("IB/srp: Convert to new registration API")
>> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
>> Cc: Sagi Grimberg <sai-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
>> Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
>> Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # v4.4+
>> ---
>>   drivers/infiniband/ulp/srp/ib_srp.c | 7 ++++++-
>>   1 file changed, 6 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
>> index e088a49..74e3ec8 100644
>> --- a/drivers/infiniband/ulp/srp/ib_srp.c
>> +++ b/drivers/infiniband/ulp/srp/ib_srp.c
>> @@ -1330,8 +1330,13 @@ static int srp_map_finish_fr(struct srp_map_state *state,
>>   	ib_update_fast_reg_key(desc->mr, rkey);
>>
>>   	n = ib_map_mr_sg(desc->mr, state->sg, sg_nents, 0, dev->mr_page_size);
>> -	if (unlikely(n < 0))
>> +	if (unlikely(n < 0)) {
>
> The ib_map_mr_sg can return 0 which is not error, but still pretty
> useless number of mapped SGE. I didn't look on the srp code close enough, but will
> the code handle this case correctly?

Hello Leon,

ib_map_mr_sg() can only return 0 after patch 4/6 of this series has been 
applied. xfstests triggers I/O requests that are large enough to make 
ib_map_mr_sg() return 0 when enabling register_always and when using 
fast registration with an mlx4 HCA. This is because for mlx4 the number 
of pages per fast registration request is limited to 511. 511 * 4096 = 
2044 KB. This is less than the value I used for max_sectors in my tests 
(4 MB).

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 3/6] IB/srp: Fix srp_create_target() error handling

[Bart Van Assche]

Avoid that the following kernel oops occurs if memory pool
allocation fails:

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffffa048d0a0>] ib_drain_rq+0x0/0x20 [ib_core]
Call Trace:
 [<ffffffffa04af386>] srp_create_target+0xca6/0x13a9 [ib_srp]
 [<ffffffff813cc863>] dev_attr_store+0x13/0x20
 [<ffffffff81214b50>] sysfs_kf_write+0x40/0x50
 [<ffffffff81213f1c>] kernfs_fop_write+0x13c/0x180
 [<ffffffff81197683>] __vfs_write+0x23/0xf0
 [<ffffffff81198744>] vfs_write+0xa4/0x1a0
 [<ffffffff81199a44>] SyS_write+0x44/0xa0
 [<ffffffff8159e3e9>] entry_SYSCALL_64_fastpath+0x1c/0xac

Fixes: 1dc7b1f10dcb ("IB/srp: use the new CQ API")
Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # v4.5+
---
 drivers/infiniband/ulp/srp/ib_srp.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 74e3ec8..f4dc6f9 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -447,16 +447,16 @@ static struct srp_fr_pool *srp_alloc_fr_pool(struct srp_target_port *target)
 
 /**
  * srp_destroy_qp() - destroy an RDMA queue pair
- * @ch: SRP RDMA channel.
+ * @qp: RDMA queue pair.
  *
  * Drain the qp before destroying it.  This avoids that the receive
  * completion handler can access the queue pair while it is
  * being destroyed.
  */
-static void srp_destroy_qp(struct srp_rdma_ch *ch)
+static void srp_destroy_qp(struct ib_qp *qp)
 {
-	ib_drain_rq(ch->qp);
-	ib_destroy_qp(ch->qp);
+	ib_drain_rq(qp);
+	ib_destroy_qp(qp);
 }
 
 static int srp_create_ch_ib(struct srp_rdma_ch *ch)
@@ -529,7 +529,7 @@ static int srp_create_ch_ib(struct srp_rdma_ch *ch)
 	}
 
 	if (ch->qp)
-		srp_destroy_qp(ch);
+		srp_destroy_qp(ch->qp);
 	if (ch->recv_cq)
 		ib_free_cq(ch->recv_cq);
 	if (ch->send_cq)
@@ -553,7 +553,7 @@ static int srp_create_ch_ib(struct srp_rdma_ch *ch)
 	return 0;
 
 err_qp:
-	srp_destroy_qp(ch);
+	srp_destroy_qp(qp);
 
 err_send_cq:
 	ib_free_cq(send_cq);
@@ -596,7 +596,7 @@ static void srp_free_ch_ib(struct srp_target_port *target,
 			ib_destroy_fmr_pool(ch->fmr_pool);
 	}
 
-	srp_destroy_qp(ch);
+	srp_destroy_qp(ch->qp);
 	ib_free_cq(ch->send_cq);
 	ib_free_cq(ch->recv_cq);
 
-- 
2.8.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/6] IB/srp: Fix srp_create_target() error handling

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1222 bytes --]

On Tue, May 10, 2016 at 05:13:52PM -0700, Bart Van Assche wrote:
> Avoid that the following kernel oops occurs if memory pool
> allocation fails:
> 
> BUG: unable to handle kernel NULL pointer dereference at (null)
> IP: [<ffffffffa048d0a0>] ib_drain_rq+0x0/0x20 [ib_core]
> Call Trace:
>  [<ffffffffa04af386>] srp_create_target+0xca6/0x13a9 [ib_srp]
>  [<ffffffff813cc863>] dev_attr_store+0x13/0x20
>  [<ffffffff81214b50>] sysfs_kf_write+0x40/0x50
>  [<ffffffff81213f1c>] kernfs_fop_write+0x13c/0x180
>  [<ffffffff81197683>] __vfs_write+0x23/0xf0
>  [<ffffffff81198744>] vfs_write+0xa4/0x1a0
>  [<ffffffff81199a44>] SyS_write+0x44/0xa0
>  [<ffffffff8159e3e9>] entry_SYSCALL_64_fastpath+0x1c/0xac
> 
> Fixes: 1dc7b1f10dcb ("IB/srp: use the new CQ API")
> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # v4.5+

Looks good,
Reviewed-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[PATCH 4/6] IB/core: Enhance ib_map_mr_sg()

[Bart Van Assche]

The SRP initiator allows to set max_sectors to a value that exceeds
the largest amount of data that can be mapped at once with an mlx4
HCA using fast registration and a page size of 4 KB. Hence modify
ib_map_mr_sg() such that it can map partial sg-elements. If an
sg-element has been mapped partially, let the caller know
which fraction has been mapped by adjusting *sg_offset.

Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 drivers/infiniband/core/rw.c                |  2 +-
 drivers/infiniband/core/verbs.c             | 26 +++++++++++++++++++++-----
 drivers/infiniband/hw/cxgb3/iwch_provider.c |  2 +-
 drivers/infiniband/hw/cxgb4/iw_cxgb4.h      |  2 +-
 drivers/infiniband/hw/cxgb4/mem.c           |  2 +-
 drivers/infiniband/hw/i40iw/i40iw_verbs.c   |  2 +-
 drivers/infiniband/hw/mlx4/mlx4_ib.h        |  2 +-
 drivers/infiniband/hw/mlx4/mr.c             |  2 +-
 drivers/infiniband/hw/mlx5/mlx5_ib.h        |  2 +-
 drivers/infiniband/hw/mlx5/mr.c             |  8 ++++++--
 drivers/infiniband/hw/nes/nes_verbs.c       |  2 +-
 drivers/infiniband/hw/ocrdma/ocrdma_verbs.c |  2 +-
 drivers/infiniband/hw/ocrdma/ocrdma_verbs.h |  2 +-
 drivers/infiniband/ulp/iser/iser_memory.c   |  4 ++--
 drivers/infiniband/ulp/srp/ib_srp.c         |  2 +-
 include/rdma/ib_verbs.h                     |  8 ++++----
 net/sunrpc/xprtrdma/frwr_ops.c              |  2 +-
 net/sunrpc/xprtrdma/svc_rdma_recvfrom.c     |  2 +-
 18 files changed, 47 insertions(+), 27 deletions(-)

diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c
index 6fc50bf..1eb9b12 100644
--- a/drivers/infiniband/core/rw.c
+++ b/drivers/infiniband/core/rw.c
@@ -92,7 +92,7 @@ static int rdma_rw_init_one_mr(struct ib_qp *qp, u8 port_num,
 		reg->inv_wr.next = NULL;
 	}
 
-	ret = ib_map_mr_sg(reg->mr, sg, nents, offset, PAGE_SIZE);
+	ret = ib_map_mr_sg(reg->mr, sg, nents, &offset, PAGE_SIZE);
 	if (ret < nents) {
 		ib_mr_pool_put(qp, &qp->rdma_mrs, reg->mr);
 		return -EINVAL;
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index 3d7b266..ffb9863 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -1655,7 +1655,7 @@ EXPORT_SYMBOL(ib_set_vf_guid);
  * is ready for registration.
  */
 int ib_map_mr_sg(struct ib_mr *mr, struct scatterlist *sg, int sg_nents,
-		unsigned int sg_offset, unsigned int page_size)
+		 unsigned int *sg_offset, unsigned int page_size)
 {
 	if (unlikely(!mr->device->map_mr_sg))
 		return -ENOSYS;
@@ -1672,7 +1672,10 @@ EXPORT_SYMBOL(ib_map_mr_sg);
  * @mr:            memory region
  * @sgl:           dma mapped scatterlist
  * @sg_nents:      number of entries in sg
- * @sg_offset:     offset in bytes into sg
+ * @sg_offset_p:   IN:  start offset in bytes into sg
+ *                 OUT: offset in bytes for element n of the sg of the first
+ *                      byte that has not been processed where n is the return
+ *                      value of this function.
  * @set_page:      driver page assignment function pointer
  *
  * Core service helper for drivers to convert the largest
@@ -1684,19 +1687,24 @@ EXPORT_SYMBOL(ib_map_mr_sg);
  * a page vector.
  */
 int ib_sg_to_pages(struct ib_mr *mr, struct scatterlist *sgl, int sg_nents,
-		unsigned int sg_offset, int (*set_page)(struct ib_mr *, u64))
+		unsigned int *sg_offset_p, int (*set_page)(struct ib_mr *, u64))
 {
 	struct scatterlist *sg;
 	u64 last_end_dma_addr = 0;
+	unsigned int sg_offset = sg_offset_p ? *sg_offset_p : 0;
 	unsigned int last_page_off = 0;
 	u64 page_mask = ~((u64)mr->page_size - 1);
 	int i, ret;
 
+	if (unlikely(sg_nents <= 0 || sg_offset > sg_dma_len(&sgl[0])))
+		return -EINVAL;
+
 	mr->iova = sg_dma_address(&sgl[0]) + sg_offset;
 	mr->length = 0;
 
 	for_each_sg(sgl, sg, sg_nents, i) {
 		u64 dma_addr = sg_dma_address(sg) + sg_offset;
+		u64 prev_addr = dma_addr;
 		unsigned int dma_len = sg_dma_len(sg) - sg_offset;
 		u64 end_dma_addr = dma_addr + dma_len;
 		u64 page_addr = dma_addr & page_mask;
@@ -1721,8 +1729,14 @@ int ib_sg_to_pages(struct ib_mr *mr, struct scatterlist *sgl, int sg_nents,
 
 		do {
 			ret = set_page(mr, page_addr);
-			if (unlikely(ret < 0))
-				return i ? : ret;
+			if (unlikely(ret < 0)) {
+				sg_offset = prev_addr - dma_addr;
+				mr->length += sg_offset;
+				if (sg_offset_p)
+					*sg_offset_p = sg_offset;
+				return i || sg_offset ? i : ret;
+			}
+			prev_addr = page_addr;
 next_page:
 			page_addr += mr->page_size;
 		} while (page_addr < end_dma_addr);
@@ -1734,6 +1748,8 @@ next_page:
 		sg_offset = 0;
 	}
 
+	if (sg_offset_p)
+		*sg_offset_p = 0;
 	return i;
 }
 EXPORT_SYMBOL(ib_sg_to_pages);
diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c
index 608aa0c..47cb927 100644
--- a/drivers/infiniband/hw/cxgb3/iwch_provider.c
+++ b/drivers/infiniband/hw/cxgb3/iwch_provider.c
@@ -784,7 +784,7 @@ static int iwch_set_page(struct ib_mr *ibmr, u64 addr)
 }
 
 static int iwch_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg,
-		int sg_nents, unsigned sg_offset)
+			  int sg_nents, unsigned int *sg_offset)
 {
 	struct iwch_mr *mhp = to_iwch_mr(ibmr);
 
diff --git a/drivers/infiniband/hw/cxgb4/iw_cxgb4.h b/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
index 067cb3f..1ff3ba8 100644
--- a/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
+++ b/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
@@ -918,7 +918,7 @@ struct ib_mr *c4iw_alloc_mr(struct ib_pd *pd,
 			    enum ib_mr_type mr_type,
 			    u32 max_num_sg);
 int c4iw_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
-		unsigned int sg_offset);
+		   unsigned int *sg_offset);
 int c4iw_dealloc_mw(struct ib_mw *mw);
 struct ib_mw *c4iw_alloc_mw(struct ib_pd *pd, enum ib_mw_type type,
 			    struct ib_udata *udata);
diff --git a/drivers/infiniband/hw/cxgb4/mem.c b/drivers/infiniband/hw/cxgb4/mem.c
index 38afb3d..83960df 100644
--- a/drivers/infiniband/hw/cxgb4/mem.c
+++ b/drivers/infiniband/hw/cxgb4/mem.c
@@ -691,7 +691,7 @@ static int c4iw_set_page(struct ib_mr *ibmr, u64 addr)
 }
 
 int c4iw_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
-		unsigned int sg_offset)
+		   unsigned int *sg_offset)
 {
 	struct c4iw_mr *mhp = to_c4iw_mr(ibmr);
 
diff --git a/drivers/infiniband/hw/i40iw/i40iw_verbs.c b/drivers/infiniband/hw/i40iw/i40iw_verbs.c
index 141eaba..4a740f7 100644
--- a/drivers/infiniband/hw/i40iw/i40iw_verbs.c
+++ b/drivers/infiniband/hw/i40iw/i40iw_verbs.c
@@ -1574,7 +1574,7 @@ static int i40iw_set_page(struct ib_mr *ibmr, u64 addr)
  * @sg_nents: number of sg pages
  */
 static int i40iw_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg,
-		int sg_nents, unsigned int sg_offset)
+			   int sg_nents, unsigned int *sg_offset)
 {
 	struct i40iw_mr *iwmr = to_iwmr(ibmr);
 
diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
index ba328177..6c5ac5d 100644
--- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
+++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
@@ -718,7 +718,7 @@ struct ib_mr *mlx4_ib_alloc_mr(struct ib_pd *pd,
 			       enum ib_mr_type mr_type,
 			       u32 max_num_sg);
 int mlx4_ib_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
-		unsigned int sg_offset);
+		      unsigned int *sg_offset);
 int mlx4_ib_modify_cq(struct ib_cq *cq, u16 cq_count, u16 cq_period);
 int mlx4_ib_resize_cq(struct ib_cq *ibcq, int entries, struct ib_udata *udata);
 struct ib_cq *mlx4_ib_create_cq(struct ib_device *ibdev,
diff --git a/drivers/infiniband/hw/mlx4/mr.c b/drivers/infiniband/hw/mlx4/mr.c
index b04f623..6312721 100644
--- a/drivers/infiniband/hw/mlx4/mr.c
+++ b/drivers/infiniband/hw/mlx4/mr.c
@@ -529,7 +529,7 @@ static int mlx4_set_page(struct ib_mr *ibmr, u64 addr)
 }
 
 int mlx4_ib_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
-		unsigned int sg_offset)
+		      unsigned int *sg_offset)
 {
 	struct mlx4_ib_mr *mr = to_mmr(ibmr);
 	int rc;
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 8c835b2..f05cf57 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -713,7 +713,7 @@ struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd,
 			       enum ib_mr_type mr_type,
 			       u32 max_num_sg);
 int mlx5_ib_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
-		unsigned int sg_offset);
+		      unsigned int *sg_offset);
 int mlx5_ib_process_mad(struct ib_device *ibdev, int mad_flags, u8 port_num,
 			const struct ib_wc *in_wc, const struct ib_grh *in_grh,
 			const struct ib_mad_hdr *in, size_t in_mad_size,
diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index b678eac..8cf2ce5 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1752,10 +1752,11 @@ static int
 mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
 		   struct scatterlist *sgl,
 		   unsigned short sg_nents,
-		   unsigned int sg_offset)
+		   unsigned int *sg_offset_p)
 {
 	struct scatterlist *sg = sgl;
 	struct mlx5_klm *klms = mr->descs;
+	unsigned int sg_offset = sg_offset_p ? *sg_offset_p : 0;
 	u32 lkey = mr->ibmr.pd->local_dma_lkey;
 	int i;
 
@@ -1774,6 +1775,9 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
 		sg_offset = 0;
 	}
 
+	if (sg_offset_p)
+		*sg_offset_p = sg_offset;
+
 	return i;
 }
 
@@ -1792,7 +1796,7 @@ static int mlx5_set_page(struct ib_mr *ibmr, u64 addr)
 }
 
 int mlx5_ib_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
-		unsigned int sg_offset)
+		      unsigned int *sg_offset)
 {
 	struct mlx5_ib_mr *mr = to_mmr(ibmr);
 	int n;
diff --git a/drivers/infiniband/hw/nes/nes_verbs.c b/drivers/infiniband/hw/nes/nes_verbs.c
index 698aab6..4ebea4c 100644
--- a/drivers/infiniband/hw/nes/nes_verbs.c
+++ b/drivers/infiniband/hw/nes/nes_verbs.c
@@ -403,7 +403,7 @@ static int nes_set_page(struct ib_mr *ibmr, u64 addr)
 }
 
 static int nes_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg,
-		int sg_nents, unsigned int sg_offset)
+			 int sg_nents, unsigned int *sg_offset)
 {
 	struct nes_mr *nesmr = to_nesmr(ibmr);
 
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
index 9ddd550..b1a3d91 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
@@ -3082,7 +3082,7 @@ static int ocrdma_set_page(struct ib_mr *ibmr, u64 addr)
 }
 
 int ocrdma_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
-		unsigned int sg_offset)
+		     unsigned int *sg_offset)
 {
 	struct ocrdma_mr *mr = get_ocrdma_mr(ibmr);
 
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.h b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.h
index b290e5d..704ef1e 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.h
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.h
@@ -123,6 +123,6 @@ struct ib_mr *ocrdma_alloc_mr(struct ib_pd *pd,
 			      enum ib_mr_type mr_type,
 			      u32 max_num_sg);
 int ocrdma_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
-		unsigned sg_offset);
+		     unsigned int *sg_offset);
 
 #endif				/* __OCRDMA_VERBS_H__ */
diff --git a/drivers/infiniband/ulp/iser/iser_memory.c b/drivers/infiniband/ulp/iser/iser_memory.c
index 44cc85f..90be568 100644
--- a/drivers/infiniband/ulp/iser/iser_memory.c
+++ b/drivers/infiniband/ulp/iser/iser_memory.c
@@ -236,7 +236,7 @@ int iser_fast_reg_fmr(struct iscsi_iser_task *iser_task,
 	page_vec->npages = 0;
 	page_vec->fake_mr.page_size = SIZE_4K;
 	plen = ib_sg_to_pages(&page_vec->fake_mr, mem->sg,
-			      mem->size, 0, iser_set_page);
+			      mem->size, NULL, iser_set_page);
 	if (unlikely(plen < mem->size)) {
 		iser_err("page vec too short to hold this SG\n");
 		iser_data_buf_dump(mem, device->ib_device);
@@ -446,7 +446,7 @@ static int iser_fast_reg_mr(struct iscsi_iser_task *iser_task,
 
 	ib_update_fast_reg_key(mr, ib_inc_rkey(mr->rkey));
 
-	n = ib_map_mr_sg(mr, mem->sg, mem->size, 0, SIZE_4K);
+	n = ib_map_mr_sg(mr, mem->sg, mem->size, NULL, SIZE_4K);
 	if (unlikely(n != mem->size)) {
 		iser_err("failed to map sg (%d/%d)\n",
 			 n, mem->size);
diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index f4dc6f9..6440469 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -1329,7 +1329,7 @@ static int srp_map_finish_fr(struct srp_map_state *state,
 	rkey = ib_inc_rkey(desc->mr->rkey);
 	ib_update_fast_reg_key(desc->mr, rkey);
 
-	n = ib_map_mr_sg(desc->mr, state->sg, sg_nents, 0, dev->mr_page_size);
+	n = ib_map_mr_sg(desc->mr, state->sg, sg_nents, NULL, dev->mr_page_size);
 	if (unlikely(n < 0)) {
 		srp_fr_pool_put(ch->fr_pool, &desc, 1);
 		pr_debug("%s: ib_map_mr_sg(%d) returned %d.\n",
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 544c55b..56bb0f3 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -1848,7 +1848,7 @@ struct ib_device {
 	int                        (*map_mr_sg)(struct ib_mr *mr,
 						struct scatterlist *sg,
 						int sg_nents,
-						unsigned sg_offset);
+						unsigned int *sg_offset);
 	struct ib_mw *             (*alloc_mw)(struct ib_pd *pd,
 					       enum ib_mw_type type,
 					       struct ib_udata *udata);
@@ -3145,11 +3145,11 @@ struct net_device *ib_get_net_dev_by_params(struct ib_device *dev, u8 port,
 					    const struct sockaddr *addr);
 
 int ib_map_mr_sg(struct ib_mr *mr, struct scatterlist *sg, int sg_nents,
-		unsigned int sg_offset, unsigned int page_size);
+		 unsigned int *sg_offset, unsigned int page_size);
 
 static inline int
 ib_map_mr_sg_zbva(struct ib_mr *mr, struct scatterlist *sg, int sg_nents,
-		unsigned int sg_offset, unsigned int page_size)
+		  unsigned int *sg_offset, unsigned int page_size)
 {
 	int n;
 
@@ -3160,7 +3160,7 @@ ib_map_mr_sg_zbva(struct ib_mr *mr, struct scatterlist *sg, int sg_nents,
 }
 
 int ib_sg_to_pages(struct ib_mr *mr, struct scatterlist *sgl, int sg_nents,
-		unsigned int sg_offset, int (*set_page)(struct ib_mr *, u64));
+		unsigned int *sg_offset, int (*set_page)(struct ib_mr *, u64));
 
 void ib_drain_rq(struct ib_qp *qp);
 void ib_drain_sq(struct ib_qp *qp);
diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c
index 3274a4a..94c3fa9 100644
--- a/net/sunrpc/xprtrdma/frwr_ops.c
+++ b/net/sunrpc/xprtrdma/frwr_ops.c
@@ -421,7 +421,7 @@ frwr_op_map(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr_seg *seg,
 		return -ENOMEM;
 	}
 
-	n = ib_map_mr_sg(mr, frmr->sg, frmr->sg_nents, 0, PAGE_SIZE);
+	n = ib_map_mr_sg(mr, frmr->sg, frmr->sg_nents, NULL, PAGE_SIZE);
 	if (unlikely(n != frmr->sg_nents)) {
 		pr_err("RPC:       %s: failed to map mr %p (%u/%u)\n",
 		       __func__, frmr->fr_mr, n, frmr->sg_nents);
diff --git a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
index 19a74e9..fbe7444 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
@@ -281,7 +281,7 @@ int rdma_read_chunk_frmr(struct svcxprt_rdma *xprt,
 	}
 	atomic_inc(&xprt->sc_dma_used);
 
-	n = ib_map_mr_sg(frmr->mr, frmr->sg, frmr->sg_nents, 0, PAGE_SIZE);
+	n = ib_map_mr_sg(frmr->mr, frmr->sg, frmr->sg_nents, NULL, PAGE_SIZE);
 	if (unlikely(n != frmr->sg_nents)) {
 		pr_err("svcrdma: failed to map mr %p (%d/%d elements)\n",
 		       frmr->mr, n, frmr->sg_nents);
-- 
2.8.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1445 bytes --]

On Tue, May 10, 2016 at 05:14:57PM -0700, Bart Van Assche wrote:
> The SRP initiator allows to set max_sectors to a value that exceeds
> the largest amount of data that can be mapped at once with an mlx4
> HCA using fast registration and a page size of 4 KB. Hence modify
> ib_map_mr_sg() such that it can map partial sg-elements. If an
> sg-element has been mapped partially, let the caller know
> which fraction has been mapped by adjusting *sg_offset.
> 
> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> ---
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1752,10 +1752,11 @@ static int
>  mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
>  		   struct scatterlist *sgl,
>  		   unsigned short sg_nents,
> -		   unsigned int sg_offset)
> +		   unsigned int *sg_offset_p)
>  {

I wonder on which tree are you basing?
In Linus (4.6-rc7) the function signature is different [1], the same
goes for my tree and Doug's for-4.7 branch [2].

[1] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/hw/mlx5/mr.c#n1752
[2] https://github.com/dledford/linux/blob/k.o/for-4.7/drivers/infiniband/hw/mlx5/mr.c#L1752

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()

[Bart Van Assche]

On 05/11/2016 12:54 AM, Leon Romanovsky wrote:
> On Tue, May 10, 2016 at 05:14:57PM -0700, Bart Van Assche wrote:
>> The SRP initiator allows to set max_sectors to a value that exceeds
>> the largest amount of data that can be mapped at once with an mlx4
>> HCA using fast registration and a page size of 4 KB. Hence modify
>> ib_map_mr_sg() such that it can map partial sg-elements. If an
>> sg-element has been mapped partially, let the caller know
>> which fraction has been mapped by adjusting *sg_offset.
>>
>> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
>> Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
>> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
>> Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>> ---
>> --- a/drivers/infiniband/hw/mlx5/mr.c
>> +++ b/drivers/infiniband/hw/mlx5/mr.c
>> @@ -1752,10 +1752,11 @@ static int
>>   mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
>>   		   struct scatterlist *sgl,
>>   		   unsigned short sg_nents,
>> -		   unsigned int sg_offset)
>> +		   unsigned int *sg_offset_p)
>>   {
>
> I wonder on which tree are you basing?
> In Linus (4.6-rc7) the function signature is different [1], the same
> goes for my tree and Doug's for-4.7 branch [2].

Hello Leon,

Sorry that I hadn't mentioned this explicitly in the cover letter of 
this patch series but this patch series is based on Christoph's generic 
RDMA READ/WRITE API work.

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()

[Laurence Oberman]

----- Original Message -----
> From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Christoph Hellwig" <hch-jcswGhMUV9g@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
> "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Or Gerlitz" <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Sent: Wednesday, May 11, 2016 11:22:29 AM
> Subject: Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()
> 
> On 05/11/2016 12:54 AM, Leon Romanovsky wrote:
> > On Tue, May 10, 2016 at 05:14:57PM -0700, Bart Van Assche wrote:
> >> The SRP initiator allows to set max_sectors to a value that exceeds
> >> the largest amount of data that can be mapped at once with an mlx4
> >> HCA using fast registration and a page size of 4 KB. Hence modify
> >> ib_map_mr_sg() such that it can map partial sg-elements. If an
> >> sg-element has been mapped partially, let the caller know
> >> which fraction has been mapped by adjusting *sg_offset.
> >>
> >> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> >> Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
> >> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> >> Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> >> ---
> >> --- a/drivers/infiniband/hw/mlx5/mr.c
> >> +++ b/drivers/infiniband/hw/mlx5/mr.c
> >> @@ -1752,10 +1752,11 @@ static int
> >>   mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
> >>   		   struct scatterlist *sgl,
> >>   		   unsigned short sg_nents,
> >> -		   unsigned int sg_offset)
> >> +		   unsigned int *sg_offset_p)
> >>   {
> >
> > I wonder on which tree are you basing?
> > In Linus (4.6-rc7) the function signature is different [1], the same
> > goes for my tree and Doug's for-4.7 branch [2].
> 
> Hello Leon,
> 
> Sorry that I hadn't mentioned this explicitly in the cover letter of
> this patch series but this patch series is based on Christoph's generic
> RDMA READ/WRITE API work.
> 
> Bart.
> 
> 

I chased that for a while too.:)
Landed up pulling the latest next, applying all of Christoph's 11 RDMA patches, then the first 11 of Barts and the latest 6.
I had to hand fix some stuff.
Kernel is building now for testing :)


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()

[Bart Van Assche]

On 05/11/2016 08:31 AM, Laurence Oberman wrote:
> I chased that for a while too.:)
> Landed up pulling the latest next, applying all of Christoph's 11 RDMA patches, then the first 11 of Barts and the latest 6.
> I had to hand fix some stuff.
> Kernel is building now for testing :)

Hello Laurence,

Please wait with starting your tests until I have made a kernel tree 
with this patch series available.

Thanks,

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()

[Laurence Oberman]

----- Original Message -----
> From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Christoph Hellwig" <hch-jcswGhMUV9g@public.gmane.org>, "Sagi Grimberg"
> <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Or Gerlitz" <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Sent: Wednesday, May 11, 2016 11:41:39 AM
> Subject: Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()
> 
> On 05/11/2016 08:31 AM, Laurence Oberman wrote:
> > I chased that for a while too.:)
> > Landed up pulling the latest next, applying all of Christoph's 11 RDMA
> > patches, then the first 11 of Barts and the latest 6.
> > I had to hand fix some stuff.
> > Kernel is building now for testing :)
> 
> Hello Laurence,
> 
> Please wait with starting your tests until I have made a kernel tree
> with this patch series available.
> 
> Thanks,
> 
> Bart.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hello Bart

I had started already, and its looking awesomely stable so far.
Awesome work from all of you guys.

### RECORD   84 >>> jumpclient <<< (1462981973.001) (Wed May 11 11:52:53 2016) ###
# DISK STATISTICS (/sec)
#                   <---------reads---------><---------writes---------><--------averages--------> Pct
#Time     Name       KBytes Merged  IOs Size  KBytes Merged  IOs Size  RWSize  QLen  Wait SvcTim Util
11:52:53 sdc              0      0    0    0  163840      0   40 4096    4096     1    10     10   42
11:52:53 dm-6             0      0    0    0  327680    320   80 4096    4096     1    11     11   90
11:52:53 sdd              0      0    0    0  176128      0   43 4096    4096     1    10     10   44
11:52:53 dm-7             0      0    0    0  348160    336   85 4096    4096     1    11     10   92
11:52:53 sde              0      0    0    0  159744      0   39 4096    4096     1    11     11   43
11:52:53 dm-8             0      0    0    0  319488    312   78 4096    4096     1    11     11   89
11:52:53 sdf              4      0    1    4  167936      0   41 4096    3998     1    10     10   44
11:52:53 sdg              4      0    1    4  163840      0   40 4096    3996     1    10     10   44
11:52:53 dm-9             0      0    0    0  335872    328   82 4096    4096     1    11     11   91
11:52:53 dm-10            0      0    0    0  331776    324   81 4096    4096     1    11     11   91
11:52:53 sdh              4      0    1    4  159744      0   39 4096    3993     1    11     11   45
11:52:53 dm-11            0      0    0    0  319488    308   78 4096    4096     1    11     11   91
11:52:53 sdi              0      0    0    0  167936      0   41 4096    4096     1    10     10   43
11:52:53 dm-12            0      0    0    0  335872    328   82 4096    4096     1    11     11   93
11:52:53 sdj              0      0    0    0  172032      0   42 4096    4096     1    10     10   44
11:52:53 dm-13            0      0    0    0  344064    332   84 4096    4096     1    10     10   91
11:52:53 sdk              0      0    0    0  176128      0   43 4096    4096     1    11     11   47
11:52:53 dm-14            0      0    0    0  352256    344   86 4096    4096     1    10     10   91
11:52:53 sdl              0      0    0    0  163840      0   40 4096    4096     1    10     11   43
11:52:53 dm-15            0      0    0    0  331776    324   81 4096    4096     1    11     11   91
11:52:53 sdm              0      0    0    0  163840      0   40 4096    4096     1    11     11   45
11:52:53 sdn              0      0    0    0  172032      0   42 4096    4096     1    10     10   45
11:52:53 sdo              0      0    0    0  159744      0   39 4096    4096     1    11     11   44
11:52:53 sdp              4      0    1    4  167936      0   41 4096    3998     1    10     10   45
11:52:53 sdq              0      0    0    0  167936      0   41 4096    4096     1    11     11   45
11:52:53 sdr              4      0    1    4  159744      0   39 4096    3993     1    11     10   43
11:52:53 sds              0      0    0    0  167936      0   41 4096    4096     1    11     11   46
11:52:53 sdt              0      0    0    0  172032      0   42 4096    4096     1    10     10   44
11:52:53 sdu              0      0    0    0  176128      0   43 4096    4096     1     9      9   42
11:52:53 sdv              0      0    0    0  167936      0   41 4096    4096     1    11     11   45

Whoop, 3.2GBytes/sec and no errors :)

#         <----CPU[HYPER]-----><----------Disks-----------><----------Network---------->
#Time     cpu sys inter  ctxsw KBRead  Reads KBWrit Writes   KBIn  PktIn  KBOut  PktOut 
11:55:21    7   7  2652   7721     12      3  3223K    974      0      3      0       3 
11:55:22    7   7  2714   7984      4      1  3336K    834      0      1      0       1 
11:55:23    6   6  2545   7698      0      0  3216K    804      0      1      0       1 
11:55:24    7   7  2576   7455      0      0  3012K    758      0      3      0       1 
11:55:25    6   6  2717   8096     24      6  3314K    900      0      1      0       1 
11:55:26    7   7  2651   7807      0      0  3118K    955      1      9      2      11 
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()

[Laurence Oberman]

----- Original Message -----
> From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Christoph Hellwig" <hch-jcswGhMUV9g@public.gmane.org>, "Sagi Grimberg"
> <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Or Gerlitz" <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Sent: Wednesday, May 11, 2016 11:41:39 AM
> Subject: Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()
> 
> On 05/11/2016 08:31 AM, Laurence Oberman wrote:
> > I chased that for a while too.:)
> > Landed up pulling the latest next, applying all of Christoph's 11 RDMA
> > patches, then the first 11 of Barts and the latest 6.
> > I had to hand fix some stuff.
> > Kernel is building now for testing :)
> 
> Hello Laurence,
> 
> Please wait with starting your tests until I have made a kernel tree
> with this patch series available.
> 
> Thanks,
> 
> Bart.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

For Barts latest set of patches see subject, using Barts's tree the mapping failures are gone and its run for over 24 hours stable.
This is with multiple parallel reads of 4MB issued direct and multiple parallel writes to the same mpath devices issued buffered.
The only variation from what I am used to seeing (when its not failing :) on prior ib_srp) is that the I/O sizes reach 4MB often but are also often smaller.
Some of this could be issues with my LIO target and next week will have an enterprise array directly connected I will be testing with.

Its a huge improvement and seems good to me as I know these failures very well.

Tested-by: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

Example with direct reads and buffered writes

mpath view

### RECORD  556 >>> jumpclient <<< (1463070126.001) (Thu May 12 12:22:06 2016) ###
# DISK STATISTICS (/sec)
#                   <---------reads---------><---------writes---------><--------averages--------> Pct
#Time     Name       KBytes Merged  IOs Size  KBytes Merged  IOs Size  RWSize  QLen  Wait SvcTim Util
12:22:06 dm-6        212992    128  247  862  130200     64  126 1033     920     7    20      2   99
12:22:06 dm-7        213180     52  885  241  118784     29  406  293     257    25    20      0   99
12:22:06 dm-8        217088     53  424  512  122880     60  270  455     489    14    21      1   99
12:22:06 dm-9        211968     52  465  456  119760     30  409  293     379    19    22      1   99
12:22:06 dm-10       212992     52  364  585  121340     60  146  831     655    10    20      1   99
12:22:06 dm-11       221184    162   54 4096  135168     33  297  455    1015     9    26      2   99  *** Reads reach 4MB here, often smaller though
12:22:06 dm-12       229376    168  280  819  126976     93   62 2048    1041     6    17      2   99
12:22:06 dm-13       229376    168  112 2048  131072     64  128 1024    1501     5    23      4   99
12:22:06 dm-14       225280    110  385  585  122880     29  185  664     610    11    20      1   99
12:22:06 dm-15       203912     50  549  371  118792     29  379  313     347    20    21      1   99

individual path view

### RECORD  556 >>> jumpclient <<< (1463070126.001) (Thu May 12 12:22:06 2016) ###
# DISK STATISTICS (/sec)
#                   <---------reads---------><---------writes---------><--------averages--------> Pct
#Time     Name       KBytes Merged  IOs Size  KBytes Merged  IOs Size  RWSize  QLen  Wait SvcTim Util
12:26:37 sdc         128092      0  348  368   44468      0  482   92     207    18    21      1   95
12:26:37 sdd         103476      0  461  224   62956      0  266  237     228    14    20      1   97
12:26:37 sde         118100      0  145  814   65488      0  127  516     674     5    19      3   97
12:26:37 sdf         121660      0  651  187   53960      0  277  195     189    17    18      1   97
12:26:37 sdg          87276      0  305  286   76020      0  191  398     329    10    20      1   97
12:26:37 sdh          98688      0  166  595   67400      0  554  122     230    17    23      1   95
12:26:37 sdi         197112      0  945  209   16208      0   81  200     207    16    15      0   99
12:26:37 sdj           1776      0   16  111  143360      0   35 4096    2845     1    24     18   95
12:26:37 sdk              0      0    0    0  139264      0  374  372     372    10    26      2   96
12:26:37 sdl          77600      0  572  136   76624      0  268  286     183    17    19      1   93
12:26:37 sdm          80804      0  264  306   74316      0  533  139     194    19    24      1   98
12:26:37 sdn         101324      0  439  231   64020      0  385  166     200    18    22      1   98
12:26:37 sdo         107180      0  130  824   73776      0  145  509     658     5    21      3   97
12:26:37 sdp          80788      0  389  208   69376      0  339  205     206    17    22      1   99
12:26:37 sdq         116704      0  390  299   55052      0  129  427     330    10    20      1   99
12:26:37 sdr         113280      0  249  455   53816      0  401  134     257    16    24      1   99
12:26:37 sds          13192      0   93  142  110768      0  291  381     322    10    26      2   95
12:26:37 sdt         165184      0 1197  138       0      0    0    0     137    23    18      0   99
12:26:37 sdu         241664      0   59 4096       0      0    0    0    4096     1    16     16   97
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Christoph Hellwig" <hch-jcswGhMUV9g@public.gmane.org>, "Sagi Grimberg"
> <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Or Gerlitz" <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Sent: Thursday, May 12, 2016 12:28:34 PM
> Subject: Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()
> 
> 
> 
> ----- Original Message -----
> > From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > Cc: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Christoph
> > Hellwig" <hch-jcswGhMUV9g@public.gmane.org>, "Sagi Grimberg"
> > <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Or Gerlitz"
> > <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > Sent: Wednesday, May 11, 2016 11:41:39 AM
> > Subject: Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()
> > 
> > On 05/11/2016 08:31 AM, Laurence Oberman wrote:
> > > I chased that for a while too.:)
> > > Landed up pulling the latest next, applying all of Christoph's 11 RDMA
> > > patches, then the first 11 of Barts and the latest 6.
> > > I had to hand fix some stuff.
> > > Kernel is building now for testing :)
> > 
> > Hello Laurence,
> > 
> > Please wait with starting your tests until I have made a kernel tree
> > with this patch series available.
> > 
> > Thanks,
> > 
> > Bart.
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> For Barts latest set of patches see subject, using Barts's tree the mapping
> failures are gone and its run for over 24 hours stable.
> This is with multiple parallel reads of 4MB issued direct and multiple
> parallel writes to the same mpath devices issued buffered.
> The only variation from what I am used to seeing (when its not failing :) on
> prior ib_srp) is that the I/O sizes reach 4MB often but are also often
> smaller.
> Some of this could be issues with my LIO target and next week will have an
> enterprise array directly connected I will be testing with.
> 
> Its a huge improvement and seems good to me as I know these failures very
> well.
> 
> Tested-by: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> 
> Example with direct reads and buffered writes
> 
> mpath view
> 
> ### RECORD  556 >>> jumpclient <<< (1463070126.001) (Thu May 12 12:22:06
> 2016) ###
> # DISK STATISTICS (/sec)
> #
> <---------reads---------><---------writes---------><--------averages-------->
> Pct
> #Time     Name       KBytes Merged  IOs Size  KBytes Merged  IOs Size  RWSize
> QLen  Wait SvcTim Util
> 12:22:06 dm-6        212992    128  247  862  130200     64  126 1033     920
> 7    20      2   99
> 12:22:06 dm-7        213180     52  885  241  118784     29  406  293     257
> 25    20      0   99
> 12:22:06 dm-8        217088     53  424  512  122880     60  270  455     489
> 14    21      1   99
> 12:22:06 dm-9        211968     52  465  456  119760     30  409  293     379
> 19    22      1   99
> 12:22:06 dm-10       212992     52  364  585  121340     60  146  831     655
> 10    20      1   99
> 12:22:06 dm-11       221184    162   54 4096  135168     33  297  455    1015
> 9    26      2   99  *** Reads reach 4MB here, often smaller though
> 12:22:06 dm-12       229376    168  280  819  126976     93   62 2048    1041
> 6    17      2   99
> 12:22:06 dm-13       229376    168  112 2048  131072     64  128 1024    1501
> 5    23      4   99
> 12:22:06 dm-14       225280    110  385  585  122880     29  185  664     610
> 11    20      1   99
> 12:22:06 dm-15       203912     50  549  371  118792     29  379  313     347
> 20    21      1   99
> 
> individual path view
> 
> ### RECORD  556 >>> jumpclient <<< (1463070126.001) (Thu May 12 12:22:06
> 2016) ###
> # DISK STATISTICS (/sec)
> #
> <---------reads---------><---------writes---------><--------averages-------->
> Pct
> #Time     Name       KBytes Merged  IOs Size  KBytes Merged  IOs Size  RWSize
> QLen  Wait SvcTim Util
> 12:26:37 sdc         128092      0  348  368   44468      0  482   92     207
> 18    21      1   95
> 12:26:37 sdd         103476      0  461  224   62956      0  266  237     228
> 14    20      1   97
> 12:26:37 sde         118100      0  145  814   65488      0  127  516     674
> 5    19      3   97
> 12:26:37 sdf         121660      0  651  187   53960      0  277  195     189
> 17    18      1   97
> 12:26:37 sdg          87276      0  305  286   76020      0  191  398     329
> 10    20      1   97
> 12:26:37 sdh          98688      0  166  595   67400      0  554  122     230
> 17    23      1   95
> 12:26:37 sdi         197112      0  945  209   16208      0   81  200     207
> 16    15      0   99
> 12:26:37 sdj           1776      0   16  111  143360      0   35 4096    2845
> 1    24     18   95
> 12:26:37 sdk              0      0    0    0  139264      0  374  372     372
> 10    26      2   96
> 12:26:37 sdl          77600      0  572  136   76624      0  268  286     183
> 17    19      1   93
> 12:26:37 sdm          80804      0  264  306   74316      0  533  139     194
> 19    24      1   98
> 12:26:37 sdn         101324      0  439  231   64020      0  385  166     200
> 18    22      1   98
> 12:26:37 sdo         107180      0  130  824   73776      0  145  509     658
> 5    21      3   97
> 12:26:37 sdp          80788      0  389  208   69376      0  339  205     206
> 17    22      1   99
> 12:26:37 sdq         116704      0  390  299   55052      0  129  427     330
> 10    20      1   99
> 12:26:37 sdr         113280      0  249  455   53816      0  401  134     257
> 16    24      1   99
> 12:26:37 sds          13192      0   93  142  110768      0  291  381     322
> 10    26      2   95
> 12:26:37 sdt         165184      0 1197  138       0      0    0    0     137
> 23    18      0   99
> 12:26:37 sdu         241664      0   59 4096       0      0    0    0    4096
> 1    16     16   97
> 

Bart,
I meant to mention that that was with ib_srp untuned.
My next set of tests will be with indirect_sg_entries=512 and cmd_sg_entries=64 for a start.
Then I will max them out and see how we do.

Thanks!!!
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Christoph Hellwig" <hch-jcswGhMUV9g@public.gmane.org>, "Sagi Grimberg"
> <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Or Gerlitz" <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Sent: Thursday, May 12, 2016 12:38:13 PM
> Subject: Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()
> 
> 
> 
> ----- Original Message -----
> > From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > Cc: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Christoph
> > Hellwig" <hch-jcswGhMUV9g@public.gmane.org>, "Sagi Grimberg"
> > <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Or Gerlitz"
> > <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > Sent: Thursday, May 12, 2016 12:28:34 PM
> > Subject: Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > > To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > > Cc: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Christoph
> > > Hellwig" <hch-jcswGhMUV9g@public.gmane.org>, "Sagi Grimberg"
> > > <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Or Gerlitz"
> > > <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > Sent: Wednesday, May 11, 2016 11:41:39 AM
> > > Subject: Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()
> > > 
> > > On 05/11/2016 08:31 AM, Laurence Oberman wrote:
> > > > I chased that for a while too.:)
> > > > Landed up pulling the latest next, applying all of Christoph's 11 RDMA
> > > > patches, then the first 11 of Barts and the latest 6.
> > > > I had to hand fix some stuff.
> > > > Kernel is building now for testing :)
> > > 
> > > Hello Laurence,
> > > 
> > > Please wait with starting your tests until I have made a kernel tree
> > > with this patch series available.
> > > 
> > > Thanks,
> > > 
> > > Bart.
> > > 
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > 
> > For Barts latest set of patches see subject, using Barts's tree the mapping
> > failures are gone and its run for over 24 hours stable.
> > This is with multiple parallel reads of 4MB issued direct and multiple
> > parallel writes to the same mpath devices issued buffered.
> > The only variation from what I am used to seeing (when its not failing :)
> > on
> > prior ib_srp) is that the I/O sizes reach 4MB often but are also often
> > smaller.
> > Some of this could be issues with my LIO target and next week will have an
> > enterprise array directly connected I will be testing with.
> > 
> > Its a huge improvement and seems good to me as I know these failures very
> > well.
> > 
> > Tested-by: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > 
> > Example with direct reads and buffered writes
> > 
> > mpath view
> > 
> > ### RECORD  556 >>> jumpclient <<< (1463070126.001) (Thu May 12 12:22:06
> > 2016) ###
> > # DISK STATISTICS (/sec)
> > #
> > <---------reads---------><---------writes---------><--------averages-------->
> > Pct
> > #Time     Name       KBytes Merged  IOs Size  KBytes Merged  IOs Size
> > RWSize
> > QLen  Wait SvcTim Util
> > 12:22:06 dm-6        212992    128  247  862  130200     64  126 1033
> > 920
> > 7    20      2   99
> > 12:22:06 dm-7        213180     52  885  241  118784     29  406  293
> > 257
> > 25    20      0   99
> > 12:22:06 dm-8        217088     53  424  512  122880     60  270  455
> > 489
> > 14    21      1   99
> > 12:22:06 dm-9        211968     52  465  456  119760     30  409  293
> > 379
> > 19    22      1   99
> > 12:22:06 dm-10       212992     52  364  585  121340     60  146  831
> > 655
> > 10    20      1   99
> > 12:22:06 dm-11       221184    162   54 4096  135168     33  297  455
> > 1015
> > 9    26      2   99  *** Reads reach 4MB here, often smaller though
> > 12:22:06 dm-12       229376    168  280  819  126976     93   62 2048
> > 1041
> > 6    17      2   99
> > 12:22:06 dm-13       229376    168  112 2048  131072     64  128 1024
> > 1501
> > 5    23      4   99
> > 12:22:06 dm-14       225280    110  385  585  122880     29  185  664
> > 610
> > 11    20      1   99
> > 12:22:06 dm-15       203912     50  549  371  118792     29  379  313
> > 347
> > 20    21      1   99
> > 
> > individual path view
> > 
> > ### RECORD  556 >>> jumpclient <<< (1463070126.001) (Thu May 12 12:22:06
> > 2016) ###
> > # DISK STATISTICS (/sec)
> > #
> > <---------reads---------><---------writes---------><--------averages-------->
> > Pct
> > #Time     Name       KBytes Merged  IOs Size  KBytes Merged  IOs Size
> > RWSize
> > QLen  Wait SvcTim Util
> > 12:26:37 sdc         128092      0  348  368   44468      0  482   92
> > 207
> > 18    21      1   95
> > 12:26:37 sdd         103476      0  461  224   62956      0  266  237
> > 228
> > 14    20      1   97
> > 12:26:37 sde         118100      0  145  814   65488      0  127  516
> > 674
> > 5    19      3   97
> > 12:26:37 sdf         121660      0  651  187   53960      0  277  195
> > 189
> > 17    18      1   97
> > 12:26:37 sdg          87276      0  305  286   76020      0  191  398
> > 329
> > 10    20      1   97
> > 12:26:37 sdh          98688      0  166  595   67400      0  554  122
> > 230
> > 17    23      1   95
> > 12:26:37 sdi         197112      0  945  209   16208      0   81  200
> > 207
> > 16    15      0   99
> > 12:26:37 sdj           1776      0   16  111  143360      0   35 4096
> > 2845
> > 1    24     18   95
> > 12:26:37 sdk              0      0    0    0  139264      0  374  372
> > 372
> > 10    26      2   96
> > 12:26:37 sdl          77600      0  572  136   76624      0  268  286
> > 183
> > 17    19      1   93
> > 12:26:37 sdm          80804      0  264  306   74316      0  533  139
> > 194
> > 19    24      1   98
> > 12:26:37 sdn         101324      0  439  231   64020      0  385  166
> > 200
> > 18    22      1   98
> > 12:26:37 sdo         107180      0  130  824   73776      0  145  509
> > 658
> > 5    21      3   97
> > 12:26:37 sdp          80788      0  389  208   69376      0  339  205
> > 206
> > 17    22      1   99
> > 12:26:37 sdq         116704      0  390  299   55052      0  129  427
> > 330
> > 10    20      1   99
> > 12:26:37 sdr         113280      0  249  455   53816      0  401  134
> > 257
> > 16    24      1   99
> > 12:26:37 sds          13192      0   93  142  110768      0  291  381
> > 322
> > 10    26      2   95
> > 12:26:37 sdt         165184      0 1197  138       0      0    0    0
> > 137
> > 23    18      0   99
> > 12:26:37 sdu         241664      0   59 4096       0      0    0    0
> > 4096
> > 1    16     16   97
> > 
> 
> Bart,
> I meant to mention that that was with ib_srp untuned.
> My next set of tests will be with indirect_sg_entries=512 and
> cmd_sg_entries=64 for a start.
> Then I will max them out and see how we do.
> 
> Thanks!!!
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Replying to my own message

Tuning ib_srp and I am back to full 4MB as expected so now we should be all set.

Bart, and all, thanks for all the assistance with this.
Awesome work Bart on your part as always.

### RECORD    3 >>> jumpclient <<< (1463071707.001) (Thu May 12 12:48:27 2016) ###
# DISK STATISTICS (/sec)
#                   <---------reads---------><---------writes---------><--------averages--------> Pct
#Time     Name       KBytes Merged  IOs Size  KBytes Merged  IOs Size  RWSize  QLen  Wait SvcTim Util
12:48:27 dm-6             0      0    0    0  286720    284   70 4096    4096     1    12     12   89
12:48:27 dm-7             0      0    0    0  290816    284   71 4096    4096     1    12     12   89
12:48:27 dm-8             0      0    0    0  286720    280   70 4096    4096     1    12     12   89
12:48:27 dm-9             0      0    0    0  294912    288   72 4096    4096     1    13     12   93
12:48:27 dm-10            0      0    0    0  290816    284   71 4096    4096     1    13     13   93
12:48:27 dm-11            0      0    0    0  286720    284   70 4096    4096     1    12     12   88
12:48:27 dm-12            0      0    0    0  290816    284   71 4096    4096     1    12     12   89
12:48:27 dm-13            0      0    0    0  290816    288   71 4096    4096     1    12     12   90
12:48:27 dm-14            0      0    0    0  286720    280   70 4096    4096     1    12     12   89
12:48:27 dm-15            0      0    0    0  282624    280   69 4096    4096     1    12     12   88
12:48:27 sdm              0      0    0    0  143360      0   35 4096    4096     1    12     12   42
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()

[Bart Van Assche]

On 05/12/2016 09:50 AM, Laurence Oberman wrote:
> Tuning ib_srp and I am back to full 4MB as expected so now we should be all set.
>
> Bart, and all, thanks for all the assistance with this.
> Awesome work Bart on your part as always.

Hello Laurence,

Thank you for having tested this patch series so quickly. I assume that 
this means that I can add your Tested-by when I repost this patch series?

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()

[Laurence Oberman]

----- Original Message -----
> From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Christoph Hellwig" <hch-jcswGhMUV9g@public.gmane.org>, "Sagi Grimberg"
> <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Or Gerlitz" <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Sent: Thursday, May 12, 2016 1:00:20 PM
> Subject: Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg()
> 
> On 05/12/2016 09:50 AM, Laurence Oberman wrote:
> > Tuning ib_srp and I am back to full 4MB as expected so now we should be all
> > set.
> >
> > Bart, and all, thanks for all the assistance with this.
> > Awesome work Bart on your part as always.
> 
> Hello Laurence,
> 
> Thank you for having tested this patch series so quickly. I assume that
> this means that I can add your Tested-by when I repost this patch series?
> 
> Bart.
> 

Hello Bart

Absolutely.
And in fact now I am running with options ib_srp cmd_sg_entries=255 indirect_sg_entries=2048 and it singing along
I have never reached that before without issues.
Its rock solid from what I can see.
Thanks!!!
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 5/6] IB/srp: Swap two code blocks in srp_add_one()

[Bart Van Assche]

This patch does not change any functionality but makes the next
patch in this series easier to read.

Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
---
 drivers/infiniband/ulp/srp/ib_srp.c | 23 ++++++++++++-----------
 1 file changed, 12 insertions(+), 11 deletions(-)

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 6440469..e530a77 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -3430,17 +3430,6 @@ static void srp_add_one(struct ib_device *device)
 	if (!srp_dev)
 		return;
 
-	srp_dev->has_fmr = (device->alloc_fmr && device->dealloc_fmr &&
-			    device->map_phys_fmr && device->unmap_fmr);
-	srp_dev->has_fr = (device->attrs.device_cap_flags &
-			   IB_DEVICE_MEM_MGT_EXTENSIONS);
-	if (!srp_dev->has_fmr && !srp_dev->has_fr)
-		dev_warn(&device->dev, "neither FMR nor FR is supported\n");
-
-	srp_dev->use_fast_reg = (srp_dev->has_fr &&
-				 (!srp_dev->has_fmr || prefer_fr));
-	srp_dev->use_fmr = !srp_dev->use_fast_reg && srp_dev->has_fmr;
-
 	/*
 	 * Use the smallest page size supported by the HCA, down to a
 	 * minimum of 4096 bytes. We're unlikely to build large sglists
@@ -3453,6 +3442,18 @@ static void srp_add_one(struct ib_device *device)
 	do_div(max_pages_per_mr, srp_dev->mr_page_size);
 	srp_dev->max_pages_per_mr = min_t(u64, SRP_MAX_PAGES_PER_MR,
 					  max_pages_per_mr);
+
+	srp_dev->has_fmr = (device->alloc_fmr && device->dealloc_fmr &&
+			    device->map_phys_fmr && device->unmap_fmr);
+	srp_dev->has_fr = (device->attrs.device_cap_flags &
+			   IB_DEVICE_MEM_MGT_EXTENSIONS);
+	if (!srp_dev->has_fmr && !srp_dev->has_fr)
+		dev_warn(&device->dev, "neither FMR nor FR is supported\n");
+
+	srp_dev->use_fast_reg = (srp_dev->has_fr &&
+				 (!srp_dev->has_fmr || prefer_fr));
+	srp_dev->use_fmr = !srp_dev->use_fast_reg && srp_dev->has_fmr;
+
 	if (srp_dev->use_fast_reg) {
 		srp_dev->max_pages_per_mr =
 			min_t(u32, srp_dev->max_pages_per_mr,
-- 
2.8.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 6/6] IB/srp: Prevent mapping failures

[Bart Van Assche]

If both max_sectors and the queue_depth are high enough it can
happen that the MR pool is depleted temporarily. This causes
the SRP initiator to report mapping failures. Although the SRP
initiator recovers from such mapping failures, prevent that
this can happen by allocating more memory regions.

Additionally, only enable memory registration if at least two
pages can be registered per memory region.

Reported-by: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
---
 drivers/infiniband/ulp/srp/ib_srp.c | 102 +++++++++++++++++++++++++++++-------
 drivers/infiniband/ulp/srp/ib_srp.h |   1 +
 2 files changed, 84 insertions(+), 19 deletions(-)

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index e530a77..075cd7a 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -468,7 +468,7 @@ static int srp_create_ch_ib(struct srp_rdma_ch *ch)
 	struct ib_qp *qp;
 	struct ib_fmr_pool *fmr_pool = NULL;
 	struct srp_fr_pool *fr_pool = NULL;
-	const int m = dev->use_fast_reg ? 3 : 1;
+	const int m = 1 + dev->use_fast_reg * target->mr_per_cmd * 2;
 	int ret;
 
 	init_attr = kzalloc(sizeof *init_attr, GFP_KERNEL);
@@ -849,7 +849,7 @@ static int srp_alloc_req_data(struct srp_rdma_ch *ch)
 
 	for (i = 0; i < target->req_ring_size; ++i) {
 		req = &ch->req_ring[i];
-		mr_list = kmalloc(target->cmd_sg_cnt * sizeof(void *),
+		mr_list = kmalloc(target->mr_per_cmd * sizeof(void *),
 				  GFP_KERNEL);
 		if (!mr_list)
 			goto out;
@@ -1298,9 +1298,16 @@ static void srp_reg_mr_err_done(struct ib_cq *cq, struct ib_wc *wc)
 	srp_handle_qp_err(cq, wc, "FAST REG");
 }
 
+/*
+ * Map up to sg_nents elements of state->sg where *sg_offset_p is the offset
+ * where to start in the first element. If sg_offset_p != NULL then
+ * *sg_offset_p is updated to the offset in state->sg[retval] of the first
+ * byte that has not yet been mapped.
+ */
 static int srp_map_finish_fr(struct srp_map_state *state,
 			     struct srp_request *req,
-			     struct srp_rdma_ch *ch, int sg_nents)
+			     struct srp_rdma_ch *ch, int sg_nents,
+			     unsigned int *sg_offset_p)
 {
 	struct srp_target_port *target = ch->target;
 	struct srp_device *dev = target->srp_host->srp_dev;
@@ -1316,9 +1323,13 @@ static int srp_map_finish_fr(struct srp_map_state *state,
 	WARN_ON_ONCE(!dev->use_fast_reg);
 
 	if (sg_nents == 1 && target->global_mr) {
-		srp_map_desc(state, sg_dma_address(state->sg),
-			     sg_dma_len(state->sg),
+		unsigned int sg_offset = sg_offset_p ? *sg_offset_p : 0;
+
+		srp_map_desc(state, sg_dma_address(state->sg) + sg_offset,
+			     sg_dma_len(state->sg) - sg_offset,
 			     target->global_mr->rkey);
+		if (sg_offset_p)
+			*sg_offset_p = 0;
 		return 1;
 	}
 
@@ -1329,15 +1340,18 @@ static int srp_map_finish_fr(struct srp_map_state *state,
 	rkey = ib_inc_rkey(desc->mr->rkey);
 	ib_update_fast_reg_key(desc->mr, rkey);
 
-	n = ib_map_mr_sg(desc->mr, state->sg, sg_nents, NULL, dev->mr_page_size);
+	n = ib_map_mr_sg(desc->mr, state->sg, sg_nents, sg_offset_p,
+			 dev->mr_page_size);
 	if (unlikely(n < 0)) {
 		srp_fr_pool_put(ch->fr_pool, &desc, 1);
-		pr_debug("%s: ib_map_mr_sg(%d) returned %d.\n",
+		pr_debug("%s: ib_map_mr_sg(%d, %d) returned %d.\n",
 			 dev_name(&req->scmnd->device->sdev_gendev), sg_nents,
-			 n);
+			 sg_offset_p ? *sg_offset_p : -1, n);
 		return n;
 	}
 
+	WARN_ON_ONCE(desc->mr->length == 0);
+
 	req->reg_cqe.done = srp_reg_mr_err_done;
 
 	wr.wr.next = NULL;
@@ -1358,8 +1372,10 @@ static int srp_map_finish_fr(struct srp_map_state *state,
 		     desc->mr->length, desc->mr->rkey);
 
 	err = ib_post_send(ch->qp, &wr.wr, &bad_wr);
-	if (unlikely(err))
+	if (unlikely(err)) {
+		WARN_ON_ONCE(err == -ENOMEM);
 		return err;
+	}
 
 	return n;
 }
@@ -1416,7 +1432,7 @@ static int srp_map_sg_fmr(struct srp_map_state *state, struct srp_rdma_ch *ch,
 
 	state->pages = req->map_page;
 	state->fmr.next = req->fmr_list;
-	state->fmr.end = req->fmr_list + ch->target->cmd_sg_cnt;
+	state->fmr.end = req->fmr_list + ch->target->mr_per_cmd;
 
 	for_each_sg(scat, sg, count, i) {
 		ret = srp_map_sg_entry(state, ch, sg, i);
@@ -1435,8 +1451,10 @@ static int srp_map_sg_fr(struct srp_map_state *state, struct srp_rdma_ch *ch,
 			 struct srp_request *req, struct scatterlist *scat,
 			 int count)
 {
+	unsigned int sg_offset = 0;
+
 	state->fr.next = req->fr_list;
-	state->fr.end = req->fr_list + ch->target->cmd_sg_cnt;
+	state->fr.end = req->fr_list + ch->target->mr_per_cmd;
 	state->sg = scat;
 
 	if (count == 0)
@@ -1445,7 +1463,7 @@ static int srp_map_sg_fr(struct srp_map_state *state, struct srp_rdma_ch *ch,
 	while (count) {
 		int i, n;
 
-		n = srp_map_finish_fr(state, req, ch, count);
+		n = srp_map_finish_fr(state, req, ch, count, &sg_offset);
 		if (unlikely(n < 0))
 			return n;
 
@@ -1509,9 +1527,10 @@ static int srp_map_idb(struct srp_rdma_ch *ch, struct srp_request *req,
 #ifdef CONFIG_NEED_SG_DMA_LENGTH
 		idb_sg->dma_length = idb_sg->length;	      /* hack^2 */
 #endif
-		ret = srp_map_finish_fr(&state, req, ch, 1);
+		ret = srp_map_finish_fr(&state, req, ch, 1, NULL);
 		if (ret < 0)
 			return ret;
+		WARN_ON_ONCE(ret < 1);
 	} else if (dev->use_fmr) {
 		state.pages = idb_pages;
 		state.pages[0] = (req->indirect_dma_addr &
@@ -2579,6 +2598,20 @@ static int srp_reset_host(struct scsi_cmnd *scmnd)
 	return srp_reconnect_rport(target->rport) == 0 ? SUCCESS : FAILED;
 }
 
+static int srp_slave_alloc(struct scsi_device *sdev)
+{
+	struct Scsi_Host *shost = sdev->host;
+	struct srp_target_port *target = host_to_target(shost);
+	struct srp_device *srp_dev = target->srp_host->srp_dev;
+	struct ib_device *ibdev = srp_dev->dev;
+
+	if (!(ibdev->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG))
+		blk_queue_virt_boundary(sdev->request_queue,
+					~srp_dev->mr_page_mask);
+
+	return 0;
+}
+
 static int srp_slave_configure(struct scsi_device *sdev)
 {
 	struct Scsi_Host *shost = sdev->host;
@@ -2770,6 +2803,7 @@ static struct scsi_host_template srp_template = {
 	.module				= THIS_MODULE,
 	.name				= "InfiniBand SRP initiator",
 	.proc_name			= DRV_NAME,
+	.slave_alloc			= srp_slave_alloc,
 	.slave_configure		= srp_slave_configure,
 	.info				= srp_target_info,
 	.queuecommand			= srp_queuecommand,
@@ -3176,6 +3210,7 @@ static ssize_t srp_create_target(struct device *dev,
 	struct srp_device *srp_dev = host->srp_dev;
 	struct ib_device *ibdev = srp_dev->dev;
 	int ret, node_idx, node, cpu, i;
+	unsigned int max_sectors_per_mr, mr_per_cmd = 0;
 	bool multich = false;
 
 	target_host = scsi_host_alloc(&srp_template,
@@ -3232,8 +3267,33 @@ static ssize_t srp_create_target(struct device *dev,
 		target->sg_tablesize = target->cmd_sg_cnt;
 	}
 
+	if (srp_dev->use_fast_reg || srp_dev->use_fmr) {
+		/*
+		 * FR and FMR can only map one HCA page per entry. If the
+		 * start address is not aligned on a HCA page boundary two
+		 * entries will be used for the head and the tail although
+		 * these two entries combined contain at most one HCA page of
+		 * data. Hence the "+ 1" in the calculation below.
+		 *
+		 * The indirect data buffer descriptor is contiguous so the
+		 * memory for that buffer will only be registered if
+		 * register_always is true. Hence add one to mr_per_cmd if
+		 * register_always has been set.
+		 */
+		max_sectors_per_mr = srp_dev->max_pages_per_mr <<
+				  (ilog2(srp_dev->mr_page_size) - 9);
+		mr_per_cmd = register_always +
+			(target->scsi_host->max_sectors + 1 +
+			 max_sectors_per_mr - 1) / max_sectors_per_mr;
+		pr_debug("max_sectors = %u; max_pages_per_mr = %u; mr_page_size = %u; max_sectors_per_mr = %u; mr_per_cmd = %u\n",
+			 target->scsi_host->max_sectors,
+			 srp_dev->max_pages_per_mr, srp_dev->mr_page_size,
+			 max_sectors_per_mr, mr_per_cmd);
+	}
+
 	target_host->sg_tablesize = target->sg_tablesize;
-	target->mr_pool_size = target->scsi_host->can_queue;
+	target->mr_pool_size = target->scsi_host->can_queue * mr_per_cmd;
+	target->mr_per_cmd = mr_per_cmd;
 	target->indirect_size = target->sg_tablesize *
 				sizeof (struct srp_direct_buf);
 	target->max_iu_len = sizeof (struct srp_cmd) +
@@ -3440,6 +3500,9 @@ static void srp_add_one(struct ib_device *device)
 	srp_dev->mr_page_mask	= ~((u64) srp_dev->mr_page_size - 1);
 	max_pages_per_mr	= device->attrs.max_mr_size;
 	do_div(max_pages_per_mr, srp_dev->mr_page_size);
+	pr_debug("%s: %llu / %u = %llu <> %u\n", __func__,
+		 device->attrs.max_mr_size, srp_dev->mr_page_size,
+		 max_pages_per_mr, SRP_MAX_PAGES_PER_MR);
 	srp_dev->max_pages_per_mr = min_t(u64, SRP_MAX_PAGES_PER_MR,
 					  max_pages_per_mr);
 
@@ -3447,12 +3510,13 @@ static void srp_add_one(struct ib_device *device)
 			    device->map_phys_fmr && device->unmap_fmr);
 	srp_dev->has_fr = (device->attrs.device_cap_flags &
 			   IB_DEVICE_MEM_MGT_EXTENSIONS);
-	if (!srp_dev->has_fmr && !srp_dev->has_fr)
+	if (!srp_dev->has_fmr && !srp_dev->has_fr) {
 		dev_warn(&device->dev, "neither FMR nor FR is supported\n");
-
-	srp_dev->use_fast_reg = (srp_dev->has_fr &&
-				 (!srp_dev->has_fmr || prefer_fr));
-	srp_dev->use_fmr = !srp_dev->use_fast_reg && srp_dev->has_fmr;
+	} else if (device->attrs.max_mr_size >= 2 * srp_dev->mr_page_size) {
+		srp_dev->use_fast_reg = (srp_dev->has_fr &&
+					 (!srp_dev->has_fmr || prefer_fr));
+		srp_dev->use_fmr = !srp_dev->use_fast_reg && srp_dev->has_fmr;
+	}
 
 	if (srp_dev->use_fast_reg) {
 		srp_dev->max_pages_per_mr =
diff --git a/drivers/infiniband/ulp/srp/ib_srp.h b/drivers/infiniband/ulp/srp/ib_srp.h
index a00914c..26bb9b0 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.h
+++ b/drivers/infiniband/ulp/srp/ib_srp.h
@@ -203,6 +203,7 @@ struct srp_target_port {
 	unsigned int		scsi_id;
 	unsigned int		sg_tablesize;
 	int			mr_pool_size;
+	int			mr_per_cmd;
 	int			queue_size;
 	int			req_ring_size;
 	int			comp_vector;
-- 
2.8.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html