Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Bart Van Assche]

On Wed, 2017-02-15 at 05:50 -0500, Laurence Oberman wrote:
> Bart only has mlx4.

I'll try to find an mlx5 setup.

Bart.

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Laurence Oberman]

----- Original Message -----
> From: "Bart Van Assche" <Bart.VanAssche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
> Cc: artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, "yuval shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>,
> stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
> Sent: Wednesday, February 15, 2017 10:31:20 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> 
> On Wed, 2017-02-15 at 05:50 -0500, Laurence Oberman wrote:
> > Bart only has mlx4.
> 
> I'll try to find an mlx5 setup.
> 
> Bart.

Hello Bart

Its fine, I tested it already. See later email from me.
I am good to do the mlx5 tests, I have this test bed in place permanently.

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 0/8] IB/srp bug fixes

[Bart Van Assche]

Hello Doug,

The patches in this series are the initiator patches I came up with while
testing the SRP initiator and target drivers. Please consider these patches
for inclusion in the upstream kernel.

The changes compared to the previous version of this series are:
- Left out the "Use the IB_DEVICE_SG_GAPS_REG HCA feature if supported" patch.
- Added a patch to stop using memory regions with gaps since Laurence ran
  into an issue with these memory regions that cannot be explained by
  analzying the SRP initiator code only.
- Addressed Leon's comments about patch "Avoid that duplicate responses
  trigger a kernel bug" and "Add support for draining IB_POLL_DIRECT
  completion queues".

The patches in this series are:

Bart Van Assche (8):
  IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
  IB/srp: Avoid that duplicate responses trigger a kernel bug
  IB/srp: Fix race conditions related to task management
  IB/srp: Document locking conventions
  IB/srp: Make a diagnostic message more informative
  IB/srp: Improve an error path
  IB/core: Add support for draining IB_POLL_DIRECT completion queues
  IB/srp: Drain the send queue before destroying a QP

 drivers/infiniband/core/cq.c        |  4 +-
 drivers/infiniband/core/verbs.c     | 35 ++++++--------
 drivers/infiniband/ulp/srp/ib_srp.c | 92 ++++++++++++++++++++++++-------------
 drivers/infiniband/ulp/srp/ib_srp.h |  1 +
 4 files changed, 77 insertions(+), 55 deletions(-)

Thanks,

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Bart Van Assche]

Tests have shown that the following error message is reported when
using SG-GAPS registration with an mlx5 adapter:

scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 0f007806 2500002a ad9fafd1
scsi host1: ib_srp: reconnect succeeded
mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 0f007806 25000032 00105dd0
scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138

Hence avoid using SG-GAPS memory registrations. Additionally,
always configure the blk_queue_virt_boundary() to avoid to trigger
a mapping failure when using adapters that support SG-GAPS (e.g.
mlx5).

Fixes: commit ad8e66b4a801 ("IB/srp: fix mr allocation when the device supports sg gaps")
Fixes: commit 509c5f33f4f6 ("IB/srp: Prevent mapping failures")
Reported-by: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Mark Bloch <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Yuval Shaia <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # 4.7+
---
 drivers/infiniband/ulp/srp/ib_srp.c | 11 +++--------
 1 file changed, 3 insertions(+), 8 deletions(-)

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 79bf48477ddb..07e3613e4798 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -371,7 +371,6 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
 	struct srp_fr_desc *d;
 	struct ib_mr *mr;
 	int i, ret = -EINVAL;
-	enum ib_mr_type mr_type;
 
 	if (pool_size <= 0)
 		goto err;
@@ -385,13 +384,9 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
 	spin_lock_init(&pool->lock);
 	INIT_LIST_HEAD(&pool->free_list);
 
-	if (device->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG)
-		mr_type = IB_MR_TYPE_SG_GAPS;
-	else
-		mr_type = IB_MR_TYPE_MEM_REG;
-
 	for (i = 0, d = &pool->desc[0]; i < pool->size; i++, d++) {
-		mr = ib_alloc_mr(pd, mr_type, max_page_list_len);
+		mr = ib_alloc_mr(pd, IB_MR_TYPE_MEM_REG,
+				 max_page_list_len);
 		if (IS_ERR(mr)) {
 			ret = PTR_ERR(mr);
 			if (ret == -ENOMEM)
@@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)
 	struct srp_device *srp_dev = target->srp_host->srp_dev;
 	struct ib_device *ibdev = srp_dev->dev;
 
-	if (!(ibdev->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG))
+	if (true)
 		blk_queue_virt_boundary(sdev->request_queue,
 					~srp_dev->mr_page_mask);
 
-- 
2.11.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Bart Van Assche]

On Tue, 2017-02-14 at 10:56 -0800, Bart Van Assche wrote:
> @@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)
>  	struct srp_device *srp_dev = target->srp_host->srp_dev;
>  	struct ib_device *ibdev = srp_dev->dev;
>  
> -	if (!(ibdev->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG))
> +	if (true)
>  		blk_queue_virt_boundary(sdev->request_queue,
>  					~srp_dev->mr_page_mask);

Hello Doug,

This patch introduces a new warning when building with W=1, namely that
variable ibdev is not used. The build bot figured this out. Do you want me
to address this?

Thanks,

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 3655 bytes --]

On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> Tests have shown that the following error message is reported when
> using SG-GAPS registration with an mlx5 adapter:
>
> scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 2500002a ad9fafd1
> scsi host1: ib_srp: reconnect succeeded
> mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 25000032 00105dd0
> scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
>
> Hence avoid using SG-GAPS memory registrations. Additionally,
> always configure the blk_queue_virt_boundary() to avoid to trigger
> a mapping failure when using adapters that support SG-GAPS (e.g.
> mlx5).

According to the error dump, we have an issue with max_page_list_len supplied and/or
internal calculations from that value to the UMR byte count.

I assume that iser works with SG_GAPS.

>
> Fixes: commit ad8e66b4a801 ("IB/srp: fix mr allocation when the device supports sg gaps")
> Fixes: commit 509c5f33f4f6 ("IB/srp: Prevent mapping failures")
> Reported-by: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: Mark Bloch <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: Yuval Shaia <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # 4.7+
> ---
>  drivers/infiniband/ulp/srp/ib_srp.c | 11 +++--------
>  1 file changed, 3 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
> index 79bf48477ddb..07e3613e4798 100644
> --- a/drivers/infiniband/ulp/srp/ib_srp.c
> +++ b/drivers/infiniband/ulp/srp/ib_srp.c
> @@ -371,7 +371,6 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
>  	struct srp_fr_desc *d;
>  	struct ib_mr *mr;
>  	int i, ret = -EINVAL;
> -	enum ib_mr_type mr_type;
>
>  	if (pool_size <= 0)
>  		goto err;
> @@ -385,13 +384,9 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
>  	spin_lock_init(&pool->lock);
>  	INIT_LIST_HEAD(&pool->free_list);
>
> -	if (device->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG)
> -		mr_type = IB_MR_TYPE_SG_GAPS;
> -	else
> -		mr_type = IB_MR_TYPE_MEM_REG;
> -
>  	for (i = 0, d = &pool->desc[0]; i < pool->size; i++, d++) {
> -		mr = ib_alloc_mr(pd, mr_type, max_page_list_len);
> +		mr = ib_alloc_mr(pd, IB_MR_TYPE_MEM_REG,
> +				 max_page_list_len);
>  		if (IS_ERR(mr)) {
>  			ret = PTR_ERR(mr);
>  			if (ret == -ENOMEM)
> @@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)
>  	struct srp_device *srp_dev = target->srp_host->srp_dev;
>  	struct ib_device *ibdev = srp_dev->dev;
>
> -	if (!(ibdev->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG))
> +	if (true)
>  		blk_queue_virt_boundary(sdev->request_queue,
>  					~srp_dev->mr_page_mask);
>
> --
> 2.11.0
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2163 bytes --]

On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > Tests have shown that the following error message is reported when
> > using SG-GAPS registration with an mlx5 adapter:
> >
> > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 2500002a ad9fafd1
> > scsi host1: ib_srp: reconnect succeeded
> > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 25000032 00105dd0
> > scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
> >
> > Hence avoid using SG-GAPS memory registrations. Additionally,
> > always configure the blk_queue_virt_boundary() to avoid to trigger
> > a mapping failure when using adapters that support SG-GAPS (e.g.
> > mlx5).
>
> According to the error dump, we have an issue with max_page_list_len supplied and/or
> internal calculations from that value to the UMR byte count.

Hi Bart,

Do you mind to try your test on my branch rdma-next [1] with the following
fixup?

diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index 3c1f483d003f..3e59dce10d5e 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx, int npages,
 	for (pages_mapped = 0;
 	     pages_mapped < pages_to_map && !err;
 	     pages_mapped += pages_iter, idx += pages_iter) {
+		npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
 		dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
-		npages = populate_xlt(mr, idx, pages_iter, xlt,
+		npages = populate_xlt(mr, idx, npages, xlt,
 				      page_shift, size, flags);

 		dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);

[1] https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Laurence Oberman]

----- Original Message -----
> From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark
> Bloch" <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "Artemy Kovalyov" <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "# 4
> . 7+" <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> Sent: Wednesday, February 15, 2017 3:19:45 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> 
> On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > Tests have shown that the following error message is reported when
> > > using SG-GAPS registration with an mlx5 adapter:
> > >
> > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > ffff880bd4270eb0
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 2500002a ad9fafd1
> > > scsi host1: ib_srp: reconnect succeeded
> > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 25000032 00105dd0
> > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > error (6) for CQE ffff880b92860138
> > >
> > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > mlx5).
> >
> > According to the error dump, we have an issue with max_page_list_len
> > supplied and/or
> > internal calculations from that value to the UMR byte count.
> 
> Hi Bart,
> 
> Do you mind to try your test on my branch rdma-next [1] with the following
> fixup?
> 
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 3c1f483d003f..3e59dce10d5e 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx,
> int npages,
>  	for (pages_mapped = 0;
>  	     pages_mapped < pages_to_map && !err;
>  	     pages_mapped += pages_iter, idx += pages_iter) {
> +		npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
>  		dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> -		npages = populate_xlt(mr, idx, pages_iter, xlt,
> +		npages = populate_xlt(mr, idx, npages, xlt,
>  				      page_shift, size, flags);
> 
>  		dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> 
> [1]
> https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> 
> Thanks
> 

Hello Leon
Replied earlier but I dont know if my reply made it.
I will have to test this.

is this repo https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next already patched with the change you want.
If not can I just take the patch and apply to my earlier tree based just on Linus's tree where I reverted the patch.

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman@redhat.com>
> To: "Leon Romanovsky" <leonro@mellanox.com>
> Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy" <maxg@mellanox.com>, "Doug Ledford"
> <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Mark Bloch"
> <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> <stable@vger.kernel.org>
> Sent: Wednesday, February 15, 2017 8:33:35 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> 
> 
> 
> ----- Original Message -----
> > From: "Leon Romanovsky" <leonro@mellanox.com>
> > To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > <maxg@mellanox.com>
> > Cc: "Doug Ledford" <dledford@redhat.com>, linux-rdma@vger.kernel.org,
> > "Israel Rukshin" <israelr@mellanox.com>, "Mark
> > Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>,
> > "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4
> > . 7+" <stable@vger.kernel.org>
> > Sent: Wednesday, February 15, 2017 3:19:45 AM
> > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > 
> > On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > > Tests have shown that the following error message is reported when
> > > > using SG-GAPS registration with an mlx5 adapter:
> > > >
> > > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > > ffff880bd4270eb0
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 0f007806 2500002a ad9fafd1
> > > > scsi host1: ib_srp: reconnect succeeded
> > > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 0f007806 25000032 00105dd0
> > > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > > error (6) for CQE ffff880b92860138
> > > >
> > > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > > mlx5).
> > >
> > > According to the error dump, we have an issue with max_page_list_len
> > > supplied and/or
> > > internal calculations from that value to the UMR byte count.
> > 
> > Hi Bart,
> > 
> > Do you mind to try your test on my branch rdma-next [1] with the following
> > fixup?
> > 
> > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > b/drivers/infiniband/hw/mlx5/mr.c
> > index 3c1f483d003f..3e59dce10d5e 100644
> > --- a/drivers/infiniband/hw/mlx5/mr.c
> > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64
> > idx,
> > int npages,
> >  	for (pages_mapped = 0;
> >  	     pages_mapped < pages_to_map && !err;
> >  	     pages_mapped += pages_iter, idx += pages_iter) {
> > +		npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> >  		dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> > -		npages = populate_xlt(mr, idx, pages_iter, xlt,
> > +		npages = populate_xlt(mr, idx, npages, xlt,
> >  				      page_shift, size, flags);
> > 
> >  		dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> > 
> > [1]
> > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > 
> > Thanks
> > 
> 
> Hello Leon
> Replied earlier but I dont know if my reply made it.
> I will have to test this.
> 
> is this repo
> https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> already patched with the change you want.
> If not can I just take the patch and apply to my earlier tree based just on
> Linus's tree where I reverted the patch.
> 
> Thanks
> Laurence
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

I guess not, I looked at applying this patch to Linus's tree but the function 
mlx5_ib_update_xlt() is not in there yet.

I will see if I can get your tree staged and test this for you.

Thanks
Laurence

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 5018 bytes --]

On Wed, Feb 15, 2017 at 08:43:38AM -0500, Laurence Oberman wrote:
>
>
> ----- Original Message -----
> > From: "Laurence Oberman" <loberman@redhat.com>
> > To: "Leon Romanovsky" <leonro@mellanox.com>
> > Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy" <maxg@mellanox.com>, "Doug Ledford"
> > <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Mark Bloch"
> > <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> > <stable@vger.kernel.org>
> > Sent: Wednesday, February 15, 2017 8:33:35 AM
> > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> >
> >
> >
> > ----- Original Message -----
> > > From: "Leon Romanovsky" <leonro@mellanox.com>
> > > To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > > <maxg@mellanox.com>
> > > Cc: "Doug Ledford" <dledford@redhat.com>, linux-rdma@vger.kernel.org,
> > > "Israel Rukshin" <israelr@mellanox.com>, "Mark
> > > Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>,
> > > "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4
> > > . 7+" <stable@vger.kernel.org>
> > > Sent: Wednesday, February 15, 2017 3:19:45 AM
> > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > >
> > > On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > > > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > > > Tests have shown that the following error message is reported when
> > > > > using SG-GAPS registration with an mlx5 adapter:
> > > > >
> > > > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > > > ffff880bd4270eb0
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 0f007806 2500002a ad9fafd1
> > > > > scsi host1: ib_srp: reconnect succeeded
> > > > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 0f007806 25000032 00105dd0
> > > > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > > > error (6) for CQE ffff880b92860138
> > > > >
> > > > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > > > mlx5).
> > > >
> > > > According to the error dump, we have an issue with max_page_list_len
> > > > supplied and/or
> > > > internal calculations from that value to the UMR byte count.
> > >
> > > Hi Bart,
> > >
> > > Do you mind to try your test on my branch rdma-next [1] with the following
> > > fixup?
> > >
> > > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > > b/drivers/infiniband/hw/mlx5/mr.c
> > > index 3c1f483d003f..3e59dce10d5e 100644
> > > --- a/drivers/infiniband/hw/mlx5/mr.c
> > > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > > @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64
> > > idx,
> > > int npages,
> > >  	for (pages_mapped = 0;
> > >  	     pages_mapped < pages_to_map && !err;
> > >  	     pages_mapped += pages_iter, idx += pages_iter) {
> > > +		npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> > >  		dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> > > -		npages = populate_xlt(mr, idx, pages_iter, xlt,
> > > +		npages = populate_xlt(mr, idx, npages, xlt,
> > >  				      page_shift, size, flags);
> > >
> > >  		dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> > >
> > > [1]
> > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > >
> > > Thanks
> > >
> >
> > Hello Leon
> > Replied earlier but I dont know if my reply made it.
> > I will have to test this.
> >
> > is this repo
> > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > already patched with the change you want.
> > If not can I just take the patch and apply to my earlier tree based just on
> > Linus's tree where I reverted the patch.
> >
> > Thanks
> > Laurence
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
>
> I guess not, I looked at applying this patch to Linus's tree but the function
> mlx5_ib_update_xlt() is not in there yet.

This is why I asked to use my tree, there is a chance that wrong
calculation was before we introduced mlx5_ib_update_xlt() function.

>
> I will see if I can get your tree staged and test this for you.

Thanks a lot.

>
> Thanks
> Laurence
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Laurence Oberman]

----- Original Message -----
> From: "Leon Romanovsky" <leonro@mellanox.com>
> To: "Laurence Oberman" <loberman@redhat.com>
> Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy" <maxg@mellanox.com>, "Doug Ledford"
> <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Mark Bloch"
> <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> <stable@vger.kernel.org>
> Sent: Wednesday, February 15, 2017 8:47:07 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> 
> On Wed, Feb 15, 2017 at 08:43:38AM -0500, Laurence Oberman wrote:
> >
> >
> > ----- Original Message -----
> > > From: "Laurence Oberman" <loberman@redhat.com>
> > > To: "Leon Romanovsky" <leonro@mellanox.com>
> > > Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > > <maxg@mellanox.com>, "Doug Ledford"
> > > <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin"
> > > <israelr@mellanox.com>, "Mark Bloch"
> > > <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy
> > > Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> > > <stable@vger.kernel.org>
> > > Sent: Wednesday, February 15, 2017 8:33:35 AM
> > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > >
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Leon Romanovsky" <leonro@mellanox.com>
> > > > To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > > > <maxg@mellanox.com>
> > > > Cc: "Doug Ledford" <dledford@redhat.com>, linux-rdma@vger.kernel.org,
> > > > "Israel Rukshin" <israelr@mellanox.com>, "Mark
> > > > Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>,
> > > > "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4
> > > > . 7+" <stable@vger.kernel.org>
> > > > Sent: Wednesday, February 15, 2017 3:19:45 AM
> > > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > > >
> > > > On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > > > > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > > > > Tests have shown that the following error message is reported when
> > > > > > using SG-GAPS registration with an mlx5 adapter:
> > > > > >
> > > > > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > > > > ffff880bd4270eb0
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 0f007806 2500002a ad9fafd1
> > > > > > scsi host1: ib_srp: reconnect succeeded
> > > > > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 0f007806 25000032 00105dd0
> > > > > > scsi host1: ib_srp: failed FAST REG status memory management
> > > > > > operation
> > > > > > error (6) for CQE ffff880b92860138
> > > > > >
> > > > > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > > > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > > > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > > > > mlx5).
> > > > >
> > > > > According to the error dump, we have an issue with max_page_list_len
> > > > > supplied and/or
> > > > > internal calculations from that value to the UMR byte count.
> > > >
> > > > Hi Bart,
> > > >
> > > > Do you mind to try your test on my branch rdma-next [1] with the
> > > > following
> > > > fixup?
> > > >
> > > > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > > > b/drivers/infiniband/hw/mlx5/mr.c
> > > > index 3c1f483d003f..3e59dce10d5e 100644
> > > > --- a/drivers/infiniband/hw/mlx5/mr.c
> > > > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > > > @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64
> > > > idx,
> > > > int npages,
> > > >  	for (pages_mapped = 0;
> > > >  	     pages_mapped < pages_to_map && !err;
> > > >  	     pages_mapped += pages_iter, idx += pages_iter) {
> > > > +		npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> > > >  		dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> > > > -		npages = populate_xlt(mr, idx, pages_iter, xlt,
> > > > +		npages = populate_xlt(mr, idx, npages, xlt,
> > > >  				      page_shift, size, flags);
> > > >
> > > >  		dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> > > >
> > > > [1]
> > > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > > >
> > > > Thanks
> > > >
> > >
> > > Hello Leon
> > > Replied earlier but I dont know if my reply made it.
> > > I will have to test this.
> > >
> > > is this repo
> > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > > already patched with the change you want.
> > > If not can I just take the patch and apply to my earlier tree based just
> > > on
> > > Linus's tree where I reverted the patch.
> > >
> > > Thanks
> > > Laurence
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > >
> >
> > I guess not, I looked at applying this patch to Linus's tree but the
> > function
> > mlx5_ib_update_xlt() is not in there yet.
> 
> This is why I asked to use my tree, there is a chance that wrong
> calculation was before we introduced mlx5_ib_update_xlt() function.
> 
> >
> > I will see if I can get your tree staged and test this for you.
> 
> Thanks a lot.
> 
> >
> > Thanks
> > Laurence
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hello Leon
I pulled your tree, ran git checkout rdma-next
I applied the patch manually, built the kernel and started the tests.

Pretty soon I ran into

ibclient login: [  132.640142] mlx5_1:dump_cqe:262:(pid 11417): dump error cqe
[  132.640185] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd86a97b0
[  132.718106] 00000000 00000000 00000000 00000000
[  132.743616] 00000000 00000000 00000000 00000000
[  132.767790] 00000000 00000000 00000000 00000000
[  132.793626] 00000000 0f007806 2500002a 8ad015d1
[  136.181512] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/bin/loginctl) failed: Too many levels of symbolic links
[  136.250779] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/lib/systemd/system/dracut-emergency.service) failed: Too many levels of symbolic links
[  147.791028] scsi host2: ib_srp: reconnect succeeded
[  147.827012] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330
[  162.908764] scsi host2: ib_srp: reconnect succeeded
[  162.944244] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330
[  166.409523] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec570
[  177.292496] scsi host2: ib_srp: reconnect succeeded
[  177.334396] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
[  177.363793] 00000000 00000000 00000000 00000000
[  177.388434] 00000000 00000000 00000000 00000000
[  177.413918] 00000000 00000000 00000000 00000000
[  177.438911] 00000000 0f007806 25000042 00102dd0
[  177.465048] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138
[  181.386124] scsi host1: ib_srp: reconnect succeeded
[  181.422892] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[  192.827678] fast_io_fail_tmo expired for SRP port-2:1 / host2.
[  193.230036] scsi host2: ib_srp: reconnect succeeded
[  193.329290] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
[  193.356245] 00000000 00000000 00000000 00000000
[  193.379488] 00000000 00000000 00000000 00000000
[  193.404853] 00000000 00000000 00000000 00000000
[  193.429633] 00000000 0f007806 2500004a 006c46d0
[  193.455183] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8
[  196.178111] scsi host1: ib_srp: reconnect succeeded
[  196.215197] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[  208.277396] scsi host2: ib_srp: reconnect succeeded
[  208.318454] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
[  208.349012] 00000000 00000000 00000000 00000000
[  208.374873] 00000000 00000000 00000000 00000000
[  208.401070] 00000000 00000000 00000000 00000000
[  208.426873] 00000000 0f007806 25000052 00103dd0
[  208.452788] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138
[  211.066945] scsi host1: ib_srp: reconnect succeeded
[  211.101645] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[  223.296126] scsi host2: ib_srp: reconnect succeeded
[  223.396340] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
[  223.423310] 00000000 00000000 00000000 00000000
[  223.448590] 00000000 00000000 00000000 00000000
[  223.473520] 00000000 00000000 00000000 00000000
[  223.499008] 00000000 0f007806 2500005a 006c56d0
[  223.524787] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8
[  225.463312] scsi host1: ib_srp: reconnect succeeded
[  225.499915] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[  238.912126] fast_io_fail_tmo expired for SRP port-2:1 / host2.
[  239.254388] scsi host2: ib_srp: reconnect succeeded
[  239.291843] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0
[  240.619914] scsi host1: ib_srp: reconnect succeeded
[  240.654445] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[  254.620679] scsi host2: ib_srp: reconnect succeeded
[  254.658021] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0
[  255.798616] scsi host1: ib_srp: reconnect succeeded
[  255.832897] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[  269.228783] scsi host2: ib_srp: reconnect succeeded
[  269.272055] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
[  269.300655] 00000000 00000000 00000000 00000000
[  269.325885] 00000000 00000000 00000000 00000000
[  269.350457] 00000000 00000000 00000000 00000000
[  269.375757] 00000000 0f007806 25000072 005847d0
[  269.402013] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c905b8
[  269.856464] scsi host1: ib_srp: reconnect succeeded
[  269.893203] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0

Thanks
Laurence

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 12548 bytes --]

On Wed, Feb 15, 2017 at 09:53:56AM -0500, Laurence Oberman wrote:
>
>
> ----- Original Message -----
> > From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Doug Ledford"
> > <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark Bloch"
> > <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "Artemy Kovalyov" <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "# 4 . 7+"
> > <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> > Sent: Wednesday, February 15, 2017 8:47:07 AM
> > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> >
> > On Wed, Feb 15, 2017 at 08:43:38AM -0500, Laurence Oberman wrote:
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > > > To: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > > Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Max Gurtovoy"
> > > > <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Doug Ledford"
> > > > <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Israel Rukshin"
> > > > <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark Bloch"
> > > > <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "Artemy
> > > > Kovalyov" <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "# 4 . 7+"
> > > > <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> > > > Sent: Wednesday, February 15, 2017 8:33:35 AM
> > > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > > >
> > > >
> > > >
> > > > ----- Original Message -----
> > > > > From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > > > To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Max Gurtovoy"
> > > > > <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > > > Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
> > > > > "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark
> > > > > Bloch" <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>,
> > > > > "Artemy Kovalyov" <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "# 4
> > > > > . 7+" <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> > > > > Sent: Wednesday, February 15, 2017 3:19:45 AM
> > > > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > > > >
> > > > > On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > > > > > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > > > > > Tests have shown that the following error message is reported when
> > > > > > > using SG-GAPS registration with an mlx5 adapter:
> > > > > > >
> > > > > > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > > > > > ffff880bd4270eb0
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 0f007806 2500002a ad9fafd1
> > > > > > > scsi host1: ib_srp: reconnect succeeded
> > > > > > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 0f007806 25000032 00105dd0
> > > > > > > scsi host1: ib_srp: failed FAST REG status memory management
> > > > > > > operation
> > > > > > > error (6) for CQE ffff880b92860138
> > > > > > >
> > > > > > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > > > > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > > > > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > > > > > mlx5).
> > > > > >
> > > > > > According to the error dump, we have an issue with max_page_list_len
> > > > > > supplied and/or
> > > > > > internal calculations from that value to the UMR byte count.
> > > > >
> > > > > Hi Bart,
> > > > >
> > > > > Do you mind to try your test on my branch rdma-next [1] with the
> > > > > following
> > > > > fixup?
> > > > >
> > > > > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > > > > b/drivers/infiniband/hw/mlx5/mr.c
> > > > > index 3c1f483d003f..3e59dce10d5e 100644
> > > > > --- a/drivers/infiniband/hw/mlx5/mr.c
> > > > > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > > > > @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64
> > > > > idx,
> > > > > int npages,
> > > > >  	for (pages_mapped = 0;
> > > > >  	     pages_mapped < pages_to_map && !err;
> > > > >  	     pages_mapped += pages_iter, idx += pages_iter) {
> > > > > +		npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> > > > >  		dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> > > > > -		npages = populate_xlt(mr, idx, pages_iter, xlt,
> > > > > +		npages = populate_xlt(mr, idx, npages, xlt,
> > > > >  				      page_shift, size, flags);
> > > > >
> > > > >  		dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> > > > >
> > > > > [1]
> > > > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > > > >
> > > > > Thanks
> > > > >
> > > >
> > > > Hello Leon
> > > > Replied earlier but I dont know if my reply made it.
> > > > I will have to test this.
> > > >
> > > > is this repo
> > > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > > > already patched with the change you want.
> > > > If not can I just take the patch and apply to my earlier tree based just
> > > > on
> > > > Linus's tree where I reverted the patch.
> > > >
> > > > Thanks
> > > > Laurence
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > >
> > >
> > > I guess not, I looked at applying this patch to Linus's tree but the
> > > function
> > > mlx5_ib_update_xlt() is not in there yet.
> >
> > This is why I asked to use my tree, there is a chance that wrong
> > calculation was before we introduced mlx5_ib_update_xlt() function.
> >
> > >
> > > I will see if I can get your tree staged and test this for you.
> >
> > Thanks a lot.
> >
> > >
> > > Thanks
> > > Laurence
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
>
> Hello Leon
> I pulled your tree, ran git checkout rdma-next
> I applied the patch manually, built the kernel and started the tests.
>
> Pretty soon I ran into
>
> ibclient login: [  132.640142] mlx5_1:dump_cqe:262:(pid 11417): dump error cqe
> [  132.640185] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd86a97b0
> [  132.718106] 00000000 00000000 00000000 00000000
> [  132.743616] 00000000 00000000 00000000 00000000
> [  132.767790] 00000000 00000000 00000000 00000000
> [  132.793626] 00000000 0f007806 2500002a 8ad015d1
> [  136.181512] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/bin/loginctl) failed: Too many levels of symbolic links
> [  136.250779] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/lib/systemd/system/dracut-emergency.service) failed: Too many levels of symbolic links
> [  147.791028] scsi host2: ib_srp: reconnect succeeded
> [  147.827012] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330
> [  162.908764] scsi host2: ib_srp: reconnect succeeded
> [  162.944244] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330
> [  166.409523] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec570
> [  177.292496] scsi host2: ib_srp: reconnect succeeded
> [  177.334396] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [  177.363793] 00000000 00000000 00000000 00000000
> [  177.388434] 00000000 00000000 00000000 00000000
> [  177.413918] 00000000 00000000 00000000 00000000
> [  177.438911] 00000000 0f007806 25000042 00102dd0
> [  177.465048] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138
> [  181.386124] scsi host1: ib_srp: reconnect succeeded
> [  181.422892] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  192.827678] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> [  193.230036] scsi host2: ib_srp: reconnect succeeded
> [  193.329290] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [  193.356245] 00000000 00000000 00000000 00000000
> [  193.379488] 00000000 00000000 00000000 00000000
> [  193.404853] 00000000 00000000 00000000 00000000
> [  193.429633] 00000000 0f007806 2500004a 006c46d0
> [  193.455183] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8
> [  196.178111] scsi host1: ib_srp: reconnect succeeded
> [  196.215197] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  208.277396] scsi host2: ib_srp: reconnect succeeded
> [  208.318454] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [  208.349012] 00000000 00000000 00000000 00000000
> [  208.374873] 00000000 00000000 00000000 00000000
> [  208.401070] 00000000 00000000 00000000 00000000
> [  208.426873] 00000000 0f007806 25000052 00103dd0
> [  208.452788] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138
> [  211.066945] scsi host1: ib_srp: reconnect succeeded
> [  211.101645] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  223.296126] scsi host2: ib_srp: reconnect succeeded
> [  223.396340] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [  223.423310] 00000000 00000000 00000000 00000000
> [  223.448590] 00000000 00000000 00000000 00000000
> [  223.473520] 00000000 00000000 00000000 00000000
> [  223.499008] 00000000 0f007806 2500005a 006c56d0
> [  223.524787] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8
> [  225.463312] scsi host1: ib_srp: reconnect succeeded
> [  225.499915] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  238.912126] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> [  239.254388] scsi host2: ib_srp: reconnect succeeded
> [  239.291843] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0
> [  240.619914] scsi host1: ib_srp: reconnect succeeded
> [  240.654445] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  254.620679] scsi host2: ib_srp: reconnect succeeded
> [  254.658021] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0
> [  255.798616] scsi host1: ib_srp: reconnect succeeded
> [  255.832897] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  269.228783] scsi host2: ib_srp: reconnect succeeded
> [  269.272055] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [  269.300655] 00000000 00000000 00000000 00000000
> [  269.325885] 00000000 00000000 00000000 00000000
> [  269.350457] 00000000 00000000 00000000 00000000
> [  269.375757] 00000000 0f007806 25000072 005847d0
> [  269.402013] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c905b8
> [  269.856464] scsi host1: ib_srp: reconnect succeeded
> [  269.893203] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
>

Thanks a lot for your effort, we tried fast path to debug, but it didn't
work :(.

> Thanks
> Laurence
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Sagi Grimberg]

> Hello Leon
> I pulled your tree, ran git checkout rdma-next
> I applied the patch manually, built the kernel and started the tests.
>
> Pretty soon I ran into
>
> ibclient login: [  132.640142] mlx5_1:dump_cqe:262:(pid 11417): dump error cqe
> [  132.640185] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd86a97b0
> [  132.718106] 00000000 00000000 00000000 00000000
> [  132.743616] 00000000 00000000 00000000 00000000
> [  132.767790] 00000000 00000000 00000000 00000000
> [  132.793626] 00000000 0f007806 2500002a 8ad015d1

Not surprising given that the ib_map_mr_sg code never visits there...

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Sagi Grimberg]

> Tests have shown that the following error message is reported when
> using SG-GAPS registration with an mlx5 adapter:
>
> scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 2500002a ad9fafd1
> scsi host1: ib_srp: reconnect succeeded
> mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 25000032 00105dd0
> scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
>
> Hence avoid using SG-GAPS memory registrations. Additionally,
> always configure the blk_queue_virt_boundary() to avoid to trigger
> a mapping failure when using adapters that support SG-GAPS (e.g.
> mlx5).

Hi Guys,

Sorry for addressing this late, but has this failure been investigated?

Max, Israel, what does this error syndrome map to?

Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
incremented. Does the following change fix the problem?
--
diff --git a/drivers/infiniband/hw/mlx5/mr.c 
b/drivers/infiniband/hw/mlx5/mr.c
index 8f608debe141..c21c9eee37f6 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
                 klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
                 klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
                 klms[i].key = cpu_to_be32(lkey);
-               mr->ibmr.length += sg_dma_len(sg);
+               mr->ibmr.length += sg_dma_len(sg) - sg_offset;

                 sg_offset = 0;
         }

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Laurence Oberman]

----- Original Message -----
> From: "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Leon
> Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark Bloch" <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "# 4 .
> 7+" <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> Sent: Wednesday, February 15, 2017 10:38:06 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> 
> 
> > Tests have shown that the following error message is reported when
> > using SG-GAPS registration with an mlx5 adapter:
> >
> > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > ffff880bd4270eb0
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 2500002a ad9fafd1
> > scsi host1: ib_srp: reconnect succeeded
> > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 25000032 00105dd0
> > scsi host1: ib_srp: failed FAST REG status memory management operation
> > error (6) for CQE ffff880b92860138
> >
> > Hence avoid using SG-GAPS memory registrations. Additionally,
> > always configure the blk_queue_virt_boundary() to avoid to trigger
> > a mapping failure when using adapters that support SG-GAPS (e.g.
> > mlx5).
> 
> Hi Guys,
> 
> Sorry for addressing this late, but has this failure been investigated?
> 
> Max, Israel, what does this error syndrome map to?
> 
> Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> incremented. Does the following change fix the problem?
> --
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 8f608debe141..c21c9eee37f6 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
>                  klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
>                  klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
>                  klms[i].key = cpu_to_be32(lkey);
> -               mr->ibmr.length += sg_dma_len(sg);
> +               mr->ibmr.length += sg_dma_len(sg) - sg_offset;
> 
>                  sg_offset = 0;
>          }
> --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hello Sagi,

I will get this tested.
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Max Gurtovoy]

On 2/15/2017 5:38 PM, Sagi Grimberg wrote:
>
>> Tests have shown that the following error message is reported when
>> using SG-GAPS registration with an mlx5 adapter:
>>
>> scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
>> ffff880bd4270eb0
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 0f007806 2500002a ad9fafd1
>> scsi host1: ib_srp: reconnect succeeded
>> mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 0f007806 25000032 00105dd0
>> scsi host1: ib_srp: failed FAST REG status memory management operation
>> error (6) for CQE ffff880b92860138
>>
>> Hence avoid using SG-GAPS memory registrations. Additionally,
>> always configure the blk_queue_virt_boundary() to avoid to trigger
>> a mapping failure when using adapters that support SG-GAPS (e.g.
>> mlx5).
>
> Hi Guys,
>
> Sorry for addressing this late, but has this failure been investigated?
>
> Max, Israel, what does this error syndrome map to?

Sagi,
this syndrome says that number of klms to write is bigger than number of 
mtts.

Artemy started investigating it and proposed solution that were tested 
by Laurence.
Let's see if your fix will help.

>
> Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> incremented. Does the following change fix the problem?
> --
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 8f608debe141..c21c9eee37f6 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
>                 klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
>                 klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
>                 klms[i].key = cpu_to_be32(lkey);
> -               mr->ibmr.length += sg_dma_len(sg);
> +               mr->ibmr.length += sg_dma_len(sg) - sg_offset;
>
>                 sg_offset = 0;
>         }
> --
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Sagi Grimberg]

>> Hi Guys,
>>
>> Sorry for addressing this late, but has this failure been investigated?
>>
>> Max, Israel, what does this error syndrome map to?
>
> Sagi,
> this syndrome says that number of klms to write is bigger than number of
> mtts.

That is strange, given that we check for it explicitly...

if this is indeed the case, then mlx5 reports wrong
max_fast_reg_page_list_len.

> Artemy started investigating it and proposed solution that were tested
> by Laurence.
> Let's see if your fix will help.

Not sure it will help if the syndrome is what you say it is...

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2582 bytes --]

On Wed, Feb 15, 2017 at 06:18:02PM +0200, Max Gurtovoy wrote:
>
>
> On 2/15/2017 5:38 PM, Sagi Grimberg wrote:
> >
> > > Tests have shown that the following error message is reported when
> > > using SG-GAPS registration with an mlx5 adapter:
> > >
> > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > ffff880bd4270eb0
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 2500002a ad9fafd1
> > > scsi host1: ib_srp: reconnect succeeded
> > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 25000032 00105dd0
> > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > error (6) for CQE ffff880b92860138
> > >
> > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > mlx5).
> >
> > Hi Guys,
> >
> > Sorry for addressing this late, but has this failure been investigated?
> >
> > Max, Israel, what does this error syndrome map to?
>
> Sagi,
> this syndrome says that number of klms to write is bigger than number of
> mtts.
>
> Artemy started investigating it and proposed solution that were tested by
> Laurence.
> Let's see if your fix will help.

No, Artemy's change doesn't fix it.

>
> >
> > Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> > incremented. Does the following change fix the problem?
> > --
> > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > b/drivers/infiniband/hw/mlx5/mr.c
> > index 8f608debe141..c21c9eee37f6 100644
> > --- a/drivers/infiniband/hw/mlx5/mr.c
> > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
> >                 klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
> >                 klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
> >                 klms[i].key = cpu_to_be32(lkey);
> > -               mr->ibmr.length += sg_dma_len(sg);
> > +               mr->ibmr.length += sg_dma_len(sg) - sg_offset;
> >
> >                 sg_offset = 0;
> >         }
> > --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Laurence Oberman]

----- Original Message -----
> From: "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Leon
> Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark Bloch" <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "# 4 .
> 7+" <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> Sent: Wednesday, February 15, 2017 10:38:06 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> 
> 
> > Tests have shown that the following error message is reported when
> > using SG-GAPS registration with an mlx5 adapter:
> >
> > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > ffff880bd4270eb0
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 2500002a ad9fafd1
> > scsi host1: ib_srp: reconnect succeeded
> > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 25000032 00105dd0
> > scsi host1: ib_srp: failed FAST REG status memory management operation
> > error (6) for CQE ffff880b92860138
> >
> > Hence avoid using SG-GAPS memory registrations. Additionally,
> > always configure the blk_queue_virt_boundary() to avoid to trigger
> > a mapping failure when using adapters that support SG-GAPS (e.g.
> > mlx5).
> 
> Hi Guys,
> 
> Sorry for addressing this late, but has this failure been investigated?
> 
> Max, Israel, what does this error syndrome map to?
> 
> Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> incremented. Does the following change fix the problem?
> --
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 8f608debe141..c21c9eee37f6 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
>                  klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
>                  klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
>                  klms[i].key = cpu_to_be32(lkey);
> -               mr->ibmr.length += sg_dma_len(sg);
> +               mr->ibmr.length += sg_dma_len(sg) - sg_offset;
> 
>                  sg_offset = 0;
>          }
> --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.

Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux

Very quickly get to this

[  180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
[  181.016899] 00000000 00000000 00000000 00000000
[  181.040949] 00000000 00000000 00000000 00000000
[  181.066960] 00000000 00000000 00000000 00000000
[  181.092030] 00000000 0f007806 2500002a bf1913d0
[  181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
[  196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
[  197.090886] scsi host2: ib_srp: reconnect succeeded
[  197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30

So does not help.
I think my and Barts suggestion to revert for now is the best way forward.
I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Sagi Grimberg]

> Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
>
> Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
>
> Very quickly get to this
>
> [  180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> [  181.016899] 00000000 00000000 00000000 00000000
> [  181.040949] 00000000 00000000 00000000 00000000
> [  181.066960] 00000000 00000000 00000000 00000000
> [  181.092030] 00000000 0f007806 2500002a bf1913d0
> [  181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
> [  196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> [  197.090886] scsi host2: ib_srp: reconnect succeeded
> [  197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30
>
> So does not help.
> I think my and Barts suggestion to revert for now is the best way forward.
> I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.

Yea, probably this is the best way forward.

Bart, I think the change I suggested is still needed regardless,
do you agree?

Max, Leon, is it possible that the max number of klms pr mr is
less than what reported in device capabilities for page_list_len?

If so, this means that either:
1. mlx5 needs to expose the minimum between pages and sg elems (sucks)
2. we need yet another capability for SG_GAPS (sucks^2 because the
whole point was to make it transparent to the user)
3. mlx5 does not support SG_GAPS (sucks^3 because we now have something
thats not supported by any device).

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Bart Van Assche]

On Wed, 2017-02-15 at 18:55 +0200, Sagi Grimberg wrote:
> Bart, I think the change I suggested is still needed regardless,
> do you agree?

I'm not sure. I have not yet had a close look at that part of the mlx5 driver.

Bart.

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2157 bytes --]

On Wed, Feb 15, 2017 at 06:55:52PM +0200, Sagi Grimberg wrote:
>
> > Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
> >
> > Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
> >
> > Very quickly get to this
> >
> > [  180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> > [  181.016899] 00000000 00000000 00000000 00000000
> > [  181.040949] 00000000 00000000 00000000 00000000
> > [  181.066960] 00000000 00000000 00000000 00000000
> > [  181.092030] 00000000 0f007806 2500002a bf1913d0
> > [  181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
> > [  196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> > [  197.090886] scsi host2: ib_srp: reconnect succeeded
> > [  197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30
> >
> > So does not help.
> > I think my and Barts suggestion to revert for now is the best way forward.
> > I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.
>
> Yea, probably this is the best way forward.
>
> Bart, I think the change I suggested is still needed regardless,
> do you agree?
>
> Max, Leon, is it possible that the max number of klms pr mr is
> less than what reported in device capabilities for page_list_len?

I hope no and we will check.
I already asked it, but didn't get any response, and I'll repeat it again.
ISER has similar code with SG_GAPS, does it work?

>
> If so, this means that either:
> 1. mlx5 needs to expose the minimum between pages and sg elems (sucks)
> 2. we need yet another capability for SG_GAPS (sucks^2 because the
> whole point was to make it transparent to the user)
> 3. mlx5 does not support SG_GAPS (sucks^3 because we now have something
> thats not supported by any device).
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Max Gurtovoy]

On 2/16/2017 8:14 AM, Leon Romanovsky wrote:
> On Wed, Feb 15, 2017 at 06:55:52PM +0200, Sagi Grimberg wrote:
>>
>>> Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
>>>
>>> Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> Very quickly get to this
>>>
>>> [  180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
>>> [  181.016899] 00000000 00000000 00000000 00000000
>>> [  181.040949] 00000000 00000000 00000000 00000000
>>> [  181.066960] 00000000 00000000 00000000 00000000
>>> [  181.092030] 00000000 0f007806 2500002a bf1913d0
>>> [  181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
>>> [  196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
>>> [  197.090886] scsi host2: ib_srp: reconnect succeeded
>>> [  197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30
>>>
>>> So does not help.
>>> I think my and Barts suggestion to revert for now is the best way forward.
>>> I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.
>>
>> Yea, probably this is the best way forward.
>>
>> Bart, I think the change I suggested is still needed regardless,
>> do you agree?
>>
>> Max, Leon, is it possible that the max number of klms pr mr is
>> less than what reported in device capabilities for page_list_len?
>
> I hope no and we will check.
> I already asked it, but didn't get any response, and I'll repeat it again.
> ISER has similar code with SG_GAPS, does it work?

Yes, I haven't seen issues with that in iSER.
We need to continue with the debug.

>
>>
>> If so, this means that either:
>> 1. mlx5 needs to expose the minimum between pages and sg elems (sucks)
>> 2. we need yet another capability for SG_GAPS (sucks^2 because the
>> whole point was to make it transparent to the user)
>> 3. mlx5 does not support SG_GAPS (sucks^3 because we now have something
>> thats not supported by any device).
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html