[PATCH 0/2] Expose max_sge_rd correctly

[PATCH 0/2] Expose max_sge_rd correctly

[Sagi Grimberg]

This addresses a specific mlx4 issue where the max_sge_rd
is actually smaller than max_sge (rdma reads with max_sge
entries completes with error).

The second patch removes the explicit work-around from the
iser target code.

This applies on top of Christoph's device attributes modification.

Sagi Grimberg (2):
  mlx4: Expose correct max_sge_rd limit
  iser-target: Remove explicit mlx4 work-around

 drivers/infiniband/hw/mlx4/main.c       |    3 ++-
 drivers/infiniband/ulp/isert/ib_isert.c |   10 ++--------
 2 files changed, 4 insertions(+), 9 deletions(-)

[PATCH 1/2] mlx4: Expose correct max_sge_rd limit

[Sagi Grimberg]

mlx4 devices (ConnectX-2, ConnectX-3) can not issue
max_sge in a single RDMA_READ request (resulting in
a completion error). Thus, expose lower max_sge_rd
to avoid this issue.

Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
 drivers/infiniband/hw/mlx4/main.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index 3889723..46305dc 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -499,7 +499,8 @@ static int mlx4_ib_init_device_flags(struct ib_device *ibdev)
 	ibdev->max_qp_wr	   = dev->dev->caps.max_wqes - MLX4_IB_SQ_MAX_SPARE;
 	ibdev->max_sge		   = min(dev->dev->caps.max_sq_sg,
 					 dev->dev->caps.max_rq_sg);
-	ibdev->max_sge_rd	   = ibdev->max_sge;
+	/* reserve 2 sge slots for rdma reads */
+	ibdev->max_sge_rd	   = ibdev->max_sge - 2;
 	ibdev->max_cq		   = dev->dev->quotas.cq;
 	ibdev->max_cqe		   = dev->dev->caps.max_cqes;
 	ibdev->max_mr		   = dev->dev->quotas.mpt;
-- 
1.7.1

Re: [PATCH 1/2] mlx4: Expose correct max_sge_rd limit

[Or Gerlitz]

On 10/27/2015 11:40 AM, Sagi Grimberg wrote:
> mlx4 devices (ConnectX-2, ConnectX-3) can not issue
> max_sge in a single RDMA_READ request (resulting in
> a completion error). Thus, expose lower max_sge_rd
> to avoid this issue.

Sagi,

I can hear your pain when wearing the iser target driver maintainer hat.

Still, this patch is currently pure WA b/c we didn't do RCA (Root Cause 
Analysis)

Lets wait for RCA (which might yield the same patch, BTW) and keep 
suffering in LIO

Or.


>
> Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> ---
>   drivers/infiniband/hw/mlx4/main.c |    3 ++-
>   1 files changed, 2 insertions(+), 1 deletions(-)
>
> diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
> index 3889723..46305dc 100644
> --- a/drivers/infiniband/hw/mlx4/main.c
> +++ b/drivers/infiniband/hw/mlx4/main.c
> @@ -499,7 +499,8 @@ static int mlx4_ib_init_device_flags(struct ib_device *ibdev)
>   	ibdev->max_qp_wr	   = dev->dev->caps.max_wqes - MLX4_IB_SQ_MAX_SPARE;
>   	ibdev->max_sge		   = min(dev->dev->caps.max_sq_sg,
>   					 dev->dev->caps.max_rq_sg);
> -	ibdev->max_sge_rd	   = ibdev->max_sge;
> +	/* reserve 2 sge slots for rdma reads */
> +	ibdev->max_sge_rd	   = ibdev->max_sge - 2;
>   	ibdev->max_cq		   = dev->dev->quotas.cq;
>   	ibdev->max_cqe		   = dev->dev->caps.max_cqes;
>   	ibdev->max_mr		   = dev->dev->quotas.mpt;

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/2] mlx4: Expose correct max_sge_rd limit

[Sagi Grimberg]

On 27/10/2015 16:39, Or Gerlitz wrote:
> On 10/27/2015 11:40 AM, Sagi Grimberg wrote:
>> mlx4 devices (ConnectX-2, ConnectX-3) can not issue
>> max_sge in a single RDMA_READ request (resulting in
>> a completion error). Thus, expose lower max_sge_rd
>> to avoid this issue.
>
> Sagi,

Hey Or,

> Still, this patch is currently pure WA b/c we didn't do RCA (Root Cause
> Analysis)

So from my discussions with the HW folks a RDMA_READ wqe cannot exceed
512B. The wqe control segment is 16 bytes, the rdma section is 12 bytes
(rkey + raddr) and each sge is 16 bytes so the computation is:

(512B-16B-12B)/16B = 30.

The reason is that the HW needs to fetch the rdma_read wqe on the RX
path (rdma_read response) and it has a limited buffer at that point.

Perhaps a dedicated #define for that is needed here.

I'll add that in the change log in v1.

Cheers,
Sagi.

Re: [PATCH 1/2] mlx4: Expose correct max_sge_rd limit

[Or Gerlitz]

On 10/27/2015 6:03 PM, Sagi Grimberg wrote:
> So from my discussions with the HW folks a RDMA_READ wqe cannot exceed
> 512B. The wqe control segment is 16 bytes, the rdma section is 12 bytes
> (rkey + raddr) and each sge is 16 bytes so the computation is:
>
> (512B-16B-12B)/16B = 30. 

But AFAIR, the magic number was 28... how this goes hand in hand with 
your findings?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/2] mlx4: Expose correct max_sge_rd limit

[Sagi Grimberg]

> But AFAIR, the magic number was 28... how this goes hand in hand with
> your findings?

mlx4 max_sge is 32, and isert does max_sge - 2 = 30.
So it always used 30... and I run it reliably with this for a while now.

This thing exists before I was involved so I might not be familiar with
all the details...

Sagi.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/2] mlx4: Expose correct max_sge_rd limit

[Bart Van Assche]

On 10/27/2015 02:40 AM, Sagi Grimberg wrote:
> mlx4 devices (ConnectX-2, ConnectX-3) can not issue
> max_sge in a single RDMA_READ request (resulting in
> a completion error). Thus, expose lower max_sge_rd
> to avoid this issue.
>
> Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
> ---
>   drivers/infiniband/hw/mlx4/main.c |    3 ++-
>   1 files changed, 2 insertions(+), 1 deletions(-)
>
> diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
> index 3889723..46305dc 100644
> --- a/drivers/infiniband/hw/mlx4/main.c
> +++ b/drivers/infiniband/hw/mlx4/main.c
> @@ -499,7 +499,8 @@ static int mlx4_ib_init_device_flags(struct ib_device *ibdev)
>   	ibdev->max_qp_wr	   = dev->dev->caps.max_wqes - MLX4_IB_SQ_MAX_SPARE;
>   	ibdev->max_sge		   = min(dev->dev->caps.max_sq_sg,
>   					 dev->dev->caps.max_rq_sg);
> -	ibdev->max_sge_rd	   = ibdev->max_sge;
> +	/* reserve 2 sge slots for rdma reads */
> +	ibdev->max_sge_rd	   = ibdev->max_sge - 2;
>   	ibdev->max_cq		   = dev->dev->quotas.cq;
>   	ibdev->max_cqe		   = dev->dev->caps.max_cqes;
>   	ibdev->max_mr		   = dev->dev->quotas.mpt;

Hello Sagi,

Is this the same issue as what has been discussed in 
http://www.spinics.net/lists/linux-rdma/msg21799.html ?

Thanks,

Bart.

Re: [PATCH 1/2] mlx4: Expose correct max_sge_rd limit

[Sagi Grimberg]

> Hello Sagi,
>
> Is this the same issue as what has been discussed in
> http://www.spinics.net/lists/linux-rdma/msg21799.html ?

Looks like it.

I think this patch addresses this issue, but lets CC Eli
to comment if I'm missing something.

Thanks for digging this up...

Sagi.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 1/2] mlx4: Expose correct max_sge_rd limit

[Eli Cohen]

Just discussed the issue with Sagi. Sagi will follow up with a small correction.

-----Original Message-----
From: Sagi Grimberg [mailto:sagig-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org] 
Sent: Tuesday, October 27, 2015 11:32 AM
To: Bart Van Assche; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; target-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Cc: Steve Wise; Nicholas A. Bellinger; Or Gerlitz; Doug Ledford; Eli Cohen
Subject: Re: [PATCH 1/2] mlx4: Expose correct max_sge_rd limit

> Hello Sagi,
>
> Is this the same issue as what has been discussed in 
> http://www.spinics.net/lists/linux-rdma/msg21799.html ?

Looks like it.

I think this patch addresses this issue, but lets CC Eli to comment if I'm missing something.

Thanks for digging this up...

Sagi.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 2/2] iser-target: Remove explicit mlx4 work-around

[Sagi Grimberg]

The driver now exposes sufficient limits so we can
avoid having mlx4 specific work-around.

Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
 drivers/infiniband/ulp/isert/ib_isert.c |   10 ++--------
 1 files changed, 2 insertions(+), 8 deletions(-)

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 96336a9..303cea7 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -141,14 +141,8 @@ isert_create_qp(struct isert_conn *isert_conn,
 	attr.recv_cq = comp->cq;
 	attr.cap.max_send_wr = ISERT_QP_MAX_REQ_DTOS;
 	attr.cap.max_recv_wr = ISERT_QP_MAX_RECV_DTOS + 1;
-	/*
-	 * FIXME: Use devattr.max_sge - 2 for max_send_sge as
-	 * work-around for RDMA_READs with ConnectX-2.
-	 *
-	 * Also, still make sure to have at least two SGEs for
-	 * outgoing control PDU responses.
-	 */
-	attr.cap.max_send_sge = max(2, device->ib_device->max_sge - 2);
+	attr.cap.max_send_sge = min(device->ib_device->max_sge,
+				    device->ib_device->max_sge_rd);
 	isert_conn->max_sge = attr.cap.max_send_sge;
 
 	attr.cap.max_recv_sge = 1;
-- 
1.7.1

RE: [PATCH 0/2] Expose max_sge_rd correctly

[Steve Wise]

> -----Original Message-----
> From: Sagi Grimberg [mailto:sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org]
> Sent: Tuesday, October 27, 2015 4:41 AM
> To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; target-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Cc: Steve Wise; Nicholas A. Bellinger; Or Gerlitz; Doug Ledford
> Subject: [PATCH 0/2] Expose max_sge_rd correctly
> 
> This addresses a specific mlx4 issue where the max_sge_rd
> is actually smaller than max_sge (rdma reads with max_sge
> entries completes with error).
> 
> The second patch removes the explicit work-around from the
> iser target code.
> 
> This applies on top of Christoph's device attributes modification.
> 


Looks correct to me.

Series Reviewed-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html