* [PATCH] svcrdma: limit FRMR page list lengths to device max
@ 2015-08-07 16:11 ` Steve Wise
0 siblings, 0 replies; 6+ messages in thread
From: Steve Wise @ 2015-08-07 16:11 UTC (permalink / raw)
To: dledford-H+wXaHxf7aLQT0dZR+AlfA
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA,
linux-nfs-u79uwXL29TY76Z2rM5mHXA, bfields-uC3wQj2KruNg9hUCZPvPmw
Svcrdma was incorrectly allocating fastreg MRs and page lists using
RPCSVC_MAXPAGES, which can exceed the device capabilities. So limit
the depth to the minimum of RPCSVC_MAXPAGES and xprt->sc_frmr_pg_list_len.
Signed-off-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
---
Doug, this patch needs to be added after this commit:
e20684a xprtrdma, svcrdma: Convert to ib_alloc_mr
and before these commits:
af78181 cxgb3: Support ib_alloc_mr verb
b7e06cd iw_cxgb4: Support ib_alloc_mr verb
This will avoid a bisect window where NFSRDMA over cxgb4 is broken.
Bruce, please ACK if this commit looks good, and also if you're ok with
this flowing through Doug's rdma tree due to the dependencies.
---
net/sunrpc/xprtrdma/svc_rdma_transport.c | 6 ++++--
1 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c b/net/sunrpc/xprtrdma/svc_rdma_transport.c
index 8752a2d..11d5133 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_transport.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c
@@ -733,17 +733,19 @@ static struct svc_rdma_fastreg_mr *rdma_alloc_frmr(struct svcxprt_rdma *xprt)
struct ib_mr *mr;
struct ib_fast_reg_page_list *pl;
struct svc_rdma_fastreg_mr *frmr;
+ u32 num_sg;
frmr = kmalloc(sizeof(*frmr), GFP_KERNEL);
if (!frmr)
goto err;
- mr = ib_alloc_mr(xprt->sc_pd, IB_MR_TYPE_MEM_REG, RPCSVC_MAXPAGES);
+ num_sg = min_t(u32, RPCSVC_MAXPAGES, xprt->sc_frmr_pg_list_len);
+ mr = ib_alloc_mr(xprt->sc_pd, IB_MR_TYPE_MEM_REG, num_sg);
if (IS_ERR(mr))
goto err_free_frmr;
pl = ib_alloc_fast_reg_page_list(xprt->sc_cm_id->device,
- RPCSVC_MAXPAGES);
+ num_sg);
if (IS_ERR(pl))
goto err_free_mr;
--
1.7.1
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] svcrdma: limit FRMR page list lengths to device max
@ 2015-08-07 17:36 ` J. Bruce Fields
0 siblings, 0 replies; 6+ messages in thread
From: J. Bruce Fields @ 2015-08-07 17:36 UTC (permalink / raw)
To: Steve Wise; +Cc: dledford, linux-rdma, linux-nfs
On Fri, Aug 07, 2015 at 11:11:20AM -0500, Steve Wise wrote:
> Svcrdma was incorrectly allocating fastreg MRs and page lists using
> RPCSVC_MAXPAGES, which can exceed the device capabilities. So limit
> the depth to the minimum of RPCSVC_MAXPAGES and xprt->sc_frmr_pg_list_len.
>
> Signed-off-by: Steve Wise <swise@opengridcomputing.com>
> ---
> Doug, this patch needs to be added after this commit:
>
> e20684a xprtrdma, svcrdma: Convert to ib_alloc_mr
>
> and before these commits:
>
> af78181 cxgb3: Support ib_alloc_mr verb
> b7e06cd iw_cxgb4: Support ib_alloc_mr verb
>
> This will avoid a bisect window where NFSRDMA over cxgb4 is broken.
>
> Bruce, please ACK if this commit looks good, and also if you're ok with
> this flowing through Doug's rdma tree due to the dependencies.
Fine by me. On a quick check the only pending change in my tree nearby
there is 31193fe5f6fb "svcrdma: Remove svc_rdma_fastreg()", which
shouldn't conflict.
Acked-by: J. Bruce Fields <bfields@redhat.com>
--b.
> ---
> net/sunrpc/xprtrdma/svc_rdma_transport.c | 6 ++++--
> 1 files changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c b/net/sunrpc/xprtrdma/svc_rdma_transport.c
> index 8752a2d..11d5133 100644
> --- a/net/sunrpc/xprtrdma/svc_rdma_transport.c
> +++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c
> @@ -733,17 +733,19 @@ static struct svc_rdma_fastreg_mr *rdma_alloc_frmr(struct svcxprt_rdma *xprt)
> struct ib_mr *mr;
> struct ib_fast_reg_page_list *pl;
> struct svc_rdma_fastreg_mr *frmr;
> + u32 num_sg;
>
> frmr = kmalloc(sizeof(*frmr), GFP_KERNEL);
> if (!frmr)
> goto err;
>
> - mr = ib_alloc_mr(xprt->sc_pd, IB_MR_TYPE_MEM_REG, RPCSVC_MAXPAGES);
> + num_sg = min_t(u32, RPCSVC_MAXPAGES, xprt->sc_frmr_pg_list_len);
> + mr = ib_alloc_mr(xprt->sc_pd, IB_MR_TYPE_MEM_REG, num_sg);
> if (IS_ERR(mr))
> goto err_free_frmr;
>
> pl = ib_alloc_fast_reg_page_list(xprt->sc_cm_id->device,
> - RPCSVC_MAXPAGES);
> + num_sg);
> if (IS_ERR(pl))
> goto err_free_mr;
>
> --
> 1.7.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] svcrdma: limit FRMR page list lengths to device max
@ 2015-08-07 17:36 ` J. Bruce Fields
0 siblings, 0 replies; 6+ messages in thread
From: J. Bruce Fields @ 2015-08-07 17:36 UTC (permalink / raw)
To: Steve Wise
Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
linux-nfs-u79uwXL29TY76Z2rM5mHXA
On Fri, Aug 07, 2015 at 11:11:20AM -0500, Steve Wise wrote:
> Svcrdma was incorrectly allocating fastreg MRs and page lists using
> RPCSVC_MAXPAGES, which can exceed the device capabilities. So limit
> the depth to the minimum of RPCSVC_MAXPAGES and xprt->sc_frmr_pg_list_len.
>
> Signed-off-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
> ---
> Doug, this patch needs to be added after this commit:
>
> e20684a xprtrdma, svcrdma: Convert to ib_alloc_mr
>
> and before these commits:
>
> af78181 cxgb3: Support ib_alloc_mr verb
> b7e06cd iw_cxgb4: Support ib_alloc_mr verb
>
> This will avoid a bisect window where NFSRDMA over cxgb4 is broken.
>
> Bruce, please ACK if this commit looks good, and also if you're ok with
> this flowing through Doug's rdma tree due to the dependencies.
Fine by me. On a quick check the only pending change in my tree nearby
there is 31193fe5f6fb "svcrdma: Remove svc_rdma_fastreg()", which
shouldn't conflict.
Acked-by: J. Bruce Fields <bfields-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
--b.
> ---
> net/sunrpc/xprtrdma/svc_rdma_transport.c | 6 ++++--
> 1 files changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c b/net/sunrpc/xprtrdma/svc_rdma_transport.c
> index 8752a2d..11d5133 100644
> --- a/net/sunrpc/xprtrdma/svc_rdma_transport.c
> +++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c
> @@ -733,17 +733,19 @@ static struct svc_rdma_fastreg_mr *rdma_alloc_frmr(struct svcxprt_rdma *xprt)
> struct ib_mr *mr;
> struct ib_fast_reg_page_list *pl;
> struct svc_rdma_fastreg_mr *frmr;
> + u32 num_sg;
>
> frmr = kmalloc(sizeof(*frmr), GFP_KERNEL);
> if (!frmr)
> goto err;
>
> - mr = ib_alloc_mr(xprt->sc_pd, IB_MR_TYPE_MEM_REG, RPCSVC_MAXPAGES);
> + num_sg = min_t(u32, RPCSVC_MAXPAGES, xprt->sc_frmr_pg_list_len);
> + mr = ib_alloc_mr(xprt->sc_pd, IB_MR_TYPE_MEM_REG, num_sg);
> if (IS_ERR(mr))
> goto err_free_frmr;
>
> pl = ib_alloc_fast_reg_page_list(xprt->sc_cm_id->device,
> - RPCSVC_MAXPAGES);
> + num_sg);
> if (IS_ERR(pl))
> goto err_free_mr;
>
> --
> 1.7.1
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] svcrdma: limit FRMR page list lengths to device max
@ 2015-08-15 1:31 ` Doug Ledford
0 siblings, 0 replies; 6+ messages in thread
From: Doug Ledford @ 2015-08-15 1:31 UTC (permalink / raw)
To: Steve Wise; +Cc: linux-rdma, linux-nfs, bfields
[-- Attachment #1: Type: text/plain, Size: 1974 bytes --]
On 08/07/2015 12:11 PM, Steve Wise wrote:
> Svcrdma was incorrectly allocating fastreg MRs and page lists using
> RPCSVC_MAXPAGES, which can exceed the device capabilities. So limit
> the depth to the minimum of RPCSVC_MAXPAGES and xprt->sc_frmr_pg_list_len.
>
> Signed-off-by: Steve Wise <swise@opengridcomputing.com>
> ---
> Doug, this patch needs to be added after this commit:
>
> e20684a xprtrdma, svcrdma: Convert to ib_alloc_mr
>
> and before these commits:
>
> af78181 cxgb3: Support ib_alloc_mr verb
> b7e06cd iw_cxgb4: Support ib_alloc_mr verb
Done.
> This will avoid a bisect window where NFSRDMA over cxgb4 is broken.
>
> Bruce, please ACK if this commit looks good, and also if you're ok with
> this flowing through Doug's rdma tree due to the dependencies.
> ---
> net/sunrpc/xprtrdma/svc_rdma_transport.c | 6 ++++--
> 1 files changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c b/net/sunrpc/xprtrdma/svc_rdma_transport.c
> index 8752a2d..11d5133 100644
> --- a/net/sunrpc/xprtrdma/svc_rdma_transport.c
> +++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c
> @@ -733,17 +733,19 @@ static struct svc_rdma_fastreg_mr *rdma_alloc_frmr(struct svcxprt_rdma *xprt)
> struct ib_mr *mr;
> struct ib_fast_reg_page_list *pl;
> struct svc_rdma_fastreg_mr *frmr;
> + u32 num_sg;
>
> frmr = kmalloc(sizeof(*frmr), GFP_KERNEL);
> if (!frmr)
> goto err;
>
> - mr = ib_alloc_mr(xprt->sc_pd, IB_MR_TYPE_MEM_REG, RPCSVC_MAXPAGES);
> + num_sg = min_t(u32, RPCSVC_MAXPAGES, xprt->sc_frmr_pg_list_len);
> + mr = ib_alloc_mr(xprt->sc_pd, IB_MR_TYPE_MEM_REG, num_sg);
> if (IS_ERR(mr))
> goto err_free_frmr;
>
> pl = ib_alloc_fast_reg_page_list(xprt->sc_cm_id->device,
> - RPCSVC_MAXPAGES);
> + num_sg);
> if (IS_ERR(pl))
> goto err_free_mr;
>
>
--
Doug Ledford <dledford@redhat.com>
GPG KeyID: 0E572FDD
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] svcrdma: limit FRMR page list lengths to device max
@ 2015-08-15 1:31 ` Doug Ledford
0 siblings, 0 replies; 6+ messages in thread
From: Doug Ledford @ 2015-08-15 1:31 UTC (permalink / raw)
To: Steve Wise
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA,
linux-nfs-u79uwXL29TY76Z2rM5mHXA, bfields-uC3wQj2KruNg9hUCZPvPmw
[-- Attachment #1: Type: text/plain, Size: 2031 bytes --]
On 08/07/2015 12:11 PM, Steve Wise wrote:
> Svcrdma was incorrectly allocating fastreg MRs and page lists using
> RPCSVC_MAXPAGES, which can exceed the device capabilities. So limit
> the depth to the minimum of RPCSVC_MAXPAGES and xprt->sc_frmr_pg_list_len.
>
> Signed-off-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
> ---
> Doug, this patch needs to be added after this commit:
>
> e20684a xprtrdma, svcrdma: Convert to ib_alloc_mr
>
> and before these commits:
>
> af78181 cxgb3: Support ib_alloc_mr verb
> b7e06cd iw_cxgb4: Support ib_alloc_mr verb
Done.
> This will avoid a bisect window where NFSRDMA over cxgb4 is broken.
>
> Bruce, please ACK if this commit looks good, and also if you're ok with
> this flowing through Doug's rdma tree due to the dependencies.
> ---
> net/sunrpc/xprtrdma/svc_rdma_transport.c | 6 ++++--
> 1 files changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c b/net/sunrpc/xprtrdma/svc_rdma_transport.c
> index 8752a2d..11d5133 100644
> --- a/net/sunrpc/xprtrdma/svc_rdma_transport.c
> +++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c
> @@ -733,17 +733,19 @@ static struct svc_rdma_fastreg_mr *rdma_alloc_frmr(struct svcxprt_rdma *xprt)
> struct ib_mr *mr;
> struct ib_fast_reg_page_list *pl;
> struct svc_rdma_fastreg_mr *frmr;
> + u32 num_sg;
>
> frmr = kmalloc(sizeof(*frmr), GFP_KERNEL);
> if (!frmr)
> goto err;
>
> - mr = ib_alloc_mr(xprt->sc_pd, IB_MR_TYPE_MEM_REG, RPCSVC_MAXPAGES);
> + num_sg = min_t(u32, RPCSVC_MAXPAGES, xprt->sc_frmr_pg_list_len);
> + mr = ib_alloc_mr(xprt->sc_pd, IB_MR_TYPE_MEM_REG, num_sg);
> if (IS_ERR(mr))
> goto err_free_frmr;
>
> pl = ib_alloc_fast_reg_page_list(xprt->sc_cm_id->device,
> - RPCSVC_MAXPAGES);
> + num_sg);
> if (IS_ERR(pl))
> goto err_free_mr;
>
>
--
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
GPG KeyID: 0E572FDD
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread