[PATCH] ib: mlx4 map fix for large hugetlb areas

[PATCH] ib: mlx4 map fix for large hugetlb areas

[Dan aloni]

When an application tries to map a large area of hugetlb pages, the
mlx4 code rightfully reachs handle_hugetlb_user_mr(). This function
uses an intermediate allocation in order to pass the list of big pages
to mlx4_write_mtt(), however that this allocation, when passing over
for example, 128GB of hugetlb memory (63365 pages), maps to 65536*8
bytes of memory, which is not supported by kmalloc() under various
kernel configurations.

It would be prudent to use vmalloc() here.

NOTE: core_umem will still hold 1GB of descriptors for mapping 128GB
of memory, even though it's hugetlb. This needs to be optimized as
well.

Signed-off-by: Dan Aloni <alonid-IGuEOkGjbyfby3iVrkZq2A@public.gmane.org>
---
 drivers/infiniband/hw/mlx4/mr.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/hw/mlx4/mr.c b/drivers/infiniband/hw/mlx4/mr.c
index e6c096c..59f2909 100644
--- a/drivers/infiniband/hw/mlx4/mr.c
+++ b/drivers/infiniband/hw/mlx4/mr.c
@@ -140,7 +140,7 @@ static int handle_hugetlb_user_mr(struct ib_pd *pd, struct mlx4_ib_mr *mr,
 	int off = start & (HPAGE_SIZE - 1);
 
 	n = DIV_ROUND_UP(off + umem->length, HPAGE_SIZE);
-	arr = kmalloc(n * sizeof *arr, GFP_KERNEL);
+	arr = vmalloc(n * sizeof *arr);
 	if (!arr)
 		return -ENOMEM;
 
@@ -178,7 +178,7 @@ static int handle_hugetlb_user_mr(struct ib_pd *pd, struct mlx4_ib_mr *mr,
 	err = mlx4_write_mtt(dev->dev, &mr->mmr.mtt, 0, n, arr);
 
 out:
-	kfree(arr);
+	vfree(arr);
 	return err;
 #else
 	return -ENOSYS;
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] ib: mlx4 map fix for large hugetlb areas

[Roland Dreier]

On Mon, Jun 6, 2011 at 4:29 AM, Dan aloni <alonid-IGuEOkGjbyfby3iVrkZq2A@public.gmane.org> wrote:
> --- a/drivers/infiniband/hw/mlx4/mr.c
> +++ b/drivers/infiniband/hw/mlx4/mr.c
> @@ -140,7 +140,7 @@ static int handle_hugetlb_user_mr(struct ib_pd *pd, struct mlx4_ib_mr *mr,
>        int off = start & (HPAGE_SIZE - 1);
>
>        n = DIV_ROUND_UP(off + umem->length, HPAGE_SIZE);
> -       arr = kmalloc(n * sizeof *arr, GFP_KERNEL);
> +       arr = vmalloc(n * sizeof *arr);
>        if (!arr)
>                return -ENOMEM;
>

I'm not able to find anywhere in the mlx4 driver where this patch
would have a chance at applying??

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] ib: mlx4 map fix for large hugetlb areas

[Eli Cohen]

On Mon, Jun 06, 2011 at 09:09:42AM -0700, Roland Dreier wrote:
> On Mon, Jun 6, 2011 at 4:29 AM, Dan aloni <alonid-IGuEOkGjbyfby3iVrkZq2A@public.gmane.org> wrote:
> > --- a/drivers/infiniband/hw/mlx4/mr.c
> > +++ b/drivers/infiniband/hw/mlx4/mr.c
> > @@ -140,7 +140,7 @@ static int handle_hugetlb_user_mr(struct ib_pd *pd, struct mlx4_ib_mr *mr,
> >        int off = start & (HPAGE_SIZE - 1);
> >
> >        n = DIV_ROUND_UP(off + umem->length, HPAGE_SIZE);
> > -       arr = kmalloc(n * sizeof *arr, GFP_KERNEL);
> > +       arr = vmalloc(n * sizeof *arr);
> >        if (!arr)
> >                return -ENOMEM;
> >
> 
> I'm not able to find anywhere in the mlx4 driver where this patch
> would have a chance at applying??
> 

I submitted this patch about 2 years ago. As far as I recall there was
a problem to obtain the huge page size for powerpc archs and that's
were the patch died. We're are using it in OFED for x86_64 and it
gives significant improvement when registering huge pages. I will
re-submit an up to date version.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html