memory region limit at 32 GB?

memory region limit at 32 GB?

[Albert Strasheim]

Hello all

Is there some kind of limit that would prevent me from registering
more than 32 GiB worth of memory regions with ibv_reg_mr in
libibverbs?

>From strace I can see:

open("/dev/infiniband/uverbs0", O_RDWR) = 8
...
write(8, "\t\0\0\0\f\0\3\0\340o\255\35\377\177\0\0\0P\211\336\26\177\0\0\0\0\0@\0\0\0\0\0P\211\336\26\177\0\0\1\0\0\0\3\0\0\0",
48) = -1 ENOMEM (Cannot allocate memory)

when trying to register my 33rd 1 GiB buffer.

cat /proc/meminfo
MemTotal:       198075136 kB
MemFree:        186688448 kB
CommitLimit:    185892796 kB

so it doesn't look like a memory thing.

IB adapter details:

hca_id: mlx4_0
        transport:                      InfiniBand (0)
        fw_ver:                         2.8.600
        vendor_id:                      0x02c9
        vendor_part_id:                 26428
        hw_ver:                         0xB0
        board_id:                       MT_0FC0110009

I'm using libibverbs 1.1.6 on kernel 3.2.1-1.fc16.x86_64.

Regards

Albert
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: memory region limit at 32 GB?

[Roland Dreier]

On Fri, Jan 20, 2012 at 9:39 AM, Albert Strasheim <fullung-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Is there some kind of limit that would prevent me from registering
> more than 32 GiB worth of memory regions with ibv_reg_mr in
> libibverbs?

Yes, by default mlx4 allocates a limited amount of adapter resources
for tracking memory regions.  I forget the exact limits but 32GB looks
reasonable...

In mlx4/main.c, there is

static struct mlx4_profile default_profile = {
        .num_qp         = 1 << 18,
        .num_srq        = 1 << 16,
        .rdmarc_per_qp  = 1 << 4,
        .num_cq         = 1 << 16,
        .num_mcg        = 1 << 13,
        .num_mpt        = 1 << 19,
        .num_mtt        = 1 << 20,
};

and I think if you bump num_mtt up a few powers of 2 (eg
1 << 22) then you should be able to register more.

(num_mpt controls the number of MRs but I guess 33 is
not near the limit of 512K yet ;)

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: memory region limit at 32 GB?

[Roland Dreier]

By the way, I wonder if we should auto-tune num_mtt so
we have enough MTTs to cover, say, 4X of the amount of
physical memory.

How much RAM do you have in your system?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: memory region limit at 32 GB?

[Albert Strasheim]

Hello

On Fri, Jan 20, 2012 at 8:03 PM, Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
> By the way, I wonder if we should auto-tune num_mtt so
> we have enough MTTs to cover, say, 4X of the amount of
> physical memory.
> How much RAM do you have in your system?

192 GB.

FYI, new Sandy Bridge motherboards will be out soon that do 512 GB and
even 768 GB.

Regards

Albert
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: memory region limit at 32 GB?

[Roland Dreier]

On Fri, Jan 20, 2012 at 10:30 AM, Albert Strasheim <fullung-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> FYI, new Sandy Bridge motherboards will be out soon that do 512 GB and
> even 768 GB.

Yeah, Cisco UCS C260 goes up to 1TB in a 2socket 2U already so we really
should fix this.  I'll cook something up.

The good news is that the overhead is that we use more memory for adapter
context, but if you have 1TB of RAM you probably don't care about a few
more MB over overhead :)

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: memory region limit at 32 GB?

[Joseph Glanville]

Hi,

Romley is likely to support >1TB in 2 and 4 socket boards.
I rememeber when it was hard to fit 1TB in a rack let alone a 1U chasis..

Joseph.

On 21 January 2012 05:38, Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
> On Fri, Jan 20, 2012 at 10:30 AM, Albert Strasheim <fullung-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> FYI, new Sandy Bridge motherboards will be out soon that do 512 GB and
>> even 768 GB.
>
> Yeah, Cisco UCS C260 goes up to 1TB in a 2socket 2U already so we really
> should fix this.  I'll cook something up.
>
> The good news is that the overhead is that we use more memory for adapter
> context, but if you have 1TB of RAM you probably don't care about a few
> more MB over overhead :)
>
>  - R.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Founder | Director | VP Research
Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56
99 52 | Mobile: 0428 754 846
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: memory region limit at 32 GB?

[Albert Strasheim]

Hello

On Fri, Jan 20, 2012 at 8:00 PM, Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
> On Fri, Jan 20, 2012 at 9:39 AM, Albert Strasheim <fullung-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> Is there some kind of limit that would prevent me from registering
>> more than 32 GiB worth of memory regions with ibv_reg_mr in
>> libibverbs?
> Yes, by default mlx4 allocates a limited amount of adapter resources
> for tracking memory regions.  I forget the exact limits but 32GB looks
> reasonable...

I think we've run into this in a "real" application (i.e., not a test).

Is this limit per-process or per-HCA or per-machine? It seems it might
not be per-process.

Any chance to get this limit increased upstream?

Regards

Albert
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: memory region limit at 32 GB?

[Roland Dreier]

On Thu, Aug 2, 2012 at 8:03 AM, Albert Strasheim <fullung-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Did something happen with this discussion?
>
> possible bug when scaling MTT table size with system ram
> http://www.mail-archive.com/linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg11728.html

I thought I had merged a patch to use vmalloc for bigger allocations
but looking at what's actually upstream I don't see it.

Or, can you resend that if indeed it is missing?

Thanks,
  Roland
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: memory region limit at 32 GB?

[Yishai Hadas]

Hi Albert,
I have sent yesterday a patch that may enable working with 1TB of registered memory.
Its subject is: [PATCH] net/mlx4_core: Fix num mtt issues.

Can you test it and let us know whether your issue was solved ?
In the meanwhile I have found the root cause of the limit of log_num_mtt of 28.
Plan to send in coming days an extra patch that enables value of 31 which match to 8TB.
Yishai


On 8/2/2012 5:31 PM, Albert Strasheim wrote:
> Hello
>
> On Fri, Jan 20, 2012 at 8:00 PM, Roland Dreier<roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org>  wrote:
>> On Fri, Jan 20, 2012 at 9:39 AM, Albert Strasheim<fullung-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>  wrote:
>>> Is there some kind of limit that would prevent me from registering
>>> more than 32 GiB worth of memory regions with ibv_reg_mr in
>>> libibverbs?
>> Yes, by default mlx4 allocates a limited amount of adapter resources
>> for tracking memory regions.  I forget the exact limits but 32GB looks
>> reasonable...
> I think we've run into this in a "real" application (i.e., not a test).
>
> Is this limit per-process or per-HCA or per-machine? It seems it might
> not be per-process.
>
> Any chance to get this limit increased upstream?
>
> Regards
>
> Albert
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: memory region limit at 32 GB?

[Or Gerlitz]

On Mon, Aug 6, 2012 at 6:51 PM, Yishai Hadas <yishaih-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
> In the meanwhile I have found the root cause of the limit of log_num_mtt of 28. Plan to
> send in coming days an extra patch that enables value of 31 which match to 8TB.

nice doing. Just an ordering comment, for the upstream inclusion, it
seems that what we need is two patches

1. fix for some issues e.g around ints/longs and friends which is
likely to be the source of the bug which once fixed eliminated the
crashes you hit in the 28-30/31 range

2. the vmalloc etc patch you posted here

This way you avoid reverting some code from patch #2 in patch #1 - it
helps with maintainance

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: memory region limit at 32 GB?

[Yishai Hadas]

On 8/7/2012 10:37 AM, Or Gerlitz wrote:
> On Mon, Aug 6, 2012 at 6:51 PM, Yishai Hadas<yishaih-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>  wrote:
>> In the meanwhile I have found the root cause of the limit of log_num_mtt of 28. Plan to
>> send in coming days an extra patch that enables value of 31 which match to 8TB.
> nice doing. Just an ordering comment, for the upstream inclusion, it
> seems that what we need is two patches
>
> 1. fix for some issues e.g around ints/longs and friends which is
> likely to be the source of the bug which once fixed eliminated the
> crashes you hit in the 28-30/31 range
>
> 2. the vmalloc etc patch you posted here
>
> This way you avoid reverting some code from patch #2 in patch #1 - it
> helps with maintainance
>
> Or.
I prefer that first patch will hold the limit of 28 as part of - 
otherwise system with large memory may fail to load,and the vmalloc fix 
won't help.
Second patch fixes the limit and enables mapping of 8TB.

Second patch was already sent.
Yishai
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html