Maximum size for memory registration (ibv_reg_mr)

Maximum size for memory registration (ibv_reg_mr)

[Bharath Ramesh]

[-- Attachment #1: Type: text/plain, Size: 318 bytes --]

Is there any limit on the maximum size of memory that can be registered
using ibv_reg_mr. I am using OFED-1.4.2 stack, and when I try to pass a size
greater than 16GB ibv_reg_mr returns NULL. The total amount of physical
memory on the node is 48GB, max locked memory(ulimit -l) is set to
unlimited.

Regards,

Bharath

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 4440 bytes --]

Re: Maximum size for memory registration (ibv_reg_mr)

[Roland Dreier]

 > Is there any limit on the maximum size of memory that can be registered
 > using ibv_reg_mr. I am using OFED-1.4.2 stack, and when I try to pass a size
 > greater than 16GB ibv_reg_mr returns NULL. The total amount of physical
 > memory on the node is 48GB, max locked memory(ulimit -l) is set to
 > unlimited.

The limit depends on the HCA.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Maximum size for memory registration (ibv_reg_mr)

[Bharath Ramesh]

[-- Attachment #1: Type: text/plain, Size: 601 bytes --]

> -----Original Message-----
> From: Roland Dreier [mailto:rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org]
> 
>  > Is there any limit on the maximum size of memory that can be
> registered
>  > using ibv_reg_mr. I am using OFED-1.4.2 stack, and when I try to
> pass a size
>  > greater than 16GB ibv_reg_mr returns NULL. The total amount of
> physical
>  > memory on the node is 48GB, max locked memory(ulimit -l) is set to
>  > unlimited.
> 
> The limit depends on the HCA.

max_mr_size returned by calling ibv_devinfo is 0xffffffffffffffff. Does this
mean the size is unlimited?

Regards,

Bharath

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 4440 bytes --]

Re: Maximum size for memory registration (ibv_reg_mr)

[Roland Dreier]

 > max_mr_size returned by calling ibv_devinfo is 0xffffffffffffffff. Does this
 > mean the size is unlimited?

It means that the query function might be returning bad data.  What
adapter is it?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Maximum size for memory registration (ibv_reg_mr)

[Bharath Ramesh]

[-- Attachment #1: Type: text/plain, Size: 532 bytes --]

> -----Original Message-----
> From: Roland Dreier [mailto:rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org]
> 
>  > max_mr_size returned by calling ibv_devinfo is 0xffffffffffffffff.
> Does this
>  > mean the size is unlimited?
> 
> It means that the query function might be returning bad data.  What
> adapter is it?

I don't know the exact adapter it is a QDR ConnectX adapter with board_id:
MT_0C40110009. Is there any documentations where one can find such
information if the query function returns bad data.

Regards,

Bharath

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 4440 bytes --]

Re: Maximum size for memory registration (ibv_reg_mr)

[Roland Dreier]

 > I don't know the exact adapter it is a QDR ConnectX adapter with board_id:
 > MT_0C40110009. Is there any documentations where one can find such
 > information if the query function returns bad data.

All Mellanox adapters have some limit on the amount of memory they can
register.  You can mess around with module parameters to adjust it.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Maximum size for memory registration (ibv_reg_mr)

[Tziporet Koren]

On 9/25/2010 12:57 AM, Roland Dreier wrote:
> All Mellanox adapters have some limit on the amount of memory they can
> register.  You can mess around with module parameters to adjust it.
>
>    
You can use mlx4_core module parameter (log_mtts_per_seg) to enlarge 
number of MTTs per segment.
This enable to register more memory with the same number of segments.
In OFED 1.5.2 the max value was enlarged to 7

Tziporet


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Maximum size for memory registration (ibv_reg_mr)

[Or Gerlitz]

Roland Dreier wrote:
>> max_mr_size returned by calling ibv_devinfo is 0xffffffffffffffff. Does this mean the size is unlimited?
> It means that the query function might be returning bad data.  What adapter is it?

Yep, mlx4_ib_query_device does props->max_mr_size = ~0ull ... 

Jack, what would it take to fix that? I took a look and other settings are taken from 
dev->dev->caps.xxx where caps is of type mlx4_caps which is in turn filled @ drivers/net/mlx4/main.c 
:: mlx4_dev_cap I wasn't sure what field/computation would exactly account for later setting max_mr_size

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Maximum size for memory registration (ibv_reg_mr)

[Eli Cohen]

On Thu, Oct 07, 2010 at 11:40:02AM +0200, Or Gerlitz wrote:
> Roland Dreier wrote:
> >> max_mr_size returned by calling ibv_devinfo is 0xffffffffffffffff. Does this mean the size is unlimited?
> > It means that the query function might be returning bad data.  What adapter is it?
> 
> Yep, mlx4_ib_query_device does props->max_mr_size = ~0ull ... 
> 
> Jack, what would it take to fix that? I took a look and other settings are taken from 
> dev->dev->caps.xxx where caps is of type mlx4_caps which is in turn filled @ drivers/net/mlx4/main.c 
> :: mlx4_dev_cap I wasn't sure what field/computation would exactly account for later setting max_mr_size
> 
It is not that trivial to calculate this value.
If you create a MR in kernel, it covers the entire address space and
the HCA does not pose any limit since you do not consume MTTs.
And if you use MTTs then the page size is a parameter in this
calculation - huge page, regular page etc.

Do leaving it as is seems to be the most accurate thing...
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Maximum size for memory registration (ibv_reg_mr)

[Or Gerlitz]

Eli Cohen wrote:
> If you create a MR in kernel, it covers the entire address space and 
> the HCA does not pose any limit since you do not consume MTTs. And if 
> you use MTTs then the page size is a parameter in this calculation - 
> huge page, regular page etc.
I agree that the kernel case is not of large interest, even though what 
you wrote only applies for dma mr, when some FMR scheme is used, MTTs 
are consumed, ofcourse. But, typically, kernel code will not go to the 
order of giga-bytes, and in other words will not hit the HCA limit.

> Do leaving it as is seems to be the most accurate thing...
I would implement it for regular pages and drop a note in the libibverbs 
man page that if huge pages are used (well, the huge pages patch set 
isn't fully merged, maybe its about time to make this happen...) then 
the actual limit is bigger, e.g follows the proportion between the 
regular to the huge pages used.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Maximum size for memory registration (ibv_reg_mr)

[Eli Cohen]

On Thu, Oct 07, 2010 at 01:21:36PM +0200, Or Gerlitz wrote:
> I would implement it for regular pages and drop a note in the
> libibverbs man page that if huge pages are used (well, the huge
> pages patch set isn't fully merged, maybe its about time to make
> this happen...) then the actual limit is bigger, e.g follows the
> proportion between the regular to the huge pages used.
> 
What you're proposing is to change something that is not accurate in
another not accurate thing. Even when you don't use huge pages, a
calculation based on regular pages would be inaccurate too. That's
because the allocator allocates MTTs in 2^n quantities while the
registered memory need not be sized like that. Moreover, such
calculation is less accurate then what we have now.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html