mlx4 kernel parameters

mlx4 kernel parameters

[Rui Machado]

Hi list,

on trying to change the mlx4_core parameters log_num_mtt and
log_num_mpt to 21 and 18 respectively, I see the following message:

mlx4_core 0000:07:00.0: Failed to initialize memory region table, aborting.

The objective is to be able to memory-pin large portion of a machine
with lots of RAM, otherwise we are limited to about 16GB.
Does anyone has an hint or thoughts on what might be the problem?

Thanks in advance,
Rui
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mlx4 kernel parameters

[Eli Cohen]

On Wed, Feb 03, 2010 at 11:02:07AM +0100, Rui Machado wrote:
> 
> on trying to change the mlx4_core parameters log_num_mtt and
> log_num_mpt to 21 and 18 respectively, I see the following message:

The MPTs and MTTs are managed by bitmap allocators within the mlx4
core drivers. These allocators requires physically contigous memory
whose size is proportional to the number of controlled resources.
Since Linux does has a limitation on the order of allocated contigous
memory, the allocator might fail to initialize.
> 
> mlx4_core 0000:07:00.0: Failed to initialize memory region table, aborting.
> 
> The objective is to be able to memory-pin large portion of a machine
> with lots of RAM, otherwise we are limited to about 16GB.
> Does anyone has an hint or thoughts on what might be the problem?

For this you can use the parameter log_mtts_per_seg which allows to
define the granulairty of MTTs allocated. So for a given number of MTT
segements (which is what the allocator really allocates), you control
more registered memeory.
As for the number of MPTs, please don't use too large numbers - you
usually don't have to increase this number. 


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mlx4 kernel parameters

[Rui Machado]

Hi there,

>>
>> The objective is to be able to memory-pin large portion of a machine
>> with lots of RAM, otherwise we are limited to about 16GB.
>> Does anyone has an hint or thoughts on what might be the problem?
>
> For this you can use the parameter log_mtts_per_seg which allows to
> define the granulairty of MTTs allocated. So for a given number of MTT
> segements (which is what the allocator really allocates), you control
> more registered memeory.

Great, thanks. I increased the log_mtts_per_seg to 5 and was able to
register more memory. But one question remaining: what is then the
maximum memory that one is able to register? 5 is the maximum value to
this parameter, so what does it mean?

cheers,
Rui
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mlx4 kernel parameters

[Eli Cohen]

On Wed, Feb 03, 2010 at 03:17:17PM +0100, Rui Machado wrote:
> 
> Great, thanks. I increased the log_mtts_per_seg to 5 and was able to
> register more memory. But one question remaining: what is then the
> maximum memory that one is able to register? 5 is the maximum value to
> this parameter, so what does it mean?
> 
The overall size of memory that can be registered, from the HCA's
perspective, depends on the number of MTT segments, the number of
MTTs in each segment and PAGE_SIZE. The number of MTT segments however
is limited by the MTT allocator's ability to allocate memory for
managing allocations. Roughly, this is 128K * 8 MTT segments.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html