Question on mlx4 log_num_mtt and log_mtt_per_seg setting

Question on mlx4 log_num_mtt and log_mtt_per_seg setting

[Bharath Ramesh]

[-- Attachment #1: Type: text/plain, Size: 996 bytes --]

I am trying to find the correct setting for log_num_mtt and 
log_mtt_per_seg should be for our cluster. Each node has 64G of RAM. We 
are having issues when running MPI applications, the error is related to 
registering memory. The current setting log_num_mtt=24 and 
log_mtt_per_seg=1. There are lot of conflicting documentations available 
with regards how these settings should be changed. I was wondering if 
the community could explain how these settings works so that we can come 
up with the correct settings that will work in our environment. This 
document [1] specifically says that log_mtt_per_seg should always be 1. 
However, OpenMPI mailing list post [2] talks about different value. Any 
help on this is appreciated. I am not subscribed to the list so would 
really appreciate if I am copied in the replies.

[1] http://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem
[2] http://www.open-mpi.org/community/lists/users/2011/09/17222.php

-- 
Bharath



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3738 bytes --]

Re: Question on mlx4 log_num_mtt and log_mtt_per_seg setting

[Eli Cohen]

Hi Bharath,

each MTT entry is used to cover a whole page. Assuming page size in
you system equals 4K, the max total of registered memory is 2^(12+24)
which is 64 GB.
The log_mtt_per_seg param denotes the granularity of the allocator
managing MTTs. So, in your case, MTTs are allocated in granularity of
2. So if you try to allocate 3 MTTs, you'll end up consuming 4.
Moreover, the MTT allocator is a buddy allocator that allocates power
of 2 MTT segments. So if you register memory that requires (2^15 + 1)
MTTs, you'll actually consume 2^16 MTTs!

In your case I would try to use 2^25 MTTs and log_mtt_per_seg=1. If
the driver fails to load (probably because the allocator fails to
allocate memory), try to use 2^24 and 2^2 etc.

I hope that helps.



On Mon, Jan 21, 2013 at 5:08 AM, Bharath Ramesh <bramesh-PjAqaU27lzQ@public.gmane.org> wrote:
> I am trying to find the correct setting for log_num_mtt and log_mtt_per_seg
> should be for our cluster. Each node has 64G of RAM. We are having issues
> when running MPI applications, the error is related to registering memory.
> The current setting log_num_mtt=24 and log_mtt_per_seg=1. There are lot of
> conflicting documentations available with regards how these settings should
> be changed. I was wondering if the community could explain how these
> settings works so that we can come up with the correct settings that will
> work in our environment. This document [1] specifically says that
> log_mtt_per_seg should always be 1. However, OpenMPI mailing list post [2]
> talks about different value. Any help on this is appreciated. I am not
> subscribed to the list so would really appreciate if I am copied in the
> replies.
>
> [1] http://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem
> [2] http://www.open-mpi.org/community/lists/users/2011/09/17222.php
>
> --
> Bharath
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Question on mlx4 log_num_mtt and log_mtt_per_seg setting

[Bharath Ramesh]

[-- Attachment #1: Type: text/plain, Size: 2243 bytes --]

Hi Eli,
     Thanks a lot for this explanation of the MTT and how 
log_mtt_per_seg log_num_mtt work. I will play around with these values 
and figure which one works best for our cluster + application environment.

--
Bharath

On 1/21/2013 11:57 AM, Eli Cohen wrote:
> Hi Bharath,
>
> each MTT entry is used to cover a whole page. Assuming page size in
> you system equals 4K, the max total of registered memory is 2^(12+24)
> which is 64 GB.
> The log_mtt_per_seg param denotes the granularity of the allocator
> managing MTTs. So, in your case, MTTs are allocated in granularity of
> 2. So if you try to allocate 3 MTTs, you'll end up consuming 4.
> Moreover, the MTT allocator is a buddy allocator that allocates power
> of 2 MTT segments. So if you register memory that requires (2^15 + 1)
> MTTs, you'll actually consume 2^16 MTTs!
>
> In your case I would try to use 2^25 MTTs and log_mtt_per_seg=1. If
> the driver fails to load (probably because the allocator fails to
> allocate memory), try to use 2^24 and 2^2 etc.
>
> I hope that helps.
>
>
>
> On Mon, Jan 21, 2013 at 5:08 AM, Bharath Ramesh <bramesh-PjAqaU27lzQ@public.gmane.org> wrote:
>> I am trying to find the correct setting for log_num_mtt and log_mtt_per_seg
>> should be for our cluster. Each node has 64G of RAM. We are having issues
>> when running MPI applications, the error is related to registering memory.
>> The current setting log_num_mtt=24 and log_mtt_per_seg=1. There are lot of
>> conflicting documentations available with regards how these settings should
>> be changed. I was wondering if the community could explain how these
>> settings works so that we can come up with the correct settings that will
>> work in our environment. This document [1] specifically says that
>> log_mtt_per_seg should always be 1. However, OpenMPI mailing list post [2]
>> talks about different value. Any help on this is appreciated. I am not
>> subscribed to the list so would really appreciate if I am copied in the
>> replies.
>>
>> [1] http://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem
>> [2] http://www.open-mpi.org/community/lists/users/2011/09/17222.php
>>
>> --
>> Bharath
>>
>>


-- 
Bharath



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3738 bytes --]