Re: [PATCH] librdmacm: lazy initialization for ib devices]

Re: [PATCH] librdmacm: lazy initialization for ib devices]

[Shamir Rabinovith]

----- Forwarded message from Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> -----

So what exact capability is under the spot here? and how it's related
to a certain HCA driver and not to the kernel uverbs layer?

----- End forwarded message -----

ibv_open_device fail when opening the hca many times. ucma_init call
this api a-priori on all the devices for any process that use the
library. The patch remove this a-priori open and open the device only
when verbs are really needed. I do not think its issue at the verbs
but more issue in how the librdmacm use the verbs.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] librdmacm: lazy initialization for ib devices]

[Or Gerlitz]

On 27/03/2014 10:23, Shamir Rabinovith wrote:
> So what exact capability is under the spot here? and how it's related
> to a certain HCA driver and not to the kernel uverbs layer?
>
> ----- End forwarded message -----
>
> ibv_open_device fail when opening the hca many times.
sorry, but you didn't provide an answer to my question - why is 
ibv_open_device failing if called from N > M (what's
N and what's M) processes/threads.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] librdmacm: lazy initialization for ib devices]

[Shamir Rabinovith]

On Thu, Mar 27, 2014 at 10:38:19AM +0200, Or Gerlitz wrote:
> sorry, but you didn't provide an answer to my question - why is
> ibv_open_device failing if called from N > M (what's
> N and what's M) processes/threads.
> 
> Or.

The limit is due to firmware UAR BAR size. The issue happen here:
ib_uverbs_get_context -> ibdev->alloc_ucontext -> mlx4_ib_alloc_ucontext
-> mlx4_uar_alloc -> mlx4_bitmap_alloc . mlx4_QUERY_DEV_CAP return value
of 4194304 when reading the UAR from the firmware which turn to UAR
table size of 1024 in mlx4_init_uar_table. This is the result of
dividing the UAR BAR size by page size. We can try to extend the size of
the UAR BAR for Oracle servers as we know the HW and we can verify it
will boot with this change but anyhow we will still hit the limit of
number of processes that can open the library. More then this - UAR is
set to value that will allow less stronger machines to boot properly so
if you do not know which machine will use the firmware you cannot change
this limit as all. So the overall is that I think we need this change
anyhow, with or without the UAR BAR size change.

BR, Shamir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] librdmacm: lazy initialization for ib devices]

[Or Gerlitz]

On Sun, Mar 30, 2014 at 1:27 PM, Shamir Rabinovith
<shamir.rabinovitch-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote:
> On Thu, Mar 27, 2014 at 10:38:19AM +0200, Or Gerlitz wrote:
>> sorry, but you didn't provide an answer to my question - why is
>> ibv_open_device failing if called from N > M (what's
>> N and what's M) processes/threads.
>>
>> Or.
>
> The limit is due to firmware UAR BAR size. The issue happen here:
> ib_uverbs_get_context -> ibdev->alloc_ucontext -> mlx4_ib_alloc_ucontext
> -> mlx4_uar_alloc -> mlx4_bitmap_alloc .
[...]

So what **exact** operation in librdmacm fails after N processes
loaded the library on a node with M HCAs and what are N and M?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html