RE: rdma_create_qp() and max_send_wr

[Yann Droneaud <ydroneaud-RlY5vtjFyJ3QT0dZR+AlfA@public.gmane.org>]

Hi,

Le jeudi 21 avril 2011 à 11:53 -0700, cait-DpaxOq6QOWMAvxtiuMwx3w@public.gmane.org a écrit :
> An ENOMEM return does not mean that the subsystem *just* failed to
> allocate system memory.

> The memory that could not be allocated could be device memory.
>         

I'm also having some difficulties with system memory allocation.

In my test, a user is allowed to lock 4MBytes of memory, but not all
this memory is available to ibv_reg_mr() since ibv_create_cq() and
ibv_create_qp()/rdma_create_qp() lock memory respectively for CQ and QP.
The question is how much memory is needed for the CQ and QP queues ?

In my case, the maximum message of size is 4MBytes - 20KBytes, for a CQ
and QP (half duplex) queues length of 1.

Using message size of 128 bytes and less hit the QP WR limit of 16351
length.

When using messages of size 256 bytes, I'm only able to register 2609152
bytes, then CQ and QP (half duplex) queues are 10192 entries length. So
they seems to requires about 1585152 bytes. Taking in account a fixed
amount of reserved memory of 20KBytes, this give about 154 bytes per (CQ
+ QP (half duplex)) entry.

When doing the same math with size 512 and 1024, the size of (CQ + QP
(half duplex)) is going down.

msg  512 memory 3395584 length 6632
msg 1024 memory 3788800 length 3700
msg 2048 memory 3985408 length 1946

Note that the memory used for the message is allocated as an aligned big
chunk and registered as whole, and then sliced to be posted in WR. 

But the memory required for the CQ and QP elements (and other) is also
subject to alignment to a page size.

At least, I known that CQ / QP "overhead" is not going to hurt users, if
they are allocated "modern" memory limits, let's say 1GBytes ;)

Regards.

-- 
Yann Droneaud
OPTEYA



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html