Re: rdma_create_qp() and max_send_wr

[Eli Cohen <eli-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>]

Hi Yan,
it appears that you're using quite an old firmware. Could you upgrade
the firmware to the latest version and check again the failure to
create a QP with the max depth. FW and burning tools can be downloaded
from www.mellanox.com.

Another possible reason for the failures you were seeing is the 4 MB
limit on locking memory from usersapce. Could you repeat the
experiment as root?

P.S: I checked on my setup and was able to create a QP with the max
size.

On Fri, Apr 22, 2011 at 12:20:45PM +0200, Yann Droneaud wrote:
> Hi,
> 
> Le jeudi 21 avril 2011 à 11:53 -0700, cait-DpaxOq6QOWMAvxtiuMwx3w@public.gmane.org a écrit :
> > An ENOMEM return does not mean that the subsystem *just* failed to
> > allocate system memory.
> 
> > The memory that could not be allocated could be device memory.
> >         
> 
> I'm also having some difficulties with system memory allocation.
> 
> In my test, a user is allowed to lock 4MBytes of memory, but not all
> this memory is available to ibv_reg_mr() since ibv_create_cq() and
> ibv_create_qp()/rdma_create_qp() lock memory respectively for CQ and QP.
> The question is how much memory is needed for the CQ and QP queues ?
> 
> In my case, the maximum message of size is 4MBytes - 20KBytes, for a CQ
> and QP (half duplex) queues length of 1.
> 
> Using message size of 128 bytes and less hit the QP WR limit of 16351
> length.
> 
> When using messages of size 256 bytes, I'm only able to register 2609152
> bytes, then CQ and QP (half duplex) queues are 10192 entries length. So
> they seems to requires about 1585152 bytes. Taking in account a fixed
> amount of reserved memory of 20KBytes, this give about 154 bytes per (CQ
> + QP (half duplex)) entry.
> 
> When doing the same math with size 512 and 1024, the size of (CQ + QP
> (half duplex)) is going down.
> 
> msg  512 memory 3395584 length 6632
> msg 1024 memory 3788800 length 3700
> msg 2048 memory 3985408 length 1946
> 
> Note that the memory used for the message is allocated as an aligned big
> chunk and registered as whole, and then sliced to be posted in WR. 
> 
> But the memory required for the CQ and QP elements (and other) is also
> subject to alignment to a page size.
> 
> At least, I known that CQ / QP "overhead" is not going to hurt users, if
> they are allocated "modern" memory limits, let's say 1GBytes ;)
> 
> Regards.
> 
> -- 
> Yann Droneaud
> OPTEYA
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html