Re: rdma_create_qp() and max_send_wr

[Yann Droneaud <ydroneaud-RlY5vtjFyJ3QT0dZR+AlfA@public.gmane.org>]

Le jeudi 19 mai 2011 à 18:46 +0300, Eli Cohen a écrit :
> On Thu, May 19, 2011 at 05:03:45PM +0200, Yann Droneaud wrote:
> > 
> > To sum up:
> > 
> > - ibv_qp_init_attr.max_recv_wr can be set to ibv_device_attr.max_qp_wr,
> > 16384 in my case,
> >  
> > - ibv_qp_init_attr.max_send_wr *cannot* be set to
> > ibv_device_attr.max_qp_wr, but it can be set to 16351.
> > 
> Thanks for investigating this. We'll check and send a fix. Meanwhile
> you can workaround this in your code.

I understand the problem (which is probably not really a problem):

In libmlx4/src/verbs.c:mlx4_create_qp()

        mlx4_calc_sq_wqe_size(&attr->cap, attr->qp_type, qp);

        /*
         * We need to leave 2 KB + 1 WQE of headroom in the SQ to
         * allow HW to prefetch.
         */
        qp->sq_spare_wqes = (2048 >> qp->sq.wqe_shift) + 1;
        qp->sq.wqe_cnt = align_queue_size(attr->cap.max_send_wr +
qp->sq_spare_wqes);
        qp->rq.wqe_cnt = align_queue_size(attr->cap.max_recv_wr);

For a requested SQ length of 1:

 qp->sq.wge_shift : 6
 qp->sq_spare_wges : 33
 attr->cap.max_send_wr : 1
 qp->sq_wge_cnt : 64

For a requested SQ length of 16351:

 qp->sq.wge_shift : 6
 qp->sq_spare_wges : 33
 attr->cap.max_send_wr : 16351
 qp->sq_wge_cnt : 16384

For a requested SQ length of 16384:

 qp->sq.wge_shift : 6
 qp->sq_spare_wges : 33
 attr->cap.max_send_wr : 16384
 qp->sq_wge_cnt : 32768

32768 is clearly above the ibv_device_attr.max_qp_wr limit, so the
creation of the QP failed later:

        ret = ibv_cmd_create_qp(pd, &qp->ibv_qp, attr, &cmd.ibv_cmd,
sizeof cmd,
                                &resp, sizeof resp);

ibv_cmd_create_qp() fails and returns 22.


If spare WQEs are taken in account here, it should be taken in account
in the data reported by ibv_query_device().

Regards.

-- 
Yann Droneaud
OPTEYA


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html