Configuration of cq->cqe is lower than entries by 1

Configuration of cq->cqe is lower than entries by 1

[Amrani, Ram]

Hi Leon, All,
While inspecting MLX code as well as other vendors' I see that the actual number of cq->cqe is configured to be less by 1 than 'entries'. Why is that?

e.g.
struct ib_cq *mlx4_ib_create_cq(struct ib_device *ibdev,
                                const struct ib_cq_init_attr *attr,
                                struct ib_ucontext *context,
                                struct ib_udata *udata)
{
        ...
        cq->ibcq.cqe = entries - 1;
        ...
}


static int mlx4_alloc_resize_buf(struct mlx4_ib_dev *dev, struct mlx4_ib_cq *cq,
                                  int entries)
{
        ...
        cq->resize_buf->cqe = entries - 1;	// this is later copied to cq->ibcq.cqe
        ...
}

Thanks,
Ram

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Configuration of cq->cqe is lower than entries by 1

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1058 bytes --]

On Mon, Nov 14, 2016 at 11:07:53AM +0000, Amrani, Ram wrote:
> Hi Leon, All,
> While inspecting MLX code as well as other vendors' I see that the actual number of cq->cqe is configured to be less by 1 than 'entries'. Why is that?

There is addition of 1 in mlx4_ib_create_cq():
192         entries      = roundup_pow_of_two(entries + 1);
193         cq->ibcq.cqe = entries - 1;

The same goes for mlx4_alloc_resize_buf, just earlier in the stack.

>
> e.g.
> struct ib_cq *mlx4_ib_create_cq(struct ib_device *ibdev,
>                                 const struct ib_cq_init_attr *attr,
>                                 struct ib_ucontext *context,
>                                 struct ib_udata *udata)
> {
>         ...
>         cq->ibcq.cqe = entries - 1;
>         ...
> }
>
>
> static int mlx4_alloc_resize_buf(struct mlx4_ib_dev *dev, struct mlx4_ib_cq *cq,
>                                   int entries)
> {
>         ...
>         cq->resize_buf->cqe = entries - 1;	// this is later copied to cq->ibcq.cqe
>         ...
> }
>
> Thanks,
> Ram
>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

RE: Configuration of cq->cqe is lower than entries by 1

[Amrani, Ram]

> There is addition of 1 in mlx4_ib_create_cq():
> 192         entries      = roundup_pow_of_two(entries + 1);
> 193         cq->ibcq.cqe = entries - 1;

I thought something else might hide there.
Thanks,
Ram

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Configuration of cq->cqe is lower than entries by 1

[Amrani, Ram]

>> 192         entries      = roundup_pow_of_two(entries + 1);
>> 193         cq->ibcq.cqe = entries - 1;
>> 
>> I thought something else might hide there.
>> Hi Ram,
>> 
> For CQs, we always reserve an extra CQE, and thus report one less CQE to the
> user.
> This CQE is used for resize CQ operations.
> When the CQ is resized, the HW posts a CQE to indicate that the operation was
> completed on the original CQ buffer.
> Hope now it's clear.

Thanks. I had a guess this was related. 
Note that there might be different behavior than what you are describing. If the user requested for (2^n) -1 then it'll receive the exact number of entries it requested, without the (at least one) extra entries.
For example, if the user requests 63 entries (n=6) it'll receive the same number of entries:
192         entries      = roundup_pow_of_two(63+ 1);	// this gives 64
193         cq->ibcq.cqe = entries - 1;			// this gives 63 again.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Configuration of cq->cqe is lower than entries by 1

[Amrani, Ram]

 > Correct. We create it with 64(the result of the roundup) and let him work with
> 63 only.

ACK
 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html