Re: CQ creation for RDMA use case

Re: CQ creation for RDMA use case

[Jason Gunthorpe]

On Tue, Jul 12, 2022 at 07:55:21PM +0000, Ajay Sharma wrote:
>    Hello,
> 
>    We have a situation where the hardware treats cq creation differently
>    when attaching it to RAW_QP use case vs the RC_QP use case. The
>    situation is as follows :
> 
> 
>    When creating the cq for RAW_QP_TYPE use case , the hw requires that
>    memory be registered with the hw but it does not yet attaches it to the
>    qp meaning it does not yet allocate any hw side resources for the cq.
> 
>    The hardware will allocate resources for this qp type only when it
>    see’s the work queue creation request and that’s when it allocates
>    resources for both cq and wq.
> 
> 
>    When creating cq for the RC_QP_TYPE use case the hardware creates
>    resources immediately on cq creation request.
> 
> 
>    The problem is from the user mode side there is no way to tell whether
>    this cq is intended for RAW QP use case or RC QP use case. Can we add a
>    flag in the  ibv_cq_init_attr_ex to differentiate the use case we are
>    trying to address.
> 
>    We have explored the option of customizing in the firmware side and at
>    this point it seems a very expensive operation and would like to avoid
>    that.

I don't have a good advice for you, this is very non-compliant if CQs
can only be used in certain configurations. The entire verbs is
designed around the idea that the CQ is a sharable object - having a
device that creates a CQ for every queue is very far away.

Why can you not correct this?

A flag is not so sufficient, it is an entire set of bad behavior that you
have to opt into.

You may consider a dv for creating a QP/CQ pair if that models your
HW.

Jason