Re: Lockless behavior for CQs in userspace

[Jason Gunthorpe <jgg@mellanox.com>]

On Wed, Mar 18, 2020 at 02:36:35PM +0200, Yishai Hadas wrote:
> On 3/18/2020 1:43 PM, Jason Gunthorpe wrote:
> > On Wed, Mar 18, 2020 at 09:52:27AM +0200, Yishai Hadas wrote:
> > 
> > > We can really consider extending the functionality of a parent domain that
> > > was just added for CQ in case it holds a thread domain for this purpose.
> > > However, current code enforces creating a dedicated BF as part of thread
> > > domain unless we extend ibv_alloc_td() to ask only for marking the single
> > > thread functionality.
> > 
> > Doesn't the CQ need a BF too?
> 
> Not really, it uses a per-allocated device one for writing 8 bytes for
> doorbell, applicable only in the arm/events mode.

But this also wastes a BF, if the app is only running a single
lockless TD then the CQ could use the BF from the TD, shared with the
QP and not allocate another one

> > > The existing alternative is or to use the legacy ENV that mentioned above or
> > > to use the ibv_cq_ex functionality which upon its creation gets an explicit
> > > flag for single threaded one (i.e. IBV_CREATE_CQ_ATTR_SINGLE_THREADED).
> > 
> > An apps that don't want the BF can always use
> > IBV_CREATE_CQ_ATTR_SINGLE_THREADED
> > 
> 
> This requires moving to the new polling mechanism where this option is
> really applicable (see a3a31e8dc6a3fbf577dd8b12742d0c70cf63bd29).

cq_ex is pollable the normal way if converted with ibv_cq_ex_to_cq()

It is also a bug if the old env driven lock elimination doesn't trigger
for normal CQ poll if IBV_CREATE_CQ_ATTR_SINGLE_THREADED is specified.

Jason