Re: ibv_req_notify_cq clarification

[Jason Gunthorpe <jgg@ziepe.ca>]

On Sun, Feb 21, 2021 at 11:25:02AM +0200, Gal Pressman wrote:
> On 18/02/2021 18:23, Jason Gunthorpe wrote:
> > On Thu, Feb 18, 2021 at 05:52:16PM +0200, Gal Pressman wrote:
> >> On 18/02/2021 14:53, Jason Gunthorpe wrote:
> >>> On Thu, Feb 18, 2021 at 11:13:43AM +0200, Gal Pressman wrote:
> >>>> I'm a bit confused about the meaning of the ibv_req_notify_cq() verb:
> >>>> "Upon the addition of a new CQ entry (CQE) to cq, a completion event will be
> >>>> added to the completion channel associated with the CQ."
> >>>>
> >>>> What is considered a new CQE in this case?
> >>>> The next CQE from the user's perspective, i.e. any new CQE that wasn't consumed
> >>>> by the user's poll cq?
> >>>> Or any new CQE from the device's perspective?
> >>>
> >>> new CQE from the device perspective.
> >>>
> >>>> For example, if at the time of ibv_req_notify_cq() call the CQ has received 100
> >>>> completions, but the user hasn't polled his CQ yet, when should he be notified?
> >>>> On the 101 completion or immediately (since there are completions waiting on the
> >>>> CQ)?
> >>>
> >>> 101 completion
> >>>
> >>> It is only meaningful to call it when the CQ is empty.
> >>
> >> Thanks, so there's an inherent race between the user's CQ poll and the next arm?
> > 
> > I think the specs or man pages talk about this, the application has to
> > observe empty, do arm, then poll again then sleep on the cq if empty.
> > 
> >> Do you know what's the purpose of the consumer index in the arm doorbell that's
> >> implemented by many providers?
> > 
> > The consumer index is needed by HW to prevent CQ overflow, presumably
> > the drivers push to reduce the cases where the HW has to read it from
> > PCI
> 
> Thanks, that makes sense.
> 
> I found the following sentence in CX PRM:
> "If new CQEs are posted to the CQ after the reporting of a completion event and
> these CQEs are not yet consumed, then an event will be generated immediately
> after the request for notification is executed."
> 
> Doesn't that contradict the expected behavior?

I read it as confirming it?

Only *new* CQEs trigger an event, and new CQE's always trigger an
event regardless of the full/empty state of the queue.

This paragraph is an obtuse way of warning of the race I described.

Jason