Re: About a shortcoming of the verbs API

Roland Dreier
2010-07-26T19:22:35Z

> 2. Double completion processing loop
>
> * Initialization:
> ib_req_notify_cq(cq, IB_CQ_NEXT_COMP);
>
> * Notification handler:
>
> struct ib_wc wc;
> do {
>     while (ib_poll_cq(cq, 1, &wc) > 0)
>         /* process wc */
> } while (ib_req_notify_cq(cq, IB_CQ_NEXT_COMP |
> IB_CQ_REPORT_MISSED_EVENTS) > 0);

This approach can be used to have race-free in-order processing of
completions using a scheme such as the NAPI processing loop used by the
IPoIB driver (with help from the core networking stack). Essentially a
completion notification just marks the completion processing routine as
runnable, and the networking core schedules that processing routine in a
single-threaded way until the CQ is drained.

Another approach is to just always run the completion processing for a
given CQ on a single CPU and avoid locking entirely. If you want more
CPUs to spread the work, just use multiple CQs and multiple event vectors.

> see e.g. VipCQNotify() in the Virtual Interface Architecture
> Specification.

I don't know of an efficient way to implement this type of "atomic
dequeue completion or enable completions" with any existing hardware.
Do you have an idea how this could be done?

- R.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to: