Re: About a shortcoming of the verbs API

Bart Van Assche
2010-07-27T08:33:42Z

On Mon, Jul 26, 2010 at 9:22 PM, Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
> [ ... ]
>
> Another approach is to just always run the completion processing for a
> given CQ on a single CPU and avoid locking entirely.  If you want more
> CPUs to spread the work, just use multiple CQs and multiple event vectors.

In the applications I'm familiar with InfiniBand is being used not
only because of its low latency but also because of its high
throughput. In order to handle such loads efficiently, interrupts have
to be spread over multiple CPUs.

Switching from a single receive queue to multiple receive queues is an
interesting alternative, but is not possible without changing the
communication protocol between client and server. Changing the
communication protocol is not always possible, especially when the
communication protocol has been defined by a standards organization.

>  > see e.g. VipCQNotify() in the Virtual Interface Architecture
>  > Specification.
>
> I don't know of an efficient way to implement this type of "atomic
> dequeue completion or enable completions" with any existing hardware.
> Do you have an idea how this could be done?

I am not an expert with regard to HCA programming. But I assume the
above should refer to "reprogrammable firmware" instead of "hardware"
?

Bart.