Re: [PATCH 2/9] IB: add a proper completion queue abstraction

Bart Van Assche
2015-11-13T19:58:07Z

On 11/13/2015 10:25 AM, Jason Gunthorpe wrote:
> On Fri, Nov 13, 2015 at 02:46:43PM +0100, Christoph Hellwig wrote:
>> This adds an abstraction that allows ULP to simply pass a completion
>> object and completion callback with each submitted WR and let the RDMA
>> core handle the nitty gritty details of how to handle completion
>> interrupts and poll the CQ.
>
> This looks pretty nice, I'd really like to look it over carefully
> after SC|15..
>
> I know Bart and others have attempted to have switching between event
> and polling driven operation, but there were problems resolving the
> races. Would be nice to review that conversation.. Do you remember the
> details Bart?

Hello Jason,

I think this is the conversation you are referring to: "About a
shortcoming of the verbs API"
(http://thread.gmane.org/gmane.linux.drivers.rdma/5028). That
conversation occurred five years ago, which means that you have an
excellent memory :-)

I doesn't seem to me like Christoph wanted to support dynamic switching
between the IB_POLL_DIRECT, IB_POLL_SOFTIRQ and IB_POLL_WORKQUEUE
polling modes. I think this should have been mentioned in the patch
description.

The implementation of this patch makes it clear that it is essential
that all polling is serialized. The WC array that is used for polling is
embedded in the CQ and is not protected against concurrent access. This
means that it is essential that _ib_process_cq() calls are serialized. I
need some more time to verify whether such serialization is always
guaranteed by this patch.

Bart.