ibv_req_notify_cq and multithreading

ibv_req_notify_cq and multithreading

[Flavio Baronti]

I'm trying to have N threads reading from the same completion channel, bounded to M completion queues. I would like to 
have N << M, and to ensure that only a single thread at time can call ibv_poll_cq() on a given queue, to process the 
events in the same order they were put in the queue.

I can't understand how to properly achieve this, since:
1- If I call ibv_req_notify_cq() before ibv_poll_cq(), I might end up with two threads polling the same queue.
2- If I call ibv_req_notify_cq() after ibv_poll_cq(), I could end up with events in the cq not being notified in the 
channel (I read this on the IBTA 11.4.2.2, and I *think* I actually experienced this under load).

I can use option 1 with an additional lock before ibv_req_notify_cq(), but I would like to know if there is a simpler 
way which I can't see.

Thanks
Flavio
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: ibv_req_notify_cq and multithreading

[Hefty, Sean]

> I'm trying to have N threads reading from the same completion channel, bounded
> to M completion queues. I would like to
> have N << M, and to ensure that only a single thread at time can call
> ibv_poll_cq() on a given queue, to process the
> events in the same order they were put in the queue.
> 
> I can't understand how to properly achieve this, since:
> 1- If I call ibv_req_notify_cq() before ibv_poll_cq(), I might end up with two
> threads polling the same queue.
> 2- If I call ibv_req_notify_cq() after ibv_poll_cq(), I could end up with
> events in the cq not being notified in the
> channel (I read this on the IBTA 11.4.2.2, and I *think* I actually
> experienced this under load).
> 
> I can use option 1 with an additional lock before ibv_req_notify_cq(), but I
> would like to know if there is a simpler
> way which I can't see.

I can't think of a simpler way.  You just don't have any idea which CQ will be returned from the completion channel.  Does your traffic pattern work to create N completion channels and distributed the CQs among them?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ibv_req_notify_cq and multithreading

[Flavio Baronti]

Il 1/12/2012 16:52 PM, Hefty, Sean ha scritto:
>> I'm trying to have N threads reading from the same completion channel, bounded
>> to M completion queues. I would like to
>> have N<<  M, and to ensure that only a single thread at time can call
>> ibv_poll_cq() on a given queue, to process the
>> events in the same order they were put in the queue.
>>
>> I can't understand how to properly achieve this, since:
>> 1- If I call ibv_req_notify_cq() before ibv_poll_cq(), I might end up with two
>> threads polling the same queue.
>> 2- If I call ibv_req_notify_cq() after ibv_poll_cq(), I could end up with
>> events in the cq not being notified in the
>> channel (I read this on the IBTA 11.4.2.2, and I *think* I actually
>> experienced this under load).
>>
>> I can use option 1 with an additional lock before ibv_req_notify_cq(), but I
>> would like to know if there is a simpler
>> way which I can't see.
>
> I can't think of a simpler way.  You just don't have any idea which CQ will be returned from the completion channel.  Does your traffic pattern work to create N completion channels and distributed the CQs among them?
> --

Each CQ is related to a separate connection; putting two on the same channel, and making a single thread read the 
channel, would force an arbitrary coupling between connections which I'm trying to avoid.
Point 2 is correct though? ibv_req_notify_cq() should be called before ibv_poll_cq()?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: ibv_req_notify_cq and multithreading

[Hefty, Sean]

> Point 2 is correct though? ibv_req_notify_cq() should be called before
> ibv_poll_cq()?

To be technical, I would state is as you need to poll after calling notify.  For example, the sequence poll - notify - poll is okay.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html