Re: asynchronous operation with poll()

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Wed, Nov 10, 2010 at 09:43:12AM -0800, Roland Dreier wrote:
>  > Could I get some clarification on where there is no ordering
>  > guarantee? The WC's do not necessarily come back in the order that the
>  > sends were posted?
> 
> For a given queue, completions are always returned in the order that
> work requests were posted.  However there is no ordering between
> different queues.

Right, so if you design a scheme where getting a recv grants a send
credit then you are doing

  Post Send #1    ---> Recv WC
  Recv WC        <---- Post Send #2
  Send WC #2           Send WC #2
  PostSend #3     ---> Recv WC

and there is no order guarentee for when you get a Recv WC vs a Send
WC, even though by protocol design they are in fact ordered. If you
load the HCA the Send Complete WC's do get behind Recv WCs.

This is true even if you use the same CQ for Recv WC and Send Complete
WCs.

In practice this can make the implementation quite troublesome since
the posting of a new Send can't necessarily be done in the Recv WC
path as no send buffers could be available at that time.

I've found properly placing all the activities in a RDMA system to be
the hardest challenge of the whole design. Avoiding deadlock and
starvation and untangling the various data dependencies can be
hard :(

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html