Re: [PATCH rdma-next 0/3] Support out of order data placement

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Mon, Jun 12, 2017 at 03:57:29PM -0500, Steve Wise wrote:
> > > > When transmitter and receiver is enabled to do so, as I described in
> > > overview section of Documentation, it helps
> > > > (a) to avoid retransmission - improves network utilization
> > > > (b) reduces latency due to timers not kicking in.
> > >
> > > Yes those benefits are clear. I see no reason why it shouldn't always
> > > be
> > > done is my point. Application shouldn't have to care and there is no
> > > need to make this an additional flag.
> > 
> > The app cares when data from write 2 can be written at the target before data
> > from write 1, especially if the writes target the same memory buffers.  (At least I
> > think this is the intent of exposing this to the app.)
> > 
> > Note that the provider can always provide stronger ordering than what the app
> > needs.
> 
> My understanding is that IB or IW apps should never assume ingress
> write or read response data is _placed_ into local memory in the
> order it was transmitted from the peer.  The only guarantee is that
> the _indication_ of the arrived data preserve the sender's ordering.
> However, I'm thinking that there are applications out there that
> spin polling local memory that is the target of a write or read
> response and assume the last bit of that memory will get written
> last...

That is with respect to the CPU, but IB requires strong ordering
between messages within the same QP, eg if I do

RDMA WRITE addr=0 data=1
RDMA WRITE addr=0 data=2
RDMA WRITE addr=0 data=3
RDMA READ  addr=0

I must always get 3, not something else.

It would be notable if this 'out of order' feature violated that
invariant, but many ULPs would probably still be OK.

Frankly, Parav's original message doesn't seem to describe at all what
this is about, so maybe we should all wait until v2, and maybe more
people from Mellanox could contribute to sensibly describing it if
they want it in ibverbs.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html