Re: [PATCH rdma-next 0/3] Support out of order data placement

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Tue, Aug 01, 2017 at 10:06:14PM +0000, Parav Pandit wrote:

> > > >  Initial Condition VA=0 Data = 0
> > > >  RDMA-W VA=0 Data=1
> > > >  RDMA-R VA=0
> > > >
> > > > Spec says 1 must be returned, but sounds like this relaxed version could
> > return 0.
> > 
> > > No. Table 76 stays as is as described before.
> > 
> > How is this possible?

> I am not sure what more can I explain you Jason.
> Requester side HCA follows HCA Table-76.
> Incoming read responses are not processed until previous writes are
> ACKed implicitly (in read responses) or explicitly by ACK packets.
> Same as before described in spec. No extra description needed for
> this patchset.

But doing that pretty much destroys much of the entire point of having
a relaxed ordering :P

> > > >  RDMA-W VA=0 Data=1
> > > >  RDMA-W VA=0 Data=2
> > > >  SEND
> > > >
> > > > Sounds like with the relaxed version the app could see 1 at SEND CQ time.
> > > >
> > > > So RDMA-W -> RDMA-W degrades to a F
> > 
> > > No. Table-76 is based on  how requester sees the execution.
> > > So it stays as '#'.
> > 
> > How is this possible?
> > 
> Please don't mix requester side ordering with responder side execution.
> C9-28 on responder side is relaxed - as explained few times before.

No, I see what you are tring to say now. I disagree with
this. Table-76 and C9-28 are describing the same thing, you cannot
weaken C9-28 without also restating Table 76.

Table 76 is clearly talking about the entire system, including the
execution and memory subsystem of the completer.

> It covers only requester side.
> Send with invalidate execution on responder side is described in 9.4.1.1.1

I suppose 9.4.1.1.1 point #1 already allows the out of order behavior.

> You are proposing a different behavior and attribute which may be
> done for a HCA that support such thing.  Please submit a different
> patch for it whenever its appropriate.  Current query HCA attribute
> is bit field for future relaxation. May be what you described can be
> done.

Since you guys are hell bent on avoiding the IBTA for your new verbs
features, you do have to define them in a sensible and usefully broad
way using the community process.

> Other out of order atomics such as
> Atomic->Read
> Write->Atomic may be done in future under different attribute.

I think that is a mistake, you should start with them being out of
order and require the app to fence to bring order back, even if the
current HCAs execute them in order anyhow.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html