Re: [PATCH rdma-next 0/3] Support out of order data placement

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Sat, Jul 22, 2017 at 05:32:05PM +0000, Parav Pandit wrote:
> > 
> > Eg
> > 
> >  RDMA-W VA=0 Data=1
> >  RDMA-R VA=0
> >  RDMA-W VA=0 Data=2
> > 
> > What does the read return? Spec says 1, but it sounds like this relaxed ordering
> > could return 2.
> > 
> Spec says Data=1 on RDMA-R only if Fence is set on read operation in Table-76.
> Otherwise duplicate read request after executing RDMA-W2 of Data=2 can return Data=2 on read request.

Erm, I should have written it like this

 Initial Condition VA=0 Data = 0
 RDMA-W VA=0 Data=1
 RDMA-R VA=0

Spec says 1 must be returned, but sounds like this relaxed version
could return 0. So RDMA Write -> RDMA Read degrades to a F

Similarly,

 RDMA-W VA=0 Data=1
 RDMA-W VA=0 Data=2
 SEND

Sounds like with the relaxed version the app could see 1 at SEND CQ
time.

So RDMA-W -> RDMA-W degrades to a F

> > Whta about
> > 
> >  RDMA-W VA=0 Data=1
> >  SEND WITH INVALIDATE VA=0
> >  RDMA-W VA=0 Data=2
> > 
> > Spec says the second RDMA-W must fail, 
> Right.
> 
> > but it sounds like this relaxed ordering would allow it to happen.

> No. Table 76 is followed in this case.
> 1st operation Write.
> 2nd operation send.
> There is implicit fence defined by '#' in Table, which is followed.
> So 2nd RDMA-W continue to fail.

So, I expect what is happening here is that the SEND RCQ is delayed
until the sequence numbers catch up, eg guarenteeing that all
packets prior to the SEND have been seen and committed to memory.
Which is what table 76 is primarily talking about.

However, SEND WITH INVALIDATE is a special cases that impacts the
processing of work itself, not just the CPU observation, which is a
bit outside what table 76 is talking about.

I'd advocate for allowing this to be out of order (but documented as
such), as impliclty fencing SEND WITH INVALIDATE is not acceptable for
performance and most workloads using that feature do not care about
this strict ordering.

The requirement is really that by the time the SEND RCQ is seen that
the INVALIDATE has taken effect.

Atomic are basically similar, sounds like Atomic Op -> RDMA Read should
degrade to a F as well. I'd say that is desired as well.

The point is we want a definition for this feature that is broad
enough to allow future hardware optimization, and not just some some
narrow defintion that follows what mlx5 happened to implement.

> I completely understand and agree that storage protocols who depend
> on RDMA-CM would like to have this.  But again, unavailability of
> this bit in RDMA-CM is not a blocker for user land apps and let them
> start using it.  Once RDMA-CM has it, storage kernel ULPs can also
> be enabled.

If there is no path to get this into the RDMA CM then it is just
another vendor feature and it does not belong in the common API.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html