Re: [PATCH rdma-next 0/3] Support out of order data placement

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Tue, Aug 01, 2017 at 06:14:08PM +0000, Parav Pandit wrote:

> >  Initial Condition VA=0 Data = 0
> >  RDMA-W VA=0 Data=1
> >  RDMA-R VA=0
> > 
> > Spec says 1 must be returned, but sounds like this relaxed version could return 0.

> No. Table 76 stays as is as described before.

How is this possible?

> >  RDMA-W VA=0 Data=1
> >  RDMA-W VA=0 Data=2
> >  SEND
> > 
> > Sounds like with the relaxed version the app could see 1 at SEND CQ time.
> > 
> > So RDMA-W -> RDMA-W degrades to a F

> No. Table-76 is based on  how requester sees the execution.
> So it stays as '#'.

How is this possible?

You've clearly stated this feature allows out of order execution
across packet boundaries, there is no way to know at the responder
what the missed packets where, so ineventiably, both of these cases
must be possible. Or you are wrong about the statement on out of
order.

Frankly, I still don't think you know what this feature actually does
and consequently cannot document it properly, which is not
acceptable for a common verbs feature.

> > However, SEND WITH INVALIDATE is a special cases that impacts the processing
> > of work itself, not just the CPU observation, which is a bit outside what table 76
> > is talking about.
 
> SEND, SEND WITH IMM, SEND WITH INVALIDATE falls in same category as send as first column in Table76.

Not really, I don't think you understand how this all fits together..

> > I'd advocate for allowing this to be out of order (but documented as such), as
> > impliclty fencing SEND WITH INVALIDATE is not acceptable for
> > performance and

> It is as per first column of Table-76.

Dn't understand you remark, it is clearly ordered..

> > most workloads using that feature do not care about this strict ordering.
> > 
> nvme fabrics do care.
> nvme fabrics target does RDMA-W, RDMA_S_INV sequence on the same memory key that is being used in RDMA-W without waiting for RDMA-W completion for good reason.
> I recall SMB doing the same as well.
> RDMA-S_INV after RDMA-W cannot break the order.

They don't care, because RDMA_WRITE, SEND_WITH_INVALIDATE on the same
rkey does not try to write to the rkey memory twice, which is the only
case where adding out of order execution really matters.

They only care that the invalidate guarentees no DMA is possible once
it reaches the receiver's CQ.

> > The requirement is really that by the time the SEND RCQ is seen that the
> > INVALIDATE has taken effect.
> > 
> Current Table-76 requirement already relaxes for 
> RDMA-R-> RDMA_S_INV.
> However most users won't do above sequence because users would not like to fail duplicate read requests.
> So let's continue with Table-76 for SEND as 2nd operation as defined today. (first column stays as is)

As I said, table 76 does not really capture the full behavior of
INVALIDATE.

The spec requires WRITE,INV,WRITE to fail, but it would be just a fine
for storage protocols if WRITE,INV,WRITE could succeed, so long as
delivering the INV to the CQ fences the DMA, which can be done in a
high performance way. Fencing the WRITE,INV,WRITE can not be done
with high performance.

> > Atomic are basically similar, sounds like Atomic Op -> RDMA Read should
> > degrade to a F as well. I'd say that is desired as well.
> 
> No. Table-76 stays as is.
> Atomic->Atomic is already 'F'.
> Atomic->RDMA_R is continues as '#'. (Similar to RDMA_W->RDMA_R).

Same argument as above, many apps will tolerate out of order for
atomics, thee default for an out of order mode should be to allow it,
and let apps request in order with fence.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html