Re: [RFC PATCH rdma-next 08/10] RDMA/rxe: Implement flush execution in responder side

[Jason Gunthorpe <jgg@ziepe.ca>]

On Thu, Dec 30, 2021 at 09:32:06PM -0500, Tom Talpey wrote:

> The global visibility is oriented toward supporting distributed
> shared memory workloads, or publish/subscribe on high scale systems.
> It's mainly about ensuring that all devices and CPUs see the data,
> something ordinary RDMA Write does not guarantee. This often means
> flushing write pipelines, possibly followed by invalidating caches.

Isn't that what that new ATOMIC_WRITE does? Why do I need to flush if
ATOMIC WRITE was specified as a release? All I need to do is acquire
on the ATOMIC_WRITE site and I'm good?

So what does FLUSH do here, and how does a CPU 'acquire' against this
kind of flush? The flush doesn't imply any ordering right, so how is
it useful on its own?

The write to persistance aspect I understand, but this notion of
global viability seems peculiar.

> Well, higher level wrappers may signal errors, for example if they're
> not available or unreliable, and you will need to handle them. Also,
> they may block. Is that ok in this context?

I'm not sure we have any other tools here beyond a release barrier
like wmb() or the special pmem cache flush. Neither are blocking or
can fail.

In pmem systems storage failure is handled via the memory failure
stuff asynchronously.

Jason