Re: [PATCH rdma-core 12/14] qedr: Update to use new udma write barriers

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Thu, Feb 23, 2017 at 01:49:47PM +0000, Amrani, Ram wrote:
> Thanks, Jason, for this work.
> 
> Per this change, I don't see any need in using mmio_wc_start() in
> qedr.

The required pattern when dealing with WC memory is this:

 mmio_wc_start();
 writel_to_wc(..);
 mmio_flush_writes();

On x86-64 this translates to

 sfence
 mov ..
 sfence

As far as I know, on x86-64 both sfences are required to ensure that
all stores are visible to the device in strictly program order. x86-64
does not strongly order accesses to WC relative to other memory types
without SFENCE.

mmio_wc_start() is defined to be a superset of the functionality of
udma_to_device_barrier() - and it is given a unique name to make it
very clear where to use it: directly before writing to WC memory.

> wmb() is invoked to make sure that the CPU doesn't issue the
> doorbell before the data was flushed (that's why I expected its
> implementation to be as the kernel's i.e. sfence. See e-mail

Exactly. mmio_wc_start() is defined as the barrier to do that.  It
specifically guarentees that all writes to C and UC memory are ordered
before upcoming writes to WC memory.

It is SFENCE on x86-64, which as you say, is what you expected in the
first place, so this fixes a bug :\..

FWIW the

#define mmio_wc_start() mmio_flush_writes()

Is just a stand in that happens to get a safe instruction on every
arch.

If required each of the macros could have a unique implementation if
the arch requires it, however in most cases I would expect that
udma_to_device_barrier(), mmio_flush_writes() and mmio_wc_start() are
the same instruction.

The use of different names is a purely descriptive to aid in
understanding what the barriers do and when to use them.

Does that address your concern?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html