Re: [PATCH 2/4] arch: Add lightweight memory barriers fast_rmb() and fast_wmb()

[Benjamin Herrenschmidt <benh@kernel.crashing.org>]

On Mon, 2014-11-17 at 12:18 -0800, Paul E. McKenney wrote:
> On Mon, Nov 17, 2014 at 09:18:13AM -0800, Alexander Duyck wrote:
> > There are a number of situations where the mandatory barriers rmb() and
> > wmb() are used to order memory/memory operations in the device drivers
> > and those barriers are much heavier than they actually need to be.  For
> > example in the case of PowerPC wmb() calls the heavy-weight sync
> > instruction when for memory/memory operations all that is really needed is
> > an lsync or eieio instruction.
> 
> Is this still the case if one of the memory operations is MMIO?  Last
> I knew, it was not.

I *think* (Alexander, correct me if I'm wrong), that what he wants is
the memory<->memory barriers (the smp_* ones) basically for ordering his
loads or stores from/to the DMA area.

The problem is that the smp_* ones aren't compiled for !CONFIG_SMP

IE. Something like:

  - Read valid bit from descriptor

  - Read rest of descriptor

That needs an rmb of some sort in between, but a full blown "rmb" will
also order vs. MMIOs and end up being a full sync, while an smp_rmb is a
lwsync which is more lightweight.

Similarily:

 - Populate descriptor

 - Write valid bit

Same deal with wmb ...

Basically, rmb and wmb order both cachable and non-cachable (memory and
MMIO) which makes them needlessly heavy on powerpc and possibly others
when all you need is to order memory accesses to some DMA data
structures. In that case you really want the normal smp_* variants
except they may not be around...

Cheers,
Ben.