Re: SMP barriers semantics

[Nick Piggin <npiggin@suse.de>]

On Mon, Mar 22, 2010 at 11:02:03PM +1100, Nick Piggin wrote:
> On Wed, Mar 17, 2010 at 01:42:43PM +0000, Jamie Lokier wrote:
> > Benjamin Herrenschmidt wrote:
> > > Maybe we can agree on a set of relaxed accessors defined specifically
> > > with those semantics (more relaxed implies use of raw_*) :
> > > 
> > >   - order is guaranteed between MMIOs
> > >   - no order is guaranteed between MMIOs and spinlocks
> > 
> > No order between MMIOs and spinlocks will be fun :-)
> 
> There isn't anyway, and things are pretty messed up in this area
> already. We have mmiowb. Some architectures do a bit of mmio vs
> lock synchronization. Eg. powerpc. But it doesn't do barriers on
> some other locks like bit spinlocks or mutexes or rwsems or
> semaphores blah blah.
> 
> When this came up I grepped a couple of drivers and found possible
> problems straight away.

Ie. with drivers using a mutex to synchronise mmios, which breaks on
powerpc (even though spinlocks would work due to their hack in there).

And on some other architectures the driver would be broken whether it
uses spinlocks or not because it is not using the right barriers.

I think we have basically everything we need right now except to
have things clearly documented and consistent over architectures, and
to have mmio acquire/release to be paired with _any_ locking function
(bitops, atomics, spinlocks, mutexes, etc).

If we enforce *strong* ordering with cacheable accesses and other IO
for regular IO accessors, then a driver would not have to do anything
special for ios or locks or anything. This can be very expensive on
some archs, but IMO it is the best default because any driver that would
work on x86 using these will work on any other arch.

It is much easier for a powerpc or altix developer to take a working
but underperforming driver, identify some of the critical ios and relax
them than it is for them to try to fix a strangely broken driver (and
where the driver developer and 99% of driver testers doesn't have their
hardware).

For relaxed accessors, I think acquire/release (versus cacheable), and
then traditional mb, rmb, wmb should be pretty good start.

Anyone disagree? I would volunteer to post some patches if not...