Re: [rfc] add io barriers, remove mmiowb

[Nick Piggin <npiggin@suse.de>]

On Thu, May 22, 2008 at 04:34:51AM -0400, Jes Sorensen wrote:
> >>>>> "Nick" == Nick Piggin <npiggin@suse.de> writes:
> 
> Nick> mb and wmb are now no longer guaranteed to order system memory
> Nick> operations with device memory stores. mmiowb has been introduced
> Nick> to provide this ordering (when combined with a mb, wmb, or
> Nick> spin_unlock). Unfortunately, it appears to be rather less well
> Nick> understood among both users and implementors than even the old
> Nick> memory barrier scheme. It also subtly breaks existing code that
> Nick> uses mb or wmb (if only on sn2). I really think it is not a good
> Nick> solution.
> 
> Nick> The alternative I propose is to restore mb and wmb to their full
> Nick> strength.  This does mean that sn2 has to do the equivalent of
> Nick> mb+mmiowb, wmb+mmiowb respectively, but that's the price you pay
> Nick> for weak memory ordering.
> 
> Nick,
> 
> Introducing this constraint would make me less than pleased I have to
> admit. It's a very expensive operation to do since it requires going
> out talking to the PCI bridge, doing that on every wmb() is going to
> really hurt :-(

Right, but probably the large majority of wmb() callers actually
just want io_wmb(). This would relieve much of the performance
problem, I'd say.

Of those that really want a wmb() and cannot be converted to
io_wmb(), I don't think it is a good option to actually just weaken
wmb() because we deem that doing what the caller asked for is too
expensive.

I guess with the ia64_mf(), Altix probably does not reorder PCI
stores past earlier cacheable stores, so _some_ wmb()s actually
do not require the full mmiowb case (if we only need to order
an earlier RAM store with a later PCI store). However, again,
weakening wmb() is not a good option because it really requires
an audit of the entire tree to do that.

We _could_ introduce partial barriers like store/iostore iostore/store,
but really, I think the io_wmb is a pretty good first step, and I
haven't actually seen any numbers indicating it would be a performance
problem.