I/O write ordering

[Matthew Wilcox <matthew@wil.cx>]

On Tue, Sep 21, 2004 at 07:58:05PM -0700, Jeremy Higdon wrote:
> I think I'm convinced that we want separate solutions for
> write ordering and (I'll call it) write flushing.

Yes, I think we have consensus on that.

> I'd define write posting as: a store instruction completing
> before the write is completed.  Write ordering is a problem
> if posted writes from two CPUs can complete to the device
> out of order.  Write flushing would be the process of
> assuring that a posted write has indeed been completed.
> A write flush takes care of write ordering.

Clear, succinct, understandable and correct.

> I think the only way to perform a write flush is to issue
> a register read to the card in question.  This makes it
> very heavy weight when all you need is ordering, especially
> since ordering is apparently not a problem for many (most?)
> platforms that do write posting.

Yes, the PCI specs make it clear that bridges can hang on to the
posted write until a read request goes through the bridge, not
just to the bridge.

> So I think we need one solution for write ordering, and
> a separate heavier-weight solution for write flushing.  For
> platforms in which write posting does not create ordering problems
> the write ordering primitive would be a no-op.

Yes.

> The solution for write ordering on the two arches I know about
> is processor/CPU chipset specific (SGI MIPS is a "sync" instruction,
> Altix is a read of a SHub register).
> 
> Whether we call it io_mwb() or mmiob() is not important to me.
> Does io_mwb stand for I/O Memory-mapped Write Barrier?

I think the only disagreement is whether we have it implicit or explicit.
I see three potential solutions:

1. iomb() [we have mb(), rmb(), wmb() already.  Does mb() imply iomb()?
   Vice versa?]
2. io_spin_unlock() which is an io barrier as well as a memory barrier.
3. Make spin_unlock an io barrier.  Expensive.

I'd quite like to hear opinions from the PPC and Alpha people, they seem to
be the most out-of-order architectures.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain