Re: [PATCH] Document Linux's memory barriers

Linus Torvalds
2006-03-08T03:31:04Z

On Wed, 8 Mar 2006, Paul Mackerras wrote:
>
> Linus explained recently that wmb() on x86 does not order stores to
> system memory w.r.t. stores to stores to prefetchable I/O memory (at
> least that's what I think he said ;).

In fact, it won't order stores to normal memory even wrt any
_non-prefetchable_ IO memory.

PCI (and any other sane IO fabric, for that matter) will do IO posting, so
the fact that the CPU _core_ may order them due to a wmb() doesn't
actually mean anything.

The only way to _really_ synchronize with a store to an IO device is
literally to read from that device (*). No amount of memory barriers will
do it.

So you can really only order stores to regular memory wrt each other, and
stores to IO memory wrt each other. For the former, "smp_wmb()" does it.

For IO memory, normal IO memory is _always_ supposed to be in program
order (at least for PCI. It's part of how the bus is supposed to work),
unless the IO range allows prefetching (and you've set some MTRR). And if
you do, that, currently you're kind of screwed. mmiowb() should do it, but
nobody really uses it, and I think it's broken on x86 (it's a no-op, it
really should be an "sfence").

A full "mb()" is probably most likely to work in practice. And yes, we
should clean this up.

Linus

(*) The "read" can of course be any event that tells you that the store
has happened - it doesn't necessarily have to be an actual "read[bwl]()"
operation. Eg the store might start a command, and when you get the
completion interrupt, you obviously know that the store is done, just from
a causal reason.