Memory vs. MMIO ordering, again

[Robert Hancock <hancockr@shaw.ca>]

A question came up recently about memory ordering in libata. Essentially 
we have the classic case (ata_bmdma_setup in drivers/ata/libata-sff.c) 
where we do some writes to memory (specifically the PRD table) and want 
to make sure the device will see those writes before we poke it to go 
and use that memory. Right now this is using an mb() (which seems like 
overkill, even in the worst case it seems like only a wmb() should be 
needed).

I just finished digging through the 2006 LKML discussion "Opinion on 
ordering of writel vs. stores to RAM" which ironically failed to clearly 
answer the question asked in its subject.

Documentation/memory-barriers.txt does not answer this question either. 
The only thing it contains that seems to pertain to this question is the 
following:

"Memory mapped I/O usually takes place through memory locations that are 
part of a window in the CPU's memory space that has different properties 
assigned than the usual RAM directed window.

Amongst these properties is usually the fact that such accesses bypass 
the caching entirely and go directly to the device buses.  This means 
MMIO accesses may, in effect, overtake accesses to cached memory that 
were emitted earlier. A memory barrier isn't sufficient in such a case, 
but rather the cache must be flushed between the cached memory write and 
the MMIO access if the two are in any way dependent."

This seems like BS to me.. Flush the cache? How is a driver supposed to 
know how to do that? Furthermore, why should it need to worry about this 
  kind of detail? This seems wrong or at least a low-level detail that 
normal code should not have to be concerned with.

 From what I can vaguely infer from that 2006 discussion it seems like 
powerpc was going to be fixed so that writel, etc. would provide the 
expected ordering with respect to memory writes, however I don't know if 
this is actually the case. The documentation that driver writers would 
rely on should be updated to be explicit on this question...