Re: How to write portable MMIO code?

[Jesse Barnes <jbarnes@sgi.com>]

On Tue, Apr 30, 2002 at 02:41:56PM -0500, Kai Germaschewski wrote:
> Well, my understanding is the following: (if I get something wrong, 
> hopefully somebody who's reading this will correct me)
> 
> the barrier(), {,r,w}mb() stuff is for actually for normal memory 
> accesses.
> 
> About barrier():
> 
> If you have
> 
> 	*p1 = 1; *p2 = 2;
> 
> the compiler may decide to reorder this to (if it knows that p1 != p2)
> 
> 	*p2 = 2; *p1 = 1;
> 
> A barrier() in between will inhibit this reordering.
> 
> For some archs, even the barrier() is not sufficient to serialize the
> accesses to RAM. The compiler may generate something like
> 
> 	mov [p1], 1
> 	mov [p2], 2
> 
> but on e.g. alpha (where the asm would look differently ;-), the processor
> may decide to put the second instruction on the memory bus before the 
> first one. Only an mb in between will guarantee the ordering on the
> memory bus.

That's a good summary of the memory ordering issues one normally runs
into.

> Now, for IO, basically the same holds, though I wouldn't want to guarantee 
> that the macros designed for the memory bus would work on the PCI bus as 
> expected.

Right.  In fact, waiting on I/O busses can take a bit longer than
making sure the processor executes memory transactions in the order
you'd like.

> However, I do *believe*, that the readl/writel functions implicitly do the
> right thing and introduce barriers where needed. On x86 e.g., the macros
> do a cast to (volatile *), which will ensure that these functions are
> compiled without reordering. As x86 is strongly ordered, no additional
> mb() or whatever is necessary (nor does it exist) to make sure that this
> ordering will propagate to the PCI bus.

Right, readl/writel will order things wrt the compiler and
processor, but not necessarily the I/O bus.  On IA64, we've introduced
mmiob() to address this.  It acts just like mb(), but wrt I/O address
space.  The ia64 patch for 2.5 includes some documentation about it,
I'd love to see other arches implement something similar (even as a
simple nop) so that machines with weakly ordered I/O busses will work
as expected.

Jesse