How to write portable MMIO code?

How to write portable MMIO code?

[José Fonseca]

Dear kernel developers,

I'm currently trying to get the Mach64 DRI driver to run on PowerPC. It's 
mostly working but there are some strange behaviors (DMA works, MMIO not 
really unless you make long waits when submiting, etc.). This is most 
likely related with the MMIO programming macros in the kernel module.

My question is: How to code MMIO to be portable across all platforms, 
i.e., taking in consideration the endian format and memory caches?

I've search thorougly the answer to this question but found 
incomplete/contraditory answers:

  - should one use readl/writel or dereference the address directly?
  - is the use of readl/writel macros suficient to account for endian 
correctness or it's also needed to use the cpu_to_le32/le32_to_cpu macros?
  - should one in general (i.e., assuming the worst case) do wmb() on 
writes, and mb() on reads?

Although I appreciate answers concerning future developments on this 
matter, I need an answer to the current stable kernel release.


Regards,

José Fonseca


PS: Please CC me as I'm not subscribed.

Re: How to write portable MMIO code?

[Kai Germaschewski]

On Tue, 30 Apr 2002, José Fonseca wrote:

> I'm currently trying to get the Mach64 DRI driver to run on PowerPC. It's 
> mostly working but there are some strange behaviors (DMA works, MMIO not 
> really unless you make long waits when submiting, etc.). This is most 
> likely related with the MMIO programming macros in the kernel module.
> 
> My question is: How to code MMIO to be portable across all platforms, 
> i.e., taking in consideration the endian format and memory caches?
> 
> I've search thorougly the answer to this question but found 
> incomplete/contraditory answers:
> 
>   - should one use readl/writel or dereference the address directly?

Use readl/writel, that's why they exist.

>   - is the use of readl/writel macros suficient to account for endian 
> correctness or it's also needed to use the cpu_to_le32/le32_to_cpu macros?

They'll give you little endian always (i.e. they byteswap on big endian 
archs), so normally (register accesses) you shouldn't need additional 
swapping.

>   - should one in general (i.e., assuming the worst case) do wmb() on 
> writes, and mb() on reads?

I don't think mb() will help you. You're probably experiencing PCI posting 
problems - when a writel() has executed, that doesn't necessarily mean 
that the transaction has actually happened it may (and will) be buffered 
for a potentially long time.

However, PCI won't reorder reads vs. writes, so you when you want to be 
sure that a write() actually reached the hardware, do a dummy read() 
afterwards, that'll flush the write buffer.

--Kai

Re: How to write portable MMIO code?

[José Fonseca]

Kai,

On 2002.04.30 17:19 Kai Germaschewski wrote:
> On Tue, 30 Apr 2002, José Fonseca wrote:
> >   - should one in general (i.e., assuming the worst case) do wmb() on
> > writes, and mb() on reads?
> 
> I don't think mb() will help you. You're probably experiencing PCI
> posting
> problems - when a writel() has executed, that doesn't necessarily mean
> that the transaction has actually happened it may (and will) be buffered
> for a potentially long time.
> 
> However, PCI won't reorder reads vs. writes, so you when you want to be
> sure that a write() actually reached the hardware, do a dummy read()
> afterwards, that'll flush the write buffer.

Unfortunately one of the problems occurs in a idle wait loop, when a 
register is being sucessively read.

And if so, how the wmb() example in "Linux Device Drivers" 
(http://www.xml.com/ldd/chapter/book/ch08.html#t1) can be explained? The 
"Bus-Independent Device Accesses" 
(http://www.kernelnewbies.org/documents/kdoc/deviceiobook/x44.html) also 
refers what you suggested, but it also mentions the use of memory 
barriers. So how and when should they be used?

Regards,

José Fonseca

Re: How to write portable MMIO code?

[Kai Germaschewski]

On Tue, 30 Apr 2002, José Fonseca wrote:

> Unfortunately one of the problems occurs in a idle wait loop, when a 
> register is being sucessively read.

Hmmh, suppose you have different problem (most likely elsewhere) then.

> And if so, how the wmb() example in "Linux Device Drivers" 
> (http://www.xml.com/ldd/chapter/book/ch08.html#t1) can be explained? The 
> "Bus-Independent Device Accesses" 
> (http://www.kernelnewbies.org/documents/kdoc/deviceiobook/x44.html) also 
> refers what you suggested, but it also mentions the use of memory 
> barriers. So how and when should they be used?

Well, my understanding is the following: (if I get something wrong, 
hopefully somebody who's reading this will correct me)

the barrier(), {,r,w}mb() stuff is for actually for normal memory 
accesses.

About barrier():

If you have

	*p1 = 1; *p2 = 2;

the compiler may decide to reorder this to (if it knows that p1 != p2)

	*p2 = 2; *p1 = 1;

A barrier() in between will inhibit this reordering.

For some archs, even the barrier() is not sufficient to serialize the
accesses to RAM. The compiler may generate something like

	mov [p1], 1
	mov [p2], 2

but on e.g. alpha (where the asm would look differently ;-), the processor
may decide to put the second instruction on the memory bus before the 
first one. Only an mb in between will guarantee the ordering on the
memory bus.

Now, for IO, basically the same holds, though I wouldn't want to guarantee 
that the macros designed for the memory bus would work on the PCI bus as 
expected.

However, I do *believe*, that the readl/writel functions implicitly do the
right thing and introduce barriers where needed. On x86 e.g., the macros
do a cast to (volatile *), which will ensure that these functions are
compiled without reordering. As x86 is strongly ordered, no additional
mb() or whatever is necessary (nor does it exist) to make sure that this
ordering will propagate to the PCI bus.

There's definitely people here who know that stuff better than I do, 
though.

--Kai

Re: How to write portable MMIO code?

[Jesse Barnes]

On Tue, Apr 30, 2002 at 02:41:56PM -0500, Kai Germaschewski wrote:
> Well, my understanding is the following: (if I get something wrong, 
> hopefully somebody who's reading this will correct me)
> 
> the barrier(), {,r,w}mb() stuff is for actually for normal memory 
> accesses.
> 
> About barrier():
> 
> If you have
> 
> 	*p1 = 1; *p2 = 2;
> 
> the compiler may decide to reorder this to (if it knows that p1 != p2)
> 
> 	*p2 = 2; *p1 = 1;
> 
> A barrier() in between will inhibit this reordering.
> 
> For some archs, even the barrier() is not sufficient to serialize the
> accesses to RAM. The compiler may generate something like
> 
> 	mov [p1], 1
> 	mov [p2], 2
> 
> but on e.g. alpha (where the asm would look differently ;-), the processor
> may decide to put the second instruction on the memory bus before the 
> first one. Only an mb in between will guarantee the ordering on the
> memory bus.

That's a good summary of the memory ordering issues one normally runs
into.

> Now, for IO, basically the same holds, though I wouldn't want to guarantee 
> that the macros designed for the memory bus would work on the PCI bus as 
> expected.

Right.  In fact, waiting on I/O busses can take a bit longer than
making sure the processor executes memory transactions in the order
you'd like.

> However, I do *believe*, that the readl/writel functions implicitly do the
> right thing and introduce barriers where needed. On x86 e.g., the macros
> do a cast to (volatile *), which will ensure that these functions are
> compiled without reordering. As x86 is strongly ordered, no additional
> mb() or whatever is necessary (nor does it exist) to make sure that this
> ordering will propagate to the PCI bus.

Right, readl/writel will order things wrt the compiler and
processor, but not necessarily the I/O bus.  On IA64, we've introduced
mmiob() to address this.  It acts just like mb(), but wrt I/O address
space.  The ia64 patch for 2.5 includes some documentation about it,
I'd love to see other arches implement something similar (even as a
simple nop) so that machines with weakly ordered I/O busses will work
as expected.

Jesse

Re: How to write portable MMIO code?

[arjan]

In article <Pine.LNX.4.44.0204301112520.32217-100000@chaos.physics.uiowa.edu> you wrote:
>>   - should one in general (i.e., assuming the worst case) do wmb() on 
>> writes, and mb() on reads?

readl() and writel() are specified to contain the proper barriers already.

 
> I don't think mb() will help you. You're probably experiencing PCI posting 
> problems - when a writel() has executed, that doesn't necessarily mean 
> that the transaction has actually happened it may (and will) be buffered 
> for a potentially long time.
> 
> However, PCI won't reorder reads vs. writes, so you when you want to be 
> sure that a write() actually reached the hardware, do a dummy read() 
> afterwards, that'll flush the write buffer.

yup