Linux mips and DMA

Linux mips and DMA

[Jon Dufresne]

Hello,

I am in the process of porting a known working linux driver for a pci
device from an x86 machine to a mips machine. This is my first time
developing a driver under mips (but not the first time with x86) so I am
learning some of the differences and gotchas that exist when porting a
driver like this.

My most recent problem exists when setting up dma between the host and
the device. I am using the following two websites as guides for doing
this:

https://lwn.net/Articles/28230/
https://lwn.net/Articles/28092/

In addition I am using LDD.

To create the dma memory area I am using the function
"pci_alloc_consistent". When I pass the "dma_handle" (as I understand it
the host's physical address of the dma memory), to the pci device, the
device in the x86 box correctly access this memory, not so in the mips
box.

Not sure if this is helpful, but the fuction returns the following
addresses on the mips when I use it:

dma_handle=0x026f0000 size=0x00010000 cpu_addr=0xa26f0000

Does this physical address seem abnormally low? It is well outside the
range of the PCI BARs which exist around 0x20000000.

Anything I should know about using pci_alloc_consistent on a mips?

If you need more information to understand the situation I'd be happy to
supply it but right now I'm not sure what to supply.

Thanks,
Jon

Re: Linux mips and DMA

[Ralf Baechle]

On Wed, Jan 09, 2008 at 01:57:17PM -0500, Jon Dufresne wrote:

> I am in the process of porting a known working linux driver for a pci
> device from an x86 machine to a mips machine. This is my first time
> developing a driver under mips (but not the first time with x86) so I am
> learning some of the differences and gotchas that exist when porting a
> driver like this.
> 
> My most recent problem exists when setting up dma between the host and
> the device. I am using the following two websites as guides for doing
> this:
> 
> https://lwn.net/Articles/28230/
> https://lwn.net/Articles/28092/

The articles are fairly old and the APIs and their documentation are
occasionally changing.  I suggest you use Documentation/DMA-mapping.txt
as the primary documentation - though as usual Jonathan Corbet's article
is very well written.

> In addition I am using LDD.
> 
> To create the dma memory area I am using the function
> "pci_alloc_consistent". When I pass the "dma_handle" (as I understand it
> the host's physical address of the dma memory), to the pci device, the
> device in the x86 box correctly access this memory, not so in the mips
> box.
> 
> Not sure if this is helpful, but the fuction returns the following
> addresses on the mips when I use it:
> 
> dma_handle=0x026f0000 size=0x00010000 cpu_addr=0xa26f0000
> 
> Does this physical address seem abnormally low? It is well outside the
> range of the PCI BARs which exist around 0x20000000.

There is no relation to the PCI bars whatsoever.  pci_alloc_consistent
works on a memory that is RAM range.  0xa26f0000 is an uncached address
in KSEG1 and 0x026f0000 the equivalent physical address.

All these numbers look sane which suggest your problem may be elsewhere.

One potencial problem might be that bus addresses as used on the PCI bus
and physical addresses as use by the CPU are not identical.  The
return value of pci_alloc_consistent shows that your kernel is setup
to assume that both addresses are identical.  Which is true on most
x86 and MIPS platforms but not all.

Note you may not touch a cached mapping of this range for example
0x826f0000 in KSEG0 or any other cached mapping you may have created or
data corruption will happen.

> Anything I should know about using pci_alloc_consistent on a mips?

MIPS hardware is different so pci_alloc_consistent is implemented
differently.  For correct use however this should not matter.  Any bugs
you may find porting to MIPS were already bugs on x86.

(Or in pci_alloc_consistent but I'm optimistic ;-)

  Ralf

Re: Linux mips and DMA

[Jon Dufresne]

> MIPS hardware is different so pci_alloc_consistent is implemented
> differently.  For correct use however this should not matter.  Any bugs
> you may find porting to MIPS were already bugs on x86.
> 
> (Or in pci_alloc_consistent but I'm optimistic ;-)


Is there a chance that my platform does not support coherent DMA
mappings? Or is this unheard of for a MIPs platform?

Thanks for the help!
Jon

Re: Linux mips and DMA

[Ralf Baechle]

On Thu, Jan 10, 2008 at 08:30:16AM -0500, Jon Dufresne wrote:

> > MIPS hardware is different so pci_alloc_consistent is implemented
> > differently.  For correct use however this should not matter.  Any bugs
> > you may find porting to MIPS were already bugs on x86.
> > 
> > (Or in pci_alloc_consistent but I'm optimistic ;-)
> 
> 
> Is there a chance that my platform does not support coherent DMA
> mappings? Or is this unheard of for a MIPs platform?

Hardware coherency for DMA is the exception for low-end embedded MIPS
systems andgiven the CPU address your's is no exception from that.

If your system was supporting hardware coherency for DMA I/O you would
have obtained a cachable CPU address like:

  dma_handle=0x026f0000 size=0x00010000 cpu_addr=0x826f0000
                                                  ^^^

A 0x8??????? would be in KSEG0 so cachable.

What hardware are you using anyway?

  Ralf

Re: Linux mips and DMA

[Jon Dufresne]

> Hardware coherency for DMA is the exception for low-end embedded MIPS
> systems andgiven the CPU address your's is no exception from that.
> 
> If your system was supporting hardware coherency for DMA I/O you would
> have obtained a cachable CPU address like:
> 
>   dma_handle=0x026f0000 size=0x00010000 cpu_addr=0x826f0000
>                                                   ^^^
> 
> A 0x8??????? would be in KSEG0 so cachable.

I do have a an embedded system. Are you saying that, in all likelyhood,
I do not have coherency? If I understand you correctly, this is a bad
thing right? Will I need to take extra care to work around this issue.

So are you saying I would prefer a cpu_addr in the 0x8******* range?

If it is true that I don't have hardware coherency should I still be
using the pci_*_consistent api? Or should I switch to the
dma_*_noncoherent api? Also what extra steps do I need to take to get
this to work with a non-coherent system?

I am reading Documentation/DMA-API.txt which has some discussion about
non-coherent systems to see if I can get a handle on this.


> What hardware are you using anyway?

I am using the MDS-810 platform supplied by Momentum Data Systems which
contains a Phillips/Nexperia pnx8950 chip. That chip contains a MIPS32
core. http://www.mds.com/products/product.asp?prod=MDS-810

Thanks again,
Jon

Re: Linux mips and DMA

[Ralf Baechle]

On Thu, Jan 10, 2008 at 09:12:23AM -0500, Jon Dufresne wrote:

> > Hardware coherency for DMA is the exception for low-end embedded MIPS
> > systems andgiven the CPU address your's is no exception from that.
> > 
> > If your system was supporting hardware coherency for DMA I/O you would
> > have obtained a cachable CPU address like:
> > 
> >   dma_handle=0x026f0000 size=0x00010000 cpu_addr=0x826f0000
> >                                                   ^^^
> > 
> > A 0x8??????? would be in KSEG0 so cachable.
> 
> I do have a an embedded system. Are you saying that, in all likelyhood,
> I do not have coherency? If I understand you correctly, this is a bad
> thing right? Will I need to take extra care to work around this issue.
> 
> So are you saying I would prefer a cpu_addr in the 0x8******* range?

No. because you don't seem to have hw coherency.

> If it is true that I don't have hardware coherency should I still be
> using the pci_*_consistent api? Or should I switch to the
> dma_*_noncoherent api? Also what extra steps do I need to take to get
> this to work with a non-coherent system?

What you were doing seemed to be the right thing.  The API is supposed
to do the necessary address conversion and cache flushes for the driver.
That is the unchanged driver should work on any architecture.

The dma_* API is a generalization of the slightly older pci_* API.  You
can use either one for a PCI device.

In fact on MIPS the pci_* functions are just wrappers around their dma_*
equivalents.

> I am reading Documentation/DMA-API.txt which has some discussion about
> non-coherent systems to see if I can get a handle on this.
> 
> 
> > What hardware are you using anyway?
> 
> I am using the MDS-810 platform supplied by Momentum Data Systems which
> contains a Phillips/Nexperia pnx8950 chip. That chip contains a MIPS32
> core. http://www.mds.com/products/product.asp?prod=MDS-810

According to the current kernel code the PNX8550 non-coheren (aka software
coherency).

  Ralf

Re: Linux mips and DMA

[Jon Dufresne]

> What you were doing seemed to be the right thing.  The API is supposed
> to do the necessary address conversion and cache flushes for the driver.
> That is the unchanged driver should work on any architecture.

> 
> According to the current kernel code the PNX8550 non-coheren (aka software
> coherency).
> 

Thanks Ralf,

So it sounds like you're saying even though my platform does not have HW
coherency, it shouldn't affect the driver code.

In my code, the device creates an interrupt when a message is to be
passed to the host. The message is written in the DMA memory. When I
receive that interrupt, is there some sort of flush I should do to
update the memory the cpu sees, with the data written by the device?

As of right now when I receive the interrupt, then inspect the dma
memory it appears as though nothing was written to memory. So at this
point I left thinking either the device can't correctly write to the
memory, or the memory isn't ready to be read by the host. I am having
trouble discovering which case is actually occurring.

Jon

Re: Linux mips and DMA

[Jon Dufresne]

> > > Hardware coherency for DMA is the exception for low-end embedded MIPS
> > > systems andgiven the CPU address your's is no exception from that.
> > > 
> > > If your system was supporting hardware coherency for DMA I/O you would
> > > have obtained a cachable CPU address like:
> > > 
> > >   dma_handle=0x026f0000 size=0x00010000 cpu_addr=0x826f0000
> > >                                                   ^^^
> > > 
> > > A 0x8??????? would be in KSEG0 so cachable.
> > 
> > I do have a an embedded system. Are you saying that, in all likelyhood,
> > I do not have coherency? If I understand you correctly, this is a bad
> > thing right? Will I need to take extra care to work around this issue.
> > 
> > So are you saying I would prefer a cpu_addr in the 0x8******* range?
> 
> No. because you don't seem to have hw coherency.

So, the address I received is in uncachable memory, and therefore,
should look the same to all devices on the bus without hw cache
coherency?

> 
> > If it is true that I don't have hardware coherency should I still be
> > using the pci_*_consistent api? Or should I switch to the
> > dma_*_noncoherent api? Also what extra steps do I need to take to get
> > this to work with a non-coherent system?
> 
> What you were doing seemed to be the right thing.  The API is supposed
> to do the necessary address conversion and cache flushes for the driver.
> That is the unchanged driver should work on any architecture.

Should I need to do any explicit cache flushes in my code for hardware
without cache coherency? If I wanted to do this what function should I
use?


> According to the current kernel code the PNX8550 non-coheren (aka software
> coherency).

Will this be a problem for what I'm trying to accomplish? That is dma on
a pci device.

Thanks,
Jon