Trouble with DMA on PPC linux question

Trouble with DMA on PPC linux question

[Bruce_Leonard]

Good afternoon everyone,

We're trying to get some performance gains in an older embedded design by 
adding DMA to our NAND driver.  The HW is an MPC8349 talking across a PCI 
bus to a NAND controller and we have 512Mb of RAM.  We're using the 3.18 
kernel and the Freescale "fsl,mpc8349-dma" driver.  I've verified using a 
bus analyzer that DMA transactions are occurring on the PCI bus correctly 
(correct addresses and the data I'm reading is coming across the bus to 
the processor correctly).  What's not happening is periodically the data 
being read doesn't make it to RAM.  I've narrowed this down to the 
dma_addr_t I get back from dma_map_single().

Now I'm not an expert on how memory management in the PPC linux kernel 
works, but based on some experimentation and stepping through some of the 
code, translating a kernel virtual address is essentially subtracting 
0xC0000000 from the virtual address.  I know the equations a bit more than 
that, I've dug into some of the macros, but many of the constants compile 
to zero on my setup, so the end result is just the subtraction.

On the DMA transactions that work, the virtual address I hand to 
dma_map_single() is something like 0xe0840000 and the dma_addr_t result is 
0x10840000 which is less than my 512Mb limit.  On the transactions that 
don't work, the virtual address is 0xd5390000 with the mapped result being 
0x25390000, which is past my upper bound on my RAM.  In fact it's not even 
in my memory map, there's a hole there.  (Evidently the MPC4349 DMA engine 
bypasses the TLBs, since I'm not getting an exception of any 
kind...learned something new today!)  So on the transactions that don't 
work, they fail because the physical address I give to the DMA engine 
doesn't exist.  The only error indication I get is when I get an ECC error 
because what's pointed to be the virtual address (where ever that may be) 
still contains zeros and it fails the ECC comparison check.

So my question is, where should I be looking or what config option should 
I be checking to try and figure out why the upper layers 
(MTD/UBI/UBIFS/user space) should be giving the NAND layer or my driver a 
virtual address that can't be translated into a physical address?  One 
thing I have noticed (though I don't know if it's relevant or not) is that 
when I get a "good" virtual address it's through a call to 
nand_subpage_read() and when I get a "bad" virtual address it's through a 
call to nand_read_page_swecc().

I'm not asking if someone can solve my problem for me, but any suggestions 
of what rocks I can turn over to look for clues would be greatly 
appreciated.

Thanks for you time and suggestions!

Bruce

Re: Trouble with DMA on PPC linux question

[Benjamin Herrenschmidt]

On Mon, 2016-04-18 at 14:54 -0700, Bruce_Leonard@selinc.com wrote:
> 
> On the DMA transactions that work, the virtual address I hand to 
> dma_map_single() is something like 0xe0840000 and the dma_addr_t result is 
> 0x10840000 which is less than my 512Mb limit.  On the transactions that 
> don't work, the virtual address is 0xd5390000 with the mapped result being 
> 0x25390000, which is past my upper bound on my RAM.  In fact it's not even 
> in my memory map, there's a hole there. 

Where does this virtual address come from ?

The kernel has two types of virtual addresses. Those coming from the
linear mapping (the stuff you get from kmalloc() for example, or
get_pages()) which can be translated using that simple substraction.

The other is the vmalloc space, and that is a non-linear mapping of
random pages.

If your vaddr comes from the latter it can't be passed to
dma_map_single as-is, you need to get to the underlying pages first.

Ben.

Re: Trouble with DMA on PPC linux question

[Bruce_Leonard]

Ben,

Benjamin Herrenschmidt <benh@au1.ibm.com> wrote on 04/19/2016 01:45:40 AM:

> From: Benjamin Herrenschmidt <benh@au1.ibm.com>
> To: Bruce_Leonard@selinc.com, linuxppc-dev@lists.ozlabs.org
> Date: 04/19/2016 01:46 AM
> Subject: Re: Trouble with DMA on PPC linux question
> 
> On Mon, 2016-04-18 at 14:54 -0700, Bruce_Leonard@selinc.com wrote:
> > 
> > On the DMA transactions that work, the virtual address I hand to 
> > dma_map_single() is something like 0xe0840000 and the dma_addr_t 
result is 
> > 0x10840000 which is less than my 512Mb limit.  On the transactions 
that 
> > don't work, the virtual address is 0xd5390000 with the mapped result 
being 
> > 0x25390000, which is past my upper bound on my RAM.  In fact it's not 
even 
> > in my memory map, there's a hole there. 
> 
> Where does this virtual address come from ?
> 
> The kernel has two types of virtual addresses. Those coming from the
> linear mapping (the stuff you get from kmalloc() for example, or
> get_pages()) which can be translated using that simple substraction.
> 
> The other is the vmalloc space, and that is a non-linear mapping of
> random pages.
> 
> If your vaddr comes from the latter it can't be passed to
> dma_map_single as-is, you need to get to the underlying pages first.
> 
> Ben.
> 

That's a good question.  I'm not sure where the addresses come from right 
now (they're handed to me from the MTD layer), but I'll certainly dig into 
that and see.

Thanks for the help!  I appreciate the pointer.

Bruce