Async DMA question regarding dma_async_memcpy_buf_to_buf

Async DMA question regarding dma_async_memcpy_buf_to_buf

[Bruce_Leonard]

I have a question about 
.../drivers/dma/dmaengine.c/dma_async_memcpy_buf_to_buf() in 2.6.28-rc2. 
It looks like it always assumes that the source pointer points to data 
coming from the CPU and the destination pointer always points to the DMA 
device.  I say this because of the following two lines from the function:

 479        dma_src = dma_map_single(dev->dev, src, len, DMA_TO_DEVICE);
 480        dma_dest = dma_map_single(dev->dev, dest, len, DMA_FROM_DEVICE
);


As you can see 'src' is mapped DMA_TO_DEVICE indicating the data is going 
from the CPU to the device, and 'dest' is mapped DMA_FROM_DEVICE, 
indicating the opposite direction.  Seems to me this means the function is 
assuming a certain direction (i.e., a write to the device), but there 
isn't a corresponding function for going the other direction.  This leads 
me to believe that the function is intended to operate in either 
direction.  But this is in contradiction to both the mapped directions and 
what I've read in LDD3, ch 15.

The hardware doesn't care (I'm using an MPC8347E), as far as the DMA 
engine is concerned these are just addresses.  All of this goes to cache 
coherency.  The map/unmap functions are supposed to ensure that data is 
copied and caches flushed at the right times to ensure cache coherency. So 
the question is this; is the dma_async_memcpy_buf_to_buf() function 
intended to be bi-directional and is it safe to pass it a 'src' pointer 
that's actually coming from the device, or does there need to be a second 
function for doing a copy from the device to the CPU?

Thanks.

Bruce

Async DMA question regarding dma_async_memcpy_buf_to_buf

[Bruce_Leonard]

I have a question about 
.../drivers/dma/dmaengine.c/dma_async_memcpy_buf_to_buf() in 2.6.28-rc2. 
It looks like it always assumes that the source pointer points to data 
coming from the CPU and the destination pointer always points to the DMA 
device.  I say this because of the following two lines from the function:

 dma_src = dma_map_single(dev->dev, src, len, DMA_TO_DEVICE);
 dma_dest = dma_map_single(dev->dev, dest, len, DMA_FROM_DEVICE);


As you can see 'src' is mapped DMA_TO_DEVICE indicating the data is going 
from the CPU to the device, and 'dest' is mapped DMA_FROM_DEVICE, 
indicating the opposite direction.  Seems to me this means the function is 

assuming a certain direction (i.e., a write to the device), but there 
isn't a corresponding function for going the other direction.  This leads 
me to believe that the function is intended to operate in either 
direction.  But this is in contradiction to both the mapped directions and 

what I've read in LDD3, ch 15.

The hardware doesn't care (I'm using an MPC8347E), as far as the DMA 
engine is concerned these are just addresses.  All of this goes to cache 
coherency.  The map/unmap functions are supposed to ensure that data is 
copied and caches flushed at the right times to ensure cache coherency. So 

the question is this; is the dma_async_memcpy_buf_to_buf() function 
intended to be bi-directional and is it safe to pass it a 'src' pointer 
that's actually coming from the device, or does there need to be a second 
function for doing a copy from the device to the CPU?

Thanks.

Bruce

Re: Async DMA question regarding dma_async_memcpy_buf_to_buf

[Scott Wood]

Bruce_Leonard@selinc.com wrote:
> The hardware doesn't care (I'm using an MPC8347E), as far as the DMA 
> engine is concerned these are just addresses.  All of this goes to cache 
> coherency.  The map/unmap functions are supposed to ensure that data is 
> copied and caches flushed at the right times to ensure cache coherency. So 
> 
> the question is this; is the dma_async_memcpy_buf_to_buf() function 
> intended to be bi-directional and is it safe to pass it a 'src' pointer 
> that's actually coming from the device, or does there need to be a second 
> function for doing a copy from the device to the CPU?

The "device" is the DMA engine, thus src is inherently going to the 
device and dest is inherently coming from the device.

-Scott

Re: Async DMA question regarding dma_async_memcpy_buf_to_buf

[Bruce_Leonard]

Hi Scott,

Thanks for the reply.

> > 
> > the question is this; is the dma_async_memcpy_buf_to_buf() function 
> > intended to be bi-directional and is it safe to pass it a 'src' 
pointer 
> > that's actually coming from the device, or does there need to be a 
second 
> > function for doing a copy from the device to the CPU?
> 
> The "device" is the DMA engine, thus src is inherently going to the 
> device and dest is inherently coming from the device.
>

Yes, I understand the directions and what they apply to, that's obvious 
from the macros themselves.  What I don't understand is if it's safe, from 
a kernel/cache standpoint, to pass to the function a src pointer coming 
_from_ the device and a dest pointer that's going _to_ the device?  In 
other words, can I use this function to _write_ to a DMA slave?  If not, 
there's a functionality missing from the async DMA engine.

Thanks.

Bruce

Re: Async DMA question regarding dma_async_memcpy_buf_to_buf

[Scott Wood]

Bruce_Leonard@selinc.com wrote:
> Yes, I understand the directions and what they apply to, that's obvious 
> from the macros themselves.  What I don't understand is if it's safe, from 
> a kernel/cache standpoint, to pass to the function a src pointer coming 
> _from_ the device and a dest pointer that's going _to_ the device?  In 
> other words, can I use this function to _write_ to a DMA slave?  If not, 
> there's a functionality missing from the async DMA engine.

What exactly do you mean by "the device"?  if it's the DMA engine, then 
that's a meaningless request, by the definition of "source" and 
"destination".  If you mean some other device that happens to be 
providing a buffer for you to DMA into or out of, that's irrelevant to 
the DMA API; it's just memory that happens to live somewhere else (and 
possibly not be cached).

-Scott

Re: Async DMA question regarding dma_async_memcpy_buf_to_buf

[Bruce_Leonard]

> 
> What exactly do you mean by "the device"?  if it's the DMA engine, then 
> that's a meaningless request, by the definition of "source" and 
> "destination".  If you mean some other device that happens to be 
> providing a buffer for you to DMA into or out of, that's irrelevant to 
> the DMA API; it's just memory that happens to live somewhere else (and 
> possibly not be cached).
> 

Hummm, let me try again, I may be tripping up on my ignorance of the 
kernel.  According to Linux Device Drivers 3, the DMA_TO/FROM_DEVICE 
macros impact when the dma_map/unmap_single functions copy data and do 
cache flushes to ensure cache coherency.  DMA_TO_DEVICE tells 
dma_map_single() to ensure that all data is copied out to memory and cache 
is flushed before doing the transfer and DMA_FROM_DEVICE tell 
dma_unmap_single() to ensure all data is in main memory after the 
transfer.  (My understanding is that this is really only important on 
archs that use bounce buffers which I'm not).

So, no as far as "the device" (and I do mean the DMA engine) is concerned, 
the "mapping" of the src and dest pointers make zero difference, the 
hardware doesn't care since it's just acting on raw addresses.  However, 
it does (or at least I think it does) matter to the kernel and making sure 
that the cache doesn't get screwed up.  That all being said, I may be 
worried about nothing.  I don't know enough about any of this to be sure. 
I do know that in the past I've spent weeks trying to track down problems 
that were caused by cache coherency, and LDD3 is clearly warning about 
cache problems if these macros aren't used correctly.  But it sounds like 
you're saying those macros are unimportant.  The async DMA stuff is new 
since LDD3, so do those macros not apply?  Can 
dma_async_memcpy_buf_to_buf() _safely_ be used to DMA either direction 
(CPU <=> peripheral) _without_ cache coherency problems?

Thanks for the help.

Bruce

Re: Async DMA question regarding dma_async_memcpy_buf_to_buf

[Scott Wood]

Bruce_Leonard@selinc.com wrote:
> Hummm, let me try again, I may be tripping up on my ignorance of the 
> kernel.  According to Linux Device Drivers 3, the DMA_TO/FROM_DEVICE 
> macros impact when the dma_map/unmap_single functions copy data and do 
> cache flushes to ensure cache coherency.  DMA_TO_DEVICE tells 
> dma_map_single() to ensure that all data is copied out to memory and cache 
> is flushed before doing the transfer and DMA_FROM_DEVICE tell 
> dma_unmap_single() to ensure all data is in main memory after the 
> transfer.

Right.

> (My understanding is that this is really only important on 
> archs that use bounce buffers which I'm not).

It's also important on architectures where DMA is non-coherent (mpc83xx 
DMA is coherent).

> So, no as far as "the device" (and I do mean the DMA engine) is concerned, 
> the "mapping" of the src and dest pointers make zero difference, the 
> hardware doesn't care since it's just acting on raw addresses.  However, 
> it does (or at least I think it does) matter to the kernel and making sure 
> that the cache doesn't get screwed up.

Right.

> But it sounds like 
> you're saying those macros are unimportant. 

No, they're very important on certain hardware.

> Can dma_async_memcpy_buf_to_buf() _safely_ be used to DMA either direction 
> (CPU <=> peripheral) _without_ cache coherency problems?

Any given memcpy operation involves *both* directions.  Data is copied 
out of host memory into the DMA engine's internal buffer (this is the 
src mapping), and then it its copied back into host memory at a 
different address (this is the dest mapping).

-Scott

Re: Async DMA question regarding dma_async_memcpy_buf_to_buf

[Bruce_Leonard]

> 
> Any given memcpy operation involves *both* directions.  Data is copied 
> out of host memory into the DMA engine's internal buffer (this is the 
> src mapping), and then it its copied back into host memory at a 
> different address (this is the dest mapping).
> 

Whoops!  Now we may be getting to the heart of what I'm confused about ;). 
 My bad, some more background may be in order for you to help me.

What I'm trying to do is set up a DMA transfer to/from a NAND flash 
controller hanging on a PCI bus.  So, let's say I'm doing a read of the 2K 
NAND page.  My "source" is a PCI address and is what I was assuming I 
needed to pass to the memcpy function as "src".  But that assumption 
conflicts with your statement that "Data is copied out of host memory into 
the DMA engine's internal buffer (this is the src mapping)".  Is this 
where my ignorance is jumping up and bitting me?

Maybe (with you clarification) this is starting to make sense.  It doesn't 
matter if the "src" is host memory or PCI space.  What the "src" and it's 
DMA_TO_DEVICE mapping is saying to the kernel is "this data needs to be 
moved into the DMA engine", while the "dest" and it's DMA_FROM_DEIVCE 
mapping is telling the kernel "this data needs to be moved from the DMA 
engine".

Or am I even more confused :(  ?

Bruce

Re: Async DMA question regarding dma_async_memcpy_buf_to_buf

[Scott Wood]

Bruce_Leonard@selinc.com wrote:
> Maybe (with you clarification) this is starting to make sense.  It doesn't 
> matter if the "src" is host memory or PCI space.  What the "src" and it's 
> DMA_TO_DEVICE mapping is saying to the kernel is "this data needs to be 
> moved into the DMA engine", while the "dest" and it's DMA_FROM_DEIVCE 
> mapping is telling the kernel "this data needs to be moved from the DMA 
> engine".

Right.

-Scott

Re: Async DMA question regarding dma_async_memcpy_buf_to_buf

[Bruce_Leonard]

Scott Wood <scottwood@freescale.com> wrote on 11/24/2008 01:28:20 PM:

> Bruce_Leonard@selinc.com wrote:
> > Maybe (with you clarification) this is starting to make sense.  It 
doesn't 
> > matter if the "src" is host memory or PCI space.  What the "src" and 
it's 
> > DMA_TO_DEVICE mapping is saying to the kernel is "this data needs to 
be 
> > moved into the DMA engine", while the "dest" and it's DMA_FROM_DEIVCE 
> > mapping is telling the kernel "this data needs to be moved from the 
DMA 
> > engine".
> 
> Right.
> 
> -Scott

Thank you sooooo much.  Now it makes sense.  I guess I was getting bogged 
down in the hardware sense of "src" and "dest"

Thanks again.

Bruce