dma_alloc_coherent versus streaming DMA, neither works satisfactory

Mike Looijmans
2015-04-23T11:52:34Z

?I'm writing a driver that transfers data on a Zynq-7000 between the ARM and PL
part, using a buffering scheme similar to IIO. So it allocates buffers, which
the user can memory-map and then send/receive using ioctl calls.
The trouble I have it that transitioning these buffers into user space is
costing more CPU time than actually copying them.

The self-written DMA controller is in logic, can queue up several transfers
and operates fine.

The trouble I have with the driver is that for the DMA transfers, I want to
use a zero-copy style interface, because the data usually consists of video
frames, and moving them around in memory is prohibitively expensive.

I based my implementation on what IIO (industrial IO) does, and implemented
IOCTL calls to the driver to allocate, free, enqueue and dequeue blocks that
are owned by the driver. Each block can be mmapped into user space.

Using dma_alloc_coherent to allocate the blocks, and then just using them
without any extra measures works just fine. The system can transfer data at
600MB/s to and from DDR into logic with very little CPU intervention. However,
the memory returned by dma_mmap_coherent appears to be uncached, because
accessing this area from userspace is horribly slow (e.g. reading 200MB
byte-by-byte in a simple for loop takes 20 seconds, while this takes about a
second in malloced memory)

After reading some documentation, I decided that I should use a streaming DMA
interface because my driver knows exactly when logic or CPU "owns" the data
blocks. So instead of the "coherent" functions, I just kmalloc these buffers
and then use dma_map_single_* to initialize them. Before and after DMA
transfers, I call the appropriate dma_sync_single_for_* method.

By mapping the kmalloced memory into user space, I once again have speedy
access to this memory, and caching is enabled. Data transfers also work well
and extensive testing shows that this works well. However, the in-kernel
performance is now completely crippled. The system spends so much time in the
dma_sync_single... calls, the the CPU now becomes a limiting factor. This
limits the transfer speeds to about only 150MB/s. This method is only about
20% more CPU intensive than just copying the data from the DMA buffer into a
user buffer using a copy_to_user call!

Since my DMA controller is pretty smart, I also experimented with transfers
directly from user memory. This boiled down to calling "get_user_pages" and
then constructing a scatter-gather list using sg_init_table and then adding
those user pages. Then call "dma_map_sg" to translate and coalesce the pages
into DMA requests. Just this pagetable housekeeping took about the same amount
of processing time as the copy_from_user call, which made me abandon that code
before even getting to the point of actually transferring the data.

Based on that experience, I'd think the dma_sync calls do similar things
(walking page tables and changing some attributes) and that is where they
spend so much time.

I also tried cheating by not calling the dma_sync methods at all, but this
(surprisingly) led to hangups in the driver. I would have only expected data
corruption, not "hang". I'm still investigating that route. Specifying
"dma-coherent" has the same effect as it basically "nulls" the sync methods.

Also tried to replace the "dma_mmap_coherent" call by a simple
"remap_pfn_range" so as to prevent setting the cache attributes on that
region, but that didn't have any effect at all, it appears that the
non-cachable property was already applied in the dma_alloc_coherent method.

I added some timing code to the "sync" calls, this is what I get (numbers in
microseconds) when using 1MB blocks of streaming DMA transfers:

dma_sync_single_for_device(TO_DEVICE): 3336
dma_sync_single_for_device(FROM_DEVICE): 1991
dma_sync_single_for_cpu(FROM_DEVICE): 2175
dma_sync_single_for_cpu(TO_DEVICE): 0
dma_sync_single_for_device(TO_DEVICE): 3152
dma_sync_single_for_device(FROM_DEVICE): 1990
dma_sync_single_for_cpu(FROM_DEVICE): 2193
dma_sync_single_for_cpu(TO_DEVICE): 0

As you can see, the system spends 2 or 3 ms on "housekeeping" for each
transition, except the cpu(TO_DEVICE) one which appears to be free which is
perfectly logical because returning the outgoing buffer to the CPU should not
need any special cache handling. I would have expected the
for_device(FROM_DEVICE) to be free as well, but surprisingly this one takes up
2ms as well.

Adding the numbers, it takes over 7 ms overhead to transfer 1MB data, hence
1MB/0.007s or about 150MB/s would be the maximum possible data transfer rate.

Can anyone here offer some advise on this?

Kind regards,

Mike Looijmans
System Expert

TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
Telefax: +31 (0) 499 33 69 70
E-mail: mike.looijmans at topicproducts.com