Provision for direct device to device DMA

["\"Ing. Daniel Rozsnyó\"" <daniel@rozsnyo.com>]

We are working on a product with PCIe based backplane (with PLX switch 
chips) to connect various high-bandwidth devices (namely an image sensor 
and multiple SSDs). The Linux system runs on a small (ARM based) 
processor, which has limited memory bandwidth (32bit/800MHz) and more 
importantly limited PCIe bandwidth (x1 root port). From linux sources, 
we could understand, that all DMA is towards or from system memory, 
which is the bottleneck in our use case (we peak at 2000MB/s).

What is the advised way of doing disk i/o with data coming from/to 
another PCIe device? We would prefer to keep a filesystem on the target 
drives, but it is not a hard requirement. The storage solution is AHCI 
(multiple ASM1061) and the other end are our FPGA devices. For future, 
it would be nice to stream over PCIe fabric into a 10GE nic, like i82599 
(ixgbe), from both the storage and from FPGA.

Do we need to write a kernel module to handle such transfers on a very 
low level (like a combined ahci-fpga-nic driver)?

Or a special mmap() before write() would auto-magically do the right job?


Thank you,
Daniel Rozsnyo