DMA with PCIe and very large DMA transfers

[Alex <arghness@gmail.com>]

Are there any examples (or just documentation) on providing DMA for
PCIe devices? I have read the DMA-mapping.txt document but wasn't sure
if this was all relevant to PCIe. For example, pci_set_dma_mask talks
about driving pins on the PCI bus, but PCIe doesn't work in quite the
same way. Perhaps these calls have no effect in this case (similar to
the PCI latency timers) but I just wondered.

I'm also interested in knowing if any drivers perform very large DMA
transfers. I'm putting together a driver for a specialist high-speed
data acquisition device that typically might need a DMA buffer of
100-500MB (ouch!) in the low 32 bit address space (or possibly 36 bit
address space, but I'm not sure if this is possible to allocate
without allocating as much as possible and then discarding?) but only
supports a very limited number of scatter/gather entries (between 1
and 4). The particular use-case for this is a ring buffer with
registers in IO memory that are used to keep track of read/write
pointers in the buffer. The device writes to the DMA memory when there
is space in the ring buffer i.e. the DMA transfer is only from device
to host.

I would like to perform the DMA straight from device to user-space
(probably via mmap), which I think requires a consistent/coherent
rather than streaming DMA so that I may read from the ring buffer
while the DMA may still be active (although not active in that section
of the buffer).

I assume that to allocate that much memory in physical contiguous
addresses will require a driver to be loaded as soon as possible at
startup. I was thinking about trying to grab a lot of high-order pages
and try and make them one contiguous block - is that feasible?
Browsing the archives, I found references to early allocation for
large buffers, but no direct links to existing examples or recommended
techniques on how to stitch pages together in to a single buffer. Is
there a platform independent way to ensure cache coherency with
allocated pages like this (i.e. not allocated with
pci_alloc_consistent / dma_alloc_coherent)?

I suppose that anything which takes a large chunk of physical memory
at startup isn't very recommended, but this is for a specialist device
and the host machine will probably be dedicated to using it.

As an aside, my module, driver and device are under the pci bus in
sysfs - should be PCIe device be showing under the pci_express bus?
This appears to be the PCIe Port Bus Driver and only has the aer
driver listed under it. I can't find any other drivers in the kernel
source that use it (I'm currently running 2.6.21).

Thanks,
Alex