Re: PCIe dma-able memory location in physical memory for Intel Nehalem + Tylersburg architecture

[Andi Kleen <andi@firstfloor.org>]

On Fri, Jul 10, 2009 at 09:56:29AM +0200, Rob van de Voort wrote:
> 
> Hello,
> 
> I have the following question regarding the (re-)allocation of dma-able
> memory for a PCIe device. I wish to know on which CPU / Memory
> controller the dma-able memory will be allocated. 

There are several kinds of dma-able memory.

Coherent/consistent memory (pci_alloc_consistent, dma_alloc_coherent)
This is usually allocated on the node where the device is connected to,
if the platform reports it correctly using ACPI and there is enough
memory available on that node.  In some kernels there were bugs with this.

All other memory, later mapped with pci_map_*/dma_map_*: This 
is allocated according to the current numa memory policy, unless
you override it (alloc_pages_node etc.)

> The system runs a 64bit linux kernel. Furthermore a PCIe device which
> can only use 32 bit addressing is connected to the PCIe bus. I want to

GFP_DMA32 allocations are typically only on the first node, unless
your first node has less than 3GB of memory or so.

The only thing you can do for 4GB limited allocation devices is to
enable VT-d and allocate it elsewhere (alloc_pages_node et.al.)
and then let it be remapped by pci_map*/dma_map* into the first
4GB.

> P.S  I'd be grateful if people could point me in the direction of
> resources I could read. Thus far I have gone over the PCI / DMA

http://halobates.de/numaapi3.pdf has a general overview of NUMA
policy, although from the user space perspective.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.