Re: PCIe dma-able memory location in physical memory for Intel Nehalem + Tylersburg architecture

[Robert Hancock <hancockrwd@gmail.com>]

On 07/09/2009 03:04 AM, Rob van de Voort wrote:
> Hello,
>
> I have the following question regarding the (re-)allocation of dma-able
> memory for a PCIe device. I wish to know on which CPU / Memory
> controller the dma-able memory will be allocated. In addition i would
> like to know how i can control this allocation process, or how to
> reroute it to another CPU / Memory controller.
>
> The system i use contains two Nehalem processors connected via intel QPI
> bus. Both processors have 3GB of DDR3 RAM in their memory banks. The
> PCIe bus is controlled by a tylersburg io hub which is also connected to
> the intel QPI bus
>
> The system runs a 64bit linux kernel. Furthermore a PCIe device which
> can only use 32 bit addressing is connected to the PCIe bus. I want to
> allocate dma-able memory for this device using a driver, i know how to
> do this. For maximal performance i would like to control the allocation
> of dma-able memory, in particular the allocation to a certain CPU /
> memory controller. The tylersburg IO hub has an influence on this
> allocation but can i influence this.
>
> So my questions are:
> -1- How can i know were our dma-able memory is allocated in physical
> memory?
> -2- How can i control / redirect this allocation to a certain memory
> controller / CPU?
>
> Thanks for any reply,
>
> Rob van de Voort
>
> P.S  I'd be grateful if people could point me in the direction of
> resources I could read. Thus far I have gone over the PCI / DMA
> documentation in
> the kernel, and a couple of articles on lwn.
> If this is the wrong mailing list for this type of question i hope you
> can redirect me to the correct one.

I believe that dma_alloc_coherent, etc. normally try to allocate DMA 
memory on the same node as the device is connected to. For that to 
actually happen, the motherboard's ACPI tables have to be set up to 
indicate the proper node to memory allocations. As well, since your 
device can only address 32-bit memory (below 4GB) this may prevent 
memory that gets allocated from being on the optimal CPU.