Memory allocation in NUMA system

["Yang, Xiaowei" <xiaowei.yang@intel.com>]

Currently, when alloc_domheap_pages() is called with no range specified
(which is the case for allocating domain memory during its creation), it
uses such priority to allocate memory:
1) current node, > DMA range (2^dma_bitsize = 4G).
2) other nodes, > DMA range
3) current node, all range
4) other nodes, all range

Let's say we have a 2-node system, with node0 and node1's memory range 
being 0-0xc0000000 (<4G) and 0x100000000-0x1c0000000 (>4G) respectively. 
In that case, node1's memory is always preferred for domain memory 
allocation, no matter which node the created domain is pinned to. It 
results in performance penalty.

One possible fix is to specify all range for the domain memory 
allocation, which means local memory is preferred. This change may be 
restricted only to the domain pinned to one node for less impact.

One side effect is that the DMA memory size may be smaller, which makes 
device domain unhappy. This can be addressed by reserving node0 to be 
used lastly.

Comments?

Thanks,
Xiaowei