Re: Xen 3.4.1 NUMA support

[Andre Przywara <andre.przywara@amd.com>]

George Dunlap wrote:
> Andre Przywara wrote:
>> BTW: Shouldn't we set finally numa=on as the default value?
>>   
> Is there any data to support the idea that this helps significantly on 
> common systems?
I did some tests on an 8 node machine. I will retry this later on 
4-nodes and 2-nodes systems, but I assume similar numbers. I used 
multiple guests in parallel each running bw_mem of lmbench, which is 
admittedly quite NUMA sensitive. I cannot publish real numbers (yet?), 
but the results were dramatic:
with numa=on I got the same results for each guest (the same as the 
native result) when the number of guests was smaller or equal the number 
of nodes (since each guest got it's own memory controller).
If I disabled NUMA aware placement by explicitly specifying cpus="0-31" 
  in the config file or booted with numa=off, the values dropped down by 
factor 3-5 (!) (even for a few guests) with some variance due to the 
random nature of core to memory mapping.
Overcommitting the nodes (letting multiple guests use each node) lowered 
the values to about 80% for two guests and 60% for three guests per 
node, but it never got anywhere close to the numa=off values.
So these results encourage me again to opt for numa=on as the default value.
Keir, I will check if dropping the node containment in the CPU 
overcommitment case is an option, but what would be the right strategy 
in that case?
Warn the user?
Don't contain at all?
Contain to more than onde node?

Regards,
Andre.

-- 
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 448 3567 12
----to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Andrew Bowd; Thomas M. McCoy; Giuliano Meroni
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632