[RFC] NUMA support

["Andre Przywara" <andre.przywara@amd.com>]

All,
thanks Ronghui for your patches and ideas. To make a more structured 
approach to a better NUMA support, I suggest to concentrate on 
one-node-guests first:
* introduce CPU affinity to memory allocation routines called from Dom0. 
This is basically my patch 2/4 from August. We should think about using 
a NUMA node number instead of a physical CPU, is there something to be 
said against this?

* find _some_ method of load balancing when creating guests. The method 
1 from Ronghui is a start, but a real decision based on each node's 
utilization (or free memory) would be more reasonable.

* patch the guest memory allocation routines to allocate memory from 
that specific node only (based on my patch 3/4)

* use live migration to local host to allow node migration. Assuming 
that localhost live migration works reliably (is that really true?) it 
shouldn't be too hard to implement this (basically just using node 
affinity while allocating guest memory). Since this is a rather 
expensive operation (takes twice the memory temporarily and quite some 
time), I'd suggest to trigger that explicitly from the admin via a xm 
command, maybe as an addition to migrate:
# xm migrate --live --node 1 <domid> localhost
There could be some Dom0 daemon based re-balancer to do this somewhat 
automatically later on.

I would take care of the memory allocation patch and would look into 
node migration. It would be great if Roughui or Anthony would help to 
improve the "load balancing" algorithm.

Meanwhile I will continue to patch that d*** Linux kernel to accept both 
CONFIG_NUMA and CONFIG_XEN without crashing that early ;-), this should 
allow both HVM and PV guests to support multiple NUMA nodes within one 
guest.

Also we should start a discussion on the config file options to add:
Shall we use "numanodes=<nr of nodes>", something like "numa=on" (for 
one-node-guests only), or something like "numanode=0,1" to explicitly 
specify certain nodes?

Any comments are appreciated.

> I read your patches and Anthony's commands. Write a patch based on
> 
> 1:    If guest set numanodes=n (default it will be 1 means that this
> guest   	will be restricted in one node); hypervisor will choose
> begin node to 	pin for this guest use round robin. But the method I use
> need a 	spin_lock to prevent create domain at same time. Are there any
> more 	good methods, hope for your suggestion.
That's a good start, thank you. Maybe Keir has some comments on the 
spinlock issue.
> 2:	pass node parameter use higher bits in flags when create domain.
> At  	this time, domain can record node information in domain struct
> for 	further use, i.e. show which node to pin when setup_guest.    
> 	If use this method, in your patch, can simply balance nodes just
> like 	below;
> 
>> +    for (i=0;i<=dominfo.max_vcpu_id;i++)
>> +    {
>> +        node= ( i * numanodes ) / (dominfo.max_vcpu_id+1)+ 		
>> +		domaininfo.first_node;
>> +        xc_vcpu_setaffinity (xc_handle, dom, i, nodemasks[node]);
>> +    }
How many bits do you want to use? Maybe it's not a good idea to abuse 
some variable to hold a limited number of nodes only ("640K ought to be 
enough for anybody" ;-) But the general idea is good.

Regards,
Andre.

-- 
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 277-84917
----to satisfy European Law for business letters:
AMD Saxony Limited Liability Company & Co. KG
Sitz (Geschäftsanschrift): Wilschdorfer Landstr. 101, 01109 Dresden, 
Deutschland
Registergericht Dresden: HRA 4896
vertretungsberechtigter Komplementär: AMD Saxony LLC (Sitz Wilmington, 
Delaware, USA)
Geschäftsführer der AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy