Re: NUMA-aware VM placement in Xen

[George Dunlap <george.dunlap@eu.citrix.com>]

On 24/02/12 10:12, Dario Faggioli wrote:
> Hi guys,
>
> As some of you know I'm working on putting some kind of NUMA-aware
> placement of the various VMs within Xen. This means I'm investigating
> deeply how memory allocation works, which isn't an easy task for me (I
> started completely from scratch), so forgive me if I say something
> wrong! :-P
>
> The status is I'm dealing for a while with a "design issue" I'd be very
> glad to discuss with someone, as I'm not sure which path to go for...
>
> To keep it short, what I need is a place --ideally during VM creation--
> where I can check how much memory a VM wants against how much memory is
> available in the various NUMA-nodes, and use this as the basis of my
> decision. The question is, where is this place?
>
> I traced memory related calls (e.g., for HVMs) from libxl__build_hvm to
> xc_hvm_build_target_mem to setup_guest to xc_domain_populate_physmap and
> alloc_domheap_pages. The last twos have been my target for a while, but
> I'm not so sure they would be the right choice, mainly because both of
> them are called for allocating _only_part_ of the VM's memory, i.e.,
> some extents of it at each call (am I right?).
>
> Basically, given alloc_domheap_pages uses d->node_affinity for deciding
> from which node(s) to actually take memory from, I was planning to
> either use the same mask or build a new one with similar purposes, the
> problem being _where_ to populate it with the proper nodes.
> I'm now looking at xc_domain_setmaxmem-->do_domctl(XEN_DOMCTL_max_mem),
> although I think it's too early, and I'd end up guessing wrt a lot of
> aspects... But considering xm/xend was doing the same even earlier (at
> least I think)...
So the first question is, where should the decision about NUMA placement 
be made, and the second is how that level should implement it.

Doing it at the libxc level I think is not right.  It seems to me we 
have two options:
* Have libxl do the NUMA placement on behalf of the toolstack.  In that 
case, the libxl_domain_create_new function should look at the available 
memory, the NUMA layout, &c, and then set d->node_affinity before 
calling xc_hvm_build.
* Have the toolstack do it.  In this case, you'd be modifying xl to set 
d->node_affinity before calling libxl's domain creation function.

Do those options work?  Let me know if I've misunderstood anything.

Any thoughts one way or the other from anyone?

I'd be tempted to have it be optional -- you can set "numa=auto" and the 
domain creation function will do the simple thing; or you can set 
"numa=manual" and have the toolstack / config file set the nodes 
manually.  That would translate pretty well to config files as well -- 
more "set the knobs" administrators could set the numa layout in the 
config file manually if they wanted.

Thoughts?

  -George

>
> Sorry fro writing so much... Any help/ideas you feel comfortable with
> sharing would be very appreciated! :-)
>
> Thanks a lot and regards,
> Dario
>