NUMA TODO-list for xen-devel

[Dario Faggioli <raistlin@linux.it>]

[-- Attachment #1.1: Type: text/plain, Size: 4233 bytes --]

Hi everyone,

With automatic placement finally landing into xen-unstable, I stated
thinking about what I could work on next, still in the field of
improving Xen's NUMA support. Well, it turned out that running out of
things to do is not an option! :-O

In fact, I can think of quite a bit of open issues in that area, that I'm
just braindumping here. If anyone has thoughts or idea or feedback or
whatever, I'd be happy to serve as a collector of them. I've already
created a Wiki page to help with the tracking. You can see it here
(for now it basically replicates this e-mail):

 http://wiki.xen.org/wiki/Xen_NUMA_Roadmap

I'm putting a [D] (standing for Dario) near the points I've started
working on or looking at, and again, I'd be happy to try tracking this
too, i.e., keeping the list of "who-is-doing-what" updated, in order to
ease collaboration.

So, let's cut the talking:

    - Automatic placement at guest creation time. Basics are there and
      will be shipping with 4.2. However, a lot of other things are
      missing and/or can be improved, for instance:
[D]    * automated verification and testing of the placement;
       * benchmarks and improvements of the placement heuristic;
[D]    * choosing/building up some measure of node load (more accurate
         than just counting vcpus) onto which to rely during placement;
       * consider IONUMA during placement;
       * automatic placement of Dom0, if possible (my current series is
         only affecting DomU)
       * having internal xen data structure honour the placement (e.g., 
         I've been told that right now vcpu stacks are always allocated
         on node 0... Andrew?).

[D] - NUMA aware scheduling in Xen. Don't pin vcpus on nodes' pcpus,
      just have them _prefer_ running on the nodes where their memory
      is.

[D] - Dynamic memory migration between different nodes of the host. As
      the counter-part of the NUMA-aware scheduler.

    - Virtual NUMA topology exposure to guests (a.k.a guest-numa). If a
      guest ends up on more than one nodes, make sure it knows it's
      running on a NUMA platform (smaller than the actual host, but
      still NUMA). This interacts with some of the above points:
       * consider this during automatic placement for
         resuming/migrating domains (if they have a virtual topology,
         better not to change it);
       * consider this during memory migration (it can change the
         actual topology, should we update it on-line or disable memory
         migration?)

    - NUMA and ballooning and memory sharing. In some more details:
       * page sharing on NUMA boxes: it's probably sane to make it
         possible disabling sharing pages across nodes;
       * ballooning and its interaction with placement (races, amount of
         memory needed and reported being different at different time,
         etc.).

    - Inter-VM dependencies and communication issues. If a workload is
      made up of more than just a VM and they all share the same (NUMA)
      host, it might be best to have them sharing the nodes as much as
      possible, or perhaps do right the opposite, depending on the
      specific characteristics of he workload itself, and this might be
      considered during placement, memory migration and perhaps
      scheduling.

    - Benchmarking and performances evaluation in general. Meaning both
      agreeing on a (set of) relevant workload(s) and on how to extract
      meaningful performances data from there (and maybe how to do that
      automatically?).

So, what do you think?

Thanks and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel