Re: [PATCH 00/15] Implement per-vcpu NUMA node-affinity for credit1

[George Dunlap <george.dunlap@eu.citrix.com>]

On 11/05/2013 11:29 AM, Dario Faggioli wrote:
> On gio, 2013-10-03 at 19:45 +0200, Dario Faggioli wrote:
>> Hi everyone,
>>
> Ping?
>
> Despite being a rather big series, it shouldn't be too controversial.
> For that reason, I was hoping hat it could go in before code freeze, but
> I need some review ASAP. :-P
>
> Some minor and independent patches have been already applied by IanC...
> I'm happy to resend the series, if that would be helpful, just let me
> know.

This was basically next on my list. :-)

Have you rebased locally -- does this still apply fairly cleanly to tip? 
  If not, a re-send would be nice; otherwise I can just pull your git 
tree and rebase myself.

  -George


>
> Thanks and Regards,
> Dario
>
>> So, this series introduces the concept of per-vcpu NUMA node-affinity. In fact,
>> up to now, node-affinity has only been "per-domain". That means it was the
>> domain that had a node-affinity and:
>>   - that node-affinity was used to decide where to allocate the memory for the
>>     domain;
>>   - that node-affinity was used to decide on what nodes _all_ the vcpus of the
>>     domain prefer to be scheduled.
>>
>> After this series this changes like this:
>>   - each vcpu of a domain has (well, may have) its own node-affinity, and that
>>     is what is used to determine (if the credit1 scheduler is used) where each
>>     specific vcpu prefers to run;
>>   - the node-affinity of the whole domain is the _union_ of all the
>>     node-affinities of the domain's vcpus;
>>   - the memory is still allocated following what the node-affinity of the whole
>>     domain (so, the union of vcpu node-affinities, as said above) says.
>>
>> In practise, it's not such a big change, I'm just extending at the per-vcpu
>> level what we already had at the domain level. This is also making
>> node-affinity a lot more similar to vcpu-pinning, both in terms of functioning
>> and user interface. As a side efect, that simplify the scheduling code (at
>> least the NUMA-aware part) by quite a bit. Finally, and most important, this is
>> something that will become really important when we will start to support
>> virtual NUMA topologies, as, a that point, having the same node-affinity for
>> all the vcpus in a domain won't be enough any longer (we'll want the vcpus from
>> a particular vnode to have node-afinity with a particular pnode).
>>
>> More detailed description of the mechanism and of the implementation choices
>> are provided in the changelogs and in the documentation (docs/misc and
>> manpages).
>>
>> One last thing is that this series relies on some other patches and series that
>> I sent on xen-devel already, but have not been applied yet.  I'm re-sending
>> them here, as a part of this series, so feel free to pick them up from here, if
>> wanting to apply them, or comment on them in this thread, if you want me to
>> change them.  In particular, patches 01 and 03, I already sent as single
>> patches, patches 04-07, I already sent them as a series. Sorry if that is a bit
>> clumsy, but I couldn't find a better way to do it. :-)
>>
>> In the detailed list of patches below, 'x' means previously submitted, '*'
>> means already acked/reviewed-by.
>>
>> Finally, Elena, that is not super important, but perhaps, in the next release
>> of your vNUMA series, you could try to integrate it with this (and of course,
>> ask if you need anything while trying to do that).
>>
>> Matt, if/when you eventually get to release, even as RFC or something like
>> that, your HVM vNUMA series, we can try to figure out how to integrate that
>> with this, so to use node-affinity instead than pinning.
>>
>> The series is also available at the following git coordinates:
>>
>>   git://xenbits.xen.org/people/dariof/xen.git numa/per-vcpu-affinity-v1
>>   http://xenbits.xen.org/gitweb/?p=people/dariof/xen.git;a=shortlog;h=refs/heads/numa/per-vcpu-affinity-v1
>>
>> Let me know what you think about all this.
>>
>> Regards,
>> Dario
>>
>> PS. Someone of you probably received part of this series as a direct message
>> (i.e., with your address in 'To', rather than in 'Cc'). I'm sincerely sorry for
>> that, messed up with `stg mail'... Won't happen again, I promise! :-P
>>
>> ---
>>
>> Dario Faggioli (15):
>>   x *  xl: update the manpage about "cpus=" and NUMA node-affinity
>>        xl: fix a typo in main_vcpulist()
>>   x *  xen: numa-sched: leave node-affinity alone if not in "auto" mode
>>   x *  libxl: introduce libxl_node_to_cpumap
>>   x    xl: allow for node-wise specification of vcpu pinning
>>   x *  xl: implement and enable dryrun mode for `xl vcpu-pin'
>>   x    xl: test script for the cpumap parser (for vCPU pinning)
>>        xen: numa-sched: make space for per-vcpu node-affinity
>>        xen: numa-sched: domain node-affinity always comes from vcpu node-affinity
>>        xen: numa-sched: use per-vcpu node-affinity for actual scheduling
>>        xen: numa-sched: enable getting/specifying per-vcpu node-affinity
>>        libxc: numa-sched: enable getting/specifying per-vcpu node-affinity
>>        libxl: numa-sched: enable getting/specifying per-vcpu node-affinity
>>        xl: numa-sched: enable getting/specifying per-vcpu node-affinity
>>        xl: numa-sched: enable specifying node-affinity in VM config file
>>
>>
>>   docs/man/xl.cfg.pod.5                           |   88 ++++
>>   docs/man/xl.pod.1                               |   25 +
>>   docs/misc/xl-numa-placement.markdown            |  124 ++++--
>>   tools/libxc/xc_domain.c                         |   90 ++++-
>>   tools/libxc/xenctrl.h                           |   19 +
>>   tools/libxl/check-xl-vcpupin-parse              |  294 +++++++++++++++
>>   tools/libxl/check-xl-vcpupin-parse.data-example |   53 +++
>>   tools/libxl/libxl.c                             |   28 +
>>   tools/libxl/libxl.h                             |   11 +
>>   tools/libxl/libxl_dom.c                         |   18 +
>>   tools/libxl/libxl_numa.c                        |   14 -
>>   tools/libxl/libxl_types.idl                     |    1
>>   tools/libxl/libxl_utils.c                       |   22 +
>>   tools/libxl/libxl_utils.h                       |   15 +
>>   tools/libxl/xl.h                                |    1
>>   tools/libxl/xl_cmdimpl.c                        |  458 +++++++++++++++++++----
>>   tools/libxl/xl_cmdtable.c                       |   11 -
>>   xen/common/domain.c                             |   97 ++---
>>   xen/common/domctl.c                             |   47 ++
>>   xen/common/keyhandler.c                         |    6
>>   xen/common/sched_credit.c                       |   63 ---
>>   xen/common/schedule.c                           |   55 +++
>>   xen/include/public/domctl.h                     |    8
>>   xen/include/xen/sched-if.h                      |    2
>>   xen/include/xen/sched.h                         |   13 +
>>   xen/xsm/flask/hooks.c                           |    2
>>   26 files changed, 1282 insertions(+), 283 deletions(-)
>>   create mode 100755 tools/libxl/check-xl-vcpupin-parse
>>   create mode 100644 tools/libxl/check-xl-vcpupin-parse.data-example
>>
>
>