xen-devel.lists.xenproject.org archive mirror
 help / color / mirror / Atom feed
* [DOC RFC] Heterogeneous Multi Processing Support in Xen
@ 2016-12-07 18:29 Dario Faggioli
  2016-12-08  6:12 ` Juergen Gross
                   ` (3 more replies)
  0 siblings, 4 replies; 22+ messages in thread
From: Dario Faggioli @ 2016-12-07 18:29 UTC (permalink / raw)
  To: Xen Devel
  Cc: Jürgen Groß, Peng Fan, Stefano Stabellini,
	George Dunlap, Andrew Cooper, Dario Faggioli, anastassios.nanos,
	Jan Beulich, Peng Fan


[-- Attachment #1.1: Type: text/plain, Size: 16542 bytes --]

% Heterogeneous Multi Processing Support in Xen
% Revision 1

\clearpage

# Basics

---------------- ------------------------
         Status: **Design Document**

Architecture(s): x86, arm

   Component(s): Hypervisor and toolstack
---------------- ------------------------

# Overview

HMP (Heterogeneous Multi Processing) and AMP (Asymmetric Multi Processing)
refer to systems where physical CPUs are not exactly equal. It may be that
they have different processing power, or capabilities, or that each is
specifically designed to run a particular system component.
Most of the times the CPUs have different Instruction Set Architectures (ISA)
or Application Binary Interfaces (ABIs). But they may *just* be different
implementations of the same ISA, in which case they typically differ in
speed, power efficiency or handling of special things (e.g., erratas).

An example is ARM big.LITTLE, which in fact, is the use case that got the
discussion about HMP started. This document, however, is generic, and does
not target only big.LITTLE.

What need proper Xen support are systems and use cases where virtual CPUs
can not be seamlessly moved around all the physical CPUs. In fact, in these
cases, there must be a way to:

* decide and specify on what (set of) physical CPU(s), each vCPU can execute on;
* enforce that a vCPU that can only run on a certain (set of) pCPUs, is never
  actually run anywhere else.

**N.B.:** it is becoming common to refer as AMP or HMP also to systems which
have various kind of co-processors (from crypto engines to graphic hardware),
integrated with the CPUs on the same chip. This is not what this design 
document is about.

# Classes of CPUs

A *class of CPUs* is defined as follows:

1. each pCPU in the system belongs to a class;
2. a class can consist of one or more pCPUs;
3. each pCPU can only be in one class;
4. CPUs belonging to the same class are homogeneous enough that a virtual
   CPU that blocks/is preempted while running on a pCPU of a class can,
   **seamlessly**, unblock/be scheduler on any pCPU of that same class;
5. when a virtual CPU is associated with a (set of) class(es) of CPUs, it
   means that the vCPU can run on all the pCPUs belonging to the said
   class(es).

So, for instance, in architecture Foobar two classes of CPUs exist, class
foo and class bar. If a virtual CPU running on a CPU 0, which is of class
foo, blocks (or is preempted), it can, when it unblocks (or is selected by
the scheduler to run  again), run on CPU 3, still of class foo, but not on
CPU 6, which is of class bar.

## Defining classes

How a class is defined, i.e., what are the specific characteristics that
determine what CPUs belong to which class, is highly architecture specific.

### x86

There is no HMP platform of relevance, for now, in x86 world. Therefore,
only one class will exist, and all the CPUs will be set to belong to it.
**TODO X86:** is this correct?

### ARM

**TODO ARM:** I know nothing about what specifically should be used to
form classes, so I'm deferring this to ARM people.

So far, in the original thread the following ideas came up (well, there's
more, but I don't know enough of ARM to judge what is really relevant about
this topic):

* [Julien](https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg02153.html)
  "I don't think an hardcoded list of processor in Xen is the right solution.
   There are many existing processors and combinations for big.LITTLE so it
   will nearly be impossible to keep updated."
* [Julien](https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg02256.html)
  "Well, before trying to do something clever like that (i.e naming "big" and
  "little"), we need to have upstreamed bindings available to acknowledge the
  difference. AFAICT, it is not yet upstreamed for Device Tree and I don't
  know any static ACPI tables providing the similar information."
* [Peng](https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg02194.html)
  "For how to differentiate cpus, I am looking the linaro eas cpu topology code"

# User details

## Classes of CPUs for the users

It will be possible, in a VM config file, to specify the (set of) class(es)
of each vCPU. This allows creating HMP VMs.

E.g., on ARM, it will be possible to create big.LITTLE VMs which, if run on
big.LITTLE hosts, could leverage the big.LITTLE support of the guest OS kernel
and tools.

For such purpose, a new option will be added to xl config file:

    vcpus = "8"
    vcpuclass = ["0-2:class0", "3,4:class1,class3", "5:class0, class2", "8:class4"]

with the following meaning:

* vCPUs 0, 1, 2 can only run on pcpus of class class0
* vCPUs 3, 4 can run on pcpus of class class1 **and** on pcpus of class class3
* vCPUs 5 can run on pcpus of class class0 **and** on pCPUs of class class2
* for vCPUs 7, since they're not mentioned, default applies
* vCPUs 8 can only run on pcpus of class class4

For the vCPUs for which no class is specified, default behavior applies.

**TODO:** note that I think it must be possible to associate more than
one class to a vCPU. This is expressed in the example above, and assumed
to be true throughout the document. It might be, though, that, at least at
early stages (see implementation phases below), we will enable only 1-to-1
mapping.

**TODO:** default can be, either:

1. the vCPU can run on any CPU of any class,
2. the vCPU can only run on a specific, arbitrary decided, class (and I'd say
   that should be class 0).

The former seems a better interface. It looks to me like the most natural
and less surprising, from the user point of view, and the most future proof
(see phase 3 of implementation below).
The latter may be more practical, though. In fact, with the former, we risk
crashing (the guest or the hypervisor) if one creates a VM and forgets to
specify the vCPU classes --which does not look ideal.

It will be possible to gather information about what classes exist, and what
pCPUs belong to each class, by issuing the `xl info -n' command:

    cpu_topology           :
    cpu:    core    socket     node     class
      0:       0        1        0        0
      1:       0        1        0        1
      2:       1        1        0        2
      3:       1        1        0        3
      4:       9        1        0        3
      5:       9        1        0        0
      6:      10        1        0        1
      7:      10        1        0        2
      8:       0        0        1        3
      9:       0        0        1        3
     10:       1        0        1        1
     11:       1        0        1        0
     12:       9        0        1        1
     13:       9        0        1        0
     14:      10        0        1        2
     15:      10        0        1        2

**TODO:** do we want to keep using `-n`, or add another switch, like -c or
something? I'm not sure I like using `-n` as, e.g., on x86, this would most
of the times result in just a column full of `0`, and it may raise confusion
among users about what that actually means.
Also, do we want to print the class ids, or some more abstract class names?
(or support both, and have a way to decide which one to see)? 

# Technical details

## Hypervisor

The hypervisor needs to know within which class each of the present CPUs
falls. At boot (or, in general, CPU bringup) time, while identifying the CPU,
a list of classes is constructed, and the mapping between each CPU and the
class it is determined it should belong, established.

The list of classes is kept ordered from the more powerful to the less
powerful.
**TODO:** this has been [proposed by George](https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg02212.html).
I like the idea, what do others think? If we agree on that, note that there
has been no discussion on defining what "more powerful" means, neither on
x86 (although, not really that interesting, for now, I'd say), nor on ARM.

The mapping between CPUs and classes will be kept in memory in the following
data structures:

    uint16_t cpu_to_class[NR_CPUS] __read_mostly;
    cpumask_t class_to_cpumask[NR_CPUS] __read_mostly;

**TODO:** it's probably better to allocate the cpumask array dynamically,
to avoid wasting too much space.

**TODO:** if we want the ordering, structure needs to be kept ordered too
(or additional structures should be used for the purpose).

Each virtual CPU must know on what class of CPUs it can run on. Since a
vCPU can be associated to more than one class, the best way to keep track
of this information is a bitamp. That will be a new `cpumask` typed member
in `struct vcpu`. were the i-eth bit set means the vCPU can
run on CPUs of class i.

If a vCPU is found running on a pCPU of a class that is not associated to
the vCPU itself, an exception should be raised.
**TODO:** What kind? BUG_ON? Crash the guest? The guest would probably crash
--or become unreliable-- by its own, I guess.

Setting and getting the CPU class of a vCPU will happen via two new
hypercalls:

* `XEN_DOMCTL_setvcpuclass`
* `XEN_DOMCTL_setvcpuclass`

Information about CPU classes will be propagated to toolstak by adding a
new field in xen_sysctl_cputopo, which will become:

    struct xen_sysctl_cputopo {
        uint32_t core;
        uint32_t socket;
        uint32_t node;
        unit32_t class;
    };

For homogeneous and SMP systems, the value of the new class field will
be 0 for all the cores.

## Toolstack

It will be possible for the toolstack to retrieve from Xen the list of
existing CPU classes, their names, and the information about to which
class each present CPU belongs to.

**TODO:** [George suggested](https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg02212.html)
to allow a richer set of labels, at the toolstack level, and I like
the idea very much. It's not clear to me, though, in what component
this list of names, and the mapping between them and the classes as
they're known inside Xen should live.

Libxl and libxc interfaces will be introduced for associating a vCPU to
a (set of) class(es):

* `libxl_set_vcpuclass()`, `libxl_get_vcpuclass()`;
* `xc_vcpu_setclass()`, `xc_vcpu_getclass()`.

In libxl, class information will be added in `struct libxl_cputopology`,
which is filled by `libxl_get_cpu_topology()`.

# Implementation

Implementation can proceed in phases.

## Phase 1

Class definition, identification and mapping of CPUs to classes, inside
Xen, will be implemented. And so they will be libxc and libxl interfaces
for retrieving such information.

Parsing of the new `vcpuclass` parameter will be implemented in `xl`. The
result of such parsing will then be used as if it were the hard-affinity of
the various vCPUs. That is, we will set the hard-affinity of each vCPU, to
the pCPUs that are part of the class(es) the vCPU itself is being assigned,
according to `vcpuclass`.

This would *Just Work(TM)*, as soon as the user does not try to change the
hard-affinity, during the VM lifetime (e.g., with `xl vcpu-pin').

**TODO:** It may be useful, for avoiding the above to happen, to add another
`xl` config option that, if set, disallows changing the affinity from what it
was at VM creation time (something like `immutable_affinity=1`). Thoughts?
I'm leaning toward doing that, as it may even be something useful to have
in other usecases.

### Phase 1.5

Library (libxc and libxl) calls and hypercalls that are necessary to associate
a class to the vCPUs will be implemented.

At which point, when parsing `vcpuclass` in `xl`, we will call both (with the
same bitmap as input):

* `libxl_set_vcpuclass()`
* `libxl_set_vcpuaffinity()`

`libxl__set_vcpuaffinity()` will be modified in such a way that, when setting
hard-affinity for a vCPU:

* it will get the CPU class(es) associated to the vCPU;
* it will check what pCPUs that belong to the class(es);
* it will filter out, from the new hard-affinity being set, the pCPUs that
   are not in the vCPU's class(es)'.

As a safety measure, `vcpu_set_hard_affinity()` in Xen will also be modified
such that, if someone somehow manages to pass down an hard-affinity mask
which contains pCPUs outside from the proper classes, it will error out
with -EINVAL.

### Phase 2

Inside Xen, the various schedulers will be modified to deal internally with
the fact that vCPUs can only run on pCPUs from the class(es) they are
associated with. This allows for more efficient implementation, and paves
the way for enabling more intelligent logic (e.g., for minimizing power
consumption) in *phase 3*.

Calling `libxl_set_vcpuaffinity()` from `xl` / libxl is therefore no longer
necessary and will be avoided (i.e., only `libxl_set_vcpuclass()` will be
called).

### Phase 3

Moving vCPUs between classes will be implemented. This means that, e.g.,
on ARM big.LITTLE, it will be possible for a vCPU to block on a big core
and wakeup on a LITTLE core.

**TODO:** About what this takes, see [Julien's email](https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg02345.html).

This means it will be no longer necessary to specify the class of the
vCPUs via `vcpuclass` in `xl`, although that will of course remain
supported. So:

1. if one wants (sticking with big.LITTLE as example) a big.LITTLE VM,
   and wants to make sure that make sure that big vCPUs will run on big
   pCPUs, and that LITTLE vCPUs will run on LITTLE pCPUs, she will use:

    vcpus = "8"
    vcpuclass = ["0-3:big", "4-7:little"]

2. if one does not care, and is happy to let the Xen scheduler decide
   where to run the various vCPUs, in order, for instance, to be sure
   to get the best power efficiency for the host as a whole, he can
   just avoid specifying any `vcpuclass`, or doing something like this:

    vcpuclass = ["all:all"]

# Limitations

* Until in *phase 1*, it won't be possible to use vCPU hard-affinity
  for anything else than HMP support;
* until before *phase 3*, since HMP support is basically the same as
  setting hard-affinity, performance may not be ideal;
* until before *phase 3*, vCPUs can't move between classes. This means.
  for instance, in the big.LITTLE world, Xen's scheduler can't move a
  vCPU running on a big core on a LITTLE core (e.g., to try save power).

# Testing

Testing requires an actual AMP/HMP system. On such a system, we at least
want to:

* create a VM **without** specifying `vcpuclass` in its config file, and
  check that the default policy is correctly applied to all vCPUs;
* create a VM **specifying** `vcpuclass` in its config file and check that
  the classes are assegned to vCPUs appropriately;
* create a VM **specifying** `vcpuclass` in its config file and check that
  the various vCPUs are not running on any pCPU outside of their respective
  classes.

# Areas for improvement

* Make it possible to test even on non-HMP systems. That could be done by
  making it possible to provide Xen with fake CPU classes for the system
  CPUs (e.g., with boot time parameters);
* implement a way to view the class the vCPUs have been assigned (either as
  past of the output of `xl vcpu-list`, or as a dedicated `xl` subcommand);
* make it possible to dynamically change the class of vCPUs at runtime, with
  `xl` (either via a new parameter to `vcpu-pin` subcommand, or via a new
  subcommand).

# Known issues

*TBD*.

# References

* [Asymetric Multi Processing](https://en.wikipedia.org/wiki/Asymmetric_multiprocessing)
* [Heterogeneous Multi Processing](https://en.wikipedia.org/wiki/Heterogeneous_computing)
* [ARM big.LITTLE](https://www.arm.com/products/processors/technologies/biglittleprocessing.php)

# History

------------------------------------------------------------------------
Date       Revision Version  Notes
---------- -------- -------- -------------------------------------------
2016-12-02 1                 RFC of design document
---------- -------- -------- -------------------------------------------

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 22+ messages in thread
end of thread, other threads:[~2017-03-01 18:58 UTC | newest]

Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-12-07 18:29 [DOC RFC] Heterogeneous Multi Processing Support in Xen Dario Faggioli
2016-12-08  6:12 ` Juergen Gross
2016-12-08 10:27   ` Dario Faggioli
2016-12-08 10:38     ` Juergen Gross
2016-12-08 21:45       ` Dario Faggioli
2016-12-15 18:41       ` Dario Faggioli
2016-12-16  7:44         ` Juergen Gross
2016-12-08 10:14 ` Jan Beulich
2016-12-08 10:23   ` Dario Faggioli
2016-12-08 10:41     ` Jan Beulich
2016-12-08 19:09   ` Stefano Stabellini
2016-12-08 21:54   ` Dario Faggioli
2016-12-09  8:13     ` Jan Beulich
2016-12-09  8:29       ` Dario Faggioli
2016-12-09  9:09         ` Jan Beulich
2016-12-09 19:20           ` Stefano Stabellini
2016-12-16  8:00           ` George Dunlap
2016-12-16  8:05 ` George Dunlap
2016-12-16  8:07   ` George Dunlap
2017-03-01  0:05 ` Anastassios Nanos
2017-03-01 17:38   ` Dario Faggioli
2017-03-01 18:58     ` Stefano Stabellini

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).