Re: [Qemu-devel] CPU hotplug

[Christian Borntraeger <borntraeger@de.ibm.com>]

On 02/01/2016 06:35 AM, David Gibson wrote:
> Hi,
> 
> It seems to me we're getting rather bogged down in how to proceed with
> an improved CPU hotplug (and hot unplug) interface, both generically
> and for ppc in particular.

Yes, s390 also needs this. 
Can you add Matthew in any cpu hotplug discussion?


> 
> So here's a somewhat more concrete suggestion of a way forward, to see
> if we can get some consensus.
> 
> The biggest difficulty I think we're grappling with is that device-add
> is actually *not* a great interface to cpu hotplug.  Or rather, it's
> not great as the _only_ interface: in order to represent the many
> different constraints on how cpus can be plugged on various platforms,
> it's natural to use a heirarchy of cpu core / socket / package types
> specific to the specific platform or real-world cpu package being
> modeled.  However, for the normal case of a regular homogenous (and at
> least slightly para-virtualized) server, that interface is nasty for
> management layers because they have to know the right type to
> instantiate.
> 
> To address this, I'm proposing this two layer interface:
> 
> Layer 1: Low-level, device-add based
> 
>     * a new, generic cpu-package QOM type represents a group of 1 or
>       more cpu threads which can be hotplugged as a unit
>     * cpu-package is abstract and can't be instantiated directly
>     * archs and/or individual platforms have specific subtypes of
>       cpu-package which can be instantiated
>     * for platforms attempting to be faithful representations of real
>       hardware these subtypes would match the specific characteristics
>       of the real hardware devices.  In addition to the cpu threads,
>       they may have other on chip devices as sub-objects.
>     * for platforms which are paravirtual - or which have existing
>       firmware abstractions for cpu cores/sockets/packages/whatever -
>       these could be more abstract, but would still be tied to that
>       platform's constraints
>     * Depending on the platform the cpu-package object could have
>       further internal structure (e.g. a package object representing a
>       socket contains package objects representing each core, which in
>       turn contain cpu objects for each thread)
>         * Some crazy platform that has multiple daughterboards each with
>           several multi-chip-modules each with several chips, each
> 	  with several cores each with several threads could represent
> 	  that too.
> 
> What would be common to all the cpu-package subtypes is:
>     * A boolean "present" attribute ("realized" might already be
>       suitable, but I'm not certain)
>     * A generic means of determining the number of cpu threads in the
>       package, and enumerating those
>     * A generic means of determining if the package is hotpluggable or
>       not
>     * They'd get listed in a standard place in the QOM tree
> 
> This interface is suitable if you want complete control over
> constructing the system, including weird cases like heterogeneous
> machines (either totally different cpu types, or just different
> numbers of threads in different packages).
> 
> The intention is that these objects would never look at the global cpu
> type or sockets/cores/threads numbers.  The next level up would
> instead configure the packages to match those for the common case.
> 
> Layer 2: Higher-level
> 
>     * not all machine types need support this model, but I'd expect
>       all future versions of machine types designed for production use
>       to do so
>     * machine types don't construct cpu objects directly
>     * instead they create enough cpu-package objects - of a subtype
>       suitable for this machine - to provide maxcpus threads
>     * the machine type would set the "present" bit on enough of the
>       cpu packages to provide the base number of cpu threads
> 
> Management layers can then manage hotplug without knowing platform
> specifics by using qmp to toggle the "present" bit on packages.
> Platforms that allow thread-level pluggability can expose a package
> for every thread, those that allow core-level expose a package per
> core, those that have even less granularity expose a package at
> whatever grouping they can do hotplug on.
> 
> Examples:
> 
> For use with pc (or q35 or whatever) machine type, I'd expect a
> cpu-package subtype called, say "acpi-thread" which represents a
> single thread in the ACPI sense.  Toggling those would trigger ACPI
> hotplug events as cpu_add does now.
> 
> For use with pseries, I'd expect a "papr-core" cpu-package subtype,
> which represents a single (paravirtual) core.  Toggling present on
> this would trigger the PAPR hotplug events.  A property would control
> the number of threads in the core (only settable before enabling
> present).
> 
> For use with the powernv machine type (once ready for merge) I'd
> expect "POWER8-package" type which represents a POWER8 chip / module
> as close to the real hardware as we can get.  It would have a fixed
> number of cores and threads within it as per the real hardware, and
> would also include xscoms and other per-module logic.
> 
> From here to there:
> 
> A suggested order of implementation to get there without too much risk
> of breaking things.
> 
>   1. Fix bugs with creation / removal of CPU objects (Bharata's cpu
>      hotplug series already has this)
>   2. Split creation and realization of CPU objects, so machine types
>      must explicitly perform both steps (Bharata's series has this
>      too)
>   3. Add the abstract cpu-package type, and define the generic
>      interfaces it needs (Bharata's series has something that could be
>      changed to this fairly easily)
>   4. For each machine type we care to convert:
>       4.1. Add platform suitable cpu-package subtypes
>       4.2. Convert the (latest version) machine type to instantiate packages instead of
>            cpu threads directly
>       4.3. Add any necessary backwards compat goo
>   5. Teach libvirt how to toggle cpu-packages
>