Minutes for KVM call 2013-01-15

Minutes for KVM call 2013-01-15

[Juan Quintela]

* cpu hot plug
  - use qdev propierties conected to a set of socket objects (anthony)
  - cpusets are the wrong interface (anthony)
  - make a link between cpu <-> socket instead of a propierty?
  - how far are we from being able to describe a cpu with -device?
    (didn't heare the answer, andreas?)
  - perhaps the best approach?
  - After soft-freeze, exceptions depend on the maintainer
  - After hard-freeze, no exceptions
  -device don't require a bus, just an implementation detail, we can change that
  - use cpuset as an intermediate step until full vision is implemented
  - several approaches from where we are now, to have something before
    we get a full solution


At this point, Andreas agreed to write a better summary of the
discussion and suggestions O:-)

Later, Juan.

vCPU hotplug roadmap (was: Minutes for KVM call 2013-01-15)

[Andreas Färber]

Am 15.01.2013 17:16, schrieb Juan Quintela:
> 
> * cpu hot plug
>   - use qdev propierties conected to a set of socket objects (anthony)
>   - cpusets are the wrong interface (anthony)
>   - make a link between cpu <-> socket instead of a propierty?
>   - how far are we from being able to describe a cpu with -device?
>     (didn't heare the answer, andreas?)
>   - perhaps the best approach?
>   - After soft-freeze, exceptions depend on the maintainer
>   - After hard-freeze, no exceptions
>   -device don't require a bus, just an implementation detail, we can change that
>   - use cpuset as an intermediate step until full vision is implemented
>   - several approaches from where we are now, to have something before
>     we get a full solution
> 
> 
> At this point, Andreas agreed to write a better summary of the
> discussion and suggestions O:-)

Got buried, here we go:

== vCPU hot-plug user interfaces ==

=== cpu_set ===

Previously available in qemu-kvm.git:
`cpu_set <n+1> online` via HMP

Pros:
* Hides QOM/qdev implementation details (afaerber)
* Thus: Doesn't depend on QOM CPUState refactoring (imammedo)
* Opens a fast route to implementing vCPU unplug in KVM (imammedo)
* Unintrusive to add and easy to obsolete/remove in future (imammedo)
* Existing virt-test cases (afaerber)
* Supported by libvirt (imammedo)
* Prevents confusing guests by hot-plugging random mix of CPUs (agraf)

Cons:
* Cannot express topologies (ehabkost)

=== device_add ===

`device_add driver=Haswell-x86_64-cpu id=qdevid`
[You can try this today and see it failing / not working.]

Pros:
* QMP/HMP command available today and known to users (afaerber)
* Unified command for device and CPU hot-plug (imammedo)
* Would allow first doing thread-level vCPU hotplug (imammedo)
* Could be extended to support socket-level hot-plug (aliguori/imammedo)

Cons:
* Operates on raw QOM type name unlike -cpu (afaerber)
* Needs support in libvirt for device_add driver=<CPU> (imammedo)
* libvirt needs means to enumerate CPU types (imammedo) => QMP? (AF)

Challenges:
* No CPU qbus (afaerber)
  => "should work" without (aliguori)
* CPU subclasses needed for identifying type name (afaerber/imammedo)
  => "Haswell-x86_64-cpu" does not exist yet, just "x86_64-cpu"
* CPU class_init for -cpu host requires KVM init (imammedo)
  [suggestion by ehabkost to use kvm_arch_vcpu_init, WIP by afaerber]
* Conversion of CPU features to static properties needed (imammedo)
  => device_add driver=foo,level=x,xlevel=y,...
* Alternatively conversion to global properties (imammedo)
* Cements type names - rename for 1.4? (afaerber) => permissable (alig.)
  [patches for arm, m68k, openrisc, unicore32 on list]

=== qom-set ===

`qom-set` via QMP w/ link<CPUSocket> property (aliguori)

Topology represented in QOM:
CPUSocket has-a    CPUCore has-a    CPUThread a.k.a. CPUState, or
CPUSocket links-to CPUCore links-to CPUThread a.k.a. CPUState

Challenges (afaerber):
* No CPUSocket/CPUCore objects yet and may take a while to get there...
  topology fields being moved to CPUState for 1.4 [done, more WIP]
* No decisions on canonical paths for CPUs: CPU? machine? unassigned?
* Duality of thread-level device types and socket-level? (afaerber)
  => fine to have, e.g., quad-core Xeon 500 device (aliguori)
* CPUState is no_user (afaerber)
  => need to generally drop no_user for QOM (aliguori)

=== libvirt ===

libvirt's XML topology modelling is closer to today's -smp than to the
desired QOM modelling:
http://www.libvirt.org/formatcaps.html

`virsh setvcpus <domain> <n>`
http://libvirt.org/sources/virshcmdref/html/sect-setvcpus.html

== qom-cpu course of action (afaerber) ==

It was requested to have vCPU hot-plug in v1.5.

For device_add we need to move code from cpu_init() into QOM facilities.
=> QOM realize support would help [applied by aliguori]
=> cleanups piggy-backed onto CPU realizefn [applied to qom-cpu-next]

Agreement on goal of X86CPU subclasses, but conflicts how to get there:
* Refactor x86_def_t to X86CPUInfo for X86CPUClass class_init? (AF 2012)
* Refactor x86_def_t to X86CPU instance_init as done for arm?
* Refactor x86_def_t to class_inits? (afaerber)
  -> heavy merge conflicts due to bug fixes / cleanups
  Pro: We can get things into a consistent QOM'ish state across targets.
  Con: We will refactor again on top for machine-compat properties.
* Keep x86_def_t within X86CPUClass as done for ppc? (WIP: afaerber)
  => smallest common denominator, separates x86 from cross-target work

APIC ID topology fixes are being reviewed for 1.4. [merged]
X86CPU wave 4 cleanups by Igor are being reviewed for 1.4. [merged]

Rename CPU types according to unified <name>-<arch>-cpu scheme for 1.4?
(aliguori: permissable) [patches on list]

VMState series by Juan being rebased - subset for 1.4, rest for 1.5.
[1.4 part on list, WIP for 1.5]

Remainder is considered 1.5 material, qom-cpu-next avail. during Freeze.

== Common issues (imammedo) ==

- back-port CPU hot-plug ACPI notification
- hot-plug is not allowed on SysBus:
  - APIC that is created by CPU on hot-plug is a SysBus device,
    so essentially APIC is hot-plugged to SysBus anyway which causes
    abort in qdev_try_create() -> qdev_set_parent_bus(SysBus) ->
    bus_add_child() [no hot-plug on SysBus], which is hard to fix
    without allowing hot-plug on SysBus.
    [AF: change the APIC to not be a SysBusDevice (like CPUState) then?]
  - qdev_device_add() and qdev_try_create() assign SysBus by default to
    device if no explicit bus was provided.
    - qbus_find_recursive() in qdev_device_add() always returns SysBus
      if last 2 args are NULL, which is case with busless CPU
    - qdev_try_create() adds device SysBus if no bus was provided in
      1st arg
- vapic is not working with hot-plug in current master
  (do not know why yet)

Experimental CPU hot-add tree using device_add implemented only for
qemu64 CPU:
https://github.com/imammedo/qemu/tree/x86-cpu-hot-add.WIP

---
That completes Igor's and my notes, amended by status updates of mine.
Sorry for the delay, I did try to address some of the discussed issues
in form of patches in the meantime though. ;)

CC'ing s390x, ppc and arm KVM folks that were not on the call for
feedback on their requirements and thoughts.

Regards,
Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: vCPU hotplug roadmap (was: Minutes for KVM call 2013-01-15)

[Eduardo Habkost]

On Wed, Jan 30, 2013 at 11:58:56AM +0100, Andreas Färber wrote:
> Am 15.01.2013 17:16, schrieb Juan Quintela:
> > 
> > * cpu hot plug
> >   - use qdev propierties conected to a set of socket objects (anthony)
> >   - cpusets are the wrong interface (anthony)
> >   - make a link between cpu <-> socket instead of a propierty?
> >   - how far are we from being able to describe a cpu with -device?
> >     (didn't heare the answer, andreas?)
> >   - perhaps the best approach?
> >   - After soft-freeze, exceptions depend on the maintainer
> >   - After hard-freeze, no exceptions
> >   -device don't require a bus, just an implementation detail, we can change that
> >   - use cpuset as an intermediate step until full vision is implemented
> >   - several approaches from where we are now, to have something before
> >     we get a full solution
> > 
> > 
> > At this point, Andreas agreed to write a better summary of the
> > discussion and suggestions O:-)
> 
> Got buried, here we go:
> 
> == vCPU hot-plug user interfaces ==
> 
> === cpu_set ===
> 
> Previously available in qemu-kvm.git:
> `cpu_set <n+1> online` via HMP
> 
> Pros:
> * Hides QOM/qdev implementation details (afaerber)
> * Thus: Doesn't depend on QOM CPUState refactoring (imammedo)
> * Opens a fast route to implementing vCPU unplug in KVM (imammedo)
> * Unintrusive to add and easy to obsolete/remove in future (imammedo)
> * Existing virt-test cases (afaerber)
> * Supported by libvirt (imammedo)
> * Prevents confusing guests by hot-plugging random mix of CPUs (agraf)
> 
> Cons:
> * Cannot express topologies (ehabkost)

Actually, I believe this is not the main problem (we will have exactly
the same limitation if using thread-level device_add). To me, the main
problem is that we are creating a new QMP command that should be
eventually obsoleted by device_add.


> 
> === device_add ===
> 
> `device_add driver=Haswell-x86_64-cpu id=qdevid`
> [You can try this today and see it failing / not working.]
> 
> Pros:
> * QMP/HMP command available today and known to users (afaerber)
> * Unified command for device and CPU hot-plug (imammedo)
> * Would allow first doing thread-level vCPU hotplug (imammedo)
> * Could be extended to support socket-level hot-plug (aliguori/imammedo)
> 
> Cons:
> * Operates on raw QOM type name unlike -cpu (afaerber)
> * Needs support in libvirt for device_add driver=<CPU> (imammedo)
> * libvirt needs means to enumerate CPU types (imammedo) => QMP? (AF)
> 
> Challenges:
> * No CPU qbus (afaerber)
>   => "should work" without (aliguori)
> * CPU subclasses needed for identifying type name (afaerber/imammedo)
>   => "Haswell-x86_64-cpu" does not exist yet, just "x86_64-cpu"
> * CPU class_init for -cpu host requires KVM init (imammedo)
>   [suggestion by ehabkost to use kvm_arch_vcpu_init, WIP by afaerber]

I don't know what you mean by "use kvm_arch_vcpu_init()". I sent a RFC
following somebody's suggestion of simply make kvm_arch_init() call a
function to finish the -cpu host initialization, as we can't initialize
everything inside class_init.

See x86_cpu_finish_host_class_init() at:
 Message-Id: <1357329382-20944-7-git-send-email-ehabkost@redhat.com>
 http://article.gmane.org/gmane.comp.emulators.qemu/186778


> * Conversion of CPU features to static properties needed (imammedo)
>   => device_add driver=foo,level=x,xlevel=y,...
> * Alternatively conversion to global properties (imammedo)
> * Cements type names - rename for 1.4? (afaerber) => permissable (alig.)
>   [patches for arm, m68k, openrisc, unicore32 on list]
> 
> === qom-set ===
> 
> `qom-set` via QMP w/ link<CPUSocket> property (aliguori)
> 
> Topology represented in QOM:
> CPUSocket has-a    CPUCore has-a    CPUThread a.k.a. CPUState, or
> CPUSocket links-to CPUCore links-to CPUThread a.k.a. CPUState
> 
> Challenges (afaerber):
> * No CPUSocket/CPUCore objects yet and may take a while to get there...
>   topology fields being moved to CPUState for 1.4 [done, more WIP]
> * No decisions on canonical paths for CPUs: CPU? machine? unassigned?
> * Duality of thread-level device types and socket-level? (afaerber)
>   => fine to have, e.g., quad-core Xeon 500 device (aliguori)
> * CPUState is no_user (afaerber)
>   => need to generally drop no_user for QOM (aliguori)

I would like to drop no_user on 1.5 even if we don't manage to finish
CPU hotplug, as exposing the CPU objects and classes will be very useful
to allow libvirt to probe for the available CPU models and features.


> 
> === libvirt ===
> 
> libvirt's XML topology modelling is closer to today's -smp than to the
> desired QOM modelling:
> http://www.libvirt.org/formatcaps.html
> 
> `virsh setvcpus <domain> <n>`
> http://libvirt.org/sources/virshcmdref/html/sect-setvcpus.html
> 
> == qom-cpu course of action (afaerber) ==
> 
> It was requested to have vCPU hot-plug in v1.5.
> 
> For device_add we need to move code from cpu_init() into QOM facilities.
> => QOM realize support would help [applied by aliguori]
> => cleanups piggy-backed onto CPU realizefn [applied to qom-cpu-next]
> 
> Agreement on goal of X86CPU subclasses, but conflicts how to get there:
> * Refactor x86_def_t to X86CPUInfo for X86CPUClass class_init? (AF 2012)
> * Refactor x86_def_t to X86CPU instance_init as done for arm?
> * Refactor x86_def_t to class_inits? (afaerber)
>   -> heavy merge conflicts due to bug fixes / cleanups
>   Pro: We can get things into a consistent QOM'ish state across targets.
>   Con: We will refactor again on top for machine-compat properties.
> * Keep x86_def_t within X86CPUClass as done for ppc? (WIP: afaerber)
>   => smallest common denominator, separates x86 from cross-target work
> 
> APIC ID topology fixes are being reviewed for 1.4. [merged]
> X86CPU wave 4 cleanups by Igor are being reviewed for 1.4. [merged]
> 
> Rename CPU types according to unified <name>-<arch>-cpu scheme for 1.4?
> (aliguori: permissable) [patches on list]
> 
> VMState series by Juan being rebased - subset for 1.4, rest for 1.5.
> [1.4 part on list, WIP for 1.5]
> 
> Remainder is considered 1.5 material, qom-cpu-next avail. during Freeze.
> 
> == Common issues (imammedo) ==
> 
> - back-port CPU hot-plug ACPI notification
> - hot-plug is not allowed on SysBus:
>   - APIC that is created by CPU on hot-plug is a SysBus device,
>     so essentially APIC is hot-plugged to SysBus anyway which causes
>     abort in qdev_try_create() -> qdev_set_parent_bus(SysBus) ->
>     bus_add_child() [no hot-plug on SysBus], which is hard to fix
>     without allowing hot-plug on SysBus.
>     [AF: change the APIC to not be a SysBusDevice (like CPUState) then?]
>   - qdev_device_add() and qdev_try_create() assign SysBus by default to
>     device if no explicit bus was provided.
>     - qbus_find_recursive() in qdev_device_add() always returns SysBus
>       if last 2 args are NULL, which is case with busless CPU
>     - qdev_try_create() adds device SysBus if no bus was provided in
>       1st arg
> - vapic is not working with hot-plug in current master
>   (do not know why yet)
> 
> Experimental CPU hot-add tree using device_add implemented only for
> qemu64 CPU:
> https://github.com/imammedo/qemu/tree/x86-cpu-hot-add.WIP
> 
> ---
> That completes Igor's and my notes, amended by status updates of mine.
> Sorry for the delay, I did try to address some of the discussed issues
> in form of patches in the meantime though. ;)
> 
> CC'ing s390x, ppc and arm KVM folks that were not on the call for
> feedback on their requirements and thoughts.
> 
> Regards,
> Andreas
> 
> -- 
> SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

-- 
Eduardo

Re: vCPU hotplug roadmap

[Andreas Färber]

Am 30.01.2013 13:49, schrieb Eduardo Habkost:
> On Wed, Jan 30, 2013 at 11:58:56AM +0100, Andreas Färber wrote:
>> * CPU class_init for -cpu host requires KVM init (imammedo)
>>   [suggestion by ehabkost to use kvm_arch_vcpu_init, WIP by afaerber]
> 
> I don't know what you mean by "use kvm_arch_vcpu_init()".

Sorry, scratch the _vcpu. I.e., the x86-specific KVM init hook.

> I sent a RFC
> following somebody's suggestion of simply make kvm_arch_init() call a
> function to finish the -cpu host initialization, as we can't initialize
> everything inside class_init.
> 
> See x86_cpu_finish_host_class_init() at:
>  Message-Id: <1357329382-20944-7-git-send-email-ehabkost@redhat.com>
>  http://article.gmane.org/gmane.comp.emulators.qemu/186778

...and I have been working on making it even simpler for the
still-x86_def_t-based approach. I'm still busy looking at 1.4 issues
currently though.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: vCPU hotplug roadmap

[Igor Mammedov]

On Wed, 30 Jan 2013 14:02:16 +0100
Andreas Färber <afaerber@suse.de> wrote:

> Am 30.01.2013 13:49, schrieb Eduardo Habkost:
> > On Wed, Jan 30, 2013 at 11:58:56AM +0100, Andreas Färber wrote:
[...]
> >  http://article.gmane.org/gmane.comp.emulators.qemu/186778
> 
> ...and I have been working on making it even simpler for the
> still-x86_def_t-based approach. I'm still busy looking at 1.4 issues
> currently though.
> 
> Andreas
> 

I'll try to cook series that would do properties and classes in one seamless
approach without intermediate steps. Perhaps it would work out better.