From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?ISO-8859-15?Q?Andreas_F=E4rber?= Subject: vCPU hotplug roadmap (was: Minutes for KVM call 2013-01-15) Date: Wed, 30 Jan 2013 11:58:56 +0100 Message-ID: <5108FCF0.6080001@suse.de> References: <874niito98.fsf@elfo.mitica> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: Juan Quintela , libvir-list@redhat.com, Igor Mammedov , Christian Borntraeger , qemu-ppc , Benjamin Herrenschmidt , David Gibson , Peter Maydell , Eduardo Habkost , Alexander Graf , Anthony Liguori To: qemu-devel qemu-devel , KVM devel mailing list Return-path: Received: from cantor2.suse.de ([195.135.220.15]:42716 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754192Ab3A3K7L (ORCPT ); Wed, 30 Jan 2013 05:59:11 -0500 In-Reply-To: <874niito98.fsf@elfo.mitica> Sender: kvm-owner@vger.kernel.org List-ID: Am 15.01.2013 17:16, schrieb Juan Quintela: >=20 > * cpu hot plug > - use qdev propierties conected to a set of socket objects (anthony= ) > - cpusets are the wrong interface (anthony) > - make a link between cpu <-> socket instead of a propierty? > - how far are we from being able to describe a cpu with -device? > (didn't heare the answer, andreas?) > - perhaps the best approach? > - After soft-freeze, exceptions depend on the maintainer > - After hard-freeze, no exceptions > -device don't require a bus, just an implementation detail, we can = change that > - use cpuset as an intermediate step until full vision is implement= ed > - several approaches from where we are now, to have something befor= e > we get a full solution >=20 >=20 > At this point, Andreas agreed to write a better summary of the > discussion and suggestions O:-) Got buried, here we go: =3D=3D vCPU hot-plug user interfaces =3D=3D =3D=3D=3D cpu_set =3D=3D=3D Previously available in qemu-kvm.git: `cpu_set online` via HMP Pros: * Hides QOM/qdev implementation details (afaerber) * Thus: Doesn't depend on QOM CPUState refactoring (imammedo) * Opens a fast route to implementing vCPU unplug in KVM (imammedo) * Unintrusive to add and easy to obsolete/remove in future (imammedo) * Existing virt-test cases (afaerber) * Supported by libvirt (imammedo) * Prevents confusing guests by hot-plugging random mix of CPUs (agraf) Cons: * Cannot express topologies (ehabkost) =3D=3D=3D device_add =3D=3D=3D `device_add driver=3DHaswell-x86_64-cpu id=3Dqdevid` [You can try this today and see it failing / not working.] Pros: * QMP/HMP command available today and known to users (afaerber) * Unified command for device and CPU hot-plug (imammedo) * Would allow first doing thread-level vCPU hotplug (imammedo) * Could be extended to support socket-level hot-plug (aliguori/imammedo= ) Cons: * Operates on raw QOM type name unlike -cpu (afaerber) * Needs support in libvirt for device_add driver=3D (imammedo) * libvirt needs means to enumerate CPU types (imammedo) =3D> QMP? (AF) Challenges: * No CPU qbus (afaerber) =3D> "should work" without (aliguori) * CPU subclasses needed for identifying type name (afaerber/imammedo) =3D> "Haswell-x86_64-cpu" does not exist yet, just "x86_64-cpu" * CPU class_init for -cpu host requires KVM init (imammedo) [suggestion by ehabkost to use kvm_arch_vcpu_init, WIP by afaerber] * Conversion of CPU features to static properties needed (imammedo) =3D> device_add driver=3Dfoo,level=3Dx,xlevel=3Dy,... * Alternatively conversion to global properties (imammedo) * Cements type names - rename for 1.4? (afaerber) =3D> permissable (ali= g.) [patches for arm, m68k, openrisc, unicore32 on list] =3D=3D=3D qom-set =3D=3D=3D `qom-set` via QMP w/ link property (aliguori) Topology represented in QOM: CPUSocket has-a CPUCore has-a CPUThread a.k.a. CPUState, or CPUSocket links-to CPUCore links-to CPUThread a.k.a. CPUState Challenges (afaerber): * No CPUSocket/CPUCore objects yet and may take a while to get there... topology fields being moved to CPUState for 1.4 [done, more WIP] * No decisions on canonical paths for CPUs: CPU? machine? unassigned? * Duality of thread-level device types and socket-level? (afaerber) =3D> fine to have, e.g., quad-core Xeon 500 device (aliguori) * CPUState is no_user (afaerber) =3D> need to generally drop no_user for QOM (aliguori) =3D=3D=3D libvirt =3D=3D=3D libvirt's XML topology modelling is closer to today's -smp than to the desired QOM modelling: http://www.libvirt.org/formatcaps.html `virsh setvcpus ` http://libvirt.org/sources/virshcmdref/html/sect-setvcpus.html =3D=3D qom-cpu course of action (afaerber) =3D=3D It was requested to have vCPU hot-plug in v1.5. =46or device_add we need to move code from cpu_init() into QOM faciliti= es. =3D> QOM realize support would help [applied by aliguori] =3D> cleanups piggy-backed onto CPU realizefn [applied to qom-cpu-next] Agreement on goal of X86CPU subclasses, but conflicts how to get there: * Refactor x86_def_t to X86CPUInfo for X86CPUClass class_init? (AF 2012= ) * Refactor x86_def_t to X86CPU instance_init as done for arm? * Refactor x86_def_t to class_inits? (afaerber) -> heavy merge conflicts due to bug fixes / cleanups Pro: We can get things into a consistent QOM'ish state across targets= =2E Con: We will refactor again on top for machine-compat properties. * Keep x86_def_t within X86CPUClass as done for ppc? (WIP: afaerber) =3D> smallest common denominator, separates x86 from cross-target wor= k APIC ID topology fixes are being reviewed for 1.4. [merged] X86CPU wave 4 cleanups by Igor are being reviewed for 1.4. [merged] Rename CPU types according to unified --cpu scheme for 1.4? (aliguori: permissable) [patches on list] VMState series by Juan being rebased - subset for 1.4, rest for 1.5. [1.4 part on list, WIP for 1.5] Remainder is considered 1.5 material, qom-cpu-next avail. during Freeze= =2E =3D=3D Common issues (imammedo) =3D=3D - back-port CPU hot-plug ACPI notification - hot-plug is not allowed on SysBus: - APIC that is created by CPU on hot-plug is a SysBus device, so essentially APIC is hot-plugged to SysBus anyway which causes abort in qdev_try_create() -> qdev_set_parent_bus(SysBus) -> bus_add_child() [no hot-plug on SysBus], which is hard to fix without allowing hot-plug on SysBus. [AF: change the APIC to not be a SysBusDevice (like CPUState) then?= ] - qdev_device_add() and qdev_try_create() assign SysBus by default to device if no explicit bus was provided. - qbus_find_recursive() in qdev_device_add() always returns SysBus if last 2 args are NULL, which is case with busless CPU - qdev_try_create() adds device SysBus if no bus was provided in 1st arg - vapic is not working with hot-plug in current master (do not know why yet) Experimental CPU hot-add tree using device_add implemented only for qemu64 CPU: https://github.com/imammedo/qemu/tree/x86-cpu-hot-add.WIP --- That completes Igor's and my notes, amended by status updates of mine. Sorry for the delay, I did try to address some of the discussed issues in form of patches in the meantime though. ;) CC'ing s390x, ppc and arm KVM folks that were not on the call for feedback on their requirements and thoughts. Regards, Andreas --=20 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 N=FCrnberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imend=F6rffer; HRB 16746 AG N=FCrn= berg