Re: [PATCH] cpu hotplug issue

Gleb Natapov
2011-07-25T13:21:45Z

On Mon, Jul 25, 2011 at 03:18:19PM +0200, Jan Kiszka wrote:
> On 2011-07-24 18:11, Jan Kiszka wrote:
> >>> I had a closer look and identified two further issues, one generic, one
> >>> CPU-hotplug-specific:
> >>> - (qdev) devices that are hotplugged do not receive any reset. That
> >>> does not only apply to the APIC in case of CPU hotplugging, it is
> >>> also broken for NICs, storage controllers, etc. when doing PCI
> >>> hot-add as I just checked via gdb.
> >>> - CPU hotplugging was always (or at least for a fairly long time),
> >>> well, fragile as it failed to make CPU thread creation and CPU
> >>> initialization atomic against APIC addition and other initialization
> >>> steps. IOW, we need to create CPUs stopped, finish all init work,
> >>> sync their states completely to the kernel
> >>> (cpu_synchronize_post_init), and then kick them of. Actually I'm
> >> Syncing the state to the kernel should be done by vcpu thread, so I it
> >> cannot be stopped while the sync is done. May be I misunderstood what
> >> you mean here.
> >
> > Stopped first of all means not entering kvm_cpu_exec before the whole
> > setup and the initial sync are done.
> >
> > Syncing the initial state may also happen over the creating context as
> > long as the vcpus are stopped (analogously to
> > kvm_cpu_synchronize_post_init).
>
> OK, hacks below plus the following three patches make CPU hotplug work
> again - with some exceptions. Here are the patches:
>
> 1. http://thread.gmane.org/gmane.comp.emulators.kvm.devel/76484
> 2. http://thread.gmane.org/gmane.comp.emulators.qemu/110272
> 3. http://thread.gmane.org/gmane.comp.emulators.qemu/110426
>
> And here are the hacks (well, the first hunk is clearly a fix, the last
> one clearly a hack, /me still undecided about the rest):
>
> diff --git a/hw/acpi_piix4.c b/hw/acpi_piix4.c
> index c30a050..f650250 100644
> --- a/hw/acpi_piix4.c
> +++ b/hw/acpi_piix4.c
> @@ -92,7 +92,8 @@ static void pm_update_sci(PIIX4PMState *s)
> ACPI_BITMASK_POWER_BUTTON_ENABLE |
> ACPI_BITMASK_GLOBAL_LOCK_ENABLE |
> ACPI_BITMASK_TIMER_ENABLE)) != 0) ||
> - (((s->gpe.sts[0] & s->gpe.en[0]) & PIIX4_PCI_HOTPLUG_STATUS) != 0);
> + (((s->gpe.sts[0] & s->gpe.en[0]) &
> + (PIIX4_PCI_HOTPLUG_STATUS | PIIX4_CPU_HOTPLUG_STATUS)) != 0);
>
> qemu_set_irq(s->irq, sci_level);
> /* schedule a timer interruption if needed */
> diff --git a/hw/pc.c b/hw/pc.c
> index c0a88e1..e5371be 100644
> --- a/hw/pc.c
> +++ b/hw/pc.c
> @@ -42,6 +42,7 @@
> #include "kvm.h"
> #include "blockdev.h"
> #include "ui/qemu-spice.h"
> +#include "cpus.h"
>
> /* output Bochs bios info messages */
> //#define DEBUG_BIOS
> @@ -936,6 +937,10 @@ CPUState *pc_new_cpu(const char *cpu_model)
> #endif
> }
>
> + if (vm_running) {
> + pause_all_vcpus();
> + }
> +
> env = cpu_init(cpu_model);
> if (!env) {
> fprintf(stderr, "Unable to find x86 CPU definition\n");
> @@ -947,6 +952,11 @@ CPUState *pc_new_cpu(const char *cpu_model)
> }
> qemu_register_reset(pc_cpu_reset, env);
> pc_cpu_reset(env);
> +
> + cpu_synchronize_post_init(env);
> + if (vm_running) {
> + resume_all_vcpus();
> + }
> return env;
> }
>
> diff --git a/hw/qdev.c b/hw/qdev.c
> index 1626131..b91e2c2 100644
> --- a/hw/qdev.c
> +++ b/hw/qdev.c
> @@ -330,6 +330,7 @@ BusState *sysbus_get_default(void)
> if (!main_system_bus) {
> main_system_bus = qbus_create(&system_bus_info, NULL,
> "main-system-bus");
> + main_system_bus->allow_hotplug = 1;
> }
> return main_system_bus;
> }
>
> I see two remaining problems:
> - kvmclock is somehow broken, either in my guest kernel (OpenSUSE HEAD
> 3.0.0-2) or the host, -cpu host,-kvmclock works around sporadic
> guest lockups on echo 1 > /sys...
> - Seabios tends to lock up once every few system_reset after some
> CPU has been hot-added - also in TCG mode. It seems to dislike any
> setup of #CPUs > smp_cpus (whatever that implies in details).
>
Have you specified maxcpus? Something like -smp 1,maxcpus=4.

--
Gleb.