Re: [PATCH] cpu hotplug issue

[Gleb Natapov <gleb@redhat.com>]

On Mon, Jul 25, 2011 at 03:18:19PM +0200, Jan Kiszka wrote:
> On 2011-07-24 18:11, Jan Kiszka wrote:
> >>> I had a closer look and identified two further issues, one generic, one
> >>> CPU-hotplug-specific:
> >>>  - (qdev) devices that are hotplugged do not receive any reset. That
> >>>    does not only apply to the APIC in case of CPU hotplugging, it is
> >>>    also broken for NICs, storage controllers, etc. when doing PCI
> >>>    hot-add as I just checked via gdb.
> >>>  - CPU hotplugging was always (or at least for a fairly long time),
> >>>    well, fragile as it failed to make CPU thread creation and CPU
> >>>    initialization atomic against APIC addition and other initialization
> >>>    steps. IOW, we need to create CPUs stopped, finish all init work,
> >>>    sync their states completely to the kernel
> >>>    (cpu_synchronize_post_init), and then kick them of. Actually I'm
> >> Syncing the state to the kernel should be done by vcpu thread, so I it
> >> cannot be stopped while the sync is done. May be I misunderstood what
> >> you mean here.
> > 
> > Stopped first of all means not entering kvm_cpu_exec before the whole
> > setup and the initial sync are done.
> > 
> > Syncing the initial state may also happen over the creating context as
> > long as the vcpus are stopped (analogously to
> > kvm_cpu_synchronize_post_init).
> 
> OK, hacks below plus the following three patches make CPU hotplug work
> again - with some exceptions. Here are the patches:
> 
> 1. http://thread.gmane.org/gmane.comp.emulators.kvm.devel/76484
> 2. http://thread.gmane.org/gmane.comp.emulators.qemu/110272
> 3. http://thread.gmane.org/gmane.comp.emulators.qemu/110426
> 
> And here are the hacks (well, the first hunk is clearly a fix, the last
> one clearly a hack, /me still undecided about the rest):
> 
> diff --git a/hw/acpi_piix4.c b/hw/acpi_piix4.c
> index c30a050..f650250 100644
> --- a/hw/acpi_piix4.c
> +++ b/hw/acpi_piix4.c
> @@ -92,7 +92,8 @@ static void pm_update_sci(PIIX4PMState *s)
>                     ACPI_BITMASK_POWER_BUTTON_ENABLE |
>                     ACPI_BITMASK_GLOBAL_LOCK_ENABLE |
>                     ACPI_BITMASK_TIMER_ENABLE)) != 0) ||
> -        (((s->gpe.sts[0] & s->gpe.en[0]) & PIIX4_PCI_HOTPLUG_STATUS) != 0);
> +        (((s->gpe.sts[0] & s->gpe.en[0]) &
> +          (PIIX4_PCI_HOTPLUG_STATUS | PIIX4_CPU_HOTPLUG_STATUS)) != 0);
>  
>      qemu_set_irq(s->irq, sci_level);
>      /* schedule a timer interruption if needed */
> diff --git a/hw/pc.c b/hw/pc.c
> index c0a88e1..e5371be 100644
> --- a/hw/pc.c
> +++ b/hw/pc.c
> @@ -42,6 +42,7 @@
>  #include "kvm.h"
>  #include "blockdev.h"
>  #include "ui/qemu-spice.h"
> +#include "cpus.h"
>  
>  /* output Bochs bios info messages */
>  //#define DEBUG_BIOS
> @@ -936,6 +937,10 @@ CPUState *pc_new_cpu(const char *cpu_model)
>  #endif
>      }
>  
> +    if (vm_running) {
> +        pause_all_vcpus();
> +    }
> +
>      env = cpu_init(cpu_model);
>      if (!env) {
>          fprintf(stderr, "Unable to find x86 CPU definition\n");
> @@ -947,6 +952,11 @@ CPUState *pc_new_cpu(const char *cpu_model)
>      }
>      qemu_register_reset(pc_cpu_reset, env);
>      pc_cpu_reset(env);
> +
> +    cpu_synchronize_post_init(env);
> +    if (vm_running) {
> +        resume_all_vcpus();
> +    }
>      return env;
>  }
>  
> diff --git a/hw/qdev.c b/hw/qdev.c
> index 1626131..b91e2c2 100644
> --- a/hw/qdev.c
> +++ b/hw/qdev.c
> @@ -330,6 +330,7 @@ BusState *sysbus_get_default(void)
>      if (!main_system_bus) {
>          main_system_bus = qbus_create(&system_bus_info, NULL,
>                                        "main-system-bus");
> +        main_system_bus->allow_hotplug = 1;
>      }
>      return main_system_bus;
>  }
> 
> I see two remaining problems:
>  - kvmclock is somehow broken, either in my guest kernel (OpenSUSE HEAD
>    3.0.0-2) or the host, -cpu host,-kvmclock works around sporadic
>    guest lockups on echo 1 > /sys...
>  - Seabios tends to lock up once every few system_reset after some
>    CPU has been hot-added - also in TCG mode. It seems to dislike any
>    setup of #CPUs > smp_cpus (whatever that implies in details).
> 
Have you specified maxcpus? Something like -smp 1,maxcpus=4.

--
			Gleb.