Re: [PATCH] cpu hotplug issue

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2011-07-24 18:11, Jan Kiszka wrote:
>>> I had a closer look and identified two further issues, one generic, one
>>> CPU-hotplug-specific:
>>>  - (qdev) devices that are hotplugged do not receive any reset. That
>>>    does not only apply to the APIC in case of CPU hotplugging, it is
>>>    also broken for NICs, storage controllers, etc. when doing PCI
>>>    hot-add as I just checked via gdb.
>>>  - CPU hotplugging was always (or at least for a fairly long time),
>>>    well, fragile as it failed to make CPU thread creation and CPU
>>>    initialization atomic against APIC addition and other initialization
>>>    steps. IOW, we need to create CPUs stopped, finish all init work,
>>>    sync their states completely to the kernel
>>>    (cpu_synchronize_post_init), and then kick them of. Actually I'm
>> Syncing the state to the kernel should be done by vcpu thread, so I it
>> cannot be stopped while the sync is done. May be I misunderstood what
>> you mean here.
> 
> Stopped first of all means not entering kvm_cpu_exec before the whole
> setup and the initial sync are done.
> 
> Syncing the initial state may also happen over the creating context as
> long as the vcpus are stopped (analogously to
> kvm_cpu_synchronize_post_init).

OK, hacks below plus the following three patches make CPU hotplug work
again - with some exceptions. Here are the patches:

1. http://thread.gmane.org/gmane.comp.emulators.kvm.devel/76484
2. http://thread.gmane.org/gmane.comp.emulators.qemu/110272
3. http://thread.gmane.org/gmane.comp.emulators.qemu/110426

And here are the hacks (well, the first hunk is clearly a fix, the last
one clearly a hack, /me still undecided about the rest):

diff --git a/hw/acpi_piix4.c b/hw/acpi_piix4.c
index c30a050..f650250 100644
--- a/hw/acpi_piix4.c
+++ b/hw/acpi_piix4.c
@@ -92,7 +92,8 @@ static void pm_update_sci(PIIX4PMState *s)
                    ACPI_BITMASK_POWER_BUTTON_ENABLE |
                    ACPI_BITMASK_GLOBAL_LOCK_ENABLE |
                    ACPI_BITMASK_TIMER_ENABLE)) != 0) ||
-        (((s->gpe.sts[0] & s->gpe.en[0]) & PIIX4_PCI_HOTPLUG_STATUS) != 0);
+        (((s->gpe.sts[0] & s->gpe.en[0]) &
+          (PIIX4_PCI_HOTPLUG_STATUS | PIIX4_CPU_HOTPLUG_STATUS)) != 0);
 
     qemu_set_irq(s->irq, sci_level);
     /* schedule a timer interruption if needed */
diff --git a/hw/pc.c b/hw/pc.c
index c0a88e1..e5371be 100644
--- a/hw/pc.c
+++ b/hw/pc.c
@@ -42,6 +42,7 @@
 #include "kvm.h"
 #include "blockdev.h"
 #include "ui/qemu-spice.h"
+#include "cpus.h"
 
 /* output Bochs bios info messages */
 //#define DEBUG_BIOS
@@ -936,6 +937,10 @@ CPUState *pc_new_cpu(const char *cpu_model)
 #endif
     }
 
+    if (vm_running) {
+        pause_all_vcpus();
+    }
+
     env = cpu_init(cpu_model);
     if (!env) {
         fprintf(stderr, "Unable to find x86 CPU definition\n");
@@ -947,6 +952,11 @@ CPUState *pc_new_cpu(const char *cpu_model)
     }
     qemu_register_reset(pc_cpu_reset, env);
     pc_cpu_reset(env);
+
+    cpu_synchronize_post_init(env);
+    if (vm_running) {
+        resume_all_vcpus();
+    }
     return env;
 }
 
diff --git a/hw/qdev.c b/hw/qdev.c
index 1626131..b91e2c2 100644
--- a/hw/qdev.c
+++ b/hw/qdev.c
@@ -330,6 +330,7 @@ BusState *sysbus_get_default(void)
     if (!main_system_bus) {
         main_system_bus = qbus_create(&system_bus_info, NULL,
                                       "main-system-bus");
+        main_system_bus->allow_hotplug = 1;
     }
     return main_system_bus;
 }

I see two remaining problems:
 - kvmclock is somehow broken, either in my guest kernel (OpenSUSE HEAD
   3.0.0-2) or the host, -cpu host,-kvmclock works around sporadic
   guest lockups on echo 1 > /sys...
 - Seabios tends to lock up once every few system_reset after some
   CPU has been hot-added - also in TCG mode. It seems to dislike any
   setup of #CPUs > smp_cpus (whatever that implies in details).

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux