Re: [PATCH] cpu hotplug issue

Jan Kiszka
2011-07-25T13:18:24Z

On 2011-07-24 18:11, Jan Kiszka wrote:
>>> I had a closer look and identified two further issues, one generic, one
>>> CPU-hotplug-specific:
>>> - (qdev) devices that are hotplugged do not receive any reset. That
>>> does not only apply to the APIC in case of CPU hotplugging, it is
>>> also broken for NICs, storage controllers, etc. when doing PCI
>>> hot-add as I just checked via gdb.
>>> - CPU hotplugging was always (or at least for a fairly long time),
>>> well, fragile as it failed to make CPU thread creation and CPU
>>> initialization atomic against APIC addition and other initialization
>>> steps. IOW, we need to create CPUs stopped, finish all init work,
>>> sync their states completely to the kernel
>>> (cpu_synchronize_post_init), and then kick them of. Actually I'm
>> Syncing the state to the kernel should be done by vcpu thread, so I it
>> cannot be stopped while the sync is done. May be I misunderstood what
>> you mean here.
>
> Stopped first of all means not entering kvm_cpu_exec before the whole
> setup and the initial sync are done.
>
> Syncing the initial state may also happen over the creating context as
> long as the vcpus are stopped (analogously to
> kvm_cpu_synchronize_post_init).

OK, hacks below plus the following three patches make CPU hotplug work
again - with some exceptions. Here are the patches:

1. http://thread.gmane.org/gmane.comp.emulators.kvm.devel/76484
2. http://thread.gmane.org/gmane.comp.emulators.qemu/110272
3. http://thread.gmane.org/gmane.comp.emulators.qemu/110426

And here are the hacks (well, the first hunk is clearly a fix, the last
one clearly a hack, /me still undecided about the rest):

diff --git a/hw/acpi_piix4.c b/hw/acpi_piix4.c
index c30a050..f650250 100644
--- a/hw/acpi_piix4.c
+++ b/hw/acpi_piix4.c
@@ -92,7 +92,8 @@ static void pm_update_sci(PIIX4PMState *s)
ACPI_BITMASK_POWER_BUTTON_ENABLE |
ACPI_BITMASK_GLOBAL_LOCK_ENABLE |
ACPI_BITMASK_TIMER_ENABLE)) != 0) ||
- (((s->gpe.sts[0] & s->gpe.en[0]) & PIIX4_PCI_HOTPLUG_STATUS) != 0);
+ (((s->gpe.sts[0] & s->gpe.en[0]) &
+ (PIIX4_PCI_HOTPLUG_STATUS | PIIX4_CPU_HOTPLUG_STATUS)) != 0);

qemu_set_irq(s->irq, sci_level);
/* schedule a timer interruption if needed */
diff --git a/hw/pc.c b/hw/pc.c
index c0a88e1..e5371be 100644
--- a/hw/pc.c
+++ b/hw/pc.c
@@ -42,6 +42,7 @@
#include "kvm.h"
#include "blockdev.h"
#include "ui/qemu-spice.h"
+#include "cpus.h"

/* output Bochs bios info messages */
//#define DEBUG_BIOS
@@ -936,6 +937,10 @@ CPUState *pc_new_cpu(const char *cpu_model)
#endif
}

+ if (vm_running) {
+ pause_all_vcpus();
+ }
+
env = cpu_init(cpu_model);
if (!env) {
fprintf(stderr, "Unable to find x86 CPU definition\n");
@@ -947,6 +952,11 @@ CPUState *pc_new_cpu(const char *cpu_model)
}
qemu_register_reset(pc_cpu_reset, env);
pc_cpu_reset(env);
+
+ cpu_synchronize_post_init(env);
+ if (vm_running) {
+ resume_all_vcpus();
+ }
return env;
}

diff --git a/hw/qdev.c b/hw/qdev.c
index 1626131..b91e2c2 100644
--- a/hw/qdev.c
+++ b/hw/qdev.c
@@ -330,6 +330,7 @@ BusState *sysbus_get_default(void)
if (!main_system_bus) {
main_system_bus = qbus_create(&system_bus_info, NULL,
"main-system-bus");
+ main_system_bus->allow_hotplug = 1;
}
return main_system_bus;
}

I see two remaining problems:
- kvmclock is somehow broken, either in my guest kernel (OpenSUSE HEAD
3.0.0-2) or the host, -cpu host,-kvmclock works around sporadic
guest lockups on echo 1 > /sys...
- Seabios tends to lock up once every few system_reset after some
CPU has been hot-added - also in TCG mode. It seems to dislike any
setup of #CPUs > smp_cpus (whatever that implies in details).

Jan

--
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux