Re: [PATCH] cpu hotplug issue

Jan Kiszka
2011-07-25T13:26:23Z

On 2011-07-25 15:21, Gleb Natapov wrote:
> On Mon, Jul 25, 2011 at 03:18:19PM +0200, Jan Kiszka wrote:
>> On 2011-07-24 18:11, Jan Kiszka wrote:
>>>>> I had a closer look and identified two further issues, one generic, one
>>>>> CPU-hotplug-specific:
>>>>> - (qdev) devices that are hotplugged do not receive any reset. That
>>>>> does not only apply to the APIC in case of CPU hotplugging, it is
>>>>> also broken for NICs, storage controllers, etc. when doing PCI
>>>>> hot-add as I just checked via gdb.
>>>>> - CPU hotplugging was always (or at least for a fairly long time),
>>>>> well, fragile as it failed to make CPU thread creation and CPU
>>>>> initialization atomic against APIC addition and other initialization
>>>>> steps. IOW, we need to create CPUs stopped, finish all init work,
>>>>> sync their states completely to the kernel
>>>>> (cpu_synchronize_post_init), and then kick them of. Actually I'm
>>>> Syncing the state to the kernel should be done by vcpu thread, so I it
>>>> cannot be stopped while the sync is done. May be I misunderstood what
>>>> you mean here.
>>>
>>> Stopped first of all means not entering kvm_cpu_exec before the whole
>>> setup and the initial sync are done.
>>>
>>> Syncing the initial state may also happen over the creating context as
>>> long as the vcpus are stopped (analogously to
>>> kvm_cpu_synchronize_post_init).
>>
>> OK, hacks below plus the following three patches make CPU hotplug work
>> again - with some exceptions. Here are the patches:
>>
>> 1. http://thread.gmane.org/gmane.comp.emulators.kvm.devel/76484
>> 2. http://thread.gmane.org/gmane.comp.emulators.qemu/110272
>> 3. http://thread.gmane.org/gmane.comp.emulators.qemu/110426
>>
>> And here are the hacks (well, the first hunk is clearly a fix, the last
>> one clearly a hack, /me still undecided about the rest):
>>
>> diff --git a/hw/acpi_piix4.c b/hw/acpi_piix4.c
>> index c30a050..f650250 100644
>> --- a/hw/acpi_piix4.c
>> +++ b/hw/acpi_piix4.c
>> @@ -92,7 +92,8 @@ static void pm_update_sci(PIIX4PMState *s)
>> ACPI_BITMASK_POWER_BUTTON_ENABLE |
>> ACPI_BITMASK_GLOBAL_LOCK_ENABLE |
>> ACPI_BITMASK_TIMER_ENABLE)) != 0) ||
>> - (((s->gpe.sts[0] & s->gpe.en[0]) & PIIX4_PCI_HOTPLUG_STATUS) != 0);
>> + (((s->gpe.sts[0] & s->gpe.en[0]) &
>> + (PIIX4_PCI_HOTPLUG_STATUS | PIIX4_CPU_HOTPLUG_STATUS)) != 0);
>>
>> qemu_set_irq(s->irq, sci_level);
>> /* schedule a timer interruption if needed */
>> diff --git a/hw/pc.c b/hw/pc.c
>> index c0a88e1..e5371be 100644
>> --- a/hw/pc.c
>> +++ b/hw/pc.c
>> @@ -42,6 +42,7 @@
>> #include "kvm.h"
>> #include "blockdev.h"
>> #include "ui/qemu-spice.h"
>> +#include "cpus.h"
>>
>> /* output Bochs bios info messages */
>> //#define DEBUG_BIOS
>> @@ -936,6 +937,10 @@ CPUState *pc_new_cpu(const char *cpu_model)
>> #endif
>> }
>>
>> + if (vm_running) {
>> + pause_all_vcpus();
>> + }
>> +
>> env = cpu_init(cpu_model);
>> if (!env) {
>> fprintf(stderr, "Unable to find x86 CPU definition\n");
>> @@ -947,6 +952,11 @@ CPUState *pc_new_cpu(const char *cpu_model)
>> }
>> qemu_register_reset(pc_cpu_reset, env);
>> pc_cpu_reset(env);
>> +
>> + cpu_synchronize_post_init(env);
>> + if (vm_running) {
>> + resume_all_vcpus();
>> + }
>> return env;
>> }
>>
>> diff --git a/hw/qdev.c b/hw/qdev.c
>> index 1626131..b91e2c2 100644
>> --- a/hw/qdev.c
>> +++ b/hw/qdev.c
>> @@ -330,6 +330,7 @@ BusState *sysbus_get_default(void)
>> if (!main_system_bus) {
>> main_system_bus = qbus_create(&system_bus_info, NULL,
>> "main-system-bus");
>> + main_system_bus->allow_hotplug = 1;
>> }
>> return main_system_bus;
>> }
>>
>> I see two remaining problems:
>> - kvmclock is somehow broken, either in my guest kernel (OpenSUSE HEAD
>> 3.0.0-2) or the host, -cpu host,-kvmclock works around sporadic
>> guest lockups on echo 1 > /sys...
>> - Seabios tends to lock up once every few system_reset after some
>> CPU has been hot-added - also in TCG mode. It seems to dislike any
>> setup of #CPUs > smp_cpus (whatever that implies in details).
>>
> Have you specified maxcpus? Something like -smp 1,maxcpus=4.

Yes, for sure.

BTW, cpu_set completely lacks any parameter sanity checks. That
interface looks like "for gurus only". Hope we can do better via qdev
and properties.

Jan

--
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux