Re: [PATCH] cpu hotplug issue

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2011-07-25 15:21, Gleb Natapov wrote:
> On Mon, Jul 25, 2011 at 03:18:19PM +0200, Jan Kiszka wrote:
>> On 2011-07-24 18:11, Jan Kiszka wrote:
>>>>> I had a closer look and identified two further issues, one generic, one
>>>>> CPU-hotplug-specific:
>>>>>  - (qdev) devices that are hotplugged do not receive any reset. That
>>>>>    does not only apply to the APIC in case of CPU hotplugging, it is
>>>>>    also broken for NICs, storage controllers, etc. when doing PCI
>>>>>    hot-add as I just checked via gdb.
>>>>>  - CPU hotplugging was always (or at least for a fairly long time),
>>>>>    well, fragile as it failed to make CPU thread creation and CPU
>>>>>    initialization atomic against APIC addition and other initialization
>>>>>    steps. IOW, we need to create CPUs stopped, finish all init work,
>>>>>    sync their states completely to the kernel
>>>>>    (cpu_synchronize_post_init), and then kick them of. Actually I'm
>>>> Syncing the state to the kernel should be done by vcpu thread, so I it
>>>> cannot be stopped while the sync is done. May be I misunderstood what
>>>> you mean here.
>>>
>>> Stopped first of all means not entering kvm_cpu_exec before the whole
>>> setup and the initial sync are done.
>>>
>>> Syncing the initial state may also happen over the creating context as
>>> long as the vcpus are stopped (analogously to
>>> kvm_cpu_synchronize_post_init).
>>
>> OK, hacks below plus the following three patches make CPU hotplug work
>> again - with some exceptions. Here are the patches:
>>
>> 1. http://thread.gmane.org/gmane.comp.emulators.kvm.devel/76484
>> 2. http://thread.gmane.org/gmane.comp.emulators.qemu/110272
>> 3. http://thread.gmane.org/gmane.comp.emulators.qemu/110426
>>
>> And here are the hacks (well, the first hunk is clearly a fix, the last
>> one clearly a hack, /me still undecided about the rest):
>>
>> diff --git a/hw/acpi_piix4.c b/hw/acpi_piix4.c
>> index c30a050..f650250 100644
>> --- a/hw/acpi_piix4.c
>> +++ b/hw/acpi_piix4.c
>> @@ -92,7 +92,8 @@ static void pm_update_sci(PIIX4PMState *s)
>>                     ACPI_BITMASK_POWER_BUTTON_ENABLE |
>>                     ACPI_BITMASK_GLOBAL_LOCK_ENABLE |
>>                     ACPI_BITMASK_TIMER_ENABLE)) != 0) ||
>> -        (((s->gpe.sts[0] & s->gpe.en[0]) & PIIX4_PCI_HOTPLUG_STATUS) != 0);
>> +        (((s->gpe.sts[0] & s->gpe.en[0]) &
>> +          (PIIX4_PCI_HOTPLUG_STATUS | PIIX4_CPU_HOTPLUG_STATUS)) != 0);
>>  
>>      qemu_set_irq(s->irq, sci_level);
>>      /* schedule a timer interruption if needed */
>> diff --git a/hw/pc.c b/hw/pc.c
>> index c0a88e1..e5371be 100644
>> --- a/hw/pc.c
>> +++ b/hw/pc.c
>> @@ -42,6 +42,7 @@
>>  #include "kvm.h"
>>  #include "blockdev.h"
>>  #include "ui/qemu-spice.h"
>> +#include "cpus.h"
>>  
>>  /* output Bochs bios info messages */
>>  //#define DEBUG_BIOS
>> @@ -936,6 +937,10 @@ CPUState *pc_new_cpu(const char *cpu_model)
>>  #endif
>>      }
>>  
>> +    if (vm_running) {
>> +        pause_all_vcpus();
>> +    }
>> +
>>      env = cpu_init(cpu_model);
>>      if (!env) {
>>          fprintf(stderr, "Unable to find x86 CPU definition\n");
>> @@ -947,6 +952,11 @@ CPUState *pc_new_cpu(const char *cpu_model)
>>      }
>>      qemu_register_reset(pc_cpu_reset, env);
>>      pc_cpu_reset(env);
>> +
>> +    cpu_synchronize_post_init(env);
>> +    if (vm_running) {
>> +        resume_all_vcpus();
>> +    }
>>      return env;
>>  }
>>  
>> diff --git a/hw/qdev.c b/hw/qdev.c
>> index 1626131..b91e2c2 100644
>> --- a/hw/qdev.c
>> +++ b/hw/qdev.c
>> @@ -330,6 +330,7 @@ BusState *sysbus_get_default(void)
>>      if (!main_system_bus) {
>>          main_system_bus = qbus_create(&system_bus_info, NULL,
>>                                        "main-system-bus");
>> +        main_system_bus->allow_hotplug = 1;
>>      }
>>      return main_system_bus;
>>  }
>>
>> I see two remaining problems:
>>  - kvmclock is somehow broken, either in my guest kernel (OpenSUSE HEAD
>>    3.0.0-2) or the host, -cpu host,-kvmclock works around sporadic
>>    guest lockups on echo 1 > /sys...
>>  - Seabios tends to lock up once every few system_reset after some
>>    CPU has been hot-added - also in TCG mode. It seems to dislike any
>>    setup of #CPUs > smp_cpus (whatever that implies in details).
>>
> Have you specified maxcpus? Something like -smp 1,maxcpus=4.

Yes, for sure.

BTW, cpu_set completely lacks any parameter sanity checks. That
interface looks like "for gurus only". Hope we can do better via qdev
and properties.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux