Re: PVHVM VCPU hotplug mechanism via ACPI hotplug doesn't work in Xen 4.[1, 2, 3]

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

> >>[   41.517099] ------------[ cut here ]------------
> >>[   41.519532] kernel BUG at /home/konrad/ssd/konrad/linux/arch/x86/xen/time.c:337!
> >>[   41.519532] invalid opcode: 0000 [#1] SMP
> >>[   41.519532] Modules linked in: dm_multipath dm_mod iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi libcrc32c crc32c sg sr_mod cdrom ata_generic crc32c_intel ata_piix libata scsi_mod xen_blkfront xen_netfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xenfs xen_privcmd
> >>[   41.519532] CPU 1
> >>[   41.519532] Pid: 0, comm: swapper/1 Not tainted 3.9.0upstream-00022-g49c1bf1-dirty #3 Xen HVM domU
> >>[   41.519532] RIP: 0010:[<ffffffff81044f5c>]  [<ffffffff81044f5c>] xen_vcpuop_set_mode+0x3c/0x80
> >>[   41.519532] RSP: 0000:ffff880074467db8  EFLAGS: 00010082
> >>[   41.519532] RAX: ffffffffffffffea RBX: ffff880074a2be40 RCX: 0000000000000001
> >>[   41.519532] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000009
> >>[   41.519532] RBP: ffff880074467db8 R08: 0000000000000001 R09: 0000000000000000
> >>[   41.519532] R10: 0000000000000008 R11: 0000000000000001 R12: 0000000000000001
> >>[   41.519532] R13: 0000000000000082 R14: 0000000000000000 R15: 0000000000000082
> >>[   41.519532] FS:  0000000000000000(0000) GS:ffff880074a20000(0000) knlGS:0000000000000000
> >>[   41.519532] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >>[   41.519532] CR2: 0000000000000000 CR3: 0000000001c0c000 CR4: 00000000000406e0
> >>[   41.519532] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> >>[   41.519532] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> >>[   41.519532] Process swapper/1 (pid: 0, threadinfo ffff880074466000, task ffff880074464300)
> >>[   41.519532] Stack:
> >>[   41.519532]  ffff880074467dd8 ffffffff810e8385 ffff880074a2be40 ffff880074a2be40
> >>[   41.519532]  ffff880074467df8 ffffffff810e83e6 ffff880074467df8 0000000000000000
> >>[   41.519532]  ffff880074467e28 ffffffff810e84b7 ffffffff810e8ae4 ffff880074a2be40
> >>[   41.519532] Call Trace:
> >>[   41.519532]  [<ffffffff810e8385>] clockevents_set_mode+0x25/0x70
> >>[   41.519532]  [<ffffffff810e83e6>] clockevents_shutdown+0x16/0x30
> >>[   41.519532]  [<ffffffff810e84b7>] clockevents_exchange_device+0xb7/0x110
> >>[   41.519532]  [<ffffffff810e8ae4>] ? tick_notify+0x114/0x420
> >>[   41.519532]  [<ffffffff810e8b99>] tick_notify+0x1c9/0x420
> >>[   41.519532]  [<ffffffff810e8221>] ? clockevents_register_device+0x31/0x170
> >>[   41.519532]  [<ffffffff8169dd9d>] notifier_call_chain+0x4d/0x70
> >>[   41.519532]  [<ffffffff810be851>] raw_notifier_call_chain+0x11/0x20
> >>[   41.519532]  [<ffffffff810e82d0>] clockevents_register_device+0xe0/0x170
> >>[   41.519532]  [<ffffffff8104502c>] xen_setup_cpu_clockevents+0x2c/0x50
> >>[   41.519532]  [<ffffffff81045066>] xen_hvm_setup_cpu_clockevents+0x16/0x20
> >>[   41.519532]  [<ffffffff8168d48f>] start_secondary+0x1ea/0x1f9
> >>[   41.519532] Code: 73 2d 48 63 c9 bf 09 00 00 00 31 d2 48 89 ce e8 bb c3 fb ff 85 c0 75 13 bf 07 00 00 00 48 89 ce 31 d2 e8 a8 c3 fb ff 85 c0 74 09 <0f> 0b eb fe 83 ff 03 74 02 c9 c3 bf 07 00 00 00 48 63 f1 31 d2
> >>[   41.519532] RIP  [<ffffffff81044f5c>] xen_vcpuop_set_mode+0x3c/0x80
> >>[   41.519532]  RSP <ffff880074467db8>
> >>[   41.519532] ---[ end trace a182694869545b1a ]---
> >>[   41.519532] Kernel panic - not syncing: Attempted to kill the idle task!
> >>
> >>Thought this is using xend, as you cannot in xl have maxvcpus != vcpus.
> 
> Did you mean maxvcpus != vcpus, or maxvcpus > pcpus?

It is true either way from a boolean perspective :-)

In the past I couldn't get maxvcpus != vcpus to work with xl, but that seems
to have fixed itself. Then I ran in the maxvcpus != vcpus for which I had
posted a patch. The xen_vcpuop_set_mode shows up with maxvcpus != vcpus
(and it sometimes shows up as maxvcpus > pcpus).

However I seem to have run in two more issues:
 - QEMU's updating the offline/online CPU races with ACPI causing it to
   online/offline some of the CPUS but not all.
 - The hypervisor's vcpu id is different from what 'smp_processor_id()' in the
   guest shows (that is the vcpu_set_mode above). This looks to be happening
   also with Xen 4.1.