[Qemu-devel] Runtime-modified DIMMs and live migration issue

[Qemu-devel] Runtime-modified DIMMs and live migration issue

[Andrey Korolyov]

[-- Attachment #1: Type: text/plain, Size: 1177 bytes --]

Hello Igor,

the current hotplug code for dimms effectively prohibiting a
successful migration for VM if memory was added after startup:

- start a VM with certain amount of empty memory slots,
- add some dimms and online them in guest (I am transitioning from 2
to 16G with 512Mb DIMMs),
- migrate a VM and observe guest null pointer dereference (or BSOD
with reboot, for Windows).

Issue is currently touching all stable versions and assumingly master,
as there are no related fixes/RFCs since 2.3 I`m currently using for
testing. The issue is related to an incorrect population of the
regions during runtime hotplugging, hopefully 2.4 will get the fix.

You may run some workload in guest to achieve one hundred percent
certainty of hitting the issue, for example, fio against
http://xdel.ru/downloads/fio.txt . QEMU args are simular to '... -m
512,slots=31,maxmem=16384M -object
memory-backend-ram,id=mem0,size=512M -device
pc-dimm,id=dimm0,node=0,memdev=mem0 -object
memory-backend-ram,id=mem1,size=512M -device
pc-dimm,id=dimm1,node=0,memdev=mem1 -object
memory-backend-ram,id=mem2,size=512M -device
pc-dimm,id=dimm2,node=0,memdev=mem2...'

Thanks for looking into this!

[-- Attachment #2: fio-migrate-hotplug.trace.txt --]
[-- Type: text/plain, Size: 5677 bytes --]

11 June 2015, 19:50:14 	[ 141.005630] fio[2742]: segfault at 0 ip (null) sp 00007f841ab5aeb8 error 14
11 June 2015, 19:50:14 	in fio[400000+58000]
11 June 2015, 19:50:14 	NULL pointer dereference
11 June 2015, 19:50:14 	at 0000000000000028
11 June 2015, 19:50:14 	[ 141.006282] IP:
11 June 2015, 19:50:14 	[ 141.006316] PGD 107ccc067
11 June 2015, 19:50:14 	PUD 106056067
11 June 2015, 19:50:14 	[ 141.006319] Oops: 0000 [#1]
11 June 2015, 19:50:14 	SMP
11 June 2015, 19:50:14 	nfsd
11 June 2015, 19:50:14 	auth_rpcgss
11 June 2015, 19:50:14 	oid_registry
11 June 2015, 19:50:14 	nfs
11 June 2015, 19:50:14 	lockd
11 June 2015, 19:50:14 	fscache
11 June 2015, 19:50:14 	netconsole
11 June 2015, 19:50:14 	configfs
11 June 2015, 19:50:14 	crct10dif_pclmul
11 June 2015, 19:50:14 	crct10dif_common
11 June 2015, 19:50:14 	ghash_clmulni_intel
11 June 2015, 19:50:14 	aesni_intel
11 June 2015, 19:50:14 	lrw
11 June 2015, 19:50:14 	gf128mul
11 June 2015, 19:50:14 	ablk_helper
11 June 2015, 19:50:14 	psmouse
11 June 2015, 19:50:14 	parport_pc
11 June 2015, 19:50:14 	virtio_console
11 June 2015, 19:50:14 	serio_raw
11 June 2015, 19:50:14 	evdev
11 June 2015, 19:50:14 	pcspkr
11 June 2015, 19:50:14 	processor
11 June 2015, 19:50:14 	thermal_sys
11 June 2015, 19:50:14 	button
11 June 2015, 19:50:14 	ext4
11 June 2015, 19:50:14 	mbcache
11 June 2015, 19:50:14 	ata_generic
11 June 2015, 19:50:14 	virtio_blk
11 June 2015, 19:50:14 	crc32c_intel
11 June 2015, 19:50:14 	floppy
11 June 2015, 19:50:14 	xhci_hcd
11 June 2015, 19:50:14 	libata
11 June 2015, 19:50:14 	virtio_ring
11 June 2015, 19:50:14 	usbcore
11 June 2015, 19:50:14 	usb_common
11 June 2015, 19:50:14 	[ 141.006396] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.16.0-0.bpo.4-amd64 #1 Debian 3.16.7-ckt7-1~bpo70+1
11 June 2015, 19:50:14 	[ 141.006397] Hardware name: SuperMicro Virtual Appliance, BIOS 1.1
11 June 2015, 19:50:14 	[ 141.006403] RIP: 0010:[<ffffffffa015ba38>]
11 June 2015, 19:50:14 	[<ffffffffa015ba38>] ext4_finish_bio+0xd8/0x220 [ext4]
11 June 2015, 19:50:14 	[ 141.006415] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000001000
11 June 2015, 19:50:14 	[ 141.006415] RDX: 000000000000000d RSI: ffffea0010ea6818 RDI: ffff88001c291300
11 June 2015, 19:50:14 	[ 141.006417] R10: 0000000000000002 R11: 0000000000000040 R12: ffff8804a41aaf98
11 June 2015, 19:50:14 	[ 141.006419] FS: 0000000000000000(0000) GS:ffff88001fc00000(0000) knlGS:0000000000000000
11 June 2015, 19:50:14 	[ 141.006420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
11 June 2015, 19:50:14 	[ 141.006434] Stack:
11 June 2015, 19:50:14 	[ 141.006435] ffffffff0000005e
11 June 2015, 19:50:14 	0000000000000007
11 June 2015, 19:50:14 	ffff88001fdd2ec0
11 June 2015, 19:50:14 	ffff88001c291300
11 June 2015, 19:50:14 	ffff88001d84c240
11 June 2015, 19:50:14 	[ 141.006439] 0000000000000093
11 June 2015, 19:50:14 	ffff88001d84c940
11 June 2015, 19:50:14 	d794c666350eb7d9
11 June 2015, 19:50:14 	[ 141.006442] <IRQ>
11 June 2015, 19:50:14 	[<ffffffffa015c036>] ? ext4_end_bio+0xc6/0x130 [ext4]
11 June 2015, 19:50:14 	[<ffffffff8129dcfb>] ? blk_update_request+0x9b/0x310
11 June 2015, 19:50:14 	[ 141.006488]
11 June 2015, 19:50:14 	[ 141.006494]
11 June 2015, 19:50:14 	[<ffffffff812a76c9>] ? __blk_mq_complete_request+0x79/0x110
11 June 2015, 19:50:14 	[<ffffffffa01461ed>] ? virtblk_done+0x4d/0xb0 [virtio_blk]
11 June 2015, 19:50:14 	[ 141.006506]
11 June 2015, 19:50:14 	[ 141.006512]
11 June 2015, 19:50:14 	[<ffffffff810c4f54>] ? handle_irq_event_percpu+0x54/0x1e0
11 June 2015, 19:50:14 	[<ffffffff810a73aa>] ? update_blocked_averages+0x24a/0x5f0
11 June 2015, 19:50:14 	[ 141.006540]
11 June 2015, 19:50:14 	[ 141.006542]
11 June 2015, 19:50:14 	[<ffffffff810c7f7d>] ? handle_edge_irq+0x7d/0x120
11 June 2015, 19:50:14 	[<ffffffff810175ed>] ? handle_irq+0x1d/0x30
11 June 2015, 19:50:14 	[ 141.006559]
11 June 2015, 19:50:14 	[ 141.006579]
11 June 2015, 19:50:15 	[ 141.006582]
11 June 2015, 19:50:15 	[<ffffffff81072bf8>] ? __do_softirq+0x88/0x2e0
11 June 2015, 19:50:15 	[<ffffffff81072b8d>] ? __do_softirq+0x1d/0x2e0
11 June 2015, 19:50:15 	[ 141.006592]
11 June 2015, 19:50:15 	[ 141.006600]
11 June 2015, 19:50:15 	[<ffffffff810730a6>] ? irq_exit+0x86/0xb0
11 June 2015, 19:50:15 	[<ffffffff8154d1d6>] ? do_IRQ+0x66/0x110
11 June 2015, 19:50:15 	[<ffffffff8154b06d>] ? common_interrupt+0x6d/0x6d
11 June 2015, 19:50:15 	[ 141.006609] <EOI>
11 June 2015, 19:50:15 	[ 141.006616]
11 June 2015, 19:50:15 	[ 141.006619]
11 June 2015, 19:50:15 	[<ffffffff8101f6b2>] ? default_idle+0x22/0xf0
11 June 2015, 19:50:15 	[<ffffffff810b1818>] ? cpu_startup_entry+0x2e8/0x4b0
11 June 2015, 19:50:15 	[ 141.006624]
11 June 2015, 19:50:15 	[ 141.006629]
11 June 2015, 19:50:15 	[<ffffffff81900a1a>] ? set_init_arg+0x4d/0x4d
11 June 2015, 19:50:15 	[<ffffffff81900120>] ? early_idt_handlers+0x120/0x120
11 June 2015, 19:50:15 	[<ffffffff8190072b>] ? x86_64_start_kernel+0x150/0x15f
11 June 2015, 19:50:15 	[ 141.006635] Code:
11 June 2015, 19:50:15 	4c 89 e3 eb 26 66 0f 1f 44 00 00 48 03 43 20 48 39 c8 77 25 f0 80 63 01 fe 45 85 c0 0f 85 a1 00 00 00 48 8b 5b 08 49 39 dc 74 26 <48> 8b 43 28 25 ff 0f 00 00 4c 39 e8 73 d2 48 8b 03 48 8b 5b 08
11 June 2015, 19:50:15 	[<ffffffffa015ba38>] ext4_finish_bio+0xd8/0x220 [ext4]
11 June 2015, 19:50:15 	[ 141.006682] CR2: 0000000000000028
11 June 2015, 19:50:15 	[ 141.006705] Kernel panic - not syncing: Fatal exception in interrupt
11 June 2015, 19:50:15 	[ 141.009665] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff)
11 June 2015, 19:50:15 	[ 141.009665] ---[ end Kernel panic - not syncing: Fatal exception in interrupt

Re: [Qemu-devel] Runtime-modified DIMMs and live migration issue

[Andrey Korolyov]

[-- Attachment #1: Type: text/plain, Size: 1884 bytes --]

On Thu, Jun 11, 2015 at 8:14 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> Hello Igor,
>
> the current hotplug code for dimms effectively prohibiting a
> successful migration for VM if memory was added after startup:
>
> - start a VM with certain amount of empty memory slots,
> - add some dimms and online them in guest (I am transitioning from 2
> to 16G with 512Mb DIMMs),
> - migrate a VM and observe guest null pointer dereference (or BSOD
> with reboot, for Windows).
>
> Issue is currently touching all stable versions and assumingly master,
> as there are no related fixes/RFCs since 2.3 I`m currently using for
> testing. The issue is related to an incorrect population of the
> regions during runtime hotplugging, hopefully 2.4 will get the fix.
>
> You may run some workload in guest to achieve one hundred percent
> certainty of hitting the issue, for example, fio against
> http://xdel.ru/downloads/fio.txt . QEMU args are simular to '... -m
> 512,slots=31,maxmem=16384M -object
> memory-backend-ram,id=mem0,size=512M -device
> pc-dimm,id=dimm0,node=0,memdev=mem0 -object
> memory-backend-ram,id=mem1,size=512M -device
> pc-dimm,id=dimm1,node=0,memdev=mem1 -object
> memory-backend-ram,id=mem2,size=512M -device
> pc-dimm,id=dimm2,node=0,memdev=mem2...'
>
> Thanks for looking into this!

Please find the full cli args and two guest logs for DIMM
initalization attached. As you can see, the freshly populated DIMMs
are probably misplaced in SRAT ('already populated' messages), despite
the fact that the initialized ranges are looking correct at a glance.
When VM is migrated to the destination (with equal RAM device
configuration) which is simular to a VM with 16G RAM this misplacement
is causing the mentioned panic in the guest. This should be very
easily reproducible and the issue is very critical as well, I don`t
even understand why I missed this issue earlier.

[-- Attachment #2: hotplug-qemu-args.txt --]
[-- Type: text/plain, Size: 2433 bytes --]

qemu-system-x86_64 -enable-kvm -name vm33090 -S -machine pc-i440fx-2.3,accel=kvm,usb=off -cpu SandyBridge -bios /usr/share/seabios/bios.bin -m 512 -realtime mlock=off -smp 8,sockets=1,cores=8,threads=8 -numa node,nodeid=0,cpus=0-7,mem=512 -uuid 922e9c4f-1100-441c-ac8d-f7361a3ab987 -nographic -no-user-config -nodefaults -device sga -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm33090.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device nec-usb-xhci,id=usb,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=rbd:dev-rack2/vm33090-Iuy:id=qemukvm:key=xxxx:auth_supported=cephx\;none:mon_host=10.6.0.1\:6789\;10.6.0.3\:6789\;10.6.0.4\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writeback,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=39,id=hostnet0,vhost=on,vhostfd=40 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:10:08:e4,bus=pci.0,addr=0x2 -netdev tap,fd=41,id=hostnet1,vhost=on,vhostfd=42 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:10:08:e3,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/vm33090.sock,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1 -object iothread,id=vm33090blk0 -set device.virtio-disk0.config-wce=off -set device.virtio-disk0.scsi=off -set device.virtio-disk0.iothread=vm33090blk0 -m 512,slots=31,maxmem=16384M -object memory-backend-ram,id=mem0,size=512M -device pc-dimm,id=dimm0,node=0,memdev=mem0 -object memory-backend-ram,id=mem1,size=512M -device pc-dimm,id=dimm1,node=0,memdev=mem1 -object memory-backend-ram,id=mem2,size=512M -device pc-dimm,id=dimm2,node=0,memdev=mem2 -object memory-backend-ram,id=mem3,size=512M -device pc-dimm,id=dimm3,node=0,memdev=mem3 -object memory-backend-ram,id=mem4,size=512M -device pc-dimm,id=dimm4,node=0,memdev=mem4 -object memory-backend-ram,id=mem5,size=512M -device pc-dimm,id=dimm5,node=0,memdev=mem5 -object memory-backend-ram,id=mem6,size=512M -device pc-dimm,id=dimm6,node=0,memdev=mem6 -msg timestamp=on

[-- Attachment #3: 16G-no-hotplug.txt --]
[-- Type: text/plain, Size: 11163 bytes --]

16G (31 dimm populated) boot-up:
[    0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x02 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x03 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x04 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x05 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x06 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x07 -> Node 0
[    0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff]
[    0.000000] SRAT: Node 0 PXM 0 [mem 0x00100000-0x1fffffff]
[    0.000000] SRAT: Node 0 PXM 0 [mem 0x100000000-0xc9fffffff] hotplug
[    0.000000] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x1ffdffff] -> [mem 0x00000000-0x1ffdffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x1ffdffff]
[    0.000000]   NODE_DATA [mem 0x1ffdb000-0x1ffdffff]
[    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[    0.000000] kvm-clock: cpu 0, msr 0:1ffd3001, primary cpu clock
[    0.000000]  [ffffea0000000000-ffffea00007fffff] PMD -> [ffff88001e000000-ffff88001e7fffff] on node 0
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0009efff]
[    0.000000]   node   0: [mem 0x00100000-0x1ffdffff]
[    0.000000] On node 0 totalpages: 130942
[    0.000000]   DMA zone: 56 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3998 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 1736 pages used for memmap
[    0.000000]   DMA32 zone: 126944 pages, LIFO batch:31
...
[    0.296858] init_memory_mapping: [mem 0x100000000-0x11fffffff]
[    0.296862]  [mem 0x100000000-0x11fffffff] page 2M
[    0.304921]  [ffffea0003800000-ffffea0003bfffff] PMD -> [ffff88001d400000-ffff88001d7fffff] on node 0
[    0.309060] init_memory_mapping: [mem 0x120000000-0x13fffffff]
[    0.309063]  [mem 0x120000000-0x13fffffff] page 2M
[    0.311319]  [ffffea0003c00000-ffffea0003ffffff] PMD -> [ffff88001d000000-ffff88001d3fffff] on node 0
[    0.317435]  [ffffea0004000000-ffffea00043fffff] PMD -> [ffff88001cc00000-ffff88001cffffff] on node 0
[    0.319384] init_memory_mapping: [mem 0x140000000-0x15fffffff]
[    0.319387]  [mem 0x140000000-0x15fffffff] page 2M
[    0.324745]  [ffffea0004400000-ffffea00047fffff] PMD -> [ffff88001c800000-ffff88001cbfffff] on node 0
[    0.330871]  [ffffea0004800000-ffffea0004bfffff] PMD -> [ffff88001c400000-ffff88001c7fffff] on node 0
[    0.332037] init_memory_mapping: [mem 0x160000000-0x17fffffff]
[    0.332039]  [mem 0x160000000-0x17fffffff] page 2M
[    0.337378]  [ffffea0004c00000-ffffea0004ffffff] PMD -> [ffff88001c000000-ffff88001c3fffff] on node 0
[    0.342348] init_memory_mapping: [mem 0x180000000-0x19fffffff]
[    0.342351]  [mem 0x180000000-0x19fffffff] page 2M
[    0.344662]  [ffffea0005000000-ffffea00053fffff] PMD -> [ffff88001bc00000-ffff88001bffffff] on node 0
[    0.350783]  [ffffea0005400000-ffffea00057fffff] PMD -> [ffff88001b800000-ffff88001bbfffff] on node 0
[    0.354997] init_memory_mapping: [mem 0x1a0000000-0x1bfffffff]
[    0.355000]  [mem 0x1a0000000-0x1bfffffff] page 2M
[    0.357282]  [ffffea0005800000-ffffea0005bfffff] PMD -> [ffff88001b400000-ffff88001b7fffff] on node 0
[    0.363398]  [ffffea0005c00000-ffffea0005ffffff] PMD -> [ffff88001b000000-ffff88001b3fffff] on node 0
[    0.365342] init_memory_mapping: [mem 0x1c0000000-0x1dfffffff]
[    0.365345]  [mem 0x1c0000000-0x1dfffffff] page 2M
[    0.370653]  [ffffea0006000000-ffffea00063fffff] PMD -> [ffff88001ac00000-ffff88001affffff] on node 0
[    0.376775]  [ffffea0006400000-ffffea00067fffff] PMD -> [ffff88001a800000-ffff88001abfffff] on node 0
[    0.377938] init_memory_mapping: [mem 0x1e0000000-0x1ffffffff]
[    0.377940]  [mem 0x1e0000000-0x1ffffffff] page 2M
[    0.383289]  [ffffea0006800000-ffffea0006bfffff] PMD -> [ffff88001a400000-ffff88001a7fffff] on node 0
[    0.388294] init_memory_mapping: [mem 0x200000000-0x21fffffff]
[    0.388297]  [mem 0x200000000-0x21fffffff] page 2M
[    0.390566]  [ffffea0006c00000-ffffea0006ffffff] PMD -> [ffff88001a000000-ffff88001a3fffff] on node 0
[    0.396736]  [ffffea0007000000-ffffea00073fffff] PMD -> [ffff880019c00000-ffff880019ffffff] on node 0
[    0.400940] init_memory_mapping: [mem 0x220000000-0x23fffffff]
[    0.400943]  [mem 0x220000000-0x23fffffff] page 2M
[    0.403223]  [ffffea0007400000-ffffea00077fffff] PMD -> [ffff880019800000-ffff880019bfffff] on node 0
[    0.409343]  [ffffea0007800000-ffffea0007bfffff] PMD -> [ffff880019400000-ffff8800197fffff] on node 0
[    0.411285] init_memory_mapping: [mem 0x240000000-0x25fffffff]
[    0.411288]  [mem 0x240000000-0x25fffffff] page 2M
[    0.416637]  [ffffea0007c00000-ffffea0007ffffff] PMD -> [ffff880019000000-ffff8800193fffff] on node 0
[    0.422727]  [ffffea0008000000-ffffea00083fffff] PMD -> [ffff880018c00000-ffff880018ffffff] on node 0
[    0.423881] init_memory_mapping: [mem 0x260000000-0x27fffffff]
[    0.423884]  [mem 0x260000000-0x27fffffff] page 2M
[    0.429223]  [ffffea0008400000-ffffea00087fffff] PMD -> [ffff880018800000-ffff880018bfffff] on node 0
[    0.434221] init_memory_mapping: [mem 0x280000000-0x29fffffff]
[    0.434224]  [mem 0x280000000-0x29fffffff] page 2M
[    0.436492]  [ffffea0008800000-ffffea0008bfffff] PMD -> [ffff880018400000-ffff8800187fffff] on node 0
[    0.442594]  [ffffea0008c00000-ffffea0008ffffff] PMD -> [ffff880018000000-ffff8800183fffff] on node 0
[    0.446799] init_memory_mapping: [mem 0x2a0000000-0x2bfffffff]
[    0.446802]  [mem 0x2a0000000-0x2bfffffff] page 2M
[    0.449112]  [ffffea0009000000-ffffea00093fffff] PMD -> [ffff880017c00000-ffff880017ffffff] on node 0
[    0.455257]  [ffffea0009400000-ffffea00097fffff] PMD -> [ffff880017800000-ffff880017bfffff] on node 0
[    0.457206] init_memory_mapping: [mem 0x2c0000000-0x2dfffffff]
[    0.457209]  [mem 0x2c0000000-0x2dfffffff] page 2M
[    0.462530]  [ffffea0009800000-ffffea0009bfffff] PMD -> [ffff880017400000-ffff8800177fffff] on node 0
[    0.468670]  [ffffea0009c00000-ffffea0009ffffff] PMD -> [ffff880017000000-ffff8800173fffff] on node 0
[    0.469823] init_memory_mapping: [mem 0x2e0000000-0x2ffffffff]
[    0.469826]  [mem 0x2e0000000-0x2ffffffff] page 2M
[    0.475118]  [ffffea000a000000-ffffea000a3fffff] PMD -> [ffff880016c00000-ffff880016ffffff] on node 0
[    0.480181] init_memory_mapping: [mem 0x300000000-0x31fffffff]
[    0.480184]  [mem 0x300000000-0x31fffffff] page 2M
[    0.482467]  [ffffea000a400000-ffffea000a7fffff] PMD -> [ffff880016800000-ffff880016bfffff] on node 0
[    0.488661]  [ffffea000a800000-ffffea000abfffff] PMD -> [ffff880016400000-ffff8800167fffff] on node 0
[    0.492875] init_memory_mapping: [mem 0x320000000-0x33fffffff]
[    0.492878]  [mem 0x320000000-0x33fffffff] page 2M
[    0.495167]  [ffffea000ac00000-ffffea000affffff] PMD -> [ffff880016000000-ffff8800163fffff] on node 0
[    0.501307]  [ffffea000b000000-ffffea000b3fffff] PMD -> [ffff880015c00000-ffff880015ffffff] on node 0
[    0.503260] init_memory_mapping: [mem 0x340000000-0x35fffffff]
[    0.503263]  [mem 0x340000000-0x35fffffff] page 2M
[    0.508605]  [ffffea000b400000-ffffea000b7fffff] PMD -> [ffff880015800000-ffff880015bfffff] on node 0
[    0.514731]  [ffffea000b800000-ffffea000bbfffff] PMD -> [ffff880015400000-ffff8800157fffff] on node 0
[    0.515880] init_memory_mapping: [mem 0x360000000-0x37fffffff]
[    0.515883]  [mem 0x360000000-0x37fffffff] page 2M
[    0.521190]  [ffffea000bc00000-ffffea000bffffff] PMD -> [ffff880015000000-ffff8800153fffff] on node 0
[    0.526201] init_memory_mapping: [mem 0x380000000-0x39fffffff]
[    0.526204]  [mem 0x380000000-0x39fffffff] page 2M
[    0.528493]  [ffffea000c000000-ffffea000c3fffff] PMD -> [ffff880014c00000-ffff880014ffffff] on node 0
[    0.534595]  [ffffea000c400000-ffffea000c7fffff] PMD -> [ffff880014800000-ffff880014bfffff] on node 0
[    0.538803] init_memory_mapping: [mem 0x3a0000000-0x3bfffffff]
[    0.538806]  [mem 0x3a0000000-0x3bfffffff] page 2M
[    0.541118]  [ffffea000c800000-ffffea000cbfffff] PMD -> [ffff880014400000-ffff8800147fffff] on node 0
[    0.547272]  [ffffea000cc00000-ffffea000cffffff] PMD -> [ffff880014000000-ffff8800143fffff] on node 0
[    0.549219] init_memory_mapping: [mem 0x3c0000000-0x3dfffffff]
[    0.549222]  [mem 0x3c0000000-0x3dfffffff] page 2M
[    0.554564]  [ffffea000d000000-ffffea000d3fffff] PMD -> [ffff880013c00000-ffff880013ffffff] on node 0
[    0.560686]  [ffffea000d400000-ffffea000d7fffff] PMD -> [ffff880013800000-ffff880013bfffff] on node 0
[    0.561835] init_memory_mapping: [mem 0x3e0000000-0x3ffffffff]
[    0.561837]  [mem 0x3e0000000-0x3ffffffff] page 2M
[    0.567128]  [ffffea000d800000-ffffea000dbfffff] PMD -> [ffff880013400000-ffff8800137fffff] on node 0
[    0.572102] init_memory_mapping: [mem 0x400000000-0x41fffffff]
[    0.572105]  [mem 0x400000000-0x41fffffff] page 2M
[    0.574361]  [ffffea000dc00000-ffffea000dffffff] PMD -> [ffff880013000000-ffff8800133fffff] on node 0
[    0.580439]  [ffffea000e000000-ffffea000e3fffff] PMD -> [ffff880012c00000-ffff880012ffffff] on node 0
[    0.584652] init_memory_mapping: [mem 0x420000000-0x43fffffff]
[    0.584655]  [mem 0x420000000-0x43fffffff] page 2M
[    0.586870]  [ffffea000e400000-ffffea000e7fffff] PMD -> [ffff880012800000-ffff880012bfffff] on node 0
[    0.592935]  [ffffea000e800000-ffffea000ebfffff] PMD -> [ffff880012400000-ffff8800127fffff] on node 0
[    0.594876] init_memory_mapping: [mem 0x440000000-0x45fffffff]
[    0.594879]  [mem 0x440000000-0x45fffffff] page 2M
[    0.600120]  [ffffea000ec00000-ffffea000effffff] PMD -> [ffff880012000000-ffff8800123fffff] on node 0
[    0.606152]  [ffffea000f000000-ffffea000f3fffff] PMD -> [ffff880011c00000-ffff880011ffffff] on node 0
[    0.607313] init_memory_mapping: [mem 0x460000000-0x47fffffff]
[    0.607315]  [mem 0x460000000-0x47fffffff] page 2M
[    0.612601]  [ffffea000f400000-ffffea000f7fffff] PMD -> [ffff880011800000-ffff880011bfffff] on node 0
[    0.617570] init_memory_mapping: [mem 0x480000000-0x49fffffff]
[    0.617573]  [mem 0x480000000-0x49fffffff] page 2M
[    0.619827]  [ffffea000f800000-ffffea000fbfffff] PMD -> [ffff880011400000-ffff8800117fffff] on node 0
[    0.625878]  [ffffea000fc00000-ffffea000fffffff] PMD -> [ffff880011000000-ffff8800113fffff] on node 0
[    0.630058] init_memory_mapping: [mem 0x4a0000000-0x4bfffffff]
[    0.630061]  [mem 0x4a0000000-0x4bfffffff] page 2M
[    0.632309]  [ffffea0010000000-ffffea00103fffff] PMD -> [ffff880010c00000-ffff880010ffffff] on node 0
[    0.638371]  [ffffea0010400000-ffffea00107fffff] PMD -> [ffff880010800000-ffff880010bfffff] on node 0
[    0.640341] init_memory_mapping: [mem 0x4c0000000-0x4dfffffff]
[    0.640344]  [mem 0x4c0000000-0x4dfffffff] page 2M
[    0.645651]  [ffffea0010800000-ffffea0010bfffff] PMD -> [ffff880010400000-ffff8800107fffff] on node 0
[    0.651737]  [ffffea0010c00000-ffffea0010ffffff] PMD -> [ffff880010000000-ffff8800103fffff] on node 0


[-- Attachment #4: 4G-hotplug-12G.txt --]
[-- Type: text/plain, Size: 13650 bytes --]

4G (7 dimms populated, then 24 are plugged and onlined by udev)
[    0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff]
[    0.000000] SRAT: Node 0 PXM 0 [mem 0x00100000-0x1fffffff]
[    0.000000] SRAT: Node 0 PXM 0 [mem 0x100000000-0xc9fffffff] hotplug
[    0.000000] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x1ffdffff] -> [mem 0x00000000-0x1ffdffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x1ffdffff]
[    0.000000]   NODE_DATA [mem 0x1ffdb000-0x1ffdffff]
[    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[    0.000000] kvm-clock: cpu 0, msr 0:1ffd3001, primary cpu clock
[    0.000000]  [ffffea0000000000-ffffea00007fffff] PMD -> [ffff88001e000000-ffff88001e7fffff] on node 0
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0009efff]
[    0.000000]   node   0: [mem 0x00100000-0x1ffdffff]
[    0.000000] On node 0 totalpages: 130942
[    0.000000]   DMA zone: 56 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3998 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 1736 pages used for memmap
[    0.000000]   DMA32 zone: 126944 pages, LIFO batch:31
[    0.271850] init_memory_mapping: [mem 0x100000000-0x11fffffff]
[    0.271854]  [mem 0x100000000-0x11fffffff] page 2M
[    0.280953]  [ffffea0003800000-ffffea0003bfffff] PMD -> [ffff88001d400000-ffff88001d7fffff] on node 0
[    0.285504] init_memory_mapping: [mem 0x120000000-0x13fffffff]
[    0.285509]  [mem 0x120000000-0x13fffffff] page 2M
[    0.287933]  [ffffea0003c00000-ffffea0003ffffff] PMD -> [ffff88001d000000-ffff88001d3fffff] on node 0
[    0.294339]  [ffffea0004000000-ffffea00043fffff] PMD -> [ffff88001cc00000-ffff88001cffffff] on node 0
[    0.296475] init_memory_mapping: [mem 0x140000000-0x15fffffff]
[    0.296479]  [mem 0x140000000-0x15fffffff] page 2M
[    0.302158]  [ffffea0004400000-ffffea00047fffff] PMD -> [ffff88001c800000-ffff88001cbfffff] on node 0
[    0.308651]  [ffffea0004800000-ffffea0004bfffff] PMD -> [ffff88001c400000-ffff88001c7fffff] on node 0
[    0.309918] init_memory_mapping: [mem 0x160000000-0x17fffffff]
[    0.309922]  [mem 0x160000000-0x17fffffff] page 2M
[    0.315531]  [ffffea0004c00000-ffffea0004ffffff] PMD -> [ffff88001c000000-ffff88001c3fffff] on node 0
[    0.320874] init_memory_mapping: [mem 0x180000000-0x19fffffff]
[    0.320878]  [mem 0x180000000-0x19fffffff] page 2M
[    0.323320]  [ffffea0005000000-ffffea00053fffff] PMD -> [ffff88001bc00000-ffff88001bffffff] on node 0
[    0.329845]  [ffffea0005400000-ffffea00057fffff] PMD -> [ffff88001b800000-ffff88001bbfffff] on node 0
[    0.334444] init_memory_mapping: [mem 0x1a0000000-0x1bfffffff]
[    0.334447]  [mem 0x1a0000000-0x1bfffffff] page 2M
[    0.336965]  [ffffea0005800000-ffffea0005bfffff] PMD -> [ffff88001b400000-ffff88001b7fffff] on node 0
[    0.343517]  [ffffea0005c00000-ffffea0005ffffff] PMD -> [ffff88001b000000-ffff88001b3fffff] on node 0
[    0.345663] init_memory_mapping: [mem 0x1c0000000-0x1dfffffff]
[    0.345667]  [mem 0x1c0000000-0x1dfffffff] page 2M
[    0.351333]  [ffffea0006000000-ffffea00063fffff] PMD -> [ffff88001ac00000-ffff88001affffff] on node 0
[    0.357778]  [ffffea0006400000-ffffea00067fffff] PMD -> [ffff88001a800000-ffff88001abfffff] on node 0
...
[  102.518123] acpi PNP0C80:00: Already enumerated
[  102.521254] acpi PNP0C80:01: Already enumerated
[  102.523272] acpi PNP0C80:02: Already enumerated
[  102.526427] acpi PNP0C80:03: Already enumerated
[  102.532864] acpi PNP0C80:04: Already enumerated
[  102.535744] acpi PNP0C80:05: Already enumerated
[  102.539084] acpi PNP0C80:06: Already enumerated
[  102.599661] init_memory_mapping: [mem 0x1e0000000-0x1ffffffff]
[  102.603111]  [mem 0x1e0000000-0x1ffffffff] page 2M
[  102.605778]  [ffffea0006800000-ffffea00069fffff] PMD -> [ffff88001a400000-ffff88001a5fffff] on node 0
[  102.607825]  [ffffea0006a00000-ffffea0006bfffff] PMD -> [ffff88015f000000-ffff88015f1fffff] on node 0
[  102.612527]  [ffffea0006c00000-ffffea0006dfffff] PMD -> [ffff880018c00000-ffff880018dfffff] on node 0
[  102.682492] init_memory_mapping: [mem 0x200000000-0x21fffffff]
[  102.686115]  [mem 0x200000000-0x21fffffff] page 2M
[  102.688712]  [ffffea0006e00000-ffffea0006ffffff] PMD -> [ffff88015f200000-ffff88015f3fffff] on node 0
[  102.690567]  [ffffea0007000000-ffffea00071fffff] PMD -> [ffff88015ec00000-ffff88015edfffff] on node 0
[  102.694309]  [ffffea0007200000-ffffea00073fffff] PMD -> [ffff880018800000-ffff8800189fffff] on node 0
[  102.697784]  [ffffea0007400000-ffffea00075fffff] PMD -> [ffff88015ee00000-ffff88015effffff] on node 0
[  102.741802] init_memory_mapping: [mem 0x220000000-0x23fffffff]
[  102.746035]  [mem 0x220000000-0x23fffffff] page 2M
[  102.750317]  [ffffea0007600000-ffffea00079fffff] PMD -> [ffff88015e800000-ffff88015ebfffff] on node 0
[  102.753983]  [ffffea0007a00000-ffffea0007bfffff] PMD -> [ffff880019400000-ffff8800195fffff] on node 0
[  102.757100] init_memory_mapping: [mem 0x240000000-0x25fffffff]
[  102.794016]  [mem 0x240000000-0x25fffffff] page 2M
[  102.798456]  [ffffea0007c00000-ffffea0007ffffff] PMD -> [ffff88015e400000-ffff88015e7fffff] on node 0
[  102.801853]  [ffffea0008000000-ffffea00081fffff] PMD -> [ffff880019600000-ffff8800197fffff] on node 0
[  102.807243] init_memory_mapping: [mem 0x260000000-0x27fffffff]
[  102.841825]  [mem 0x260000000-0x27fffffff] page 2M
[  102.844362]  [ffffea0008200000-ffffea00085fffff] PMD -> [ffff88015e000000-ffff88015e3fffff] on node 0
[  102.846513]  [ffffea0008600000-ffffea00087fffff] PMD -> [ffff88015de00000-ffff88015dffffff] on node 0
[  102.849983]  [ffffea0008800000-ffffea00089fffff] PMD -> [ffff880018400000-ffff8800185fffff] on node 0
[  102.853676] init_memory_mapping: [mem 0x280000000-0x29fffffff]
[  102.858936]  [mem 0x280000000-0x29fffffff] page 2M
[  102.863144]  [ffffea0008a00000-ffffea0008dfffff] PMD -> [ffff88015d800000-ffff88015dbfffff] on node 0
[  102.866493]  [ffffea0008e00000-ffffea0008ffffff] PMD -> [ffff880018600000-ffff8800187fffff] on node 0
[  102.965377] init_memory_mapping: [mem 0x2a0000000-0x2bfffffff]
[  102.969629]  [mem 0x2a0000000-0x2bfffffff] page 2M
[  102.972637]  [ffffea0009000000-ffffea00093fffff] PMD -> [ffff88015d400000-ffff88015d7fffff] on node 0
[  102.976852]  [ffffea0009400000-ffffea00095fffff] PMD -> [ffff88015d000000-ffff88015d1fffff] on node 0
[  102.980431]  [ffffea0009600000-ffffea00097fffff] PMD -> [ffff880014800000-ffff8800149fffff] on node 0
[  103.058514] init_memory_mapping: [mem 0x2c0000000-0x2dfffffff]
[  103.062312]  [mem 0x2c0000000-0x2dfffffff] page 2M
[  103.068583]  [ffffea0009800000-ffffea0009bfffff] PMD -> [ffff88015cc00000-ffff88015cffffff] on node 0
[  103.071846]  [ffffea0009c00000-ffffea0009dfffff] PMD -> [ffff880014a00000-ffff880014bfffff] on node 0
[  103.166436] init_memory_mapping: [mem 0x2e0000000-0x2ffffffff]
[  103.170236]  [mem 0x2e0000000-0x2ffffffff] page 2M
[  103.172696]  [ffffea0009e00000-ffffea000a1fffff] PMD -> [ffff88015c800000-ffff88015cbfffff] on node 0
[  103.176613]  [ffffea000a200000-ffffea000a3fffff] PMD -> [ffff88015c400000-ffff88015c5fffff] on node 0
[  103.181122]  [ffffea000a400000-ffffea000a5fffff] PMD -> [ffff880014400000-ffff8800145fffff] on node 0
[  103.246497] init_memory_mapping: [mem 0x300000000-0x31fffffff]
[  103.250685]  [mem 0x300000000-0x31fffffff] page 2M
[  103.253605]  [ffffea000a600000-ffffea000a7fffff] PMD -> [ffff88015c600000-ffff88015c7fffff] on node 0
[  103.257327]  [ffffea000a800000-ffffea000a9fffff] PMD -> [ffff880014600000-ffff8800147fffff] on node 0
[  103.260699]  [ffffea000aa00000-ffffea000abfffff] PMD -> [ffff88015c200000-ffff88015c3fffff] on node 0
[  103.318340] init_memory_mapping: [mem 0x320000000-0x33fffffff]
[  103.323420]  [mem 0x320000000-0x33fffffff] page 2M
[  103.326588]  [ffffea000ac00000-ffffea000affffff] PMD -> [ffff88015bc00000-ffff88015bffffff] on node 0
[  103.330788]  [ffffea000b000000-ffffea000b1fffff] PMD -> [ffff88015b800000-ffff88015b9fffff] on node 0
[  103.334163]  [ffffea000b200000-ffffea000b3fffff] PMD -> [ffff880014000000-ffff8800141fffff] on node 0
[  103.442482] init_memory_mapping: [mem 0x340000000-0x35fffffff]
[  103.446194]  [mem 0x340000000-0x35fffffff] page 2M
[  103.448630]  [ffffea000b400000-ffffea000b5fffff] PMD -> [ffff88015ba00000-ffff88015bbfffff] on node 0
[  103.452409]  [ffffea000b600000-ffffea000b7fffff] PMD -> [ffff88015b600000-ffff88015b7fffff] on node 0
[  103.455739]  [ffffea000b800000-ffffea000b9fffff] PMD -> [ffff880014200000-ffff8800143fffff] on node 0
[  103.494067] init_memory_mapping: [mem 0x360000000-0x37fffffff]
[  103.498402]  [mem 0x360000000-0x37fffffff] page 2M
[  103.500871]  [ffffea000ba00000-ffffea000bdfffff] PMD -> [ffff88015b000000-ffff88015b3fffff] on node 0
[  103.505072]  [ffffea000be00000-ffffea000bffffff] PMD -> [ffff88015ac00000-ffff88015adfffff] on node 0
[  103.508488]  [ffffea000c000000-ffffea000c1fffff] PMD -> [ffff880013c00000-ffff880013dfffff] on node 0
[  103.546396] init_memory_mapping: [mem 0x380000000-0x39fffffff]
[  103.550058]  [mem 0x380000000-0x39fffffff] page 2M
[  103.552676]  [ffffea000c200000-ffffea000c3fffff] PMD -> [ffff88015ae00000-ffff88015affffff] on node 0
[  103.556587]  [ffffea000c400000-ffffea000c5fffff] PMD -> [ffff880013e00000-ffff880013ffffff] on node 0
[  103.559993]  [ffffea000c600000-ffffea000c7fffff] PMD -> [ffff88015aa00000-ffff88015abfffff] on node 0
[  103.636973] min_free_kbytes is not updated to 8282 because user defined value 12288 is preferred
[  103.643535] init_memory_mapping: [mem 0x3a0000000-0x3bfffffff]
[  103.647995]  [mem 0x3a0000000-0x3bfffffff] page 2M
[  103.650479]  [ffffea000c800000-ffffea000cbfffff] PMD -> [ffff88015a400000-ffff88015a7fffff] on node 0
[  103.657814]  [ffffea000cc00000-ffffea000cffffff] PMD -> [ffff8801ff800000-ffff8801ffbfffff] on node 0
[  103.722520] init_memory_mapping: [mem 0x3c0000000-0x3dfffffff]
[  103.726238]  [mem 0x3c0000000-0x3dfffffff] page 2M
[  103.728877]  [ffffea000d000000-ffffea000d1fffff] PMD -> [ffff8801ff400000-ffff8801ff5fffff] on node 0
[  103.732895]  [ffffea000d200000-ffffea000d3fffff] PMD -> [ffff8801ff000000-ffff8801ff1fffff] on node 0
[  103.736415]  [ffffea000d400000-ffffea000d5fffff] PMD -> [ffff880013800000-ffff8800139fffff] on node 0
[  103.739990]  [ffffea000d600000-ffffea000d7fffff] PMD -> [ffff8801ff200000-ffff8801ff3fffff] on node 0
[  103.780874] min_free_kbytes is not updated to 8408 because user defined value 12288 is preferred
[  103.792279] min_free_kbytes is not updated to 8532 because user defined value 12288 is preferred
[  103.833842] min_free_kbytes is not updated to 8654 because user defined value 12288 is preferred
[  103.842111] init_memory_mapping: [mem 0x3e0000000-0x3ffffffff]
[  103.895028]  [mem 0x3e0000000-0x3ffffffff] page 2M
[  103.901457]  [ffffea000d800000-ffffea000dbfffff] PMD -> [ffff8801fec00000-ffff8801feffffff] on node 0
[  103.966197] init_memory_mapping: [mem 0x400000000-0x41fffffff]
[  103.970518]  [mem 0x400000000-0x41fffffff] page 2M
[  103.972993]  [ffffea000dc00000-ffffea000dffffff] PMD -> [ffff880247800000-ffff880247bfffff] on node 0
[  103.980076]  [ffffea000e000000-ffffea000e3fffff] PMD -> [ffff880247400000-ffff8802477fffff] on node 0
[  104.050325] init_memory_mapping: [mem 0x420000000-0x43fffffff]
[  104.054193]  [mem 0x420000000-0x43fffffff] page 2M
[  104.056667]  [ffffea000e400000-ffffea000e7fffff] PMD -> [ffff880247000000-ffff8802473fffff] on node 0
[  104.060685]  [ffffea000e800000-ffffea000e9fffff] PMD -> [ffff880246c00000-ffff880246dfffff] on node 0
[  104.064064]  [ffffea000ea00000-ffffea000ebfffff] PMD -> [ffff880013a00000-ffff880013bfffff] on node 0
[  104.109937] init_memory_mapping: [mem 0x440000000-0x45fffffff]
[  104.114568]  [mem 0x440000000-0x45fffffff] page 2M
[  104.117218]  [ffffea000ec00000-ffffea000edfffff] PMD -> [ffff880246e00000-ffff880246ffffff] on node 0
[  104.124499]  [ffffea000ee00000-ffffea000f1fffff] PMD -> [ffff880246800000-ffff880246bfffff] on node 0
[  104.182762] init_memory_mapping: [mem 0x460000000-0x47fffffff]
[  104.186862]  [mem 0x460000000-0x47fffffff] page 2M
[  104.189403]  [ffffea000f200000-ffffea000f5fffff] PMD -> [ffff880246400000-ffff8802467fffff] on node 0
[  104.193322]  [ffffea000f600000-ffffea000f7fffff] PMD -> [ffff880246200000-ffff8802463fffff] on node 0
[  104.196818]  [ffffea000f800000-ffffea000f9fffff] PMD -> [ffff880013400000-ffff8800135fffff] on node 0
[  104.245925] init_memory_mapping: [mem 0x480000000-0x49fffffff]
[  104.251035]  [mem 0x480000000-0x49fffffff] page 2M
[  104.257012]  [ffffea000fa00000-ffffea000fdfffff] PMD -> [ffff880245c00000-ffff880245ffffff] on node 0
[  104.263878]  [ffffea000fe00000-ffffea00101fffff] PMD -> [ffff880245800000-ffff880245bfffff] on node 0
[  104.265623] init_memory_mapping: [mem 0x4a0000000-0x4bfffffff]
[  104.267831]  [mem 0x4a0000000-0x4bfffffff] page 2M
[  104.273894]  [ffffea0010200000-ffffea00105fffff] PMD -> [ffff880245400000-ffff8802457fffff] on node 0
[  104.280261] init_memory_mapping: [mem 0x4c0000000-0x4dfffffff]
[  104.282361]  [mem 0x4c0000000-0x4dfffffff] page 2M
[  104.285066]  [ffffea0010600000-ffffea00109fffff] PMD -> [ffff880245000000-ffff8802453fffff] on node 0
[  104.288570]  [ffffea0010a00000-ffffea0010bfffff] PMD -> [ffff880013600000-ffff8800137fffff] on node 0
[  104.292091]  [ffffea0010c00000-ffffea0010dfffff] PMD -> [ffff880244e00000-ffff880244ffffff] on node 0

Re: [Qemu-devel] Runtime-modified DIMMs and live migration issue

[Andrey Korolyov]

>
> Please find the full cli args and two guest logs for DIMM
> initalization attached. As you can see, the freshly populated DIMMs
> are probably misplaced in SRAT ('already populated' messages), despite
> the fact that the initialized ranges are looking correct at a glance.
> When VM is migrated to the destination (with equal RAM device
> configuration) which is simular to a VM with 16G RAM this misplacement
> is causing the mentioned panic in the guest. This should be very
> easily reproducible and the issue is very critical as well, I don`t
> even understand why I missed this issue earlier.

Answering back to myself - I made a wrong statement before, the
physical mapping *are* different with different cases, of course!
Therefore, the issue looks much simpler and I`d have a patch over a
couple of days if nobody fix this earlier.

[  102.757100] init_memory_mapping: [mem 0x240000000-0x25fffffff]
[  102.794016]  [mem 0x240000000-0x25fffffff] page 2M
[  102.798456]  [ffffea0007c00000-ffffea0007ffffff] PMD ->
[ffff88015e400000-ffff88015e7fffff] on node 0
[  102.801853]  [ffffea0008000000-ffffea00081fffff] PMD ->
[ffff880019600000-ffff8800197fffff] on node 0

vs

[    0.411285] init_memory_mapping: [mem 0x240000000-0x25fffffff]
[    0.411288]  [mem 0x240000000-0x25fffffff] page 2M
[    0.416637]  [ffffea0007c00000-ffffea0007ffffff] PMD ->
[ffff880019000000-ffff8800193fffff] on node 0
[    0.422727]  [ffffea0008000000-ffffea00083fffff] PMD ->
[ffff880018c00000-ffff880018ffffff] on node 0

Re: [Qemu-devel] Runtime-modified DIMMs and live migration issue

[Andrey Korolyov]

> Answering back to myself - I made a wrong statement before, the
> physical mapping *are* different with different cases, of course!
> Therefore, the issue looks much simpler and I`d have a patch over a
> couple of days if nobody fix this earlier.
>

... and another (possibly last) update. This is not a memory
misplacement but a quirky race - if no internal workload applied to
the virtual machine during migration, no harm is done - after its
completion the VM passes all imaginable tests as well. If no device
I/O is involved (perf bench numa for 1000s), guest is not falling with
guts out as with disk-test case, just crashing a process instead:

[  784.613032] thread 0/0[2750]: segfault at 0 ip           (null) sp
00007fda4ea6a940 error 14 in perf_3.16[400000+144000]

I think we are facing a very interesting memory access race during a
live migration but there are no visible reasons for it to be bound
only for a runtime-plugged memory case. All possible cases where
either userspace or kernel driver are involved showing null as an
instruction pointer for trace, may be this can be a hint for someone.

Re: [Qemu-devel] Runtime-modified DIMMs and live migration issue

[Igor Mammedov]

On Tue, 16 Jun 2015 17:41:03 +0300
Andrey Korolyov <andrey@xdel.ru> wrote:

> > Answering back to myself - I made a wrong statement before, the
> > physical mapping *are* different with different cases, of course!
> > Therefore, the issue looks much simpler and I`d have a patch over a
> > couple of days if nobody fix this earlier.
> >
> 
> ... and another (possibly last) update. This is not a memory
> misplacement but a quirky race - if no internal workload applied to
> the virtual machine during migration, no harm is done - after its
> completion the VM passes all imaginable tests as well. If no device
> I/O is involved (perf bench numa for 1000s), guest is not falling with
> guts out as with disk-test case, just crashing a process instead:
> 
> [  784.613032] thread 0/0[2750]: segfault at 0 ip           (null) sp
> 00007fda4ea6a940 error 14 in perf_3.16[400000+144000]
> 
> I think we are facing a very interesting memory access race during a
> live migration but there are no visible reasons for it to be bound
> only for a runtime-plugged memory case. All possible cases where
> either userspace or kernel driver are involved showing null as an
> instruction pointer for trace, may be this can be a hint for someone.

I've checked logs, so far I don't see anything suspicious there
except of "acpi PNP0C80:00: Already enumerated" lines,
probably rising log level might show more info
 + upload full logs
 + enable ACPI debug info to so that dimm device's _CRS would show up
 + QEMU's CLI that was used to produce such log

wrt migration:
could you provide exact CLI args on source and destination along with
used intermediate mem hotplug commands or even better if it's just
reproduced with migration of cold-plugged dimm-s for simplification
+ steps to reproduce (and guest kernel versions).

Re: [Qemu-devel] Runtime-modified DIMMs and live migration issue

[Andrey Korolyov]

[-- Attachment #1: Type: text/plain, Size: 2550 bytes --]

> I've checked logs, so far I don't see anything suspicious there
> except of "acpi PNP0C80:00: Already enumerated" lines,
> probably rising log level might show more info
>  + upload full logs
>  + enable ACPI debug info to so that dimm device's _CRS would show up
>  + QEMU's CLI that was used to produce such log
>
> wrt migration:
> could you provide exact CLI args on source and destination along with
> used intermediate mem hotplug commands or even better if it's just
> reproduced with migration of cold-plugged dimm-s for simplification
> + steps to reproduce (and guest kernel versions).

Thanks Igor,

I am using 3.10 and 3.16 guest kernels lately, but it seems that the
issue is hitting every OS. Issue is not reproducible with cold-plugged
DIMMs at all which is kinda confusing, bearing in mind race-like
behavior described previously, either the guest kernel is partially
responsible for the issue or its nature will be ultimately weird. You
can borrow full cli arg set from the message containing 'Please find
the full cli args and two guest logs for DIMM' three days ago in this
chain. The destination emulator launch string is identical to source
plus device/object pairs in the args for hotplugged memory; mem
devices are getting onlined automatically via udev script. My
colleague suggested me to disable CONFIG_SPARSEMEM_VMEMMAP to remove
the side mess of printks from sparse hotplug mapping and, as it was
shown with that, there is nothing wrong with per-dimm memory
population map, the runtime and coldplugged maps are identical in this
case.

Another trace with null IP is attached, it is produced by running fio.
The easiest way to set up the test bed and to reproduce the issue is
to launch an attached VM with xml (add disk and optionally framebuffer
for convenience), ripping out two or three dimms, then stop libvirt,
add those dimms back in a runtime config, launch libvirt back, add
those dimms, put the workload on VM and migrate a VM with live flag.
Or, if it would be more acceptable for you, launch bare qemu with some
empty slots, plug appropriate objects and devices in (object_add
memory-backend-ram,id=memX,size=512M,
pc-dimm,id=dimmX,node=0,memdev=memX) and migrate to a receiver with
same dimms added to the args. Please not forget to online dimms in
guest as well.

I don`t think that it could be ACPI-related in any way, instead, it
looks like race in vhost or simular mm-touching mechanism. The
repeated hits you mentioned should be fixed as well indeed, but they
can be barely the reason for this problem.

[-- Attachment #2: fio-trace-no-IP.txt --]
[-- Type: text/plain, Size: 5925 bytes --]

[   76.906896] random: nonblocking pool is initialized
[   89.508346] BUG: unable to handle kernel NULL pointer dereference at           (null)
[   89.511075] IP: [<ffffffffa0150f6b>] mpage_process_page_bufs+0x2b/0x120 [ext4]
[   89.512089] PGD 1b755f067 PUD 167e1e067 PMD 0 
[   89.512089] Oops: 0000 [#1] SMP 
[   89.512089] Modules linked in: nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc netconsole configfs loop crct10dif_pclmul crct10dif_common crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw psmouse gf128mul glue_helper i2c_piix4 virtio_console ablk_helper serio_raw pcspkr parport_pc pvpanic cryptd i2c_core evdev parport processor thermal_sys button ext4 crc16
[   89.530893] mon-agent[2185]: segfault at 10 ip 00007fa05ae592a8 sp 00007fff1f5ed2c0 error 4
[   89.530985]  in libc-2.13.so[7fa05ade1000+181000]

[   89.532418] fio[2710]: segfault at 0 ip 0000000000439970 sp 00007fffa3f4e808 error 6
[   89.532452]  in fio[400000+58000]

[   89.512089]  mbcache
[   89.512089]  jbd2 ata_generic virtio_blk virtio_net floppy crc32c_intel xhci_hcd ata_piix usbcore libata usb_common virtio_pci virtio_ring virtio scsi_mod
[   89.512089] CPU: 4 PID: 2715 Comm: fio Not tainted 3.16.7-ckt9 #1
[   89.512089] Hardware name: SuperMicro Virtual Appliance, BIOS 1.1 
[   89.512089] task: ffff8801b0d00210 ti: ffff8801b3f70000 task.ti: ffff8801b3f70000
[   89.512089] RIP: 0010:[<ffffffffa0150f6b>]  [<ffffffffa0150f6b>] mpage_process_page_bufs+0x2b/0x120 [ext4]
[   89.512089] RSP: 0018:ffff8801b3f73cb8  EFLAGS: 00010213
[   89.512089] RAX: 0000000000000000 RBX: 0000000000007824 RCX: 000000000000000c
[   89.512089] RDX: 0000000000000000 RSI: ffff8804a7c01f98 RDI: ffff8801b3f73e58
[   89.512089] RBP: 000000000000c000 R08: 0000000000000000 R09: 20004c74b3000000
[   89.512089] R10: dfff3b8b601d2cc0 R11: ffff88001ffdbe00 R12: 000000000000bfff
[   89.512089] R13: ffff8801b3f73d88 R14: ffff8801b3f73e58 R15: ffff8800131d2d00
[   89.512089] FS:  00007fba4235e700(0000) GS:ffff88001fd00000(0000) knlGS:0000000000000000
[   89.512089] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   89.512089] CR2: 0000000000000000 CR3: 000000014e9a5000 CR4: 00000000000406e0
[   89.512089] Stack:
[   89.512089]  ffff8800131d2b80 ffff8801b3f73d48 7fffffffffffcbdc ffffffffa0151506
[   89.512089]  0000000000000000 ffff8801b784f600 ffff8801b3f73d20 0000000000000000
[   89.512089]  0000000000000004 000000000000782c 000000000000000e 0000000000000000
[   89.512089] Call Trace:
[   89.512089]  [<ffffffffa0151506>] ? mpage_prepare_extent_to_map+0x1d6/0x280 [ext4]
[   89.512089]  [<ffffffffa0157ebf>] ? ext4_writepages+0x3ef/0xd00 [ext4]
[   89.512089]  [<ffffffff812508ef>] ? security_file_permission+0x2f/0xd0
[   89.512089]  [<ffffffff8114d601>] ? __filemap_fdatawrite_range+0x51/0x60
[   89.512089]  [<ffffffff811507fd>] ? SyS_fadvise64+0x24d/0x260
[   89.512089]  [<ffffffff8154d5cd>] ? system_call_fast_compare_end+0x10/0x15
[   89.512089] Code: 66 66 66 66 90 55 bd 01 00 00 00 53 89 cb 48 83 ec 08 48 8b 07 8b 88 90 00 00 00 d3 e5 48 63 ed 48 03 68 50 48 83 ed 01 48 d3 fd <48> 8b 02 a8 04 0f 85 e7 00 00 00 39 eb 0f 83 a2 00 00 00 48 8b 
[   89.512089] RIP  [<ffffffffa0150f6b>] mpage_process_page_bufs+0x2b/0x120 [ext4]
[   89.512089]  RSP <ffff8801b3f73cb8>
[   89.512089] CR2: 0000000000000000
[   89.609566] ---[ end trace 213ee878070f2ba5 ]---
[   89.610884] ------------[ cut here ]------------
[   89.612246] WARNING: CPU: 4 PID: 2715 at kernel/exit.c:669 do_exit+0x4d/0xa30()
[   89.614252] Modules linked in: nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc netconsole configfs loop crct10dif_pclmul crct10dif_common crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw psmouse gf128mul glue_helper i2c_piix4 virtio_console ablk_helper serio_raw pcspkr parport_pc pvpanic cryptd i2c_core evdev parport processor thermal_sys button ext4 crc16 mbcache jbd2 ata_generic virtio_blk virtio_net floppy crc32c_intel xhci_hcd ata_piix usbcore libata usb_common virtio_pci virtio_ring virtio scsi_mod
[   89.635186] CPU: 4 PID: 2715 Comm: fio Tainted: G      D       3.16.7-ckt9 #1
[   89.637099] Hardware name: SuperMicro Virtual Appliance, BIOS 1.1 
[   89.638779]  0000000000000000 0000000000000009 ffffffff8154731e 0000000000000000
[   89.641533]  ffffffff8106cd0b 0000000000000009 ffff8801b3f73c08 0000000000000296
[   89.644158]  0000000000000000 0000000000000296 ffffffff8106eecd 0000000000000000
[   89.646974] Call Trace:
[   89.647753]  [<ffffffff8154731e>] ? dump_stack+0x41/0x51
[   89.649242]  [<ffffffff8106cd0b>] ? warn_slowpath_common+0x8b/0xc0
[   89.650943]  [<ffffffff8106eecd>] ? do_exit+0x4d/0xa30
[   89.652391]  [<ffffffff815449e6>] ? printk+0x54/0x59
[   89.653824]  [<ffffffff8101775b>] ? oops_end+0x9b/0xe0
[   89.655234]  [<ffffffff81543e2f>] ? no_context+0x2a4/0x2cf
[   89.656773]  [<ffffffff8105bcc3>] ? __do_page_fault+0x423/0x520
[   89.658438]  [<ffffffff812a1952>] ? blk_account_io_start+0x112/0x180
[   89.660150]  [<ffffffff8114f0f9>] ? mempool_alloc+0x69/0x190
[   89.661753]  [<ffffffff811b57bd>] ? mem_cgroup_update_page_stat+0x1d/0x60
[   89.663595]  [<ffffffff8154f618>] ? async_page_fault+0x28/0x30
[   89.665214]  [<ffffffffa0150f6b>] ? mpage_process_page_bufs+0x2b/0x120 [ext4]
[   89.667172]  [<ffffffffa0151030>] ? mpage_process_page_bufs+0xf0/0x120 [ext4]
[   89.669104]  [<ffffffffa0151506>] ? mpage_prepare_extent_to_map+0x1d6/0x280 [ext4]
[   89.671132]  [<ffffffffa0157ebf>] ? ext4_writepages+0x3ef/0xd00 [ext4]
[   89.672909]  [<ffffffff812508ef>] ? security_file_permission+0x2f/0xd0
[   89.674683]  [<ffffffff8114d601>] ? __filemap_fdatawrite_range+0x51/0x60
[   89.676513]  [<ffffffff811507fd>] ? SyS_fadvise64+0x24d/0x260
[   89.678115]  [<ffffffff8154d5cd>] ? system_call_fast_compare_end+0x10/0x15
[   89.679936] ---[ end trace 213ee878070f2ba6 ]---

[-- Attachment #3: sample-vm-for-hotplug.xml --]
[-- Type: text/xml, Size: 7584 bytes --]

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>test</name>
  <memory unit='KiB'>524288</memory>
  <currentMemory unit='KiB'>524288</currentMemory>
  </memtune>
  <os>
    <type arch='x86_64' machine='pc'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic eoi='on'/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
    <vendor>Intel</vendor>
    <topology sockets='1' cores='8' threads='8'/>
    <numa>
      <cell cpus='0-7' memory='524288'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <controller type='usb' index='0' model='nec-xhci'>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
    </controller>
    <serial type='pty'>
      <target type='isa-serial' port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <memballoon model='none'/>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-object'/>
    <qemu:arg value='iothread,id=vm33090blk0'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.config-wce=off'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.scsi=off'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.iothread=vm33090blk0'/>
    <qemu:arg value='-m'/>
    <qemu:arg value='512,slots=31,maxmem=16384M'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem0,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm0,node=0,memdev=mem0'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem1,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm1,node=0,memdev=mem1'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem2,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm2,node=0,memdev=mem2'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem3,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm3,node=0,memdev=mem3'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem4,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm4,node=0,memdev=mem4'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem5,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm5,node=0,memdev=mem5'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem6,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm6,node=0,memdev=mem6'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem7,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm7,node=0,memdev=mem7'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem8,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm8,node=0,memdev=mem8'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem9,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm9,node=0,memdev=mem9'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem10,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm10,node=0,memdev=mem10'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem11,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm11,node=0,memdev=mem11'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem12,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm12,node=0,memdev=mem12'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem13,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm13,node=0,memdev=mem13'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem14,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm14,node=0,memdev=mem14'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem15,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm15,node=0,memdev=mem15'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem17,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm17,node=0,memdev=mem17'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem16,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm16,node=0,memdev=mem16'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem19,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm19,node=0,memdev=mem19'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem18,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm18,node=0,memdev=mem18'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem21,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm21,node=0,memdev=mem21'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem20,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm20,node=0,memdev=mem20'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem23,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm23,node=0,memdev=mem23'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem22,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm22,node=0,memdev=mem22'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem25,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm25,node=0,memdev=mem25'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem24,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm24,node=0,memdev=mem24'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem27,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm27,node=0,memdev=mem27'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem26,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm26,node=0,memdev=mem26'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem29,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm29,node=0,memdev=mem29'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem28,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm28,node=0,memdev=mem28'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-ram,id=mem30,size=512M'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='pc-dimm,id=dimm30,node=0,memdev=mem30'/>
  </qemu:commandline>
</domain>

Re: [Qemu-devel] Runtime-modified DIMMs and live migration issue

[Andrey Korolyov]

[-- Attachment #1: Type: text/plain, Size: 456 bytes --]

> I don`t think that it could be ACPI-related in any way, instead, it
> looks like race in vhost or simular mm-touching mechanism. The
> repeated hits you mentioned should be fixed as well indeed, but they
> can be barely the reason for this problem.

Please find a trace from a single dimm plugging in attached. The
configuration is -m 512 + three 512-Mb dimms on start, then a 512-Mb
dimm is plugged in, the NUMA topology is created with a single node0.

[-- Attachment #2: acpi-hp.txt --]
[-- Type: text/plain, Size: 2256 bytes --]

[   94.751646] 
**** Context Switch from TID 4294967295 to TID 496836688 ****

[   94.755180]     utils-0291 evaluate_integer      : Return value [15]
[   94.757146]       bus-0130 bus_get_status        : Device [MP00] status [0000000f]
[   94.760755] acpi PNP0C80:00: Already enumerated
[   94.762478]     utils-0291 evaluate_integer      : Return value [15]
[   94.767924]       bus-0130 bus_get_status        : Device [MP01] status [0000000f]
[   94.770232] acpi PNP0C80:01: Already enumerated
[   94.771766]     utils-0291 evaluate_integer      : Return value [15]
[   94.773867]       bus-0130 bus_get_status        : Device [MP02] status [0000000f]
[   94.776149] acpi PNP0C80:02: Already enumerated
[   94.777678]     utils-0291 evaluate_integer      : Return value [15]
[   94.779669]       bus-0130 bus_get_status        : Device [MP03] status [0000000f]
[   94.782686]      scan-1144 get_device_data       : No context for object [ffff88001d83fe50]
[   94.785908]      scan-1144 get_device_data       : No context for object [ffff88001d83fe28]
[   94.789194]      scan-1144 get_device_data       : No context for object [ffff88001d83fe00]
[   94.792431]      scan-1144 get_device_data       : No context for object [ffff88001d83fdd8]
[   94.795605]      scan-1144 get_device_data       : No context for object [ffff88001d83fdb0]
[   94.798956]      scan-1144 get_device_data       : No context for object [ffff88001d83fd88]
[   94.802337]     utils-0291 evaluate_integer      : Return value [15]
[   94.804961]       bus-0130 bus_get_status        : Device [MP03] status [0000000f]
[   94.807953] device_pm-0128 device_get_power      : Device [MP03] power state is D0
[   94.811318]     utils-0291 evaluate_integer      : Return value [15]
[   94.814056]     utils-0291 evaluate_integer      : Return value [0]
[   94.816629] init_memory_mapping: [mem 0x160000000-0x17fffffff]
[   94.818714]  [mem 0x160000000-0x17fffffff] page 2M
[   94.819539]  [ffffea0004c00000-ffffea0004dfffff] PMD -> [ffff88001b800000-ffff88001b9fffff] on node 0
[   94.823406]  [ffffea0004e00000-ffffea0004ffffff] PMD -> [ffff88001aa00000-ffff88001abfffff] on node 0
[   94.827062]  [ffffea0005000000-ffffea00051fffff] PMD -> [ffff880000400000-ffff8800005fffff] on node 0

Re: [Qemu-devel] Runtime-modified DIMMs and live migration issue

[Andrey Korolyov]

On Fri, Jun 19, 2015 at 7:57 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
>> I don`t think that it could be ACPI-related in any way, instead, it
>> looks like race in vhost or simular mm-touching mechanism. The
>> repeated hits you mentioned should be fixed as well indeed, but they
>> can be barely the reason for this problem.
>
> Please find a trace from a single dimm plugging in attached. The
> configuration is -m 512 + three 512-Mb dimms on start, then a 512-Mb
> dimm is plugged in, the NUMA topology is created with a single node0.

Tried the same thing without vhost being involved, with same result.
What is interesting, the second and consequent migrations are
successful not regarding if the workload was applied or not during
those migrations. So, the only first migration after DIMM hotplug may
fail (if the DIMMs are plugged separately between migrations, only the
migration+workload following the hotplug event may crash guest
kernel).

Re: [Qemu-devel] Runtime-modified DIMMs and live migration issue

[Andrey Korolyov]

"Fixed" with cherry-pick of the
7a72f7a140bfd3a5dae73088947010bfdbcf6a40 and its predecessor
7103f60de8bed21a0ad5d15d2ad5b7a333dda201. Of course this is not a real
fix as the only race precondition is shifted/disappeared by a clear
assumption. Though there are not too many hotplug users around, I hope
this information would be useful for those who would experience the
same in a next year or so, until 3.18+ will be stable enough for
hypervisor kernel role. Any suggestions on a further debug/race
re-exposition are of course very welcomed.

CCing kvm@ as it looks as a hypervisor subsystem issue then. The
entire discussion can be found at
https://lists.gnu.org/archive/html/qemu-devel/2015-06/msg03117.html .

Re: [Qemu-devel] Runtime-modified DIMMs and live migration issue

[Andrey Korolyov]

On Tue, Aug 18, 2015 at 5:51 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> "Fixed" with cherry-pick of the
> 7a72f7a140bfd3a5dae73088947010bfdbcf6a40 and its predecessor
> 7103f60de8bed21a0ad5d15d2ad5b7a333dda201. Of course this is not a real
> fix as the only race precondition is shifted/disappeared by a clear
> assumption. Though there are not too many hotplug users around, I hope
> this information would be useful for those who would experience the
> same in a next year or so, until 3.18+ will be stable enough for
> hypervisor kernel role. Any suggestions on a further debug/race
> re-exposition are of course very welcomed.
>
> CCing kvm@ as it looks as a hypervisor subsystem issue then. The
> entire discussion can be found at
> https://lists.gnu.org/archive/html/qemu-devel/2015-06/msg03117.html .

So no, the issue is still there, though appearance rate is lower. What
could be interesting, non-smp guests are affected as well, before that
I suspected that the vCPUs has been resumed in a racy manner to
trigger a memory corruption. Also the chance to hit the problem is
increased at least faster than linear with number of plugged DIMMs, at
8G total it is almost impossible to catch the issue for now (which is
better than the state of things at the beginning of this thread) and
at 16G total reproduction has a fairly high rate with active memory
operations.

Migration of the suspended VM resulting in same corruption being seen,
so it is very likely that the core analysis could reveal the root of
the issue, the problem is that I have a zero clues of what exactly
could be wrong there and how this thing could be dependent on a
machine size, if we are not taking race conditions in a view.