Re: [Qemu-devel] Unresponsive linux guest once migrated

["Marcin Gibuła" <m.gibula@beyond.pl>]

> Can you give:
>    1) A backtrace from the guest
>       thread apply all bt full
>       in gdb

You mean from gdb attached to hanged guest? I'll try to get it. From 
what I remember it looks rather "normal" - busy executing guest code.

>    2) What's the earliest/newest qemu versions you've seen this on?

1.4 - 1.6
Don't know about earlier versions because I didn't use migration on 
them. Haven't tried 1.7 yet (I know about XBZRLE fixes, but it happened 
without it as well...).

>    3) What guest OS are you running?

All flavors of Centos, Ubuntu, Redhat, etc. Also Windows. But never seen 
a crash with Windows so far.

Seems that few people who also have this issue, reports success with 
kvmclock disabled (either in qemu or kernel command line).

>    4) What host OS are you running?

Distro is Gentoo based (with no crazy compiler options). I've been using 
kernel 3.4 - 3.10.

>    5) What CPU are you running on?

AMD Opteron(tm) Processor 6164 HE

>    6) What does your qemu command line look like?

Example VM:
/usr/bin/qemu-system-x86_64 -machine accel=kvm -name 
3b5e37ea-04be-4a6b-8d63-f1a5853f2138 -S -machine 
pc-i440fx-1.5,accel=kvm,usb=off -cpu 
qemu64,+misalignsse,+abm,+lahf_lm,+rdtscp,+popcnt,+x2apic,-svm,+kvmclock 
-m 1024 -realtime mlock=on -smp 2,sockets=4,cores=12,threads=1 -uuid 
3b5e37ea-04be-4a6b-8d63-f1a5853f2138 -smbios type=0,vendor=HAL 9000 
-smbios type=1,manufacturer=cloud -no-user-config -nodefaults -chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/3b5e37ea-04be-4a6b-8d63-f1a5853f2138.monitor,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc 
base=utc,clock=vm,driftfix=slew -no-hpet -no-kvm-pit-reinjection 
-no-shutdown -boot menu=off -device 
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device 
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive 
file=/dev/stor1c/2e7fd7aa-8588-47ed-a091-af2b81c9e935,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native,bps_rd=57671680,bps_wr=57671680,iops_rd=275,iops_wr=275 
-device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 
-drive if=none,id=drive-ide0-0-0,readonly=on,format=raw -device 
ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 
-netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=27 -device 
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:11:11:11:11,bus=pci.0,addr=0x3 
-chardev pty,id=charserial0 -device 
isa-serial,chardev=charserial0,id=serial0 -chardev 
socket,id=charchannel0,path=/var/lib/libvirt/qemu/f16x86_64.agent,server,nowait 
-device 
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1 
-device usb-tablet,id=input0 -vnc 0.0.0.0:4,password -vga cirrus -device 
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -sandbox on

I've tried playing with different CPU model (Opteron_G3) and flags, it 
didn't make any difference.

>    7) How exactly are you migrating?

Via libvirt live migration. Seen it with and without XBZRLE enabled.

>    8) You talk about having to wait a few hours to trigger it - do
>       you have a more exact description of a test?

Yes, that's where it gets weird. I've never seen this on fresh VM. It 
needs to be idle for couple of hours at least. And even then it doesn't 
always hang.

>    9) Is there any output from qemu stderr/stdout in your qemu logs?

Nothing unusual. From QEMU point of view guest is up and running. Only 
its OS is hanged (but not panicked, there is no backtrace,  oops or BUG 
on its screen).

-- 
mg