Re: one out of four existing kvm guest's not starting after system upgrade

[Thomas Fjellstrom <thomas@fjellstrom.ca>]

On Sat Feb 18, 2012, Thomas Fjellstrom wrote:
> On Sat Feb 18, 2012, Jan Kiszka wrote:
> > On 2012-02-18 09:50, Thomas Fjellstrom wrote:
> > > On Sat Feb 18, 2012, Jan Kiszka wrote:
> > >> On 2012-02-18 05:49, Thomas Fjellstrom wrote:
> > >>> I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2,
> > >>> and qemu+qemu-kvm updated (not sure from what to what). But after
> > >>> the upgrade, one of my guests will not start up. It gets stuck with
> > >>> 60-80% cpu use, almost no memory is allocated by qemu/kvm, and no
> > >>> output of any kind is seen (in the host console, or a bunch of the
> > >>> guest output options like curses display, stdio output, virsh
> > >>> console, vnc or sdl output). I normally use libvirt to manage the
> > >>> guests, but I've attempted to run qemu manually, and have the same
> > >>> problems.
> > >>> 
> > >>> What can cause this?
> > >>> 
> > >>> Just tested booting back into the old kernel, the one guest still
> > >>> won't start, while the rest do. I'm thoroughly confused.
> > >> 
> > >> You mean if you only update qemu-kvm, the problem persists, just with
> > >> lower probability? In that case, we definitely need the version of
> > >> your current qemu-kvm installation. Also, it would be nice to attach
> > >> gdb to the stuck qemu-kvm process, issuing a "thread apply all
> > >> backtrace" in that state.
> > >> 
> > >> Jan
> > > 
> > > Sorry I wasn't clear. If I just update qemu-kvm (And qemu with it, and
> > > not the kernel), it always just hangs on load.
> > > 
> > > current version:
> > > QEMU emulator version 1.0 (qemu-kvm-1.0 Debian 1.0+dfsg-8), Copyright
> > > (c) 2003-2008 Fabrice Bellard
> > > 
> > > gdb thread apply all bt:
> > > Thread 2 (Thread 0x7fe7b810c700 (LWP 14650)):
> > > #0  0x00007fe7c0a11957 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6
> > > #1  0x00007fe7c53155e9 in kvm_vcpu_ioctl (env=<optimized out>,
> > > type=<optimized out>) at
> > > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-
> > > all.c:1101
> > > #2  0x00007fe7c5315731 in kvm_cpu_exec (env=0x7fe7c61cb350) at
> > > /build/buildd-
> > > qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-all.c:987 #3
> > > 0x00007fe7c52ecf31 in qemu_kvm_cpu_thread_fn (arg=0x7fe7c61cb350) at
> > > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/cpus.c
> > > : 740 #4  0x00007fe7c0ccdb50 in start_thread () from /lib/x86_64-linux-
> > > gnu/libpthread.so.0
> > > #5  0x00007fe7c0a1890d in clone () from /lib/x86_64-linux-gnu/libc.so.6
> > > #6  0x0000000000000000 in ?? ()
> > > 
> > > Thread 1 (Thread 0x7fe7c5101900 (LWP 14648)):
> > > #0  0x00007fe7c0a12403 in select () from
> > > /lib/x86_64-linux-gnu/libc.so.6 #1  0x00007fe7c525b56c in
> > > main_loop_wait (nonblocking=<optimized out>) at
> > > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/main-
> > > loop.c:456
> > > #2  0x00007fe7c51a372f in main_loop () at
> > > /build/buildd-qemu-kvm_1.0+dfsg-8-
> > > amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:1482
> > > #3  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized
> > > out>) at
> > > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:3
> > > 5 23
> > > 
> > > Thanks :)
> > 
> > OK, then we need a kernel view on this. Can you try
> > 
> > http://www.linux-kvm.org/page/Tracing
> > 
> > ?
> > 
> > Thanks,
> > Jan
> 
> It made a rather large trace file. I'm pretty sure no-one wants me to
> actually link (or attach) the full 1.5G file, but the last 5000 lines
> might be useful, so I've compressed and attached them. (but just in case,
> I'm lzma'ing the entire 1.5G trace data file)

I'm pretty much stumped on this. So I decided to try re-creating the vm 
through virt-manager. Its up and running now. The only to major differences I 
can see in the old and new config is the machine (-M pc-0.12 vs -M pc-1.0) 
parameter, and the uuid. The rest of the parameters I played with a lot trying 
to get it to work by starting up the vm manually from the cli. I can't really 
see how those two changes would do much of anything considering the other 
three VM's still are configured to use -M pc-0.12, and they work fine.

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca