Re: one out of four existing kvm guest's not starting after system upgrade

[Thomas Fjellstrom <thomas@fjellstrom.ca>]

[-- Attachment #1: Type: Text/Plain, Size: 3468 bytes --]

On Sat Feb 18, 2012, Jan Kiszka wrote:
> On 2012-02-18 09:50, Thomas Fjellstrom wrote:
> > On Sat Feb 18, 2012, Jan Kiszka wrote:
> >> On 2012-02-18 05:49, Thomas Fjellstrom wrote:
> >>> I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2, and
> >>> qemu+qemu-kvm updated (not sure from what to what). But after the
> >>> upgrade, one of my guests will not start up. It gets stuck with 60-80%
> >>> cpu use, almost no memory is allocated by qemu/kvm, and no output of
> >>> any kind is seen (in the host console, or a bunch of the guest output
> >>> options like curses display, stdio output, virsh console, vnc or sdl
> >>> output). I normally use libvirt to manage the guests, but I've
> >>> attempted to run qemu manually, and have the same problems.
> >>> 
> >>> What can cause this?
> >>> 
> >>> Just tested booting back into the old kernel, the one guest still won't
> >>> start, while the rest do. I'm thoroughly confused.
> >> 
> >> You mean if you only update qemu-kvm, the problem persists, just with
> >> lower probability? In that case, we definitely need the version of your
> >> current qemu-kvm installation. Also, it would be nice to attach gdb to
> >> the stuck qemu-kvm process, issuing a "thread apply all backtrace" in
> >> that state.
> >> 
> >> Jan
> > 
> > Sorry I wasn't clear. If I just update qemu-kvm (And qemu with it, and
> > not the kernel), it always just hangs on load.
> > 
> > current version:
> > QEMU emulator version 1.0 (qemu-kvm-1.0 Debian 1.0+dfsg-8), Copyright (c)
> > 2003-2008 Fabrice Bellard
> > 
> > gdb thread apply all bt:
> > Thread 2 (Thread 0x7fe7b810c700 (LWP 14650)):
> > #0  0x00007fe7c0a11957 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6
> > #1  0x00007fe7c53155e9 in kvm_vcpu_ioctl (env=<optimized out>,
> > type=<optimized out>) at
> > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-
> > all.c:1101
> > #2  0x00007fe7c5315731 in kvm_cpu_exec (env=0x7fe7c61cb350) at
> > /build/buildd-
> > qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-all.c:987 #3 
> > 0x00007fe7c52ecf31 in qemu_kvm_cpu_thread_fn (arg=0x7fe7c61cb350) at
> > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/cpus.c:
> > 740 #4  0x00007fe7c0ccdb50 in start_thread () from /lib/x86_64-linux-
> > gnu/libpthread.so.0
> > #5  0x00007fe7c0a1890d in clone () from /lib/x86_64-linux-gnu/libc.so.6
> > #6  0x0000000000000000 in ?? ()
> > 
> > Thread 1 (Thread 0x7fe7c5101900 (LWP 14648)):
> > #0  0x00007fe7c0a12403 in select () from /lib/x86_64-linux-gnu/libc.so.6
> > #1  0x00007fe7c525b56c in main_loop_wait (nonblocking=<optimized out>) at
> > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/main-
> > loop.c:456
> > #2  0x00007fe7c51a372f in main_loop () at
> > /build/buildd-qemu-kvm_1.0+dfsg-8-
> > amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:1482
> > #3  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized
> > out>) at
> > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:35
> > 23
> > 
> > Thanks :)
> 
> OK, then we need a kernel view on this. Can you try
> 
> http://www.linux-kvm.org/page/Tracing
> 
> ?
> 
> Thanks,
> Jan

It made a rather large trace file. I'm pretty sure no-one wants me to actually 
link (or attach) the full 1.5G file, but the last 5000 lines might be useful, 
so I've compressed and attached them. (but just in case, I'm lzma'ing the 
entire 1.5G trace data file)

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

[-- Attachment #2: trace-cmd.report1-tail.lzma --]
[-- Type: application/x-lzma, Size: 17117 bytes --]