one out of four existing kvm guest's not starting after system upgrade

one out of four existing kvm guest's not starting after system upgrade

[Thomas Fjellstrom]

I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2, and 
qemu+qemu-kvm updated (not sure from what to what). But after the upgrade, one 
of my guests will not start up. It gets stuck with 60-80% cpu use, almost no 
memory is allocated by qemu/kvm, and no output of any kind is seen (in the 
host console, or a bunch of the guest output options like curses display, 
stdio output, virsh console, vnc or sdl output). I normally use libvirt to 
manage the guests, but I've attempted to run qemu manually, and have the same 
problems.

What can cause this? 

Just tested booting back into the old kernel, the one guest still won't start, 
while the rest do. I'm thoroughly confused.

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Re: one out of four existing kvm guest's not starting after system upgrade

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1091 bytes --]

On 2012-02-18 05:49, Thomas Fjellstrom wrote:
> I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2, and 
> qemu+qemu-kvm updated (not sure from what to what). But after the upgrade, one 
> of my guests will not start up. It gets stuck with 60-80% cpu use, almost no 
> memory is allocated by qemu/kvm, and no output of any kind is seen (in the 
> host console, or a bunch of the guest output options like curses display, 
> stdio output, virsh console, vnc or sdl output). I normally use libvirt to 
> manage the guests, but I've attempted to run qemu manually, and have the same 
> problems.
> 
> What can cause this? 
> 
> Just tested booting back into the old kernel, the one guest still won't start, 
> while the rest do. I'm thoroughly confused.

You mean if you only update qemu-kvm, the problem persists, just with
lower probability? In that case, we definitely need the version of your
current qemu-kvm installation. Also, it would be nice to attach gdb to
the stuck qemu-kvm process, issuing a "thread apply all backtrace" in
that state.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: one out of four existing kvm guest's not starting after system upgrade

[Thomas Fjellstrom]

On Sat Feb 18, 2012, Jan Kiszka wrote:
> On 2012-02-18 05:49, Thomas Fjellstrom wrote:
> > I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2, and
> > qemu+qemu-kvm updated (not sure from what to what). But after the
> > upgrade, one of my guests will not start up. It gets stuck with 60-80%
> > cpu use, almost no memory is allocated by qemu/kvm, and no output of any
> > kind is seen (in the host console, or a bunch of the guest output
> > options like curses display, stdio output, virsh console, vnc or sdl
> > output). I normally use libvirt to manage the guests, but I've attempted
> > to run qemu manually, and have the same problems.
> > 
> > What can cause this?
> > 
> > Just tested booting back into the old kernel, the one guest still won't
> > start, while the rest do. I'm thoroughly confused.
> 
> You mean if you only update qemu-kvm, the problem persists, just with
> lower probability? In that case, we definitely need the version of your
> current qemu-kvm installation. Also, it would be nice to attach gdb to
> the stuck qemu-kvm process, issuing a "thread apply all backtrace" in
> that state.
> 
> Jan

Sorry I wasn't clear. If I just update qemu-kvm (And qemu with it, and not the 
kernel), it always just hangs on load. 

current version:
QEMU emulator version 1.0 (qemu-kvm-1.0 Debian 1.0+dfsg-8), Copyright (c) 
2003-2008 Fabrice Bellard

gdb thread apply all bt:
Thread 2 (Thread 0x7fe7b810c700 (LWP 14650)):
#0  0x00007fe7c0a11957 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fe7c53155e9 in kvm_vcpu_ioctl (env=<optimized out>, type=<optimized 
out>) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-
all.c:1101
#2  0x00007fe7c5315731 in kvm_cpu_exec (env=0x7fe7c61cb350) at /build/buildd-
qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-all.c:987
#3  0x00007fe7c52ecf31 in qemu_kvm_cpu_thread_fn (arg=0x7fe7c61cb350) at 
/build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/cpus.c:740
#4  0x00007fe7c0ccdb50 in start_thread () from /lib/x86_64-linux-
gnu/libpthread.so.0
#5  0x00007fe7c0a1890d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fe7c5101900 (LWP 14648)):
#0  0x00007fe7c0a12403 in select () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fe7c525b56c in main_loop_wait (nonblocking=<optimized out>) at 
/build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/main-
loop.c:456
#2  0x00007fe7c51a372f in main_loop () at /build/buildd-qemu-kvm_1.0+dfsg-8-
amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:1482
#3  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at 
/build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:3523

Thanks :)

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Re: one out of four existing kvm guest's not starting after system upgrade

[Thomas Fjellstrom]

On Sat Feb 18, 2012, Thomas Fjellstrom wrote:
> On Sat Feb 18, 2012, Jan Kiszka wrote:
> > On 2012-02-18 05:49, Thomas Fjellstrom wrote:
> > > I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2, and
> > > qemu+qemu-kvm updated (not sure from what to what). But after the
> > > upgrade, one of my guests will not start up. It gets stuck with 60-80%
> > > cpu use, almost no memory is allocated by qemu/kvm, and no output of
> > > any kind is seen (in the host console, or a bunch of the guest output
> > > options like curses display, stdio output, virsh console, vnc or sdl
> > > output). I normally use libvirt to manage the guests, but I've
> > > attempted to run qemu manually, and have the same problems.
> > > 
> > > What can cause this?
> > > 
> > > Just tested booting back into the old kernel, the one guest still won't
> > > start, while the rest do. I'm thoroughly confused.
> > 
> > You mean if you only update qemu-kvm, the problem persists, just with
> > lower probability? In that case, we definitely need the version of your
> > current qemu-kvm installation. Also, it would be nice to attach gdb to
> > the stuck qemu-kvm process, issuing a "thread apply all backtrace" in
> > that state.
> > 
> > Jan
> 
> Sorry I wasn't clear. If I just update qemu-kvm (And qemu with it, and not
> the kernel), it always just hangs on load.

where "it" is the one vm I mentioned before. Three others are just fine. The 
stranger bit, is I'm pretty sure I created them all at the same time, in very 
similar ways. At the very least the libvirt config for them all are near 
identical.

[snip]

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Re: one out of four existing kvm guest's not starting after system upgrade

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3041 bytes --]

On 2012-02-18 09:50, Thomas Fjellstrom wrote:
> On Sat Feb 18, 2012, Jan Kiszka wrote:
>> On 2012-02-18 05:49, Thomas Fjellstrom wrote:
>>> I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2, and
>>> qemu+qemu-kvm updated (not sure from what to what). But after the
>>> upgrade, one of my guests will not start up. It gets stuck with 60-80%
>>> cpu use, almost no memory is allocated by qemu/kvm, and no output of any
>>> kind is seen (in the host console, or a bunch of the guest output
>>> options like curses display, stdio output, virsh console, vnc or sdl
>>> output). I normally use libvirt to manage the guests, but I've attempted
>>> to run qemu manually, and have the same problems.
>>>
>>> What can cause this?
>>>
>>> Just tested booting back into the old kernel, the one guest still won't
>>> start, while the rest do. I'm thoroughly confused.
>>
>> You mean if you only update qemu-kvm, the problem persists, just with
>> lower probability? In that case, we definitely need the version of your
>> current qemu-kvm installation. Also, it would be nice to attach gdb to
>> the stuck qemu-kvm process, issuing a "thread apply all backtrace" in
>> that state.
>>
>> Jan
> 
> Sorry I wasn't clear. If I just update qemu-kvm (And qemu with it, and not the 
> kernel), it always just hangs on load. 
> 
> current version:
> QEMU emulator version 1.0 (qemu-kvm-1.0 Debian 1.0+dfsg-8), Copyright (c) 
> 2003-2008 Fabrice Bellard
> 
> gdb thread apply all bt:
> Thread 2 (Thread 0x7fe7b810c700 (LWP 14650)):
> #0  0x00007fe7c0a11957 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6
> #1  0x00007fe7c53155e9 in kvm_vcpu_ioctl (env=<optimized out>, type=<optimized 
> out>) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-
> all.c:1101
> #2  0x00007fe7c5315731 in kvm_cpu_exec (env=0x7fe7c61cb350) at /build/buildd-
> qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-all.c:987
> #3  0x00007fe7c52ecf31 in qemu_kvm_cpu_thread_fn (arg=0x7fe7c61cb350) at 
> /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/cpus.c:740
> #4  0x00007fe7c0ccdb50 in start_thread () from /lib/x86_64-linux-
> gnu/libpthread.so.0
> #5  0x00007fe7c0a1890d in clone () from /lib/x86_64-linux-gnu/libc.so.6
> #6  0x0000000000000000 in ?? ()
> 
> Thread 1 (Thread 0x7fe7c5101900 (LWP 14648)):
> #0  0x00007fe7c0a12403 in select () from /lib/x86_64-linux-gnu/libc.so.6
> #1  0x00007fe7c525b56c in main_loop_wait (nonblocking=<optimized out>) at 
> /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/main-
> loop.c:456
> #2  0x00007fe7c51a372f in main_loop () at /build/buildd-qemu-kvm_1.0+dfsg-8-
> amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:1482
> #3  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at 
> /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:3523
> 
> Thanks :)
> 

OK, then we need a kernel view on this. Can you try

http://www.linux-kvm.org/page/Tracing

?

Thanks,
Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: one out of four existing kvm guest's not starting after system upgrade

[Thomas Fjellstrom]

On Sat Feb 18, 2012, Jan Kiszka wrote:
> On 2012-02-18 09:50, Thomas Fjellstrom wrote:
> > On Sat Feb 18, 2012, Jan Kiszka wrote:
> >> On 2012-02-18 05:49, Thomas Fjellstrom wrote:
> >>> I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2, and
> >>> qemu+qemu-kvm updated (not sure from what to what). But after the
> >>> upgrade, one of my guests will not start up. It gets stuck with 60-80%
> >>> cpu use, almost no memory is allocated by qemu/kvm, and no output of
> >>> any kind is seen (in the host console, or a bunch of the guest output
> >>> options like curses display, stdio output, virsh console, vnc or sdl
> >>> output). I normally use libvirt to manage the guests, but I've
> >>> attempted to run qemu manually, and have the same problems.
> >>> 
> >>> What can cause this?
> >>> 
> >>> Just tested booting back into the old kernel, the one guest still won't
> >>> start, while the rest do. I'm thoroughly confused.
> >> 
> >> You mean if you only update qemu-kvm, the problem persists, just with
> >> lower probability? In that case, we definitely need the version of your
> >> current qemu-kvm installation. Also, it would be nice to attach gdb to
> >> the stuck qemu-kvm process, issuing a "thread apply all backtrace" in
> >> that state.
> >> 
> >> Jan
> > 
> > Sorry I wasn't clear. If I just update qemu-kvm (And qemu with it, and
> > not the kernel), it always just hangs on load.
> > 
> > current version:
> > QEMU emulator version 1.0 (qemu-kvm-1.0 Debian 1.0+dfsg-8), Copyright (c)
> > 2003-2008 Fabrice Bellard
> > 
> > gdb thread apply all bt:
> > Thread 2 (Thread 0x7fe7b810c700 (LWP 14650)):
> > #0  0x00007fe7c0a11957 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6
> > #1  0x00007fe7c53155e9 in kvm_vcpu_ioctl (env=<optimized out>,
> > type=<optimized out>) at
> > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-
> > all.c:1101
> > #2  0x00007fe7c5315731 in kvm_cpu_exec (env=0x7fe7c61cb350) at
> > /build/buildd-
> > qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-all.c:987 #3 
> > 0x00007fe7c52ecf31 in qemu_kvm_cpu_thread_fn (arg=0x7fe7c61cb350) at
> > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/cpus.c:
> > 740 #4  0x00007fe7c0ccdb50 in start_thread () from /lib/x86_64-linux-
> > gnu/libpthread.so.0
> > #5  0x00007fe7c0a1890d in clone () from /lib/x86_64-linux-gnu/libc.so.6
> > #6  0x0000000000000000 in ?? ()
> > 
> > Thread 1 (Thread 0x7fe7c5101900 (LWP 14648)):
> > #0  0x00007fe7c0a12403 in select () from /lib/x86_64-linux-gnu/libc.so.6
> > #1  0x00007fe7c525b56c in main_loop_wait (nonblocking=<optimized out>) at
> > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/main-
> > loop.c:456
> > #2  0x00007fe7c51a372f in main_loop () at
> > /build/buildd-qemu-kvm_1.0+dfsg-8-
> > amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:1482
> > #3  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized
> > out>) at
> > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:35
> > 23
> > 
> > Thanks :)
> 
> OK, then we need a kernel view on this. Can you try
> 
> http://www.linux-kvm.org/page/Tracing
> 
> ?
> 
> Thanks,
> Jan

Before I get done with that, does it help that I see some dmesg warnings (with 
the 3.2 kernel) relating to some ioctls? 

[   26.694981] kvm: sending ioctl 80200204 to a partition!
[   26.695036] kvm: sending ioctl 5326 to a partition!

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Re: one out of four existing kvm guest's not starting after system upgrade

[Thomas Fjellstrom]

[-- Attachment #1: Type: Text/Plain, Size: 3468 bytes --]

On Sat Feb 18, 2012, Jan Kiszka wrote:
> On 2012-02-18 09:50, Thomas Fjellstrom wrote:
> > On Sat Feb 18, 2012, Jan Kiszka wrote:
> >> On 2012-02-18 05:49, Thomas Fjellstrom wrote:
> >>> I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2, and
> >>> qemu+qemu-kvm updated (not sure from what to what). But after the
> >>> upgrade, one of my guests will not start up. It gets stuck with 60-80%
> >>> cpu use, almost no memory is allocated by qemu/kvm, and no output of
> >>> any kind is seen (in the host console, or a bunch of the guest output
> >>> options like curses display, stdio output, virsh console, vnc or sdl
> >>> output). I normally use libvirt to manage the guests, but I've
> >>> attempted to run qemu manually, and have the same problems.
> >>> 
> >>> What can cause this?
> >>> 
> >>> Just tested booting back into the old kernel, the one guest still won't
> >>> start, while the rest do. I'm thoroughly confused.
> >> 
> >> You mean if you only update qemu-kvm, the problem persists, just with
> >> lower probability? In that case, we definitely need the version of your
> >> current qemu-kvm installation. Also, it would be nice to attach gdb to
> >> the stuck qemu-kvm process, issuing a "thread apply all backtrace" in
> >> that state.
> >> 
> >> Jan
> > 
> > Sorry I wasn't clear. If I just update qemu-kvm (And qemu with it, and
> > not the kernel), it always just hangs on load.
> > 
> > current version:
> > QEMU emulator version 1.0 (qemu-kvm-1.0 Debian 1.0+dfsg-8), Copyright (c)
> > 2003-2008 Fabrice Bellard
> > 
> > gdb thread apply all bt:
> > Thread 2 (Thread 0x7fe7b810c700 (LWP 14650)):
> > #0  0x00007fe7c0a11957 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6
> > #1  0x00007fe7c53155e9 in kvm_vcpu_ioctl (env=<optimized out>,
> > type=<optimized out>) at
> > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-
> > all.c:1101
> > #2  0x00007fe7c5315731 in kvm_cpu_exec (env=0x7fe7c61cb350) at
> > /build/buildd-
> > qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-all.c:987 #3 
> > 0x00007fe7c52ecf31 in qemu_kvm_cpu_thread_fn (arg=0x7fe7c61cb350) at
> > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/cpus.c:
> > 740 #4  0x00007fe7c0ccdb50 in start_thread () from /lib/x86_64-linux-
> > gnu/libpthread.so.0
> > #5  0x00007fe7c0a1890d in clone () from /lib/x86_64-linux-gnu/libc.so.6
> > #6  0x0000000000000000 in ?? ()
> > 
> > Thread 1 (Thread 0x7fe7c5101900 (LWP 14648)):
> > #0  0x00007fe7c0a12403 in select () from /lib/x86_64-linux-gnu/libc.so.6
> > #1  0x00007fe7c525b56c in main_loop_wait (nonblocking=<optimized out>) at
> > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/main-
> > loop.c:456
> > #2  0x00007fe7c51a372f in main_loop () at
> > /build/buildd-qemu-kvm_1.0+dfsg-8-
> > amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:1482
> > #3  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized
> > out>) at
> > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:35
> > 23
> > 
> > Thanks :)
> 
> OK, then we need a kernel view on this. Can you try
> 
> http://www.linux-kvm.org/page/Tracing
> 
> ?
> 
> Thanks,
> Jan

It made a rather large trace file. I'm pretty sure no-one wants me to actually 
link (or attach) the full 1.5G file, but the last 5000 lines might be useful, 
so I've compressed and attached them. (but just in case, I'm lzma'ing the 
entire 1.5G trace data file)

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

[-- Attachment #2: trace-cmd.report1-tail.lzma --]
[-- Type: application/x-lzma, Size: 17117 bytes --]

Re: one out of four existing kvm guest's not starting after system upgrade

[Thomas Fjellstrom]

On Sat Feb 18, 2012, Thomas Fjellstrom wrote:
> On Sat Feb 18, 2012, Jan Kiszka wrote:
> > On 2012-02-18 09:50, Thomas Fjellstrom wrote:
> > > On Sat Feb 18, 2012, Jan Kiszka wrote:
> > >> On 2012-02-18 05:49, Thomas Fjellstrom wrote:
> > >>> I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2,
> > >>> and qemu+qemu-kvm updated (not sure from what to what). But after
> > >>> the upgrade, one of my guests will not start up. It gets stuck with
> > >>> 60-80% cpu use, almost no memory is allocated by qemu/kvm, and no
> > >>> output of any kind is seen (in the host console, or a bunch of the
> > >>> guest output options like curses display, stdio output, virsh
> > >>> console, vnc or sdl output). I normally use libvirt to manage the
> > >>> guests, but I've attempted to run qemu manually, and have the same
> > >>> problems.
> > >>> 
> > >>> What can cause this?
> > >>> 
> > >>> Just tested booting back into the old kernel, the one guest still
> > >>> won't start, while the rest do. I'm thoroughly confused.
> > >> 
> > >> You mean if you only update qemu-kvm, the problem persists, just with
> > >> lower probability? In that case, we definitely need the version of
> > >> your current qemu-kvm installation. Also, it would be nice to attach
> > >> gdb to the stuck qemu-kvm process, issuing a "thread apply all
> > >> backtrace" in that state.
> > >> 
> > >> Jan
> > > 
> > > Sorry I wasn't clear. If I just update qemu-kvm (And qemu with it, and
> > > not the kernel), it always just hangs on load.
> > > 
> > > current version:
> > > QEMU emulator version 1.0 (qemu-kvm-1.0 Debian 1.0+dfsg-8), Copyright
> > > (c) 2003-2008 Fabrice Bellard
> > > 
> > > gdb thread apply all bt:
> > > Thread 2 (Thread 0x7fe7b810c700 (LWP 14650)):
> > > #0  0x00007fe7c0a11957 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6
> > > #1  0x00007fe7c53155e9 in kvm_vcpu_ioctl (env=<optimized out>,
> > > type=<optimized out>) at
> > > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-
> > > all.c:1101
> > > #2  0x00007fe7c5315731 in kvm_cpu_exec (env=0x7fe7c61cb350) at
> > > /build/buildd-
> > > qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-all.c:987 #3
> > > 0x00007fe7c52ecf31 in qemu_kvm_cpu_thread_fn (arg=0x7fe7c61cb350) at
> > > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/cpus.c
> > > : 740 #4  0x00007fe7c0ccdb50 in start_thread () from /lib/x86_64-linux-
> > > gnu/libpthread.so.0
> > > #5  0x00007fe7c0a1890d in clone () from /lib/x86_64-linux-gnu/libc.so.6
> > > #6  0x0000000000000000 in ?? ()
> > > 
> > > Thread 1 (Thread 0x7fe7c5101900 (LWP 14648)):
> > > #0  0x00007fe7c0a12403 in select () from
> > > /lib/x86_64-linux-gnu/libc.so.6 #1  0x00007fe7c525b56c in
> > > main_loop_wait (nonblocking=<optimized out>) at
> > > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/main-
> > > loop.c:456
> > > #2  0x00007fe7c51a372f in main_loop () at
> > > /build/buildd-qemu-kvm_1.0+dfsg-8-
> > > amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:1482
> > > #3  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized
> > > out>) at
> > > /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:3
> > > 5 23
> > > 
> > > Thanks :)
> > 
> > OK, then we need a kernel view on this. Can you try
> > 
> > http://www.linux-kvm.org/page/Tracing
> > 
> > ?
> > 
> > Thanks,
> > Jan
> 
> It made a rather large trace file. I'm pretty sure no-one wants me to
> actually link (or attach) the full 1.5G file, but the last 5000 lines
> might be useful, so I've compressed and attached them. (but just in case,
> I'm lzma'ing the entire 1.5G trace data file)

I'm pretty much stumped on this. So I decided to try re-creating the vm 
through virt-manager. Its up and running now. The only to major differences I 
can see in the old and new config is the machine (-M pc-0.12 vs -M pc-1.0) 
parameter, and the uuid. The rest of the parameters I played with a lot trying 
to get it to work by starting up the vm manually from the cli. I can't really 
see how those two changes would do much of anything considering the other 
three VM's still are configured to use -M pc-0.12, and they work fine.

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Re: one out of four existing kvm guest's not starting after system upgrade

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 4392 bytes --]

On 2012-02-19 21:13, Thomas Fjellstrom wrote:
> On Sat Feb 18, 2012, Thomas Fjellstrom wrote:
>> On Sat Feb 18, 2012, Jan Kiszka wrote:
>>> On 2012-02-18 09:50, Thomas Fjellstrom wrote:
>>>> On Sat Feb 18, 2012, Jan Kiszka wrote:
>>>>> On 2012-02-18 05:49, Thomas Fjellstrom wrote:
>>>>>> I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2,
>>>>>> and qemu+qemu-kvm updated (not sure from what to what). But after
>>>>>> the upgrade, one of my guests will not start up. It gets stuck with
>>>>>> 60-80% cpu use, almost no memory is allocated by qemu/kvm, and no
>>>>>> output of any kind is seen (in the host console, or a bunch of the
>>>>>> guest output options like curses display, stdio output, virsh
>>>>>> console, vnc or sdl output). I normally use libvirt to manage the
>>>>>> guests, but I've attempted to run qemu manually, and have the same
>>>>>> problems.
>>>>>>
>>>>>> What can cause this?
>>>>>>
>>>>>> Just tested booting back into the old kernel, the one guest still
>>>>>> won't start, while the rest do. I'm thoroughly confused.
>>>>>
>>>>> You mean if you only update qemu-kvm, the problem persists, just with
>>>>> lower probability? In that case, we definitely need the version of
>>>>> your current qemu-kvm installation. Also, it would be nice to attach
>>>>> gdb to the stuck qemu-kvm process, issuing a "thread apply all
>>>>> backtrace" in that state.
>>>>>
>>>>> Jan
>>>>
>>>> Sorry I wasn't clear. If I just update qemu-kvm (And qemu with it, and
>>>> not the kernel), it always just hangs on load.
>>>>
>>>> current version:
>>>> QEMU emulator version 1.0 (qemu-kvm-1.0 Debian 1.0+dfsg-8), Copyright
>>>> (c) 2003-2008 Fabrice Bellard
>>>>
>>>> gdb thread apply all bt:
>>>> Thread 2 (Thread 0x7fe7b810c700 (LWP 14650)):
>>>> #0  0x00007fe7c0a11957 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6
>>>> #1  0x00007fe7c53155e9 in kvm_vcpu_ioctl (env=<optimized out>,
>>>> type=<optimized out>) at
>>>> /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-
>>>> all.c:1101
>>>> #2  0x00007fe7c5315731 in kvm_cpu_exec (env=0x7fe7c61cb350) at
>>>> /build/buildd-
>>>> qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-all.c:987 #3
>>>> 0x00007fe7c52ecf31 in qemu_kvm_cpu_thread_fn (arg=0x7fe7c61cb350) at
>>>> /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/cpus.c
>>>> : 740 #4  0x00007fe7c0ccdb50 in start_thread () from /lib/x86_64-linux-
>>>> gnu/libpthread.so.0
>>>> #5  0x00007fe7c0a1890d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>> #6  0x0000000000000000 in ?? ()
>>>>
>>>> Thread 1 (Thread 0x7fe7c5101900 (LWP 14648)):
>>>> #0  0x00007fe7c0a12403 in select () from
>>>> /lib/x86_64-linux-gnu/libc.so.6 #1  0x00007fe7c525b56c in
>>>> main_loop_wait (nonblocking=<optimized out>) at
>>>> /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/main-
>>>> loop.c:456
>>>> #2  0x00007fe7c51a372f in main_loop () at
>>>> /build/buildd-qemu-kvm_1.0+dfsg-8-
>>>> amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:1482
>>>> #3  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized
>>>> out>) at
>>>> /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:3
>>>> 5 23
>>>>
>>>> Thanks :)
>>>
>>> OK, then we need a kernel view on this. Can you try
>>>
>>> http://www.linux-kvm.org/page/Tracing
>>>
>>> ?
>>>
>>> Thanks,
>>> Jan
>>
>> It made a rather large trace file. I'm pretty sure no-one wants me to
>> actually link (or attach) the full 1.5G file, but the last 5000 lines
>> might be useful, so I've compressed and attached them. (but just in case,
>> I'm lzma'ing the entire 1.5G trace data file)
> 
> I'm pretty much stumped on this. So I decided to try re-creating the vm 
> through virt-manager. Its up and running now. The only to major differences I 
> can see in the old and new config is the machine (-M pc-0.12 vs -M pc-1.0) 
> parameter, and the uuid. The rest of the parameters I played with a lot trying 
> to get it to work by starting up the vm manually from the cli. I can't really 
> see how those two changes would do much of anything considering the other 
> three VM's still are configured to use -M pc-0.12, and they work fine.
> 

Sorry, on vacation with limited IT access, thus wasn't able to look that
traces yet. Hope someone else can pick this up.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: one out of four existing kvm guest's not starting after system upgrade

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1221 bytes --]

On 2012-02-19 21:13, Thomas Fjellstrom wrote:
> I'm pretty much stumped on this. So I decided to try re-creating the vm 
> through virt-manager. Its up and running now. The only to major differences I 
> can see in the old and new config is the machine (-M pc-0.12 vs -M pc-1.0) 
> parameter, and the uuid. The rest of the parameters I played with a lot trying 
> to get it to work by starting up the vm manually from the cli. I can't really 
> see how those two changes would do much of anything considering the other 
> three VM's still are configured to use -M pc-0.12, and they work fine.
> 

To pick up this topic again: The trace contains no clear indication what
is going on. Now I'm trying to understand what works and what not.
Please correct / extend as required:

 - qemu-kvm-0.12 problematic-vm.img		[OK]
 - qemu-kvm-1.0 -M pc-0.12 problematic-vm.img	[HANG]
 - qemu-kvm-1.0 -M pc-1.0 problematic-vm.img	[OK]

In all cases, the image is the same, never reinstalled?

BTW, what is your guest again? What is your VM configuration?

That IOCTL error messages you find in the kernel log likely relate to
direct cdrom access from the qemu process. Do you pass a host drive through?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: one out of four existing kvm guest's not starting after system upgrade

[Thomas Fjellstrom]

On Tue Feb 28, 2012, you wrote:
> On 2012-02-19 21:13, Thomas Fjellstrom wrote:
> > I'm pretty much stumped on this. So I decided to try re-creating the vm
> > through virt-manager. Its up and running now. The only to major
> > differences I can see in the old and new config is the machine (-M
> > pc-0.12 vs -M pc-1.0) parameter, and the uuid. The rest of the
> > parameters I played with a lot trying to get it to work by starting up
> > the vm manually from the cli. I can't really see how those two changes
> > would do much of anything considering the other three VM's still are
> > configured to use -M pc-0.12, and they work fine.
> 
> To pick up this topic again: The trace contains no clear indication what
> is going on. Now I'm trying to understand what works and what not.
> Please correct / extend as required:
> 
>  - qemu-kvm-0.12 problematic-vm.img		[OK]
>  - qemu-kvm-1.0 -M pc-0.12 problematic-vm.img	[HANG]
>  - qemu-kvm-1.0 -M pc-1.0 problematic-vm.img	[OK]
> 
> In all cases, the image is the same, never reinstalled?

Right. same exact disk image.

> 
> BTW, what is your guest again? What is your VM configuration?

Guest is debian squeeze. (with a trace of sid, but not a whole lot)
 
> That IOCTL error messages you find in the kernel log likely relate to
> direct cdrom access from the qemu process. Do you pass a host drive
> through?

Not a CDROM drive no. The host doesn't even have a cdrom drive. There are some 
virtio lvm disk images passed through.
 
> Jan


-- 
Thomas Fjellstrom
thomas@fjellstrom.ca