Re: one out of four existing kvm guest's not starting after system upgrade

[Jan Kiszka <jan.kiszka@web.de>]

[-- Attachment #1: Type: text/plain, Size: 4392 bytes --]

On 2012-02-19 21:13, Thomas Fjellstrom wrote:
> On Sat Feb 18, 2012, Thomas Fjellstrom wrote:
>> On Sat Feb 18, 2012, Jan Kiszka wrote:
>>> On 2012-02-18 09:50, Thomas Fjellstrom wrote:
>>>> On Sat Feb 18, 2012, Jan Kiszka wrote:
>>>>> On 2012-02-18 05:49, Thomas Fjellstrom wrote:
>>>>>> I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2,
>>>>>> and qemu+qemu-kvm updated (not sure from what to what). But after
>>>>>> the upgrade, one of my guests will not start up. It gets stuck with
>>>>>> 60-80% cpu use, almost no memory is allocated by qemu/kvm, and no
>>>>>> output of any kind is seen (in the host console, or a bunch of the
>>>>>> guest output options like curses display, stdio output, virsh
>>>>>> console, vnc or sdl output). I normally use libvirt to manage the
>>>>>> guests, but I've attempted to run qemu manually, and have the same
>>>>>> problems.
>>>>>>
>>>>>> What can cause this?
>>>>>>
>>>>>> Just tested booting back into the old kernel, the one guest still
>>>>>> won't start, while the rest do. I'm thoroughly confused.
>>>>>
>>>>> You mean if you only update qemu-kvm, the problem persists, just with
>>>>> lower probability? In that case, we definitely need the version of
>>>>> your current qemu-kvm installation. Also, it would be nice to attach
>>>>> gdb to the stuck qemu-kvm process, issuing a "thread apply all
>>>>> backtrace" in that state.
>>>>>
>>>>> Jan
>>>>
>>>> Sorry I wasn't clear. If I just update qemu-kvm (And qemu with it, and
>>>> not the kernel), it always just hangs on load.
>>>>
>>>> current version:
>>>> QEMU emulator version 1.0 (qemu-kvm-1.0 Debian 1.0+dfsg-8), Copyright
>>>> (c) 2003-2008 Fabrice Bellard
>>>>
>>>> gdb thread apply all bt:
>>>> Thread 2 (Thread 0x7fe7b810c700 (LWP 14650)):
>>>> #0  0x00007fe7c0a11957 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6
>>>> #1  0x00007fe7c53155e9 in kvm_vcpu_ioctl (env=<optimized out>,
>>>> type=<optimized out>) at
>>>> /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-
>>>> all.c:1101
>>>> #2  0x00007fe7c5315731 in kvm_cpu_exec (env=0x7fe7c61cb350) at
>>>> /build/buildd-
>>>> qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-all.c:987 #3
>>>> 0x00007fe7c52ecf31 in qemu_kvm_cpu_thread_fn (arg=0x7fe7c61cb350) at
>>>> /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/cpus.c
>>>> : 740 #4  0x00007fe7c0ccdb50 in start_thread () from /lib/x86_64-linux-
>>>> gnu/libpthread.so.0
>>>> #5  0x00007fe7c0a1890d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>> #6  0x0000000000000000 in ?? ()
>>>>
>>>> Thread 1 (Thread 0x7fe7c5101900 (LWP 14648)):
>>>> #0  0x00007fe7c0a12403 in select () from
>>>> /lib/x86_64-linux-gnu/libc.so.6 #1  0x00007fe7c525b56c in
>>>> main_loop_wait (nonblocking=<optimized out>) at
>>>> /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/main-
>>>> loop.c:456
>>>> #2  0x00007fe7c51a372f in main_loop () at
>>>> /build/buildd-qemu-kvm_1.0+dfsg-8-
>>>> amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:1482
>>>> #3  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized
>>>> out>) at
>>>> /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:3
>>>> 5 23
>>>>
>>>> Thanks :)
>>>
>>> OK, then we need a kernel view on this. Can you try
>>>
>>> http://www.linux-kvm.org/page/Tracing
>>>
>>> ?
>>>
>>> Thanks,
>>> Jan
>>
>> It made a rather large trace file. I'm pretty sure no-one wants me to
>> actually link (or attach) the full 1.5G file, but the last 5000 lines
>> might be useful, so I've compressed and attached them. (but just in case,
>> I'm lzma'ing the entire 1.5G trace data file)
> 
> I'm pretty much stumped on this. So I decided to try re-creating the vm 
> through virt-manager. Its up and running now. The only to major differences I 
> can see in the old and new config is the machine (-M pc-0.12 vs -M pc-1.0) 
> parameter, and the uuid. The rest of the parameters I played with a lot trying 
> to get it to work by starting up the vm manually from the cli. I can't really 
> see how those two changes would do much of anything considering the other 
> three VM's still are configured to use -M pc-0.12, and they work fine.
> 

Sorry, on vacation with limited IT access, thus wasn't able to look that
traces yet. Hope someone else can pick this up.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]