[Qemu-devel] qemu crashes since today

[Qemu-devel] qemu crashes since today

[Stefan Berger]

Qemu is crashing for me. Bisecting the code I found this here as the 
reason. Stepping one commit further into the past does not cause the crash.

6792a57bf19ab37f61f5acf0f8e3003cf08814af is the first bad commit
commit 6792a57bf19ab37f61f5acf0f8e3003cf08814af
Author: Jan Kiszka <jan.kiszka@siemens.com>
Date:   Mon Feb 7 12:19:18 2011 +0100


I get crashes while booting FC14 on a FC14 host, both 64bit. Host 
running 2.6.35.11-83.fc14 . Crashes typically happen when udev has 
stared in the guest.

This is the command line I was using:

./x86_64-softmmu/qemu-system-x86_64 \
       -hda ~images/fc14-x86_64.img \
       -m 2048 \
       -enable-kvm \
       -boot menu=on \
       -chardev stdio,id=seabios \
       -device isa-debugcon,iobase=0x402,chardev=seabios

Regards,
   Stefan

[Qemu-devel] Re: qemu crashes since today

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1015 bytes --]

On 2011-02-16 23:18, Stefan Berger wrote:
> Qemu is crashing for me. Bisecting the code I found this here as the
> reason. Stepping one commit further into the past does not cause the crash.
> 
> 6792a57bf19ab37f61f5acf0f8e3003cf08814af is the first bad commit
> commit 6792a57bf19ab37f61f5acf0f8e3003cf08814af
> Author: Jan Kiszka <jan.kiszka@siemens.com>
> Date:   Mon Feb 7 12:19:18 2011 +0100
> 
> 
> I get crashes while booting FC14 on a FC14 host, both 64bit. Host
> running 2.6.35.11-83.fc14 . Crashes typically happen when udev has
> stared in the guest.
> 
> This is the command line I was using:
> 
> ./x86_64-softmmu/qemu-system-x86_64 \
>       -hda ~images/fc14-x86_64.img \
>       -m 2048 \
>       -enable-kvm \
>       -boot menu=on \
>       -chardev stdio,id=seabios \
>       -device isa-debugcon,iobase=0x402,chardev=seabios
> 

With our without --enable-io-thread? How does the crash look like,
specifically when caught by gdb (backtrace, variable states)?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [Qemu-devel] Re: qemu crashes since today

[Stefan Berger]

On 02/16/2011 05:45 PM, Jan Kiszka wrote:
> On 2011-02-16 23:18, Stefan Berger wrote:
>> Qemu is crashing for me. Bisecting the code I found this here as the
>> reason. Stepping one commit further into the past does not cause the crash.
>>
>> 6792a57bf19ab37f61f5acf0f8e3003cf08814af is the first bad commit
>> commit 6792a57bf19ab37f61f5acf0f8e3003cf08814af
>> Author: Jan Kiszka<jan.kiszka@siemens.com>
>> Date:   Mon Feb 7 12:19:18 2011 +0100
>>
>>
>> I get crashes while booting FC14 on a FC14 host, both 64bit. Host
>> running 2.6.35.11-83.fc14 . Crashes typically happen when udev has
>> stared in the guest.
>>
>> This is the command line I was using:
>>
>> ./x86_64-softmmu/qemu-system-x86_64 \
>>        -hda ~images/fc14-x86_64.img \
>>        -m 2048 \
>>        -enable-kvm \
>>        -boot menu=on \
>>        -chardev stdio,id=seabios \
>>        -device isa-debugcon,iobase=0x402,chardev=seabios
>>
> With our without --enable-io-thread? How does the crash look like,
> specifically when caught by gdb (backtrace, variable states)?

./configure --target-list="x86_64-softmmu" --enable-kvm

the backtrace:

Booting from Hard Disk...
Booting from 0000:7c00
[New Thread 0x7fff51244700 (LWP 28825)]
*** stack smashing detected ***: 
/home/stefanb/qemu/qemu-git/x86_64-softmmu/qemu
-system-x86_64 terminated

Program received signal SIGSEGV, Segmentation fault.
0x0000003883a0f09c in ?? () from /lib64/libgcc_s.so.1
(gdb) thread apply all bt

Thread 2 (Thread 0x7fff51244700 (LWP 28825)):
#0  0x00007ffff79c571e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
    from /lib64/libpthread.so.0
#1  0x000000000043b1eb in cond_timedwait (unused=<value optimized out>)
     at posix-aio-compat.c:104
#2  aio_thread (unused=<value optimized out>) at posix-aio-compat.c:325
#3  0x00007ffff79c0ccb in start_thread () from /lib64/libpthread.so.0
#4  0x00007ffff6a2cc2d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7ffff7fc8820 (LWP 28818)):
#0  0x0000003883a0f09c in ?? () from /lib64/libgcc_s.so.1
#1  0x0000003883a0fc59 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
#2  0x00007ffff6a42520 in backtrace () from /lib64/libc.so.6
#3  0x00007ffff69bbd0f in __libc_message () from /lib64/libc.so.6
#4  0x00007ffff6a46127 in __fortify_fail () from /lib64/libc.so.6
#5  0x00007ffff6a460f0 in __stack_chk_fail () from /lib64/libc.so.6
#6  0x000000000042c669 in kvm_cpu_exec (env=0x1179310)
     at /home/stefanb/qemu/qemu-git/kvm-all.c:1005
#7  0x0000000000000001 in ?? ()
#8  0x000000000040bc18 in cpu_exec_all ()
     at /home/stefanb/qemu/qemu-git/cpus.c:1097
#9  0x000000000056d914 in main_loop (argc=<value optimized out>,
     argv=<value optimized out>, envp=<value optimized out>)
     at /home/stefanb/qemu/qemu-git/vl.c:1429
#10 main (argc=<value optimized out>, argv=<value optimized out>,
     envp=<value optimized out>) at /home/stefanb/qemu/qemu-git/vl.c:3148

I have previously seen another backtrace, but having tried this now 5 
times, this one keeps on coming back.

This is one that I have seen previously:

Program received signal SIGSEGV, Segmentation fault.
0x0000000001181345 in ?? ()
(gdb) bt
#0  0x0000000001181345 in ?? ()
#1  0x0000000000564d58 in load_segment (selector1=<value optimized out>)
     at /home/stefanb/qemu/qemu-git/target-i386/op_helper.c:149
#2  helper_verw (selector1=<value optimized out>)
     at /home/stefanb/qemu/qemu-git/target-i386/op_helper.c:3411
  #3  0x0000000000000000 in ?? ()

I hope this helps...

Regards,
     Stefan

Re: [Qemu-devel] Re: qemu crashes since today

[Anthony Liguori]

On 02/16/2011 07:35 PM, Stefan Berger wrote:
> On 02/16/2011 05:45 PM, Jan Kiszka wrote:
>> On 2011-02-16 23:18, Stefan Berger wrote:
>>> Qemu is crashing for me. Bisecting the code I found this here as the
>>> reason. Stepping one commit further into the past does not cause the 
>>> crash.
>>>
>>> 6792a57bf19ab37f61f5acf0f8e3003cf08814af is the first bad commit
>>> commit 6792a57bf19ab37f61f5acf0f8e3003cf08814af
>>> Author: Jan Kiszka<jan.kiszka@siemens.com>
>>> Date:   Mon Feb 7 12:19:18 2011 +0100
>>>
>>>
>>> I get crashes while booting FC14 on a FC14 host, both 64bit. Host
>>> running 2.6.35.11-83.fc14 . Crashes typically happen when udev has
>>> stared in the guest.
>>>
>>> This is the command line I was using:
>>>
>>> ./x86_64-softmmu/qemu-system-x86_64 \
>>>        -hda ~images/fc14-x86_64.img \
>>>        -m 2048 \
>>>        -enable-kvm \
>>>        -boot menu=on \
>>>        -chardev stdio,id=seabios \
>>>        -device isa-debugcon,iobase=0x402,chardev=seabios
>>>
>> With our without --enable-io-thread? How does the crash look like,
>> specifically when caught by gdb (backtrace, variable states)?
>
> ./configure --target-list="x86_64-softmmu" --enable-kvm
>
> the backtrace:
>
> Booting from Hard Disk...
> Booting from 0000:7c00
> [New Thread 0x7fff51244700 (LWP 28825)]
> *** stack smashing detected ***: 
> /home/stefanb/qemu/qemu-git/x86_64-softmmu/qemu
> -system-x86_64 terminated
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x0000003883a0f09c in ?? () from /lib64/libgcc_s.so.1
> (gdb) thread apply all bt
>
> Thread 2 (Thread 0x7fff51244700 (LWP 28825)):
> #0  0x00007ffff79c571e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>    from /lib64/libpthread.so.0
> #1  0x000000000043b1eb in cond_timedwait (unused=<value optimized out>)
>     at posix-aio-compat.c:104
> #2  aio_thread (unused=<value optimized out>) at posix-aio-compat.c:325
> #3  0x00007ffff79c0ccb in start_thread () from /lib64/libpthread.so.0
> #4  0x00007ffff6a2cc2d in clone () from /lib64/libc.so.6
>
> Thread 1 (Thread 0x7ffff7fc8820 (LWP 28818)):
> #0  0x0000003883a0f09c in ?? () from /lib64/libgcc_s.so.1
> #1  0x0000003883a0fc59 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
> #2  0x00007ffff6a42520 in backtrace () from /lib64/libc.so.6
> #3  0x00007ffff69bbd0f in __libc_message () from /lib64/libc.so.6
> #4  0x00007ffff6a46127 in __fortify_fail () from /lib64/libc.so.6
> #5  0x00007ffff6a460f0 in __stack_chk_fail () from /lib64/libc.so.6
> #6  0x000000000042c669 in kvm_cpu_exec (env=0x1179310)
>     at /home/stefanb/qemu/qemu-git/kvm-all.c:1005
> #7  0x0000000000000001 in ?? ()
> #8  0x000000000040bc18 in cpu_exec_all ()
>     at /home/stefanb/qemu/qemu-git/cpus.c:1097
> #9  0x000000000056d914 in main_loop (argc=<value optimized out>,
>     argv=<value optimized out>, envp=<value optimized out>)
>     at /home/stefanb/qemu/qemu-git/vl.c:1429
> #10 main (argc=<value optimized out>, argv=<value optimized out>,
>     envp=<value optimized out>) at /home/stefanb/qemu/qemu-git/vl.c:3148
>
> I have previously seen another backtrace, but having tried this now 5 
> times, this one keeps on coming back.
>
> This is one that I have seen previously:
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x0000000001181345 in ?? ()
> (gdb) bt
> #0  0x0000000001181345 in ?? ()
> #1  0x0000000000564d58 in load_segment (selector1=<value optimized out>)
>     at /home/stefanb/qemu/qemu-git/target-i386/op_helper.c:149
> #2  helper_verw (selector1=<value optimized out>)
>     at /home/stefanb/qemu/qemu-git/target-i386/op_helper.c:3411
>  #3  0x0000000000000000 in ?? ()
>
> I hope this helps...

It's a 64-bit fedora 14 guest.  I can't reproduce this with a grub1 or 
grub2 Ubuntu guest or a Fedora 13 Live CD.

I haven't tried a F14 guest yet though.

Regards,

Anthony Liguori

> Regards,
>     Stefan
>
>

Re: [Qemu-devel] Re: qemu crashes since today

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 4454 bytes --]

On 2011-02-17 04:09, Anthony Liguori wrote:
> On 02/16/2011 07:35 PM, Stefan Berger wrote:
>> On 02/16/2011 05:45 PM, Jan Kiszka wrote:
>>> On 2011-02-16 23:18, Stefan Berger wrote:
>>>> Qemu is crashing for me. Bisecting the code I found this here as the
>>>> reason. Stepping one commit further into the past does not cause the
>>>> crash.
>>>>
>>>> 6792a57bf19ab37f61f5acf0f8e3003cf08814af is the first bad commit
>>>> commit 6792a57bf19ab37f61f5acf0f8e3003cf08814af
>>>> Author: Jan Kiszka<jan.kiszka@siemens.com>
>>>> Date:   Mon Feb 7 12:19:18 2011 +0100
>>>>
>>>>
>>>> I get crashes while booting FC14 on a FC14 host, both 64bit. Host
>>>> running 2.6.35.11-83.fc14 . Crashes typically happen when udev has
>>>> stared in the guest.
>>>>
>>>> This is the command line I was using:
>>>>
>>>> ./x86_64-softmmu/qemu-system-x86_64 \
>>>>        -hda ~images/fc14-x86_64.img \
>>>>        -m 2048 \
>>>>        -enable-kvm \
>>>>        -boot menu=on \
>>>>        -chardev stdio,id=seabios \
>>>>        -device isa-debugcon,iobase=0x402,chardev=seabios
>>>>
>>> With our without --enable-io-thread? How does the crash look like,
>>> specifically when caught by gdb (backtrace, variable states)?
>>
>> ./configure --target-list="x86_64-softmmu" --enable-kvm
>>
>> the backtrace:
>>
>> Booting from Hard Disk...
>> Booting from 0000:7c00
>> [New Thread 0x7fff51244700 (LWP 28825)]
>> *** stack smashing detected ***:
>> /home/stefanb/qemu/qemu-git/x86_64-softmmu/qemu
>> -system-x86_64 terminated
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x0000003883a0f09c in ?? () from /lib64/libgcc_s.so.1
>> (gdb) thread apply all bt
>>
>> Thread 2 (Thread 0x7fff51244700 (LWP 28825)):
>> #0  0x00007ffff79c571e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>>    from /lib64/libpthread.so.0
>> #1  0x000000000043b1eb in cond_timedwait (unused=<value optimized out>)
>>     at posix-aio-compat.c:104
>> #2  aio_thread (unused=<value optimized out>) at posix-aio-compat.c:325
>> #3  0x00007ffff79c0ccb in start_thread () from /lib64/libpthread.so.0
>> #4  0x00007ffff6a2cc2d in clone () from /lib64/libc.so.6
>>
>> Thread 1 (Thread 0x7ffff7fc8820 (LWP 28818)):
>> #0  0x0000003883a0f09c in ?? () from /lib64/libgcc_s.so.1
>> #1  0x0000003883a0fc59 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
>> #2  0x00007ffff6a42520 in backtrace () from /lib64/libc.so.6
>> #3  0x00007ffff69bbd0f in __libc_message () from /lib64/libc.so.6
>> #4  0x00007ffff6a46127 in __fortify_fail () from /lib64/libc.so.6
>> #5  0x00007ffff6a460f0 in __stack_chk_fail () from /lib64/libc.so.6
>> #6  0x000000000042c669 in kvm_cpu_exec (env=0x1179310)
>>     at /home/stefanb/qemu/qemu-git/kvm-all.c:1005
>> #7  0x0000000000000001 in ?? ()
>> #8  0x000000000040bc18 in cpu_exec_all ()
>>     at /home/stefanb/qemu/qemu-git/cpus.c:1097
>> #9  0x000000000056d914 in main_loop (argc=<value optimized out>,
>>     argv=<value optimized out>, envp=<value optimized out>)
>>     at /home/stefanb/qemu/qemu-git/vl.c:1429
>> #10 main (argc=<value optimized out>, argv=<value optimized out>,
>>     envp=<value optimized out>) at /home/stefanb/qemu/qemu-git/vl.c:3148
>>
>> I have previously seen another backtrace, but having tried this now 5
>> times, this one keeps on coming back.
>>
>> This is one that I have seen previously:
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x0000000001181345 in ?? ()
>> (gdb) bt
>> #0  0x0000000001181345 in ?? ()
>> #1  0x0000000000564d58 in load_segment (selector1=<value optimized out>)
>>     at /home/stefanb/qemu/qemu-git/target-i386/op_helper.c:149
>> #2  helper_verw (selector1=<value optimized out>)
>>     at /home/stefanb/qemu/qemu-git/target-i386/op_helper.c:3411
>>  #3  0x0000000000000000 in ?? ()
>>
>> I hope this helps...
> 
> It's a 64-bit fedora 14 guest.  I can't reproduce this with a grub1 or
> grub2 Ubuntu guest or a Fedora 13 Live CD.
> 
> I haven't tried a F14 guest yet though.

I suspect (not only because it's a more comfortable explanation :) ) the
patch you bisected is only changing memory layouts to turn the stack
smashing done elsewhere into a segfault.

Does F14 happen to use the vmport/vmmount? Maybe this crash is related
to [1].

Jan (who still can't reproduce this as well, even with all stack
protections enabled)

[1] http://article.gmane.org/gmane.comp.emulators.qemu/94132


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [Qemu-devel] Re: qemu crashes since today

[Gerd Hoffmann]

   Hi,

> Does F14 happen to use the vmport/vmmount? Maybe this crash is related
> to [1].

Yes.  Try 'rpm -e xorg-x11-drv-vmmouse' in the guest and check whenever 
this avoids the crash to confirm vmmouse is the culprit.

cheers,
   Gerd

Re: [Qemu-devel] Re: qemu crashes since today

[Stefan Berger]

On 02/17/2011 04:21 AM, Gerd Hoffmann wrote:
>   Hi,
>
>> Does F14 happen to use the vmport/vmmount? Maybe this crash is related
>> to [1].
>
> Yes.  Try 'rpm -e xorg-x11-drv-vmmouse' in the guest and check 
> whenever this avoids the crash to confirm vmmouse is the culprit.
>
I removed this one and it's dependency xorg-x11-drivers and can confirm 
that it is the culprit. It boots fine without it and once I had 
reinstalled both of them and rebooted I saw the crash again.

   Stefan

> cheers,
>   Gerd
>
>