[Qemu-devel] coroutine: hung when using gthread backend

[Qemu-devel] coroutine: hung when using gthread backend

[Wenchao Xia]

  When I using tcg with coroutine backend = gthread,
x86_64-softmmu/qemu-system-x86_64 will hung. Any one
working on it?

System: RH6.3
gdb:

Reading symbols from
/home/xiawenc/Workdir/Src/qemu/build/master/x86_64-softmmu/qemu-system-x86_64...done.
(gdb) r
Starting program:
/home/xiawenc/Workdir/Src/qemu/build/master/x86_64-softmmu/qemu-system-x86_64

[Thread debugging using libthread_db enabled]
[New Thread 0x7fffecd5c700 (LWP 971)]
[New Thread 0x7fffe415a700 (LWP 972)]
[New Thread 0x7fffe3759700 (LWP 973)]
[Thread 0x7fffe415a700 (LWP 972) exited]
[Thread 0x7fffe3759700 (LWP 973) exited]
^C
Program received signal SIGINT, Interrupt.
0x00007ffff60f843c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install
SDL-1.2.14-3.el6.x86_64 cyrus-sasl-lib-2.1.23-13.el6.x86_64
glib2-2.22.5-7.el6.x86_64 glibc-2.12-1.80.el6.x86_64
gnutls-2.8.5-4.el6_2.2.x86_64 keyutils-libs-1.4-4.el6.x86_64
krb5-libs-1.9-33.el6.x86_64 libX11-1.3-2.el6.x86_64
libXau-1.0.5-1.el6.x86_64 libcom_err-1.41.12-12.el6.x86_64
libcurl-7.19.7-26.el6_2.4.x86_64 libgcrypt-1.4.5-9.el6_2.2.x86_64
libgpg-error-1.7-4.el6.x86_64 libidn-1.18-2.el6.x86_64
libjpeg-6b-46.el6.x86_64 libpng-1.2.49-1.el6_2.x86_64
libselinux-2.0.94-5.3.el6.x86_64 libssh2-1.2.2-7.el6_2.3.x86_64
libtasn1-2.3-3.el6_2.1.x86_64 libuuid-2.17.2-12.7.el6.x86_64
libxcb-1.5-1.el6.x86_64 ncurses-libs-5.7-3.20090208.el6.x86_64
nspr-4.9-1.el6.x86_64 nss-3.13.3-6.el6.x86_64
nss-softokn-freebl-3.12.9-11.el6.x86_64 nss-util-3.13.3-2.el6.x86_64
openldap-2.4.23-26.el6.x86_64 openssl-1.0.0-20.el6_2.5.x86_64
pixman-0.18.4-1.el6_0.1.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb) thread
[Current thread is 1 (Thread 0x7ffff7bf7940 (LWP 965))]
(gdb) info thread
  2 Thread 0x7fffecd5c700 (LWP 971)  0x00007ffff60f843c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
* 1 Thread 0x7ffff7bf7940 (LWP 965)  0x00007ffff60f843c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007ffff60f843c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff7ce73c8 in coroutine_wait_runnable_locked (co=0x7ffff8be4f30)
    at /home/xiawenc/Workdir/Src/qemu/qemu/coroutine-gthread.c:133
#2  0x00007ffff7ce74a8 in qemu_coroutine_switch (from_=0x7ffff8be4f30, to_=
    0x7ffff8be7b30, action=COROUTINE_YIELD)
    at /home/xiawenc/Workdir/Src/qemu/qemu/coroutine-gthread.c:191
#3  0x00007ffff7ddc092 in coroutine_swap (from=<value optimized out>, to=
    0x7ffff8be7b30) at
/home/xiawenc/Workdir/Src/qemu/qemu/qemu-coroutine.c:72
#4  0x00007ffff7cb8c05 in bdrv_rw_co (bs=0x7ffff8bcda60,
    sector_num=<value optimized out>, buf=<value optimized out>,
    nb_sectors=<value optimized out>, is_write=<value optimized out>)
    at /home/xiawenc/Workdir/Src/qemu/qemu/block.c:2171
#5  0x00007ffff7d615f7 in pflash_cfi01_init (dev=0x7ffff8bd0e60)
    at /home/xiawenc/Workdir/Src/qemu/qemu/hw/pflash_cfi01.c:583
#6  0x00007ffff7d66a76 in device_realize (dev=0x7ffff8bd0e60, err=
    0x7fffffffd9a0) at /home/xiawenc/Workdir/Src/qemu/qemu/hw/qdev.c:175
#7  0x00007ffff7d671b7 in device_set_realized (obj=0x7ffff8bd0e60,
value=true,
    err=0x7fffffffdaa0) at /home/xiawenc/Workdir/Src/qemu/qemu/hw/qdev.c:673
#8  0x00007ffff7de824e in property_set_bool (obj=0x7ffff8bd0e60,
    v=<value optimized out>, opaque=0x7ffff8be4ae0,
    name=<value optimized out>, errp=0x7fffffffdaa0)
    at /home/xiawenc/Workdir/Src/qemu/qemu/qom/object.c:1224
---Type <return> to continue, or q <return> to quit---
#9  0x00007ffff7dea765 in object_property_set_qobject (obj=0x7ffff8bd0e60,
    value=<value optimized out>, name=0x7ffff7f5ab27 "realized", errp=
    0x7fffffffdaa0) at
/home/xiawenc/Workdir/Src/qemu/qemu/qom/qom-qobject.c:24
#10 0x00007ffff7de8cbe in object_property_set_bool (obj=0x7ffff8bd0e60,
    value=<value optimized out>, name=0x7ffff7f5ab27 "realized", errp=
    0x7fffffffdaa0) at /home/xiawenc/Workdir/Src/qemu/qemu/qom/object.c:767
#11 0x00007ffff7d67efa in qdev_init (dev=0x7ffff8bd0e60)
    at /home/xiawenc/Workdir/Src/qemu/qemu/hw/qdev.c:161
#12 0x00007ffff7d67f91 in qdev_init_nofail (dev=0x7ffff8bd0e60)
    at /home/xiawenc/Workdir/Src/qemu/qemu/hw/qdev.c:261
#13 0x00007ffff7d60b6e in pflash_cfi01_register (base=4294836224,
    qdev=<value optimized out>, name=0x7ffff7f78604 "system.flash",
    size=<value optimized out>, bs=0x7ffff8bcda60, sector_len=4096,
nb_blocs=
    32, width=1, id0=0, id1=0, id2=0, id3=0, be=0)
    at /home/xiawenc/Workdir/Src/qemu/qemu/hw/pflash_cfi01.c:752
#14 0x00007ffff7e52154 in pc_system_flash_init (rom_memory=0x7ffff8be7250)
    at /home/xiawenc/Workdir/Src/qemu/qemu/hw/i386/../pc_sysfw.c:134
#15 pc_system_firmware_init (rom_memory=0x7ffff8be7250)
    at /home/xiawenc/Workdir/Src/qemu/qemu/hw/i386/../pc_sysfw.c:234
#16 0x00007ffff7e58b73 in pc_memory_init (system_memory=0x7ffff8badd50,
    kernel_filename=0x0, kernel_cmdline=0x7ffff7f87af2 "", initrd_filename=
    0x0, below_4g_mem_size=134217728, above_4g_mem_size=0, rom_memory=
    0x7ffff8be7250, ram_memory=0x7fffffffdcf0)
---Type <return> to continue, or q <return> to quit---
    at /home/xiawenc/Workdir/Src/qemu/qemu/hw/i386/pc.c:948
#17 0x00007ffff7e590e7 in pc_init1 (system_memory=0x7ffff8badd50, system_io=
    0x7ffff8baea20, ram_size=134217728, boot_device=0x7ffff7f5560b "cad",
    kernel_filename=0x0, kernel_cmdline=0x7ffff7f87af2 "", initrd_filename=
    0x0, cpu_model=0x0, pci_enabled=1, kvmclock_enabled=1)
    at /home/xiawenc/Workdir/Src/qemu/qemu/hw/i386/pc_piix.c:115
#18 0x00007ffff7e59800 in pc_init_pci (args=<value optimized out>)
    at /home/xiawenc/Workdir/Src/qemu/qemu/hw/i386/pc_piix.c:230
#19 0x00007ffff7e1e8db in main (argc=<value optimized out>,
    argv=<value optimized out>, envp=<value optimized out>)
    at /home/xiawenc/Workdir/Src/qemu/qemu/vl.c:4306
(gdb) thread 2
[Switching to thread 2 (Thread 0x7fffecd5c700 (LWP 971))]#0
0x00007ffff60f843c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
(gdb) bt
#0  0x00007ffff60f843c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff7f2f739 in qemu_cond_wait (cond=<value optimized out>,
    mutex=<value optimized out>)
    at /home/xiawenc/Workdir/Src/qemu/qemu/util/qemu-thread-posix.c:116
#2  0x00007ffff7e2845c in qemu_tcg_cpu_thread_fn (arg=<value optimized out>)
    at /home/xiawenc/Workdir/Src/qemu/qemu/cpus.c:834
#3  0x00007ffff60f4851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007ffff3de567d in clone () from /lib64/libc.so.6

-- 
Best Regards

Wenchao Xia

Re: [Qemu-devel] coroutine: hung when using gthread backend

[Paolo Bonzini]

Il 26/03/2013 03:50, Wenchao Xia ha scritto:
>   When I using tcg with coroutine backend = gthread,
> x86_64-softmmu/qemu-system-x86_64 will hung. Any one
> working on it?

coroutine backend gthread hardly works for qemu, only qemu-io and qemu-img.

Paolo

Re: [Qemu-devel] coroutine: hung when using gthread backend

[Stefan Hajnoczi]

On Tue, Mar 26, 2013 at 08:03:50AM +0100, Paolo Bonzini wrote:
> Il 26/03/2013 03:50, Wenchao Xia ha scritto:
> >   When I using tcg with coroutine backend = gthread,
> > x86_64-softmmu/qemu-system-x86_64 will hung. Any one
> > working on it?
> 
> coroutine backend gthread hardly works for qemu, only qemu-io and qemu-img.

Do you know why it doesn't work?

Stefan

Re: [Qemu-devel] coroutine: hung when using gthread backend

[Peter Maydell]

On 26 March 2013 09:54, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Tue, Mar 26, 2013 at 08:03:50AM +0100, Paolo Bonzini wrote:
>> coroutine backend gthread hardly works for qemu, only qemu-io and qemu-img.
>
> Do you know why it doesn't work?

Because nobody tests it?

-- PMM

Re: [Qemu-devel] coroutine: hung when using gthread backend

[Paolo Bonzini]

Il 26/03/2013 10:54, Stefan Hajnoczi ha scritto:
> On Tue, Mar 26, 2013 at 08:03:50AM +0100, Paolo Bonzini wrote:
>> Il 26/03/2013 03:50, Wenchao Xia ha scritto:
>>>   When I using tcg with coroutine backend = gthread,
>>> x86_64-softmmu/qemu-system-x86_64 will hung. Any one
>>> working on it?
>>
>> coroutine backend gthread hardly works for qemu, only qemu-io and qemu-img.
> 
> Do you know why it doesn't work?

Because it screws up the signals.  Coroutines can start in a thread and
later move to another.  If you use the gthread backend, coroutines keep
the signal mask of the thread where they were created.

It is basically the same bug that was fixed in commit 6ab7e54 (Replace
all setjmp()/longjmp() with sigsetjmp()/siglongjmp(), 2013-02-20).

Paolo

Re: [Qemu-devel] coroutine: hung when using gthread backend

[Stefan Hajnoczi]

On Tue, Mar 26, 2013 at 11:03:53AM +0100, Paolo Bonzini wrote:
> Il 26/03/2013 10:54, Stefan Hajnoczi ha scritto:
> > On Tue, Mar 26, 2013 at 08:03:50AM +0100, Paolo Bonzini wrote:
> >> Il 26/03/2013 03:50, Wenchao Xia ha scritto:
> >>>   When I using tcg with coroutine backend = gthread,
> >>> x86_64-softmmu/qemu-system-x86_64 will hung. Any one
> >>> working on it?
> >>
> >> coroutine backend gthread hardly works for qemu, only qemu-io and qemu-img.
> > 
> > Do you know why it doesn't work?
> 
> Because it screws up the signals.  Coroutines can start in a thread and
> later move to another.  If you use the gthread backend, coroutines keep
> the signal mask of the thread where they were created.
> 
> It is basically the same bug that was fixed in commit 6ab7e54 (Replace
> all setjmp()/longjmp() with sigsetjmp()/siglongjmp(), 2013-02-20).

That's interesting.  Maybe it can be fixed if the gthread backend is
actually needed.

Stefan

Re: [Qemu-devel] coroutine: hung when using gthread backend

[Wenchao Xia]

于 2013-3-27 0:21, Stefan Hajnoczi 写道:
> On Tue, Mar 26, 2013 at 11:03:53AM +0100, Paolo Bonzini wrote:
>> Il 26/03/2013 10:54, Stefan Hajnoczi ha scritto:
>>> On Tue, Mar 26, 2013 at 08:03:50AM +0100, Paolo Bonzini wrote:
>>>> Il 26/03/2013 03:50, Wenchao Xia ha scritto:
>>>>>    When I using tcg with coroutine backend = gthread,
>>>>> x86_64-softmmu/qemu-system-x86_64 will hung. Any one
>>>>> working on it?
>>>>
>>>> coroutine backend gthread hardly works for qemu, only qemu-io and qemu-img.
>>>
>>> Do you know why it doesn't work?
>>
>> Because it screws up the signals.  Coroutines can start in a thread and
>> later move to another.  If you use the gthread backend, coroutines keep
>> the signal mask of the thread where they were created.
>>
>> It is basically the same bug that was fixed in commit 6ab7e54 (Replace
>> all setjmp()/longjmp() with sigsetjmp()/siglongjmp(), 2013-02-20).
>
> That's interesting.  Maybe it can be fixed if the gthread backend is
> actually needed.
>
> Stefan
>
   If not hard I hope gthread backend works, since it works with gdb
and valgrind. I guess with default backend, gdb and valgrind need
some tricks to make they work.

-- 
Best Regards

Wenchao Xia

Re: [Qemu-devel] coroutine: hung when using gthread backend

[Wenchao Xia]

于 2013-3-26 17:56, Peter Maydell 写道:
> On 26 March 2013 09:54, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Tue, Mar 26, 2013 at 08:03:50AM +0100, Paolo Bonzini wrote:
>>> coroutine backend gthread hardly works for qemu, only qemu-io and qemu-img.
>>
>> Do you know why it doesn't work?
>
> Because nobody tests it?
>
> -- PMM
>
   It is not enabled by default in configure, so missed in tests. I feel
a full regression test suit covering different configure case is
missing.

-- 
Best Regards

Wenchao Xia