Re: [Qemu-devel] qemu 2.2 stuck on condition mutex locks on all threads

[Neil McGill <nmcgill@cisco.com>]

Thanks Stefan, will check and update

neil

On 12/10/15, 3:47 AM, Stefan Hajnoczi wrote:
> On Mon, Dec 07, 2015 at 07:44:24PM -0500, Neil McGill wrote:
>> Has anyone seen anything like this ? all 4 qemu threads are stuck on a
>> pthread condition
>
> This is incorrect.  Thread 4 is blocked in poll(2).
>
>> QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9.3~cloud0), Copyright (c) 2003-2008 Fabrice Bellard
>>
>> I'll try and get the qemu source built on this machine to debug and get
>> some useful symbols out of gdb, but just wondering if there is a known
>> issue like this on:
>>
>> Linux virl 3.19.0-33-generic #38~14.04.1-Ubuntu SMP Fri Nov 6 18:17:28 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>>
>> tx
>>
>> neil
>>
>> (gdb) where
>> #0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
>> #1  0x00007f7a17524657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
>> #2  0x00007f7a17524480 in __GI___pthread_mutex_lock (mutex=0x7f7a2150f9a0) at ../nptl/pthread_mutex_lock.c:79
>> #3  0x00007f7a21036909 in ?? ()
>> #4  0x00007f7a20fc1fdb in ?? ()
>> #5  0x00007f7a20d153fe in ?? ()
>> #6  0x00007f7a17176ec5 in __libc_start_main (main=0x7f7a20d13eb0, argc=94, argv=0x7ffcb0b0b948, init=<optimized out>,
>>      fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffcb0b0b938) at libc-start.c:287
>> #7  0x00007f7a20d1b96c in ?? ()
>
> This is the QEMU main loop.  It's probably trying to lock the global
> mutex.
>
>> (gdb) thread 2
>> [Switching to thread 2 (Thread 0x7f793e3ff700 (LWP 65315))]
>> #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>> 185	../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
>> (gdb) where
>> #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>> #1  0x00007f7a21036b29 in ?? ()
>> #2  0x00007f7a20fb10c3 in ?? ()
>> #3  0x00007f7a20fb14c0 in ?? ()
>> #4  0x00007f7a17522182 in start_thread (arg=0x7f793e3ff700) at pthread_create.c:312
>> #5  0x00007f7a1724f47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>>
>> (gdb) thread 3
>> [Switching to thread 3 (Thread 0x7f7a0cbf8700 (LWP 65311))]
>> #0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
>> 135	../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
>> (gdb) where
>> #0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
>> #1  0x00007f7a17524657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
>> #2  0x00007f7a17524480 in __GI___pthread_mutex_lock (mutex=0x7f7a2150f9a0) at ../nptl/pthread_mutex_lock.c:79
>> #3  0x00007f7a21036909 in ?? ()
>> #4  0x00007f7a20d57d3c in ?? ()
>> #5  0x00007f7a20d42db2 in ?? ()
>> #6  0x00007f7a17522182 in start_thread (arg=0x7f7a0cbf8700) at pthread_create.c:312
>> #7  0x00007f7a1724f47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>
> This could be a vcpu thread.  It's also trying to lock the global mutex.
>
>>
>> (gdb) thread 4
>> [Switching to thread 4 (Thread 0x7f7a0d3f9700 (LWP 65310))]
>> #0  0x00007f7a172421ef in __GI_ppoll (fds=0x7f7a234db150, nfds=1, timeout=<optimized out>, sigmask=0x0)
>>      at ../sysdeps/unix/sysv/linux/ppoll.c:56
>> 56	../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.
>> (gdb) where
>> #0  0x00007f7a172421ef in __GI_ppoll (fds=0x7f7a234db150, nfds=1, timeout=<optimized out>, sigmask=0x0)
>>      at ../sysdeps/unix/sysv/linux/ppoll.c:56
>> #1  0x00007f7a20fc2b2b in ?? ()
>> #2  0x00007f7a20fc3d04 in ?? ()
>> #3  0x00007f7a20fb5e5f in ?? ()
>> #4  0x00007f7a20fb78f4 in ?? ()
>> #5  0x00007f7a20ef1732 in ?? ()
>> #6  0x00007f7a20d58e81 in ?? ()
>> #7  0x00007f7a20d5e947 in ?? ()
>> #8  0x00007f7a20d1f933 in ?? ()
>> #9  0x00007f7a20d57fa1 in ?? ()
>> #10 0x00007f7a20d42db2 in ?? ()
>> #11 0x00007f7a17522182 in start_thread (arg=0x7f7a0d3f9700) at pthread_create.c:312
>> #12 0x00007f7a1724f47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>> (gdb)
>
> This is the interesting thread.  It is blocked in poll(2) waiting for
> activity on just 1 file descriptor.  You could check fds[0] and then
> look at ls -l /proc/fd/<n> to see what the file is being monitored.
>
> My guess is thread 4 holds the global mutex and is therefore blocking
> all the other threads.
>
> Stefan
>