Re: [Qemu-devel] qemu 2.2 stuck on condition mutex locks on all threads

[Stefan Hajnoczi <stefanha@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 4328 bytes --]

On Mon, Dec 07, 2015 at 07:44:24PM -0500, Neil McGill wrote:
> Has anyone seen anything like this ? all 4 qemu threads are stuck on a
> pthread condition

This is incorrect.  Thread 4 is blocked in poll(2).

> QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9.3~cloud0), Copyright (c) 2003-2008 Fabrice Bellard
> 
> I'll try and get the qemu source built on this machine to debug and get
> some useful symbols out of gdb, but just wondering if there is a known
> issue like this on:
> 
> Linux virl 3.19.0-33-generic #38~14.04.1-Ubuntu SMP Fri Nov 6 18:17:28 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
> 
> tx
> 
> neil
> 
> (gdb) where
> #0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
> #1  0x00007f7a17524657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
> #2  0x00007f7a17524480 in __GI___pthread_mutex_lock (mutex=0x7f7a2150f9a0) at ../nptl/pthread_mutex_lock.c:79
> #3  0x00007f7a21036909 in ?? ()
> #4  0x00007f7a20fc1fdb in ?? ()
> #5  0x00007f7a20d153fe in ?? ()
> #6  0x00007f7a17176ec5 in __libc_start_main (main=0x7f7a20d13eb0, argc=94, argv=0x7ffcb0b0b948, init=<optimized out>,
>     fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffcb0b0b938) at libc-start.c:287
> #7  0x00007f7a20d1b96c in ?? ()

This is the QEMU main loop.  It's probably trying to lock the global
mutex.

> (gdb) thread 2
> [Switching to thread 2 (Thread 0x7f793e3ff700 (LWP 65315))]
> #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> 185	../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
> (gdb) where
> #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> #1  0x00007f7a21036b29 in ?? ()
> #2  0x00007f7a20fb10c3 in ?? ()
> #3  0x00007f7a20fb14c0 in ?? ()
> #4  0x00007f7a17522182 in start_thread (arg=0x7f793e3ff700) at pthread_create.c:312
> #5  0x00007f7a1724f47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
> 
> (gdb) thread 3
> [Switching to thread 3 (Thread 0x7f7a0cbf8700 (LWP 65311))]
> #0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
> 135	../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
> (gdb) where
> #0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
> #1  0x00007f7a17524657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
> #2  0x00007f7a17524480 in __GI___pthread_mutex_lock (mutex=0x7f7a2150f9a0) at ../nptl/pthread_mutex_lock.c:79
> #3  0x00007f7a21036909 in ?? ()
> #4  0x00007f7a20d57d3c in ?? ()
> #5  0x00007f7a20d42db2 in ?? ()
> #6  0x00007f7a17522182 in start_thread (arg=0x7f7a0cbf8700) at pthread_create.c:312
> #7  0x00007f7a1724f47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

This could be a vcpu thread.  It's also trying to lock the global mutex.

> 
> (gdb) thread 4
> [Switching to thread 4 (Thread 0x7f7a0d3f9700 (LWP 65310))]
> #0  0x00007f7a172421ef in __GI_ppoll (fds=0x7f7a234db150, nfds=1, timeout=<optimized out>, sigmask=0x0)
>     at ../sysdeps/unix/sysv/linux/ppoll.c:56
> 56	../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.
> (gdb) where
> #0  0x00007f7a172421ef in __GI_ppoll (fds=0x7f7a234db150, nfds=1, timeout=<optimized out>, sigmask=0x0)
>     at ../sysdeps/unix/sysv/linux/ppoll.c:56
> #1  0x00007f7a20fc2b2b in ?? ()
> #2  0x00007f7a20fc3d04 in ?? ()
> #3  0x00007f7a20fb5e5f in ?? ()
> #4  0x00007f7a20fb78f4 in ?? ()
> #5  0x00007f7a20ef1732 in ?? ()
> #6  0x00007f7a20d58e81 in ?? ()
> #7  0x00007f7a20d5e947 in ?? ()
> #8  0x00007f7a20d1f933 in ?? ()
> #9  0x00007f7a20d57fa1 in ?? ()
> #10 0x00007f7a20d42db2 in ?? ()
> #11 0x00007f7a17522182 in start_thread (arg=0x7f7a0d3f9700) at pthread_create.c:312
> #12 0x00007f7a1724f47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
> (gdb)

This is the interesting thread.  It is blocked in poll(2) waiting for
activity on just 1 file descriptor.  You could check fds[0] and then
look at ls -l /proc/fd/<n> to see what the file is being monitored.

My guess is thread 4 holds the global mutex and is therefore blocking
all the other threads.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]