Re: [Qemu-devel] qemu 2.0, deadlock in block-commit

["Marcin Gibuła" <m.gibula@beyond.pl>]

On 23.05.2014 10:19, Paolo Bonzini wrote:
> Il 22/05/2014 23:05, Marcin Gibuła ha scritto:
>> Some more info.
>> VM was doing lot of write IO during this test.
>
> QEMU is waiting for librados to complete I/O.  Can you reproduce it with
> a different driver?

Hi,

I've reproduced it without RBD. Backtrace below:

(gdb) thread apply all backtrace

Thread 4 (Thread 0x7f9c8cccd700 (LWP 2017)):
#0  0x00007f9c907717a4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9c9076d19c in _L_lock_518 () from /lib64/libpthread.so.0
#2  0x00007f9c9076cfeb in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f9c947addf9 in qemu_mutex_lock 
(mutex=mutex@entry=0x7f9c95002660 <qemu_global_mutex>) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/util/qemu-thread-posix.c:76
#4  0x00007f9c946b3a10 in qemu_mutex_lock_iothread () at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/cpus.c:1043
#5  0x00007f9c9470cf3d in kvm_cpu_exec (cpu=cpu@entry=0x7f9c968bf290) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/kvm-all.c:1683
#6  0x00007f9c946b271c in qemu_kvm_cpu_thread_fn (arg=0x7f9c968bf290) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/cpus.c:873
#7  0x00007f9c9076af3a in start_thread () from /lib64/libpthread.so.0
#8  0x00007f9c904a4dad in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f9c87fff700 (LWP 2018)):
#0  0x00007f9c9049c897 in ioctl () from /lib64/libc.so.6
#1  0x00007f9c9470cdf9 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f9c968fa300, 
type=type@entry=44672) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/kvm-all.c:1796
#2  0x00007f9c9470cf35 in kvm_cpu_exec (cpu=cpu@entry=0x7f9c968fa300) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/kvm-all.c:1681
#3  0x00007f9c946b271c in qemu_kvm_cpu_thread_fn (arg=0x7f9c968fa300) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/cpus.c:873
#4  0x00007f9c9076af3a in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9c904a4dad in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f9c869ff700 (LWP 2020)):
#0  0x00007f9c9076ed0c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007f9c947ae019 in qemu_cond_wait 
(cond=cond@entry=0x7f9c9695a250, mutex=mutex@entry=0x7f9c9695a280)
     at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/util/qemu-thread-posix.c:135
#2  0x00007f9c946a270b in vnc_worker_thread_loop 
(queue=queue@entry=0x7f9c9695a250) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/ui/vnc-jobs.c:222
#3  0x00007f9c946a2ae0 in vnc_worker_thread (arg=0x7f9c9695a250) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/ui/vnc-jobs.c:323
#4  0x00007f9c9076af3a in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9c904a4dad in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f9c94448900 (LWP 2013)):
#0  0x00007f9c9049b286 in ppoll () from /lib64/libc.so.6
#1  0x00007f9c9466ed9b in ppoll (__ss=0x0, __timeout=0x0, 
__nfds=<optimized out>, __fds=<optimized out>) at 
/usr/include/bits/poll2.h:77
#2  qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, 
timeout=<optimized out>) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/qemu-timer.c:311
#3  0x00007f9c945027e0 in aio_poll (ctx=0x7f9c95d5bc00, 
blocking=blocking@entry=true) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/aio-posix.c:221
#4  0x00007f9c94510c0a in bdrv_drain_all () at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/block.c:1805
#5  0x00007f9c9451787e in bdrv_close (bs=bs@entry=0x7f9c969b7d90) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/block.c:1695
#6  0x00007f9c945175fa in bdrv_delete (bs=0x7f9c969b7d90) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/block.c:1978
#7  bdrv_unref (bs=0x7f9c969b7d90) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/block.c:5198
#8  0x00007f9c94517812 in bdrv_drop_intermediate 
(active=active@entry=0x7f9c9648f490, top=top@entry=0x7f9c969b7d90, 
base=base@entry=0x7f9c96756500)
     at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/block.c:2567
#9  0x00007f9c9451c963 in commit_run (opaque=0x7f9c96a1e280) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/block/commit.c:144
#10 0x00007f9c9455bdca in coroutine_trampoline (i0=<optimized out>, 
i1=<optimized out>) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/coroutine-ucontext.c:118
#11 0x00007f9c904009f0 in ?? () from /lib64/libc.so.6
#12 0x00007fffe4bcfee0 in ?? ()
#13 0x0000000000000000 in ?? ()

I still have this process running (hanging ;) ) if you need any more 
info. I also have no problems with reproducing it.

-- 
mg