From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38313) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z2FPO-00036f-8B for qemu-devel@nongnu.org; Tue, 09 Jun 2015 05:01:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Z2FPI-0000JU-2i for qemu-devel@nongnu.org; Tue, 09 Jun 2015 05:01:22 -0400 Received: from e06smtp12.uk.ibm.com ([195.75.94.108]:57811) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z2FPH-0000GZ-N1 for qemu-devel@nongnu.org; Tue, 09 Jun 2015 05:01:15 -0400 Received: from /spool/local by e06smtp12.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 9 Jun 2015 10:01:13 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (d06relay13.portsmouth.uk.ibm.com [9.149.109.198]) by d06dlp01.portsmouth.uk.ibm.com (Postfix) with ESMTP id CB63317D8059 for ; Tue, 9 Jun 2015 10:02:11 +0100 (BST) Received: from d06av10.portsmouth.uk.ibm.com (d06av10.portsmouth.uk.ibm.com [9.149.37.251]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id t5991Amk14745804 for ; Tue, 9 Jun 2015 09:01:10 GMT Received: from d06av10.portsmouth.uk.ibm.com (localhost [127.0.0.1]) by d06av10.portsmouth.uk.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id t59919Bq023814 for ; Tue, 9 Jun 2015 03:01:09 -0600 Message-ID: <5576AB52.8090708@de.ibm.com> Date: Tue, 09 Jun 2015 11:01:06 +0200 From: Christian Borntraeger MIME-Version: 1.0 References: <556DBF87.2020908@de.ibm.com> <20150609022832.GA12817@cpc-pc.redhat.com> In-Reply-To: <20150609022832.GA12817@cpc-pc.redhat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] "iothread: release iothread around aio_poll" causes random hangs at startup List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Fam Zheng Cc: Kevin Wolf , Paolo Bonzini , qemu-devel , Stefan Hajnoczi Am 09.06.2015 um 04:28 schrieb Fam Zheng: > On Tue, 06/02 16:36, Christian Borntraeger wrote: >> Paolo, >> >> I bisected >> commit a0710f7995f914e3044e5899bd8ff6c43c62f916 >> Author: Paolo Bonzini >> AuthorDate: Fri Feb 20 17:26:52 2015 +0100 >> Commit: Kevin Wolf >> CommitDate: Tue Apr 28 15:36:08 2015 +0200 >> >> iothread: release iothread around aio_poll >> >> to cause a problem with hanging guests. >> >> Having many guests all with a kernel/ramdisk (via -kernel) and >> several null block devices will result in hangs. All hanging >> guests are in partition detection code waiting for an I/O to return >> so very early maybe even the first I/O. >> >> Reverting that commit "fixes" the hangs. >> Any ideas? > > Christian, I can't reproduce this on my x86 box with virtio-blk-pci. Do you > have a reproducer for x86? Or could you collect backtraces for all the threads > in QEMU when it hangs? > > My long shot is that the main loop is blocked at aio_context_acquire(ctx), > while the iothread of that ctx is blocked at aio_poll(ctx, blocking). Here is a backtrace on s390. I need 2 or more disks, (one is not enough). Thread 5 (Thread 0x3fffb406910 (LWP 74602)): #0 0x000003fffc0bde8e in syscall () from /lib64/libc.so.6 #1 0x00000000801dd282 in futex_wait (val=4294967295, ev=0x8079c6c4 ) at /home/cborntra/REPOS/qemu/util/qemu-thread-posix.c:301 #2 qemu_event_wait (ev=ev@entry=0x8079c6c4 ) at /home/cborntra/REPOS/qemu/util/qemu-thread-posix.c:399 #3 0x00000000801ec75c in call_rcu_thread (opaque=) at /home/cborntra/REPOS/qemu/util/rcu.c:233 #4 0x000003fffc16f4e6 in start_thread () from /lib64/libpthread.so.0 #5 0x000003fffc0c30fa in thread_start () from /lib64/libc.so.6 Thread 4 (Thread 0x3fffabf7910 (LWP 74604)): #0 0x000003fffc0b75d6 in ppoll () from /lib64/libc.so.6 #1 0x000000008016fbd0 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=, nfds=, timeout=timeout@entry=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:310 #3 0x0000000080170d32 in aio_poll (ctx=0x807d6a70, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:274 #4 0x00000000800b3758 in iothread_run (opaque=0x807d6690) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffc16f4e6 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffc0c30fa in thread_start () from /lib64/libc.so.6 Thread 3 (Thread 0x3fffa3f7910 (LWP 74605)): #0 0x000003fffc0b75d6 in ppoll () from /lib64/libc.so.6 #1 0x000000008016fbd0 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=, nfds=, timeout=timeout@entry=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:310 #3 0x0000000080170d32 in aio_poll (ctx=0x807d94a0, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:274 #4 0x00000000800b3758 in iothread_run (opaque=0x807d6f60) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffc16f4e6 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffc0c30fa in thread_start () from /lib64/libc.so.6 Thread 2 (Thread 0x3fff8a21910 (LWP 74625)): #0 0x000003fffc0b90a2 in ioctl () from /lib64/libc.so.6 #1 0x0000000080056a46 in kvm_vcpu_ioctl (cpu=cpu@entry=0x81f3b620, type=type@entry=44672) at /home/cborntra/REPOS/qemu/kvm-all.c:1916 #2 0x0000000080056b08 in kvm_cpu_exec (cpu=cpu@entry=0x81f3b620) at /home/cborntra/REPOS/qemu/kvm-all.c:1775 #3 0x00000000800445de in qemu_kvm_cpu_thread_fn (arg=0x81f3b620) at /home/cborntra/REPOS/qemu/cpus.c:979 #4 0x000003fffc16f4e6 in start_thread () from /lib64/libpthread.so.0 #5 0x000003fffc0c30fa in thread_start () from /lib64/libc.so.6 Thread 1 (Thread 0x3fffb408bc0 (LWP 74580)): #0 0x000003fffc0b75d6 in ppoll () from /lib64/libc.so.6 #1 0x000000008016fbb0 in ppoll (__ss=0x0, __timeout=0x3ffffd64438, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=, nfds=, timeout=timeout@entry=999000000) at /home/cborntra/REPOS/qemu/qemu-timer.c:322 #3 0x000000008016f230 in os_host_main_loop_wait (timeout=999000000) at /home/cborntra/REPOS/qemu/main-loop.c:239 #4 main_loop_wait (nonblocking=) at /home/cborntra/REPOS/qemu/main-loop.c:494 #5 0x000000008001346a in main_loop () at /home/cborntra/REPOS/qemu/vl.c:1789 #6 main (argc=, argv=, envp=) at /home/cborntra/REPOS/qemu/vl.c:4391