From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:41792)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <famz@redhat.com>) id 1Z29Gn-00014l-Pl
	for qemu-devel@nongnu.org; Mon, 08 Jun 2015 22:28:07 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <famz@redhat.com>) id 1Z29Gk-0007vw-IQ
	for qemu-devel@nongnu.org; Mon, 08 Jun 2015 22:28:05 -0400
Received: from mx1.redhat.com ([209.132.183.28]:43676)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <famz@redhat.com>) id 1Z29Gk-0007vp-Ai
	for qemu-devel@nongnu.org; Mon, 08 Jun 2015 22:28:02 -0400
Date: Tue, 9 Jun 2015 10:28:32 +0800
From: Fam Zheng <famz@redhat.com>
Message-ID: <20150609022832.GA12817@cpc-pc.redhat.com>
References: <556DBF87.2020908@de.ibm.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <556DBF87.2020908@de.ibm.com>
Subject: Re: [Qemu-devel] "iothread: release iothread around aio_poll"
 causes random hangs at startup
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: Christian Borntraeger <borntraeger@de.ibm.com>
Cc: Kevin Wolf <kwolf@redhat.com>, Paolo Bonzini <pbonzini@redhat.com>, qemu-devel <qemu-devel@nongnu.org>, Stefan Hajnoczi <stefanha@redhat.com>

On Tue, 06/02 16:36, Christian Borntraeger wrote:
> Paolo,
> 
> I bisected 
> commit a0710f7995f914e3044e5899bd8ff6c43c62f916
> Author:     Paolo Bonzini <pbonzini@redhat.com>
> AuthorDate: Fri Feb 20 17:26:52 2015 +0100
> Commit:     Kevin Wolf <kwolf@redhat.com>
> CommitDate: Tue Apr 28 15:36:08 2015 +0200
> 
>     iothread: release iothread around aio_poll
> 
> to cause a problem with hanging guests.
> 
> Having many guests all with a kernel/ramdisk (via -kernel) and
> several null block devices will result in hangs. All hanging 
> guests are in partition detection code waiting for an I/O to return
> so very early maybe even the first I/O.
> 
> Reverting that commit "fixes" the hangs.
> Any ideas?

Christian, I can't reproduce this on my x86 box with virtio-blk-pci. Do you
have a reproducer for x86? Or could you collect backtraces for all the threads
in QEMU when it hangs?

My long shot is that the main loop is blocked at aio_context_acquire(ctx),
while the iothread of that ctx is blocked at aio_poll(ctx, blocking).

Thanks,
Fam

> 
> Christian
> 
> PS: A guest xml looks like
> 
> 
> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
>   <name>test2</name>
>   <uuid>13bd8253-9abb-4be8-9399-73b899762aaa</uuid>
>   <memory unit='KiB'>286720</memory>
>   <currentMemory unit='KiB'>286720</currentMemory>
>   <vcpu placement='static'>10</vcpu>
>   <iothreads>8</iothreads>
>   <cputune>
>     <shares>12</shares>
>   </cputune>
>   <os>
>     <type arch='s390x' machine='s390-ccw-virtio'>hvm</type>
>     <kernel>/boot/vmlinux-4.0.0+</kernel>
>     <initrd>/boot/ramdisk.reboot</initrd>
>     <cmdline>root=/dev/ram0</cmdline>
>     <boot dev='hd'/>
>   </os>
>   <clock offset='utc'/>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>preserve</on_crash>
>   <devices>
>     <controller type='usb' index='0' model='none'/>
>     <console type='pty'>
>       <target type='sclp' port='0'/>
>     </console>
>     <memballoon model='none'/>
>   </devices>
>   <qemu:commandline>
>     <qemu:arg value='-drive'/>
>     <qemu:arg value='driver=null-aio,id=null1,if=none,size=100G'/>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='virtio-blk-ccw,drive=null1,serial=null1,iothread=iothread1'/>
>     <qemu:arg value='-drive'/>
>     <qemu:arg value='driver=null-aio,id=null2,if=none,size=100G'/>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='virtio-blk-ccw,drive=null2,serial=null2,iothread=iothread2'/>
>     <qemu:arg value='-drive'/>
>     <qemu:arg value='driver=null-aio,id=null3,if=none,size=100G'/>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='virtio-blk-ccw,drive=null3,serial=null3,iothread=iothread3'/>
>     <qemu:arg value='-drive'/>
>     <qemu:arg value='driver=null-aio,id=null4,if=none,size=100G'/>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='virtio-blk-ccw,drive=null4,serial=null4,iothread=iothread4'/>
>     <qemu:arg value='-drive'/>
>     <qemu:arg value='driver=null-aio,id=null5,if=none,size=100G'/>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='virtio-blk-ccw,drive=null5,serial=null5,iothread=iothread5'/>
>     <qemu:arg value='-drive'/>
>     <qemu:arg value='driver=null-aio,id=null6,if=none,size=100G'/>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='virtio-blk-ccw,drive=null6,serial=null6,iothread=iothread6'/>
>     <qemu:arg value='-drive'/>
>     <qemu:arg value='driver=null-aio,id=null7,if=none,size=100G'/>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='virtio-blk-ccw,drive=null7,serial=null7,iothread=iothread7'/>
>     <qemu:arg value='-drive'/>
>     <qemu:arg value='driver=null-aio,id=null8,if=none,size=100G'/>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='virtio-blk-ccw,drive=null8,serial=null8,iothread=iothread8'/>
>   </qemu:commandline>
> </domain>
> 
>