From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33075) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z2dIa-0001Ug-Nm for qemu-devel@nongnu.org; Wed, 10 Jun 2015 06:31:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Z2dIY-0005Z8-1b for qemu-devel@nongnu.org; Wed, 10 Jun 2015 06:31:56 -0400 Received: from e06smtp11.uk.ibm.com ([195.75.94.107]:51681) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z2dIX-0005Xl-OZ for qemu-devel@nongnu.org; Wed, 10 Jun 2015 06:31:53 -0400 Received: from /spool/local by e06smtp11.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 10 Jun 2015 11:31:51 +0100 Received: from b06cxnps4074.portsmouth.uk.ibm.com (d06relay11.portsmouth.uk.ibm.com [9.149.109.196]) by d06dlp02.portsmouth.uk.ibm.com (Postfix) with ESMTP id 1AD9E219005F for ; Wed, 10 Jun 2015 11:31:29 +0100 (BST) Received: from d06av12.portsmouth.uk.ibm.com (d06av12.portsmouth.uk.ibm.com [9.149.37.247]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id t5AAVnjH24641620 for ; Wed, 10 Jun 2015 10:31:50 GMT Received: from d06av12.portsmouth.uk.ibm.com (localhost [127.0.0.1]) by d06av12.portsmouth.uk.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id t5AAVnte021970 for ; Wed, 10 Jun 2015 04:31:49 -0600 Message-ID: <55781215.5030309@de.ibm.com> Date: Wed, 10 Jun 2015 12:31:49 +0200 From: Christian Borntraeger MIME-Version: 1.0 References: <556DBF87.2020908@de.ibm.com> <20150609022832.GA12817@cpc-pc.redhat.com> <5576AB52.8090708@de.ibm.com> <20150610021224.GE10873@ad.nay.redhat.com> <557800E0.5020202@de.ibm.com> <20150610093408.GC11648@ad.nay.redhat.com> In-Reply-To: <20150610093408.GC11648@ad.nay.redhat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] "iothread: release iothread around aio_poll" causes random hangs at startup List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Fam Zheng Cc: Kevin Wolf , Paolo Bonzini , qemu-devel , Stefan Hajnoczi Am 10.06.2015 um 11:34 schrieb Fam Zheng: > On Wed, 06/10 11:18, Christian Borntraeger wrote: >> Am 10.06.2015 um 04:12 schrieb Fam Zheng: >>> On Tue, 06/09 11:01, Christian Borntraeger wrote: >>>> Am 09.06.2015 um 04:28 schrieb Fam Zheng: >>>>> On Tue, 06/02 16:36, Christian Borntraeger wrote: >>>>>> Paolo, >>>>>> >>>>>> I bisected >>>>>> commit a0710f7995f914e3044e5899bd8ff6c43c62f916 >>>>>> Author: Paolo Bonzini >>>>>> AuthorDate: Fri Feb 20 17:26:52 2015 +0100 >>>>>> Commit: Kevin Wolf >>>>>> CommitDate: Tue Apr 28 15:36:08 2015 +0200 >>>>>> >>>>>> iothread: release iothread around aio_poll >>>>>> >>>>>> to cause a problem with hanging guests. >>>>>> >>>>>> Having many guests all with a kernel/ramdisk (via -kernel) and >>>>>> several null block devices will result in hangs. All hanging >>>>>> guests are in partition detection code waiting for an I/O to return >>>>>> so very early maybe even the first I/O. >>>>>> >>>>>> Reverting that commit "fixes" the hangs. >>>>>> Any ideas? >>>>> >>>>> Christian, I can't reproduce this on my x86 box with virtio-blk-pci. Do you >>>>> have a reproducer for x86? Or could you collect backtraces for all the threads >>>>> in QEMU when it hangs? >>>>> >>>>> My long shot is that the main loop is blocked at aio_context_acquire(ctx), >>>>> while the iothread of that ctx is blocked at aio_poll(ctx, blocking). >>>> >>>> Here is a backtrace on s390. I need 2 or more disks, (one is not enough). >>> >>> It shows iothreads and main loop are all waiting for events, and the vcpu >>> threads are running guest code. >>> >>> It could be the requests being leaked. Do you see this problem with a regular >>> file based image or null-co driver? Maybe we're missing something about the >>> AioContext in block/null.c. >> >> It seems to run with normal file based images. As soon as I have two or more null-aio >> devices it hangs pretty soon when doing a reboot loop. >> > > Ahh! If it's a reboot loop, the device reset thing may get fishy. I suspect the > completion BH used by null-aio may be messed up, that's why I wonder whether > null-co:// would work for you. Could you test that? null-co also fails. > > Also, could you try below patch with null-aio://, too? The same. Guests still get stuck. > > Thanks, > Fam > > --- > > diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c > index cd539aa..c87b444 100644 > --- a/hw/block/virtio-blk.c > +++ b/hw/block/virtio-blk.c > @@ -652,15 +652,11 @@ static void virtio_blk_reset(VirtIODevice *vdev) > { > VirtIOBlock *s = VIRTIO_BLK(vdev); > > - if (s->dataplane) { > - virtio_blk_data_plane_stop(s->dataplane); > - } > - > - /* > - * This should cancel pending requests, but can't do nicely until there > - * are per-device request lists. > - */ > blk_drain_all(); > + if (s->dataplane) { > + virtio_blk_data_plane_stop(s->dataplane); > + } > + > blk_set_enable_write_cache(s->blk, s->original_wce); > } >