From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=36382 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOFv8-0006gW-Um for qemu-devel@nongnu.org; Mon, 14 Jun 2010 16:06:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOFv3-0008Vq-LJ for qemu-devel@nongnu.org; Mon, 14 Jun 2010 16:06:10 -0400 Received: from mx1.redhat.com ([209.132.183.28]:15022) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOFv3-0008VV-ER for qemu-devel@nongnu.org; Mon, 14 Jun 2010 16:06:05 -0400 Date: Mon, 14 Jun 2010 17:05:59 -0300 From: Glauber Costa Subject: Re: [Qemu-devel] [PATCH] stop cpus before forking. Message-ID: <20100614200559.GE19104@mothafucka.localdomain> References: <1276543644-32689-1-git-send-email-glommer@redhat.com> <4C1683EC.3010609@codemonkey.ws> <20100614194237.GD19104@mothafucka.localdomain> <4C1689F7.4040004@codemonkey.ws> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C1689F7.4040004@codemonkey.ws> List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Anthony Liguori Cc: qemu-devel@nongnu.org, avi@redhat.com On Mon, Jun 14, 2010 at 02:58:47PM -0500, Anthony Liguori wrote: > On 06/14/2010 02:42 PM, Glauber Costa wrote: > >On Mon, Jun 14, 2010 at 02:33:00PM -0500, Anthony Liguori wrote: > >>On 06/14/2010 02:27 PM, Glauber Costa wrote: > >>>This patch fixes a bug that happens with kvm, irqchip-in-kernel, > >>>while adding a netdev. Despite the situations of reproduction being > >>>specific to kvm, I believe this fix is pretty generic, and fits here. > >>>Specially if we ever want to have our own irqchip in kernel too. > >>> > >>>The problem happens after the fork system call, and although it is not > >>>100 % reproduceable, happens pretty often. After fork, the memory where > >>>the apic is mapped is present in both processes. It ends up confusing > >>>the vcpus somewhere in the irq<-> ack path, and qemu hangs, with no > >>>irqs being delivered at all from that point on. > >>> > >>>Making sure the vcpus are stopped before forking makes the problem go > >>>away. Besides, this is a pretty unfrequent operation, which already hangs > >>>the io-thread for a while. So it should not hurt performance. > >>> > >>>Signed-off-by: Glauber Costa > >>This doesn't make very much sense to me but smells like a kernel bug to me. > >My interpretation is that by doing that, we make sure no in-flight > >requests are happening. Actually, a sleep(x), with x sufficiently big > >is enough to make this problem go away, but that is too hacky. > > vm_stop() is probably just acting a glorified sleep() since it has > to wait for each thread to stop. > > >I do agree that this is most likely a kernel bug. But as with any other > >kernel bugs, I believe this is a easy workaround to have things working > >even in older kernels until we fix it. > > If we don't know what the bug is, then we do not know whether this > is a work around. Rather, this change happens to make the bug more > difficult to reproduce with your test case. > > >>Even if it isn't, I can't rationalize why stopping the vm like this > >>is enough to fix such a problem. Is the problem that the KVM VCPU > >>threads get duplicated while potentially running or something like > >>that? > >I doubt fork is duplicating the vcpu threads. More than that, this > >bug does not happen with userspace irqchip. > >So I believe that either irq request or the ack itself is reaching the > >wrong process, forever stalling the apic. > > That sounds more like a signal delivery issue. It's not obvious to > me that we're doing the wrong thing with signal mask though. > > If it's a signal mask related issue, then vm_stop isn't a proper fix > as there would be still be a race. I do can investigate it further, but I doubt it is signal-delivery related. I spent the first days believing it was, but now, I believe it is much more likely to be apic-related. We don't need to wait for the child to exit for this bug to happen, so SIGCHLD is never raised. And with in-kernel irqchip, we don't deliver signals during normal vcpu execution.