From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=35377 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOFoJ-0000Gx-Vd for qemu-devel@nongnu.org; Mon, 14 Jun 2010 15:59:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOFo1-0007KP-4M for qemu-devel@nongnu.org; Mon, 14 Jun 2010 15:58:50 -0400 Received: from mail-iw0-f173.google.com ([209.85.214.173]:39684) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOFo0-0007KI-Vt for qemu-devel@nongnu.org; Mon, 14 Jun 2010 15:58:49 -0400 Received: by iwn10 with SMTP id 10so4529879iwn.4 for ; Mon, 14 Jun 2010 12:58:48 -0700 (PDT) Message-ID: <4C1689F7.4040004@codemonkey.ws> Date: Mon, 14 Jun 2010 14:58:47 -0500 From: Anthony Liguori MIME-Version: 1.0 Subject: Re: [Qemu-devel] [PATCH] stop cpus before forking. References: <1276543644-32689-1-git-send-email-glommer@redhat.com> <4C1683EC.3010609@codemonkey.ws> <20100614194237.GD19104@mothafucka.localdomain> In-Reply-To: <20100614194237.GD19104@mothafucka.localdomain> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Glauber Costa Cc: qemu-devel@nongnu.org, avi@redhat.com On 06/14/2010 02:42 PM, Glauber Costa wrote: > On Mon, Jun 14, 2010 at 02:33:00PM -0500, Anthony Liguori wrote: > >> On 06/14/2010 02:27 PM, Glauber Costa wrote: >> >>> This patch fixes a bug that happens with kvm, irqchip-in-kernel, >>> while adding a netdev. Despite the situations of reproduction being >>> specific to kvm, I believe this fix is pretty generic, and fits here. >>> Specially if we ever want to have our own irqchip in kernel too. >>> >>> The problem happens after the fork system call, and although it is not >>> 100 % reproduceable, happens pretty often. After fork, the memory where >>> the apic is mapped is present in both processes. It ends up confusing >>> the vcpus somewhere in the irq<-> ack path, and qemu hangs, with no >>> irqs being delivered at all from that point on. >>> >>> Making sure the vcpus are stopped before forking makes the problem go >>> away. Besides, this is a pretty unfrequent operation, which already hangs >>> the io-thread for a while. So it should not hurt performance. >>> >>> Signed-off-by: Glauber Costa >>> >> This doesn't make very much sense to me but smells like a kernel bug to me. >> > My interpretation is that by doing that, we make sure no in-flight > requests are happening. Actually, a sleep(x), with x sufficiently big > is enough to make this problem go away, but that is too hacky. > vm_stop() is probably just acting a glorified sleep() since it has to wait for each thread to stop. > I do agree that this is most likely a kernel bug. But as with any other > kernel bugs, I believe this is a easy workaround to have things working > even in older kernels until we fix it. > If we don't know what the bug is, then we do not know whether this is a work around. Rather, this change happens to make the bug more difficult to reproduce with your test case. >> Even if it isn't, I can't rationalize why stopping the vm like this >> is enough to fix such a problem. Is the problem that the KVM VCPU >> threads get duplicated while potentially running or something like >> that? >> > I doubt fork is duplicating the vcpu threads. More than that, this > bug does not happen with userspace irqchip. > So I believe that either irq request or the ack itself is reaching the > wrong process, forever stalling the apic. > That sounds more like a signal delivery issue. It's not obvious to me that we're doing the wrong thing with signal mask though. If it's a signal mask related issue, then vm_stop isn't a proper fix as there would be still be a race. Regards, Anthony Liguori