From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 14373C00A5A for ; Thu, 19 Jan 2023 14:03:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Subject:Cc:To:From:Message-ID:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xxKV/7H0NPW83d64PyqNqM2SD4OB5d/RXZqZ4Yc12qk=; b=2ADGKx1yOtltSU +9rTzOejfjEEi65KnO4mygFuqgN9ulSa1JEtNfQ4KAkMtTAGf9VlEmnpvdjA58b8rZAerwQ1VsD3O 0/FFuGZ+uwMlWFUWTOfuIWaUCGD7zvLeX3YuwzZN+wJ3/j2uIfC35RBIJp8AJ1v4wn2o8ZMeOxVu/ pD0ou2ZRI/QI682n0aw8dEeHjTn6naFvCb/PwZhPZwOQu1iXc9rcqz5PHZsaB1zuyq0BGjEfFnWFn t7W903qT6ZmwpKgcX8Bn3cD+aym7dO+elok3B3MLIorbQZOEO+n0fN4TLv3lxJx1AKBG5wfIaiicR jYDQGYBWdE13qQuLOlWw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pIVUZ-005Cx5-C0; Thu, 19 Jan 2023 14:02:11 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pIVUM-005Cql-AL for linux-arm-kernel@lists.infradead.org; Thu, 19 Jan 2023 14:02:00 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 554A7B82424; Thu, 19 Jan 2023 14:01:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0CE7DC433F0; Thu, 19 Jan 2023 14:01:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1674136913; bh=ERwloxsMur2+1ZyE7gnOd8YyhrMcQ6tIXzM9/y1/rDM=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=HLRHn7MBfhC0W22B9ymDJLKzDe7Y/fGinfwy2LBfQJ1b2p6mCx54AL3tIvPcmWsAZ YyCJcgl7uAZMy8NvxLW93uj+GX2CH5MOdjgX4FaIms6a2uUuAvycGtiCYsaRhBcuvG cjrEGtjuVosd9NQIc8C9VAdm9axnSpeOOlFkQW17txpGngqqYPJ3F0+bikKcitkQ6C 2vN6lBlwGeXoDyOEV8XbPGr9U+lMjVZQPMVGsuf0/CfoEZHk+XCBH7CZyeeg8j3pKM nMXM1jCjQLYXg1o0fZPA72JUHt7hHxS+OVY40qZgK4bM422VMcgKT4g56R0X1ocU5G ybzvbn13qM9MA== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pIVUE-0036Tt-OQ; Thu, 19 Jan 2023 14:01:50 +0000 Date: Thu, 19 Jan 2023 14:01:50 +0000 Message-ID: <86wn5imxm9.wl-maz@kernel.org> From: Marc Zyngier To: Shanker Donthineni Cc: James Morse , Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, Vikram Sethi , Zenghui Yu , Oliver Upton , Suzuki K Poulose , Ard Biesheuvel Subject: Re: [PATCH] KVM: arm64: vgic: Fix soft lockup during VM teardown In-Reply-To: <2e0c971a-0199-ff0d-c13c-d007d9f03122@nvidia.com> References: <20230118022348.4137094-1-sdonthineni@nvidia.com> <863588njmt.wl-maz@kernel.org> <28061ceb-a7ce-0aca-a97d-8227dcfe6800@nvidia.com> <87bkmvdmna.wl-maz@kernel.org> <2e0c971a-0199-ff0d-c13c-d007d9f03122@nvidia.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/28.2 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: sdonthineni@nvidia.com, james.morse@arm.com, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, vsethi@nvidia.com, yuzenghui@huawei.com, oliver.upton@linux.dev, suzuki.poulose@arm.com, ardb@kernel.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230119_060158_675580_65F80CA8 X-CRM114-Status: GOOD ( 35.51 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, 19 Jan 2023 13:00:49 +0000, Shanker Donthineni wrote: > > > > On 1/19/23 01:11, Marc Zyngier wrote: > > So you can see the VM being torn down while the vgic save sequence is > > still in progress? > > > > If you can actually see that, then this is a much bigger bug than the > > simple race you are describing, and we're missing a reference on the > > kvm structure. This would be a *MAJOR* bug. > > > How do we know vGIC save sequence is in progress while VM is being > teardown? I'm launching/terminating ~32 VMs in a loop to reproduce > the issue. Errr... *you* know when you are issuing the save ioctl, right? You also know when you are terminating the VM (closing its fd or killing the VMM). > > > Please post the full traces, not snippets. The absolutely full kernel > > log, the configuration, what you run, how you run it, *EVERYTHING*. I > > need to be able to reproduce this. > Sure, I'll share the complete boot log messages of host kernel next run. > > > > >> > >>>> > >>>> irqreturn_t handle_irq_event(struct irq_desc *desc) > >>>> { > >>>> irqd_set(&desc->irq_data, IRQD_IRQ_INPROGRESS); > >>>> raw_spin_unlock(&desc->lock); > >>>> > >>>> ret = handle_irq_event_percpu(desc); > >>>> > >>>> raw_spin_lock(&desc->lock); > >>>> irqd_clear(&desc->irq_data, IRQD_IRQ_INPROGRESS); > >>>> } > >>> > >>> How is that relevant to this trace? Do you see this function running > >>> concurrently with the teardown? If it matters here, it must be a VPE > >>> doorbell, right? But you claim that this is on a GICv4 platform, while > >>> this would only affect GICv4.1... Or are you using GICv4.1? > >>> > >> handle_irq_event() is running concurrently with irq_domain_activate_irq() > >> which happens before free_irq() called. Corruption at [78.983544] and > >> teardown started at [87.360891]. > > > > But that doesn't match the description you made of concurrent > > events. Does it take more than 9 seconds for the vgic state to be > > saved to memory? > > Are there any other possibilities of corrupting IRQD_IRQ_INPROGRESS > state bit other than concurrent accesses? Forget about this bit. You said that we could see the VM teardown happening *at the same time* as the vgic state saving, despite the vgic device holding a reference on the kvm structure. If that's the case, this bit is the least of our worries. Think of the consequences for a second... [...] > Using the below steps for launching/terminating 32 VMs in loop. The > failure is intermittent. The same issue is reproducible with KVMTOOL > also. kvmtool never issue a KVM_DEV_ARM_VGIC_GRP_CTRL with the KVM_DEV_ARM_ITS_SAVE_TABLES argument, so the code path we discussed is never used. What is the exact problem you're observing with kvmtool as the VMM? M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel