From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3A13DEB64DD for ; Mon, 3 Jul 2023 10:37:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To: Subject:MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UQzKsrG7MF1XQwl3ldGDoAbNvckCKu7lKE59ljrAAdg=; b=FSvDLDPF5LYfSf 7WJLt0GJhzEQQgp/XhiGJKaL8+nBhw6pHfcuySFpH5pE5FMhquwWWPtzFma2Pbk7/evAKjX6QcqVn QIoafyf7O360baZeSyH38hLnKWnXfVwVhBHdiD/tFVKUKO17q+iwuM6jyiTv+gC2F2atVJlr4+Yjt 97Dt+moPsSUuAvfOepx7utXmDxxQhWbR8gwuRtBGE1yr+azQICjuYxpAi/Z2cfhiG9T/hz2d+ZRv7 KHICpq0D7zZeV1xUqbrv7lgsEFmiEdk7qYYWMOqSi2iVq41vv/HUwKuCHZwvAFalXzoxUh3mQbFwR 9kXH7W5Ml1sO62wHp+JQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qGGvk-00AGuI-2N; Mon, 03 Jul 2023 10:37:16 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qGGvi-00AGtR-29 for linux-arm-kernel@lists.infradead.org; Mon, 03 Jul 2023 10:37:16 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CDE442F4; Mon, 3 Jul 2023 03:37:53 -0700 (PDT) Received: from [10.57.28.201] (unknown [10.57.28.201]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 32D513F762; Mon, 3 Jul 2023 03:37:09 -0700 (PDT) Message-ID: Date: Mon, 3 Jul 2023 11:36:51 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Subject: Re: KVM CPU hotplug notifier triggers BUG_ON on arm64 Content-Language: en-US To: Marc Zyngier , Oliver Upton Cc: isaku.yamahata@intel.com, seanjc@google.com, pbonzini@redhat.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, James Morse References: <867crhxr9l.wl-maz@kernel.org> From: Kristina Martsenko In-Reply-To: <867crhxr9l.wl-maz@kernel.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230703_033714_773241_1294CCF6 X-CRM114-Status: GOOD ( 28.04 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 03/07/2023 10:45, Marc Zyngier wrote: > On Sat, 01 Jul 2023 18:42:28 +0100, > Oliver Upton wrote: >> >> Hi Kristina, >> >> Thanks for the bug report. >> >> On Sat, Jul 01, 2023 at 01:50:52PM +0100, Kristina Martsenko wrote: >>> Hi, >>> >>> When I try to online a CPU on arm64 while a KVM guest is running, I hit a >>> BUG_ON(preemptible()) (as well as a WARN_ON). See below for the full log. >>> >>> This is on kvmarm/next, but seems to have been broken since 6.3. Bisecting it >>> points at commit: >>> >>> 0bf50497f03b ("KVM: Drop kvm_count_lock and instead protect kvm_usage_count with kvm_lock") >> >> Makes sense. We were using a spinlock before, which implictly disables >> preemption. >> >> Well, one way to hack around the problem would be to just cram >> preempt_{disable,enable}() into kvm_arch_hardware_disable(), but that's >> kinda gross in the context of cpuhp which isn't migratable in the first >> place. Let me have a look... > > An alternative would be to replace the preemptible() checks with a one > that looks at the migration state, but I'm not sure that's much better > (it certainly looks more costly). > > There is also the fact that most of our per-CPU accessors are already > using preemption disabling, and this code has a bunch of them. So I'm > not sure there is a lot to be gained from not disabling preemption > upfront. > > Anyway, as I was able to reproduce the issue under NV, I tested the > hack below. If anything, I expect it to be a reasonable fix for > 6.3/6.4, and until we come up with a better approach. > > Thanks, > > M. > > diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c > index aaeae1145359..a28c4ffe4932 100644 > --- a/arch/arm64/kvm/arm.c > +++ b/arch/arm64/kvm/arm.c > @@ -1894,8 +1894,17 @@ static void _kvm_arch_hardware_enable(void *discard) > > int kvm_arch_hardware_enable(void) > { > - int was_enabled = __this_cpu_read(kvm_arm_hardware_enabled); > + int was_enabled; > > + /* > + * Most calls to this function are made with migration > + * disabled, but not with preemption disabled. The former is > + * enough to ensure correctness, but most of the helpers > + * expect the later and will throw a tantrum otherwise. > + */ > + preempt_disable(); > + > + was_enabled = __this_cpu_read(kvm_arm_hardware_enabled); > _kvm_arch_hardware_enable(NULL); > > if (!was_enabled) { > @@ -1903,6 +1912,8 @@ int kvm_arch_hardware_enable(void) > kvm_timer_cpu_up(); > } > > + preempt_enable(); > + > return 0; > } This fixes the issue for me. Tested-by: Kristina Martsenko Thanks, Kristina _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel