From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8E9E9C02181 for ; Mon, 20 Jan 2025 16:15:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:MIME-Version: References:In-Reply-To:Subject:Cc:To:From:Message-ID:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ezA128kN3DbOsMEo6R/fP3sEwsyFq+LurIo96wI+RZE=; b=bMow4bJwbFuoHr6QXrewFIvvfu apEkJaeiA7PI0lsnx19PROLFgB294So/2KRIcJ9GNfMtgIqWOl6Tw5kEonK6hgPf6kiDP8oO3GjuP xJ5xrU+Do35KoJ9R3ArpMMQV8/6An2B1b54dHdNat0F4WVQjUPHncr9LSdW+332AnyHL5CzVpjWnM 39epeZm1WMfbAmx9eaogu3so2HmKbfcguySiYZOdJmgvcLv2Kj6Bd3kPMo5nbQJ2gi9L+V54gtbl5 W0IbS+QKlVpmWgXOf3AMYiqvtaXu01UnpOXQ1079C4ImadBuhjrucLD5iWmNfW8dFk/HI7sv8LknG jRPvb4Og==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tZuQu-00000005ySX-1auO; Mon, 20 Jan 2025 16:15:24 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tZuPZ-00000005y0i-0Tep for linux-arm-kernel@lists.infradead.org; Mon, 20 Jan 2025 16:14:02 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id C5C715C5CE4; Mon, 20 Jan 2025 16:13:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F21DCC4CEDD; Mon, 20 Jan 2025 16:13:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1737389640; bh=dicmTpGKvGRXydkJyA8oNSHXAQRDfmL3ALzcwM2h/WQ=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=oqP7OfGMqMzURcSeBNIVeKnwIDMUVULDAJQ1n9MWo51RuG9UD3xAmRGubWaVePZhQ MXnOTQ9UatSTr7S9tBTOtsHBmVYX/t9p2g6WZLmQvy93Xl1RDJbWnXxnAoDRNBx7Wm bK3UmlyaE0SzKjYtpE0ZylFlVsxuPzMaAyM4C5PnXXTnkqDUvyE+Tn37f0DDhwFn76 xbebHmZUotdfSsARzX/sKDBDFpr/MGX/VHuWoIzIHIIKMxJlcPzO0g4twa0IwPcIzQ KHHh02P1yg4/wXrK5hpKmArOfXi3UpF8fyeSN5QSRZVwj54qGQSdPj5WX7az4R08jP 8WdEgtW2VNmxQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1tZuPV-00DofX-10; Mon, 20 Jan 2025 16:13:57 +0000 Date: Mon, 20 Jan 2025 16:13:56 +0000 Message-ID: <86sepdv4xn.wl-maz@kernel.org> From: Marc Zyngier To: Lokesh Vutla Cc: Oliver Upton , Joey Gouly , Suzuki K Poulose , Catalin Marinas , Will Deacon , Fuad Tabba , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, sachinparekh@google.com Subject: Re: [PATCH] KVM: arm64: Flush hyp bss section after initialization of variables in bss In-Reply-To: <20250120151514.1769597-1-lokeshvutla@google.com> References: <20250120151514.1769597-1-lokeshvutla@google.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/29.4 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: lokeshvutla@google.com, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org, tabba@google.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, sachinparekh@google.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250120_081401_235785_396AF56F X-CRM114-Status: GOOD ( 30.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, 20 Jan 2025 15:15:14 +0000, Lokesh Vutla wrote: > > To determine CPU features during initialization, the NVHE hypervisor s/NVHE/nVHE/ > utilizes sanitized values of the host's CPU features registers. These > values, stored in u64 idaa64*_el1_sys_val variables are updated by the > kvm_hyp_init_symbols() function at EL1. To ensure EL2 visibility, the visibility *with the MMU off* > data cache needs to be flushed after these updates. However, > individually flushing each variable using kvm_flush_dcache_to_poc() is > inefficient. > > These cpu feature variables would be part of the bss section of > the hypervisor. Hence, flush the entire bss section of hypervisor > once the initialization is complete. > > Motivation for this change: > * Since the existing variables are not flushed from EL1, the > id_aa64pfr0_el1_sys_val is seen as 0 from EL2. > * based on this value check_override macro in hypervisor skips > updating the sve (cpacr_el1) at finalise_el2_state. > * The default value for cpacr_el1 enables the sve traps to EL2. > * With sve enabled, during the context switch from EL0 -> EL1 (which is > much later in the boot process), the sve registers are saved/restored. > * Since sve traps are enabled, accessing sve registers at EL1 caused a > trap to EL2. > * However, hypervisor is not ready to handle sve traps at this stage > causing the below kernel crash during the boot: Drop this section, it doesn't bring much to the discussion. > > [ 0.320850][ T1] Run /init as init process > [ 0.321392][ T1] kvm [1]: nVHE hyp BUG at: [] __kvm_nvhe_$x.24+0x254/0x254! > [ 0.321522][ T1] kvm [1]: Cannot dump pKVM nVHE stacktrace: !CONFIG_PROTECTED_NVHE_STACKTRACE > [ 0.321635][ T1] kvm [1]: Hyp Offset: 0xffffff6e60000000 > [ 0.321710][ T1] Kernel panic - not syncing: HYP panic: > [ 0.321710][ T1] PS:634023c9 PC:000000522112ee8c ESR:00000000f2000800 > [ 0.321710][ T1] FAR:0000000000000000 CPACR:0000000000310000 PAR:0000000000000800 > [ 0.321710][ T1] VCPU:0000000000000000 > [...] > [ 0.322251][ T1] Call trace: > [ 0.322292][ T1] dump_backtrace+0x100/0x180 > [ 0.322355][ T1] show_stack+0x20/0x30 > [ 0.322410][ T1] dump_stack_lvl+0x40/0x88 > [ 0.322471][ T1] dump_stack+0x18/0x24 > [ 0.322523][ T1] panic+0x13c/0x364 > [ 0.322578][ T1] nvhe_hyp_panic_handler+0x148/0x1cc > [ 0.322646][ T1] do_sve_acc+0xec/0x260 > [ 0.322706][ T1] el0_sve_acc+0x34/0x68 This is essentially content-free, given that there is no backtrace. Please drop this. > > Fixes: 6c30bfb18d0b ("KVM: arm64: Add handlers for protected VM System Registers") > Suggested-by: Fuad Tabba > Signed-off-by: Lokesh Vutla > --- > arch/arm64/kvm/arm.c | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c > index a102c3aebdbc..5d3b2069a2d5 100644 > --- a/arch/arm64/kvm/arm.c > +++ b/arch/arm64/kvm/arm.c > @@ -2661,6 +2661,12 @@ static int __init init_hyp_mode(void) > } > } > > + /* > + * Flush entire BSS since part of its data is read while the MMU is off. > + */ > + kvm_flush_dcache_to_poc(kvm_ksym_ref(__hyp_bss_start), > + kvm_ksym_ref(__hyp_bss_end) - kvm_ksym_ref(__hyp_bss_start)); > + > return 0; > > out_err: I don't understand how this fixes anything. At this stage, the hypervisor has already been initialised, and I expect it will have evaluated the wrong values. Even worse, I strongly suspect that by the time you perform this, S2 is enabled on the host, and that the BSS is off-limit. Which means it could fault and send you to lalaland. Have you actually tested this with upstream? I would have expected the clean operations to be called from kvm_hyp_init_symbols(), which runs before EL2 gets initialised in protected mode. M. -- Without deviation from the norm, progress is not possible.