From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ECC0E27F72E for ; Tue, 10 Jun 2025 07:35:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749540905; cv=none; b=S42yg4PCP6tfvvRYyBFM/IFVluiPFYriwgK/jQy0fAfXO76SV+sXqnMIYnSGteDcBajadwgFQ36ptwX4xogQIDSaDH3DPQjgfIO0zt1GAxdbq8+qPIE222+WEBI77cmiJkb7Dg5VVQHa2vKpU2TNxlH6YaYtFBHgbiDwecDbLJ4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749540905; c=relaxed/simple; bh=gy5bzrQ2yPlP6+aqbDxNMn7VUHz5wr3arme8eeRkcZg=; h=Date:Message-ID:From:To:Cc:Subject:In-Reply-To:References: MIME-Version:Content-Type; b=OnjVN9pWRyLf4+mWj7E453fU+/eUdY4qn5JWf3GDszrIFi1oge1F70sZctWIcUjy/DS6cjn2d2SjPYADZmgTTnpOWJIroDa+qDUtbZuFAsjTvAdDfBHp2NJzImhwZ09a0yCipbWM138ZHqIcK++V0KcFs0EpLtJKP17rXfa1o5E= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=pZARxZxF; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="pZARxZxF" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 86878C4CEEF; Tue, 10 Jun 2025 07:35:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1749540904; bh=gy5bzrQ2yPlP6+aqbDxNMn7VUHz5wr3arme8eeRkcZg=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=pZARxZxFpno0IXqPjqIsuSTWv80hS09Z7YjCUVZbYgiuvH0jNMM1CG/9fxHaiNmRZ AkUDSf9UWGxGJmCaips1Z8IbSqajd1R08MM704EQIHjvsgdzDZVm7/521o65t0mCG4 G27cqp/NJB0YOwmw0o50WS9v0U7Weq3715z5utCeMRQhLTirdo2Wkj2Jcm+k2JvDUl q01huFXqhvKB6xOBbD9o/A/9qbvKRLdIR4bsS/Ll7tVQbM6VkcBuRx7AfLvuRBegGc 2AxWDlwIvYOuJKOxG7ZPOGX2MirwcmhmNItjkec74mk46daQA/iGiS8XkDhyxq462b 4h7AosVYGhk+A== Received: from 91-161-240-24.subs.proxad.net ([91.161.240.24] helo=lobster-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1uOtVe-005PgI-5V; Tue, 10 Jun 2025 08:35:02 +0100 Date: Tue, 10 Jun 2025 08:34:58 +0100 Message-ID: <87ldq0f3rx.wl-maz@kernel.org> From: Marc Zyngier To: Mostafa Saleh Cc: "Aneesh Kumar K.V" , kvmarm@lists.linux.dev, Will Deacon , Quentin Perret Subject: Re: pkvm boot failures In-Reply-To: References: User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/30.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 91.161.240.24 X-SA-Exim-Rcpt-To: smostafa@google.com, aneesh.kumar@kernel.org, kvmarm@lists.linux.dev, will@kernel.org, qperret@google.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Hi Mostafa, Thanks for looking into this. On Mon, 09 Jun 2025 18:25:15 +0100, Mostafa Saleh wrote: >=20 > On Mon, Jun 09, 2025 at 06:53:40PM +0530, Aneesh Kumar K.V wrote: > >=20 > > I am hitting the below failure with v6.15 (I tried other kernel versions > > with similar results). I disabled CONFIG_PROTECTED_NVHE_STACKTRACE > > because with CONFIG_NVHE_EL2_DEBUG, the stack was pointing at > > hyp_assert_lock_held() . > >=20 > > [ 0.664457] kvm [1]: nVHE hyp panic at: [] __kvm_n= vhe_handle_trap+0x34/0x10c! > > [ 0.664538] kvm [1]: Cannot dump pKVM nVHE stacktrace: !CONFIG_PROTE= CTED_NVHE_STACKTRACE > > [ 0.664566] kvm [1]: Hyp Offset: 0xffff000007c00000 > > [ 0.664631] Kernel panic - not syncing: HYP panic: > > [ 0.664631] PS:614023c9 PC:000080007890b10c ESR:0000000096000007 > > [ 0.664631] FAR:0000800078c252f0 HPFAR:0000000000000000 PAR:00000000= 00000000 > > [ 0.664631] VCPU:0000000000000000 > > [ 0.664938] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.15.0-= rc1 #594 NONE=20 > > [ 0.665068] Hardware name: FVP Base RevC (DT) > > [ 0.665140] Call trace: > > [ 0.665196] show_stack+0x18/0x24 (C) > > [ 0.665346] dump_stack_lvl+0x3c/0x80 > > [ 0.665468] dump_stack+0x18/0x24 > > [ 0.665588] panic+0x124/0x2d8 > > [ 0.665699] nvhe_hyp_panic_handler+0x108/0x180 > > [ 0.665825] do_pkvm_init+0xb0/0x124 > > [ 0.665957] do_pkvm_init+0xb0/0x124 > > [ 0.666089] kvm_hyp_init_protection+0x5c/0x6c > > [ 0.666226] init_hyp_mode+0x760/0x790 > > [ 0.666362] kvm_arm_init+0xac/0x23c > > [ 0.666492] do_one_initcall+0xa0/0x1f0 > > [ 0.666617] do_initcall_level+0x8c/0xac > > [ 0.666753] do_initcalls+0x54/0x94 > > [ 0.666885] do_basic_setup+0x18/0x24 > > [ 0.667019] kernel_init_freeable+0xc0/0x10c > > [ 0.667157] kernel_init+0x20/0x118 > > [ 0.667271] ret_from_fork+0x10/0x20 > > [ 0.667400] SMP: stopping secondary CPUs > > [ 0.667475] Kernel Offset: disabled > > [ 0.667534] CPU features: 0x0000,00000140,064dc298,cb7a552f > > [ 0.667619] Memory Limit: none > > [ 0.667681] ---[ end Kernel panic - not syncing: HYP panic: > > [ 0.667681] PS:614023c9 PC:000080007890b10c ESR:0000000096000007 > > [ 0.667681] FAR:0000800078c252f0 HPFAR:0000000000000000 PAR:00000000= 00000000 > > [ 0.667681] VCPU:0000000000000000 ] > >=20 > > I was able to locate a .config that make the pkvm work, But i am not > > able to identify which config dependency is making the difference. I am > > attaching below the working and non working kernel configs. I am using > > FVP to test this. > >=20 >=20 > I had a look at this and tracked the issue to "CONFIG_JUMP_LABEL=3Dn" > It seems that it panics at > if (static_branch_unlikely(&kvm_protected_mode_initialized)) > Where "kvm_protected_mode_initialized" is mapped in the initial PGD for t= he > hypervisor, but not mapped in the hypervisor created one. > As the variable is defined outside the hypervisor namespace, it doesn=E2= =80=99t exist > in the hyp bss section. > And in case of "CONFIG_JUMP_LABEL=3Dn" it won't be patched in this case,= causing > next access to read the variable and panicking. It really begs the question: why do we even support JUMP_LABEL=3Dn? It really feels like a backward configuration, and I'd be very glad to either mark it as "always on", or make KVM depend on it. > I guess moving this key to hyp would cause problems with kernel access af= ter > de-privilege as cases from kvm_share_hyp(), So I can only think of having= a > different key for the hypervisor as >=20 > diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe= /hyp-main.c > index 8e8848de4d47..8945b335bcea 100644 > --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c > +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c > @@ -21,6 +21,7 @@ > #include > =20 > DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); > +DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized_hyp); > =20 > void __kvm_hyp_host_forward_smc(struct kvm_cpu_context *host_ctxt); > =20 > @@ -626,7 +627,7 @@ static void handle_host_hcall(struct kvm_cpu_context = *host_ctxt) > * basis. This is all fine, however, since __pkvm_prot_finalize > * returns -EPERM after the first call for a given CPU. > */ > - if (static_branch_unlikely(&kvm_protected_mode_initialized)) > + if (static_branch_unlikely(&kvm_protected_mode_initialized_hyp)) > hcall_min =3D __KVM_HOST_SMCCC_FUNC___pkvm_prot_finalize; > =20 > id &=3D ~ARM_SMCCC_CALL_HINTS; > diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c > index fcd70bfe44fb..af0854e98902 100644 > --- a/arch/arm64/kvm/pkvm.c > +++ b/arch/arm64/kvm/pkvm.c > @@ -17,6 +17,7 @@ > #include "hyp_constants.h" > =20 > DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized); > +DECLARE_STATIC_KEY_FALSE(kvm_nvhe_sym(kvm_protected_mode_initialized_hyp= )); > =20 > static struct memblock_region *hyp_memory =3D kvm_nvhe_sym(hyp_memory); > static unsigned int *hyp_memblock_nr_ptr =3D &kvm_nvhe_sym(hyp_memblock_= nr); > @@ -229,6 +230,7 @@ static int __init pkvm_drop_host_privileges(void) > * once the host stage 2 is installed. > */ > static_branch_enable(&kvm_protected_mode_initialized); > + static_branch_enable(&kvm_nvhe_sym(kvm_protected_mode_initialized_hyp)); > on_each_cpu(_kvm_host_prot_finalize, &ret, 1); > return ret; > } > I don't really enjoy this duplication, and unless we have a good reason not too, I'd rather have something like: diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 713248f240e0..66d232e7c894 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -37,6 +37,7 @@ menuconfig KVM select HAVE_KVM_VCPU_RUN_PID_CHANGE select SCHED_INFO select GUEST_PERF_EVENTS if PERF_EVENTS + select JUMP_LABEL help Support hosting virtualized guest machines. =20 It should be OK now that all the supported compilers have asm goto support. Thanks, M. --=20 Jazz isn't dead. It just smells funny.