From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C09E91339A4 for ; Wed, 8 Oct 2025 09:28:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759915688; cv=none; b=K6OJvVsga7XESItu1NTt57e12s2zjFRo+OKCPtwtjJSM+SPPSoHS3LSbLTlkBcK6EJJChQIn/w5B1HyVY1flMtPjjsom3NoJpac630qCYFj7gTaNOWODamdzn9/+HymwMWebgVMDOnPxeeaTsSoR+LH9x219uyLh6xpKDTLG1F8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759915688; c=relaxed/simple; bh=cfS9Je3tPGiIOoD/0inqgXV1e6swXVq6ZFjHKzhN2d0=; h=Date:Message-ID:From:To:Cc:Subject:In-Reply-To:References: MIME-Version:Content-Type; b=I7NnqDm59Wn2BL64/8oLZYG9EVBnZMjIRAf0ARq8eyA9oqRiQ15eQcNhSM2g+msjXXExsg3u13tUbv7xrc/+cBgG+3fHobioAPJrwn3qyCJYqhWhAOhP5+jlGSmrzSzr5lnaJsHh/fqXIfdQV6dzHcPZ8qXXb8DRjvur8i5y73g= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=acvM4Hqv; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="acvM4Hqv" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 942A5C4CEF4; Wed, 8 Oct 2025 09:28:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1759915688; bh=cfS9Je3tPGiIOoD/0inqgXV1e6swXVq6ZFjHKzhN2d0=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=acvM4HqvRI/CaygIgLdWvtX5bDgA/wFnz1UaVh2Wz2Ioh8X/Tooy2ghIy1aESEC+2 boYMAvEC3wHd5dpyT9LPPYnR30QjMzeJgSFV0o4JsxLZSxFknwWZrg3FCZ0bU34gLI i++i+5WK9Y9mA9I7q1Du+jkGGrIfBCtEAd+gRu/cd3KMUfJRnU/pD51Q/GTX2A0PvE xO3Gk3I15O0ftEsFIa7xYf6TZYdbWzNKcm4FsIfFB/Jg+wr9tjwxOZkLa92RnwE53A Eai8m8byo710oZVedOCVHmXcTu0yvw1oPaOLLsoUyiRe8wwTRDNxH/Z50V9VO7MNz2 MF/rLAWoNhi1A== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1v6QSs-0000000CJEW-1UI5; Wed, 08 Oct 2025 09:28:06 +0000 Date: Wed, 08 Oct 2025 10:28:06 +0100 Message-ID: <864is9zqs9.wl-maz@kernel.org> From: Marc Zyngier To: Jan Kotas Cc: Oliver Upton , "kvmarm@lists.linux.dev" Subject: Re: KVM NV + SVE host OS warning In-Reply-To: References: <799DD5E5-8BC2-47B3-A919-33429D3FB2F1@global.cadence.com> <865xd61tt5.wl-maz@kernel.org> <864isq1r66.wl-maz@kernel.org> <25C5E00D-62BC-4188-8642-21913446B32C@global.cadence.com> <1271032F-41BB-4896-AAED-8660D5459E7D@global.cadence.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/30.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: jank@cadence.com, oliver.upton@linux.dev, kvmarm@lists.linux.dev X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false On Wed, 08 Oct 2025 08:29:38 +0100, Jan Kotas wrote: >=20 >=20 >=20 > >=20 > > On 8 Oct 2025, at 08:32, Jan Kotas wrote: > >=20 > > Hi Oliver, > >=20 > >> On 8 Oct 2025, at 01:26, Oliver Upton wrote: > >>=20 > >> EXTERNAL MAIL > >>=20 > >>=20 > >> On Tue, Oct 07, 2025 at 11:12:31AM +0000, Jan Kotas wrote: > >>> Hello, > >>>=20 > >>> I was finally able to do some validation, sorry for a long delay. > >>=20 > >> No worries, thanks for testing. > >>=20 > >>> First I applied the "Don't advance PC" patch on top of 6.16.9. > >>> It fixed the error message, but the Guest didn=E2=80=99t boot. > >>> I didn=E2=80=99t debug it further. > >>>=20 > >>> Then I applied it on top of 6.17 along with Oliver=E2=80=99s second p= atch. > >>> Guest OS stops booting because of an exception, when accessing ZCR_EL= 2. > >>>=20 > >>> I checked the ESR_EL2 register and it has 0x66000000: > >>>=20 > >>> Access to SVE functionality trapped as a result of CPACR_EL1.ZEN, > >>> CPTR_EL2.ZEN, CPTR_EL2.TZ, or CPTR_EL3.EZ > >>>=20 > >>> I=E2=80=99ll continue the debug to make sure the issue is not on our = end. > >>=20 > >> Could you please share the repro steps? Also, is the guest kernel > >> unmodified? FWIW, I tested kvmarm/next as the kernel at all levels, > >> kvmtool as the VMM and E2H=3DRES1. > >=20 > > I=E2=80=99m running a minimal, unmodified Linux 6.16.0, for my integrat= ion tests. > > It only has a CPU, a GIC, a few UARTs, and uses initramfs. > >=20 > > In my VMM I only set KVM_ARM_VCPU_HAS_EL2, without HAS_EL2_E2H0. > > To start the kernel I set the X0-X3 registers. > > It worked fine, so far, for all other cases than SVE && NV. > >=20 > >> While the trap to L0 is unavoidable, reinjecting the SVE trap depends = on > >> the L0 view of CPTR_EL2 which originates from the in-memory value. > >> Unless there's a bug lurking this should always be in agreement with t= he > >> effective value programmed in CPACR_EL1. > >=20 > > I checked the place, where it=E2=80=99s failing. > > It looks like Guest clears CPTR_EL2 from 0x33ff to 0. > > CPACR_EL1 is 0 at this point. This doesn=E2=80=99t seem to be correct. > >=20 > > 8062192c: mrs x0, cptr_el2 > > 80621930: and x0, x0, #0xfffffffffffffeff > > 80621934: msr cptr_el2, x0 > > 80621938: isb > >=20 > > And accessing ZCR_EL2 is trapped, and causes an exception. > > 8062193c: mov x1, #0xf > > 80621940: msr zcr_el2, x1 > >=20 > > Looks like the Guest OS executes > > .Lcptr_nvhe_\@: // nVHE case > > from el2_setup.h. >=20 > I did some more debugging. > I checked __check_hvhe and it looks like it jumps to fail. > HCR_EL2 has 0x100030080000000. > E2H is set to 0, even though it should be RES1? The value itself doesn't matter for KVM. However, the guest can write anything it wants, and that value will hold *in the register*. > I changed the value using a debugger, so the check passed. > The code took a different branch and it seems the SVE init passed. Oh crap. You are on V2, which doesn't have FGT, so ID_AA64MMFR4_EL1 doesn't trap, and the kernel end-up assuming that the CPU is nVHE capable. Nothing works after that. I'm afraid there is no real fix for this, only hacks... M. --=20 Without deviation from the norm, progress is not possible.