From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 67B941F92E for ; Wed, 8 Oct 2025 11:58:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759924736; cv=none; b=S0mDNPgjIiKGAuH6rAX44pE66erKFmiZ49DIYZ0t2xGmnG2ZsGjR5x9JK5k666V+6aw6/nnMfnlZKox3XlSC7r42tU4tOmv8j5Vi6vNRf25DcQ/vQAWPwqR+ZKK5muNK7iO3UyyzcLPTsEri1cGuNOTOZJFzkx0toLvlayQihPE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759924736; c=relaxed/simple; bh=jYHJTWOjN36HBHpKSmLEi103ZytK5+Ab5/2OP7gKmPM=; h=Date:Message-ID:From:To:Cc:Subject:In-Reply-To:References: MIME-Version:Content-Type; b=Vkv4fvFHnjfp5ZQOcUe+1jlIsqkiXNEQjjWojCbc7t2AuI/NtJ/z7Lk4vev8nFOOBmPaIdVqtYrZtq8pNEoPMYqZA5PWoYIQZxoEDMIx5Tyodg1Y51uOGzdFlLgwP/DcZ5wdcpZM1B24VFeMne3HiNChvtvtgYzO18xjRFomV0M= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=bnld+RtI; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="bnld+RtI" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C6C33C4CEF4; Wed, 8 Oct 2025 11:58:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1759924735; bh=jYHJTWOjN36HBHpKSmLEi103ZytK5+Ab5/2OP7gKmPM=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=bnld+RtIWBvzt5IiG5dqR3SYixjW9lu+r7cwQiXws5ow24hSbEwZyEWb4Cf1PV9Iy lYu0XENMx1S2gJ8oCGIZB/ubOYLj8IfvZg1OeFpJAqECQsgMjKfYFuEucj+4xgVclP FVZuXcxV4jIjt+HjgcMQzQfrNrDojh7riCxghGZB1ZzxGLMQYeQTNCA+32d/w4ZO6z cQRXF1VtH19PUj7UVruDwl9na++3cvPrWgULGdRTs8DeBWo5pL0KFhyCZ5g1pYrO8C q9u4uEmpGJDuWB4lEv9jGv+uvCwtdqoeyT8UeoLidGH8ATckvasdZcJKkNFnDPxplQ VNogsr8aOhFMg== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1v6Son-0000000CL6n-30ag; Wed, 08 Oct 2025 11:58:53 +0000 Date: Wed, 08 Oct 2025 12:58:53 +0100 Message-ID: <86zfa1y58i.wl-maz@kernel.org> From: Marc Zyngier To: Jan Kotas Cc: Oliver Upton , "kvmarm@lists.linux.dev" Subject: Re: KVM NV + SVE host OS warning In-Reply-To: <677A529C-B3D7-4BD1-BEED-D8414D961BBD@global.cadence.com> References: <799DD5E5-8BC2-47B3-A919-33429D3FB2F1@global.cadence.com> <865xd61tt5.wl-maz@kernel.org> <864isq1r66.wl-maz@kernel.org> <25C5E00D-62BC-4188-8642-21913446B32C@global.cadence.com> <1271032F-41BB-4896-AAED-8660D5459E7D@global.cadence.com> <864is9zqs9.wl-maz@kernel.org> <677A529C-B3D7-4BD1-BEED-D8414D961BBD@global.cadence.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/30.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: jank@cadence.com, oliver.upton@linux.dev, kvmarm@lists.linux.dev X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false On Wed, 08 Oct 2025 10:45:51 +0100, Jan Kotas wrote: >=20 > Hi Marc, >=20 > Thanks for looking into this. >=20 >=20 > > On 8 Oct 2025, at 11:28, Marc Zyngier wrote: > >=20 > > EXTERNAL MAIL > >=20 > >=20 > > On Wed, 08 Oct 2025 08:29:38 +0100, > > Jan Kotas wrote: > >>=20 > >>=20 > >>=20 > >>>=20 > >>> On 8 Oct 2025, at 08:32, Jan Kotas wrote: > >>>=20 > >>> Hi Oliver, > >>>=20 > >>>> On 8 Oct 2025, at 01:26, Oliver Upton wrote: > >>>>=20 > >>>> EXTERNAL MAIL > >>>>=20 > >>>>=20 > >>>> On Tue, Oct 07, 2025 at 11:12:31AM +0000, Jan Kotas wrote: > >>>>> Hello, > >>>>>=20 > >>>>> I was finally able to do some validation, sorry for a long delay. > >>>>=20 > >>>> No worries, thanks for testing. > >>>>=20 > >>>>> First I applied the "Don't advance PC" patch on top of 6.16.9. > >>>>> It fixed the error message, but the Guest didn=E2=80=99t boot. > >>>>> I didn=E2=80=99t debug it further. > >>>>>=20 > >>>>> Then I applied it on top of 6.17 along with Oliver=E2=80=99s second= patch. > >>>>> Guest OS stops booting because of an exception, when accessing ZCR_= EL2. > >>>>>=20 > >>>>> I checked the ESR_EL2 register and it has 0x66000000: > >>>>>=20 > >>>>> Access to SVE functionality trapped as a result of CPACR_EL1.ZEN, > >>>>> CPTR_EL2.ZEN, CPTR_EL2.TZ, or CPTR_EL3.EZ > >>>>>=20 > >>>>> I=E2=80=99ll continue the debug to make sure the issue is not on ou= r end. > >>>>=20 > >>>> Could you please share the repro steps? Also, is the guest kernel > >>>> unmodified? FWIW, I tested kvmarm/next as the kernel at all levels, > >>>> kvmtool as the VMM and E2H=3DRES1. > >>>=20 > >>> I=E2=80=99m running a minimal, unmodified Linux 6.16.0, for my integr= ation tests. > >>> It only has a CPU, a GIC, a few UARTs, and uses initramfs. > >>>=20 > >>> In my VMM I only set KVM_ARM_VCPU_HAS_EL2, without HAS_EL2_E2H0. > >>> To start the kernel I set the X0-X3 registers. > >>> It worked fine, so far, for all other cases than SVE && NV. > >>>=20 > >>>> While the trap to L0 is unavoidable, reinjecting the SVE trap depend= s on > >>>> the L0 view of CPTR_EL2 which originates from the in-memory value. > >>>> Unless there's a bug lurking this should always be in agreement with= the > >>>> effective value programmed in CPACR_EL1. > >>>=20 > >>> I checked the place, where it=E2=80=99s failing. > >>> It looks like Guest clears CPTR_EL2 from 0x33ff to 0. > >>> CPACR_EL1 is 0 at this point. This doesn=E2=80=99t seem to be correct. > >>>=20 > >>> 8062192c: mrs x0, cptr_el2 > >>> 80621930: and x0, x0, #0xfffffffffffffeff > >>> 80621934: msr cptr_el2, x0 > >>> 80621938: isb > >>>=20 > >>> And accessing ZCR_EL2 is trapped, and causes an exception. > >>> 8062193c: mov x1, #0xf > >>> 80621940: msr zcr_el2, x1 > >>>=20 > >>> Looks like the Guest OS executes > >>> .Lcptr_nvhe_\@: // nVHE case > >>> from el2_setup.h. > >>=20 > >> I did some more debugging. > >> I checked __check_hvhe and it looks like it jumps to fail. > >> HCR_EL2 has 0x100030080000000. > >> E2H is set to 0, even though it should be RES1? > >=20 > > The value itself doesn't matter for KVM. However, the guest can write > > anything it wants, and that value will hold *in the register*. > >=20 > >> I changed the value using a debugger, so the check passed. > >> The code took a different branch and it seems the SVE init passed. > >=20 > > Oh crap. You are on V2, which doesn't have FGT, so ID_AA64MMFR4_EL1 > > doesn't trap, and the kernel end-up assuming that the CPU is nVHE > > capable. Nothing works after that. > >=20 > > I'm afraid there is no real fix for this, only hacks... >=20 > Is it possible to trap/emulate HCR_EL2, and make sure E2H is always 1, > in cases when only KVM_ARM_VCPU_HAS_EL2 is set? > Would that help? There are no traps for HCR_EL2. > As a test, I tried forcing this bit before running VCPUs, but > reading it via MRS got a value without it being set. Where did you read that from? If you apply the following change to your guest, does it start behaving? M. diff --git a/arch/arm64/include/asm/el2_setup.h b/arch/arm64/include/asm/el= 2_setup.h index 46033027510cc..392c9f4016f2e 100644 --- a/arch/arm64/include/asm/el2_setup.h +++ b/arch/arm64/include/asm/el2_setup.h @@ -34,7 +34,7 @@ mrs_s x1, SYS_ID_AA64MMFR4_EL1 sbfx x1, x1, #ID_AA64MMFR4_EL1_E2H0_SHIFT, #ID_AA64MMFR4_EL1_E2H0_WIDTH cmp x1, #0 - b.ge .LnVHE_\@ +// b.ge .LnVHE_\@ =20 orr x0, x0, #HCR_E2H .LnVHE_\@: --=20 Without deviation from the norm, progress is not possible.