Re: KVM NV + SVE host OS warning

[Marc Zyngier <maz@kernel.org>]

On Wed, 08 Oct 2025 10:45:51 +0100,
Jan Kotas <jank@cadence.com> wrote:
> 
> Hi Marc,
> 
> Thanks for looking into this.
> 
> 
> > On 8 Oct 2025, at 11:28, Marc Zyngier <maz@kernel.org> wrote:
> > 
> > EXTERNAL MAIL
> > 
> > 
> > On Wed, 08 Oct 2025 08:29:38 +0100,
> > Jan Kotas <jank@cadence.com> wrote:
> >> 
> >> 
> >> 
> >>> 
> >>> On 8 Oct 2025, at 08:32, Jan Kotas <jank@cadence.com> wrote:
> >>> 
> >>> Hi Oliver,
> >>> 
> >>>> On 8 Oct 2025, at 01:26, Oliver Upton <oliver.upton@linux.dev> wrote:
> >>>> 
> >>>> EXTERNAL MAIL
> >>>> 
> >>>> 
> >>>> On Tue, Oct 07, 2025 at 11:12:31AM +0000, Jan Kotas wrote:
> >>>>> Hello,
> >>>>> 
> >>>>> I was finally able to do some validation, sorry for a long delay.
> >>>> 
> >>>> No worries, thanks for testing.
> >>>> 
> >>>>> First I applied the "Don't advance PC" patch on top of 6.16.9.
> >>>>> It fixed the error message, but the Guest didn’t boot.
> >>>>> I didn’t debug it further.
> >>>>> 
> >>>>> Then I applied it on top of 6.17 along with Oliver’s second patch.
> >>>>> Guest OS stops booting because of an exception, when accessing ZCR_EL2.
> >>>>> 
> >>>>> I checked the ESR_EL2 register and it has 0x66000000:
> >>>>> 
> >>>>> Access to SVE functionality trapped as a result of CPACR_EL1.ZEN,
> >>>>> CPTR_EL2.ZEN, CPTR_EL2.TZ, or CPTR_EL3.EZ
> >>>>> 
> >>>>> I’ll continue the debug to make sure the issue is not on our end.
> >>>> 
> >>>> Could you please share the repro steps? Also, is the guest kernel
> >>>> unmodified? FWIW, I tested kvmarm/next as the kernel at all levels,
> >>>> kvmtool as the VMM and E2H=RES1.
> >>> 
> >>> I’m running a minimal, unmodified Linux 6.16.0, for my integration tests.
> >>> It only has a CPU, a GIC, a few UARTs, and uses initramfs.
> >>> 
> >>> In my VMM I only set KVM_ARM_VCPU_HAS_EL2, without HAS_EL2_E2H0.
> >>> To start the kernel I set the X0-X3 registers.
> >>> It worked fine, so far, for all other cases than SVE && NV.
> >>> 
> >>>> While the trap to L0 is unavoidable, reinjecting the SVE trap depends on
> >>>> the L0 view of CPTR_EL2 which originates from the in-memory value.
> >>>> Unless there's a bug lurking this should always be in agreement with the
> >>>> effective value programmed in CPACR_EL1.
> >>> 
> >>> I checked the place, where it’s failing.
> >>> It looks like Guest clears CPTR_EL2 from 0x33ff to 0.
> >>> CPACR_EL1 is 0 at this point. This doesn’t seem to be correct.
> >>> 
> >>> 8062192c:   mrs     x0, cptr_el2
> >>> 80621930:   and     x0, x0, #0xfffffffffffffeff
> >>> 80621934:   msr     cptr_el2, x0
> >>> 80621938:   isb
> >>> 
> >>> And accessing ZCR_EL2 is trapped, and causes an exception.
> >>> 8062193c:   mov     x1, #0xf
> >>> 80621940:   msr     zcr_el2, x1
> >>> 
> >>> Looks like the Guest OS executes
> >>> .Lcptr_nvhe_\@: // nVHE case
> >>> from el2_setup.h.
> >> 
> >> I did some more debugging.
> >> I checked __check_hvhe and it looks like it jumps to fail.
> >> HCR_EL2 has 0x100030080000000.
> >> E2H is set to 0, even though it should be RES1?
> > 
> > The value itself doesn't matter for KVM. However, the guest can write
> > anything it wants, and that value will hold *in the register*.
> > 
> >> I changed the value using a debugger, so the check passed.
> >> The code took a different branch and it seems the SVE init passed.
> > 
> > Oh crap. You are on V2, which doesn't have FGT, so ID_AA64MMFR4_EL1
> > doesn't trap, and the kernel end-up assuming that the CPU is nVHE
> > capable. Nothing works after that.
> > 
> > I'm afraid there is no real fix for this, only hacks...
> 
> Is it possible to trap/emulate HCR_EL2, and make sure E2H is always 1,
> in cases when only KVM_ARM_VCPU_HAS_EL2 is set?
> Would that help?

There are no traps for HCR_EL2.

> As a test, I tried forcing this bit before running VCPUs, but
> reading it via MRS got a value without it being set.

Where did you read that from?

If you apply the following change to your guest, does it start
behaving?

	M.

diff --git a/arch/arm64/include/asm/el2_setup.h b/arch/arm64/include/asm/el2_setup.h
index 46033027510cc..392c9f4016f2e 100644
--- a/arch/arm64/include/asm/el2_setup.h
+++ b/arch/arm64/include/asm/el2_setup.h
@@ -34,7 +34,7 @@
 	mrs_s	x1, SYS_ID_AA64MMFR4_EL1
 	sbfx	x1, x1, #ID_AA64MMFR4_EL1_E2H0_SHIFT, #ID_AA64MMFR4_EL1_E2H0_WIDTH
 	cmp	x1, #0
-	b.ge	.LnVHE_\@
+//	b.ge	.LnVHE_\@
 
 	orr	x0, x0, #HCR_E2H
 .LnVHE_\@:

-- 
Without deviation from the norm, progress is not possible.