Re: KVM NV + SVE host OS warning

[Marc Zyngier <maz@kernel.org>]

On Wed, 08 Oct 2025 08:29:38 +0100,
Jan Kotas <jank@cadence.com> wrote:
> 
> 
> 
> > 
> > On 8 Oct 2025, at 08:32, Jan Kotas <jank@cadence.com> wrote:
> > 
> > Hi Oliver,
> > 
> >> On 8 Oct 2025, at 01:26, Oliver Upton <oliver.upton@linux.dev> wrote:
> >> 
> >> EXTERNAL MAIL
> >> 
> >> 
> >> On Tue, Oct 07, 2025 at 11:12:31AM +0000, Jan Kotas wrote:
> >>> Hello,
> >>> 
> >>> I was finally able to do some validation, sorry for a long delay.
> >> 
> >> No worries, thanks for testing.
> >> 
> >>> First I applied the "Don't advance PC" patch on top of 6.16.9.
> >>> It fixed the error message, but the Guest didn’t boot.
> >>> I didn’t debug it further.
> >>> 
> >>> Then I applied it on top of 6.17 along with Oliver’s second patch.
> >>> Guest OS stops booting because of an exception, when accessing ZCR_EL2.
> >>> 
> >>> I checked the ESR_EL2 register and it has 0x66000000:
> >>> 
> >>> Access to SVE functionality trapped as a result of CPACR_EL1.ZEN,
> >>> CPTR_EL2.ZEN, CPTR_EL2.TZ, or CPTR_EL3.EZ
> >>> 
> >>> I’ll continue the debug to make sure the issue is not on our end.
> >> 
> >> Could you please share the repro steps? Also, is the guest kernel
> >> unmodified? FWIW, I tested kvmarm/next as the kernel at all levels,
> >> kvmtool as the VMM and E2H=RES1.
> > 
> > I’m running a minimal, unmodified Linux 6.16.0, for my integration tests.
> > It only has a CPU, a GIC, a few UARTs, and uses initramfs.
> > 
> > In my VMM I only set KVM_ARM_VCPU_HAS_EL2, without HAS_EL2_E2H0.
> > To start the kernel I set the X0-X3 registers.
> > It worked fine, so far, for all other cases than SVE && NV.
> > 
> >> While the trap to L0 is unavoidable, reinjecting the SVE trap depends on
> >> the L0 view of CPTR_EL2 which originates from the in-memory value.
> >> Unless there's a bug lurking this should always be in agreement with the
> >> effective value programmed in CPACR_EL1.
> > 
> > I checked the place, where it’s failing.
> > It looks like Guest clears CPTR_EL2 from 0x33ff to 0.
> > CPACR_EL1 is 0 at this point. This doesn’t seem to be correct.
> > 
> > 8062192c:   mrs     x0, cptr_el2
> > 80621930:   and     x0, x0, #0xfffffffffffffeff
> > 80621934:   msr     cptr_el2, x0
> > 80621938:   isb
> > 
> > And accessing ZCR_EL2 is trapped, and causes an exception.
> > 8062193c:   mov     x1, #0xf
> > 80621940:   msr     zcr_el2, x1
> > 
> > Looks like the Guest OS executes
> > .Lcptr_nvhe_\@: // nVHE case
> > from el2_setup.h.
> 
> I did some more debugging.
> I checked __check_hvhe and it looks like it jumps to fail.
> HCR_EL2 has 0x100030080000000.
> E2H is set to 0, even though it should be RES1?

The value itself doesn't matter for KVM. However, the guest can write
anything it wants, and that value will hold *in the register*.

> I changed the value using a debugger, so the check passed.
> The code took a different branch and it seems the SVE init passed.

Oh crap. You are on V2, which doesn't have FGT, so ID_AA64MMFR4_EL1
doesn't trap, and the kernel end-up assuming that the CPU is nVHE
capable. Nothing works after that.

I'm afraid there is no real fix for this, only hacks...

	M.

-- 
Without deviation from the norm, progress is not possible.