Re: [kvm-unit-tests PATCH v3 00/10] arm64: EL2 support

[Joey Gouly <joey.gouly@arm.com>]

On Thu, Nov 27, 2025 at 01:04:08PM +0100, Eric Auger wrote:
> Hi Joey,
> 
> On 11/27/25 12:08 PM, Joey Gouly wrote:
> > On Thu, Nov 27, 2025 at 11:04:43AM +0100, Eric Auger wrote:
> >> Hi Joey,
> >>
> >> On 9/25/25 4:19 PM, Joey Gouly wrote:
> >>> Hi all,
> >>>
> >>> This series is for adding support to running the kvm-unit-tests at EL2. These
> >>> have been tested with Linux 6.17-rc6 KVM nested virt.
> >>>
> >>> This latest round I also tested using the run_tests.sh script with QEMU TCG,
> >>> running at EL2.
> >>>
> >>> The goal is to later extend and add new tests for Nested Virtualisation,
> >>> however they should also work with bare metal as well.
> >>>
> >>> Changes since v2[1]:
> >>> 	- Move the sctlr setup in EFI to a function.
> >>> 	- Decided to not re-use el2_setup.h from Linux, looked more
> >>> 	  complicated to use than needed for KUT.
> >>> 	- Add EL2 env variable for testing, open to feedback for that.
> >>> 	  This was untested with kvmtool as my testing setup only has
> >>> 	  busybox ash currently, and the run_tests.sh script needs bash.
> >>>
> >>> Issues (that I think are fine to investigate/fix later):
> >>> 	- Some of the debug tests fail with QEMU at EL2 and kvmtool.
> >>> 	- The gic ipi test times out with QEMU at EL2, but works with kvmtool.
> >> Have you noticed any failure with migration tests. On my end, as soon as
> >> I set EL2=1 migration tests do fail.
> > Yes migration also fails here, forgot to mention that.
> >
> > Seems like migration completes, but then something bad happens on the first
> > interrupt. I will investigate a bit now and see if it's an easy fix.
> 
> while you do that I will go through the series. My apologies for the delay

Thanks!

I found the issue, and I thought it was one I had solved before, but maybe I lost the fix somehow:

diff --git a/arm/cstart64.S b/arm/cstart64.S
index af7c81c1..fd8b6690 100644
--- a/arm/cstart64.S
+++ b/arm/cstart64.S
@@ -235,6 +235,11 @@ get_mmu_off:
 .globl secondary_entry
 secondary_entry:
        init_el x0
+
+       /* set SCTLR_EL1 to a known value */
+       ldr     x4, =INIT_SCTLR_EL1_MMU_OFF
+       msr     sctlr_el1, x4
+       isb
+
        /* enable FP/ASIMD and SVE */
        mov     x0, #(3 << 20)
        orr     x0, x0, #(3 << 16)

The secondary core code doesn't fully initialise SCTLR_ELx, which means that
SCTLR_ELx.SPAN=0. That's why there was a data abort after the IRQ.

This doesn't affect EL1 (only) because KVM has a different reset value for
SCTLR_EL1 and SCTLR_EL2.

I will send out a new version with this fix, missing sob/reviews/acks, after you've had a look!

Thanks,
Joey

> 
> Eric
> >
> > Thanks,
> > Joey
> >
> >> Eric
> >>> Thanks,
> >>> Joey
> >>>
> >>> [1] https://lore.kernel.org/kvmarm/20250529135557.2439500-1-joey.gouly@arm.com/
> >>>
> >>> Alexandru Elisei (2):
> >>>   arm64: micro-bench: use smc when at EL2
> >>>   arm64: selftest: update test for running at EL2
> >>>
> >>> Joey Gouly (8):
> >>>   arm64: drop to EL1 if booted at EL2
> >>>   arm64: efi: initialise SCTLR_ELx fully
> >>>   arm64: efi: initialise the EL
> >>>   arm64: timer: use hypervisor timers when at EL2
> >>>   arm64: micro-bench: fix timer IRQ
> >>>   arm64: pmu: count EL2 cycles
> >>>   arm64: run at EL2 if supported
> >>>   arm64: add EL2 environment variable
> >>>
> >>>  arm/cstart64.S             | 56 ++++++++++++++++++++++++++++++++++++--
> >>>  arm/efi/crt0-efi-aarch64.S |  5 ++++
> >>>  arm/micro-bench.c          | 26 ++++++++++++++++--
> >>>  arm/pmu.c                  | 13 ++++++---
> >>>  arm/run                    |  7 +++++
> >>>  arm/selftest.c             | 18 ++++++++----
> >>>  arm/timer.c                | 10 +++++--
> >>>  lib/acpi.h                 |  2 ++
> >>>  lib/arm/asm/setup.h        |  8 ++++++
> >>>  lib/arm/asm/timer.h        | 11 ++++++++
> >>>  lib/arm/setup.c            |  4 +++
> >>>  lib/arm/timer.c            | 19 +++++++++++--
> >>>  lib/arm64/asm/sysreg.h     | 19 +++++++++++++
> >>>  lib/arm64/processor.c      | 12 ++++++++
> >>>  14 files changed, 191 insertions(+), 19 deletions(-)
> >>>
>