kvm: nVHE hyp panic at: __kvm_nvhe_kvm_hyp_handle_sysre

kvm: nVHE hyp panic at: __kvm_nvhe_kvm_hyp_handle_sysre

[Naresh Kamboju]

Regression on rk3399-rock-pi-4b while running kvm-unit-tests with
nvhe, protected and vhe mode with virtualization enabled.

First seen on next-20250120
Good: next-20250117
Bad: next-20250120 till today's next-20250210

This is always reproducible.

Regression on these devices with kernel command line boot modes.
* rk3399-rock-pi-4b-nvhe
* rk3399-rock-pi-4b-protected
* rk3399-rock-pi-4b-vhe

Test regression: kvm nVHE hyp panic at __kvm_nvhe_kvm_hyp_handle_sysreg
Reported-by: Linux Kernel Functional Testing <lkft@linaro.org>

## Test log
ok 139 - psci: psci: cpu-off
[  674.423092] kvm [6196]: nVHE hyp panic at:
__kvm_nvhe_kvm_hyp_handle_sysreg (include/kvm/arm_arch_timer.h:174
arch/arm64/kvm/hyp/include/hyp/switch.h:506
arch/arm64/kvm/hyp/include/hyp/switch.h:559
arch/arm64/kvm/hyp/include/hyp/switch.h:604)
[  674.423987] kvm [6196]: nVHE call trace:
[  674.424331] kvm [6196]: __kvm_nvhe_hyp_panic
(arch/arm64/kvm/hyp/nvhe/switch.c:415)
[  674.424950] kvm [6196]: __kvm_nvhe___kvm_vcpu_run
(arch/arm64/kvm/hyp/include/hyp/switch.h:750 (discriminator 1)
arch/arm64/kvm/hyp/nvhe/switch.c:355 (discriminator 1))
[  674.425619] kvm [6196]: __kvm_nvhe___kvm_vcpu_run
(arch/arm64/kvm/hyp/include/hyp/switch.h:750 (discriminator 1)
arch/arm64/kvm/hyp/nvhe/switch.c:355 (discriminator 1))
[  674.426288] kvm [6196]: __kvm_nvhe_handle___kvm_vcpu_run
(arch/arm64/kvm/hyp/nvhe/hyp-main.c:231)
[  674.427004] kvm [6196]: __kvm_nvhe_handle_trap
(arch/arm64/kvm/hyp/nvhe/hyp-main.c:640
arch/arm64/kvm/hyp/nvhe/hyp-main.c:673)
[  674.427651] kvm [6196]: __kvm_nvhe___skip_pauth_save
(arch/arm64/kvm/hyp/nvhe/host.S:67)
[  674.428313] kvm [6196]: ---[ end nVHE call trace ]---
[  674.428755] kvm [6196]: Hyp Offset: 0xfffea0df7d800000
[  674.429208] Kernel panic - not syncing: HYP panic:
[  674.429208] PS:600003c9 PC:0000df2103dc1610 ESR:0000000096000004
[  674.429208] FAR:ffff00000d031ba0 HPFAR:0000000000000000 PAR:1d00007edbadc0de
[  674.429208] VCPU:0000df21077a1bc0
[  674.431040] CPU: 4 UID: 0 PID: 6196 Comm: qemu-system-aar Not
tainted 6.14.0-rc2-next-20250210 #1
[  674.431815] Hardware name: Radxa ROCK Pi 4B (DT)
[  674.432219] Call trace:
[  674.432437] show_stack (arch/arm64/kernel/stacktrace.c:468) (C)
[  674.432763] dump_stack_lvl (lib/dump_stack.c:124)
[  674.433090] dump_stack (lib/dump_stack.c:130)
[  674.433384] panic (kernel/panic.c:354)
[  674.433657] nvhe_hyp_panic_handler
(arch/arm64/include/asm/current.h:19 arch/arm64/kvm/handle_exit.c:452)
[  674.434056] kvm_arm_vcpu_enter_exit
(arch/arm64/include/asm/alternative-macros.h:232
arch/arm64/include/asm/cpufeature.h:453
arch/arm64/include/asm/cpufeature.h:498
arch/arm64/include/asm/virt.h:139 arch/arm64/kvm/arm.c:1084)
[  674.434448] kvm_arch_vcpu_ioctl_run (arch/arm64/kvm/arm.c:1191)
[  674.434855] kvm_vcpu_ioctl (arch/arm64/kvm/../../../virt/kvm/kvm_main.c:4377)
[  674.435194] __arm64_sys_ioctl (fs/ioctl.c:51 (discriminator 1)
fs/ioctl.c:906 (discriminator 1) fs/ioctl.c:892 (discriminator 1)
fs/ioctl.c:892 (discriminator 1))
[  674.435542] invoke_syscall (arch/arm64/include/asm/current.h:19
arch/arm64/kernel/syscall.c:54)
[  674.435875] el0_svc_common.constprop.0
(include/linux/thread_info.h:135 (discriminator 2)
arch/arm64/kernel/syscall.c:140 (discriminator 2))
[  674.436289] do_el0_svc (arch/arm64/kernel/syscall.c:152)
[  674.436583] el0_svc (arch/arm64/include/asm/irqflags.h:82
(discriminator 1) arch/arm64/include/asm/irqflags.h:123 (discriminator
1) arch/arm64/include/asm/irqflags.h:136 (discriminator 1)
arch/arm64/kernel/entry-common.c:165 (discriminator 1)
arch/arm64/kernel/entry-common.c:178 (discriminator 1)
arch/arm64/kernel/entry-common.c:745 (discriminator 1))
[  674.436860] el0t_64_sync_handler (arch/arm64/kernel/entry-common.c:763)
[  674.437243] el0t_64_sync (arch/arm64/kernel/entry.S:600)
[  674.437567] SMP: stopping secondary CPUs
[  674.438032] Kernel Offset: disabled
[  674.438338] CPU features: 0x100,0002082c,00800000,8200421b
[  674.438819] Memory Limit: none
[  674.439090] ---[ end Kernel panic - not syncing: HYP panic:
[  674.439090] PS:600003c9 PC:0000df2103dc1610 ESR:0000000096000004
[  674.439090] FAR:ffff00000d031ba0 HPFAR:0000000000000000 PAR:1d00007edbadc0de
[  674.439090] VCPU:0000df21077a1bc0 ]---

## Source
* kernel version: 6.14.0-rc2-next-20250210
* git tree: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
* git sha: df5d6180169ae06a2eac57e33b077ad6f6252440
* git describe: next-20250210
* project details:
https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20250210/
* architecture: arm64
* device: rk3399-rock-pi-4b
* toolchain: gcc-13
* build config:
https://storage.tuxsuite.com/public/linaro/lkft/builds/2spp0brZaSpWe7z5aIqZfQx4GhT/config
* build: https://storage.tuxsuite.com/public/linaro/lkft/builds/2spp0brZaSpWe7z5aIqZfQx4GhT/

## Boot log
* test link-1: https://lkft.validation.linaro.org/scheduler/job/8117232#L1395
* test link-2: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20250210/testrun/27250834/suite/log-parser-test/test/panic-multiline-kernel-panic-not-syncing-hyp-panic/log
* build link: https://storage.tuxsuite.com/public/linaro/lkft/builds/2spp0brZaSpWe7z5aIqZfQx4GhT/
* config link: https://storage.tuxsuite.com/public/linaro/lkft/builds/2spp0brZaSpWe7z5aIqZfQx4GhT/config
* linux-next-history:
https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20250207/testrun/27226044/suite/log-parser-test/test/panic-multiline-kernel-panic-not-syncing-hyp-panic/history/
* linux-mainline-history:
https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.14-rc2/testrun/27248845/suite/log-parser-test/test/panic-multiline-kernel-panic-not-syncing-hyp-panic/history/

--
Linaro LKFT
https://lkft.linaro.org

Re: kvm: nVHE hyp panic at: __kvm_nvhe_kvm_hyp_handle_sysre

[Marc Zyngier]

On Tue, 11 Feb 2025 11:24:06 +0000,
Naresh Kamboju <naresh.kamboju@linaro.org> wrote:
> 
> Regression on rk3399-rock-pi-4b while running kvm-unit-tests with
> nvhe, protected and vhe mode with virtualization enabled.

I do not buy this. RK3399 only has ARMv8.0 cores, which by definition
do not have VHE.

> 
> First seen on next-20250120
> Good: next-20250117
> Bad: next-20250120 till today's next-20250210
> 
> This is always reproducible.

What about vanilla upstream?

> 
> Regression on these devices with kernel command line boot modes.
> * rk3399-rock-pi-4b-nvhe
> * rk3399-rock-pi-4b-protected
> * rk3399-rock-pi-4b-vhe

Please show me this device running in VHE mode.

For the crash at hand, which clearly shows nVHE, can you report
whether the following hack fixes it for you?

	M.

diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
index f838a45665f26..cb57420a07de2 100644
--- a/arch/arm64/kvm/hyp/include/hyp/switch.h
+++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
@@ -556,7 +556,7 @@ static bool kvm_handle_cntxct(struct kvm_vcpu *vcpu)
 		return false;
 	}
 
-	val = compute_counter_value(ctxt);
+	val = compute_counter_value(kern_hyp_va(ctxt));
 
 	vcpu_set_reg(vcpu, kvm_vcpu_sys_get_rt(vcpu), val);
 	__kvm_skip_instr(vcpu);

-- 
Without deviation from the norm, progress is not possible.

Re: kvm: nVHE hyp panic at: __kvm_nvhe_kvm_hyp_handle_sysre

[Dan Carpenter]

On Tue, Feb 11, 2025 at 11:36:31AM +0000, Marc Zyngier wrote:
> On Tue, 11 Feb 2025 11:24:06 +0000,
> Naresh Kamboju <naresh.kamboju@linaro.org> wrote:
> > 
> > Regression on rk3399-rock-pi-4b while running kvm-unit-tests with
> > nvhe, protected and vhe mode with virtualization enabled.
> 
> I do not buy this. RK3399 only has ARMv8.0 cores, which by definition
> do not have VHE.
> 

The links to the dmesg are at the bottom of the email.  It does seem to
be a RK3399 with ARMv8.0 cores though as you say...

https://lkft.validation.linaro.org/scheduler/job/8117232#L1395

We'll test the patch.

regards,
dan carpenter

Re: kvm: nVHE hyp panic at: __kvm_nvhe_kvm_hyp_handle_sysre

[Dan Carpenter]

On Tue, Feb 11, 2025 at 11:36:31AM +0000, Marc Zyngier wrote:
> For the crash at hand, which clearly shows nVHE, can you report
> whether the following hack fixes it for you?
> 
> 	M.

No luck, I'm afraid.  It still crashes the same way.

https://tuxapi.tuxsuite.com/v1/groups/linaro/projects/anders/tests/2steTT4f2J8ZJjc3jNJFMuow9Cb

Click on "Logs: html" for the dmesg.

regards,
dan carpenter

> 
> diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
> index f838a45665f26..cb57420a07de2 100644
> --- a/arch/arm64/kvm/hyp/include/hyp/switch.h
> +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
> @@ -556,7 +556,7 @@ static bool kvm_handle_cntxct(struct kvm_vcpu *vcpu)
>  		return false;
>  	}
>  
> -	val = compute_counter_value(ctxt);
> +	val = compute_counter_value(kern_hyp_va(ctxt));
>  
>  	vcpu_set_reg(vcpu, kvm_vcpu_sys_get_rt(vcpu), val);
>  	__kvm_skip_instr(vcpu);
> 
> -- 
> Without deviation from the norm, progress is not possible.

Re: kvm: nVHE hyp panic at: __kvm_nvhe_kvm_hyp_handle_sysre

[Naresh Kamboju]

On Wed, 12 Feb 2025 at 17:11, Dan Carpenter <dan.carpenter@linaro.org> wrote:
>
> On Tue, Feb 11, 2025 at 11:36:31AM +0000, Marc Zyngier wrote:
> > For the crash at hand, which clearly shows nVHE, can you report
> > whether the following hack fixes it for you?
> >
> >       M.
>
> No luck, I'm afraid.  It still crashes the same way.
>
> https://tuxapi.tuxsuite.com/v1/groups/linaro/projects/anders/tests/2steTT4f2J8ZJjc3jNJFMuow9Cb
>
> Click on "Logs: html" for the dmesg.

Anders tested this patch and the reported issue did not fix,

Link: https://lkft.validation.linaro.org/scheduler/job/8118315#L1379

- Naresh

Re: kvm: nVHE hyp panic at: __kvm_nvhe_kvm_hyp_handle_sysre

[Marc Zyngier]

On Wed, 12 Feb 2025 11:41:38 +0000,
Dan Carpenter <dan.carpenter@linaro.org> wrote:
> 
> On Tue, Feb 11, 2025 at 11:36:31AM +0000, Marc Zyngier wrote:
> > For the crash at hand, which clearly shows nVHE, can you report
> > whether the following hack fixes it for you?
> > 
> > 	M.
> 
> No luck, I'm afraid.  It still crashes the same way.

Right. It was one level deeper. The following change fixes it for
me. YMMV.

	M.

diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
index f838a45665f26..b899d815d272f 100644
--- a/arch/arm64/kvm/hyp/include/hyp/switch.h
+++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
@@ -501,9 +501,22 @@ static inline bool handle_tx2_tvm(struct kvm_vcpu *vcpu)
 	return true;
 }
 
+/* Open-coded version of timer_get_offset() to allow for kern_hyp_va() */
+static inline u64 hyp_timer_get_offset(struct arch_timer_context *ctxt)
+{
+	u64 offset = 0;
+
+	if (ctxt->offset.vm_offset)
+		offset += *kern_hyp_va(ctxt->offset.vm_offset);
+	if (ctxt->offset.vcpu_offset)
+		offset += *kern_hyp_va(ctxt->offset.vcpu_offset);
+
+	return offset;
+}
+
 static inline u64 compute_counter_value(struct arch_timer_context *ctxt)
 {
-	return arch_timer_read_cntpct_el0() - timer_get_offset(ctxt);
+	return arch_timer_read_cntpct_el0() - hyp_timer_get_offset(ctxt);
 }
 
 static bool kvm_handle_cntxct(struct kvm_vcpu *vcpu)

-- 
Without deviation from the norm, progress is not possible.

Re: kvm: nVHE hyp panic at: __kvm_nvhe_kvm_hyp_handle_sysre

[Naresh Kamboju]

On Wed, 12 Feb 2025 at 19:30, Marc Zyngier <maz@kernel.org> wrote:
>
> On Wed, 12 Feb 2025 11:41:38 +0000,
> Dan Carpenter <dan.carpenter@linaro.org> wrote:
> >
> > On Tue, Feb 11, 2025 at 11:36:31AM +0000, Marc Zyngier wrote:
> > > For the crash at hand, which clearly shows nVHE, can you report
> > > whether the following hack fixes it for you?
> > >
> > >     M.
> >
> > No luck, I'm afraid.  It still crashes the same way.
>
> Right. It was one level deeper. The following change fixes it for
> me. YMMV.

Anders applied this patch and tested on rk3399-rock-pi-4b,
and confirmed that the regression is resolved.

Tested-by: Linux Kernel Functional Testing <lkft@linaro.org>

Link:
 - https://lkft.validation.linaro.org/scheduler/job/8119717#L1251

>
>         M.
>
> diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
> index f838a45665f26..b899d815d272f 100644
> --- a/arch/arm64/kvm/hyp/include/hyp/switch.h
> +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
> @@ -501,9 +501,22 @@ static inline bool handle_tx2_tvm(struct kvm_vcpu *vcpu)
>         return true;
>  }
>
> +/* Open-coded version of timer_get_offset() to allow for kern_hyp_va() */
> +static inline u64 hyp_timer_get_offset(struct arch_timer_context *ctxt)
> +{
> +       u64 offset = 0;
> +
> +       if (ctxt->offset.vm_offset)
> +               offset += *kern_hyp_va(ctxt->offset.vm_offset);
> +       if (ctxt->offset.vcpu_offset)
> +               offset += *kern_hyp_va(ctxt->offset.vcpu_offset);
> +
> +       return offset;
> +}
> +
>  static inline u64 compute_counter_value(struct arch_timer_context *ctxt)
>  {
> -       return arch_timer_read_cntpct_el0() - timer_get_offset(ctxt);
> +       return arch_timer_read_cntpct_el0() - hyp_timer_get_offset(ctxt);
>  }
>
>  static bool kvm_handle_cntxct(struct kvm_vcpu *vcpu)
>
> --
> Without deviation from the norm, progress is not possible.

--
Linaro LKFT
https://lkft.linaro.org