Re: KVM NV + SVE host OS warning

Re: KVM NV + SVE host OS warning

[Marc Zyngier]

[+Oliver for the SVE stuff]

Hi Jan,

On Thu, 25 Sep 2025 15:02:20 +0100,
Jan Kotas <jank@cadence.com> wrote:
> Hello,
> 
> I’m experimenting with Nested Virtualization.
> I use Linux kernel 6.16.3 from Debian backports running on Neoverse-V2.
> 
> When I try to boot a GuestOS, it hangs,
> and I can see a warning in Host's dmesg:
> 
> [52417.934951] ------------[ cut here ]------------
> [52417.934990] WARNING: CPU: 120 PID: 44115 at arch/arm64/include/asm/kvm_emulate.h:553 perform_access+0x14c/0x160
> [52417.935087] Modules linked in: nfsv3 nfs netfs snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore rfkill qrtr binfmt_misc nls_ascii nls_cp437 vfat fat aes_ce_blk aes_ce_cipher polyval_ce ghash_ce gf128mul sha3_ce sha512_ce sha1_ce acpi_ipmi dax_hmem arm_smccc_trng cxl_acpi ipmi_ssif i2c_smbus arm_spe_pmu arm_smmuv3_pmu coresight_trbe spi_nor mtd ipmi_devintf ipmi_msghandler coresight_stm coresight_tmc coresight_funnel stm_core coresight_etm4x coresight joydev evdev cppc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc efi_pstore configfs nfnetlink efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_cryptoapi hid_generic usbhid hid rndis_host cdc_ether usbnet mii dm_mod ast ixgbe drm_shmem_helper xhci_pci_renesas i2c_algo_bit xfrm_algo xhci_pci drm_client_lib mdio_devres drm_kms_helper xhci_hcd of_mdio nvme fixed_phy drm fwnode_mdio usbcore nvme_core libphy sbsa_gwdt mdio_bus nvme_keyring usb_common nvme_auth mdio i2c_tegra
> [52417.935818] CPU: 120 UID: 254353 PID: 44115 Comm: kvm_vcpu0 Tainted: G        W           6.16.3+deb13-arm64 #1 PREEMPTLAZY  Debian 6.16.3-1~bpo13+1
> [52417.935855] Tainted: [W]=WARN
> [52417.935866] Hardware name: Supermicro ARS-121L-DNR/G1SMH, BIOS 2.1 04/17/2025

Fancy HW (/me goes selling a kidney...)

> [52417.935879] pstate: 62400009 (nZCv daif +PAN -UAO +TCO -DIT -SSBS BTYPE=--)
> [52417.935906] pc : perform_access+0x14c/0x160
> [52417.935933] lr : perform_access+0x4c/0x160
> [52417.935956] sp : ffff8000f30db850
> [52417.935967] x29: ffff8000f30db850 x28: ffff000097245000 x27: 0000000000000000
> [52417.936004] x26: 0000000000000000 x25: 0000000000000000 x24: ffff10002c701c28
> [52417.936036] x23: 0000000000000000 x22: ffff000097245000 x21: ffff8000f30db8a0
> [52417.936065] x20: ffffdbf14a19eac0 x19: ffff10002c701be0 x18: 0000000000000014
> [52417.936095] x17: 000000040044ffff x16: 00100075b5503510 x15: 0000000000000000
> [52417.936127] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
> [52417.936157] x11: 0000000000001348 x10: 00000000000013b0 x9 : ffffdbf1491608b4
> [52417.936188] x8 : 0000000000000001 x7 : 0000000000000000 x6 : 00000000000fffff
> [52417.936218] x5 : 000000000036cb76 x4 : ffff10027148f7c0 x3 : ffffdbf14915f04c
> [52417.936249] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000009
> [52417.936280] Call trace:
> [52417.936291]  perform_access+0x14c/0x160 (P)
> [52417.936325]  kvm_handle_sys_reg+0x12c/0x2a0
> [52417.936366]  handle_exit+0x68/0x190
> [52417.936408]  kvm_arch_vcpu_ioctl_run+0x2d8/0xa10
> [52417.936436]  kvm_vcpu_ioctl+0x1a8/0xb18
> [52417.936459]  __arm64_sys_ioctl+0xb4/0x120
> [52417.936510]  invoke_syscall+0x6c/0x100
> [52417.936547]  el0_svc_common.constprop.0+0x48/0xf0
> [52417.936581]  do_el0_svc+0x24/0x38
> [52417.936613]  el0_svc+0xd4/0x190
> [52417.936643]  el0t_64_sync_handler+0x10c/0x138
> [52417.936667]  el0t_64_sync+0x198/0x1a0
> [52417.936690] ---[ end trace 0000000000000000 ]---
> 
> 
> The tracing revealed, it may be caused by a ZCR_EL2 write:
> [109] ..... 52068.375927: kvm_sys_access: PC: 806608b8 SYS_ZCR_EL2 (3,4,1,2,0) write
> 
> The instruction from ELR also matches: msr zcr_el2, x1
> 
> The reason might be CPTR_EL2, its value just before this instruction is executed, is 0.
> However before the start of the VM execution, it has 0x22ff.
> 
> I can see accesses to this register in the trace log as well, just before ZCR_EL2 is accessed.
> [109] ..... 52068.375922: kvm_sys_access: PC: 806608a4 SYS_CPTR_EL2 (3,4,1,1,2) read
> [109] ..... 52068.375925: kvm_sys_access: PC: 806608ac SYS_CPTR_EL2 (3,4,1,1,2) write
> 
> I’m running Linux 6.16.0 as my Guest.
> Nested Virtualization works fine with SVE disabled, so does SVE without NV.
> Could it be caused by a bug in userspace hypervisor code?

Unlikely. The warning indicates that we are incrementing PC while
there is a pending exception. Having both at the same time is a very
bad bug -- hence the warning.

Looking at the code with the above in mind, something immediately
jumps at me. Can you try the following (against 6.17, but you'll
surely be able to apply it against 6.16):

diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 91053aa832d08..a07ad5c92583d 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -2705,7 +2705,7 @@ static bool access_zcr_el2(struct kvm_vcpu *vcpu,
 
 	if (guest_hyp_sve_traps_enabled(vcpu)) {
 		kvm_inject_nested_sve_trap(vcpu);
-		return true;
+		return false;
 	}
 
 	if (!p->is_write) {

This should make the warning go away -- not sure about anything else.
Note that I do not have access to an NV+SVE capable machine, so you're
are basically on your own, unless Oliver has a box he can reproduce
this on.

I would also recommend to update to 6.17 -- 6.16 was the first release
with NV, and while it may work, it will also have a lot of ugly bugs.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: KVM NV + SVE host OS warning

[Jan Kotas]

Hi Marc,

Thank you for a quick response.

> On 25 Sep 2025, at 16:38, Marc Zyngier <maz@kernel.org> wrote:
> 
> EXTERNAL MAIL
> 
> 
> [+Oliver for the SVE stuff]
> 
> Hi Jan,
> 
> On Thu, 25 Sep 2025 15:02:20 +0100,
> Jan Kotas <jank@cadence.com> wrote:
>> Hello,
>> 
>> I’m experimenting with Nested Virtualization.
>> I use Linux kernel 6.16.3 from Debian backports running on Neoverse-V2.
>> 
>> When I try to boot a GuestOS, it hangs,
>> and I can see a warning in Host's dmesg:
>> 
>> [52417.934951] ------------[ cut here ]------------
>> [52417.934990] WARNING: CPU: 120 PID: 44115 at arch/arm64/include/asm/kvm_emulate.h:553 perform_access+0x14c/0x160
>> [52417.935087] Modules linked in: nfsv3 nfs netfs snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore rfkill qrtr binfmt_misc nls_ascii nls_cp437 vfat fat aes_ce_blk aes_ce_cipher polyval_ce ghash_ce gf128mul sha3_ce sha512_ce sha1_ce acpi_ipmi dax_hmem arm_smccc_trng cxl_acpi ipmi_ssif i2c_smbus arm_spe_pmu arm_smmuv3_pmu coresight_trbe spi_nor mtd ipmi_devintf ipmi_msghandler coresight_stm coresight_tmc coresight_funnel stm_core coresight_etm4x coresight joydev evdev cppc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc efi_pstore configfs nfnetlink efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_cryptoapi hid_generic usbhid hid rndis_host cdc_ether usbnet mii dm_mod ast ixgbe drm_shmem_helper xhci_pci_renesas i2c_algo_bit xfrm_algo xhci_pci drm_client_lib mdio_devres drm_kms_helper xhci_hcd of_mdio nvme fixed_phy drm fwnode_mdio usbcore nvme_core libphy sbsa_gwdt mdio_bus nvme_keyring usb_common nvme_auth mdio i2c_tegra
>> [52417.935818] CPU: 120 UID: 254353 PID: 44115 Comm: kvm_vcpu0 Tainted: G        W           6.16.3+deb13-arm64 #1 PREEMPTLAZY  Debian 6.16.3-1~bpo13+1
>> [52417.935855] Tainted: [W]=WARN
>> [52417.935866] Hardware name: Supermicro ARS-121L-DNR/G1SMH, BIOS 2.1 04/17/2025
> 
> Fancy HW (/me goes selling a kidney...)
> 
>> [52417.935879] pstate: 62400009 (nZCv daif +PAN -UAO +TCO -DIT -SSBS BTYPE=--)
>> [52417.935906] pc : perform_access+0x14c/0x160
>> [52417.935933] lr : perform_access+0x4c/0x160
>> [52417.935956] sp : ffff8000f30db850
>> [52417.935967] x29: ffff8000f30db850 x28: ffff000097245000 x27: 0000000000000000
>> [52417.936004] x26: 0000000000000000 x25: 0000000000000000 x24: ffff10002c701c28
>> [52417.936036] x23: 0000000000000000 x22: ffff000097245000 x21: ffff8000f30db8a0
>> [52417.936065] x20: ffffdbf14a19eac0 x19: ffff10002c701be0 x18: 0000000000000014
>> [52417.936095] x17: 000000040044ffff x16: 00100075b5503510 x15: 0000000000000000
>> [52417.936127] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
>> [52417.936157] x11: 0000000000001348 x10: 00000000000013b0 x9 : ffffdbf1491608b4
>> [52417.936188] x8 : 0000000000000001 x7 : 0000000000000000 x6 : 00000000000fffff
>> [52417.936218] x5 : 000000000036cb76 x4 : ffff10027148f7c0 x3 : ffffdbf14915f04c
>> [52417.936249] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000009
>> [52417.936280] Call trace:
>> [52417.936291]  perform_access+0x14c/0x160 (P)
>> [52417.936325]  kvm_handle_sys_reg+0x12c/0x2a0
>> [52417.936366]  handle_exit+0x68/0x190
>> [52417.936408]  kvm_arch_vcpu_ioctl_run+0x2d8/0xa10
>> [52417.936436]  kvm_vcpu_ioctl+0x1a8/0xb18
>> [52417.936459]  __arm64_sys_ioctl+0xb4/0x120
>> [52417.936510]  invoke_syscall+0x6c/0x100
>> [52417.936547]  el0_svc_common.constprop.0+0x48/0xf0
>> [52417.936581]  do_el0_svc+0x24/0x38
>> [52417.936613]  el0_svc+0xd4/0x190
>> [52417.936643]  el0t_64_sync_handler+0x10c/0x138
>> [52417.936667]  el0t_64_sync+0x198/0x1a0
>> [52417.936690] ---[ end trace 0000000000000000 ]---
>> 
>> 
>> The tracing revealed, it may be caused by a ZCR_EL2 write:
>> [109] ..... 52068.375927: kvm_sys_access: PC: 806608b8 SYS_ZCR_EL2 (3,4,1,2,0) write
>> 
>> The instruction from ELR also matches: msr zcr_el2, x1
>> 
>> The reason might be CPTR_EL2, its value just before this instruction is executed, is 0.
>> However before the start of the VM execution, it has 0x22ff.
>> 
>> I can see accesses to this register in the trace log as well, just before ZCR_EL2 is accessed.
>> [109] ..... 52068.375922: kvm_sys_access: PC: 806608a4 SYS_CPTR_EL2 (3,4,1,1,2) read
>> [109] ..... 52068.375925: kvm_sys_access: PC: 806608ac SYS_CPTR_EL2 (3,4,1,1,2) write
>> 
>> I’m running Linux 6.16.0 as my Guest.
>> Nested Virtualization works fine with SVE disabled, so does SVE without NV.
>> Could it be caused by a bug in userspace hypervisor code?
> 
> Unlikely. The warning indicates that we are incrementing PC while
> there is a pending exception. Having both at the same time is a very
> bad bug -- hence the warning.
> 
> Looking at the code with the above in mind, something immediately
> jumps at me. Can you try the following (against 6.17, but you'll
> surely be able to apply it against 6.16):
> 
> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
> index 91053aa832d08..a07ad5c92583d 100644
> --- a/arch/arm64/kvm/sys_regs.c
> +++ b/arch/arm64/kvm/sys_regs.c
> @@ -2705,7 +2705,7 @@ static bool access_zcr_el2(struct kvm_vcpu *vcpu,
> 
> if (guest_hyp_sve_traps_enabled(vcpu)) {
> kvm_inject_nested_sve_trap(vcpu);
> - return true;
> + return false;
> }
> 
> if (!p->is_write) {
> 
> This should make the warning go away -- not sure about anything else.
> Note that I do not have access to an NV+SVE capable machine, so you're
> are basically on your own, unless Oliver has a box he can reproduce
> this on.
> 
> I would also recommend to update to 6.17 -- 6.16 was the first release
> with NV, and while it may work, it will also have a lot of ugly bugs.

I will try, but as I don’t own this machine,
changing kernels might be… challenging.
If there’s anything else I can provide to help track it down,
please let me know.

> 
> Thanks,
> 
> M.
> 
> -- 
> Without deviation from the norm, progress is not possible.

Regards,
Jan

Re: KVM NV + SVE host OS warning

[Marc Zyngier]

On Thu, 25 Sep 2025 16:10:22 +0100,
Jan Kotas <jank@cadence.com> wrote:
> 
> Hi Marc,
> 
> Thank you for a quick response.
> 
> > On 25 Sep 2025, at 16:38, Marc Zyngier <maz@kernel.org> wrote:
> > 
> > EXTERNAL MAIL
> > 
> > 
> > [+Oliver for the SVE stuff]
> > 
> > Hi Jan,

[...]

> > 
> > I would also recommend to update to 6.17 -- 6.16 was the first release
> > with NV, and while it may work, it will also have a lot of ugly bugs.
> 
> I will try, but as I don’t own this machine, changing kernels might
> be… challenging.

Well, you're already passing something on the command-line that is
documented as "experimental and should be used with extreme caution".
Changing the kernel is a walk in the park compared to that ;-).

> If there’s anything else I can provide to help track it down,
> please let me know.

Testing the provided patch would be good -- I would if I could.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: KVM NV + SVE host OS warning

[Oliver Upton]

On Thu, Sep 25, 2025 at 04:35:13PM +0100, Marc Zyngier wrote:
> On Thu, 25 Sep 2025 16:10:22 +0100,
> Jan Kotas <jank@cadence.com> wrote:
> > 
> > Hi Marc,
> > 
> > Thank you for a quick response.
> > 
> > > On 25 Sep 2025, at 16:38, Marc Zyngier <maz@kernel.org> wrote:
> > > 
> > > EXTERNAL MAIL
> > > 
> > > 
> > > [+Oliver for the SVE stuff]
> > > 
> > > Hi Jan,
> 
> [...]
> 
> > > 
> > > I would also recommend to update to 6.17 -- 6.16 was the first release
> > > with NV, and while it may work, it will also have a lot of ugly bugs.
> > 
> > I will try, but as I don’t own this machine, changing kernels might
> > be… challenging.
> 
> Well, you're already passing something on the command-line that is
> documented as "experimental and should be used with extreme caution".
> Changing the kernel is a walk in the park compared to that ;-).

+1, also it'd be good to know what you're doing in the L1 and the L2
that leads to this bug.

I just tried the original test setup I used when doing NV SVE and things
appear to work (kvmarm/next + initramfs at L1 + L2, tested with
sve-stress selftest).

> > If there’s anything else I can provide to help track it down,
> > please let me know.
> 
> Testing the provided patch would be good -- I would if I could.

Although I didn't repro the bug, the patch itself LGTM.

Thanks,
Oliver

Re: KVM NV + SVE host OS warning

[Jan Kotas]

Hello,

I was finally able to do some validation, sorry for a long delay.

First I applied the "Don't advance PC" patch on top of 6.16.9.
It fixed the error message, but the Guest didn’t boot.
I didn’t debug it further.

Then I applied it on top of 6.17 along with Oliver’s second patch.
Guest OS stops booting because of an exception, when accessing ZCR_EL2.

I checked the ESR_EL2 register and it has 0x66000000:

Access to SVE functionality trapped as a result of CPACR_EL1.ZEN,
CPTR_EL2.ZEN, CPTR_EL2.TZ, or CPTR_EL3.EZ

I’ll continue the debug to make sure the issue is not on our end.


Regards,
Jan

> On 26 Sep 2025, at 00:46, Oliver Upton <oliver.upton@linux.dev> wrote:
> 
> EXTERNAL MAIL
> 
> 
> On Thu, Sep 25, 2025 at 04:35:13PM +0100, Marc Zyngier wrote:
>> On Thu, 25 Sep 2025 16:10:22 +0100,
>> Jan Kotas <jank@cadence.com> wrote:
>>> 
>>> Hi Marc,
>>> 
>>> Thank you for a quick response.
>>> 
>>>> On 25 Sep 2025, at 16:38, Marc Zyngier <maz@kernel.org> wrote:
>>>> 
>>>> EXTERNAL MAIL
>>>> 
>>>> 
>>>> [+Oliver for the SVE stuff]
>>>> 
>>>> Hi Jan,
>> 
>> [...]
>> 
>>>> 
>>>> I would also recommend to update to 6.17 -- 6.16 was the first release
>>>> with NV, and while it may work, it will also have a lot of ugly bugs.
>>> 
>>> I will try, but as I don’t own this machine, changing kernels might
>>> be… challenging.
>> 
>> Well, you're already passing something on the command-line that is
>> documented as "experimental and should be used with extreme caution".
>> Changing the kernel is a walk in the park compared to that ;-).
> 
> +1, also it'd be good to know what you're doing in the L1 and the L2
> that leads to this bug.
> 
> I just tried the original test setup I used when doing NV SVE and things
> appear to work (kvmarm/next + initramfs at L1 + L2, tested with
> sve-stress selftest).
> 
>>> If there’s anything else I can provide to help track it down,
>>> please let me know.
>> 
>> Testing the provided patch would be good -- I would if I could.
> 
> Although I didn't repro the bug, the patch itself LGTM.
> 
> Thanks,
> Oliver

Re: KVM NV + SVE host OS warning

[Oliver Upton]

On Tue, Oct 07, 2025 at 11:12:31AM +0000, Jan Kotas wrote:
> Hello,
> 
> I was finally able to do some validation, sorry for a long delay.

No worries, thanks for testing.

> First I applied the "Don't advance PC" patch on top of 6.16.9.
> It fixed the error message, but the Guest didn’t boot.
> I didn’t debug it further.
> 
> Then I applied it on top of 6.17 along with Oliver’s second patch.
> Guest OS stops booting because of an exception, when accessing ZCR_EL2.
> 
> I checked the ESR_EL2 register and it has 0x66000000:
> 
> Access to SVE functionality trapped as a result of CPACR_EL1.ZEN,
> CPTR_EL2.ZEN, CPTR_EL2.TZ, or CPTR_EL3.EZ
> 
> I’ll continue the debug to make sure the issue is not on our end.

Could you please share the repro steps? Also, is the guest kernel
unmodified? FWIW, I tested kvmarm/next as the kernel at all levels,
kvmtool as the VMM and E2H=RES1.

While the trap to L0 is unavoidable, reinjecting the SVE trap depends on
the L0 view of CPTR_EL2 which originates from the in-memory value.
Unless there's a bug lurking this should always be in agreement with the
effective value programmed in CPACR_EL1.

Thanks,
Oliver

Re: KVM NV + SVE host OS warning

[Jan Kotas]

Hi Oliver,

> On 8 Oct 2025, at 01:26, Oliver Upton <oliver.upton@linux.dev> wrote:
> 
> EXTERNAL MAIL
> 
> 
> On Tue, Oct 07, 2025 at 11:12:31AM +0000, Jan Kotas wrote:
>> Hello,
>> 
>> I was finally able to do some validation, sorry for a long delay.
> 
> No worries, thanks for testing.
> 
>> First I applied the "Don't advance PC" patch on top of 6.16.9.
>> It fixed the error message, but the Guest didn’t boot.
>> I didn’t debug it further.
>> 
>> Then I applied it on top of 6.17 along with Oliver’s second patch.
>> Guest OS stops booting because of an exception, when accessing ZCR_EL2.
>> 
>> I checked the ESR_EL2 register and it has 0x66000000:
>> 
>> Access to SVE functionality trapped as a result of CPACR_EL1.ZEN,
>> CPTR_EL2.ZEN, CPTR_EL2.TZ, or CPTR_EL3.EZ
>> 
>> I’ll continue the debug to make sure the issue is not on our end.
> 
> Could you please share the repro steps? Also, is the guest kernel
> unmodified? FWIW, I tested kvmarm/next as the kernel at all levels,
> kvmtool as the VMM and E2H=RES1.

I’m running a minimal, unmodified Linux 6.16.0, for my integration tests.
It only has a CPU, a GIC, a few UARTs, and uses initramfs.

In my VMM I only set KVM_ARM_VCPU_HAS_EL2, without HAS_EL2_E2H0.
To start the kernel I set the X0-X3 registers.
It worked fine, so far, for all other cases than SVE && NV.

> While the trap to L0 is unavoidable, reinjecting the SVE trap depends on
> the L0 view of CPTR_EL2 which originates from the in-memory value.
> Unless there's a bug lurking this should always be in agreement with the
> effective value programmed in CPACR_EL1.

I checked the place, where it’s failing.
It looks like Guest clears CPTR_EL2 from 0x33ff to 0.
CPACR_EL1 is 0 at this point. This doesn’t seem to be correct.

8062192c:   mrs     x0, cptr_el2
80621930:   and     x0, x0, #0xfffffffffffffeff
80621934:   msr     cptr_el2, x0
80621938:   isb

And accessing ZCR_EL2 is trapped, and causes an exception.
8062193c:   mov     x1, #0xf
80621940:   msr     zcr_el2, x1

Looks like the Guest OS executes
.Lcptr_nvhe_\@: // nVHE case
from el2_setup.h.

> Thanks,
> Oliver

Regards,
Jan

Re: KVM NV + SVE host OS warning

[Jan Kotas]

> 
> On 8 Oct 2025, at 08:32, Jan Kotas <jank@cadence.com> wrote:
> 
> Hi Oliver,
> 
>> On 8 Oct 2025, at 01:26, Oliver Upton <oliver.upton@linux.dev> wrote:
>> 
>> EXTERNAL MAIL
>> 
>> 
>> On Tue, Oct 07, 2025 at 11:12:31AM +0000, Jan Kotas wrote:
>>> Hello,
>>> 
>>> I was finally able to do some validation, sorry for a long delay.
>> 
>> No worries, thanks for testing.
>> 
>>> First I applied the "Don't advance PC" patch on top of 6.16.9.
>>> It fixed the error message, but the Guest didn’t boot.
>>> I didn’t debug it further.
>>> 
>>> Then I applied it on top of 6.17 along with Oliver’s second patch.
>>> Guest OS stops booting because of an exception, when accessing ZCR_EL2.
>>> 
>>> I checked the ESR_EL2 register and it has 0x66000000:
>>> 
>>> Access to SVE functionality trapped as a result of CPACR_EL1.ZEN,
>>> CPTR_EL2.ZEN, CPTR_EL2.TZ, or CPTR_EL3.EZ
>>> 
>>> I’ll continue the debug to make sure the issue is not on our end.
>> 
>> Could you please share the repro steps? Also, is the guest kernel
>> unmodified? FWIW, I tested kvmarm/next as the kernel at all levels,
>> kvmtool as the VMM and E2H=RES1.
> 
> I’m running a minimal, unmodified Linux 6.16.0, for my integration tests.
> It only has a CPU, a GIC, a few UARTs, and uses initramfs.
> 
> In my VMM I only set KVM_ARM_VCPU_HAS_EL2, without HAS_EL2_E2H0.
> To start the kernel I set the X0-X3 registers.
> It worked fine, so far, for all other cases than SVE && NV.
> 
>> While the trap to L0 is unavoidable, reinjecting the SVE trap depends on
>> the L0 view of CPTR_EL2 which originates from the in-memory value.
>> Unless there's a bug lurking this should always be in agreement with the
>> effective value programmed in CPACR_EL1.
> 
> I checked the place, where it’s failing.
> It looks like Guest clears CPTR_EL2 from 0x33ff to 0.
> CPACR_EL1 is 0 at this point. This doesn’t seem to be correct.
> 
> 8062192c:   mrs     x0, cptr_el2
> 80621930:   and     x0, x0, #0xfffffffffffffeff
> 80621934:   msr     cptr_el2, x0
> 80621938:   isb
> 
> And accessing ZCR_EL2 is trapped, and causes an exception.
> 8062193c:   mov     x1, #0xf
> 80621940:   msr     zcr_el2, x1
> 
> Looks like the Guest OS executes
> .Lcptr_nvhe_\@: // nVHE case
> from el2_setup.h.

I did some more debugging.
I checked __check_hvhe and it looks like it jumps to fail.
HCR_EL2 has 0x100030080000000.
E2H is set to 0, even though it should be RES1?
I changed the value using a debugger, so the check passed.
The code took a different branch and it seems the SVE init passed.


>> Thanks,
>> Oliver
> 
> Regards,
> Jan

Re: KVM NV + SVE host OS warning

[Marc Zyngier]

On Wed, 08 Oct 2025 08:29:38 +0100,
Jan Kotas <jank@cadence.com> wrote:
> 
> 
> 
> > 
> > On 8 Oct 2025, at 08:32, Jan Kotas <jank@cadence.com> wrote:
> > 
> > Hi Oliver,
> > 
> >> On 8 Oct 2025, at 01:26, Oliver Upton <oliver.upton@linux.dev> wrote:
> >> 
> >> EXTERNAL MAIL
> >> 
> >> 
> >> On Tue, Oct 07, 2025 at 11:12:31AM +0000, Jan Kotas wrote:
> >>> Hello,
> >>> 
> >>> I was finally able to do some validation, sorry for a long delay.
> >> 
> >> No worries, thanks for testing.
> >> 
> >>> First I applied the "Don't advance PC" patch on top of 6.16.9.
> >>> It fixed the error message, but the Guest didn’t boot.
> >>> I didn’t debug it further.
> >>> 
> >>> Then I applied it on top of 6.17 along with Oliver’s second patch.
> >>> Guest OS stops booting because of an exception, when accessing ZCR_EL2.
> >>> 
> >>> I checked the ESR_EL2 register and it has 0x66000000:
> >>> 
> >>> Access to SVE functionality trapped as a result of CPACR_EL1.ZEN,
> >>> CPTR_EL2.ZEN, CPTR_EL2.TZ, or CPTR_EL3.EZ
> >>> 
> >>> I’ll continue the debug to make sure the issue is not on our end.
> >> 
> >> Could you please share the repro steps? Also, is the guest kernel
> >> unmodified? FWIW, I tested kvmarm/next as the kernel at all levels,
> >> kvmtool as the VMM and E2H=RES1.
> > 
> > I’m running a minimal, unmodified Linux 6.16.0, for my integration tests.
> > It only has a CPU, a GIC, a few UARTs, and uses initramfs.
> > 
> > In my VMM I only set KVM_ARM_VCPU_HAS_EL2, without HAS_EL2_E2H0.
> > To start the kernel I set the X0-X3 registers.
> > It worked fine, so far, for all other cases than SVE && NV.
> > 
> >> While the trap to L0 is unavoidable, reinjecting the SVE trap depends on
> >> the L0 view of CPTR_EL2 which originates from the in-memory value.
> >> Unless there's a bug lurking this should always be in agreement with the
> >> effective value programmed in CPACR_EL1.
> > 
> > I checked the place, where it’s failing.
> > It looks like Guest clears CPTR_EL2 from 0x33ff to 0.
> > CPACR_EL1 is 0 at this point. This doesn’t seem to be correct.
> > 
> > 8062192c:   mrs     x0, cptr_el2
> > 80621930:   and     x0, x0, #0xfffffffffffffeff
> > 80621934:   msr     cptr_el2, x0
> > 80621938:   isb
> > 
> > And accessing ZCR_EL2 is trapped, and causes an exception.
> > 8062193c:   mov     x1, #0xf
> > 80621940:   msr     zcr_el2, x1
> > 
> > Looks like the Guest OS executes
> > .Lcptr_nvhe_\@: // nVHE case
> > from el2_setup.h.
> 
> I did some more debugging.
> I checked __check_hvhe and it looks like it jumps to fail.
> HCR_EL2 has 0x100030080000000.
> E2H is set to 0, even though it should be RES1?

The value itself doesn't matter for KVM. However, the guest can write
anything it wants, and that value will hold *in the register*.

> I changed the value using a debugger, so the check passed.
> The code took a different branch and it seems the SVE init passed.

Oh crap. You are on V2, which doesn't have FGT, so ID_AA64MMFR4_EL1
doesn't trap, and the kernel end-up assuming that the CPU is nVHE
capable. Nothing works after that.

I'm afraid there is no real fix for this, only hacks...

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: KVM NV + SVE host OS warning

[Jan Kotas]

Hi Marc,

Thanks for looking into this.


> On 8 Oct 2025, at 11:28, Marc Zyngier <maz@kernel.org> wrote:
> 
> EXTERNAL MAIL
> 
> 
> On Wed, 08 Oct 2025 08:29:38 +0100,
> Jan Kotas <jank@cadence.com> wrote:
>> 
>> 
>> 
>>> 
>>> On 8 Oct 2025, at 08:32, Jan Kotas <jank@cadence.com> wrote:
>>> 
>>> Hi Oliver,
>>> 
>>>> On 8 Oct 2025, at 01:26, Oliver Upton <oliver.upton@linux.dev> wrote:
>>>> 
>>>> EXTERNAL MAIL
>>>> 
>>>> 
>>>> On Tue, Oct 07, 2025 at 11:12:31AM +0000, Jan Kotas wrote:
>>>>> Hello,
>>>>> 
>>>>> I was finally able to do some validation, sorry for a long delay.
>>>> 
>>>> No worries, thanks for testing.
>>>> 
>>>>> First I applied the "Don't advance PC" patch on top of 6.16.9.
>>>>> It fixed the error message, but the Guest didn’t boot.
>>>>> I didn’t debug it further.
>>>>> 
>>>>> Then I applied it on top of 6.17 along with Oliver’s second patch.
>>>>> Guest OS stops booting because of an exception, when accessing ZCR_EL2.
>>>>> 
>>>>> I checked the ESR_EL2 register and it has 0x66000000:
>>>>> 
>>>>> Access to SVE functionality trapped as a result of CPACR_EL1.ZEN,
>>>>> CPTR_EL2.ZEN, CPTR_EL2.TZ, or CPTR_EL3.EZ
>>>>> 
>>>>> I’ll continue the debug to make sure the issue is not on our end.
>>>> 
>>>> Could you please share the repro steps? Also, is the guest kernel
>>>> unmodified? FWIW, I tested kvmarm/next as the kernel at all levels,
>>>> kvmtool as the VMM and E2H=RES1.
>>> 
>>> I’m running a minimal, unmodified Linux 6.16.0, for my integration tests.
>>> It only has a CPU, a GIC, a few UARTs, and uses initramfs.
>>> 
>>> In my VMM I only set KVM_ARM_VCPU_HAS_EL2, without HAS_EL2_E2H0.
>>> To start the kernel I set the X0-X3 registers.
>>> It worked fine, so far, for all other cases than SVE && NV.
>>> 
>>>> While the trap to L0 is unavoidable, reinjecting the SVE trap depends on
>>>> the L0 view of CPTR_EL2 which originates from the in-memory value.
>>>> Unless there's a bug lurking this should always be in agreement with the
>>>> effective value programmed in CPACR_EL1.
>>> 
>>> I checked the place, where it’s failing.
>>> It looks like Guest clears CPTR_EL2 from 0x33ff to 0.
>>> CPACR_EL1 is 0 at this point. This doesn’t seem to be correct.
>>> 
>>> 8062192c:   mrs     x0, cptr_el2
>>> 80621930:   and     x0, x0, #0xfffffffffffffeff
>>> 80621934:   msr     cptr_el2, x0
>>> 80621938:   isb
>>> 
>>> And accessing ZCR_EL2 is trapped, and causes an exception.
>>> 8062193c:   mov     x1, #0xf
>>> 80621940:   msr     zcr_el2, x1
>>> 
>>> Looks like the Guest OS executes
>>> .Lcptr_nvhe_\@: // nVHE case
>>> from el2_setup.h.
>> 
>> I did some more debugging.
>> I checked __check_hvhe and it looks like it jumps to fail.
>> HCR_EL2 has 0x100030080000000.
>> E2H is set to 0, even though it should be RES1?
> 
> The value itself doesn't matter for KVM. However, the guest can write
> anything it wants, and that value will hold *in the register*.
> 
>> I changed the value using a debugger, so the check passed.
>> The code took a different branch and it seems the SVE init passed.
> 
> Oh crap. You are on V2, which doesn't have FGT, so ID_AA64MMFR4_EL1
> doesn't trap, and the kernel end-up assuming that the CPU is nVHE
> capable. Nothing works after that.
> 
> I'm afraid there is no real fix for this, only hacks...

Is it possible to trap/emulate HCR_EL2, and make sure E2H is always 1,
in cases when only KVM_ARM_VCPU_HAS_EL2 is set?
Would that help?

As a test, I tried forcing this bit before running VCPUs,
but reading it via MRS got a value without it being set.


> 
> M.
> 
> -- 
> Without deviation from the norm, progress is not possible.

Regards,
Jan

Re: KVM NV + SVE host OS warning

[Marc Zyngier]

On Wed, 08 Oct 2025 10:45:51 +0100,
Jan Kotas <jank@cadence.com> wrote:
> 
> Hi Marc,
> 
> Thanks for looking into this.
> 
> 
> > On 8 Oct 2025, at 11:28, Marc Zyngier <maz@kernel.org> wrote:
> > 
> > EXTERNAL MAIL
> > 
> > 
> > On Wed, 08 Oct 2025 08:29:38 +0100,
> > Jan Kotas <jank@cadence.com> wrote:
> >> 
> >> 
> >> 
> >>> 
> >>> On 8 Oct 2025, at 08:32, Jan Kotas <jank@cadence.com> wrote:
> >>> 
> >>> Hi Oliver,
> >>> 
> >>>> On 8 Oct 2025, at 01:26, Oliver Upton <oliver.upton@linux.dev> wrote:
> >>>> 
> >>>> EXTERNAL MAIL
> >>>> 
> >>>> 
> >>>> On Tue, Oct 07, 2025 at 11:12:31AM +0000, Jan Kotas wrote:
> >>>>> Hello,
> >>>>> 
> >>>>> I was finally able to do some validation, sorry for a long delay.
> >>>> 
> >>>> No worries, thanks for testing.
> >>>> 
> >>>>> First I applied the "Don't advance PC" patch on top of 6.16.9.
> >>>>> It fixed the error message, but the Guest didn’t boot.
> >>>>> I didn’t debug it further.
> >>>>> 
> >>>>> Then I applied it on top of 6.17 along with Oliver’s second patch.
> >>>>> Guest OS stops booting because of an exception, when accessing ZCR_EL2.
> >>>>> 
> >>>>> I checked the ESR_EL2 register and it has 0x66000000:
> >>>>> 
> >>>>> Access to SVE functionality trapped as a result of CPACR_EL1.ZEN,
> >>>>> CPTR_EL2.ZEN, CPTR_EL2.TZ, or CPTR_EL3.EZ
> >>>>> 
> >>>>> I’ll continue the debug to make sure the issue is not on our end.
> >>>> 
> >>>> Could you please share the repro steps? Also, is the guest kernel
> >>>> unmodified? FWIW, I tested kvmarm/next as the kernel at all levels,
> >>>> kvmtool as the VMM and E2H=RES1.
> >>> 
> >>> I’m running a minimal, unmodified Linux 6.16.0, for my integration tests.
> >>> It only has a CPU, a GIC, a few UARTs, and uses initramfs.
> >>> 
> >>> In my VMM I only set KVM_ARM_VCPU_HAS_EL2, without HAS_EL2_E2H0.
> >>> To start the kernel I set the X0-X3 registers.
> >>> It worked fine, so far, for all other cases than SVE && NV.
> >>> 
> >>>> While the trap to L0 is unavoidable, reinjecting the SVE trap depends on
> >>>> the L0 view of CPTR_EL2 which originates from the in-memory value.
> >>>> Unless there's a bug lurking this should always be in agreement with the
> >>>> effective value programmed in CPACR_EL1.
> >>> 
> >>> I checked the place, where it’s failing.
> >>> It looks like Guest clears CPTR_EL2 from 0x33ff to 0.
> >>> CPACR_EL1 is 0 at this point. This doesn’t seem to be correct.
> >>> 
> >>> 8062192c:   mrs     x0, cptr_el2
> >>> 80621930:   and     x0, x0, #0xfffffffffffffeff
> >>> 80621934:   msr     cptr_el2, x0
> >>> 80621938:   isb
> >>> 
> >>> And accessing ZCR_EL2 is trapped, and causes an exception.
> >>> 8062193c:   mov     x1, #0xf
> >>> 80621940:   msr     zcr_el2, x1
> >>> 
> >>> Looks like the Guest OS executes
> >>> .Lcptr_nvhe_\@: // nVHE case
> >>> from el2_setup.h.
> >> 
> >> I did some more debugging.
> >> I checked __check_hvhe and it looks like it jumps to fail.
> >> HCR_EL2 has 0x100030080000000.
> >> E2H is set to 0, even though it should be RES1?
> > 
> > The value itself doesn't matter for KVM. However, the guest can write
> > anything it wants, and that value will hold *in the register*.
> > 
> >> I changed the value using a debugger, so the check passed.
> >> The code took a different branch and it seems the SVE init passed.
> > 
> > Oh crap. You are on V2, which doesn't have FGT, so ID_AA64MMFR4_EL1
> > doesn't trap, and the kernel end-up assuming that the CPU is nVHE
> > capable. Nothing works after that.
> > 
> > I'm afraid there is no real fix for this, only hacks...
> 
> Is it possible to trap/emulate HCR_EL2, and make sure E2H is always 1,
> in cases when only KVM_ARM_VCPU_HAS_EL2 is set?
> Would that help?

There are no traps for HCR_EL2.

> As a test, I tried forcing this bit before running VCPUs, but
> reading it via MRS got a value without it being set.

Where did you read that from?

If you apply the following change to your guest, does it start
behaving?

	M.

diff --git a/arch/arm64/include/asm/el2_setup.h b/arch/arm64/include/asm/el2_setup.h
index 46033027510cc..392c9f4016f2e 100644
--- a/arch/arm64/include/asm/el2_setup.h
+++ b/arch/arm64/include/asm/el2_setup.h
@@ -34,7 +34,7 @@
 	mrs_s	x1, SYS_ID_AA64MMFR4_EL1
 	sbfx	x1, x1, #ID_AA64MMFR4_EL1_E2H0_SHIFT, #ID_AA64MMFR4_EL1_E2H0_WIDTH
 	cmp	x1, #0
-	b.ge	.LnVHE_\@
+//	b.ge	.LnVHE_\@
 
 	orr	x0, x0, #HCR_E2H
 .LnVHE_\@:

-- 
Without deviation from the norm, progress is not possible.

Re: KVM NV + SVE host OS warning

[Jan Kotas]

> On 8 Oct 2025, at 13:58, Marc Zyngier <maz@kernel.org> wrote:
> 
> EXTERNAL MAIL
> 
> 
> On Wed, 08 Oct 2025 10:45:51 +0100,
> Jan Kotas <jank@cadence.com> wrote:
>> 
>> Hi Marc,
>> 
>> Thanks for looking into this.
>> 
>> 
>>> On 8 Oct 2025, at 11:28, Marc Zyngier <maz@kernel.org> wrote:
>>> 
>>> EXTERNAL MAIL
>>> 
>>> 
>>> On Wed, 08 Oct 2025 08:29:38 +0100,
>>> Jan Kotas <jank@cadence.com> wrote:
>>>> 
>>>> 
>>>> 
>>>>> 
>>>>> On 8 Oct 2025, at 08:32, Jan Kotas <jank@cadence.com> wrote:
>>>>> 
>>>>> Hi Oliver,
>>>>> 
>>>>>> On 8 Oct 2025, at 01:26, Oliver Upton <oliver.upton@linux.dev> wrote:
>>>>>> 
>>>>>> EXTERNAL MAIL
>>>>>> 
>>>>>> 
>>>>>> On Tue, Oct 07, 2025 at 11:12:31AM +0000, Jan Kotas wrote:
>>>>>>> Hello,
>>>>>>> 
>>>>>>> I was finally able to do some validation, sorry for a long delay.
>>>>>> 
>>>>>> No worries, thanks for testing.
>>>>>> 
>>>>>>> First I applied the "Don't advance PC" patch on top of 6.16.9.
>>>>>>> It fixed the error message, but the Guest didn’t boot.
>>>>>>> I didn’t debug it further.
>>>>>>> 
>>>>>>> Then I applied it on top of 6.17 along with Oliver’s second patch.
>>>>>>> Guest OS stops booting because of an exception, when accessing ZCR_EL2.
>>>>>>> 
>>>>>>> I checked the ESR_EL2 register and it has 0x66000000:
>>>>>>> 
>>>>>>> Access to SVE functionality trapped as a result of CPACR_EL1.ZEN,
>>>>>>> CPTR_EL2.ZEN, CPTR_EL2.TZ, or CPTR_EL3.EZ
>>>>>>> 
>>>>>>> I’ll continue the debug to make sure the issue is not on our end.
>>>>>> 
>>>>>> Could you please share the repro steps? Also, is the guest kernel
>>>>>> unmodified? FWIW, I tested kvmarm/next as the kernel at all levels,
>>>>>> kvmtool as the VMM and E2H=RES1.
>>>>> 
>>>>> I’m running a minimal, unmodified Linux 6.16.0, for my integration tests.
>>>>> It only has a CPU, a GIC, a few UARTs, and uses initramfs.
>>>>> 
>>>>> In my VMM I only set KVM_ARM_VCPU_HAS_EL2, without HAS_EL2_E2H0.
>>>>> To start the kernel I set the X0-X3 registers.
>>>>> It worked fine, so far, for all other cases than SVE && NV.
>>>>> 
>>>>>> While the trap to L0 is unavoidable, reinjecting the SVE trap depends on
>>>>>> the L0 view of CPTR_EL2 which originates from the in-memory value.
>>>>>> Unless there's a bug lurking this should always be in agreement with the
>>>>>> effective value programmed in CPACR_EL1.
>>>>> 
>>>>> I checked the place, where it’s failing.
>>>>> It looks like Guest clears CPTR_EL2 from 0x33ff to 0.
>>>>> CPACR_EL1 is 0 at this point. This doesn’t seem to be correct.
>>>>> 
>>>>> 8062192c:   mrs     x0, cptr_el2
>>>>> 80621930:   and     x0, x0, #0xfffffffffffffeff
>>>>> 80621934:   msr     cptr_el2, x0
>>>>> 80621938:   isb
>>>>> 
>>>>> And accessing ZCR_EL2 is trapped, and causes an exception.
>>>>> 8062193c:   mov     x1, #0xf
>>>>> 80621940:   msr     zcr_el2, x1
>>>>> 
>>>>> Looks like the Guest OS executes
>>>>> .Lcptr_nvhe_\@: // nVHE case
>>>>> from el2_setup.h.
>>>> 
>>>> I did some more debugging.
>>>> I checked __check_hvhe and it looks like it jumps to fail.
>>>> HCR_EL2 has 0x100030080000000.
>>>> E2H is set to 0, even though it should be RES1?
>>> 
>>> The value itself doesn't matter for KVM. However, the guest can write
>>> anything it wants, and that value will hold *in the register*.
>>> 
>>>> I changed the value using a debugger, so the check passed.
>>>> The code took a different branch and it seems the SVE init passed.
>>> 
>>> Oh crap. You are on V2, which doesn't have FGT, so ID_AA64MMFR4_EL1
>>> doesn't trap, and the kernel end-up assuming that the CPU is nVHE
>>> capable. Nothing works after that.
>>> 
>>> I'm afraid there is no real fix for this, only hacks...
>> 
>> Is it possible to trap/emulate HCR_EL2, and make sure E2H is always 1,
>> in cases when only KVM_ARM_VCPU_HAS_EL2 is set?
>> Would that help?
> 
> There are no traps for HCR_EL2.
> 
>> As a test, I tried forcing this bit before running VCPUs, but
>> reading it via MRS got a value without it being set.
> 
> Where did you read that from?

I checked the X1 register, after: mrs x1, hcr_el2 in __check_hvhe.


> If you apply the following change to your guest, does it start
> behaving?
> 
> M.
> 
> diff --git a/arch/arm64/include/asm/el2_setup.h b/arch/arm64/include/asm/el2_setup.h
> index 46033027510cc..392c9f4016f2e 100644
> --- a/arch/arm64/include/asm/el2_setup.h
> +++ b/arch/arm64/include/asm/el2_setup.h
> @@ -34,7 +34,7 @@
> mrs_s x1, SYS_ID_AA64MMFR4_EL1
> sbfx x1, x1, #ID_AA64MMFR4_EL1_E2H0_SHIFT, #ID_AA64MMFR4_EL1_E2H0_WIDTH
> cmp x1, #0
> - b.ge .LnVHE_\@
> +// b.ge .LnVHE_\@
> 
> orr x0, x0, #HCR_E2H
> .LnVHE_\@:
> 

With this change Guest OS starts the boot process.

[ 0.000000] CPU features: detected: Virtualization Host Extensions
[ 6.998696] SMP: Total of 4 processors activated.
[ 7.257045] CPU: All CPU(s) started at EL2
[ 185.781062] SVE: maximum available vector length 16 bytes per vector
[ 186.104159] SVE: default vector length 16 bytes per vector

I also observed something interesting.
When PSCI method is set to HVC, I get a kernel panic.
This wasn’t a problem in all other test cases.

[ 3.391773] smp: Bringing up secondary CPUs ...
[ 3.680981] Unhandled 64-bit el1h sync exception on CPU0, ESR 0x000000005a000000 -- HVC (AArch64)
[ 3.681680] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.16.0 #6 VOLUNTARY 
[ 3.682305] Hardware name: TEST_GUEST (DT)
[ 3.682747] pstate: a0000009 (NzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 3.683057] pc : __arm_smccc_hvc+0x8/0x30
[ 3.684216] lr : __invoke_psci_fn_hvc+0x3c/0x68
[ 3.685649] sp : ffff800080cabb00
[ 3.685739] x29: ffff800080cabb10 x28: 0000000000000028 x27: 0000000000000000
[ 3.686066] x26: 0000000000000000 x25: ffff80008003cac8 x24: ffff7fffbf294000
[ 3.686275] x23: 0000000000000001 x22: ffff800080c414f8 x21: ffff8000803ab888
[ 3.686445] x20: 0000000000000001 x19: ffff0000010d8000 x18: 00000000363e3f75
[ 3.686616] x17: 000000003a5017c7 x16: 00000000c25a262c x15: 0000b6183cc00fe2
[ 3.686806] x14: 0000000000000000 x13: 000000000000003e x12: ffff0000010d8000
[ 3.686981] x11: ffff00003fd97d54 x10: 0000000000000001 x9 : 0000000000000000
[ 3.687175] x8 : 0000000000000001 x7 : 0000000000000000 x6 : 0000000000000000
[ 3.687346] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[ 3.687505] x2 : 000000008073e454 x1 : 0000000000000100 x0 : 00000000c4000003
[ 3.687723] Kernel panic - not syncing: Unhandled exception
[ 3.687835] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.16.0 #6 VOLUNTARY 
[ 3.688028] Hardware name: TEST_GUEST (DT)
[ 3.688105] Call trace:
[ 3.688164] show_stack+0x18/0x24 (C)
[ 3.688377] dump_stack_lvl+0x38/0x90
[ 3.689093] dump_stack+0x18/0x24
[ 3.689339] panic+0x378/0x38c
[ 3.689724] arm64_exit_nmi.isra.0+0x0/0x94
[ 3.690216] el1h_64_sync_handler+0x8c/0x118
[ 3.690506] el1h_64_sync+0x6c/0x70
[ 3.690647] __arm_smccc_hvc+0x8/0x30 (P)
[ 3.691187] psci_0_1_cpu_on+0x2c/0x60
[ 3.691500] cpu_psci_cpu_boot+0x40/0x7c
[ 3.692400] __cpu_up+0x44/0x1d4
[ 3.692703] bringup_cpu+0x48/0x190
[ 3.692924] cpuhp_invoke_callback+0x120/0x230
[ 3.693127] __cpuhp_invoke_callback_range+0xa8/0x13c
[ 3.693337] _cpu_up+0x16c/0x398
[ 3.693550] cpu_up+0xa8/0x11c
[ 3.693762] bringup_nonboot_cpus+0x68/0xd4
[ 3.694451] smp_init+0x30/0x88
[ 3.694629] kernel_init_freeable+0x150/0x348
[ 3.694945] kernel_init+0x20/0x128
[ 3.695265] ret_from_fork+0x10/0x20
[ 15.461588] ---[ end Kernel panic - not syncing: Unhandled exception ]---


On a side note, thank you for giving me some pointers.
With HAS_EL2_E2H the original Guest OS boots fine in nVHE.

[    0.204744] CPU: All CPU(s) started at EL2
[   18.483737] SVE: maximum available vector length 16 bytes per vector
[   18.500959] SVE: default vector length 16 bytes per vector
[   25.999475] kvm [1]: Hyp nVHE mode initialized successfully

> -- 
> Without deviation from the norm, progress is not possible.

Regards,
Jan

Re: KVM NV + SVE host OS warning

[Marc Zyngier]

On Wed, 08 Oct 2025 14:43:29 +0100,
Jan Kotas <jank@cadence.com> wrote:
> >>
> >> Is it possible to trap/emulate HCR_EL2, and make sure E2H is always 1,
> >> in cases when only KVM_ARM_VCPU_HAS_EL2 is set?
> >> Would that help?
> > 
> > There are no traps for HCR_EL2.
> > 
> >> As a test, I tried forcing this bit before running VCPUs, but
> >> reading it via MRS got a value without it being set.
> > 
> > Where did you read that from?
> 
> I checked the X1 register, after: mrs x1, hcr_el2 in __check_hvhe.

That's too late. At this point, the write to HCR_EL2 will have already
occurred,

> 
> 
> > If you apply the following change to your guest, does it start
> > behaving?
> > 
> > M.
> > 
> > diff --git a/arch/arm64/include/asm/el2_setup.h b/arch/arm64/include/asm/el2_setup.h
> > index 46033027510cc..392c9f4016f2e 100644
> > --- a/arch/arm64/include/asm/el2_setup.h
> > +++ b/arch/arm64/include/asm/el2_setup.h
> > @@ -34,7 +34,7 @@
> > mrs_s x1, SYS_ID_AA64MMFR4_EL1
> > sbfx x1, x1, #ID_AA64MMFR4_EL1_E2H0_SHIFT, #ID_AA64MMFR4_EL1_E2H0_WIDTH
> > cmp x1, #0
> > - b.ge .LnVHE_\@
> > +// b.ge .LnVHE_\@
> > 
> > orr x0, x0, #HCR_E2H
> > .LnVHE_\@:
> > 
> 
> With this change Guest OS starts the boot process.
> 
> [ 0.000000] CPU features: detected: Virtualization Host Extensions
> [ 6.998696] SMP: Total of 4 processors activated.
> [ 7.257045] CPU: All CPU(s) started at EL2
> [ 185.781062] SVE: maximum available vector length 16 bytes per vector
> [ 186.104159] SVE: default vector length 16 bytes per vector

Great. That confirms my suspicion that we cannot advertise VHE on CPUs
that do not have FEAT_FGT. Oh well. I'll post a patch to forbid VHE
guests on these machines.

> I also observed something interesting.
> When PSCI method is set to HVC, I get a kernel panic.
> This wasn’t a problem in all other test cases.

Well, that's completely expected. HVC is handled by EL2. You run at
EL2. You end-up calling yourself, which you don't really expect.

SMC is the way for EL2 guests, as if they were on bare-metal.

> On a side note, thank you for giving me some pointers.
> With HAS_EL2_E2H the original Guest OS boots fine in nVHE.
> 
> [    0.204744] CPU: All CPU(s) started at EL2
> [   18.483737] SVE: maximum available vector length 16 bytes per vector
> [   18.500959] SVE: default vector length 16 bytes per vector
> [   25.999475] kvm [1]: Hyp nVHE mode initialized successfully

Yup. I'll kill the ability to run VHE guests on crappy old HW then.

Thanks for giving it a go!

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: KVM NV + SVE host OS warning

[Jan Kotas]

> On 8 Oct 2025, at 17:22, Marc Zyngier <maz@kernel.org> wrote:
> 
> EXTERNAL MAIL
> 
> 
> On Wed, 08 Oct 2025 14:43:29 +0100,
> Jan Kotas <jank@cadence.com> wrote:
>>>> 
>>>> Is it possible to trap/emulate HCR_EL2, and make sure E2H is always 1,
>>>> in cases when only KVM_ARM_VCPU_HAS_EL2 is set?
>>>> Would that help?
>>> 
>>> There are no traps for HCR_EL2.
>>> 
>>>> As a test, I tried forcing this bit before running VCPUs, but
>>>> reading it via MRS got a value without it being set.
>>> 
>>> Where did you read that from?
>> 
>> I checked the X1 register, after: mrs x1, hcr_el2 in __check_hvhe.
> 
> That's too late. At this point, the write to HCR_EL2 will have already
> occurred,

I see, thanks.

>> 
>> 
>>> If you apply the following change to your guest, does it start
>>> behaving?
>>> 
>>> M.
>>> 
>>> diff --git a/arch/arm64/include/asm/el2_setup.h b/arch/arm64/include/asm/el2_setup.h
>>> index 46033027510cc..392c9f4016f2e 100644
>>> --- a/arch/arm64/include/asm/el2_setup.h
>>> +++ b/arch/arm64/include/asm/el2_setup.h
>>> @@ -34,7 +34,7 @@
>>> mrs_s x1, SYS_ID_AA64MMFR4_EL1
>>> sbfx x1, x1, #ID_AA64MMFR4_EL1_E2H0_SHIFT, #ID_AA64MMFR4_EL1_E2H0_WIDTH
>>> cmp x1, #0
>>> - b.ge .LnVHE_\@
>>> +// b.ge .LnVHE_\@
>>> 
>>> orr x0, x0, #HCR_E2H
>>> .LnVHE_\@:
>>> 
>> 
>> With this change Guest OS starts the boot process.
>> 
>> [ 0.000000] CPU features: detected: Virtualization Host Extensions
>> [ 6.998696] SMP: Total of 4 processors activated.
>> [ 7.257045] CPU: All CPU(s) started at EL2
>> [ 185.781062] SVE: maximum available vector length 16 bytes per vector
>> [ 186.104159] SVE: default vector length 16 bytes per vector
> 
> Great. That confirms my suspicion that we cannot advertise VHE on CPUs
> that do not have FEAT_FGT. Oh well. I'll post a patch to forbid VHE
> guests on these machines.
OK. Makes sense. I’ll test the patch once it’s posted.

> 
>> I also observed something interesting.
>> When PSCI method is set to HVC, I get a kernel panic.
>> This wasn’t a problem in all other test cases.
> 
> Well, that's completely expected. HVC is handled by EL2. You run at
> EL2. You end-up calling yourself, which you don't really expect.
> 
> SMC is the way for EL2 guests, as if they were on bare-metal.
> 
>> On a side note, thank you for giving me some pointers.
>> With HAS_EL2_E2H the original Guest OS boots fine in nVHE.
>> 
>> [    0.204744] CPU: All CPU(s) started at EL2
>> [   18.483737] SVE: maximum available vector length 16 bytes per vector
>> [   18.500959] SVE: default vector length 16 bytes per vector
>> [   25.999475] kvm [1]: Hyp nVHE mode initialized successfully
> 
> Yup. I'll kill the ability to run VHE guests on crappy old HW then.
> 
> Thanks for giving it a go!
> 
> M.

Thank you for these issues!

Regards,
Jan

Re: KVM NV + SVE host OS warning

[Marc Zyngier]

On Wed, 08 Oct 2025 16:22:31 +0100,
Marc Zyngier <maz@kernel.org> wrote:
> 
> On Wed, 08 Oct 2025 14:43:29 +0100,
> Jan Kotas <jank@cadence.com> wrote:
> > >>
> > >> Is it possible to trap/emulate HCR_EL2, and make sure E2H is always 1,
> > >> in cases when only KVM_ARM_VCPU_HAS_EL2 is set?
> > >> Would that help?
> > > 
> > > There are no traps for HCR_EL2.
> > > 
> > >> As a test, I tried forcing this bit before running VCPUs, but
> > >> reading it via MRS got a value without it being set.
> > > 
> > > Where did you read that from?
> > 
> > I checked the X1 register, after: mrs x1, hcr_el2 in __check_hvhe.
> 
> That's too late. At this point, the write to HCR_EL2 will have already
> occurred,
> 
> > 
> > 
> > > If you apply the following change to your guest, does it start
> > > behaving?
> > > 
> > > M.
> > > 
> > > diff --git a/arch/arm64/include/asm/el2_setup.h b/arch/arm64/include/asm/el2_setup.h
> > > index 46033027510cc..392c9f4016f2e 100644
> > > --- a/arch/arm64/include/asm/el2_setup.h
> > > +++ b/arch/arm64/include/asm/el2_setup.h
> > > @@ -34,7 +34,7 @@
> > > mrs_s x1, SYS_ID_AA64MMFR4_EL1
> > > sbfx x1, x1, #ID_AA64MMFR4_EL1_E2H0_SHIFT, #ID_AA64MMFR4_EL1_E2H0_WIDTH
> > > cmp x1, #0
> > > - b.ge .LnVHE_\@
> > > +// b.ge .LnVHE_\@
> > > 
> > > orr x0, x0, #HCR_E2H
> > > .LnVHE_\@:
> > > 
> > 
> > With this change Guest OS starts the boot process.
> > 
> > [ 0.000000] CPU features: detected: Virtualization Host Extensions
> > [ 6.998696] SMP: Total of 4 processors activated.
> > [ 7.257045] CPU: All CPU(s) started at EL2
> > [ 185.781062] SVE: maximum available vector length 16 bytes per vector
> > [ 186.104159] SVE: default vector length 16 bytes per vector
> 
> Great. That confirms my suspicion that we cannot advertise VHE on CPUs
> that do not have FEAT_FGT. Oh well. I'll post a patch to forbid VHE
> guests on these machines.

As an alternative, and in an effort to keep at least VHE guests
running on V2 (and other machines suffering from the same situation),
I have posted a potential workaround at [1] for the guest to reliably
detect the situation.

Could you please test it with your setup and report whether this
allows you to boot a VHE guest? You will need to patch the guest.

Note that this still won't let you recursively virtualise EL2 (the HW
can do it, but KVM in the guest doesn't know it), but that's better
than nothing.

Thanks,

	M.

[1] https://lore.kernel.org/r/20251009121239.29370-1-maz@kernel.org

-- 
Without deviation from the norm, progress is not possible.

Re: KVM NV + SVE host OS warning

[Jan Kotas]

> On 9 Oct 2025, at 14:22, Marc Zyngier <maz@kernel.org> wrote:
> 
> EXTERNAL MAIL
> 
> 
> On Wed, 08 Oct 2025 16:22:31 +0100,
> Marc Zyngier <maz@kernel.org> wrote:
>> 
>> On Wed, 08 Oct 2025 14:43:29 +0100,
>> Jan Kotas <jank@cadence.com> wrote:
>>>>> 
>>>>> Is it possible to trap/emulate HCR_EL2, and make sure E2H is always 1,
>>>>> in cases when only KVM_ARM_VCPU_HAS_EL2 is set?
>>>>> Would that help?
>>>> 
>>>> There are no traps for HCR_EL2.
>>>> 
>>>>> As a test, I tried forcing this bit before running VCPUs, but
>>>>> reading it via MRS got a value without it being set.
>>>> 
>>>> Where did you read that from?
>>> 
>>> I checked the X1 register, after: mrs x1, hcr_el2 in __check_hvhe.
>> 
>> That's too late. At this point, the write to HCR_EL2 will have already
>> occurred,
>> 
>>> 
>>> 
>>>> If you apply the following change to your guest, does it start
>>>> behaving?
>>>> 
>>>> M.
>>>> 
>>>> diff --git a/arch/arm64/include/asm/el2_setup.h b/arch/arm64/include/asm/el2_setup.h
>>>> index 46033027510cc..392c9f4016f2e 100644
>>>> --- a/arch/arm64/include/asm/el2_setup.h
>>>> +++ b/arch/arm64/include/asm/el2_setup.h
>>>> @@ -34,7 +34,7 @@
>>>> mrs_s x1, SYS_ID_AA64MMFR4_EL1
>>>> sbfx x1, x1, #ID_AA64MMFR4_EL1_E2H0_SHIFT, #ID_AA64MMFR4_EL1_E2H0_WIDTH
>>>> cmp x1, #0
>>>> - b.ge .LnVHE_\@
>>>> +// b.ge .LnVHE_\@
>>>> 
>>>> orr x0, x0, #HCR_E2H
>>>> .LnVHE_\@:
>>>> 
>>> 
>>> With this change Guest OS starts the boot process.
>>> 
>>> [ 0.000000] CPU features: detected: Virtualization Host Extensions
>>> [ 6.998696] SMP: Total of 4 processors activated.
>>> [ 7.257045] CPU: All CPU(s) started at EL2
>>> [ 185.781062] SVE: maximum available vector length 16 bytes per vector
>>> [ 186.104159] SVE: default vector length 16 bytes per vector
>> 
>> Great. That confirms my suspicion that we cannot advertise VHE on CPUs
>> that do not have FEAT_FGT. Oh well. I'll post a patch to forbid VHE
>> guests on these machines.
> 
> As an alternative, and in an effort to keep at least VHE guests
> running on V2 (and other machines suffering from the same situation),
> I have posted a potential workaround at [1] for the guest to reliably
> detect the situation.
> 
> Could you please test it with your setup and report whether this
> allows you to boot a VHE guest? You will need to patch the guest.
> 
> Note that this still won't let you recursively virtualise EL2 (the HW
> can do it, but KVM in the guest doesn't know it), but that's better
> than nothing.

Thanks Marc,
I tried it in my Guest, on top of 6.17.0. It seems to be working.


Without HAS_EL2_E2H0:
CPU features: detected: Virtualization Host Extensions
CPU: All CPU(s) started at EL2
kvm [1]: VHE mode initialized successfully


With HAS_EL2_E2H0:
CPU: All CPU(s) started at EL2
kvm [1]: Hyp nVHE mode initialized successfully

In both cases SVE is also initialized properly.

Thanks,
Jan

> 
> Thanks,
> 
> M.
> 
> [1] https://urldefense.com/v3/__https://lore.kernel.org/r/20251009121239.29370-1-maz@kernel.org__;!!EHscmS1ygiU1lA!CSDHrEzxKt79BarWMdScZJ0yCSntOw-6rDX_3Xz6ic8fKw0huyf-Gu3y6C84cPTu-4Zu1KUk$
> 
> -- 
> Without deviation from the norm, progress is not possible.

Re: KVM NV + SVE host OS warning

[Marc Zyngier]

On Thu, 09 Oct 2025 15:41:28 +0100,
Jan Kotas <jank@cadence.com> wrote:
> 
> > On 9 Oct 2025, at 14:22, Marc Zyngier <maz@kernel.org> wrote:
> > 
> > As an alternative, and in an effort to keep at least VHE guests
> > running on V2 (and other machines suffering from the same situation),
> > I have posted a potential workaround at [1] for the guest to reliably
> > detect the situation.
> > 
> > Could you please test it with your setup and report whether this
> > allows you to boot a VHE guest? You will need to patch the guest.
> > 
> > Note that this still won't let you recursively virtualise EL2 (the HW
> > can do it, but KVM in the guest doesn't know it), but that's better
> > than nothing.
> 
> Thanks Marc,
> I tried it in my Guest, on top of 6.17.0. It seems to be working.
> 
> 
> Without HAS_EL2_E2H0:
> CPU features: detected: Virtualization Host Extensions
> CPU: All CPU(s) started at EL2
> kvm [1]: VHE mode initialized successfully
> 
> 
> With HAS_EL2_E2H0:
> CPU: All CPU(s) started at EL2
> kvm [1]: Hyp nVHE mode initialized successfully
> 
> In both cases SVE is also initialized properly.

Great. Would you mind replying to the patch with a Tested-by: tag?

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.