Re: KVM NV + SVE host OS warning

[Marc Zyngier <maz@kernel.org>]

[+Oliver for the SVE stuff]

Hi Jan,

On Thu, 25 Sep 2025 15:02:20 +0100,
Jan Kotas <jank@cadence.com> wrote:
> Hello,
> 
> I’m experimenting with Nested Virtualization.
> I use Linux kernel 6.16.3 from Debian backports running on Neoverse-V2.
> 
> When I try to boot a GuestOS, it hangs,
> and I can see a warning in Host's dmesg:
> 
> [52417.934951] ------------[ cut here ]------------
> [52417.934990] WARNING: CPU: 120 PID: 44115 at arch/arm64/include/asm/kvm_emulate.h:553 perform_access+0x14c/0x160
> [52417.935087] Modules linked in: nfsv3 nfs netfs snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore rfkill qrtr binfmt_misc nls_ascii nls_cp437 vfat fat aes_ce_blk aes_ce_cipher polyval_ce ghash_ce gf128mul sha3_ce sha512_ce sha1_ce acpi_ipmi dax_hmem arm_smccc_trng cxl_acpi ipmi_ssif i2c_smbus arm_spe_pmu arm_smmuv3_pmu coresight_trbe spi_nor mtd ipmi_devintf ipmi_msghandler coresight_stm coresight_tmc coresight_funnel stm_core coresight_etm4x coresight joydev evdev cppc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc efi_pstore configfs nfnetlink efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_cryptoapi hid_generic usbhid hid rndis_host cdc_ether usbnet mii dm_mod ast ixgbe drm_shmem_helper xhci_pci_renesas i2c_algo_bit xfrm_algo xhci_pci drm_client_lib mdio_devres drm_kms_helper xhci_hcd of_mdio nvme fixed_phy drm fwnode_mdio usbcore nvme_core libphy sbsa_gwdt mdio_bus nvme_keyring usb_common nvme_auth mdio i2c_tegra
> [52417.935818] CPU: 120 UID: 254353 PID: 44115 Comm: kvm_vcpu0 Tainted: G        W           6.16.3+deb13-arm64 #1 PREEMPTLAZY  Debian 6.16.3-1~bpo13+1
> [52417.935855] Tainted: [W]=WARN
> [52417.935866] Hardware name: Supermicro ARS-121L-DNR/G1SMH, BIOS 2.1 04/17/2025

Fancy HW (/me goes selling a kidney...)

> [52417.935879] pstate: 62400009 (nZCv daif +PAN -UAO +TCO -DIT -SSBS BTYPE=--)
> [52417.935906] pc : perform_access+0x14c/0x160
> [52417.935933] lr : perform_access+0x4c/0x160
> [52417.935956] sp : ffff8000f30db850
> [52417.935967] x29: ffff8000f30db850 x28: ffff000097245000 x27: 0000000000000000
> [52417.936004] x26: 0000000000000000 x25: 0000000000000000 x24: ffff10002c701c28
> [52417.936036] x23: 0000000000000000 x22: ffff000097245000 x21: ffff8000f30db8a0
> [52417.936065] x20: ffffdbf14a19eac0 x19: ffff10002c701be0 x18: 0000000000000014
> [52417.936095] x17: 000000040044ffff x16: 00100075b5503510 x15: 0000000000000000
> [52417.936127] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
> [52417.936157] x11: 0000000000001348 x10: 00000000000013b0 x9 : ffffdbf1491608b4
> [52417.936188] x8 : 0000000000000001 x7 : 0000000000000000 x6 : 00000000000fffff
> [52417.936218] x5 : 000000000036cb76 x4 : ffff10027148f7c0 x3 : ffffdbf14915f04c
> [52417.936249] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000009
> [52417.936280] Call trace:
> [52417.936291]  perform_access+0x14c/0x160 (P)
> [52417.936325]  kvm_handle_sys_reg+0x12c/0x2a0
> [52417.936366]  handle_exit+0x68/0x190
> [52417.936408]  kvm_arch_vcpu_ioctl_run+0x2d8/0xa10
> [52417.936436]  kvm_vcpu_ioctl+0x1a8/0xb18
> [52417.936459]  __arm64_sys_ioctl+0xb4/0x120
> [52417.936510]  invoke_syscall+0x6c/0x100
> [52417.936547]  el0_svc_common.constprop.0+0x48/0xf0
> [52417.936581]  do_el0_svc+0x24/0x38
> [52417.936613]  el0_svc+0xd4/0x190
> [52417.936643]  el0t_64_sync_handler+0x10c/0x138
> [52417.936667]  el0t_64_sync+0x198/0x1a0
> [52417.936690] ---[ end trace 0000000000000000 ]---
> 
> 
> The tracing revealed, it may be caused by a ZCR_EL2 write:
> [109] ..... 52068.375927: kvm_sys_access: PC: 806608b8 SYS_ZCR_EL2 (3,4,1,2,0) write
> 
> The instruction from ELR also matches: msr zcr_el2, x1
> 
> The reason might be CPTR_EL2, its value just before this instruction is executed, is 0.
> However before the start of the VM execution, it has 0x22ff.
> 
> I can see accesses to this register in the trace log as well, just before ZCR_EL2 is accessed.
> [109] ..... 52068.375922: kvm_sys_access: PC: 806608a4 SYS_CPTR_EL2 (3,4,1,1,2) read
> [109] ..... 52068.375925: kvm_sys_access: PC: 806608ac SYS_CPTR_EL2 (3,4,1,1,2) write
> 
> I’m running Linux 6.16.0 as my Guest.
> Nested Virtualization works fine with SVE disabled, so does SVE without NV.
> Could it be caused by a bug in userspace hypervisor code?

Unlikely. The warning indicates that we are incrementing PC while
there is a pending exception. Having both at the same time is a very
bad bug -- hence the warning.

Looking at the code with the above in mind, something immediately
jumps at me. Can you try the following (against 6.17, but you'll
surely be able to apply it against 6.16):

diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 91053aa832d08..a07ad5c92583d 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -2705,7 +2705,7 @@ static bool access_zcr_el2(struct kvm_vcpu *vcpu,
 
 	if (guest_hyp_sve_traps_enabled(vcpu)) {
 		kvm_inject_nested_sve_trap(vcpu);
-		return true;
+		return false;
 	}
 
 	if (!p->is_write) {

This should make the warning go away -- not sure about anything else.
Note that I do not have access to an NV+SVE capable machine, so you're
are basically on your own, unless Oliver has a box he can reproduce
this on.

I would also recommend to update to 6.17 -- 6.16 was the first release
with NV, and while it may work, it will also have a lot of ugly bugs.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.