[PATCH kvmtool 0/5] Fix diagnostics and capability probes for protected VMs

[PATCH kvmtool 0/5] Fix diagnostics and capability probes for protected VMs

[Fuad Tabba]

Hi folks,

These are the kvmtool changes that go with the pKVM series confining a
protected VM's vCPU state to EL2 [1].

That series stops the host reading a protected vCPU's registers once it
has run, and stops it setting the PVTIME attribute. kvmtool dies on
both. Its arm64 register-dump helpers call die() when KVM_GET_ONE_REG
fails, which turns a diagnostic into a VMM abort.
kvm_cpu__setup_pvtime() runs into the new -EPERM and fails vCPU init.

The PVTIME failure is a probe on the wrong fd. kvmtool asks the global
/dev/kvm fd whether steal-time is available, and that fd advertises it
for a protected VM, which cannot use it.
The last patch fixes the same wrong-fd probe for the counter offset,
where "lkvm run --protected --counter-offset" already fails on Linux
7.3-rc1, without the kernel series.
The same conversion went in for the vCPU feature probes in July [2].

Patches 2 and 3 are not specific to protected VMs. They fix the
exit-reason table on the same panic path as patch 1. The lookup ran off
the end of it, and the table had stopped naming anything KVM added after
KVM_EXIT_INTERNAL_ERROR, which on arm64 includes KVM_EXIT_SYSTEM_EVENT
and KVM_EXIT_ARM_NISV.

Patch 1's path needs a protected VM that is already running: "lkvm debug
-d" against one kills the VMM without this series, and prints the errno
with it.

Based on kvmtool master (f67bc0b).

Cheers,
/fuad

[1] https://lore.kernel.org/all/20260831163421.272420-1-fuad.tabba@linux.dev/
[2] https://lore.kernel.org/all/20260714110329.12113-1-fuad.tabba@linux.dev/

Fuad Tabba (5):
  arm64: Do not abort on register-dump failures
  kvm: Bound-check the exit-reason string lookup
  kvm: Name every exit reason the UAPI header defines
  arm64: Query steal-time support on the VM fd
  arm64: Query counter-offset support on the VM fd

 arm64/kvm-cpu.c   | 36 ++++++++++++++++++++++++------------
 arm64/kvm.c       |  2 +-
 arm64/pvtime.c    |  4 ++--
 builtin-run.c     |  2 +-
 include/kvm/kvm.h |  2 +-
 kvm.c             | 37 ++++++++++++++++++++++++++++++++++---
 6 files changed, 63 insertions(+), 20 deletions(-)


base-commit: f67bc0bdae9433a9cfd05e65ea2c1bb6102566d9
-- 
2.39.5

[PATCH kvmtool 1/5] arm64: Do not abort on register-dump failures

[Fuad Tabba]

kvm_cpu__show_registers() and kvm_cpu__show_code() read the vCPU's core
registers with KVM_GET_ONE_REG, and die() if the ioctl fails. Both are
diagnostics. They run from the KVM_EXIT_DEBUG case in kvm_cpu__start(),
from the "lkvm debug -d" dump path in handle_sigusr1(), and from the
panic dump in kvm_cpu_thread().

Once a protected vCPU has run, its registers belong to the guest and
KVM_GET_ONE_REG returns -EPERM. Dying on that turns a diagnostic into a
VMM abort.

Report that the register state is unavailable, with the errno, and
return instead of dying.

Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---
 arm64/kvm-cpu.c | 36 ++++++++++++++++++++++++------------
 1 file changed, 24 insertions(+), 12 deletions(-)

diff --git a/arm64/kvm-cpu.c b/arm64/kvm-cpu.c
index 3aa7684..c58d61f 100644
--- a/arm64/kvm-cpu.c
+++ b/arm64/kvm-cpu.c
@@ -476,15 +476,19 @@ void kvm_cpu__show_code(struct kvm_cpu *vcpu)
 
 	dprintf(debug_fd, "\n*pc:\n");
 	reg.id = ARM64_CORE_REG(regs.pc);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (show_code @ PC)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (pc): %s", strerror(errno));
+		return;
+	}
 
 	kvm__dump_mem(vcpu->kvm, data, 32, debug_fd);
 
 	dprintf(debug_fd, "\n*lr:\n");
 	reg.id = ARM64_CORE_REG(regs.regs[30]);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (show_code @ LR)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (lr): %s", strerror(errno));
+		return;
+	}
 
 	kvm__dump_mem(vcpu->kvm, data, 32, debug_fd);
 }
@@ -499,23 +503,31 @@ void kvm_cpu__show_registers(struct kvm_cpu *vcpu)
 	dprintf(debug_fd, "\n Registers:\n");
 
 	reg.id		= ARM64_CORE_REG(regs.pc);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (pc)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (pc): %s", strerror(errno));
+		return;
+	}
 	dprintf(debug_fd, " PC:    0x%lx\n", data);
 
 	reg.id		= ARM64_CORE_REG(regs.pstate);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (pstate)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (pstate): %s", strerror(errno));
+		return;
+	}
 	dprintf(debug_fd, " PSTATE:    0x%lx\n", data);
 
 	reg.id		= ARM64_CORE_REG(sp_el1);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (sp_el1)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (sp_el1): %s", strerror(errno));
+		return;
+	}
 	dprintf(debug_fd, " SP_EL1:    0x%lx\n", data);
 
 	reg.id		= ARM64_CORE_REG(regs.regs[30]);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (lr)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (lr): %s", strerror(errno));
+		return;
+	}
 	dprintf(debug_fd, " LR:    0x%lx\n", data);
 }
 
-- 
2.39.5

[PATCH kvmtool 2/5] kvm: Bound-check the exit-reason string lookup

[Fuad Tabba]

kvm_cpu_thread()'s panic path uses the kernel's exit_reason to index
kvm_exit_reasons[] without checking it against the array's size. The
table does not cover every exit reason KVM can return, so a reason past
its last entry reads out of bounds. KVM_EXIT_SYSTEM_EVENT and
KVM_EXIT_ARM_NISV are both past it.

Move the table behind kvm__exit_reason_str(), which checks the index
first. The array now has a single caller inside kvm.c, so make it
static const.

Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---
 builtin-run.c     |  2 +-
 include/kvm/kvm.h |  2 +-
 kvm.c             | 10 +++++++++-
 3 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/builtin-run.c b/builtin-run.c
index 81f255f..f36d3e2 100644
--- a/builtin-run.c
+++ b/builtin-run.c
@@ -298,7 +298,7 @@ static void *kvm_cpu_thread(void *arg)
 panic_kvm:
 	pr_err("KVM exit reason: %u (\"%s\")",
 		current_kvm_cpu->kvm_run->exit_reason,
-		kvm_exit_reasons[current_kvm_cpu->kvm_run->exit_reason]);
+		kvm__exit_reason_str(current_kvm_cpu->kvm_run->exit_reason));
 
 	if (current_kvm_cpu->kvm_run->exit_reason == KVM_EXIT_UNKNOWN) {
 		pr_err("KVM exit code: %llu",
diff --git a/include/kvm/kvm.h b/include/kvm/kvm.h
index a9376b6..8619070 100644
--- a/include/kvm/kvm.h
+++ b/include/kvm/kvm.h
@@ -255,7 +255,7 @@ int kvm__for_each_mem_bank(struct kvm *kvm, enum kvm_mem_type type,
  */
 void kvm__dump_mem(struct kvm *kvm, unsigned long addr, unsigned long size, int debug_fd);
 
-extern const char *kvm_exit_reasons[];
+const char *kvm__exit_reason_str(__u32 exit_reason);
 
 static inline bool host_ptr_in_ram(struct kvm *kvm, void *p)
 {
diff --git a/kvm.c b/kvm.c
index 96583f9..b416f6f 100644
--- a/kvm.c
+++ b/kvm.c
@@ -33,7 +33,7 @@
 
 #define DEFINE_KVM_EXIT_REASON(reason) [reason] = #reason
 
-const char *kvm_exit_reasons[] = {
+static const char * const kvm_exit_reasons[] = {
 	DEFINE_KVM_EXIT_REASON(KVM_EXIT_UNKNOWN),
 	DEFINE_KVM_EXIT_REASON(KVM_EXIT_EXCEPTION),
 	DEFINE_KVM_EXIT_REASON(KVM_EXIT_IO),
@@ -57,6 +57,14 @@ const char *kvm_exit_reasons[] = {
 #endif
 };
 
+const char *kvm__exit_reason_str(__u32 exit_reason)
+{
+	if (exit_reason >= ARRAY_SIZE(kvm_exit_reasons))
+		return "UNKNOWN";
+
+	return kvm_exit_reasons[exit_reason];
+}
+
 static int pause_event;
 static DEFINE_MUTEX(pause_lock);
 static struct kvm_cpu *pause_req_cpu;
-- 
2.39.5

[PATCH kvmtool 3/5] kvm: Name every exit reason the UAPI header defines

[Fuad Tabba]

kvm_exit_reasons[] stops at KVM_EXIT_INTERNAL_ERROR, so the panic path
prints "UNKNOWN" for every reason KVM has gained since. On arm64 that
covers KVM_EXIT_SYSTEM_EVENT and KVM_EXIT_ARM_NISV, and the newer
KVM_EXIT_MEMORY_FAULT, KVM_EXIT_ARM_SEA and KVM_EXIT_ARM_LDST64B.

Name the rest of the reasons the synced uapi header defines, up to
KVM_EXIT_SNP_REQ_CERTS. KVM_EXIT_PAPR_HCALL also loses its CONFIG_PPC64
guard, which the build never defines, so that entry was missing from
every configuration. The neighbouring PPC reasons are unconditional
already.

Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---
 kvm.c | 27 +++++++++++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)

diff --git a/kvm.c b/kvm.c
index b416f6f..828da64 100644
--- a/kvm.c
+++ b/kvm.c
@@ -52,9 +52,32 @@ static const char * const kvm_exit_reasons[] = {
 	DEFINE_KVM_EXIT_REASON(KVM_EXIT_DCR),
 	DEFINE_KVM_EXIT_REASON(KVM_EXIT_NMI),
 	DEFINE_KVM_EXIT_REASON(KVM_EXIT_INTERNAL_ERROR),
-#ifdef CONFIG_PPC64
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_OSI),
 	DEFINE_KVM_EXIT_REASON(KVM_EXIT_PAPR_HCALL),
-#endif
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_S390_UCONTROL),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_WATCHDOG),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_S390_TSCH),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_EPR),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_SYSTEM_EVENT),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_S390_STSI),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_IOAPIC_EOI),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_HYPERV),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_ARM_NISV),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_X86_RDMSR),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_X86_WRMSR),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_DIRTY_RING_FULL),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_AP_RESET_HOLD),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_X86_BUS_LOCK),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_XEN),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_RISCV_SBI),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_RISCV_CSR),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_NOTIFY),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_LOONGARCH_IOCSR),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_MEMORY_FAULT),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_TDX),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_ARM_SEA),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_ARM_LDST64B),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_SNP_REQ_CERTS),
 };
 
 const char *kvm__exit_reason_str(__u32 exit_reason)
-- 
2.39.5

[PATCH kvmtool 4/5] arm64: Query steal-time support on the VM fd

[Fuad Tabba]

kvm_cpu__setup_pvtime() probes KVM_CAP_STEAL_TIME with
kvm__supports_extension(), which issues KVM_CHECK_EXTENSION on the
global /dev/kvm fd. That reports the host's raw capabilities, unaware of
any per-VM restrictions.

pKVM does not offer steal-time to a protected VM, and reflects that on
the VM fd alone. kvmtool goes on to KVM_HAS_DEVICE_ATTR, the kernel
refuses the PVTIME attribute with -EPERM, and kvm_cpu__arch_init() dies:

  Fatal: Unable to initialise vcpu

Query it on the VM fd via kvm__supports_vm_extension(), as
commit 84464ba0246b ("arm64: Query per-VM capabilities when selecting
vCPU features") did for the vCPU feature probes. kvmtool then takes the
no_pvtime path it already has when steal-time is unsupported.

Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---
 arm64/pvtime.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arm64/pvtime.c b/arm64/pvtime.c
index 2933ac7..839aa8a 100644
--- a/arm64/pvtime.c
+++ b/arm64/pvtime.c
@@ -58,8 +58,8 @@ int kvm_cpu__setup_pvtime(struct kvm_cpu *vcpu)
 	if (kvm_cfg->no_pvtime)
 		return 0;
 
-	has_stolen_time = kvm__supports_extension(vcpu->kvm,
-						  KVM_CAP_STEAL_TIME);
+	has_stolen_time = kvm__supports_vm_extension(vcpu->kvm,
+						     KVM_CAP_STEAL_TIME);
 	if (!has_stolen_time) {
 		kvm_cfg->no_pvtime = true;
 		return 0;
-- 
2.39.5

[PATCH kvmtool 5/5] arm64: Query counter-offset support on the VM fd

[Fuad Tabba]

kvm__arch_set_counter_offset() probes KVM_CAP_COUNTER_OFFSET with
kvm__supports_extension(), which issues KVM_CHECK_EXTENSION on the
global /dev/kvm fd. That reports the host's raw capabilities, unaware of
any per-VM restrictions.

pKVM does not offer the counter offset to a protected VM, and reflects
that on the VM fd alone. kvmtool's own check passes, and the
KVM_ARM_SET_COUNTER_OFFSET that follows is refused by
commit b12b3b04f6ba ("KVM: arm64: Check whether a VM IOCTL is allowed
in pKVM"):

  KVM_ARM_SET_COUNTER_OFFSET: Invalid argument

Query it on the VM fd via kvm__supports_vm_extension(), so
--counter-offset on a protected VM fails with kvmtool's own message:

  Fatal: No support for global counter offset

Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
---
 arm64/kvm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arm64/kvm.c b/arm64/kvm.c
index 36b3284..0bb3535 100644
--- a/arm64/kvm.c
+++ b/arm64/kvm.c
@@ -155,7 +155,7 @@ static void kvm__arch_set_counter_offset(struct kvm *kvm)
 	if (!kvm->cfg.arch.counter_offset)
 		return;
 
-	if (!kvm__supports_extension(kvm, KVM_CAP_COUNTER_OFFSET))
+	if (!kvm__supports_vm_extension(kvm, KVM_CAP_COUNTER_OFFSET))
 		die("No support for global counter offset");
 
 	if (ioctl(kvm->vm_fd, KVM_ARM_SET_COUNTER_OFFSET, &offset))
-- 
2.39.5