[PATCH v2 0/4] KVM: x86: hyper-v: Forward unimplemented hypercalls to userspace

[PATCH v2 0/4] KVM: x86: hyper-v: Forward unimplemented hypercalls to userspace

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

KVM implements a fixed set of Hyper-V hypercalls and fails every other
code with HV_STATUS_INVALID_HYPERCALL_CODE. A VMM that wants to
implement more of the TLFS, such as the VSM hypercalls Windows uses for
Virtualization Based Security, never sees them.

This series adds KVM_CAP_HYPERV_FORWARD_HCALL. As with KVM_CAP_XEN_HVM
and KVM_CAP_EXIT_HYPERCALL, KVM_CHECK_EXTENSION returns a bitmask of
features and KVM_ENABLE_CAP takes the subset to enable, rejecting any
bit the kernel does not report. The first feature,
KVM_HYPERV_HCALL_FORWARD_UNKNOWN, sends the hypercalls KVM does not
implement, and extended hypercalls in fast form, out through the
existing KVM_EXIT_HYPERV_HCALL exit, whose format does not change at
all. Further mask bits will follow for HvCallVtlCall and HvCallVtlReturn
on VMs with planes, so that KVM can handle VTL transitions in the
kernel.

Patch 2 lets userspace cancel the pending completion of a hypercall
that exited to userspace, with a new KVM_VCPUEVENT_CANCEL_HCALL flag on
KVM_SET_VCPU_EVENTS. The guest then re-executes the hypercall, unless
the same ioctl queued an exception to be delivered at it. That is how a
VMM makes a hypercall raise #UD (or #PF with CR2, or anything else the
events machinery can queue), which the TLFS specifies for some invalid
uses, instead of returning a status. It is independent of the rest of
the series: in v1 this was a #UD request flag in the hcall exit itself,
but the KVM_EXIT_VMWARE series [1] wants the same for other exits and
other vectors, and KVM_SET_VCPU_EVENTS already knows how to queue every
exception shape, so v2 adds only the cancel primitive and lets the
existing machinery do the rest. KVM_SET_VCPU_EVENTS has no flag
discovery, so KVM_CHECK_EXTENSION(KVM_CAP_VCPU_EVENTS) becomes a
feature mask (bit 0 keeps truth-value checks working, bit 1 advertises
the cancel). The KVM_VCPUEVENT_ namespace so far only holds VALID_*
field gates; if the action flag wants a different name, I am not
attached to this one.

Patch 3 reports HV_X64_HYPERCALL_XMM_OUTPUT_AVAILABLE, since a VMM
handling a forwarded fast hypercall can write the XMM output itself.
Patch 4 adds the Hyper-V forwarding selftest; the generic cancel error
paths come with patch 2's own selftest.

v1->v2:
 - Drop KVM_HYPERV_HCALL_RAISE_UD, KVM_HYPERV_HCALL_UD and the hcall
   exit's flags field; cancel the completion through
   KVM_SET_VCPU_EVENTS instead (patch 2).
 - Selftests: a generic vcpu_events_cancel_test, and forwarding-test
   legs for cancel+#UD (RIP and RAX untouched), cancel alone (the
   hypercall exits a second time), and EBUSY with nothing cancellable
   pending.

The series is based on v7.4-rc2. Tested on a bare-metal AMD EPYC
(Zen 4) system: vcpu_events_cancel_test and hyperv_forward_hcall pass,
as do hyperv_clock, hyperv_cpuid, hyperv_extended_hypercalls,
hyperv_features, hyperv_ipi, hyperv_tlb_flush and hyperv_svm_test, plus
triple_fault_event_test and nested_exceptions_test for the
KVM_SET_VCPU_EVENTS change. Not yet retested on Intel.

[1] https://lore.kernel.org/all/90467782be23b83fd3e9a6afe2f1d2149eaf9db9.1794374897.git.zack.rusin@broadcom.com/

David Woodhouse (4):
  KVM: x86: hyper-v: Add KVM_CAP_HYPERV_FORWARD_HCALL
  KVM: x86: Let userspace cancel a pending hypercall completion
  KVM: x86: hyper-v: Advertise XMM fast hypercall output
  KVM: selftests: Add Hyper-V hypercall forwarding test

 Documentation/virt/kvm/api.rst                |  73 ++++
 arch/x86/include/asm/kvm_host.h               |   4 +
 arch/x86/include/uapi/asm/kvm.h               |   7 +
 arch/x86/kvm/hyperv.c                         |  14 +-
 arch/x86/kvm/hyperv.h                         |   8 +
 arch/x86/kvm/x86.c                            |  51 ++-
 include/uapi/linux/kvm.h                      |   4 +
 tools/testing/selftests/kvm/Makefile.kvm      |   2 +
 .../selftests/kvm/x86/hyperv_forward_hcall.c  | 337 ++++++++++++++++++
 .../kvm/x86/vcpu_events_cancel_test.c         | 119 +++++++
 10 files changed, 616 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86/hyperv_forward_hcall.c
 create mode 100644 tools/testing/selftests/kvm/x86/vcpu_events_cancel_test.c

-- 
2.50.1

[PATCH v2 1/4] KVM: x86: hyper-v: Add KVM_CAP_HYPERV_FORWARD_HCALL

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

A VMM that implements Hyper-V hypercalls KVM does not know about has
no way to receive them. kvm_hv_hypercall() fails an unknown code with
HV_STATUS_INVALID_HYPERCALL_CODE, and only a fixed set (the synthetic
debugger, memory-based extended hypercalls, HvCallPostMessage and the
HvCallSignalEvent fallback) ever exits to userspace.

Add a VM capability whose KVM_CHECK_EXTENSION value is a bitmask of the
hypercall handling features the kernel implements, as KVM_CAP_XEN_HVM
and KVM_CAP_EXIT_HYPERCALL do. KVM_ENABLE_CAP takes the subset to
enable and rejects any bit the kernel does not report, so a VMM can
probe for exactly the behaviour it relies on.

The first feature, KVM_HYPERV_HCALL_FORWARD_UNKNOWN, sends a hypercall
KVM would have failed as unknown, and an extended hypercall in fast
form, out through the existing KVM_EXIT_HYPERV_HCALL. That exit already
carries the control value and both parameter registers, and XMM state
is reachable through the ordinary vCPU state ioctls, so no new exit
format is needed.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 Documentation/virt/kvm/api.rst  | 49 +++++++++++++++++++++++++++++++++
 arch/x86/include/asm/kvm_host.h |  2 ++
 arch/x86/kvm/hyperv.c           |  7 ++++-
 arch/x86/kvm/hyperv.h           |  8 ++++++
 arch/x86/kvm/x86.c              | 14 ++++++++++
 include/uapi/linux/kvm.h        |  4 +++
 6 files changed, 83 insertions(+), 1 deletion(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index e95c9cda085d..092f6f9ad8a0 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -7312,6 +7312,17 @@ Hyper-V SynIC state change. Notification is used to remap SynIC
 event/message pages and to enable/disable SynIC messages/events processing
 in userspace.
 
+	- KVM_EXIT_HYPERV_HCALL -- a hypercall for userspace to handle.
+
+'input' holds the hypercall control value and 'params' the two parameter
+registers.  Userspace sets 'result' to the hypercall result value before
+resuming the vCPU; KVM then writes it to the guest and advances past the
+hypercall instruction.  HvCallPostMessage, HvCallSignalEvent for a
+connection with no registered eventfd, the synthetic debugger hypercalls
+and memory-based extended hypercalls exit this way;
+KVM_CAP_HYPERV_FORWARD_HCALL (see section 7.48) adds the hypercalls KVM
+does not implement.
+
 	- KVM_EXIT_HYPERV_SYNDBG -- synchronously notify user-space about
 
 Hyper-V Synthetic debugger state change. Notification is used to either update
@@ -9181,6 +9192,44 @@ enabled, cmma can't be enabled anymore and pfmfi and the storage key
 interpretation are disabled. If cmma has already been enabled or the
 hpage_2g module parameter is not set to 1, -EINVAL is returned.
 
+7.48 KVM_CAP_HYPERV_FORWARD_HCALL
+---------------------------------
+
+:Architectures: x86
+:Target: VM
+:Parameters: args[0] is a bitmask of KVM_HYPERV_HCALL_* features to enable
+:Returns: 0 on success, -EINVAL if args[0] contains an unsupported bit
+
+KVM_CHECK_EXTENSION for this capability returns a bitmask of the Hyper-V
+hypercall handling features that KVM implements.  KVM_ENABLE_CAP takes
+the subset that userspace wants active and replaces the previously
+enabled set; it can be called at any time.  A bit that KVM_CHECK_EXTENSION
+does not report is rejected with -EINVAL, so a VMM can enable exactly the
+features it was written for and learn which ones the kernel lacks.
+
+The defined bits are::
+
+  #define KVM_HYPERV_HCALL_FORWARD_UNKNOWN	(1ULL << 0)
+
+KVM_HYPERV_HCALL_FORWARD_UNKNOWN: a hypercall that KVM would otherwise
+fail with HV_STATUS_INVALID_HYPERCALL_CODE exits to userspace with
+KVM_EXIT_HYPERV_HCALL instead, and an extended hypercall in fast form
+exits the same way instead of failing with HV_STATUS_INVALID_PARAMETER.
+The VMM can then implement hypercalls that KVM does not.  The synthetic
+debugger hypercalls (HvCallResetDebugSession, HvCallPostDebugData,
+HvCallRetrieveDebugData) are an exception: while the synthetic debugger
+is not enabled they still fail in KVM, with HV_STATUS_INVALID_PARAMETER
+for the fast form of HvCallPostDebugData and HvCallRetrieveDebugData
+and with HV_STATUS_INVALID_HYPERCALL_CODE otherwise.
+
+For a fast hypercall, 'params' carries RDX and R8 from a 64-bit caller,
+or the EBX:ECX and EDI:ESI pairs from a 32-bit caller.  XMM input and
+output registers of a fast hypercall are read and written with the
+ordinary vCPU state ioctls (KVM_GET_XSAVE/KVM_SET_XSAVE or
+KVM_GET_FPU/KVM_SET_FPU) before the vCPU is resumed.  Hypercalls that
+KVM implements itself, and the CPUID-enforcement checks KVM applies
+before dispatching a hypercall, are unaffected.
+
 8. Other capabilities.
 ======================
 
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 1ee4b127004d..fce46ae26137 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1072,6 +1072,8 @@ struct kvm_hv {
 	struct mutex hv_lock;
 	u64 hv_guest_os_id;
 	u64 hv_hypercall;
+	/* KVM_HYPERV_HCALL_* features enabled by KVM_CAP_HYPERV_FORWARD_HCALL */
+	u64 hcall_features;
 	u64 hv_tsc_page;
 	enum hv_tsc_page_status hv_tsc_page_status;
 
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 5d2e3617c952..e72539f9acc9 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -2692,12 +2692,17 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 		goto hypercall_userspace_exit;
 	}
 	case HV_EXT_CALL_QUERY_CAPABILITIES ... HV_EXT_CALL_MAX:
-		if (unlikely(hc.fast)) {
+		if (unlikely(hc.fast) &&
+		    !kvm_hv_hcall_feature(vcpu->kvm,
+					  KVM_HYPERV_HCALL_FORWARD_UNKNOWN)) {
 			ret = HV_STATUS_INVALID_PARAMETER;
 			break;
 		}
 		goto hypercall_userspace_exit;
 	default:
+		if (kvm_hv_hcall_feature(vcpu->kvm,
+					 KVM_HYPERV_HCALL_FORWARD_UNKNOWN))
+			goto hypercall_userspace_exit;
 		ret = HV_STATUS_INVALID_HYPERCALL_CODE;
 		break;
 	}
diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h
index 622a6553e9ac..20d40261c75b 100644
--- a/arch/x86/kvm/hyperv.h
+++ b/arch/x86/kvm/hyperv.h
@@ -205,6 +205,14 @@ static inline bool kvm_hv_hypercall_enabled(struct kvm_vcpu *vcpu)
 	return vcpu->arch.hyperv_enabled && to_kvm_hv(vcpu->kvm)->hv_guest_os_id;
 }
 
+/* Features reported and accepted by KVM_CAP_HYPERV_FORWARD_HCALL. */
+#define KVM_HYPERV_HCALL_SUPPORTED	KVM_HYPERV_HCALL_FORWARD_UNKNOWN
+
+static inline bool kvm_hv_hcall_feature(struct kvm *kvm, u64 feature)
+{
+	return READ_ONCE(to_kvm_hv(kvm)->hcall_features) & feature;
+}
+
 int kvm_hv_hypercall(struct kvm_vcpu *vcpu);
 
 void kvm_hv_irq_routing_update(struct kvm *kvm);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 26e7beaff8a0..d8de7742a0da 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2305,6 +2305,11 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_EXIT_HYPERCALL:
 		r = KVM_EXIT_HYPERCALL_VALID_MASK;
 		break;
+#ifdef CONFIG_KVM_HYPERV
+	case KVM_CAP_HYPERV_FORWARD_HCALL:
+		r = KVM_HYPERV_HCALL_SUPPORTED;
+		break;
+#endif
 	case KVM_CAP_SET_GUEST_DEBUG2:
 		return KVM_GUESTDBG_VALID_MASK;
 #ifdef CONFIG_KVM_XEN
@@ -4144,6 +4149,15 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		kvm->arch.hypercall_exit_enabled = cap->args[0];
 		r = 0;
 		break;
+#ifdef CONFIG_KVM_HYPERV
+	case KVM_CAP_HYPERV_FORWARD_HCALL:
+		r = -EINVAL;
+		if (cap->args[0] & ~KVM_HYPERV_HCALL_SUPPORTED)
+			break;
+		WRITE_ONCE(to_kvm_hv(kvm)->hcall_features, cap->args[0]);
+		r = 0;
+		break;
+#endif
 	case KVM_CAP_EXIT_ON_EMULATION_FAILURE:
 		r = -EINVAL;
 		if (cap->args[0] & ~1)
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 8dff2fc1972e..79f128958df5 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -126,6 +126,9 @@ struct kvm_hyperv_exit {
 	} u;
 };
 
+/* KVM_CAP_HYPERV_FORWARD_HCALL feature bits */
+#define KVM_HYPERV_HCALL_FORWARD_UNKNOWN	(1ULL << 0)
+
 struct kvm_xen_exit {
 #define KVM_EXIT_XEN_HCALL          1
 	__u32 type;
@@ -1000,6 +1003,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_PPC_COMPAT_CAPS 250
 #define KVM_CAP_ARM_PMU_V3_STRICT 251
 #define KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES 252
+#define KVM_CAP_HYPERV_FORWARD_HCALL 255
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
-- 
2.50.1

[PATCH v2 2/4] KVM: x86: Let userspace cancel a pending hypercall completion

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

A VMM implementing Hyper-V hypercalls behind KVM_EXIT_HYPERV_HCALL can
only complete them: write a status, advance RIP.  The TLFS specifies
some hypercalls to raise an exception at the hypercall instruction
instead, and the exit has no way to say that.  The KVM_EXIT_VMWARE
series has the same problem for its backdoor exits, so solve it without
touching the exit format at all: cancel the pending completion, and
queue the exception through the machinery KVM_SET_VCPU_EVENTS already
has.

KVM_VCPUEVENT_CANCEL_HCALL on KVM_SET_VCPU_EVENTS drops the pending
completion: no result write, no RIP advance, so the guest re-executes
the hypercall on its next entry unless the same call queued an
exception to be delivered there.  Hyper-V hypercalls are restartable by
design, so both outcomes are architecturally sound.  If nothing
cancellable is pending - nothing at all, or a completion KVM armed for
its own purposes, like the MMIO and PIO emulation continuations - the
ioctl fails with EBUSY and changes nothing.  Only the Hyper-V hypercall
userspace exits are cancellable.

Only the ioctl takes the flag.  KVM_GET_VCPU_EVENTS never reports it, so
no saved or synced copy of the events carries it, and cancelling from the
events KVM_RUN syncs from kvm_run (KVM_SYNC_X86_EVENTS) would have no
use: userspace is already re-entering the vCPU.  sync_regs() rejects the
flag outright with EINVAL, so EBUSY keeps one meaning and only the ioctl
returns it.

KVM_SET_VCPU_EVENTS offers no way to probe for a new flag, so
KVM_CHECK_EXTENSION(KVM_CAP_VCPU_EVENTS) grows from 1 into a feature
mask: KVM_VCPU_EVENTS_BASE keeps existing truth-value checks working,
and KVM_VCPU_EVENTS_CANCEL_HCALL advertises the new flag.

The new selftest covers the paths that need no Hyper-V support: the
capability mask, EBUSY with nothing pending and with a PIO emulation
continuation pending (which must then still retire the instruction),
EINVAL for an undefined flags bit, and EINVAL from KVM_RUN for the flag
in the synced events.  The positive paths need a
hypercall exit and come with the Hyper-V forwarding test.

Link: https://lore.kernel.org/all/876c4f4fc442116c9bff8abd8c34f99fcfa65531.camel@infradead.org/
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 Documentation/virt/kvm/api.rst                |  24 ++++
 arch/x86/include/asm/kvm_host.h               |   2 +
 arch/x86/include/uapi/asm/kvm.h               |   7 ++
 arch/x86/kvm/hyperv.c                         |   6 +
 arch/x86/kvm/x86.c                            |  37 +++++-
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../kvm/x86/vcpu_events_cancel_test.c         | 119 ++++++++++++++++++
 7 files changed, 194 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86/vcpu_events_cancel_test.c

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 092f6f9ad8a0..cb45bdf5f1ee 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -1302,6 +1302,27 @@ If KVM_CAP_X86_TRIPLE_FAULT_EVENT is enabled, KVM_VCPUEVENT_VALID_TRIPLE_FAULT
 can be set in flags field to signal that the triple_fault field contains
 a valid state and shall be written into the VCPU.
 
+On x86, KVM_CHECK_EXTENSION(KVM_CAP_VCPU_EVENTS) returns a feature mask:
+KVM_VCPU_EVENTS_BASE for the two ioctls themselves, and
+KVM_VCPU_EVENTS_CANCEL_HCALL when KVM_SET_VCPU_EVENTS accepts
+KVM_VCPUEVENT_CANCEL_HCALL in the flags field.
+
+KVM_VCPUEVENT_CANCEL_HCALL cancels a pending, cancellable hypercall
+completion, i.e. a KVM_EXIT_HYPERV_HCALL exit that userspace is holding.
+KVM forgets the completion: the hypercall's result is never written and
+its instruction is never retired, so the vCPU re-executes the hypercall
+instruction on its next entry unless the same call queued an exception
+to be delivered there instead.  Hyper-V hypercalls are restartable by
+design, so either is architecturally sound; this is how a VMM makes a
+hypercall fault (e.g. #UD, or #PF with the address in
+exception_payload) rather than return a status.  If no completion is
+pending, or the pending completion is not cancellable (for example an
+MMIO or PIO emulation continuation), the ioctl fails with EBUSY and
+changes nothing.  Only the KVM_SET_VCPU_EVENTS ioctl takes the flag:
+KVM_GET_VCPU_EVENTS never reports it, and in the events that
+KVM_SYNC_X86_EVENTS syncs through kvm_run it makes KVM_RUN fail with
+EINVAL.
+
 ARM64:
 ^^^^^^
 
@@ -7323,6 +7344,9 @@ and memory-based extended hypercalls exit this way;
 KVM_CAP_HYPERV_FORWARD_HCALL (see section 7.48) adds the hypercalls KVM
 does not implement.
 
+Instead of completing the hypercall, userspace may cancel it with
+KVM_VCPUEVENT_CANCEL_HCALL on KVM_SET_VCPU_EVENTS (see section 4.32).
+
 	- KVM_EXIT_HYPERV_SYNDBG -- synchronously notify user-space about
 
 Hyper-V Synthetic debugger state change. Notification is used to either update
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index fce46ae26137..7e37cbadc6d4 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -856,6 +856,8 @@ struct kvm_vcpu_arch {
 	struct x86_emulate_ctxt *emulate_ctxt;
 	bool emulate_regs_need_sync_to_vcpu;
 	bool emulate_regs_need_sync_from_vcpu;
+	/* KVM_SET_VCPU_EVENTS may cancel the pending completion. */
+	bool cui_cancellable;
 	int (*complete_userspace_io)(struct kvm_vcpu *vcpu);
 	unsigned long cui_linear_rip;
 	int cui_rdmsr_imm_reg;
diff --git a/arch/x86/include/uapi/asm/kvm.h b/arch/x86/include/uapi/asm/kvm.h
index 00ce265391e1..a886830f663c 100644
--- a/arch/x86/include/uapi/asm/kvm.h
+++ b/arch/x86/include/uapi/asm/kvm.h
@@ -330,6 +330,13 @@ struct kvm_reinject_control {
 #define KVM_VCPUEVENT_VALID_PAYLOAD	0x00000010
 #define KVM_VCPUEVENT_VALID_TRIPLE_FAULT	0x00000020
 
+/* Cancels a pending, cancellable hypercall completion on KVM_SET_VCPU_EVENTS */
+#define KVM_VCPUEVENT_CANCEL_HCALL	0x00000040
+
+/* KVM_CHECK_EXTENSION(KVM_CAP_VCPU_EVENTS) feature bits */
+#define KVM_VCPU_EVENTS_BASE		0x00000001
+#define KVM_VCPU_EVENTS_CANCEL_HCALL	0x00000002
+
 /* Interrupt shadow states */
 #define KVM_X86_SHADOW_INT_MOV_SS	0x01
 #define KVM_X86_SHADOW_INT_STI		0x02
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index e72539f9acc9..858ee0f374e5 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -2717,6 +2717,12 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 	vcpu->run->hyperv.u.hcall.params[0] = hc.ingpa;
 	vcpu->run->hyperv.u.hcall.params[1] = hc.outgpa;
 	vcpu->arch.complete_userspace_io = kvm_hv_hypercall_complete_userspace;
+	/*
+	 * Userspace may revoke this completion through KVM_SET_VCPU_EVENTS
+	 * instead of completing it; the guest then re-executes the hypercall
+	 * instruction.
+	 */
+	vcpu->arch.cui_cancellable = true;
 	return 0;
 }
 
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index d8de7742a0da..2824fe8e1417 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2237,7 +2237,6 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_IOEVENTFD_NO_LENGTH:
 
 	case KVM_CAP_SET_IDENTITY_MAP_ADDR:
-	case KVM_CAP_VCPU_EVENTS:
 #ifdef CONFIG_KVM_HYPERV
 	case KVM_CAP_HYPERV:
 	case KVM_CAP_HYPERV_VAPIC:
@@ -2305,6 +2304,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_EXIT_HYPERCALL:
 		r = KVM_EXIT_HYPERCALL_VALID_MASK;
 		break;
+	case KVM_CAP_VCPU_EVENTS:
+		r = KVM_VCPU_EVENTS_BASE | KVM_VCPU_EVENTS_CANCEL_HCALL;
+		break;
 #ifdef CONFIG_KVM_HYPERV
 	case KVM_CAP_HYPERV_FORWARD_HCALL:
 		r = KVM_HYPERV_HCALL_SUPPORTED;
@@ -3052,9 +3054,21 @@ static int kvm_vcpu_ioctl_x86_set_vcpu_events(struct kvm_vcpu *vcpu,
 			      | KVM_VCPUEVENT_VALID_SHADOW
 			      | KVM_VCPUEVENT_VALID_SMM
 			      | KVM_VCPUEVENT_VALID_PAYLOAD
-			      | KVM_VCPUEVENT_VALID_TRIPLE_FAULT))
+			      | KVM_VCPUEVENT_VALID_TRIPLE_FAULT
+			      | KVM_VCPUEVENT_CANCEL_HCALL))
 		return -EINVAL;
 
+	/*
+	 * Refuse the cancel unless a cancellable completion is pending: the
+	 * flag is only valid on the vCPU whose KVM_EXIT_HYPERV_HCALL exit
+	 * userspace is holding, and a completion KVM armed for its own
+	 * purposes (MMIO and PIO emulation continuations) is not userspace's
+	 * to revoke.
+	 */
+	if ((events->flags & KVM_VCPUEVENT_CANCEL_HCALL) &&
+	    (!vcpu->arch.complete_userspace_io || !vcpu->arch.cui_cancellable))
+		return -EBUSY;
+
 	if (events->flags & KVM_VCPUEVENT_VALID_PAYLOAD) {
 		if (!vcpu->kvm->arch.exception_payload_enabled)
 			return -EINVAL;
@@ -3152,6 +3166,16 @@ static int kvm_vcpu_ioctl_x86_set_vcpu_events(struct kvm_vcpu *vcpu,
 			kvm_clear_request(KVM_REQ_TRIPLE_FAULT, vcpu);
 	}
 
+	/*
+	 * Drop the pending completion: nothing retires the hypercall
+	 * instruction, so the vCPU re-executes it on its next entry unless
+	 * this call queued an exception to be delivered there instead.
+	 */
+	if (events->flags & KVM_VCPUEVENT_CANCEL_HCALL) {
+		vcpu->arch.complete_userspace_io = NULL;
+		vcpu->arch.cui_cancellable = false;
+	}
+
 	kvm_make_request(KVM_REQ_EVENT, vcpu);
 
 	return 0;
@@ -9060,6 +9084,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 	if (unlikely(vcpu->arch.complete_userspace_io)) {
 		int (*cui)(struct kvm_vcpu *) = vcpu->arch.complete_userspace_io;
 		vcpu->arch.complete_userspace_io = NULL;
+		vcpu->arch.cui_cancellable = false;
 		r = cui(vcpu);
 		if (r <= 0)
 			goto out;
@@ -9393,6 +9418,14 @@ static int sync_regs(struct kvm_vcpu *vcpu)
 	if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_EVENTS) {
 		struct kvm_vcpu_events events = vcpu->run->s.regs.events;
 
+		/*
+		 * Cancelling a hypercall completion takes an explicit
+		 * KVM_SET_VCPU_EVENTS; KVM_GET_VCPU_EVENTS never reports the
+		 * flag, so no synced copy of the events legitimately has it.
+		 */
+		if (events.flags & KVM_VCPUEVENT_CANCEL_HCALL)
+			return -EINVAL;
+
 		if (kvm_vcpu_ioctl_x86_set_vcpu_events(vcpu, &events))
 			return -EINVAL;
 
diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 17a4256e9081..7a51ff975cfa 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -134,6 +134,7 @@ TEST_GEN_PROGS_x86 += x86/sync_regs_test
 TEST_GEN_PROGS_x86 += x86/ucna_injection_test
 TEST_GEN_PROGS_x86 += x86/userspace_io_test
 TEST_GEN_PROGS_x86 += x86/userspace_msr_exit_test
+TEST_GEN_PROGS_x86 += x86/vcpu_events_cancel_test
 TEST_GEN_PROGS_x86 += x86/vmx_apic_access_test
 TEST_GEN_PROGS_x86 += x86/vmx_apicv_updates_test
 TEST_GEN_PROGS_x86 += x86/vmx_exception_with_invalid_guest_state
diff --git a/tools/testing/selftests/kvm/x86/vcpu_events_cancel_test.c b/tools/testing/selftests/kvm/x86/vcpu_events_cancel_test.c
new file mode 100644
index 000000000000..d78f7f881031
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/vcpu_events_cancel_test.c
@@ -0,0 +1,119 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Test KVM_VCPUEVENT_CANCEL_HCALL without a cancellable completion: the
+ * cancel must fail with EBUSY and disturb nothing, whether nothing at
+ * all is pending or the pending completion belongs to KVM (a PIO
+ * emulation continuation), and an undefined flags bit must still fail
+ * with EINVAL, as must the flag in the events KVM_RUN syncs from
+ * kvm_run.  The positive paths need a hypercall exit and live in the
+ * Hyper-V forwarding test (hyperv_forward_hcall) later in this series.
+ */
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+
+#define TEST_PORT	0xd00d
+
+static void guest_code(void)
+{
+	asm volatile("outb %%al, %%dx" :: "a"(0xaa), "d"(TEST_PORT));
+	GUEST_DONE();
+}
+
+/*
+ * The cancel must fail with EBUSY and leave the vCPU's event state as
+ * KVM_GET_VCPU_EVENTS reported it.
+ */
+static void check_cancel_refused(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vcpu_events events, after;
+	int r;
+
+	vcpu_events_get(vcpu, &events);
+	events.flags |= KVM_VCPUEVENT_CANCEL_HCALL;
+	r = __vcpu_ioctl(vcpu, KVM_SET_VCPU_EVENTS, &events);
+	TEST_ASSERT(r == -1 && errno == EBUSY,
+		    "Cancel with nothing to cancel: r = %d, errno = %d",
+		    r, errno);
+
+	events.flags &= ~KVM_VCPUEVENT_CANCEL_HCALL;
+	vcpu_events_get(vcpu, &after);
+	TEST_ASSERT(!memcmp(&events, &after, sizeof(events)),
+		    "Failed cancel changed the vCPU's events");
+}
+
+int main(int argc, char *argv[])
+{
+	struct kvm_vcpu_events events;
+	struct kvm_vcpu *vcpu;
+	struct kvm_regs regs, after;
+	struct kvm_vm *vm;
+	struct ucall uc;
+	long cap;
+	int r;
+
+	cap = kvm_check_cap(KVM_CAP_VCPU_EVENTS);
+	TEST_REQUIRE(cap & KVM_VCPU_EVENTS_CANCEL_HCALL);
+	TEST_ASSERT(cap & KVM_VCPU_EVENTS_BASE,
+		    "KVM_CAP_VCPU_EVENTS lost its base bit: %#lx", cap);
+
+	vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+
+	/* Nothing pending at all. */
+	check_cancel_refused(vcpu);
+
+	/*
+	 * An undefined flags bit is still unknown.  Flags are allocated from
+	 * the bottom, so probe the top bit rather than the next free one.
+	 */
+	vcpu_events_get(vcpu, &events);
+	events.flags |= 1U << 31;
+	r = __vcpu_ioctl(vcpu, KVM_SET_VCPU_EVENTS, &events);
+	TEST_ASSERT(r == -1 && errno == EINVAL,
+		    "Undefined events flag: r = %d, errno = %d", r, errno);
+
+	/* Only the ioctl cancels: in the synced events the flag is invalid. */
+	if (kvm_check_cap(KVM_CAP_SYNC_REGS) & KVM_SYNC_X86_EVENTS) {
+		vcpu_events_get(vcpu, &vcpu->run->s.regs.events);
+		vcpu->run->s.regs.events.flags |= KVM_VCPUEVENT_CANCEL_HCALL;
+		vcpu->run->kvm_dirty_regs = KVM_SYNC_X86_EVENTS;
+		r = __vcpu_run(vcpu);
+		TEST_ASSERT(r == -1 && errno == EINVAL,
+			    "Cancel through sync_regs: r = %d, errno = %d",
+			    r, errno);
+		vcpu->run->kvm_dirty_regs = 0;
+	} else {
+		print_skip("KVM_CAP_SYNC_REGS lacks KVM_SYNC_X86_EVENTS, sync_regs cancel case");
+	}
+
+	vcpu_run(vcpu);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+	TEST_ASSERT(vcpu->run->io.port == TEST_PORT &&
+		    vcpu->run->io.direction == KVM_EXIT_IO_OUT,
+		    "Expected OUT to port %#x, got %s port %#x", TEST_PORT,
+		    vcpu->run->io.direction == KVM_EXIT_IO_OUT ? "OUT" : "IN",
+		    vcpu->run->io.port);
+
+	/*
+	 * The OUT's completion is KVM's, not userspace's to revoke, and
+	 * the refusal must not move the vCPU.
+	 */
+	vcpu_regs_get(vcpu, &regs);
+	check_cancel_refused(vcpu);
+	vcpu_regs_get(vcpu, &after);
+	TEST_ASSERT_EQ(regs.rip, after.rip);
+	TEST_ASSERT_EQ(regs.rax, after.rax);
+
+	/*
+	 * The pending completion must still retire the OUT: the next exit
+	 * is the GUEST_DONE ucall, not the OUT re-executing.
+	 */
+	vcpu_run(vcpu);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+	TEST_ASSERT(vcpu->run->io.port != TEST_PORT,
+		    "The OUT re-executed: its completion was lost");
+	TEST_ASSERT_EQ(get_ucall(vcpu, &uc), UCALL_DONE);
+
+	kvm_vm_free(vm);
+	return 0;
+}
-- 
2.50.1

[PATCH v2 3/4] KVM: x86: hyper-v: Advertise XMM fast hypercall output

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

A fast hypercall may return its output in XMM registers when the
hypervisor reports HV_X64_HYPERCALL_XMM_OUTPUT_AVAILABLE, and Windows
only uses those forms when the bit is set. KVM does not report it in
KVM_GET_SUPPORTED_HV_CPUID, although nothing stops a VMM handling a
forwarded fast hypercall from writing its output through the vCPU FPU
state before resuming the vCPU.

Report the bit. None of the hypercalls KVM implements itself returns
output, so it only matters for hypercalls the VMM handles, where the
VMM writes the output registers.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/kvm/hyperv.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 858ee0f374e5..e0d9b2b1302d 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -2870,6 +2870,7 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid,
 			ent->ebx |= HV_ENABLE_EXTENDED_HYPERCALLS;
 
 			ent->edx |= HV_X64_HYPERCALL_XMM_INPUT_AVAILABLE;
+			ent->edx |= HV_X64_HYPERCALL_XMM_OUTPUT_AVAILABLE;
 			ent->edx |= HV_FEATURE_FREQUENCY_MSRS_AVAILABLE;
 			ent->edx |= HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE;
 
-- 
2.50.1

[PATCH v2 4/4] KVM: selftests: Add Hyper-V hypercall forwarding test

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Cover KVM_CAP_HYPERV_FORWARD_HCALL and the completion cancel:

 - the feature mask reports forwarding, and rejects bits it does not
   report;
 - with nothing enabled, an unknown hypercall fails in KVM;
 - with forwarding enabled, it exits to userspace in memory-based and
   fast form with the control value and parameters, and the guest
   sees the status userspace supplies (and, for the fast form, the XMM
   output userspace writes through KVM_SET_FPU);
 - the result reaches the guest as the full 64-bit value userspace
   wrote, bit 63 included;
 - a cancel with a queued #UD faults the hypercall, leaving RIP and
   RAX untouched;
 - a cancel alone makes the same hypercall exit a second time, and
   completing the second exit works;
 - the cancel fails with EBUSY at a hypercall-less ucall exit, whose
   PIO completion is not cancellable, and leaves the vCPU's events
   unchanged;
 - disabling everything restores the in-kernel failure.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/x86/hyperv_forward_hcall.c  | 337 ++++++++++++++++++
 2 files changed, 338 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/hyperv_forward_hcall.c

diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 7a51ff975cfa..ca68943f99d4 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -89,6 +89,7 @@ TEST_GEN_PROGS_x86 += x86/hyperv_cpuid
 TEST_GEN_PROGS_x86 += x86/hyperv_evmcs
 TEST_GEN_PROGS_x86 += x86/hyperv_extended_hypercalls
 TEST_GEN_PROGS_x86 += x86/hyperv_features
+TEST_GEN_PROGS_x86 += x86/hyperv_forward_hcall
 TEST_GEN_PROGS_x86 += x86/hyperv_ipi
 TEST_GEN_PROGS_x86 += x86/hyperv_svm_test
 TEST_GEN_PROGS_x86 += x86/hyperv_tlb_flush
diff --git a/tools/testing/selftests/kvm/x86/hyperv_forward_hcall.c b/tools/testing/selftests/kvm/x86/hyperv_forward_hcall.c
new file mode 100644
index 000000000000..4c290b6a1012
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/hyperv_forward_hcall.c
@@ -0,0 +1,337 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Test KVM_CAP_HYPERV_FORWARD_HCALL: the feature mask KVM reports and
+ * accepts, Hyper-V hypercalls that KVM does not implement exiting to
+ * userspace in memory-based and fast form, with userspace supplying the
+ * result and the XMM output, and a pending completion cancelled through
+ * KVM_SET_VCPU_EVENTS - alone, so that the guest re-executes the
+ * hypercall, and together with a queued #UD, so that the hypercall
+ * faults instead.
+ */
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+#include "hyperv.h"
+
+/* A hypercall code KVM does not implement (HvCallGetVpRegisters). */
+#define HVCALL_TEST_CODE	0x50
+
+#define FAST_PARAM0		0x1122334455667788ULL
+#define FAST_PARAM1		0x99aabbccddeeff00ULL
+#define XMM_IN_MARKER		0x5157ca11c0ffee00ULL
+#define XMM_OUT_MARKER		0x0de1ba5eba5eba11ULL
+#define USERSPACE_STATUS	0x55AA
+#define FULL_RESULT		(BIT_ULL(63) | USERSPACE_STATUS)
+
+enum {
+	STAGE_DISABLED = 1,
+	STAGE_MEMORY,
+	STAGE_FAST,
+	STAGE_RESULT_FULL,
+	STAGE_CANCEL_UD,
+	STAGE_CANCEL_RETRY,
+	STAGE_AFTER_CANCEL,
+	STAGE_REDISABLED,
+};
+
+static void guest_code(gpa_t pgs_gpa)
+{
+	__sse128_u in, out;
+	u64 status;
+	u8 vector;
+
+	wrmsr(HV_X64_MSR_GUEST_OS_ID, HYPERV_LINUX_OS_ID);
+	wrmsr(HV_X64_MSR_HYPERCALL, pgs_gpa);
+
+	/* Nothing enabled: KVM fails the call itself. */
+	vector = __hyperv_hypercall(HVCALL_TEST_CODE, pgs_gpa + PAGE_SIZE,
+				    pgs_gpa + 2 * PAGE_SIZE, &status);
+	GUEST_ASSERT(!vector);
+	GUEST_ASSERT_EQ(status & 0xffff, HV_STATUS_INVALID_HYPERCALL_CODE);
+	GUEST_SYNC(STAGE_DISABLED);
+
+	/* Forwarding enabled: userspace supplies the status. */
+	vector = __hyperv_hypercall(HVCALL_TEST_CODE, pgs_gpa + PAGE_SIZE,
+				    pgs_gpa + 2 * PAGE_SIZE, &status);
+	GUEST_ASSERT(!vector);
+	GUEST_ASSERT_EQ(status, USERSPACE_STATUS);
+	GUEST_SYNC(STAGE_MEMORY);
+
+	/*
+	 * Fast form: the register parameters and XMM0 reach userspace, and
+	 * the XMM1 value userspace writes reaches the guest.
+	 */
+	in.as_u64[0] = XMM_IN_MARKER;
+	in.as_u64[1] = XMM_IN_MARKER + 1;
+	out.as_u64[0] = 0;
+	out.as_u64[1] = 0;
+	write_sse_reg(0, &in.vec);
+	write_sse_reg(1, &out.vec);
+	vector = __hyperv_hypercall(HVCALL_TEST_CODE | HV_HYPERCALL_FAST_BIT,
+				    FAST_PARAM0, FAST_PARAM1, &status);
+	read_sse_reg(1, &out.vec);
+	GUEST_ASSERT(!vector);
+	GUEST_ASSERT_EQ(status, USERSPACE_STATUS);
+	GUEST_ASSERT_EQ(out.as_u64[0], XMM_OUT_MARKER);
+	GUEST_ASSERT_EQ(out.as_u64[1], XMM_OUT_MARKER + 1);
+	GUEST_SYNC(STAGE_FAST);
+
+	/* The result reaches the guest as all 64 bits userspace wrote. */
+	vector = __hyperv_hypercall(HVCALL_TEST_CODE, pgs_gpa + PAGE_SIZE,
+				    pgs_gpa + 2 * PAGE_SIZE, &status);
+	GUEST_ASSERT(!vector);
+	GUEST_ASSERT_EQ(status, FULL_RESULT);
+	GUEST_SYNC(STAGE_RESULT_FULL);
+
+	/* Userspace cancels the completion and queues #UD: the call faults. */
+	vector = __hyperv_hypercall(HVCALL_TEST_CODE, pgs_gpa + PAGE_SIZE,
+				    pgs_gpa + 2 * PAGE_SIZE, &status);
+	GUEST_ASSERT_EQ(vector, UD_VECTOR);
+	GUEST_SYNC(STAGE_CANCEL_UD);
+
+	/*
+	 * Userspace cancels the completion without queueing anything: this
+	 * one hypercall re-executes and exits to userspace a second time,
+	 * which then completes it.
+	 */
+	vector = __hyperv_hypercall(HVCALL_TEST_CODE, pgs_gpa + PAGE_SIZE,
+				    pgs_gpa + 2 * PAGE_SIZE, &status);
+	GUEST_ASSERT(!vector);
+	GUEST_ASSERT_EQ(status, USERSPACE_STATUS);
+	GUEST_SYNC(STAGE_CANCEL_RETRY);
+
+	/* An ordinary completion still works after the cancels. */
+	vector = __hyperv_hypercall(HVCALL_TEST_CODE, pgs_gpa + PAGE_SIZE,
+				    pgs_gpa + 2 * PAGE_SIZE, &status);
+	GUEST_ASSERT(!vector);
+	GUEST_ASSERT_EQ(status, USERSPACE_STATUS);
+	GUEST_SYNC(STAGE_AFTER_CANCEL);
+
+	/* Everything disabled again. */
+	vector = __hyperv_hypercall(HVCALL_TEST_CODE, pgs_gpa + PAGE_SIZE,
+				    pgs_gpa + 2 * PAGE_SIZE, &status);
+	GUEST_ASSERT(!vector);
+	GUEST_ASSERT_EQ(status & 0xffff, HV_STATUS_INVALID_HYPERCALL_CODE);
+	GUEST_SYNC(STAGE_REDISABLED);
+
+	GUEST_DONE();
+}
+
+static void check_hcall_exit(struct kvm_vcpu *vcpu, u64 expect_control,
+			     u64 expect_in, u64 expect_out)
+{
+	struct kvm_run *run = vcpu->run;
+
+	TEST_ASSERT_EQ(run->exit_reason, KVM_EXIT_HYPERV);
+	TEST_ASSERT_EQ(run->hyperv.type, KVM_EXIT_HYPERV_HCALL);
+	TEST_ASSERT_EQ(run->hyperv.u.hcall.input, expect_control);
+	TEST_ASSERT_EQ(run->hyperv.u.hcall.params[0], expect_in);
+	TEST_ASSERT_EQ(run->hyperv.u.hcall.params[1], expect_out);
+}
+
+/*
+ * Cancel the pending hypercall completion, optionally queueing a #UD to
+ * be delivered at the (unadvanced) hypercall instruction.  The cancel
+ * must leave RIP and RAX alone.
+ */
+static void cancel_hcall(struct kvm_vcpu *vcpu, bool queue_ud)
+{
+	struct kvm_vcpu_events events;
+	struct kvm_regs before, after;
+
+	vcpu_events_get(vcpu, &events);
+	events.flags |= KVM_VCPUEVENT_CANCEL_HCALL;
+	if (queue_ud) {
+		events.flags |= KVM_VCPUEVENT_VALID_PAYLOAD;
+		events.exception.pending = 1;
+		events.exception.injected = 0;
+		events.exception.nr = UD_VECTOR;
+		events.exception.has_error_code = 0;
+		events.exception_has_payload = 0;
+	}
+
+	vcpu_regs_get(vcpu, &before);
+	vcpu_events_set(vcpu, &events);
+	vcpu_regs_get(vcpu, &after);
+	TEST_ASSERT_EQ(before.rip, after.rip);
+	TEST_ASSERT_EQ(before.rax, after.rax);
+}
+
+/*
+ * With no cancellable completion pending (nothing pending at all, or a
+ * non-cancellable completion such as the ucall PIO's), the cancel must
+ * fail with EBUSY and leave the vCPU's event state as
+ * KVM_GET_VCPU_EVENTS reported it.
+ */
+static void check_cancel_refused(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vcpu_events events, after;
+	int r;
+
+	vcpu_events_get(vcpu, &events);
+	events.flags |= KVM_VCPUEVENT_CANCEL_HCALL;
+	r = __vcpu_ioctl(vcpu, KVM_SET_VCPU_EVENTS, &events);
+	TEST_ASSERT(r == -1 && errno == EBUSY,
+		    "Cancel with nothing to cancel: r = %d, errno = %d",
+		    r, errno);
+
+	events.flags &= ~KVM_VCPUEVENT_CANCEL_HCALL;
+	vcpu_events_get(vcpu, &after);
+	TEST_ASSERT(!memcmp(&events, &after, sizeof(events)),
+		    "Failed cancel changed the vCPU's events");
+}
+
+static void handle_fast_hcall(struct kvm_vcpu *vcpu)
+{
+	struct kvm_fpu fpu;
+
+	check_hcall_exit(vcpu, HVCALL_TEST_CODE | HV_HYPERCALL_FAST_BIT,
+			 FAST_PARAM0, FAST_PARAM1);
+
+	vcpu_fpu_get(vcpu, &fpu);
+	TEST_ASSERT_EQ(*(u64 *)&fpu.xmm[0][0], XMM_IN_MARKER);
+	TEST_ASSERT_EQ(*(u64 *)&fpu.xmm[0][8], XMM_IN_MARKER + 1);
+	*(u64 *)&fpu.xmm[1][0] = XMM_OUT_MARKER;
+	*(u64 *)&fpu.xmm[1][8] = XMM_OUT_MARKER + 1;
+	vcpu_fpu_set(vcpu, &fpu);
+
+	vcpu->run->hyperv.u.hcall.result = USERSPACE_STATUS;
+}
+
+static void test_feature_mask(struct kvm_vm *vm)
+{
+	u64 supported = kvm_check_cap(KVM_CAP_HYPERV_FORWARD_HCALL);
+	int r;
+
+	TEST_ASSERT(supported & KVM_HYPERV_HCALL_FORWARD_UNKNOWN,
+		    "KVM_HYPERV_HCALL_FORWARD_UNKNOWN not reported");
+	r = __vm_enable_cap(vm, KVM_CAP_HYPERV_FORWARD_HCALL, BIT_ULL(63));
+	TEST_ASSERT(r && errno == EINVAL,
+		    "Enabling an undefined feature should fail with EINVAL");
+
+	/* Enabling nothing is allowed. */
+	vm_enable_cap(vm, KVM_CAP_HYPERV_FORWARD_HCALL, 0);
+}
+
+int main(int argc, char *argv[])
+{
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	gva_t hcall_page;
+	gpa_t hcall_gpa;
+	struct ucall uc;
+	int stage = STAGE_DISABLED;
+	bool retried = false;
+
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_HYPERV_FORWARD_HCALL));
+	TEST_REQUIRE(kvm_check_cap(KVM_CAP_VCPU_EVENTS) &
+		     KVM_VCPU_EVENTS_CANCEL_HCALL);
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_EXCEPTION_PAYLOAD));
+
+	vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+	vcpu_set_hv_cpuid(vcpu);
+	vm_enable_cap(vm, KVM_CAP_EXCEPTION_PAYLOAD, 1);
+
+	test_feature_mask(vm);
+
+	/* Hypercall page plus an input and an output page. */
+	hcall_page = vm_alloc_pages(vm, 3);
+	memset(addr_gva2hva(vm, hcall_page), 0x0, 3 * getpagesize());
+	hcall_gpa = addr_gva2gpa(vm, hcall_page);
+
+	vcpu_args_set(vcpu, 1, hcall_gpa);
+
+	for (;;) {
+		vcpu_run(vcpu);
+
+		if (vcpu->run->exit_reason == KVM_EXIT_HYPERV) {
+			switch (stage) {
+			case STAGE_MEMORY:
+				check_hcall_exit(vcpu, HVCALL_TEST_CODE,
+						 hcall_gpa + getpagesize(),
+						 hcall_gpa + 2 * getpagesize());
+				vcpu->run->hyperv.u.hcall.result = USERSPACE_STATUS;
+				break;
+			case STAGE_FAST:
+				handle_fast_hcall(vcpu);
+				break;
+			case STAGE_RESULT_FULL:
+				check_hcall_exit(vcpu, HVCALL_TEST_CODE,
+						 hcall_gpa + getpagesize(),
+						 hcall_gpa + 2 * getpagesize());
+				vcpu->run->hyperv.u.hcall.result = FULL_RESULT;
+				break;
+			case STAGE_CANCEL_UD:
+				check_hcall_exit(vcpu, HVCALL_TEST_CODE,
+						 hcall_gpa + getpagesize(),
+						 hcall_gpa + 2 * getpagesize());
+				cancel_hcall(vcpu, true);
+				/* Cancelled: the result must never reach the guest. */
+				vcpu->run->hyperv.u.hcall.result = USERSPACE_STATUS;
+				break;
+			case STAGE_CANCEL_RETRY:
+				check_hcall_exit(vcpu, HVCALL_TEST_CODE,
+						 hcall_gpa + getpagesize(),
+						 hcall_gpa + 2 * getpagesize());
+				if (!retried) {
+					/*
+					 * Cancel with nothing queued: the same
+					 * hypercall must exit a second time.
+					 */
+					cancel_hcall(vcpu, false);
+					vcpu->run->hyperv.u.hcall.result = 0;
+					retried = true;
+				} else {
+					vcpu->run->hyperv.u.hcall.result = USERSPACE_STATUS;
+				}
+				break;
+			case STAGE_AFTER_CANCEL:
+				check_hcall_exit(vcpu, HVCALL_TEST_CODE,
+						 hcall_gpa + getpagesize(),
+						 hcall_gpa + 2 * getpagesize());
+				vcpu->run->hyperv.u.hcall.result = USERSPACE_STATUS;
+				break;
+			default:
+				TEST_FAIL("Unexpected hypercall exit in stage %d", stage);
+			}
+			continue;
+		}
+
+		TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+
+		switch (get_ucall(vcpu, &uc)) {
+		case UCALL_SYNC:
+			TEST_ASSERT_EQ(uc.args[1], stage);
+			switch (stage) {
+			case STAGE_DISABLED:
+				vm_enable_cap(vm, KVM_CAP_HYPERV_FORWARD_HCALL,
+					      KVM_HYPERV_HCALL_FORWARD_UNKNOWN);
+				break;
+			case STAGE_CANCEL_RETRY:
+				/*
+				 * The ucall PIO's own completion is pending
+				 * here: not cancellable, even with the
+				 * feature enabled.
+				 */
+				check_cancel_refused(vcpu);
+				break;
+			case STAGE_AFTER_CANCEL:
+				vm_enable_cap(vm, KVM_CAP_HYPERV_FORWARD_HCALL, 0);
+				break;
+			}
+			stage++;
+			break;
+		case UCALL_ABORT:
+			REPORT_GUEST_ASSERT(uc);
+		case UCALL_DONE:
+			goto done;
+		default:
+			TEST_FAIL("Unhandled ucall: %ld", uc.cmd);
+		}
+	}
+
+done:
+	TEST_ASSERT_EQ(stage, STAGE_REDISABLED + 1);
+	kvm_vm_free(vm);
+	return 0;
+}
-- 
2.50.1

Re: [PATCH v2 4/4] KVM: selftests: Add Hyper-V hypercall forwarding test

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] KVM: selftests: missing volatile and XMM constraints in
  guest_code()
--

commit e7e0d35b7ca4b9fc386b6fe7a75c6cc9bf38431a
Author: David Woodhouse <dwmw@amazon.co.uk>

KVM: selftests: Add Hyper-V hypercall forwarding test

This commit adds a KVM selftest covering KVM_CAP_HYPERV_FORWARD_HCALL for
memory-based and fast hypercall exits as well as hypercall completion
cancellation via KVM_SET_VCPU_EVENTS.

> diff --git a/tools/testing/selftests/kvm/x86/hyperv_forward_hcall.c b/tools/testing/selftests/kvm/x86/hyperv_forward_hcall.c
> new file mode 100644
> index 0000000000000..4c290b6a10122
> --- /dev/null
> +++ b/tools/testing/selftests/kvm/x86/hyperv_forward_hcall.c
> @@ -0,0 +1,337 @@

[ ... ]

> +static void guest_code(gpa_t pgs_gpa)
> +{
> +	__sse128_u in, out;
> +	u64 status;
> +	u8 vector;

[ ... ]

> +	/*
> +	 * Fast form: the register parameters and XMM0 reach userspace, and
> +	 * the XMM1 value userspace writes reaches the guest.
> +	 */
> +	in.as_u64[0] = XMM_IN_MARKER;
> +	in.as_u64[1] = XMM_IN_MARKER + 1;
> +	out.as_u64[0] = 0;
> +	out.as_u64[1] = 0;
> +	write_sse_reg(0, &in.vec);
> +	write_sse_reg(1, &out.vec);
> +	vector = __hyperv_hypercall(HVCALL_TEST_CODE | HV_HYPERCALL_FAST_BIT,
> +				    FAST_PARAM0, FAST_PARAM1, &status);
> +	read_sse_reg(1, &out.vec);
> +	GUEST_ASSERT(!vector);
> +	GUEST_ASSERT_EQ(status, USERSPACE_STATUS);
> +	GUEST_ASSERT_EQ(out.as_u64[0], XMM_OUT_MARKER);
> +	GUEST_ASSERT_EQ(out.as_u64[1], XMM_OUT_MARKER + 1);
> +	GUEST_SYNC(STAGE_FAST);

[Severity: Low]
Can the compiler reorder read_sse_reg() across __hyperv_hypercall(), or
clobber xmm0 before the hypercall runs?

In tools/testing/selftests/kvm/include/x86/processor.h, read_sse_reg()
uses a non-volatile asm statement with an output operand and no input
operands, and neither write_sse_reg() nor __hyperv_hypercall() declares
xmm register constraints or clobbers:

tools/testing/selftests/kvm/include/x86/processor.h:read_sse_reg() {
	switch (reg) {
	...
	case 1:
		asm("movdqa %%xmm1, %0" : "=m"(*data));
		break;
}

tools/testing/selftests/kvm/include/x86/processor.h:write_sse_reg() {
	switch (reg) {
	case 0:
		asm("movdqa %0, %%xmm0" : : "m"(*data));
		break;
}

Because x86 selftests are compiled with -O2 -march=x86-64-v2 (with SSE
enabled), the compiler can schedule a vectorized zeroing of out via xmm0
after write_sse_reg(0, &in.vec) or move read_sse_reg(1, &out.vec) ahead of
__hyperv_hypercall(), causing spurious assertion failures in
handle_fast_hcall() or guest_code().

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261113170535.79246-1-dwmw2@infradead.org?part=4

Re: [PATCH v2 0/4] KVM: x86: hyper-v: Forward unimplemented hypercalls to userspace

[Sean Christopherson]

On Fri, Nov 13, 2026, David Woodhouse wrote:
> From: David Woodhouse <dwmw@amazon.co.uk>
> 
> KVM implements a fixed set of Hyper-V hypercalls and fails every other
> code with HV_STATUS_INVALID_HYPERCALL_CODE. A VMM that wants to
> implement more of the TLFS, such as the VSM hypercalls Windows uses for
> Virtualization Based Security, never sees them.
> 
> This series adds KVM_CAP_HYPERV_FORWARD_HCALL. As with KVM_CAP_XEN_HVM
> and KVM_CAP_EXIT_HYPERCALL, KVM_CHECK_EXTENSION returns a bitmask of
> features and KVM_ENABLE_CAP takes the subset to enable, rejecting any
> bit the kernel does not report. The first feature,
> KVM_HYPERV_HCALL_FORWARD_UNKNOWN, sends the hypercalls KVM does not
> implement, and extended hypercalls in fast form, out through the
> existing KVM_EXIT_HYPERV_HCALL exit, whose format does not change at
> all.

I'm not a fan of fowarding unknown hypercalls.  It seems like it would be quite
easy to end up with a mess where it's difficult for KVM to implement "native"
support without breaking userspace.

How much code (and ongoing churn) would it be to add an allowlist of hypercalls
to forward to userspace?  And then obviously require userspace to configure its
forwarding "intercepts".

Re: [PATCH v2 0/4] KVM: x86: hyper-v: Forward unimplemented hypercalls to userspace

[David Woodhouse]

On 13 November 2026 10:10:40 GMT-07:00, Sean Christopherson <seanjc@google.com> wrote:
>On Fri, Nov 13, 2026, David Woodhouse wrote:
>> From: David Woodhouse <dwmw@amazon.co.uk>
>> 
>> KVM implements a fixed set of Hyper-V hypercalls and fails every other
>> code with HV_STATUS_INVALID_HYPERCALL_CODE. A VMM that wants to
>> implement more of the TLFS, such as the VSM hypercalls Windows uses for
>> Virtualization Based Security, never sees them.
>> 
>> This series adds KVM_CAP_HYPERV_FORWARD_HCALL. As with KVM_CAP_XEN_HVM
>> and KVM_CAP_EXIT_HYPERCALL, KVM_CHECK_EXTENSION returns a bitmask of
>> features and KVM_ENABLE_CAP takes the subset to enable, rejecting any
>> bit the kernel does not report. The first feature,
>> KVM_HYPERV_HCALL_FORWARD_UNKNOWN, sends the hypercalls KVM does not
>> implement, and extended hypercalls in fast form, out through the
>> existing KVM_EXIT_HYPERV_HCALL exit, whose format does not change at
>> all.
>
>I'm not a fan of fowarding unknown hypercalls.  It seems like it would be quite
>easy to end up with a mess where it's difficult for KVM to implement "native"
>support without breaking userspace.
>
>How much code (and ongoing churn) would it be to add an allowlist of hypercalls
>to forward to userspace?  And then obviously require userspace to configure its
>forwarding "intercepts".

I'm following the model we used for Xen support, where the default assumption is that we do everything in userspace, then accelerate only what's absolutely necessary in the kernel.

Userspace can opt *in* to having the kernel support certain hypercalls, and later patches waiting in the wings already add the in-kernel VTL switching enabled via a bitmask when enabling the capability.