[RFC PATCH 00/55] KVM: x86: Hyper-V VSM (Virtual Trust Levels) on VM planes

[RFC PATCH 00/55] KVM: x86: Hyper-V VSM (Virtual Trust Levels) on VM planes

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Windows uses Virtualization Based Security for Credential Guard and
friends, and enables it by default in current releases. A guest with
VBS enabled runs its secure kernel in VTL1, a higher privilege level
which the hypervisor has to keep isolated from the normal kernel in
VTL0. KVM has no concept of that today, so Windows guests either lose
VBS or emulate it with nested virtualization, which is slower and
loses features.

There have been completely in-kernel implementations of VBS, which
did unnatural things all within a single kvm_vcpu (four APICs, etc.)

This is an attempt to build it more cleanly, more in userspace, based
on KVM planes (https://lwn.net/Articles/1087590/).

The behavioural reference was the in-kernel VSM implementation in the
Nitro Hypervisor, complete with its latest compatibility fixes since
it was last posted — largely turned into a bunch of kvm unit
tests. The result boots Windows Server 2016 through 2025 with
Credential Guard active, on both Intel and AMD hosts, with VTL1
protecting its memory from VTL0. With hibernate/resume tested, since
that part has always been extra fun.

For now I've structured the userspace part as a separable library
which can be pulled into QEMU or other VMMs. Partly because QEMU
doesn't want it when AI did the typing (but *might* tolerate it as an
external library), and partly because I think it might make sense to
have a more permissively licensed standard implementation for how
KVM-based VMMs implement VSM, and have some input from the Microsoft
side and some improved expectation that it'll *keep* working with
Windows and not keep breaking for every new release. But that's a
future conversation. For now I'm just preempting the question, "why
does it look like that?".

 • https://git.infradead.org/?p=users/dwmw2/linux.git;a=shortlog;h=refs/heads/vsm
 • https://git.infradead.org/?p=users/dwmw2/qemu.git;a=shortlog;h=refs/heads/vsm
 • https://git.infradead.org/?p=users/dwmw2/kvm-unit-tests.git;a=shortlog;h=refs/heads/vsm

Pulled into my base tree on v7.1-rc7 (not reposted here):

 • KVM planes itself (Jörg's latest from June)
   https://lore.kernel.org/kvm/20260608144252.351443-1-joro@8bytes.org/
   https://github.com/joergroedel/linux/tree/kvm-planes-v7.1
 • Paolo's memory protection attributes v3
   https://lore.kernel.org/kvm/20260918135030.171564-1-pbonzini@redhat.com/
 • Melody Wang's SEV-SNP #HV doorbell / Restricted Injection series
   https://lore.kernel.org/all/cover.1777048608.git.huibo.wang@amd.com/
 • A few fixes to the above ride just above them in the git tree,
   each citing the report it addresses; not part of this posting

Generic fixes and features in KVM before we get to VSM itself:

 • Userspace opt-in (KVM_CAP_PLANES) and plane run-state ioctls;
   MP state, guest-debug, KVM_TRANSLATE, SETUP_MCE and a filtered
   KVM_ENABLE_CAP on plane vCPU fds; unwind of a failed
   KVM_CREATE_PLANE
 • Plane mechanics: per-plane request fan-out, kicking the twin
   that is actually running, XCR0 and TSC matching shared across
   twins, runtime CPUID per plane, APICv inhibited while planes
   are enabled (for now).
 • Per-plane memory attributes and overlays, and instruction state
   on the memory-fault exit, for the VMM to build intercept
   messages from
 • Fixes found along the way: KVM_TRANSLATE reporting a real
   write-access walk, SVM keeping TPR[3:0] across exits without a
   guest CR8 write

Finally we get to the actual Hyper-V VSM bits:

 • Hypercall forwarding to userspace (KVM_CAP_HYPERV_FORWARD_HCALL),
   with completion or injected #UD, for the VSM hypercalls KVM
   doesn't handle itself
 • HvCallVtlCall/HvCallVtlReturn switching between planes in the
   kernel; forced VTL entry gated on interrupt deliverability and
   re-armed when a plane suspends with IRR set
 • Per-plane Hyper-V state: GUEST_OS_ID/HYPERCALL MSRs, the
   reference TSC page, SynIC and event routing, TLB flushes
 • VTL register intercepts, including the descriptor-table load
   traps Credential Guard requires
 • The suspended-VP state (from Nikolas Wipper's series), refusal of
   SendIpi to a higher VTL, XMM fast hypercall output
 • Selftests for both halves; a hyperv_vsm suite for kvm-unit-tests
   covering VTL call/return, protections, intercepts and eventing
   follows separately

The four WIP-marked patches need finishing and splitting; they are
included because the series does not function without them. This is
very much an RFC.

Held back in the git tree for a follow-up posting:

 • KVM_TRANSLATE2 and the page-walker flags/status work (also
   Nikolas's), and the HvCallTranslateVirtualAddress handling built
   on it
 • TLB flush inhibit (TlbLocked)

Not done at all yet: VINA, and migration of mid-VTL-switch state
(the forced-entry and captured-register state has no GET/SET
exposure). The interaction with SEV-SNP VMPLs, the other consumer of
planes, is untested from this side.

Mushahid also has a version which boots Windows 11 on arm64, which
we'll reconcile when some of the dust settles.

I think this will take a while to reconcile and merge, even after
the base dependencies (planes, memattrs, etc.) are in. I'd like to
start settling the basic functionality so we can start preparing
to merge it in little pieces, if we can reach a general consensus.


David Woodhouse (44):
  KVM: x86/mmu: Move ACC_* masks from spte.h to mmu.h
  KVM: x86: Let userspace opt in to multiple planes with KVM_CAP_PLANES
  KVM: selftests: Add x86 planes lifecycle test
  KVM: x86: hyper-v: Add KVM_CAP_HYPERV_FORWARD_HCALL
  KVM: selftests: Add Hyper-V hypercall forwarding test
  KVM: Add KVM_GET/SET_PLANE_RUN_STATE vCPU ioctls
  KVM: selftests: Cover plane run-state ioctls in planes_test
  KVM: x86: hyper-v: Let userspace fault a forwarded hypercall
  KVM: x86: Allow MP state ioctls on plane vCPU fds
  KVM: x86: Filter KVM_ENABLE_CAP on plane vCPU fds
  KVM: x86: initialise Hyper-V state on every plane vCPU
  KVM: implement per-plane memory attributes
  KVM: x86: hyper-v: bank GUEST_OS_ID and HYPERCALL MSRs per plane
  KVM: x86: hyper-v: bank the reference TSC page per plane
  KVM: x86: allow KVM_SET_GUEST_DEBUG on plane vCPU fds
  KVM: x86: hyper-v: flush the calling plane's TLB on flush hypercalls
  KVM: x86/mmu: free obsolete roots of all planes
  KVM: x86: hyperv: don't arm PV EOI for a higher plane's VP assist page
  KVM: x86: allow KVM_TRANSLATE on plane vCPU fds
  KVM: x86: hyper-v: handle VTL call/return in the kernel
  KVM: x86: load each plane twin while updating it after KVM_SET_CPUID
  WIP: userspace VTL interrupt entry and register intercepts
  WIP: KVM: x86: deliver events to suspended higher planes
  WIP: KVM: per-plane memory overlays, richer protection-fault exits
  WIP: KVM: x86: VSM event-delivery parity fixes from Windows boot
    triage
  KVM: planes: fan out per-plane requests to twin vCPUs
  KVM: x86: inhibit APICv when VM planes are enabled
  KVM: x86: hyperv: divert LLDT and LTR into register intercepts
  KVM: planes: Unwind published state on KVM_CREATE_PLANE failure
  KVM: planes: Kick the running twin's pCPU, not the request target's
  KVM: planes: Keep current_vcpu in sync in vcpu_load()
  KVM: x86: Report a real write-access walk from KVM_TRANSLATE
  KVM: planes: Share XCR0 across plane twins
  KVM: x86: hyperv: Refuse a synthetic IPI to a higher VTL
  KVM: x86: Gate forced VTL entry on interrupt deliverability
  KVM: selftests: Test a plane suspending with an undelivered interrupt
  KVM: x86: hyperv: Advertise XMM fast hypercall output
  KVM: SVM: Carry instruction state on reflected plane-protection faults
  KVM: SVM: Trap descriptor-table loads for VTL register intercepts
  KVM: SVM: Keep TPR[3:0] across exits without a guest CR8 write
  KVM: selftests: Add Hyper-V reference TSC page memory attribute test
  KVM: SVM: Hedge the v_tpr_entry comment
  KVM: Check every plane's protections before caching a gfn->hva
  KVM: planes: Annotate the lockless twin-publication protocol

Mushahid Hussain (9):
  KVM: x86: Leave TSC matching to the plane 0 vCPUs
  KVM: x86: hyperv: Track the reference TSC page per plane
  KVM: x86: hyperv: Update the reference TSC page past protections
  KVM: x86: hyperv: Publish an added reference TSC page at once
  KVM: x86: hyperv: Harden the per-plane reference TSC page updates
  KVM: x86: Derive runtime CPUID bits from the executing plane
  KVM: selftests: Test hugepage sizing over mixed protection attributes
  KVM: x86: hyperv: Do not skip a moved vCPU when completing a hypercall
  KVM: x86: Allow KVM_X86_SETUP_MCE on plane vCPUs

Nikolas Wipper (1):
  KVM: x86: Implement Hyper-V's vCPU suspended state

Paolo Bonzini (1):
  KVM: selftests: Add VM plane infrastructure and tests

 Documentation/virt/kvm/api.rst                | 241 ++++-
 arch/x86/hyperv/hv_apic.c                     |   3 +-
 arch/x86/include/asm/kvm-x86-ops.h            |   1 +
 arch/x86/include/asm/kvm_host.h               | 140 ++-
 arch/x86/kvm/cpuid.c                          | 139 ++-
 arch/x86/kvm/cpuid.h                          |   6 +-
 arch/x86/kvm/emulate.c                        |  21 +
 arch/x86/kvm/hyperv.c                         | 995 ++++++++++++++++--
 arch/x86/kvm/hyperv.h                         |  37 +-
 arch/x86/kvm/kvm_emulate.h                    |  10 +
 arch/x86/kvm/lapic.c                          |  85 +-
 arch/x86/kvm/mmu.h                            |  14 +
 arch/x86/kvm/mmu/mmu.c                        | 131 ++-
 arch/x86/kvm/mmu/paging_tmpl.h                |   7 +-
 arch/x86/kvm/mmu/spte.h                       |  14 -
 arch/x86/kvm/mmu/tdp_mmu.c                    |  23 +-
 arch/x86/kvm/svm/nested.c                     |  16 +
 arch/x86/kvm/svm/svm.c                        | 143 ++-
 arch/x86/kvm/svm/svm.h                        |  16 +-
 arch/x86/kvm/vmx/main.c                       |  13 +-
 arch/x86/kvm/vmx/nested.c                     |   1 +
 arch/x86/kvm/vmx/vmx.c                        | 118 ++-
 arch/x86/kvm/vmx/x86_ops.h                    |   1 +
 arch/x86/kvm/x86.c                            | 668 +++++++++++-
 include/hyperv/hvgdk_mini.h                   |   5 +-
 include/hyperv/hvhdk_mini.h                   |   3 +-
 include/linux/kvm_host.h                      | 149 ++-
 include/uapi/linux/kvm.h                      |  63 +-
 tools/include/uapi/linux/kvm.h                |  54 +-
 tools/testing/selftests/kvm/Makefile.kvm      |   7 +
 .../testing/selftests/kvm/include/kvm_util.h  |  74 ++
 .../selftests/kvm/include/x86/processor.h     |   1 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |  85 +-
 .../testing/selftests/kvm/lib/x86/processor.c |  15 +
 .../testing/selftests/kvm/memory_attributes.c |   6 +-
 tools/testing/selftests/kvm/plane_test.c      | 137 +++
 .../selftests/kvm/x86/hyperv_forward_hcall.c  | 144 +++
 .../selftests/kvm/x86/hyperv_tsc_page_attrs.c | 129 +++
 .../selftests/kvm/x86/memattr_hugepage_test.c | 132 +++
 .../selftests/kvm/x86/plane_x86_test.c        | 325 ++++++
 tools/testing/selftests/kvm/x86/planes_test.c | 142 +++
 .../selftests/kvm/x86/planes_vtl_entry_test.c | 188 ++++
 virt/kvm/kvm_main.c                           | 699 +++++++++++-
 virt/kvm/pfncache.c                           |   3 +-
 44 files changed, 4956 insertions(+), 248 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/plane_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/hyperv_forward_hcall.c
 create mode 100644 tools/testing/selftests/kvm/x86/hyperv_tsc_page_attrs.c
 create mode 100644 tools/testing/selftests/kvm/x86/memattr_hugepage_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/plane_x86_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/planes_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/planes_vtl_entry_test.c

-- 
2.50.1

[RFC PATCH 01/55] KVM: x86/mmu: Move ACC_* masks from spte.h to mmu.h

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The memory-attributes backport uses the ACC_* masks in mmu_internal.h,
which cannot include spte.h because spte.h includes *it*. Anything
pulling in mmu_internal.h without spte.h (debugfs.c, page_track.c) now
fails to build.

Move the ACC_* definitions to mmu.h, next to the PT_WRITABLE_MASK and
PT_USER_MASK definitions they are built from. Both headers already
include it.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/mmu.h      | 14 ++++++++++++++
 arch/x86/kvm/mmu/spte.h | 14 --------------
 2 files changed, 14 insertions(+), 14 deletions(-)

diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h
index 830f46145692..ae1068169925 100644
--- a/arch/x86/kvm/mmu.h
+++ b/arch/x86/kvm/mmu.h
@@ -32,6 +32,20 @@ extern bool __read_mostly enable_mmio_caching;
 #define PT_DIR_PAT_SHIFT 12
 #define PT_DIR_PAT_MASK (1ULL << PT_DIR_PAT_SHIFT)
 
+#define ACC_EXEC_MASK    1
+#define ACC_WRITE_MASK   PT_WRITABLE_MASK
+#define ACC_USER_MASK    PT_USER_MASK
+#define ACC_ALL          (ACC_EXEC_MASK | ACC_WRITE_MASK | ACC_USER_MASK)
+
+/*
+ * Read permission for a gfn's memory protection attributes.  Not part of
+ * ACC_ALL and never set in a PTE's access mask: PTE formats cannot express
+ * no-read, and x86 rejects protection attributes that forbid only read
+ * (kvm_arch_mem_attributes_supported_prot()), so a no-read gfn always
+ * takes a memory fault exit before any SPTE is created.
+ */
+#define ACC_READ_MASK    BIT(3)
+
 #define PT64_ROOT_5LEVEL 5
 #define PT64_ROOT_4LEVEL 4
 #define PT32_ROOT_LEVEL 2
diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h
index 217f6b40c50b..511aa0e0e222 100644
--- a/arch/x86/kvm/mmu/spte.h
+++ b/arch/x86/kvm/mmu/spte.h
@@ -45,20 +45,6 @@ static_assert(SPTE_TDP_AD_ENABLED == 0);
 #define SPTE_PERM_MASK (PT_PRESENT_MASK | PT_WRITABLE_MASK | shadow_user_mask \
 			| shadow_x_mask | shadow_nx_mask | shadow_me_mask)
 
-#define ACC_EXEC_MASK    1
-#define ACC_WRITE_MASK   PT_WRITABLE_MASK
-#define ACC_USER_MASK    PT_USER_MASK
-#define ACC_ALL          (ACC_EXEC_MASK | ACC_WRITE_MASK | ACC_USER_MASK)
-
-/*
- * Read permission for a gfn's memory protection attributes.  Not part of
- * ACC_ALL and never set in a PTE's access mask: PTE formats cannot express
- * no-read, and x86 rejects protection attributes that forbid only read
- * (kvm_arch_mem_attributes_supported_prot()), so a no-read gfn always
- * takes a memory fault exit before any SPTE is created.
- */
-#define ACC_READ_MASK    BIT(3)
-
 /* The mask for the R/X bits in EPT PTEs */
 #define SPTE_EPT_READABLE_MASK			0x1ull
 #define SPTE_EPT_EXECUTABLE_MASK		0x4ull
-- 
2.50.1

[RFC PATCH 02/55] KVM: x86: Let userspace opt in to multiple planes with KVM_CAP_PLANES

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

A VMM implementing a software privilege model such as Hyper-V VSM
needs more than one plane, but kvm_arch_max_planes() returns the
vendor ceiling — 1 everywhere except SEV-SNP, where VMPLs are
guest-architectural — so KVM_CREATE_PLANE rejects every id above zero.

Make KVM_CAP_PLANES a VM enable capability taking the number of planes
userspace is going to manage, up to KVM_MAX_PLANES. It must be set
before any vCPU or extra plane exists, so nothing is sized against the
old limit, and cannot be changed once set. Non-default VM types are
rejected: SEV-SNP keeps deriving its count from the supported VMPLs,
and protected-state VM types need their own audit first. A VM that
does not enable the capability keeps reporting a single plane.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 Documentation/virt/kvm/api.rst  | 27 +++++++++++++++++++++
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/x86.c              | 42 ++++++++++++++++++++++++++++++++-
 3 files changed, 69 insertions(+), 1 deletion(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index b5f0136bb555..bc63054ce7bd 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6598,6 +6598,12 @@ KVM_S390_KEYOP_SSKE
 
 Creates a new *plane*, i.e. a separate privilege level for the virtual machine.
 
+The range of valid plane ids is [0, N) where N is the value returned by
+KVM_CHECK_EXTENSION(KVM_CAP_PLANES) on the VM file descriptor.  Plane 0 is
+the VM itself; for VM types where the plane count is not guest-architectural,
+ids above 0 only become valid after userspace enables KVM_CAP_PLANES with
+the number of planes it is going to manage (see section 7.47).
+
 Each plane has a numeric id that is used when communicating with KVM.  While
 KVM is currently agnostic to whether low ids are more or less privileged, it is
 expected that this will not always be the case in the future.  For example KVM
@@ -8959,6 +8965,27 @@ helpful if user space wants to emulate instructions which are not
 This capability can be enabled dynamically even if VCPUs were already
 created and are running.
 
+7.47 KVM_CAP_PLANES
+-------------------
+
+:Architectures: x86
+:Target: VM
+:Parameters: args[0] contains the number of planes to make available
+
+Opts the VM in to using multiple planes, raising the limit on plane ids
+that KVM_CREATE_PLANE accepts to args[0] (between 1 and the KVM_MAX_PLANES
+value implied by KVM_CHECK_EXTENSION(KVM_CAP_PLANES) on the system fd).
+
+Planes other than plane 0 only exist when userspace manages them, so by
+default a VM reports and accepts a single plane.  VM types whose plane
+count is guest-architectural (such as SEV-SNP, where it reflects the
+supported VMPLs) report that count without this capability, and enabling
+it on them fails with -EINVAL.
+
+This capability must be enabled before any vCPU or extra plane has been
+created, and the chosen count cannot be changed afterwards; otherwise
+-EEXIST is returned.
+
 7.49 KVM_CAP_HCALL_FAULT_EXIT
 ------------------------------------
 
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 0ce814bbf1ea..e4c8691dc1e1 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1442,6 +1442,7 @@ struct kvm_arch {
 	unsigned int indirect_shadow_pages;
 	u8 mmu_valid_gen;
 	u8 vm_type;
+	unsigned int user_max_planes;
 	bool has_private_mem;
 	bool has_protected_state;
 	bool has_protected_eoi;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 9a15e06fc91e..a9073436c734 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -490,11 +490,29 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_default_max_planes);
 
 unsigned kvm_arch_max_planes(struct kvm *kvm)
 {
+	unsigned int vendor_planes;
+
 	/* For now, planes are only supported with irqchip=split */
 	if (!irqchip_split(kvm))
 		return 1;
 
-	return kvm_x86_call(max_planes)(kvm);
+	vendor_planes = kvm_x86_call(max_planes)(kvm);
+
+	/*
+	 * When the plane count is guest-architectural (SEV-SNP VMPLs),
+	 * the vendor module owns it.  Otherwise planes are a software
+	 * construct and only exist when userspace has opted in to the
+	 * number of planes it is going to manage, via KVM_CAP_PLANES.
+	 */
+	if (vendor_planes > 1)
+		return vendor_planes;
+
+	/*
+	 * Lockless readers (KVM_CHECK_EXTENSION) race the KVM_ENABLE_CAP
+	 * store; the value is set once, so a torn-free read is all that
+	 * is needed.
+	 */
+	return READ_ONCE(kvm->arch.user_max_planes) ? : 1;
 }
 
 struct kvm_plane *x86_alloc_plane(void)
@@ -6846,6 +6864,28 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		mutex_unlock(&kvm->lock);
 		break;
 	}
+	case KVM_CAP_PLANES: {
+		mutex_lock(&kvm->lock);
+		r = -EINVAL;
+		if (cap->args[0] < 1 || cap->args[0] > KVM_MAX_PLANES)
+			goto planes_unlock;
+		/*
+		 * Only ordinary VMs for now.  SEV-SNP VMs get their plane
+		 * count from the architectural VMPL support, and VM types
+		 * with protected state need per-plane audits first.
+		 */
+		if (kvm->arch.vm_type != KVM_X86_DEFAULT_VM)
+			goto planes_unlock;
+		r = -EEXIST;
+		if (kvm->created_vcpus || kvm->has_planes ||
+		    kvm->arch.user_max_planes)
+			goto planes_unlock;
+		WRITE_ONCE(kvm->arch.user_max_planes, cap->args[0]);
+		r = 0;
+planes_unlock:
+		mutex_unlock(&kvm->lock);
+		break;
+	}
 	case KVM_CAP_X2APIC_API:
 		r = -EINVAL;
 		if (cap->args[0] & ~KVM_X2APIC_API_VALID_FLAGS)
-- 
2.50.1

[RFC PATCH 03/55] KVM: selftests: Add VM plane infrastructure and tests

[David Woodhouse]

From: Paolo Bonzini <pbonzini@redhat.com>

Add plane and plane-vCPU wrappers to the selftest library and two tests:
plane_test (plane and plane vCPU creation, sibling rules, register state
isolation) and plane_x86_test (per-plane FPU, MSRs and LAPIC; MSI
delivery to a plane). Planes require a split IRQ chip, so the tests
create their VM with vm_create_barebones_irqchip(true).

The code is from the planes RFC's selftest work, already adapted to
the KVM_CREATE_VCPU on a plane fd shape of the posted series.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
[hmushi: ported onto the posted series on kvm/next]
Signed-off-by: Mushahid Hussain <hmushi@amazon.co.uk>

[dwmw2: ported to this stack's uAPI: KVM_CAP_PLANES is a VM-level
 opt-in taking the plane count and must precede vCPU creation, so the
 tests enable it with 2 planes right after VM creation. The main()
 probe uses a split-irqchip VM, because kvm_arch_max_planes() reports
 1 without one, and skips when the opt-in is refused. Two subtest
 groups of the later planes series are not carried: the foreign-thread
 EBUSY test (this kernel does not restrict vCPU ioctls to the vCPU's
 own thread) and the userspace plane-switch group (this kernel drives
 plane switches through the Hyper-V VSM hypercalls).
 kvm_force_split_irqchip already exists here; all seven remaining
 subtests pass on this kernel.]
(re-done by hand from Mushahid's port)
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 tools/testing/selftests/kvm/Makefile.kvm      |   2 +
 .../testing/selftests/kvm/include/kvm_util.h  |  73 ++++
 .../selftests/kvm/include/x86/processor.h     |   1 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |  74 +++-
 .../testing/selftests/kvm/lib/x86/processor.c |  15 +
 tools/testing/selftests/kvm/plane_test.c      | 137 ++++++++
 .../selftests/kvm/x86/plane_x86_test.c        | 325 ++++++++++++++++++
 7 files changed, 622 insertions(+), 5 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/plane_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/plane_x86_test.c

diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 03da6e59e614..61e767a1e31b 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -66,6 +66,7 @@ TEST_GEN_PROGS_COMMON += kvm_page_table_test
 TEST_GEN_PROGS_COMMON += set_memory_region_test
 TEST_GEN_PROGS_COMMON += memslot_modification_stress_test
 TEST_GEN_PROGS_COMMON += memslot_perf_test
+TEST_GEN_PROGS_COMMON += plane_test
 
 # Compiled test targets
 TEST_GEN_PROGS_x86 = $(TEST_GEN_PROGS_COMMON)
@@ -102,6 +103,7 @@ TEST_GEN_PROGS_x86 += x86/nested_tsc_adjust_test
 TEST_GEN_PROGS_x86 += x86/nested_tsc_scaling_test
 TEST_GEN_PROGS_x86 += x86/nested_vmsave_vmload_test
 TEST_GEN_PROGS_x86 += x86/platform_info_test
+TEST_GEN_PROGS_x86 += x86/plane_x86_test
 TEST_GEN_PROGS_x86 += x86/pmu_counters_test
 TEST_GEN_PROGS_x86 += x86/pmu_event_filter_test
 TEST_GEN_PROGS_x86 += x86/private_mem_conversions_test
diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h
index 4349aabc5d56..204f1474f102 100644
--- a/tools/testing/selftests/kvm/include/kvm_util.h
+++ b/tools/testing/selftests/kvm/include/kvm_util.h
@@ -74,6 +74,21 @@ struct kvm_vcpu {
 	u32 dirty_gfns_count;
 };
 
+struct kvm_plane {
+	struct list_head list;
+	uint32_t id;
+	int fd;
+	struct kvm_vm *vm;
+};
+
+struct kvm_plane_vcpu {
+	struct list_head list;
+	uint32_t id;
+	int fd;
+	struct kvm_vcpu *plane0;
+	struct kvm_plane *plane;
+};
+
 struct userspace_mem_regions {
 	struct rb_root gpa_tree;
 	struct rb_root hva_tree;
@@ -107,10 +122,13 @@ struct kvm_vm {
 	unsigned int va_bits;
 	u64 max_gfn;
 	struct list_head vcpus;
+	struct list_head planes;
+	struct list_head plane_vcpus;
 	struct userspace_mem_regions regions;
 	struct sparsebit *vpages_valid;
 	struct sparsebit *vpages_mapped;
 	bool has_irqchip;
+	bool has_split_irqchip;
 	gpa_t ucall_mmio_addr;
 	gva_t handlers;
 	u32 dirty_ring_size;
@@ -377,6 +395,21 @@ do {											\
 	__TEST_ASSERT_VM_VCPU_IOCTL(!ret, #cmd, ret, vm);		\
 })
 
+static __always_inline void static_assert_is_plane(struct kvm_plane *plane) { }
+
+#define __plane_ioctl(plane, cmd, arg)				\
+({								\
+	static_assert_is_plane(plane);				\
+	kvm_do_ioctl((plane)->fd, cmd, arg);			\
+})
+
+#define plane_ioctl(plane, cmd, arg)				\
+({								\
+	int ret = __plane_ioctl(plane, cmd, arg);		\
+								\
+	__TEST_ASSERT_VM_VCPU_IOCTL(!ret, #cmd, ret, (plane)->vm); \
+})
+
 static __always_inline void static_assert_is_vcpu(struct kvm_vcpu *vcpu) { }
 
 #define __vcpu_ioctl(vcpu, cmd, arg)				\
@@ -392,6 +425,21 @@ static __always_inline void static_assert_is_vcpu(struct kvm_vcpu *vcpu) { }
 	__TEST_ASSERT_VM_VCPU_IOCTL(!ret, #cmd, ret, (vcpu)->vm);	\
 })
 
+static __always_inline void static_assert_is_plane_vcpu(struct kvm_plane_vcpu *plane_vcpu) { }
+
+#define __plane_vcpu_ioctl(plane_vcpu, cmd, arg)		\
+({								\
+	static_assert_is_plane_vcpu(plane_vcpu);		\
+	kvm_do_ioctl((plane_vcpu)->fd, cmd, arg);		\
+})
+
+#define plane_vcpu_ioctl(plane_vcpu, cmd, arg)			\
+({								\
+	int ret = __plane_vcpu_ioctl(plane_vcpu, cmd, arg);	\
+								\
+	__TEST_ASSERT_VM_VCPU_IOCTL(!ret, #cmd, ret, (plane_vcpu)->plane0->vm); \
+})
+
 /*
  * Looks up and returns the value corresponding to the capability
  * (KVM_CAP_*) given by cap.
@@ -667,6 +715,17 @@ static inline bool is_smt_on(void)
 
 void vm_create_irqchip(struct kvm_vm *vm);
 
+/*
+ * Record the irqchip mode on the VM so that kvm_vm_restart() recreates
+ * the same mode, whatever kvm_force_split_irqchip says by then.
+ */
+static inline void vm_enable_split_irqchip(struct kvm_vm *vm)
+{
+	vm_enable_cap(vm, KVM_CAP_SPLIT_IRQCHIP, 24);
+	vm->has_irqchip = true;
+	vm->has_split_irqchip = true;
+}
+
 static inline int __vm_create_guest_memfd(struct kvm_vm *vm, u64 size,
 					  u64 flags)
 {
@@ -717,6 +776,8 @@ void vm_mem_region_reload(struct kvm_vm *vm, u32 slot);
 void vm_mem_region_move(struct kvm_vm *vm, u32 slot, u64 new_gpa);
 void vm_mem_region_delete(struct kvm_vm *vm, u32 slot);
 struct kvm_vcpu *__vm_vcpu_add(struct kvm_vm *vm, u32 vcpu_id);
+struct kvm_plane *vm_plane_add(struct kvm_vm *vm, int plane_id);
+struct kvm_plane_vcpu *__vm_plane_vcpu_add(struct kvm_vcpu *vcpu, struct kvm_plane *plane);
 void vm_populate_gva_bitmap(struct kvm_vm *vm);
 gva_t vm_unused_gva_gap(struct kvm_vm *vm, size_t sz, gva_t min_gva);
 gva_t vm_alloc(struct kvm_vm *vm, size_t sz, gva_t min_gva);
@@ -1034,6 +1095,18 @@ static inline struct kvm_vm *vm_create_barebones(void)
 	return ____vm_create(VM_SHAPE_DEFAULT);
 }
 
+static inline struct kvm_vm *vm_create_barebones_irqchip(bool split)
+{
+	struct kvm_vm *vm = vm_create_barebones();
+
+	if (split)
+		vm_enable_split_irqchip(vm);
+	else
+		vm_create_irqchip(vm);
+
+	return vm;
+}
+
 static inline struct kvm_vm *vm_create_barebones_type(unsigned long type)
 {
 	const struct vm_shape shape = {
diff --git a/tools/testing/selftests/kvm/include/x86/processor.h b/tools/testing/selftests/kvm/include/x86/processor.h
index 7cc09721e149..43be2d3faed9 100644
--- a/tools/testing/selftests/kvm/include/x86/processor.h
+++ b/tools/testing/selftests/kvm/include/x86/processor.h
@@ -1136,6 +1136,7 @@ static inline void vcpu_clear_cpuid_feature(struct kvm_vcpu *vcpu,
 
 u64 vcpu_get_msr(struct kvm_vcpu *vcpu, u64 msr_index);
 int _vcpu_set_msr(struct kvm_vcpu *vcpu, u64 msr_index, u64 msr_value);
+int _plane_vcpu_set_msr(struct kvm_plane_vcpu *plane_vcpu, u64 msr_index, u64 msr_value);
 
 /*
  * Assert on an MSR access(es) and pretty print the MSR name when possible.
diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c
index e08967ef7b7b..3972c9e0486a 100644
--- a/tools/testing/selftests/kvm/lib/kvm_util.c
+++ b/tools/testing/selftests/kvm/lib/kvm_util.c
@@ -284,6 +284,8 @@ struct kvm_vm *____vm_create(struct vm_shape shape)
 	TEST_ASSERT(vm != NULL, "Insufficient Memory");
 
 	INIT_LIST_HEAD(&vm->vcpus);
+	INIT_LIST_HEAD(&vm->planes);
+	INIT_LIST_HEAD(&vm->plane_vcpus);
 	vm->regions.gpa_tree = RB_ROOT;
 	vm->regions.hva_tree = RB_ROOT;
 	hash_init(vm->regions.slot_hash);
@@ -596,7 +598,9 @@ void kvm_vm_restart(struct kvm_vm *vmp)
 	struct userspace_mem_region *region;
 
 	vm_open(vmp);
-	if (vmp->has_irqchip)
+	if (vmp->has_split_irqchip)
+		vm_enable_split_irqchip(vmp);
+	else if (vmp->has_irqchip)
 		vm_create_irqchip(vmp);
 
 	hash_for_each(vmp->regions.slot_hash, ctr, region, slot_node) {
@@ -788,11 +792,25 @@ static void vm_vcpu_rm(struct kvm_vm *vm, struct kvm_vcpu *vcpu)
 
 void kvm_vm_release(struct kvm_vm *vmp)
 {
+	struct kvm_plane_vcpu *plane_vcpu, *tmp_plane_vcpu;
+	struct kvm_plane *plane, *tmp_plane;
 	struct kvm_vcpu *vcpu, *tmp;
 
+	list_for_each_entry_safe(plane_vcpu, tmp_plane_vcpu, &vmp->plane_vcpus, list) {
+		kvm_close(plane_vcpu->fd);
+		list_del(&plane_vcpu->list);
+		free(plane_vcpu);
+	}
+
 	list_for_each_entry_safe(vcpu, tmp, &vmp->vcpus, list)
 		vm_vcpu_rm(vmp, vcpu);
 
+	list_for_each_entry_safe(plane, tmp_plane, &vmp->planes, list) {
+		kvm_close(plane->fd);
+		list_del(&plane->list);
+		free(plane);
+	}
+
 	kvm_close(vmp->fd);
 	kvm_close(vmp->kvm_fd);
 
@@ -1365,6 +1383,51 @@ struct kvm_vcpu *__vm_vcpu_add(struct kvm_vm *vm, u32 vcpu_id)
 	return vcpu;
 }
 
+/*
+ * Creates the plane with the ID given by plane_id in the VM specified by vm.
+ * No additional setup is done.  Returns the plane.
+ */
+struct kvm_plane *vm_plane_add(struct kvm_vm *vm, int plane_id)
+{
+	struct kvm_plane *plane;
+
+	plane = calloc(1, sizeof(*plane));
+	TEST_ASSERT(plane != NULL, "Insufficient Memory");
+
+	plane->fd = __vm_ioctl(vm, KVM_CREATE_PLANE, (void *)(unsigned long)plane_id);
+	TEST_ASSERT_VM_VCPU_IOCTL(plane->fd >= 0, KVM_CREATE_PLANE, plane->fd, vm);
+	plane->vm = vm;
+	plane->id = plane_id;
+
+	/* Add to linked-list of planes. */
+	list_add(&plane->list, &vm->planes);
+
+	return plane;
+}
+
+/*
+ * Adds @vcpu's sibling in @plane.  No additional setup is done.  Returns the
+ * plane vCPU.
+ */
+struct kvm_plane_vcpu *__vm_plane_vcpu_add(struct kvm_vcpu *vcpu, struct kvm_plane *plane)
+{
+	struct kvm_plane_vcpu *plane_vcpu;
+
+	plane_vcpu = calloc(1, sizeof(*plane_vcpu));
+	TEST_ASSERT(plane_vcpu != NULL, "Insufficient Memory");
+
+	plane_vcpu->fd = __plane_ioctl(plane, KVM_CREATE_VCPU, (void *)(unsigned long)vcpu->id);
+	TEST_ASSERT_VM_VCPU_IOCTL(plane_vcpu->fd >= 0, KVM_CREATE_VCPU, plane_vcpu->fd, plane->vm);
+	plane_vcpu->id = vcpu->id;
+	plane_vcpu->plane0 = vcpu;
+	plane_vcpu->plane = plane;
+
+	/* Add to linked-list of extra-plane VCPUs. */
+	list_add(&plane_vcpu->list, &plane->vm->plane_vcpus);
+
+	return plane_vcpu;
+}
+
 /*
  * Within the VM specified by @vm, locates the lowest starting guest virtual
  * address >= @min_gva, that has at least @sz unallocated bytes.  A
@@ -1644,11 +1707,12 @@ void vm_create_irqchip(struct kvm_vm *vm)
 	 * support for KVM_CREATE_IRQCHIP).
 	 */
 	r = __vm_ioctl(vm, KVM_CREATE_IRQCHIP, NULL);
-	if (r && errno == ENOTTY && kvm_has_cap(KVM_CAP_SPLIT_IRQCHIP))
-		vm_enable_cap(vm, KVM_CAP_SPLIT_IRQCHIP, 24);
-	else
-		TEST_ASSERT_VM_VCPU_IOCTL(!r, KVM_CREATE_IRQCHIP, r, vm);
+	if (r && errno == ENOTTY && kvm_has_cap(KVM_CAP_SPLIT_IRQCHIP)) {
+		vm_enable_split_irqchip(vm);
+		return;
+	}
 
+	TEST_ASSERT_VM_VCPU_IOCTL(!r, KVM_CREATE_IRQCHIP, r, vm);
 	vm->has_irqchip = true;
 }
 
diff --git a/tools/testing/selftests/kvm/lib/x86/processor.c b/tools/testing/selftests/kvm/lib/x86/processor.c
index 947fc31c0acb..8d8eb3552be1 100644
--- a/tools/testing/selftests/kvm/lib/x86/processor.c
+++ b/tools/testing/selftests/kvm/lib/x86/processor.c
@@ -1092,6 +1092,21 @@ int _vcpu_set_msr(struct kvm_vcpu *vcpu, u64 msr_index, u64 msr_value)
 	return __vcpu_ioctl(vcpu, KVM_SET_MSRS, &buffer.header);
 }
 
+int _plane_vcpu_set_msr(struct kvm_plane_vcpu *plane_vcpu, u64 msr_index, u64 msr_value)
+{
+	struct {
+		struct kvm_msrs header;
+		struct kvm_msr_entry entry;
+	} buffer = {};
+
+	memset(&buffer, 0, sizeof(buffer));
+	buffer.header.nmsrs = 1;
+	buffer.entry.index = msr_index;
+	buffer.entry.data = msr_value;
+
+	return __plane_vcpu_ioctl(plane_vcpu, KVM_SET_MSRS, &buffer.header);
+}
+
 void vcpu_args_set(struct kvm_vcpu *vcpu, unsigned int num, ...)
 {
 	va_list ap;
diff --git a/tools/testing/selftests/kvm/plane_test.c b/tools/testing/selftests/kvm/plane_test.c
new file mode 100644
index 000000000000..cfa90275b323
--- /dev/null
+++ b/tools/testing/selftests/kvm/plane_test.c
@@ -0,0 +1,137 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2025 Red Hat, Inc.
+ *
+ * Test for architecture-neutral VM plane functionality
+ */
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "test_util.h"
+
+#include "kvm_util.h"
+#include "asm/kvm.h"
+#include "linux/kvm.h"
+
+/*
+ * Port note (dwmw stack): KVM_CAP_PLANES is a VM-level opt-in that takes
+ * the plane count and must precede vCPU creation; planes are not
+ * always-on.  The upstream planes series' foreign-thread EBUSY subtest is
+ * omitted: this kernel does not restrict vCPU ioctls to the vCPU's own
+ * thread.  The assertions below are otherwise unchanged.
+ */
+#define NR_PLANES 2
+
+static void enable_planes(struct kvm_vm *vm)
+{
+	vm_enable_cap(vm, KVM_CAP_PLANES, NR_PLANES);
+}
+
+void test_create_plane_errors(int max_planes)
+{
+	struct kvm_vm *vm;
+	struct kvm_vcpu *vcpu;
+	int planefd, plane_vcpufd;
+
+	/* Planes require an in-kernel (split) IRQ chip. */
+	vm = vm_create_barebones_irqchip(true);
+	enable_planes(vm);
+	vcpu = __vm_vcpu_add(vm, 0);
+
+	planefd = __vm_ioctl(vm, KVM_CREATE_PLANE, (void *)(unsigned long)0);
+	TEST_ASSERT(planefd == -1 && errno == EEXIST,
+		    "Creating existing plane, expecting EEXIST. ret: %d, errno: %d",
+		    planefd, errno);
+
+	planefd = __vm_ioctl(vm, KVM_CREATE_PLANE, (void *)(unsigned long)max_planes);
+	TEST_ASSERT(planefd == -1 && errno == EINVAL,
+		    "Creating plane %d, expecting EINVAL. ret: %d, errno: %d",
+		    max_planes, planefd, errno);
+
+	plane_vcpufd = __vm_ioctl(vm, KVM_CREATE_VCPU, (void *)(unsigned long)vcpu->id);
+	TEST_ASSERT(plane_vcpufd == -1 && errno == EEXIST,
+		    "Creating existing vCPU for plane 0, expecting EEXIST. ret: %d, errno: %d",
+		    plane_vcpufd, errno);
+
+	kvm_vm_free(vm);
+	ksft_test_result_pass("error conditions\n");
+}
+
+void test_create_plane(void)
+{
+	struct kvm_vm *vm;
+	struct kvm_vcpu *vcpu;
+	struct kvm_plane *plane;
+	int r;
+
+	vm = vm_create_barebones_irqchip(true);
+	enable_planes(vm);
+	vcpu = __vm_vcpu_add(vm, 0);
+
+	plane = vm_plane_add(vm, 1);
+
+	/* A plane fd takes plane ioctls only; capabilities are a VM matter. */
+	r = __plane_ioctl(plane, KVM_CHECK_EXTENSION, (void *)(unsigned long)KVM_CAP_PLANES);
+	TEST_ASSERT(r == -1 && errno == ENOTTY,
+		    "KVM_CHECK_EXTENSION on a plane fd, expecting ENOTTY. ret: %d, errno: %d",
+		    r, errno);
+
+	r = __vm_ioctl(vm, KVM_CREATE_PLANE, (void *)(unsigned long)1);
+	TEST_ASSERT(r == -1 && errno == EEXIST,
+		    "Creating existing plane, expecting EEXIST. ret: %d, errno: %d",
+		    r, errno);
+
+	__vm_plane_vcpu_add(vcpu, plane);
+
+	r = __plane_ioctl(plane, KVM_CREATE_VCPU, (void *)(unsigned long)vcpu->id);
+	TEST_ASSERT(r == -1 && errno == EEXIST,
+		    "Creating vCPU again for plane 1. ret: %d, errno: %d",
+		    r, errno);
+
+	r = __plane_ioctl(plane, KVM_RUN, (void *)(unsigned long)0);
+	TEST_ASSERT(r == -1 && errno == ENOTTY,
+		    "Running plane vCPU again for plane 1. ret: %d, errno: %d",
+		    r, errno);
+
+	kvm_vm_free(vm);
+	ksft_test_result_pass("basic planefd and plane_vcpufd operation\n");
+}
+
+int main(int argc, char *argv[])
+{
+	struct kvm_vm *vm;
+	int cap_planes = 0;
+
+	/*
+	 * kvm_arch_max_planes() reports 1 without a split irqchip, so the
+	 * probe VM needs one even though the opt-in itself does not.  On
+	 * architectures without KVM_CAP_SPLIT_IRQCHIP the vm_enable_cap()
+	 * inside vm_create_barebones_irqchip() would assert, turning the
+	 * missing feature into a failure instead of a skip.
+	 */
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_SPLIT_IRQCHIP));
+
+	vm = vm_create_barebones_irqchip(true);
+
+	if (!__vm_enable_cap(vm, KVM_CAP_PLANES, NR_PLANES))
+		cap_planes = vm_check_cap(vm, KVM_CAP_PLANES);
+
+	kvm_vm_free(vm);
+	TEST_REQUIRE(cap_planes);
+
+	ksft_print_header();
+	ksft_set_plan(2);
+
+	pr_info("# KVM_CAP_PLANES: %d\n", cap_planes);
+
+	test_create_plane_errors(cap_planes);
+
+	if (cap_planes > 1)
+		test_create_plane();
+	else
+		ksft_test_result_skip("plane creation requires KVM_CAP_PLANES > 1\n");
+
+	ksft_finished();
+}
diff --git a/tools/testing/selftests/kvm/x86/plane_x86_test.c b/tools/testing/selftests/kvm/x86/plane_x86_test.c
new file mode 100644
index 000000000000..870be9c68050
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/plane_x86_test.c
@@ -0,0 +1,325 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2025 Red Hat, Inc.
+ *
+ * Test for x86-specific VM plane functionality
+ */
+#include <fcntl.h>
+#include <limits.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "test_util.h"
+
+#include "kvm_util.h"
+#include "processor.h"
+#include "apic.h"
+#include "asm/kvm.h"
+#include "linux/kvm.h"
+
+/*
+ * Port note (dwmw stack): KVM_CAP_PLANES is a VM-level opt-in that takes
+ * the plane count and must precede vCPU creation; planes are not
+ * always-on.  The KVM_CAP_PLANES_USERSPACE_SWITCH subtests of the
+ * upstream planes series (userspace plane switch, lower/not-runnable
+ * interrupt events, CPUID runtime bits, hugepage mixed attributes) are
+ * omitted: this kernel drives plane switches through the Hyper-V VSM
+ * hypercalls and has no userspace-switch capability.  The remaining
+ * subtests' assertions are unchanged.
+ */
+#define NR_PLANES 2
+
+static void enable_planes(struct kvm_vm *vm)
+{
+	vm_enable_cap(vm, KVM_CAP_PLANES, NR_PLANES);
+}
+
+static void test_plane_regs(void)
+{
+	struct kvm_vm *vm;
+	struct kvm_vcpu *vcpu;
+	struct kvm_plane *plane;
+	struct kvm_plane_vcpu *plane_vcpu;
+
+	struct kvm_regs regs0, regs1;
+
+	vm = vm_create_barebones_irqchip(true);
+	enable_planes(vm);
+	vcpu = __vm_vcpu_add(vm, 0);
+	plane = vm_plane_add(vm, 1);
+	plane_vcpu = __vm_plane_vcpu_add(vcpu, plane);
+
+	vcpu_ioctl(vcpu, KVM_GET_REGS, &regs0);
+	plane_vcpu_ioctl(plane_vcpu, KVM_GET_REGS, &regs1);
+	regs0.rax = 0x12345678;
+	regs1.rax = 0x87654321;
+
+	vcpu_ioctl(vcpu, KVM_SET_REGS, &regs0);
+	plane_vcpu_ioctl(plane_vcpu, KVM_SET_REGS, &regs1);
+
+	vcpu_ioctl(vcpu, KVM_GET_REGS, &regs0);
+	plane_vcpu_ioctl(plane_vcpu, KVM_GET_REGS, &regs1);
+	TEST_ASSERT_EQ(regs0.rax, 0x12345678);
+	TEST_ASSERT_EQ(regs1.rax, 0x87654321);
+
+	kvm_vm_free(vm);
+	ksft_test_result_pass("get/set regs for planes\n");
+}
+
+/*
+ * The planes of a VP share one TSC: a plane vCPU created later must take the
+ * TSC offset of its sibling, not start a new one.  Read the guest TSC on both
+ * fds back to back; the difference is the elapsed host time, not an offset.
+ */
+static void test_plane_tsc_offset(void)
+{
+	struct {
+		struct kvm_msrs header;
+		struct kvm_msr_entry entry;
+	} msrs0 = { .header.nmsrs = 1, .entry.index = MSR_IA32_TSC };
+	struct kvm_plane_vcpu *plane_vcpu;
+	struct kvm_plane *plane;
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	u64 tsc0, tsc1, bound;
+
+	vm = vm_create_barebones_irqchip(true);
+	enable_planes(vm);
+	vcpu = __vm_vcpu_add(vm, 0);
+	plane = vm_plane_add(vm, 1);
+	plane_vcpu = __vm_plane_vcpu_add(vcpu, plane);
+
+	TEST_ASSERT_EQ(__vcpu_ioctl(vcpu, KVM_GET_MSRS, &msrs0.header), 1);
+	tsc0 = msrs0.entry.data;
+	TEST_ASSERT_EQ(__plane_vcpu_ioctl(plane_vcpu, KVM_GET_MSRS, &msrs0.header), 1);
+	tsc1 = msrs0.entry.data;
+
+	/* One second of guest TSC; an unshared offset is off by far more. */
+	bound = (u64)__vcpu_ioctl(vcpu, KVM_GET_TSC_KHZ, NULL) * 1000;
+	TEST_ASSERT(tsc1 >= tsc0 && tsc1 - tsc0 < bound,
+		    "plane 1 TSC 0x%lx is not plane 0 TSC 0x%lx plus elapsed time",
+		    tsc1, tsc0);
+
+	kvm_vm_free(vm);
+	ksft_test_result_pass("plane vCPU shares the TSC offset\n");
+}
+
+/* Offset of XMM0 in the legacy XSAVE area.  */
+#define XSTATE_BV_OFFSET	(0x200/4)
+#define XMM_OFFSET		(0xa0/4)
+
+static void test_plane_fpu_nonshared(void)
+{
+	struct kvm_vm *vm;
+	struct kvm_vcpu *vcpu;
+	struct kvm_plane *plane;
+	struct kvm_plane_vcpu *plane_vcpu;
+
+	struct kvm_xsave xsave0, xsave1;
+
+	vm = vm_create_barebones_irqchip(true);
+	enable_planes(vm);
+
+	vcpu = __vm_vcpu_add(vm, 0);
+	vcpu_init_cpuid(vcpu, kvm_get_supported_cpuid());
+	vcpu_set_cpuid(vcpu);
+
+	plane = vm_plane_add(vm, 1);
+	plane_vcpu = __vm_plane_vcpu_add(vcpu, plane);
+
+	vcpu_ioctl(vcpu, KVM_GET_XSAVE, &xsave0);
+	xsave0.region[XSTATE_BV_OFFSET] |= XFEATURE_MASK_FP | XFEATURE_MASK_SSE;
+	xsave0.region[XMM_OFFSET] = 0x12345678;
+	vcpu_ioctl(vcpu, KVM_SET_XSAVE, &xsave0);
+
+	plane_vcpu_ioctl(plane_vcpu, KVM_GET_XSAVE, &xsave1);
+	xsave1.region[XSTATE_BV_OFFSET] |= XFEATURE_MASK_FP | XFEATURE_MASK_SSE;
+	xsave1.region[XMM_OFFSET] = 0x87654321;
+	plane_vcpu_ioctl(plane_vcpu, KVM_SET_XSAVE, &xsave1);
+
+	memset(&xsave0, 0, sizeof(xsave0));
+	vcpu_ioctl(vcpu, KVM_GET_XSAVE, &xsave0);
+	TEST_ASSERT_EQ(xsave0.region[XMM_OFFSET], 0x12345678);
+
+	memset(&xsave1, 0, sizeof(xsave0));
+	plane_vcpu_ioctl(plane_vcpu, KVM_GET_XSAVE, &xsave1);
+	TEST_ASSERT_EQ(xsave1.region[XMM_OFFSET], 0x87654321);
+
+	kvm_vm_free(vm);
+	ksft_test_result_pass("get/set FPU not shared across planes\n");
+}
+
+#define APIC_SPIV		0xF0
+#define APIC_IRR		0x200
+
+#define MYVEC			192
+
+/* Physical destination mode: destination id in address bits 19:12. */
+#define MAKE_MSI(cpu, vector) ((struct kvm_msi){			\
+	.address_lo = APIC_DEFAULT_GPA + (((cpu) & 0xff) << 12),	\
+	.address_hi = (cpu) & ~0xff,					\
+	.data = (vector),						\
+})
+
+static bool has_irr(struct kvm_lapic_state *apic, int vector)
+{
+	int word = vector >> 5;
+	int bit_in_word = vector & 31;
+	int bit = (APIC_IRR + word * 16) * CHAR_BIT + (bit_in_word & 31);
+
+	return apic->regs[bit >> 3] & (1 << (bit & 7));
+}
+
+static void do_enable_lapic(struct kvm_lapic_state *apic)
+{
+	/* set bit 8 */
+	apic->regs[APIC_SPIV + 1] |= 1;
+}
+
+static void test_plane_msi(void)
+{
+	struct kvm_vm *vm;
+	struct kvm_vcpu *vcpu;
+	struct kvm_plane *plane;
+	struct kvm_plane_vcpu *plane_vcpu;
+	int r;
+
+	struct kvm_msi msi = MAKE_MSI(0, MYVEC);
+	struct kvm_lapic_state lapic0, lapic1;
+
+	vm = vm_create_barebones_irqchip(true);
+	enable_planes(vm);
+
+	vcpu = __vm_vcpu_add(vm, 0);
+	vcpu_init_cpuid(vcpu, kvm_get_supported_cpuid());
+	vcpu_set_cpuid(vcpu);
+
+	plane = vm_plane_add(vm, 1);
+	plane_vcpu = __vm_plane_vcpu_add(vcpu, plane);
+
+	vcpu_set_msr(vcpu, MSR_IA32_APICBASE,
+		     APIC_DEFAULT_GPA | MSR_IA32_APICBASE_ENABLE | X2APIC_ENABLE);
+	vcpu_ioctl(vcpu, KVM_GET_LAPIC, &lapic0);
+	do_enable_lapic(&lapic0);
+	vcpu_ioctl(vcpu, KVM_SET_LAPIC, &lapic0);
+
+	_plane_vcpu_set_msr(plane_vcpu, MSR_IA32_APICBASE,
+			    APIC_DEFAULT_GPA | MSR_IA32_APICBASE_ENABLE | X2APIC_ENABLE);
+	plane_vcpu_ioctl(plane_vcpu, KVM_GET_LAPIC, &lapic1);
+	do_enable_lapic(&lapic1);
+	plane_vcpu_ioctl(plane_vcpu, KVM_SET_LAPIC, &lapic1);
+
+	/* Deliver to plane 1 (via the plane fd); it must land only in plane 1. */
+	r = __plane_ioctl(plane, KVM_SIGNAL_MSI, &msi);
+	TEST_ASSERT(r == 1,
+		   "Delivering interrupt to plane 1. ret: %d, errno: %d", r, errno);
+
+	vcpu_ioctl(vcpu, KVM_GET_LAPIC, &lapic0);
+	TEST_ASSERT(!has_irr(&lapic0, MYVEC), "Vector clear in plane 0");
+	plane_vcpu_ioctl(plane_vcpu, KVM_GET_LAPIC, &lapic1);
+	TEST_ASSERT(has_irr(&lapic1, MYVEC), "Vector set in plane 1");
+
+	/* Deliver to plane 0 (via the vm fd); it must land in plane 0. */
+	r = __vm_ioctl(vm, KVM_SIGNAL_MSI, &msi);
+	TEST_ASSERT(r == 1,
+		   "Delivering interrupt to plane 0. ret: %d, errno: %d", r, errno);
+	vcpu_ioctl(vcpu, KVM_GET_LAPIC, &lapic0);
+	TEST_ASSERT(has_irr(&lapic0, MYVEC), "Vector set in plane 0");
+
+	kvm_vm_free(vm);
+	ksft_test_result_pass("signal MSI routed per plane\n");
+}
+
+/*
+ * MP_STATE is per plane vCPU: a plane vCPU whose id is not the BSP's starts
+ * uninitialized while its plane-0 sibling runs; setting it does not touch the
+ * sibling; an INIT signalled to the plane's LAPIC is accepted on the plane fd.
+ */
+static void test_plane_mp_state(void)
+{
+	struct kvm_vm *vm;
+	struct kvm_vcpu *vcpu0, *vcpu1;
+	struct kvm_plane *plane;
+	struct kvm_plane_vcpu *plane_vcpu0, *plane_vcpu1;
+	struct kvm_mp_state mp_state;
+	struct kvm_msi init_msi = MAKE_MSI(1, 0);
+	int r;
+
+	vm = vm_create_barebones_irqchip(true);
+	enable_planes(vm);
+
+	vcpu0 = __vm_vcpu_add(vm, 0);
+	vcpu_init_cpuid(vcpu0, kvm_get_supported_cpuid());
+	vcpu_set_cpuid(vcpu0);
+	vcpu1 = __vm_vcpu_add(vm, 1);
+	vcpu_init_cpuid(vcpu1, kvm_get_supported_cpuid());
+	vcpu_set_cpuid(vcpu1);
+
+	plane = vm_plane_add(vm, 1);
+	plane_vcpu0 = __vm_plane_vcpu_add(vcpu0, plane);
+	plane_vcpu1 = __vm_plane_vcpu_add(vcpu1, plane);
+
+	/* BSP id starts runnable in every plane, the others uninitialized. */
+	plane_vcpu_ioctl(plane_vcpu0, KVM_GET_MP_STATE, &mp_state);
+	TEST_ASSERT_EQ(mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
+	plane_vcpu_ioctl(plane_vcpu1, KVM_GET_MP_STATE, &mp_state);
+	TEST_ASSERT_EQ(mp_state.mp_state, KVM_MP_STATE_UNINITIALIZED);
+	vcpu_ioctl(vcpu1, KVM_GET_MP_STATE, &mp_state);
+	TEST_ASSERT_EQ(mp_state.mp_state, KVM_MP_STATE_UNINITIALIZED);
+
+	/* Setting plane 1's state leaves plane 0's alone, and vice versa. */
+	mp_state.mp_state = KVM_MP_STATE_RUNNABLE;
+	plane_vcpu_ioctl(plane_vcpu1, KVM_SET_MP_STATE, &mp_state);
+	vcpu_ioctl(vcpu1, KVM_GET_MP_STATE, &mp_state);
+	TEST_ASSERT_EQ(mp_state.mp_state, KVM_MP_STATE_UNINITIALIZED);
+	plane_vcpu_ioctl(plane_vcpu1, KVM_GET_MP_STATE, &mp_state);
+	TEST_ASSERT_EQ(mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
+
+	mp_state.mp_state = KVM_MP_STATE_HALTED;
+	vcpu_ioctl(vcpu1, KVM_SET_MP_STATE, &mp_state);
+	plane_vcpu_ioctl(plane_vcpu1, KVM_GET_MP_STATE, &mp_state);
+	TEST_ASSERT_EQ(mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
+
+	mp_state.mp_state = KVM_MP_STATE_INIT_RECEIVED;
+	plane_vcpu_ioctl(plane_vcpu1, KVM_SET_MP_STATE, &mp_state);
+	plane_vcpu_ioctl(plane_vcpu1, KVM_GET_MP_STATE, &mp_state);
+	TEST_ASSERT_EQ(mp_state.mp_state, KVM_MP_STATE_INIT_RECEIVED);
+
+	/* An INIT to the plane's LAPIC is accepted on the plane fd. */
+	init_msi.data |= APIC_DM_INIT;
+	r = __plane_ioctl(plane, KVM_SIGNAL_MSI, &init_msi);
+	TEST_ASSERT(r == 1, "INIT to plane 1 vCPU 1. ret: %d, errno: %d", r, errno);
+
+	kvm_vm_free(vm);
+	ksft_test_result_pass("MP_STATE per plane vCPU\n");
+}
+
+int main(int argc, char *argv[])
+{
+	/*
+	 * kvm_arch_max_planes() reports 1 without a split irqchip, so the
+	 * probe VM needs one even though the opt-in itself does not.
+	 */
+	struct kvm_vm *vm = vm_create_barebones_irqchip(true);
+	int cap_planes = 0;
+
+	if (!__vm_enable_cap(vm, KVM_CAP_PLANES, NR_PLANES))
+		cap_planes = vm_check_cap(vm, KVM_CAP_PLANES);
+
+	kvm_vm_free(vm);
+	TEST_REQUIRE(cap_planes > 1);
+
+	ksft_print_header();
+	ksft_set_plan(5);
+
+	pr_info("# KVM_CAP_PLANES: %d\n", cap_planes);
+
+	test_plane_regs();
+	test_plane_tsc_offset();
+	test_plane_fpu_nonshared();
+	test_plane_msi();
+	test_plane_mp_state();
+
+	ksft_finished();
+}
-- 
2.50.1

[RFC PATCH 04/55] KVM: selftests: Add x86 planes lifecycle test

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Cover KVM_CAP_PLANES argument validation and ordering, plane creation
bounds, plane vCPU creation, and per-plane isolation of vCPU register
state.

The tools copy of linux/kvm.h is synced to pick up the VM planes
definitions the test builds against.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 tools/include/uapi/linux/kvm.h                |  40 +++++-
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 tools/testing/selftests/kvm/x86/planes_test.c | 120 ++++++++++++++++++
 3 files changed, 156 insertions(+), 5 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86/planes_test.c

diff --git a/tools/include/uapi/linux/kvm.h b/tools/include/uapi/linux/kvm.h
index fe2f500c932d..dcd12f58186e 100644
--- a/tools/include/uapi/linux/kvm.h
+++ b/tools/include/uapi/linux/kvm.h
@@ -11,6 +11,7 @@
 #include <linux/const.h>
 #include <linux/types.h>
 #include <linux/compiler.h>
+#include <linux/stddef.h>
 #include <linux/ioctl.h>
 #include <asm/kvm.h>
 
@@ -145,6 +146,13 @@ struct kvm_exit_snp_req_certs {
 	__u64 ret;
 };
 
+struct kvm_plane_event_exit {
+#define KVM_PLANE_EVENT_CREATE_VCPU	1
+	__u32 cause;
+	__u32 plane;
+	__u64 extra[8];
+};
+
 #define KVM_S390_GET_SKEYS_NONE   1
 #define KVM_S390_SKEYS_MAX        1048576
 
@@ -192,6 +200,7 @@ struct kvm_exit_snp_req_certs {
 #define KVM_EXIT_ARM_SEA          41
 #define KVM_EXIT_ARM_LDST64B      42
 #define KVM_EXIT_SNP_REQ_CERTS    43
+#define KVM_EXIT_PLANE_EVENT      44
 
 /* For KVM_EXIT_INTERNAL_ERROR */
 /* Emulate instruction failed. */
@@ -456,6 +465,9 @@ struct kvm_run {
 		} notify;
 		/* KVM_EXIT_MEMORY_FAULT */
 		struct {
+#define KVM_MEMORY_EXIT_FLAG_READ       (1ULL << 0)
+#define KVM_MEMORY_EXIT_FLAG_WRITE      (1ULL << 1)
+#define KVM_MEMORY_EXIT_FLAG_EXEC       (1ULL << 2)
 #define KVM_MEMORY_EXIT_FLAG_PRIVATE	(1ULL << 3)
 			__u64 flags;
 			__u64 gpa;
@@ -496,6 +508,8 @@ struct kvm_run {
 		} arm_sea;
 		/* KVM_EXIT_SNP_REQ_CERTS */
 		struct kvm_exit_snp_req_certs snp_req_certs;
+		/* KVM_EXIT_PLANE_EVENT */
+		struct kvm_plane_event_exit plane_event;
 		/* Fix the size of the union. */
 		char padding[256];
 	};
@@ -542,7 +556,7 @@ struct kvm_coalesced_mmio {
 
 struct kvm_coalesced_mmio_ring {
 	__u32 first, last;
-	struct kvm_coalesced_mmio coalesced_mmio[];
+	__DECLARE_FLEX_ARRAY(struct kvm_coalesced_mmio, coalesced_mmio);
 };
 
 #define KVM_COALESCED_MMIO_MAX \
@@ -592,7 +606,7 @@ struct kvm_clear_dirty_log {
 /* for KVM_SET_SIGNAL_MASK */
 struct kvm_signal_mask {
 	__u32 len;
-	__u8  sigset[];
+	__DECLARE_FLEX_ARRAY(__u8, sigset);
 };
 
 /* for KVM_TPR_ACCESS_REPORTING */
@@ -703,6 +717,11 @@ struct kvm_enable_cap {
 #define KVM_VM_TYPE_ARM_IPA_SIZE_MASK	0xffULL
 #define KVM_VM_TYPE_ARM_IPA_SIZE(x)		\
 	((x) & KVM_VM_TYPE_ARM_IPA_SIZE_MASK)
+
+#define KVM_VM_TYPE_ARM_PROTECTED	(1UL << 31)
+#define KVM_VM_TYPE_ARM_MASK		(KVM_VM_TYPE_ARM_IPA_SIZE_MASK | \
+					 KVM_VM_TYPE_ARM_PROTECTED)
+
 /*
  * ioctls for /dev/kvm fds:
  */
@@ -724,6 +743,11 @@ struct kvm_enable_cap {
 #define KVM_GET_EMULATED_CPUID	  _IOWR(KVMIO, 0x09, struct kvm_cpuid2)
 #define KVM_GET_MSR_FEATURE_INDEX_LIST    _IOWR(KVMIO, 0x0a, struct kvm_msr_list)
 
+/*
+ * Maximum number of supported planes
+ */
+#define KVM_MAX_PLANES	16
+
 /*
  * Extension capability list.
  */
@@ -989,6 +1013,8 @@ struct kvm_enable_cap {
 #define KVM_CAP_ARM_SEA_TO_USER 245
 #define KVM_CAP_S390_USER_OPEREXEC 246
 #define KVM_CAP_S390_KEYOP 247
+#define KVM_CAP_S390_VSIE_ESAMODE 248
+#define KVM_CAP_PLANES 249
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
@@ -1051,7 +1077,7 @@ struct kvm_irq_routing_entry {
 struct kvm_irq_routing {
 	__u32 nr;
 	__u32 flags;
-	struct kvm_irq_routing_entry entries[];
+	__DECLARE_FLEX_ARRAY(struct kvm_irq_routing_entry, entries);
 };
 
 #define KVM_IRQFD_FLAG_DEASSIGN (1 << 0)
@@ -1142,7 +1168,7 @@ struct kvm_dirty_tlb {
 
 struct kvm_reg_list {
 	__u64 n; /* number of regs */
-	__u64 reg[];
+	__DECLARE_FLEX_ARRAY(__u64, reg);
 };
 
 struct kvm_one_reg {
@@ -1224,6 +1250,8 @@ enum kvm_device_type {
 #define KVM_DEV_TYPE_LOONGARCH_EIOINTC	KVM_DEV_TYPE_LOONGARCH_EIOINTC
 	KVM_DEV_TYPE_LOONGARCH_PCHPIC,
 #define KVM_DEV_TYPE_LOONGARCH_PCHPIC	KVM_DEV_TYPE_LOONGARCH_PCHPIC
+	KVM_DEV_TYPE_LOONGARCH_DMSINTC,
+#define KVM_DEV_TYPE_LOONGARCH_DMSINTC	KVM_DEV_TYPE_LOONGARCH_DMSINTC
 	KVM_DEV_TYPE_ARM_VGIC_V5,
 #define KVM_DEV_TYPE_ARM_VGIC_V5	KVM_DEV_TYPE_ARM_VGIC_V5
 
@@ -1340,6 +1368,8 @@ struct kvm_s390_keyop {
 #define KVM_GET_DEVICE_ATTR	  _IOW(KVMIO,  0xe2, struct kvm_device_attr)
 #define KVM_HAS_DEVICE_ATTR	  _IOW(KVMIO,  0xe3, struct kvm_device_attr)
 
+#define KVM_CREATE_PLANE	  _IO(KVMIO, 0xe4)
+
 /*
  * ioctls for vcpu fds
  */
@@ -1610,7 +1640,7 @@ struct kvm_stats_desc {
 #ifdef __KERNEL__
 	char name[KVM_STATS_NAME_SIZE];
 #else
-	char name[];
+	__DECLARE_FLEX_ARRAY(char, name);
 #endif
 };
 
diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 61e767a1e31b..5883359c1a5d 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -104,6 +104,7 @@ TEST_GEN_PROGS_x86 += x86/nested_tsc_scaling_test
 TEST_GEN_PROGS_x86 += x86/nested_vmsave_vmload_test
 TEST_GEN_PROGS_x86 += x86/platform_info_test
 TEST_GEN_PROGS_x86 += x86/plane_x86_test
+TEST_GEN_PROGS_x86 += x86/planes_test
 TEST_GEN_PROGS_x86 += x86/pmu_counters_test
 TEST_GEN_PROGS_x86 += x86/pmu_event_filter_test
 TEST_GEN_PROGS_x86 += x86/private_mem_conversions_test
diff --git a/tools/testing/selftests/kvm/x86/planes_test.c b/tools/testing/selftests/kvm/x86/planes_test.c
new file mode 100644
index 000000000000..3517175580d3
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/planes_test.c
@@ -0,0 +1,120 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Test the KVM_CAP_PLANES opt-in and the plane lifecycle uAPI: plane
+ * creation bounds, plane vCPU creation, and the independence of
+ * per-plane vCPU register state.
+ */
+#include <errno.h>
+#include <fcntl.h>
+#include <string.h>
+#include <unistd.h>
+
+#include "kvm_util.h"
+
+static int plane_vcpu_fd(int plane_fd, int vcpu_id)
+{
+	return __kvm_ioctl(plane_fd, KVM_CREATE_VCPU, (void *)(unsigned long)vcpu_id);
+}
+
+static void test_enable_cap_bounds(void)
+{
+	struct kvm_vm *vm = vm_create_barebones();
+
+	/* Planes require a split irqchip; the cap itself does not. */
+	TEST_ASSERT_EQ(__vm_enable_cap(vm, KVM_CAP_PLANES, 0), -1);
+	TEST_ASSERT_EQ(errno, EINVAL);
+
+	TEST_ASSERT_EQ(__vm_enable_cap(vm, KVM_CAP_PLANES, KVM_MAX_PLANES + 1), -1);
+	TEST_ASSERT_EQ(errno, EINVAL);
+
+	vm_enable_cap(vm, KVM_CAP_PLANES, KVM_MAX_PLANES);
+
+	kvm_vm_free(vm);
+}
+
+static void test_enable_cap_too_late(void)
+{
+	struct kvm_vm *vm = vm_create_barebones();
+	int vcpu_fd;
+
+	vcpu_fd = __vm_ioctl(vm, KVM_CREATE_VCPU, NULL);
+	TEST_ASSERT(vcpu_fd >= 0, "KVM_CREATE_VCPU failed: errno %d", errno);
+
+	TEST_ASSERT_EQ(__vm_enable_cap(vm, KVM_CAP_PLANES, 2), -1);
+	TEST_ASSERT_EQ(errno, EEXIST);
+
+	close(vcpu_fd);
+	kvm_vm_free(vm);
+}
+
+static void test_plane_lifecycle(void)
+{
+	struct kvm_vm *vm = vm_create_barebones();
+	struct kvm_regs regs0, regs1;
+	int plane_fd, vcpu0_fd, vcpu1_fd;
+	int r;
+
+	vm_enable_cap(vm, KVM_CAP_SPLIT_IRQCHIP, 24);
+
+	/* Without the opt-in a VM has a single plane. */
+	TEST_ASSERT_EQ(vm_check_cap(vm, KVM_CAP_PLANES), 1);
+	TEST_ASSERT_EQ(__vm_ioctl(vm, KVM_CREATE_PLANE, (void *)1UL), -1);
+	TEST_ASSERT_EQ(errno, EINVAL);
+
+	vm_enable_cap(vm, KVM_CAP_PLANES, 2);
+	TEST_ASSERT_EQ(vm_check_cap(vm, KVM_CAP_PLANES), 2);
+
+	/* The configured count is a hard bound, and reconfiguration a hard no. */
+	TEST_ASSERT_EQ(__vm_ioctl(vm, KVM_CREATE_PLANE, (void *)2UL), -1);
+	TEST_ASSERT_EQ(errno, EINVAL);
+
+	plane_fd = __vm_ioctl(vm, KVM_CREATE_PLANE, (void *)1UL);
+	TEST_ASSERT(plane_fd >= 0, "KVM_CREATE_PLANE failed: errno %d", errno);
+
+	TEST_ASSERT_EQ(__vm_enable_cap(vm, KVM_CAP_PLANES, 3), -1);
+	TEST_ASSERT_EQ(errno, EEXIST);
+
+	/*
+	 * A plane cannot host planes of its own.  Plane fds that do not
+	 * implement KVM_CHECK_EXTENSION at all satisfy that trivially.
+	 */
+	r = __kvm_ioctl(plane_fd, KVM_CHECK_EXTENSION, (void *)KVM_CAP_PLANES);
+	TEST_ASSERT(r == 0 || (r == -1 && errno == ENOTTY),
+		    "plane fd reports nested plane support: %d", r);
+
+	vcpu0_fd = __vm_ioctl(vm, KVM_CREATE_VCPU, NULL);
+	TEST_ASSERT(vcpu0_fd >= 0, "KVM_CREATE_VCPU failed: errno %d", errno);
+
+	vcpu1_fd = plane_vcpu_fd(plane_fd, 0);
+	TEST_ASSERT(vcpu1_fd >= 0,
+		    "KVM_CREATE_VCPU on plane 1 failed: errno %d", errno);
+
+	/* Register state is per plane: write plane 1, plane 0 unaffected. */
+	kvm_ioctl(vcpu1_fd, KVM_GET_REGS, &regs1);
+	regs1.rbx = 0x5157ca11;	/* arbitrary marker */
+	kvm_ioctl(vcpu1_fd, KVM_SET_REGS, &regs1);
+
+	kvm_ioctl(vcpu0_fd, KVM_GET_REGS, &regs0);
+	TEST_ASSERT(regs0.rbx != 0x5157ca11,
+		    "plane 0 RBX changed by plane 1 KVM_SET_REGS");
+
+	memset(&regs1, 0, sizeof(regs1));
+	kvm_ioctl(vcpu1_fd, KVM_GET_REGS, &regs1);
+	TEST_ASSERT_EQ(regs1.rbx, 0x5157ca11);
+
+	close(vcpu1_fd);
+	close(vcpu0_fd);
+	close(plane_fd);
+	kvm_vm_free(vm);
+}
+
+int main(int argc, char *argv[])
+{
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_PLANES));
+
+	test_enable_cap_bounds();
+	test_enable_cap_too_late();
+	test_plane_lifecycle();
+
+	return 0;
+}
-- 
2.50.1

[RFC PATCH 05/55] KVM: x86: hyper-v: Add KVM_CAP_HYPERV_FORWARD_HCALL

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

A VMM implementing Hyper-V hypercalls that KVM does not know about,
such as the VSM set, has no way to receive them: kvm_hv_hypercall()
fails unknown codes with HV_STATUS_INVALID_HYPERCALL_CODE, and only
syndbg and slow-form extended hypercalls ever exit to userspace.

Add a VM capability which routes any hypercall KVM would have failed
as unknown out through the existing KVM_EXIT_HYPERV_HCALL exit, fast
forms included. The exit already carries the control value and both
parameters (RDX and R8 for fast calls), and XMM state is reachable
through the ordinary vCPU ioctls, so no new exit format is needed.
A future in-kernel accelerator for a specific hypercall can still
claim its code behind its own capability, with userspace handling as
the fallback.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 Documentation/virt/kvm/api.rst  | 26 ++++++++++++++++++++++++++
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/hyperv.c           |  5 ++++-
 arch/x86/kvm/x86.c              | 10 ++++++++++
 include/uapi/linux/kvm.h        |  1 +
 5 files changed, 42 insertions(+), 1 deletion(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index bc63054ce7bd..ade8c91ef24b 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -8986,6 +8986,32 @@ This capability must be enabled before any vCPU or extra plane has been
 created, and the chosen count cannot be changed afterwards; otherwise
 -EEXIST is returned.
 
+7.48 KVM_CAP_HYPERV_FORWARD_HCALL
+---------------------------------
+
+:Architectures: x86
+:Target: VM
+:Parameters: args[0] is 1 to enable forwarding, 0 to disable it
+
+When enabled, a Hyper-V hypercall that KVM would otherwise have failed
+with HV_STATUS_INVALID_HYPERCALL_CODE instead exits to userspace with
+KVM_EXIT_HYPERV_HCALL, and extended hypercalls in fast form exit the
+same way instead of failing with HV_STATUS_INVALID_PARAMETER.  This
+lets a VMM implement hypercalls KVM does not know, such as the Hyper-V
+VSM set.  The synthetic-debugger hypercalls (HvCallResetDebugSession,
+HvCallPostDebugData, HvCallRetrieveDebugData) are an exception: while
+the synthetic debugger is not enabled they still fail in KVM with
+HV_STATUS_INVALID_HYPERCALL_CODE and are not forwarded.
+
+The exit carries the control value and the input and output parameters
+(for fast hypercalls, RDX and R8 from a 64-bit caller, or the EBX:ECX
+and EDI:ESI pairs from a 32-bit caller).  Register argument and
+return state of fast hypercalls, such as the XMM registers, is read and
+written with the ordinary vCPU state ioctls before resuming the vCPU
+with ``hyperv.u.hcall.result`` set.  Hypercalls KVM implements itself,
+and the ordering of CPUID-enforcement checks against hypercall
+dispatch, are unaffected.
+
 7.49 KVM_CAP_HCALL_FAULT_EXIT
 ------------------------------------
 
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index e4c8691dc1e1..05670b5ae28a 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1204,6 +1204,7 @@ struct kvm_hv {
 	struct mutex hv_lock;
 	u64 hv_guest_os_id;
 	u64 hv_hypercall;
+	bool hv_forward_hcalls;
 	u64 hv_tsc_page;
 	enum hv_tsc_page_status hv_tsc_page_status;
 
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 98a3c2aefdcd..ec669dafda6e 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -2777,12 +2777,15 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 		return kvm_hv_hypercall_userspace_exit(vcpu, &hc);
 	}
 	case HV_EXT_CALL_QUERY_CAPABILITIES ... HV_EXT_CALL_MAX:
-		if (unlikely(hc.fast)) {
+		if (unlikely(hc.fast) &&
+		    !to_kvm_hv(vcpu->kvm)->hv_forward_hcalls) {
 			ret = HV_STATUS_INVALID_PARAMETER;
 			break;
 		}
 		return kvm_hv_hypercall_userspace_exit(vcpu, &hc);
 	default:
+		if (to_kvm_hv(vcpu->kvm)->hv_forward_hcalls)
+			return kvm_hv_hypercall_userspace_exit(vcpu, &hc);
 		ret = HV_STATUS_INVALID_HYPERCALL_CODE;
 		break;
 	}
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index a9073436c734..8de596e00441 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -4883,6 +4883,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_HYPERV_CPUID:
 	case KVM_CAP_HYPERV_ENFORCE_CPUID:
 	case KVM_CAP_SYS_HYPERV_CPUID:
+	case KVM_CAP_HYPERV_FORWARD_HCALL:
 #endif
 	case KVM_CAP_HCALL_FAULT_EXIT:
 	case KVM_CAP_PCI_SEGMENT:
@@ -6886,6 +6887,15 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		mutex_unlock(&kvm->lock);
 		break;
 	}
+#ifdef CONFIG_KVM_HYPERV
+	case KVM_CAP_HYPERV_FORWARD_HCALL:
+		r = -EINVAL;
+		if (cap->args[0] & ~1ULL)
+			break;
+		to_kvm_hv(kvm)->hv_forward_hcalls = !!cap->args[0];
+		r = 0;
+		break;
+#endif
 	case KVM_CAP_X2APIC_API:
 		r = -EINVAL;
 		if (cap->args[0] & ~KVM_X2APIC_API_VALID_FLAGS)
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 839e9e83d21d..5567cf352907 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1015,6 +1015,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_S390_KEYOP 247
 #define KVM_CAP_S390_VSIE_ESAMODE 248
 #define KVM_CAP_PLANES 249
+#define KVM_CAP_HYPERV_FORWARD_HCALL 250
 #define KVM_CAP_HCALL_FAULT_EXIT 252
 
 struct kvm_irq_routing_irqchip {
-- 
2.50.1

[RFC PATCH 06/55] KVM: selftests: Add Hyper-V hypercall forwarding test

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Drive an unimplemented hypercall code through the three interesting
shapes: forwarding disabled (the guest sees
HV_STATUS_INVALID_HYPERCALL_CODE from KVM itself), a slow-form call
(the exit carries the control value and GPAs, and the userspace-chosen
result reaches the guest), and a fast-form call (RDX and R8 in the
exit parameters, XMM0 visible through KVM_GET_FPU).

The tools copy of linux/kvm.h picks up the KVM_CAP_HYPERV_FORWARD_HCALL
capability number for the test.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 tools/include/uapi/linux/kvm.h                |   1 +
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/x86/hyperv_forward_hcall.c  | 144 ++++++++++++++++++
 3 files changed, 146 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/hyperv_forward_hcall.c

diff --git a/tools/include/uapi/linux/kvm.h b/tools/include/uapi/linux/kvm.h
index dcd12f58186e..586c0d6c168c 100644
--- a/tools/include/uapi/linux/kvm.h
+++ b/tools/include/uapi/linux/kvm.h
@@ -1015,6 +1015,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_S390_KEYOP 247
 #define KVM_CAP_S390_VSIE_ESAMODE 248
 #define KVM_CAP_PLANES 249
+#define KVM_CAP_HYPERV_FORWARD_HCALL 250
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 5883359c1a5d..14e385750ed1 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -85,6 +85,7 @@ TEST_GEN_PROGS_x86 += x86/hyperv_cpuid
 TEST_GEN_PROGS_x86 += x86/hyperv_evmcs
 TEST_GEN_PROGS_x86 += x86/hyperv_extended_hypercalls
 TEST_GEN_PROGS_x86 += x86/hyperv_features
+TEST_GEN_PROGS_x86 += x86/hyperv_forward_hcall
 TEST_GEN_PROGS_x86 += x86/hyperv_ipi
 TEST_GEN_PROGS_x86 += x86/hyperv_svm_test
 TEST_GEN_PROGS_x86 += x86/hyperv_tlb_flush
diff --git a/tools/testing/selftests/kvm/x86/hyperv_forward_hcall.c b/tools/testing/selftests/kvm/x86/hyperv_forward_hcall.c
new file mode 100644
index 000000000000..47f786ca3e4b
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/hyperv_forward_hcall.c
@@ -0,0 +1,144 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Test KVM_CAP_HYPERV_FORWARD_HCALL: Hyper-V hypercalls that KVM does
+ * not implement exit to userspace instead of failing with
+ * HV_STATUS_INVALID_HYPERCALL_CODE, in both slow and fast form, and
+ * userspace supplies the result the guest observes.
+ */
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+#include "hyperv.h"
+
+/* A hypercall code KVM does not implement: HvCallGetVpRegisters. */
+#define HVCALL_TEST_CODE	0x50
+
+#define FAST_PARAM0		0x1122334455667788ULL
+#define FAST_PARAM1		0x99aabbccddeeff00ULL
+#define XMM_MARKER		0x5157ca11c0ffee00ULL
+#define USERSPACE_STATUS	0x55AA
+
+static void guest_code(gpa_t pgs_gpa)
+{
+	__sse128_u marker;
+	u64 status;
+	u8 vector;
+
+	marker.as_u64[0] = XMM_MARKER;
+	marker.as_u64[1] = XMM_MARKER + 1;
+
+	wrmsr(HV_X64_MSR_GUEST_OS_ID, HYPERV_LINUX_OS_ID);
+	wrmsr(HV_X64_MSR_HYPERCALL, pgs_gpa);
+
+	/* Forwarding disabled: KVM itself fails the call. */
+	vector = __hyperv_hypercall(HVCALL_TEST_CODE, pgs_gpa,
+				    pgs_gpa + PAGE_SIZE, &status);
+	GUEST_ASSERT(!vector);
+	GUEST_ASSERT_EQ(status & 0xffff, HV_STATUS_INVALID_HYPERCALL_CODE);
+	GUEST_SYNC(1);
+
+	/* Forwarding enabled: userspace supplies the status. */
+	vector = __hyperv_hypercall(HVCALL_TEST_CODE, pgs_gpa,
+				    pgs_gpa + PAGE_SIZE, &status);
+	GUEST_ASSERT(!vector);
+	GUEST_ASSERT_EQ(status & 0xffff, 0);
+	GUEST_SYNC(2);
+
+	/* Fast form: register parameters reach userspace. */
+	write_sse_reg(0, &marker.vec);
+	vector = __hyperv_hypercall(HVCALL_TEST_CODE | HV_HYPERCALL_FAST_BIT,
+				    FAST_PARAM0, FAST_PARAM1, &status);
+	GUEST_ASSERT(!vector);
+	GUEST_ASSERT_EQ(status & 0xffff, USERSPACE_STATUS);
+
+	GUEST_DONE();
+}
+
+static void handle_forwarded_hcall(struct kvm_vcpu *vcpu, u64 expect_control,
+				   u64 expect_in, u64 expect_out, u64 result)
+{
+	struct kvm_run *run = vcpu->run;
+
+	TEST_ASSERT_EQ(run->exit_reason, KVM_EXIT_HYPERV);
+	TEST_ASSERT_EQ(run->hyperv.type, KVM_EXIT_HYPERV_HCALL);
+	TEST_ASSERT_EQ(run->hyperv.u.hcall.input, expect_control);
+	TEST_ASSERT_EQ(run->hyperv.u.hcall.params[0], expect_in);
+	TEST_ASSERT_EQ(run->hyperv.u.hcall.params[1], expect_out);
+
+	run->hyperv.u.hcall.result = result;
+}
+
+int main(int argc, char *argv[])
+{
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	gva_t hcall_page;
+	gpa_t hcall_gpa;
+	struct kvm_fpu fpu;
+	struct ucall uc;
+	int stage;
+
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_HYPERV_FORWARD_HCALL));
+
+	vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+	vcpu_set_hv_cpuid(vcpu);
+
+	/* Hypercall page plus an input and an output page. */
+	hcall_page = vm_alloc_pages(vm, 3);
+	memset(addr_gva2hva(vm, hcall_page), 0x0, 3 * getpagesize());
+	hcall_gpa = addr_gva2gpa(vm, hcall_page);
+
+	vcpu_args_set(vcpu, 1, hcall_gpa);
+
+	for (stage = 1;; stage++) {
+		vcpu_run(vcpu);
+
+		if (vcpu->run->exit_reason == KVM_EXIT_HYPERV) {
+			switch (stage) {
+			case 2:
+				handle_forwarded_hcall(vcpu, HVCALL_TEST_CODE,
+						       hcall_gpa,
+						       hcall_gpa + getpagesize(),
+						       0);
+				break;
+			case 3:
+				handle_forwarded_hcall(vcpu,
+						       HVCALL_TEST_CODE | HV_HYPERCALL_FAST_BIT,
+						       FAST_PARAM0, FAST_PARAM1,
+						       USERSPACE_STATUS);
+				/*
+				 * Fast-call register state is reachable with
+				 * the ordinary vCPU state ioctls.
+				 */
+				vcpu_fpu_get(vcpu, &fpu);
+				TEST_ASSERT_EQ(*(u64 *)&fpu.xmm[0][0], XMM_MARKER);
+				TEST_ASSERT_EQ(*(u64 *)&fpu.xmm[0][8], XMM_MARKER + 1);
+				break;
+			default:
+				TEST_FAIL("Unexpected hypercall exit in stage %d", stage);
+			}
+			stage--;	/* same guest stage continues */
+			continue;
+		}
+
+		TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+
+		switch (get_ucall(vcpu, &uc)) {
+		case UCALL_SYNC:
+			TEST_ASSERT_EQ(uc.args[1], stage);
+			if (stage == 1)
+				vm_enable_cap(vm, KVM_CAP_HYPERV_FORWARD_HCALL, 1);
+			break;
+		case UCALL_ABORT:
+			REPORT_GUEST_ASSERT(uc);
+		case UCALL_DONE:
+			goto done;
+		default:
+			TEST_FAIL("Unhandled ucall: %ld", uc.cmd);
+		}
+	}
+
+done:
+	kvm_vm_free(vm);
+	return 0;
+}
-- 
2.50.1

[RFC PATCH 07/55] KVM: Add KVM_GET/SET_PLANE_RUN_STATE vCPU ioctls

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

A VMM doing VSM in userspace has to switch the executing plane when
the guest asks for a privilege transition, but a vCPU's plane run
state is kernel-internal — set only by the SEV-SNP VMPL switch,
plane-0 event delivery, and vCPU creation. A VTL call forwarded to
userspace is a dead end.

Add ioctls to read and write the run state as stopped/runnable.
KVM_RUN already executes the lowest-numbered runnable plane, so a
userspace plane switch is just: mark the target plane's vCPU runnable,
mark the current one stopped, re-enter KVM_RUN.

A plane 0 stopped this way *stays* stopped. Forcing it runnable on a
pending event fits the SEV-SNP model, but under a userspace privilege
model plane 0 would preempt the higher VTL on any interrupt and resume
mid-hypercall with whatever completion value the shared kvm_run last
held; the events stay pending until the VMM switches back. In-kernel
transitions never set the stopped-by-userspace flag, so SNP behaviour
is unchanged.

The ioctls return -EINVAL unless kvm_arch_max_planes() reports plane
support, the same condition KVM_CREATE_PLANE enforces: only x86
consumes KVM_REQ_PLANE_RESCHED, and on the other architectures a stray
request bit would abort every subsequent entry attempt, leaving KVM_RUN
spinning at 100% with no way for the VMM to recover the vCPU.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 Documentation/virt/kvm/api.rst | 26 ++++++++++++++++++++
 arch/x86/kvm/x86.c             |  3 ++-
 include/linux/kvm_host.h       |  6 +++++
 include/uapi/linux/kvm.h       |  4 ++++
 virt/kvm/kvm_main.c            | 44 ++++++++++++++++++++++++++++++++++
 5 files changed, 82 insertions(+), 1 deletion(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index ade8c91ef24b..f8d9bca1baea 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6611,6 +6611,32 @@ in the future may use the plane id when planes are supported by hardware (as is
 the case for VMPLs in AMD), or if KVM supports accelerated plane switch
 operations (as might be the case for Hyper-V VTLs).
 
+4.146 KVM_GET_PLANE_RUN_STATE / KVM_SET_PLANE_RUN_STATE
+--------------------------------------------------------
+
+:Capability: KVM_CAP_PLANES
+:Architectures: x86
+:Type: vcpu ioctl
+:Parameters: pointer to a __u32 run state
+:Returns: 0 on success, <0 on error
+
+Reads or sets the per-plane run state of a vCPU:
+KVM_PLANE_RUN_STATE_STOPPED (0) or KVM_PLANE_RUN_STATE_RUNNABLE (1).
+
+When a vCPU thread calls KVM_RUN, KVM executes the lowest-numbered plane
+whose vCPU is runnable, falling back to plane 0.  A VMM implementing a
+software privilege model (such as Hyper-V VSM) switches planes by
+marking the target plane's vCPU runnable and the current one stopped
+before re-entering KVM_RUN.  Pending events for plane 0 make it runnable
+again on their own only as long as userspace has not explicitly stopped
+it with this ioctl; a VMM-stopped plane 0 keeps its events pending until
+the VMM makes it runnable, so a software privilege model can keep a
+higher plane running undisturbed.  The ioctl is accepted on any plane's
+vCPU fd.
+
+The run state is distinct from KVM_GET/SET_MP_STATE, which describes
+the architectural state of a vCPU within its plane.
+
 5. The kvm_run structure
 ========================
 
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 8de596e00441..ca5d8e081fdc 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -11508,7 +11508,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 		}
 	}
 
-	if (kvm_check_plane0_events(vcpu)) {
+	if (kvm_check_plane0_events(vcpu) &&
+	    !vcpu->common->vcpus[0]->plane_user_stopped) {
 		kvm_vcpu_set_plane_runnable(vcpu->common->vcpus[0]);
 
 		kvm_make_request(KVM_REQ_EVENT, vcpu);
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 7ed79479b4a1..5bdae2127578 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -413,6 +413,12 @@ struct kvm_vcpu {
 
 	u64 plane_requests;
 	enum kvm_vcpu_state plane_state;
+	/*
+	 * Set when userspace stopped this plane with
+	 * KVM_SET_PLANE_RUN_STATE: pending plane-0 events then no longer
+	 * make plane 0 runnable on their own, the VMM owns the switch.
+	 */
+	bool plane_user_stopped;
 
 	/* S390 only */
 	bool valid_wakeup;
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 5567cf352907..2aeb8c5172cb 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1371,6 +1371,10 @@ struct kvm_s390_keyop {
 #define KVM_HAS_DEVICE_ATTR	  _IOW(KVMIO,  0xe3, struct kvm_device_attr)
 
 #define KVM_CREATE_PLANE	  _IO(KVMIO, 0xe4)
+#define KVM_GET_PLANE_RUN_STATE	  _IOR(KVMIO, 0xe5, __u32)
+#define KVM_SET_PLANE_RUN_STATE	  _IOW(KVMIO, 0xe6, __u32)
+#define   KVM_PLANE_RUN_STATE_STOPPED	0
+#define   KVM_PLANE_RUN_STATE_RUNNABLE	1
 
 /*
  * ioctls for vcpu fds
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index c8b0722ac35c..2d2dc7828e1f 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -4788,6 +4788,8 @@ static inline bool kvm_is_vcpu_plane_ioctl(unsigned ioctl)
 	case KVM_GET_SREGS:
 	case KVM_SET_SREGS:
 	case KVM_TRANSLATE:
+	case KVM_GET_PLANE_RUN_STATE:
+	case KVM_SET_PLANE_RUN_STATE:
 		return true;
 	default:
 		return kvm_arch_is_vcpu_plane_ioctl(ioctl);
@@ -4928,6 +4930,48 @@ static long kvm_vcpu_ioctl(struct file *filp,
 		r = kvm_arch_vcpu_ioctl_set_sregs(vcpu, kvm_sregs);
 		break;
 	}
+	case KVM_GET_PLANE_RUN_STATE: {
+		u32 state = vcpu->plane_state == RUNNABLE ?
+			    KVM_PLANE_RUN_STATE_RUNNABLE :
+			    KVM_PLANE_RUN_STATE_STOPPED;
+
+		r = -EINVAL;
+		if (kvm_arch_max_planes(vcpu->kvm) <= 1)
+			goto out;
+		r = -EFAULT;
+		if (copy_to_user(argp, &state, sizeof(state)))
+			goto out;
+		r = 0;
+		break;
+	}
+	case KVM_SET_PLANE_RUN_STATE: {
+		u32 state;
+
+		/*
+		 * KVM_SET_PLANE_RUN_STATE raises KVM_REQ_PLANE_RESCHED,
+		 * which only plane-aware architecture code consumes; on a
+		 * planes-less VM the stuck request would abort every guest
+		 * entry and spin KVM_RUN at 100%.
+		 */
+		r = -EINVAL;
+		if (kvm_arch_max_planes(vcpu->kvm) <= 1)
+			goto out;
+		r = -EFAULT;
+		if (copy_from_user(&state, argp, sizeof(state)))
+			goto out;
+		r = -EINVAL;
+		if (state > KVM_PLANE_RUN_STATE_RUNNABLE)
+			goto out;
+		if (state == KVM_PLANE_RUN_STATE_RUNNABLE) {
+			vcpu->plane_user_stopped = false;
+			kvm_vcpu_set_plane_runnable(vcpu);
+		} else {
+			vcpu->plane_user_stopped = true;
+			kvm_vcpu_set_plane_stopped(vcpu);
+		}
+		r = 0;
+		break;
+	}
 	case KVM_GET_MP_STATE: {
 		struct kvm_mp_state mp_state;
 
-- 
2.50.1

[RFC PATCH 08/55] KVM: selftests: Cover plane run-state ioctls in planes_test

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Assert the initial stopped state of a new plane-1 vCPU, a runnable and
back-to-stopped round trip, and rejection of out-of-range states.

The tools copy of linux/kvm.h picks up the plane run-state ioctl
definitions.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 tools/include/uapi/linux/kvm.h                |  4 ++++
 tools/testing/selftests/kvm/x86/planes_test.c | 22 +++++++++++++++++++
 2 files changed, 26 insertions(+)

diff --git a/tools/include/uapi/linux/kvm.h b/tools/include/uapi/linux/kvm.h
index 586c0d6c168c..8d9f7e6bba19 100644
--- a/tools/include/uapi/linux/kvm.h
+++ b/tools/include/uapi/linux/kvm.h
@@ -1370,6 +1370,10 @@ struct kvm_s390_keyop {
 #define KVM_HAS_DEVICE_ATTR	  _IOW(KVMIO,  0xe3, struct kvm_device_attr)
 
 #define KVM_CREATE_PLANE	  _IO(KVMIO, 0xe4)
+#define KVM_GET_PLANE_RUN_STATE	  _IOR(KVMIO, 0xe5, __u32)
+#define KVM_SET_PLANE_RUN_STATE	  _IOW(KVMIO, 0xe6, __u32)
+#define   KVM_PLANE_RUN_STATE_STOPPED	0
+#define   KVM_PLANE_RUN_STATE_RUNNABLE	1
 
 /*
  * ioctls for vcpu fds
diff --git a/tools/testing/selftests/kvm/x86/planes_test.c b/tools/testing/selftests/kvm/x86/planes_test.c
index 3517175580d3..b28ed4077715 100644
--- a/tools/testing/selftests/kvm/x86/planes_test.c
+++ b/tools/testing/selftests/kvm/x86/planes_test.c
@@ -89,6 +89,28 @@ static void test_plane_lifecycle(void)
 	TEST_ASSERT(vcpu1_fd >= 0,
 		    "KVM_CREATE_VCPU on plane 1 failed: errno %d", errno);
 
+	/* Plane 1 starts stopped; run state is readable and writable. */
+	{
+		u32 state = 0xff;
+
+		kvm_ioctl(vcpu1_fd, KVM_GET_PLANE_RUN_STATE, &state);
+		TEST_ASSERT_EQ(state, KVM_PLANE_RUN_STATE_STOPPED);
+
+		state = KVM_PLANE_RUN_STATE_RUNNABLE;
+		kvm_ioctl(vcpu1_fd, KVM_SET_PLANE_RUN_STATE, &state);
+		state = 0xff;
+		kvm_ioctl(vcpu1_fd, KVM_GET_PLANE_RUN_STATE, &state);
+		TEST_ASSERT_EQ(state, KVM_PLANE_RUN_STATE_RUNNABLE);
+
+		state = KVM_PLANE_RUN_STATE_STOPPED;
+		kvm_ioctl(vcpu1_fd, KVM_SET_PLANE_RUN_STATE, &state);
+
+		state = 2;
+		TEST_ASSERT_EQ(__kvm_ioctl(vcpu1_fd, KVM_SET_PLANE_RUN_STATE,
+					   &state), -1);
+		TEST_ASSERT_EQ(errno, EINVAL);
+	}
+
 	/* Register state is per plane: write plane 1, plane 0 unaffected. */
 	kvm_ioctl(vcpu1_fd, KVM_GET_REGS, &regs1);
 	regs1.rbx = 0x5157ca11;	/* arbitrary marker */
-- 
2.50.1

[RFC PATCH 09/55] KVM: x86: hyper-v: Let userspace fault a forwarded hypercall

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The architectural failure mode for an invalid VTL transition is #UD at
the VMCALL, not a hypercall status — a status would advance RIP and
complete the call. Define a result flag for the forwarded-hypercall
completion which queues #UD at the unadvanced RIP and ignores the rest
of the result.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/hyperv.c          | 14 +++++++++++++-
 include/uapi/linux/kvm.h       |  8 ++++++++
 tools/include/uapi/linux/kvm.h |  9 +++++++++
 3 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index ec669dafda6e..e41d101e327a 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -2473,7 +2473,19 @@ static int kvm_hv_hypercall_complete(struct kvm_vcpu *vcpu, u64 result)
 
 static int kvm_hv_hypercall_complete_userspace(struct kvm_vcpu *vcpu)
 {
-	return kvm_hv_hypercall_complete(vcpu, vcpu->run->hyperv.u.hcall.result);
+	u64 result = vcpu->run->hyperv.u.hcall.result;
+
+	/*
+	 * A hypercall the VMM judged malformed beyond returning a status
+	 * (such as a VTL call that is invalid in the current state) faults
+	 * at the instruction instead of completing.
+	 */
+	if (result & KVM_HV_HCALL_RESULT_RAISE_UD) {
+		kvm_queue_exception(vcpu, UD_VECTOR);
+		return 1;
+	}
+
+	return kvm_hv_hypercall_complete(vcpu, result);
 }
 
 static int kvm_hvcall_signal_event(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 2aeb8c5172cb..b55e28be5181 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1016,6 +1016,14 @@ struct kvm_enable_cap {
 #define KVM_CAP_S390_VSIE_ESAMODE 248
 #define KVM_CAP_PLANES 249
 #define KVM_CAP_HYPERV_FORWARD_HCALL 250
+
+/*
+ * Userspace completion of a forwarded Hyper-V hypercall (KVM_EXIT_HYPERV_HCALL
+ * under KVM_CAP_HYPERV_FORWARD_HCALL): setting this bit in the result makes
+ * KVM raise #UD at the faulting instruction instead of completing the
+ * hypercall; all other result bits are then ignored.
+ */
+#define KVM_HV_HCALL_RESULT_RAISE_UD	(1ULL << 63)
 #define KVM_CAP_HCALL_FAULT_EXIT 252
 
 struct kvm_irq_routing_irqchip {
diff --git a/tools/include/uapi/linux/kvm.h b/tools/include/uapi/linux/kvm.h
index 8d9f7e6bba19..b55e28be5181 100644
--- a/tools/include/uapi/linux/kvm.h
+++ b/tools/include/uapi/linux/kvm.h
@@ -1017,6 +1017,15 @@ struct kvm_enable_cap {
 #define KVM_CAP_PLANES 249
 #define KVM_CAP_HYPERV_FORWARD_HCALL 250
 
+/*
+ * Userspace completion of a forwarded Hyper-V hypercall (KVM_EXIT_HYPERV_HCALL
+ * under KVM_CAP_HYPERV_FORWARD_HCALL): setting this bit in the result makes
+ * KVM raise #UD at the faulting instruction instead of completing the
+ * hypercall; all other result bits are then ignored.
+ */
+#define KVM_HV_HCALL_RESULT_RAISE_UD	(1ULL << 63)
+#define KVM_CAP_HCALL_FAULT_EXIT 252
+
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
 	__u32 pin;
-- 
2.50.1

[RFC PATCH 10/55] KVM: x86: Allow MP state ioctls on plane vCPU fds

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

A new plane vCPU for an AP starts, like any non-boot x86 vCPU, in the
uninitialised wait-for-INIT state, and nothing ever sends INIT/SIPI to
the higher plane of an AP. The VMM placing an initial VTL context on
it has to mark it runnable itself.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/x86.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index ca5d8e081fdc..970c738558f2 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -6278,6 +6278,8 @@ bool kvm_arch_is_vcpu_plane_ioctl(unsigned ioctl)
 	case KVM_SET_ONE_REG:
 	case KVM_GET_SREGS2:
 	case KVM_SET_SREGS2:
+	case KVM_GET_MP_STATE:
+	case KVM_SET_MP_STATE:
 	case KVM_GET_VCPU_EVENTS:
 	case KVM_SET_VCPU_EVENTS:
 	case KVM_GET_XCRS:
-- 
2.50.1

[RFC PATCH 11/55] KVM: x86: Filter KVM_ENABLE_CAP on plane vCPU fds

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The plane-fd ioctl allowlist admits KVM_ENABLE_CAP wholesale, so a
plane twin's fd reaches every vCPU cap handler: enlightened VMCS,
direct TLB flush, register intercepts.  The nested paths behind the
first two have never been audited for plane_level > 0, the register
intercept only ever diverts plane 0 (hv_divert_dt_write() skips higher
planes, so arming it on a twin buys native-emulation overhead and no
diversion), and any future vCPU cap would become plane-enabled by
default without anyone deciding that.

Accept only the audited caps on a plane fd: the per-plane SynIC pair
and the two CPUID-enforcement toggles, which mutate state the planes
genuinely bank per twin.  Everything else fails with -EINVAL, matching
how non-allowlisted ioctls already behave on plane fds.

The exposure is widened by our per-plane Hyper-V state: the VTL code
assumes per-vCPU capability state is uniform across a vCPU's twins
unless a cap is explicitly plane-aware.

Found by a local Sashiko review:

  ### M17 — whole KVM_ENABLE_CAP surface admitted on plane fds
  kvm_arch_is_vcpu_plane_ioctl() (x86.c) lists KVM_ENABLE_CAP with no
  per-cap filter in kvm_vcpu_ioctl_enable_cap(), so plane fds reach
  enable_evmcs, enable_l2_tlb_flush, ENFORCE_CPUID, REG_INTERCEPT etc.
  No demonstrated bug (all mutate per-vCPU state), but the vendor
  nested paths are unaudited for plane_level>0 and any future vCPU cap
  becomes plane-enabled implicitly.
  Fix shape: per-cap allowlist for plane vCPUs, or a comment recording
  the audit.

Fixes: 21d7ad7e2ba8 ("KVM: Implement KVM_CREATE_VCPU ioctl for planes")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/x86.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 970c738558f2..2c41c53e4d4f 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -6088,6 +6088,26 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
 	if (cap->flags)
 		return -EINVAL;
 
+	/*
+	 * KVM_ENABLE_CAP is accepted on plane vCPU fds, but only the caps
+	 * below have been audited for plane_level > 0.  The nested paths
+	 * behind eVMCS and direct TLB flush have not, and the register
+	 * intercept only ever diverts plane 0 (hv_divert_dt_write() skips
+	 * higher planes); refuse those rather than implicitly extending
+	 * every current and future vCPU cap to planes.
+	 */
+	if (vcpu->plane_level) {
+		switch (cap->cap) {
+		case KVM_CAP_HYPERV_SYNIC:
+		case KVM_CAP_HYPERV_SYNIC2:
+		case KVM_CAP_HYPERV_ENFORCE_CPUID:
+		case KVM_CAP_ENFORCE_PV_FEATURE_CPUID:
+			break;
+		default:
+			return -EINVAL;
+		}
+	}
+
 	switch (cap->cap) {
 #ifdef CONFIG_KVM_HYPERV
 	case KVM_CAP_HYPERV_SYNIC2:
-- 
2.50.1

[RFC PATCH 12/55] KVM: x86: initialise Hyper-V state on every plane vCPU

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

When userspace sets a Hyper-V CPUID, kvm_set_cpuid() walks the vCPU's
planes to allocate the per-plane kvm_vcpu_hv, but passes the plane-0
vCPU to kvm_hv_vcpu_init() on every iteration. Non-zero plane vCPUs
are left without hv_vcpu, which trips the
WARN_ON_ONCE(vcpu->arch.hyperv_enabled) in kvm_hv_set_cpuid() and
leaves their vp_index and SynIC state uninitialised until a lazy init
path happens to run. Pass the plane vCPU being iterated.

A twin created after KVM_SET_CPUID never passes through
kvm_set_cpuid() at all: it shares the common CPUID, and
kvm_arch_vcpu_create() syncs it through kvm_vcpu_after_set_cpuid(),
so that walk cannot cover it and the twin hits the same WARN with an
empty cpuid_cache. Allocate the context in kvm_vcpu_after_set_cpuid()
when the CPUID has the Hyper-V signature and the vCPU doesn't have
one yet. The allocation is idempotent, so the kvm_set_cpuid() path is
unaffected; if it fails, treat Hyper-V as disabled for the vCPU
exactly as a failing kvm_set_cpuid() would have.

Also admit KVM_ENABLE_CAP on plane vCPU fds, so userspace can enable
per-vCPU capabilities such as KVM_CAP_HYPERV_SYNIC2 on a plane vCPU
before it first runs.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/cpuid.c | 18 +++++++++++++++---
 arch/x86/kvm/cpuid.h |  1 +
 arch/x86/kvm/x86.c   | 20 +++++++++++++++++++-
 3 files changed, 35 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index fab075bb6fdc..fe8a793b1643 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -324,7 +324,7 @@ static void kvm_update_cpuid_runtime(struct kvm_vcpu *vcpu)
 						 vcpu->arch.ia32_xss, true);
 }
 
-static bool kvm_cpuid_has_hyperv(struct kvm_vcpu *vcpu)
+bool kvm_cpuid_has_hyperv(struct kvm_vcpu *vcpu)
 {
 #ifdef CONFIG_KVM_HYPERV
 	struct kvm_cpuid_entry2 *entry;
@@ -380,6 +380,7 @@ void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
 	struct kvm_cpuid_entry2 *best;
 	struct kvm_cpuid_entry2 *entry;
 	bool allow_gbpages;
+	bool hyperv_enabled;
 	int i;
 
 	memset(common->arch.cpu_caps, 0, sizeof(common->arch.cpu_caps));
@@ -456,7 +457,18 @@ void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
 					 __cr4_reserved_bits(guest_cpu_cap_has, vcpu);
 #undef __kvm_cpu_cap_has
 
-	kvm_hv_set_cpuid(vcpu, kvm_cpuid_has_hyperv(vcpu));
+	hyperv_enabled = kvm_cpuid_has_hyperv(vcpu);
+
+	/*
+	 * A plane twin created after KVM_SET_CPUID shares the common CPUID
+	 * without passing through kvm_set_cpuid(), which is what allocates
+	 * the Hyper-V context for the planes that exist at set time;
+	 * allocate it here for a late-created twin.
+	 */
+	if (hyperv_enabled && kvm_hv_vcpu_init(vcpu))
+		hyperv_enabled = false;
+
+	kvm_hv_set_cpuid(vcpu, hyperv_enabled);
 
 	/* Invoke the vendor callback only after the above state is updated. */
 	kvm_x86_call(vcpu_after_set_cpuid)(vcpu);
@@ -565,7 +577,7 @@ static int kvm_set_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid_entry2 *e2,
 #ifdef CONFIG_KVM_HYPERV
 	if (kvm_cpuid_has_hyperv(vcpu)) {
 		vcpu_for_each_plane(common, i, v) {
-			r = kvm_hv_vcpu_init(vcpu);
+			r = kvm_hv_vcpu_init(v);
 			if (r)
 				goto err;
 		}
diff --git a/arch/x86/kvm/cpuid.h b/arch/x86/kvm/cpuid.h
index 75abf447eabf..873377404899 100644
--- a/arch/x86/kvm/cpuid.h
+++ b/arch/x86/kvm/cpuid.h
@@ -19,6 +19,7 @@ static inline void kvm_finalize_cpu_caps(void)
 }
 
 void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu);
+bool kvm_cpuid_has_hyperv(struct kvm_vcpu *vcpu);
 struct kvm_cpuid_entry2 *kvm_find_cpuid_entry2(struct kvm_cpuid_entry2 *entries,
 					       int nent, u32 function, u64 index);
 /*
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 2c41c53e4d4f..0828889db149 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -6286,6 +6286,7 @@ static int kvm_get_reg_list(struct kvm_vcpu *vcpu,
 bool kvm_arch_is_vcpu_plane_ioctl(unsigned ioctl)
 {
 	switch (ioctl) {
+	case KVM_ENABLE_CAP:
 	case KVM_GET_DEBUGREGS:
 	case KVM_SET_DEBUGREGS:
 	case KVM_GET_LAPIC:
@@ -13135,9 +13136,24 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 	vcpu->arch.hv_root_tdp = INVALID_PAGE;
 #endif
 
+	/*
+	 * A plane twin created after KVM_SET_CPUID shares CPUID that may
+	 * already advertise Hyper-V.  Allocate the Hyper-V context now so
+	 * a failure fails the creation; left to kvm_vcpu_after_set_cpuid()
+	 * below, an -ENOMEM silently downgrades just this twin to no
+	 * Hyper-V (its MSRs fail) while the shared CPUID still advertises
+	 * the Hyper-V signature, and nothing short of another full
+	 * KVM_SET_CPUID repairs it.
+	 */
+	if (kvm_cpuid_has_hyperv(vcpu)) {
+		r = kvm_hv_vcpu_init(vcpu);
+		if (r)
+			goto free_guest_fpu;
+	}
+
 	r = kvm_x86_call(vcpu_create)(vcpu);
 	if (r)
-		goto free_guest_fpu;
+		goto free_hv;
 
 	kvm_xen_init_vcpu(vcpu);
 	vcpu_load(vcpu);
@@ -13148,6 +13164,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 	vcpu_put(vcpu);
 	return 0;
 
+free_hv:
+	kvm_hv_vcpu_uninit(vcpu);
 free_guest_fpu:
 	fpu_free_guest_fpstate(&vcpu->arch.guest_fpu);
 free_emulate_ctxt:
-- 
2.50.1

[RFC PATCH 13/55] KVM: implement per-plane memory attributes

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Memory attributes lived solely on struct kvm and the page role carried
no plane identity, so every plane shared one attribute array and one
set of TDP roots. A VTL1 guest protecting VTL0 pages therefore revoked
its *own* access too: the Windows secure kernel's first
HvCallModifyVtlProtectionMask made its own hypercall page
non-executable, and the next VTL transition died on an unhandled
protection fault.

Give each non-zero plane its own attribute xarray (plane 0 keeps
kvm->mem_attrs, which also carries partition-wide attributes such as
KVM_MEMORY_ATTRIBUTE_PRIVATE), accept KVM_SET_MEMORY_ATTRIBUTES on
plane fds restricted to the NR/NW/NX protection bits, and add the
plane to kvm_mmu_page_role so each plane builds its own roots with its
own protections applied at fault time.

Guest accesses are checked against the attributes of the plane doing
the accessing, not plane 0's: the vCPU-context hva lookups,
kvm_vcpu_map(), the guest page read/write helpers (which gain a
kvm_plane argument; the VM-level variants keep checking plane 0), and
the software page walker's PTE reads. Without that, a higher plane is
refused access to pages it has itself protected against plane 0 — the
secure kernel's VP assist page registration and the page walks through
its own protected page tables both fail exactly there.

Hugepage mixed-attribute tracking still follows plane 0 only; faults
on other planes clamp to 4KiB whenever the plane carries any attribute
within the maximal mapping range.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/include/asm/kvm_host.h |   2 +-
 arch/x86/kvm/mmu/mmu.c          |  78 +++++++++++----
 arch/x86/kvm/mmu/paging_tmpl.h  |   4 +-
 include/linux/kvm_host.h        |  52 ++++++++++
 virt/kvm/kvm_main.c             | 172 +++++++++++++++++++++++++-------
 5 files changed, 251 insertions(+), 57 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 05670b5ae28a..17dfd232ba58 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -366,7 +366,7 @@ union kvm_mmu_page_role {
 		unsigned guest_mode:1;
 		unsigned passthrough:1;
 		unsigned is_mirror:1;
-		unsigned :4;
+		unsigned plane:4;
 
 		/*
 		 * This is left at the top of the word so that
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index d94c34af52ee..177b2e71bf35 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -3143,7 +3143,9 @@ static bool kvm_mmu_prefetch_sptes(struct kvm_vcpu *vcpu, gfn_t gfn, u64 *sptep,
 		return false;
 
 	for (i = 0; i < nr_pages; i++, gfn++, sptep++) {
-		if (kvm_get_memory_attributes(vcpu->kvm, gfn)) {
+		if (kvm_get_memory_attributes(vcpu->kvm, gfn) ||
+		    (vcpu->plane_level &&
+		     kvm_plane_get_memory_attributes(vcpu->plane, gfn))) {
 			kvm_release_page_clean(pages[i]);
 			continue;
 		}
@@ -4626,7 +4628,7 @@ static inline unsigned kvm_get_gfn_protections(struct kvm_vcpu *vcpu, gfn_t gfn)
 {
 	struct kvm *kvm = vcpu->kvm;
 	unsigned int access = ACC_ALL | ACC_READ_MASK;
-	unsigned long attrs = kvm_get_memory_attributes(kvm, gfn);
+	unsigned long attrs = kvm_plane_get_memory_attributes(vcpu->plane, gfn);
 	if (!attrs)
 		return access;
 
@@ -4651,6 +4653,19 @@ static int kvm_faultin_memory_protections(struct kvm_vcpu *vcpu,
 	if (unlikely(!fault->slot))
 		return RET_PF_CONTINUE;
 
+	/*
+	 * Hugepage mixed-attribute tracking only covers plane 0; clamp
+	 * non-zero-plane faults to 4KiB when the plane carries any attribute
+	 * in the largest possible mapping range.
+	 */
+	if (vcpu->plane_level && fault->max_level > PG_LEVEL_4K) {
+		gfn_t base = gfn_round_for_level(fault->gfn, fault->max_level);
+
+		if (kvm_plane_range_has_any_attrs(vcpu->plane, base,
+				base + KVM_PAGES_PER_HPAGE(fault->max_level)))
+			fault->max_level = PG_LEVEL_4K;
+	}
+
 	access = kvm_get_gfn_protections(vcpu, fault->gfn);
 	if (access == (ACC_ALL | ACC_READ_MASK))
 		return RET_PF_CONTINUE;
@@ -5783,6 +5798,8 @@ static union kvm_cpu_role kvm_calc_cpu_role(struct kvm_vcpu *vcpu,
 	role.base.guest_mode = is_guest_mode(vcpu);
 	role.ext.valid = 1;
 
+	role.base.plane = vcpu->plane_level;
+
 	if (!____is_cr0_pg(regs)) {
 		role.base.direct = 1;
 		return role;
@@ -5868,6 +5885,7 @@ kvm_calc_tdp_mmu_root_page_role(struct kvm_vcpu *vcpu,
 	role.level = kvm_mmu_get_tdp_level(vcpu);
 	role.direct = true;
 	role.has_4_byte_gpte = false;
+	role.plane = vcpu->plane_level;
 
 	return role;
 }
@@ -5991,6 +6009,7 @@ kvm_calc_shadow_ept_root_page_role(struct kvm_vcpu *vcpu, bool accessed_dirty,
 	role.base.ad_disabled = !accessed_dirty;
 	role.base.guest_mode = true;
 	role.base.access = ACC_ALL;
+	role.base.plane = vcpu->plane_level;
 
 	role.ext.word = 0;
 	role.ext.execonly = execonly;
@@ -8113,12 +8132,36 @@ static bool hugepage_has_attrs(struct kvm *kvm, struct kvm_memory_slot *slot,
 	return true;
 }
 
+/*
+ * There are special considerations when applying a memory protection
+ * attribute against a page containing guest PTEs, because virtual
+ * addresses that require traversing that page need to fault.
+ *
+ * When using shadow paging, the PTEs that the CPU actually walks
+ * do not have any information about attributes (unlike TDP where
+ * the attributes become part of the TDP permission bits).  Since
+ * this should be a rare occasion, just invalidate all the mmu roots
+ * if shadow paging is in use and attributes do cover a guest PTE.
+ */
+static bool kvm_mmu_zap_if_gpte_page(struct kvm *kvm,
+				     struct kvm_gfn_range *range)
+{
+	struct kvm_mmu_page *sp;
+
+	for (gfn_t gfn = range->start; gfn < range->end; gfn++) {
+		for_each_gfn_valid_sp_with_gptes(kvm, sp, gfn) {
+			__kvm_mmu_zap_all_fast_front_half(kvm);
+			return false;
+		}
+	}
+	return false;
+}
+
 bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
 					 struct kvm_gfn_range *range)
 {
 	unsigned long attrs = range->arg.attributes;
 	struct kvm_memory_slot *slot = range->slot;
-	struct kvm_mmu_page *sp;
 	int level;
 
 	lockdep_assert_held_write(&kvm->mmu_lock);
@@ -8175,24 +8218,21 @@ bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
 		}
 	}
 
+	return kvm_mmu_zap_if_gpte_page(kvm, range);
+}
+
+bool kvm_arch_post_set_plane_memory_attributes(struct kvm *kvm,
+					       struct kvm_gfn_range *range)
+{
+	lockdep_assert_held_write(&kvm->mmu_lock);
+	lockdep_assert_held(&kvm->slots_lock);
+
 	/*
-	 * There are special considerations when applying an memory protection
-	 * attibute against a page containing guest PTEs, because virtual
-	 * addresses that requires traversing that page need to fault.
-	 *
-	 * When using shadow paging, the PTEs that the CPU actually walks
-	 * do not have any information about attributes (unlike TDP where
-	 * the attributes become part of the TDP permission bits).  Since
-	 * this should be a rare occasion, just invalidate all the mmu roots
-	 * if shadow paging is in use and attributes do cover a guest PTE.
+	 * Non-zero planes get no hugepage mixed-attribute tracking (their
+	 * faults are clamped to 4KiB while attributes are present); only
+	 * the shadow-paging gPTE invalidation applies.
 	 */
-	for (gfn_t gfn = range->start; gfn < range->end; gfn++) {
-		for_each_gfn_valid_sp_with_gptes(kvm, sp, gfn) {
-			__kvm_mmu_zap_all_fast_front_half(kvm);
-			return false;
-		}
-	}
-	return false;
+	return kvm_mmu_zap_if_gpte_page(kvm, range);
 }
 
 void kvm_arch_post_set_memory_attributes_unlocked(struct kvm *kvm)
diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h
index e48b8db54961..78d1e5c9fbe4 100644
--- a/arch/x86/kvm/mmu/paging_tmpl.h
+++ b/arch/x86/kvm/mmu/paging_tmpl.h
@@ -403,13 +403,13 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,
 		if (unlikely(kvm_is_error_hva(host_addr)))
 			goto error;
 
-		if (!kvm_mem_attributes_may_read_gfn(vcpu->kvm, gpa_to_gfn(real_gpa))) {
+		if (!kvm_vcpu_mem_attributes_may_read_gfn(vcpu, gpa_to_gfn(real_gpa))) {
 			walker->memory_attributes_fault = true;
 			walker->gfn = gpa_to_gfn(real_gpa);
 			goto error;
 		}
 
-		if (!kvm_mem_attributes_may_write_gfn(vcpu->kvm, gpa_to_gfn(real_gpa)))
+		if (!kvm_vcpu_mem_attributes_may_write_gfn(vcpu, gpa_to_gfn(real_gpa)))
 			walker->pte_writable[walker->level - 1] = false;
 
 		ptep_user = (pt_element_t __user *)((void *)host_addr + offset);
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 5bdae2127578..7071ff74d4ea 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -884,6 +884,16 @@ struct kvm_plane {
 	/* Per-Plane VCPU array */
 	struct xarray vcpu_array;
 
+#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+	/*
+	 * Per-plane memory attributes, used only for non-zero planes.
+	 * Plane 0's attributes live in kvm->mem_attrs, which also carries
+	 * partition-wide attributes such as KVM_MEMORY_ATTRIBUTE_PRIVATE.
+	 * Protected by slots_lock (for writes) and RCU (for reads).
+	 */
+	struct xarray mem_attrs;
+#endif
+
 	struct kvm_arch_plane arch;
 };
 
@@ -2736,6 +2746,18 @@ static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn
 	return xa_to_value(xa_load(&kvm->mem_attrs.array, gfn));
 }
 
+static inline struct xarray *kvm_plane_mem_attr_array(struct kvm_plane *plane)
+{
+	return plane->level ? &plane->mem_attrs :
+			      &plane->kvm->mem_attrs.array;
+}
+
+static inline unsigned long kvm_plane_get_memory_attributes(struct kvm_plane *plane,
+							    gfn_t gfn)
+{
+	return xa_to_value(xa_load(kvm_plane_mem_attr_array(plane), gfn));
+}
+
 static inline u64 kvm_mem_attributes_generation(struct kvm *kvm)
 {
 	RCU_LOCKDEP_WARN(!lockdep_is_held(&kvm->slots_lock) &&
@@ -2752,12 +2774,21 @@ static inline u64 kvm_mem_attributes_generation(struct kvm *kvm)
 
 bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 				     unsigned long mask, unsigned long attrs);
+bool kvm_plane_range_has_any_attrs(struct kvm_plane *plane, gfn_t start,
+				   gfn_t end);
 bool kvm_range_has_any_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 					unsigned long mask);
 bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
 					struct kvm_gfn_range *range);
 bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
 					 struct kvm_gfn_range *range);
+/*
+ * As kvm_arch_post_set_memory_attributes(), but for an update of a
+ * non-zero plane's attributes, which has no hugepage mixed-attribute
+ * tracking to maintain.
+ */
+bool kvm_arch_post_set_plane_memory_attributes(struct kvm *kvm,
+					       struct kvm_gfn_range *range);
 void kvm_arch_post_set_memory_attributes_unlocked(struct kvm *kvm);
 bool kvm_mem_attributes_valid(struct kvm *kvm, unsigned long attrs);
 u64 kvm_supported_mem_attributes(struct kvm *kvm);
@@ -2810,6 +2841,27 @@ static inline int kvm_mem_attributes_may_write_gfn(struct kvm *kvm, gfn_t gfn)
 	return kvm_mem_attributes_may_write(attrs);
 }
 
+/*
+ * vCPU-context accesses are checked against the vCPU's own plane's
+ * memory attributes: a higher plane is not subject to the protections
+ * it imposes on a lower plane's view of a page.
+ */
+static inline int kvm_vcpu_mem_attributes_may_read_gfn(struct kvm_vcpu *vcpu,
+							gfn_t gfn)
+{
+	unsigned long attrs = kvm_plane_get_memory_attributes(vcpu->plane, gfn);
+
+	return kvm_mem_attributes_may_read(attrs);
+}
+
+static inline int kvm_vcpu_mem_attributes_may_write_gfn(struct kvm_vcpu *vcpu,
+							 gfn_t gfn)
+{
+	unsigned long attrs = kvm_plane_get_memory_attributes(vcpu->plane, gfn);
+
+	return kvm_mem_attributes_may_write(attrs);
+}
+
 static inline bool kvm_range_has_rw_memory_protections(struct kvm *kvm,
 						       gfn_t start, gfn_t end)
 {
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 2d2dc7828e1f..ef23ee3ce04a 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -1231,6 +1231,9 @@ static struct kvm_plane *kvm_create_plane(struct kvm *kvm, unsigned plane_level)
 	plane->level = plane_level;
 
 	xa_init(&plane->vcpu_array);
+#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+	xa_init(&plane->mem_attrs);
+#endif
 
 	if (kvm_arch_plane_init(kvm, plane, plane_level))
 		goto out_free_plane;
@@ -1247,6 +1250,9 @@ static struct kvm_plane *kvm_create_plane(struct kvm *kvm, unsigned plane_level)
 
 static void kvm_destroy_one_plane(struct kvm_plane *plane)
 {
+#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+	xa_destroy(&plane->mem_attrs);
+#endif
 	kvm_arch_plane_destroy(plane);
 	kvm_free_plane(plane);
 }
@@ -2646,6 +2652,28 @@ bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 	return true;
 }
 
+/* Returns true if _any_ gfn in [@start, @end) has any attribute in @plane. */
+bool kvm_plane_range_has_any_attrs(struct kvm_plane *plane, gfn_t start,
+				   gfn_t end)
+{
+	XA_STATE(xas, kvm_plane_mem_attr_array(plane), start);
+	void *entry;
+
+	guard(rcu)();
+	xas_for_each(&xas, entry, end - 1) {
+		if (xas_retry(&xas, entry))
+			continue;
+		/*
+		 * Only count real attribute values: xa_reserve()
+		 * placeholders from an in-flight (or failed) attribute
+		 * update are not attributes.
+		 */
+		if (xa_is_value(entry))
+			return true;
+	}
+	return false;
+}
+
 /*
  * Returns true if _any_ gfns in the range [@start, @end) have attributes that
  * match _any_ bit in @mask.
@@ -2679,7 +2707,28 @@ bool kvm_range_has_any_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end
 	return false;
 }
 
-static bool kvm_range_memory_attributes_need_sync(struct kvm *kvm,
+/*
+ * Returns true if _any_ gfn in [@start, @end) has attributes matching
+ * _any_ bit in @mask in @plane's attribute array.
+ */
+static bool kvm_plane_range_has_memory_attributes(struct kvm_plane *plane,
+						  gfn_t start, gfn_t end,
+						  unsigned long mask)
+{
+	XA_STATE(xas, kvm_plane_mem_attr_array(plane), start);
+	void *entry;
+
+	guard(rcu)();
+	xas_for_each(&xas, entry, end - 1) {
+		if (xas_retry(&xas, entry))
+			continue;
+		if (xa_is_value(entry) && (xa_to_value(entry) & mask))
+			return true;
+	}
+	return false;
+}
+
+static bool kvm_range_memory_attributes_need_sync(struct kvm_plane *plane,
 						  gfn_t start, gfn_t end,
 						  unsigned long attributes)
 {
@@ -2688,7 +2737,7 @@ static bool kvm_range_memory_attributes_need_sync(struct kvm *kvm,
 	if (attributes & mask)
 		return true;
 
-	return kvm_range_has_any_memory_attributes(kvm, start, end, mask);
+	return kvm_plane_range_has_memory_attributes(plane, start, end, mask);
 }
 
 static __always_inline bool kvm_handle_gfn_range(struct kvm *kvm,
@@ -2763,9 +2812,11 @@ static bool kvm_pre_set_memory_attributes(struct kvm *kvm,
 }
 
 /* Set @attributes for the gfn range [@start, @end). */
-static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
-				     unsigned long attributes)
+static int kvm_plane_set_mem_attributes(struct kvm_plane *plane, gfn_t start,
+					gfn_t end, unsigned long attributes)
 {
+	struct kvm *kvm = plane->kvm;
+	struct xarray *attr_array = kvm_plane_mem_attr_array(plane);
 	struct kvm_mmu_notifier_range pre_set_range = {
 		.start = start,
 		.end = end,
@@ -2791,13 +2842,17 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 
 	entry = attributes ? xa_mk_value(attributes) : NULL;
 
+	if (plane->level)
+		post_set_range.handler = kvm_arch_post_set_plane_memory_attributes;
+
 	mutex_lock(&kvm->slots_lock);
 
 	/* Nothing to do if the entire range has the desired attributes. */
-	if (kvm_range_has_memory_attributes(kvm, start, end, ~0, attributes))
+	if (!plane->level &&
+	    kvm_range_has_memory_attributes(kvm, start, end, ~0, attributes))
 		goto out_unlock;
 
-	sync = kvm_range_memory_attributes_need_sync(kvm, start, end,
+	sync = kvm_range_memory_attributes_need_sync(plane, start, end,
 						     attributes);
 
 	trace_kvm_vm_set_mem_attributes(start, end, attributes, sync,
@@ -2808,9 +2863,19 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 	 * partway through setting the new attributes.
 	 */
 	for (i = start; i < end; i++) {
-		r = xa_reserve(&kvm->mem_attrs.array, i, GFP_KERNEL_ACCOUNT);
-		if (r)
+		r = xa_reserve(attr_array, i, GFP_KERNEL_ACCOUNT);
+		if (r) {
+			/*
+			 * Nothing was stored yet; drop the reservations
+			 * already made so they don't sit in the array
+			 * (and pin their nodes) for the VM's lifetime.
+			 * xa_release() only removes reserved entries, so
+			 * pre-existing attribute values are untouched.
+			 */
+			while (i-- > start)
+				xa_release(attr_array, i);
 			goto out_unlock;
+		}
 
 		cond_resched();
 	}
@@ -2818,8 +2883,7 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 	found_memslot = kvm_handle_gfn_range(kvm, &pre_set_range);
 
 	for (i = start; i < end; i++) {
-		r = xa_err(xa_store(&kvm->mem_attrs.array, i, entry,
-				    GFP_KERNEL_ACCOUNT));
+		r = xa_err(xa_store(attr_array, i, entry, GFP_KERNEL_ACCOUNT));
 		KVM_BUG_ON(r, kvm);
 		cond_resched();
 	}
@@ -2829,6 +2893,15 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 			  kvm->mem_attrs.generation + 1);
 
 	if (found_memslot) {
+		/*
+		 * The post-set handler differs by plane: hugepage
+		 * mixed-attribute tracking only follows plane 0's
+		 * attributes (faults on other planes with attributes
+		 * present are clamped to 4KiB mappings instead), but the
+		 * shadow-paging invalidation of mappings through
+		 * attribute-covered guest page tables is needed on every
+		 * plane.
+		 */
 		kvm_handle_gfn_range(kvm, &post_set_range);
 		KVM_MMU_LOCK(kvm);
 		kvm_mmu_invalidate_end(kvm);
@@ -2850,6 +2923,16 @@ bool __weak kvm_arch_mem_attributes_supported_prot(struct kvm *kvm, unsigned lon
 	return false;
 }
 
+/*
+ * Only reachable on architectures which support planes; those provide a
+ * real implementation.
+ */
+bool __weak kvm_arch_post_set_plane_memory_attributes(struct kvm *kvm,
+						      struct kvm_gfn_range *range)
+{
+	return false;
+}
+
 bool kvm_mem_attributes_valid(struct kvm *kvm, unsigned long attrs)
 {
 	if (attrs & ~kvm_supported_mem_attributes(kvm))
@@ -2862,9 +2945,10 @@ bool kvm_mem_attributes_valid(struct kvm *kvm, unsigned long attrs)
 	return true;
 }
 
-static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
+static int kvm_vm_ioctl_set_mem_attributes(struct kvm_plane *plane,
 					   struct kvm_memory_attributes *attrs)
 {
+	struct kvm *kvm = plane->kvm;
 	gfn_t start, end;
 
 	/* flags is currently not used. */
@@ -2872,6 +2956,10 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 		return -EINVAL;
 	if (!kvm_mem_attributes_valid(kvm, attrs->attributes))
 		return -EINVAL;
+	/* Partition-wide attributes have no meaning on a non-zero plane. */
+	if (plane->level &&
+	    (attrs->attributes & ~(unsigned long)KVM_MEMORY_ATTRIBUTE_PROT))
+		return -EINVAL;
 	if (attrs->size == 0 || attrs->address + attrs->size < attrs->address)
 		return -EINVAL;
 	if (!PAGE_ALIGNED(attrs->address) || !PAGE_ALIGNED(attrs->size))
@@ -2887,7 +2975,8 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 	 */
 	BUILD_BUG_ON(sizeof(attrs->attributes) != sizeof(unsigned long));
 
-	return kvm_vm_set_mem_attributes(kvm, start, end, attrs->attributes);
+	return kvm_plane_set_mem_attributes(plane, start, end,
+					    attrs->attributes);
 }
 #endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
 
@@ -3024,17 +3113,16 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(gfn_to_hva);
 
 unsigned long kvm_vcpu_gfn_to_hva(struct kvm_vcpu *vcpu, gfn_t gfn)
 {
-	struct kvm *kvm = vcpu->kvm;
 	unsigned long addr;
 
 	addr = gfn_to_hva_many(kvm_vcpu_gfn_to_memslot(vcpu, gfn), gfn, NULL);
 	if (kvm_is_error_hva(addr))
 		return addr;
 
-	if (!kvm_mem_attributes_may_read_gfn(kvm, gfn))
+	if (!kvm_vcpu_mem_attributes_may_read_gfn(vcpu, gfn))
 		return KVM_HVA_ERR_BAD;
 
-	if (!kvm_mem_attributes_may_write_gfn(kvm, gfn))
+	if (!kvm_vcpu_mem_attributes_may_write_gfn(vcpu, gfn))
 		return KVM_HVA_ERR_RO_BAD;
 
 	return addr;
@@ -3087,10 +3175,10 @@ unsigned long kvm_vcpu_gfn_to_hva_prot(struct kvm_vcpu *vcpu, gfn_t gfn, bool *w
 	if (kvm_is_error_hva(addr))
 		return addr;
 
-	if (!kvm_mem_attributes_may_read_gfn(vcpu->kvm, gfn))
+	if (!kvm_vcpu_mem_attributes_may_read_gfn(vcpu, gfn))
 		return KVM_HVA_ERR_BAD;
 
-	if (writable && !kvm_mem_attributes_may_write_gfn(vcpu->kvm, gfn))
+	if (writable && !kvm_vcpu_mem_attributes_may_write_gfn(vcpu, gfn))
 		*writable = false;
 
 	return addr;
@@ -3431,9 +3519,9 @@ int __kvm_vcpu_map(struct kvm_vcpu *vcpu, gfn_t gfn, struct kvm_host_map *map,
 		.pin = true,
 	};
 
-	if (!kvm_mem_attributes_may_read_gfn(vcpu->kvm, gfn))
+	if (!kvm_vcpu_mem_attributes_may_read_gfn(vcpu, gfn))
 		return -EFAULT;
-	if (writable && !kvm_mem_attributes_may_write_gfn(vcpu->kvm, gfn))
+	if (writable && !kvm_vcpu_mem_attributes_may_write_gfn(vcpu, gfn))
 		return -EFAULT;
 
 	map->pinned_page = NULL;
@@ -3496,7 +3584,8 @@ static int next_segment(unsigned long len, int offset)
 }
 
 /* Copy @len bytes from guest memory at '(@gfn * PAGE_SIZE) + @offset' to @data */
-static int __kvm_read_guest_page(struct kvm *kvm, struct kvm_memory_slot *slot,
+static int __kvm_read_guest_page(struct kvm_plane *plane,
+				 struct kvm_memory_slot *slot,
 				 gfn_t gfn, void *data, int offset, int len,
 				 unsigned long attr)
 {
@@ -3511,7 +3600,7 @@ static int __kvm_read_guest_page(struct kvm *kvm, struct kvm_memory_slot *slot,
 	if (kvm_is_error_hva(addr))
 		return -EFAULT;
 
-	attrs = kvm_get_memory_attributes(kvm, gfn);
+	attrs = kvm_plane_get_memory_attributes(plane, gfn);
 	if (attrs & attr)
 		return -EFAULT;
 
@@ -3526,8 +3615,8 @@ int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,
 {
 	struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
 
-	return __kvm_read_guest_page(kvm, slot, gfn, data, offset, len,
-				     KVM_MEMORY_ATTRIBUTE_NR);
+	return __kvm_read_guest_page(kvm->planes[0], slot, gfn, data, offset,
+				     len, KVM_MEMORY_ATTRIBUTE_NR);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_read_guest_page);
 
@@ -3536,8 +3625,8 @@ int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,
 {
 	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
 
-	return __kvm_read_guest_page(vcpu->kvm, slot, gfn, data, offset, len,
-				     KVM_MEMORY_ATTRIBUTE_NR);
+	return __kvm_read_guest_page(vcpu->plane, slot, gfn, data, offset,
+				     len, KVM_MEMORY_ATTRIBUTE_NR);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_read_guest_page);
 
@@ -3546,8 +3635,8 @@ int kvm_vcpu_fetch_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,
 {
 	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
 
-	return __kvm_read_guest_page(vcpu->kvm, slot, gfn, data, offset, len,
-				     KVM_MEMORY_ATTRIBUTE_NX);
+	return __kvm_read_guest_page(vcpu->plane, slot, gfn, data, offset,
+				     len, KVM_MEMORY_ATTRIBUTE_NX);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_fetch_guest_page);
 
@@ -3591,7 +3680,7 @@ int kvm_vcpu_read_guest(struct kvm_vcpu *vcpu, gpa_t gpa, void *data, unsigned l
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_read_guest);
 
-static int __kvm_read_guest_atomic(struct kvm *kvm,
+static int __kvm_read_guest_atomic(struct kvm_plane *plane,
 				   struct kvm_memory_slot *slot, gfn_t gfn,
 				   void *data, int offset, unsigned long len)
 {
@@ -3601,7 +3690,7 @@ static int __kvm_read_guest_atomic(struct kvm *kvm,
 	if (WARN_ON_ONCE(offset + len > PAGE_SIZE))
 		return -EFAULT;
 
-	if (!kvm_mem_attributes_may_read_gfn(kvm, gfn))
+	if (!kvm_mem_attributes_may_read(kvm_plane_get_memory_attributes(plane, gfn)))
 		return -EFAULT;
 
 	addr = gfn_to_hva_memslot_prot(slot, gfn, NULL);
@@ -3622,12 +3711,12 @@ int kvm_vcpu_read_guest_atomic(struct kvm_vcpu *vcpu, gpa_t gpa,
 	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
 	int offset = offset_in_page(gpa);
 
-	return __kvm_read_guest_atomic(vcpu->kvm, slot, gfn, data, offset, len);
+	return __kvm_read_guest_atomic(vcpu->plane, slot, gfn, data, offset, len);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_read_guest_atomic);
 
 /* Copy @len bytes from @data into guest memory at '(@gfn * PAGE_SIZE) + @offset' */
-static int __kvm_write_guest_page(struct kvm *kvm,
+static int __kvm_write_guest_page(struct kvm_plane *plane, struct kvm *kvm,
 				  struct kvm_memory_slot *memslot, gfn_t gfn,
 			          const void *data, int offset, int len)
 {
@@ -3641,7 +3730,7 @@ static int __kvm_write_guest_page(struct kvm *kvm,
 	if (kvm_is_error_hva(addr))
 		return -EFAULT;
 
-	if (!kvm_mem_attributes_may_write_gfn(kvm, gfn))
+	if (!kvm_mem_attributes_may_write(kvm_plane_get_memory_attributes(plane, gfn)))
 		return -EFAULT;
 
 	r = __copy_to_user((void __user *)addr + offset, data, len);
@@ -3656,7 +3745,8 @@ int kvm_write_guest_page(struct kvm *kvm, gfn_t gfn,
 {
 	struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
 
-	return __kvm_write_guest_page(kvm, slot, gfn, data, offset, len);
+	return __kvm_write_guest_page(kvm->planes[0], kvm, slot, gfn, data,
+				      offset, len);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_write_guest_page);
 
@@ -3665,7 +3755,8 @@ int kvm_vcpu_write_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn,
 {
 	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
 
-	return __kvm_write_guest_page(vcpu->kvm, slot, gfn, data, offset, len);
+	return __kvm_write_guest_page(vcpu->plane, vcpu->kvm, slot, gfn, data,
+				      offset, len);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_write_guest_page);
 
@@ -5182,6 +5273,17 @@ static long __kvm_plane_ioctl(struct kvm_plane *plane, unsigned int ioctl, unsig
 		break;
 	}
 #endif /* CONFIG_HAVE_KVM_IRQ_ROUTING */
+#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+	case KVM_SET_MEMORY_ATTRIBUTES: {
+		void __user *argp = (void __user *)arg;
+		struct kvm_memory_attributes attrs;
+
+		if (copy_from_user(&attrs, argp, sizeof(attrs)))
+			return -EFAULT;
+		r = kvm_vm_ioctl_set_mem_attributes(plane, &attrs);
+		break;
+	}
+#endif
 	default:
 		r = -ENOTTY;
 	}
@@ -5926,7 +6028,7 @@ static long kvm_vm_ioctl(struct file *filp,
 		if (copy_from_user(&attrs, argp, sizeof(attrs)))
 			goto out;
 
-		r = kvm_vm_ioctl_set_mem_attributes(kvm, &attrs);
+		r = kvm_vm_ioctl_set_mem_attributes(kvm->planes[0], &attrs);
 		break;
 	}
 #endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
-- 
2.50.1

[RFC PATCH 14/55] KVM: x86: hyper-v: bank GUEST_OS_ID and HYPERCALL MSRs per plane

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The TLFS makes HV_X64_MSR_GUEST_OS_ID and HV_X64_MSR_HYPERCALL
separate instances per VTL; KVM kept a single partition-wide copy, so
the Windows secure kernel enabling its own hypercall page from VTL1
clobbered VTL0's.

Index both MSRs by the vCPU's plane. The hypercall page contents are
still written into guest RAM on enable, now once per plane's page.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/include/asm/kvm_host.h |  9 +++++++--
 arch/x86/kvm/hyperv.c           | 19 ++++++++++---------
 arch/x86/kvm/hyperv.h           |  3 ++-
 3 files changed, 19 insertions(+), 12 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 17dfd232ba58..1c4395f73acb 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1202,8 +1202,13 @@ enum hv_tsc_page_status {
 /* Hyper-V emulation context */
 struct kvm_hv {
 	struct mutex hv_lock;
-	u64 hv_guest_os_id;
-	u64 hv_hypercall;
+	/*
+	 * GUEST_OS_ID and the hypercall page are per-VTL instances in the
+	 * TLFS; bank them per plane so a higher VTL enabling its own
+	 * hypercall page does not clobber the lower VTL's.
+	 */
+	u64 hv_guest_os_id[KVM_MAX_PLANES];
+	u64 hv_hypercall[KVM_MAX_PLANES];
 	bool hv_forward_hcalls;
 	u64 hv_tsc_page;
 	enum hv_tsc_page_status hv_tsc_page_status;
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index e41d101e327a..e703d8999ff0 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -1342,7 +1342,7 @@ static void __kvm_hv_xsaves_xsavec_maybe_warn(struct kvm_vcpu *vcpu)
 	if (hv->xsaves_xsavec_checked)
 		return;
 
-	if ((hv->hv_guest_os_id & KVM_HV_WIN2016_GUEST_ID_MASK) !=
+	if ((hv->hv_guest_os_id[0] & KVM_HV_WIN2016_GUEST_ID_MASK) !=
 	    KVM_HV_WIN2016_GUEST_ID)
 		return;
 
@@ -1384,10 +1384,11 @@ static int kvm_hv_set_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data,
 
 	switch (msr) {
 	case HV_X64_MSR_GUEST_OS_ID:
-		hv->hv_guest_os_id = data;
+		hv->hv_guest_os_id[vcpu->plane_level] = data;
 		/* setting guest os id to zero disables hypercall page */
-		if (!hv->hv_guest_os_id)
-			hv->hv_hypercall &= ~HV_X64_MSR_HYPERCALL_ENABLE;
+		if (!hv->hv_guest_os_id[vcpu->plane_level])
+			hv->hv_hypercall[vcpu->plane_level] &=
+				~HV_X64_MSR_HYPERCALL_ENABLE;
 		break;
 	case HV_X64_MSR_HYPERCALL: {
 		u8 instructions[9];
@@ -1395,10 +1396,10 @@ static int kvm_hv_set_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data,
 		u64 addr;
 
 		/* if guest os id is not set hypercall should remain disabled */
-		if (!hv->hv_guest_os_id)
+		if (!hv->hv_guest_os_id[vcpu->plane_level])
 			break;
 		if (!(data & HV_X64_MSR_HYPERCALL_ENABLE)) {
-			hv->hv_hypercall = data;
+			hv->hv_hypercall[vcpu->plane_level] = data;
 			break;
 		}
 
@@ -1427,7 +1428,7 @@ static int kvm_hv_set_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data,
 		addr = data & HV_X64_MSR_HYPERCALL_PAGE_ADDRESS_MASK;
 		if (kvm_vcpu_write_guest(vcpu, addr, instructions, i))
 			return 1;
-		hv->hv_hypercall = data;
+		hv->hv_hypercall[vcpu->plane_level] = data;
 		break;
 	}
 	case HV_X64_MSR_REFERENCE_TSC:
@@ -1640,10 +1641,10 @@ static int kvm_hv_get_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata,
 
 	switch (msr) {
 	case HV_X64_MSR_GUEST_OS_ID:
-		data = hv->hv_guest_os_id;
+		data = hv->hv_guest_os_id[vcpu->plane_level];
 		break;
 	case HV_X64_MSR_HYPERCALL:
-		data = hv->hv_hypercall;
+		data = hv->hv_hypercall[vcpu->plane_level];
 		break;
 	case HV_X64_MSR_TIME_REF_COUNT:
 		data = get_time_ref_counter(kvm);
diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h
index 6301f79fcbae..0593fe4333d3 100644
--- a/arch/x86/kvm/hyperv.h
+++ b/arch/x86/kvm/hyperv.h
@@ -97,7 +97,8 @@ int kvm_hv_get_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata, bool host)
 
 static inline bool kvm_hv_hypercall_enabled(struct kvm_vcpu *vcpu)
 {
-	return vcpu->arch.hyperv_enabled && to_kvm_hv(vcpu->kvm)->hv_guest_os_id;
+	return vcpu->arch.hyperv_enabled &&
+	       to_kvm_hv(vcpu->kvm)->hv_guest_os_id[vcpu->plane_level];
 }
 
 int kvm_hv_hypercall(struct kvm_vcpu *vcpu);
-- 
2.50.1

[RFC PATCH 15/55] KVM: x86: hyper-v: bank the reference TSC page per plane

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

HV_X64_MSR_REFERENCE_TSC is a per-VTL instance like GUEST_OS_ID and
HYPERCALL. With a single partition-wide copy, the secure kernel
enabling its own reference TSC page from VTL1 invalidated VTL0's, and
the normal kernel fell back to a TIME_REF_COUNT MSR read — a VM exit —
on every clock read.

Keep the masterclock-maintained page bound to plane 0, and give
non-zero planes a banked MSR whose page gets a one-shot copy of the
current parameters; those stay valid while the masterclock is stable,
which is enough for VTL1's bounded execution windows.

The copy is made only when hv->tsc_ref is host-computed. With
HV_X64_MSR_TSC_EMULATION_CONTROL set (reenlightenment, restored across
migration), kvm_hv_setup_tsc_page() repopulates hv->tsc_ref from the
plane-0 page in guest RAM, which the lowest VTL can write; handing a
copy of that to a higher plane with hypervisor authority would let
VTL0 forge the sequence/scale/offset the secure kernel computes time
from. In that case write the plane's page with a zero sequence, making
it fall back to the TIME_REF_COUNT MSR. The reference VSM host keeps
fully separate per-VTL pages and never crosses VTLs here.

The copy uses the same zero-sequence/body/sequence protocol as the
plane-0 path, so a concurrent reader never sees torn parameters, and
the MSR write fails when the guest write faults, as the
HV_X64_MSR_HYPERCALL case does.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 Documentation/virt/kvm/api.rst  |  7 ++++
 arch/x86/include/asm/kvm_host.h |  8 +++++
 arch/x86/kvm/hyperv.c           | 58 ++++++++++++++++++++++++++++++++-
 3 files changed, 72 insertions(+), 1 deletion(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index f8d9bca1baea..570db83844c7 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6611,6 +6611,13 @@ in the future may use the plane id when planes are supported by hardware (as is
 the case for VMPLs in AMD), or if KVM supports accelerated plane switch
 operations (as might be the case for Hyper-V VTLs).
 
+On x86, the Hyper-V partition-wide MSRs HV_X64_MSR_GUEST_OS_ID,
+HV_X64_MSR_HYPERCALL and HV_X64_MSR_REFERENCE_TSC are banked per plane,
+matching their per-VTL instances in the TLFS: an MSR access reads or
+writes the instance belonging to the plane of the vCPU fd that performs
+it.  A migrating VMM must therefore save and restore these MSRs through
+a vCPU fd of every plane, not just plane 0.
+
 4.146 KVM_GET_PLANE_RUN_STATE / KVM_SET_PLANE_RUN_STATE
 --------------------------------------------------------
 
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 1c4395f73acb..6ee85022cdfc 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1211,6 +1211,8 @@ struct kvm_hv {
 	u64 hv_hypercall[KVM_MAX_PLANES];
 	bool hv_forward_hcalls;
 	u64 hv_tsc_page;
+	/* Banked REFERENCE_TSC MSR values for non-zero planes (VTLs). */
+	u64 hv_tsc_page_plane[KVM_MAX_PLANES];
 	enum hv_tsc_page_status hv_tsc_page_status;
 
 	/* Hyper-v based guest crash (NT kernel bugcheck) parameters */
@@ -1218,6 +1220,12 @@ struct kvm_hv {
 	u64 hv_crash_ctl;
 
 	struct ms_hyperv_tsc_page tsc_ref;
+	/*
+	 * tsc_ref was read back from the plane-0 page (guest TSC emulation
+	 * after migration) rather than computed by the host; such values
+	 * are plane-0-writable and must never seed another plane's page.
+	 */
+	bool tsc_ref_from_guest;
 
 	struct idr conn_to_evt;
 
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index e703d8999ff0..eefc18c750f1 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -1191,6 +1191,7 @@ void kvm_hv_setup_tsc_page(struct kvm *kvm,
 		if (kvm_read_guest(kvm, gfn_to_gpa(gfn), &hv->tsc_ref, sizeof(hv->tsc_ref)))
 			goto out_err;
 
+		hv->tsc_ref_from_guest = true;
 		hv->hv_tsc_page_status = HV_TSC_PAGE_SET;
 		return;
 	}
@@ -1227,6 +1228,7 @@ void kvm_hv_setup_tsc_page(struct kvm *kvm,
 			    &hv->tsc_ref, sizeof(hv->tsc_ref.tsc_sequence)))
 		goto out_err;
 
+	hv->tsc_ref_from_guest = false;
 	hv->hv_tsc_page_status = HV_TSC_PAGE_SET;
 	return;
 
@@ -1432,6 +1434,58 @@ static int kvm_hv_set_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data,
 		break;
 	}
 	case HV_X64_MSR_REFERENCE_TSC:
+		/*
+		 * The reference TSC page is a per-VTL instance.  Plane 0 keeps
+		 * the live masterclock-maintained page; a non-zero plane gets
+		 * a one-shot copy of the current parameters, which stay valid
+		 * while the masterclock is stable.
+		 */
+		if (vcpu->plane_level) {
+			struct ms_hyperv_tsc_page tsc_ref = {};
+			u32 tsc_seq;
+			u64 gfn;
+
+			hv->hv_tsc_page_plane[vcpu->plane_level] = data;
+			if (!(data & HV_X64_MSR_TSC_REFERENCE_ENABLE))
+				break;
+
+			/*
+			 * Copy only host-computed parameters.  After a
+			 * masterclock change under guest TSC emulation,
+			 * hv->tsc_ref holds values read back from the
+			 * plane-0 page, which the lowest VTL can write;
+			 * presenting those to another plane would let it
+			 * forge that plane's clock.  Without usable
+			 * parameters, publish an invalid (zero) sequence so
+			 * the plane falls back to the TIME_REF_COUNT MSR.
+			 */
+			if (hv->hv_tsc_page_status == HV_TSC_PAGE_SET &&
+			    !hv->tsc_ref_from_guest)
+				tsc_ref = hv->tsc_ref;
+
+			gfn = data >> HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT;
+			tsc_seq = tsc_ref.tsc_sequence;
+
+			/*
+			 * Write the body under a zero sequence, then the
+			 * real sequence, so a concurrent reader on this
+			 * plane never consumes torn parameters.
+			 */
+			tsc_ref.tsc_sequence = 0;
+			if (kvm_vcpu_write_guest(vcpu, gfn_to_gpa(gfn),
+						 &tsc_ref, sizeof(tsc_ref)))
+				return 1;
+
+			if (!tsc_seq)
+				break;
+
+			tsc_ref.tsc_sequence = tsc_seq;
+			if (kvm_vcpu_write_guest(vcpu, gfn_to_gpa(gfn),
+						 &tsc_ref,
+						 sizeof(tsc_ref.tsc_sequence)))
+				return 1;
+			break;
+		}
 		hv->hv_tsc_page = data;
 		if (hv->hv_tsc_page & HV_X64_MSR_TSC_REFERENCE_ENABLE) {
 			if (!host)
@@ -1650,7 +1704,9 @@ static int kvm_hv_get_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata,
 		data = get_time_ref_counter(kvm);
 		break;
 	case HV_X64_MSR_REFERENCE_TSC:
-		data = hv->hv_tsc_page;
+		data = vcpu->plane_level ?
+		       hv->hv_tsc_page_plane[vcpu->plane_level] :
+		       hv->hv_tsc_page;
 		break;
 	case HV_X64_MSR_CRASH_P0 ... HV_X64_MSR_CRASH_P4:
 		return kvm_hv_msr_get_crash_data(kvm,
-- 
2.50.1

[RFC PATCH 16/55] KVM: x86: allow KVM_SET_GUEST_DEBUG on plane vCPU fds

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Debugging a non-zero plane needs single-step and breakpoint control on
the plane vCPU. The debug exits surface through the plane-0 KVM_RUN
loop like any other plane exit.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 Documentation/virt/kvm/api.rst | 6 +++---
 arch/x86/kvm/x86.c             | 1 +
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 570db83844c7..8232356b1bb0 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -1536,7 +1536,7 @@ for vm-wide capabilities.
 
 :Capability: KVM_CAP_MP_STATE
 :Architectures: x86, s390, arm64, riscv, loongarch
-:Type: vcpu ioctl
+:Type: vcpu ioctl (all planes)
 :Parameters: struct kvm_mp_state (out)
 :Returns: 0 on success; -1 on error
 
@@ -1617,7 +1617,7 @@ whether the vcpu is runnable.
 
 :Capability: KVM_CAP_MP_STATE
 :Architectures: x86, s390, arm64, riscv, loongarch
-:Type: vcpu ioctl
+:Type: vcpu ioctl (all planes)
 :Parameters: struct kvm_mp_state (in)
 :Returns: 0 on success; -1 on error
 
@@ -3746,7 +3746,7 @@ handled.
 
 :Capability: KVM_CAP_SET_GUEST_DEBUG
 :Architectures: x86, s390, ppc, arm64
-:Type: vcpu ioctl
+:Type: vcpu ioctl (all planes)
 :Parameters: struct kvm_guest_debug (in)
 :Returns: 0 on success; -1 on error
 
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 0828889db149..eb5d4932641e 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -6287,6 +6287,7 @@ bool kvm_arch_is_vcpu_plane_ioctl(unsigned ioctl)
 {
 	switch (ioctl) {
 	case KVM_ENABLE_CAP:
+	case KVM_SET_GUEST_DEBUG:
 	case KVM_GET_DEBUGREGS:
 	case KVM_SET_DEBUGREGS:
 	case KVM_GET_LAPIC:
-- 
2.50.1

[RFC PATCH 17/55] KVM: x86: hyper-v: flush the calling plane's TLB on flush hypercalls

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The Hyper-V TLB flush hypercalls resolve their target vCPUs through
the plane-0 array, so a flush issued from a non-zero plane flushed
only the plane-0 vCPUs and the caller's own plane kept stale
translations. The Windows secure kernel repopulated a PFN-database
page table entry, flushed, then took a fatal page fault on the very
address it had just mapped, because VTL1's TLB still held the old
read-only zero-page translation.

Resolve each flush target to the vCPU of the calling plane (falling
back to plane 0 where no such plane vCPU exists — it then has no TLB
to flush either) and raise the flush request on that vCPU directly.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/hyperv.c | 34 ++++++++++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index eefc18c750f1..797c5cbfbbaf 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -2122,6 +2122,23 @@ static int kvm_hv_hypercall_check_params(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
+/*
+ * A TLB flush hypercall flushes the caller's VTL view of the target VPs:
+ * resolve the target to the vCPU of the caller's plane, which has its own
+ * roots and TLB tag.  Fall back to plane 0 when the plane vCPU does not
+ * exist (it then has no TLB to flush either).
+ */
+static struct kvm_vcpu *hv_flush_plane_vcpu(struct kvm_vcpu *caller,
+					    struct kvm_vcpu *v)
+{
+	struct kvm_vcpu *pv;
+
+	if (!caller->plane_level)
+		return v;
+	pv = v->common->vcpus[caller->plane_level];
+	return pv ?: v;
+}
+
 static s64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 {
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
@@ -2257,12 +2274,18 @@ static s64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 	 */
 	if (all_cpus && !is_guest_mode(vcpu)) {
 		kvm_for_each_vcpu(i, v, kvm) {
+			v = hv_flush_plane_vcpu(vcpu, v);
 			tlb_flush_fifo = kvm_hv_get_tlb_flush_fifo(v, false);
 			hv_tlb_flush_enqueue(v, tlb_flush_fifo,
 					     tlb_flush_entries, hc->rep_cnt);
+			if (vcpu->plane_level) {
+				kvm_make_request(KVM_REQ_HV_TLB_FLUSH, v);
+				kvm_vcpu_kick(v);
+			}
 		}
 
-		kvm_make_all_cpus_request(kvm, KVM_REQ_HV_TLB_FLUSH);
+		if (!vcpu->plane_level)
+			kvm_make_all_cpus_request(kvm, KVM_REQ_HV_TLB_FLUSH);
 	} else if (!is_guest_mode(vcpu)) {
 		sparse_set_to_vcpu_mask(kvm, sparse_banks, valid_bank_mask, vcpu_mask);
 
@@ -2270,12 +2293,19 @@ static s64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 			v = kvm_get_vcpu(kvm, i);
 			if (!v)
 				continue;
+			v = hv_flush_plane_vcpu(vcpu, v);
 			tlb_flush_fifo = kvm_hv_get_tlb_flush_fifo(v, false);
 			hv_tlb_flush_enqueue(v, tlb_flush_fifo,
 					     tlb_flush_entries, hc->rep_cnt);
+			if (vcpu->plane_level) {
+				kvm_make_request(KVM_REQ_HV_TLB_FLUSH, v);
+				kvm_vcpu_kick(v);
+			}
 		}
 
-		kvm_make_vcpus_request_mask(kvm, KVM_REQ_HV_TLB_FLUSH, vcpu_mask);
+		if (!vcpu->plane_level)
+			kvm_make_vcpus_request_mask(kvm, KVM_REQ_HV_TLB_FLUSH,
+						    vcpu_mask);
 	} else {
 		struct kvm_vcpu_hv *hv_v;
 
-- 
2.50.1

[RFC PATCH 18/55] KVM: x86/mmu: free obsolete roots of all planes

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

KVM_REQ_MMU_FREE_OBSOLETE_ROOTS lives in the request set shared by all
planes of a vCPU. A fast zap (e.g. memslot deletion) invalidates
*every* plane's TDP roots and raises the request once per vCPU, but
whichever plane enters the guest next consumes the shared bit and
frees only its own obsolete roots. The other planes keep an obsolete
root with a valid-looking root.hpa, kvm_mmu_reload() does nothing for
them, and every page fault bails out in is_page_fault_stale() with
RET_PF_RETRY: the vCPU livelocks taking the same EPT violation
forever. Observed with a Windows VSM guest spinning at ~18k EPT
violations/s after a memslot update during boot, the root obsolete
(inv=1 obs=1) and the request already consumed (req=0).

Service every plane twin in kvm_mmu_free_obsolete_roots(). The
inactive twins cannot be running (all planes of a vCPU share one
thread), so their MMUs are stable.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/mmu/mmu.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 177b2e71bf35..cdebb5c324fe 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6234,8 +6234,23 @@ static void __kvm_mmu_free_obsolete_roots(struct kvm *kvm, struct kvm_mmu *mmu)
 
 void kvm_mmu_free_obsolete_roots(struct kvm_vcpu *vcpu)
 {
-	__kvm_mmu_free_obsolete_roots(vcpu->kvm, &vcpu->arch.root_mmu);
-	__kvm_mmu_free_obsolete_roots(vcpu->kvm, &vcpu->arch.guest_mmu);
+	struct kvm_vcpu *v;
+	int i;
+
+	/*
+	 * KVM_REQ_MMU_FREE_OBSOLETE_ROOTS is tracked in the request set
+	 * shared by all of the vCPU's planes, so the plane that consumes
+	 * the request is not necessarily the plane whose root is obsolete.
+	 * Service every plane: the inactive twins cannot be running (all
+	 * planes of a vCPU share one thread), so their MMUs are stable.
+	 */
+	for (i = 0; i < KVM_MAX_PLANES; i++) {
+		v = vcpu->common->vcpus[i];
+		if (!v)
+			continue;
+		__kvm_mmu_free_obsolete_roots(v->kvm, &v->arch.root_mmu);
+		__kvm_mmu_free_obsolete_roots(v->kvm, &v->arch.guest_mmu);
+	}
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_free_obsolete_roots);
 
-- 
2.50.1

[RFC PATCH 19/55] KVM: x86: hyperv: don't arm PV EOI for a higher plane's VP assist page

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Registering the VP assist page arms the PV EOI / APIC assist
optimisation through a gfn_to_hva cache, and that cache refuses ranges
carrying RW memory protections in plane 0's view. The Windows secure
kernel places its VTL1 assist page in memory it has protected against
VTL0, so the MSR write failed and VTL1 had no assist page at all: no
VTL entry reasons, no VTL-return register control, no boot.

Register the page without arming PV EOI when the vCPU is in a plane
above 0. KVM then never sets the no-EOI-required bit for that plane,
so the guest always performs a real EOI; only the fast path is lost.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/hyperv.c | 24 ++++++++++++++++++++----
 1 file changed, 20 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 797c5cbfbbaf..ea80536caa48 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -1609,7 +1609,8 @@ static int kvm_hv_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data, bool host)
 
 		if (!(data & HV_X64_MSR_VP_ASSIST_PAGE_ENABLE)) {
 			hv_vcpu->hv_vapic = data;
-			if (kvm_lapic_set_pv_eoi(vcpu, 0, 0))
+			if (!vcpu->plane_level &&
+			    kvm_lapic_set_pv_eoi(vcpu, 0, 0))
 				return 1;
 			break;
 		}
@@ -1627,9 +1628,24 @@ static int kvm_hv_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data, bool host)
 			return 1;
 		hv_vcpu->hv_vapic = data;
 		kvm_vcpu_mark_page_dirty(vcpu, gfn);
-		if (kvm_lapic_set_pv_eoi(vcpu,
-					    gfn_to_gpa(gfn) | KVM_MSR_ENABLED,
-					    sizeof(struct hv_vp_assist_page)))
+
+		/*
+		 * The gfn_to_hva cache behind the PV EOI / APIC assist
+		 * optimization validates against plane 0's memory
+		 * attributes, and a higher plane's VP assist page may
+		 * live on a page the plane protected against plane 0
+		 * (the Windows secure kernel's does). Don't arm PV EOI
+		 * there: pv_eoi.msr_val stays clear, which keeps
+		 * kvm_hv_assist_page_enabled() false for the plane, so
+		 * every assist-page consumer behind it (PV EOI, nested
+		 * enlightened VMCS, the SVM VP-assist read) is disabled
+		 * and KVM never touches the page's contents.  The guest
+		 * always performs a real EOI.
+		 */
+		if (!vcpu->plane_level &&
+		    kvm_lapic_set_pv_eoi(vcpu,
+					 gfn_to_gpa(gfn) | KVM_MSR_ENABLED,
+					 sizeof(struct hv_vp_assist_page)))
 			return 1;
 		break;
 	}
-- 
2.50.1

[RFC PATCH 20/55] KVM: x86: allow KVM_TRANSLATE on plane vCPU fds

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The VMM needs to translate guest virtual addresses in the context of a
specific plane to implement HvCallTranslateVirtualAddress, which a
higher VTL uses to resolve a lower VTL's buffer addresses.
KVM_TRANSLATE already works on any vCPU; permit it on plane fds too.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/x86.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index eb5d4932641e..1a9a241264ca 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -6310,6 +6310,7 @@ bool kvm_arch_is_vcpu_plane_ioctl(unsigned ioctl)
 	case KVM_GET_XSAVE2:
 	case KVM_SET_XSAVE:
 	case KVM_GET_REG_LIST:
+	case KVM_TRANSLATE:
 		return true;
 	default:
 		return false;
-- 
2.50.1

[RFC PATCH 21/55] KVM: x86: hyper-v: handle VTL call/return in the kernel

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Every VTL transition exits to userspace, which performs the switch
with a dozen cross-thread vCPU ioctls. A Windows guest boots VSM with
hundreds of thousands of transitions, so boot takes *minutes* of
wall-clock time in switch overhead alone.

Handle HvCallVtlCall and HvCallVtlReturn in kvm_hv_hypercall() when
userspace opts in with KVM_CAP_HYPERV_VTL_SWITCH. The switch copies
the VTL-shared state directly between the plane vCPUs (GPRs except
RSP, CR2), maintains the entered VTL's assist-page control area (entry
reason, caller's RAX/RCX), and restores RAX/RCX on a non-fast VTL
return. The suspended plane keeps its deferred hypercall completion
pending, RIP still at the vmcall, exactly as userspace-driven
switching leaves it.

The shared guest FPU is carried across in the plane run loop, once the
outgoing plane's FPU has been saved, by copying the legacy FXSAVE area
between the twins. The source's FP and SSE XSTATE_BV bits are
propagated set-or-clear: when those components are in init state,
XSAVES skipped them and the vzalloc'ed legacy area is zeroes, and
blindly marking them valid would load MXCSR=0 (every SIMD exception
unmasked) and FCW=0 into the other VTL as architectural state.

Transitions the kernel cannot handle (no twin plane vCPU) still go to
userspace, with the vmcall retired and the completion dropped before
the exit. A completion left armed across the suspension fires on
whatever entry next resumes the plane — after a forced higher-VTL
entry that is an arbitrary interrupted instruction, which must not
have RAX overwritten and RIP skipped. The reference hypervisor
likewise retires the instruction before switching, which also makes
the suspended RIP a higher VTL observes through GetVpRegisters point
past the vmcall. An hv_vtl_switch_limit module parameter caps how many
transitions the kernel handles, with the rest forwarded; that is the
bisection knob for behavioural divergence between the two switch
paths.

Also report the originating plane of every forwarded hypercall in a
previously-padding field of struct kvm_hyperv_exit, so userspace can
tell which VTL issued a hypercall without tracking every switch
itself.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 Documentation/virt/kvm/api.rst  |  59 +++++++++-
 arch/x86/include/asm/kvm_host.h |  14 +++
 arch/x86/kvm/hyperv.c           | 193 ++++++++++++++++++++++++++++++++
 arch/x86/kvm/x86.c              |  69 ++++++++++++
 include/hyperv/hvgdk_mini.h     |   2 +
 include/linux/kvm_host.h        |  21 ++++
 include/uapi/linux/kvm.h        |   3 +-
 virt/kvm/kvm_main.c             |   1 +
 8 files changed, 360 insertions(+), 2 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 8232356b1bb0..c9283994014a 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -7126,8 +7126,10 @@ EOI was received.
   #define KVM_EXIT_HYPERV_SYNIC          1
   #define KVM_EXIT_HYPERV_HCALL          2
   #define KVM_EXIT_HYPERV_SYNDBG         3
+  #define KVM_EXIT_HYPERV_VTL_INTERRUPT  4
+  #define KVM_EXIT_HYPERV_REG_INTERCEPT  5
 			__u32 type;
-			__u32 pad1;
+			__u32 plane;
 			union {
 				struct {
 					__u32 msr;
@@ -7141,6 +7143,15 @@ EOI was received.
 					__u64 result;
 					__u64 params[2];
 				} hcall;
+				struct {
+					__u32 reg;
+					__u8 insn_len;
+					__u8 pad[3];
+					__u64 base;
+					__u32 limit;
+					__u32 pad2;
+					__u64 rip;
+				} reg_intercept;
 				struct {
 					__u32 msr;
 					__u32 pad2;
@@ -7158,6 +7169,11 @@ EOI was received.
 Indicates that the VCPU exits into userspace to process some tasks
 related to Hyper-V emulation.
 
+'plane' identifies the plane the exit concerns: the plane of the vCPU
+that triggered the exit, or for KVM_EXIT_HYPERV_VTL_INTERRUPT the
+higher plane to be entered.  On kernels without plane support the
+field was padding and reads as zero.
+
 Valid values for 'type' are:
 
 	- KVM_EXIT_HYPERV_SYNIC -- synchronously notify user-space about
@@ -7172,6 +7188,25 @@ Hyper-V Synthetic debugger state change. Notification is used to either update
 the pending_page location or to send a control command (send the buffer located
 in send_page or recv a buffer to recv_page).
 
+	- KVM_EXIT_HYPERV_VTL_INTERRUPT -- the plane named by 'plane' has a
+
+pending event that preempts the current, lower plane (TLFS: higher VTL
+interrupts preempt lower VTLs).  The VMM is expected to perform a VTL
+entry into that plane.  Delivered only while in-kernel VTL switching
+(KVM_CAP_HYPERV_VTL_SWITCH, see section 7.50) is disabled, and
+single-shot: if the VMM declines the entry, the event waits in the
+target plane for a cooperative entry rather than re-triggering the
+exit.
+
+	- KVM_EXIT_HYPERV_REG_INTERCEPT -- a descriptor-table write that a
+
+higher plane intercepted (KVM_CAP_HYPERV_REG_INTERCEPT) has been
+diverted to userspace instead of being applied.  'reg_intercept'
+carries the HV_X64_REGISTER_* id of the written register, the
+descriptor-table base and limit the guest tried to load, and the RIP
+and length of the writing instruction.  The vCPU's RIP has been
+rewound to that instruction; the higher plane owns its completion.
+
 ::
 
 		/* KVM_EXIT_ARM_NISV / KVM_EXIT_ARM_LDST64B */
@@ -9064,6 +9099,28 @@ For Hyper-V, unknown hypercalls are passed to userspace without checking their
 parameter pages.  The fault GPA is in the physical address space of the
 VM managed by userspace, after nested GPA translation.
 
+7.50 KVM_CAP_HYPERV_VTL_SWITCH
+------------------------------
+
+:Architectures: x86
+:Target: VM
+:Parameters: args[0] is 1 to enable in-kernel VTL switches, 0 to disable them
+
+When enabled, KVM handles the non-fast forms of the HvCallVtlCall and
+HvCallVtlReturn hypercalls itself where it can: the calling plane's
+vCPU is suspended mid-hypercall and its twin in the target plane
+resumed, with the VTL-shared register state (GPRs except RSP, CR2, FPU)
+carried across and the entered VTL's assist page updated as the TLFS
+specifies.  A transition KVM cannot handle (a fast-form call, no twin
+vCPU in the target plane, a target that cannot be entered) falls back
+to the KVM_CAP_HYPERV_FORWARD_HCALL path, so a VMM enabling this
+capability must still implement the userspace switch and should enable
+forwarding too.
+
+While this capability is disabled, a pending event for a higher plane
+instead surfaces as a KVM_EXIT_HYPERV_VTL_INTERRUPT exit (see section
+5) telling the VMM to enter that plane.
+
 8. Other capabilities.
 ======================
 
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 6ee85022cdfc..537ea6cded7a 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -731,6 +731,14 @@ struct kvm_vcpu_hv {
 		u64 vm_id;
 		u32 vp_id;
 	} nested;
+
+	/*
+	 * RAX/RCX of the lower VTL captured at VTL entry, restored on a
+	 * non-fast VTL return when this plane has no VP assist page to
+	 * carry them in its VTL control area.
+	 */
+	u64 vtl_entry_rax;
+	u64 vtl_entry_rcx;
 };
 
 struct kvm_hypervisor_cpuid {
@@ -1202,6 +1210,12 @@ enum hv_tsc_page_status {
 /* Hyper-V emulation context */
 struct kvm_hv {
 	struct mutex hv_lock;
+	bool hv_inkernel_vtl_switch;
+	/*
+	 * In-kernel VTL switches performed by this VM, counted against the
+	 * hv_vtl_switch_limit debug knob; saturates instead of wrapping.
+	 */
+	atomic_t vtl_switch_count;
 	/*
 	 * GUEST_OS_ID and the hypercall page are per-VTL instances in the
 	 * TLFS; bank them per plane so a higher VTL enabling its own
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index ea80536caa48..2f57f89cd8ea 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -2725,6 +2725,7 @@ static int kvm_hv_hypercall_userspace_exit(struct kvm_vcpu *vcpu,
 
 	vcpu->run->exit_reason = KVM_EXIT_HYPERV;
 	vcpu->run->hyperv.type = KVM_EXIT_HYPERV_HCALL;
+	vcpu->run->hyperv.plane = vcpu->plane_level;
 	vcpu->run->hyperv.u.hcall.input = hc->param;
 	vcpu->run->hyperv.u.hcall.params[0] = hc->ingpa;
 	vcpu->run->hyperv.u.hcall.params[1] = hc->outgpa;
@@ -2732,6 +2733,150 @@ static int kvm_hv_hypercall_userspace_exit(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
+
+/* VTL control area in the VP assist page (TLFS struct hv_vp_vtl_control). */
+#define HV_VTL_CONTROL_OFFSET		8
+#define HV_VTL_ENTRY_REASON_VTL_CALL	1
+#define HV_VTL_RETURN_FAST_BIT		BIT_ULL(0)
+
+struct hv_vtl_control {
+	u32 vtl_entry_reason;
+	u8 vina_asserted;
+	u8 reserved[3];
+	u64 vtl_ret_x64rax;
+	u64 vtl_ret_x64rcx;
+} __packed;
+
+static u64 hv_vtl_assist_page_gpa(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
+
+	if (!hv_vcpu ||
+	    !(hv_vcpu->hv_vapic & HV_X64_MSR_VP_ASSIST_PAGE_ENABLE))
+		return 0;
+	return hv_vcpu->hv_vapic & HV_X64_MSR_VP_ASSIST_PAGE_ADDRESS_MASK;
+}
+
+/*
+ * In-kernel VTL call/return: switch between the plane vCPUs directly
+ * instead of bouncing every transition through userspace.  Keeps the
+ * userspace contract: VTL-shared register state (GPRs except RSP, CR2,
+ * FPU) follows the switch, the entered VTL's assist page gets the entry
+ * reason and the caller's RAX/RCX, and a non-fast VTL return restores
+ * the lower VTL's RAX/RCX from the returning VTL's control area.  The
+ * suspending plane stays suspended mid-hypercall, with
+ * hcall_resume_pending set: its pending completion delivers the
+ * instruction skip and the resume RAX when a later switch re-enters
+ * the plane.
+ *
+ * Returns -ENOENT when the transition cannot be handled here (no twin
+ * plane); the caller forwards the hypercall to userspace unchanged.
+ */
+static int hv_vtl_switch_limit = -1;
+module_param(hv_vtl_switch_limit, int, 0644);
+
+static int kvm_hv_vtl_switch(struct kvm_vcpu *vcpu, bool is_return)
+{
+	static const int shared_gprs[] = {
+		VCPU_REGS_RAX, VCPU_REGS_RBX, VCPU_REGS_RCX, VCPU_REGS_RDX,
+		VCPU_REGS_RSI, VCPU_REGS_RDI, VCPU_REGS_RBP,
+		VCPU_REGS_R8, VCPU_REGS_R9, VCPU_REGS_R10, VCPU_REGS_R11,
+		VCPU_REGS_R12, VCPU_REGS_R13, VCPU_REGS_R14, VCPU_REGS_R15,
+	};
+	int cur = vcpu->plane_level;
+	int to = is_return ? cur - 1 : cur + 1;
+	u64 rax = kvm_rax_read(vcpu), rcx = kvm_rcx_read(vcpu);
+	bool fast = is_return && (rax & HV_VTL_RETURN_FAST_BIT);
+	int limit = READ_ONCE(hv_vtl_switch_limit);
+	struct hv_vtl_control ctl;
+	struct kvm_vcpu *target;
+	u64 assist, resume;
+	int i;
+
+	if (to < 0 || to >= KVM_MAX_PLANES || !vcpu->common->vcpus[to])
+		return -ENOENT;
+	target = vcpu->common->vcpus[to];
+
+	/*
+	 * Debug budget on in-kernel switches, counted per VM and only for
+	 * switches that pass the eligibility checks above (a forwarded
+	 * switch consumes none).  The counter saturates so a guest cannot
+	 * wrap it back under an operator-set limit.
+	 */
+	if (limit >= 0 &&
+	    atomic_fetch_add_unless(&to_kvm_hv(vcpu->kvm)->vtl_switch_count,
+				    1, INT_MAX) >= limit)
+		return -ENOENT;
+
+	if (is_return && !fast) {
+		assist = hv_vtl_assist_page_gpa(vcpu);
+		if (assist) {
+			if (kvm_vcpu_read_guest(vcpu,
+						assist + HV_VTL_CONTROL_OFFSET,
+						&ctl, sizeof(ctl)))
+				return -ENOENT;
+		} else {
+			ctl.vtl_ret_x64rax = to_hv_vcpu(vcpu)->vtl_entry_rax;
+			ctl.vtl_ret_x64rcx = to_hv_vcpu(vcpu)->vtl_entry_rcx;
+		}
+	}
+
+	/* VTL-shared state: GPRs except RSP (RIP/RFLAGS stay private). */
+	for (i = 0; i < ARRAY_SIZE(shared_gprs); i++)
+		kvm_register_write_raw(target, shared_gprs[i],
+				       kvm_register_read_raw(vcpu,
+							     shared_gprs[i]));
+	target->arch.cr2 = vcpu->arch.cr2;
+	vcpu->common->vtl_fpu_from = cur;
+
+	if (!is_return) {
+		assist = hv_vtl_assist_page_gpa(target);
+		if (assist) {
+			ctl = (struct hv_vtl_control) {
+				.vtl_entry_reason = HV_VTL_ENTRY_REASON_VTL_CALL,
+				.vtl_ret_x64rax = rax,
+				.vtl_ret_x64rcx = rcx,
+			};
+			kvm_vcpu_write_guest(vcpu,
+					     assist + HV_VTL_CONTROL_OFFSET,
+					     &ctl, sizeof(ctl));
+		}
+		if (to_hv_vcpu(target)) {
+			to_hv_vcpu(target)->vtl_entry_rax = rax;
+			to_hv_vcpu(target)->vtl_entry_rcx = rcx;
+		}
+		resume = rax;
+	} else {
+		if (!fast)
+			kvm_rcx_write(target, ctl.vtl_ret_x64rcx);
+		resume = fast ? rax : ctl.vtl_ret_x64rax;
+	}
+
+	/*
+	 * The target resumes in a hypercall of its own (any time after its
+	 * very first entry).  Its pending completion delivers the deferred
+	 * instruction skip and the resume RAX from the shared run struct,
+	 * exactly as when userspace drives the switch; until then its
+	 * suspended RIP stays at the vmcall, which the higher VTL can
+	 * observe (and rewrite) through HvCallGetVpRegisters.
+	 */
+	if (target->arch.complete_userspace_io)
+		vcpu->run->hyperv.u.hcall.result = resume;
+
+	/*
+	 * Suspend the current plane in its uncompleted hypercall; the
+	 * switch back will provide its resume value the same way.
+	 */
+	vcpu->arch.complete_userspace_io = kvm_hv_hypercall_complete_userspace;
+
+	vcpu->plane_user_stopped = true;
+	kvm_vcpu_set_plane_stopped(vcpu);
+	target->plane_user_stopped = false;
+	kvm_vcpu_set_plane_runnable(target);
+	kvm_make_request(KVM_REQ_PLANE_RESCHED, vcpu);
+	return 0;
+}
+
 int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
@@ -2898,6 +3043,54 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 			break;
 		}
 		return kvm_hv_hypercall_userspace_exit(vcpu, &hc);
+	case HVCALL_VTL_CALL:
+	case HVCALL_VTL_RETURN:
+		/* Neither call takes a variable header or rep count. */
+		if (unlikely(hc.rep || hc.var_cnt)) {
+			ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
+			break;
+		}
+		if (to_kvm_hv(vcpu->kvm)->hv_inkernel_vtl_switch &&
+		    !hc.fast &&
+		    !kvm_hv_vtl_switch(vcpu, hc.code == HVCALL_VTL_RETURN)) {
+			/*
+			 * Suspended mid-hypercall; the pending completion
+			 * delivers the result when the plane resumes.
+			 */
+			return 1;
+		}
+		if (to_kvm_hv(vcpu->kvm)->hv_forward_hcalls) {
+			/*
+			 * Retire the vmcall before the plane switch, like the
+			 * reference hypervisor: the VMM fully owns the resumed
+			 * context (GPR copy, control-area RAX/RCX restore).  A
+			 * completion deferred across the suspension fires on
+			 * whatever entry resumes this plane next - after a
+			 * forced higher-VTL entry that can be an arbitrary
+			 * interrupted instruction, which then has RAX and RIP
+			 * rewritten with stale hypercall results.
+			 */
+			ret = kvm_hv_hypercall_userspace_exit(vcpu, &hc);
+			if (!ret) {
+				/*
+				 * With guest TF and KVM_GUESTDBG_SINGLESTEP
+				 * the skip prepares KVM_EXIT_DEBUG in the
+				 * same run union, overwriting the exit just
+				 * prepared; the retired vmcall would then be
+				 * silently dropped and the VMM never perform
+				 * the switch.  Re-prepare the hypercall exit
+				 * in that case: the VMM must see it, at the
+				 * price of one missed step event.
+				 */
+				if (!kvm_skip_emulated_instruction(vcpu))
+					kvm_hv_hypercall_userspace_exit(vcpu,
+									&hc);
+				vcpu->arch.complete_userspace_io = NULL;
+				return 0;
+			}
+			break;
+		}
+		fallthrough;
 	default:
 		if (to_kvm_hv(vcpu->kvm)->hv_forward_hcalls)
 			return kvm_hv_hypercall_userspace_exit(vcpu, &hc);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 1a9a241264ca..aa8e14a45568 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -4884,6 +4884,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_HYPERV_ENFORCE_CPUID:
 	case KVM_CAP_SYS_HYPERV_CPUID:
 	case KVM_CAP_HYPERV_FORWARD_HCALL:
+	case KVM_CAP_HYPERV_VTL_SWITCH:
 #endif
 	case KVM_CAP_HCALL_FAULT_EXIT:
 	case KVM_CAP_PCI_SEGMENT:
@@ -5912,6 +5913,12 @@ static int kvm_vcpu_ioctl_x86_set_xsave(struct kvm_vcpu *vcpu,
 	 */
 	xstate->xsave.header.xfeatures &= ~vcpu->arch.guest_fpu.fpstate->xfd;
 
+	/*
+	 * Cancel a pending VTL FPU carry: it would overwrite the state
+	 * installed here on the next KVM_RUN.
+	 */
+	kvm_vcpu_cancel_vtl_fpu_carry(vcpu->common);
+
 	return fpu_copy_uabi_to_guest_fpstate(&vcpu->arch.guest_fpu,
 					      guest_xsave->region,
 					      kvm_caps.supported_xcr0,
@@ -6920,6 +6927,13 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		to_kvm_hv(kvm)->hv_forward_hcalls = !!cap->args[0];
 		r = 0;
 		break;
+	case KVM_CAP_HYPERV_VTL_SWITCH:
+		r = -EINVAL;
+		if (cap->args[0] & ~1ULL)
+			break;
+		to_kvm_hv(kvm)->hv_inkernel_vtl_switch = !!cap->args[0];
+		r = 0;
+		break;
 #endif
 	case KVM_CAP_X2APIC_API:
 		r = -EINVAL;
@@ -12366,6 +12380,55 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu_plane0)
 		if (vcpu == NULL)
 			return -EINVAL;
 
+#ifdef CONFIG_KVM_HYPERV
+		/*
+		 * An in-kernel VTL switch carries the shared guest FPU
+		 * state to the newly selected plane.  The source plane's
+		 * FPU was saved to its fpstate when its inner run loop
+		 * exited, so the buffers are current.
+		 */
+		if (common->vtl_fpu_from >= 0) {
+			struct kvm_vcpu *prev;
+
+			prev = common->vtl_fpu_from < KVM_MAX_PLANES ?
+			       common->vcpus[common->vtl_fpu_from] : NULL;
+			common->vtl_fpu_from = -1;
+			if (prev && prev != vcpu &&
+			    prev->arch.guest_fpu.fpstate &&
+			    vcpu->arch.guest_fpu.fpstate) {
+				struct fpstate *dst = vcpu->arch.guest_fpu.fpstate;
+				struct fpstate *src = prev->arch.guest_fpu.fpstate;
+
+				memcpy(&dst->regs.fxsave, &src->regs.fxsave,
+				       sizeof(struct fxregs_state));
+				/*
+				 * Only the legacy FXSAVE area (x87, XMM,
+				 * MXCSR) is VTL-shared; the XSAVE header and
+				 * extended components (PKRU in particular)
+				 * stay per-VTL.  Carry the source's FP/SSE
+				 * xfeatures bits along with the area: when
+				 * the source's components are in init state
+				 * the saved legacy area holds no valid
+				 * register image (XSAVES skipped it), and
+				 * marking it present would make XRSTOR load
+				 * zeroes as architectural state, MXCSR=0
+				 * included.  Without XSAVE the fpstate is
+				 * only the 512-byte legacy area and there is
+				 * no header to maintain.
+				 */
+				if (boot_cpu_has(X86_FEATURE_XSAVE)) {
+					u64 carry = src->regs.xsave.header.xfeatures &
+						    (XFEATURE_MASK_FP | XFEATURE_MASK_SSE);
+
+					dst->regs.xsave.header.xfeatures =
+						(dst->regs.xsave.header.xfeatures &
+						 ~(XFEATURE_MASK_FP | XFEATURE_MASK_SSE)) |
+						carry;
+				}
+			}
+		}
+#endif
+
 		common->plane_switch = false;
 
 		ret = __kvm_arch_vcpu_ioctl_run(vcpu);
@@ -12986,6 +13049,12 @@ int kvm_arch_vcpu_ioctl_set_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu)
 	fxsave->rdp = fpu->last_dp;
 	memcpy(fxsave->xmm_space, fpu->xmm, sizeof(fxsave->xmm_space));
 
+	/*
+	 * Cancel a pending VTL FPU carry: it would overwrite the state
+	 * installed here on the next KVM_RUN.
+	 */
+	kvm_vcpu_cancel_vtl_fpu_carry(vcpu->common);
+
 	vcpu_put(vcpu);
 	return 0;
 }
diff --git a/include/hyperv/hvgdk_mini.h b/include/hyperv/hvgdk_mini.h
index 6a4e8b9d570f..e1b6ee863777 100644
--- a/include/hyperv/hvgdk_mini.h
+++ b/include/hyperv/hvgdk_mini.h
@@ -482,6 +482,8 @@ union hv_vp_assist_msr_contents {	 /* HV_REGISTER_VP_ASSIST_PAGE */
 #define HVCALL_ASSERT_VIRTUAL_INTERRUPT			0x0094
 #define HVCALL_CREATE_PORT				0x0095
 #define HVCALL_CONNECT_PORT				0x0096
+#define HVCALL_VTL_CALL				0x0011
+#define HVCALL_VTL_RETURN			0x0012
 #define HVCALL_START_VP					0x0099
 #define HVCALL_GET_VP_INDEX_FROM_APIC_ID		0x009a
 #define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_SPACE	0x00af
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 7071ff74d4ea..79b3ec2b95b8 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -338,6 +338,16 @@ struct kvm_vcpu_common {
 	/* Currently active VCPU */
 	struct kvm_vcpu *current_vcpu;
 
+#ifdef CONFIG_KVM_HYPERV
+	/*
+	 * An in-kernel VTL switch moved execution away from this plane
+	 * (-1: none pending); the run loop copies the (VTL-shared) guest
+	 * FPU state from it when it selects the new plane, after the old
+	 * one's FPU has been saved.
+	 */
+	int vtl_fpu_from;
+#endif
+
 	struct kvm_vcpu *vcpus[KVM_MAX_PLANES];
 
 	/* Locks */
@@ -458,6 +468,17 @@ struct kvm_vcpu {
 	unsigned plane_level;
 };
 
+/*
+ * Cancel (or initialise away) a pending in-kernel VTL switch FPU carry.
+ * A no-op on configurations without the Hyper-V VTL switch.
+ */
+static inline void kvm_vcpu_cancel_vtl_fpu_carry(struct kvm_vcpu_common *common)
+{
+#ifdef CONFIG_KVM_HYPERV
+	common->vtl_fpu_from = -1;
+#endif
+}
+
 void kvm_vcpu_set_plane_runnable(struct kvm_vcpu *vcpu);
 void kvm_vcpu_set_plane_stopped(struct kvm_vcpu *vcpu);
 struct kvm_vcpu *kvm_vcpu_select_plane(struct kvm_vcpu *vcpu);
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index b55e28be5181..b50f92135acd 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -100,7 +100,7 @@ struct kvm_hyperv_exit {
 #define KVM_EXIT_HYPERV_HCALL          2
 #define KVM_EXIT_HYPERV_SYNDBG         3
 	__u32 type;
-	__u32 pad1;
+	__u32 plane;
 	union {
 		struct {
 			__u32 msr;
@@ -1025,6 +1025,7 @@ struct kvm_enable_cap {
  */
 #define KVM_HV_HCALL_RESULT_RAISE_UD	(1ULL << 63)
 #define KVM_CAP_HCALL_FAULT_EXIT 252
+#define KVM_CAP_HYPERV_VTL_SWITCH 253
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index ef23ee3ce04a..ef6e59a15846 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -474,6 +474,7 @@ static int kvm_vcpu_init_common(struct kvm_vcpu *vcpu, struct kvm *kvm, unsigned
 		goto out_drop_counter;
 	}
 	common->run = page_address(page);
+	kvm_vcpu_cancel_vtl_fpu_carry(common);
 
 	mutex_init(&common->mutex);
 
-- 
2.50.1

[RFC PATCH 22/55] KVM: x86: load each plane twin while updating it after KVM_SET_CPUID

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

kvm_vcpu_ioctl_set_cpuid() runs kvm_vcpu_after_set_cpuid() on every
plane twin, but only the calling vCPU is loaded. The vendor callback
writes VMCS/VMCB controls, and on VMX those writes go to whatever VMCS
is loaded on the current pCPU: the caller's, not the twin's. The
twin's controls shadow is updated anyway, so the stale values in its
real VMCS can never be corrected later; a twin created before
KVM_SET_CPUID2 keeps its empty-CPUID controls (RDSEED/RDRAND exiting
among them), and the guest's higher VTL takes a spurious #UD on its
first RDSEED.

Swap the twin in as the loaded vCPU around its update. Plane ioctls
all share the common vCPU mutex, so the twin cannot be loaded
elsewhere concurrently.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/cpuid.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index fe8a793b1643..f744e7845843 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -592,7 +592,20 @@ static int kvm_set_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid_entry2 *e2,
 #ifdef CONFIG_KVM_XEN
 		v->arch.xen.cpuid = kvm_get_hypervisor_cpuid(vcpu, XEN_SIGNATURE);
 #endif
+		/*
+		 * The ioctl loaded the calling vCPU; updating a twin's
+		 * VMCS/VMCB controls requires that twin to be the loaded
+		 * one, so swap it in around the update.
+		 */
+		if (v != vcpu) {
+			vcpu_put(vcpu);
+			vcpu_load(v);
+		}
 		kvm_vcpu_after_set_cpuid(v);
+		if (v != vcpu) {
+			vcpu_put(v);
+			vcpu_load(vcpu);
+		}
 	}
 
 success:
-- 
2.50.1

[RFC PATCH 23/55] WIP: userspace VTL interrupt entry and register intercepts

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Windows VSM boot stalls and PatchGuard-induced triple faults traced to
missing host behaviours, all confirmed against the reference VSM host:

- A pending interrupt for a suspended higher VTL must preempt the
  lower VTL. In userspace-switch mode, report it as a new
  KVM_EXIT_HYPERV_VTL_INTERRUPT exit — from the halted/block path too,
  since a halted lower VTL never reaches vcpu_enter_guest() — so
  userspace enters the higher VTL itself and its view of the active
  VTL stays in sync. kvm_higher_plane_with_event() tests raw
  pending-ness (kvm_apic_has_interrupt), because a suspended plane
  carries a stale apicv_active that makes
  kvm_cpu_has_injectable_intr() report false.

- The in-kernel forced entry, like the in-kernel VTL switch, writes
  the entered VTL's control area through the *entered* plane's
  context, since the assist page may be protected against the lower
  VTL; on failure a VTL call forwards to userspace and a forced entry
  stays pending. The refusal conditions live in
  kvm_hv_vtl_entry_blocked(), which also gates
  kvm_higher_plane_with_event() in in-kernel mode: an armed event on a
  plane that cannot be entered (never run, stopped by userspace,
  parked on a non-hypercall completion owning the shared run union)
  must not keep waking a halted lower VTL that has no way to consume
  it. A plane which has never entered the guest holds reset state and
  userspace owns the HvCallEnableVpVtl tracking (the reference host
  fails such calls with HV_STATUS_INVALID_VP_STATE), so hand it the
  switch and leave the event pending. The resume value is
  guest-derived in every switch flavour, so
  KVM_HV_HCALL_RESULT_RAISE_UD is masked out of it, and a target with
  no completion armed receives it directly in RAX, as the TLFS
  register-restore contract on VTL return requires.

- RAX/RCX of the lower VTL are captured at in-kernel VTL entry and
  restored on an assist-less, non-fast return; vtl_entry_regs_valid
  tracks whether the capture is live. A return finding it clear (the
  entry was performed by userspace, or predates a migration — the
  capture has no GET/SET exposure) is forwarded to userspace, which
  owns restoring the lower VTL's registers.

- plane_user_stopped is a pure userspace-policy latch: only
  KVM_SET_PLANE_RUN_STATE writes it, and no runnable plane can carry
  it (it is cleared at the single RUNNABLE choke point). In-kernel
  suspensions get their own plane_kernel_stopped flag, reported and
  restorable as KVM_PLANE_RUN_STATE_KERNEL_STOPPED, with the same
  plane-0 event gating. The in-kernel VTL switch and the forced entry
  refuse a user-stopped target (forwarding the switch to userspace or
  leaving the event pending), and the kvm_vcpu_select_plane() fallback
  refuses a user-stopped plane 0: KVM_RUN fails with -EINVAL,
  surfacing the VMM's ordering bug instead of resuming a plane the VMM
  parked mid-hypercall.

- The secure kernel arms HvRegisterCrInterceptControl
  descriptor-table write intercepts for VTL0, then PatchGuard runs its
  protected patching window under an intercepted lidt. The write must
  *not* be applied — the window would execute with a genuinely null
  IDT and the first event triple-faults the guest. Add a per-vCPU
  KVM_CAP_HYPERV_REG_INTERCEPT that arms the control word, trap the
  descriptor-table instructions (SECONDARY_EXEC_DESC, dropped again
  when the intercepts disarm), and divert armed writes into a
  KVM_EXIT_HYPERV_REG_INTERCEPT exit with RIP rewound to the
  intercepted instruction: the higher VTL owns completion, and
  Windows' patch-protection probe matches its reflected #GP(0) against
  the probe RIP.

- HvCallSendSyntheticClusterIpi(Ex) honours the input's in_vtl field
  (both the memory and fast forms carry it after the vector),
  delivering to the resolved plane's twins; reserved bits 7:5 are
  rejected with HV_STATUS_INVALID_HYPERCALL_INPUT before anything is
  delivered, as on the reference host.

- hv_inkernel_vtl_switch is written under kvm->lock but read
  locklessly from vCPU context; the accesses carry
  WRITE_ONCE/READ_ONCE so each individual check reads a coherent
  value. A VMM that flips the cap while vCPUs run owns the
  consequences of catching a transition in flight.

Also includes debug instrumentation, to be stripped before posting.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 Documentation/virt/kvm/api.rst  |  31 ++--
 arch/x86/hyperv/hv_apic.c       |   3 +-
 arch/x86/include/asm/kvm_host.h |  21 ++-
 arch/x86/kvm/hyperv.c           | 257 +++++++++++++++++++++++++++++---
 arch/x86/kvm/hyperv.h           |   3 +
 arch/x86/kvm/vmx/vmx.c          |  23 ++-
 arch/x86/kvm/x86.c              | 210 +++++++++++++++++++++++++-
 include/hyperv/hvgdk_mini.h     |   3 +-
 include/hyperv/hvhdk_mini.h     |   3 +-
 include/linux/kvm_host.h        |  12 ++
 include/uapi/linux/kvm.h        |  13 ++
 virt/kvm/kvm_main.c             |  35 ++++-
 12 files changed, 571 insertions(+), 43 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index c9283994014a..3ef57b3b050b 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6628,19 +6628,32 @@ a vCPU fd of every plane, not just plane 0.
 :Returns: 0 on success, <0 on error
 
 Reads or sets the per-plane run state of a vCPU:
-KVM_PLANE_RUN_STATE_STOPPED (0) or KVM_PLANE_RUN_STATE_RUNNABLE (1).
+KVM_PLANE_RUN_STATE_STOPPED (0), KVM_PLANE_RUN_STATE_RUNNABLE (1) or
+KVM_PLANE_RUN_STATE_KERNEL_STOPPED (2).
 
 When a vCPU thread calls KVM_RUN, KVM executes the lowest-numbered plane
-whose vCPU is runnable, falling back to plane 0.  A VMM implementing a
-software privilege model (such as Hyper-V VSM) switches planes by
-marking the target plane's vCPU runnable and the current one stopped
-before re-entering KVM_RUN.  Pending events for plane 0 make it runnable
-again on their own only as long as userspace has not explicitly stopped
-it with this ioctl; a VMM-stopped plane 0 keeps its events pending until
-the VMM makes it runnable, so a software privilege model can keep a
-higher plane running undisturbed.  The ioctl is accepted on any plane's
+whose vCPU is runnable, falling back to plane 0.  If no plane is
+runnable and userspace has stopped plane 0, KVM_RUN fails with -EINVAL
+instead of overriding the stop.  A VMM implementing a software privilege
+model (such as Hyper-V VSM) switches planes by marking the target
+plane's vCPU runnable and the current one stopped before re-entering
+KVM_RUN.  Pending events for plane 0 make it runnable again on their own
+only as long as userspace has not explicitly stopped it with this ioctl;
+a VMM-stopped plane 0 keeps its events pending until the VMM makes it
+runnable, so a software privilege model can keep a higher plane running
+undisturbed.  In-kernel plane transitions never override a userspace
+stop: a guest-initiated VTL switch targeting a stopped plane is
+forwarded to userspace instead.  The ioctl is accepted on any plane's
 vCPU fd.
 
+KVM_PLANE_RUN_STATE_KERNEL_STOPPED is reported for a plane that KVM
+itself suspended (such as the lower plane of an in-kernel VTL switch);
+such a plane becomes runnable again on the matching in-kernel transition
+without userspace involvement.  Setting it stops the plane without
+recording a userspace stop, so restoring previously read run states
+round-trips a kernel suspension without converting it into userspace
+stop policy.
+
 The run state is distinct from KVM_GET/SET_MP_STATE, which describes
 the architectural state of a vCPU within its plane.
 
diff --git a/arch/x86/hyperv/hv_apic.c b/arch/x86/hyperv/hv_apic.c
index a8de503def37..533304c29603 100644
--- a/arch/x86/hyperv/hv_apic.c
+++ b/arch/x86/hyperv/hv_apic.c
@@ -127,7 +127,8 @@ static bool __send_ipi_mask_ex(const struct cpumask *mask, int vector,
 		goto ipi_mask_ex_done;
 
 	ipi_arg->vector = vector;
-	ipi_arg->reserved = 0;
+	ipi_arg->in_vtl.as_uint8 = 0;
+	memset(ipi_arg->reserved, 0, sizeof(ipi_arg->reserved));
 	ipi_arg->vp_set.valid_bank_mask = 0;
 
 	/*
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 537ea6cded7a..74810e0e9ed7 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -732,13 +732,32 @@ struct kvm_vcpu_hv {
 		u32 vp_id;
 	} nested;
 
+	/*
+	 * HvRegisterCrInterceptControl value the higher VTL armed for the
+	 * lower VTL (userspace VSM): descriptor-table write bits divert
+	 * the emulated write into a KVM_EXIT_HYPERV_REG_INTERCEPT exit.
+	 */
+	u64 reg_intercept_control;
+	struct {
+		u32 reg;
+		u8 insn_len;
+		bool pending;
+		u64 base;
+		u32 limit;
+		u64 rip;
+	} reg_intercept;
+
 	/*
 	 * RAX/RCX of the lower VTL captured at VTL entry, restored on a
 	 * non-fast VTL return when this plane has no VP assist page to
-	 * carry them in its VTL control area.
+	 * carry them in its VTL control area.  Only valid between an
+	 * in-kernel VTL entry and the matching return: a plane entered by
+	 * userspace, or restored from migration, has no captured values
+	 * and its assist-less return is forwarded to userspace.
 	 */
 	u64 vtl_entry_rax;
 	u64 vtl_entry_rcx;
+	bool vtl_entry_regs_valid;
 };
 
 struct kvm_hypervisor_cpuid {
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 2f57f89cd8ea..ca31054ed065 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -2364,7 +2364,8 @@ static s64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 }
 
 static void kvm_hv_send_ipi_to_many(struct kvm *kvm, u32 vector,
-				    u64 *sparse_banks, u64 valid_bank_mask)
+				    u64 *sparse_banks, u64 valid_bank_mask,
+				    int target_plane)
 {
 	struct kvm_lapic_irq irq = {
 		.delivery_mode = APIC_DM_FIXED,
@@ -2374,13 +2375,25 @@ static void kvm_hv_send_ipi_to_many(struct kvm *kvm, u32 vector,
 	unsigned long i;
 
 	kvm_for_each_vcpu(i, vcpu, kvm) {
+		struct kvm_vcpu *target = vcpu;
+
 		if (sparse_banks &&
 		    !hv_is_vp_in_sparse_set(kvm_hv_get_vpindex(vcpu),
 					    valid_bank_mask, sparse_banks))
 			continue;
 
+		/*
+		 * TLFS: the IPI targets the VPs' in_vtl (or the sender's
+		 * VTL): deliver into that plane's APIC.
+		 */
+		if (target_plane) {
+			target = vcpu->common->vcpus[target_plane];
+			if (!target)
+				continue;
+		}
+
 		/* We fail only when APIC is disabled */
-		kvm_apic_set_irq(vcpu, &irq, NULL);
+		kvm_apic_set_irq(target, &irq, NULL);
 	}
 }
 
@@ -2393,6 +2406,8 @@ static int kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 	struct hv_send_ipi send_ipi;
 	u64 valid_bank_mask;
 	u32 vector;
+	u8 in_vtl = 0;
+	int target_plane;
 	bool all_cpus;
 	int r;
 
@@ -2410,13 +2425,25 @@ static int kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 				return HV_STATUS_INVALID_HYPERCALL_INPUT;
 			sparse_banks[0] = send_ipi.cpu_mask;
 			vector = send_ipi.vector;
+			in_vtl = send_ipi.in_vtl.as_uint8;
 		} else {
 			/* 'reserved' part of hv_send_ipi should be 0 */
-			if (unlikely(hc->ingpa >> 32 != 0))
+			if (unlikely(hc->ingpa >> 40 != 0))
 				return HV_STATUS_INVALID_HYPERCALL_INPUT;
 			sparse_banks[0] = hc->outgpa;
 			vector = (u32)hc->ingpa;
+			in_vtl = (hc->ingpa >> 32) & 0xff;
 		}
+
+		/*
+		 * Only hv_input_vtl.target_vtl (bits 3:0) and
+		 * use_target_vtl (bit 4) are defined; reserved_z (7:5)
+		 * must be zero in every form of the hypercall, before
+		 * anything is delivered.
+		 */
+		if (unlikely(in_vtl & 0xe0))
+			return HV_STATUS_INVALID_HYPERCALL_INPUT;
+
 		all_cpus = false;
 		valid_bank_mask = BIT_ULL(0);
 
@@ -2426,12 +2453,18 @@ static int kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 			if (unlikely(kvm_read_guest(kvm, hc->ingpa, &send_ipi_ex,
 						    sizeof(send_ipi_ex))))
 				return HV_STATUS_INVALID_HYPERCALL_INPUT;
+			in_vtl = send_ipi_ex.in_vtl.as_uint8;
 		} else {
 			send_ipi_ex.vector = (u32)hc->ingpa;
 			send_ipi_ex.vp_set.format = hc->outgpa;
 			send_ipi_ex.vp_set.valid_bank_mask = sse128_lo(hc->xmm[0]);
+			in_vtl = (hc->ingpa >> 32) & 0xff;
 		}
 
+		/* As above: hv_input_vtl bits 7:5 are reserved. */
+		if (unlikely(in_vtl & 0xe0))
+			return HV_STATUS_INVALID_HYPERCALL_INPUT;
+
 		trace_kvm_hv_send_ipi_ex(send_ipi_ex.vector,
 					 send_ipi_ex.vp_set.format,
 					 send_ipi_ex.vp_set.valid_bank_mask);
@@ -2463,10 +2496,19 @@ static int kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 	if ((vector < HV_IPI_LOW_VECTOR) || (vector > HV_IPI_HIGH_VECTOR))
 		return HV_STATUS_INVALID_HYPERCALL_INPUT;
 
+	/* TLFS: in_vtl bit 4 selects an explicit target VTL. */
+	if (in_vtl & 0x10)
+		target_plane = in_vtl & 0xf;
+	else
+		target_plane = vcpu->plane_level;
+	if (target_plane >= KVM_MAX_PLANES)
+		return HV_STATUS_INVALID_HYPERCALL_INPUT;
+
 	if (all_cpus)
-		kvm_hv_send_ipi_to_many(kvm, vector, NULL, 0);
+		kvm_hv_send_ipi_to_many(kvm, vector, NULL, 0, target_plane);
 	else
-		kvm_hv_send_ipi_to_many(kvm, vector, sparse_banks, valid_bank_mask);
+		kvm_hv_send_ipi_to_many(kvm, vector, sparse_banks,
+					valid_bank_mask, target_plane);
 
 ret_success:
 	return HV_STATUS_SUCCESS;
@@ -2772,6 +2814,116 @@ static u64 hv_vtl_assist_page_gpa(struct kvm_vcpu *vcpu)
  * Returns -ENOENT when the transition cannot be handled here (no twin
  * plane); the caller forwards the hypercall to userspace unchanged.
  */
+/*
+ * Enter a higher VTL because it has a pending interrupt (TLFS: higher
+ * VTL interrupts preempt lower VTL execution).  The lower VTL is
+ * suspended mid-instruction, not in a hypercall, so no completion is
+ * pended for it: it resumes at the preempted RIP when the higher VTL
+ * returns, and the entered VTL's control area carries the INTERRUPT
+ * entry reason with the preempted RAX/RCX.
+ */
+#define HV_VTL_ENTRY_REASON_INTERRUPT	2
+
+/*
+ * States in which the in-kernel paths must not enter a plane:
+ *
+ *  - never entered the guest: it still holds reset state (userspace owns
+ *    HvCallEnableVpVtl and runs a plane's first entry itself);
+ *  - stopped by userspace with KVM_SET_PLANE_RUN_STATE: the VMM owns
+ *    restarting it;
+ *  - parked on a non-hypercall completion (MMIO, PIO): the completion
+ *    owns the shared run union, which the resume protocol would clobber.
+ *
+ * The forced interrupt entry refuses such a target (the event stays
+ * pending), and kvm_higher_plane_with_event() must not report its
+ * pending event as a wakeup either, or a halted lower VTL spins through
+ * hlt/wake with nothing able to consume the event.
+ */
+bool kvm_hv_vtl_entry_blocked(struct kvm_vcpu *target)
+{
+	if (target->arch.last_vmentry_cpu == -1)
+		return true;
+
+	if (target->plane_user_stopped)
+		return true;
+
+	if (target->arch.complete_userspace_io &&
+	    target->arch.complete_userspace_io != kvm_hv_hypercall_complete_userspace)
+		return true;
+
+	return false;
+}
+
+bool kvm_hv_vtl_interrupt_entry(struct kvm_vcpu *vcpu,
+				struct kvm_vcpu *target)
+{
+	u64 rax = kvm_rax_read(vcpu), rcx = kvm_rcx_read(vcpu);
+	struct hv_vtl_control ctl;
+	u64 assist;
+	int i;
+
+	static const int shared_gprs[] = {
+		VCPU_REGS_RAX, VCPU_REGS_RBX, VCPU_REGS_RCX, VCPU_REGS_RDX,
+		VCPU_REGS_RSI, VCPU_REGS_RDI, VCPU_REGS_RBP,
+		VCPU_REGS_R8, VCPU_REGS_R9, VCPU_REGS_R10, VCPU_REGS_R11,
+		VCPU_REGS_R12, VCPU_REGS_R13, VCPU_REGS_R14, VCPU_REGS_R15,
+	};
+
+	/* The event stays pending until the plane can be entered. */
+	if (kvm_hv_vtl_entry_blocked(target))
+		return false;
+
+	/*
+	 * Write the entered plane's control area through the entered
+	 * plane's vCPU: the write must pass that plane's own memory
+	 * attributes and memslot view, not the suspending plane's (which
+	 * typically has no access to the higher VTL's assist page).  On
+	 * failure leave both planes untouched; the event stays pending in
+	 * the target and the entry is retried.
+	 */
+	assist = hv_vtl_assist_page_gpa(target);
+	if (assist) {
+		ctl = (struct hv_vtl_control) {
+			.vtl_entry_reason = HV_VTL_ENTRY_REASON_INTERRUPT,
+			.vtl_ret_x64rax = rax,
+			.vtl_ret_x64rcx = rcx,
+		};
+		if (kvm_vcpu_write_guest(target, assist + HV_VTL_CONTROL_OFFSET,
+					 &ctl, sizeof(ctl)))
+			return false;
+	}
+
+	for (i = 0; i < ARRAY_SIZE(shared_gprs); i++)
+		kvm_register_write_raw(target, shared_gprs[i],
+				       kvm_register_read_raw(vcpu,
+							     shared_gprs[i]));
+	target->arch.cr2 = vcpu->arch.cr2;
+	vcpu->common->vtl_fpu_from = vcpu->plane_level;
+
+	if (to_hv_vcpu(target)) {
+		to_hv_vcpu(target)->vtl_entry_rax = rax;
+		to_hv_vcpu(target)->vtl_entry_rcx = rcx;
+		to_hv_vcpu(target)->vtl_entry_regs_valid = true;
+	}
+
+	/*
+	 * The target resumes its suspended hypercall, if it has one.  The
+	 * resume value is the shared RAX as copied from the entering plane
+	 * just above: GPRs are VTL-shared, so the retired hypercall's RAX
+	 * must match what every other shared register already shows.  The
+	 * value is guest-derived; strip the VMM-only RAISE_UD control bit
+	 * so the completion cannot be made to inject #UD by the guest.
+	 */
+	if (target->arch.complete_userspace_io)
+		vcpu->run->hyperv.u.hcall.result = kvm_rax_read(target) &
+						   ~KVM_HV_HCALL_RESULT_RAISE_UD;
+
+	vcpu->plane_kernel_stopped = true;
+	kvm_vcpu_set_plane_stopped(vcpu);
+	kvm_vcpu_set_plane_runnable(target);
+	return true;
+}
+
 static int hv_vtl_switch_limit = -1;
 module_param(hv_vtl_switch_limit, int, 0644);
 
@@ -2797,6 +2949,35 @@ static int kvm_hv_vtl_switch(struct kvm_vcpu *vcpu, bool is_return)
 		return -ENOENT;
 	target = vcpu->common->vcpus[to];
 
+	/*
+	 * A VMM-stopped plane stays stopped until the VMM restarts it.
+	 * Forward the switch so userspace can apply its own policy.
+	 */
+	if (target->plane_user_stopped)
+		return -ENOENT;
+
+	/*
+	 * The resume protocol below delivers the result through the
+	 * target's armed hypercall completion.  A target parked on any
+	 * other completion (MMIO, PIO) owns the shared run union; writing
+	 * hcall.result over it would feed the pending completion garbage
+	 * and lose the resume value.  Forward such a switch to userspace.
+	 */
+	if (target->arch.complete_userspace_io &&
+	    target->arch.complete_userspace_io != kvm_hv_hypercall_complete_userspace)
+		return -ENOENT;
+
+	/*
+	 * Userspace owns HvCallEnableVpVtl state; the kernel only knows a
+	 * plane vCPU exists, and VMMs create those at setup time.  Refuse
+	 * to call into a plane that has never entered the guest: it still
+	 * holds reset state, and the reference host fails such a call with
+	 * HV_STATUS_INVALID_VP_STATE.  Forwarding lets userspace apply its
+	 * enablement tracking (and run the plane's first entry itself).
+	 */
+	if (!is_return && target->arch.last_vmentry_cpu == -1)
+		return -ENOENT;
+
 	/*
 	 * Debug budget on in-kernel switches, counted per VM and only for
 	 * switches that pass the eligibility checks above (a forwarded
@@ -2816,9 +2997,39 @@ static int kvm_hv_vtl_switch(struct kvm_vcpu *vcpu, bool is_return)
 						&ctl, sizeof(ctl)))
 				return -ENOENT;
 		} else {
+			/*
+			 * No captured entry registers to restore: this plane
+			 * was last entered by userspace (or the capture was
+			 * lost across migration).  Forward the return so
+			 * userspace restores the lower VTL's RAX/RCX itself.
+			 */
+			if (!to_hv_vcpu(vcpu)->vtl_entry_regs_valid)
+				return -ENOENT;
+
 			ctl.vtl_ret_x64rax = to_hv_vcpu(vcpu)->vtl_entry_rax;
 			ctl.vtl_ret_x64rcx = to_hv_vcpu(vcpu)->vtl_entry_rcx;
 		}
+	} else if (!is_return) {
+		/*
+		 * Write the entered VTL's control area through the entered
+		 * plane's vCPU: the write must pass that plane's memory
+		 * attributes and memslot view, and the higher VTL normally
+		 * protects its assist page against the caller's plane.
+		 * Fail before any state moves (mirroring the read above);
+		 * userspace then owns the switch.
+		 */
+		assist = hv_vtl_assist_page_gpa(target);
+		if (assist) {
+			ctl = (struct hv_vtl_control) {
+				.vtl_entry_reason = HV_VTL_ENTRY_REASON_VTL_CALL,
+				.vtl_ret_x64rax = rax,
+				.vtl_ret_x64rcx = rcx,
+			};
+			if (kvm_vcpu_write_guest(target,
+						 assist + HV_VTL_CONTROL_OFFSET,
+						 &ctl, sizeof(ctl)))
+				return -ENOENT;
+		}
 	}
 
 	/* VTL-shared state: GPRs except RSP (RIP/RFLAGS stay private). */
@@ -2830,26 +3041,17 @@ static int kvm_hv_vtl_switch(struct kvm_vcpu *vcpu, bool is_return)
 	vcpu->common->vtl_fpu_from = cur;
 
 	if (!is_return) {
-		assist = hv_vtl_assist_page_gpa(target);
-		if (assist) {
-			ctl = (struct hv_vtl_control) {
-				.vtl_entry_reason = HV_VTL_ENTRY_REASON_VTL_CALL,
-				.vtl_ret_x64rax = rax,
-				.vtl_ret_x64rcx = rcx,
-			};
-			kvm_vcpu_write_guest(vcpu,
-					     assist + HV_VTL_CONTROL_OFFSET,
-					     &ctl, sizeof(ctl));
-		}
 		if (to_hv_vcpu(target)) {
 			to_hv_vcpu(target)->vtl_entry_rax = rax;
 			to_hv_vcpu(target)->vtl_entry_rcx = rcx;
+			to_hv_vcpu(target)->vtl_entry_regs_valid = true;
 		}
 		resume = rax;
 	} else {
 		if (!fast)
 			kvm_rcx_write(target, ctl.vtl_ret_x64rcx);
 		resume = fast ? rax : ctl.vtl_ret_x64rax;
+		to_hv_vcpu(vcpu)->vtl_entry_regs_valid = false;
 	}
 
 	/*
@@ -2859,9 +3061,27 @@ static int kvm_hv_vtl_switch(struct kvm_vcpu *vcpu, bool is_return)
 	 * exactly as when userspace drives the switch; until then its
 	 * suspended RIP stays at the vmcall, which the higher VTL can
 	 * observe (and rewrite) through HvCallGetVpRegisters.
+	 *
+	 * The resume value is guest-derived in all three cases (the
+	 * caller's RAX on a call or a fast return, the control area on a
+	 * non-fast return): strip the VMM-only RAISE_UD control bit so
+	 * the completion cannot be made to inject #UD into the suspended
+	 * plane.
 	 */
+	resume &= ~KVM_HV_HCALL_RESULT_RAISE_UD;
 	if (target->arch.complete_userspace_io)
 		vcpu->run->hyperv.u.hcall.result = resume;
+	else
+		/*
+		 * No completion is armed when the target's suspension
+		 * already retired its vmcall (forwarded switch) or was a
+		 * forced interrupt entry: RIP is already where the plane
+		 * resumes, so deliver the resume RAX directly.  Without
+		 * this the GPR copy above leaves the switching plane's raw
+		 * RAX in place and a non-fast return never restores the
+		 * control-area value.
+		 */
+		kvm_rax_write(target, resume);
 
 	/*
 	 * Suspend the current plane in its uncompleted hypercall; the
@@ -2869,9 +3089,8 @@ static int kvm_hv_vtl_switch(struct kvm_vcpu *vcpu, bool is_return)
 	 */
 	vcpu->arch.complete_userspace_io = kvm_hv_hypercall_complete_userspace;
 
-	vcpu->plane_user_stopped = true;
+	vcpu->plane_kernel_stopped = true;
 	kvm_vcpu_set_plane_stopped(vcpu);
-	target->plane_user_stopped = false;
 	kvm_vcpu_set_plane_runnable(target);
 	kvm_make_request(KVM_REQ_PLANE_RESCHED, vcpu);
 	return 0;
@@ -3050,7 +3269,7 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 			ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
 			break;
 		}
-		if (to_kvm_hv(vcpu->kvm)->hv_inkernel_vtl_switch &&
+		if (READ_ONCE(to_kvm_hv(vcpu->kvm)->hv_inkernel_vtl_switch) &&
 		    !hc.fast &&
 		    !kvm_hv_vtl_switch(vcpu, hc.code == HVCALL_VTL_RETURN)) {
 			/*
diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h
index 0593fe4333d3..0b81ba482201 100644
--- a/arch/x86/kvm/hyperv.h
+++ b/arch/x86/kvm/hyperv.h
@@ -102,6 +102,9 @@ static inline bool kvm_hv_hypercall_enabled(struct kvm_vcpu *vcpu)
 }
 
 int kvm_hv_hypercall(struct kvm_vcpu *vcpu);
+bool kvm_hv_vtl_interrupt_entry(struct kvm_vcpu *vcpu,
+				struct kvm_vcpu *target);
+bool kvm_hv_vtl_entry_blocked(struct kvm_vcpu *target);
 
 void kvm_hv_irq_routing_update(struct kvm *kvm);
 int kvm_hv_synic_set_irq(struct kvm_kernel_irq_routing_entry *e, struct kvm *kvm,
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index d10aa5f60cad..e1c5f1608b09 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -5640,7 +5640,14 @@ static int handle_desc(struct kvm_vcpu *vcpu)
 	 */
 	BUILD_BUG_ON(KVM_POSSIBLE_CR4_GUEST_BITS & X86_CR4_UMIP);
 
-	WARN_ON_ONCE(!kvm_is_cr4_bit_set(vcpu, X86_CR4_UMIP));
+	/*
+	 * Descriptor-table exits are also armed when a higher VTL registers
+	 * DT intercepts (reg_intercept_control); only an exit with neither
+	 * UMIP emulation nor an armed intercept indicates a stale
+	 * SECONDARY_EXEC_DESC control.
+	 */
+	WARN_ON_ONCE(!kvm_is_cr4_bit_set(vcpu, X86_CR4_UMIP) &&
+		     !(to_hv_vcpu(vcpu) && to_hv_vcpu(vcpu)->reg_intercept_control));
 	return kvm_emulate_instruction(vcpu, 0);
 }
 
@@ -8044,6 +8051,20 @@ void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
 
 	/* Refresh #PF interception to account for MAXPHYADDR changes. */
 	vmx_update_exception_bitmap(vcpu);
+
+	/*
+	 * Trap descriptor-table instructions when a higher VTL armed
+	 * register intercepts on them, and stop trapping when the
+	 * intercepts are disarmed.  Preserve the control when UMIP
+	 * emulation needs it (CR4.UMIP owns it then).
+	 */
+	if (vmcs_config.cpu_based_2nd_exec_ctrl & SECONDARY_EXEC_DESC) {
+		if (to_hv_vcpu(vcpu) && to_hv_vcpu(vcpu)->reg_intercept_control)
+			secondary_exec_controls_setbit(vmx, SECONDARY_EXEC_DESC);
+		else if (!(vmx_umip_emulated() &&
+			   kvm_is_cr4_bit_set(vcpu, X86_CR4_UMIP)))
+			secondary_exec_controls_clearbit(vmx, SECONDARY_EXEC_DESC);
+	}
 }
 
 static __init u64 vmx_get_perf_capabilities(void)
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index aa8e14a45568..374330f61eb2 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -4885,6 +4885,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_SYS_HYPERV_CPUID:
 	case KVM_CAP_HYPERV_FORWARD_HCALL:
 	case KVM_CAP_HYPERV_VTL_SWITCH:
+	case KVM_CAP_HYPERV_REG_INTERCEPT:
 #endif
 	case KVM_CAP_HCALL_FAULT_EXIT:
 	case KVM_CAP_PCI_SEGMENT:
@@ -6144,6 +6145,19 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
 			}
 			return r;
 		}
+	case KVM_CAP_HYPERV_REG_INTERCEPT: {
+		struct kvm_vcpu_hv *hv_vcpu;
+		int r;
+
+		r = kvm_hv_vcpu_init(vcpu);
+		if (r)
+			return r;
+		hv_vcpu = to_hv_vcpu(vcpu);
+		hv_vcpu->reg_intercept_control = cap->args[0];
+		/* Refresh the descriptor-table exiting controls. */
+		kvm_x86_call(vcpu_after_set_cpuid)(vcpu);
+		return 0;
+	}
 	case KVM_CAP_HYPERV_DIRECT_TLBFLUSH:
 		if (!kvm_x86_ops.enable_l2_tlb_flush)
 			return -ENOTTY;
@@ -6931,7 +6945,7 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		r = -EINVAL;
 		if (cap->args[0] & ~1ULL)
 			break;
-		to_kvm_hv(kvm)->hv_inkernel_vtl_switch = !!cap->args[0];
+		WRITE_ONCE(to_kvm_hv(kvm)->hv_inkernel_vtl_switch, !!cap->args[0]);
 		r = 0;
 		break;
 #endif
@@ -8814,14 +8828,55 @@ static void emulator_get_idt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt)
 	kvm_x86_call(get_idt)(emul_to_vcpu(ctxt), dt);
 }
 
+/*
+ * Divert a descriptor-table write into a register-intercept exit when the
+ * higher VTL armed one: the write is not applied, and userspace delivers
+ * the TLFS register-intercept message to the higher VTL.
+ */
+static bool hv_divert_dt_write(struct x86_emulate_ctxt *ctxt, u32 hv_reg,
+			       const struct desc_ptr *dt, u64 ctl_bit)
+{
+#ifdef CONFIG_KVM_HYPERV
+	struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt);
+	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
+
+	if (!hv_vcpu || vcpu->plane_level ||
+	    !(hv_vcpu->reg_intercept_control & ctl_bit))
+		return false;
+
+	hv_vcpu->reg_intercept.reg = hv_reg;
+	hv_vcpu->reg_intercept.base = dt->address;
+	hv_vcpu->reg_intercept.limit = dt->size;
+	hv_vcpu->reg_intercept.rip = ctxt->eip;
+	hv_vcpu->reg_intercept.insn_len = ctxt->_eip - ctxt->eip;
+	hv_vcpu->reg_intercept.pending = true;
+	return true;
+#else
+	return false;
+#endif
+}
+
+#define HV_CR_INTERCEPT_GDTR_WRITE	BIT_ULL(15)
+#define HV_CR_INTERCEPT_IDTR_WRITE	BIT_ULL(16)
+#define HV_X64_REGISTER_IDTR_ID		0x00070000
+#define HV_X64_REGISTER_GDTR_ID		0x00070001
+
 static void emulator_set_gdt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt)
 {
+	if (hv_divert_dt_write(ctxt, HV_X64_REGISTER_GDTR_ID, dt,
+			       HV_CR_INTERCEPT_GDTR_WRITE))
+		return;
 	kvm_x86_call(set_gdt)(emul_to_vcpu(ctxt), dt);
 }
 
 static void emulator_set_idt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt)
 {
-	kvm_x86_call(set_idt)(emul_to_vcpu(ctxt), dt);
+	struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt);
+
+	if (hv_divert_dt_write(ctxt, HV_X64_REGISTER_IDTR_ID, dt,
+			       HV_CR_INTERCEPT_IDTR_WRITE))
+		return;
+	kvm_x86_call(set_idt)(vcpu, dt);
 }
 
 static unsigned long emulator_get_cached_segment_base(
@@ -9607,6 +9662,44 @@ int x86_decode_emulated_instruction(struct kvm_vcpu *vcpu, int emulation_type,
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(x86_decode_emulated_instruction);
 
+#ifdef CONFIG_KVM_HYPERV
+/*
+ * Complete a descriptor-table write that hv_divert_dt_write() diverted
+ * during the emulation that just finished: rewind RIP and build the
+ * KVM_EXIT_HYPERV_REG_INTERCEPT exit.  Consuming the record before
+ * x86_emulate_instruction() returns means it never outlives the emulation
+ * call: no stale rewind can fire on a later KVM_RUN, and disarming the cap
+ * or migrating the vCPU can never see a half-delivered intercept.
+ */
+static int complete_hv_reg_intercept(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
+
+	hv_vcpu->reg_intercept.pending = false;
+
+	/*
+	 * The intercepted instruction must not retire: the emulator advanced
+	 * RIP when it diverted the write, but the higher VTL owns
+	 * completion.  It either applies the write and advances RIP itself,
+	 * disarms the intercept so re-execution succeeds, or queues an
+	 * exception that must fault AT the instruction (the
+	 * kernel-patch-protection LIDT probe matches the faulting RIP
+	 * against the probe site).
+	 */
+	kvm_rip_write(vcpu, hv_vcpu->reg_intercept.rip);
+	vcpu->run->exit_reason = KVM_EXIT_HYPERV;
+	vcpu->run->hyperv.type = KVM_EXIT_HYPERV_REG_INTERCEPT;
+	vcpu->run->hyperv.plane = vcpu->plane_level;
+	vcpu->run->hyperv.u.reg_intercept.reg = hv_vcpu->reg_intercept.reg;
+	vcpu->run->hyperv.u.reg_intercept.insn_len =
+		hv_vcpu->reg_intercept.insn_len;
+	vcpu->run->hyperv.u.reg_intercept.base = hv_vcpu->reg_intercept.base;
+	vcpu->run->hyperv.u.reg_intercept.limit = hv_vcpu->reg_intercept.limit;
+	vcpu->run->hyperv.u.reg_intercept.rip = hv_vcpu->reg_intercept.rip;
+	return 0;
+}
+#endif
+
 int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 			    int emulation_type, void *insn, int insn_len)
 {
@@ -9826,6 +9919,15 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 	} else
 		vcpu->arch.emulate_regs_need_sync_to_vcpu = true;
 
+#ifdef CONFIG_KVM_HYPERV
+	{
+		struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
+
+		if (hv_vcpu && hv_vcpu->reg_intercept.pending)
+			r = complete_hv_reg_intercept(vcpu);
+	}
+#endif
+
 	return r;
 }
 
@@ -11083,6 +11185,77 @@ static int kvm_check_and_inject_events(struct kvm_vcpu *vcpu,
 	return r;
 }
 
+/*
+ * A pending interrupt for a higher plane preempts this one: the higher
+ * VTL's interrupts take priority over lower-VTL execution (TLFS 15.2.4),
+ * and the secure kernel relies on it, e.g. to freeze the other processors'
+ * VTL1 while starting a new one.
+ */
+static struct kvm_vcpu *kvm_higher_plane_with_event(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vcpu *higher;
+	int i;
+
+	for (i = vcpu->plane_level + 1; i < KVM_MAX_PLANES; i++) {
+		higher = vcpu->common->vcpus[i];
+		if (!higher)
+			continue;
+#ifdef CONFIG_KVM_HYPERV
+		/*
+		 * With in-kernel switching, an event on a plane the forced
+		 * entry would refuse (never run, user-stopped, parked on a
+		 * foreign completion) has no consumer until userspace
+		 * starts the plane; reporting it would wake a halted lower
+		 * VTL into an endless hlt/wake loop.  With userspace
+		 * switching the event is consumable via the interrupt exit,
+		 * so it does count.
+		 */
+		if (READ_ONCE(to_kvm_hv(vcpu->kvm)->hv_inkernel_vtl_switch) &&
+		    kvm_hv_vtl_entry_blocked(higher))
+			continue;
+#endif
+		/*
+		 * Test raw pending-ness, not injectability:
+		 * kvm_cpu_has_injectable_intr() reports false on an
+		 * APICv-active vCPU, and a suspended plane may carry a
+		 * stale apicv_active until it next runs.  A pending
+		 * synthetic-timer request counts as well: it can only be
+		 * processed (and its interrupt delivered) once the higher
+		 * plane runs, so it would otherwise never fire while the
+		 * plane is suspended.
+		 */
+		if (kvm_cpu_has_extint(higher) ||
+		    kvm_apic_has_interrupt(higher) != -1 ||
+		    kvm_test_request(KVM_REQ_HV_STIMER, higher))
+			return higher;
+	}
+	return NULL;
+}
+
+#ifdef CONFIG_KVM_HYPERV
+/*
+ * With userspace owning VTL transitions, report a pending higher-plane
+ * event as an exit so userspace can enter the higher VTL.  Returns true
+ * when the exit was prepared.
+ */
+static bool kvm_hv_vtl_interrupt_exit(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vcpu *higher;
+
+	if (READ_ONCE(to_kvm_hv(vcpu->kvm)->hv_inkernel_vtl_switch))
+		return false;
+
+	higher = kvm_higher_plane_with_event(vcpu);
+	if (!higher)
+		return false;
+
+	vcpu->run->exit_reason = KVM_EXIT_HYPERV;
+	vcpu->run->hyperv.type = KVM_EXIT_HYPERV_VTL_INTERRUPT;
+	vcpu->run->hyperv.plane = higher->plane_level;
+	return true;
+}
+#endif
+
 static inline bool kvm_check_plane0_events(struct kvm_vcpu *vcpu)
 {
 	struct kvm_vcpu *vcpu_plane0;
@@ -11548,13 +11721,30 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 	}
 
 	if (kvm_check_plane0_events(vcpu) &&
-	    !vcpu->common->vcpus[0]->plane_user_stopped) {
+	    !vcpu->common->vcpus[0]->plane_user_stopped &&
+	    !vcpu->common->vcpus[0]->plane_kernel_stopped) {
 		kvm_vcpu_set_plane_runnable(vcpu->common->vcpus[0]);
 
 		kvm_make_request(KVM_REQ_EVENT, vcpu);
 		kvm_make_request(KVM_REQ_PLANE_RESCHED, vcpu);
 	}
 
+#ifdef CONFIG_KVM_HYPERV
+	if (kvm_hv_vtl_interrupt_exit(vcpu)) {
+		r = 0;
+		goto out;
+	}
+
+	if (READ_ONCE(to_kvm_hv(vcpu->kvm)->hv_inkernel_vtl_switch)) {
+		struct kvm_vcpu *higher = kvm_higher_plane_with_event(vcpu);
+
+		if (higher && kvm_hv_vtl_interrupt_entry(vcpu, higher)) {
+			kvm_make_request(KVM_REQ_EVENT, higher);
+			kvm_make_request(KVM_REQ_PLANE_RESCHED, vcpu);
+		}
+	}
+#endif
+
 	if (kvm_check_request(KVM_REQ_PLANE_RESCHED, vcpu)) {
 		vcpu->common->plane_switch = true;
 		r = 0;
@@ -11904,7 +12094,8 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_has_events);
 int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu)
 {
 	return kvm_vcpu_running(vcpu) || vcpu->arch.pv.pv_unhalted ||
-	       kvm_vcpu_has_events(vcpu);
+	       kvm_vcpu_has_events(vcpu) ||
+	       kvm_higher_plane_with_event(vcpu);
 }
 
 /* Called within kvm->srcu read side.  */
@@ -11999,6 +12190,17 @@ static int vcpu_run(struct kvm_vcpu *vcpu)
 		if (r <= 0)
 			break;
 
+#ifdef CONFIG_KVM_HYPERV
+		/*
+		 * A halted lower VTL never reaches vcpu_enter_guest, but a
+		 * pending higher-VTL event must still preempt it.
+		 */
+		if (!kvm_vcpu_running(vcpu) && kvm_hv_vtl_interrupt_exit(vcpu)) {
+			r = 0;
+			break;
+		}
+#endif
+
 		kvm_clear_request(KVM_REQ_UNBLOCK, vcpu);
 		if (kvm_xen_has_pending_events(vcpu))
 			kvm_xen_inject_pending_events(vcpu);
diff --git a/include/hyperv/hvgdk_mini.h b/include/hyperv/hvgdk_mini.h
index e1b6ee863777..956e9fb7ec33 100644
--- a/include/hyperv/hvgdk_mini.h
+++ b/include/hyperv/hvgdk_mini.h
@@ -1346,7 +1346,8 @@ struct hv_input_set_vp_registers {
 /* HvCallSendSyntheticClusterIpi hypercall */
 struct hv_send_ipi {	 /* HV_INPUT_SEND_SYNTHETIC_CLUSTER_IPI */
 	u32 vector;
-	u32 reserved;
+	union hv_input_vtl in_vtl;
+	u8 reserved[3];
 	u64 cpu_mask;
 } __packed;
 
diff --git a/include/hyperv/hvhdk_mini.h b/include/hyperv/hvhdk_mini.h
index b4cb2fa26e9b..a2180b35f646 100644
--- a/include/hyperv/hvhdk_mini.h
+++ b/include/hyperv/hvhdk_mini.h
@@ -464,7 +464,8 @@ struct hv_input_unmap_device_interrupt {
 
 struct hv_send_ipi_ex { /* HV_INPUT_SEND_SYNTHETIC_CLUSTER_IPI_EX */
 	u32 vector;
-	u32 reserved;
+	union hv_input_vtl in_vtl;
+	u8 reserved[3];
 	struct hv_vpset vp_set;
 } __packed;
 
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 79b3ec2b95b8..a76615f2c3dc 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -427,8 +427,20 @@ struct kvm_vcpu {
 	 * Set when userspace stopped this plane with
 	 * KVM_SET_PLANE_RUN_STATE: pending plane-0 events then no longer
 	 * make plane 0 runnable on their own, the VMM owns the switch.
+	 * Only the ioctl sets it; in-kernel plane transitions never
+	 * override it, and kvm_vcpu_set_plane_runnable() clears it so a
+	 * runnable plane cannot carry a stale latch.
 	 */
 	bool plane_user_stopped;
+	/*
+	 * Set when an in-kernel VTL switch suspends this plane: plane-0
+	 * events are gated exactly as for a userspace stop, but
+	 * KVM_GET_PLANE_RUN_STATE reports the state as
+	 * KVM_PLANE_RUN_STATE_KERNEL_STOPPED so that a save/restore
+	 * round trip does not convert a transient kernel suspension
+	 * into userspace stop policy.
+	 */
+	bool plane_kernel_stopped;
 
 	/* S390 only */
 	bool valid_wakeup;
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index b50f92135acd..9e453f215e68 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -99,6 +99,8 @@ struct kvm_hyperv_exit {
 #define KVM_EXIT_HYPERV_SYNIC          1
 #define KVM_EXIT_HYPERV_HCALL          2
 #define KVM_EXIT_HYPERV_SYNDBG         3
+#define KVM_EXIT_HYPERV_VTL_INTERRUPT  4
+#define KVM_EXIT_HYPERV_REG_INTERCEPT  5
 	__u32 type;
 	__u32 plane;
 	union {
@@ -114,6 +116,15 @@ struct kvm_hyperv_exit {
 			__u64 result;
 			__u64 params[2];
 		} hcall;
+		struct {
+			__u32 reg;
+			__u8 insn_len;
+			__u8 pad[3];
+			__u64 base;
+			__u32 limit;
+			__u32 pad2;
+			__u64 rip;
+		} reg_intercept;
 		struct {
 			__u32 msr;
 			__u32 pad2;
@@ -1026,6 +1037,7 @@ struct kvm_enable_cap {
 #define KVM_HV_HCALL_RESULT_RAISE_UD	(1ULL << 63)
 #define KVM_CAP_HCALL_FAULT_EXIT 252
 #define KVM_CAP_HYPERV_VTL_SWITCH 253
+#define KVM_CAP_HYPERV_REG_INTERCEPT 254
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
@@ -1384,6 +1396,7 @@ struct kvm_s390_keyop {
 #define KVM_SET_PLANE_RUN_STATE	  _IOW(KVMIO, 0xe6, __u32)
 #define   KVM_PLANE_RUN_STATE_STOPPED	0
 #define   KVM_PLANE_RUN_STATE_RUNNABLE	1
+#define   KVM_PLANE_RUN_STATE_KERNEL_STOPPED	2
 
 /*
  * ioctls for vcpu fds
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index ef6e59a15846..4197496f7176 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -5023,9 +5023,14 @@ static long kvm_vcpu_ioctl(struct file *filp,
 		break;
 	}
 	case KVM_GET_PLANE_RUN_STATE: {
-		u32 state = vcpu->plane_state == RUNNABLE ?
-			    KVM_PLANE_RUN_STATE_RUNNABLE :
-			    KVM_PLANE_RUN_STATE_STOPPED;
+		u32 state;
+
+		if (vcpu->plane_state == RUNNABLE)
+			state = KVM_PLANE_RUN_STATE_RUNNABLE;
+		else if (vcpu->plane_user_stopped)
+			state = KVM_PLANE_RUN_STATE_STOPPED;
+		else
+			state = KVM_PLANE_RUN_STATE_KERNEL_STOPPED;
 
 		r = -EINVAL;
 		if (kvm_arch_max_planes(vcpu->kvm) <= 1)
@@ -5052,13 +5057,15 @@ static long kvm_vcpu_ioctl(struct file *filp,
 		if (copy_from_user(&state, argp, sizeof(state)))
 			goto out;
 		r = -EINVAL;
-		if (state > KVM_PLANE_RUN_STATE_RUNNABLE)
+		if (state > KVM_PLANE_RUN_STATE_KERNEL_STOPPED)
 			goto out;
 		if (state == KVM_PLANE_RUN_STATE_RUNNABLE) {
-			vcpu->plane_user_stopped = false;
 			kvm_vcpu_set_plane_runnable(vcpu);
 		} else {
-			vcpu->plane_user_stopped = true;
+			vcpu->plane_user_stopped =
+				state == KVM_PLANE_RUN_STATE_STOPPED;
+			vcpu->plane_kernel_stopped =
+				state == KVM_PLANE_RUN_STATE_KERNEL_STOPPED;
 			kvm_vcpu_set_plane_stopped(vcpu);
 		}
 		r = 0;
@@ -5319,6 +5326,13 @@ static struct file_operations kvm_plane_fops = {
 
 void kvm_vcpu_set_plane_runnable(struct kvm_vcpu *vcpu)
 {
+	/*
+	 * A runnable plane must not carry a stale stop latch: the latches
+	 * gate the plane-0 event wakeup, and KVM_GET_PLANE_RUN_STATE could
+	 * not report the inconsistency.
+	 */
+	vcpu->plane_user_stopped = false;
+	vcpu->plane_kernel_stopped = false;
 	vcpu->plane_state = RUNNABLE;
 	vcpu->common->plane_switch = true;
 	kvm_make_request(KVM_REQ_PLANE_RESCHED, vcpu);
@@ -5349,7 +5363,16 @@ struct kvm_vcpu *kvm_vcpu_select_plane(struct kvm_vcpu *vcpu)
 	}
 
 	if (ret == NULL) {
+		/*
+		 * Never force a userspace-stopped plane 0 to run: it may be
+		 * suspended mid-hypercall, and the VMM owns restarting it.
+		 * KVM_RUN fails instead, surfacing the VMM's ordering bug.
+		 */
 		ret = common->vcpus[0];
+		if (ret->plane_user_stopped)
+			return NULL;
+
+		ret->plane_kernel_stopped = false;
 		ret->plane_state = RUNNABLE;
 	}
 
-- 
2.50.1

[RFC PATCH 24/55] WIP: KVM: x86: deliver events to suspended higher planes

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

A suspended plane's events must preempt the lower plane running on the
same vCPU, but nothing told the running twin:

- An interrupt, local-APIC timer or Hyper-V stimer landing on a
  stopped twin only kicked the twin itself — a no-op while it is
  parked. Kick the lower twins too, with KVM_REQ_UNBLOCK so a sibling
  already past its event checks aborts the in-flight vmentry.
- A parked plane's lapic timer rode the VMX preemption timer, which
  only counts in guest mode, so it never fired while the plane was
  suspended. Hand it to the hrtimer when a non-zero plane's KVM_RUN
  ends.
- KVM_REQ_HV_STIMER was a shared request, so the running plane
  consumed the twin's request without processing the twin's stimers.
  Make it per-plane.
- kvm_higher_plane_with_event() now also reports a
  pending-but-not-injected apic timer.
- Suppress KVM_EXIT_HYPERV_SYNIC for non-zero planes: userspace's
  SynIC device models plane 0 only, and treated a twin's SIMP move as
  a plane-0 overlay relocation, carrying stale overlay contents to the
  new GPA.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/include/asm/kvm_host.h |  2 +-
 arch/x86/kvm/hyperv.c           | 27 +++++++++++++++++++
 arch/x86/kvm/lapic.c            | 46 ++++++++++++++++++++++++++++++++-
 arch/x86/kvm/x86.c              | 19 ++++++++++++--
 4 files changed, 90 insertions(+), 4 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 74810e0e9ed7..c54f47c52a59 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -112,7 +112,7 @@
 #define KVM_REQ_IOAPIC_EOI_EXIT		KVM_ARCH_PLANE_REQ(19)
 #define KVM_REQ_HV_RESET		KVM_ARCH_REQ(20)
 #define KVM_REQ_HV_EXIT			KVM_ARCH_REQ(21)
-#define KVM_REQ_HV_STIMER		KVM_ARCH_REQ(22)
+#define KVM_REQ_HV_STIMER		KVM_ARCH_PLANE_REQ(22)
 #define KVM_REQ_LOAD_EOI_EXITMAP	KVM_ARCH_REQ(23)
 #define KVM_REQ_GET_NESTED_STATE_PAGES	KVM_ARCH_REQ(24)
 #define KVM_REQ_APICV_UPDATE \
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index ca31054ed065..4551e011bf56 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -247,6 +247,15 @@ static void synic_exit(struct kvm_vcpu_hv_synic *synic, u32 msr)
 	struct kvm_vcpu *vcpu = hv_synic_to_vcpu(synic);
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
 
+	/*
+	 * Userspace's SynIC device models plane 0 only; a higher plane's
+	 * exit would be mistaken for a plane-0 page move and relocate the
+	 * overlay the VMM keeps for SIMP/SIEFP.  Higher-plane SynICs are
+	 * handled fully in-kernel.
+	 */
+	if (vcpu->plane_level)
+		return;
+
 	hv_vcpu->exit.type = KVM_EXIT_HYPERV_SYNIC;
 	hv_vcpu->exit.u.synic.msr = msr;
 	hv_vcpu->exit.u.synic.control = synic->control;
@@ -598,6 +607,24 @@ static void stimer_mark_pending(struct kvm_vcpu_hv_stimer *stimer,
 	kvm_make_request(KVM_REQ_HV_STIMER, vcpu);
 	if (vcpu_kick)
 		kvm_vcpu_kick(vcpu);
+
+	/*
+	 * A pending stimer on a suspended higher plane is only processed
+	 * once that plane runs; kick the lower twins so the running one
+	 * re-evaluates pending higher-plane events and forces entry.
+	 */
+	if (vcpu_kick && vcpu->plane_level) {
+		struct kvm_vcpu *twin;
+		int p;
+
+		for (p = 0; p < vcpu->plane_level; p++) {
+			twin = vcpu->common->vcpus[p];
+			if (twin) {
+				kvm_make_request(KVM_REQ_UNBLOCK, twin);
+				kvm_vcpu_kick(twin);
+			}
+		}
+	}
 }
 
 static void stimer_cleanup(struct kvm_vcpu_hv_stimer *stimer)
diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index dc7a08831a54..f89156c26972 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -1436,6 +1436,26 @@ static int __apic_accept_irq(struct kvm_lapic *apic, int delivery_mode,
 
 		kvm_x86_call(deliver_interrupt)(apic, delivery_mode,
 						trig_mode, vector);
+
+		/*
+		 * An interrupt for a higher plane must preempt a lower
+		 * plane running on the same vCPU: kicking only the (stopped)
+		 * target twin wakes a blocked sibling via the shared wait,
+		 * but a sibling executing in guest mode needs an explicit
+		 * kick to re-evaluate pending higher-plane events.
+		 */
+		if (vcpu->plane_level) {
+			struct kvm_vcpu *twin;
+			int p;
+
+			for (p = 0; p < vcpu->plane_level; p++) {
+				twin = vcpu->common->vcpus[p];
+				if (twin) {
+					kvm_make_request(KVM_REQ_UNBLOCK, twin);
+					kvm_vcpu_kick(twin);
+				}
+			}
+		}
 		break;
 
 	case APIC_DM_REMRD:
@@ -2060,6 +2080,24 @@ static void apic_timer_expired(struct kvm_lapic *apic, bool from_timer_fn)
 	kvm_make_request(KVM_REQ_UNBLOCK, vcpu);
 	if (from_timer_fn)
 		kvm_vcpu_kick(vcpu);
+
+	/*
+	 * A pending timer on a suspended higher plane only gets injected
+	 * once that plane runs again; kick the lower twins so the running
+	 * one re-evaluates pending higher-plane events and forces entry.
+	 */
+	if (vcpu->plane_level) {
+		struct kvm_vcpu *twin;
+		int p;
+
+		for (p = 0; p < vcpu->plane_level; p++) {
+			twin = vcpu->common->vcpus[p];
+			if (twin) {
+				kvm_make_request(KVM_REQ_UNBLOCK, twin);
+				kvm_vcpu_kick(twin);
+			}
+		}
+	}
 }
 
 static void start_sw_tscdeadline(struct kvm_lapic *apic)
@@ -3107,7 +3145,13 @@ int kvm_create_lapic(struct kvm_vcpu *vcpu)
 	 * that vCPU creation is guaranteed to run with a deterministic value,
 	 * the request will ensure the vCPU gets the correct state before VM-Entry.
 	 */
-	if (enable_apicv) {
+	/*
+	 * Higher-plane twins are software-switched and mostly suspended;
+	 * hardware interrupt virtualization on them only leaves pending
+	 * vectors in the PIR where a suspended plane's state cannot be
+	 * reliably observed.  Keep them on the legacy delivery path.
+	 */
+	if (enable_apicv && !vcpu->plane_level) {
 		apic->apicv_active = true;
 		kvm_make_request(KVM_REQ_APICV_UPDATE, vcpu);
 	}
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 374330f61eb2..4b12ec7ee100 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -11226,7 +11226,9 @@ static struct kvm_vcpu *kvm_higher_plane_with_event(struct kvm_vcpu *vcpu)
 		 */
 		if (kvm_cpu_has_extint(higher) ||
 		    kvm_apic_has_interrupt(higher) != -1 ||
-		    kvm_test_request(KVM_REQ_HV_STIMER, higher))
+		    kvm_test_request(KVM_REQ_HV_STIMER, higher) ||
+		    (higher->arch.apic &&
+		     atomic_read(&higher->arch.apic->lapic_timer.pending)))
 			return higher;
 	}
 	return NULL;
@@ -11336,8 +11338,12 @@ void __kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu)
 	down_read(&vcpu->kvm->arch.apicv_update_lock);
 	preempt_disable();
 
-	/* Do not activate APICV when APIC is disabled */
+	/*
+	 * Do not activate APICV when APIC is disabled, nor on higher-plane
+	 * twins (see kvm_create_lapic).
+	 */
 	activate = kvm_vcpu_apicv_activated(vcpu) &&
+		   !vcpu->plane_level &&
 		   (kvm_get_apic_mode(vcpu) != LAPIC_MODE_DISABLED);
 
 	if (apic->apicv_active == activate)
@@ -12566,6 +12572,15 @@ static int __kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 	post_kvm_run_save(vcpu);
 	kvm_vcpu_srcu_read_unlock(vcpu);
 
+	/*
+	 * A non-zero plane may stay suspended indefinitely after this run
+	 * ends, and its hv timer only counts in VMX non-root mode.  Hand
+	 * the lapic timer to the hrtimer so it keeps firing while the
+	 * plane is parked; firing is what forces re-entry into the plane.
+	 */
+	if (vcpu->plane_level && kvm_lapic_hv_timer_in_use(vcpu))
+		kvm_lapic_switch_to_sw_timer(vcpu);
+
 	kvm_sigset_deactivate(vcpu);
 	vcpu_put(vcpu);
 	return r;
-- 
2.50.1

[RFC PATCH 25/55] WIP: KVM: per-plane memory overlays, richer protection-fault exits

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The pieces the VSM contract tests need:

- KVM_SET_PLANE_MEMORY_OVERLAY maps one page of userspace memory over
  a GPA for a single plane. TLFS overlay pages (SynIC SIMP in
  particular) show each VTL a private page while the other VTLs keep
  seeing the guest RAM underneath; per-plane EPT roots make a
  per-plane pfn override in the fault path sufficient. Mappings for a
  plane with overlays stay 4KiB so a neighbouring fault's huge page
  cannot span an overlaid gfn, and kvm_tdp_mmu_recover_huge_pages()
  skips nonzero-plane roots so a later promotion cannot widen the
  plane's attribute or overlay clamps back out (planes keep no lpage
  tracking by design; their fault path rebuilds each mapping at the
  largest level the plane allows). The overlay update runs under the
  kvm_mmu_invalidate_begin()/end() protocol with the gfn registered,
  as the mmu notifiers do, so a concurrent lockless fault cannot
  install an SPTE for the old backing after the zap.

- SynIC message delivery reads and writes the overlay through the same
  lookup, and fails with -EFAULT when a higher plane's SIMP page has
  no overlay installed yet, rather than falling through to
  kvm_vcpu_write_guest_page() and corrupting underlay RAM that plane 0
  owns; the sender retries once the overlay appears. A non-zero
  plane's SynIC exits carry the plane instead of being suppressed, so
  the VMM can keep its plane-0 device state separate.

- A plane-protection fault reflected as KVM_EXIT_MEMORY_FAULT now
  carries the guest-linear address and the faulting instruction's
  length and bytes (via VM_EXIT_INSTRUCTION_LEN and a guest-virt
  read), which the TLFS GPA-intercept message requires. LLDT joins
  LGDT/LIDT in the descriptor-write divert, reporting the written
  selector.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 Documentation/virt/kvm/api.rst |  62 ++++++++
 arch/x86/kvm/hyperv.c          |  82 +++++++---
 arch/x86/kvm/mmu/mmu.c         |  29 +++-
 arch/x86/kvm/mmu/tdp_mmu.c     |  23 ++-
 arch/x86/kvm/vmx/vmx.c         |  51 ++++++-
 arch/x86/kvm/x86.c             |  14 ++
 include/linux/kvm_host.h       |  17 +++
 include/uapi/linux/kvm.h       |  17 +++
 virt/kvm/kvm_main.c            | 271 +++++++++++++++++++++++++++++++++
 9 files changed, 541 insertions(+), 25 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 3ef57b3b050b..215547802377 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6657,6 +6657,38 @@ stop policy.
 The run state is distinct from KVM_GET/SET_MP_STATE, which describes
 the architectural state of a vCPU within its plane.
 
+4.149 KVM_SET_PLANE_MEMORY_OVERLAY
+----------------------------------
+
+:Capability: KVM_CAP_PLANE_MEMORY_OVERLAYS
+:Architectures: x86
+:Type: vm ioctl
+:Parameters: struct kvm_plane_memory_overlay (in)
+:Returns: 0 on success, <0 on error
+
+::
+
+  struct kvm_plane_memory_overlay {
+	__u32 plane;
+	__u32 flags;
+	__u64 gfn;
+	__u64 uaddr;		/* page-aligned; 0 removes the overlay */
+  };
+
+Installs, replaces or removes a memory overlay for a single guest page
+frame on one plane.  While an overlay is installed, translations of
+'gfn' for vCPUs of 'plane' resolve to the (pinned) page at 'uaddr' in
+the calling process instead of to the memslot backing; other planes are
+unaffected.  Hyper-V VSM uses this for per-VTL overlay pages such as
+the SynIC message and event-flags pages.
+
+'flags' must be zero and 'uaddr' must be page-aligned.  At most
+KVM_CHECK_EXTENSION(KVM_CAP_PLANE_MEMORY_OVERLAYS) distinct gfns can
+carry an overlay per plane at any one time; further installs fail with
+-ENOSPC until an overlay is removed.  An overlay cannot be installed
+over a private or guest_memfd-only gfn (-EINVAL).  Passing 'uaddr' == 0
+removes the overlay for 'gfn', if any.
+
 5. The kvm_run structure
 ========================
 
@@ -7387,9 +7419,15 @@ spec refer, https://github.com/riscv/riscv-sbi-doc.
   #define KVM_MEMORY_EXIT_FLAG_WRITE    (1ULL << 1)
   #define KVM_MEMORY_EXIT_FLAG_EXEC     (1ULL << 2)
   #define KVM_MEMORY_EXIT_FLAG_PRIVATE	(1ULL << 3)
+  #define KVM_MEMORY_EXIT_FLAG_GVA_VALID	(1ULL << 4)
 			__u64 flags;
 			__u64 gpa;
 			__u64 size;
+			/* set on plane-protection faults when available */
+			__u64 gva;
+			__u32 insn_len;
+			__u8  insn_bytes[16];
+			__u32 pad;
 		} memory_fault;
 
 KVM_EXIT_MEMORY_FAULT indicates the vCPU has encountered a memory fault that
@@ -7402,6 +7440,14 @@ describes properties of the faulting access that are likely pertinent:
  - KVM_MEMORY_EXIT_FLAG_PRIVATE - When set, indicates the memory fault occurred
    on a private memory access.  When clear, indicates the fault occurred on a
    shared access.
+ - KVM_MEMORY_EXIT_FLAG_GVA_VALID - When set, 'gva' holds the faulting guest
+   virtual address.
+
+The 'gva', 'insn_len' and 'insn_bytes' fields exist only on kernels
+which advertise KVM_CAP_PLANE_MEMORY_OVERLAYS (see section 8.48).  On
+plane memory-protection faults, 'insn_len' and 'insn_bytes' carry a
+best-effort copy of the faulting instruction when KVM could fetch it;
+an 'insn_len' of 0 means no instruction bytes are available.
 
 Note!  KVM_EXIT_MEMORY_FAULT is unique among all KVM exit reasons in that it
 accompanies a return code of '-1', not '0'!  errno will always be set to EFAULT
@@ -9676,6 +9722,22 @@ take care to differentiate between these cases.
 The presence of this capability indicates that the nested KVM guest can
 start in ESA mode.
 
+8.48 KVM_CAP_PLANE_MEMORY_OVERLAYS
+----------------------------------
+
+:Architectures: x86
+
+A positive return from KVM_CHECK_EXTENSION(KVM_CAP_PLANE_MEMORY_OVERLAYS)
+announces two things:
+
+ - The KVM_SET_PLANE_MEMORY_OVERLAY vm ioctl (section 4.149).  The
+   returned value is the number of gfns which can carry an overlay
+   simultaneously on each plane.
+
+ - The extended KVM_EXIT_MEMORY_FAULT layout: 'memory_fault' carries
+   the additional 'gva', 'insn_len' and 'insn_bytes' fields, with
+   KVM_MEMORY_EXIT_FLAG_GVA_VALID qualifying 'gva' (see section 5).
+
 9. Known KVM API problems
 =========================
 
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 4551e011bf56..d4e44ae3f6e8 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -247,16 +247,8 @@ static void synic_exit(struct kvm_vcpu_hv_synic *synic, u32 msr)
 	struct kvm_vcpu *vcpu = hv_synic_to_vcpu(synic);
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
 
-	/*
-	 * Userspace's SynIC device models plane 0 only; a higher plane's
-	 * exit would be mistaken for a plane-0 page move and relocate the
-	 * overlay the VMM keeps for SIMP/SIEFP.  Higher-plane SynICs are
-	 * handled fully in-kernel.
-	 */
-	if (vcpu->plane_level)
-		return;
-
 	hv_vcpu->exit.type = KVM_EXIT_HYPERV_SYNIC;
+	hv_vcpu->exit.plane = vcpu->plane_level;
 	hv_vcpu->exit.u.synic.msr = msr;
 	hv_vcpu->exit.u.synic.control = synic->control;
 	hv_vcpu->exit.u.synic.evt_page = synic->evt_page;
@@ -358,6 +350,7 @@ static void syndbg_exit(struct kvm_vcpu *vcpu, u32 msr)
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
 
 	hv_vcpu->exit.type = KVM_EXIT_HYPERV_SYNDBG;
+	hv_vcpu->exit.plane = vcpu->plane_level;
 	hv_vcpu->exit.u.syndbg.msr = msr;
 	hv_vcpu->exit.u.syndbg.control = syndbg->control.control;
 	hv_vcpu->exit.u.syndbg.send_page = syndbg->control.send_page;
@@ -783,6 +776,51 @@ static int stimer_get_count(struct kvm_vcpu_hv_stimer *stimer, u64 *pcount)
 	return 0;
 }
 
+/*
+ * SIMP accessors: a plane overlay page, when installed by userspace,
+ * carries the live message page for this plane; the memslot holds only
+ * the underlay the other planes see.
+ */
+static int synic_msg_page_read(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,
+			       int offset, int len)
+{
+	unsigned long uaddr = kvm_plane_overlay_uaddr(vcpu->plane, gfn);
+
+	if (uaddr)
+		return copy_from_user(data, (void __user *)(uaddr + offset),
+				      len) ? -EFAULT : 0;
+	/*
+	 * Mirror the write side: without an overlay a higher plane's
+	 * message page is backed by underlay memory that plane 0 controls.
+	 * Deciding deliverability from plane-0-writable bytes would let a
+	 * lower VTL plant a non-HVMSG_NONE type and silently swallow a
+	 * no_retry (periodic stimer) message; fail so the delivery is
+	 * retried once the overlay appears.
+	 */
+	if (vcpu->plane_level)
+		return -EFAULT;
+	return kvm_vcpu_read_guest_page(vcpu, gfn, data, offset, len);
+}
+
+static int synic_msg_page_write(struct kvm_vcpu *vcpu, gfn_t gfn,
+				const void *data, int offset, int len)
+{
+	unsigned long uaddr = kvm_plane_overlay_uaddr(vcpu->plane, gfn);
+
+	if (uaddr)
+		return copy_to_user((void __user *)(uaddr + offset), data,
+				    len) ? -EFAULT : 0;
+	/*
+	 * Never write a higher plane's messages into guest RAM: without an
+	 * overlay installed that memory is the underlay every other plane
+	 * sees, and silently corrupting plane-0-owned pages is far worse
+	 * than retrying the message once the overlay appears.
+	 */
+	if (vcpu->plane_level)
+		return -EFAULT;
+	return kvm_vcpu_write_guest_page(vcpu, gfn, data, offset, len);
+}
+
 static int synic_deliver_msg(struct kvm_vcpu_hv_synic *synic, u32 sint,
 			     struct hv_message *src_msg, bool no_retry)
 {
@@ -803,10 +841,10 @@ static int synic_deliver_msg(struct kvm_vcpu_hv_synic *synic, u32 sint,
 	 * is only called in vcpu context so the entire update is atomic from
 	 * guest POV and thus the exact order here doesn't matter.
 	 */
-	r = kvm_vcpu_read_guest_page(vcpu, msg_page_gfn, &hv_hdr.message_type,
-				     msg_off + offsetof(struct hv_message,
-							header.message_type),
-				     sizeof(hv_hdr.message_type));
+	r = synic_msg_page_read(vcpu, msg_page_gfn, &hv_hdr.message_type,
+				msg_off + offsetof(struct hv_message,
+						   header.message_type),
+				sizeof(hv_hdr.message_type));
 	if (r < 0)
 		return r;
 
@@ -815,20 +853,20 @@ static int synic_deliver_msg(struct kvm_vcpu_hv_synic *synic, u32 sint,
 			return 0;
 
 		hv_hdr.message_flags.msg_pending = 1;
-		r = kvm_vcpu_write_guest_page(vcpu, msg_page_gfn,
-					      &hv_hdr.message_flags,
-					      msg_off +
-					      offsetof(struct hv_message,
-						       header.message_flags),
-					      sizeof(hv_hdr.message_flags));
+		r = synic_msg_page_write(vcpu, msg_page_gfn,
+					 &hv_hdr.message_flags,
+					 msg_off +
+					 offsetof(struct hv_message,
+						  header.message_flags),
+					 sizeof(hv_hdr.message_flags));
 		if (r < 0)
 			return r;
 		return -EAGAIN;
 	}
 
-	r = kvm_vcpu_write_guest_page(vcpu, msg_page_gfn, src_msg, msg_off,
-				      sizeof(src_msg->header) +
-				      src_msg->header.payload_size);
+	r = synic_msg_page_write(vcpu, msg_page_gfn, src_msg, msg_off,
+				 sizeof(src_msg->header) +
+				 src_msg->header.payload_size);
 	if (r < 0)
 		return r;
 
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index cdebb5c324fe..4b11631d9e99 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -4620,7 +4620,13 @@ static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu,
 	if (fault->slot->flags & KVM_MEM_READONLY)
 		fault->host_access &= ~ACC_WRITE_MASK;
 
-	fault->max_level = kvm_max_level_for_order(max_order);
+	/*
+	 * Only ever narrow max_level: the overlay check in
+	 * __kvm_mmu_faultin_pfn() may already have clamped it to 4KiB, and
+	 * a huge gmem mapping must not span an overlaid gfn.
+	 */
+	fault->max_level = min(fault->max_level,
+			       kvm_max_level_for_order(max_order));
 	return RET_PF_CONTINUE;
 }
 
@@ -4682,8 +4688,29 @@ static int __kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 				 struct kvm_page_fault *fault)
 {
 	unsigned int foll = fault->write ? FOLL_WRITE : 0;
+	struct page *overlay;
 	bool writable;
 
+	/*
+	 * A plane overlay page (VSM SynIC SIMP/SIEFP etc.) replaces the
+	 * memslot translation for this plane only.  The page is pinned
+	 * for the overlay's lifetime, so no refcounting here.
+	 */
+	/*
+	 * Keep this plane's mappings at 4KiB while it has overlays: a huge
+	 * mapping established by a neighbouring gfn's fault would span the
+	 * overlaid gfn and bypass the override below.
+	 */
+	if (READ_ONCE(vcpu->plane->nr_overlays)) {
+		fault->max_level = PG_LEVEL_4K;
+
+		overlay = kvm_plane_overlay_page(vcpu->plane, fault->gfn);
+		if (overlay) {
+			fault->pfn = page_to_pfn(overlay);
+			return RET_PF_CONTINUE;
+		}
+	}
+
 	if (fault->is_private || kvm_memslot_is_gmem_only(fault->slot))
 		return kvm_mmu_faultin_pfn_gmem(vcpu, fault);
 
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index 1e56a4784160..d5d0b2bc547c 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -1871,8 +1871,29 @@ void kvm_tdp_mmu_recover_huge_pages(struct kvm *kvm,
 	struct kvm_mmu_page *root;
 
 	lockdep_assert_held_read(&kvm->mmu_lock);
-	for_each_valid_tdp_mmu_root_yield_safe(kvm, root, slot->as_id)
+	for_each_valid_tdp_mmu_root_yield_safe(kvm, root, slot->as_id) {
+		/*
+		 * kvm_mmu_max_mapping_level() consults only plane-0 state
+		 * (lpage_info and plane-0 private attributes).  Nonzero
+		 * planes carry per-plane memory attributes and memory
+		 * overlays whose fault-time 4KiB clamps a huge SPTE would
+		 * undo, extending one page's protections or overlay
+		 * backing over gfns the plane maps differently.  Leave
+		 * those roots alone; their fault path rebuilds mappings at
+		 * the largest level the plane allows.
+		 *
+		 * Plane 0 can carry overlays too (kvm->planes[0] always
+		 * exists and KVM_SET_PLANE_MEMORY_OVERLAY accepts plane 0),
+		 * and the same reasoning applies: a recovered huge SPTE
+		 * would either span an overlaid gfn with the memslot
+		 * backing, or huge-align the pinned overlay page's pfn and
+		 * map ~2MiB of unrelated host memory around it.
+		 */
+		if (root->role.plane ||
+		    READ_ONCE(kvm->planes[root->role.plane]->nr_overlays))
+			continue;
 		recover_huge_pages_range(kvm, root, slot);
+	}
 }
 
 /*
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index e1c5f1608b09..248d799b46e9 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -70,6 +70,7 @@
 #include "vmcs12.h"
 #include "vmx.h"
 #include "x86.h"
+#include "kvm_emulate.h"
 #include "x86_ops.h"
 #include "smm.h"
 #include "vmx_onhyperv.h"
@@ -5949,6 +5950,7 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu)
 {
 	unsigned long exit_qualification = vmx_get_exit_qual(vcpu);
 	gpa_t gpa;
+	int ret;
 
 	/*
 	 * EPT violation happened while executing iret from NMI,
@@ -5975,7 +5977,54 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu)
 	if (unlikely(allow_smaller_maxphyaddr && !kvm_vcpu_is_legal_gpa(vcpu, gpa)))
 		return kvm_emulate_instruction(vcpu, 0);
 
-	return __vmx_handle_ept_violation(vcpu, gpa, exit_qualification);
+	ret = __vmx_handle_ept_violation(vcpu, gpa, exit_qualification);
+
+	/*
+	 * A plane-protection fault reflected to userspace becomes a TLFS
+	 * GPA intercept message, which carries the guest-linear address
+	 * and the faulting instruction's bytes.  Harvest them while the
+	 * exit state is still at hand.
+	 */
+	if (ret == -EFAULT &&
+	    vcpu->run->exit_reason == KVM_EXIT_MEMORY_FAULT &&
+	    (vcpu->run->memory_fault.flags & (KVM_MEMORY_EXIT_FLAG_READ |
+					      KVM_MEMORY_EXIT_FLAG_WRITE |
+					      KVM_MEMORY_EXIT_FLAG_EXEC))) {
+		unsigned int len;
+
+		if (exit_qualification & EPT_VIOLATION_GVA_IS_VALID) {
+			vcpu->run->memory_fault.gva =
+				vmcs_readl(GUEST_LINEAR_ADDRESS);
+			vcpu->run->memory_fault.flags |=
+				KVM_MEMORY_EXIT_FLAG_GVA_VALID;
+		}
+		/*
+		 * Architecturally undefined for EPT violations, but the
+		 * hardware provides the faulting (for a fetch: the
+		 * fetching) instruction's length here, and the TLFS
+		 * message carries exactly that.
+		 */
+		len = vmcs_read32(VM_EXIT_INSTRUCTION_LEN);
+		len = min_t(unsigned int, len,
+			    sizeof(vcpu->run->memory_fault.insn_bytes));
+		vcpu->run->memory_fault.insn_len = len;
+		if (!(exit_qualification & EPT_VIOLATION_ACC_INSTR)) {
+			struct x86_exception e;
+
+			/*
+			 * kvm_read_guest_virt() takes a linear address: RIP
+			 * is CS-relative, and the two coincide only when
+			 * CS.base is zero (64-bit mode, or a flat 32-bit
+			 * code segment).
+			 */
+			if (kvm_read_guest_virt(vcpu, kvm_get_linear_rip(vcpu),
+						vcpu->run->memory_fault.insn_bytes,
+						len, &e))
+				memset(vcpu->run->memory_fault.insn_bytes, 0,
+				       sizeof(vcpu->run->memory_fault.insn_bytes));
+		}
+	}
+	return ret;
 }
 
 static int handle_ept_misconfig(struct kvm_vcpu *vcpu)
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 4b12ec7ee100..10e14ab3e2ca 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -8858,8 +8858,10 @@ static bool hv_divert_dt_write(struct x86_emulate_ctxt *ctxt, u32 hv_reg,
 
 #define HV_CR_INTERCEPT_GDTR_WRITE	BIT_ULL(15)
 #define HV_CR_INTERCEPT_IDTR_WRITE	BIT_ULL(16)
+#define HV_CR_INTERCEPT_LDTR_WRITE	BIT_ULL(17)
 #define HV_X64_REGISTER_IDTR_ID		0x00070000
 #define HV_X64_REGISTER_GDTR_ID		0x00070001
+#define HV_X64_REGISTER_LDTR_ID		0x00060006
 
 static void emulator_set_gdt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt)
 {
@@ -8947,6 +8949,18 @@ static void emulator_set_segment(struct x86_emulate_ctxt *ctxt, u16 selector,
 	var.unusable = !var.present;
 	var.padding = 0;
 
+	if (seg == VCPU_SREG_LDTR) {
+		struct desc_ptr sel_as_dt = {
+			.address = selector,
+			.size = 0,
+		};
+
+		/* The intercept message carries the written selector. */
+		if (hv_divert_dt_write(ctxt, HV_X64_REGISTER_LDTR_ID,
+				       &sel_as_dt, HV_CR_INTERCEPT_LDTR_WRITE))
+			return;
+	}
+
 	kvm_set_segment(vcpu, &var, seg);
 	return;
 }
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index a76615f2c3dc..cb8c4798df12 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -927,9 +927,26 @@ struct kvm_plane {
 	struct xarray mem_attrs;
 #endif
 
+	/*
+	 * VSM overlay pages: per-plane gfn -> pinned user page overriding
+	 * the memslot translation for this plane only.  Written under
+	 * kvm->slots_lock; readers tolerate a transient stale view, the
+	 * post-update zap makes the EPT coherent.
+	 */
+#define KVM_PLANE_MAX_OVERLAYS 64
+	struct kvm_plane_overlay {
+		gfn_t gfn;
+		unsigned long uaddr;
+		struct page *page;
+	} overlays[KVM_PLANE_MAX_OVERLAYS];
+	int nr_overlays;
+
 	struct kvm_arch_plane arch;
 };
 
+struct page *kvm_plane_overlay_page(struct kvm_plane *plane, gfn_t gfn);
+unsigned long kvm_plane_overlay_uaddr(struct kvm_plane *plane, gfn_t gfn);
+
 struct kvm {
 #ifdef KVM_HAVE_MMU_RWLOCK
 	rwlock_t mmu_lock;
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 9e453f215e68..843cef8f4cbf 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -480,9 +480,15 @@ struct kvm_run {
 #define KVM_MEMORY_EXIT_FLAG_WRITE      (1ULL << 1)
 #define KVM_MEMORY_EXIT_FLAG_EXEC       (1ULL << 2)
 #define KVM_MEMORY_EXIT_FLAG_PRIVATE	(1ULL << 3)
+#define KVM_MEMORY_EXIT_FLAG_GVA_VALID	(1ULL << 4)
 			__u64 flags;
 			__u64 gpa;
 			__u64 size;
+			/* set on plane-protection faults when available */
+			__u64 gva;
+			__u32 insn_len;
+			__u8  insn_bytes[16];
+			__u32 pad;
 		} memory_fault;
 		/* KVM_EXIT_TDX */
 		struct {
@@ -1038,6 +1044,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_HCALL_FAULT_EXIT 252
 #define KVM_CAP_HYPERV_VTL_SWITCH 253
 #define KVM_CAP_HYPERV_REG_INTERCEPT 254
+#define KVM_CAP_PLANE_MEMORY_OVERLAYS 257
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
@@ -1394,6 +1401,16 @@ struct kvm_s390_keyop {
 #define KVM_CREATE_PLANE	  _IO(KVMIO, 0xe4)
 #define KVM_GET_PLANE_RUN_STATE	  _IOR(KVMIO, 0xe5, __u32)
 #define KVM_SET_PLANE_RUN_STATE	  _IOW(KVMIO, 0xe6, __u32)
+
+/* Map one page of userspace memory over a GPA for a single plane (VSM
+ * overlay pages: SynIC SIMP/SIEFP, hypercall page, VP assist page). */
+struct kvm_plane_memory_overlay {
+	__u32 plane;
+	__u32 flags;
+	__u64 gfn;
+	__u64 uaddr;		/* page-aligned; 0 removes the overlay */
+};
+#define KVM_SET_PLANE_MEMORY_OVERLAY	  _IOW(KVMIO, 0xe7, struct kvm_plane_memory_overlay)
 #define   KVM_PLANE_RUN_STATE_STOPPED	0
 #define   KVM_PLANE_RUN_STATE_RUNNABLE	1
 #define   KVM_PLANE_RUN_STATE_KERNEL_STOPPED	2
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 4197496f7176..5856672bbf6d 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -1251,6 +1251,14 @@ static struct kvm_plane *kvm_create_plane(struct kvm *kvm, unsigned plane_level)
 
 static void kvm_destroy_one_plane(struct kvm_plane *plane)
 {
+	int i;
+
+	for (i = 0; i < plane->nr_overlays; i++)
+		if (plane->overlays[i].page)
+			unpin_user_pages_dirty_lock(&plane->overlays[i].page,
+						    1, true);
+	plane->nr_overlays = 0;
+
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
 	xa_destroy(&plane->mem_attrs);
 #endif
@@ -5629,6 +5637,8 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
 		 * VM actually has.
 		 */
 		return KVM_MAX_PLANES;
+	case KVM_CAP_PLANE_MEMORY_OVERLAYS:
+		return KVM_PLANE_MAX_OVERLAYS;
 	default:
 		break;
 	}
@@ -5896,6 +5906,258 @@ do {										\
 		     sizeof_field(struct kvm_userspace_memory_region2, field));	\
 } while (0)
 
+struct page *kvm_plane_overlay_page(struct kvm_plane *plane, gfn_t gfn)
+{
+	int i, n = READ_ONCE(plane->nr_overlays);
+
+	/* pairs with the smp_wmb() publishing a new entry */
+	smp_rmb();
+	for (i = 0; i < n; i++) {
+		if (READ_ONCE(plane->overlays[i].gfn) != gfn)
+			continue;
+		/* pairs with the smp_store_release() updating an entry */
+		return smp_load_acquire(&plane->overlays[i].page);
+	}
+	return NULL;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_plane_overlay_page);
+
+unsigned long kvm_plane_overlay_uaddr(struct kvm_plane *plane, gfn_t gfn)
+{
+	int i, n = READ_ONCE(plane->nr_overlays);
+
+	smp_rmb();
+	for (i = 0; i < n; i++) {
+		if (READ_ONCE(plane->overlays[i].gfn) != gfn)
+			continue;
+		/*
+		 * The acquire pairs with the smp_store_release() updating
+		 * an entry: a reader which observes the new page also
+		 * observes the uaddr written before it.
+		 */
+		if (smp_load_acquire(&plane->overlays[i].page))
+			return READ_ONCE(plane->overlays[i].uaddr);
+		return 0;
+	}
+	return 0;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_plane_overlay_uaddr);
+
+/*
+ * The invalidation window registered around an overlay update must
+ * cover every gfn whose in-flight fault could install a huge leaf
+ * spanning the overlaid gfn, i.e. the naturally-aligned region of the
+ * widest stage-2 huge mapping any plane-capable architecture supports
+ * (a 1GiB page on x86).  A narrower window lets a racing fault on a
+ * neighbouring gfn, which sampled nr_overlays and mmu_invalidate_seq
+ * before the update, pass mmu_invalidate_retry_gfn() and install a
+ * huge SPTE over the overlay after the zap.
+ */
+#define KVM_PLANE_OVERLAY_INVAL_PAGES	(SZ_1G >> PAGE_SHIFT)
+
+static void kvm_plane_overlay_invalidate_add(struct kvm *kvm, gfn_t gfn)
+{
+	gfn_t start = ALIGN_DOWN(gfn, KVM_PLANE_OVERLAY_INVAL_PAGES);
+
+	kvm_mmu_invalidate_range_add(kvm, start,
+				     start + KVM_PLANE_OVERLAY_INVAL_PAGES);
+}
+
+/*
+ * Drop every translation of @gfn, in all address spaces and all planes'
+ * roots.  A huge leaf spanning the gfn intersects the single-page range
+ * and is zapped with it.  Hardware TLBs are not flushed here.
+ */
+static void kvm_plane_overlay_unmap(struct kvm *kvm, gfn_t gfn)
+{
+#ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER
+	struct kvm_memslot_iter iter;
+	struct kvm_memslots *slots;
+	int i;
+
+	KVM_MMU_LOCK(kvm);
+	for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) {
+		slots = __kvm_memslots(kvm, i);
+
+		kvm_for_each_memslot_in_gfn_range(&iter, slots, gfn, gfn + 1) {
+			struct kvm_gfn_range range = {
+				.slot = iter.slot,
+				.start = gfn,
+				.end = gfn + 1,
+				.attr_filter = KVM_FILTER_PRIVATE |
+					       KVM_FILTER_SHARED,
+				.may_block = true,
+			};
+
+			kvm_unmap_gfn_range(kvm, &range);
+		}
+	}
+	KVM_MMU_UNLOCK(kvm);
+#else
+	kvm_arch_flush_shadow_all(kvm);
+#endif
+}
+
+static int kvm_vm_ioctl_set_plane_overlay(struct kvm *kvm,
+					  struct kvm_plane_memory_overlay *ov)
+{
+	struct kvm_plane_overlay *e = NULL;
+	struct page *page = NULL, *old_page = NULL;
+	struct kvm_plane *plane;
+	bool moved = false;
+	gfn_t moved_gfn = 0;
+	int i, r;
+
+	if (ov->flags || ov->plane >= KVM_MAX_PLANES)
+		return -EINVAL;
+	if (ov->uaddr & ~PAGE_MASK)
+		return -EINVAL;
+	plane = kvm->planes[ov->plane];
+	if (!plane)
+		return -EINVAL;
+
+	if (ov->uaddr) {
+		r = pin_user_pages_fast(ov->uaddr, 1,
+					FOLL_WRITE | FOLL_LONGTERM, &page);
+		if (r != 1)
+			return r < 0 ? r : -EFAULT;
+	}
+
+	mutex_lock(&kvm->slots_lock);
+	for (i = 0; i < plane->nr_overlays; i++) {
+		if (plane->overlays[i].gfn == ov->gfn) {
+			e = &plane->overlays[i];
+			break;
+		}
+	}
+
+	/*
+	 * Removing an overlay which was never installed changes nothing;
+	 * don't zap and flush for it.
+	 */
+	if (!ov->uaddr && !e) {
+		mutex_unlock(&kvm->slots_lock);
+		return 0;
+	}
+
+	/*
+	 * guest_memfd-backed and private gfns have their own backing
+	 * contract which an ordinary pinned user page cannot substitute
+	 * for, and kvm_mmu_faultin_pfn_gmem() does not consult overlays.
+	 * Both the private attribute and the memslot flags are stable
+	 * under slots_lock.
+	 */
+	if (ov->uaddr) {
+		struct kvm_memory_slot *slot = gfn_to_memslot(kvm, ov->gfn);
+
+		if ((slot && kvm_memslot_is_gmem_only(slot)) ||
+		    kvm_mem_is_private(kvm, ov->gfn)) {
+			mutex_unlock(&kvm->slots_lock);
+			unpin_user_page(page);
+			return -EINVAL;
+		}
+	}
+
+	if (ov->uaddr) {
+		if (!e) {
+			if (plane->nr_overlays >= KVM_PLANE_MAX_OVERLAYS) {
+				mutex_unlock(&kvm->slots_lock);
+				unpin_user_page(page);
+				return -ENOSPC;
+			}
+			e = &plane->overlays[plane->nr_overlays];
+			e->gfn = ov->gfn;
+			e->uaddr = ov->uaddr;
+			e->page = page;
+			/*
+			 * Publish the fully-initialised entry before
+			 * raising the count; pairs with the smp_rmb() in
+			 * the lockless readers.
+			 */
+			smp_wmb();
+			WRITE_ONCE(plane->nr_overlays, plane->nr_overlays + 1);
+		} else {
+			old_page = e->page;
+			WRITE_ONCE(e->uaddr, ov->uaddr);
+			/*
+			 * Order the uaddr store before the page store; a
+			 * reader which acquires the new page must see the
+			 * uaddr it belongs to.
+			 */
+			smp_store_release(&e->page, page);
+		}
+	} else if (e) {
+		struct kvm_plane_overlay *tail =
+			&plane->overlays[plane->nr_overlays - 1];
+
+		old_page = e->page;
+		WRITE_ONCE(e->page, NULL);
+		WRITE_ONCE(e->uaddr, 0);
+
+		/*
+		 * Reclaim the slot: move the tail entry into the vacated
+		 * one and shrink the array.  A lockless reader scanning
+		 * with the old count can miss the moved entry entirely
+		 * (it may already have passed the vacated slot), so the
+		 * moved gfn is added to the invalidation window below and
+		 * any such in-flight fault retries.  The cleared tail is
+		 * harmless to a stale-count reader: the live copy sits at
+		 * a lower index and is found first.
+		 */
+		if (e != tail) {
+			moved = true;
+			moved_gfn = tail->gfn;
+			WRITE_ONCE(e->gfn, tail->gfn);
+			WRITE_ONCE(e->uaddr, tail->uaddr);
+			/* pairs with smp_load_acquire() in the readers */
+			smp_store_release(&e->page, tail->page);
+		}
+		WRITE_ONCE(plane->nr_overlays, plane->nr_overlays - 1);
+		WRITE_ONCE(tail->page, NULL);
+		WRITE_ONCE(tail->uaddr, 0);
+		WRITE_ONCE(tail->gfn, 0);
+	}
+	/*
+	 * Bump the invalidation sequence across the update and zap: a
+	 * concurrent fault resolves the overlay pfn locklessly before
+	 * taking mmu_lock, and without a sequence change
+	 * is_page_fault_stale() would let it install an SPTE for the OLD
+	 * backing after the zap, mapping a page this function is about
+	 * to unpin.
+	 */
+	KVM_MMU_LOCK(kvm);
+	kvm_mmu_invalidate_begin(kvm);
+	kvm_plane_overlay_invalidate_add(kvm, ov->gfn);
+	if (moved)
+		kvm_plane_overlay_invalidate_add(kvm, moved_gfn);
+	KVM_MMU_UNLOCK(kvm);
+
+	mutex_unlock(&kvm->slots_lock);
+
+	/*
+	 * Drop any translation mapping the old or new backing.  The unmap
+	 * removes the SPTEs but does not flush hardware TLBs: a vCPU that
+	 * mapped the gfn before the update still holds a translation to
+	 * the old backing in its TLB (tagged by the live ASID on SVM,
+	 * which nothing else invalidates).  Flush before the old page is
+	 * unpinned: KVM_REQ_TLB_FLUSH is a per-plane request, so the
+	 * fan-out in kvm_make_vcpu_request() reaches every plane twin and
+	 * each flushes its own hardware context.
+	 */
+	kvm_plane_overlay_unmap(kvm, ov->gfn);
+	if (moved)
+		kvm_plane_overlay_unmap(kvm, moved_gfn);
+	kvm_flush_remote_tlbs(kvm);
+
+	KVM_MMU_LOCK(kvm);
+	kvm_mmu_invalidate_end(kvm);
+	KVM_MMU_UNLOCK(kvm);
+
+	if (old_page)
+		unpin_user_pages_dirty_lock(&old_page, 1, true);
+	return 0;
+}
+
 static long kvm_vm_ioctl(struct file *filp,
 			   unsigned int ioctl, unsigned long arg)
 {
@@ -5906,6 +6168,15 @@ static long kvm_vm_ioctl(struct file *filp,
 	if (kvm->mm != current->mm || kvm->vm_dead)
 		return -EIO;
 	switch (ioctl) {
+	case KVM_SET_PLANE_MEMORY_OVERLAY: {
+		struct kvm_plane_memory_overlay ov;
+
+		r = -EFAULT;
+		if (copy_from_user(&ov, argp, sizeof(ov)))
+			goto out;
+		r = kvm_vm_ioctl_set_plane_overlay(kvm, &ov);
+		break;
+	}
 	case KVM_CREATE_PLANE:
 		r = kvm_vm_ioctl_create_plane(kvm, arg);
 		break;
-- 
2.50.1

[RFC PATCH 26/55] WIP: KVM: x86: VSM event-delivery parity fixes from Windows boot triage

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

All derived from the reference host's behaviour, and verified against
its VSM unit-test suite (still green) and Windows Server 2025 VBS
boot:

- Forced entry into a higher plane is edge-triggered, as on the
  reference host, which attempts one forced entry per posted
  interrupt. Delivery of a fixed vector or NMI to a suspended higher
  plane sets hv_apic_event_pending; the evaluation triggers on that
  marker and the consumable requests (stimer, NMI, APIC timer), not
  on a scan of the plane's IRR — a vector still latched after a
  refused entry waits for the target's next cooperative entry or next
  posted event instead of retrying on every run of the lower plane.
  The marker is cleared only by a run that actually reaches event
  evaluation; a run aborted by the VTL-exit or KVM_REQ_PLANE_RESCHED
  bail-outs leaves it set, since delivery never re-arms it. A vector
  masked by the higher VTL's TPR still forces exactly one entry.
- One exception to single-shot: a plane that suspends (VtlReturn or
  the run-state ioctl) with a vector still latched in its IRR re-arms
  its marker, via kvm_arch_vcpu_plane_stopped() from
  kvm_vcpu_set_plane_stopped(). Evaluating does not imply delivering:
  Windows hibernation posts vector 0xE0 to a twin mid-run with IF
  clear, the run consumes the marker without injecting, and the
  suspended twin's vector is stranded — VTL0 freezes all processors,
  spins on the secure kernel and wedges the VM at 100% host CPU. A
  re-armed marker cannot storm: the deliverability gate still applies,
  and a plane that reached suspension has run before, so userspace can
  enter it again.
- NMIs aimed at a higher plane fold back to plane 0 for delivery
  after the VTL returns; the higher plane only sees the forced entry.
  A real NMI inside VTL1 bugchecks the secure kernel (0x18B).
- VTL1 exempted from the reserved-bits check on Hyper-V vAPIC MSR
  writes: the secure kernel writes HV_X64_MSR_EOI with stale upper
  bits, and the reference host truncates for VTL1.
- KVM_REQ_HV_TLB_FLUSH made per-plane: a shared request could be
  consumed by the running twin, leaving the targeted plane's TLB
  stale.
- Lower-twin kicks on the stimer retry (vcpu_kick=false) paths.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/include/asm/kvm_host.h | 12 ++++-
 arch/x86/kvm/hyperv.c           |  3 +-
 arch/x86/kvm/lapic.c            | 39 ++++++++++++++-
 arch/x86/kvm/x86.c              | 89 +++++++++++++++++++++++++++++----
 include/linux/kvm_host.h        |  1 +
 virt/kvm/kvm_main.c             |  5 ++
 6 files changed, 135 insertions(+), 14 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index c54f47c52a59..834597791fe1 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -127,7 +127,8 @@
 #define KVM_REQ_MMU_FREE_OBSOLETE_ROOTS \
 	KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
 #define KVM_REQ_HV_TLB_FLUSH \
-	KVM_ARCH_REQ_FLAGS(32, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
+	KVM_ARCH_REQ_FLAGS(32, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP | \
+			   KVM_REQUEST_PER_PLANE)
 #define KVM_REQ_UPDATE_PROTECTED_GUEST_STATE \
 	KVM_ARCH_REQ_FLAGS(34, KVM_REQUEST_WAIT | KVM_REQUEST_PER_PLANE)
 
@@ -854,6 +855,15 @@ int kvm_arch_vcpu_common_init(struct kvm_vcpu_common *common);
 void kvm_arch_vcpu_common_destroy(struct kvm_vcpu_common *common);
 
 struct kvm_vcpu_arch {
+	/*
+	 * Hyper-V VSM: an APIC event (fixed vector or NMI) was delivered to
+	 * this plane while it was suspended.  Set at delivery, cleared when
+	 * the plane next runs; a lower plane's run loop uses it to force a
+	 * VTL entry even when the vector is currently masked by this
+	 * plane's TPR (matching the reference host's apic_pending_vtls).
+	 */
+	bool hv_apic_event_pending;
+
 	/*
 	 * rip and regs accesses must go through
 	 * kvm_{register,rip}_{read,write} functions.
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index d4e44ae3f6e8..0d239f054856 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -605,8 +605,9 @@ static void stimer_mark_pending(struct kvm_vcpu_hv_stimer *stimer,
 	 * A pending stimer on a suspended higher plane is only processed
 	 * once that plane runs; kick the lower twins so the running one
 	 * re-evaluates pending higher-plane events and forces entry.
+	 * This applies to retry paths too (vcpu_kick == false).
 	 */
-	if (vcpu_kick && vcpu->plane_level) {
+	if (vcpu->plane_level) {
 		struct kvm_vcpu *twin;
 		int p;
 
diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index f89156c26972..32cace3c89f4 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -1448,6 +1448,7 @@ static int __apic_accept_irq(struct kvm_lapic *apic, int delivery_mode,
 			struct kvm_vcpu *twin;
 			int p;
 
+			WRITE_ONCE(vcpu->arch.hv_apic_event_pending, true);
 			for (p = 0; p < vcpu->plane_level; p++) {
 				twin = vcpu->common->vcpus[p];
 				if (twin) {
@@ -1474,8 +1475,31 @@ static int __apic_accept_irq(struct kvm_lapic *apic, int delivery_mode,
 
 	case APIC_DM_NMI:
 		result = 1;
-		kvm_inject_nmi(vcpu);
-		kvm_vcpu_kick(vcpu);
+		/*
+		 * An NMI aimed at a higher plane forces entry into that
+		 * plane (reason INTERRUPT, below) but the NMI itself is
+		 * folded back to plane 0: the reference host never delivers
+		 * an NMI inside a higher VTL, it re-queues it to VTL0's
+		 * context for delivery after the VTL returns.
+		 */
+		if (vcpu->plane_level) {
+			struct kvm_vcpu *twin;
+			int p;
+
+			WRITE_ONCE(vcpu->arch.hv_apic_event_pending, true);
+			if (vcpu->common->vcpus[0])
+				kvm_inject_nmi(vcpu->common->vcpus[0]);
+			for (p = 0; p < vcpu->plane_level; p++) {
+				twin = vcpu->common->vcpus[p];
+				if (twin) {
+					kvm_make_request(KVM_REQ_UNBLOCK, twin);
+					kvm_vcpu_kick(twin);
+				}
+			}
+		} else {
+			kvm_inject_nmi(vcpu);
+			kvm_vcpu_kick(vcpu);
+		}
 		break;
 
 	case APIC_DM_INIT:
@@ -3535,6 +3559,17 @@ int kvm_hv_vapic_msr_write(struct kvm_vcpu *vcpu, u32 reg, u64 data)
 	if (!lapic_in_kernel(vcpu))
 		return 1;
 
+	/*
+	 * Bits 63:32 are reserved in non-ICR registers, but the Windows
+	 * secure kernel leaves stale data in the high bits when writing
+	 * the Hyper-V vAPIC MSRs from a higher VTL.  The reference host
+	 * exempts those writes and truncates; a #GP here is a VTL1
+	 * bugcheck.  The exemption covers only this PV path: a native
+	 * x2APIC WRMSR takes the architectural #GP whatever the plane.
+	 */
+	if (vcpu->plane_level && reg != APIC_ICR)
+		data = (u32)data;
+
 	return kvm_lapic_msr_write(vcpu->arch.apic, reg, data);
 }
 
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 10e14ab3e2ca..3e2465b2fa2b 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -11229,18 +11229,26 @@ static struct kvm_vcpu *kvm_higher_plane_with_event(struct kvm_vcpu *vcpu)
 			continue;
 #endif
 		/*
-		 * Test raw pending-ness, not injectability:
-		 * kvm_cpu_has_injectable_intr() reports false on an
-		 * APICv-active vCPU, and a suspended plane may carry a
-		 * stale apicv_active until it next runs.  A pending
-		 * synthetic-timer request counts as well: it can only be
-		 * processed (and its interrupt delivered) once the higher
-		 * plane runs, so it would otherwise never fire while the
-		 * plane is suspended.
+		 * Forced entry is edge-triggered on the posting of an
+		 * event, not level-triggered on the target's IRR, matching
+		 * the reference host's apic_pending_vtls bitmap: one entry
+		 * attempt per posted interrupt, via the hv_apic_event_pending
+		 * marker __apic_accept_irq sets.  A vector that stays
+		 * latched after a refused attempt (no VP assist page yet)
+		 * or that was masked by the plane's TPR waits for the next
+		 * cooperative entry or the next posted event; scanning the
+		 * IRR here instead would replay it the moment the assist
+		 * page is re-enabled.  A pending synthetic-timer request
+		 * counts as well: it can only be processed (and its
+		 * interrupt delivered) once the higher plane runs, so it
+		 * would otherwise never fire while the plane is suspended.
 		 */
-		if (kvm_cpu_has_extint(higher) ||
-		    kvm_apic_has_interrupt(higher) != -1 ||
+		if (READ_ONCE(higher->arch.hv_apic_event_pending) ||
+		    kvm_cpu_has_extint(higher) ||
 		    kvm_test_request(KVM_REQ_HV_STIMER, higher) ||
+		    kvm_test_request(KVM_REQ_NMI, higher) ||
+		    atomic_read(&higher->arch.nmi_queued) ||
+		    higher->arch.nmi_pending ||
 		    (higher->arch.apic &&
 		     atomic_read(&higher->arch.apic->lapic_timer.pending)))
 			return higher;
@@ -11265,6 +11273,14 @@ static bool kvm_hv_vtl_interrupt_exit(struct kvm_vcpu *vcpu)
 	if (!higher)
 		return false;
 
+	/*
+	 * Single-shot, as in the reference host: if userspace refuses
+	 * the entry (e.g. the target VTL has no VP assist page yet),
+	 * the event waits in the target's IRR for a cooperative entry
+	 * rather than re-triggering an exit storm.
+	 */
+	WRITE_ONCE(higher->arch.hv_apic_event_pending, false);
+
 	vcpu->run->exit_reason = KVM_EXIT_HYPERV;
 	vcpu->run->hyperv.type = KVM_EXIT_HYPERV_VTL_INTERRUPT;
 	vcpu->run->hyperv.plane = higher->plane_level;
@@ -11286,6 +11302,50 @@ static inline bool kvm_check_plane0_events(struct kvm_vcpu *vcpu)
 		vcpu_plane0->arch.smi_pending;
 }
 
+/*
+ * A higher plane that suspends with an interrupt still latched in its IRR
+ * must keep its hv_apic_event_pending marker armed.  The running plane
+ * consumes the marker in vcpu_enter_guest() before evaluating events, but
+ * evaluating does not imply delivering: a plane that keeps IF clear until
+ * it suspends (e.g. the secure kernel finishing a VTL round trip), or
+ * whose PPR masks the vector for the whole run, parks the vector in its
+ * IRR with the marker already consumed.  Nothing re-posts the interrupt,
+ * so the deliverability-gated forced entry in kvm_higher_plane_with_event()
+ * would never fire again and the plane would never see the vector; a lower
+ * VTL spinning on the suspended VTL's response then wedges the VM.
+ *
+ * Re-arming cannot re-create the refused-entry exit storm the single-shot
+ * marker guards against: the forced-entry gate still requires the vector
+ * to be deliverable by the plane's own priority state.
+ */
+void kvm_arch_vcpu_plane_stopped(struct kvm_vcpu *vcpu)
+{
+	/*
+	 * The hv_apic_event_pending marker drives the Hyper-V VSM forced
+	 * entry (see kvm_higher_plane_with_event()).  This hook also runs
+	 * from __sev_snp_run_vmpl() for guest-architectural VMPL switches:
+	 * without the Hyper-V gate, every SNP VMPL switch away from a
+	 * plane with a latched IRR would feed VSM-flavoured wakeups and
+	 * exits to a VM that has no VSM at all.
+	 */
+	if (!vcpu->arch.hyperv_enabled)
+		return;
+
+	/*
+	 * A plane that reached suspension normally has run before; the
+	 * exception is userspace parking a never-run plane it had already
+	 * injected an interrupt into.  Forced entry into a never-run
+	 * plane is always refused, so don't arm an attempt that cannot
+	 * succeed; the IRR is evaluated on the plane's first real entry.
+	 */
+	if (vcpu->arch.last_vmentry_cpu == -1)
+		return;
+
+	if (vcpu->plane_level && lapic_in_kernel(vcpu) &&
+	    kvm_lapic_find_highest_irr(vcpu) != -1)
+		WRITE_ONCE(vcpu->arch.hv_apic_event_pending, true);
+}
+
 static void process_nmi(struct kvm_vcpu *vcpu)
 {
 	unsigned int limit;
@@ -11771,6 +11831,15 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 		goto out;
 	}
 
+	/*
+	 * This plane is running and about to evaluate its events: its
+	 * pended-while-suspended marker is consumed.  The clear must come
+	 * after the VTL-exit and plane-reschedule bail-outs above: a run
+	 * aborted there never evaluates the pended event, and nothing
+	 * re-arms the marker for a TPR-masked vector.
+	 */
+	WRITE_ONCE(vcpu->arch.hv_apic_event_pending, false);
+
 	if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win ||
 	    kvm_xen_has_interrupt(vcpu)) {
 		++vcpu->stat.req_event;
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index cb8c4798df12..30dbc28885fb 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -493,6 +493,7 @@ static inline void kvm_vcpu_cancel_vtl_fpu_carry(struct kvm_vcpu_common *common)
 
 void kvm_vcpu_set_plane_runnable(struct kvm_vcpu *vcpu);
 void kvm_vcpu_set_plane_stopped(struct kvm_vcpu *vcpu);
+void kvm_arch_vcpu_plane_stopped(struct kvm_vcpu *vcpu);
 struct kvm_vcpu *kvm_vcpu_select_plane(struct kvm_vcpu *vcpu);
 
 static inline bool kvm_vcpu_wants_to_run(struct kvm_vcpu *vcpu)
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 5856672bbf6d..fac30b8b6518 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -5347,9 +5347,14 @@ void kvm_vcpu_set_plane_runnable(struct kvm_vcpu *vcpu)
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_set_plane_runnable);
 
+void __weak kvm_arch_vcpu_plane_stopped(struct kvm_vcpu *vcpu)
+{
+}
+
 void kvm_vcpu_set_plane_stopped(struct kvm_vcpu *vcpu)
 {
 	vcpu->plane_state = STOPPED;
+	kvm_arch_vcpu_plane_stopped(vcpu);
 	kvm_make_request(KVM_REQ_PLANE_RESCHED, vcpu);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_set_plane_stopped);
-- 
2.50.1

[RFC PATCH 27/55] KVM: planes: fan out per-plane requests to twin vCPUs

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

A request raised on one plane's vCPU for a VM-wide reason — TLB flush
after a host-side zap, APIC page reload, obsolete-roots release — is
consumed by whichever twin runs first; the planes series does not yet
fan requests out to twins, so the other planes keep stale
translations. The plane-1 twin of a Windows VSM guest then runs on a
zapped root and the guest bugchecks.

Mark those three requests per-plane and make kvm_make_vcpu_request()
replicate any KVM_REQUEST_PER_PLANE request to every twin of the
target vCPU. Mode, pCPU and wait live in the shared common, so the
existing kick logic already reaches the twins.

Move the Hyper-V TLB-flush hypercalls onto the same fan-out. The
reference hypervisor's TLB is VTL-blind: both VTLs share one hardware
translation context, a flush from either VTL covers both, and the
hypercall does not return until the targets have left guest mode. The
secure kernel relies on all of that when it edits mappings the normal
kernel also uses — with per-plane TLB tags and the previous unwaited
kvm_make_request() plus kick for non-zero-plane callers, VTL0 read
stale PTEs and Windows bugchecked 0x1A (MEMORY_MANAGEMENT, 0x6102)
minutes into an SMP boot. Enqueue the flush entries for every twin of
each target VP and dispatch KVM_REQ_HV_TLB_FLUSH through the waiting
fan-out for callers in either VTL.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/include/asm/kvm_host.h |  6 ++-
 arch/x86/kvm/hyperv.c           | 65 ++++++++++++++++-----------------
 arch/x86/kvm/mmu/mmu.c          | 13 ++++---
 include/linux/kvm_host.h        |  3 +-
 virt/kvm/kvm_main.c             | 22 ++++++++++-
 5 files changed, 66 insertions(+), 43 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 834597791fe1..1ba77d9375cd 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -107,7 +107,8 @@
 	KVM_ARCH_REQ_FLAGS(15, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
 #define KVM_REQ_GLOBAL_CLOCK_UPDATE	KVM_ARCH_REQ(16)
 #define KVM_REQ_APIC_PAGE_RELOAD \
-	KVM_ARCH_REQ_FLAGS(17, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
+	KVM_ARCH_REQ_FLAGS(17, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP | \
+			   KVM_REQUEST_PER_PLANE)
 #define KVM_REQ_HV_CRASH		KVM_ARCH_REQ(18)
 #define KVM_REQ_IOAPIC_EOI_EXIT		KVM_ARCH_PLANE_REQ(19)
 #define KVM_REQ_HV_RESET		KVM_ARCH_REQ(20)
@@ -125,7 +126,8 @@
 #define KVM_REQ_UPDATE_CPU_DIRTY_LOGGING \
 	KVM_ARCH_REQ_FLAGS(30, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
 #define KVM_REQ_MMU_FREE_OBSOLETE_ROOTS \
-	KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
+	KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP | \
+			   KVM_REQUEST_PER_PLANE)
 #define KVM_REQ_HV_TLB_FLUSH \
 	KVM_ARCH_REQ_FLAGS(32, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP | \
 			   KVM_REQUEST_PER_PLANE)
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 0d239f054856..759589c6de87 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -2205,20 +2205,26 @@ static int kvm_hv_hypercall_check_params(struct kvm_vcpu *vcpu,
 }
 
 /*
- * A TLB flush hypercall flushes the caller's VTL view of the target VPs:
- * resolve the target to the vCPU of the caller's plane, which has its own
- * roots and TLB tag.  Fall back to plane 0 when the plane vCPU does not
- * exist (it then has no TLB to flush either).
+ * The reference hypervisor's TLB is VTL-blind: both VTLs share one hardware
+ * translation context, so a flush issued from either VTL covers both and
+ * the secure kernel relies on that when it edits mappings the normal kernel
+ * also uses.  Planes have separate roots and TLB tags, so match the
+ * contract by enqueueing the flush for every twin of the target VP.
  */
-static struct kvm_vcpu *hv_flush_plane_vcpu(struct kvm_vcpu *caller,
-					    struct kvm_vcpu *v)
+static void hv_tlb_flush_enqueue_all_planes(struct kvm_vcpu *v,
+					    u64 *entries, int count)
 {
-	struct kvm_vcpu *pv;
+	struct kvm_vcpu_hv_tlb_flush_fifo *fifo;
+	struct kvm_vcpu *twin;
+	int p;
 
-	if (!caller->plane_level)
-		return v;
-	pv = v->common->vcpus[caller->plane_level];
-	return pv ?: v;
+	for (p = 0; p < KVM_MAX_PLANES; p++) {
+		twin = v->common->vcpus[p];
+		if (!twin || !to_hv_vcpu(twin))
+			continue;
+		fifo = kvm_hv_get_tlb_flush_fifo(twin, false);
+		hv_tlb_flush_enqueue(twin, fifo, entries, count);
+	}
 }
 
 static s64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
@@ -2355,19 +2361,17 @@ static s64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 	 * analyze it here, flush TLB regardless of the specified address space.
 	 */
 	if (all_cpus && !is_guest_mode(vcpu)) {
-		kvm_for_each_vcpu(i, v, kvm) {
-			v = hv_flush_plane_vcpu(vcpu, v);
-			tlb_flush_fifo = kvm_hv_get_tlb_flush_fifo(v, false);
-			hv_tlb_flush_enqueue(v, tlb_flush_fifo,
-					     tlb_flush_entries, hc->rep_cnt);
-			if (vcpu->plane_level) {
-				kvm_make_request(KVM_REQ_HV_TLB_FLUSH, v);
-				kvm_vcpu_kick(v);
-			}
-		}
+		kvm_for_each_vcpu(i, v, kvm)
+			hv_tlb_flush_enqueue_all_planes(v, tlb_flush_entries,
+							hc->rep_cnt);
 
-		if (!vcpu->plane_level)
-			kvm_make_all_cpus_request(kvm, KVM_REQ_HV_TLB_FLUSH);
+		/*
+		 * KVM_REQ_HV_TLB_FLUSH is per-plane and fans out to every
+		 * twin; the WAIT in the fan-out is the barrier the TLFS
+		 * requires before the hypercall returns, for callers in
+		 * either VTL.
+		 */
+		kvm_make_all_cpus_request(kvm, KVM_REQ_HV_TLB_FLUSH);
 	} else if (!is_guest_mode(vcpu)) {
 		sparse_set_to_vcpu_mask(kvm, sparse_banks, valid_bank_mask, vcpu_mask);
 
@@ -2375,19 +2379,12 @@ static s64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 			v = kvm_get_vcpu(kvm, i);
 			if (!v)
 				continue;
-			v = hv_flush_plane_vcpu(vcpu, v);
-			tlb_flush_fifo = kvm_hv_get_tlb_flush_fifo(v, false);
-			hv_tlb_flush_enqueue(v, tlb_flush_fifo,
-					     tlb_flush_entries, hc->rep_cnt);
-			if (vcpu->plane_level) {
-				kvm_make_request(KVM_REQ_HV_TLB_FLUSH, v);
-				kvm_vcpu_kick(v);
-			}
+			hv_tlb_flush_enqueue_all_planes(v, tlb_flush_entries,
+							hc->rep_cnt);
 		}
 
-		if (!vcpu->plane_level)
-			kvm_make_vcpus_request_mask(kvm, KVM_REQ_HV_TLB_FLUSH,
-						    vcpu_mask);
+		kvm_make_vcpus_request_mask(kvm, KVM_REQ_HV_TLB_FLUSH,
+					    vcpu_mask);
 	} else {
 		struct kvm_vcpu_hv *hv_v;
 
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 4b11631d9e99..07a8de28c73d 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6265,11 +6265,14 @@ void kvm_mmu_free_obsolete_roots(struct kvm_vcpu *vcpu)
 	int i;
 
 	/*
-	 * KVM_REQ_MMU_FREE_OBSOLETE_ROOTS is tracked in the request set
-	 * shared by all of the vCPU's planes, so the plane that consumes
-	 * the request is not necessarily the plane whose root is obsolete.
-	 * Service every plane: the inactive twins cannot be running (all
-	 * planes of a vCPU share one thread), so their MMUs are stable.
+	 * KVM_REQ_MMU_FREE_OBSOLETE_ROOTS fans out to every plane twin,
+	 * but each twin only consumes its own copy when it next runs, and
+	 * a plane that userspace never re-enters would pin its obsolete
+	 * roots indefinitely.  Service every plane from whichever twin
+	 * consumes its request first: the inactive twins cannot be
+	 * running (all planes of a vCPU share one thread), so their MMUs
+	 * are stable, and a twin's own later request finds nothing left
+	 * to free.
 	 */
 	for (i = 0; i < KVM_MAX_PLANES; i++) {
 		v = vcpu->common->vcpus[i];
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 30dbc28885fb..fdcf401ccb46 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -164,7 +164,8 @@ static inline bool kvm_is_error_gpa(gpa_t gpa)
  * Architecture-independent vcpu->requests bit members
  * Bits 3-7 are reserved for more arch-independent bits.
  */
-#define KVM_REQ_TLB_FLUSH		(0 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
+#define KVM_REQ_TLB_FLUSH \
+	(0 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP | KVM_REQUEST_PER_PLANE)
 #define KVM_REQ_VM_DEAD			(1 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
 #define KVM_REQ_UNBLOCK			2
 #define KVM_REQ_DIRTY_RING_SOFT_FULL	3
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index fac30b8b6518..3e021683c8a8 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -218,9 +218,29 @@ static void kvm_make_vcpu_request(struct kvm_vcpu *vcpu, unsigned int req,
 {
 	int cpu;
 
-	if (likely(!(req & KVM_REQUEST_NO_ACTION)))
+	if (likely(!(req & KVM_REQUEST_NO_ACTION))) {
 		__kvm_make_request(req, vcpu);
 
+		/*
+		 * A per-plane request raised on "the vCPU" (TLB flush after
+		 * a host-side zap, obsolete-roots reload, ...) concerns
+		 * every plane: fan it out to the twins, or the plane that
+		 * runs next consumes a bit the others still need.  Mode,
+		 * pCPU and wait are shared with the common, so the kick
+		 * logic below covers the twins too.
+		 */
+		if (req & KVM_REQUEST_PER_PLANE) {
+			struct kvm_vcpu *twin;
+			int p;
+
+			for (p = 0; p < KVM_MAX_PLANES; p++) {
+				twin = vcpu->common->vcpus[p];
+				if (twin && twin != vcpu)
+					__kvm_make_request(req, twin);
+			}
+		}
+	}
+
 	if (!(req & KVM_REQUEST_NO_WAKEUP) && kvm_vcpu_wake_up(vcpu))
 		return;
 
-- 
2.50.1

[RFC PATCH 28/55] KVM: x86: inhibit APICv when VM planes are enabled

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

With APICv live on plane-0 vCPUs while their plane-1 twins use
software injection, the twins trip the APICv coherency checks: stale
vcpu-vs-VM activation state at vcpu_enter_guest(), and timer interrupt
injection against a vCPU that is not the running one. Cross-plane
event delivery (forced VTL entry, twin kicks, NMI folding) depends on
software-injection bookkeeping that hardware acceleration bypasses.

Add APICV_INHIBIT_REASON_PLANES, set it when a plane above 0 is
created, and include it in the VMX and AVIC required-inhibit masks;
__kvm_set_or_clear_apicv_inhibit() silently ignores a reason the
vendor mask does not carry. The reference VSM hypervisor likewise
force-disables APICv when VTLs are enabled.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/include/asm/kvm_host.h | 10 +++++++++-
 arch/x86/kvm/svm/svm.h          |  3 ++-
 arch/x86/kvm/vmx/main.c         |  3 ++-
 arch/x86/kvm/x86.c              | 10 ++++++++++
 4 files changed, 23 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 1ba77d9375cd..5945b57c091d 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1436,6 +1436,13 @@ enum kvm_apicv_inhibit {
 	 */
 	APICV_INHIBIT_REASON_PHYSICAL_ID_TOO_BIG,
 
+	/*
+	 * APICv is inhibited when the VM has multiple planes (Hyper-V VTLs).
+	 * Cross-plane event delivery relies on software injection; hardware
+	 * acceleration on one twin would bypass it.
+	 */
+	APICV_INHIBIT_REASON_PLANES,
+
 	NR_APICV_INHIBIT_REASONS,
 };
 
@@ -1455,7 +1462,8 @@ enum kvm_apicv_inhibit {
 	__APICV_INHIBIT_REASON(PIT_REINJ),		\
 	__APICV_INHIBIT_REASON(SEV),			\
 	__APICV_INHIBIT_REASON(LOGICAL_ID_ALIASED),	\
-	__APICV_INHIBIT_REASON(PHYSICAL_ID_TOO_BIG)
+	__APICV_INHIBIT_REASON(PHYSICAL_ID_TOO_BIG),	\
+	__APICV_INHIBIT_REASON(PLANES)
 
 struct kvm_possible_nx_huge_pages {
 	/*
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 7a1bf4f5b8e0..64438317d148 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -925,7 +925,8 @@ extern struct kvm_x86_nested_ops svm_nested_ops;
 	BIT(APICV_INHIBIT_REASON_APIC_ID_MODIFIED) |	\
 	BIT(APICV_INHIBIT_REASON_APIC_BASE_MODIFIED) |	\
 	BIT(APICV_INHIBIT_REASON_LOGICAL_ID_ALIASED) |	\
-	BIT(APICV_INHIBIT_REASON_PHYSICAL_ID_TOO_BIG)	\
+	BIT(APICV_INHIBIT_REASON_PHYSICAL_ID_TOO_BIG) |	\
+	BIT(APICV_INHIBIT_REASON_PLANES)		\
 )
 
 bool __init avic_hardware_setup(void);
diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c
index 572921bdfb32..81a8d398068b 100644
--- a/arch/x86/kvm/vmx/main.c
+++ b/arch/x86/kvm/vmx/main.c
@@ -876,7 +876,8 @@ static int vt_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn,
 	 BIT(APICV_INHIBIT_REASON_BLOCKIRQ) |			\
 	 BIT(APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED) |	\
 	 BIT(APICV_INHIBIT_REASON_APIC_ID_MODIFIED) |		\
-	 BIT(APICV_INHIBIT_REASON_APIC_BASE_MODIFIED))
+	 BIT(APICV_INHIBIT_REASON_APIC_BASE_MODIFIED) |	\
+	 BIT(APICV_INHIBIT_REASON_PLANES))
 
 struct kvm_x86_ops vt_x86_ops __initdata = {
 	.name = KBUILD_MODNAME,
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 3e2465b2fa2b..052821799120 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -13979,6 +13979,16 @@ int kvm_arch_plane_init(struct kvm *kvm, struct kvm_plane *plane,
 {
 	mutex_init(&plane->arch.apic_map_lock);
 
+	/*
+	 * Cross-plane event delivery (forced VTL entry, twin kicks, NMI
+	 * folding) relies on software injection via KVM_REQ_EVENT; hardware
+	 * APIC virtualization on one twin would bypass the bookkeeping the
+	 * other twin's plane depends on.  Mirrors the reference VSM
+	 * hypervisor, which force-disables APICv when VTLs are enabled.
+	 */
+	if (plane_level > 0)
+		kvm_set_apicv_inhibit(kvm, APICV_INHIBIT_REASON_PLANES);
+
 	return 0;
 }
 
-- 
2.50.1

[RFC PATCH 29/55] KVM: x86: hyperv: divert LLDT and LTR into register intercepts

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

When a higher VTL arms the ldtr_write or tr_write bits of
HvRegisterCrInterceptControl, hv_divert_dt_write() covers GDTR and
IDTR writes, but an LTR in the lower VTL retires natively, and the
LDTR divert sat in emulator_set_segment(), which task-switch emulation
also reaches via __load_segment_descriptor(). A protected-mode task
switch with the intercept armed — reachable by a malicious lower VTL,
even though 64-bit Windows takes no task switches — had its LDTR load
silently dropped, and the higher VTL received a bogus mid-task-switch
intercept carrying the wrong RIP and instruction length. The TLFS
intercepts cover the LLDT and LTR instructions only.

Add a divert_seg_write(ctxt, selector, seg) emulator op and call it
from em_lldt() and em_ltr() before load_segment_descriptor().
Diverting from the instruction handlers, before any architectural
effect, keeps hardware task switches out of the intercept; for LTR it
also matters that __load_segment_descriptor() marks the TSS descriptor
busy in the GDT first, and a busy descriptor would #GP the LTR when it
re-executes after a cooperative VTL return. emulator_set_segment() no
longer diverts anything.

No vmx change is needed: SECONDARY_EXEC_DESC already exits on LTR
(handle_desc covers EXIT_REASON_LDTR_TR) whenever any
reg_intercept_control bit is set.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/emulate.c     | 21 +++++++++++++++++++++
 arch/x86/kvm/kvm_emulate.h | 10 ++++++++++
 arch/x86/kvm/x86.c         | 33 +++++++++++++++++++++------------
 3 files changed, 52 insertions(+), 12 deletions(-)

diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
index 8013dccb3110..86d165b8185c 100644
--- a/arch/x86/kvm/emulate.c
+++ b/arch/x86/kvm/emulate.c
@@ -3387,6 +3387,16 @@ static int em_lldt(struct x86_emulate_ctxt *ctxt)
 
 	/* Disable writeback. */
 	ctxt->dst.type = OP_NONE;
+
+	/*
+	 * A higher VTL's LDTR-write intercept suspends the LLDT before any
+	 * architectural effect.  Diverting here rather than in
+	 * emulator_set_segment() keeps task switches, which load LDTR
+	 * through set_segment as well, out of the intercept.
+	 */
+	if (ctxt->ops->divert_seg_write(ctxt, sel, VCPU_SREG_LDTR))
+		return X86EMUL_CONTINUE;
+
 	return load_segment_descriptor(ctxt, sel, VCPU_SREG_LDTR);
 }
 
@@ -3401,6 +3411,17 @@ static int em_ltr(struct x86_emulate_ctxt *ctxt)
 
 	/* Disable writeback. */
 	ctxt->dst.type = OP_NONE;
+
+	/*
+	 * A higher VTL's TR-write intercept suspends the LTR before any
+	 * architectural effect.  The divert must precede
+	 * load_segment_descriptor(): that path marks the TSS descriptor
+	 * busy in the GDT, and a busy descriptor would #GP the re-executed
+	 * LTR after a cooperative VTL return.
+	 */
+	if (ctxt->ops->divert_seg_write(ctxt, sel, VCPU_SREG_TR))
+		return X86EMUL_CONTINUE;
+
 	return load_segment_descriptor(ctxt, sel, VCPU_SREG_TR);
 }
 
diff --git a/arch/x86/kvm/kvm_emulate.h b/arch/x86/kvm/kvm_emulate.h
index 0abff36d0994..8a3b9a40934b 100644
--- a/arch/x86/kvm/kvm_emulate.h
+++ b/arch/x86/kvm/kvm_emulate.h
@@ -202,6 +202,16 @@ struct x86_emulate_ops {
 			    struct desc_struct *desc, u32 *base3, int seg);
 	void (*set_segment)(struct x86_emulate_ctxt *ctxt, u16 selector,
 			    struct desc_struct *desc, u32 base3, int seg);
+	/*
+	 * Divert an LTR or LLDT into a Hyper-V VSM register-intercept exit
+	 * when the higher VTL armed one.  Returns true when the write was
+	 * diverted: the instruction must then have no architectural effect,
+	 * so this is checked before descriptor validation (and, for LTR,
+	 * the busy-bit write).  Only the instructions divert; task-switch
+	 * loads of LDTR/TR are not covered by the TLFS intercepts.
+	 */
+	bool (*divert_seg_write)(struct x86_emulate_ctxt *ctxt, u16 selector,
+				 int seg);
 	unsigned long (*get_cached_segment_base)(struct x86_emulate_ctxt *ctxt,
 						 int seg);
 	void (*get_gdt)(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 052821799120..c5122e6f42bd 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -8859,9 +8859,11 @@ static bool hv_divert_dt_write(struct x86_emulate_ctxt *ctxt, u32 hv_reg,
 #define HV_CR_INTERCEPT_GDTR_WRITE	BIT_ULL(15)
 #define HV_CR_INTERCEPT_IDTR_WRITE	BIT_ULL(16)
 #define HV_CR_INTERCEPT_LDTR_WRITE	BIT_ULL(17)
+#define HV_CR_INTERCEPT_TR_WRITE	BIT_ULL(18)
 #define HV_X64_REGISTER_IDTR_ID		0x00070000
 #define HV_X64_REGISTER_GDTR_ID		0x00070001
 #define HV_X64_REGISTER_LDTR_ID		0x00060006
+#define HV_X64_REGISTER_TR_ID		0x00060007
 
 static void emulator_set_gdt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt)
 {
@@ -8949,22 +8951,28 @@ static void emulator_set_segment(struct x86_emulate_ctxt *ctxt, u16 selector,
 	var.unusable = !var.present;
 	var.padding = 0;
 
-	if (seg == VCPU_SREG_LDTR) {
-		struct desc_ptr sel_as_dt = {
-			.address = selector,
-			.size = 0,
-		};
-
-		/* The intercept message carries the written selector. */
-		if (hv_divert_dt_write(ctxt, HV_X64_REGISTER_LDTR_ID,
-				       &sel_as_dt, HV_CR_INTERCEPT_LDTR_WRITE))
-			return;
-	}
-
 	kvm_set_segment(vcpu, &var, seg);
 	return;
 }
 
+static bool emulator_divert_seg_write(struct x86_emulate_ctxt *ctxt,
+				      u16 selector, int seg)
+{
+	struct desc_ptr sel_as_dt = {
+		.address = selector,
+		.size = 0,
+	};
+
+	/* The intercept message carries the written selector. */
+	if (seg == VCPU_SREG_LDTR)
+		return hv_divert_dt_write(ctxt, HV_X64_REGISTER_LDTR_ID,
+					  &sel_as_dt,
+					  HV_CR_INTERCEPT_LDTR_WRITE);
+
+	return hv_divert_dt_write(ctxt, HV_X64_REGISTER_TR_ID, &sel_as_dt,
+				  HV_CR_INTERCEPT_TR_WRITE);
+}
+
 static int emulator_get_msr_with_filter(struct x86_emulate_ctxt *ctxt,
 					u32 msr_index, u64 *pdata)
 {
@@ -9167,6 +9175,7 @@ static const struct x86_emulate_ops emulate_ops = {
 	.pio_out_emulated    = emulator_pio_out_emulated,
 	.get_segment         = emulator_get_segment,
 	.set_segment         = emulator_set_segment,
+	.divert_seg_write    = emulator_divert_seg_write,
 	.get_cached_segment_base = emulator_get_cached_segment_base,
 	.get_gdt             = emulator_get_gdt,
 	.get_idt	     = emulator_get_idt,
-- 
2.50.1

[RFC PATCH 30/55] KVM: planes: Unwind published state on KVM_CREATE_PLANE failure

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

kvm_create_plane() publishes the new plane in kvm->planes[] and takes
the APICV_INHIBIT_REASON_PLANES inhibit before
kvm_vm_ioctl_create_plane() can still fail at anon_inode_getfile(),
and the error path freed the plane without unpublishing it: a VMM
retrying plane setup after such a failure wrote overlays through the
freed plane via KVM_SET_PLANE_MEMORY_OVERLAY, and VM teardown
destroyed the plane a second time. The VM also kept APICv inhibited
with no nonzero plane left to justify it.

Clear the kvm->planes[] slot before freeing, and drop the APICv
inhibit in kvm_arch_plane_destroy() when the last nonzero plane of a
live VM goes away (at VM teardown plane 0 is already gone, which skips
the update). Make the overlay ioctl read kvm->planes[] under
kvm->lock, which the create path holds across publish and unpublish,
so a plane pointer it observes stays valid until VM teardown; publish
with smp_store_release() for the remaining lock-free readers.

Found by Sashiko review.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/x86.c  | 32 ++++++++++++++++++++++++++++++++
 virt/kvm/kvm_main.c | 25 ++++++++++++++++++++++---
 2 files changed, 54 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index c5122e6f42bd..afd668c6db3f 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -14003,7 +14003,39 @@ int kvm_arch_plane_init(struct kvm *kvm, struct kvm_plane *plane,
 
 void kvm_arch_plane_destroy(struct kvm_plane *plane)
 {
+	struct kvm *kvm = plane->kvm;
+	int i;
+
 	kvfree(rcu_dereference_check(plane->arch.apic_map, 1));
+
+	if (plane->level == 0)
+		return;
+
+	/*
+	 * Drop the APICv inhibit taken in kvm_arch_plane_init() when the
+	 * last nonzero plane goes away.  Only the KVM_CREATE_PLANE error
+	 * path destroys a nonzero plane while the VM is live; at VM
+	 * teardown plane 0 is destroyed first and kvm->planes[0] is
+	 * already NULL, which skips the update (the inhibit machinery
+	 * needs plane 0 and SRCU, both gone by then).
+	 */
+	if (!kvm->planes[0])
+		return;
+
+	/*
+	 * Reaching here means a nonzero plane died while the VM is live,
+	 * which only the KVM_CREATE_PLANE error path does - under
+	 * kvm->lock.  A future kvm_destroy_planes() reordered to tear
+	 * nonzero planes down before plane 0 would get here locklessly;
+	 * catch that instead of silently poking freed APICv state.
+	 */
+	lockdep_assert_held(&kvm->lock);
+
+	for (i = 1; i < KVM_MAX_PLANES; i++)
+		if (kvm->planes[i] && kvm->planes[i] != plane)
+			return;
+
+	kvm_clear_apicv_inhibit(kvm, APICV_INHIBIT_REASON_PLANES);
 }
 
 int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 3e021683c8a8..ae998cef3c36 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -1259,7 +1259,12 @@ static struct kvm_plane *kvm_create_plane(struct kvm *kvm, unsigned plane_level)
 	if (kvm_arch_plane_init(kvm, plane, plane_level))
 		goto out_free_plane;
 
-	kvm->planes[plane_level] = plane;
+	/*
+	 * Pairs with the lock-free readers of kvm->planes[] (MSI delivery,
+	 * SEV VMPL targeting): the plane's fields must be visible before
+	 * the pointer is.
+	 */
+	smp_store_release(&kvm->planes[plane_level], plane);
 
 	return plane;
 
@@ -5915,8 +5920,15 @@ static int kvm_vm_ioctl_create_plane(struct kvm *kvm, unsigned id)
 	return fd;
 
 put_kvm:
-	kvm_put_kvm(kvm);
+	/*
+	 * kvm_create_plane() published the plane; unpublish it before
+	 * freeing (kvm->lock is held throughout, so readers who take it
+	 * never observe the dead pointer) and so that VM teardown does
+	 * not destroy it a second time.
+	 */
+	kvm->planes[id] = NULL;
 	kvm_destroy_one_plane(plane);
+	kvm_put_kvm(kvm);
 put_fd:
 	put_unused_fd(fd);
 	return r;
@@ -6037,7 +6049,14 @@ static int kvm_vm_ioctl_set_plane_overlay(struct kvm *kvm,
 		return -EINVAL;
 	if (ov->uaddr & ~PAGE_MASK)
 		return -EINVAL;
-	plane = kvm->planes[ov->plane];
+	/*
+	 * Read under kvm->lock: KVM_CREATE_PLANE publishes and, on its
+	 * error path, unpublishes-and-frees the plane under that lock, so
+	 * a pointer observed here is either NULL or a plane that lives
+	 * until VM teardown.
+	 */
+	scoped_guard(mutex, &kvm->lock)
+		plane = kvm->planes[ov->plane];
 	if (!plane)
 		return -EINVAL;
 
-- 
2.50.1

[RFC PATCH 31/55] KVM: planes: Kick the running twin's pCPU, not the request target's

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The remote kick paths (kvm_make_vcpus_request_mask(),
kvm_make_all_cpus_request(), __kvm_vcpu_kick()) read
READ_ONCE(vcpu->cpu) of the plane-0 vCPU. But vcpu->cpu is per plane
vCPU — kvm_arch_vcpu_load() refreshes only the loaded twin's copy —
while vcpu->mode is shared in kvm_vcpu_common. When the vCPU thread
has migrated pCPUs while a non-zero plane was loaded, the shared mode
reads IN_GUEST_MODE but the IPI goes to plane 0's stale pCPU: a flush
hypercall returns HV_STATUS_SUCCESS while the twin keeps running on
stale translations, breaking the TLFS guarantee that the flush covers
the whole VP.

Read the pCPU from common->current_vcpu, whose cpu field every
kvm_arch_vcpu_load() refreshes, in both kick paths. Also read the twin
array with READ_ONCE in the per-plane request fan-out and correct the
comment there: mode and wait are shared with the common, the pCPU is
not.

Found by Sashiko review.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 virt/kvm/kvm_main.c | 25 +++++++++++++++++++------
 1 file changed, 19 insertions(+), 6 deletions(-)

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index ae998cef3c36..fa639a659138 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -225,16 +225,16 @@ static void kvm_make_vcpu_request(struct kvm_vcpu *vcpu, unsigned int req,
 		 * A per-plane request raised on "the vCPU" (TLB flush after
 		 * a host-side zap, obsolete-roots reload, ...) concerns
 		 * every plane: fan it out to the twins, or the plane that
-		 * runs next consumes a bit the others still need.  Mode,
-		 * pCPU and wait are shared with the common, so the kick
-		 * logic below covers the twins too.
+		 * runs next consumes a bit the others still need.  Mode and
+		 * wait live in the common and the kick below targets the
+		 * running twin's pCPU, so it covers the twins too.
 		 */
 		if (req & KVM_REQUEST_PER_PLANE) {
 			struct kvm_vcpu *twin;
 			int p;
 
 			for (p = 0; p < KVM_MAX_PLANES; p++) {
-				twin = vcpu->common->vcpus[p];
+				twin = READ_ONCE(vcpu->common->vcpus[p]);
 				if (twin && twin != vcpu)
 					__kvm_make_request(req, twin);
 			}
@@ -255,7 +255,14 @@ static void kvm_make_vcpu_request(struct kvm_vcpu *vcpu, unsigned int req,
 	 * finalized. See kvm_vcpu_kick() for more details on handling requests.
 	 */
 	if (kvm_request_needs_ipi(vcpu, req)) {
-		cpu = READ_ONCE(vcpu->cpu);
+		/*
+		 * vcpu->cpu is per plane vCPU and kvm_arch_vcpu_load()
+		 * refreshes only the loaded twin's, so this plane's copy
+		 * can point at a pCPU the thread has long left while a
+		 * twin runs elsewhere.  The mode just checked is shared;
+		 * the IPI must go where the running twin actually is.
+		 */
+		cpu = READ_ONCE(READ_ONCE(vcpu->common->current_vcpu)->cpu);
 		if (cpu != -1 && cpu != current_cpu)
 			__cpumask_set_cpu(cpu, tmp);
 	}
@@ -4334,7 +4341,13 @@ void __kvm_vcpu_kick(struct kvm_vcpu *vcpu, bool wait)
 	 * vCPU also requires it to leave IN_GUEST_MODE.
 	 */
 	if (kvm_arch_vcpu_should_kick(vcpu)) {
-		cpu = READ_ONCE(vcpu->cpu);
+		/*
+		 * Target the running twin's pCPU: this plane's vcpu->cpu
+		 * goes stale whenever the thread migrates while another
+		 * plane is loaded, and the IN_GUEST_MODE just observed is
+		 * the (shared) mode of whichever twin runs.
+		 */
+		cpu = READ_ONCE(READ_ONCE(vcpu->common->current_vcpu)->cpu);
 		if (cpu != me && (unsigned int)cpu < nr_cpu_ids && cpu_online(cpu)) {
 			/*
 			 * Use a reschedule IPI to kick the vCPU if the caller
-- 
2.50.1

[RFC PATCH 32/55] KVM: planes: Keep current_vcpu in sync in vcpu_load()

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The preempt notifiers save and restore common->current_vcpu
(kvm_sched_out()/kvm_sched_in()), but vcpu_load() does not update it.
kvm_vcpu_ioctl_set_cpuid() swaps the loaded vCPU to each twin around
kvm_vcpu_after_set_cpuid(), a window that is preemptible and sleeps
(kvm_pmu_refresh, mmu_lock): a reschedule there arch-put the
already-put caller a second time, never saved the twin, and reloaded
the caller's VMCS, so the vendor callback wrote the twin's controls
into the caller's VMCS and the trailing vcpu_put() unloaded a
non-resident vCPU, corrupting the per-pCPU loaded_vmcs tracking. Any
KVM_SET_CPUID2 on a multi-plane VM reaches this; ioctls taken on a
plane-N fd while current_vcpu pointed at another plane had the same
exposure, as do the kick paths that now read current_vcpu->cpu.

Set common->current_vcpu in vcpu_load(), making every load path (run
loop, vcpu ioctls, the cpuid twin swap) agree with what the preempt
notifiers will operate on. kvm_vcpu_select_plane() keeps its own
assignment; both stores use WRITE_ONCE to pair with the lock-free
readers in the kick paths.

Found by Sashiko review.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 virt/kvm/kvm_main.c | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index fa639a659138..d37961cc5235 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -165,6 +165,16 @@ void vcpu_load(struct kvm_vcpu *vcpu)
 {
 	int cpu = get_cpu();
 
+	/*
+	 * The preempt notifiers operate on common->current_vcpu, so it
+	 * must name the plane vCPU being loaded here: an ioctl taken on
+	 * another plane's fd than the one that ran last would otherwise
+	 * have a preemption arch-put the stale twin (whose state is not
+	 * resident) and arch-load it back over the state this load makes
+	 * resident.  The kick paths also read it to find the running
+	 * twin's pCPU.
+	 */
+	WRITE_ONCE(vcpu->common->current_vcpu, vcpu);
 	__this_cpu_write(kvm_running_vcpu, vcpu->common);
 	preempt_notifier_register(&vcpu->common->preempt_notifier);
 	kvm_arch_vcpu_load(vcpu, cpu);
@@ -4719,7 +4729,26 @@ static int kvm_plane_ioctl_create_vcpu(struct kvm_plane *plane, unsigned long id
 
 	kvm_vcpu_init(vcpu, kvm, id);
 
+	/*
+	 * A non-zero plane shares 'common' with a plane-0 twin which may
+	 * already be running.  kvm_arch_vcpu_create() calls vcpu_load(),
+	 * which publishes the half-created vCPU through
+	 * common->current_vcpu and registers the shared preempt notifier,
+	 * so it must not run concurrently with a twin's KVM_RUN: hold the
+	 * shared vCPU mutex across it.  Drop it before kvm->lock is taken
+	 * below, to respect the kvm->lock -> vcpu mutex ordering.
+	 */
+	if (plane->level > 0) {
+		r = mutex_lock_killable(kvm_vcpu_mutex(vcpu));
+		if (r)
+			goto vcpu_free_common;
+	}
+
 	r = kvm_arch_vcpu_create(vcpu);
+
+	if (plane->level > 0)
+		kvm_vcpu_unlock(vcpu);
+
 	if (r)
 		goto vcpu_free_common;
 
@@ -5427,7 +5456,7 @@ struct kvm_vcpu *kvm_vcpu_select_plane(struct kvm_vcpu *vcpu)
 		ret->plane_state = RUNNABLE;
 	}
 
-	common->current_vcpu = ret;
+	WRITE_ONCE(common->current_vcpu, ret);
 
 	common->plane_switch = false;
 
-- 
2.50.1

[RFC PATCH 33/55] KVM: x86: Report a real write-access walk from KVM_TRANSLATE

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

KVM_TRANSLATE hardcodes writeable = 1, so userspace answering
HvCallTranslateVirtualAddress from this ioctl cannot honour the
hypercall's write-validation flag: validating write access against a
read-only PTE succeeds instead of reporting the address unmapped, and
the Windows secure kernel's page verification trusts a mapping it
cannot write.

Walk the address a second time with PFERR_WRITE_MASK at supervisor
level and report the outcome, matching the reference hypervisor's
validation walk.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/x86.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index afd668c6db3f..491a0360c403 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -13305,10 +13305,19 @@ int kvm_arch_vcpu_ioctl_translate(struct kvm_vcpu *vcpu,
 
 	idx = srcu_read_lock(&vcpu->kvm->srcu);
 	gpa = kvm_mmu_gva_to_gpa_system(vcpu, vaddr, NULL);
+	/*
+	 * Report write access from a supervisor-level walk instead of the
+	 * hardcoded 1: Hyper-V's TranslateVirtualAddress write validation
+	 * is answered from this ioctl, and the reference hypervisor walks
+	 * with PFERR_WRITE_MASK and no user bit.
+	 */
+	tr->writeable = vcpu->arch.walk_mmu->gva_to_gpa(vcpu,
+							vcpu->arch.walk_mmu,
+							vaddr, PFERR_WRITE_MASK,
+							NULL) != INVALID_GPA;
 	srcu_read_unlock(&vcpu->kvm->srcu, idx);
 	tr->physical_address = gpa;
 	tr->valid = gpa != INVALID_GPA;
-	tr->writeable = 1;
 	tr->usermode = 0;
 
 	vcpu_put(vcpu);
-- 
2.50.1

[RFC PATCH 34/55] KVM: planes: Share XCR0 across plane twins

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

XCR0/XFEM is shared between VTLs: the TLFS banks no vector state and
the reference host keeps a single XCR0 (its store_vtl/load_vtl carry
no XSAVE state). The planes series shares the FPU payload between
twins but keeps arch.xcr0 per-vCPU, so an XSETBV in one VTL was
invisible in the other: VTL1 read a stale XFEM, and state enabled by
VTL0 (AVX, for the Windows secure kernel) silently vanished across a
VTL switch.

Propagate the validated value to every plane twin in __kvm_set_xcr(),
covering both guest XSETBV and the KVM_SET_XCRS ioctl path. The CPUID
dynamic-bits dirty marker lives in the shared common state, so the
existing single set covers the twins.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/x86.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 491a0360c403..0f8a149ba0eb 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1361,6 +1361,28 @@ int __kvm_set_xcr(struct kvm_vcpu *vcpu, u32 index, u64 xcr)
 
 	vcpu->arch.xcr0 = xcr0;
 
+	/*
+	 * XCR0/XFEM is shared across VTLs (TLFS 15.4; the reference host
+	 * banks no vector state): propagate the new value to every plane
+	 * twin so the other VTLs observe it.  This covers both guest
+	 * XSETBV and the KVM_SET_XCRS ioctl path.  The CPUID dirty marker
+	 * below lives in the shared common state, so one set covers the
+	 * twins too.
+	 *
+	 * Skip the fan-out until this vCPU is published in
+	 * common->vcpus[]: the architectural reset of a twin still being
+	 * created must not clobber a live sibling's XCR0 (the new twin
+	 * inherits the shared value instead, in kvm_arch_vcpu_create()).
+	 */
+	if (vcpu->common &&
+	    vcpu->common->vcpus[vcpu->plane_level] == vcpu) {
+		struct kvm_vcpu *twin;
+		int i;
+
+		vcpu_for_each_plane(vcpu->common, i, twin)
+			twin->arch.xcr0 = xcr0;
+	}
+
 	if ((xcr0 ^ old_xcr0) & XFEATURE_MASK_EXTEND)
 		cpuid_set_dirty(vcpu);
 	return 0;
@@ -13549,6 +13571,14 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 	kvm_vcpu_after_set_cpuid(vcpu);
 	kvm_set_tsc_khz(vcpu, vcpu->kvm->arch.default_tsc_khz);
 	kvm_vcpu_reset(vcpu, false);
+	/*
+	 * XCR0 is shared across planes.  A twin created after its plane-0
+	 * sibling has run (Hyper-V VSM enables VTL1 at runtime) inherits
+	 * the current shared value; the architectural reset value only
+	 * stands when no sibling exists yet.
+	 */
+	if (vcpu->plane_level && vcpu->common->vcpus[0])
+		vcpu->arch.xcr0 = vcpu->common->vcpus[0]->arch.xcr0;
 	kvm_init_mmu(vcpu);
 	vcpu_put(vcpu);
 	return 0;
-- 
2.50.1

[RFC PATCH 35/55] KVM: x86: Leave TSC matching to the plane 0 vCPUs

[David Woodhouse]

From: Mushahid Hussain <hmushi@amazon.co.uk>

KVM enables the master clock when every vCPU has a matching TSC, judged
by nr_vcpus_matched_tsc + 1 == online_vcpus.  online_vcpus counts a vCPU
once, but each of its plane vCPUs synchronizes its TSC at creation and
counts as another matching vCPU.  With one higher plane the count can
never equal the number of vCPUs again, the master clock stays off and
kvmclock loses PVCLOCK_TSC_STABLE_BIT.  Among other things the Hyper-V
reference TSC page then breaks: KVM writes sequence 0 to it, finds the
clock unstable and leaves it that way.

The planes of a vCPU share its TSC.  Give a plane vCPU the current
generation and leave the matching count and generation changes to its
plane 0 sibling.

Assisted-by: LLM
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/kvm/x86.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 0f8a149ba0eb..ba468f7c4d4a 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2858,6 +2858,31 @@ static void __kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 offset, u64 tsc,
 	if (vcpu->arch.guest_tsc_protected)
 		return;
 
+	/*
+	 * The planes of a vCPU share its TSC and online_vcpus counts the
+	 * vCPU once, so a plane vCPU neither counts as a matching vCPU nor
+	 * starts a generation; its plane 0 sibling speaks for it.  Nor may
+	 * it perturb the global matching window (last_tsc_ and
+	 * user_set_tsc): KVM_SET_MSRS is accepted on plane fds, and a TSC
+	 * write on a plane fd
+	 * moving the window would corrupt the next plane-0 write's match
+	 * decision.  Latch the generation from the plane-0 sibling, which
+	 * exists by construction for a plane twin, rather than from the
+	 * VM-global cur_tsc_* that may belong to a different vCPU's
+	 * generation entirely.
+	 */
+	if (vcpu->plane_level) {
+		struct kvm_vcpu *sibling = vcpu->common->vcpus[0];
+
+		vcpu->arch.last_guest_tsc = tsc;
+		kvm_vcpu_write_tsc_offset(vcpu, offset);
+
+		vcpu->arch.this_tsc_generation = sibling->arch.this_tsc_generation;
+		vcpu->arch.this_tsc_nsec = sibling->arch.this_tsc_nsec;
+		vcpu->arch.this_tsc_write = sibling->arch.this_tsc_write;
+		return;
+	}
+
 	if (user_set_tsc)
 		vcpu->kvm->arch.user_set_tsc = true;
 
-- 
2.50.1

[RFC PATCH 36/55] KVM: x86: hyperv: Track the reference TSC page per plane

[David Woodhouse]

From: Mushahid Hussain <hmushi@amazon.co.uk>

HV_X64_MSR_REFERENCE_TSC is private to each Virtual Trust Level: the
secure VTL maps its own page.  The reference time behind the pages is
one clock per partition, so every page must show the same scale, offset
and sequence.

KVM keeps a single page address per VM.  A write from another plane
replaces it, the old page stops being updated and the new page starts a
sequence of its own.  Keep the MSR value per plane and write the shared
page contents to every enabled page.  The lowest enabled plane owns the
sequence number; disabling one plane's page keeps the others updated.

Assisted-by: LLM
[dwmw2: our tree already banks the MSR per plane (fbdffa08a242) with a
  one-shot copy for higher planes; replace that with the maintained
  pages, keeping the bank in kvm_hv rather than kvm_arch_plane. ]
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/include/asm/kvm_host.h |   6 +-
 arch/x86/kvm/hyperv.c           | 118 +++++++++++++++-----------------
 2 files changed, 58 insertions(+), 66 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 5945b57c091d..d913c68b18a7 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1255,8 +1255,10 @@ struct kvm_hv {
 	u64 hv_guest_os_id[KVM_MAX_PLANES];
 	u64 hv_hypercall[KVM_MAX_PLANES];
 	bool hv_forward_hcalls;
-	u64 hv_tsc_page;
-	/* Banked REFERENCE_TSC MSR values for non-zero planes (VTLs). */
+	/*
+	 * HV_X64_MSR_REFERENCE_TSC is private to each VTL, the reference time
+	 * it exposes is not: every enabled page gets the same contents.
+	 */
 	u64 hv_tsc_page_plane[KVM_MAX_PLANES];
 	enum hv_tsc_page_status hv_tsc_page_status;
 
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 759589c6de87..2098daede7d3 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -1224,6 +1224,49 @@ static inline bool tsc_page_update_unsafe(struct kvm_hv *hv)
 		hv->hv_tsc_emulation_control;
 }
 
+static bool hv_tsc_page_gfn(struct kvm_hv *hv, int plane, u64 *gfn)
+{
+	u64 msr = hv->hv_tsc_page_plane[plane];
+
+	if (!(msr & HV_X64_MSR_TSC_REFERENCE_ENABLE))
+		return false;
+
+	*gfn = msr >> HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT;
+	return true;
+}
+
+/* The lowest plane with an enabled page; it owns the sequence number. */
+static bool hv_first_tsc_page_gfn(struct kvm *kvm, u64 *gfn)
+{
+	struct kvm_hv *hv = to_kvm_hv(kvm);
+	int i;
+
+	for (i = 0; i < KVM_MAX_PLANES; i++) {
+		if (hv_tsc_page_gfn(hv, i, gfn))
+			return true;
+	}
+
+	return false;
+}
+
+/* Write the first @len bytes of the reference page to every enabled plane. */
+static int hv_write_tsc_pages(struct kvm *kvm, size_t len)
+{
+	struct kvm_hv *hv = to_kvm_hv(kvm);
+	u64 gfn;
+	int i;
+
+	for (i = 0; i < KVM_MAX_PLANES; i++) {
+		if (!hv_tsc_page_gfn(hv, i, &gfn))
+			continue;
+
+		if (kvm_write_guest(kvm, gfn_to_gpa(gfn), &hv->tsc_ref, len))
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
 void kvm_hv_setup_tsc_page(struct kvm *kvm,
 			   struct pvclock_vcpu_time_info *hv_clock)
 {
@@ -1241,10 +1284,9 @@ void kvm_hv_setup_tsc_page(struct kvm *kvm,
 	    hv->hv_tsc_page_status == HV_TSC_PAGE_UNSET)
 		return;
 
-	if (!(hv->hv_tsc_page & HV_X64_MSR_TSC_REFERENCE_ENABLE))
+	if (!hv_first_tsc_page_gfn(kvm, &gfn))
 		return;
 
-	gfn = hv->hv_tsc_page >> HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT;
 	/*
 	 * Because the TSC parameters only vary when there is a
 	 * change in the master clock, do not bother with caching.
@@ -1267,8 +1309,7 @@ void kvm_hv_setup_tsc_page(struct kvm *kvm,
 	 * guest to use the time reference count MSR.
 	 */
 	hv->tsc_ref.tsc_sequence = 0;
-	if (kvm_write_guest(kvm, gfn_to_gpa(gfn),
-			    &hv->tsc_ref, sizeof(hv->tsc_ref.tsc_sequence)))
+	if (hv_write_tsc_pages(kvm, sizeof(hv->tsc_ref.tsc_sequence)))
 		goto out_err;
 
 	if (!compute_tsc_page_parameters(hv_clock, &hv->tsc_ref))
@@ -1276,7 +1317,7 @@ void kvm_hv_setup_tsc_page(struct kvm *kvm,
 
 	/* Ensure sequence is zero before writing the rest of the struct.  */
 	smp_wmb();
-	if (kvm_write_guest(kvm, gfn_to_gpa(gfn), &hv->tsc_ref, sizeof(hv->tsc_ref)))
+	if (hv_write_tsc_pages(kvm, sizeof(hv->tsc_ref)))
 		goto out_err;
 
 	/*
@@ -1290,8 +1331,7 @@ void kvm_hv_setup_tsc_page(struct kvm *kvm,
 	smp_wmb();
 
 	hv->tsc_ref.tsc_sequence = tsc_seq;
-	if (kvm_write_guest(kvm, gfn_to_gpa(gfn),
-			    &hv->tsc_ref, sizeof(hv->tsc_ref.tsc_sequence)))
+	if (hv_write_tsc_pages(kvm, sizeof(hv->tsc_ref.tsc_sequence)))
 		goto out_err;
 
 	hv->tsc_ref_from_guest = false;
@@ -1499,61 +1539,12 @@ static int kvm_hv_set_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data,
 		hv->hv_hypercall[vcpu->plane_level] = data;
 		break;
 	}
-	case HV_X64_MSR_REFERENCE_TSC:
-		/*
-		 * The reference TSC page is a per-VTL instance.  Plane 0 keeps
-		 * the live masterclock-maintained page; a non-zero plane gets
-		 * a one-shot copy of the current parameters, which stay valid
-		 * while the masterclock is stable.
-		 */
-		if (vcpu->plane_level) {
-			struct ms_hyperv_tsc_page tsc_ref = {};
-			u32 tsc_seq;
-			u64 gfn;
-
-			hv->hv_tsc_page_plane[vcpu->plane_level] = data;
-			if (!(data & HV_X64_MSR_TSC_REFERENCE_ENABLE))
-				break;
-
-			/*
-			 * Copy only host-computed parameters.  After a
-			 * masterclock change under guest TSC emulation,
-			 * hv->tsc_ref holds values read back from the
-			 * plane-0 page, which the lowest VTL can write;
-			 * presenting those to another plane would let it
-			 * forge that plane's clock.  Without usable
-			 * parameters, publish an invalid (zero) sequence so
-			 * the plane falls back to the TIME_REF_COUNT MSR.
-			 */
-			if (hv->hv_tsc_page_status == HV_TSC_PAGE_SET &&
-			    !hv->tsc_ref_from_guest)
-				tsc_ref = hv->tsc_ref;
-
-			gfn = data >> HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT;
-			tsc_seq = tsc_ref.tsc_sequence;
-
-			/*
-			 * Write the body under a zero sequence, then the
-			 * real sequence, so a concurrent reader on this
-			 * plane never consumes torn parameters.
-			 */
-			tsc_ref.tsc_sequence = 0;
-			if (kvm_vcpu_write_guest(vcpu, gfn_to_gpa(gfn),
-						 &tsc_ref, sizeof(tsc_ref)))
-				return 1;
-
-			if (!tsc_seq)
-				break;
+	case HV_X64_MSR_REFERENCE_TSC: {
+		u64 gfn;
 
-			tsc_ref.tsc_sequence = tsc_seq;
-			if (kvm_vcpu_write_guest(vcpu, gfn_to_gpa(gfn),
-						 &tsc_ref,
-						 sizeof(tsc_ref.tsc_sequence)))
-				return 1;
-			break;
-		}
-		hv->hv_tsc_page = data;
-		if (hv->hv_tsc_page & HV_X64_MSR_TSC_REFERENCE_ENABLE) {
+		hv->hv_tsc_page_plane[vcpu->plane_level] = data;
+		/* Another plane's page may still be enabled: keep it updated. */
+		if (hv_first_tsc_page_gfn(kvm, &gfn)) {
 			if (!host)
 				hv->hv_tsc_page_status = HV_TSC_PAGE_GUEST_CHANGED;
 			else
@@ -1563,6 +1554,7 @@ static int kvm_hv_set_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data,
 			hv->hv_tsc_page_status = HV_TSC_PAGE_UNSET;
 		}
 		break;
+	}
 	case HV_X64_MSR_CRASH_P0 ... HV_X64_MSR_CRASH_P4:
 		return kvm_hv_msr_set_crash_data(kvm,
 						 msr - HV_X64_MSR_CRASH_P0,
@@ -1786,9 +1778,7 @@ static int kvm_hv_get_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata,
 		data = get_time_ref_counter(kvm);
 		break;
 	case HV_X64_MSR_REFERENCE_TSC:
-		data = vcpu->plane_level ?
-		       hv->hv_tsc_page_plane[vcpu->plane_level] :
-		       hv->hv_tsc_page;
+		data = hv->hv_tsc_page_plane[vcpu->plane_level];
 		break;
 	case HV_X64_MSR_CRASH_P0 ... HV_X64_MSR_CRASH_P4:
 		return kvm_hv_msr_get_crash_data(kvm,
-- 
2.50.1

[RFC PATCH 37/55] KVM: x86: hyperv: Update the reference TSC page past protections

[David Woodhouse]

From: Mushahid Hussain <hmushi@amazon.co.uk>

The reference TSC page is a page the hypervisor maintains for the guest.
A higher Virtual Trust Level keeps its own page, and it protects the
memory that holds that page against the lower levels. KVM writes the
page through kvm_write_guest(), which refuses a write to protected
memory. The refusal aborts the update of every page: KVM has written
sequence zero to all of them and leaves them that way for the rest of
the guest's life, so the guest falls back to the time reference count
register for every read of time.

Memory protections restrict the guest. They do not restrict the
hypervisor's update of a page the hypervisor owns. Add
kvm_write_guest_unprotected() for such a page and use it for the
reference TSC page.

Assisted-by: LLM
[dwmw2: the original message called the function kvm_write_guest_hv();
  it is kvm_write_guest_unprotected() in the code, say so. ]
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/kvm/hyperv.c    |  2 +-
 include/linux/kvm_host.h |  2 ++
 virt/kvm/kvm_main.c      | 27 +++++++++++++++++++++++++++
 3 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 2098daede7d3..eebe48f5be23 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -1260,7 +1260,7 @@ static int hv_write_tsc_pages(struct kvm *kvm, size_t len)
 		if (!hv_tsc_page_gfn(hv, i, &gfn))
 			continue;
 
-		if (kvm_write_guest(kvm, gfn_to_gpa(gfn), &hv->tsc_ref, len))
+		if (kvm_write_guest_unprotected(kvm, gfn_to_gpa(gfn), &hv->tsc_ref, len))
 			return -EFAULT;
 	}
 
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index fdcf401ccb46..beae4f2a982a 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1551,6 +1551,8 @@ int kvm_write_guest_page(struct kvm *kvm, gfn_t gfn, const void *data,
 			 int offset, int len);
 int kvm_write_guest(struct kvm *kvm, gpa_t gpa, const void *data,
 		    unsigned long len);
+int kvm_write_guest_unprotected(struct kvm *kvm, gpa_t gpa, const void *data,
+				unsigned long len);
 int kvm_write_guest_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 			   void *data, unsigned long len);
 int kvm_write_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index d37961cc5235..72fe64e5d4a1 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -3832,6 +3832,33 @@ int kvm_write_guest(struct kvm *kvm, gpa_t gpa, const void *data,
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_write_guest);
 
+/* Write a page KVM maintains for the guest; plane protections do not apply. */
+int kvm_write_guest_unprotected(struct kvm *kvm, gpa_t gpa, const void *data,
+				unsigned long len)
+{
+	gfn_t gfn = gpa >> PAGE_SHIFT;
+	int seg;
+	int offset = offset_in_page(gpa);
+
+	while ((seg = next_segment(len, offset)) != 0) {
+		struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
+		unsigned long addr;
+
+		addr = gfn_to_hva_many(slot, gfn, NULL);
+		if (kvm_is_error_hva(addr))
+			return -EFAULT;
+		if (__copy_to_user((void __user *)addr + offset, data, seg))
+			return -EFAULT;
+		mark_page_dirty_in_slot(kvm, slot, gfn);
+		offset = 0;
+		len -= seg;
+		data += seg;
+		++gfn;
+	}
+	return 0;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_write_guest_unprotected);
+
 int kvm_vcpu_write_guest(struct kvm_vcpu *vcpu, gpa_t gpa, const void *data,
 		         unsigned long len)
 {
-- 
2.50.1

[RFC PATCH 38/55] KVM: x86: hyperv: Publish an added reference TSC page at once

[David Woodhouse]

From: Mushahid Hussain <hmushi@amazon.co.uk>

When a plane enables its reference TSC page while another plane's page
is valid, KVM marks the partition's parameters changed and requests a
master clock update. The update writes sequence zero to every enabled
page, computes the parameters again, and writes them to every page. The
parameters have not changed, so the valid pages lose their validity for
nothing, and a guest that reads the time in that window falls back to
the time reference count register.

Write the current parameters to the page the plane enables, the struct
first and the sequence last, and leave the other pages and the status as
they are. Only the first page of the partition, or a change of the
parameters, takes the full update.

Assisted-by: LLM
[dwmw2: keep the tsc_ref_from_guest guard from fbdffa08a242: parameters
  read back from the plane-0 page never seed a higher plane's page; it
  gets a zero sequence instead, as the one-shot copy used to do. ]
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/kvm/hyperv.c | 44 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index eebe48f5be23..2cf73f72836c 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -1235,6 +1235,39 @@ static bool hv_tsc_page_gfn(struct kvm_hv *hv, int plane, u64 *gfn)
 	return true;
 }
 
+/* Publish the current parameters to one plane's page: struct first, then sequence. */
+static int hv_publish_tsc_page(struct kvm *kvm, unsigned int plane, u64 gfn)
+{
+	struct kvm_hv *hv = to_kvm_hv(kvm);
+	struct ms_hyperv_tsc_page page = hv->tsc_ref;
+	u32 tsc_seq = page.tsc_sequence;
+
+	/*
+	 * Parameters read back from the plane-0 page under guest TSC
+	 * emulation are plane-0-writable; handing them to a higher plane
+	 * would let the lowest VTL forge that plane's clock.  Publish a
+	 * zero sequence there instead: the plane falls back to the
+	 * TIME_REF_COUNT MSR.
+	 */
+	if (plane && hv->tsc_ref_from_guest)
+		tsc_seq = 0;
+
+	page.tsc_sequence = 0;
+	if (kvm_write_guest_unprotected(kvm, gfn_to_gpa(gfn), &page, sizeof(page)))
+		return -EFAULT;
+
+	if (!tsc_seq)
+		return 0;
+
+	/* The struct is complete before the sequence makes it valid. */
+	smp_wmb();
+	page.tsc_sequence = tsc_seq;
+	if (kvm_write_guest_unprotected(kvm, gfn_to_gpa(gfn), &page,
+					sizeof(page.tsc_sequence)))
+		return -EFAULT;
+	return 0;
+}
+
 /* The lowest plane with an enabled page; it owns the sequence number. */
 static bool hv_first_tsc_page_gfn(struct kvm *kvm, u64 *gfn)
 {
@@ -1542,6 +1575,17 @@ static int kvm_hv_set_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data,
 	case HV_X64_MSR_REFERENCE_TSC: {
 		u64 gfn;
 
+		/* The parameters are valid already: publish them to this page only. */
+		if (!host && hv->hv_tsc_page_status == HV_TSC_PAGE_SET &&
+		    (data & HV_X64_MSR_TSC_REFERENCE_ENABLE)) {
+			/* The caller holds hv_lock. */
+			hv->hv_tsc_page_plane[vcpu->plane_level] = data;
+			gfn = data >> HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT;
+			if (hv_publish_tsc_page(kvm, vcpu->plane_level, gfn))
+				hv->hv_tsc_page_status = HV_TSC_PAGE_BROKEN;
+			break;
+		}
+
 		hv->hv_tsc_page_plane[vcpu->plane_level] = data;
 		/* Another plane's page may still be enabled: keep it updated. */
 		if (hv_first_tsc_page_gfn(kvm, &gfn)) {
-- 
2.50.1

[RFC PATCH 39/55] KVM: x86: hyperv: Harden the per-plane reference TSC page updates

[David Woodhouse]

From: Mushahid Hussain <hmushi@amazon.co.uk>

The per-plane reference TSC pages were updated through
kvm_write_guest_unprotected(), which bypasses the memory attribute
checks entirely: a plane writing HV_X64_MSR_REFERENCE_TSC had KVM
write the first 24 bytes of an arbitrary gfn of its choosing even
where that plane's own attributes forbid it, which is precisely the
protection a higher VTL counts on.  The sequence stream was also
seeded from the lowest enabled plane's page — guest-writable memory,
normally VTL0's — so VTL0 could stage S-1 on its own page (S being the
sequence live on VTL1's page), force a rewrite with an MSR write and
have a VTL1 reader which sampled S accept a torn body: guest-steered
time in a higher trust level.  And a failed update marked the whole
partition HV_TSC_PAGE_BROKEN, leaving every page at sequence zero for
the rest of the guest's life although the previous parameters still
described a valid clock.

Access each plane's page as the plane that owns it, with
kvm_plane_write_guest() and kvm_plane_read_guest() checking the given
plane's attributes (not exported to vendor modules: their only
callers live in kvm.ko, and the helpers bypass the usual "as the
vCPU's own plane" accessors).  Author the sequence host-side
(hv_tsc_sequence: monotonic, never read back from a page), invalidate
a page with a discrete zero-sequence write before touching its body,
invalidate synchronously on guest enables, make a guest MSR write
under TSC emulation take the recompute path, and invalidate any
enabled page whose content does not match the adopted parameters.

A refused access or a failed parameter computation now degrades only
the affected plane: its page is tracked in a per-plane
hv_tsc_page_broken bitmap and skipped until the plane writes its MSR
again or the next clock update retries, while the other planes' pages
are still maintained.  The pages are left invalid rather than
republishing the previous parameters: that fallback (TIME_REF_COUNT
answering from kvmclock) is always correct, and the previous
scale/offset would advertise a clock that had just stopped being
TSC-based.  Nothing marks the partition broken any more.

Assisted-by: LLM
[dwmw2: folded in the plane-aware accessor conversion, the host-side
 sequence authorship and invalidation protocol, the per-plane broken
 tracking replacing the partition-wide HV_TSC_PAGE_BROKEN restore
 logic, and the removal of the helper exports; the original keep-the-
 previous-values restore was dropped because parameters read back from
 the plane-0 page (tsc_ref_from_guest) written to a higher plane's
 page with hypervisor authority would let VTL0 forge its clock.]
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/include/asm/kvm_host.h |  14 ++-
 arch/x86/kvm/hyperv.c           | 216 ++++++++++++++++++++++++--------
 include/linux/kvm_host.h        |   6 +-
 virt/kvm/kvm_main.c             |  48 +++++--
 4 files changed, 218 insertions(+), 66 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index d913c68b18a7..153c076e91bc 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1233,8 +1233,6 @@ enum hv_tsc_page_status {
 	HV_TSC_PAGE_HOST_CHANGED,
 	/* TSC page was properly set up and is currently active  */
 	HV_TSC_PAGE_SET,
-	/* TSC page was set up with an inaccessible GPA */
-	HV_TSC_PAGE_BROKEN,
 };
 
 #ifdef CONFIG_KVM_HYPERV
@@ -1260,6 +1258,18 @@ struct kvm_hv {
 	 * it exposes is not: every enabled page gets the same contents.
 	 */
 	u64 hv_tsc_page_plane[KVM_MAX_PLANES];
+	/*
+	 * Planes whose reference TSC page could not be written, typically
+	 * because the plane's own memory attributes forbid the write.  Such
+	 * a page is left alone until the plane writes its MSR again.
+	 */
+	DECLARE_BITMAP(hv_tsc_page_broken, KVM_MAX_PLANES);
+	/*
+	 * The last sequence number published to the planes' pages.  Host-
+	 * authored and monotonic; never seeded from the (guest-writable)
+	 * pages themselves.
+	 */
+	u32 hv_tsc_sequence;
 	enum hv_tsc_page_status hv_tsc_page_status;
 
 	/* Hyper-v based guest crash (NT kernel bugcheck) parameters */
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 2cf73f72836c..e6ef0f93096e 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -1231,12 +1231,20 @@ static bool hv_tsc_page_gfn(struct kvm_hv *hv, int plane, u64 *gfn)
 	if (!(msr & HV_X64_MSR_TSC_REFERENCE_ENABLE))
 		return false;
 
+	if (test_bit(plane, hv->hv_tsc_page_broken))
+		return false;
+
 	*gfn = msr >> HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT;
 	return true;
 }
 
-/* Publish the current parameters to one plane's page: struct first, then sequence. */
-static int hv_publish_tsc_page(struct kvm *kvm, unsigned int plane, u64 gfn)
+/*
+ * Publish the current parameters to one plane's page: struct first, then
+ * sequence.  The write is subject to the owning plane's own memory
+ * attributes; a plane whose page cannot be written loses only that page,
+ * and keeps whatever the page last held, until it writes its MSR again.
+ */
+static void hv_publish_tsc_page(struct kvm *kvm, unsigned int plane, u64 gfn)
 {
 	struct kvm_hv *hv = to_kvm_hv(kvm);
 	struct ms_hyperv_tsc_page page = hv->tsc_ref;
@@ -1252,38 +1260,55 @@ static int hv_publish_tsc_page(struct kvm *kvm, unsigned int plane, u64 gfn)
 	if (plane && hv->tsc_ref_from_guest)
 		tsc_seq = 0;
 
+	/*
+	 * The page may hold a live sequence from an earlier enable, over
+	 * parameters that have since changed.  Invalidate it with a discrete
+	 * write first: the stores of one long copy are not ordered as seen
+	 * by another vCPU, so folding the zero sequence into the body write
+	 * would let a reader pair the old sequence with a half-written body.
+	 */
 	page.tsc_sequence = 0;
-	if (kvm_write_guest_unprotected(kvm, gfn_to_gpa(gfn), &page, sizeof(page)))
-		return -EFAULT;
+	if (kvm_plane_write_guest(kvm->planes[plane], gfn_to_gpa(gfn), &page,
+				  sizeof(page.tsc_sequence)))
+		goto broken;
+
+	/* Ensure the sequence is zero before the body changes. */
+	smp_wmb();
+	if (kvm_plane_write_guest(kvm->planes[plane], gfn_to_gpa(gfn), &page,
+				  sizeof(page)))
+		goto broken;
 
 	if (!tsc_seq)
-		return 0;
+		return;
 
 	/* The struct is complete before the sequence makes it valid. */
 	smp_wmb();
 	page.tsc_sequence = tsc_seq;
-	if (kvm_write_guest_unprotected(kvm, gfn_to_gpa(gfn), &page,
-					sizeof(page.tsc_sequence)))
-		return -EFAULT;
-	return 0;
+	if (kvm_plane_write_guest(kvm->planes[plane], gfn_to_gpa(gfn), &page,
+				  sizeof(page.tsc_sequence)))
+		goto broken;
+	return;
+
+broken:
+	set_bit(plane, hv->hv_tsc_page_broken);
 }
 
 /* The lowest plane with an enabled page; it owns the sequence number. */
-static bool hv_first_tsc_page_gfn(struct kvm *kvm, u64 *gfn)
+static int hv_first_tsc_page(struct kvm *kvm, u64 *gfn)
 {
 	struct kvm_hv *hv = to_kvm_hv(kvm);
 	int i;
 
 	for (i = 0; i < KVM_MAX_PLANES; i++) {
 		if (hv_tsc_page_gfn(hv, i, gfn))
-			return true;
+			return i;
 	}
 
-	return false;
+	return -1;
 }
 
 /* Write the first @len bytes of the reference page to every enabled plane. */
-static int hv_write_tsc_pages(struct kvm *kvm, size_t len)
+static void hv_write_tsc_pages(struct kvm *kvm, size_t len)
 {
 	struct kvm_hv *hv = to_kvm_hv(kvm);
 	u64 gfn;
@@ -1293,11 +1318,54 @@ static int hv_write_tsc_pages(struct kvm *kvm, size_t len)
 		if (!hv_tsc_page_gfn(hv, i, &gfn))
 			continue;
 
-		if (kvm_write_guest_unprotected(kvm, gfn_to_gpa(gfn), &hv->tsc_ref, len))
-			return -EFAULT;
+		/*
+		 * A plane whose own attributes refuse the write loses only
+		 * its page; the other planes' pages stay maintained.
+		 */
+		if (kvm_plane_write_guest(kvm->planes[i], gfn_to_gpa(gfn),
+					  &hv->tsc_ref, len))
+			set_bit(i, hv->hv_tsc_page_broken);
 	}
+}
 
-	return 0;
+/* Invalidate one plane's page: its readers fall back to TIME_REF_COUNT. */
+static void hv_invalidate_tsc_page(struct kvm *kvm, unsigned int plane, u64 gfn)
+{
+	struct kvm_hv *hv = to_kvm_hv(kvm);
+	u32 tsc_seq = 0;
+
+	if (kvm_plane_write_guest(kvm->planes[plane], gfn_to_gpa(gfn),
+				  &tsc_seq, sizeof(tsc_seq)))
+		set_bit(plane, hv->hv_tsc_page_broken);
+}
+
+/*
+ * Under guest TSC emulation the parameters come from plane @src_plane's
+ * page, which is guest-writable: they must not reach a higher plane's page
+ * with hypervisor authority.  A higher plane's page either already holds
+ * the same contents (every page gets the same stream, so a page carried
+ * across migration matches) or was never published — enabled from the host
+ * side while the update was pending — and would serve whatever its memory
+ * last held.  Invalidate the pages that do not match.
+ */
+static void hv_invalidate_mismatched_tsc_pages(struct kvm *kvm, int src_plane)
+{
+	struct kvm_hv *hv = to_kvm_hv(kvm);
+	struct ms_hyperv_tsc_page page;
+	u64 gfn;
+	int i;
+
+	for (i = src_plane + 1; i < KVM_MAX_PLANES; i++) {
+		if (!hv_tsc_page_gfn(hv, i, &gfn))
+			continue;
+
+		if (!kvm_plane_read_guest(kvm->planes[i], gfn_to_gpa(gfn),
+					  &page, sizeof(page)) &&
+		    !memcmp(&page, &hv->tsc_ref, sizeof(page)))
+			continue;
+
+		hv_invalidate_tsc_page(kvm, i, gfn);
+	}
 }
 
 void kvm_hv_setup_tsc_page(struct kvm *kvm,
@@ -1305,6 +1373,7 @@ void kvm_hv_setup_tsc_page(struct kvm *kvm,
 {
 	struct kvm_hv *hv = to_kvm_hv(kvm);
 	u32 tsc_seq;
+	int plane;
 	u64 gfn;
 
 	BUILD_BUG_ON(sizeof(tsc_seq) != sizeof(hv->tsc_ref.tsc_sequence));
@@ -1312,28 +1381,46 @@ void kvm_hv_setup_tsc_page(struct kvm *kvm,
 
 	guard(mutex)(&hv->hv_lock);
 
-	if (hv->hv_tsc_page_status == HV_TSC_PAGE_BROKEN ||
-	    hv->hv_tsc_page_status == HV_TSC_PAGE_SET ||
+	if (hv->hv_tsc_page_status == HV_TSC_PAGE_SET ||
 	    hv->hv_tsc_page_status == HV_TSC_PAGE_UNSET)
 		return;
 
-	if (!hv_first_tsc_page_gfn(kvm, &gfn))
-		return;
+	if (tsc_page_update_unsafe(hv)) {
+		/*
+		 * The guest has opted for TSC emulation after migration: the
+		 * lowest enabled plane's page carries the parameters to keep.
+		 * The reads are subject to the owning plane's read attribute;
+		 * a plane whose page cannot be read loses that page and the
+		 * next enabled plane takes over.
+		 */
+		for (;;) {
+			plane = hv_first_tsc_page(kvm, &gfn);
+			if (plane < 0)
+				return;
+
+			if (likely(!kvm_plane_read_guest(kvm->planes[plane],
+							 gfn_to_gpa(gfn), &tsc_seq,
+							 sizeof(tsc_seq))))
+				break;
 
-	/*
-	 * Because the TSC parameters only vary when there is a
-	 * change in the master clock, do not bother with caching.
-	 */
-	if (unlikely(kvm_read_guest(kvm, gfn_to_gpa(gfn),
-				    &tsc_seq, sizeof(tsc_seq))))
-		goto out_err;
+			set_bit(plane, hv->hv_tsc_page_broken);
+		}
 
-	if (tsc_seq && tsc_page_update_unsafe(hv)) {
-		if (kvm_read_guest(kvm, gfn_to_gpa(gfn), &hv->tsc_ref, sizeof(hv->tsc_ref)))
-			goto out_err;
+		if (tsc_seq) {
+			if (kvm_plane_read_guest(kvm->planes[plane],
+						 gfn_to_gpa(gfn), &hv->tsc_ref,
+						 sizeof(hv->tsc_ref))) {
+				/* Status is unchanged: the next update retries. */
+				set_bit(plane, hv->hv_tsc_page_broken);
+				return;
+			}
 
-		hv->tsc_ref_from_guest = true;
-		hv->hv_tsc_page_status = HV_TSC_PAGE_SET;
+			hv->tsc_ref_from_guest = true;
+			hv->hv_tsc_page_status = HV_TSC_PAGE_SET;
+			hv_invalidate_mismatched_tsc_pages(kvm, plane);
+			return;
+		}
+	} else if (hv_first_tsc_page(kvm, &gfn) < 0) {
 		return;
 	}
 
@@ -1342,37 +1429,43 @@ void kvm_hv_setup_tsc_page(struct kvm *kvm,
 	 * guest to use the time reference count MSR.
 	 */
 	hv->tsc_ref.tsc_sequence = 0;
-	if (hv_write_tsc_pages(kvm, sizeof(hv->tsc_ref.tsc_sequence)))
-		goto out_err;
+	hv_write_tsc_pages(kvm, sizeof(hv->tsc_ref.tsc_sequence));
 
+	/*
+	 * If the master clock is not currently TSC-based, the pages cannot
+	 * describe it: leave them invalid, with their readers on the
+	 * TIME_REF_COUNT fallback, and retry on the next clock update.
+	 * Republishing the previous parameters would advertise a clock that
+	 * no longer matches the one TIME_REF_COUNT answers from.
+	 */
 	if (!compute_tsc_page_parameters(hv_clock, &hv->tsc_ref))
-		goto out_err;
+		return;
 
 	/* Ensure sequence is zero before writing the rest of the struct.  */
 	smp_wmb();
-	if (hv_write_tsc_pages(kvm, sizeof(hv->tsc_ref)))
-		goto out_err;
+	hv_write_tsc_pages(kvm, sizeof(hv->tsc_ref));
 
 	/*
 	 * Now switch to the TSC page mechanism by writing the sequence.
+	 * The sequence is host-authored and monotonic, never read back from
+	 * the pages themselves: they are guest-writable, and a lower plane
+	 * which could steer the next value could stage it on a higher
+	 * plane's page in advance, and have that plane's seqlock accept a
+	 * torn read across this rewrite.
 	 */
-	tsc_seq++;
+	tsc_seq = hv->hv_tsc_sequence + 1;
 	if (tsc_seq == 0xFFFFFFFF || tsc_seq == 0)
 		tsc_seq = 1;
+	hv->hv_tsc_sequence = tsc_seq;
 
 	/* Write the struct entirely before the non-zero sequence.  */
 	smp_wmb();
 
 	hv->tsc_ref.tsc_sequence = tsc_seq;
-	if (hv_write_tsc_pages(kvm, sizeof(hv->tsc_ref.tsc_sequence)))
-		goto out_err;
+	hv_write_tsc_pages(kvm, sizeof(hv->tsc_ref.tsc_sequence));
 
 	hv->tsc_ref_from_guest = false;
 	hv->hv_tsc_page_status = HV_TSC_PAGE_SET;
-	return;
-
-out_err:
-	hv->hv_tsc_page_status = HV_TSC_PAGE_BROKEN;
 }
 
 void kvm_hv_request_tsc_page_update(struct kvm *kvm)
@@ -1575,20 +1668,43 @@ static int kvm_hv_set_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data,
 	case HV_X64_MSR_REFERENCE_TSC: {
 		u64 gfn;
 
-		/* The parameters are valid already: publish them to this page only. */
+		/* A newly written MSR is a fresh attempt at the page. */
+		clear_bit(vcpu->plane_level, hv->hv_tsc_page_broken);
+
+		hv->hv_tsc_page_plane[vcpu->plane_level] = data;
+
+		/*
+		 * The parameters are valid already: publish them to this page
+		 * only.  Not under guest TSC emulation: those parameters are
+		 * the frozen migration-time snapshot, and a guest MSR write
+		 * is the trigger for computing fresh ones (GUEST_CHANGED
+		 * below), as it was before the pages were tracked per plane.
+		 */
 		if (!host && hv->hv_tsc_page_status == HV_TSC_PAGE_SET &&
+		    !hv->tsc_ref_from_guest &&
 		    (data & HV_X64_MSR_TSC_REFERENCE_ENABLE)) {
 			/* The caller holds hv_lock. */
-			hv->hv_tsc_page_plane[vcpu->plane_level] = data;
 			gfn = data >> HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT;
-			if (hv_publish_tsc_page(kvm, vcpu->plane_level, gfn))
-				hv->hv_tsc_page_status = HV_TSC_PAGE_BROKEN;
+			hv_publish_tsc_page(kvm, vcpu->plane_level, gfn);
 			break;
 		}
 
-		hv->hv_tsc_page_plane[vcpu->plane_level] = data;
+		/*
+		 * The publish below is deferred to a plane-0 vCPU processing
+		 * the masterclock update; until then the page holds whatever
+		 * its memory last held — possibly a forged clock a lower
+		 * plane staged there.  Invalidate it now, so this plane reads
+		 * TIME_REF_COUNT until the real parameters arrive.  Not for
+		 * host writes: a page restored across migration keeps its
+		 * contents for the TSC-emulation readback.
+		 */
+		if (!host && (data & HV_X64_MSR_TSC_REFERENCE_ENABLE)) {
+			gfn = data >> HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT;
+			hv_invalidate_tsc_page(kvm, vcpu->plane_level, gfn);
+		}
+
 		/* Another plane's page may still be enabled: keep it updated. */
-		if (hv_first_tsc_page_gfn(kvm, &gfn)) {
+		if (hv_first_tsc_page(kvm, &gfn) >= 0) {
 			if (!host)
 				hv->hv_tsc_page_status = HV_TSC_PAGE_GUEST_CHANGED;
 			else
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index beae4f2a982a..bc576889c6f0 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1551,8 +1551,10 @@ int kvm_write_guest_page(struct kvm *kvm, gfn_t gfn, const void *data,
 			 int offset, int len);
 int kvm_write_guest(struct kvm *kvm, gpa_t gpa, const void *data,
 		    unsigned long len);
-int kvm_write_guest_unprotected(struct kvm *kvm, gpa_t gpa, const void *data,
-				unsigned long len);
+int kvm_plane_write_guest(struct kvm_plane *plane, gpa_t gpa, const void *data,
+			  unsigned long len);
+int kvm_plane_read_guest(struct kvm_plane *plane, gpa_t gpa, void *data,
+			 unsigned long len);
 int kvm_write_guest_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 			   void *data, unsigned long len);
 int kvm_write_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 72fe64e5d4a1..1c70bb70500c 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -3832,24 +3832,49 @@ int kvm_write_guest(struct kvm *kvm, gpa_t gpa, const void *data,
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_write_guest);
 
-/* Write a page KVM maintains for the guest; plane protections do not apply. */
-int kvm_write_guest_unprotected(struct kvm *kvm, gpa_t gpa, const void *data,
-				unsigned long len)
+/*
+ * Write a page KVM maintains for the guest on behalf of plane @plane,
+ * honouring that plane's memory attributes rather than plane 0's.
+ */
+int kvm_plane_write_guest(struct kvm_plane *plane, gpa_t gpa, const void *data,
+			  unsigned long len)
 {
+	struct kvm *kvm = plane->kvm;
 	gfn_t gfn = gpa >> PAGE_SHIFT;
 	int seg;
 	int offset = offset_in_page(gpa);
+	int ret;
 
 	while ((seg = next_segment(len, offset)) != 0) {
-		struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
-		unsigned long addr;
+		ret = __kvm_write_guest_page(plane, kvm,
+					     gfn_to_memslot(kvm, gfn), gfn,
+					     data, offset, seg);
+		if (ret < 0)
+			return ret;
+		offset = 0;
+		len -= seg;
+		data += seg;
+		++gfn;
+	}
+	return 0;
+}
 
-		addr = gfn_to_hva_many(slot, gfn, NULL);
-		if (kvm_is_error_hva(addr))
-			return -EFAULT;
-		if (__copy_to_user((void __user *)addr + offset, data, seg))
-			return -EFAULT;
-		mark_page_dirty_in_slot(kvm, slot, gfn);
+/* Read counterpart of kvm_plane_write_guest(). */
+int kvm_plane_read_guest(struct kvm_plane *plane, gpa_t gpa, void *data,
+			 unsigned long len)
+{
+	struct kvm *kvm = plane->kvm;
+	gfn_t gfn = gpa >> PAGE_SHIFT;
+	int seg;
+	int offset = offset_in_page(gpa);
+	int ret;
+
+	while ((seg = next_segment(len, offset)) != 0) {
+		ret = __kvm_read_guest_page(plane, gfn_to_memslot(kvm, gfn),
+					    gfn, data, offset, seg,
+					    KVM_MEMORY_ATTRIBUTE_NR);
+		if (ret < 0)
+			return ret;
 		offset = 0;
 		len -= seg;
 		data += seg;
@@ -3857,7 +3882,6 @@ int kvm_write_guest_unprotected(struct kvm *kvm, gpa_t gpa, const void *data,
 	}
 	return 0;
 }
-EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_write_guest_unprotected);
 
 int kvm_vcpu_write_guest(struct kvm_vcpu *vcpu, gpa_t gpa, const void *data,
 		         unsigned long len)
-- 
2.50.1

[RFC PATCH 40/55] KVM: x86: Implement Hyper-V's vCPU suspended state

[David Woodhouse]

From: Nikolas Wipper <nikwip@amazon.de>

Introduce a suspension state for Hyper-V enlightened vCPUs. Microsoft's
"Hypervisor Top Level Functional Specification" (TLFS) introduces this
state as a "vCPU that is stopped on a instruction guest boundary, either
explicitly or implicitly due to an intercept". The only documented
explicit suspension is caused in response to a TLB Flush hypercall, which
will use the state switching API in subsequent patches.

Each Hyper-V vCPU now has a 'suspended' boolean field, which is checked
before entering the guest. When set, it forces the vCPU to block. Once in
that state, the vCPU ignores any events. The vCPU is unsuspended by
clearing 'suspend' and issuing a request to force the vCPU out of sleep.

Suspensions are issued as a mechanism to halt execution until state change
is observed on a remote vCPU. Hyper-V vCPUs track this with 'waiting_on',
which holds the 'vcpu_id' of the remote vCPU that forced the vCPU to enter
the suspended state. It's the remote vCPU's responsibility to wake up the
suspended vCPUs when ready. 'waiting_on' ensures the remote vCPU can
selectively unsuspend vCPUs that blocked on its behalf while leaving other
suspended vCPUs undisturbed. One vCPU can only be suspended due to a
single remote vCPU, but different vCPUs can be suspended on behalf of
different or the same remote vCPU(s). The guest is responsible for
avoiding circular dependencies between suspended vCPUs.

Callers of the suspend API are responsible for ensuring that suspend and
unsuspend aren't called in parallel while targeting the same pair of
vCPUs. Otherwise kvm_hv_vcpu_{un}suspend_tlb_flush() ensure 'waiting_on'
and 'suspended' are updated and accessed in the correct order. This, for
example, avoids races where the unsuspended vCPU re-suspends before
kvm_hv_vcpu_unsuspend_tlb_flush() is done updating 'waiting_on'.

Signed-off-by: Nikolas Wipper <nikwip@amazon.de>

(cherry picked from the posted series, re-done by hand)
[hmushi: struct kvm_vcpu_hv now lives in arch/x86/kvm/hyperv.h;
 bitmap_zero(vcpu_mask) added (the original used an uninitialised stack
 bitmap)]
Signed-off-by: Mushahid Hussain <hmushi@amazon.co.uk>
(re-done by hand from the reference VSM implementation)
[dwmw2: struct kvm_vcpu_hv is in arch/x86/include/asm/kvm_host.h on
 this branch.  Unsuspend walks the suspended waiters' own plane and
 kicks each one directly: kvm_make_vcpus_request_mask() resolves
 indices against plane 0, which is the wrong twin for a waiter in a
 higher plane.]
Assisted-by: LLM
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/include/asm/kvm_host.h |  3 +++
 arch/x86/kvm/hyperv.c           | 28 ++++++++++++++++++++++++++++
 arch/x86/kvm/hyperv.h           | 15 +++++++++++++++
 arch/x86/kvm/x86.c              |  2 +-
 4 files changed, 47 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 153c076e91bc..6d864aa71b5b 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -761,6 +761,9 @@ struct kvm_vcpu_hv {
 	u64 vtl_entry_rax;
 	u64 vtl_entry_rcx;
 	bool vtl_entry_regs_valid;
+
+	bool suspended;
+	int waiting_on;
 };
 
 struct kvm_hypervisor_cpuid {
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index e6ef0f93096e..0bc9435ea3b9 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -1040,6 +1040,7 @@ int kvm_hv_vcpu_init(struct kvm_vcpu *vcpu)
 
 	vcpu->arch.hyperv = hv_vcpu;
 	hv_vcpu->vcpu = vcpu;
+	hv_vcpu->waiting_on = -1;
 
 	synic_init(&hv_vcpu->synic);
 
@@ -3764,3 +3765,30 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid,
 
 	return 0;
 }
+
+void kvm_hv_vcpu_suspend_tlb_flush(struct kvm_vcpu *vcpu, int vcpu_id)
+{
+	/* waiting_on's store should happen before suspended's */
+	WRITE_ONCE(vcpu->arch.hyperv->waiting_on, vcpu_id);
+	WRITE_ONCE(vcpu->arch.hyperv->suspended, true);
+}
+
+void kvm_hv_vcpu_unsuspend_tlb_flush(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vcpu_hv *vcpu_hv;
+	struct kvm_vcpu *v;
+	unsigned long i;
+
+	plane_for_each_vcpu(i, v, vcpu->plane) {
+		vcpu_hv = to_hv_vcpu(v);
+
+		if (vcpu_hv && kvm_hv_vcpu_suspended(v) &&
+		    READ_ONCE(vcpu_hv->waiting_on) == vcpu->vcpu_id) {
+			/* waiting_on's store should happen before suspended's */
+			WRITE_ONCE(vcpu_hv->waiting_on, -1);
+			WRITE_ONCE(vcpu_hv->suspended, false);
+			kvm_make_request(KVM_REQ_EVENT, v);
+			kvm_vcpu_kick(v);
+		}
+	}
+}
diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h
index 0b81ba482201..5ba2bb2e649b 100644
--- a/arch/x86/kvm/hyperv.h
+++ b/arch/x86/kvm/hyperv.h
@@ -228,6 +228,15 @@ static inline bool guest_hv_cpuid_has_l2_tlb_flush(struct kvm_vcpu *vcpu)
 		(hv_vcpu->cpuid_cache.nested_eax & HV_X64_NESTED_DIRECT_FLUSH);
 }
 
+void kvm_hv_vcpu_suspend_tlb_flush(struct kvm_vcpu *vcpu, int vcpu_id);
+void kvm_hv_vcpu_unsuspend_tlb_flush(struct kvm_vcpu *vcpu);
+
+static inline bool kvm_hv_vcpu_suspended(struct kvm_vcpu *vcpu)
+{
+	return vcpu->arch.hyperv_enabled &&
+	       READ_ONCE(vcpu->arch.hyperv->suspended);
+}
+
 static inline bool kvm_hv_is_tlb_flush_hcall(struct kvm_vcpu *vcpu)
 {
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
@@ -309,6 +318,12 @@ static inline bool kvm_hv_has_stimer_pending(struct kvm_vcpu *vcpu)
 {
 	return false;
 }
+static inline void kvm_hv_vcpu_suspend_tlb_flush(struct kvm_vcpu *vcpu, int vcpu_id) {}
+static inline void kvm_hv_vcpu_unsuspend_tlb_flush(struct kvm_vcpu *vcpu) {}
+static inline bool kvm_hv_vcpu_suspended(struct kvm_vcpu *vcpu)
+{
+	return false;
+}
 static inline int kvm_hv_verify_vp_assist(struct kvm_vcpu *vcpu)
 {
 	return 0;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index ba468f7c4d4a..d82b457e6714 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -12182,7 +12182,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 static bool kvm_vcpu_running(struct kvm_vcpu *vcpu)
 {
 	return (vcpu->arch.mp_state == KVM_MP_STATE_RUNNABLE &&
-		!vcpu->arch.apf.halted);
+		!vcpu->arch.apf.halted && !kvm_hv_vcpu_suspended(vcpu));
 }
 
 bool kvm_vcpu_has_events(struct kvm_vcpu *vcpu)
-- 
2.50.1

[RFC PATCH 41/55] KVM: x86: hyperv: Refuse a synthetic IPI to a higher VTL

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

HvCallSendSyntheticClusterIpi(Ex) honours the target VTL in its input
but does not check it against the caller's VTL. A guest running in a
lower VTL can therefore assert any fixed interrupt in a higher VTL on
any VP, which crosses the boundary that VTLs exist to enforce.

Refuse a target VTL above the caller's with HV_STATUS_ACCESS_DENIED.
A VTL may still interrupt its own level and lower ones.  The old
upper-bound check on the target plane goes away: a target at or below
the caller's level is below KVM_MAX_PLANES by construction.

Matches the equivalent fix in the reference VSM implementation.  The
production Nitro Hypervisor permits the injection; the conformance
suite probes for either behaviour.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/hyperv.c | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 0bc9435ea3b9..e3ac840c8e5e 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -2710,13 +2710,21 @@ static int kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 	if ((vector < HV_IPI_LOW_VECTOR) || (vector > HV_IPI_HIGH_VECTOR))
 		return HV_STATUS_INVALID_HYPERCALL_INPUT;
 
-	/* TLFS: in_vtl bit 4 selects an explicit target VTL. */
-	if (in_vtl & 0x10)
+	/*
+	 * TLFS: in_vtl bit 4 selects an explicit target VTL.  A VTL may
+	 * interrupt its own level and lower ones, never a higher one
+	 * (TLFS 15.9, "VTL Interrupt Management": each VTL sends and
+	 * receives interrupts without interference from a less secure
+	 * VTL).  The reference host omits this check; the suite's
+	 * SendIpi test probes for both behaviours.
+	 */
+	if (in_vtl & 0x10) {
 		target_plane = in_vtl & 0xf;
-	else
+		if (target_plane > vcpu->plane_level)
+			return HV_STATUS_ACCESS_DENIED;
+	} else {
 		target_plane = vcpu->plane_level;
-	if (target_plane >= KVM_MAX_PLANES)
-		return HV_STATUS_INVALID_HYPERCALL_INPUT;
+	}
 
 	if (all_cpus)
 		kvm_hv_send_ipi_to_many(kvm, vector, NULL, 0, target_plane);
-- 
2.50.1

[RFC PATCH 42/55] KVM: x86: Gate forced VTL entry on interrupt deliverability

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

A fixed vector posted to a higher plane forces a VTL entry even when
the higher VTL's TPR masks it, so the secure kernel is entered with
reason INTERRUPT and nothing to deliver.  The TLFS (15.9, "VTL
Interrupt Management") holds such an interrupt pending without
switching: only a deliverable interrupt preempts the lower VTL.

Gate the forced-entry evaluation on the higher plane's APIC reporting
a deliverable vector under its PPR.  A masked vector keeps its posted
marker instead of consuming the single entry attempt, so the entry
fires if a later VTL round trip leaves the vector deliverable; the
higher plane still consumes its own marker when it runs.  Each marker
is consumed only by its own approved clause, in both the userspace
exit and the in-kernel refusal paths: an exit triggered by extint, a
stimer or the LAPIC timer while the posted vector is still masked
leaves the APIC marker armed.

NMIs move to a marker of their own, hv_nmi_event_pending, and keep
forcing entry unconditionally: the NMI is folded back to plane 0 at
acceptance, so the higher plane's APIC carries nothing the PPR gate
could see.  The marker is armed wherever NMI state is posted to a
non-zero plane (kvm_inject_nmi() and the vCPU-events ioctl, which is
allowlisted on plane fds), and the level-triggered NMI state
(KVM_REQ_NMI, nmi_queued, nmi_pending) does not feed the gate: a
posting earns exactly one wakeup, like the APIC event path, instead
of keeping the gate permanently true for a parked higher plane.

The in-kernel switch consumes the markers on a refused entry attempt
too.  The only refusal left on that path is a failing VTL-control
write (blocked planes are filtered in kvm_higher_plane_with_event()),
which stays failed until an overlay or memory-attribute change; the
event remains latched in the plane's own pending state for its next
entry, rather than repeating the failing guest write on every exit of
the lower plane and keeping the vCPU from ever blocking.

Matches the deliverability gating of the reference VSM
implementation (its kvm_plane_has_interrupt); the production Nitro
Hypervisor remains PPR-blind, and the conformance suite's TPR-masked
test probes for either behaviour.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/include/asm/kvm_host.h |  21 +++++--
 arch/x86/kvm/lapic.c            |   2 +-
 arch/x86/kvm/x86.c              | 104 ++++++++++++++++++++++++--------
 3 files changed, 96 insertions(+), 31 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 6d864aa71b5b..6d3bae46ecab 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -861,14 +861,25 @@ void kvm_arch_vcpu_common_destroy(struct kvm_vcpu_common *common);
 
 struct kvm_vcpu_arch {
 	/*
-	 * Hyper-V VSM: an APIC event (fixed vector or NMI) was delivered to
-	 * this plane while it was suspended.  Set at delivery, cleared when
-	 * the plane next runs; a lower plane's run loop uses it to force a
-	 * VTL entry even when the vector is currently masked by this
-	 * plane's TPR (matching the reference host's apic_pending_vtls).
+	 * Hyper-V VSM: a fixed APIC vector was delivered to this plane
+	 * while it was suspended.  Set at delivery, cleared when the plane
+	 * next runs; a lower plane's run loop uses it to force a VTL entry
+	 * once the vector is deliverable under this plane's PPR (TLFS 15.9:
+	 * an interrupt the higher VTL's priority masks is held pending and
+	 * the VTL does not switch).  While masked, the marker is held, so
+	 * the entry fires when a VTL round trip leaves the vector
+	 * deliverable.
 	 */
 	bool hv_apic_event_pending;
 
+	/*
+	 * Hyper-V VSM: an NMI was aimed at this (higher) plane.  Forces a
+	 * VTL entry unconditionally: the NMI itself is folded back to
+	 * plane 0 (reference host behaviour), so the higher plane's APIC
+	 * carries no trace of it and the PPR gate above cannot apply.
+	 */
+	bool hv_nmi_event_pending;
+
 	/*
 	 * rip and regs accesses must go through
 	 * kvm_{register,rip}_{read,write} functions.
diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index 32cace3c89f4..03894f302985 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -1486,7 +1486,7 @@ static int __apic_accept_irq(struct kvm_lapic *apic, int delivery_mode,
 			struct kvm_vcpu *twin;
 			int p;
 
-			WRITE_ONCE(vcpu->arch.hv_apic_event_pending, true);
+			WRITE_ONCE(vcpu->arch.hv_nmi_event_pending, true);
 			if (vcpu->common->vcpus[0])
 				kvm_inject_nmi(vcpu->common->vcpus[0]);
 			for (p = 0; p < vcpu->plane_level; p++) {
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index d82b457e6714..89203909db36 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1071,6 +1071,13 @@ void kvm_inject_nmi(struct kvm_vcpu *vcpu)
 {
 	atomic_inc(&vcpu->arch.nmi_queued);
 	kvm_make_request(KVM_REQ_NMI, vcpu);
+	/*
+	 * NMI state posted to a suspended higher plane earns one entry
+	 * attempt through the single-shot marker, like the APIC event
+	 * path; the queued NMI itself is delivered when the plane runs.
+	 */
+	if (vcpu->plane_level)
+		WRITE_ONCE(vcpu->arch.hv_nmi_event_pending, true);
 }
 
 void kvm_queue_exception_e(struct kvm_vcpu *vcpu, unsigned nr, u32 error_code)
@@ -5809,8 +5816,13 @@ static int kvm_vcpu_ioctl_x86_set_vcpu_events(struct kvm_vcpu *vcpu,
 	if (events->flags & KVM_VCPUEVENT_VALID_NMI_PENDING) {
 		vcpu->arch.nmi_pending = 0;
 		atomic_set(&vcpu->arch.nmi_queued, events->nmi.pending);
-		if (events->nmi.pending)
+		if (events->nmi.pending) {
 			kvm_make_request(KVM_REQ_NMI, vcpu);
+			/* One entry attempt for a parked higher plane. */
+			if (vcpu->plane_level)
+				WRITE_ONCE(vcpu->arch.hv_nmi_event_pending,
+					   true);
+		}
 	}
 	kvm_x86_call(set_nmi_mask)(vcpu, events->nmi.masked);
 
@@ -11286,25 +11298,36 @@ static struct kvm_vcpu *kvm_higher_plane_with_event(struct kvm_vcpu *vcpu)
 #endif
 		/*
 		 * Forced entry is edge-triggered on the posting of an
-		 * event, not level-triggered on the target's IRR, matching
-		 * the reference host's apic_pending_vtls bitmap: one entry
-		 * attempt per posted interrupt, via the hv_apic_event_pending
-		 * marker __apic_accept_irq sets.  A vector that stays
+		 * event, not level-triggered on the target's IRR: one
+		 * entry attempt per posted interrupt, via the
+		 * hv_apic_event_pending marker __apic_accept_irq sets.
+		 * The attempt is gated on deliverability (TLFS 15.9: an
+		 * interrupt the higher VTL's priority mask blocks is held
+		 * pending and the VTL does not switch).  A masked vector
+		 * holds its marker, so the entry fires if a VTL round
+		 * trip leaves the vector deliverable; the plane consumes
+		 * its own marker when it runs.  A vector that stays
 		 * latched after a refused attempt (no VP assist page yet)
-		 * or that was masked by the plane's TPR waits for the next
-		 * cooperative entry or the next posted event; scanning the
-		 * IRR here instead would replay it the moment the assist
-		 * page is re-enabled.  A pending synthetic-timer request
-		 * counts as well: it can only be processed (and its
-		 * interrupt delivered) once the higher plane runs, so it
-		 * would otherwise never fire while the plane is suspended.
+		 * waits for the next cooperative entry or the next posted
+		 * event; scanning the IRR here instead would replay it
+		 * the moment the assist page is re-enabled.  NMIs force
+		 * entry ungated through their own single-shot marker, set
+		 * wherever NMI state is posted to a higher plane (the
+		 * higher APIC carries nothing to gate on; the NMI itself
+		 * is folded back to plane 0 on the APIC path).  A pending
+		 * synthetic-timer request counts as well: it can only be
+		 * processed (and its interrupt delivered) once the higher
+		 * plane runs, so it would otherwise never fire while the
+		 * plane is suspended.
+		 *
+		 * The reference host forces entry even for a TPR-masked
+		 * vector; the suite's TPR test probes for both behaviours.
 		 */
-		if (READ_ONCE(higher->arch.hv_apic_event_pending) ||
+		if ((READ_ONCE(higher->arch.hv_apic_event_pending) &&
+		     higher->arch.apic && kvm_apic_has_interrupt(higher) >= 0) ||
+		    READ_ONCE(higher->arch.hv_nmi_event_pending) ||
 		    kvm_cpu_has_extint(higher) ||
 		    kvm_test_request(KVM_REQ_HV_STIMER, higher) ||
-		    kvm_test_request(KVM_REQ_NMI, higher) ||
-		    atomic_read(&higher->arch.nmi_queued) ||
-		    higher->arch.nmi_pending ||
 		    (higher->arch.apic &&
 		     atomic_read(&higher->arch.apic->lapic_timer.pending)))
 			return higher;
@@ -11333,9 +11356,17 @@ static bool kvm_hv_vtl_interrupt_exit(struct kvm_vcpu *vcpu)
 	 * Single-shot, as in the reference host: if userspace refuses
 	 * the entry (e.g. the target VTL has no VP assist page yet),
 	 * the event waits in the target's IRR for a cooperative entry
-	 * rather than re-triggering an exit storm.
-	 */
-	WRITE_ONCE(higher->arch.hv_apic_event_pending, false);
+	 * rather than re-triggering an exit storm.  Each marker is
+	 * consumed only by its own approved attempt: when this exit was
+	 * triggered by another clause (extint, stimer, timer) while the
+	 * posted vector is still masked, the APIC marker stays armed so
+	 * a VTL round trip that unmasks the vector can retrigger the
+	 * entry.
+	 */
+	if (READ_ONCE(higher->arch.hv_apic_event_pending) &&
+	    higher->arch.apic && kvm_apic_has_interrupt(higher) >= 0)
+		WRITE_ONCE(higher->arch.hv_apic_event_pending, false);
+	WRITE_ONCE(higher->arch.hv_nmi_event_pending, false);
 
 	vcpu->run->exit_reason = KVM_EXIT_HYPERV;
 	vcpu->run->hyperv.type = KVM_EXIT_HYPERV_VTL_INTERRUPT;
@@ -11874,9 +11905,32 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 	if (READ_ONCE(to_kvm_hv(vcpu->kvm)->hv_inkernel_vtl_switch)) {
 		struct kvm_vcpu *higher = kvm_higher_plane_with_event(vcpu);
 
-		if (higher && kvm_hv_vtl_interrupt_entry(vcpu, higher)) {
-			kvm_make_request(KVM_REQ_EVENT, higher);
-			kvm_make_request(KVM_REQ_PLANE_RESCHED, vcpu);
+		if (higher) {
+			if (kvm_hv_vtl_interrupt_entry(vcpu, higher)) {
+				kvm_make_request(KVM_REQ_EVENT, higher);
+				kvm_make_request(KVM_REQ_PLANE_RESCHED, vcpu);
+			} else {
+				/*
+				 * Single-shot in in-kernel mode too: the
+				 * refusal (a failing control-area write;
+				 * blocked planes are filtered in
+				 * kvm_higher_plane_with_event()) cannot
+				 * succeed until external action, so consume
+				 * the markers rather than repeating the
+				 * failing guest write on every exit.  The
+				 * event stays latched in the plane's own
+				 * pending state for its next entry.  A
+				 * still-masked vector keeps its marker; only
+				 * its own approved attempt consumes it.
+				 */
+				if (READ_ONCE(higher->arch.hv_apic_event_pending) &&
+				    higher->arch.apic &&
+				    kvm_apic_has_interrupt(higher) >= 0)
+					WRITE_ONCE(higher->arch.hv_apic_event_pending,
+						   false);
+				WRITE_ONCE(higher->arch.hv_nmi_event_pending,
+					   false);
+			}
 		}
 	}
 #endif
@@ -11889,12 +11943,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 
 	/*
 	 * This plane is running and about to evaluate its events: its
-	 * pended-while-suspended marker is consumed.  The clear must come
+	 * pended-while-suspended markers are consumed.  The clear must come
 	 * after the VTL-exit and plane-reschedule bail-outs above: a run
-	 * aborted there never evaluates the pended event, and nothing
-	 * re-arms the marker for a TPR-masked vector.
+	 * aborted there never evaluates the pended event.
 	 */
 	WRITE_ONCE(vcpu->arch.hv_apic_event_pending, false);
+	WRITE_ONCE(vcpu->arch.hv_nmi_event_pending, false);
 
 	if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win ||
 	    kvm_xen_has_interrupt(vcpu)) {
-- 
2.50.1

[RFC PATCH 43/55] KVM: x86: Derive runtime CPUID bits from the executing plane

[David Woodhouse]

From: Mushahid Hussain <hmushi@amazon.co.uk>

Some CPUID bits follow registers of the vCPU: CPUID.1:ECX.OSXSAVE
follows CR4.OSXSAVE, CPUID.7:ECX.OSPKE follows CR4.PKE, CPUID.1:EDX.APIC
follows the APIC enable, CPUID.1:ECX.MWAIT follows MISC_ENABLE, and the
XSAVE area sizes in leaf 0xD follow XCR0 and XSS. KVM keeps the CPUID
table of a vCPU in the state its planes share, and refreshes those bits
into the table from the state of whichever vCPU runs when the table is
dirty. The planes of a vCPU have their own control registers, so after
one plane sets CR4.OSXSAVE, CPUID on every plane of that vCPU reports
OSXSAVE until a plane with it clear refreshes the table, and the other
way round. The architecture requires CPUID to describe the processor
that executes it, and for a plane that is the plane's own registers.

A Windows guest with Virtual Secure Mode shows the leak. Its secure
kernel runs on a plane whose CR4.OSXSAVE is clear while the normal
kernel on the other plane has set it. The secure kernel executes the
usual probe, CPUID.1 followed by a test of OSXSAVE, reads the bit set
and executes XGETBV, which raises #UD because its own CR4 does not
permit it.

Compute the bits that follow vCPU registers when CPUID is emulated,
from the registers of the vCPU that executes it, and no longer refresh
the shared table on that path. The table keeps the view of plane 0 for
KVM_GET_CPUID2 and KVM_SET_CPUID2, which act on the plane 0 vCPU, and
their refresh stays as it is.

Assisted-by: LLM

(re-done by hand from the reference VSM implementation)
[dwmw2: dropped the plane selftest hunk; this branch has no
 plane_x86_test.c.  The per-VTL OSXSAVE contract is pinned by the VSM
 conformance suite instead.]
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 Documentation/virt/kvm/api.rst |   7 +++
 arch/x86/kvm/cpuid.c           | 108 +++++++++++++++++++++++----------
 arch/x86/kvm/cpuid.h           |   5 +-
 3 files changed, 87 insertions(+), 33 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 215547802377..d75987c1d325 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -370,6 +370,13 @@ cpu's hardware control block.
 VCPUs for non-zero planes can only be created if the VCPU the same index has
 already been created for plane zero.
 
+The planes of a vCPU share its CPUID model, set through the plane 0 fd with
+KVM_SET_CPUID2.  The CPUID bits that follow a vCPU's own registers (OSXSAVE
+from CR4.OSXSAVE, OSPKE from CR4.PKE, the APIC enable, MWAIT from
+MISC_ENABLE, and the XSAVE area sizes from XCR0 and XSS) reflect the plane
+that executes CPUID, since every plane has these registers of its own.
+KVM_GET_CPUID2 returns them as plane 0 sees them.
+
 4.8 KVM_GET_DIRTY_LOG
 ---------------------
 
diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index f744e7845843..a71888749a29 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -171,6 +171,9 @@ static int kvm_check_cpuid(struct kvm_vcpu *vcpu)
 
 static u32 kvm_apply_cpuid_pv_features_quirk(struct kvm_vcpu *vcpu);
 static void kvm_update_cpuid_runtime(struct kvm_vcpu *vcpu);
+static void kvm_cpuid_apply_runtime(struct kvm_vcpu *vcpu, u32 function,
+				    u32 index, u32 *eax, u32 *ebx, u32 *ecx,
+				    u32 *edx);
 
 /* Check whether the supplied CPUID data is equal to what is already set for the vCPU. */
 static int kvm_cpuid_check_equal(struct kvm_vcpu *vcpu, struct kvm_cpuid_entry2 *e2,
@@ -278,50 +281,51 @@ static u64 cpuid_get_supported_xss(struct kvm_vcpu *vcpu)
 	return (best->ecx | ((u64)best->edx << 32)) & kvm_caps.supported_xss;
 }
 
-static __always_inline void kvm_update_feature_runtime(struct kvm_vcpu *vcpu,
-						       struct kvm_cpuid_entry2 *entry,
-						       unsigned int x86_feature,
-						       bool has_feature)
-{
-	cpuid_entry_change(entry, x86_feature, has_feature);
-	guest_cpu_cap_change(vcpu->common, x86_feature, has_feature);
-}
-
+/* The table keeps plane 0's view of the runtime bits for KVM_GET_CPUID2. */
 static void kvm_update_cpuid_runtime(struct kvm_vcpu *vcpu)
 {
 	struct kvm_cpuid_entry2 *best;
 
+	/*
+	 * Compute from the plane-0 twin no matter which plane triggered
+	 * the refresh: KVM_SET_CPUID runs this for every plane in turn,
+	 * and KVM_GET_CPUID2 is reachable from any plane's fd.
+	 */
+	if (vcpu->common->vcpus[0])
+		vcpu = vcpu->common->vcpus[0];
+
 	vcpu->common->arch.cpuid_dynamic_bits_dirty = false;
 
 	best = kvm_find_cpuid_entry(vcpu, 1);
 	if (best) {
-		kvm_update_feature_runtime(vcpu, best, X86_FEATURE_OSXSAVE,
-					   kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE));
-
-		kvm_update_feature_runtime(vcpu, best, X86_FEATURE_APIC,
-					   vcpu->arch.apic_base & MSR_IA32_APICBASE_ENABLE);
-
+		kvm_cpuid_apply_runtime(vcpu, 1, 0, &best->eax, &best->ebx,
+					&best->ecx, &best->edx);
+		guest_cpu_cap_change(vcpu->common, X86_FEATURE_OSXSAVE,
+				     cpuid_entry_has(best, X86_FEATURE_OSXSAVE));
+		guest_cpu_cap_change(vcpu->common, X86_FEATURE_APIC,
+				     cpuid_entry_has(best, X86_FEATURE_APIC));
 		if (!kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT))
-			kvm_update_feature_runtime(vcpu, best, X86_FEATURE_MWAIT,
-						   vcpu->arch.ia32_misc_enable_msr &
-						   MSR_IA32_MISC_ENABLE_MWAIT);
+			guest_cpu_cap_change(vcpu->common, X86_FEATURE_MWAIT,
+					     cpuid_entry_has(best, X86_FEATURE_MWAIT));
 	}
 
 	best = kvm_find_cpuid_entry_index(vcpu, 7, 0);
-	if (best)
-		kvm_update_feature_runtime(vcpu, best, X86_FEATURE_OSPKE,
-					   kvm_is_cr4_bit_set(vcpu, X86_CR4_PKE));
-
+	if (best) {
+		kvm_cpuid_apply_runtime(vcpu, 7, 0, &best->eax, &best->ebx,
+					&best->ecx, &best->edx);
+		guest_cpu_cap_change(vcpu->common, X86_FEATURE_OSPKE,
+				     cpuid_entry_has(best, X86_FEATURE_OSPKE));
+	}
 
 	best = kvm_find_cpuid_entry_index(vcpu, 0xD, 0);
 	if (best)
-		best->ebx = xstate_required_size(vcpu->arch.xcr0, false);
+		kvm_cpuid_apply_runtime(vcpu, 0xd, 0, &best->eax, &best->ebx,
+					&best->ecx, &best->edx);
 
 	best = kvm_find_cpuid_entry_index(vcpu, 0xD, 1);
-	if (best && (cpuid_entry_has(best, X86_FEATURE_XSAVES) ||
-		     cpuid_entry_has(best, X86_FEATURE_XSAVEC)))
-		best->ebx = xstate_required_size(vcpu->arch.xcr0 |
-						 vcpu->arch.ia32_xss, true);
+	if (best)
+		kvm_cpuid_apply_runtime(vcpu, 0xd, 1, &best->eax, &best->ebx,
+					&best->ecx, &best->edx);
 }
 
 bool kvm_cpuid_has_hyperv(struct kvm_vcpu *vcpu)
@@ -2121,17 +2125,58 @@ get_out_of_range_cpuid_entry(struct kvm_vcpu *vcpu, u32 *fn_ptr, u32 index)
 	return kvm_find_cpuid_entry_index(vcpu, basic->eax, index);
 }
 
+/*
+ * The planes share the CPUID table but not CR4, the APIC enable, MISC_ENABLE
+ * or XCR0/XSS: the bits that follow those come from the executing plane.
+ */
+static void kvm_cpuid_apply_runtime(struct kvm_vcpu *vcpu, u32 function,
+				    u32 index, u32 *eax, u32 *ebx, u32 *ecx,
+				    u32 *edx)
+{
+	switch (function) {
+	case 1:
+		*ecx &= ~feature_bit(OSXSAVE);
+		if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE))
+			*ecx |= feature_bit(OSXSAVE);
+
+		*edx &= ~feature_bit(APIC);
+		if (vcpu->arch.apic_base & MSR_IA32_APICBASE_ENABLE)
+			*edx |= feature_bit(APIC);
+
+		if (!kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT)) {
+			*ecx &= ~feature_bit(MWAIT);
+			if (vcpu->arch.ia32_misc_enable_msr &
+			    MSR_IA32_MISC_ENABLE_MWAIT)
+				*ecx |= feature_bit(MWAIT);
+		}
+		break;
+	case 7:
+		if (index)
+			break;
+		*ecx &= ~feature_bit(OSPKE);
+		if (kvm_is_cr4_bit_set(vcpu, X86_CR4_PKE))
+			*ecx |= feature_bit(OSPKE);
+		break;
+	case 0xd:
+		if (index == 0)
+			*ebx = xstate_required_size(vcpu->arch.xcr0, false);
+		else if (index == 1 &&
+			 (*eax & (feature_bit(XSAVES) | feature_bit(XSAVEC))))
+			*ebx = xstate_required_size(vcpu->arch.xcr0 |
+						    vcpu->arch.ia32_xss, true);
+		break;
+	default:
+		break;
+	}
+}
+
 bool kvm_cpuid(struct kvm_vcpu *vcpu, u32 *eax, u32 *ebx,
 	       u32 *ecx, u32 *edx, bool exact_only)
 {
 	u32 orig_function = *eax, function = *eax, index = *ecx;
-	struct kvm_vcpu_common *common = vcpu->common;
 	struct kvm_cpuid_entry2 *entry;
 	bool exact, used_max_basic = false;
 
-	if (common->arch.cpuid_dynamic_bits_dirty)
-		kvm_update_cpuid_runtime(vcpu);
-
 	entry = kvm_find_cpuid_entry_index(vcpu, function, index);
 	exact = !!entry;
 
@@ -2145,6 +2190,7 @@ bool kvm_cpuid(struct kvm_vcpu *vcpu, u32 *eax, u32 *ebx,
 		*ebx = entry->ebx;
 		*ecx = entry->ecx;
 		*edx = entry->edx;
+		kvm_cpuid_apply_runtime(vcpu, function, index, eax, ebx, ecx, edx);
 		if (function == 7 && index == 0) {
 			u64 data;
 			if ((*ebx & (feature_bit(RTM) | feature_bit(HLE))) &&
diff --git a/arch/x86/kvm/cpuid.h b/arch/x86/kvm/cpuid.h
index 873377404899..2720e1820d7d 100644
--- a/arch/x86/kvm/cpuid.h
+++ b/arch/x86/kvm/cpuid.h
@@ -273,8 +273,9 @@ static __always_inline bool guest_cpu_cap_has(struct kvm_vcpu *vcpu,
 	struct kvm_vcpu_common *common = vcpu->common;
 
 	/*
-	 * Except for MWAIT, querying dynamic feature bits is disallowed, so
-	 * that KVM can defer runtime updates until the next CPUID emulation.
+	 * Except for MWAIT, querying dynamic feature bits is disallowed: the
+	 * guest gets them from the state of the plane that executes CPUID, and
+	 * the table copy is refreshed only for the CPUID ioctls.
 	 */
 	BUILD_BUG_ON(x86_feature == X86_FEATURE_APIC ||
 		     x86_feature == X86_FEATURE_OSXSAVE ||
-- 
2.50.1

[RFC PATCH 44/55] KVM: selftests: Test hugepage sizing over mixed protection attributes

[David Woodhouse]

From: Mushahid Hussain <hmushi@amazon.co.uk>

Back a 2 MiB-aligned region with a transparent hugepage and give it
alternating NX and NW protection attributes on its 4 KiB pages.  The
guest reads an NX page, which maps the region, then writes the NW page
next to it: the write must exit with KVM_EXIT_MEMORY_FAULT and leave
the page unchanged, instead of going through a 2 MiB leaf that
inherited the NX page's permissions.  Clearing the attribute lets the
write through.

This pins the hugepage side of the VSM memory protection boundary: a
single 4 KiB protection set by a higher VTL must not be widened away
by a neighbouring gfn's fault installing a hugepage whose access was
computed from the neighbour alone.  The host refuses that hugepage via
the KVM_LPAGE_MIXED_FLAG tracking which
kvm_arch_post_set_memory_attributes() maintains over the whole
attribute word, NR/NW/NX included.

[dwmw2: re-done by hand from Mushahid's original, adapted from the
 KVM_SET_PLANE_MEMORY_ATTRIBUTES uAPI to the per-plane memory
 attributes design: plane 0 attributes are set through
 KVM_SET_MEMORY_ATTRIBUTES with the NR/NW/NX protection bits, no
 plane runtime is needed, and the fault exit carries no plane field]

Assisted-by: LLM
Signed-off-by: Mushahid Hussain <hmushi@amazon.co.uk>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/x86/memattr_hugepage_test.c | 132 ++++++++++++++++++
 2 files changed, 133 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/memattr_hugepage_test.c

diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 14e385750ed1..66b16987d6e2 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -92,6 +92,7 @@ TEST_GEN_PROGS_x86 += x86/hyperv_tlb_flush
 TEST_GEN_PROGS_x86 += x86/kvm_clock_test
 TEST_GEN_PROGS_x86 += x86/kvm_pv_test
 TEST_GEN_PROGS_x86 += x86/kvm_buslock_test
+TEST_GEN_PROGS_x86 += x86/memattr_hugepage_test
 TEST_GEN_PROGS_x86 += x86/monitor_mwait_test
 TEST_GEN_PROGS_x86 += x86/msrs_test
 TEST_GEN_PROGS_x86 += x86/nested_close_kvm_test
diff --git a/tools/testing/selftests/kvm/x86/memattr_hugepage_test.c b/tools/testing/selftests/kvm/x86/memattr_hugepage_test.c
new file mode 100644
index 000000000000..156036eda5db
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/memattr_hugepage_test.c
@@ -0,0 +1,132 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Test hugepage sizing over mixed memory protection attributes on x86.
+ *
+ * Back a 2 MiB-aligned region with a transparent hugepage and give it
+ * alternating KVM_MEMORY_ATTRIBUTE_NX and _NW on its 4 KiB pages.  The
+ * guest reads an NX page, which maps the region, then writes the NW page
+ * next to it: the write must exit with KVM_EXIT_MEMORY_FAULT and leave
+ * the page unchanged, instead of going through a 2 MiB leaf that
+ * inherited the NX page's permissions.  Clearing the attribute lets the
+ * write through.
+ *
+ * This pins the VSM memory protection boundary: a VTL1 guest protecting
+ * one 4 KiB page of VTL0 memory must not have the protection widened
+ * away by a neighbouring fault's hugepage.
+ */
+#include "kvm_util.h"
+#include "processor.h"
+#include "test_util.h"
+#include "ucall_common.h"
+
+#define MIXED_GPA	0x40000000ul
+#define MIXED_SLOT	10
+#define MIXED_PAGES	1024	/* 4 MiB: a whole 2 MiB hugepage inside */
+
+/*
+ * MADV_HUGEPAGE is advisory and thp_configured() only checks that THP
+ * exists in the host kernel: if no 2 MiB THP actually backs the region
+ * (runtime policy "never", fragmentation), KVM maps 4 KiB leaves and
+ * the test "passes" without covering the hugepage path it exists for.
+ * Read the smaps entry covering @hva and report whether at least one
+ * anonymous hugepage backs it.
+ */
+static bool anon_hugepage_backed(void *hva)
+{
+	unsigned long start, end, kb, addr = (unsigned long)hva;
+	bool in_vma = false;
+	char line[256];
+	FILE *f;
+
+	f = fopen("/proc/self/smaps", "r");
+	TEST_ASSERT(f, "Failed to open /proc/self/smaps");
+
+	while (fgets(line, sizeof(line), f)) {
+		if (sscanf(line, "%lx-%lx ", &start, &end) == 2) {
+			in_vma = addr >= start && addr < end;
+			continue;
+		}
+		if (in_vma && sscanf(line, "AnonHugePages: %lu kB", &kb) == 1) {
+			fclose(f);
+			return kb >= 2048;
+		}
+	}
+
+	fclose(f);
+	TEST_FAIL("No smaps entry covers %p", hva);
+	return false;
+}
+
+static void guest_mixed_attrs(void)
+{
+	u64 *nx_page = (u64 *)MIXED_GPA;
+	u64 *nw_page = (u64 *)(MIXED_GPA + PAGE_SIZE);
+
+	/* The first touch maps the region: a read of a no-exec page. */
+	GUEST_SYNC(READ_ONCE(*nx_page));
+	/* A write to the no-write page next to it must be denied. */
+	WRITE_ONCE(*nw_page, 0xbad);
+	GUEST_SYNC(READ_ONCE(*nw_page));
+	GUEST_DONE();
+}
+
+int main(int argc, char *argv[])
+{
+	struct kvm_vcpu *vcpu;
+	struct kvm_run *run;
+	struct kvm_vm *vm;
+	struct ucall uc;
+	u64 *host;
+	int i, r;
+
+	TEST_REQUIRE(kvm_check_cap(KVM_CAP_MEMORY_ATTRIBUTES) &
+		     KVM_MEMORY_ATTRIBUTE_NW);
+	TEST_REQUIRE(thp_configured());
+
+	vm = vm_create_with_one_vcpu(&vcpu, guest_mixed_attrs);
+	run = vcpu->run;
+
+	vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS_THP, MIXED_GPA,
+				    MIXED_SLOT, MIXED_PAGES, 0);
+	virt_map(vm, MIXED_GPA, MIXED_GPA, MIXED_PAGES);
+	host = addr_gpa2hva(vm, MIXED_GPA);
+	memset(host, 0, MIXED_PAGES * getpagesize());
+	host[0] = 0x11;
+	host[PAGE_SIZE / sizeof(u64)] = 0x22;
+
+	/* The memset faulted the region in; a THP must have materialised. */
+	TEST_REQUIRE(anon_hugepage_backed(host));
+
+	/* As VTL1 would: alternate NX and NW over the hugepage's 4 KiB pages. */
+	for (i = 0; i < 512; i++)
+		vm_set_memory_attributes(vm, MIXED_GPA + i * PAGE_SIZE,
+					 PAGE_SIZE,
+					 (i & 1) ? KVM_MEMORY_ATTRIBUTE_NW :
+						   KVM_MEMORY_ATTRIBUTE_NX);
+
+	vcpu_run(vcpu);
+	TEST_ASSERT_EQ(get_ucall(vcpu, &uc), UCALL_SYNC);
+	TEST_ASSERT_EQ(uc.args[1], 0x11);
+
+	r = _vcpu_run(vcpu);
+	TEST_ASSERT(r == -1 && errno == EFAULT,
+		    "write to the NW page: ret %d errno %d, exit %u",
+		    r, errno, run->exit_reason);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_MEMORY_FAULT);
+	TEST_ASSERT_EQ(run->memory_fault.gpa, MIXED_GPA + PAGE_SIZE);
+	TEST_ASSERT(run->memory_fault.flags & KVM_MEMORY_EXIT_FLAG_WRITE,
+		    "expected a write fault, flags 0x%llx",
+		    run->memory_fault.flags);
+	TEST_ASSERT_EQ(host[PAGE_SIZE / sizeof(u64)], 0x22);
+
+	/* Lift the protection: the write goes through at the right level. */
+	vm_set_memory_attributes(vm, MIXED_GPA + PAGE_SIZE, PAGE_SIZE, 0);
+	vcpu_run(vcpu);
+	TEST_ASSERT_EQ(get_ucall(vcpu, &uc), UCALL_SYNC);
+	TEST_ASSERT_EQ(uc.args[1], 0xbad);
+	vcpu_run(vcpu);
+	TEST_ASSERT_EQ(get_ucall(vcpu, &uc), UCALL_DONE);
+
+	kvm_vm_free(vm);
+	return 0;
+}
-- 
2.50.1

[RFC PATCH 45/55] KVM: selftests: Test a plane suspending with an undelivered interrupt

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Pin the forced-VTL-entry re-arm: a plane-1 twin self-IPIs with IF
clear, runs on (consuming its single-shot hv_apic_event_pending
marker), and is then suspended by KVM_SET_PLANE_RUN_STATE exactly like
a userspace VTL switch.  Running plane 0 must report the suspended
twin's deliverable interrupt as a KVM_EXIT_HYPERV_VTL_INTERRUPT exit,
and the re-entered twin must receive the vector once it sets IF.  On a
kernel without the re-arm, plane 0 reaches its guest code and the
twin's interrupt is stranded in its IRR, which is the Windows VSM
hibernate deadlock in miniature.

The default selftest VM owns a fully in-kernel irqchip, which planes
refuse; add a kvm_force_split_irqchip override to the library (as in
the public planes tree) so a planes test can use the stock VM setup.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../testing/selftests/kvm/include/kvm_util.h  |   1 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |  11 +
 .../selftests/kvm/x86/planes_vtl_entry_test.c | 188 ++++++++++++++++++
 4 files changed, 201 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/planes_vtl_entry_test.c

diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 66b16987d6e2..47c7a4c732eb 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -107,6 +107,7 @@ TEST_GEN_PROGS_x86 += x86/nested_vmsave_vmload_test
 TEST_GEN_PROGS_x86 += x86/platform_info_test
 TEST_GEN_PROGS_x86 += x86/plane_x86_test
 TEST_GEN_PROGS_x86 += x86/planes_test
+TEST_GEN_PROGS_x86 += x86/planes_vtl_entry_test
 TEST_GEN_PROGS_x86 += x86/pmu_counters_test
 TEST_GEN_PROGS_x86 += x86/pmu_event_filter_test
 TEST_GEN_PROGS_x86 += x86/private_mem_conversions_test
diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h
index 204f1474f102..d628e31f8bb4 100644
--- a/tools/testing/selftests/kvm/include/kvm_util.h
+++ b/tools/testing/selftests/kvm/include/kvm_util.h
@@ -713,6 +713,7 @@ static inline bool is_smt_on(void)
 	return false;
 }
 
+extern bool kvm_force_split_irqchip;
 void vm_create_irqchip(struct kvm_vm *vm);
 
 /*
diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c
index 3972c9e0486a..0bd8b29af6f7 100644
--- a/tools/testing/selftests/kvm/lib/kvm_util.c
+++ b/tools/testing/selftests/kvm/lib/kvm_util.c
@@ -1696,11 +1696,22 @@ void *addr_gpa2alias(struct kvm_vm *vm, gpa_t gpa)
 	return (void *) ((uintptr_t) region->host_alias + offset);
 }
 
+/*
+ * Tests that need a split IRQ chip (x86 only), e.g. for VM planes, set this
+ * around VM creation to override the fully in-kernel default.
+ */
+bool kvm_force_split_irqchip;
+
 /* Create an interrupt controller chip for the specified VM. */
 void vm_create_irqchip(struct kvm_vm *vm)
 {
 	int r;
 
+	if (kvm_force_split_irqchip && kvm_has_cap(KVM_CAP_SPLIT_IRQCHIP)) {
+		vm_enable_split_irqchip(vm);
+		return;
+	}
+
 	/*
 	 * Allocate a fully in-kernel IRQ chip by default, but fall back to a
 	 * split model (x86 only) if that fails (KVM x86 allows compiling out
diff --git a/tools/testing/selftests/kvm/x86/planes_vtl_entry_test.c b/tools/testing/selftests/kvm/x86/planes_vtl_entry_test.c
new file mode 100644
index 000000000000..3ad402d32ed1
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/planes_vtl_entry_test.c
@@ -0,0 +1,188 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Test that an interrupt posted to a higher plane survives the plane
+ * suspending without having delivered it.
+ *
+ * A higher plane's pending interrupt forces entry into that plane via a
+ * single-shot marker (hv_apic_event_pending) that the plane consumes when
+ * it runs.  A plane that runs with IF clear until it suspends consumes
+ * the marker without delivering the vector, which used to strand the
+ * vector in the plane's IRR forever: with the marker gone, the lower
+ * plane never again exits with KVM_EXIT_HYPERV_VTL_INTERRUPT, and a
+ * guest waiting on the higher plane deadlocks.
+ *
+ * The test models the failing sequence: the plane-1 twin self-IPIs with
+ * IF clear, keeps running (consuming the marker), then is suspended by
+ * the run-state ioctl exactly like a userspace VTL switch.  Running
+ * plane 0 must then report a pending VTL interrupt for plane 1, and the
+ * re-entered twin must receive the vector once it sets IF.
+ */
+#include <errno.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <unistd.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+#include "apic.h"
+
+#ifndef KVM_EXIT_HYPERV_VTL_INTERRUPT
+#define KVM_EXIT_HYPERV_VTL_INTERRUPT  4
+#endif
+
+#define TEST_VECTOR		0xe0
+#define TWIN_SYNC_PORT		0x7e
+#define TWIN_STACK_PAGES	4
+
+static volatile bool twin_got_ipi;
+
+static void twin_sync(uint8_t stage)
+{
+	asm volatile("outb %0, %1" :: "a"(stage), "Nd"((uint16_t)TWIN_SYNC_PORT));
+}
+
+static void twin_ipi_handler(struct ex_regs *regs)
+{
+	twin_got_ipi = true;
+	x2apic_write_reg(APIC_EOI, 0);
+}
+
+/* Runs with IF clear: the initial RFLAGS has IF unset and nothing sets it. */
+static void twin_guest_code(void)
+{
+	x2apic_enable();
+	x2apic_write_reg(APIC_SELF_IPI, TEST_VECTOR);
+	twin_sync(1);
+	twin_sync(2);
+
+	/* Re-entered after suspension: open the window, take the vector. */
+	sti();
+	while (!twin_got_ipi)
+		cpu_relax();
+	twin_sync(3);
+	for (;;)
+		twin_sync(4);
+}
+
+static void plane0_guest_code(void)
+{
+	GUEST_SYNC(0);
+	GUEST_DONE();
+}
+
+static void set_plane_run_state(int vcpu_fd, u32 state)
+{
+	int r = ioctl(vcpu_fd, KVM_SET_PLANE_RUN_STATE, &state);
+
+	TEST_ASSERT(!r, "KVM_SET_PLANE_RUN_STATE(%u) failed: errno %d",
+		    state, errno);
+}
+
+static void run_twin_expect_sync(struct kvm_vcpu *vcpu0,
+				 struct kvm_run *twin_run, uint8_t stage)
+{
+	int r = ioctl(vcpu0->fd, KVM_RUN, NULL);
+
+	TEST_ASSERT(!r, "KVM_RUN (twin runnable) failed: errno %d", errno);
+	TEST_ASSERT_EQ(twin_run->exit_reason, KVM_EXIT_IO);
+	TEST_ASSERT_EQ(twin_run->io.port, TWIN_SYNC_PORT);
+	TEST_ASSERT_EQ(*((uint8_t *)twin_run + twin_run->io.data_offset), stage);
+}
+
+int main(int argc, char *argv[])
+{
+	struct kvm_sregs sregs;
+	struct kvm_regs regs;
+	struct kvm_run *twin_run;
+	struct kvm_vcpu *vcpu0;
+	struct kvm_vm *vm;
+	gva_t twin_stack;
+	int plane_fd, twin_fd, mmap_sz, r;
+
+	kvm_force_split_irqchip = true;
+	vm = __vm_create(VM_SHAPE_DEFAULT, 1, 0);
+	kvm_force_split_irqchip = false;
+
+	TEST_REQUIRE(vm_check_cap(vm, KVM_CAP_PLANES) >= 1);
+	TEST_REQUIRE(__vm_enable_cap(vm, KVM_CAP_PLANES, 2) == 0);
+
+	vcpu0 = vm_vcpu_add(vm, 0, plane0_guest_code);
+
+	plane_fd = __vm_ioctl(vm, KVM_CREATE_PLANE, (void *)1UL);
+	TEST_ASSERT(plane_fd >= 0, "KVM_CREATE_PLANE failed: errno %d", errno);
+	twin_fd = __kvm_ioctl(plane_fd, KVM_CREATE_VCPU, NULL);
+	TEST_ASSERT(twin_fd >= 0, "KVM_CREATE_VCPU on plane 1 failed: errno %d",
+		    errno);
+
+	mmap_sz = ioctl(vm->kvm_fd, KVM_GET_VCPU_MMAP_SIZE, NULL);
+	TEST_ASSERT(mmap_sz >= (int)sizeof(*twin_run),
+		    "KVM_GET_VCPU_MMAP_SIZE failed");
+	twin_run = mmap(NULL, mmap_sz, PROT_READ | PROT_WRITE, MAP_SHARED,
+			twin_fd, 0);
+	TEST_ASSERT(twin_run != MAP_FAILED, "mmap of twin kvm_run failed");
+
+	vm_install_exception_handler(vm, TEST_VECTOR, twin_ipi_handler);
+
+	/*
+	 * The twin shares the mode and page tables with plane 0: reuse
+	 * plane 0's segment state and point it at its own code and stack.
+	 * RFLAGS keeps IF clear.
+	 */
+	vcpu_sregs_get(vcpu0, &sregs);
+	r = ioctl(twin_fd, KVM_SET_SREGS, &sregs);
+	TEST_ASSERT(!r, "KVM_SET_SREGS on twin failed: errno %d", errno);
+
+	twin_stack = vm_alloc_pages(vm, TWIN_STACK_PAGES);
+	r = ioctl(twin_fd, KVM_GET_REGS, &regs);
+	TEST_ASSERT(!r, "KVM_GET_REGS on twin failed: errno %d", errno);
+	regs.rip = (unsigned long)twin_guest_code;
+	regs.rsp = twin_stack + TWIN_STACK_PAGES * getpagesize() - 16;
+	regs.rflags = 0x2;
+	r = ioctl(twin_fd, KVM_SET_REGS, &regs);
+	TEST_ASSERT(!r, "KVM_SET_REGS on twin failed: errno %d", errno);
+
+	/*
+	 * Run the twin as a userspace VTL switch would: twin runnable,
+	 * plane 0 stopped.  The twin self-IPIs with IF clear, so the
+	 * vector lands in its IRR without being delivered, and the
+	 * second sync guarantees a fresh guest entry consumed the
+	 * forced-entry marker.
+	 */
+	set_plane_run_state(twin_fd, KVM_PLANE_RUN_STATE_RUNNABLE);
+	set_plane_run_state(vcpu0->fd, KVM_PLANE_RUN_STATE_STOPPED);
+	run_twin_expect_sync(vcpu0, twin_run, 1);
+	run_twin_expect_sync(vcpu0, twin_run, 2);
+
+	/* Suspend the twin with the vector still latched in its IRR. */
+	set_plane_run_state(twin_fd, KVM_PLANE_RUN_STATE_STOPPED);
+	set_plane_run_state(vcpu0->fd, KVM_PLANE_RUN_STATE_RUNNABLE);
+
+	/*
+	 * Plane 0 must not run the guest: the suspended twin still has a
+	 * deliverable interrupt, which must surface as a VTL interrupt
+	 * exit.  On a broken kernel the marker was consumed and plane 0
+	 * reaches its guest code.
+	 */
+	r = ioctl(vcpu0->fd, KVM_RUN, NULL);
+	TEST_ASSERT(!r, "KVM_RUN (plane 0) failed: errno %d", errno);
+	TEST_ASSERT(vcpu0->run->exit_reason == KVM_EXIT_HYPERV,
+		    "Expected KVM_EXIT_HYPERV, got %u (%s): the twin's pending "
+		    "interrupt was lost when it suspended",
+		    vcpu0->run->exit_reason,
+		    exit_reason_str(vcpu0->run->exit_reason));
+	TEST_ASSERT_EQ(vcpu0->run->hyperv.type, KVM_EXIT_HYPERV_VTL_INTERRUPT);
+	TEST_ASSERT_EQ(vcpu0->run->hyperv.plane, 1);
+
+	/* Enter the twin as userspace would, and let it take the vector. */
+	set_plane_run_state(twin_fd, KVM_PLANE_RUN_STATE_RUNNABLE);
+	set_plane_run_state(vcpu0->fd, KVM_PLANE_RUN_STATE_STOPPED);
+	run_twin_expect_sync(vcpu0, twin_run, 3);
+
+	kvm_munmap(twin_run, mmap_sz);
+	close(twin_fd);
+	close(plane_fd);
+	kvm_vm_free(vm);
+	return 0;
+}
-- 
2.50.1

[RFC PATCH 46/55] KVM: x86: hyperv: Do not skip a moved vCPU when completing a hypercall

[David Woodhouse]

From: Mushahid Hussain <hmushi@amazon.co.uk>

A Hyper-V hypercall that exits to userspace is completed on the next
KVM_RUN of the vCPU, and the completion skips the VMCALL. With planes,
the lower plane is not run again until the secure kernel returns, and
userspace applies the registers the secure kernel sets for it in
between. On a resume from hibernation the secure kernel sets VTL0's
RIP to the hibernate continuation. The deferred completion then skips
an instruction on top of that RIP.

The two vendors fail differently. SVM writes the stale next_rip of the
old VMCALL exit, so the VP resumes at the hypercall page's RET with
the new RSP, pops a zero and faults at RIP 0: Windows Server 2025
with Credential Guard bugchecks 0xD1 (0, 0xff, 0, 0) on every resume
on AMD. VMX adds the VMCALL length to the new RIP and lands three
bytes into PopSaveHiberContextWrapper+0x35, where the bytes happen to
decode as a harmless instruction before resynchronising, so Intel
passes by luck.

Record the RIP of the hypercall when exiting to userspace and complete
without skipping when the vCPU is no longer there. Applies to both
the VTL switch completion and the generic userspace completion. The
generic KVM_EXIT_HYPERCALL and Xen completions have the same shape
but userspace does not move the vCPU there; on arm64 the HVC is
skipped at exit time, so the planes series is unaffected there.

Measured on an m6a.metal (AMD) planes bench, WS2025 CG hibernate:
SK SETs VTL0 RIP = wrapper+0x35, RSP = cont; first plane-0 exits after
the return are at hypercall page+3, then RIP 0; KiBugCheckData 0xD1.

Assisted-by: LLM
Signed-off-by: Mushahid Hussain <hmushi@amazon.co.uk>

[dwmw2: this stack retires a forwarded VtlCall/VtlReturn eagerly at
 exit time with no pending completion, so that path is unexposed; the
 exposures are the FORWARD_HCALL completion and the plane an in-kernel
 VTL switch suspends at its vmcall, whose RIP the higher VTL rewrites
 via HvCallSetVpRegisters. Both resume through
 kvm_hv_hypercall_complete_userspace(), so one moved-check there
 covers both; the RIP is recorded in kvm_hv_hypercall_userspace_exit()
 and kvm_hv_vtl_switch(). The check yields to the TLB-flush
 suspension retry (a suspended vCPU re-executes the instruction, so
 there is no skip to suppress).]
(re-done by hand)
[dwmw2: folded in the RAISE_UD completion-contract tightening: the
 KVM_CAP_HYPERV_FORWARD_HCALL extension check returns 2 to advertise
 that the kernel honours KVM_HV_HCALL_RESULT_RAISE_UD, the bit is
 honoured only while forwarding is enabled (the legacy syndbg and
 slow-form completions treat bit 63 as part of the result) and only
 after the suspended and moved guards, and an in-kernel-armed
 completion is tagged with hcall_resume_pending so it re-executes the
 hypercall instead of consuming stale run-union bytes left by an
 unrelated exit.]
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 Documentation/virt/kvm/api.rst  |  6 +++
 arch/x86/include/asm/kvm_host.h | 17 ++++++++
 arch/x86/kvm/hyperv.c           | 77 ++++++++++++++++++++++++++++++---
 arch/x86/kvm/x86.c              | 11 ++++-
 include/uapi/linux/kvm.h        |  4 +-
 5 files changed, 106 insertions(+), 9 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index d75987c1d325..494600236360 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -9146,6 +9146,12 @@ with ``hyperv.u.hcall.result`` set.  Hypercalls KVM implements itself,
 and the ordering of CPUID-enforcement checks against hypercall
 dispatch, are unaffected.
 
+When KVM_CHECK_EXTENSION reports a value of 2 or more, a result with
+KVM_HV_HCALL_RESULT_RAISE_UD set makes KVM raise #UD at the hypercall
+instruction instead of completing it (honoured only while forwarding
+is enabled).  A VMM must not set that bit on kernels reporting 1,
+where it lands in the guest's RAX as part of the result.
+
 7.49 KVM_CAP_HCALL_FAULT_EXIT
 ------------------------------------
 
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 6d3bae46ecab..e03ae6493641 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -762,6 +762,23 @@ struct kvm_vcpu_hv {
 	u64 vtl_entry_rcx;
 	bool vtl_entry_regs_valid;
 
+	/*
+	 * Linear RIP of the hypercall instruction whose deferred
+	 * completion (userspace exit or in-kernel VTL switch suspension)
+	 * is pending; linear so that a CS.base change while the vCPU is
+	 * parked cannot alias the recorded offset.
+	 */
+	u64 hcall_exit_rip;
+
+	/*
+	 * Set when an in-kernel VTL switch armed the deferred completion:
+	 * the resume value only lands in run->hyperv.u.hcall.result when
+	 * the opposite-direction switch provides it.  While set, the run
+	 * union holds unrelated bytes and the completion must retry the
+	 * hypercall instead of consuming them.
+	 */
+	bool hcall_resume_pending;
+
 	bool suspended;
 	int waiting_on;
 };
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index e3ac840c8e5e..02866b9b7e9b 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -2838,16 +2838,67 @@ static int kvm_hv_hypercall_complete(struct kvm_vcpu *vcpu, u64 result)
 	return ret;
 }
 
+/*
+ * A hypercall whose completion was deferred (a userspace exit, or an
+ * in-kernel VTL switch suspending this plane) completes on the vCPU's next
+ * run.  The vCPU may have been moved in between: the higher VTL can rewrite
+ * the suspended plane's RIP through HvCallSetVpRegisters (on a hibernate
+ * resume the secure kernel points VTL0 at the continuation), and userspace
+ * applies that through KVM_SET_REGS.  Skipping an instruction there would
+ * land past the new RIP, so only skip when the vCPU still sits at the RIP
+ * recorded when the completion was armed.
+ */
+static bool kvm_hv_hypercall_moved(struct kvm_vcpu *vcpu)
+{
+	return !kvm_is_linear_rip(vcpu, to_hv_vcpu(vcpu)->hcall_exit_rip);
+}
+
 static int kvm_hv_hypercall_complete_userspace(struct kvm_vcpu *vcpu)
 {
+	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
 	u64 result = vcpu->run->hyperv.u.hcall.result;
 
+	/* Still suspended: nothing completes yet, the vmcall is retried. */
+	if (kvm_hv_vcpu_suspended(vcpu))
+		return 1;
+
+	/*
+	 * An in-kernel VTL switch armed this completion, and only the
+	 * opposite-direction switch writes the resume value.  Resumed any
+	 * other way (a run-state ioctl, a switch forwarded to userspace),
+	 * the run union holds leftover bytes from an unrelated exit:
+	 * don't consume them, re-execute the hypercall instead.
+	 */
+	if (hv_vcpu->hcall_resume_pending) {
+		hv_vcpu->hcall_resume_pending = false;
+		return 1;
+	}
+
+	if (kvm_hv_hypercall_moved(vcpu)) {
+		/*
+		 * The parked plane was repointed (HvCallSetVpRegisters via
+		 * the higher VTL): deliver the result without the deferred
+		 * skip.  Strip the VMM-only control bit; an exception aimed
+		 * at the hypercall instruction has no target once RIP
+		 * moved, and bit 63 must not leak into guest RAX.
+		 */
+		result &= ~KVM_HV_HCALL_RESULT_RAISE_UD;
+		trace_kvm_hv_hypercall_done(result);
+		kvm_hv_hypercall_set_result(vcpu, result);
+		++vcpu->stat.hypercalls;
+		return 1;
+	}
+
 	/*
 	 * A hypercall the VMM judged malformed beyond returning a status
 	 * (such as a VTL call that is invalid in the current state) faults
-	 * at the instruction instead of completing.
+	 * at the instruction instead of completing.  Only completions the
+	 * VMM owns under KVM_CAP_HYPERV_FORWARD_HCALL honour the control
+	 * bit; legacy exits (syndbg, slow-form hypercalls) predate it and
+	 * keep bit 63 as part of the result.
 	 */
-	if (result & KVM_HV_HCALL_RESULT_RAISE_UD) {
+	if (to_kvm_hv(vcpu->kvm)->hv_forward_hcalls &&
+	    (result & KVM_HV_HCALL_RESULT_RAISE_UD)) {
 		kvm_queue_exception(vcpu, UD_VECTOR);
 		return 1;
 	}
@@ -2993,6 +3044,9 @@ static int kvm_hv_hypercall_userspace_exit(struct kvm_vcpu *vcpu,
 	vcpu->run->hyperv.u.hcall.input = hc->param;
 	vcpu->run->hyperv.u.hcall.params[0] = hc->ingpa;
 	vcpu->run->hyperv.u.hcall.params[1] = hc->outgpa;
+	to_hv_vcpu(vcpu)->hcall_exit_rip = kvm_get_linear_rip(vcpu);
+	/* Userspace owns this exit and writes hcall.result before KVM_RUN. */
+	to_hv_vcpu(vcpu)->hcall_resume_pending = false;
 	vcpu->arch.complete_userspace_io = kvm_hv_hypercall_complete_userspace;
 	return 0;
 }
@@ -3136,9 +3190,11 @@ bool kvm_hv_vtl_interrupt_entry(struct kvm_vcpu *vcpu,
 	 * value is guest-derived; strip the VMM-only RAISE_UD control bit
 	 * so the completion cannot be made to inject #UD by the guest.
 	 */
-	if (target->arch.complete_userspace_io)
+	if (target->arch.complete_userspace_io) {
 		vcpu->run->hyperv.u.hcall.result = kvm_rax_read(target) &
 						   ~KVM_HV_HCALL_RESULT_RAISE_UD;
+		to_hv_vcpu(target)->hcall_resume_pending = false;
+	}
 
 	vcpu->plane_kernel_stopped = true;
 	kvm_vcpu_set_plane_stopped(vcpu);
@@ -3282,7 +3338,9 @@ static int kvm_hv_vtl_switch(struct kvm_vcpu *vcpu, bool is_return)
 	 * instruction skip and the resume RAX from the shared run struct,
 	 * exactly as when userspace drives the switch; until then its
 	 * suspended RIP stays at the vmcall, which the higher VTL can
-	 * observe (and rewrite) through HvCallGetVpRegisters.
+	 * observe (and rewrite) through HvCallGetVpRegisters.  A rewritten
+	 * RIP suppresses the deferred skip (kvm_hv_hypercall_moved()): the
+	 * plane resumes exactly where the higher VTL pointed it.
 	 *
 	 * The resume value is guest-derived in all three cases (the
 	 * caller's RAX on a call or a fast return, the control area on a
@@ -3291,9 +3349,10 @@ static int kvm_hv_vtl_switch(struct kvm_vcpu *vcpu, bool is_return)
 	 * plane.
 	 */
 	resume &= ~KVM_HV_HCALL_RESULT_RAISE_UD;
-	if (target->arch.complete_userspace_io)
+	if (target->arch.complete_userspace_io) {
 		vcpu->run->hyperv.u.hcall.result = resume;
-	else
+		to_hv_vcpu(target)->hcall_resume_pending = false;
+	} else
 		/*
 		 * No completion is armed when the target's suspension
 		 * already retired its vmcall (forwarded switch) or was a
@@ -3307,8 +3366,12 @@ static int kvm_hv_vtl_switch(struct kvm_vcpu *vcpu, bool is_return)
 
 	/*
 	 * Suspend the current plane in its uncompleted hypercall; the
-	 * switch back will provide its resume value the same way.
+	 * switch back will provide its resume value the same way.  Until
+	 * it does, hcall_resume_pending marks the run union as not holding
+	 * this plane's result.
 	 */
+	to_hv_vcpu(vcpu)->hcall_exit_rip = kvm_get_linear_rip(vcpu);
+	to_hv_vcpu(vcpu)->hcall_resume_pending = true;
 	vcpu->arch.complete_userspace_io = kvm_hv_hypercall_complete_userspace;
 
 	vcpu->plane_kernel_stopped = true;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 89203909db36..dc6196107c23 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -4937,7 +4937,6 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_HYPERV_CPUID:
 	case KVM_CAP_HYPERV_ENFORCE_CPUID:
 	case KVM_CAP_SYS_HYPERV_CPUID:
-	case KVM_CAP_HYPERV_FORWARD_HCALL:
 	case KVM_CAP_HYPERV_VTL_SWITCH:
 	case KVM_CAP_HYPERV_REG_INTERCEPT:
 #endif
@@ -4985,6 +4984,16 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_ONE_REG:
 		r = 1;
 		break;
+#ifdef CONFIG_KVM_HYPERV
+	case KVM_CAP_HYPERV_FORWARD_HCALL:
+		/*
+		 * 2: KVM_HV_HCALL_RESULT_RAISE_UD is honoured on hypercall
+		 * completion.  A VMM must not set the bit on kernels
+		 * reporting 1, where it lands in guest RAX instead.
+		 */
+		r = 2;
+		break;
+#endif
 	case KVM_CAP_PRE_FAULT_MEMORY:
 		r = tdp_enabled;
 		break;
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 843cef8f4cbf..8224ef335ba1 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1038,7 +1038,9 @@ struct kvm_enable_cap {
  * Userspace completion of a forwarded Hyper-V hypercall (KVM_EXIT_HYPERV_HCALL
  * under KVM_CAP_HYPERV_FORWARD_HCALL): setting this bit in the result makes
  * KVM raise #UD at the faulting instruction instead of completing the
- * hypercall; all other result bits are then ignored.
+ * hypercall; all other result bits are then ignored.  Only honoured while
+ * forwarding is enabled, and only on kernels where
+ * KVM_CHECK_EXTENSION(KVM_CAP_HYPERV_FORWARD_HCALL) returns >= 2.
  */
 #define KVM_HV_HCALL_RESULT_RAISE_UD	(1ULL << 63)
 #define KVM_CAP_HCALL_FAULT_EXIT 252
-- 
2.50.1

[RFC PATCH 47/55] KVM: x86: Allow KVM_X86_SETUP_MCE on plane vCPUs

[David Woodhouse]

From: Mushahid Hussain <hmushi@amazon.co.uk>

mcg_cap is per vCPU. kvm_arch_vcpu_create() gives every vCPU the
KVM_MAX_MCE_BANKS default, plane twins included, but KVM_X86_SETUP_MCE
was only accepted on the plane-0 fd: userspace programs plane 0 and a
higher plane keeps running with the default, so the MCG view diverges
across planes. Hyper-V defines the MCG state as shared across Virtual
Trust Levels, so userspace needs to program the same value on every
plane: accept the ioctl on plane vCPUs.

Assisted-by: LLM

[dwmw2: ours gates plane vCPU fd ioctls in kvm_arch_is_vcpu_plane_ioctl()
 in the same shape; same one-line addition.]
(re-done by hand)
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 Documentation/virt/kvm/api.rst | 2 +-
 arch/x86/kvm/x86.c             | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 494600236360..24050c79d95c 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -4594,7 +4594,7 @@ capabilities will have the corresponding bits set.
 
 :Capability: KVM_CAP_MCE
 :Architectures: x86
-:Type: vcpu ioctl
+:Type: vcpu ioctl (all planes)
 :Parameters: u64 mcg_cap (in)
 :Returns: 0 on success,
          -EFAULT if u64 mcg_cap cannot be read,
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index dc6196107c23..e2e54f06b092 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -6400,6 +6400,7 @@ bool kvm_arch_is_vcpu_plane_ioctl(unsigned ioctl)
 	case KVM_SET_XSAVE:
 	case KVM_GET_REG_LIST:
 	case KVM_TRANSLATE:
+	case KVM_X86_SETUP_MCE:
 		return true;
 	default:
 		return false;
-- 
2.50.1

[RFC PATCH 48/55] KVM: x86: hyperv: Advertise XMM fast hypercall output

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The Hyper-V VSM register hypercalls return data in XMM registers when
the guest uses their fast forms, and the userspace handler answering
them writes those registers through the vCPU FPU state. Report
HV_X64_HYPERCALL_XMM_OUTPUT_AVAILABLE in KVM_GET_SUPPORTED_HV_CPUID so
the VMM can expose it; the reference hypervisor advertises it, and the
Windows secure kernel only uses the fast forms when it is present.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/hyperv.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 02866b9b7e9b..dbe1b83c1f28 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -3751,6 +3751,7 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid,
 			ent->ebx |= HV_ENABLE_EXTENDED_HYPERCALLS;
 
 			ent->edx |= HV_X64_HYPERCALL_XMM_INPUT_AVAILABLE;
+			ent->edx |= HV_X64_HYPERCALL_XMM_OUTPUT_AVAILABLE;
 			ent->edx |= HV_FEATURE_FREQUENCY_MSRS_AVAILABLE;
 			ent->edx |= HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE;
 
-- 
2.50.1

[RFC PATCH 49/55] KVM: SVM: Carry instruction state on reflected plane-protection faults

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

A plane-protection fault reflected to userspace becomes a TLFS GPA
intercept message, which carries the faulting instruction's length,
its bytes (for data accesses), and the fault-adjusted RFLAGS.  The
VMX exit handler harvests all of that from the VMCS, but
npf_interception() reflected the bare fault: on AMD the higher VTL
received intercept messages with instruction length 0, no instruction
bytes and RFLAGS.RF clear, failing the hyperv_vsm KUT
intercept-message tests ("Seen an expected read/write intercept
message").

Decode assists stash the raw fetch window (up to 15 bytes, not the
decoded length) in the VMCB for data accesses; run it through the
instruction decoder and report the decoded length and that many
bytes.  Set RFLAGS.RF before exiting: fault delivery gives the
restarted instruction RF=1, and VMX saves that fault-adjusted value
in guest RFLAGS on fault-type exits while SVM saves RF as it was
during execution.

An execute fault stores no bytes, and the length of the branch that
fetched the faulting page is not architecturally available on SVM
(VMX hardware reports it), so execute intercepts carry length 0.

The validity of the harvested state gets its own flag,
KVM_MEMORY_EXIT_FLAG_INSN_VALID, set by the VMX and SVM harvests only
when they populated the fields: VM_EXIT_INSTRUCTION_LEN is undefined
for EPT violations, all-zero insn_bytes is itself a valid x86
encoding, and kvm_prepare_memory_fault_exit() now clears
gva/insn_len/insn_bytes so no other memory-fault exit carries stale
values.  The flag bit (and PLANE_PROT before it) is enumerated by the
plane memory overlay capability, which also covers the extended
memory_fault layout.  The memory_attributes selftest asserted strict
equality on the exit flags; it now masks the informational flags out
and compares only the access that faulted.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/mmu/mmu.c                        |  2 +
 arch/x86/kvm/mmu/paging_tmpl.h                |  3 +
 arch/x86/kvm/svm/svm.c                        | 66 +++++++++++++++++++
 arch/x86/kvm/vmx/vmx.c                        | 13 ++--
 include/linux/kvm_host.h                      | 12 ++++
 include/uapi/linux/kvm.h                      | 17 ++++-
 .../testing/selftests/kvm/memory_attributes.c |  6 +-
 7 files changed, 112 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 07a8de28c73d..9a0a5d3ad995 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -4787,6 +4787,8 @@ static int kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 
 	if (kvm_faultin_memory_protections(vcpu, fault)) {
 		kvm_mmu_prepare_memory_fault_exit(vcpu, fault);
+		if (kvm_arch_max_planes(kvm) > 1)
+			vcpu->run->memory_fault.flags |= KVM_MEMORY_EXIT_FLAG_PLANE_PROT;
 		return -EFAULT;
 	}
 
diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h
index 78d1e5c9fbe4..f56ed74d4228 100644
--- a/arch/x86/kvm/mmu/paging_tmpl.h
+++ b/arch/x86/kvm/mmu/paging_tmpl.h
@@ -795,6 +795,9 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault
 		if (walker.memory_attributes_fault) {
 			kvm_prepare_memory_fault_exit(vcpu, walker.gfn << PAGE_SHIFT,
 						      PAGE_SIZE, false, false, false);
+			if (kvm_arch_max_planes(vcpu->kvm) > 1)
+				vcpu->run->memory_fault.flags |=
+					KVM_MEMORY_EXIT_FLAG_PLANE_PROT;
 			return -EFAULT;
 		}
 
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 31e5b597c3b0..a30f6f888b51 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -2012,6 +2012,72 @@ static int npf_interception(struct kvm_vcpu *vcpu)
 				svm->vmcb->control.insn_bytes : NULL,
 				svm->vmcb->control.insn_len);
 
+	/*
+	 * A plane-protection fault reflected to userspace becomes a TLFS
+	 * GPA intercept message carrying the faulting instruction's length
+	 * and bytes; see the equivalent harvest in handle_ept_violation().
+	 * Decode assists stash the raw fetch window (up to 15 bytes, not
+	 * the decoded length) in the VMCB for data accesses; run it
+	 * through the decoder to report the real length.  A fetch fault
+	 * stores no bytes, and the length of the branch that fetched the
+	 * faulting page is not architecturally available on SVM, so an
+	 * execute intercept reports length 0 (VMX hardware supplies the
+	 * fetching instruction's length there).
+	 */
+	if (rc == -EFAULT &&
+	    vcpu->run->exit_reason == KVM_EXIT_MEMORY_FAULT &&
+	    (vcpu->run->memory_fault.flags & KVM_MEMORY_EXIT_FLAG_PLANE_PROT)) {
+		unsigned int len = 0;
+
+		if (static_cpu_has(X86_FEATURE_DECODEASSISTS) &&
+		    svm->vmcb->control.insn_len &&
+		    x86_decode_emulated_instruction(vcpu, 0,
+						    svm->vmcb->control.insn_bytes,
+						    svm->vmcb->control.insn_len) ==
+								EMULATION_OK) {
+			struct x86_emulate_ctxt *ctxt = vcpu->arch.emulate_ctxt;
+
+			len = min_t(unsigned int, ctxt->_eip - ctxt->eip,
+				    sizeof(vcpu->run->memory_fault.insn_bytes));
+		}
+
+		vcpu->run->memory_fault.insn_len = len;
+		if (len) {
+			/*
+			 * Copy from the decoder's fetch cache, not from the
+			 * VMCB window: the decoder fetches any bytes beyond
+			 * the window from guest memory, so the decoded
+			 * length can exceed control.insn_len, and the
+			 * VMCB-sourced tail would be whatever an earlier
+			 * intercept left behind.
+			 */
+			memcpy(vcpu->run->memory_fault.insn_bytes,
+			       vcpu->arch.emulate_ctxt->fetch.data, len);
+			vcpu->run->memory_fault.flags |=
+				KVM_MEMORY_EXIT_FLAG_INSN_VALID;
+		}
+
+		/*
+		 * The reflected fault is delivered to the higher VTL like an
+		 * exception, and fault delivery gives the to-be-restarted
+		 * instruction RFLAGS.RF=1 (no instruction-breakpoint
+		 * re-trigger on the retry).  VMX saves the fault-adjusted RF
+		 * in guest RFLAGS on a fault-type exit and the TLFS message
+		 * carries that value; SVM saves RF as it was during
+		 * execution, so apply the fault adjustment by hand.
+		 *
+		 * Under NMI-window single-stepping the vmcb RFLAGS carries
+		 * KVM's own TF/RF, and the guest view lives in
+		 * nmi_singlestep_guest_rflags: set RF there too, or
+		 * svm_get_rflags() hides it from userspace and
+		 * disable_nmi_singlestep() wipes it from the vmcb when the
+		 * window closes.
+		 */
+		if (unlikely(svm->nmi_singlestep))
+			svm->nmi_singlestep_guest_rflags |= X86_EFLAGS_RF;
+		svm_set_rflags(vcpu, svm_get_rflags(vcpu) | X86_EFLAGS_RF);
+	}
+
 	if (rc > 0 && error_code & PFERR_GUEST_RMP_MASK)
 		sev_handle_rmp_fault(vcpu, gpa, error_code);
 
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 248d799b46e9..ec50e46404af 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -5987,9 +5987,7 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu)
 	 */
 	if (ret == -EFAULT &&
 	    vcpu->run->exit_reason == KVM_EXIT_MEMORY_FAULT &&
-	    (vcpu->run->memory_fault.flags & (KVM_MEMORY_EXIT_FLAG_READ |
-					      KVM_MEMORY_EXIT_FLAG_WRITE |
-					      KVM_MEMORY_EXIT_FLAG_EXEC))) {
+	    (vcpu->run->memory_fault.flags & KVM_MEMORY_EXIT_FLAG_PLANE_PROT)) {
 		unsigned int len;
 
 		if (exit_qualification & EPT_VIOLATION_GVA_IS_VALID) {
@@ -6007,7 +6005,6 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu)
 		len = vmcs_read32(VM_EXIT_INSTRUCTION_LEN);
 		len = min_t(unsigned int, len,
 			    sizeof(vcpu->run->memory_fault.insn_bytes));
-		vcpu->run->memory_fault.insn_len = len;
 		if (!(exit_qualification & EPT_VIOLATION_ACC_INSTR)) {
 			struct x86_exception e;
 
@@ -6019,10 +6016,16 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu)
 			 */
 			if (kvm_read_guest_virt(vcpu, kvm_get_linear_rip(vcpu),
 						vcpu->run->memory_fault.insn_bytes,
-						len, &e))
+						len, &e)) {
 				memset(vcpu->run->memory_fault.insn_bytes, 0,
 				       sizeof(vcpu->run->memory_fault.insn_bytes));
+				len = 0;
+			}
 		}
+		vcpu->run->memory_fault.insn_len = len;
+		if (len)
+			vcpu->run->memory_fault.flags |=
+				KVM_MEMORY_EXIT_FLAG_INSN_VALID;
 	}
 	return ret;
 }
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index bc576889c6f0..60923494eea5 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2755,6 +2755,18 @@ static inline void kvm_prepare_memory_fault_exit(struct kvm_vcpu *vcpu,
 	vcpu->run->memory_fault.gpa = gpa;
 	vcpu->run->memory_fault.size = size;
 
+	/*
+	 * Clear the extended fields: only the x86 plane-protection harvest
+	 * fills them in (with the matching validity flags), and a leftover
+	 * from an earlier exit would otherwise masquerade as this fault's
+	 * instruction.  All-zero insn_bytes is a valid x86 encoding, so
+	 * zeroing alone is not a distinguishable "not provided" marker.
+	 */
+	vcpu->run->memory_fault.gva = 0;
+	vcpu->run->memory_fault.insn_len = 0;
+	memset(vcpu->run->memory_fault.insn_bytes, 0,
+	       sizeof(vcpu->run->memory_fault.insn_bytes));
+
 	vcpu->run->memory_fault.flags = 0;
 
 	if (is_write)
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 8224ef335ba1..76d7c87172fd 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -481,10 +481,25 @@ struct kvm_run {
 #define KVM_MEMORY_EXIT_FLAG_EXEC       (1ULL << 2)
 #define KVM_MEMORY_EXIT_FLAG_PRIVATE	(1ULL << 3)
 #define KVM_MEMORY_EXIT_FLAG_GVA_VALID	(1ULL << 4)
+/* The fault was caused by the vCPU's plane's memory protections. */
+#define KVM_MEMORY_EXIT_FLAG_PLANE_PROT	(1ULL << 5)
+/*
+ * insn_len/insn_bytes are valid.  For an EXEC fault the faulting
+ * instruction itself cannot be read: insn_bytes stays zero, and
+ * insn_len is the length of the instruction which performed the
+ * fetch (reported by VMX hardware; not available on SVM).
+ */
+#define KVM_MEMORY_EXIT_FLAG_INSN_VALID	(1ULL << 6)
 			__u64 flags;
 			__u64 gpa;
 			__u64 size;
-			/* set on plane-protection faults when available */
+			/*
+			 * gva/insn_len/insn_bytes are set on plane-protection
+			 * faults when available, and are zero on every other
+			 * memory-fault exit.  They are enumerated, along with
+			 * the PLANE_PROT and INSN_VALID flags, by the plane
+			 * memory overlay capability.
+			 */
 			__u64 gva;
 			__u32 insn_len;
 			__u8  insn_bytes[16];
diff --git a/tools/testing/selftests/kvm/memory_attributes.c b/tools/testing/selftests/kvm/memory_attributes.c
index 63dec49c2888..17b5bc6bddda 100644
--- a/tools/testing/selftests/kvm/memory_attributes.c
+++ b/tools/testing/selftests/kvm/memory_attributes.c
@@ -198,7 +198,11 @@ static void test_page_restricted(struct kvm_vcpu *vcpu, int op,
 	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_MEMORY_FAULT);
 	TEST_ASSERT_EQ(vcpu->run->memory_fault.gpa, fault_paddr);
 	if (fault_reason)
-		TEST_ASSERT_EQ(vcpu->run->memory_fault.flags, fault_reason);
+		TEST_ASSERT_EQ(vcpu->run->memory_fault.flags &
+			       ~(KVM_MEMORY_EXIT_FLAG_GVA_VALID |
+				 KVM_MEMORY_EXIT_FLAG_PLANE_PROT |
+				 KVM_MEMORY_EXIT_FLAG_INSN_VALID),
+			       fault_reason);
 	TEST_ASSERT_EQ(vcpu->run->memory_fault.size, vm->page_size);
 }
 
-- 
2.50.1

[RFC PATCH 50/55] KVM: SVM: Trap descriptor-table loads for VTL register intercepts

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

A higher VTL arms register intercepts on IDTR/GDTR/LDTR/TR writes
(KVM_CAP_HYPERV_REG_INTERCEPT), and the emulator diverts an
intercepted load into a KVM_EXIT_HYPERV_REG_INTERCEPT exit instead of
applying it (hv_divert_dt_write()).  Only VMX armed the hardware exits
(SECONDARY_EXEC_DESC), so on AMD a VTL0 lidt/lgdt/lldt/ltr executed
unintercepted and the higher VTL never received its register intercept
messages: some twenty hyperv_vsm KUT register-intercept tests fail,
and a VTL1 that relies on the intercepts for kernel patch protection
silently loses them.

Arm the four VMCB descriptor-table load intercepts whenever
reg_intercept_control is non-zero, mirroring the all-or-nothing VMX
control, and route the exits into kvm_emulate_instruction(): a load
the divert check declines is applied by the emulator as before.
Store intercepts stay untrapped; the control has no bits for them.

The (dis)arming gets a dedicated kvm_x86_ops.update_hv_reg_intercepts()
op touching only the L1 controls, instead of riding on hooks with
lifecycles of their own: vmx_set_cr4() owned SECONDARY_EXEC_DESC for
UMIP emulation and a VTL0 set-then-clear of CR4.UMIP disarmed the very
lidt/lgdt trap hv_divert_dt_write() exists to catch; enabling the cap
through vcpu_after_set_cpuid() after an exit from L2 rewrote vmcs02
state that the next nested VM-exit threw away.  On VMX the op defers
while vmcs02 is loaded and nested_vmx_vmexit() refreshes vmcs01 after
switching back; on SVM it arms vmcb01, and
nested_vmcb02_recalc_intercepts() strips the LOAD_* bits from vmcb02
unless vmcb12 wants them, matching the VMX DESC strip.  The CR4 path
preserves the control while the intercepts are armed, and
hv_divert_dt_write() bails in guest mode so an L2 descriptor exit
which still reaches the emulator (UMIP emulation, for example) applies
the load normally.

Arming is refused with -EINVAL on SEV guests, failing KVM_ENABLE_CAP
before anything is armed (disarming still succeeds): the diverted
load needs full emulation, which encrypted guests cannot have.  On
SEV-ES desc_interception() hits svm_check_emulate_instruction()
returning X86EMUL_RETRY_INSTR, resuming at the same RIP with the
intercept still armed, so one descriptor-table load spins a vCPU (and
a host CPU) forever; on plain SEV the emulator has no instruction
bytes and injects #UD on an architecturally valid lgdt.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/include/asm/kvm-x86-ops.h |  1 +
 arch/x86/include/asm/kvm_host.h    |  7 ++++
 arch/x86/kvm/hyperv.h              | 16 ++++++++
 arch/x86/kvm/svm/nested.c          | 16 ++++++++
 arch/x86/kvm/svm/svm.c             | 66 ++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/main.c            | 10 +++++
 arch/x86/kvm/vmx/nested.c          |  1 +
 arch/x86/kvm/vmx/vmx.c             | 65 ++++++++++++++++++++++-------
 arch/x86/kvm/vmx/x86_ops.h         |  1 +
 arch/x86/kvm/x86.c                 | 19 ++++++---
 10 files changed, 183 insertions(+), 19 deletions(-)

diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
index 4f96090c04c9..428daf81f258 100644
--- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
@@ -140,6 +140,7 @@ KVM_X86_OP(get_feature_msr)
 KVM_X86_OP(check_emulate_instruction)
 KVM_X86_OP(apic_init_signal_blocked)
 KVM_X86_OP_OPTIONAL(enable_l2_tlb_flush)
+KVM_X86_OP(update_hv_reg_intercepts)
 KVM_X86_OP_OPTIONAL(migrate_timers)
 KVM_X86_OP(recalc_intercepts)
 KVM_X86_OP(complete_emulated_msr)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index e03ae6493641..cf36baceaabc 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -2109,6 +2109,13 @@ struct kvm_x86_ops {
 	bool (*apic_init_signal_blocked)(struct kvm_vcpu *vcpu);
 	int (*enable_l2_tlb_flush)(struct kvm_vcpu *vcpu);
 
+	/*
+	 * Bring the L1 (vmcs01/vmcb01) descriptor-table intercepts in line
+	 * with hv_vcpu->reg_intercept_control.  Never touches L2 controls;
+	 * safe to call at any point, including with a vCPU in guest mode.
+	 */
+	int (*update_hv_reg_intercepts)(struct kvm_vcpu *vcpu);
+
 	void (*migrate_timers)(struct kvm_vcpu *vcpu);
 	void (*recalc_intercepts)(struct kvm_vcpu *vcpu);
 	int (*complete_emulated_msr)(struct kvm_vcpu *vcpu, int err);
diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h
index 5ba2bb2e649b..810055c71301 100644
--- a/arch/x86/kvm/hyperv.h
+++ b/arch/x86/kvm/hyperv.h
@@ -237,6 +237,18 @@ static inline bool kvm_hv_vcpu_suspended(struct kvm_vcpu *vcpu)
 	       READ_ONCE(vcpu->arch.hyperv->suspended);
 }
 
+/*
+ * True when a higher VTL armed descriptor-table write intercepts
+ * (HvRegisterCrInterceptControl) on this vCPU, i.e. when the vendor code
+ * must trap the L1 guest's descriptor-table loads for hv_divert_dt_write().
+ */
+static inline bool kvm_hv_dt_intercepts_armed(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
+
+	return hv_vcpu && hv_vcpu->reg_intercept_control;
+}
+
 static inline bool kvm_hv_is_tlb_flush_hcall(struct kvm_vcpu *vcpu)
 {
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
@@ -328,6 +340,10 @@ static inline int kvm_hv_verify_vp_assist(struct kvm_vcpu *vcpu)
 {
 	return 0;
 }
+static inline bool kvm_hv_dt_intercepts_armed(struct kvm_vcpu *vcpu)
+{
+	return false;
+}
 static inline u32 kvm_hv_get_vpindex(struct kvm_vcpu *vcpu)
 {
 	return vcpu->vcpu_idx;
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index b340dc9991ad..09b57ea115b0 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -156,6 +156,22 @@ void nested_vmcb02_recalc_intercepts(struct vcpu_svm *svm)
 	for (i = 0; i < MAX_INTERCEPT; i++)
 		vmcb02->control.intercepts[i] |= vmcb12_ctrl->intercepts[i];
 
+	/*
+	 * KVM arms the descriptor-table load intercepts in vmcb01 to divert
+	 * L1's loads into Hyper-V register-intercept exits (see
+	 * hv_divert_dt_write()); they do not apply to L2.  Drop them from
+	 * vmcb02 unless L1 wants the intercept itself, matching the
+	 * SECONDARY_EXEC_DESC strip in VMX's prepare_vmcs02().
+	 */
+	if (!vmcb12_is_intercept(vmcb12_ctrl, INTERCEPT_LOAD_IDTR))
+		vmcb_clr_intercept(&vmcb02->control, INTERCEPT_LOAD_IDTR);
+	if (!vmcb12_is_intercept(vmcb12_ctrl, INTERCEPT_LOAD_GDTR))
+		vmcb_clr_intercept(&vmcb02->control, INTERCEPT_LOAD_GDTR);
+	if (!vmcb12_is_intercept(vmcb12_ctrl, INTERCEPT_LOAD_LDTR))
+		vmcb_clr_intercept(&vmcb02->control, INTERCEPT_LOAD_LDTR);
+	if (!vmcb12_is_intercept(vmcb12_ctrl, INTERCEPT_LOAD_TR))
+		vmcb_clr_intercept(&vmcb02->control, INTERCEPT_LOAD_TR);
+
 	/* If SMI is not intercepted, ignore guest SMI intercept as well  */
 	if (!intercept_smi)
 		vmcb_clr_intercept(&vmcb02->control, INTERCEPT_SMI);
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index a30f6f888b51..42a12f2ad612 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -2084,6 +2084,17 @@ static int npf_interception(struct kvm_vcpu *vcpu)
 	return rc;
 }
 
+static int desc_interception(struct kvm_vcpu *vcpu)
+{
+	/*
+	 * Descriptor-table write exits are armed only while a higher VTL
+	 * has register intercepts on them (reg_intercept_control); the
+	 * emulator diverts the write into a register-intercept exit via
+	 * hv_divert_dt_write() instead of applying it.
+	 */
+	return kvm_emulate_instruction(vcpu, 0);
+}
+
 static int db_interception(struct kvm_vcpu *vcpu)
 {
 	struct kvm_run *kvm_run = vcpu->run;
@@ -3365,6 +3376,10 @@ static int (*const svm_exit_handlers[])(struct kvm_vcpu *vcpu) = {
 	[SVM_EXIT_WRITE_CR3]			= cr_interception,
 	[SVM_EXIT_WRITE_CR4]			= cr_interception,
 	[SVM_EXIT_WRITE_CR8]			= cr8_write_interception,
+	[SVM_EXIT_IDTR_WRITE]			= desc_interception,
+	[SVM_EXIT_GDTR_WRITE]			= desc_interception,
+	[SVM_EXIT_LDTR_WRITE]			= desc_interception,
+	[SVM_EXIT_TR_WRITE]			= desc_interception,
 	[SVM_EXIT_READ_DR0]			= dr_interception,
 	[SVM_EXIT_READ_DR1]			= dr_interception,
 	[SVM_EXIT_READ_DR2]			= dr_interception,
@@ -4758,6 +4773,49 @@ static bool svm_has_emulated_msr(struct kvm *kvm, u32 index)
 	return true;
 }
 
+static int svm_update_hv_reg_intercepts(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_svm *svm = to_svm(vcpu);
+
+	/*
+	 * Trap descriptor-table loads while a higher VTL has register
+	 * intercepts armed on them (reg_intercept_control), and stop
+	 * trapping when they are disarmed; the emulator diverts a trapped
+	 * load into a register-intercept exit (hv_divert_dt_write()).
+	 * VMX arms the all-or-nothing SECONDARY_EXEC_DESC; the SVM
+	 * intercepts are per-register but armed as a block to match, and
+	 * an exit the emulator does not divert applies the load normally.
+	 *
+	 * The intercepts are armed in vmcb01 only: an L2 descriptor load
+	 * is never diverted, and nested_vmcb02_recalc_intercepts() keeps
+	 * these bits out of vmcb02 unless L1 asked for them itself.
+	 */
+	if (kvm_hv_dt_intercepts_armed(vcpu)) {
+		/*
+		 * The diverted load needs full emulation, which encrypted
+		 * guests cannot have: on SEV-ES and SNP the emulator cannot
+		 * read the registers, so svm_check_emulate_instruction()
+		 * retries at the same RIP with the intercept still armed and
+		 * every descriptor-table load livelocks the vCPU; on plain
+		 * SEV there are no instruction bytes to decode and a valid
+		 * lgdt takes a spurious #UD.  Refuse to arm.
+		 */
+		if (is_sev_guest(vcpu))
+			return -EINVAL;
+
+		svm_set_intercept(svm, INTERCEPT_LOAD_IDTR);
+		svm_set_intercept(svm, INTERCEPT_LOAD_GDTR);
+		svm_set_intercept(svm, INTERCEPT_LOAD_LDTR);
+		svm_set_intercept(svm, INTERCEPT_LOAD_TR);
+	} else {
+		svm_clr_intercept(svm, INTERCEPT_LOAD_IDTR);
+		svm_clr_intercept(svm, INTERCEPT_LOAD_GDTR);
+		svm_clr_intercept(svm, INTERCEPT_LOAD_LDTR);
+		svm_clr_intercept(svm, INTERCEPT_LOAD_TR);
+	}
+	return 0;
+}
+
 static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
@@ -4784,6 +4842,13 @@ static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
 	if (guest_cpuid_is_intel_compatible(vcpu))
 		guest_cpu_cap_clear(vcpu->common, X86_FEATURE_V_VMSAVE_VMLOAD);
 
+	/*
+	 * Re-arm the descriptor-table intercepts; setting guest CPUID does
+	 * not change reg_intercept_control, but keep the controls in sync
+	 * the same way vmx_vcpu_after_set_cpuid() does.
+	 */
+	svm_update_hv_reg_intercepts(vcpu);
+
 	if (is_sev_guest(vcpu))
 		sev_vcpu_after_set_cpuid(svm);
 }
@@ -5538,6 +5603,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = {
 	.apic_init_signal_blocked = svm_apic_init_signal_blocked,
 
 	.recalc_intercepts = svm_recalc_intercepts,
+	.update_hv_reg_intercepts = svm_update_hv_reg_intercepts,
 	.complete_emulated_msr = svm_complete_emulated_msr,
 
 	.vcpu_deliver_sipi_vector = svm_vcpu_deliver_sipi_vector,
diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c
index 81a8d398068b..24663bfc14b8 100644
--- a/arch/x86/kvm/vmx/main.c
+++ b/arch/x86/kvm/vmx/main.c
@@ -207,6 +207,15 @@ static void vt_recalc_intercepts(struct kvm_vcpu *vcpu)
 	vmx_recalc_intercepts(vcpu);
 }
 
+static int vt_update_hv_reg_intercepts(struct kvm_vcpu *vcpu)
+{
+	/* TDX doesn't allow descriptor-table loads to be intercepted. */
+	if (is_td_vcpu(vcpu))
+		return -EINVAL;
+
+	return vmx_update_hv_reg_intercepts(vcpu);
+}
+
 static int vt_complete_emulated_msr(struct kvm_vcpu *vcpu, int err)
 {
 	if (is_td_vcpu(vcpu))
@@ -1018,6 +1027,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 
 	.check_emulate_instruction = vt_op(check_emulate_instruction),
 	.apic_init_signal_blocked = vt_op(apic_init_signal_blocked),
+	.update_hv_reg_intercepts = vt_op(update_hv_reg_intercepts),
 	.migrate_timers = vmx_migrate_timers,
 
 	.recalc_intercepts = vt_op(recalc_intercepts),
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index 3fe88f29be7a..de5d72502e70 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -5142,6 +5142,7 @@ void __nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,
 	 * Update any VMCS fields that might have changed while vmcs02 was the
 	 * active VMCS.  The tracking is per-vCPU, not per-VMCS.
 	 */
+	vmx_update_hv_reg_intercepts(vcpu);
 	vmcs_write32(VM_EXIT_MSR_STORE_COUNT, vmx->msr_autostore.nr);
 	vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, vmx->msr_autoload.host.nr);
 	vmcs_write32(VM_ENTRY_MSR_LOAD_COUNT, vmx->msr_autoload.guest.nr);
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index ec50e46404af..bc5b08ebaefb 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -3535,6 +3535,42 @@ bool vmx_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
 	return true;
 }
 
+/*
+ * SECONDARY_EXEC_DESC has two consumers: UMIP emulation (descriptor-table
+ * reads must #GP when CR4.UMIP is set and the CPU cannot do it) and the
+ * Hyper-V VTL register intercepts (descriptor-table writes divert into
+ * KVM_EXIT_HYPERV_REG_INTERCEPT, see hv_divert_dt_write()).  Every arm or
+ * disarm of the control for one consumer must preserve the other's claim.
+ */
+static bool vmx_need_desc_exiting(struct kvm_vcpu *vcpu)
+{
+	return (vmx_umip_emulated() &&
+		kvm_is_cr4_bit_set(vcpu, X86_CR4_UMIP)) ||
+	       kvm_hv_dt_intercepts_armed(vcpu);
+}
+
+int vmx_update_hv_reg_intercepts(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_vmx *vmx = to_vmx(vcpu);
+
+	if (!(vmcs_config.cpu_based_2nd_exec_ctrl & SECONDARY_EXEC_DESC))
+		return 0;
+
+	/*
+	 * Only vmcs01 carries the register intercepts: prepare_vmcs02()
+	 * strips SECONDARY_EXEC_DESC unless vmcs12 or L2's UMIP emulation
+	 * wants it, so an L2 descriptor load is never diverted.  While
+	 * vmcs02 is loaded, leave it alone; nested_vmx_vmexit() refreshes
+	 * vmcs01 after switching back.
+	 */
+	if (vmx->loaded_vmcs != &vmx->vmcs01)
+		return 0;
+
+	secondary_exec_controls_changebit(vmx, SECONDARY_EXEC_DESC,
+					  vmx_need_desc_exiting(vcpu));
+	return 0;
+}
+
 void vmx_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
 {
 	unsigned long old_cr4 = kvm_read_cr4(vcpu);
@@ -3558,8 +3594,16 @@ void vmx_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
 		if (cr4 & X86_CR4_UMIP) {
 			secondary_exec_controls_setbit(vmx, SECONDARY_EXEC_DESC);
 			hw_cr4 &= ~X86_CR4_UMIP;
-		} else if (!is_guest_mode(vcpu) ||
-			!nested_cpu_has2(get_vmcs12(vcpu), SECONDARY_EXEC_DESC)) {
+		} else if (is_guest_mode(vcpu)) {
+			if (!nested_cpu_has2(get_vmcs12(vcpu), SECONDARY_EXEC_DESC))
+				secondary_exec_controls_clearbit(vmx,
+								 SECONDARY_EXEC_DESC);
+		} else if (!kvm_hv_dt_intercepts_armed(vcpu)) {
+			/*
+			 * Clearing CR4.UMIP must not disarm the control
+			 * while a higher VTL holds register intercepts on
+			 * the descriptor tables.
+			 */
 			secondary_exec_controls_clearbit(vmx, SECONDARY_EXEC_DESC);
 		}
 	}
@@ -5648,7 +5692,7 @@ static int handle_desc(struct kvm_vcpu *vcpu)
 	 * SECONDARY_EXEC_DESC control.
 	 */
 	WARN_ON_ONCE(!kvm_is_cr4_bit_set(vcpu, X86_CR4_UMIP) &&
-		     !(to_hv_vcpu(vcpu) && to_hv_vcpu(vcpu)->reg_intercept_control));
+		     !kvm_hv_dt_intercepts_armed(vcpu));
 	return kvm_emulate_instruction(vcpu, 0);
 }
 
@@ -8105,18 +8149,11 @@ void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
 	vmx_update_exception_bitmap(vcpu);
 
 	/*
-	 * Trap descriptor-table instructions when a higher VTL armed
-	 * register intercepts on them, and stop trapping when the
-	 * intercepts are disarmed.  Preserve the control when UMIP
-	 * emulation needs it (CR4.UMIP owns it then).
+	 * Setting the secondary exec controls above recomputed them from
+	 * scratch; re-arm descriptor-table exiting for the VTL register
+	 * intercepts and UMIP emulation.
 	 */
-	if (vmcs_config.cpu_based_2nd_exec_ctrl & SECONDARY_EXEC_DESC) {
-		if (to_hv_vcpu(vcpu) && to_hv_vcpu(vcpu)->reg_intercept_control)
-			secondary_exec_controls_setbit(vmx, SECONDARY_EXEC_DESC);
-		else if (!(vmx_umip_emulated() &&
-			   kvm_is_cr4_bit_set(vcpu, X86_CR4_UMIP)))
-			secondary_exec_controls_clearbit(vmx, SECONDARY_EXEC_DESC);
-	}
+	vmx_update_hv_reg_intercepts(vcpu);
 }
 
 static __init u64 vmx_get_perf_capabilities(void)
diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h
index b75dfe7f039d..10db312faa8e 100644
--- a/arch/x86/kvm/vmx/x86_ops.h
+++ b/arch/x86/kvm/vmx/x86_ops.h
@@ -53,6 +53,7 @@ void vmx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,
 void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu);
 bool vmx_has_emulated_msr(struct kvm *kvm, u32 index);
 void vmx_recalc_intercepts(struct kvm_vcpu *vcpu);
+int vmx_update_hv_reg_intercepts(struct kvm_vcpu *vcpu);
 void vmx_prepare_switch_to_guest(struct kvm_vcpu *vcpu);
 void vmx_update_exception_bitmap(struct kvm_vcpu *vcpu);
 int vmx_get_feature_msr(u32 msr, u64 *data);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index e2e54f06b092..5190aa7025e3 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -6222,9 +6222,16 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
 			return r;
 		hv_vcpu = to_hv_vcpu(vcpu);
 		hv_vcpu->reg_intercept_control = cap->args[0];
-		/* Refresh the descriptor-table exiting controls. */
-		kvm_x86_call(vcpu_after_set_cpuid)(vcpu);
-		return 0;
+		/*
+		 * Refresh the descriptor-table exiting controls.  The vendor
+		 * op touches only the L1 controls, so it is safe here even
+		 * when the vCPU last exited from a nested guest and
+		 * vcpu_load() restored vmcs02/vmcb02.
+		 */
+		r = kvm_x86_call(update_hv_reg_intercepts)(vcpu);
+		if (r)
+			hv_vcpu->reg_intercept_control = 0;
+		return r;
 	}
 	case KVM_CAP_HYPERV_DIRECT_TLBFLUSH:
 		if (!kvm_x86_ops.enable_l2_tlb_flush)
@@ -8900,7 +8907,9 @@ static void emulator_get_idt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt)
 /*
  * Divert a descriptor-table write into a register-intercept exit when the
  * higher VTL armed one: the write is not applied, and userspace delivers
- * the TLFS register-intercept message to the higher VTL.
+ * the TLFS register-intercept message to the higher VTL.  The intercept
+ * contract covers only the L1 guest; a descriptor load emulated on behalf
+ * of a nested (L2) guest, however its exit arrived, is applied normally.
  */
 static bool hv_divert_dt_write(struct x86_emulate_ctxt *ctxt, u32 hv_reg,
 			       const struct desc_ptr *dt, u64 ctl_bit)
@@ -8909,7 +8918,7 @@ static bool hv_divert_dt_write(struct x86_emulate_ctxt *ctxt, u32 hv_reg,
 	struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt);
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
 
-	if (!hv_vcpu || vcpu->plane_level ||
+	if (!hv_vcpu || vcpu->plane_level || is_guest_mode(vcpu) ||
 	    !(hv_vcpu->reg_intercept_control & ctl_bit))
 		return false;
 
-- 
2.50.1

[RFC PATCH 51/55] KVM: SVM: Keep TPR[3:0] across exits without a guest CR8 write

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

V_TPR holds only TPR[7:4], and sync_cr8_to_lapic() pushed it back into
the APIC after every VMRUN.  A guest that sets TASKPRI through the
APIC page or MSR loses the low nibble at the next exit: VTL1 in the
hyperv_vsm KUT writes 0xff and reads back 0xf0 ("VTL1 TASKPRI write
sticks"), and interrupts in the sub-class the guest meant to mask are
let through.  VMX keeps all eight TPR bits in the virtual-APIC page
and does not have this problem.

Hardware changes V_TPR only on a guest CR8 write.  Record the value
sync_lapic_to_cr8() loads on entry and push V_TPR back into the APIC
only when it differs, i.e. only when the guest actually wrote CR8.  A
CR8 write whose value matches the loaded TPR[7:4] is indistinguishable
from no write and now preserves TPR[3:0], where architecturally it
would clear it; the previous behaviour cleared TPR[3:0] on every exit
regardless.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/svm/svm.c | 11 ++++++++++-
 arch/x86/kvm/svm/svm.h |  9 +++++++++
 2 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 42a12f2ad612..46ab1f8e330f 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -4320,7 +4320,15 @@ static inline void sync_cr8_to_lapic(struct kvm_vcpu *vcpu)
 
 	if (!svm_is_intercept(svm, INTERCEPT_CR8_WRITE)) {
 		int cr8 = svm->vmcb->control.int_ctl & V_TPR_MASK;
-		kvm_set_cr8(vcpu, cr8);
+
+		/*
+		 * Hardware changes V_TPR only on a guest CR8 write.  Pushing
+		 * an unchanged value into the APIC would zero TPR[3:0] (CR8
+		 * maps TPR[7:4] alone), wiping the low nibble of a TASKPRI
+		 * value the guest set through the APIC page or MSR.
+		 */
+		if (cr8 != svm->v_tpr_entry)
+			kvm_set_cr8(vcpu, cr8);
 	}
 }
 
@@ -4335,6 +4343,7 @@ static inline void sync_lapic_to_cr8(struct kvm_vcpu *vcpu)
 	cr8 = kvm_get_cr8(vcpu);
 	svm->vmcb->control.int_ctl &= ~V_TPR_MASK;
 	svm->vmcb->control.int_ctl |= cr8 & V_TPR_MASK;
+	svm->v_tpr_entry = cr8 & V_TPR_MASK;
 }
 
 static void svm_complete_soft_interrupt(struct kvm_vcpu *vcpu, u8 vector,
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 64438317d148..746a2ae98c4c 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -316,6 +316,15 @@ struct vcpu_svm {
 
 	void *msrpm;
 
+	/*
+	 * V_TPR value written by sync_lapic_to_cr8() on the last entry.
+	 * sync_cr8_to_lapic() pushes V_TPR back into the APIC only when it
+	 * differs, i.e. when the guest wrote CR8 in guest mode: V_TPR holds
+	 * TPR[7:4] only, so an unconditional back-sync would zero TPR[3:0]
+	 * after every exit, losing the low nibble of an APIC TASKPRI write.
+	 */
+	u8 v_tpr_entry;
+
 	ulong nmi_iret_rip;
 
 	struct svm_nested_state nested;
-- 
2.50.1

[RFC PATCH 52/55] KVM: selftests: Add Hyper-V reference TSC page memory attribute test

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The reference TSC page is written by the host, not by the guest, so
the guest-side memory attribute tests never cover it.  Add a test
that a page at an unprotected gfn is maintained as usual, that
pointing HV_X64_MSR_REFERENCE_TSC at a gfn carrying
KVM_MEMORY_ATTRIBUTE_NW leaves the page contents untouched — on the
MSR write itself and on a subsequent KVM_SET_CLOCK-forced update —
and that re-pointing the MSR at a writable page recovers.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/x86/hyperv_tsc_page_attrs.c | 129 ++++++++++++++++++
 2 files changed, 130 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/hyperv_tsc_page_attrs.c

diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 47c7a4c732eb..6edeb6693375 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -89,6 +89,7 @@ TEST_GEN_PROGS_x86 += x86/hyperv_forward_hcall
 TEST_GEN_PROGS_x86 += x86/hyperv_ipi
 TEST_GEN_PROGS_x86 += x86/hyperv_svm_test
 TEST_GEN_PROGS_x86 += x86/hyperv_tlb_flush
+TEST_GEN_PROGS_x86 += x86/hyperv_tsc_page_attrs
 TEST_GEN_PROGS_x86 += x86/kvm_clock_test
 TEST_GEN_PROGS_x86 += x86/kvm_pv_test
 TEST_GEN_PROGS_x86 += x86/kvm_buslock_test
diff --git a/tools/testing/selftests/kvm/x86/hyperv_tsc_page_attrs.c b/tools/testing/selftests/kvm/x86/hyperv_tsc_page_attrs.c
new file mode 100644
index 000000000000..efa47ae69780
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/hyperv_tsc_page_attrs.c
@@ -0,0 +1,129 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Test that KVM honours plane memory attributes when it maintains the
+ * Hyper-V reference TSC page.  A page whose gfn carries
+ * KVM_MEMORY_ATTRIBUTE_NW in the owning plane's attributes must never
+ * be written by the host, on the HV_X64_MSR_REFERENCE_TSC write or on
+ * any later masterclock update; an unprotected page keeps working, and
+ * re-pointing the MSR at a writable page recovers.
+ */
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+#include "hyperv.h"
+
+struct ms_hyperv_tsc_page {
+	volatile u32 tsc_sequence;
+	u32 reserved1;
+	volatile u64 tsc_scale;
+	volatile s64 tsc_offset;
+} __packed;
+
+#define SENTINEL_SEQ	0xdeadbeef
+#define SENTINEL_U64	0xcafef00dcafef00dULL
+
+static void guest_main(struct ms_hyperv_tsc_page *page_a, gpa_t page_a_gpa,
+		       struct ms_hyperv_tsc_page *page_b, gpa_t page_b_gpa)
+{
+	/* Set Guest OS id to enable Hyper-V emulation */
+	wrmsr(HV_X64_MSR_GUEST_OS_ID, HYPERV_LINUX_OS_ID);
+
+	/* An unprotected page is maintained by KVM as usual. */
+	wrmsr(HV_X64_MSR_REFERENCE_TSC, page_a_gpa | 0x1);
+	GUEST_ASSERT(page_a->tsc_sequence != 0);
+
+	/* Fill the page about to be protected, while it is still writable. */
+	page_b->tsc_sequence = SENTINEL_SEQ;
+	page_b->tsc_scale = SENTINEL_U64;
+	page_b->tsc_offset = SENTINEL_U64;
+
+	GUEST_SYNC(1);	/* host sets KVM_MEMORY_ATTRIBUTE_NW on page_b */
+
+	/*
+	 * Pointing the reference TSC MSR at the protected page must not
+	 * have KVM write to it: not on the MSR write itself, and not on
+	 * the masterclock update the write requests.
+	 */
+	wrmsr(HV_X64_MSR_REFERENCE_TSC, page_b_gpa | 0x1);
+	GUEST_ASSERT(page_b->tsc_sequence == SENTINEL_SEQ);
+	GUEST_ASSERT(page_b->tsc_scale == SENTINEL_U64);
+	GUEST_ASSERT(page_b->tsc_offset == SENTINEL_U64);
+
+	GUEST_SYNC(2);	/* host resets kvmclock, forcing a TSC page update */
+
+	GUEST_ASSERT(page_b->tsc_sequence == SENTINEL_SEQ);
+	GUEST_ASSERT(page_b->tsc_scale == SENTINEL_U64);
+	GUEST_ASSERT(page_b->tsc_offset == SENTINEL_U64);
+
+	/* Re-pointing the MSR at a writable page recovers. */
+	wrmsr(HV_X64_MSR_REFERENCE_TSC, page_a_gpa | 0x1);
+	GUEST_ASSERT(page_a->tsc_sequence != 0);
+
+	GUEST_DONE();
+}
+
+int main(void)
+{
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	struct ucall uc;
+	gva_t page_a_gva, page_b_gva;
+	gpa_t page_b_gpa;
+	int stage;
+
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_HYPERV_TIME));
+	TEST_REQUIRE(kvm_check_cap(KVM_CAP_MEMORY_ATTRIBUTES) &
+		     KVM_MEMORY_ATTRIBUTE_NW);
+	TEST_REQUIRE(sys_clocksource_is_based_on_tsc());
+
+	vm = vm_create_with_one_vcpu(&vcpu, guest_main);
+
+	vcpu_set_hv_cpuid(vcpu);
+
+	page_a_gva = vm_alloc_page(vm);
+	page_b_gva = vm_alloc_page(vm);
+	memset(addr_gva2hva(vm, page_a_gva), 0x0, getpagesize());
+	memset(addr_gva2hva(vm, page_b_gva), 0x0, getpagesize());
+	page_b_gpa = addr_gva2gpa(vm, page_b_gva);
+
+	vcpu_args_set(vcpu, 4, page_a_gva, addr_gva2gpa(vm, page_a_gva),
+		      page_b_gva, page_b_gpa);
+
+	for (stage = 1;; stage++) {
+		vcpu_run(vcpu);
+		TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+
+		switch (get_ucall(vcpu, &uc)) {
+		case UCALL_ABORT:
+			REPORT_GUEST_ASSERT(uc);
+			/* NOT REACHED */
+		case UCALL_SYNC:
+			break;
+		case UCALL_DONE:
+			/* Keep in sync with guest_main() */
+			TEST_ASSERT(stage == 3, "Testing ended prematurely, stage %d",
+				    stage);
+			goto out;
+		default:
+			TEST_FAIL("Unknown ucall %lu", uc.cmd);
+		}
+
+		TEST_ASSERT(uc.args[1] == stage,
+			    "Stage %d: Unexpected sync, got %lx",
+			    stage, (ulong)uc.args[1]);
+
+		if (stage == 1)
+			vm_set_memory_attributes(vm, page_b_gpa, getpagesize(),
+						 KVM_MEMORY_ATTRIBUTE_NW);
+
+		/* Reset kvmclock triggering TSC page update */
+		if (stage == 2) {
+			struct kvm_clock_data clock = {0};
+
+			vm_ioctl(vm, KVM_SET_CLOCK, &clock);
+		}
+	}
+
+out:
+	kvm_vm_free(vm);
+}
-- 
2.50.1

[RFC PATCH 53/55] KVM: SVM: Hedge the v_tpr_entry comment

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The comment claimed V_TPR "differs, i.e. when the guest wrote CR8":
differing implies a write, but a write does not imply differing. A
same-value CR8 write is invisible to the back-sync, so TPR[3:0]
survives where the architectural write would clear it — a documented
tradeoff of commit f34c595cb8a5 ("KVM: SVM: Keep TPR[3:0] across
exits without a guest CR8 write"), which deserves stating at the
field rather than only in that commit's message. No functional
change.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/svm/svm.h | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 746a2ae98c4c..24d34697fe65 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -319,9 +319,13 @@ struct vcpu_svm {
 	/*
 	 * V_TPR value written by sync_lapic_to_cr8() on the last entry.
 	 * sync_cr8_to_lapic() pushes V_TPR back into the APIC only when it
-	 * differs, i.e. when the guest wrote CR8 in guest mode: V_TPR holds
-	 * TPR[7:4] only, so an unconditional back-sync would zero TPR[3:0]
-	 * after every exit, losing the low nibble of an APIC TASKPRI write.
+	 * differs: V_TPR holds TPR[7:4] only, so an unconditional back-sync
+	 * would zero TPR[3:0] after every exit, losing the low nibble of an
+	 * APIC TASKPRI write.  A differing V_TPR implies a guest CR8 write,
+	 * but not the converse: a CR8 write of the same value is
+	 * indistinguishable from no write, so TPR[3:0] survives it where
+	 * the architectural write (and the intercepted path through
+	 * kvm_lapic_set_tpr()) would clear it.
 	 */
 	u8 v_tpr_entry;
 
-- 
2.50.1

[RFC PATCH 54/55] KVM: Check every plane's protections before caching a gfn->hva

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The gfn_to_hva_cache init and the pfncache refresh refuse to cache a
range carrying NR/NW attributes, but consulted only plane 0's array.
Cached accesses made in vCPU context for a non-zero plane vCPU (steal
time, PV EOI, pvclock) therefore enforced plane 0's protections only: a
plane-N guest could point a PV MSR at a gfn its own plane marked NW and
kvm_write_guest_offset_cached() wrote through it, while the uncached
paths correctly use the vCPU's own plane.

A cached translation has no plane context at use time, so refuse the
cache when ANY plane protects the range.  The attribute generation that
invalidates these caches is already partition-wide, so updates on any
plane force a re-check.  The plane array walk follows the existing
lock-free kvm->planes[] reader pattern (acquire paired with the
publishing release).

Fixes: a407d03e5697 ("KVM: Take memory protections into account for accesses with cached gfn->hva")
Fixes: dc506e2ff987 ("KVM: pfncache: Fail to refresh if it contains memory protections")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 include/linux/kvm_host.h |  8 ++++++++
 virt/kvm/kvm_main.c      | 31 ++++++++++++++++++++++++++++++-
 virt/kvm/pfncache.c      |  3 ++-
 3 files changed, 40 insertions(+), 2 deletions(-)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 60923494eea5..6a1ac96d64ba 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2846,6 +2846,8 @@ bool kvm_plane_range_has_any_attrs(struct kvm_plane *plane, gfn_t start,
 				   gfn_t end);
 bool kvm_range_has_any_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 					unsigned long mask);
+bool kvm_any_plane_range_has_rw_memory_protections(struct kvm *kvm,
+						   gfn_t start, gfn_t end);
 bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
 					struct kvm_gfn_range *range);
 bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
@@ -2889,6 +2891,12 @@ static inline bool kvm_range_has_any_memory_attributes(struct kvm *kvm,
 {
 	return false;
 }
+static inline bool kvm_any_plane_range_has_rw_memory_protections(struct kvm *kvm,
+								 gfn_t start,
+								 gfn_t end)
+{
+	return false;
+}
 static inline u64 kvm_mem_attributes_generation(struct kvm *kvm)
 {
 	return 0;
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 1c70bb70500c..f0b474b0547d 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2779,6 +2779,34 @@ static bool kvm_plane_range_has_memory_attributes(struct kvm_plane *plane,
 	return false;
 }
 
+/*
+ * Returns true if any plane's attributes deny read or write anywhere in
+ * [@start, @end).  Cached gfn->hva translations carry no plane context
+ * at use time, so a cache must be refused when ANY plane protects the
+ * range, not just plane 0.
+ */
+bool kvm_any_plane_range_has_rw_memory_protections(struct kvm *kvm,
+						   gfn_t start, gfn_t end)
+{
+	unsigned long mask = KVM_MEMORY_ATTRIBUTE_NR | KVM_MEMORY_ATTRIBUTE_NW;
+	int i;
+
+	if (kvm_range_has_rw_memory_protections(kvm, start, end))
+		return true;
+
+	for (i = 1; i < KVM_MAX_PLANES; i++) {
+		/* Pairs with the release in kvm_create_plane(). */
+		struct kvm_plane *plane = smp_load_acquire(&kvm->planes[i]);
+
+		if (plane &&
+		    kvm_plane_range_has_memory_attributes(plane, start, end,
+							  mask))
+			return true;
+	}
+
+	return false;
+}
+
 static bool kvm_range_memory_attributes_need_sync(struct kvm_plane *plane,
 						  gfn_t start, gfn_t end,
 						  unsigned long attributes)
@@ -3939,7 +3967,8 @@ static int __kvm_gfn_to_hva_cache_init(struct kvm *kvm,
 	 * check has to happen after storing the generation number.
 	 */
 	ghc->attrs_generation = kvm_mem_attributes_generation(kvm);
-	if (kvm_range_has_rw_memory_protections(kvm, start_gfn, end_gfn + 1)) {
+	if (kvm_any_plane_range_has_rw_memory_protections(kvm, start_gfn,
+							  end_gfn + 1)) {
 		ghc->hva = KVM_HVA_ERR_BAD;
 		return -EFAULT;
 	}
diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c
index 39935740136d..6ad1c62e9cd8 100644
--- a/virt/kvm/pfncache.c
+++ b/virt/kvm/pfncache.c
@@ -304,7 +304,8 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l
 			 * generation number.
 			 */
 			gpc->attrs_generation = kvm_mem_attributes_generation(kvm);
-			if (kvm_range_has_rw_memory_protections(kvm, gfn, gfn + 1)) {
+			if (kvm_any_plane_range_has_rw_memory_protections(kvm, gfn,
+									  gfn + 1)) {
 				gpc->uhva = KVM_HVA_ERR_BAD;
 				ret = -EFAULT;
 				goto out;
-- 
2.50.1

[RFC PATCH 55/55] KVM: planes: Annotate the lockless twin-publication protocol

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Plane twins are published with a plain store to common->vcpus[] plus an
smp_wmb() that pairs with online_vcpus, not with twin initialisation,
while the lockless readers grown across the series (the twin-kick
paths, the Hyper-V TLB-flush and stimer twin lookups, the SEND_IPI
target lookup, vcpu_for_each_plane) read the slots with a mixture of
READ_ONCE and bare loads.  x86-TSO hides the lot, but the protocol is
unannotated generic code: KCSAN reports it, and nothing stops a
reordered publication from exposing a half-initialised twin to a kick.

Publish the slot with smp_store_release(), make every lockless reader
use READ_ONCE (vcpu_for_each_plane included), and document the pairing
at the iterator.  Mutex-serialised lookups (the in-kernel VTL switch
runs under the shared vCPU mutex that twin creation also takes) stay
plain.  Also document KVM_REQUEST_PER_PLANE's two scopes at its
definition: the fan-out to twins happens in kvm_make_vcpu_request(),
while a direct kvm_make_request() marks only the twin it was given - a
distinction a future direct-raise caller would otherwise trip over.

No functional change on x86.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
 arch/x86/kvm/hyperv.c    |  6 +++---
 include/linux/kvm_host.h | 13 ++++++++++++-
 virt/kvm/kvm_main.c      |  8 +++++++-
 3 files changed, 22 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index dbe1b83c1f28..8abc61f46608 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -612,7 +612,7 @@ static void stimer_mark_pending(struct kvm_vcpu_hv_stimer *stimer,
 		int p;
 
 		for (p = 0; p < vcpu->plane_level; p++) {
-			twin = vcpu->common->vcpus[p];
+			twin = READ_ONCE(vcpu->common->vcpus[p]);
 			if (twin) {
 				kvm_make_request(KVM_REQ_UNBLOCK, twin);
 				kvm_vcpu_kick(twin);
@@ -2370,7 +2370,7 @@ static void hv_tlb_flush_enqueue_all_planes(struct kvm_vcpu *v,
 	int p;
 
 	for (p = 0; p < KVM_MAX_PLANES; p++) {
-		twin = v->common->vcpus[p];
+		twin = READ_ONCE(v->common->vcpus[p]);
 		if (!twin || !to_hv_vcpu(twin))
 			continue;
 		fifo = kvm_hv_get_tlb_flush_fifo(twin, false);
@@ -2601,7 +2601,7 @@ static void kvm_hv_send_ipi_to_many(struct kvm *kvm, u32 vector,
 		 * VTL): deliver into that plane's APIC.
 		 */
 		if (target_plane) {
-			target = vcpu->common->vcpus[target_plane];
+			target = READ_ONCE(vcpu->common->vcpus[target_plane]);
 			if (!target)
 				continue;
 		}
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 6a1ac96d64ba..4d98de4c7523 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -159,6 +159,12 @@ static inline bool kvm_is_error_gpa(gpa_t gpa)
 #define KVM_REQUEST_NO_WAKEUP      BIT(8)
 #define KVM_REQUEST_WAIT           BIT(9)
 #define KVM_REQUEST_NO_ACTION      BIT(10)
+/*
+ * A KVM_REQUEST_PER_PLANE request has two scopes: raised through
+ * kvm_make_vcpu_request() (kicks, mass requests) it fans out to every
+ * plane twin of the vCPU, while a direct kvm_make_request() marks only
+ * the twin it was given.  A direct caller must pick the twin it means.
+ */
 #define KVM_REQUEST_PER_PLANE      BIT(11)
 /*
  * Architecture-independent vcpu->requests bit members
@@ -400,9 +406,14 @@ struct kvm_vcpu_common {
 	struct kvm_vcpu_arch_common arch;
 };
 
+/*
+ * The slot read pairs with the smp_store_release() publication in
+ * kvm_vcpu_finish_common() for iterators running outside the locks that
+ * serialise twin creation.
+ */
 #define vcpu_for_each_plane(common, i, v)			\
 	for ((i) = 0; (i) < KVM_MAX_PLANES; ++(i))		\
-		if (((v) = common->vcpus[(i)]) != NULL)
+		if (((v) = READ_ONCE(common->vcpus[(i)])) != NULL)
 
 /* Tracked per plane-VCPU - used for deciding which plane-vcpu to run */
 enum kvm_vcpu_state {
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index f0b474b0547d..6031ba9a6667 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -563,7 +563,13 @@ static int kvm_vcpu_init_common(struct kvm_vcpu *vcpu, struct kvm *kvm, unsigned
 static void kvm_vcpu_finish_common(struct kvm_vcpu *vcpu)
 {
 	WARN_ON(vcpu->common->vcpus[vcpu->plane_level] != NULL);
-	vcpu->common->vcpus[vcpu->plane_level] = vcpu;
+	/*
+	 * Publish with release semantics so the lockless twin readers
+	 * (the kick paths, vcpu_for_each_plane, the hyperv twin lookups)
+	 * observe a fully initialised vCPU through their READ_ONCE of the
+	 * slot; the plain store relied on x86-TSO.
+	 */
+	smp_store_release(&vcpu->common->vcpus[vcpu->plane_level], vcpu);
 	smp_wmb();
 	if (vcpu->plane_level == 0) {
 		/*
-- 
2.50.1

Re: [RFC PATCH 00/55] KVM: x86: Hyper-V VSM (Virtual Trust Levels) on VM planes

[Sriram Nambakam]

Hi David,

Thanks for posting this. I've been trying the series and the QEMU VSM
implementation in my local integration trees. With the fixes below,
Windows now reports "Virtualization-based security: Running" in msinfo32.

This was on a v7.3-based kernel tree with existing local VM-planes/LVBS
changes, rather than an otherwise unmodified copy of your v7.1-rc7-based
tree. I don't want to imply that all of the integration changes were
defects in the posted series.

The main runtime issue I found was descriptor-table interception while
OVMF was executing in SMM. hv_divert_dt_write() was diverting firmware
descriptor-table writes through the VTL register-intercept path. The
failure trace showed a reflected #GP delivered in SMM with IDT base and
limit both zero, followed by a triple fault/reset. Adding an is_smm()
exclusion to the helper fixed that boot failure. I'll reply to patch 23
with the change.

On the QEMU side, I also changed vsm_glue_reset() to make plane 0 runnable
when stopping plane 1. Otherwise a reset following a VTL switch can leave
the normal-world vCPU userspace-stopped. I'll send that change separately
in this thread, since the QEMU glue isn't part of the kernel posting.

Two build/header items from the newer kernel integration:

  - In patch 49, I used cpu_feature_enabled(X86_FEATURE_DECODEASSISTS)
    instead of static_cpu_has(), matching the current SVM code.
  - I synchronized the tools UAPI header with the PLANE_PROT/INSN_VALID
    flags and their field-validity comments, and the capability-version
    requirement for KVM_HV_HCALL_RESULT_RAISE_UD.

I also encountered an issue in the underlying planes dependency:
"kvm: Make KVM_SIGNAL_MSI per plane" guards the VM-fd and plane-fd ioctl
dispatch with CONFIG_HAVE_KVM_MSI. Those guards need to use __KVM_HAVE_MSI;
otherwise KVM_SIGNAL_MSI is compiled out and returns ENOTTY. The existing
test_plane_msi case exposed this in my tree. This isn't introduced by one
of the 55 patches here.

Finally, my older LVBS tree had raised the default plane count to two.
I reverted that local change to respect the KVM_CAP_PLANES opt-in contract
and added early opt-in for my LVBS userspace path. Your QEMU VSM path
already has its own opt-in, so I am not reporting that as a missing VSM
change.

My runtime result here is the msinfo32 VBS-running status; I am not
claiming separate Credential Guard, migration, or hibernate/resume
validation.

Thanks,
Sriram

Re: [RFC PATCH 23/55] WIP: userspace VTL interrupt entry and register intercepts

[Sriram Nambakam]

Hi David,

I hit a Windows VBS boot failure involving hv_divert_dt_write() when OVMF
was executing in SMM.

The helper was diverting firmware descriptor-table writes into the VTL
register-intercept path. The failure trace showed a reflected #GP being
delivered in an SMM context with IDT base and limit both zero, followed
by a triple fault and guest reset.

Excluding SMM from this helper lets those firmware descriptor-table
updates proceed normally, without changing interception for ordinary
VTL0 execution:

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -5884,7 +5884,7 @@ static bool hv_divert_dt_write(struct x86_emulate_ctxt *ctxt, u32 hv_reg,
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
 
 	if (!hv_vcpu || vcpu->plane_level || is_guest_mode(vcpu) ||
-	    !(hv_vcpu->reg_intercept_control & ctl_bit))
+	    is_smm(vcpu) || !(hv_vcpu->reg_intercept_control & ctl_bit))
 		return false;
 
 	hv_vcpu->reg_intercept.reg = hv_reg;

The context above includes the is_guest_mode() check added later in the
series by patch 50.

With this and the accompanying integration/QEMU fixes, msinfo32 reports
VBS running. This was an integrated v7.3-based tree, not a test of patch
23 in isolation.

Thanks,
Sriram

Re: [RFC PATCH 49/55] KVM: SVM: Carry instruction state on reflected plane-protection faults

[Sriram Nambakam]

Hi David,

One build adjustment when integrating this onto my v7.3-based tree:
static_cpu_has() is no longer available to this caller. I changed the
decode-assist test to cpu_feature_enabled(), matching the surrounding
SVM code:

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -2094,7 +2094,7 @@ static int npf_interception(struct kvm_vcpu *vcpu)
 	    (vcpu->run->memory_fault.flags & KVM_MEMORY_EXIT_FLAG_PLANE_PROT)) {
 		unsigned int len = 0;
 
-		if (static_cpu_has(X86_FEATURE_DECODEASSISTS) &&
+		if (cpu_feature_enabled(X86_FEATURE_DECODEASSISTS) &&
 		    svm->vmcb->control.insn_len &&
 		    x86_decode_emulated_instruction(vcpu, 0,
 						    svm->vmcb->control.insn_bytes,

This is a newer-base integration adjustment, rather than a claim that
the original v7.1-rc7-based posting fails to build.

Thanks,
Sriram

Re: [RFC PATCH 00/55] KVM: x86: Hyper-V VSM (Virtual Trust Levels) on VM planes

[Sriram Nambakam]

Hi David,

One fix in the associated QEMU VSM glue: vsm_glue_reset() stops plane 1
but also needs to make plane 0 runnable.

After a VTL switch, the normal-world vCPU can still be marked
userspace-stopped. Stopping only the higher plane during reset leaves
that state behind, rather than restoring the VTL0 execution path for
the fresh boot.

The functional change from my local QEMU commit is:

diff --git a/target/i386/kvm/hv-vsm-qemu.c b/target/i386/kvm/hv-vsm-qemu.c
--- a/target/i386/kvm/hv-vsm-qemu.c
+++ b/target/i386/kvm/hv-vsm-qemu.c
@@ -974,6 +974,7 @@ static void vsm_glue_reset(void *opaque)
     for (uint32_t vp = 0; vp < vsm_glue.nr_vps; vp++) {
         if (vsm_glue.plane1_vcpu_fd[vp] >= 0) {
             vsm_vcpu_set_run_state(&vsm_glue, 1, vp, false);
+            vsm_vcpu_set_run_state(&vsm_glue, 0, vp, true);
         }
     }
     qemu_mutex_unlock(&vsm_glue.lock);

I also added temporary exception-delivery diagnostics to print the SMM
state, CS selector/base, CR0, and IDT base/limit. Those exposed the
SMM/#GP failure described in my reply to kernel patch 23, but aren't
needed for the reset fix above.

Thanks,
Sriram