[PATCH] KVM: x86: Rebuild nested TDP shadows from L1's memory after INVEPT and nSVM TLB flushes

[PATCH] KVM: x86: Rebuild nested TDP shadows from L1's memory after INVEPT and nSVM TLB flushes

[Diptanu Choudhury]

KVM keeps shadow copies of L1's nested EPT (nVMX) and NPT (nSVM) tables,
which it keeps in sync by write-protecting the guest pages it shadows:
vCPU writes to those pages fault, and emulated writes notify the shadow
MMU via kvm_page_track_write().  Writes from host userspace into guest
memory reach none of that: a VMM completing a virtio request, or any
vhost-user backend, writes through the memslot's userspace mapping
without ever touching write tracking.  A device write that lands on a
page L1 uses as a page table therefore leaves KVM's shadow of that page
stale.

Architecturally, INVEPT and SVM's VMCB TLB_CONTROL flushes require the
logical processor to stop using cached guest-physical translations, so
after L1 invalidates, L2 must see the updated tables, the same way it
would on hardware, where the CPU resumes walking the tables from
memory.  KVM does not provide that: handle_invept() only drops root
references, and kvm_mmu_find_shadow_page() then finds the old,
still-valid shadow pages in the page hash and reuses them, stale entry
included; nSVM ignores L1's tlb_ctl entirely.  A stale translation
therefore outlives the architectural invalidation, and an L2 can keep
using a mapping that its own hypervisor explicitly invalidated.
Android emulators and nested KVM, which update and restore their
guest's EPT/NPT tables with device writes, are the exposed workloads.

We hit this in production: a red-team probe had L1 update one entry of
an L1-built EPT table for its L2 through a virtio-blk read whose
destination buffer was the table page.  The vCPU-store control was
always correct; the device-write case served the stale page to L2
after single-context and global INVEPT, 69 times out of 83 runs on a
6.18.63 host.

Fix it with a nested TDP generation tracked per vCPU, because both
invalidations are per-logical-processor.  INVEPT (single-context and
global) and the TLB_CONTROL flush advance the executing vCPU's
generation and drop its nested root references;
kvm_mmu_find_shadow_page() no longer serves a vCPU a guest_mode page
whose generation predates the vCPU's, so the affected tables are
rebuilt from L1's memory on the next nested faults; and a nested root
whose last reference is dropped is zapped, so an invalidated shadow
subtree is reclaimed through the standard invalid-root lifecycle
instead of lingering in the page hash.  Other vCPUs keep their cached
translations, as they do on hardware, until they invalidate themselves.

A note on the design, since the obvious alternatives look simpler but
do not survive a nested-boot soak:

- Making the pages obsolete through a VM-wide generation and
  is_obsolete_sp() also marks the roots other vCPUs still have loaded,
  which forces freeing every vCPU's roots (is_page_fault_stale()
  retries otherwise), i.e. a remote root-free IPI per INVEPT; our soak
  hit KVM_INTERNAL_ERROR_DELIVERY_EV event-vectoring failures under
  those IPI storms.
- Eagerly zapping roots that other vCPUs still reference runs the
  shared root through the prepare_zap zapped-root lifecycle twice and
  corrupts the active page list.
- Marking a shared root invalid without delisting breaks the
  role.invalid implies delisted assumption and double-lists the page.

The shipped revision touches no shadow-page lifecycle state outside
the stock paths: no role.invalid writes, no delisting, no remote
requests.  The at-drop zap uses the stock invalid-root list_move
branch, and the per-vCPU generation only gates page-hash lookups.
Over-invalidation is possible per lookup: a vCPU that INVEPTs one EPTP
will also rebuild rather than reuse pages of another EPTP it holds;
pages are keyed by (gfn, role) and shared across EPTPs, so per-EPTP
precision is not worth the complexity.  Rebuilding is the demand-paged
work INVEPT buys on hardware.

Testing: new selftests for both architectures.  vmx_invept_test and
svm_nested_tlb_ctl_test build an L1 EPT/NPT table for L2, let L2 read
a marker page, rewrite one entry of that table (the leaf PTE, or the
upper entry pointing at the leaf table) either from the vCPU (the
control) or from host userspace through the memslot mapping (the
virtio completion case), then execute INVEPT / request a TLB_CONTROL
flush and require L2 to observe the new mapping.  Without the fix, the
host-write case keeps serving the stale marker after the invalidation
on both Intel (i7i.metal-24xl, Xeon 8559C) and AMD (r8a.metal-24xl,
EPYC 9R45).  vmx_invept_test also proves containment: after the host
frees the page behind a stale entry, L2 faults through the new entry
and never sees a byte of the reclaimed page.

Soak: a nested KVM boot loop (L1 = Debian trixie on a 6.18 host
kernel, 8 vCPU/8G; L2 = 4 vCPU/2G, 20 boots per run) crashed or
wedged the host for each of the rejected designs above; with this
revision, three runs per architecture completed 60/60 boots cleanly on
both Intel and AMD, with L2 boot time at parity with the unpatched
host kernel (median 1068ms before and after on Intel).  A synthetic
INVEPT-saturated loop (20 runs of the selftest) pays +7.9% wall time.

The patch is against 6.18.63 (our shipping kernel) with no other
KVM-local changes in the touched files; happy to rebase it wherever it
fits the queue.

Reported-by: Datadog Security Labs red team
Link: https://github.com/tensorlakeai/compute-engine-internal/issues/3291
Signed-off-by: Diptanu Choudhury <diptanu@tensorlake.ai>

KVM keeps shadow copies of L1's nested EPT (nVMX) and NPT (nSVM) tables,
which it keeps in sync by write-protecting the guest pages it shadows:
vCPU writes to those pages fault, and emulated writes notify the shadow
MMU via kvm_page_track_write().  Writes from host userspace into guest
memory reach none of that: a VMM completing a virtio request, or any
vhost-user backend, writes through the memslot's userspace mapping
without ever touching write tracking.  A device write that lands on a
page L1 uses as a page table therefore leaves KVM's shadow of that page
stale.

Architecturally, INVEPT and SVM's VMCB TLB_CONTROL flushes require the
logical processor to stop using cached guest-physical translations, so
after L1 invalidates, L2 must see the updated tables, the same way it
would on hardware, where the CPU resumes walking the tables from
memory.  KVM does not provide that: handle_invept() only drops root
references, and kvm_mmu_find_shadow_page() then finds the old,
still-valid shadow pages in the page hash and reuses them, stale entry
included; nSVM ignores L1's tlb_ctl entirely (its own TODO notes the
gap).  A stale translation therefore outlives the architectural
invalidation, and an L2 can keep using a mapping that its own
hypervisor explicitly invalidated.  Android emulators and nested KVM,
which update and restore their guest's EPT/NPT tables with device
writes, are the exposed workloads.

Track a nested TDP generation per vCPU.  INVEPT and the TLB_CONTROL
flush advance the vCPU's generation, and kvm_mmu_find_shadow_page()
no longer serves a guest_mode page whose generation predates the
vCPU's, so the next nested faults rebuild the affected tables from
L1's memory instead of reusing the cached ones.  Generations are
per-vCPU because both invalidations are per-logical-processor: the
other vCPUs keep their cached translations, as they do on hardware,
until they invalidate themselves.  A nested root whose last reference
is dropped is zapped, so an invalidated shadow subtree is reclaimed
instead of lingering in the page hash until the next memslot
generation change.

Add KVM selftests for both architectures.  vmx_invept_test and
svm_nested_tlb_ctl_test build an L1 EPT/NPT table for L2, let L2 read a
marker page, rewrite one entry of that table (the leaf PTE, or the
upper entry pointing at the leaf table) either from the vCPU (the
control) or from host userspace through the memslot mapping (the
virtio completion case), then execute INVEPT / request a TLB_CONTROL
flush and require L2 to observe the new mapping.  Without the fix, the
host-write case keeps serving the stale marker after the invalidation.
vmx_invept_test also proves containment: after the host frees (munmap)
the page behind a stale entry, L2's next access faults through the new
entry and never sees a byte of the reclaimed page, so the effect cannot
reach memory outside the sandbox even before the fix.

Reported-by: Datadog Security Labs red team
Link: https://github.com/tensorlakeai/compute-engine-internal/issues/3291
Signed-off-by: Diptanu Choudhury <diptanu@tensorlake.ai>

Backport note: Tensorlake-authored fix, not yet merged upstream.  Line
numbers are relative to Linux 6.18.63 with patches 0001..0012 of this
tree applied.  When the fix is accepted upstream, replace the two lines
above with the upstream commit id and a Link: line.
---
 arch/x86/include/asm/kvm_host.h                    |   8 +
 arch/x86/kvm/mmu/mmu.c                             |  33 +-
 arch/x86/kvm/mmu/mmu_internal.h                    |   7 +
 arch/x86/kvm/svm/nested.c                          |  27 +-
 arch/x86/kvm/vmx/nested.c                          |  21 +-
 tools/testing/selftests/kvm/Makefile.kvm           |   2 +
 .../selftests/kvm/x86/svm_nested_tlb_ctl_test.c    | 227 +++++++++++++++++++++
 .../selftests/kvm/x86/vmx_invept_test.c            | 216 +++++++++++++++++++++
 8 files changed, 535 insertions(+), 6 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 319aeac..82ec068 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -471,6 +471,14 @@ struct kvm_mmu {
 	union kvm_cpu_role cpu_role;
 	union kvm_mmu_page_role root_role;

+	/*
+	 * Nested TDP (nVMX/nSVM) cache generation for the vCPU: bumped when
+	 * L1 invalidates its cached guest-physical translations (INVEPT, VMCB
+	 * TLB_CONTROL flush).  Shadow pages stamped with an older generation
+	 * are not reused for this vCPU, see kvm_mmu_find_shadow_page().
+	 */
+	u64 nested_tdp_epoch;
+
 	/*
 	* The pkru_mask indicates if protection key checks are needed.  It
 	* consists of 16 domains indexed by page fault error code bits [4:1],
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 835b0c2..a9ac2af 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -2279,6 +2279,23 @@ static struct kvm_mmu_page
*kvm_mmu_find_shadow_page(struct kvm *kvm,
 			continue;
 		}

+		/*
+		 * A nested TDP (nVMX/nSVM) shadow page caches L1's
+		 * guest-physical translations for the logical processor of
+		 * the vCPU it was built for.  INVEPT and the VMCB
+		 * TLB_CONTROL flush are per-logical-processor
+		 * invalidations: once the vCPU has executed one, reusing its
+		 * older pages would keep translations alive that L1
+		 * invalidated, and those pages can be stale in ways that
+		 * shadowing a vCPU write would have caught, because host
+		 * userspace and device writes to the L1 page tables bypass
+		 * guest write tracking.  Rebuild from L1's memory instead.
+		 */
+		if (sp->role.guest_mode &&
+		    (!vcpu ||
+		     sp->nested_tdp_epoch != vcpu->arch.guest_mmu.nested_tdp_epoch))
+			continue;
+
 		/*
 		 * Note!  Unlike fully direct MMUs, direct SPs in indirect MMUs
 		 * can have an arbitrary number of parents, e.g if the guest is
@@ -2399,6 +2416,8 @@ static struct kvm_mmu_page
*__kvm_mmu_get_shadow_page(struct kvm *kvm,
 	if (!sp) {
 		created = true;
 		sp = kvm_mmu_alloc_shadow_page(kvm, caches, gfn, sp_list, role);
+		sp->nested_tdp_epoch = role.guest_mode && vcpu
+			? vcpu->arch.guest_mmu.nested_tdp_epoch : 0;
 	}

 	trace_kvm_mmu_get_page(sp, created);
@@ -3819,7 +3838,19 @@ static void mmu_free_root_page(struct kvm *kvm,
hpa_t *root_hpa,
 		kvm_tdp_mmu_put_root(kvm, sp);
 	} else {
 		lockdep_assert_held_write(&kvm->mmu_lock);
-		if (!--sp->root_count && sp->role.invalid)
+		/*
+		 * Nested TDP (nVMX/nSVM) roots cache L1's guest-physical
+		 * translations for this vCPU.  Once the last reference is
+		 * dropped the pages must not linger in the page hash, as
+		 * they may be stale (host userspace and device writes to
+		 * the L1 page tables bypass guest write tracking) and stale
+		 * reuse after INVEPT or a TLB_CONTROL flush resurrects
+		 * translations that L1 invalidated.  Zap the root so its
+		 * shadow pages are reclaimed and the next nested entry
+		 * rebuilds from L1's memory.
+		 */
+		if (!--sp->root_count &&
+		    (sp->role.invalid || sp->role.guest_mode))
 			kvm_mmu_prepare_zap_page(kvm, sp, invalid_list);
 	}

diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_internal.h
index ed5c01d..5c29c9c 100644
--- a/arch/x86/kvm/mmu/mmu_internal.h
+++ b/arch/x86/kvm/mmu/mmu_internal.h
@@ -82,6 +82,13 @@ struct kvm_mmu_page {
 	union kvm_mmu_page_role role;
 	gfn_t gfn;

+	/*
+	 * Nested TDP (nVMX/nSVM) cache generation of the vCPU this page was
+	 * built for; see struct kvm_mmu.nested_tdp_epoch.  Only meaningful
+	 * when role.guest_mode, all other pages stamp 0.
+	 */
+	u64 nested_tdp_epoch;
+
 	u64 *spt;

 	/*
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 064a2bb..691679a 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -578,12 +578,8 @@ static void
nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)
 	 *
 	 *  - Flush TLBs for both L1 and L2 remote TLB flush
 	 *  - Honor L1's request to flush an ASID on nested VMRUN
-	 *  - Sync nested NPT MMU on VMRUN that flushes L2's ASID[*]
 	 *  - Don't crush a pending TLB flush in vmcb02 on nested VMRUN
 	 *  - Flush L1's ASID on KVM_REQ_TLB_FLUSH_GUEST
-	 *
-	 * [*] Unlike nested EPT, SVM's ASID management can invalidate nested
-	 *     NPT guest-physical mappings on VMRUN.
 	 */
 	kvm_make_request(KVM_REQ_MMU_SYNC, vcpu);
 	kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
@@ -812,6 +808,29 @@ static void nested_vmcb02_prepare_control(struct
vcpu_svm *svm)
 	/* Also overwritten later if necessary.  */
 	vmcb02->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;

+	/*
+	 * A TLB_CONTROL flush requested by L1 is performed at VMRUN and
+	 * architecturally invalidates the flushed ASIDs' cached nested NPT
+	 * (guest-physical) translations, including on this logical
+	 * processor.  Userspace and device writes to L1's NPT tables bypass
+	 * guest write tracking, so the cached shadow pages can be stale.
+	 * Advance the vCPU's nested TDP generation so its older shadow
+	 * pages are no longer reused (kvm_mmu_find_shadow_page), and drop
+	 * the nested root references so that L2's shadow page tables are
+	 * rebuilt from L1's memory on this VMRUN.
+	 */
+	if (svm->nested.ctl.tlb_ctl == TLB_CONTROL_FLUSH_ALL_ASID ||
+	    (nested_npt_enabled(svm) &&
+	     (svm->nested.ctl.tlb_ctl == TLB_CONTROL_FLUSH_ASID ||
+	      svm->nested.ctl.tlb_ctl == TLB_CONTROL_FLUSH_ASID_LOCAL))) {
+		write_lock(&vcpu->kvm->mmu_lock);
+		vcpu->arch.guest_mmu.nested_tdp_epoch++;
+		write_unlock(&vcpu->kvm->mmu_lock);
+
+		kvm_mmu_free_roots(vcpu->kvm, &vcpu->arch.guest_mmu,
+				   KVM_MMU_ROOTS_ALL);
+	}
+
 	/* nested_cr3.  */
 	if (nested_npt_enabled(svm))
 		nested_svm_init_mmu_context(vcpu);
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index ffca5fe..ac12546 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -6164,8 +6164,27 @@ static int handle_invept(struct kvm_vcpu *vcpu)
 		break;
 	}

-	if (roots_to_free)
+	if (roots_to_free) {
+		/*
+		 * INVEPT architecturally invalidates the logical processor's
+		 * cached guest-physical translations for the affected
+		 * EPTP(s): the processor resumes walking the tables from
+		 * memory.  Freeing the root references is not enough for
+		 * that: the shadow pages stay in the page hash and are
+		 * reused as-is by the next nested entry, and they can be
+		 * stale because userspace and device writes to L1's EPT
+		 * tables bypass guest write tracking.  Advance the vCPU's
+		 * nested TDP generation so its older shadow pages are no
+		 * longer reused (kvm_mmu_find_shadow_page), and drop the
+		 * root references so the tables are rebuilt from L1's
+		 * memory on the next nested entry.
+		 */
+		write_lock(&vcpu->kvm->mmu_lock);
+		mmu->nested_tdp_epoch++;
+		write_unlock(&vcpu->kvm->mmu_lock);
+
 		kvm_mmu_free_roots(vcpu->kvm, mmu, roots_to_free);
+	}

 	return nested_vmx_succeed(vcpu);
 }
diff --git a/tools/testing/selftests/kvm/Makefile.kvm
b/tools/testing/selftests/kvm/Makefile.kvm
index 1715efe..71fdbc4 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -105,6 +105,7 @@ TEST_GEN_PROGS_x86 += x86/svm_vmcall_test
 TEST_GEN_PROGS_x86 += x86/svm_int_ctl_test
 TEST_GEN_PROGS_x86 += x86/svm_nested_shutdown_test
 TEST_GEN_PROGS_x86 += x86/svm_nested_soft_inject_test
+TEST_GEN_PROGS_x86 += x86/svm_nested_tlb_ctl_test
 TEST_GEN_PROGS_x86 += x86/tsc_scaling_sync
 TEST_GEN_PROGS_x86 += x86/sync_regs_test
 TEST_GEN_PROGS_x86 += x86/ucna_injection_test
@@ -112,6 +113,7 @@ TEST_GEN_PROGS_x86 += x86/userspace_io_test
 TEST_GEN_PROGS_x86 += x86/userspace_msr_exit_test
 TEST_GEN_PROGS_x86 += x86/vmx_apic_access_test
 TEST_GEN_PROGS_x86 += x86/vmx_close_while_nested_test
+TEST_GEN_PROGS_x86 += x86/vmx_invept_test
 TEST_GEN_PROGS_x86 += x86/vmx_dirty_log_test
 TEST_GEN_PROGS_x86 += x86/vmx_exception_with_invalid_guest_state
 TEST_GEN_PROGS_x86 += x86/vmx_msrs_test
diff --git a/tools/testing/selftests/kvm/x86/svm_nested_tlb_ctl_test.c
b/tools/testing/selftests/kvm/x86/svm_nested_tlb_ctl_test.c
new file mode 100644
index 0000000..82e37f3
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/svm_nested_tlb_ctl_test.c
@@ -0,0 +1,227 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Host writes to L1's nested NPT tables must become visible to L2 after L1
+ * requests a TLB_CONTROL flush at VMRUN.  Userspace writes into guest memory
+ * bypass guest write tracking, so KVM's shadow of L1's NPT tables can be
+ * stale; the VMCB TLB_CONTROL flush architecturally invalidates the incoming
+ * ASID's cached guest-physical translations, so L2 must observe the new
+ * mapping after a flush, even when the table entry was written from host
+ * userspace.  The vCPU-write control proves the test can observe an update;
+ * the flush is what makes the host write visible.
+ */
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+#include "svm_util.h"
+
+#define DATA_GPA 0xc0000000ul
+#define ENTRY_GVA 0xd0000000ul
+#define GPA_MASK 0x000ffffffffff000ul
+#define MARKER_A 0xaaaaaaaaaaaaaaaaul
+#define MARKER_B 0xbbbbbbbbbbbbbbbbul
+
+struct npt_desc {
+	u64 npt_root_gpa;
+};
+
+static volatile u64 observed;
+static u8 tlb_ctl;
+static bool cpu_write;
+static u64 *guest_entry;
+static u64 replacement;
+static uint64_t l2_stack[256];
+
+static void l2_guest_code(void)
+{
+	observed = READ_ONCE(*(u64 *)DATA_GPA);
+	asm volatile("vmmcall");
+	GUEST_FAIL("L2 ran past VMMCALL");
+}
+
+static void l1_guest_code(struct svm_test_data *svm, struct npt_desc *desc)
+{
+	struct vmcb *vmcb = svm->vmcb;
+	uint64_t *l2_stack_top = &l2_stack[ARRAY_SIZE(l2_stack)];
+
+	/* Prepare for L2 execution with NPT. */
+	generic_svm_setup(svm, l2_guest_code, l2_stack_top);
+	vmcb->control.nested_ctl = SVM_NESTED_CTL_NP_ENABLE;
+	vmcb->control.nested_cr3 = desc->npt_root_gpa;
+
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_VMMCALL);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_1, 0);
+	GUEST_ASSERT_EQ(observed, MARKER_A);
+
+	GUEST_SYNC(1);
+	if (cpu_write)
+		WRITE_ONCE(*guest_entry, replacement);
+
+	vmcb->control.tlb_ctl = tlb_ctl;
+	vmcb->save.rip = (u64)l2_guest_code;
+	vmcb->save.rsp = (u64)l2_stack_top;
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_VMMCALL);
+	GUEST_ASSERT_EQ(observed, MARKER_B);
+	GUEST_DONE();
+}
+
+static u64 *npt_new_table(struct kvm_vm *vm, u64 *gpa)
+{
+	vm_vaddr_t gva = vm_vaddr_alloc_page(vm);
+	u64 *hva = addr_gva2hva(vm, gva);
+
+	memset(hva, 0, vm->page_size);
+	*gpa = addr_gva2gpa(vm, gva);
+	return hva;
+}
+
+/* Identity-map @gpa in the 4-level NPT rooted at @root_hva. */
+static void npt_map(struct kvm_vm *vm, u64 *root_hva, u64 gpa)
+{
+	u64 *table = root_hva;
+	int level;
+
+	for (level = 4; level > 1; level--) {
+		int idx = (gpa >> (12 + 9 * (level - 1))) & 511;
+		u64 pte = READ_ONCE(table[idx]);
+
+		if (!(pte & 0x1)) {
+			u64 child_gpa;
+
+			npt_new_table(vm, &child_gpa);
+			pte = child_gpa | 0x27;
+			WRITE_ONCE(table[idx], pte);
+		}
+		table = addr_gpa2hva(vm, pte & GPA_MASK);
+	}
+	WRITE_ONCE(table[(gpa >> 12) & 511], (gpa & GPA_MASK) | 0x67);
+}
+
+/* Identity-map every backing page of @memslot in the NPT. */
+static void npt_map_memslot(struct kvm_vm *vm, u64 *root_hva, uint32_t memslot)
+{
+	struct userspace_mem_region *region = memslot2region(vm, memslot);
+	sparsebit_idx_t i, last;
+
+	i = (region->region.guest_phys_addr >> vm->page_shift) - 1;
+	last = i + (region->region.memory_size >> vm->page_shift);
+	for (;;) {
+		i = sparsebit_next_clear(region->unused_phy_pages, i);
+		if (i > last)
+			break;
+
+		npt_map(vm, root_hva, (uint64_t)i << vm->page_shift);
+	}
+}
+
+/*
+ * Return the host virtual address of the NPT entry that translates @gpa at
+ * @target_level (1 = the leaf PTE, 2 = the entry pointing at the leaf table).
+ */
+static u64 *npt_entry(struct kvm_vm *vm, u64 *root_hva, u64 gpa, int
target_level)
+{
+	u64 *table = root_hva;
+	int level;
+
+	for (level = 4; level > target_level; level--)
+		table = addr_gpa2hva(vm,
+				table[(gpa >> (12 + 9 * (level - 1))) & 511] & GPA_MASK);
+	return &table[(gpa >> (12 + 9 * (target_level - 1))) & 511];
+}
+
+static void run_test(bool cpu, u8 flush, int level)
+{
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	struct svm_test_data *svm;
+	struct npt_desc *desc;
+	vm_vaddr_t svm_gva, desc_gva;
+	struct ucall uc;
+	u64 *root_hva, *entry, *alternate;
+	u64 entry_gpa, alternate_gpa;
+
+	printf("%s write, TLB_CONTROL %s, NPT level %d\n", cpu ? "vCPU" : "host",
+	       flush == TLB_CONTROL_FLUSH_ASID ? "flush ASID" : "flush all", level);
+	fflush(stdout);
+
+	vm = vm_create_with_one_vcpu(&vcpu, l1_guest_code);
+	vcpu_alloc_svm(vm, &svm_gva);
+
+	desc_gva = vm_vaddr_alloc_page(vm);
+	desc = addr_gva2hva(vm, desc_gva);
+	root_hva = npt_new_table(vm, &desc->npt_root_gpa);
+
+	vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, DATA_GPA, 1, 2, 0);
+	virt_map(vm, DATA_GPA, DATA_GPA, 2);
+	*(u64 *)addr_gpa2hva(vm, DATA_GPA) = MARKER_A;
+	*(u64 *)addr_gpa2hva(vm, DATA_GPA + vm->page_size) = MARKER_B;
+
+	/*
+	 * Identity-map the guest image so L2 can reach its code, stack, the
+	 * VMCB and the global state, plus the marker pages.
+	 */
+	npt_map_memslot(vm, root_hva, 0);
+	npt_map(vm, root_hva, DATA_GPA);
+	npt_map(vm, root_hva, DATA_GPA + vm->page_size);
+
+	entry = npt_entry(vm, root_hva, DATA_GPA, level);
+	entry_gpa = addr_hva2gpa(vm, entry);
+	virt_map(vm, ENTRY_GVA, entry_gpa & GPA_MASK, 1);
+	guest_entry = (u64 *)(ENTRY_GVA + (entry_gpa & (vm->page_size - 1)));
+
+	if (level == 1) {
+		replacement = (READ_ONCE(*entry) & ~GPA_MASK) |
+			      (DATA_GPA + vm->page_size);
+	} else {
+		alternate = npt_new_table(vm, &alternate_gpa);
+		memcpy(alternate,
+		       addr_gpa2hva(vm, READ_ONCE(*entry) & GPA_MASK),
+		       vm->page_size);
+		alternate[(DATA_GPA >> 12) & 511] =
+			(READ_ONCE(*npt_entry(vm, root_hva, DATA_GPA, 1)) & ~GPA_MASK) |
+			(DATA_GPA + vm->page_size);
+		replacement = (READ_ONCE(*entry) & ~GPA_MASK) | alternate_gpa;
+	}
+
+	cpu_write = cpu;
+	tlb_ctl = flush;
+	sync_global_to_guest(vm, cpu_write);
+	sync_global_to_guest(vm, tlb_ctl);
+	sync_global_to_guest(vm, guest_entry);
+	sync_global_to_guest(vm, replacement);
+	vcpu_args_set(vcpu, 2, svm_gva, desc_gva);
+
+	vcpu_run(vcpu);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+	if (get_ucall(vcpu, &uc) == UCALL_ABORT)
+		REPORT_GUEST_ASSERT(uc);
+	TEST_ASSERT_EQ(uc.cmd, UCALL_SYNC);
+	TEST_ASSERT_EQ(uc.args[1], 1);
+	if (!cpu)
+		WRITE_ONCE(*entry, replacement);
+
+	vcpu_run(vcpu);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+	if (get_ucall(vcpu, &uc) == UCALL_ABORT)
+		REPORT_GUEST_ASSERT(uc);
+	TEST_ASSERT_EQ(uc.cmd, UCALL_DONE);
+	kvm_vm_free(vm);
+}
+
+int main(int argc, char **argv)
+{
+	static const u8 flushes[] = {
+		TLB_CONTROL_FLUSH_ASID, TLB_CONTROL_FLUSH_ALL_ASID
+	};
+	bool cpu = argc > 1 && !strcmp(argv[1], "cpu");
+	int level, i;
+
+	TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SVM));
+	TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_NPT));
+
+	for (i = 0; i < ARRAY_SIZE(flushes); i++)
+		for (level = 1; level <= 2; level++)
+			run_test(cpu, flushes[i], level);
+	return 0;
+}
diff --git a/tools/testing/selftests/kvm/x86/vmx_invept_test.c
b/tools/testing/selftests/kvm/x86/vmx_invept_test.c
new file mode 100644
index 0000000..ea6be0d
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/vmx_invept_test.c
@@ -0,0 +1,216 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Host writes to EPT must become visible after L1 executes INVEPT. */
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+#include "vmx.h"
+
+#define DATA_GPA 0xc0000000ul
+#define ENTRY_GVA 0xd0000000ul
+#define GPA_MASK 0x000ffffffffff000ul
+#define MARKER_A 0xaaaaaaaaaaaaaaaaul
+#define MARKER_B 0xbbbbbbbbbbbbbbbbul
+
+static volatile u64 observed;
+static u64 invept_type;
+static bool cpu_write;
+static bool skip_invept;
+static u64 *guest_entry;
+static u64 replacement;
+
+static void l2_guest_code(void)
+{
+	observed = READ_ONCE(*(u64 *)DATA_GPA);
+	vmcall();
+	GUEST_FAIL("L2 ran past VMCALL");
+}
+
+static void invalidate_ept(void)
+{
+	struct { u64 eptp, reserved; } operand = { vmreadz(EPT_POINTER), 0 };
+	unsigned char fail;
+
+	asm volatile("invept %2, %1; setna %0"
+		     : "=qm"(fail) : "r"(invept_type), "m"(operand) : "cc", "memory");
+	GUEST_ASSERT(!fail);
+}
+
+static void l1_guest_code(struct vmx_pages *vmx)
+{
+	unsigned long stack[256];
+
+	GUEST_ASSERT(prepare_for_vmx_operation(vmx));
+	GUEST_ASSERT(load_vmcs(vmx));
+	prepare_vmcs(vmx, l2_guest_code, &stack[ARRAY_SIZE(stack)]);
+	GUEST_ASSERT(!vmlaunch());
+	GUEST_ASSERT_EQ(vmreadz(VM_EXIT_REASON), EXIT_REASON_VMCALL);
+	GUEST_ASSERT_EQ(observed, MARKER_A);
+
+	GUEST_SYNC(1);
+	if (cpu_write)
+		WRITE_ONCE(*guest_entry, replacement);
+	if (!skip_invept)
+		invalidate_ept();
+	vmwrite(GUEST_RIP, (unsigned long)l2_guest_code);
+	GUEST_ASSERT(!vmresume());
+	GUEST_ASSERT_EQ(vmreadz(VM_EXIT_REASON), EXIT_REASON_VMCALL);
+	GUEST_ASSERT_EQ(observed, MARKER_B);
+	GUEST_DONE();
+}
+
+static u64 *ept_entry(struct kvm_vm *vm, struct vmx_pages *vmx,
+		      u64 gpa, int target_level)
+{
+	u64 *table = vmx->eptp_hva;
+	int level;
+
+	for (level = 4; level > target_level; level--)
+		table = addr_gpa2hva(vm, table[(gpa >> (12 + 9 * (level - 1))) &
511] & GPA_MASK);
+	return &table[(gpa >> (12 + 9 * (target_level - 1))) & 511];
+}
+
+/*
+ * A stale shadow mapping must not outlive the host backing of the page it
+ * points at: freeing (or migrating) the host page behind a stale entry must
+ * tear the cached mapping down even while it is still cached, so L2 can
+ * never reach host memory that the host has taken back.  This models L1
+ * ballooning out the page behind a stale entry: L2 must observe the new
+ * entry after the host reclaim, without any INVEPT.
+ */
+static void run_stale_free_test(void)
+{
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	struct vmx_pages *vmx;
+	vm_vaddr_t vmx_gva;
+	struct ucall uc;
+	u64 *entry, stale_target;
+	u64 entry_gpa;
+
+	printf("host reclaim of the page behind a stale entry\n");
+	fflush(stdout);
+	vm = vm_create_with_one_vcpu(&vcpu, l1_guest_code);
+	vmx = vcpu_alloc_vmx(vm, &vmx_gva);
+	vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, DATA_GPA, 1, 2, 0);
+	virt_map(vm, DATA_GPA, DATA_GPA, 2);
+	*(u64 *)addr_gpa2hva(vm, DATA_GPA) = MARKER_A;
+	*(u64 *)addr_gpa2hva(vm, DATA_GPA + 4096) = MARKER_B;
+	prepare_eptp(vmx, vm, 0);
+	nested_map(vmx, vm, DATA_GPA, DATA_GPA, 4096);
+	nested_map_memslot(vmx, vm, 0);
+	entry = ept_entry(vm, vmx, DATA_GPA, 1);
+	entry_gpa = addr_hva2gpa(vm, entry);
+	virt_map(vm, ENTRY_GVA, entry_gpa & GPA_MASK, 1);
+	guest_entry = (u64 *)(ENTRY_GVA + (entry_gpa & 4095));
+	stale_target = (*entry & ~GPA_MASK) | (DATA_GPA + 4096);
+	replacement = stale_target;
+	/* Point the L1 entry at page B, but leave L1 unaware: no store, no INVEPT. */
+	cpu_write = false;
+	skip_invept = true;
+	sync_global_to_guest(vm, cpu_write);
+	sync_global_to_guest(vm, skip_invept);
+	sync_global_to_guest(vm, guest_entry);
+	sync_global_to_guest(vm, replacement);
+	vcpu_args_set(vcpu, 1, vmx_gva);
+
+	vcpu_run(vcpu);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+	if (get_ucall(vcpu, &uc) == UCALL_ABORT)
+		REPORT_GUEST_ASSERT(uc);
+	TEST_ASSERT_EQ(uc.cmd, UCALL_SYNC);
+	TEST_ASSERT_EQ(uc.args[1], 1);
+
+	/* The device write that stales the shadow, then the host reclaim. */
+	WRITE_ONCE(*entry, stale_target);
+	munmap(addr_gpa2hva(vm, DATA_GPA), 4096);
+
+	vcpu_run(vcpu);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+	if (get_ucall(vcpu, &uc) == UCALL_ABORT)
+		REPORT_GUEST_ASSERT(uc);
+	TEST_ASSERT_EQ(uc.cmd, UCALL_DONE);
+	/*
+	 * The guest's GUEST_ASSERT_EQ(observed, MARKER_B) proves L2 re-read the
+	 * page through the new entry after the host reclaimed page A, with no
+	 * INVEPT: a byte of the reclaimed page can never be served.
+	 */
+	kvm_vm_free(vm);
+}
+
+static void run_test(bool cpu, u64 type, int level)
+{
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	struct vmx_pages *vmx;
+	vm_vaddr_t vmx_gva;
+	struct ucall uc;
+	u64 *entry, *alternate;
+	u64 entry_gpa, alternate_gpa;
+
+	printf("%s write, INVEPT %s, EPT level %d\n", cpu ? "vCPU" : "host",
+	       type == 1 ? "context" : "global", level);
+	fflush(stdout);
+	vm = vm_create_with_one_vcpu(&vcpu, l1_guest_code);
+	vmx = vcpu_alloc_vmx(vm, &vmx_gva);
+	vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, DATA_GPA, 1, 2, 0);
+	virt_map(vm, DATA_GPA, DATA_GPA, 2);
+	*(u64 *)addr_gpa2hva(vm, DATA_GPA) = MARKER_A;
+	*(u64 *)addr_gpa2hva(vm, DATA_GPA + 4096) = MARKER_B;
+	prepare_eptp(vmx, vm, 0);
+	nested_map(vmx, vm, DATA_GPA, DATA_GPA, 4096);
+	entry = ept_entry(vm, vmx, DATA_GPA, level);
+	entry_gpa = addr_hva2gpa(vm, entry);
+	virt_map(vm, ENTRY_GVA, entry_gpa & GPA_MASK, 1);
+	guest_entry = (u64 *)(ENTRY_GVA + (entry_gpa & 4095));
+	if (level == 1) {
+		replacement = (*entry & ~GPA_MASK) | (DATA_GPA + 4096);
+	} else {
+		alternate_gpa = vm_alloc_page_table(vm);
+		alternate = addr_gpa2hva(vm, alternate_gpa);
+		memcpy(alternate, addr_gpa2hva(vm, *entry & GPA_MASK), 4096);
+		alternate[(DATA_GPA >> 12) & 511] =
+			(*ept_entry(vm, vmx, DATA_GPA, 1) & ~GPA_MASK) | (DATA_GPA + 4096);
+		replacement = (*entry & ~GPA_MASK) | alternate_gpa;
+	}
+	nested_map_memslot(vmx, vm, 0);
+	cpu_write = cpu;
+	skip_invept = false;
+	invept_type = type;
+	sync_global_to_guest(vm, cpu_write);
+	sync_global_to_guest(vm, skip_invept);
+	sync_global_to_guest(vm, invept_type);
+	sync_global_to_guest(vm, guest_entry);
+	sync_global_to_guest(vm, replacement);
+	vcpu_args_set(vcpu, 1, vmx_gva);
+
+	vcpu_run(vcpu);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+	if (get_ucall(vcpu, &uc) == UCALL_ABORT)
+		REPORT_GUEST_ASSERT(uc);
+	TEST_ASSERT_EQ(uc.cmd, UCALL_SYNC);
+	TEST_ASSERT_EQ(uc.args[1], 1);
+	if (!cpu)
+		WRITE_ONCE(*entry, replacement);
+	vcpu_run(vcpu);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+	if (get_ucall(vcpu, &uc) == UCALL_ABORT)
+		REPORT_GUEST_ASSERT(uc);
+	TEST_ASSERT_EQ(uc.cmd, UCALL_DONE);
+	kvm_vm_free(vm);
+}
+
+int main(int argc, char **argv)
+{
+	int level;
+	u64 type;
+	bool cpu = argc > 1 && !strcmp(argv[1], "cpu");
+
+	TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_VMX));
+	TEST_REQUIRE(kvm_cpu_has_ept());
+	for (type = 1; type <= 2; type++)
+		for (level = 1; level <= 2; level++)
+			run_test(cpu, type, level);
+	if (cpu)
+		run_stale_free_test();
+	return 0;
+}

Re: [PATCH] KVM: x86: Rebuild nested TDP shadows from L1's memory after INVEPT and nSVM TLB flushes

[Sean Christopherson]

On Fri, Nov 13, 2026, Diptanu Choudhury wrote:
> KVM keeps shadow copies of L1's nested EPT (nVMX) and NPT (nSVM) tables,
> which it keeps in sync by write-protecting the guest pages it shadows:
> vCPU writes to those pages fault, and emulated writes notify the shadow
> MMU via kvm_page_track_write().  Writes from host userspace into guest
> memory reach none of that: a VMM completing a virtio request, or any
> vhost-user backend, writes through the memslot's userspace mapping
> without ever touching write tracking.  A device write that lands on a
> page L1 uses as a page table therefore leaves KVM's shadow of that page
> stale.

...

> + * Host writes to L1's nested NPT tables must become visible to L2 after L1
> + * requests a TLB_CONTROL flush at VMRUN.

NAK.  This is not supported and has never worked.  Maybe after the onslaught of
AI vulnerabilities slows down we can consider supporting (virtual) DMA modifications
to guest PTEs.  Maybe.  But right now, absolutely not.