[RFC PATCH 0/1] KVM: x86: Skip empty TDP leaf page tables

[RFC PATCH 0/1] KVM: x86: Skip empty TDP leaf page tables

[Hao Zhang]

From: Hao Zhang <zhanghao1@kylinos.cn>

Hi,

KVM intentionally keeps the TDP page table hierarchy when zapping only
leaf SPTEs, because the backing memslot stays valid and future faults
can reuse the existing paging structure.  But when the same guest range
is invalidated repeatedly, every later zap still descends through the
retained 4K leaf page tables whose 512 SPTEs are all non-present.  Each
such descent is pure overhead: the walk scans 512 entries, zaps
nothing, and on a contended host that can also force a yield/restart
cycle.

This pattern can occur when userspace repeatedly unmaps or discards the
same guest range, e.g. ballooning or memory-discard/reset paths depending
on userspace behavior: each notifier-driven unmap zaps the range, the
guest refaults, and the next unmap zaps the same range again.

The patch adds a TDP-MMU-only hint on 4K leaf page tables, set when a
zap that covers the whole 2MiB region reaches the last entry of the
page table, and cleared whenever a leaf SPTE is installed.  If a later
zap fully covers the corresponding 2MiB range and the hint is set, the
iterator skips descending into that child page table.  The hierarchy
stays linked, so refaults can still reuse it.  The hint is
conservative: a false value only costs one extra descent, never a
missed zap.

Why not unlink the empty page table instead?  Unlinking the 2MiB
non-leaf SPTE requires a TLB flush and an RCU free of the page table,
and the next fault then has to allocate and relink a fresh one.  KVM
retains empty page tables today regardless, so the hint does not change
KVM's memory-ownership behavior; it only makes repeated zaps cheaper.
The saved work is host-side only, i.e. not directly guest-visible; the
value is CPU on the host during invalidation churn on overcommitted
machines.

Correctness rests on an existing invariant: gfn-range zaps run only
while mmu_invalidate_in_progress is elevated or slots_lock is held
(kvm_unmap_gfn_range() asserts exactly this), so no vCPU can install a
leaf SPTE between the walk zapping an entry and the walk marking the
page table empty.  Every SPTE installation goes through
__handle_changed_spte(), which clears the hint, so a stale-true hint is
not possible; a stale-false hint is harmless.

Scope notes:

 - Only 4K leaf page tables are tracked.  The 2M-level page tables are
   walked anyway for any range that spans them, so a hint there would
   buy little; the expensive part is descending into 4K page tables.
 - Partial ranges are never skipped; the hint is only consumed when the
   zap range fully covers the 2MiB region of the page table.
 - Ranges smaller than a complete 2MiB region bypass the hint checks
   entirely.
 - The hint is TDP-MMU-only.
 - On the tested configuration, the new bool lives in existing padding of
   struct kvm_mmu_page.
 - Yield/restart never preserves a stale skip decision; the restarted
   walk re-evaluates page-table state from the root.

Test setup:

 - Host: same x86 host for all runs, PREEMPT kernel, so that zaps of
   large ranges exercise the yield/restart path.
 - Guest: 4 vCPUs, 1GiB memory.
 - Workload: guest threads continuously fault and write guest memory
   while the host repeatedly invalidates guest memory with
   MADV_DONTNEED.
 - Measured paths: tdp_mmu_zap_leafs() duration and vCPU fault-side
   mmu_lock wait time, collected with perf probes.
 - Mechanism validation: kprobes on tdp_iter_skip_child() and
   tdp_mmu_zap_leafs().

Two invalidation patterns were used:

 - "same": repeatedly invalidate the same 16MiB or 4MiB host range.
   This stresses retained empty leaf page tables, because later zaps
   revisit page tables that earlier zaps emptied.
 - "sweep": invalidate rolling 16MiB, 4MiB, 2MiB and 1MiB chunks
   across the 1GiB guest.  Less favorable to the hint, since guest
   faults often repopulate page tables before the next zap reaches the
   same range.

Results from five matched repetitions on the same host, comparing the
uninstrumented patched kernel against the original kernel:

  Pattern       Original median    Patched median    Delta
  ----------    ----------------   ----------------  --------------------------
  same16        330ms              35ms              -89%
  same4         341ms              54ms              -84%
  sweep16       1.456s             1.386s            -5%
  sweep4        1.252s             1.244s            roughly flat
  sweep2        1.197s             1.246s            +4%
  sweep1        1.353s             1.138s            -16%

The same host, guest size, vCPU pinning, madvise rate, and chunk
scenarios were used for both kernels.

The result matches the intended scope: repeated same-range invalidation is
where the retained-empty hint removes substantial zap-side work; rolling
sweeps are mostly neutral, with small run-to-run variation.

An intermediate version without the sub-2MiB gate ran the hint checks on
every entry and regressed sweep1 to 1.64s zap total in one run.  The final
version bypasses the hint path for sub-2MiB ranges entirely; in the matched
five-run comparison above, sweep1 was not worse than the original kernel.

Fault-side mmu_lock wait showed no scenario with the earlier
millisecond-scale regression.  Median fault-wait p99.9 improved mainly in
same-range cases, from 78us to 61us for same16 and from 62us to 38us for
same4.  Rolling sweep fault-wait p99.9 was effectively unchanged.

Mechanism counters:

 - same16: tdp_iter_skip_child() was hit 2000 times over 867 zap calls
   in the final counter run.  The misses are page tables repopulated
   between zaps, i.e. the hint-clear path working as intended.
 - sweep1: tdp_iter_skip_child() was never hit, confirming that
   sub-2MiB ranges bypass the hint path.
 - A writeback/refault variant reduced skip hits sharply: c16 dropped from
   4177 skips over 290 zap calls without writeback to 187 skips over 245
   zap calls with writeback.  This confirms that installing a new leaf SPTE
   clears the hint and prevents skipping page tables that became non-empty
   again.

The main RFC question is whether this style of retained-tree hint is
acceptable for avoiding repeated empty 4K leaf-page-table walks, or
whether KVM should keep treating retained empty TDP leaf page tables as
intentionally reusable structure with no additional state.

Hao Zhang (1):
  KVM: x86: Skip empty TDP leaf page tables

 arch/x86/kvm/mmu/mmu_internal.h |  2 ++
 arch/x86/kvm/mmu/tdp_iter.c     | 10 +++++-
 arch/x86/kvm/mmu/tdp_iter.h     |  6 ++++
 arch/x86/kvm/mmu/tdp_mmu.c      | 74 ++++++++++++++++++++++++++++++++++++-----
 4 files changed, 82 insertions(+), 10 deletions(-)

base-commit: c21bb4193868a8de71fc4693fa741e195fdf5d86
-- 
2.15.0

[RFC PATCH 1/1] KVM: x86: Skip empty TDP leaf page tables

[Hao Zhang]

From: Hao Zhang <zhanghao1@kylinos.cn>

When KVM zaps only leaf SPTEs, the TDP page table hierarchy is
intentionally retained so that subsequent faults can reuse it.  However,
a later zap of the same range still descends through retained 4K leaf
page tables whose leaf SPTEs are all non-present.

Track whether a retained 4K leaf page table contains any present leaf
SPTEs.  If a zap fully covers the corresponding 2MiB range and the page
table is known to be empty, skip descending into it.

Keep the page table hierarchy linked so that it can still be reused by
future page faults.  Installing a new leaf SPTE clears the empty hint.
The hint checks are skipped for ranges smaller than a complete 2MiB leaf
page table.

The iterator restarts its walk from the root after yielding.  Allow the
restarted walk to mark an empty leaf page table again instead of
permanently excluding the page table that contained the yield.

Tested on a 4-vCPU, 1GB guest with five repetitions on the same host.
The main improvement is seen for repeated same-range invalidation; rolling
sweep workloads are mostly neutral.

Signed-off-by: Hao Zhang <zhanghao1@kylinos.cn>
---
 arch/x86/kvm/mmu/mmu_internal.h |  2 ++
 arch/x86/kvm/mmu/tdp_iter.c     | 10 +++++-
 arch/x86/kvm/mmu/tdp_iter.h     |  6 ++++
 arch/x86/kvm/mmu/tdp_mmu.c      | 74 ++++++++++++++++++++++++++++++++++++-----
 4 files changed, 82 insertions(+), 10 deletions(-)

diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_internal.h
index 73cdcbccc89e..5305aa3e688b 100644
--- a/arch/x86/kvm/mmu/mmu_internal.h
+++ b/arch/x86/kvm/mmu/mmu_internal.h
@@ -95,6 +95,8 @@ struct kvm_mmu_page {
 	};
 
 	bool has_mapped_host_mmio;
+	/* TDP MMU only: no present leaf SPTEs exist in this leaf page table. */
+	bool tdp_mmu_empty_leaf_pt;
 
 	union {
 		/* These two members aren't used for TDP MMU */
diff --git a/arch/x86/kvm/mmu/tdp_iter.c b/arch/x86/kvm/mmu/tdp_iter.c
index 9e17bfa80901..1d87db14a6ae 100644
--- a/arch/x86/kvm/mmu/tdp_iter.c
+++ b/arch/x86/kvm/mmu/tdp_iter.c
@@ -23,6 +23,7 @@ static void tdp_iter_refresh_sptep(struct tdp_iter *iter)
 void tdp_iter_restart(struct tdp_iter *iter)
 {
 	iter->yielded = false;
+	iter->skip_child = false;
 	iter->yielded_gfn = iter->next_last_level_gfn;
 	iter->level = iter->root_level;
 
@@ -167,9 +168,11 @@ void tdp_iter_next(struct tdp_iter *iter)
 		return;
 	}
 
-	if (try_step_down(iter))
+	if (!iter->skip_child && try_step_down(iter))
 		return;
 
+	iter->skip_child = false;
+
 	do {
 		if (try_step_side(iter))
 			return;
@@ -177,3 +180,8 @@ void tdp_iter_next(struct tdp_iter *iter)
 	iter->valid = false;
 }
 
+void tdp_iter_skip_child(struct tdp_iter *iter)
+{
+	WARN_ON_ONCE(iter->yielded);
+	iter->skip_child = true;
+}
diff --git a/arch/x86/kvm/mmu/tdp_iter.h b/arch/x86/kvm/mmu/tdp_iter.h
index 364c5da6c499..47ac8bd43794 100644
--- a/arch/x86/kvm/mmu/tdp_iter.h
+++ b/arch/x86/kvm/mmu/tdp_iter.h
@@ -114,6 +114,11 @@ struct tdp_iter {
 	 * level instead of advancing to the next entry.
 	 */
 	bool yielded;
+	/*
+	 * True if tdp_iter_next() should skip the child page table referenced by
+	 * the current SPTE, instead of descending into it.
+	 */
+	bool skip_child;
 };
 
 /*
@@ -139,5 +144,6 @@ void tdp_iter_start(struct tdp_iter *iter, struct kvm_mmu_page *root,
 		    int min_level, gfn_t next_last_level_gfn, gfn_t gfn_bits);
 void tdp_iter_next(struct tdp_iter *iter);
 void tdp_iter_restart(struct tdp_iter *iter);
+void tdp_iter_skip_child(struct tdp_iter *iter);
 
 #endif /* __KVM_X86_MMU_TDP_ITER_H */
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index c1cbae65d239..84e18bd6e777 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -238,6 +238,7 @@ static void tdp_mmu_init_sp(struct kvm_mmu_page *sp, tdp_ptep_t sptep,
 	sp->gfn = gfn;
 	sp->ptep = sptep;
 	sp->tdp_mmu_page = true;
+	sp->tdp_mmu_empty_leaf_pt = false;
 
 	trace_kvm_mmu_get_page(sp, true);
 }
@@ -527,6 +528,10 @@ static int __handle_changed_spte(struct kvm *kvm, struct kvm_mmu_page *sp,
 	if (is_leaf)
 		check_spte_writable_invariants(new_spte);
 
+	if (is_leaf && !was_leaf && sp->tdp_mmu_page &&
+	    sp->role.level == PG_LEVEL_4K)
+		WRITE_ONCE(sp->tdp_mmu_empty_leaf_pt, false);
+
 	/*
 	 * The only times a SPTE should be changed from a non-present to
 	 * non-present state is when an MMIO entry is installed/modified/
@@ -923,6 +928,39 @@ bool kvm_tdp_mmu_zap_possible_nx_huge_page(struct kvm *kvm,
 	return true;
 }
 
+static bool tdp_mmu_range_covers_leaf_pt(struct tdp_iter *iter,
+					 gfn_t start, gfn_t end)
+{
+	struct kvm_mmu_page *sp;
+
+	sp = sptep_to_sp(rcu_dereference(iter->sptep));
+	return sp->gfn >= start &&
+	       sp->gfn + KVM_PAGES_PER_HPAGE(PG_LEVEL_2M) <= end;
+}
+
+static bool tdp_mmu_can_skip_leaf_pt(struct tdp_iter *iter,
+				     gfn_t start, gfn_t end)
+{
+	struct kvm_mmu_page *child_sp;
+
+	if (iter->level != PG_LEVEL_2M ||
+	    !is_shadow_present_pte(iter->old_spte) ||
+	    is_last_spte(iter->old_spte, iter->level))
+		return false;
+
+	if (iter->gfn < start ||
+	    iter->gfn + KVM_PAGES_PER_HPAGE(PG_LEVEL_2M) > end)
+		return false;
+
+	/*
+	 * Skip retained empty 4K leaf page tables without unlinking them, so
+	 * future faults can reuse the paging structure.
+	 */
+	child_sp = spte_to_child_sp(iter->old_spte);
+	return child_sp->role.level == PG_LEVEL_4K &&
+	       READ_ONCE(child_sp->tdp_mmu_empty_leaf_pt);
+}
+
 /*
  * If can_yield is true, will release the MMU lock and reschedule if the
  * scheduler needs the CPU or there is contention on the MMU lock. If this
@@ -934,8 +972,10 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root,
 			      gfn_t start, gfn_t end, bool can_yield, bool flush)
 {
 	struct tdp_iter iter;
+	bool may_skip_leaf_pts;
 
 	end = min(end, tdp_mmu_max_gfn_exclusive());
+	may_skip_leaf_pts = end - start >= KVM_PAGES_PER_HPAGE(PG_LEVEL_2M);
 
 	lockdep_assert_held_write(&kvm->mmu_lock);
 
@@ -948,18 +988,34 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root,
 			continue;
 		}
 
-		if (!is_shadow_present_pte(iter.old_spte) ||
-		    !is_last_spte(iter.old_spte, iter.level))
+		if (may_skip_leaf_pts &&
+		    tdp_mmu_can_skip_leaf_pt(&iter, start, end)) {
+			tdp_iter_skip_child(&iter);
 			continue;
+		}
 
-		tdp_mmu_iter_set_spte(kvm, &iter, SHADOW_NONPRESENT_VALUE);
+		if (is_shadow_present_pte(iter.old_spte) &&
+		    is_last_spte(iter.old_spte, iter.level)) {
+			tdp_mmu_iter_set_spte(kvm, &iter,
+					      SHADOW_NONPRESENT_VALUE);
 
-		/*
-		 * Zappings SPTEs in invalid roots doesn't require a TLB flush,
-		 * see kvm_tdp_mmu_zap_invalidated_roots() for details.
-		 */
-		if (!root->role.invalid)
-			flush = true;
+			/*
+			 * Zappings SPTEs in invalid roots doesn't require a TLB flush,
+			 * see kvm_tdp_mmu_zap_invalidated_roots() for details.
+			 */
+			if (!root->role.invalid)
+				flush = true;
+		}
+
+		if (may_skip_leaf_pts &&
+		    iter.level == PG_LEVEL_4K &&
+		    spte_index(rcu_dereference(iter.sptep)) == SPTE_ENT_PER_PAGE - 1 &&
+		    tdp_mmu_range_covers_leaf_pt(&iter, start, end)) {
+			struct kvm_mmu_page *sp;
+
+			sp = sptep_to_sp(rcu_dereference(iter.sptep));
+			WRITE_ONCE(sp->tdp_mmu_empty_leaf_pt, true);
+		}
 	}
 
 	rcu_read_unlock();

base-commit: c21bb4193868a8de71fc4693fa741e195fdf5d86
-- 
2.15.0

Re: [RFC PATCH 1/1] KVM: x86: Skip empty TDP leaf page tables

[Sean Christopherson]

On Fri, Aug 14, 2026, Hao Zhang wrote:
> From: Hao Zhang <zhanghao1@kylinos.cn>
> 
> When KVM zaps only leaf SPTEs, the TDP page table hierarchy is
> intentionally retained so that subsequent faults can reuse it.  However,
> a later zap of the same range still descends through retained 4K leaf
> page tables whose leaf SPTEs are all non-present.
> 
> Track whether a retained 4K leaf page table contains any present leaf
> SPTEs.  If a zap fully covers the corresponding 2MiB range and the page
> table is known to be empty, skip descending into it.
> 
> Keep the page table hierarchy linked so that it can still be reused by
> future page faults.  Installing a new leaf SPTE clears the empty hint.
> The hint checks are skipped for ranges smaller than a complete 2MiB leaf
> page table.
> 
> The iterator restarts its walk from the root after yielding.  Allow the
> restarted walk to mark an empty leaf page table again instead of
> permanently excluding the page table that contained the yield.
> 
> Tested on a 4-vCPU, 1GB guest with five repetitions on the same host.
> The main improvement is seen for repeated same-range invalidation; rolling
> sweep workloads are mostly neutral.
> 
> Signed-off-by: Hao Zhang <zhanghao1@kylinos.cn>
> ---

Does the generic iterator solution I provided a (long) while back work for your
use case?  I would still strongly prefer a generic solution that doesn't rely on
storing metadata in the MMU page.

https://lore.kernel.org/all/ZxmGdhwr9BlhUQ_Y@google.com

Re: [RFC PATCH 1/1] KVM: x86: Skip empty TDP leaf page tables

[Hao Zhang]

On Fri, Aug 14, 2026, Sean Christopherson wrote:
> On Fri, Aug 14, 2026, Hao Zhang wrote:
> > From: Hao Zhang <zhanghao1@kylinos.cn>
> > 
> > When KVM zaps only leaf SPTEs, the TDP page table hierarchy is
> > intentionally retained so that subsequent faults can reuse it.  However,
> > a later zap of the same range still descends through retained 4K leaf
> > page tables whose leaf SPTEs are all non-present.
> > 
> > Track whether a retained 4K leaf page table contains any present leaf
> > SPTEs.  If a zap fully covers the corresponding 2MiB range and the page
> > table is known to be empty, skip descending into it.
> > 
> > Keep the page table hierarchy linked so that it can still be reused by
> > future page faults.  Installing a new leaf SPTE clears the empty hint.
> > The hint checks are skipped for ranges smaller than a complete 2MiB leaf
> > page table.
> > 
> > The iterator restarts its walk from the root after yielding.  Allow the
> > restarted walk to mark an empty leaf page table again instead of
> > permanently excluding the page table that contained the yield.
> > 
> > Tested on a 4-vCPU, 1GB guest with five repetitions on the same host.
> > The main improvement is seen for repeated same-range invalidation; rolling
> > sweep workloads are mostly neutral.
> > 
> > Signed-off-by: Hao Zhang <zhanghao1@kylinos.cn>
> > ---
> 
> Does the generic iterator solution I provided a (long) while back work for your
> use case?  I would still strongly prefer a generic solution that doesn't rely on
> storing metadata in the MMU page.
> 
> https://lore.kernel.org/all/ZxmGdhwr9BlhUQ_Y@google.com

Hi Sean,

I tested the generic iterator approach you pointed me at, i.e. skipping
non-present SPTEs in the TDP iterator and converting the relevant TDP MMU
walkers to use the shadow-present-only iterator.

The results do show that the generic approach helps, but it doesn't fully cover
the pathology I was trying to address.

All numbers below are medians over 5 runs on the same host/workload:

                original    generic iterator    empty-subtree hint
    same16        330 ms          222 ms              35 ms
    same4         341 ms          234 ms              54 ms
    c16          1456 ms         1696 ms            1386 ms
    c4           1252 ms         1291 ms            1244 ms
    c2           1197 ms         1174 ms            1246 ms
    c1           1353 ms         1199 ms            1138 ms

The generic iterator reduces zap time for the repeated same-range cases by
about 31-33%, but the empty-subtree hint reduces those cases by about 84-89%.

I think the reason is that the generic iterator only skips non-present SPTEs
within the current walk.  It doesn't carry any information across invalidations,
so a later zap of the same range still has to descend into the retained child
page table and rediscover that all 512 entries are non-present.  The hint avoids
that repeated discovery step by remembering that the child page table is empty
until a later fault installs a leaf SPTE.

I also measured fault wait time.  The generic iterator didn't show the severe
~1ms regression I had in an earlier experimental version that reduced yield
opportunities, but its fwait p99.9 was still higher than both the original
kernel and the empty-hint version in this workload:

                original    generic iterator    empty-subtree hint
    same16         78 us          162 us              61 us
    same4          62 us          152 us              38 us
    c16            72 us          242 us              77 us
    c4            102 us          248 us             105 us
    c2            108 us          270 us             107 us
    c1             92 us          266 us              93 us

Although the generic iterator is cleaner, but it may not enough to address 
the retained-empty-subtree case.  The key question seems to be whether KVM 
can keep a very narrow, derived hint on the shadow page: "this 4K leaf page 
table currently has no present leaf SPTEs".  The hint is cleared when a fault 
installs a leaf SPTE, and it is only used to skip a fully covered 2M subtree.

If storing that bit in struct kvm_mmu_page is still a non-starter, I can look at
a metadata-free version that checks whether the child page table is empty before
descending and skips it for the current walk.  But based on the mechanism, I
don't expect that to preserve the same benefit, because it still has to
rediscover the empty state on every invalidation.

Thanks,
Hao