Re: [RFC] KVM: x86/mmu: Prefetch forward run of pages on TDP page faults

[Sean Christopherson <seanjc@google.com>]

On Tue, Aug 25, 2026, James Houghton wrote:
> On Tue, Aug 25, 2026 at 10:38 AM Marangoni, Marco <mamarang@amazon.com> wrote:
> >
> > Thanks for the replies!
> >
> > On Tue, Aug 25, 2026, Sean Christopherson <seanjc@google.com> wrote:
> > >
> > > On Tue, Aug 25, 2026, Marco Marangoni wrote:
> > > > It's worth mentioning that I also evaluated using the existing
> > > > KVM_PRE_FAULT_MEMORY ioctl, but this doesn't work well for our use-case, as
> > > > it requires the vCPU to be paused.
> > >
> > > What about if/when KVM Userfault[*] comes along?  I.e. pre-fault memory when the
> > > vCPU exits to userspace.
> >
> > KVM Userfault + KVM_PRE_FAULT_MEMORY is a valid suggestion, however if
> > possible we'd like to have _both_ async page faults and prefetching.  I
> > haven't tested async PF together with prefetching, but tested separately,
> > both improvements yield great results, so it would be a shame to have to
> > choose.

Hrm, right.  Although _if_ we can figure out a clever way to allow prefaulting
without a vCPU (or at least, a real vCPU), that would allow both to coexist.
But at that point I'm probably just being extremely stubborn. :-)
 
> What if you used KVM_PRE_FAULT_MEMORY without KVM Userfault?
> 
> If you want to avoid pausing a vCPU, what if you made another vCPU
> (KVM_CREATE_VCPU) and used that solely for prefaulting guest memory? I
> haven't really looked into that before... I'm guessing there's
> something fundamentally wrong with this approach.

Yeah, shoving a vCPU into the VM that shouldn't exist probably won't end well.
E.g. on x86, the vCPU would kinda sorta be visible/reachable by the guest as the
fake vCPU would respond to IRQs and whatnot.

> If this doesn't work (and a VM-scoped KVM_PRE_FAULT_MEMORY doesn't
> make sense either), then perhaps the TDP MMU prefetching logic makes
> sense.
> 
> > On Tue, Aug 25, 2026, James Houghton <jthoughton@google.com> wrote:
> > > I think part of the problem in this case is that UFFDIO_COPY will
> > > install 4K pages (IIRC), I think a more natural way to fix this
> > > problem is to:
> > >
> > > 1. MADV_COLLAPSE after doing UFFDIO_COPY.
> > > 2. Make UFFDIO_COPY install PMDs when it is able to do so.
> > >
> > > These don't solve the exact same problem, but really userfaultfd
> > > should already try to install PMDs when it can (#2). If we have #2, #1
> > > is mostly a no-op.
> > >
> > > What do you think?
> >
> > Directly installing PMDs after an UFFD_COPY is something I already
> > investigated. I didn't mention it originally, since it touches exclusively
> > the MM module.  For some context, with that approach, in the same
> > benchmarks, fault latency on nested is reduced by 92%, and by 44% on metal,
> > which is significantly better than my proposal (which "only" improves by
> > 75% and 22% respectively).
> 
> Did you or one of your colleagues ever post it on list? I'm curious to see
> it. :)
> 
> > However, that approach only works when the copy is done in multiples of
> > 2MiB, and for some Firecracker use-cases, that's a no-go (I can elaborate
> > further if necessary, but the main problem is an explosion in incremental
> > snapshots size when managing memory in big chunks).  I might pursue this
> > proposal in a separate patch, however I'd love to work out a solution that
> > can be applied when userfaultfd works with smaller chunk sizes.
> 
> I see. So we really are dealing with 4K mappings.
> 
> In which case, prefetching at 2M does seem kind of arbitrary, which
> makes me even more in favor of this being mostly userspace-driven.

Letting userspace control the prefetch size is easy enough though, e.g. via a
module param or CAP.

I'm not opposed to the idea of KVM driving prefetching/prefaulting, I just don't
want to add a fourth version: indirect MMU, direct MMU, KVM_PRE_FAULT_MEMORY, and
now the TDP MMU.  Now that KVM_PRE_FAULT_MEMORY is a thing, I don't see any reason
why we can't use the core logic for KVM's own prefaulting.  At that point, using
the prefault flow would let us drop prefetching for direct shadow MMUs, i.e. would
be a net reduction in code and complexity.

Somewhat off the cuff and *very* lightly tested, but this seems to do what I want.
If it provides comparable performance, I'll write a changelog (or two?  e.g. to
have direct MMUs switch in a separate patch), and let Sashiko and other bots rip
apart my idea.

Note!  This has a hard dependency on in-flight prefaulting fixes[*].  Without
those, prefaulting will hang the vCPU if the root is invalidated.
[*] https://lore.kernel.org/all/20260806214050.78058-1-seanjc@google.com

Note #2!  The below deliberately ignores A/D-disabled MMUs.  I can't think of
any reason why it matters whether or not KVM can precisely detect accessed SPTEs,
all of the aging stuff is already extremely fuzzy.

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 79c450d677b4..7e67c5490502 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -118,6 +118,9 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(tdp_mmu_enabled);
 bool __read_mostly eager_page_split = true;
 module_param(eager_page_split, bool, 0644);
 
+unsigned int __read_mostly auto_prefault_nr_pages = KVM_PAGES_PER_HPAGE(PG_LEVEL_2M);
+module_param(auto_prefault_nr_pages, uint, 0644);
+
 static int max_huge_page_level __read_mostly;
 static int tdp_root_level __read_mostly;
 static int max_tdp_level __read_mostly;
@@ -3205,69 +3208,6 @@ static bool kvm_mmu_prefetch_sptes(struct kvm_vcpu *vcpu, gfn_t gfn, u64 *sptep,
 	return true;
 }
 
-static bool direct_pte_prefetch_many(struct kvm_vcpu *vcpu,
-				     struct kvm_mmu_page *sp,
-				     u64 *start, u64 *end)
-{
-	gfn_t gfn = kvm_mmu_page_get_gfn(sp, spte_index(start));
-	unsigned int access = sp->role.access;
-
-	return kvm_mmu_prefetch_sptes(vcpu, gfn, start, end - start, access);
-}
-
-static void __direct_pte_prefetch(struct kvm_vcpu *vcpu,
-				  struct kvm_mmu_page *sp, u64 *sptep)
-{
-	u64 *spte, *start = NULL;
-	int i;
-
-	WARN_ON_ONCE(!sp->role.direct);
-
-	i = spte_index(sptep) & ~(PTE_PREFETCH_NUM - 1);
-	spte = sp->spt + i;
-
-	for (i = 0; i < PTE_PREFETCH_NUM; i++, spte++) {
-		if (is_shadow_present_pte(*spte) || spte == sptep) {
-			if (!start)
-				continue;
-			if (!direct_pte_prefetch_many(vcpu, sp, start, spte))
-				return;
-
-			start = NULL;
-		} else if (!start)
-			start = spte;
-	}
-	if (start)
-		direct_pte_prefetch_many(vcpu, sp, start, spte);
-}
-
-static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)
-{
-	struct kvm_mmu_page *sp;
-
-	sp = sptep_to_sp(sptep);
-
-	/*
-	 * Without accessed bits, there's no way to distinguish between
-	 * actually accessed translations and prefetched, so disable pte
-	 * prefetch if accessed bits aren't available.
-	 */
-	if (sp_ad_disabled(sp))
-		return;
-
-	if (sp->role.level > PG_LEVEL_4K)
-		return;
-
-	/*
-	 * If addresses are being invalidated, skip prefetching to avoid
-	 * accidentally prefetching those addresses.
-	 */
-	if (unlikely(vcpu->kvm->mmu_invalidate_in_progress))
-		return;
-
-	__direct_pte_prefetch(vcpu, sp, sptep);
-}
-
 /*
  * Lookup the mapping level for @gfn in the current mm.
  *
@@ -6580,11 +6520,40 @@ static int kvm_mmu_write_protect_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 	return RET_PF_EMULATE;
 }
 
+static void kvm_mmu_auto_prefault(struct kvm_vcpu *vcpu, gpa_t start,
+				  u64 error_code, u8 level)
+{
+	gfn_t nr_pages = READ_ONCE(auto_prefault_nr_pages);
+	gfn_t i, o = KVM_PAGES_PER_HPAGE(level);
+	int nr_pages_msb;
+
+	if (unlikely(error_code & PFERR_RSVD_MASK))
+		return;
+
+	nr_pages = min(nr_pages, KVM_PAGES_PER_HPAGE(PG_LEVEL_1G));
+	nr_pages_msb = find_last_bit((unsigned long *)&nr_pages, sizeof(nr_pages));
+
+	start = ALIGN_DOWN(start, gfn_to_gpa(BIT_ULL(nr_pages_msb)));
+
+	for (i = KVM_PAGES_PER_HPAGE(level); i < nr_pages; i += KVM_PAGES_PER_HPAGE(level)) {
+		gpa_t gpa = start + gfn_to_gpa(i);
+
+		if (gpa < start || gpa_to_gfn(gpa) > kvm_mmu_max_gfn())
+			return;
+
+		if (kvm_tdp_page_prefault(vcpu, gpa, error_code, &level))
+			return;
+	}
+
+	pr_warn_ratelimited("Prefaulted ~%llu pages at %llx\n", i - o, start);
+}
+
 int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_code,
 		       void *insn, int insn_len)
 {
 	int r, emulation_type = EMULTYPE_PF;
 	bool direct = vcpu->arch.mmu->root_role.direct;
+	u8 level;
 
 	if (WARN_ON_ONCE(!VALID_PAGE(vcpu->arch.mmu->root.hpa)))
 		return RET_PF_RETRY;
@@ -6617,7 +6586,7 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err
 		vcpu->stat.pf_taken++;
 
 		r = kvm_mmu_do_page_fault(vcpu, cr2_or_gpa, error_code, false,
-					  &emulation_type, NULL);
+					  &emulation_type, &level);
 		if (KVM_BUG_ON(r == RET_PF_INVALID, vcpu->kvm))
 			return -EIO;
 	}
@@ -6628,6 +6597,8 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err
 	if (r == RET_PF_WRITE_PROTECTED)
 		r = kvm_mmu_write_protect_fault(vcpu, cr2_or_gpa, error_code,
 						&emulation_type);
+	else if (r == RET_PF_FIXED)
+		kvm_mmu_auto_prefault(vcpu, cr2_or_gpa, error_code, level);
 
 	if (r == RET_PF_FIXED)
 		vcpu->stat.pf_fixed++;
diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h
index 27427e7f22fa..b41b4b78cc81 100644
--- a/arch/x86/kvm/mmu/paging_tmpl.h
+++ b/arch/x86/kvm/mmu/paging_tmpl.h
@@ -617,7 +617,7 @@ static void FNAME(pte_prefetch)(struct kvm_vcpu *vcpu, struct guest_walker *gw,
 
 	sp = sptep_to_sp(sptep);
 
-	if (sp->role.level > PG_LEVEL_4K)
+	if (sp->role.level > PG_LEVEL_4K || sp->role.direct)
 		return;
 
 	/*
@@ -627,9 +627,6 @@ static void FNAME(pte_prefetch)(struct kvm_vcpu *vcpu, struct guest_walker *gw,
 	if (unlikely(vcpu->kvm->mmu_invalidate_in_progress))
 		return;
 
-	if (sp->role.direct)
-		return __direct_pte_prefetch(vcpu, sp, sptep);
-
 	i = spte_index(sptep) & ~(PTE_PREFETCH_NUM - 1);
 	spte = sp->spt + i;