[RFC] KVM: x86/mmu: Prefetch forward run of pages on TDP page faults

[RFC] KVM: x86/mmu: Prefetch forward run of pages on TDP page faults

[Marco Marangoni]

The TDP MMU installs one SPTE per fault, so faulting in guest memory via
userfaultfd (e.g. snapshot restore) without hugetlbfs costs a VM-exit per
4KiB page. Mirror the shadow MMU's prefetch: after a 4KiB fault, resolve
the forward run of host-present pages in the faulting leaf table (one
guest 2MiB region) with one non-blocking GUP and fill the empty SPTEs.
When userfaultfd populates in large chunks this maps up to 511 neighbours
per fault, cutting EPT violations up to 512x.

Touching 128MiB backed by userfaultfd with 2MiB UFFD_COPY chunks:
c8i.metal-48xl 129.1 -> 100.3 ms (-22%), nested 495.8 -> 121.5 ms (-75%).
With no batching (one copy per fault) there is a ~2-4% regression.

Best-effort and minimal: skips AD-disabled SPs, mirror (TDX) roots and
guest_memfd; installs atomically into empty entries only.

Add pf_prefetch_{called,pages,mapped,unused} stats.

Signed-off-by: Marco Marangoni <mamarang@amazon.com>
---

Hello, 
I'm Marco Marangoni, from AWS Firecracker.
This is the first time I (try to) submit a patch upstream. I'll do my best to respect your time, but I apologize in advance, since I'll get something wrong for sure :)

I'm looking for opportunities to improve VM snapshot restore time in case the guest memory is registered with UFFD, _without_ using hugetlbfs.

I found a pretty big opportunity in the x86 TDP MMU, borrowing an approach already used in the shadow MMU. By prefetching SPTEs around a faulting GFN, the number of EPT violations can be significantly reduced.
For example, if the UFFD handler copies memory in 2MiB chunks, it's theoretically possible to prefetch all 512 4KiB pages in a single VM exit, reducing the number of EPT violations by 512x.
In my tests, when using UFFD_COPY with 2MiB chunks, touching 128MiB of guest memory is 22% faster on a c8i.metal-48xl (from 129.1 ms to 100.3 ms), and 75% faster on nested virtualization (from 495.8 ms to 121.5 ms).
With UFFD using 512KiB chunks, it's 15.3% faster on metal, and 70% faster on nested with respect to baseline.

The attached patch is a proof of concept, and it's not ready to be merged.
It works by doing a fast GUP forward walk of up to 511 pages, limited to the faulting leaf page table (one guest 2 MiB region), stopping at the first already-mapped SPTE or the first host hole.
By design, this PoC is minimal; I'd be happy to extend the approach to ARM, guest_memfd, etc.

Before I polish it, I'd like your feedback on the approach chosen, specifically:
 - Since I need to store up to 512 struct page pointers returned by GUP, I added a pointer in the `kvm_vcpu_arch` struct to an auxiliary 4KiB page per vCPU. Any concerns?
 - Prefetching works forward only: in case the guest is accessing memory in reverse order, this patch won't help
 - There's a slight 2-4% performance regression when UFFD does not batch memory copy (i.e. prefetching does nothing). Do you think it's worth adding a way to enable/disable this prefetching mechanism?
 - SPTE install is open coded rather than adapting existing TDP helpers like tdp_mmu_set_spte_atomic. Is it worth adding a new helper or adapting existing ones?
 - When dirty logging is active, prefetch installs writable SPTEs and make_spte() eagerly marks each prefetched page dirty, adding pages to the dirty set that the guest never wrote. Should prefetch be suppressed on slots with dirty tracking enabled?

Some downsides can be fixed by some alternative patches I explored (let me know if you want me to share them):
 - "hierarchical prefetch": in this alternative approach, the prefetching happens in a "hierarchical" way, prefetching more and more pages in power of two increments, in batches of 8.
 - "direct prefetch": in this proof of concept, I experimented with skipping GUP entirely, and doing a lockless page walk directly in the KVM module, similarly to what `host_pfn_mapping_level` does
Performance-wise, they're similar to the attached patch on metal (within margin of error), and neither mitigates the regression in case UFFD doesn't batch memory. 
On the positive side, those can be implemented without extra allocations, and work both "forward and backward";
On the negative side, "hierarchical prefetch" is 10% slower on nested, and more complex. "direct prefetch" is relatively simple and fast, but feels like a hack, since it bypasses GUP.

It's worth mentioning that I also evaluated using the existing KVM_PRE_FAULT_MEMORY ioctl, but this doesn't work well for our use-case, as it requires the vCPU to be paused.

I look forward to your reply,
Marco

 arch/x86/include/asm/kvm_host.h | 11 ++++
 arch/x86/kvm/mmu/mmu.c          | 17 +++++-
 arch/x86/kvm/mmu/tdp_mmu.c      | 97 +++++++++++++++++++++++++++++++++
 arch/x86/kvm/x86.c              |  4 ++
 4 files changed, 128 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 6db5b5f79df9..3c369a4d0f7f 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -897,6 +897,13 @@ struct kvm_vcpu_arch {
 	 */
 	struct kvm_mmu_memory_cache mmu_external_spt_cache;
 
+	/*
+	 * Per-vCPU scratch (one page, 512 page ptrs) for the wide GUP done
+	 * during TDP fault-time prefetch.  Owning vCPU thread only; not for
+	 * zap/mmu-notifier contexts.
+	 */
+	struct page **mmu_prefetch_pages;
+
 	/*
 	 * QEMU userspace and the guest each have their own FPU state.
 	 * In vcpu_run, we switch between the user and guest FPU contexts.
@@ -1722,6 +1729,10 @@ struct kvm_vcpu_stat {
 	u64 pf_fast;
 	u64 pf_mmio_spte_created;
 	u64 pf_guest;
+	u64 pf_prefetch_called;
+	u64 pf_prefetch_pages;
+	u64 pf_prefetch_mapped;
+	u64 pf_prefetch_unused;
 	u64 tlb_flush;
 	u64 invlpg;
 
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index a61750f8e1e3..17b03eceea0b 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -636,6 +636,11 @@ static void mmu_free_memory_caches(struct kvm_vcpu *vcpu)
 	kvm_mmu_free_memory_cache(&vcpu->arch.mmu_shadowed_info_cache);
 	kvm_mmu_free_memory_cache(&vcpu->arch.mmu_external_spt_cache);
 	kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_header_cache);
+
+	if (vcpu->arch.mmu_prefetch_pages) {
+		free_page((unsigned long)vcpu->arch.mmu_prefetch_pages);
+		vcpu->arch.mmu_prefetch_pages = NULL;
+	}
 }
 
 static void mmu_free_pte_list_desc(struct pte_list_desc *pte_list_desc)
@@ -6808,6 +6813,13 @@ int kvm_mmu_create(struct kvm_vcpu *vcpu)
 {
 	int ret;
 
+	/* Prefetch scratch: one page of 512 page ptrs.  Freed via kvm_mmu_destroy(). */
+	BUILD_BUG_ON(SPTE_ENT_PER_PAGE * sizeof(struct page *) > PAGE_SIZE);
+	vcpu->arch.mmu_prefetch_pages =
+		(struct page **)__get_free_page(GFP_KERNEL_ACCOUNT);
+	if (!vcpu->arch.mmu_prefetch_pages)
+		return -ENOMEM;
+
 	vcpu->arch.mmu_pte_list_desc_cache.kmem_cache = pte_list_desc_cache;
 	vcpu->arch.mmu_pte_list_desc_cache.gfp_zero = __GFP_ZERO;
 
@@ -6824,7 +6836,7 @@ int kvm_mmu_create(struct kvm_vcpu *vcpu)
 
 	ret = __kvm_mmu_create(vcpu, &vcpu->arch.guest_mmu);
 	if (ret)
-		return ret;
+		goto fail_prefetch;
 
 	ret = __kvm_mmu_create(vcpu, &vcpu->arch.root_mmu);
 	if (ret)
@@ -6833,6 +6845,9 @@ int kvm_mmu_create(struct kvm_vcpu *vcpu)
 	return ret;
  fail_allocate_root:
 	free_mmu_pages(&vcpu->arch.guest_mmu);
+ fail_prefetch:
+	free_page((unsigned long)vcpu->arch.mmu_prefetch_pages);
+	vcpu->arch.mmu_prefetch_pages = NULL;
 	return ret;
 }
 
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index c1cbae65d239..f5c4b6bbc2a1 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -1209,6 +1209,99 @@ static int tdp_mmu_link_sp(struct kvm *kvm, struct tdp_iter *iter,
 static int tdp_mmu_split_huge_page(struct kvm *kvm, struct tdp_iter *iter,
 				   struct kvm_mmu_page *sp, bool shared);
 
+/*
+ * Prefetch the forward run of host-present pages after the fault, within the
+ * faulting leaf table (512 pages).  One non-blocking GUP fills the empty SPTEs.
+ * Forward only; capped at the first present SPTE and the first host hole.
+ */
+static void tdp_mmu_pte_prefetch(struct kvm_vcpu *vcpu,
+				 struct kvm_page_fault *fault,
+				 struct tdp_iter *iter)
+{
+	struct kvm_mmu_page *sp = sptep_to_sp(rcu_dereference(iter->sptep));
+	struct page **pages = vcpu->arch.mmu_prefetch_pages;
+	struct kvm_memory_slot *slot = fault->slot;
+	unsigned int access = sp->role.access;
+	bool host_writable = !(slot->flags & KVM_MEM_READONLY);
+	gfn_t start_gfn, slot_end;
+	int start, count, nr, i;
+
+	if (sp_ad_disabled(sp))
+		return;
+
+	/* Mirror (TDX) needs set_external_spte(); gmem pfns aren't in GUP's tables. */
+	if (is_mirror_sp(sp) || kvm_slot_has_gmem(slot))
+		return;
+
+	/* Racing invalidation may be stale.  No mmu_seq recheck: GUP is under the lock. */
+	if (unlikely(vcpu->kvm->mmu_invalidate_in_progress))
+		return;
+
+	if (WARN_ON_ONCE(!pages))
+		return;
+
+	/* Forward window: after the fault to end of table, clamped to the slot. */
+	start = spte_index(rcu_dereference(iter->sptep)) + 1;
+	if (start >= SPTE_ENT_PER_PAGE)
+		return;			/* fault on the last entry */
+
+	start_gfn = sp->gfn + start;
+	slot_end = slot->base_gfn + slot->npages;
+	if (start_gfn >= slot_end)
+		return;			/* fault on the slot's last page */
+
+	count = min_t(gfn_t, SPTE_ENT_PER_PAGE - start, slot_end - start_gfn);
+
+	/* Bound the GUP at the first present SPTE (just a bound; install re-checks). */
+	for (i = 0; i < count; i++) {
+		u64 spte = READ_ONCE(sp->spt[start + i]);
+
+		if (is_shadow_present_pte(spte) || spte != SHADOW_NONPRESENT_VALUE)
+			break;
+	}
+	count = i;
+	if (!count)
+		return;
+
+	vcpu->stat.pf_prefetch_called++;
+
+	/* Non-blocking GUP; stops at the first host hole. */
+	nr = kvm_prefetch_pages(slot, start_gfn, pages, count);
+	if (nr <= 0)
+		return;
+
+	vcpu->stat.pf_prefetch_pages += nr;
+
+	for (i = 0; i < nr; i++) {
+		u64 *sptep = sp->spt + start + i;
+		u64 old_spte = SHADOW_NONPRESENT_VALUE;
+		gfn_t gfn = start_gfn + i;
+		u64 new_spte;
+
+		make_spte(vcpu, sp, slot, access, gfn,
+			  page_to_pfn(pages[i]), old_spte,
+			  true /* prefetch */, false, host_writable, &new_spte);
+
+		/* cmpxchg from empty is the race check; present/MMIO/frozen fails it. */
+		if (try_cmpxchg64(sptep, &old_spte, new_spte)) {
+			handle_changed_spte(vcpu->kvm, sp, gfn,
+					    SHADOW_NONPRESENT_VALUE,
+					    new_spte, PG_LEVEL_4K, true);
+			vcpu->stat.pf_prefetch_mapped++;
+
+			/* Mark dirty only if mapped writable. */
+			if (host_writable)
+				kvm_release_page_dirty(pages[i]);
+			else
+				kvm_release_page_clean(pages[i]);
+		} else {
+			/* Present/raced; the winner dirties its own pin. */
+			kvm_release_page_clean(pages[i]);
+			vcpu->stat.pf_prefetch_unused++;
+		}
+	}
+}
+
 /*
  * Handle a TDP page fault (NPT/EPT violation/misconfiguration) by installing
  * page tables and SPTEs to translate the faulting guest physical address.
@@ -1297,6 +1390,10 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 map_target_level:
 	ret = tdp_mmu_map_handle_target_level(vcpu, fault, &iter);
 
+	if (ret == RET_PF_FIXED && fault->goal_level == PG_LEVEL_4K &&
+	    !fault->prefetch && fault->slot)
+		tdp_mmu_pte_prefetch(vcpu, fault, &iter);
+
 retry:
 	rcu_read_unlock();
 	return ret;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 69469bbdc84a..ed8773824f8f 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -271,6 +271,10 @@ const struct kvm_stats_desc kvm_vcpu_stats_desc[] = {
 	STATS_DESC_COUNTER(VCPU, pf_fast),
 	STATS_DESC_COUNTER(VCPU, pf_mmio_spte_created),
 	STATS_DESC_COUNTER(VCPU, pf_guest),
+	STATS_DESC_COUNTER(VCPU, pf_prefetch_called),
+	STATS_DESC_COUNTER(VCPU, pf_prefetch_pages),
+	STATS_DESC_COUNTER(VCPU, pf_prefetch_mapped),
+	STATS_DESC_COUNTER(VCPU, pf_prefetch_unused),
 	STATS_DESC_COUNTER(VCPU, tlb_flush),
 	STATS_DESC_COUNTER(VCPU, invlpg),
 	STATS_DESC_COUNTER(VCPU, exits),
-- 
2.47.3

Re: [RFC] KVM: x86/mmu: Prefetch forward run of pages on TDP page faults

[Sean Christopherson]

On Tue, Aug 25, 2026, Marco Marangoni wrote:
> It's worth mentioning that I also evaluated using the existing
> KVM_PRE_FAULT_MEMORY ioctl, but this doesn't work well for our use-case, as
> it requires the vCPU to be paused.

What about if/when KVM Userfault[*] comes along?  I.e. pre-fault memory when the
vCPU exits to userspace.

If KVM Userfault + KVM_PRE_FAULT_MEMORY is viable for Firecracker and provides
the desired performance, I would strongly prefer that approach over KVM doing its
own prefetching.  That way other architectures don't need to implement prefetching
as well (arm64 already supports KVM_PRE_FAULT_MEMORY, and RISC-V has patches posted).

I'm aiming to get back to that series "soon", this would simply be more motivation
to get KVM Userfault landed.

[*] https://lore.kernel.org/all/20250618042424.330664-1-jthoughton@google.com

Re: [RFC] KVM: x86/mmu: Prefetch forward run of pages on TDP page faults

[James Houghton]

On Tue, Aug 25, 2026 at 8:00 AM Sean Christopherson <seanjc@google.com> wrote:
>
> On Tue, Aug 25, 2026, Marco Marangoni wrote:
> > It's worth mentioning that I also evaluated using the existing
> > KVM_PRE_FAULT_MEMORY ioctl, but this doesn't work well for our use-case, as
> > it requires the vCPU to be paused.
>
> What about if/when KVM Userfault[*] comes along?  I.e. pre-fault memory when the
> vCPU exits to userspace.

My understanding is that Firecracker performs its userfaultfd
operations in a separate process. KVM Userfault doesn't break this, it
just means Firecracker would need a way to signal the other process
when we get KVM_EXIT_USERFAULTs. Firecracker might be using async PF;
using KVM Userfault basically disables async PF.

> If KVM Userfault + KVM_PRE_FAULT_MEMORY is viable for Firecracker and provides
> the desired performance, I would strongly prefer that approach over KVM doing its
> own prefetching.  That way other architectures don't need to implement prefetching
> as well (arm64 already supports KVM_PRE_FAULT_MEMORY, and RISC-V has patches posted).

I also really like this approach if it is workable for Firecracker.

> I'm aiming to get back to that series "soon", this would simply be more motivation
> to get KVM Userfault landed.

Yay!

> [*] https://lore.kernel.org/all/20250618042424.330664-1-jthoughton@google.com
>

Re: [RFC] KVM: x86/mmu: Prefetch forward run of pages on TDP page faults

[James Houghton]

On Tue, Aug 25, 2026 at 7:27 AM Marco Marangoni <mamarang@amazon.com> wrote:
>
> The TDP MMU installs one SPTE per fault, so faulting in guest memory via
> userfaultfd (e.g. snapshot restore) without hugetlbfs costs a VM-exit per
> 4KiB page. Mirror the shadow MMU's prefetch: after a 4KiB fault, resolve
> the forward run of host-present pages in the faulting leaf table (one
> guest 2MiB region) with one non-blocking GUP and fill the empty SPTEs.
> When userfaultfd populates in large chunks this maps up to 511 neighbours
> per fault, cutting EPT violations up to 512x.
>
> Touching 128MiB backed by userfaultfd with 2MiB UFFD_COPY chunks:
> c8i.metal-48xl 129.1 -> 100.3 ms (-22%), nested 495.8 -> 121.5 ms (-75%).
> With no batching (one copy per fault) there is a ~2-4% regression.

Hi Marco!

I think part of the problem in this case is that UFFDIO_COPY will
install 4K pages (IIRC), I think a more natural way to fix this
problem is to:

1. MADV_COLLAPSE after doing UFFDIO_COPY.
2. Make UFFDIO_COPY install PMDs when it is able to do so.

These don't solve the exact same problem, but really userfaultfd
should already try to install PMDs when it can (#2). If we have #2, #1
is mostly a no-op.

What do you think?

Re: [RFC] KVM: x86/mmu: Prefetch forward run of pages on TDP page faults

[Marangoni, Marco]

Thanks for the replies!

On Tue, Aug 25, 2026, Sean Christopherson <seanjc@google.com> wrote:
>
> On Tue, Aug 25, 2026, Marco Marangoni wrote:
> > It's worth mentioning that I also evaluated using the existing
> > KVM_PRE_FAULT_MEMORY ioctl, but this doesn't work well for our use-case, as
> > it requires the vCPU to be paused.
>
> What about if/when KVM Userfault[*] comes along?  I.e. pre-fault memory when the
> vCPU exits to userspace.

KVM Userfault + KVM_PRE_FAULT_MEMORY is a valid suggestion, however if possible we'd like to have _both_ async page faults and prefetching.
I haven't tested async PF together with prefetching, but tested separately, both improvements yield great results, so it would be a shame to have to choose.

On Tue, Aug 25, 2026, James Houghton <jthoughton@google.com> wrote:
> I think part of the problem in this case is that UFFDIO_COPY will
> install 4K pages (IIRC), I think a more natural way to fix this
> problem is to:
> 
> 1. MADV_COLLAPSE after doing UFFDIO_COPY.
> 2. Make UFFDIO_COPY install PMDs when it is able to do so.
> 
> These don't solve the exact same problem, but really userfaultfd
> should already try to install PMDs when it can (#2). If we have #2, #1
> is mostly a no-op.
> 
> What do you think?

Directly installing PMDs after an UFFD_COPY is something I already investigated. I didn't mention it originally, since it touches exclusively the MM module.
For some context, with that approach, in the same benchmarks, fault latency on nested is reduced by 92%, and by 44% on metal, which is significantly better than my proposal (which "only" improves by 75% and 22% respectively).
However, that approach only works when the copy is done in multiples of 2MiB, and for some Firecracker use-cases, that's a no-go (I can elaborate further if necessary, but the main problem is an explosion in incremental snapshots size when managing memory in big chunks).
I might pursue this proposal in a separate patch, however I'd love to work out a solution that can be applied when userfaultfd works with smaller chunk sizes.

Re: [RFC] KVM: x86/mmu: Prefetch forward run of pages on TDP page faults

[James Houghton]

On Tue, Aug 25, 2026 at 10:38 AM Marangoni, Marco <mamarang@amazon.com> wrote:
>
> Thanks for the replies!
>
> On Tue, Aug 25, 2026, Sean Christopherson <seanjc@google.com> wrote:
> >
> > On Tue, Aug 25, 2026, Marco Marangoni wrote:
> > > It's worth mentioning that I also evaluated using the existing
> > > KVM_PRE_FAULT_MEMORY ioctl, but this doesn't work well for our use-case, as
> > > it requires the vCPU to be paused.
> >
> > What about if/when KVM Userfault[*] comes along?  I.e. pre-fault memory when the
> > vCPU exits to userspace.
>
> KVM Userfault + KVM_PRE_FAULT_MEMORY is a valid suggestion, however if possible we'd like to have _both_ async page faults and prefetching.
> I haven't tested async PF together with prefetching, but tested separately, both improvements yield great results, so it would be a shame to have to choose.

What if you used KVM_PRE_FAULT_MEMORY without KVM Userfault?

If you want to avoid pausing a vCPU, what if you made another vCPU
(KVM_CREATE_VCPU) and used that solely for prefaulting guest memory? I
haven't really looked into that before... I'm guessing there's
something fundamentally wrong with this approach.

If this doesn't work (and a VM-scoped KVM_PRE_FAULT_MEMORY doesn't
make sense either), then perhaps the TDP MMU prefetching logic makes
sense.

> On Tue, Aug 25, 2026, James Houghton <jthoughton@google.com> wrote:
> > I think part of the problem in this case is that UFFDIO_COPY will
> > install 4K pages (IIRC), I think a more natural way to fix this
> > problem is to:
> >
> > 1. MADV_COLLAPSE after doing UFFDIO_COPY.
> > 2. Make UFFDIO_COPY install PMDs when it is able to do so.
> >
> > These don't solve the exact same problem, but really userfaultfd
> > should already try to install PMDs when it can (#2). If we have #2, #1
> > is mostly a no-op.
> >
> > What do you think?
>
> Directly installing PMDs after an UFFD_COPY is something I already investigated. I didn't mention it originally, since it touches exclusively the MM module.
> For some context, with that approach, in the same benchmarks, fault latency on nested is reduced by 92%, and by 44% on metal, which is significantly better than my proposal (which "only" improves by 75% and 22% respectively).

Did you or one of your colleagues ever post it on list? I'm curious to
see it. :)

> However, that approach only works when the copy is done in multiples of 2MiB, and for some Firecracker use-cases, that's a no-go (I can elaborate further if necessary, but the main problem is an explosion in incremental snapshots size when managing memory in big chunks).
> I might pursue this proposal in a separate patch, however I'd love to work out a solution that can be applied when userfaultfd works with smaller chunk sizes.

I see. So we really are dealing with 4K mappings.

In which case, prefetching at 2M does seem kind of arbitrary, which
makes me even more in favor of this being mostly userspace-driven.

Re: [RFC] KVM: x86/mmu: Prefetch forward run of pages on TDP page faults

[Marangoni, Marco]

On Tue, Aug 25, 2026, James Houghton <jthoughton@google.com> wrote:
>
> On Tue, Aug 25, 2026 at 10:38 AM Marangoni, Marco <mamarang@amazon.com> wrote:
> >
> > Thanks for the replies!
> >
> > On Tue, Aug 25, 2026, Sean Christopherson <seanjc@google.com> wrote:
> > >
> > > On Tue, Aug 25, 2026, Marco Marangoni wrote:
> > > > It's worth mentioning that I also evaluated using the existing
> > > > KVM_PRE_FAULT_MEMORY ioctl, but this doesn't work well for our use-case, as
> > > > it requires the vCPU to be paused.
> > >
> > > What about if/when KVM Userfault[*] comes along?  I.e. pre-fault memory when the
> > > vCPU exits to userspace.
> >
> > KVM Userfault + KVM_PRE_FAULT_MEMORY is a valid suggestion, however if possible we'd like to have _both_ async page faults and prefetching.
> > I haven't tested async PF together with prefetching, but tested separately, both improvements yield great results, so it would be a shame to have to choose.
> 
> What if you used KVM_PRE_FAULT_MEMORY without KVM Userfault?
> 
> If you want to avoid pausing a vCPU, what if you made another vCPU
> (KVM_CREATE_VCPU) and used that solely for prefaulting guest memory? I
> haven't really looked into that before... I'm guessing there's
> something fundamentally wrong with this approach.
> 
> If this doesn't work (and a VM-scoped KVM_PRE_FAULT_MEMORY doesn't
> make sense either), then perhaps the TDP MMU prefetching logic makes
> sense.

~~Unfortunately~~ Fortunately I'll be on Holidays for the next few weeks; I'll try the permanently paused vCPU approach as soon as I can.
In principle it could solve the problem without requiring any kernel changes.


On Tue, Aug 25, 2026, James Houghton <jthoughton@google.com> wrote:
>
> > On Tue, Aug 25, 2026, James Houghton <jthoughton@google.com> wrote:
> > > I think part of the problem in this case is that UFFDIO_COPY will
> > > install 4K pages (IIRC), I think a more natural way to fix this
> > > problem is to:
> > >
> > > 1. MADV_COLLAPSE after doing UFFDIO_COPY.
> > > 2. Make UFFDIO_COPY install PMDs when it is able to do so.
> > >
> > > These don't solve the exact same problem, but really userfaultfd
> > > should already try to install PMDs when it can (#2). If we have #2, #1
> > > is mostly a no-op.
> > >
> > > What do you think?
> >
> > Directly installing PMDs after an UFFD_COPY is something I already investigated. I didn't mention it originally, since it touches exclusively the MM module.
> > For some context, with that approach, in the same benchmarks, fault latency on nested is reduced by 92%, and by 44% on metal, which is significantly better than my > proposal (which "only" improves by 75% and 22% respectively).
> 
> Did you or one of your colleagues ever post it on list? I'm curious to
> see it. :)

I wanted to tackle the use-case of arbitrary UFFD chunks first, so no, not yet at least.
If you're curious, the proof-of-concept is on https://github.com/marco-marangoni/firecracker/blob/thp-uffd/resources/patches/thp_uffd.patch; it lacks mTHP support and only handles UFFD_COPY.


On Tue, Aug 25, 2026, James Houghton <jthoughton@google.com> wrote:
>
> > However, that approach only works when the copy is done in multiples of 2MiB, and for some Firecracker use-cases, that's a no-go (I can elaborate further if necessary, but the main problem is an explosion in incremental snapshots size when managing memory in big chunks).
> > I might pursue this proposal in a separate patch, however I'd love to work out a solution that can be applied when userfaultfd works with smaller chunk sizes.
> 
> I see. So we really are dealing with 4K mappings.
> 
> In which case, prefetching at 2M does seem kind of arbitrary, which
> makes me even more in favor of this being mostly userspace-driven.

Yes, 2MiB is arbitrary. 
One of the possible implementation for this prefetching patch is to skip the GUP and create the EPT PT directly from the page table, in which case 2MiB is natural.


Thanks again for the interesting replies, I'll try the KVM_PRE_FAULT_MEMORY with a paused vCPU when I'm back from PTO.

Marco

Re: [RFC] KVM: x86/mmu: Prefetch forward run of pages on TDP page faults

[Sean Christopherson]

On Tue, Aug 25, 2026, James Houghton wrote:
> On Tue, Aug 25, 2026 at 10:38 AM Marangoni, Marco <mamarang@amazon.com> wrote:
> >
> > Thanks for the replies!
> >
> > On Tue, Aug 25, 2026, Sean Christopherson <seanjc@google.com> wrote:
> > >
> > > On Tue, Aug 25, 2026, Marco Marangoni wrote:
> > > > It's worth mentioning that I also evaluated using the existing
> > > > KVM_PRE_FAULT_MEMORY ioctl, but this doesn't work well for our use-case, as
> > > > it requires the vCPU to be paused.
> > >
> > > What about if/when KVM Userfault[*] comes along?  I.e. pre-fault memory when the
> > > vCPU exits to userspace.
> >
> > KVM Userfault + KVM_PRE_FAULT_MEMORY is a valid suggestion, however if
> > possible we'd like to have _both_ async page faults and prefetching.  I
> > haven't tested async PF together with prefetching, but tested separately,
> > both improvements yield great results, so it would be a shame to have to
> > choose.

Hrm, right.  Although _if_ we can figure out a clever way to allow prefaulting
without a vCPU (or at least, a real vCPU), that would allow both to coexist.
But at that point I'm probably just being extremely stubborn. :-)
 
> What if you used KVM_PRE_FAULT_MEMORY without KVM Userfault?
> 
> If you want to avoid pausing a vCPU, what if you made another vCPU
> (KVM_CREATE_VCPU) and used that solely for prefaulting guest memory? I
> haven't really looked into that before... I'm guessing there's
> something fundamentally wrong with this approach.

Yeah, shoving a vCPU into the VM that shouldn't exist probably won't end well.
E.g. on x86, the vCPU would kinda sorta be visible/reachable by the guest as the
fake vCPU would respond to IRQs and whatnot.

> If this doesn't work (and a VM-scoped KVM_PRE_FAULT_MEMORY doesn't
> make sense either), then perhaps the TDP MMU prefetching logic makes
> sense.
> 
> > On Tue, Aug 25, 2026, James Houghton <jthoughton@google.com> wrote:
> > > I think part of the problem in this case is that UFFDIO_COPY will
> > > install 4K pages (IIRC), I think a more natural way to fix this
> > > problem is to:
> > >
> > > 1. MADV_COLLAPSE after doing UFFDIO_COPY.
> > > 2. Make UFFDIO_COPY install PMDs when it is able to do so.
> > >
> > > These don't solve the exact same problem, but really userfaultfd
> > > should already try to install PMDs when it can (#2). If we have #2, #1
> > > is mostly a no-op.
> > >
> > > What do you think?
> >
> > Directly installing PMDs after an UFFD_COPY is something I already
> > investigated. I didn't mention it originally, since it touches exclusively
> > the MM module.  For some context, with that approach, in the same
> > benchmarks, fault latency on nested is reduced by 92%, and by 44% on metal,
> > which is significantly better than my proposal (which "only" improves by
> > 75% and 22% respectively).
> 
> Did you or one of your colleagues ever post it on list? I'm curious to see
> it. :)
> 
> > However, that approach only works when the copy is done in multiples of
> > 2MiB, and for some Firecracker use-cases, that's a no-go (I can elaborate
> > further if necessary, but the main problem is an explosion in incremental
> > snapshots size when managing memory in big chunks).  I might pursue this
> > proposal in a separate patch, however I'd love to work out a solution that
> > can be applied when userfaultfd works with smaller chunk sizes.
> 
> I see. So we really are dealing with 4K mappings.
> 
> In which case, prefetching at 2M does seem kind of arbitrary, which
> makes me even more in favor of this being mostly userspace-driven.

Letting userspace control the prefetch size is easy enough though, e.g. via a
module param or CAP.

I'm not opposed to the idea of KVM driving prefetching/prefaulting, I just don't
want to add a fourth version: indirect MMU, direct MMU, KVM_PRE_FAULT_MEMORY, and
now the TDP MMU.  Now that KVM_PRE_FAULT_MEMORY is a thing, I don't see any reason
why we can't use the core logic for KVM's own prefaulting.  At that point, using
the prefault flow would let us drop prefetching for direct shadow MMUs, i.e. would
be a net reduction in code and complexity.

Somewhat off the cuff and *very* lightly tested, but this seems to do what I want.
If it provides comparable performance, I'll write a changelog (or two?  e.g. to
have direct MMUs switch in a separate patch), and let Sashiko and other bots rip
apart my idea.

Note!  This has a hard dependency on in-flight prefaulting fixes[*].  Without
those, prefaulting will hang the vCPU if the root is invalidated.
[*] https://lore.kernel.org/all/20260806214050.78058-1-seanjc@google.com

Note #2!  The below deliberately ignores A/D-disabled MMUs.  I can't think of
any reason why it matters whether or not KVM can precisely detect accessed SPTEs,
all of the aging stuff is already extremely fuzzy.

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 79c450d677b4..7e67c5490502 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -118,6 +118,9 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(tdp_mmu_enabled);
 bool __read_mostly eager_page_split = true;
 module_param(eager_page_split, bool, 0644);
 
+unsigned int __read_mostly auto_prefault_nr_pages = KVM_PAGES_PER_HPAGE(PG_LEVEL_2M);
+module_param(auto_prefault_nr_pages, uint, 0644);
+
 static int max_huge_page_level __read_mostly;
 static int tdp_root_level __read_mostly;
 static int max_tdp_level __read_mostly;
@@ -3205,69 +3208,6 @@ static bool kvm_mmu_prefetch_sptes(struct kvm_vcpu *vcpu, gfn_t gfn, u64 *sptep,
 	return true;
 }
 
-static bool direct_pte_prefetch_many(struct kvm_vcpu *vcpu,
-				     struct kvm_mmu_page *sp,
-				     u64 *start, u64 *end)
-{
-	gfn_t gfn = kvm_mmu_page_get_gfn(sp, spte_index(start));
-	unsigned int access = sp->role.access;
-
-	return kvm_mmu_prefetch_sptes(vcpu, gfn, start, end - start, access);
-}
-
-static void __direct_pte_prefetch(struct kvm_vcpu *vcpu,
-				  struct kvm_mmu_page *sp, u64 *sptep)
-{
-	u64 *spte, *start = NULL;
-	int i;
-
-	WARN_ON_ONCE(!sp->role.direct);
-
-	i = spte_index(sptep) & ~(PTE_PREFETCH_NUM - 1);
-	spte = sp->spt + i;
-
-	for (i = 0; i < PTE_PREFETCH_NUM; i++, spte++) {
-		if (is_shadow_present_pte(*spte) || spte == sptep) {
-			if (!start)
-				continue;
-			if (!direct_pte_prefetch_many(vcpu, sp, start, spte))
-				return;
-
-			start = NULL;
-		} else if (!start)
-			start = spte;
-	}
-	if (start)
-		direct_pte_prefetch_many(vcpu, sp, start, spte);
-}
-
-static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)
-{
-	struct kvm_mmu_page *sp;
-
-	sp = sptep_to_sp(sptep);
-
-	/*
-	 * Without accessed bits, there's no way to distinguish between
-	 * actually accessed translations and prefetched, so disable pte
-	 * prefetch if accessed bits aren't available.
-	 */
-	if (sp_ad_disabled(sp))
-		return;
-
-	if (sp->role.level > PG_LEVEL_4K)
-		return;
-
-	/*
-	 * If addresses are being invalidated, skip prefetching to avoid
-	 * accidentally prefetching those addresses.
-	 */
-	if (unlikely(vcpu->kvm->mmu_invalidate_in_progress))
-		return;
-
-	__direct_pte_prefetch(vcpu, sp, sptep);
-}
-
 /*
  * Lookup the mapping level for @gfn in the current mm.
  *
@@ -6580,11 +6520,40 @@ static int kvm_mmu_write_protect_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 	return RET_PF_EMULATE;
 }
 
+static void kvm_mmu_auto_prefault(struct kvm_vcpu *vcpu, gpa_t start,
+				  u64 error_code, u8 level)
+{
+	gfn_t nr_pages = READ_ONCE(auto_prefault_nr_pages);
+	gfn_t i, o = KVM_PAGES_PER_HPAGE(level);
+	int nr_pages_msb;
+
+	if (unlikely(error_code & PFERR_RSVD_MASK))
+		return;
+
+	nr_pages = min(nr_pages, KVM_PAGES_PER_HPAGE(PG_LEVEL_1G));
+	nr_pages_msb = find_last_bit((unsigned long *)&nr_pages, sizeof(nr_pages));
+
+	start = ALIGN_DOWN(start, gfn_to_gpa(BIT_ULL(nr_pages_msb)));
+
+	for (i = KVM_PAGES_PER_HPAGE(level); i < nr_pages; i += KVM_PAGES_PER_HPAGE(level)) {
+		gpa_t gpa = start + gfn_to_gpa(i);
+
+		if (gpa < start || gpa_to_gfn(gpa) > kvm_mmu_max_gfn())
+			return;
+
+		if (kvm_tdp_page_prefault(vcpu, gpa, error_code, &level))
+			return;
+	}
+
+	pr_warn_ratelimited("Prefaulted ~%llu pages at %llx\n", i - o, start);
+}
+
 int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_code,
 		       void *insn, int insn_len)
 {
 	int r, emulation_type = EMULTYPE_PF;
 	bool direct = vcpu->arch.mmu->root_role.direct;
+	u8 level;
 
 	if (WARN_ON_ONCE(!VALID_PAGE(vcpu->arch.mmu->root.hpa)))
 		return RET_PF_RETRY;
@@ -6617,7 +6586,7 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err
 		vcpu->stat.pf_taken++;
 
 		r = kvm_mmu_do_page_fault(vcpu, cr2_or_gpa, error_code, false,
-					  &emulation_type, NULL);
+					  &emulation_type, &level);
 		if (KVM_BUG_ON(r == RET_PF_INVALID, vcpu->kvm))
 			return -EIO;
 	}
@@ -6628,6 +6597,8 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err
 	if (r == RET_PF_WRITE_PROTECTED)
 		r = kvm_mmu_write_protect_fault(vcpu, cr2_or_gpa, error_code,
 						&emulation_type);
+	else if (r == RET_PF_FIXED)
+		kvm_mmu_auto_prefault(vcpu, cr2_or_gpa, error_code, level);
 
 	if (r == RET_PF_FIXED)
 		vcpu->stat.pf_fixed++;
diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h
index 27427e7f22fa..b41b4b78cc81 100644
--- a/arch/x86/kvm/mmu/paging_tmpl.h
+++ b/arch/x86/kvm/mmu/paging_tmpl.h
@@ -617,7 +617,7 @@ static void FNAME(pte_prefetch)(struct kvm_vcpu *vcpu, struct guest_walker *gw,
 
 	sp = sptep_to_sp(sptep);
 
-	if (sp->role.level > PG_LEVEL_4K)
+	if (sp->role.level > PG_LEVEL_4K || sp->role.direct)
 		return;
 
 	/*
@@ -627,9 +627,6 @@ static void FNAME(pte_prefetch)(struct kvm_vcpu *vcpu, struct guest_walker *gw,
 	if (unlikely(vcpu->kvm->mmu_invalidate_in_progress))
 		return;
 
-	if (sp->role.direct)
-		return __direct_pte_prefetch(vcpu, sp, sptep);
-
 	i = spte_index(sptep) & ~(PTE_PREFETCH_NUM - 1);
 	spte = sp->spt + i;

Re: [RFC] KVM: x86/mmu: Prefetch forward run of pages on TDP page faults

[Sean Christopherson]

On Tue, Aug 25, 2026, Sean Christopherson wrote:
> Somewhat off the cuff and *very* lightly tested, but this seems to do what I want.
> If it provides comparable performance, I'll write a changelog (or two?  e.g. to
> have direct MMUs switch in a separate patch), and let Sashiko and other bots rip
> apart my idea.
> 
> Note!  This has a hard dependency on in-flight prefaulting fixes[*].  Without
> those, prefaulting will hang the vCPU if the root is invalidated.
> [*] https://lore.kernel.org/all/20260806214050.78058-1-seanjc@google.com
> 
> Note #2!  The below deliberately ignores A/D-disabled MMUs.  I can't think of
> any reason why it matters whether or not KVM can precisely detect accessed SPTEs,
> all of the aging stuff is already extremely fuzzy.

And of course I posted an untested version (I ripped out the direct MMU prefetching
as an afterthough, and dropped a printk).  This version should actually compile.

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 79c450d677b4..88b6aa1f840f 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -118,6 +118,9 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(tdp_mmu_enabled);
 bool __read_mostly eager_page_split = true;
 module_param(eager_page_split, bool, 0644);
 
+unsigned int __read_mostly auto_prefault_nr_pages = KVM_PAGES_PER_HPAGE(PG_LEVEL_2M);
+module_param(auto_prefault_nr_pages, uint, 0644);
+
 static int max_huge_page_level __read_mostly;
 static int tdp_root_level __read_mostly;
 static int max_tdp_level __read_mostly;
@@ -3205,69 +3208,6 @@ static bool kvm_mmu_prefetch_sptes(struct kvm_vcpu *vcpu, gfn_t gfn, u64 *sptep,
 	return true;
 }
 
-static bool direct_pte_prefetch_many(struct kvm_vcpu *vcpu,
-				     struct kvm_mmu_page *sp,
-				     u64 *start, u64 *end)
-{
-	gfn_t gfn = kvm_mmu_page_get_gfn(sp, spte_index(start));
-	unsigned int access = sp->role.access;
-
-	return kvm_mmu_prefetch_sptes(vcpu, gfn, start, end - start, access);
-}
-
-static void __direct_pte_prefetch(struct kvm_vcpu *vcpu,
-				  struct kvm_mmu_page *sp, u64 *sptep)
-{
-	u64 *spte, *start = NULL;
-	int i;
-
-	WARN_ON_ONCE(!sp->role.direct);
-
-	i = spte_index(sptep) & ~(PTE_PREFETCH_NUM - 1);
-	spte = sp->spt + i;
-
-	for (i = 0; i < PTE_PREFETCH_NUM; i++, spte++) {
-		if (is_shadow_present_pte(*spte) || spte == sptep) {
-			if (!start)
-				continue;
-			if (!direct_pte_prefetch_many(vcpu, sp, start, spte))
-				return;
-
-			start = NULL;
-		} else if (!start)
-			start = spte;
-	}
-	if (start)
-		direct_pte_prefetch_many(vcpu, sp, start, spte);
-}
-
-static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)
-{
-	struct kvm_mmu_page *sp;
-
-	sp = sptep_to_sp(sptep);
-
-	/*
-	 * Without accessed bits, there's no way to distinguish between
-	 * actually accessed translations and prefetched, so disable pte
-	 * prefetch if accessed bits aren't available.
-	 */
-	if (sp_ad_disabled(sp))
-		return;
-
-	if (sp->role.level > PG_LEVEL_4K)
-		return;
-
-	/*
-	 * If addresses are being invalidated, skip prefetching to avoid
-	 * accidentally prefetching those addresses.
-	 */
-	if (unlikely(vcpu->kvm->mmu_invalidate_in_progress))
-		return;
-
-	__direct_pte_prefetch(vcpu, sp, sptep);
-}
-
 /*
  * Lookup the mapping level for @gfn in the current mm.
  *
@@ -3502,8 +3442,8 @@ static int direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 {
 	struct kvm_shadow_walk_iterator it;
 	struct kvm_mmu_page *sp;
-	int ret, access;
 	gfn_t base_gfn = fault->gfn;
+	int access;
 
 	kvm_mmu_hugepage_adjust(vcpu, fault);
 
@@ -3534,13 +3474,8 @@ static int direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 	if (WARN_ON_ONCE(it.level != fault->goal_level))
 		return -EFAULT;
 
-	ret = mmu_set_spte(vcpu, fault->slot, it.sptep, access,
-			   base_gfn, fault->pfn, fault);
-	if (ret == RET_PF_SPURIOUS)
-		return ret;
-
-	direct_pte_prefetch(vcpu, it.sptep);
-	return ret;
+	return mmu_set_spte(vcpu, fault->slot, it.sptep, access, base_gfn,
+			    fault->pfn, fault);
 }
 
 static void kvm_send_hwpoison_signal(struct kvm_memory_slot *slot, gfn_t gfn)
@@ -6580,11 +6515,38 @@ static int kvm_mmu_write_protect_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 	return RET_PF_EMULATE;
 }
 
+static void kvm_mmu_auto_prefault(struct kvm_vcpu *vcpu, gpa_t start,
+				  u64 error_code, u8 level)
+{
+	gfn_t nr_pages = READ_ONCE(auto_prefault_nr_pages);
+	int nr_pages_msb;
+	gfn_t i;
+
+	if (unlikely(error_code & PFERR_RSVD_MASK))
+		return;
+
+	nr_pages = min(nr_pages, KVM_PAGES_PER_HPAGE(PG_LEVEL_1G));
+	nr_pages_msb = find_last_bit((unsigned long *)&nr_pages, sizeof(nr_pages));
+
+	start = ALIGN_DOWN(start, gfn_to_gpa(BIT_ULL(nr_pages_msb)));
+
+	for (i = KVM_PAGES_PER_HPAGE(level); i < nr_pages; i += KVM_PAGES_PER_HPAGE(level)) {
+		gpa_t gpa = start + gfn_to_gpa(i);
+
+		if (gpa < start || gpa_to_gfn(gpa) > kvm_mmu_max_gfn())
+			return;
+
+		if (kvm_tdp_page_prefault(vcpu, gpa, error_code, &level))
+			return;
+	}
+}
+
 int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_code,
 		       void *insn, int insn_len)
 {
 	int r, emulation_type = EMULTYPE_PF;
 	bool direct = vcpu->arch.mmu->root_role.direct;
+	u8 level;
 
 	if (WARN_ON_ONCE(!VALID_PAGE(vcpu->arch.mmu->root.hpa)))
 		return RET_PF_RETRY;
@@ -6617,7 +6579,7 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err
 		vcpu->stat.pf_taken++;
 
 		r = kvm_mmu_do_page_fault(vcpu, cr2_or_gpa, error_code, false,
-					  &emulation_type, NULL);
+					  &emulation_type, &level);
 		if (KVM_BUG_ON(r == RET_PF_INVALID, vcpu->kvm))
 			return -EIO;
 	}
@@ -6628,6 +6590,8 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err
 	if (r == RET_PF_WRITE_PROTECTED)
 		r = kvm_mmu_write_protect_fault(vcpu, cr2_or_gpa, error_code,
 						&emulation_type);
+	else if (r == RET_PF_FIXED)
+		kvm_mmu_auto_prefault(vcpu, cr2_or_gpa, error_code, level);
 
 	if (r == RET_PF_FIXED)
 		vcpu->stat.pf_fixed++;
diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h
index 27427e7f22fa..b41b4b78cc81 100644
--- a/arch/x86/kvm/mmu/paging_tmpl.h
+++ b/arch/x86/kvm/mmu/paging_tmpl.h
@@ -617,7 +617,7 @@ static void FNAME(pte_prefetch)(struct kvm_vcpu *vcpu, struct guest_walker *gw,
 
 	sp = sptep_to_sp(sptep);
 
-	if (sp->role.level > PG_LEVEL_4K)
+	if (sp->role.level > PG_LEVEL_4K || sp->role.direct)
 		return;
 
 	/*
@@ -627,9 +627,6 @@ static void FNAME(pte_prefetch)(struct kvm_vcpu *vcpu, struct guest_walker *gw,
 	if (unlikely(vcpu->kvm->mmu_invalidate_in_progress))
 		return;
 
-	if (sp->role.direct)
-		return __direct_pte_prefetch(vcpu, sp, sptep);
-
 	i = spte_index(sptep) & ~(PTE_PREFETCH_NUM - 1);
 	spte = sp->spt + i;

Re: [RFC] KVM: x86/mmu: Prefetch forward run of pages on TDP page faults

[Sean Christopherson]

On Tue, Aug 25, 2026, Sean Christopherson wrote:
> On Tue, Aug 25, 2026, Sean Christopherson wrote:
> > Somewhat off the cuff and *very* lightly tested, but this seems to do what I want.
> > If it provides comparable performance, I'll write a changelog (or two?  e.g. to
> > have direct MMUs switch in a separate patch), and let Sashiko and other bots rip
> > apart my idea.
> > 
> > Note!  This has a hard dependency on in-flight prefaulting fixes[*].  Without
> > those, prefaulting will hang the vCPU if the root is invalidated.
> > [*] https://lore.kernel.org/all/20260806214050.78058-1-seanjc@google.com
> > 
> > Note #2!  The below deliberately ignores A/D-disabled MMUs.  I can't think of
> > any reason why it matters whether or not KVM can precisely detect accessed SPTEs,
> > all of the aging stuff is already extremely fuzzy.
> 
> And of course I posted an untested version (I ripped out the direct MMU prefetching
> as an afterthough, and dropped a printk).  This version should actually compile.

This breaks dirty_log_test and dirty_log_page_splitting_test, because KVM creates
writable SPTEs in direct MMUs whenever possible.  Because nothing can be simple,
the below in turn breaks pre_fault_memory_test, but I suspect that's a test flaw.

Note, this would also short-circuit async #PF completion when dirty logging is
enabled.  I think that's a good thing?  If not, we could teach kvm_mmu_do_page_fault()
to differentiate between async #PF and unprompted prefetching.

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 88b6aa1f840f..540d6583995c 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -874,19 +874,11 @@ static void unaccount_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp)
 	untrack_possible_nx_huge_page(kvm, sp, KVM_SHADOW_MMU);
 }
 
-static struct kvm_memory_slot *gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpu,
-							   gfn_t gfn,
-							   bool no_dirty_log)
+static bool kvm_is_memslot_usable_for_prefetch(struct kvm_memory_slot *slot,
+					       unsigned int access)
 {
-	struct kvm_memory_slot *slot;
-
-	slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
-	if (!slot || slot->flags & KVM_MEMSLOT_INVALID)
-		return NULL;
-	if (no_dirty_log && kvm_slot_dirty_track_enabled(slot))
-		return NULL;
-
-	return slot;
+	return slot && !(slot->flags & KVM_MEMSLOT_INVALID) &&
+	       (!(access & ACC_WRITE_MASK) || !kvm_slot_dirty_track_enabled(slot));
 }
 
 /*
@@ -3181,8 +3173,8 @@ static bool kvm_mmu_prefetch_sptes(struct kvm_vcpu *vcpu, gfn_t gfn, u64 *sptep,
 	if (WARN_ON_ONCE(nr_pages > PTE_PREFETCH_NUM))
 		return false;
 
-	slot = gfn_to_memslot_dirty_bitmap(vcpu, gfn, access & ACC_WRITE_MASK);
-	if (!slot)
+	slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
+	if (!kvm_is_memslot_usable_for_prefetch(slot, access))
 		return false;
 
 	nr_pages = kvm_prefetch_pages(slot, gfn, pages, nr_pages);
@@ -4946,6 +4938,8 @@ static int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 		 */
 		fault.gfn = gpa_to_gfn(fault.addr) & ~kvm_gfn_direct_bits(vcpu->kvm);
 		fault.slot = kvm_vcpu_gfn_to_memslot(vcpu, fault.gfn);
+		if (prefetch && !kvm_is_memslot_usable_for_prefetch(fault.slot, ACC_ALL))
+			return RET_PF_WRITE_PROTECTED;
 	}
 
 	/*

Re: [RFC] KVM: x86/mmu: Prefetch forward run of pages on TDP page faults

[Marangoni, Marco]

On Tue, Aug 25, 2026, Sean Christopherson <seanjc@google.com> wrote:
>
> If it provides comparable performance, I'll write a changelog (or two?  e.g. to
> have direct MMUs switch in a separate patch), and let Sashiko and other bots rip
> apart my idea.

Wow, thanks so much for such a quick patch.
I gave it a shot in my test setup. While it's a bit slower than my patch, it's still ~69% faster on nested (rather than ~75%), which is amazing considering how much simpler it is than my proposal.