Re: [PATCH 3/4] KVM: x86/mmu: Add automatic prefaulting/prefetching for TDP MMU faults

[Sean Christopherson <seanjc@google.com>]

On Thu, Oct 29, 2026, Marco Marangoni wrote:
> Hi Sean, thanks for the patch!
> I run some benchmarks on it, and it definitely helps! This is the time to fault 128 MiB linearly after a snapshot restore, with an UFFD page handler that faults chunks on demand; medians of 60 restores. Baseline is kvm-x86/next. Each entry is nr_pages: change vs baseline on the patched kernel; negative means faster.
> 
> - UFFD 4 KiB handler (baseline 2228 ms) 0: +1%, 1: −5%, 2: −3%, 4: +1%, 8: −3%, 16: +1%, 32: 0%, 64: 0%, 128: −2%, 256: +1%, 512: +1%
> - UFFD 512 KiB handler (baseline 543 ms) 0: 0%, 1: 0%, 2: −36%, 4: −51%, 8: −59%, 16: −64%, 32: −71%, 64: −67%, 128: −68%, 256: −67%, 512: −66%
> - UFFD 2 MiB handler (baseline 513 ms) 0: 0%, 1: +1%, 2: −34%, 4: −50%, 8: −59%, 16: −64%, 32: −68%, 64: −68%, 128: −68%, 256: −68%, 512: −68%
> 
> 8 is the series' default; 0 and 1 both disable prefetch and reproduce
> baseline. The 4 KiB row is within ±5%, i.e. noise.  It's measured on nested
> virtualization, so the improvement is probably less dramatic on metal. For
> context, my initial patch was around ~75% faster, so I'd say that
> performance-wise, your implementation is solid!
> 
> It seems like a prefetch of 32 is the sweet spot, but I agree that with James
> that with the normal GUP it's probably better to keep it off or very low by
> default.

Yeah, agreed.  Triggering zswap or any other heavy operations could regress a
lot of configurations/workloads.

Hrm, but I don't want to lose support for guest_memfd without an hva.  Oh.  That's
a pre-existing bug, kvm_prefetch_pages() should reject kvm_memslot_is_gmem_only()
slots.  But that's a side topic.

Argh, FOLL_FAST_ONLY is internal-only.  But do we actually need it to be super
strictly fast-only?  Wouldn't FOLL_NOFAULT suffice?  IRQs are enabled so taking
mmap_sem is fine; it's not be ideal, but KVM will still try fast-GUP first, and
in practice performance is going to suffer no matter what if mmap_sem is
contended, e.g. arm64 *unconditionally* takes mmap_sem for read when handling
faults, and while it shows up in micro-benchmarks, e.g. mmu_stress_test, AFAIK
no one has complained otherwise.

So isn't the fix just this?

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index b673c8130edd..1fcf4fedc0bb 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -4684,6 +4684,9 @@ static int __kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 {
 	unsigned int foll = fault->write ? FOLL_WRITE : 0;
 
+	if (fault->auto_prefetch)
+		foll |= FOLL_NOFAULT;
+
 	if (fault->is_private || kvm_memslot_is_gmem_only(fault->slot))
 		return kvm_mmu_faultin_pfn_gmem(vcpu, fault);

> From my perspective, using fast GUP has the problem that you cannot prefault
> "around" your page, but just ahead of it, so I don't mind going with normal
> GUP if it makes things easier for you.

I think I'm missing something, why would fast GUP prevent prefaulting "behind"?
E.g. is that effectively a limitation of the underlying backing store you're
using, a fundamental limitation somewhere in mm/, or a limitation in KVM itself?

> I'd say that most setup would know the size at which UFFD copies the memory,
> so having to explicitly configure is okay.
> 
> For completeness, I also measured how much the patch improves at various UFFD chunks, with nr_pages at the same size:
>   - 4 KiB: 2285 ms → 2321 ms (np1), +2% (np1 is "off", noise)
>   - 8 KiB: 1435 ms → 1441 ms (np2), 0%
>   - 16 KiB: 1036 ms → 860 ms (np4), −17%
>   - 32 KiB: 743 ms → 469 ms (np8), −37%
>   - 64 KiB: 645 ms → 319 ms (np16), −50%
>   - 128 KiB: 587 ms → 242 ms (np32), −59%
>   - 256 KiB: 561 ms → 200 ms (np64), −64%
>   - 512 KiB: 543 ms → 165 ms (np128), −70%
>   - 1 MiB: 534 ms → 171 ms (np256), −68%
>   - 2 MiB: 523 ms → 151 ms (np512), −71%
> 
> These measurement highlight a possible limitation (why didn't 8KiB improve?):
> it seems like the prefetching doesn't kick in the first time a page is
> getting faulted in the chunk, is that expected? I asked an LLM to
> investigate, and it seems to be because with async PF, the first fault
> doesn't reach the kvm_mmu_auto_prefault code path.

Hmm, right.  I deliberately hooked kvm_mmu_page_fault() instead of
kvm_mmu_do_page_fault() so that prefetching wouldn't trigger auto-prefetch, but
the existing shadow MMU prefetching does kick in for async #PFs, and it does make
sense to prefetch in that case if we buy my argument (from this changelog) that
async #PF should punch through dirty logging.

Auto-prefetching for KVM_PRE_FAULT_MEMORY feels "wrong", but we do prefetch in
that case for the shadow MMU, though I'm like 99.9% certain no one is running
TDP with the shadow MMU these days, so I don't think we should consider that a
valid datapoint.

Untested, but I think it'd be this:

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index b673c8130edd..c0ff50c52ad5 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -5024,6 +5024,9 @@ enum kvm_prefetch_type {
 	KVM_PREFETCH_AUTO,
 };
 
+static void kvm_mmu_auto_prefault(struct kvm_vcpu *vcpu, gpa_t start,
+				  u64 error_code, u8 level);
+
 static int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 				 u64 err, enum kvm_prefetch_type prefetch,
 				 int *emulation_type, u8 *level)
@@ -5098,6 +5101,9 @@ static int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 	if (level)
 		*level = fault.goal_level;
 
+	if (r == RET_PF_FIXED && prefetch != KVM_PREFETCH_AUTO)
+		kvm_mmu_auto_prefault(vcpu, cr2_or_gpa, err, fault.goal_level);
+
 	return r;
 }

or to disable it for KVM_PRE_FAULT_MEMORY:

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index b673c8130edd..ee31a35f6a55 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -5021,9 +5021,13 @@ static int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *faul
 enum kvm_prefetch_type {
 	KVM_PREFETCH_NONE,
 	KVM_PREFETCH_MANUAL,
+	KVM_PREFETCH_ASYNC,
 	KVM_PREFETCH_AUTO,
 };
 
+static void kvm_mmu_auto_prefault(struct kvm_vcpu *vcpu, gpa_t start,
+				  u64 error_code, u8 level);
+
 static int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 				 u64 err, enum kvm_prefetch_type prefetch,
 				 int *emulation_type, u8 *level)
@@ -5098,6 +5102,10 @@ static int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 	if (level)
 		*level = fault.goal_level;
 
+	if (r == RET_PF_FIXED && prefetch != KVM_PREFETCH_AUTO &&
+	    prefetch != KVM_PREFETCH_MANUAL)
+		kvm_mmu_auto_prefault(vcpu, cr2_or_gpa, err, fault.goal_level);
+
 	return r;
 }
 
@@ -5215,7 +5223,7 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work)
 		return;
 
 	r = kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, work->arch.error_code,
-				  KVM_PREFETCH_MANUAL, NULL, NULL);
+				  KVM_PREFETCH_ASYNC, NULL, NULL);
 
 	/*
 	 * Account fixed page faults, otherwise they'll never be counted, but