[PATCH v2] KVM: Ignore MMU notifiers for guest_memfd-only memslots

[PATCH v2] KVM: Ignore MMU notifiers for guest_memfd-only memslots

[Alexandru Elisei]

For guest_memfd-only memslots (kvm_memslot_is_gmem_only() is true), the
memory provider for the virtual machine is the guest_memfd file, not the
userspace mapping. Mappings in the secondary MMU are established by
obtaining folios from guest_memfd directly, not by looking the folios up
through the page tables through GUP. Consequently, there is no relationship
between the page tables and the secondary MMU: MMU notifiers do not apply.

Despite this, KVM's MMU notifiers still modify the secondary MMU page
tables, only for the same memory to be remapped the next time a guest
accesses it. Make the disconnect between the user mapping and the secondary
MMU page tables explicit by ignoring the MMU notifiers for guest_memfd-only
memslots.

Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
v1 can be found here [1].

Same testing as in v1: on arm64, mapped guest_memfd in kvmtool's address
space, used KVM_PRE_FAULT_MEMORY to map the entire stage 2, then unmapped
guest_memfd from kvmtool's address space. The memory remained mapped at
stage 2.

Changes in v2:

* Rebased on top of v7.2-rc3.

Changes in v1:

* Dropped the RFC tag.
* Fix unbalanced invalidation reported by sashiko by implementing Sean's
  approach; I've expanded it to page ageing.
* Modified the commit message as per DavidH comment.

[1] https://lore.kernel.org/kvm/20260625130902.258331-1-alexandru.elisei@arm.com/

 include/linux/kvm_host.h |  1 +
 virt/kvm/kvm_main.c      | 50 ++++++++++++++++++++++++++++++++++++----
 2 files changed, 47 insertions(+), 4 deletions(-)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index ab8cfaec82d3..ecf5c0cbf0f0 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -260,6 +260,7 @@ union kvm_mmu_notifier_arg {
 enum kvm_gfn_range_filter {
 	KVM_FILTER_SHARED		= BIT(0),
 	KVM_FILTER_PRIVATE		= BIT(1),
+	KVM_FILTER_USERSPACE_MAPPINGS	= BIT(2),
 };
 
 struct kvm_gfn_range {
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index e44c20c04961..bd469395f176 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -607,8 +607,13 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
 			/*
 			 * HVA-based notifications aren't relevant to private
 			 * mappings as they don't have a userspace mapping.
+			 *
+			 * Memslots where guest_memfd is the only memory
+			 * provider can also safely ignore changes to the
+			 * userspace mapping.
 			 */
-			gfn_range.attr_filter = KVM_FILTER_SHARED;
+			gfn_range.attr_filter = KVM_FILTER_SHARED |
+						KVM_FILTER_USERSPACE_MAPPINGS;
 
 			/*
 			 * {gfn(page) | page intersects with [hva_start, hva_end)} =
@@ -715,6 +720,21 @@ void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
 bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	kvm_mmu_invalidate_range_add(kvm, range->start, range->end);
+
+	/*
+	 * When reacting to changes in userspace mappings, don't unmap memslots
+	 * that are guest_memfd-only, in which case KVM's MMU mappings are
+	 * pulled directly from guest_memfd, i.e. don't depend on the userspace
+	 * mappings.
+	 *
+	 * TODO: Skip gmem-only memslots on mmu_notifier events entirely, once
+	 * gfn_to_pfn_cache is also wired up to directly pull from guest_memfd.
+	 */
+	if (range->attr_filter & KVM_FILTER_USERSPACE_MAPPINGS &&
+	    kvm_slot_has_gmem(range->slot) &&
+	    kvm_memslot_is_gmem_only(range->slot))
+		return false;
+
 	return kvm_unmap_gfn_range(kvm, range);
 }
 
@@ -825,12 +845,23 @@ static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn,
 		rcuwait_wake_up(&kvm->mn_memslots_update_rcuwait);
 }
 
+static bool kvm_mmu_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+{
+	/* See comment in kvm_mmu_unmap_gfn_range() */
+	if (range->attr_filter & KVM_FILTER_USERSPACE_MAPPINGS &&
+	    kvm_slot_has_gmem(range->slot) &&
+	    kvm_memslot_is_gmem_only(range->slot))
+		return false;
+
+	return kvm_age_gfn(kvm, range);
+}
+
 static bool kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn,
 		struct mm_struct *mm, unsigned long start, unsigned long end)
 {
 	trace_kvm_age_hva(start, end);
 
-	return kvm_age_hva_range(mn, start, end, kvm_age_gfn,
+	return kvm_age_hva_range(mn, start, end, kvm_mmu_age_gfn,
 				 !IS_ENABLED(CONFIG_KVM_ELIDE_TLB_FLUSH_IF_YOUNG));
 }
 
@@ -852,7 +883,18 @@ static bool kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,
 	 * cadence. If we find this inaccurate, we might come up with a
 	 * more sophisticated heuristic later.
 	 */
-	return kvm_age_hva_range_no_flush(mn, start, end, kvm_age_gfn);
+	return kvm_age_hva_range_no_flush(mn, start, end, kvm_mmu_age_gfn);
+}
+
+static bool kvm_mmu_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+{
+	/* See comment in kvm_mmu_unmap_gfn_range() */
+	if (range->attr_filter & KVM_FILTER_USERSPACE_MAPPINGS &&
+	    kvm_slot_has_gmem(range->slot) &&
+	    kvm_memslot_is_gmem_only(range->slot))
+		return false;
+
+	return kvm_test_age_gfn(kvm, range);
 }
 
 static bool kvm_mmu_notifier_test_young(struct mmu_notifier *mn,
@@ -861,7 +903,7 @@ static bool kvm_mmu_notifier_test_young(struct mmu_notifier *mn,
 	trace_kvm_test_age_hva(address);
 
 	return kvm_age_hva_range_no_flush(mn, address, address + 1,
-					  kvm_test_age_gfn);
+					  kvm_mmu_test_age_gfn);
 }
 
 static void kvm_mmu_notifier_release(struct mmu_notifier *mn,

base-commit: a13c140cc289c0b7b3770bce5b3ad42ab35074aa
-- 
2.55.0

Re: [PATCH v2] KVM: Ignore MMU notifiers for guest_memfd-only memslots

[Sean Christopherson]

On Tue, Jul 14, 2026, Alexandru Elisei wrote:
> +	 * TODO: Skip gmem-only memslots on mmu_notifier events entirely, once
> +	 * gfn_to_pfn_cache is also wired up to directly pull from guest_memfd.

So it turns out we need to solve a very related problem[*], thanks to commit
b9220d32799a ("KVM: x86/xen: allow shared_info to be mapped by fixed HVA"),
which as it says, allows populating gfn_to_pfn_cache directly with an HVA, i.e.
without a memslot.

My not-yet-tested solution for that mess is to give gfn_to_pfn_cache its own
invalidation sequence, at which point there shouldn't be a need to process
gmem-only memslots just for gfn_to_pfn_cache, because it will already be equipped
to handle memslot-less invalidation.

Given how simple this change *should* be, I'm going to put this patch on the
backburner for now, and then assuming my proposed change actually works, I'll
grab your original change (or more likely, re-post it as the last patch in the
series weans gfn_to_pfn_cache off mmu_invalidate_seq).

It'll probably take me a few weeks to circle back to this, as I finally crawled
out from underneath a pile of urgent and am just now getting back into review
mode, but this is fairly high on my todo list, so with luck we might still squeeze
this into 7.3.

[*] https://lore.kernel.org/all/aj2hi-NcqUUCaoQF@google.com

Re: [PATCH v2] KVM: Ignore MMU notifiers for guest_memfd-only memslots

[Alexandru Elisei]

Hi Sean,

(Sorry for the silence, was on holiday)

On Fri, Jul 24, 2026 at 12:52:20PM -0700, Sean Christopherson wrote:
> On Tue, Jul 14, 2026, Alexandru Elisei wrote:
> > +	 * TODO: Skip gmem-only memslots on mmu_notifier events entirely, once
> > +	 * gfn_to_pfn_cache is also wired up to directly pull from guest_memfd.
> 
> So it turns out we need to solve a very related problem[*], thanks to commit
> b9220d32799a ("KVM: x86/xen: allow shared_info to be mapped by fixed HVA"),
> which as it says, allows populating gfn_to_pfn_cache directly with an HVA, i.e.
> without a memslot.
> 
> My not-yet-tested solution for that mess is to give gfn_to_pfn_cache its own
> invalidation sequence, at which point there shouldn't be a need to process
> gmem-only memslots just for gfn_to_pfn_cache, because it will already be equipped
> to handle memslot-less invalidation.
> 
> Given how simple this change *should* be, I'm going to put this patch on the
> backburner for now, and then assuming my proposed change actually works, I'll
> grab your original change (or more likely, re-post it as the last patch in the
> series weans gfn_to_pfn_cache off mmu_invalidate_seq).

By "original change" I assume you mean the RFC patch [1], where I skip the
memslot entirely in handle_hva_range(), which would make sense since the
problem you are solving involves cache entries without a memslot.

[1] https://lore.kernel.org/kvm/20260615155244.183044-1-alexandru.elisei@arm.com/

> 
> It'll probably take me a few weeks to circle back to this, as I finally crawled
> out from underneath a pile of urgent and am just now getting back into review
> mode, but this is fairly high on my todo list, so with luck we might still squeeze
> this into 7.3.

Sounds great to me, thank you for your time and patience!

Alex

> 
> [*] https://lore.kernel.org/all/aj2hi-NcqUUCaoQF@google.com

Re: [PATCH v2] KVM: Ignore MMU notifiers for guest_memfd-only memslots

[Lorenzo Stoakes (ARM)]

[updated David's email address to @kernel.org]

On Wed, Aug 12, 2026 at 10:42:50AM +0100, Alexandru Elisei wrote:
[ ... ]
> On Fri, Jul 24, 2026 at 12:52:20PM -0700, Sean Christopherson wrote:
> > It'll probably take me a few weeks to circle back to this, as I finally crawled
> > out from underneath a pile of urgent and am just now getting back into review
> > mode, but this is fairly high on my todo list, so with luck we might still squeeze
> > this into 7.3.

Hey Sean,

I wondered whether you'd had a chance to look at this?

After discussing this with Marc and Alex it is clear that this is really a
blocker for Alex's SPE series, as we need a sensible means of ensuring
that stage-2 mappings stay in place.

So ideally would prefer to get something like this in in the next cycle,
if you don't have any objections to it? :)

If it'd be easier, I think a smaller version of the same patch would also
give us what we need? I.e. your snippet from June with the page ageing part
dropped, as below.

Thanks!

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index e7fc94e2910a..a6f37bdb0902 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -264,6 +264,7 @@ union kvm_mmu_notifier_arg {
 enum kvm_gfn_range_filter {
 	KVM_FILTER_SHARED		= BIT(0),
 	KVM_FILTER_PRIVATE		= BIT(1),
+	KVM_FILTER_USERSPACE_MAPPINGS	= BIT(2),
 };

 struct kvm_gfn_range {
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index cf4cb586f140..6fc3b443c305 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -616,8 +616,13 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
 			/*
 			 * HVA-based notifications aren't relevant to private
 			 * mappings as they don't have a userspace mapping.
+			 *
+			 * Memslots where guest_memfd is the only memory
+			 * provider can also safely ignore changes to the
+			 * userspace mapping.
 			 */
-			gfn_range.attr_filter = KVM_FILTER_SHARED;
+			gfn_range.attr_filter = KVM_FILTER_SHARED |
+						KVM_FILTER_USERSPACE_MAPPINGS;

 			/*
 			 * {gfn(page) | page intersects with [hva_start, hva_end)} =
@@ -724,6 +729,17 @@ void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
 bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	kvm_mmu_invalidate_range_add(kvm, range->start, range->end);
+
+	/*
+	 * When reacting to changes in userspace mappings, don't unmap memslots
+	 * that are guest_memfd-only, in which case KVM's MMU mappings are
+	 * pulled directly from guest_memfd, i.e. don't depend on the userspace
+	 * mappings.
+	 */
+	if (range->attr_filter & KVM_FILTER_USERSPACE_MAPPINGS &&
+	    kvm_memslot_is_gmem_only(range->slot))
+		return false;
+
 	return kvm_unmap_gfn_range(kvm, range);
 }

--
Cheers, Lorenzo

Re: [PATCH v2] KVM: Ignore MMU notifiers for guest_memfd-only memslots

[Sean Christopherson]

On Wed, Nov 04, 2026, Lorenzo Stoakes (ARM) wrote:
> [updated David's email address to @kernel.org]
> 
> On Wed, Aug 12, 2026 at 10:42:50AM +0100, Alexandru Elisei wrote:
> [ ... ]
> > On Fri, Jul 24, 2026 at 12:52:20PM -0700, Sean Christopherson wrote:
> > > It'll probably take me a few weeks to circle back to this, as I finally crawled
> > > out from underneath a pile of urgent and am just now getting back into review
> > > mode, but this is fairly high on my todo list, so with luck we might still squeeze
> > > this into 7.3.
> 
> Hey Sean,
> 
> I wondered whether you'd had a chance to look at this?
> 
> After discussing this with Marc and Alex it is clear that this is really a
> blocker for Alex's SPE series, as we need a sensible means of ensuring
> that stage-2 mappings stay in place.
> 
> So ideally would prefer to get something like this in in the next cycle,
> if you don't have any objections to it? :)
> 
> If it'd be easier, I think a smaller version of the same patch would also
> give us what we need? I.e. your snippet from June with the page ageing part
> dropped, as below.

The GPC invalidation fix landed, so we can go straight to the super simple v1.
Compile tested only at this point, I'll formally post this (hopefully today),
assuming nothing explodes in testing.

--
From: Alexandru Elisei <alexandru.elisei@arm.com>
Date: Mon, 15 Jun 2026 16:52:44 +0100
Subject: [PATCH] KVM: Skip gfn-based MMU notifier invalidations for
 guest_memfd-only memslots

Don't invalidate gmem-only memslots when responding to MMU notifier
invalidations, as the entire point of gmem-only memslots is so that KVM
doesn't need a VMA to install mappings into KVM's MMU.  Specificially,
page faults are resolved by going straight to guest_memfd, using the gfn
and the associated memslot's gmem.file.  I.e. whether or not userspace
happens to have valid mappings to the same gmem instance is irrelevant,
as are any invalidations for those mappings.

Note!  The exception is KVM's gfn_to_pfn_cache implementation which isn't
yet wired up to pull pfns directly from guest_memfd, i.e. still relies on
userspace mappings/VMAs and thus need to be invalidated, e.g. to ensure KVM
doesn't retain a kernel mapping+pointer to PRIVATE memory.  However, as of
commit da260bcb56fe ("KVM: pfncache: use a dedicated invalidation sequence
for cache refresh"), GPC invalidation no longer relies on memslot lookups,
because GPCs also support HVA-based mappings.  I.e. KVM *must* be able to
detect and handle relevant MMU notifier invalidations even if there is *no*
memslot whatsoever.

In addition to the obvious performance benefits, avoiding unnecessary and
spurious SPTE zaps and TLB flushes will allow KVM to support features like
ARM's Statistical Profiling Extension (SPE)[1] and x86's Secure AVIC[2],
both of which rely on the pages *never* being unmapped from the stage-2
page tables.

Link: https://lore.kernel.org/all/20260903160623.315525-1-alexandru.elisei@arm.com [1]
Link: https://lore.kernel.org/all/cover.1783490022.git.naveen@kernel.org [2]
Cc: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: Naveen N Rao (AMD) <naveen@kernel.org>
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
[sean: rewrite changelog to capture the new gpc behavior]
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 virt/kvm/kvm_main.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index cf4cb586f140..599dea22afb8 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -601,6 +601,10 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
 			unsigned long hva_start, hva_end;
 
 			slot = container_of(node, struct kvm_memory_slot, hva_node[slots->node_idx]);
+
+			if (kvm_slot_has_gmem(slot) && kvm_memslot_is_gmem_only(slot))
+				continue;
+
 			hva_start = max_t(unsigned long, range->start, slot->userspace_addr);
 			hva_end = min_t(unsigned long, range->end,
 					slot->userspace_addr + (slot->npages << PAGE_SHIFT));

base-commit: d21eef2084a9af87ac2a783bc7c2b175ba114e4e
--

Re: [PATCH v2] KVM: Ignore MMU notifiers for guest_memfd-only memslots

[Lorenzo Stoakes (ARM)]

On Thu, Nov 05, 2026 at 11:02:28AM -0800, Sean Christopherson wrote:
> On Wed, Nov 04, 2026, Lorenzo Stoakes (ARM) wrote:
> > [updated David's email address to @kernel.org]
> >
> > On Wed, Aug 12, 2026 at 10:42:50AM +0100, Alexandru Elisei wrote:
> > [ ... ]
> > > On Fri, Jul 24, 2026 at 12:52:20PM -0700, Sean Christopherson wrote:
> > > > It'll probably take me a few weeks to circle back to this, as I finally crawled
> > > > out from underneath a pile of urgent and am just now getting back into review
> > > > mode, but this is fairly high on my todo list, so with luck we might still squeeze
> > > > this into 7.3.
> >
> > Hey Sean,
> >
> > I wondered whether you'd had a chance to look at this?
> >
> > After discussing this with Marc and Alex it is clear that this is really a
> > blocker for Alex's SPE series, as we need a sensible means of ensuring
> > that stage-2 mappings stay in place.
> >
> > So ideally would prefer to get something like this in in the next cycle,
> > if you don't have any objections to it? :)
> >
> > If it'd be easier, I think a smaller version of the same patch would also
> > give us what we need? I.e. your snippet from June with the page ageing part
> > dropped, as below.
>
> The GPC invalidation fix landed, so we can go straight to the super simple v1.
> Compile tested only at this point, I'll formally post this (hopefully today),
> assuming nothing explodes in testing.

Perfect, thanks very much :)

>
> --
> From: Alexandru Elisei <alexandru.elisei@arm.com>
> Date: Mon, 15 Jun 2026 16:52:44 +0100
> Subject: [PATCH] KVM: Skip gfn-based MMU notifier invalidations for
>  guest_memfd-only memslots
>
> Don't invalidate gmem-only memslots when responding to MMU notifier
> invalidations, as the entire point of gmem-only memslots is so that KVM
> doesn't need a VMA to install mappings into KVM's MMU.  Specificially,
> page faults are resolved by going straight to guest_memfd, using the gfn
> and the associated memslot's gmem.file.  I.e. whether or not userspace
> happens to have valid mappings to the same gmem instance is irrelevant,
> as are any invalidations for those mappings.
>
> Note!  The exception is KVM's gfn_to_pfn_cache implementation which isn't
> yet wired up to pull pfns directly from guest_memfd, i.e. still relies on
> userspace mappings/VMAs and thus need to be invalidated, e.g. to ensure KVM
> doesn't retain a kernel mapping+pointer to PRIVATE memory.  However, as of
> commit da260bcb56fe ("KVM: pfncache: use a dedicated invalidation sequence
> for cache refresh"), GPC invalidation no longer relies on memslot lookups,
> because GPCs also support HVA-based mappings.  I.e. KVM *must* be able to
> detect and handle relevant MMU notifier invalidations even if there is *no*
> memslot whatsoever.
>
> In addition to the obvious performance benefits, avoiding unnecessary and
> spurious SPTE zaps and TLB flushes will allow KVM to support features like
> ARM's Statistical Profiling Extension (SPE)[1] and x86's Secure AVIC[2],
> both of which rely on the pages *never* being unmapped from the stage-2
> page tables.
>
> Link: https://lore.kernel.org/all/20260903160623.315525-1-alexandru.elisei@arm.com [1]
> Link: https://lore.kernel.org/all/cover.1783490022.git.naveen@kernel.org [2]
> Cc: Lorenzo Stoakes (ARM) <ljs@kernel.org>
> Cc: Naveen N Rao (AMD) <naveen@kernel.org>
> Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
> [sean: rewrite changelog to capture the new gpc behavior]
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
>  virt/kvm/kvm_main.c | 4 ++++
>  1 file changed, 4 insertions(+)
>
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index cf4cb586f140..599dea22afb8 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -601,6 +601,10 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
>  			unsigned long hva_start, hva_end;
>
>  			slot = container_of(node, struct kvm_memory_slot, hva_node[slots->node_idx]);
> +
> +			if (kvm_slot_has_gmem(slot) && kvm_memslot_is_gmem_only(slot))
> +				continue;
> +
>  			hva_start = max_t(unsigned long, range->start, slot->userspace_addr);
>  			hva_end = min_t(unsigned long, range->end,
>  					slot->userspace_addr + (slot->npages << PAGE_SHIFT));
>
> base-commit: d21eef2084a9af87ac2a783bc7c2b175ba114e4e
> --

--
Cheers, Lorenzo