[RFC PATCH 0/4] KVM: x86/mmu: don't zap secondary MMU on permission upgrade

[RFC PATCH 0/4] KVM: x86/mmu: don't zap secondary MMU on permission upgrade

[Carlos López]

Prevent x86 KVM from over-zapping its secondary MMU on PTE permission
upgrades, which leave the PFNs unchanged and do not revoke access.
Permissions for SPTEs that KVM holds are updated on the next fault, so
dropping them early gains nothing.

Do this by adding a new MMU notifier flag to signal that a
MMU_NOTIFY_PROTECTION_VMA strictly upgrades (loosens) permissions on a
particular range.

See below for benchmark results.

Background
==========

change_protection() pairs every protection change with an
MMU_NOTIFY_PROTECTION_VMA invalidation, and KVM zaps its SPTEs in
response, as it cannot tell whether permissions widened or narrowed.
This is costly e.g. during a QEMU background snapshot, which relies on
uffd-wp to write-protect guest RAM and then restore permissions as each
page is saved. Without this series, the permission restore causes the
guest to re-fault read-only memory needlessly.

Note, the reverse situation is already handled for secondary MMUs
that reuse the CPU page tables (e.g. ARM SMMU, IOMMU/ATS). They cannot
refault, so they are notified of a permission upgrade on the CPU
TLB-flush path [1].

KVM, however, owns separate page tables and faults permissions on demand,
so it needs no such notification (as Sean noted when that signalling
was first proposed on the fault path [2]). This series applies the same
principle to the change_protection() path, signalling the permission
upgrade so KVM can avoid the unnecessary zap.

RFC Considerations
==================

1. Having the zap skip in KVM's arch-independent code does not seem
   obviously correct, as s390 caches additional translation state
   that may not be rebuilt by refaults. Maybe an expert can correct
   me. The skip could be extended to arm64 and RISC-V, but I have not
   benchmarked the benefits and I'd like to get feedback first. Hence,
   x86-only for now.

2. change_protection() could simply not emit the invalidation at all
   for a pure permission upgrade (shared-page-table MMUs invalidate
   on the CPU TLB-flush path, as noted above, so they do not need it).
   But again, the s390 case could be broken, and even then, that would
   seem too much intrusion for KVM's benefit.

3. The flag reaches KVM as a bool on struct kvm_gfn_range.
   Alternatively, KVM could forward the notifier event, as Claudio
   Imbrenda suggested at LPC [3] and others have proposed in the past
   for different purposes [4] [5]. That, along with the event flags,
   could serve the same purpose. I'd be happy to take that approach if
   there is agreement that other consumers would want it.

   (The LPC talk recording has some audio issues so I am not too sure
   on what the final agreement was)

Benchmarks
==========

AMD EPYC 9654 96-Core Processor host running a recent Linux kernel
(6b4b543fa187), with a 64GB, 64-vCPU guest running stress-ng --vm while
the host takes a QEMU background snapshot. All results are aggregated
over 5 runs, and compared between the stock kernel and one with this
patch series. Guest is rebooted across each individual run, and NUMA
balancing is disabled.

pf/s and tlb/s (per-second deltas of KVM's pf_taken and remote_tlb_flush
debugfs stats) clearly improve:

  guest pattern          baseline                    patched
  -----------------------------------------------------------------------
  read64   (read)        pf/s=950,015   tlb/s=1,440  pf/s=2,503 tlb/s=0.1
  rand-set (read+write)  pf/s=1,004,353 tlb/s=862    pf/s=4,073 tlb/s=0.3
  write64  (write)       pf/s=968,018   tlb/s=94     pf/s=2,292 tlb/s=0.3

Guest read latency was sampled with a paced probe thread in the
guest, reading one byte per page with an idle gap between reads, while
stress-ng ran with read64. Each figure is the median across 5 runs, with
the run-to-run min and max, in ns:

  metric           baseline             patched
  ------------------------------------------------------
  p50 (min max)      340     (270 380)   260   (250 260)
  p99 (min max)    19789 (16885 20200)  5207 (4076 5638)
  p99.9 (min max)  23445 (22845 24036)  6850 (6259 7110)

The median improves modestly and the refault cost shows in the tail,
where p99 drops ~3.8x.

A separate stress-ng run with write64 and --verify showed no data
corruption and no livelocks.

References
==========

[1] https://lore.kernel.org/linux-mm/cover.1eca029b8603ef4eebe5b41eae51facfc5920c41.1690292440.git-series.apopple@nvidia.com/
[2] https://lore.kernel.org/linux-mm/ZGu1vsbscdO48V6h@google.com/
[3] https://lpc.events/event/20/contributions/2552/
[4] https://lore.kernel.org/all/20240801183453.57199-1-seanjc@google.com
[5] https://lore.kernel.org/all/20231110003734.1014084-1-jackyli@google.com

Carlos López (4):
  mm/mprotect: add MM_CP_PERM_UPGRADE
  mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE
  KVM: plumb MMU_NOTIFIER_RANGE_PERM_UPGRADE into kvm_gfn_range
  KVM: x86/mmu: don't zap the secondary MMU on permission upgrade

 arch/x86/kvm/mmu/mmu.c       |  7 +++++++
 include/linux/kvm_host.h     |  1 +
 include/linux/mm.h           |  8 ++++++++
 include/linux/mmu_notifier.h |  5 +++++
 mm/hugetlb.c                 |  6 +++++-
 mm/mprotect.c                | 20 +++++++++++++++++++-
 virt/kvm/kvm_main.c          |  4 ++++
 7 files changed, 49 insertions(+), 2 deletions(-)

-- 
2.51.0

[RFC PATCH 1/4] mm/mprotect: add MM_CP_PERM_UPGRADE

[Carlos López]

Add a change_protection() flag for changes that only add permissions:
the PFN stays the same and no access is revoked. mprotect() sets it when
the new VMA permissions are a superset of the old, and the protection
key is unchanged.

This will allow MMU notifier users to perform optimizations, as they
can avoid invalidating PTEs whose permissions can be faulted in.

Signed-off-by: Carlos López <clopez@suse.de>
---
 include/linux/mm.h |  2 ++
 mm/mprotect.c      | 13 +++++++++++++
 2 files changed, 15 insertions(+)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 691cef038d1e..f406ce9d1032 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3415,6 +3415,8 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen);
 #define  MM_CP_UFFD_RWP_RESOLVE            (1UL << 5) /* resolve rwp */
 #define  MM_CP_UFFD_RWP_ALL                (MM_CP_UFFD_RWP | \
 					    MM_CP_UFFD_RWP_RESOLVE)
+/* Whether this change only upgrades/adds permissions */
+#define  MM_CP_PERM_UPGRADE                (1UL << 6)
 
 bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,
 			     pte_t pte);
diff --git a/mm/mprotect.c b/mm/mprotect.c
index 2888ee638d87..e9b636809c42 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -756,6 +756,17 @@ static const struct mm_walk_ops prot_none_walk_ops = {
 	.walk_lock		= PGWALK_WRLOCK,
 };
 
+static bool vma_flags_is_perm_upgrade(vma_flags_t old_vma_flags,
+				       vma_flags_t new_vma_flags)
+{
+	vm_flags_t old = vma_flags_to_legacy(old_vma_flags);
+	vm_flags_t new = vma_flags_to_legacy(new_vma_flags);
+	vm_flags_t rwx = VM_READ | VM_WRITE | VM_EXEC;
+
+	return !(old & rwx & ~new) &&
+	       (old & ARCH_VM_PKEY_FLAGS) == (new & ARCH_VM_PKEY_FLAGS);
+}
+
 int
 mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,
 	       struct vm_area_struct *vma, struct vm_area_struct **pprev,
@@ -837,6 +848,8 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,
 	vma_flags_reset_once(vma, &new_vma_flags);
 	if (vma_wants_manual_pte_write_upgrade(vma))
 		mm_cp_flags |= MM_CP_TRY_CHANGE_WRITABLE;
+	if (vma_flags_is_perm_upgrade(old_vma_flags, new_vma_flags))
+		mm_cp_flags |= MM_CP_PERM_UPGRADE;
 	vma_set_page_prot(vma);
 
 	change_protection(tlb, vma, start, end, mm_cp_flags);
-- 
2.51.0

[RFC PATCH 2/4] mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE

[Carlos López]

Add a range flag marking an invalidation that only upgrades permissions
(that is, no PFN change and no revoked access). change_protection() sets
it for MM_CP_PERM_UPGRADE and for the userfaultfd permission resolves,
which unprotect a page and thus only restore access.

A secondary MMU can use the flag to keep its mappings rather than zap
them. Consumers that ignore it are unaffected.

Signed-off-by: Carlos López <clopez@suse.de>
---
 include/linux/mm.h           | 6 ++++++
 include/linux/mmu_notifier.h | 5 +++++
 mm/hugetlb.c                 | 6 +++++-
 mm/mprotect.c                | 7 ++++++-
 4 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index f406ce9d1032..c4fc3876a4b7 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3418,6 +3418,12 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen);
 /* Whether this change only upgrades/adds permissions */
 #define  MM_CP_PERM_UPGRADE                (1UL << 6)
 
+static inline bool mm_cp_is_perm_upgrade(unsigned long cp_flags)
+{
+	return cp_flags & (MM_CP_PERM_UPGRADE | MM_CP_UFFD_WP_RESOLVE |
+			   MM_CP_UFFD_RWP_RESOLVE);
+}
+
 bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,
 			     pte_t pte);
 extern long change_protection(struct mmu_gather *tlb,
diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h
index a11a44eef521..112be28cdf6b 100644
--- a/include/linux/mmu_notifier.h
+++ b/include/linux/mmu_notifier.h
@@ -60,6 +60,11 @@ enum mmu_notifier_event {
 };
 
 #define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0)
+/*
+ * Set on an invalidation that only expands permissions without
+ * revoking any existing access, nor freeing/moving the underlying page.
+ */
+#define MMU_NOTIFIER_RANGE_PERM_UPGRADE (1 << 1)
 
 struct mmu_notifier_ops {
 	/*
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index b017ae1dfd6f..68d44044fdd2 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -6548,19 +6548,23 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
 	long pages = 0, psize = huge_page_size(h);
 	struct mmu_notifier_range range;
 	unsigned long last_addr_mask;
+	unsigned int mn_flags = 0;
 	bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
 	bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
 	bool uffd_rwp = cp_flags & MM_CP_UFFD_RWP;
 	bool uffd_rwp_resolve = cp_flags & MM_CP_UFFD_RWP_RESOLVE;
 	struct mmu_gather tlb;
 
+	if (mm_cp_is_perm_upgrade(cp_flags))
+		mn_flags |= MMU_NOTIFIER_RANGE_PERM_UPGRADE;
+
 	/*
 	 * In the case of shared PMDs, the area to flush could be beyond
 	 * start/end.  Set range.start/range.end to cover the maximum possible
 	 * range if PMD sharing is possible.
 	 */
 	mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA,
-				0, mm, start, end);
+				mn_flags, mm, start, end);
 	adjust_range_if_pmd_sharing_possible(vma, &range.start, &range.end);
 
 	BUG_ON(address >= end);
diff --git a/mm/mprotect.c b/mm/mprotect.c
index e9b636809c42..b48d2fd3b7b5 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -597,8 +597,13 @@ static inline long change_pud_range(struct mmu_gather *tlb,
 			continue;
 
 		if (!range.start) {
+			unsigned int mn_flags = 0;
+
+			if (mm_cp_is_perm_upgrade(cp_flags))
+				mn_flags |= MMU_NOTIFIER_RANGE_PERM_UPGRADE;
+
 			mmu_notifier_range_init(&range,
-						MMU_NOTIFY_PROTECTION_VMA, 0,
+						MMU_NOTIFY_PROTECTION_VMA, mn_flags,
 						vma->vm_mm, addr, end);
 			mmu_notifier_invalidate_range_start(&range);
 		}
-- 
2.51.0

[RFC PATCH 3/4] KVM: plumb MMU_NOTIFIER_RANGE_PERM_UPGRADE into kvm_gfn_range

[Carlos López]

Propagate the permission upgrade flag from the mmu_notifier range into
struct kvm_gfn_range, so the unmap handler can react accordingly. The
value is not consumed yet, so no functional change.

Signed-off-by: Carlos López <clopez@suse.de>
---
 include/linux/kvm_host.h | 1 +
 virt/kvm/kvm_main.c      | 4 ++++
 2 files changed, 5 insertions(+)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index e7fc94e2910a..aa109818cede 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -274,6 +274,7 @@ struct kvm_gfn_range {
 	enum kvm_gfn_range_filter attr_filter;
 	bool may_block;
 	bool lockless;
+	bool perm_upgrade;
 };
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index cf4cb586f140..0d0073615fa8 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -532,6 +532,7 @@ struct kvm_mmu_notifier_range {
 	bool flush_on_ret;
 	bool may_block;
 	bool lockless;
+	bool perm_upgrade;
 };
 
 /*
@@ -613,6 +614,7 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
 			 */
 			gfn_range.arg = range->arg;
 			gfn_range.may_block = range->may_block;
+			gfn_range.perm_upgrade = range->perm_upgrade;
 			/*
 			 * HVA-based notifications aren't relevant to private
 			 * mappings as they don't have a userspace mapping.
@@ -738,6 +740,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
 		.on_lock	= kvm_mmu_invalidate_start,
 		.flush_on_ret	= true,
 		.may_block	= mmu_notifier_range_blockable(range),
+		.perm_upgrade = !!(range->flags & MMU_NOTIFIER_RANGE_PERM_UPGRADE),
 	};
 
 	trace_kvm_unmap_hva_range(range->start, range->end);
@@ -2552,6 +2555,7 @@ static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
 
 	gfn_range.arg = range->arg;
 	gfn_range.may_block = range->may_block;
+	gfn_range.perm_upgrade = range->perm_upgrade;
 
 	/*
 	 * If/when KVM supports more attributes beyond private .vs shared, this
-- 
2.51.0

[RFC PATCH 4/4] KVM: x86/mmu: don't zap the secondary MMU on permission upgrade

[Carlos López]

When an mmu_notifier invalidation only upgrades permissions
(MMU_NOTIFIER_RANGE_PERM_UPGRADE), keep the existing SPTEs instead of
zapping them. The PFN is unchanged and no access is revoked, so a held
SPTE is at most as permissive as before. Permissions for the SPTE are
updated on the next fault, so dropping them early gains nothing.

The motivating case is a userfaultfd-wp dirty tracker such as QEMU's
background snapshot, which unprotects every guest page as it saves it.
Zapping on each unprotect makes the guest refault its whole working
set. On a 64GB guest under a background snapshot, a read-mostly workload
drops secondary-MMU refaults from ~1M/s to ~2.5K/s and cross-vCPU TLB
shootdowns from ~1400/s to near zero. Uses of mprotect() that only add
access benefit likewise.

Signed-off-by: Carlos López <clopez@suse.de>
---
 arch/x86/kvm/mmu/mmu.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index dc2aef72ffdd..a4258a8c0c89 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -1719,6 +1719,13 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 	lockdep_assert_once(kvm->mmu_invalidate_in_progress ||
 			    lockdep_is_held(&kvm->slots_lock));
 
+	/*
+	 * Do not bother zapping SPTEs if this is only a protection upgrade.
+	 * Permissions will be faulted-in on demand.
+	 */
+	if (range->perm_upgrade)
+		return false;
+
 	if (kvm_memslots_have_rmaps(kvm))
 		flush = __kvm_rmap_zap_gfn_range(kvm, range->slot,
 						 range->start, range->end,
-- 
2.51.0

Re: [RFC PATCH 0/4] KVM: x86/mmu: don't zap secondary MMU on permission upgrade

[Claudio Imbrenda]

On Thu, 29 Oct 2026 15:33:33 +0100
Carlos López <clopez@suse.de> wrote:

> Prevent x86 KVM from over-zapping its secondary MMU on PTE permission
> upgrades, which leave the PFNs unchanged and do not revoke access.
> Permissions for SPTEs that KVM holds are updated on the next fault, so
> dropping them early gains nothing.
> 
> Do this by adding a new MMU notifier flag to signal that a
> MMU_NOTIFY_PROTECTION_VMA strictly upgrades (loosens) permissions on a
> particular range.
> 
> See below for benchmark results.
> 
> Background
> ==========
> 
> change_protection() pairs every protection change with an
> MMU_NOTIFY_PROTECTION_VMA invalidation, and KVM zaps its SPTEs in
> response, as it cannot tell whether permissions widened or narrowed.
> This is costly e.g. during a QEMU background snapshot, which relies on
> uffd-wp to write-protect guest RAM and then restore permissions as each
> page is saved. Without this series, the permission restore causes the
> guest to re-fault read-only memory needlessly.
> 
> Note, the reverse situation is already handled for secondary MMUs
> that reuse the CPU page tables (e.g. ARM SMMU, IOMMU/ATS). They cannot
> refault, so they are notified of a permission upgrade on the CPU
> TLB-flush path [1].
> 
> KVM, however, owns separate page tables and faults permissions on demand,
> so it needs no such notification (as Sean noted when that signalling
> was first proposed on the fault path [2]). This series applies the same
> principle to the change_protection() path, signalling the permission
> upgrade so KVM can avoid the unnecessary zap.
> 
> RFC Considerations
> ==================
> 
> 1. Having the zap skip in KVM's arch-independent code does not seem
>    obviously correct, as s390 caches additional translation state
>    that may not be rebuilt by refaults. Maybe an expert can correct
>    me. The skip could be extended to arm64 and RISC-V, but I have not
>    benchmarked the benefits and I'd like to get feedback first. Hence,
>    x86-only for now.
> 
> 2. change_protection() could simply not emit the invalidation at all
>    for a pure permission upgrade (shared-page-table MMUs invalidate
>    on the CPU TLB-flush path, as noted above, so they do not need it).
>    But again, the s390 case could be broken, and even then, that would
>    seem too much intrusion for KVM's benefit.
> 
> 3. The flag reaches KVM as a bool on struct kvm_gfn_range.
>    Alternatively, KVM could forward the notifier event, as Claudio
>    Imbrenda suggested at LPC [3] and others have proposed in the past
>    for different purposes [4] [5]. That, along with the event flags,
>    could serve the same purpose. I'd be happy to take that approach if
>    there is agreement that other consumers would want it.
> 
>    (The LPC talk recording has some audio issues so I am not too sure
>    on what the final agreement was)

AFAIR the consensus was:
- adding the reason to the kvm kvm_gfn_range is fine.
- having a return value from the notifier to pass information back is
  _not_ fine; I should add a new callback instead.

regarding this series:

on x86 you don't care what the new permissions are, because you're
updating when re-faulting (if I understood correctly).

for s390 it might be beneficial to know what the new permissions are;
of course we could look up the vma and find out, but since you are
already checking the permissions, maybe you can find a way to also
convey the new permission bits?

that would make our life on s390 easier :)

> 
> Benchmarks
> ==========
> 
> AMD EPYC 9654 96-Core Processor host running a recent Linux kernel
> (6b4b543fa187), with a 64GB, 64-vCPU guest running stress-ng --vm while
> the host takes a QEMU background snapshot. All results are aggregated
> over 5 runs, and compared between the stock kernel and one with this
> patch series. Guest is rebooted across each individual run, and NUMA
> balancing is disabled.
> 
> pf/s and tlb/s (per-second deltas of KVM's pf_taken and remote_tlb_flush
> debugfs stats) clearly improve:
> 
>   guest pattern          baseline                    patched
>   -----------------------------------------------------------------------
>   read64   (read)        pf/s=950,015   tlb/s=1,440  pf/s=2,503 tlb/s=0.1
>   rand-set (read+write)  pf/s=1,004,353 tlb/s=862    pf/s=4,073 tlb/s=0.3
>   write64  (write)       pf/s=968,018   tlb/s=94     pf/s=2,292 tlb/s=0.3
> 
> Guest read latency was sampled with a paced probe thread in the
> guest, reading one byte per page with an idle gap between reads, while
> stress-ng ran with read64. Each figure is the median across 5 runs, with
> the run-to-run min and max, in ns:
> 
>   metric           baseline             patched
>   ------------------------------------------------------
>   p50 (min max)      340     (270 380)   260   (250 260)
>   p99 (min max)    19789 (16885 20200)  5207 (4076 5638)
>   p99.9 (min max)  23445 (22845 24036)  6850 (6259 7110)
> 
> The median improves modestly and the refault cost shows in the tail,
> where p99 drops ~3.8x.
> 
> A separate stress-ng run with write64 and --verify showed no data
> corruption and no livelocks.
> 
> References
> ==========
> 
> [1] https://lore.kernel.org/linux-mm/cover.1eca029b8603ef4eebe5b41eae51facfc5920c41.1690292440.git-series.apopple@nvidia.com/
> [2] https://lore.kernel.org/linux-mm/ZGu1vsbscdO48V6h@google.com/
> [3] https://lpc.events/event/20/contributions/2552/
> [4] https://lore.kernel.org/all/20240801183453.57199-1-seanjc@google.com
> [5] https://lore.kernel.org/all/20231110003734.1014084-1-jackyli@google.com
> 
> Carlos López (4):
>   mm/mprotect: add MM_CP_PERM_UPGRADE
>   mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE
>   KVM: plumb MMU_NOTIFIER_RANGE_PERM_UPGRADE into kvm_gfn_range
>   KVM: x86/mmu: don't zap the secondary MMU on permission upgrade
> 
>  arch/x86/kvm/mmu/mmu.c       |  7 +++++++
>  include/linux/kvm_host.h     |  1 +
>  include/linux/mm.h           |  8 ++++++++
>  include/linux/mmu_notifier.h |  5 +++++
>  mm/hugetlb.c                 |  6 +++++-
>  mm/mprotect.c                | 20 +++++++++++++++++++-
>  virt/kvm/kvm_main.c          |  4 ++++
>  7 files changed, 49 insertions(+), 2 deletions(-)
> 

Re: [RFC PATCH 0/4] KVM: x86/mmu: don't zap secondary MMU on permission upgrade

[Carlos López]

On 10/29/26 5:52 PM, Claudio Imbrenda wrote:
> On Thu, 29 Oct 2026 15:33:33 +0100
> Carlos López <clopez@suse.de> wrote:
> 
>> Prevent x86 KVM from over-zapping its secondary MMU on PTE permission
>> upgrades, which leave the PFNs unchanged and do not revoke access.
>> Permissions for SPTEs that KVM holds are updated on the next fault, so
>> dropping them early gains nothing.
>>
>> Do this by adding a new MMU notifier flag to signal that a
>> MMU_NOTIFY_PROTECTION_VMA strictly upgrades (loosens) permissions on a
>> particular range.
>>
>> See below for benchmark results.
>>
>> Background
>> ==========
>>
>> change_protection() pairs every protection change with an
>> MMU_NOTIFY_PROTECTION_VMA invalidation, and KVM zaps its SPTEs in
>> response, as it cannot tell whether permissions widened or narrowed.
>> This is costly e.g. during a QEMU background snapshot, which relies on
>> uffd-wp to write-protect guest RAM and then restore permissions as each
>> page is saved. Without this series, the permission restore causes the
>> guest to re-fault read-only memory needlessly.
>>
>> Note, the reverse situation is already handled for secondary MMUs
>> that reuse the CPU page tables (e.g. ARM SMMU, IOMMU/ATS). They cannot
>> refault, so they are notified of a permission upgrade on the CPU
>> TLB-flush path [1].
>>
>> KVM, however, owns separate page tables and faults permissions on demand,
>> so it needs no such notification (as Sean noted when that signalling
>> was first proposed on the fault path [2]). This series applies the same
>> principle to the change_protection() path, signalling the permission
>> upgrade so KVM can avoid the unnecessary zap.
>>
>> RFC Considerations
>> ==================
>>
>> 1. Having the zap skip in KVM's arch-independent code does not seem
>>    obviously correct, as s390 caches additional translation state
>>    that may not be rebuilt by refaults. Maybe an expert can correct
>>    me. The skip could be extended to arm64 and RISC-V, but I have not
>>    benchmarked the benefits and I'd like to get feedback first. Hence,
>>    x86-only for now.
>>
>> 2. change_protection() could simply not emit the invalidation at all
>>    for a pure permission upgrade (shared-page-table MMUs invalidate
>>    on the CPU TLB-flush path, as noted above, so they do not need it).
>>    But again, the s390 case could be broken, and even then, that would
>>    seem too much intrusion for KVM's benefit.
>>
>> 3. The flag reaches KVM as a bool on struct kvm_gfn_range.
>>    Alternatively, KVM could forward the notifier event, as Claudio
>>    Imbrenda suggested at LPC [3] and others have proposed in the past
>>    for different purposes [4] [5]. That, along with the event flags,
>>    could serve the same purpose. I'd be happy to take that approach if
>>    there is agreement that other consumers would want it.
>>
>>    (The LPC talk recording has some audio issues so I am not too sure
>>    on what the final agreement was)
> 
> AFAIR the consensus was:
> - adding the reason to the kvm kvm_gfn_range is fine.
> - having a return value from the notifier to pass information back is
>   _not_ fine; I should add a new callback instead.
> 
> regarding this series:
> 
> on x86 you don't care what the new permissions are, because you're
> updating when re-faulting (if I understood correctly).

Correct.

> for s390 it might be beneficial to know what the new permissions are;
> of course we could look up the vma and find out, but since you are
> already checking the permissions, maybe you can find a way to also
> convey the new permission bits?
> 
> that would make our life on s390 easier :)

Right, getting the new bits from vma->flags should be trivially done
from the MMU_NOTIFY_PROTECTION_VMA callsites. There'd be no users in
this series though. The rest of architectures still need the notifier to
tell them if the change is an upgrade, since they don't have an easy way
to grab the old RWX bits. Passing the old RWX bits from the MMU notifier
does not seem trivial to me either.

So what I can do is to plumb down the event and flags into KVM, and
leave it open for you to come back and forward the permission bits you
need via the range flags, if the MM maintainers are okay with it.

However, I tried to understand the s390 code better with AI, and it
seems to think that skipping the zap is also safe there (and an
improvement). It could be further optimized by passing down the new W
bit to eagerly fix the gmap when restoring write permissions.

>>
>> Benchmarks
>> ==========
>>
>> AMD EPYC 9654 96-Core Processor host running a recent Linux kernel
>> (6b4b543fa187), with a 64GB, 64-vCPU guest running stress-ng --vm while
>> the host takes a QEMU background snapshot. All results are aggregated
>> over 5 runs, and compared between the stock kernel and one with this
>> patch series. Guest is rebooted across each individual run, and NUMA
>> balancing is disabled.
>>
>> pf/s and tlb/s (per-second deltas of KVM's pf_taken and remote_tlb_flush
>> debugfs stats) clearly improve:
>>
>>   guest pattern          baseline                    patched
>>   -----------------------------------------------------------------------
>>   read64   (read)        pf/s=950,015   tlb/s=1,440  pf/s=2,503 tlb/s=0.1
>>   rand-set (read+write)  pf/s=1,004,353 tlb/s=862    pf/s=4,073 tlb/s=0.3
>>   write64  (write)       pf/s=968,018   tlb/s=94     pf/s=2,292 tlb/s=0.3
>>
>> Guest read latency was sampled with a paced probe thread in the
>> guest, reading one byte per page with an idle gap between reads, while
>> stress-ng ran with read64. Each figure is the median across 5 runs, with
>> the run-to-run min and max, in ns:
>>
>>   metric           baseline             patched
>>   ------------------------------------------------------
>>   p50 (min max)      340     (270 380)   260   (250 260)
>>   p99 (min max)    19789 (16885 20200)  5207 (4076 5638)
>>   p99.9 (min max)  23445 (22845 24036)  6850 (6259 7110)
>>
>> The median improves modestly and the refault cost shows in the tail,
>> where p99 drops ~3.8x.
>>
>> A separate stress-ng run with write64 and --verify showed no data
>> corruption and no livelocks.
>>
>> References
>> ==========
>>
>> [1] https://lore.kernel.org/linux-mm/cover.1eca029b8603ef4eebe5b41eae51facfc5920c41.1690292440.git-series.apopple@nvidia.com/
>> [2] https://lore.kernel.org/linux-mm/ZGu1vsbscdO48V6h@google.com/
>> [3] https://lpc.events/event/20/contributions/2552/
>> [4] https://lore.kernel.org/all/20240801183453.57199-1-seanjc@google.com
>> [5] https://lore.kernel.org/all/20231110003734.1014084-1-jackyli@google.com
>>
>> Carlos López (4):
>>   mm/mprotect: add MM_CP_PERM_UPGRADE
>>   mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE
>>   KVM: plumb MMU_NOTIFIER_RANGE_PERM_UPGRADE into kvm_gfn_range
>>   KVM: x86/mmu: don't zap the secondary MMU on permission upgrade
>>
>>  arch/x86/kvm/mmu/mmu.c       |  7 +++++++
>>  include/linux/kvm_host.h     |  1 +
>>  include/linux/mm.h           |  8 ++++++++
>>  include/linux/mmu_notifier.h |  5 +++++
>>  mm/hugetlb.c                 |  6 +++++-
>>  mm/mprotect.c                | 20 +++++++++++++++++++-
>>  virt/kvm/kvm_main.c          |  4 ++++
>>  7 files changed, 49 insertions(+), 2 deletions(-)
>>
> 

Re: [RFC PATCH 0/4] KVM: x86/mmu: don't zap secondary MMU on permission upgrade

[Claudio Imbrenda]

On Fri, 30 Oct 2026 12:11:07 +0100
Carlos López <clopez@suse.de> wrote:

[...]

> > regarding this series:
> > 
> > on x86 you don't care what the new permissions are, because you're
> > updating when re-faulting (if I understood correctly).  
> 
> Correct.
> 
> > for s390 it might be beneficial to know what the new permissions are;
> > of course we could look up the vma and find out, but since you are
> > already checking the permissions, maybe you can find a way to also
> > convey the new permission bits?
> > 
> > that would make our life on s390 easier :)  
> 
> Right, getting the new bits from vma->flags should be trivially done
> from the MMU_NOTIFY_PROTECTION_VMA callsites. There'd be no users in
> this series though. The rest of architectures still need the notifier to
> tell them if the change is an upgrade, since they don't have an easy way
> to grab the old RWX bits. Passing the old RWX bits from the MMU notifier
> does not seem trivial to me either.
> 
> So what I can do is to plumb down the event and flags into KVM, and
> leave it open for you to come back and forward the permission bits you
> need via the range flags, if the MM maintainers are okay with it.
> 
> However, I tried to understand the s390 code better with AI, and it
> seems to think that skipping the zap is also safe there (and an
> improvement). It could be further optimized by passing down the new W
> bit to eagerly fix the gmap when restoring write permissions.

right, the fast path would fail (gmap_try_fixup_minor()), and the slow
path will call __kvm_faultin_pfn(), which will update the flags as
needed.

knowing whether the new mapping is writable would allow us to set the
w softbit (which is a fast operation), thus allowing the fast path to
handle the fault afterwards, instead of the slow path.

but even just avoiding the zap would surely be beneficial already

I can try to test this next week and see if anything explodes

> 
> >>
> >> Benchmarks
> >> ==========
> >>
> >> AMD EPYC 9654 96-Core Processor host running a recent Linux kernel
> >> (6b4b543fa187), with a 64GB, 64-vCPU guest running stress-ng --vm while
> >> the host takes a QEMU background snapshot. All results are aggregated
> >> over 5 runs, and compared between the stock kernel and one with this
> >> patch series. Guest is rebooted across each individual run, and NUMA
> >> balancing is disabled.
> >>
> >> pf/s and tlb/s (per-second deltas of KVM's pf_taken and remote_tlb_flush
> >> debugfs stats) clearly improve:
> >>
> >>   guest pattern          baseline                    patched
> >>   -----------------------------------------------------------------------
> >>   read64   (read)        pf/s=950,015   tlb/s=1,440  pf/s=2,503 tlb/s=0.1
> >>   rand-set (read+write)  pf/s=1,004,353 tlb/s=862    pf/s=4,073 tlb/s=0.3
> >>   write64  (write)       pf/s=968,018   tlb/s=94     pf/s=2,292 tlb/s=0.3
> >>
> >> Guest read latency was sampled with a paced probe thread in the
> >> guest, reading one byte per page with an idle gap between reads, while
> >> stress-ng ran with read64. Each figure is the median across 5 runs, with
> >> the run-to-run min and max, in ns:
> >>
> >>   metric           baseline             patched
> >>   ------------------------------------------------------
> >>   p50 (min max)      340     (270 380)   260   (250 260)
> >>   p99 (min max)    19789 (16885 20200)  5207 (4076 5638)
> >>   p99.9 (min max)  23445 (22845 24036)  6850 (6259 7110)
> >>
> >> The median improves modestly and the refault cost shows in the tail,
> >> where p99 drops ~3.8x.
> >>
> >> A separate stress-ng run with write64 and --verify showed no data
> >> corruption and no livelocks.
> >>
> >> References
> >> ==========
> >>
> >> [1] https://lore.kernel.org/linux-mm/cover.1eca029b8603ef4eebe5b41eae51facfc5920c41.1690292440.git-series.apopple@nvidia.com/
> >> [2] https://lore.kernel.org/linux-mm/ZGu1vsbscdO48V6h@google.com/
> >> [3] https://lpc.events/event/20/contributions/2552/
> >> [4] https://lore.kernel.org/all/20240801183453.57199-1-seanjc@google.com
> >> [5] https://lore.kernel.org/all/20231110003734.1014084-1-jackyli@google.com
> >>
> >> Carlos López (4):
> >>   mm/mprotect: add MM_CP_PERM_UPGRADE
> >>   mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE
> >>   KVM: plumb MMU_NOTIFIER_RANGE_PERM_UPGRADE into kvm_gfn_range
> >>   KVM: x86/mmu: don't zap the secondary MMU on permission upgrade
> >>
> >>  arch/x86/kvm/mmu/mmu.c       |  7 +++++++
> >>  include/linux/kvm_host.h     |  1 +
> >>  include/linux/mm.h           |  8 ++++++++
> >>  include/linux/mmu_notifier.h |  5 +++++
> >>  mm/hugetlb.c                 |  6 +++++-
> >>  mm/mprotect.c                | 20 +++++++++++++++++++-
> >>  virt/kvm/kvm_main.c          |  4 ++++
> >>  7 files changed, 49 insertions(+), 2 deletions(-)
> >>  
> >   
> 

Re: [RFC PATCH 1/4] mm/mprotect: add MM_CP_PERM_UPGRADE

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] mm/mprotect: MM_CP_PERM_UPGRADE set when change_protection() revokes write
- [Medium] mm/mprotect: vma_flags_is_perm_upgrade() ignores arch-specific prot flags
--

commit f3d5a380cac712469a433d985427420721eea496
Author: Carlos López <clopez@suse.de>

mm/mprotect: add MM_CP_PERM_UPGRADE

This commit introduces the MM_CP_PERM_UPGRADE flag for change_protection()
when mprotect() updates VMA permissions to a superset of the previous
permissions without changing the protection key, allowing MMU notifier
consumers to avoid invalidating PTEs.

> diff --git a/mm/mprotect.c b/mm/mprotect.c
> index 754333698c1ea..96e32e0159166 100644
> --- a/mm/mprotect.c
> +++ b/mm/mprotect.c
> @@ -756,6 +756,17 @@ static const struct mm_walk_ops prot_none_walk_ops = {
>  	.walk_lock		= PGWALK_WRLOCK,
>  };
>  
> +static bool vma_flags_is_perm_upgrade(vma_flags_t old_vma_flags,
> +				       vma_flags_t new_vma_flags)
> +{
> +	vm_flags_t old = vma_flags_to_legacy(old_vma_flags);
> +	vm_flags_t new = vma_flags_to_legacy(new_vma_flags);
> +	vm_flags_t rwx = VM_READ | VM_WRITE | VM_EXEC;
> +
> +	return !(old & rwx & ~new) &&
> +	       (old & ARCH_VM_PKEY_FLAGS) == (new & ARCH_VM_PKEY_FLAGS);
> +}

[Severity: Medium]
Does vma_flags_is_perm_upgrade() also need to check architecture-specific
protection flags that mprotect() can modify?

In do_mprotect_pkey(), newflags is computed via calc_vm_prot_bits() (which
calls arch_calc_vm_prot_bits()) and masks off VM_FLAGS_CLEAR (which includes
VM_ARCH_CLEAR):

mm/mprotect.c:do_mprotect_pkey() {
    ...
    mask_off_old_flags = VM_ACCESS_FLAGS | VM_FLAGS_CLEAR;

    new_vma_pkey = arch_override_mprotect_pkey(vma, prot, pkey);
    newflags = calc_vm_prot_bits(prot, new_vma_pkey);
    newflags |= (vma->vm_flags & ~mask_off_old_flags);
    ...
}

Several architectures expose PROT_* flags in mprotect() that set or clear
VMA flags affecting hardware PTE protection or attribute bits in
vm_get_page_prot():

- On arm64, arch_calc_vm_prot_bits() maps PROT_BTI to VM_ARM64_BTI (cleared
  via VM_ARCH_CLEAR when omitted) and PROT_MTE to VM_MTE. In
  arch/arm64/mm/mmap.c:vm_get_page_prot(), VM_ARM64_BTI sets PTE_GP (which
  restricts execution by faulting on indirect branches to non-BTI landing
  pads) and VM_MTE sets PTE_ATTRINDX(MT_NORMAL_TAGGED) (which enables MTE
  tag check faults).
- On sparc64, PROT_ADI sets VM_SPARC_ADI (cleared via VM_ARCH_CLEAR when
  omitted), which sets _PAGE_MCD_4V in
  arch/sparc/mm/init_64.c:vm_get_page_prot() to enable hardware ADI tag
  checking traps.
- On powerpc64, PROT_SAO sets VM_SAO, which sets _PAGE_SAO in
  arch/powerpc/mm/book3s64/pgtable.c:vm_get_page_prot() to alter hardware
  memory ordering attributes.

Because vma_flags_is_perm_upgrade() only inspects VM_READ | VM_WRITE |
VM_EXEC and ARCH_VM_PKEY_FLAGS, if userspace calls mprotect() to enable
PROT_BTI, PROT_MTE, PROT_ADI, or PROT_SAO while keeping the read/write/exec
bits identical (or a superset) and the pkey unchanged, won't this set
MM_CP_PERM_UPGRADE and allow MMU notifier consumers to skip secondary MMU
invalidation when access restrictions were tightened or PTE attributes
changed?

> +
>  int
>  mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,
>  	       struct vm_area_struct *vma, struct vm_area_struct **pprev,
> @@ -836,6 +847,8 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,
>  	vma_flags_reset_once(vma, &new_vma_flags);
>  	if (vma_wants_manual_pte_write_upgrade(vma))
>  		mm_cp_flags |= MM_CP_TRY_CHANGE_WRITABLE;
> +	if (vma_flags_is_perm_upgrade(old_vma_flags, new_vma_flags))
> +		mm_cp_flags |= MM_CP_PERM_UPGRADE;
>  	vma_set_page_prot(vma);
>  
>  	change_protection(tlb, vma, start, end, mm_cp_flags);

[Severity: High]
Can setting MM_CP_PERM_UPGRADE when the VMA already had VM_WRITE (or on a
hugetlb VMA) violate the invariant that no access is revoked and the PFN
stays the same?

Follow-up commits ("mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE",
"KVM: plumb MMU_NOTIFIER_RANGE_PERM_UPGRADE into kvm_gfn_range", and
"KVM: x86/mmu: don't zap the secondary MMU on permission upgrade") wire
MM_CP_PERM_UPGRADE to MMU_NOTIFIER_RANGE_PERM_UPGRADE so that
kvm_unmap_gfn_range() skips zapping secondary MMU SPTEs.

However, when a VMA already has VM_WRITE and mprotect() adds permissions
(for example, upgrading PROT_READ | PROT_WRITE to
PROT_READ | PROT_WRITE | PROT_EXEC, or PROT_WRITE to
PROT_READ | PROT_WRITE), vma_set_page_prot() computes vma->vm_page_prot
(newprot) without write permission for all MAP_PRIVATE mappings (COW) and
for MAP_SHARED mappings where vma_wants_writenotify() is true.

When change_protection() applies this read-only newprot across the VMA,
there are several paths where write permission is stripped from existing
entries or entries are unmapped in the primary MMU while MM_CP_PERM_UPGRADE
is set:

1. In mprotect_fixup(), vma_wants_manual_pte_write_upgrade(vma) is called
   after vma_flags_reset_once() but before vma_set_page_prot(vma). For
   MAP_SHARED VMAs without vm_ops->page_mkwrite that use writenotify for
   VM_UFFD_WP or soft-dirty tracking (such as shmem),
   vma_wants_writenotify(vma, vma->vm_page_prot) compares the old read-only
   vma->vm_page_prot against vma_pgprot_modify(vm_page_prot, vma->flags)
   (which still has VMA_SHARED_BIT | VMA_WRITE_BIT) and returns false. As a
   result, MM_CP_TRY_CHANGE_WRITABLE is not set, vma_set_page_prot()
   computes a read-only vma->vm_page_prot, and change_present_ptes() /
   change_huge_pmd() strip write permission from all writable PTEs and PMDs
   in the VMA.

2. In mm/hugetlb.c:hugetlb_change_protection(), MM_CP_TRY_CHANGE_WRITABLE
   is not handled, so huge_pte_modify(old_pte, newprot) unconditionally
   strips write permission from already-writable COW huge PTEs in
   MAP_PRIVATE hugetlb VMAs; in addition, huge_pmd_unshare() unmaps shared
   PMD tables via pud_clear().

3. In mm/huge_memory.c:change_huge_pud(), MM_CP_TRY_CHANGE_WRITABLE is not
   handled, so pud_modify(oldpud, newprot) strips write permission from
   writable huge PUDs in writenotify DAX mappings.

4. In mm/mprotect.c:change_softleaf_pte(),
   mm/huge_memory.c:change_non_present_huge_pmd(), and
   mm/hugetlb.c:hugetlb_change_protection(), writable migration entries
   (softleaf_is_migration_write()) and writable device-private entries
   (softleaf_is_device_private_write()) are unconditionally downgraded to
   read-only entries via make_readable_exclusive_migration_entry(),
   make_readable_migration_entry(), or make_readable_device_private_entry().

5. In change_present_ptes() and change_huge_pmd(), pte_modify() /
   pmd_modify() strip write permission and do not restore it under
   MM_CP_TRY_CHANGE_WRITABLE when !folio in a private PFN/mixed mapping or
   when a shared writenotify PTE was cleaned (!pte_dirty(ptent)).

Wouldn't skipping secondary MMU invalidation in these cases leave secondary
MMUs (such as KVM SPTEs) with stale writable mappings to pages that
change_protection() has write-protected or unmapped in the primary MMU?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261029143336.390231-3-clopez@suse.de?part=1

Re: [RFC PATCH 2/4] mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Critical] mm/hugetlb: hugetlb_change_protection() sets PERM_UPGRADE when revoking access
- [High] mm/mmu_notifier: change_protection() sets PERM_UPGRADE when revoking write
- [High] mm: change_protection() revokes write access under MMU_NOTIFIER_RANGE_PERM_UPGRADE
- [Low] mm/mmu_notifier: unused-but-set mn_flags when CONFIG_MMU_NOTIFIER=n
--

commit 22cbc27b0ff3d2d6c7ee718ba2fc52e905cded0f
Author: Carlos López <clopez@suse.de>

mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE

This commit adds the MMU_NOTIFIER_RANGE_PERM_UPGRADE range flag to mark MMU
notifier invalidations that only upgrade permissions without revoking
existing access or changing PFNs. It sets this flag in
hugetlb_change_protection() and change_pud_range() when
mm_cp_is_perm_upgrade(cp_flags) is true.

> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index da5d04edf7c10..3a988ca364e6c 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -6596,19 +6596,23 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
>  	long pages = 0, psize = huge_page_size(h);
>  	struct mmu_notifier_range range;
>  	unsigned long last_addr_mask;
> +	unsigned int mn_flags = 0;
>  	bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
>  	bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
>  	bool uffd_rwp = cp_flags & MM_CP_UFFD_RWP;
>  	bool uffd_rwp_resolve = cp_flags & MM_CP_UFFD_RWP_RESOLVE;
>  	struct mmu_gather tlb;
>  
> +	if (mm_cp_is_perm_upgrade(cp_flags))
> +		mn_flags |= MMU_NOTIFIER_RANGE_PERM_UPGRADE;
> +
>  	/*
>  	 * In the case of shared PMDs, the area to flush could be beyond
>  	 * start/end.  Set range.start/range.end to cover the maximum possible
>  	 * range if PMD sharing is possible.
>  	 */
>  	mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA,
> -				0, mm, start, end);
> +				mn_flags, mm, start, end);

[Severity: Critical]
Can setting MMU_NOTIFIER_RANGE_PERM_UPGRADE in hugetlb_change_protection()
leave dangling mappings in a secondary MMU or leave writable secondary MMU
mappings to pages that the host unmapped or downgraded to read-only?

First, when PMD sharing is active on a MAP_SHARED hugetlbfs mapping (for
example, two processes sharing a PUD_SIZE-aligned PROT_READ mapping of a 2MB
hugetlbfs folio where only one process holds the folio rmap count on the
shared PMD table) and the process with a secondary MMU calls mprotect() to
upgrade to PROT_READ | PROT_WRITE, mprotect_fixup() sets MM_CP_PERM_UPGRADE.
hugetlb_change_protection() then advertises MMU_NOTIFIER_RANGE_PERM_UPGRADE,
telling secondary MMUs such as KVM to keep their existing SPTEs, and proceeds
to unshare and unmap the PUD entry:

mm/hugetlb.c:hugetlb_change_protection()
  -> huge_pmd_unshare()
    -> __huge_pmd_unshare()
         pud_clear(pud);

Because the PUD in this mm is now cleared and detached from the shared PMD
page table without zapping the secondary MMU, when the other process later
unmaps the file, unmap_hugepage_range() only notifies that other mm and drops
the shared folio's mapcount to zero. If the folio is subsequently migrated or
freed in migrate_hugetlb_folio(), folio_mapped(src) is false and
try_to_migrate() is skipped, freeing or reusing the physical page while the
first process's secondary MMU still retains a live mapping to it.

Second, unlike change_present_ptes() and change_huge_pmd(),
hugetlb_change_protection() ignores MM_CP_TRY_CHANGE_WRITABLE and
unconditionally overwrites the huge PTE protection with newprot
(vma->vm_page_prot):

mm/hugetlb.c:hugetlb_change_protection() {
    ...
    old_pte = huge_ptep_modify_prot_start(vma, address, ptep);
    pte = huge_pte_modify(old_pte, newprot);
    pte = arch_make_huge_pte(pte, shift, vma->vm_flags);
    ...
}

For any MAP_PRIVATE hugetlb VMA, and for any MAP_SHARED hugetlb VMA with
VM_UFFD_WP enabled (where vma_wants_writenotify() is true and
vma_set_page_prot() clears VMA_SHARED_BIT from vma->vm_page_prot), newprot is
read-only. When mprotect() upgrades permissions (for example adding
PROT_EXEC with MM_CP_PERM_UPGRADE) or userfaultfd resolves write protection
(MM_CP_UFFD_WP_RESOLVE via uffd_wp_range() or userfaultfd_clear_vma()), any
already-writable huge PTE in the range has its write bit stripped to
read-only while MMU_NOTIFIER_RANGE_PERM_UPGRADE tells secondary MMUs not to
zap their writable mappings.

Similarly, hugetlb_change_protection() unconditionally downgrades writable
migration entries to readable migration entries even when
mm_cp_is_perm_upgrade(cp_flags) is true:

mm/hugetlb.c:hugetlb_change_protection() {
    ...
    if (softleaf_is_migration_write(entry)) {
        if (folio_test_anon(folio))
            entry = make_readable_exclusive_migration_entry(
                        swp_offset(entry));
        else
            entry = make_readable_migration_entry(
                        swp_offset(entry));
        newpte = swp_entry_to_pte(entry);
        pages++;
    }
    ...
}

> diff --git a/mm/mprotect.c b/mm/mprotect.c
> index 96e32e0159166..8c17875cdcfa6 100644
> --- a/mm/mprotect.c
> +++ b/mm/mprotect.c
> @@ -597,8 +597,13 @@ static inline long change_pud_range(struct mmu_gather *tlb,
>  			continue;
>  
>  		if (!range.start) {
> +			unsigned int mn_flags = 0;
> +
> +			if (mm_cp_is_perm_upgrade(cp_flags))
> +				mn_flags |= MMU_NOTIFIER_RANGE_PERM_UPGRADE;
> +
>  			mmu_notifier_range_init(&range,
> -						MMU_NOTIFY_PROTECTION_VMA, 0,
> +						MMU_NOTIFY_PROTECTION_VMA, mn_flags,
>  						vma->vm_mm, addr, end);
>  			mmu_notifier_invalidate_range_start(&range);
>  		}

[Severity: High]
Does change_pud_range() also set MMU_NOTIFIER_RANGE_PERM_UPGRADE in cases
where the underlying page table walk actually revokes write access?

1. In change_softleaf_pte() and change_non_present_huge_pmd() (as well as
hugetlb_change_protection()), cp_flags is not checked when handling writable
non-present entries. Both SOFTLEAF_MIGRATION_WRITE and
SOFTLEAF_DEVICE_PRIVATE_WRITE entries are unconditionally downgraded to
readable entries during permission upgrades:

mm/mprotect.c:change_softleaf_pte() {
    ...
    if (softleaf_is_migration_write(entry)) {
        ...
        if (folio_test_anon(folio))
            entry = make_readable_exclusive_migration_entry(swp_offset(entry));
        else
            entry = make_readable_migration_entry(swp_offset(entry));
        ...
    } else if (softleaf_is_device_private_write(entry)) {
        entry = make_readable_device_private_entry(swp_offset(entry));
        ...
    }
    ...
}

2. In change_huge_pud() (as in hugetlb_change_protection()),
MM_CP_TRY_CHANGE_WRITABLE is not handled:

mm/huge_memory.c:change_huge_pud() {
    ...
    oldpud = pudp_invalidate(vma, addr, pudp);
    entry = pud_modify(oldpud, newprot);
    set_pud_at(mm, addr, pudp, entry);
    ...
}

Whenever vma->vm_page_prot is read-only (such as a MAP_SHARED DAX PUD VMA
where vma_wants_writenotify() is true), pud_modify(oldpud, newprot) strips
write permission from already-writable huge PUDs while
MMU_NOTIFIER_RANGE_PERM_UPGRADE is advertised.

3. In change_present_ptes() and change_huge_pmd(), pte_modify() and
pmd_modify() first apply newprot (which is read-only when !VM_SHARED or
vma_wants_writenotify()), and only restore write permission if
MM_CP_TRY_CHANGE_WRITABLE is set and can_change_pte_writable() or
can_change_pmd_writable() returns true. For already-writable PTEs or PMDs
where can_change_shared_pte_writable() returns false (for example, a writable
shared PTE cleaned by clean_record_pte(), or a clean writable
VM_PFNMAP/VM_MIXEDMAP entry) or where vm_normal_page() is NULL in a
!VM_SHARED VMA, write permission is stripped while secondary MMUs are told
no access was revoked.

[Severity: High]
What happens when change_protection() runs with MM_CP_PERM_UPGRADE,
MM_CP_UFFD_WP_RESOLVE, or MM_CP_UFFD_RWP_RESOLVE on a writable MAP_SHARED VMA
without vm_ops->page_mkwrite/pfn_mkwrite (such as shmem/tmpfs) after
vma->vm_page_prot has been downgraded to PAGE_COPY for userfaultfd-wp or
soft-dirty tracking?

When userfaultfd-wp is registered or soft-dirty is cleared on a writable
MAP_SHARED shmem VMA, vma_set_page_prot() sees vma_wants_writenotify() return
true, clears VMA_SHARED_BIT on its local flags copy, and stores read-only
PAGE_COPY in vma->vm_page_prot while vma->flags retains
VMA_SHARED_BIT | VMA_WRITE_BIT.

Later, when uffd_wp_range(), userfaultfd_clear_vma(), or mprotect_fixup()
checks whether MM_CP_TRY_CHANGE_WRITABLE should be set, it calls
vma_wants_manual_pte_write_upgrade(vma):

mm/vma.h:vma_wants_manual_pte_write_upgrade()
  -> vma_wants_writenotify(vma, vma->vm_page_prot)

In vma_wants_writenotify(), because shmem has no page_mkwrite or pfn_mkwrite
callback, execution reaches:

mm/vma.c:vma_wants_writenotify() {
    ...
    /* The open routine did something to the protections that pgprot_modify
     * won't preserve? */
    if (pgprot_val(vm_page_prot) !=
        pgprot_val(vma_pgprot_modify(vm_page_prot, vma->flags)))
        return false;
    ...
}

Because vm_page_prot is already PAGE_COPY whereas
vma_pgprot_modify(vm_page_prot, vma->flags) computes PAGE_SHARED (and in
mprotect_fixup(), vma_wants_manual_pte_write_upgrade(vma) is also called
after vma_flags_reset_once() updates vma->flags but before
vma_set_page_prot(vma) updates vma->vm_page_prot), vma_wants_writenotify()
returns false. As a result, MM_CP_TRY_CHANGE_WRITABLE is not set, and
change_present_ptes() and change_huge_pmd() apply read-only newprot
(PAGE_COPY), stripping _PAGE_RW from all present writable PTEs and PMDs in
the range while change_pud_range() sets MMU_NOTIFIER_RANGE_PERM_UPGRADE.

[Severity: Low]
When CONFIG_MMU_NOTIFIER=n, does mn_flags in change_pud_range() and
hugetlb_change_protection() trigger a -Wunused-but-set-variable warning under
W=1?

In include/linux/mmu_notifier.h, when CONFIG_MMU_NOTIFIER is disabled,
mmu_notifier_range_init() is a macro that drops the flags argument:

#define mmu_notifier_range_init(range,event,flags,mm,start,end)  \
	_mmu_notifier_range_init(range, start, end)

So mn_flags is initialized and modified with |=, but never read after
preprocessing when CONFIG_MMU_NOTIFIER=n.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261029143336.390231-3-clopez@suse.de?part=2

Re: [RFC PATCH 4/4] KVM: x86/mmu: don't zap the secondary MMU on permission upgrade

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] KVM: x86/mmu: stale SPTEs in kvm_unmap_gfn_range() on hugetlb PMD unshare
- [High] mm: change_protection() revokes _PAGE_RW under MMU_NOTIFIER_RANGE_PERM_UPGRADE
- [Medium] KVM: x86/mmu: kvm_unmap_gfn_range() skips APIC page reload on perm_upgrade
--

commit eb63ff65a947838d12f405a5148038708c2ed0d7
Author: Carlos López <clopez@suse.de>

KVM: x86/mmu: don't zap the secondary MMU on permission upgrade

This commit skips zapping secondary MMU SPTEs in kvm_unmap_gfn_range() when an
mmu_notifier invalidation has range->perm_upgrade set, leaving permission
upgrades to be faulted in on demand.

> diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
> index dc2aef72ffddc..a4258a8c0c899 100644
> --- a/arch/x86/kvm/mmu/mmu.c
> +++ b/arch/x86/kvm/mmu/mmu.c
> @@ -1719,6 +1719,13 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
>  	lockdep_assert_once(kvm->mmu_invalidate_in_progress ||
>  			    lockdep_is_held(&kvm->slots_lock));
>  
> +	/*
> +	 * Do not bother zapping SPTEs if this is only a protection upgrade.
> +	 * Permissions will be faulted-in on demand.
> +	 */
> +	if (range->perm_upgrade)
> +		return false;

[Severity: Critical]
Can skipping the zap in arch/x86/kvm/mmu/mmu.c:kvm_unmap_gfn_range() leave
stale SPTEs pointing to freed 2MB hugetlbfs folios when hugetlb PMD page table
sharing is active (CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING)?

When a guest backed by a MAP_SHARED hugetlbfs VMA faults in a 2MB page,
huge_pmd_share() can map a shared PMD table into kvm->mm's PUD while KVM
installs a present 2MB leaf SPTE and drops the GUP reference in
kvm_mmu_finish_page_fault().

If host userspace later upgrades permissions via mprotect() over the VMA or a
PUD-aligned range (where hugetlb_split() does not unshare beforehand),
mprotect_fixup() sets MM_CP_PERM_UPGRADE and
mm/hugetlb.c:hugetlb_change_protection() marks the mmu_notifier range with
MMU_NOTIFIER_RANGE_PERM_UPGRADE:

mm/hugetlb.c:hugetlb_change_protection() {
	if (mm_cp_is_perm_upgrade(cp_flags))
		mn_flags |= MMU_NOTIFIER_RANGE_PERM_UPGRADE;
	...
	mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA,
				mn_flags, mm, start, end);
	adjust_range_if_pmd_sharing_possible(vma, &range.start, &range.end);
	...
		ptl = huge_pte_lock(h, mm, ptep);
		if (huge_pmd_unshare(&tlb, vma, address, ptep)) {
			...
			pages++;
			spin_unlock(ptl);
			address |= last_addr_mask;
			continue;
		}
}

With range->perm_upgrade set, kvm_unmap_gfn_range() returns false without
zapping its SPTEs. However, inside the loop,
hugetlb_change_protection() -> huge_pmd_unshare() -> __huge_pmd_unshare()
clears the PUD entry (pud_clear(pud)) in kvm->mm, decrements the shared PMD
count, and skips the rest of the 1GB PUD range without re-populating PTEs or
issuing a non-upgrade mmu_notifier invalidation.

Because the shared PMD table holds only a single rmap count across all sharing
VMAs, once the other sharing VMA unmaps the range via
__unmap_hugepage_range() -> hugetlb_remove_rmap(folio), folio_mapped(folio)
becomes false while kvm->mm is already pud_none() and KVM still holds live
SPTEs.

When that folio is subsequently migrated or truncated,
mm/migrate.c:unmap_and_move_huge_page() (or try_to_migrate() ->
mm/rmap.c:__rmap_walk_file() stopping once rwc->done / folio_not_mapped() is
true, or fs/hugetlbfs/inode.c:remove_inode_single_folio() ->
hugetlb_unmap_file_folio()) sees !folio_mapped(src) and skips mmu_notifier
invalidation on kvm->mm:

mm/migrate.c:unmap_and_move_huge_page() {
	if (folio_mapped(src)) {
		...
		try_to_migrate(src, ttu);
		was_mapped = 1;
	}

	if (!folio_mapped(src))
		rc = move_to_new_folio(dst, src, mode);
}

This leaves the guest with direct read/write access to the migrated or freed
folio, and if the guest later faults on that GFN after a new folio is mapped,
KVM attempts to replace the still-present leaf SPTE with a different PFN and
hits BUG() in arch/x86/kvm/mmu/tdp_mmu.c:__handle_changed_spte():

	if (was_leaf && is_leaf && pfn_changed) {
		...
		BUG();
	}

[Severity: High]
Is it always guaranteed that primary MMU entries retain _PAGE_RW when
range->perm_upgrade is set?

In mm/mprotect.c:change_protection(), newprot (vma->vm_page_prot) is passed to
pte_modify(), pmd_modify(), pud_modify(), and huge_pte_modify(). Because
_PAGE_RW is not in _COMMON_PAGE_CHG_MASK, those helpers overwrite _PAGE_RW from
newprot. Whenever vma->vm_page_prot has _PAGE_RW == 0 (such as MAP_PRIVATE VMAs
or MAP_SHARED VMAs where vma_wants_writenotify() was true due to VM_UFFD_WP,
soft-dirty tracking, vm_ops_needs_writenotify, or vma_fs_can_writeback),
_PAGE_RW is stripped from any already-writable entry in the range unless
explicitly restored:

1. In mm/hugetlb.c:hugetlb_change_protection() and
   mm/huge_memory.c:change_huge_pud(), MM_CP_TRY_CHANGE_WRITABLE is not
   implemented:

mm/hugetlb.c:hugetlb_change_protection() {
	...
	old_pte = huge_ptep_modify_prot_start(vma, address, ptep);
	pte = huge_pte_modify(old_pte, newprot);
	pte = arch_make_huge_pte(pte, shift, vma->vm_flags);
}

   Any already-writable hugetlb PTE or huge PUD has _PAGE_RW cleared during
   mprotect() permission upgrades (e.g. PROT_READ | PROT_WRITE to
   PROT_READ | PROT_WRITE | PROT_EXEC), UFFDIO_WRITEPROTECT(enable_wp=false)
   across a range containing already-writable hugetlb pages, or
   mm/userfaultfd.c:userfaultfd_clear_vma() on UFFDIO_UNREGISTER / close(uffd)
   (which calls change_protection() with MM_CP_UFFD_WP_RESOLVE while VM_UFFD_WP
   is still set before userfaultfd_reset_ctx() updates vma->vm_page_prot).

2. For 4KB PTEs in mm/mprotect.c:change_present_ptes() and 2MB THP PMDs in
   mm/huge_memory.c:change_huge_pmd(), restoring _PAGE_RW depends on
   MM_CP_TRY_CHANGE_WRITABLE from vma_wants_manual_pte_write_upgrade(vma),
   which calls mm/vma.c:vma_wants_writenotify(vma, vma->vm_page_prot):

mm/vma.c:vma_wants_writenotify() {
	if (pgprot_val(vm_page_prot) !=
	    pgprot_val(vma_pgprot_modify(vm_page_prot, vma->flags)))
		return false;
}

   On a MAP_SHARED VMA where vm_ops_needs_writenotify(vma->vm_ops) is false
   (such as shmem/memfd) and vma->vm_page_prot was set to read-only due to
   VM_UFFD_WP or soft-dirty tracking, or in mm/mprotect.c:mprotect_fixup()
   where vma_flags_reset_once() updates vma->flags before calling
   vma_wants_manual_pte_write_upgrade(vma) and vma_set_page_prot(vma):

mm/mprotect.c:mprotect_fixup() {
	vma_flags_reset_once(vma, &new_vma_flags);
	if (vma_wants_manual_pte_write_upgrade(vma))
		mm_cp_flags |= MM_CP_TRY_CHANGE_WRITABLE;
	if (vma_flags_is_perm_upgrade(old_vma_flags, new_vma_flags))
		mm_cp_flags |= MM_CP_PERM_UPGRADE;
	vma_set_page_prot(vma);
}

   vma_wants_writenotify() returns false and MM_CP_TRY_CHANGE_WRITABLE is not
   set, so change_present_ptes() and change_huge_pmd() strip _PAGE_RW from
   already-writable PTEs and PMDs. Even when MM_CP_TRY_CHANGE_WRITABLE is set,
   can_change_shared_pte_writable() and can_change_pmd_writable() only restore
   _PAGE_RW if the entry is dirty, downgrading writable-but-clean shared
   PTEs/PMDs to read-only.

Because kvm_unmap_gfn_range() skips zapping on range->perm_upgrade, KVM keeps
writable leaf SPTEs (PT_WRITABLE_MASK, shadow_host_writable_mask, and
shadow_mmu_writable_mask) while the host PTE is now read-only, bypassing host
write-fault tracking. If a subsequent non-write KVM fault (such as an
access-tracking fault or shadow MMU exec fault) resolves the read-only host PTE
with fault->map_writable = false, make_spte() clears shadow_host_writable_mask
and shadow_mmu_writable_mask on the present leaf SPTE and triggers the
WARN_ON_ONCE in arch/x86/kvm/mmu/tdp_mmu.c:tdp_mmu_map_handle_target_level():

	else if (is_shadow_present_pte(iter->old_spte) &&
		 (!is_last_spte(iter->old_spte, iter->level) ||
		  WARN_ON_ONCE(leaf_spte_change_needs_tlb_flush(iter->old_spte, new_spte))))
		kvm_flush_remote_tlbs_gfn(vcpu->kvm, iter->gfn, iter->level);

[Severity: Medium]
What happens to the APIC access page reload in
arch/x86/kvm/mmu/mmu.c:kvm_unmap_gfn_range() when range->perm_upgrade is true?

Further down in kvm_unmap_gfn_range(), invalidations on
APIC_ACCESS_PAGE_PRIVATE_MEMSLOT issue KVM_REQ_APIC_PAGE_RELOAD:

	if (kvm_x86_ops.set_apic_access_page_addr &&
	    range->slot->id == APIC_ACCESS_PAGE_PRIVATE_MEMSLOT)
		kvm_make_all_cpus_request(kvm, KVM_REQ_APIC_PAGE_RELOAD);

When arch/x86/kvm/vmx/vmx.c:vmx_set_apic_access_page_addr() handles
KVM_REQ_APIC_PAGE_RELOAD, it resolves the PFN with FOLL_WRITE:

	pfn = __kvm_faultin_pfn(slot, gfn, FOLL_WRITE, &writable, &refcounted_page);
	if (is_error_noslot_pfn(pfn))
		return;

	read_lock(&vcpu->kvm->mmu_lock);
	if (mmu_invalidate_retry_gfn(kvm, mmu_seq, gfn))
		kvm_make_request(KVM_REQ_APIC_PAGE_RELOAD, vcpu);
	else
		vmcs_write64(APIC_ACCESS_ADDR, pfn_to_hpa(pfn));

If the VMA backing APIC_ACCESS_PAGE_PRIVATE_MEMSLOT is made read-only (e.g. via
mprotect(..., PROT_READ)) before the page is faulted in or while the backing
page is migrated, swapped out, or unmapped, __kvm_faultin_pfn() fails after
kvm_check_request() has already cleared KVM_REQ_APIC_PAGE_RELOAD, leaving
APIC_ACCESS_ADDR uninitialized or holding a stale HPA.

When write permission on that VMA is later restored via
mprotect(..., PROT_READ | PROT_WRITE), range->perm_upgrade is true and the
early return in kvm_unmap_gfn_range() skips requesting
KVM_REQ_APIC_PAGE_RELOAD. Since APIC_ACCESS_ADDR in the VMCS is not updated on
demand during kvm_mmu_do_page_fault(), wouldn't this leave APIC_ACCESS_ADDR
stale?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261029143336.390231-3-clopez@suse.de?part=4