[PATCH v3 0/2] mm/mprotect: micro-optimization work

[PATCH v3 0/2] mm/mprotect: micro-optimization work

[Pedro Falcato]

Micro-optimize the change_protection functionality and the
change_pte_range() routine. This set of functions works in an incredibly
tight loop, and even small inefficiencies are incredibly evident when spun
hundreds, thousands or hundreds of thousands of times.

There was an attempt to keep the batching functionality as much as possible,
which introduced some part of the slowness, but not all of it. Removing it
for !arm64 architectures would speed mprotect() up even further, but could
easily pessimize cases where large folios are mapped (which is not as rare
as it seems, particularly when it comes to the page cache these days).

The micro-benchmark used for the tests was [0] (usable using google/benchmark
and g++ -O2 -lbenchmark repro.cpp)

This resulted in the following (first entry is baseline):

---------------------------------------------------------
Benchmark               Time             CPU   Iterations
---------------------------------------------------------
mprotect_bench      85967 ns        85967 ns         6935
mprotect_bench      70684 ns        70684 ns         9887


After the patchset we can observe an ~18% speedup in mprotect. Wonderful
for the elusive mprotect-based workloads!

Testing & more ideas welcome. I suspect there is plenty of improvement possible
but it would require more time than what I have on my hands right now. The
entire inlined function (which inlines into change_protection()) is gigantic
- I'm not surprised this is so finnicky.

Note: per my profiling, the next _big_ bottleneck here is modify_prot_start_ptes,
exactly on the xchg() done by x86. ptep_get_and_clear() is _expensive_. I don't think
there's a properly safe way to go about it since we do depend on the D bit
quite a lot. This might not be such an issue on other architectures.

Luke Yang reported [1]:

: On average, we see improvements ranging from a minimum of 5% to a
: maximum of 55%, with most improvements showing around a 25% speed up in
: the libmicro/mprot_tw4m micro benchmark.


Link: https://lore.kernel.org/all/aY8-XuFZ7zCvXulB@luyang-thinkpadp1gen7.toromso.csb/
Link: https://gist.github.com/heatd/1450d273005aba91fa5744f44dfcd933 [0]
Link: https://lkml.kernel.org/r/CAL2CeBxT4jtJ+LxYb6=BNxNMGinpgD_HYH5gGxOP-45Q2OncqQ@mail.gmail.com [1]

Cc: Vlastimil Babka <vbabka@kernel.org> 
Cc: Jann Horn <jannh@google.com> 
Cc: David Hildenbrand <david@kernel.org>
Cc: Dev Jain <dev.jain@arm.com> 
Cc: Luke Yang <luyang@redhat.com>
Cc: jhladky@redhat.com
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org

v3:
 - Collapse a few lines into a single line in patch 1 (David)
 - Bring the inlining to a higher level (David)
 - Pick up David's patch 1 ACK (thank you!)
 - Pick up Luke Yang's Tested-by (thank you!)
 - Add Luke's results and akpmify the Links: a bit (cover letter)
v2:
 - Addressed Sashiko's concerns
 - Picked up Lorenzo's R-b's (thank you!)
 - Squashed patch 1 and 4 into a single one (David)
 - Renamed the softleaf leaf function (David)
 - Dropped controversial noinlines & patch 3 (Lorenzo & David)

v1:
https://lore.kernel.org/linux-mm/20260319183108.1105090-1-pfalcato@suse.de/

Pedro Falcato (2):
  mm/mprotect: move softleaf code out of the main function
  mm/mprotect: special-case small folios when applying write permissions

 mm/mprotect.c | 218 ++++++++++++++++++++++++++++----------------------
 1 file changed, 124 insertions(+), 94 deletions(-)

-- 
2.53.0

[PATCH v3 1/2] mm/mprotect: move softleaf code out of the main function

[Pedro Falcato]

Move softleaf change_pte_range code into a separate function. This makes
the change_pte_range() function a good bit smaller, and lessens cognitive
load when reading through the function.

Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Tested-by: Luke Yang <luyang@redhat.com>
Signed-off-by: Pedro Falcato <pfalcato@suse.de>
---
 mm/mprotect.c | 127 ++++++++++++++++++++++++++------------------------
 1 file changed, 67 insertions(+), 60 deletions(-)

diff --git a/mm/mprotect.c b/mm/mprotect.c
index 9681f055b9fc..5929ce792c7b 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -211,6 +211,72 @@ static void set_write_prot_commit_flush_ptes(struct vm_area_struct *vma,
 	commit_anon_folio_batch(vma, folio, page, addr, ptep, oldpte, ptent, nr_ptes, tlb);
 }
 
+static long change_softleaf_pte(struct vm_area_struct *vma,
+	unsigned long addr, pte_t *pte, pte_t oldpte, unsigned long cp_flags)
+{
+	const bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
+	const bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
+	softleaf_t entry = softleaf_from_pte(oldpte);
+	pte_t newpte;
+
+	if (softleaf_is_migration_write(entry)) {
+		const struct folio *folio = softleaf_to_folio(entry);
+
+		/*
+		 * A protection check is difficult so
+		 * just be safe and disable write
+		 */
+		if (folio_test_anon(folio))
+			entry = make_readable_exclusive_migration_entry(swp_offset(entry));
+		else
+			entry = make_readable_migration_entry(swp_offset(entry));
+		newpte = swp_entry_to_pte(entry);
+		if (pte_swp_soft_dirty(oldpte))
+			newpte = pte_swp_mksoft_dirty(newpte);
+	} else if (softleaf_is_device_private_write(entry)) {
+		/*
+		 * We do not preserve soft-dirtiness. See
+		 * copy_nonpresent_pte() for explanation.
+		 */
+		entry = make_readable_device_private_entry(swp_offset(entry));
+		newpte = swp_entry_to_pte(entry);
+		if (pte_swp_uffd_wp(oldpte))
+			newpte = pte_swp_mkuffd_wp(newpte);
+	} else if (softleaf_is_marker(entry)) {
+		/*
+		 * Ignore error swap entries unconditionally,
+		 * because any access should sigbus/sigsegv
+		 * anyway.
+		 */
+		if (softleaf_is_poison_marker(entry) ||
+		    softleaf_is_guard_marker(entry))
+			return 0;
+		/*
+		 * If this is uffd-wp pte marker and we'd like
+		 * to unprotect it, drop it; the next page
+		 * fault will trigger without uffd trapping.
+		 */
+		if (uffd_wp_resolve) {
+			pte_clear(vma->vm_mm, addr, pte);
+			return 1;
+		}
+		return 0;
+	} else {
+		newpte = oldpte;
+	}
+
+	if (uffd_wp)
+		newpte = pte_swp_mkuffd_wp(newpte);
+	else if (uffd_wp_resolve)
+		newpte = pte_swp_clear_uffd_wp(newpte);
+
+	if (!pte_same(oldpte, newpte)) {
+		set_pte_at(vma->vm_mm, addr, pte, newpte);
+		return 1;
+	}
+	return 0;
+}
+
 static long change_pte_range(struct mmu_gather *tlb,
 		struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr,
 		unsigned long end, pgprot_t newprot, unsigned long cp_flags)
@@ -317,66 +383,7 @@ static long change_pte_range(struct mmu_gather *tlb,
 				pages++;
 			}
 		} else  {
-			softleaf_t entry = softleaf_from_pte(oldpte);
-			pte_t newpte;
-
-			if (softleaf_is_migration_write(entry)) {
-				const struct folio *folio = softleaf_to_folio(entry);
-
-				/*
-				 * A protection check is difficult so
-				 * just be safe and disable write
-				 */
-				if (folio_test_anon(folio))
-					entry = make_readable_exclusive_migration_entry(
-							     swp_offset(entry));
-				else
-					entry = make_readable_migration_entry(swp_offset(entry));
-				newpte = swp_entry_to_pte(entry);
-				if (pte_swp_soft_dirty(oldpte))
-					newpte = pte_swp_mksoft_dirty(newpte);
-			} else if (softleaf_is_device_private_write(entry)) {
-				/*
-				 * We do not preserve soft-dirtiness. See
-				 * copy_nonpresent_pte() for explanation.
-				 */
-				entry = make_readable_device_private_entry(
-							swp_offset(entry));
-				newpte = swp_entry_to_pte(entry);
-				if (pte_swp_uffd_wp(oldpte))
-					newpte = pte_swp_mkuffd_wp(newpte);
-			} else if (softleaf_is_marker(entry)) {
-				/*
-				 * Ignore error swap entries unconditionally,
-				 * because any access should sigbus/sigsegv
-				 * anyway.
-				 */
-				if (softleaf_is_poison_marker(entry) ||
-				    softleaf_is_guard_marker(entry))
-					continue;
-				/*
-				 * If this is uffd-wp pte marker and we'd like
-				 * to unprotect it, drop it; the next page
-				 * fault will trigger without uffd trapping.
-				 */
-				if (uffd_wp_resolve) {
-					pte_clear(vma->vm_mm, addr, pte);
-					pages++;
-				}
-				continue;
-			} else {
-				newpte = oldpte;
-			}
-
-			if (uffd_wp)
-				newpte = pte_swp_mkuffd_wp(newpte);
-			else if (uffd_wp_resolve)
-				newpte = pte_swp_clear_uffd_wp(newpte);
-
-			if (!pte_same(oldpte, newpte)) {
-				set_pte_at(vma->vm_mm, addr, pte, newpte);
-				pages++;
-			}
+			pages += change_softleaf_pte(vma, addr, pte, oldpte, cp_flags);
 		}
 	} while (pte += nr_ptes, addr += nr_ptes * PAGE_SIZE, addr != end);
 	lazy_mmu_mode_disable();
-- 
2.53.0

[PATCH v3 2/2] mm/mprotect: special-case small folios when applying write permissions

[Pedro Falcato]

The common order-0 case is important enough to want its own branch, and
avoids the hairy, large loop logic that the CPU does not seem to handle
particularly well.

While at it, encourage the compiler to inline batch PTE logic and resolve
constant branches by adding __always_inline strategically.

Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
Tested-by: Luke Yang <luyang@redhat.com>
Signed-off-by: Pedro Falcato <pfalcato@suse.de>
---
 mm/mprotect.c | 91 ++++++++++++++++++++++++++++++++-------------------
 1 file changed, 57 insertions(+), 34 deletions(-)

diff --git a/mm/mprotect.c b/mm/mprotect.c
index 5929ce792c7b..98da856e3a52 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -117,9 +117,9 @@ static int mprotect_folio_pte_batch(struct folio *folio, pte_t *ptep,
 }
 
 /* Set nr_ptes number of ptes, starting from idx */
-static void prot_commit_flush_ptes(struct vm_area_struct *vma, unsigned long addr,
-		pte_t *ptep, pte_t oldpte, pte_t ptent, int nr_ptes,
-		int idx, bool set_write, struct mmu_gather *tlb)
+static __always_inline void prot_commit_flush_ptes(struct vm_area_struct *vma,
+		unsigned long addr, pte_t *ptep, pte_t oldpte, pte_t ptent,
+		int nr_ptes, int idx, bool set_write, struct mmu_gather *tlb)
 {
 	/*
 	 * Advance the position in the batch by idx; note that if idx > 0,
@@ -143,7 +143,7 @@ static void prot_commit_flush_ptes(struct vm_area_struct *vma, unsigned long add
  * !PageAnonExclusive() pages, starting from start_idx. Caller must enforce
  * that the ptes point to consecutive pages of the same anon large folio.
  */
-static int page_anon_exclusive_sub_batch(int start_idx, int max_len,
+static __always_inline int page_anon_exclusive_sub_batch(int start_idx, int max_len,
 		struct page *first_page, bool expected_anon_exclusive)
 {
 	int idx;
@@ -169,7 +169,7 @@ static int page_anon_exclusive_sub_batch(int start_idx, int max_len,
  * pte of the batch. Therefore, we must individually check all pages and
  * retrieve sub-batches.
  */
-static void commit_anon_folio_batch(struct vm_area_struct *vma,
+static __always_inline void commit_anon_folio_batch(struct vm_area_struct *vma,
 		struct folio *folio, struct page *first_page, unsigned long addr, pte_t *ptep,
 		pte_t oldpte, pte_t ptent, int nr_ptes, struct mmu_gather *tlb)
 {
@@ -188,7 +188,7 @@ static void commit_anon_folio_batch(struct vm_area_struct *vma,
 	}
 }
 
-static void set_write_prot_commit_flush_ptes(struct vm_area_struct *vma,
+static __always_inline void set_write_prot_commit_flush_ptes(struct vm_area_struct *vma,
 		struct folio *folio, struct page *page, unsigned long addr, pte_t *ptep,
 		pte_t oldpte, pte_t ptent, int nr_ptes, struct mmu_gather *tlb)
 {
@@ -277,6 +277,45 @@ static long change_softleaf_pte(struct vm_area_struct *vma,
 	return 0;
 }
 
+static __always_inline void change_present_ptes(struct mmu_gather *tlb,
+		struct vm_area_struct *vma, unsigned long addr, pte_t *ptep,
+		int nr_ptes, unsigned long end, pgprot_t newprot,
+		struct folio *folio, struct page *page, unsigned long cp_flags)
+{
+	const bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
+	const bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
+	pte_t ptent, oldpte;
+
+	oldpte = modify_prot_start_ptes(vma, addr, ptep, nr_ptes);
+	ptent = pte_modify(oldpte, newprot);
+
+	if (uffd_wp)
+		ptent = pte_mkuffd_wp(ptent);
+	else if (uffd_wp_resolve)
+		ptent = pte_clear_uffd_wp(ptent);
+
+	/*
+	 * In some writable, shared mappings, we might want
+	 * to catch actual write access -- see
+	 * vma_wants_writenotify().
+	 *
+	 * In all writable, private mappings, we have to
+	 * properly handle COW.
+	 *
+	 * In both cases, we can sometimes still change PTEs
+	 * writable and avoid the write-fault handler, for
+	 * example, if a PTE is already dirty and no other
+	 * COW or special handling is required.
+	 */
+	if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) &&
+	     !pte_write(ptent))
+		set_write_prot_commit_flush_ptes(vma, folio, page,
+			addr, ptep, oldpte, ptent, nr_ptes, tlb);
+	else
+		prot_commit_flush_ptes(vma, addr, ptep, oldpte, ptent,
+			nr_ptes, /* idx = */ 0, /* set_write = */ false, tlb);
+}
+
 static long change_pte_range(struct mmu_gather *tlb,
 		struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr,
 		unsigned long end, pgprot_t newprot, unsigned long cp_flags)
@@ -287,7 +326,6 @@ static long change_pte_range(struct mmu_gather *tlb,
 	bool is_private_single_threaded;
 	bool prot_numa = cp_flags & MM_CP_PROT_NUMA;
 	bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
-	bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
 	int nr_ptes;
 
 	tlb_change_page_size(tlb, PAGE_SIZE);
@@ -308,7 +346,6 @@ static long change_pte_range(struct mmu_gather *tlb,
 			int max_nr_ptes = (end - addr) >> PAGE_SHIFT;
 			struct folio *folio = NULL;
 			struct page *page;
-			pte_t ptent;
 
 			/* Already in the desired state. */
 			if (prot_numa && pte_protnone(oldpte))
@@ -334,34 +371,20 @@ static long change_pte_range(struct mmu_gather *tlb,
 
 			nr_ptes = mprotect_folio_pte_batch(folio, pte, oldpte, max_nr_ptes, flags);
 
-			oldpte = modify_prot_start_ptes(vma, addr, pte, nr_ptes);
-			ptent = pte_modify(oldpte, newprot);
-
-			if (uffd_wp)
-				ptent = pte_mkuffd_wp(ptent);
-			else if (uffd_wp_resolve)
-				ptent = pte_clear_uffd_wp(ptent);
-
 			/*
-			 * In some writable, shared mappings, we might want
-			 * to catch actual write access -- see
-			 * vma_wants_writenotify().
-			 *
-			 * In all writable, private mappings, we have to
-			 * properly handle COW.
-			 *
-			 * In both cases, we can sometimes still change PTEs
-			 * writable and avoid the write-fault handler, for
-			 * example, if a PTE is already dirty and no other
-			 * COW or special handling is required.
+			 * Optimize for the small-folio common case by
+			 * special-casing it here. Compiler constant propagation
+			 * plus copious amounts of __always_inline does wonders.
 			 */
-			if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) &&
-			     !pte_write(ptent))
-				set_write_prot_commit_flush_ptes(vma, folio, page,
-				addr, pte, oldpte, ptent, nr_ptes, tlb);
-			else
-				prot_commit_flush_ptes(vma, addr, pte, oldpte, ptent,
-					nr_ptes, /* idx = */ 0, /* set_write = */ false, tlb);
+			if (likely(nr_ptes == 1)) {
+				change_present_ptes(tlb, vma, addr, pte, 1,
+					end, newprot, folio, page, cp_flags);
+			} else {
+				change_present_ptes(tlb, vma, addr, pte,
+					nr_ptes, end, newprot, folio, page,
+					cp_flags);
+			}
+
 			pages += nr_ptes;
 		} else if (pte_none(oldpte)) {
 			/*
-- 
2.53.0

Re: [PATCH v3 2/2] mm/mprotect: special-case small folios when applying write permissions

[Pedro Falcato]

Ugh, of course right after sending I notice a small problem.
Andrew, can you patch up the commit title to something like:
"mm/mprotect: special-case small folios when applying permissions"
(or protections, whatever sounds best)

After David's suggestions, this doesn't apply to write permissions
exclusively. The rest of the commit message still applies, I think.

-- 
Pedro

Re: [PATCH v3 2/2] mm/mprotect: special-case small folios when applying write permissions

[Andrew Morton]

On Thu, 2 Apr 2026 15:21:26 +0100 Pedro Falcato <pfalcato@suse.de> wrote:

> Ugh, of course right after sending I notice a small problem.
> Andrew, can you patch up the commit title to something like:
> "mm/mprotect: special-case small folios when applying permissions"
> (or protections, whatever sounds best)

np.  After scratching my head over the difference and not being able to
discern any, I went with "mm/mprotect: special-case small folios when
applying permissions"

Re: [PATCH v3 0/2] mm/mprotect: micro-optimization work

[Andrew Morton]

On Thu,  2 Apr 2026 15:16:26 +0100 Pedro Falcato <pfalcato@suse.de> wrote:

> Micro-optimize the change_protection functionality and the
> change_pte_range() routine. This set of functions works in an incredibly
> tight loop, and even small inefficiencies are incredibly evident when spun
> hundreds, thousands or hundreds of thousands of times.

Thanks, I updated mm.git's mm-unstable branch to this version.

The update is rather large.  If people think it best to spill this work
into next -rc1 then please advise.

> 
> v3:
>  - Collapse a few lines into a single line in patch 1 (David)
>  - Bring the inlining to a higher level (David)
>  - Pick up David's patch 1 ACK (thank you!)
>  - Pick up Luke Yang's Tested-by (thank you!)
>  - Add Luke's results and akpmify the Links: a bit (cover letter)

Here's how v3 altered mm.git:

 mm/mprotect.c |   98 +++++++++++++++++++++++++++---------------------
 1 file changed, 56 insertions(+), 42 deletions(-)

--- a/mm/mprotect.c~b
+++ a/mm/mprotect.c
@@ -103,7 +103,7 @@ bool can_change_pte_writable(struct vm_a
 	return can_change_shared_pte_writable(vma, pte);
 }
 
-static __always_inline int mprotect_folio_pte_batch(struct folio *folio, pte_t *ptep,
+static int mprotect_folio_pte_batch(struct folio *folio, pte_t *ptep,
 				    pte_t pte, int max_nr_ptes, fpb_t flags)
 {
 	/* No underlying folio, so cannot batch */
@@ -143,7 +143,7 @@ static __always_inline void prot_commit_
  * !PageAnonExclusive() pages, starting from start_idx. Caller must enforce
  * that the ptes point to consecutive pages of the same anon large folio.
  */
-static int page_anon_exclusive_sub_batch(int start_idx, int max_len,
+static __always_inline int page_anon_exclusive_sub_batch(int start_idx, int max_len,
 		struct page *first_page, bool expected_anon_exclusive)
 {
 	int idx;
@@ -177,13 +177,6 @@ static __always_inline void commit_anon_
 	int sub_batch_idx = 0;
 	int len;
 
-	/* Optimize for the common order-0 case. */
-	if (likely(nr_ptes == 1)) {
-		prot_commit_flush_ptes(vma, addr, ptep, oldpte, ptent, 1,
-				       0, PageAnonExclusive(first_page), tlb);
-		return;
-	}
-
 	while (nr_ptes) {
 		expected_anon_exclusive = PageAnonExclusive(first_page + sub_batch_idx);
 		len = page_anon_exclusive_sub_batch(sub_batch_idx, nr_ptes,
@@ -195,7 +188,7 @@ static __always_inline void commit_anon_
 	}
 }
 
-static void set_write_prot_commit_flush_ptes(struct vm_area_struct *vma,
+static __always_inline void set_write_prot_commit_flush_ptes(struct vm_area_struct *vma,
 		struct folio *folio, struct page *page, unsigned long addr, pte_t *ptep,
 		pte_t oldpte, pte_t ptent, int nr_ptes, struct mmu_gather *tlb)
 {
@@ -234,8 +227,7 @@ static long change_softleaf_pte(struct v
 		 * just be safe and disable write
 		 */
 		if (folio_test_anon(folio))
-			entry = make_readable_exclusive_migration_entry(
-					     swp_offset(entry));
+			entry = make_readable_exclusive_migration_entry(swp_offset(entry));
 		else
 			entry = make_readable_migration_entry(swp_offset(entry));
 		newpte = swp_entry_to_pte(entry);
@@ -246,8 +238,7 @@ static long change_softleaf_pte(struct v
 		 * We do not preserve soft-dirtiness. See
 		 * copy_nonpresent_pte() for explanation.
 		 */
-		entry = make_readable_device_private_entry(
-					swp_offset(entry));
+		entry = make_readable_device_private_entry(swp_offset(entry));
 		newpte = swp_entry_to_pte(entry);
 		if (pte_swp_uffd_wp(oldpte))
 			newpte = pte_swp_mkuffd_wp(newpte);
@@ -286,6 +277,45 @@ static long change_softleaf_pte(struct v
 	return 0;
 }
 
+static __always_inline void change_present_ptes(struct mmu_gather *tlb,
+		struct vm_area_struct *vma, unsigned long addr, pte_t *ptep,
+		int nr_ptes, unsigned long end, pgprot_t newprot,
+		struct folio *folio, struct page *page, unsigned long cp_flags)
+{
+	const bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
+	const bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
+	pte_t ptent, oldpte;
+
+	oldpte = modify_prot_start_ptes(vma, addr, ptep, nr_ptes);
+	ptent = pte_modify(oldpte, newprot);
+
+	if (uffd_wp)
+		ptent = pte_mkuffd_wp(ptent);
+	else if (uffd_wp_resolve)
+		ptent = pte_clear_uffd_wp(ptent);
+
+	/*
+	 * In some writable, shared mappings, we might want
+	 * to catch actual write access -- see
+	 * vma_wants_writenotify().
+	 *
+	 * In all writable, private mappings, we have to
+	 * properly handle COW.
+	 *
+	 * In both cases, we can sometimes still change PTEs
+	 * writable and avoid the write-fault handler, for
+	 * example, if a PTE is already dirty and no other
+	 * COW or special handling is required.
+	 */
+	if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) &&
+	     !pte_write(ptent))
+		set_write_prot_commit_flush_ptes(vma, folio, page,
+			addr, ptep, oldpte, ptent, nr_ptes, tlb);
+	else
+		prot_commit_flush_ptes(vma, addr, ptep, oldpte, ptent,
+			nr_ptes, /* idx = */ 0, /* set_write = */ false, tlb);
+}
+
 static long change_pte_range(struct mmu_gather *tlb,
 		struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr,
 		unsigned long end, pgprot_t newprot, unsigned long cp_flags)
@@ -296,7 +326,6 @@ static long change_pte_range(struct mmu_
 	bool is_private_single_threaded;
 	bool prot_numa = cp_flags & MM_CP_PROT_NUMA;
 	bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
-	bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
 	int nr_ptes;
 
 	tlb_change_page_size(tlb, PAGE_SIZE);
@@ -317,7 +346,6 @@ static long change_pte_range(struct mmu_
 			int max_nr_ptes = (end - addr) >> PAGE_SHIFT;
 			struct folio *folio = NULL;
 			struct page *page;
-			pte_t ptent;
 
 			/* Already in the desired state. */
 			if (prot_numa && pte_protnone(oldpte))
@@ -343,34 +371,20 @@ static long change_pte_range(struct mmu_
 
 			nr_ptes = mprotect_folio_pte_batch(folio, pte, oldpte, max_nr_ptes, flags);
 
-			oldpte = modify_prot_start_ptes(vma, addr, pte, nr_ptes);
-			ptent = pte_modify(oldpte, newprot);
-
-			if (uffd_wp)
-				ptent = pte_mkuffd_wp(ptent);
-			else if (uffd_wp_resolve)
-				ptent = pte_clear_uffd_wp(ptent);
-
 			/*
-			 * In some writable, shared mappings, we might want
-			 * to catch actual write access -- see
-			 * vma_wants_writenotify().
-			 *
-			 * In all writable, private mappings, we have to
-			 * properly handle COW.
-			 *
-			 * In both cases, we can sometimes still change PTEs
-			 * writable and avoid the write-fault handler, for
-			 * example, if a PTE is already dirty and no other
-			 * COW or special handling is required.
+			 * Optimize for the small-folio common case by
+			 * special-casing it here. Compiler constant propagation
+			 * plus copious amounts of __always_inline does wonders.
 			 */
-			if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) &&
-			     !pte_write(ptent))
-				set_write_prot_commit_flush_ptes(vma, folio, page,
-				addr, pte, oldpte, ptent, nr_ptes, tlb);
-			else
-				prot_commit_flush_ptes(vma, addr, pte, oldpte, ptent,
-					nr_ptes, /* idx = */ 0, /* set_write = */ false, tlb);
+			if (likely(nr_ptes == 1)) {
+				change_present_ptes(tlb, vma, addr, pte, 1,
+					end, newprot, folio, page, cp_flags);
+			} else {
+				change_present_ptes(tlb, vma, addr, pte,
+					nr_ptes, end, newprot, folio, page,
+					cp_flags);
+			}
+
 			pages += nr_ptes;
 		} else if (pte_none(oldpte)) {
 			/*
_

Re: [PATCH v3 2/2] mm/mprotect: special-case small folios when applying write permissions

[Pedro Falcato]

On Thu, Apr 02, 2026 at 11:29:59AM -0700, Andrew Morton wrote:
> On Thu, 2 Apr 2026 15:21:26 +0100 Pedro Falcato <pfalcato@suse.de> wrote:
> 
> > Ugh, of course right after sending I notice a small problem.
> > Andrew, can you patch up the commit title to something like:
> > "mm/mprotect: special-case small folios when applying permissions"
> > (or protections, whatever sounds best)
> 
> np.  After scratching my head over the difference and not being able to
> discern any, I went with "mm/mprotect: special-case small folios when
> applying permissions"

SGTM, thanks!

-- 
Pedro

Re: [PATCH v3 0/2] mm/mprotect: micro-optimization work

[Pedro Falcato]

On Thu, Apr 02, 2026 at 11:33:28AM -0700, Andrew Morton wrote:
> On Thu,  2 Apr 2026 15:16:26 +0100 Pedro Falcato <pfalcato@suse.de> wrote:
> 
> > Micro-optimize the change_protection functionality and the
> > change_pte_range() routine. This set of functions works in an incredibly
> > tight loop, and even small inefficiencies are incredibly evident when spun
> > hundreds, thousands or hundreds of thousands of times.
> 
> Thanks, I updated mm.git's mm-unstable branch to this version.
> 
> The update is rather large.  If people think it best to spill this work
> into next -rc1 then please advise.

I think it's a fairly safe change for 7.1, but I'm biased :) Perhaps others think
otherwise.

-- 
Pedro

Re: [PATCH v3 0/2] mm/mprotect: micro-optimization work

[David Hildenbrand (arm)]

On 4/6/26 10:29, Pedro Falcato wrote:
> On Thu, Apr 02, 2026 at 11:33:28AM -0700, Andrew Morton wrote:
>> On Thu,  2 Apr 2026 15:16:26 +0100 Pedro Falcato <pfalcato@suse.de> wrote:
>>
>>> Micro-optimize the change_protection functionality and the
>>> change_pte_range() routine. This set of functions works in an incredibly
>>> tight loop, and even small inefficiencies are incredibly evident when spun
>>> hundreds, thousands or hundreds of thousands of times.
>>
>> Thanks, I updated mm.git's mm-unstable branch to this version.
>>
>> The update is rather large.  If people think it best to spill this work
>> into next -rc1 then please advise.
> 
> I think it's a fairly safe change for 7.1, but I'm biased :) Perhaps others think
> otherwise.

It's mostly code movement, so it should be fine (unless we messed up 
passed parameters somehow :) )

-- 
Cheers,

David

Re: [PATCH v3 0/2] mm/mprotect: micro-optimization work

[Andrew Morton]

On Mon, 6 Apr 2026 11:09:01 +0200 "David Hildenbrand (arm)" <david@kernel.org> wrote:

> On 4/6/26 10:29, Pedro Falcato wrote:
> > On Thu, Apr 02, 2026 at 11:33:28AM -0700, Andrew Morton wrote:
> >> On Thu,  2 Apr 2026 15:16:26 +0100 Pedro Falcato <pfalcato@suse.de> wrote:
> >>
> >>> Micro-optimize the change_protection functionality and the
> >>> change_pte_range() routine. This set of functions works in an incredibly
> >>> tight loop, and even small inefficiencies are incredibly evident when spun
> >>> hundreds, thousands or hundreds of thousands of times.
> >>
> >> Thanks, I updated mm.git's mm-unstable branch to this version.
> >>
> >> The update is rather large.  If people think it best to spill this work
> >> into next -rc1 then please advise.
> > 
> > I think it's a fairly safe change for 7.1, but I'm biased :) Perhaps others think
> > otherwise.

Thanks, I've firmly added these two to the second-week-of-merge-window
pile.

> It's mostly code movement, so it should be fine (unless we messed up 
> passed parameters somehow :) )

This (or an earlier version thereof) have been in mm.git since March 24.

Re: [PATCH v3 2/2] mm/mprotect: special-case small folios when applying write permissions

[Davidlohr Bueso]

On Thu, 02 Apr 2026, Pedro Falcato wrote:

>@@ -334,34 +371,20 @@ static long change_pte_range(struct mmu_gather *tlb,
>
> 			nr_ptes = mprotect_folio_pte_batch(folio, pte, oldpte, max_nr_ptes, flags);
>
>-			oldpte = modify_prot_start_ptes(vma, addr, pte, nr_ptes);
>-			ptent = pte_modify(oldpte, newprot);
>-
>-			if (uffd_wp)
>-				ptent = pte_mkuffd_wp(ptent);
>-			else if (uffd_wp_resolve)
>-				ptent = pte_clear_uffd_wp(ptent);
>-
> 			/*
>-			 * In some writable, shared mappings, we might want
>-			 * to catch actual write access -- see
>-			 * vma_wants_writenotify().
>-			 *
>-			 * In all writable, private mappings, we have to
>-			 * properly handle COW.
>-			 *
>-			 * In both cases, we can sometimes still change PTEs
>-			 * writable and avoid the write-fault handler, for
>-			 * example, if a PTE is already dirty and no other
>-			 * COW or special handling is required.
>+			 * Optimize for the small-folio common case by
>+			 * special-casing it here. Compiler constant propagation
>+			 * plus copious amounts of __always_inline does wonders.
> 			 */
>-			if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) &&
>-			     !pte_write(ptent))
>-				set_write_prot_commit_flush_ptes(vma, folio, page,
>-				addr, pte, oldpte, ptent, nr_ptes, tlb);
>-			else
>-				prot_commit_flush_ptes(vma, addr, pte, oldpte, ptent,
>-					nr_ptes, /* idx = */ 0, /* set_write = */ false, tlb);
>+			if (likely(nr_ptes == 1)) {

Are there any numbers for this optimization? While I am all for optimizing the common
case, it seems unfair to penalize the uncommon one here. Why is nr_ptes > 1 such an
exotic use case (specially today)? ie: How does this change affect the program in
b9bf6c2872c ("mm: refactor MM_CP_PROT_NUMA skipping case into new function"),
which is a series for optimizing large folio cases.

Thanks,
Davidlohr

>+				change_present_ptes(tlb, vma, addr, pte, 1,
>+					end, newprot, folio, page, cp_flags);
>+			} else {
>+				change_present_ptes(tlb, vma, addr, pte,
>+					nr_ptes, end, newprot, folio, page,
>+					cp_flags);
>+			}
>+
> 			pages += nr_ptes;
> 		} else if (pte_none(oldpte)) {
> 			/*
>-- 
>2.53.0
>
>

Re: [PATCH v3 2/2] mm/mprotect: special-case small folios when applying write permissions

[Pedro Falcato]

(note: had to manually adjust the To: and Cc:, it seems my neomutt doesn't
like something about your email)

On Mon, Apr 06, 2026 at 05:50:26PM -0700, Davidlohr Bueso wrote:
> On Thu, 02 Apr 2026, Pedro Falcato wrote:
> 
> > @@ -334,34 +371,20 @@ static long change_pte_range(struct mmu_gather *tlb,
> > 
> > 			nr_ptes = mprotect_folio_pte_batch(folio, pte, oldpte, max_nr_ptes, flags);
> > 
> > -			oldpte = modify_prot_start_ptes(vma, addr, pte, nr_ptes);
> > -			ptent = pte_modify(oldpte, newprot);
> > -
> > -			if (uffd_wp)
> > -				ptent = pte_mkuffd_wp(ptent);
> > -			else if (uffd_wp_resolve)
> > -				ptent = pte_clear_uffd_wp(ptent);
> > -
> > 			/*
> > -			 * In some writable, shared mappings, we might want
> > -			 * to catch actual write access -- see
> > -			 * vma_wants_writenotify().
> > -			 *
> > -			 * In all writable, private mappings, we have to
> > -			 * properly handle COW.
> > -			 *
> > -			 * In both cases, we can sometimes still change PTEs
> > -			 * writable and avoid the write-fault handler, for
> > -			 * example, if a PTE is already dirty and no other
> > -			 * COW or special handling is required.
> > +			 * Optimize for the small-folio common case by
> > +			 * special-casing it here. Compiler constant propagation
> > +			 * plus copious amounts of __always_inline does wonders.
> > 			 */
> > -			if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) &&
> > -			     !pte_write(ptent))
> > -				set_write_prot_commit_flush_ptes(vma, folio, page,
> > -				addr, pte, oldpte, ptent, nr_ptes, tlb);
> > -			else
> > -				prot_commit_flush_ptes(vma, addr, pte, oldpte, ptent,
> > -					nr_ptes, /* idx = */ 0, /* set_write = */ false, tlb);
> > +			if (likely(nr_ptes == 1)) {
> 
> Are there any numbers for this optimization?

Yes, see the cover letter and the testing done by both myself, Luke and David.

> While I am all for optimizing the common
> case, it seems unfair to penalize the uncommon one here. Why is nr_ptes > 1 such an
> exotic use case (specially today)?

It's less common on most architectures due to having no real incentive for
enabling mTHP, thus having no anonymous pages with order > 0 (that aren't
PMD_ORDER and thus PMD mapped). This leaves you with pagecache folios (that may
trivially be larger), but that depends on RA and the filesystem itself (e.g
btrfs does not support large folios at all). But I would be extremely
surprised if there's a regression (not in the noise) on the order > 0 case,
as it effectively does more work per iteration (and per my measurements you
easily get order >= 3 on ext4 folios, up to maybe around order-7).

> ie: How does this change affect the program in
> b9bf6c2872c ("mm: refactor MM_CP_PROT_NUMA skipping case into new function"),

That's the series I'm "fixing" to restore performance on order-0.

-- 
Pedro

Re: [PATCH v3 1/2] mm/mprotect: move softleaf code out of the main function

[Vlastimil Babka (SUSE)]

On 4/2/26 16:16, Pedro Falcato wrote:
> Move softleaf change_pte_range code into a separate function. This makes
> the change_pte_range() function a good bit smaller, and lessens cognitive
> load when reading through the function.
> 
> Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
> Acked-by: David Hildenbrand (Arm) <david@kernel.org>
> Tested-by: Luke Yang <luyang@redhat.com>
> Signed-off-by: Pedro Falcato <pfalcato@suse.de>

Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>

Re: [PATCH v3 0/2] mm/mprotect: micro-optimization work

[Lorenzo Stoakes (Oracle)]

On Mon, Apr 06, 2026 at 10:38:54AM -0700, Andrew Morton wrote:
> On Mon, 6 Apr 2026 11:09:01 +0200 "David Hildenbrand (arm)" <david@kernel.org> wrote:
>
> > On 4/6/26 10:29, Pedro Falcato wrote:
> > > On Thu, Apr 02, 2026 at 11:33:28AM -0700, Andrew Morton wrote:
> > >> On Thu,  2 Apr 2026 15:16:26 +0100 Pedro Falcato <pfalcato@suse.de> wrote:
> > >>
> > >>> Micro-optimize the change_protection functionality and the
> > >>> change_pte_range() routine. This set of functions works in an incredibly
> > >>> tight loop, and even small inefficiencies are incredibly evident when spun
> > >>> hundreds, thousands or hundreds of thousands of times.
> > >>
> > >> Thanks, I updated mm.git's mm-unstable branch to this version.
> > >>
> > >> The update is rather large.  If people think it best to spill this work
> > >> into next -rc1 then please advise.
> > >
> > > I think it's a fairly safe change for 7.1, but I'm biased :) Perhaps others think
> > > otherwise.

I'm fine with this for 7.1 (as long as Pedro buys me a beer ;)

>
> Thanks, I've firmly added these two to the second-week-of-merge-window
> pile.

Ack!

>
> > It's mostly code movement, so it should be fine (unless we messed up
> > passed parameters somehow :) )
>
> This (or an earlier version thereof) have been in mm.git since March 24.

Also ack, yeah it's fine.

Cheers, Lorenzo

Re: [PATCH v3 2/2] mm/mprotect: special-case small folios when applying write permissions

[Vlastimil Babka (SUSE)]

On 4/2/26 16:16, Pedro Falcato wrote:
> The common order-0 case is important enough to want its own branch, and
> avoids the hairy, large loop logic that the CPU does not seem to handle
> particularly well.
> 
> While at it, encourage the compiler to inline batch PTE logic and resolve
> constant branches by adding __always_inline strategically.
> 
> Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
> Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
> Tested-by: Luke Yang <luyang@redhat.com>
> Signed-off-by: Pedro Falcato <pfalcato@suse.de>

Can't say I'm thrilled about these hacks, but seems to work, so

Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>


> +			if (likely(nr_ptes == 1)) {
> +				change_present_ptes(tlb, vma, addr, pte, 1,
> +					end, newprot, folio, page, cp_flags);
> +			} else {
> +				change_present_ptes(tlb, vma, addr, pte,
> +					nr_ptes, end, newprot, folio, page,
> +					cp_flags);
> +			}

I wonder if there's anything about this trick that ensures the compilers
will not eventually stop compiling it the way you intend. Have you talked to
compiler people? :)

> +
>  			pages += nr_ptes;
>  		} else if (pte_none(oldpte)) {
>  			/*

Re: [PATCH v3 2/2] mm/mprotect: special-case small folios when applying write permissions

[Pedro Falcato]

On Tue, Apr 07, 2026 at 02:31:29PM +0200, Vlastimil Babka (SUSE) wrote:
> On 4/2/26 16:16, Pedro Falcato wrote:
> > The common order-0 case is important enough to want its own branch, and
> > avoids the hairy, large loop logic that the CPU does not seem to handle
> > particularly well.
> > 
> > While at it, encourage the compiler to inline batch PTE logic and resolve
> > constant branches by adding __always_inline strategically.
> > 
> > Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
> > Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
> > Tested-by: Luke Yang <luyang@redhat.com>
> > Signed-off-by: Pedro Falcato <pfalcato@suse.de>
> 
> Can't say I'm thrilled about these hacks, but seems to work, so
> 
> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> 
> 
> > +			if (likely(nr_ptes == 1)) {
> > +				change_present_ptes(tlb, vma, addr, pte, 1,
> > +					end, newprot, folio, page, cp_flags);
> > +			} else {
> > +				change_present_ptes(tlb, vma, addr, pte,
> > +					nr_ptes, end, newprot, folio, page,
> > +					cp_flags);
> > +			}
> 
> I wonder if there's anything about this trick that ensures the compilers
> will not eventually stop compiling it the way you intend. Have you talked to
> compiler people? :)

These kinds of tricks are pretty ingrained across kernel code, e.g:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/include/asm/cmpxchg.h#n65

and similar. In these cases if it fails to do constant propagation you'll
get an actual build error. In our case you just get a regression back :)

-- 
Pedro