[PATCH 0/8] mm/khugepaged: several cleanups

[PATCH 0/8] mm/khugepaged: several cleanups

[Nico Pache]

The following changes stem from a number of reviews during my khugepaged
mTHP support series [1]. Some of these are minor code cleanups, issues or
reviews that we decided to deferred to a followup series, or in the case
of the more major patch of the series, changes [2] Lance Yang attempted
while my series was in-flight and we decided to wait till later to try.

The first 3 patches introduce helper functions to increase code reuse and
readability. This includes a per-scan state clearing function, extracting
the young page check into a helper, and a count_collapse_event() function
to reduce a repetative pattern used across mTHP collapse.

The 4th patch was the byproduct of me throwing Claude at all the
comments in khugepaged verifying and looking for any outdated info.

The 5th patch is based on Lance Yang's commit series [2] trying to extract
the PTE state checking into a helper function. This required a bit of
rewriting due to differences after mTHP collapse was introduced. I also
took into account the changes requested during his patches review cycle.

The remaining 3 patches were review points during my mTHP series that we
agreed can be deferred to a later series.

Thank you to those whos reviews and work I leveraged to achieve these
cleanups.

[1] - https://lore.kernel.org/all/20260605161422.213817-1-npache@redhat.com/
[2] - https://lore.kernel.org/all/20251008043748.45554-1-lance.yang@linux.dev/

Nico Pache (8):
  mm/khugepaged: refactor per-scan state clearing into
    collapse_control_init_scan()
  mm/khugepaged: extract young page check into collapse_is_young()
    helper
  mm/khugepaged: introduce a count_collapse_event() helper
  mm/khugepaged: fix outdated comments
  mm/khugepaged: Refactor the PTE state checks into a helper
  mm/khugepaged: unmap pte before releasing vma write lock
  mm/khugepaged: clarify a comment regarding max_ptes_none check
  mm: Documentation: clarify where the mTHP stats live

 Documentation/admin-guide/mm/transhuge.rst |   6 +-
 mm/khugepaged.c                            | 400 +++++++++++----------
 2 files changed, 216 insertions(+), 190 deletions(-)


base-commit: d148260a31fddf6d59cc0ea4980bd78ebe301a91
-- 
2.54.0

[PATCH 1/8] mm/khugepaged: refactor per-scan state clearing into collapse_control_init_scan()

[Nico Pache]

Extract the repeated clearing of node_load, alloc_nmask, and
mthp_present_ptes into a helper to reduce duplication in
collapse_scan_pmd() and collapse_scan_file(). Althought file scans do not
current use the bitmap, they will in the future, and clearing it now is
harmless.

Signed-off-by: Nico Pache <npache@redhat.com>
---
 mm/khugepaged.c | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 617bca76db49..b3985b854e77 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -628,6 +628,17 @@ void __khugepaged_exit(struct mm_struct *mm)
 	}
 }
 
+/*
+ * collapse_control_init_scan() - initialize/reset collapse_control variables
+ * that require being cleared once per-scan.
+ */
+static void collapse_control_init_scan(struct collapse_control *cc)
+{
+	memset(cc->node_load, 0, sizeof(cc->node_load));
+	nodes_clear(cc->alloc_nmask);
+	bitmap_zero(cc->mthp_present_ptes, MAX_PTRS_PER_PTE);
+}
+
 static void release_pte_folio(struct folio *folio)
 {
 	node_stat_mod_folio(folio,
@@ -1616,9 +1627,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 		goto out;
 	}
 
-	bitmap_zero(cc->mthp_present_ptes, MAX_PTRS_PER_PTE);
-	memset(cc->node_load, 0, sizeof(cc->node_load));
-	nodes_clear(cc->alloc_nmask);
+	collapse_control_init_scan(cc);
 
 	enabled_orders = collapse_possible_orders(vma, vma->vm_flags, tva_flags);
 
@@ -2686,8 +2695,7 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,
 
 	present = 0;
 	swap = 0;
-	memset(cc->node_load, 0, sizeof(cc->node_load));
-	nodes_clear(cc->alloc_nmask);
+	collapse_control_init_scan(cc);
 	rcu_read_lock();
 	xas_for_each(&xas, folio, start + HPAGE_PMD_NR - 1) {
 		if (xas_retry(&xas, folio))
-- 
2.54.0

Re: [PATCH 1/8] mm/khugepaged: refactor per-scan state clearing into collapse_control_init_scan()

[Usama Arif]

On Mon,  6 Jul 2026 09:44:48 -0600 Nico Pache <npache@redhat.com> wrote:

> Extract the repeated clearing of node_load, alloc_nmask, and
> mthp_present_ptes into a helper to reduce duplication in
> collapse_scan_pmd() and collapse_scan_file(). Althought file scans do not
> current use the bitmap, they will in the future, and clearing it now is
> harmless.
> 
> Signed-off-by: Nico Pache <npache@redhat.com>
> ---
>  mm/khugepaged.c | 18 +++++++++++++-----
>  1 file changed, 13 insertions(+), 5 deletions(-)
> 
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 617bca76db49..b3985b854e77 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -628,6 +628,17 @@ void __khugepaged_exit(struct mm_struct *mm)
>  	}
>  }
>  
> +/*
> + * collapse_control_init_scan() - initialize/reset collapse_control variables
> + * that require being cleared once per-scan.
> + */

Probably don't need the above comment, but apart from that

Acked-by: Usama Arif <usama.arif@linux.dev>

> +static void collapse_control_init_scan(struct collapse_control *cc)
> +{
> +	memset(cc->node_load, 0, sizeof(cc->node_load));
> +	nodes_clear(cc->alloc_nmask);
> +	bitmap_zero(cc->mthp_present_ptes, MAX_PTRS_PER_PTE);
> +}
> +
>  static void release_pte_folio(struct folio *folio)
>  {
>  	node_stat_mod_folio(folio,
> @@ -1616,9 +1627,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
>  		goto out;
>  	}
>  
> -	bitmap_zero(cc->mthp_present_ptes, MAX_PTRS_PER_PTE);
> -	memset(cc->node_load, 0, sizeof(cc->node_load));
> -	nodes_clear(cc->alloc_nmask);
> +	collapse_control_init_scan(cc);
>  
>  	enabled_orders = collapse_possible_orders(vma, vma->vm_flags, tva_flags);
>  
> @@ -2686,8 +2695,7 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,
>  
>  	present = 0;
>  	swap = 0;
> -	memset(cc->node_load, 0, sizeof(cc->node_load));
> -	nodes_clear(cc->alloc_nmask);
> +	collapse_control_init_scan(cc);
>  	rcu_read_lock();
>  	xas_for_each(&xas, folio, start + HPAGE_PMD_NR - 1) {
>  		if (xas_retry(&xas, folio))
> -- 
> 2.54.0
> 
> 

Re: [PATCH 1/8] mm/khugepaged: refactor per-scan state clearing into collapse_control_init_scan()

[Nico Pache]

On Mon, Jul 6, 2026 at 11:08 AM Usama Arif <usama.arif@linux.dev> wrote:
>
> On Mon,  6 Jul 2026 09:44:48 -0600 Nico Pache <npache@redhat.com> wrote:
>
> > Extract the repeated clearing of node_load, alloc_nmask, and
> > mthp_present_ptes into a helper to reduce duplication in
> > collapse_scan_pmd() and collapse_scan_file(). Althought file scans do not
> > current use the bitmap, they will in the future, and clearing it now is
> > harmless.
> >
> > Signed-off-by: Nico Pache <npache@redhat.com>
> > ---
> >  mm/khugepaged.c | 18 +++++++++++++-----
> >  1 file changed, 13 insertions(+), 5 deletions(-)
> >
> > diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> > index 617bca76db49..b3985b854e77 100644
> > --- a/mm/khugepaged.c
> > +++ b/mm/khugepaged.c
> > @@ -628,6 +628,17 @@ void __khugepaged_exit(struct mm_struct *mm)
> >       }
> >  }
> >
> > +/*
> > + * collapse_control_init_scan() - initialize/reset collapse_control variables
> > + * that require being cleared once per-scan.
> > + */
>
> Probably don't need the above comment, but apart from that

It was a last minute addition because I feared a review asking to add
a comment ;P I'm cool with either. The nice part of the comment is
that it clarifies these variables are per-scan.

>
> Acked-by: Usama Arif <usama.arif@linux.dev>

Thank you :)

>
> > +static void collapse_control_init_scan(struct collapse_control *cc)
> > +{
> > +     memset(cc->node_load, 0, sizeof(cc->node_load));
> > +     nodes_clear(cc->alloc_nmask);
> > +     bitmap_zero(cc->mthp_present_ptes, MAX_PTRS_PER_PTE);
> > +}
> > +
> >  static void release_pte_folio(struct folio *folio)
> >  {
> >       node_stat_mod_folio(folio,
> > @@ -1616,9 +1627,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
> >               goto out;
> >       }
> >
> > -     bitmap_zero(cc->mthp_present_ptes, MAX_PTRS_PER_PTE);
> > -     memset(cc->node_load, 0, sizeof(cc->node_load));
> > -     nodes_clear(cc->alloc_nmask);
> > +     collapse_control_init_scan(cc);
> >
> >       enabled_orders = collapse_possible_orders(vma, vma->vm_flags, tva_flags);
> >
> > @@ -2686,8 +2695,7 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,
> >
> >       present = 0;
> >       swap = 0;
> > -     memset(cc->node_load, 0, sizeof(cc->node_load));
> > -     nodes_clear(cc->alloc_nmask);
> > +     collapse_control_init_scan(cc);
> >       rcu_read_lock();
> >       xas_for_each(&xas, folio, start + HPAGE_PMD_NR - 1) {
> >               if (xas_retry(&xas, folio))
> > --
> > 2.54.0
> >
> >
>

[PATCH 2/8] mm/khugepaged: extract young page check into collapse_is_young() helper

[Nico Pache]

The change deduplicates the "is this PTE young enough to count as
referenced" condition that was repeated in both
__collapse_huge_page_isolate() and collapse_scan_pmd(), extracting it into
a single inline helper function.

Also move the comment and use it as the function header. While we are at
it, updated the comment to clarify that a young pte is a recently accessed
one.

Signed-off-by: Nico Pache <npache@redhat.com>
---
 mm/khugepaged.c | 35 +++++++++++++++++++----------------
 1 file changed, 19 insertions(+), 16 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index b3985b854e77..48b008a3c891 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -675,6 +675,23 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte,
 	}
 }
 
+/*
+ * collapse_is_young() - Check for enough young pte to justify collapsing
+ *
+ * If collapse was initiated by khugepaged, check that the page has been
+ * recently accessed (young pte) to justify collapsing the page.
+ *
+ * Return: true if the page has been recently accessed (young pte).
+ */
+static inline bool collapse_is_young(struct collapse_control *cc, pte_t pteval,
+	struct folio *folio, struct vm_area_struct *vma, unsigned long addr)
+{
+	return cc->is_khugepaged &&
+	       (pte_young(pteval) || folio_test_young(folio) ||
+		folio_test_referenced(folio) ||
+		mmu_notifier_test_young(vma->vm_mm, addr));
+}
+
 static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
 		unsigned long start_addr, pte_t *pte, struct collapse_control *cc,
 		unsigned int order, struct list_head *compound_pagelist)
@@ -813,14 +830,7 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
 		if (folio_test_large(folio))
 			list_add_tail(&folio->lru, compound_pagelist);
 next:
-		/*
-		 * If collapse was initiated by khugepaged, check that there is
-		 * enough young pte to justify collapsing the page
-		 */
-		if (cc->is_khugepaged &&
-		    (pte_young(pteval) || folio_test_young(folio) ||
-		     folio_test_referenced(folio) ||
-		     mmu_notifier_test_young(vma->vm_mm, addr)))
+		if (collapse_is_young(cc, pteval, folio, vma, addr))
 			referenced++;
 	}
 
@@ -1769,14 +1779,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 			goto out_unmap;
 		}
 
-		/*
-		 * If collapse was initiated by khugepaged, check that there is
-		 * enough young pte to justify collapsing the page
-		 */
-		if (cc->is_khugepaged &&
-		    (pte_young(pteval) || folio_test_young(folio) ||
-		     folio_test_referenced(folio) ||
-		     mmu_notifier_test_young(vma->vm_mm, addr)))
+		if (collapse_is_young(cc, pteval, folio, vma, addr))
 			referenced++;
 	}
 	if (cc->is_khugepaged &&
-- 
2.54.0

Re: [PATCH 2/8] mm/khugepaged: extract young page check into collapse_is_young() helper

[Usama Arif]

On Mon,  6 Jul 2026 09:44:49 -0600 Nico Pache <npache@redhat.com> wrote:

> The change deduplicates the "is this PTE young enough to count as
> referenced" condition that was repeated in both
> __collapse_huge_page_isolate() and collapse_scan_pmd(), extracting it into
> a single inline helper function.
> 
> Also move the comment and use it as the function header. While we are at
> it, updated the comment to clarify that a young pte is a recently accessed
> one.
> 
> Signed-off-by: Nico Pache <npache@redhat.com>

Acked-by: Usama Arif <usama.arif@linux.dev>


> ---
>  mm/khugepaged.c | 35 +++++++++++++++++++----------------
>  1 file changed, 19 insertions(+), 16 deletions(-)
> 
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index b3985b854e77..48b008a3c891 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -675,6 +675,23 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte,
>  	}
>  }
>  
> +/*
> + * collapse_is_young() - Check for enough young pte to justify collapsing
> + *
> + * If collapse was initiated by khugepaged, check that the page has been
> + * recently accessed (young pte) to justify collapsing the page.
> + *
> + * Return: true if the page has been recently accessed (young pte).
> + */
> +static inline bool collapse_is_young(struct collapse_control *cc, pte_t pteval,
> +	struct folio *folio, struct vm_area_struct *vma, unsigned long addr)
> +{
> +	return cc->is_khugepaged &&
> +	       (pte_young(pteval) || folio_test_young(folio) ||
> +		folio_test_referenced(folio) ||
> +		mmu_notifier_test_young(vma->vm_mm, addr));
> +}
> +
>  static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
>  		unsigned long start_addr, pte_t *pte, struct collapse_control *cc,
>  		unsigned int order, struct list_head *compound_pagelist)
> @@ -813,14 +830,7 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
>  		if (folio_test_large(folio))
>  			list_add_tail(&folio->lru, compound_pagelist);
>  next:
> -		/*
> -		 * If collapse was initiated by khugepaged, check that there is
> -		 * enough young pte to justify collapsing the page
> -		 */
> -		if (cc->is_khugepaged &&
> -		    (pte_young(pteval) || folio_test_young(folio) ||
> -		     folio_test_referenced(folio) ||
> -		     mmu_notifier_test_young(vma->vm_mm, addr)))
> +		if (collapse_is_young(cc, pteval, folio, vma, addr))
>  			referenced++;
>  	}
>  
> @@ -1769,14 +1779,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
>  			goto out_unmap;
>  		}
>  
> -		/*
> -		 * If collapse was initiated by khugepaged, check that there is
> -		 * enough young pte to justify collapsing the page
> -		 */
> -		if (cc->is_khugepaged &&
> -		    (pte_young(pteval) || folio_test_young(folio) ||
> -		     folio_test_referenced(folio) ||
> -		     mmu_notifier_test_young(vma->vm_mm, addr)))
> +		if (collapse_is_young(cc, pteval, folio, vma, addr))
>  			referenced++;
>  	}
>  	if (cc->is_khugepaged &&
> -- 
> 2.54.0
> 
> 

[PATCH 3/8] mm/khugepaged: introduce a count_collapse_event() helper

[Nico Pache]

Provide a simple helper function to help reduce a often used, and
duplicate pattern across the khugepaged code.

When collapsing to a PMD we need to record a vm_event and the mTHP_stat
event. When doing mTHP collapse we only update the mTHP stat.

Signed-off-by: Nico Pache <npache@redhat.com>
---
 mm/khugepaged.c | 36 ++++++++++++++++++------------------
 1 file changed, 18 insertions(+), 18 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 48b008a3c891..7047253a6fb6 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -692,6 +692,14 @@ static inline bool collapse_is_young(struct collapse_control *cc, pte_t pteval,
 		mmu_notifier_test_young(vma->vm_mm, addr));
 }
 
+static void count_collapse_event(unsigned int order, enum vm_event_item vm_event,
+		enum mthp_stat_item mthp_event)
+{
+	if (is_pmd_order(order))
+		count_vm_event(vm_event);
+	count_mthp_stat(order, mthp_event);
+}
+
 static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
 		unsigned long start_addr, pte_t *pte, struct collapse_control *cc,
 		unsigned int order, struct list_head *compound_pagelist)
@@ -712,9 +720,8 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
 		if (pte_none_or_zero(pteval)) {
 			if (++none_or_zero > max_ptes_none) {
 				result = SCAN_EXCEED_NONE_PTE;
-				if (is_pmd_order(order))
-					count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
-				count_mthp_stat(order, MTHP_STAT_COLLAPSE_EXCEED_NONE);
+				count_collapse_event(order, THP_SCAN_EXCEED_NONE_PTE,
+						MTHP_STAT_COLLAPSE_EXCEED_NONE);
 				goto out;
 			}
 			continue;
@@ -756,9 +763,8 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
 			 */
 			if (++shared > max_ptes_shared) {
 				result = SCAN_EXCEED_SHARED_PTE;
-				if (is_pmd_order(order))
-					count_vm_event(THP_SCAN_EXCEED_SHARED_PTE);
-				count_mthp_stat(order, MTHP_STAT_COLLAPSE_EXCEED_SHARED);
+				count_collapse_event(order, THP_SCAN_EXCEED_SHARED_PTE,
+						MTHP_STAT_COLLAPSE_EXCEED_SHARED);
 				goto out;
 			}
 		}
@@ -1267,15 +1273,12 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru
 	folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask);
 	if (!folio) {
 		*foliop = NULL;
-		if (is_pmd_order(order))
-			count_vm_event(THP_COLLAPSE_ALLOC_FAILED);
-		count_mthp_stat(order, MTHP_STAT_COLLAPSE_ALLOC_FAILED);
+		count_collapse_event(order, THP_COLLAPSE_ALLOC_FAILED,
+				MTHP_STAT_COLLAPSE_ALLOC_FAILED);
 		return SCAN_ALLOC_HUGE_PAGE_FAIL;
 	}
 
-	if (is_pmd_order(order))
-		count_vm_event(THP_COLLAPSE_ALLOC);
-	count_mthp_stat(order, MTHP_STAT_COLLAPSE_ALLOC);
+	count_collapse_event(order, THP_COLLAPSE_ALLOC, MTHP_STAT_COLLAPSE_ALLOC);
 
 	if (unlikely(mem_cgroup_charge(folio, mm, gfp))) {
 		folio_put(folio);
@@ -1665,8 +1668,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 		if (pte_none_or_zero(pteval)) {
 			if (++none_or_zero > max_ptes_none) {
 				result = SCAN_EXCEED_NONE_PTE;
-				count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
-				count_mthp_stat(HPAGE_PMD_ORDER,
+				count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_NONE_PTE,
 						MTHP_STAT_COLLAPSE_EXCEED_NONE);
 				goto out_unmap;
 			}
@@ -1675,8 +1677,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 		if (!pte_present(pteval)) {
 			if (++unmapped > max_ptes_swap) {
 				result = SCAN_EXCEED_SWAP_PTE;
-				count_vm_event(THP_SCAN_EXCEED_SWAP_PTE);
-				count_mthp_stat(HPAGE_PMD_ORDER,
+				count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_SWAP_PTE,
 						MTHP_STAT_COLLAPSE_EXCEED_SWAP);
 				goto out_unmap;
 			}
@@ -1734,8 +1735,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 		if (folio_maybe_mapped_shared(folio)) {
 			if (++shared > max_ptes_shared) {
 				result = SCAN_EXCEED_SHARED_PTE;
-				count_vm_event(THP_SCAN_EXCEED_SHARED_PTE);
-				count_mthp_stat(HPAGE_PMD_ORDER,
+				count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_SHARED_PTE,
 						MTHP_STAT_COLLAPSE_EXCEED_SHARED);
 				goto out_unmap;
 			}
-- 
2.54.0

Re: [PATCH 3/8] mm/khugepaged: introduce a count_collapse_event() helper

[Usama Arif]

On Mon,  6 Jul 2026 09:44:50 -0600 Nico Pache <npache@redhat.com> wrote:

> Provide a simple helper function to help reduce a often used, and
> duplicate pattern across the khugepaged code.
> 
> When collapsing to a PMD we need to record a vm_event and the mTHP_stat
> event. When doing mTHP collapse we only update the mTHP stat.
> 
> Signed-off-by: Nico Pache <npache@redhat.com>
> ---
>  mm/khugepaged.c | 36 ++++++++++++++++++------------------
>  1 file changed, 18 insertions(+), 18 deletions(-)
> 

Acked-by: Usama Arif <usama.arif@linux.dev>
 

[PATCH 4/8] mm/khugepaged: fix outdated comments

[Nico Pache]

Fix comment in collapse_scan_pmd() that still described the old
folio_mapcount() > folio_ref_count() check and a "512" false-positive
scenario. The code now uses folio_expected_ref_count() != folio_ref_count()
which doesn't suffer from the same limitation.

Fix comment in collapse_huge_page() that referenced ptep_clear_flush,
when the code actually uses pmdp_collapse_flush.

Fix comment in __collapse_huge_page_swapin() that referenced the old
function name khugepaged_scan_pmd, now collapse_scan_pmd.

Also clean up some simple typos and stale terminology (mmap_sem ->
mmap_lock, PG_lock -> folio lock, page -> folio, grammar).

Assisted-by: Cursor(claude-sonnet-4):4.6
Signed-off-by: Nico Pache <npache@redhat.com>
---
 mm/khugepaged.c | 23 ++++++++++-------------
 1 file changed, 10 insertions(+), 13 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 7047253a6fb6..676f75773a6c 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -619,7 +619,7 @@ void __khugepaged_exit(struct mm_struct *mm)
 		/*
 		 * This is required to serialize against
 		 * collapse_test_exit() (which is guaranteed to run
-		 * under mmap sem read mode). Stop here (after we return all
+		 * under mmap_lock read mode). Stop here (after we return all
 		 * pagetables will be destroyed) until khugepaged has finished
 		 * working on the pagetables under the mmap_lock.
 		 */
@@ -792,7 +792,7 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
 
 		/*
 		 * We can do it before folio_isolate_lru because the
-		 * folio can't be freed from under us. NOTE: PG_lock
+		 * folio can't be freed from under us. NOTE: folio lock
 		 * is needed to serialize against split_huge_page
 		 * when invoked from the VM.
 		 */
@@ -819,7 +819,7 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
 		}
 
 		/*
-		 * Isolate the page to avoid collapsing an hugepage
+		 * Isolate the folio to avoid collapsing a hugepage
 		 * currently in use by the VM.
 		 */
 		if (!folio_isolate_lru(folio)) {
@@ -1104,7 +1104,7 @@ static enum scan_result hugepage_vma_revalidate(struct mm_struct *mm, unsigned l
 		return SCAN_VMA_CHECK;
 	/*
 	 * Anon VMA expected, the address may be unmapped then
-	 * remapped to file after khugepaged reaquired the mmap_lock.
+	 * remapped to file after khugepaged reacquired the mmap_lock.
 	 *
 	 * thp_vma_allowable_orders may return true for qualified file
 	 * vmas.
@@ -1162,7 +1162,7 @@ static enum scan_result check_pmd_still_valid(struct mm_struct *mm,
 
 /*
  * Bring missing pages in from swap, to complete THP collapse.
- * Only done if khugepaged_scan_pmd believes it is worthwhile.
+ * Only done if collapse_scan_pmd believes it is worthwhile.
  *
  * For mTHP orders the function bails on the first swap entry, because
  * faulting pages back in during collapse could re-populate PTEs that
@@ -1354,8 +1354,8 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s
 	mmap_read_unlock(mm);
 	/*
 	 * Prevent all access to pagetables with the exception of
-	 * gup_fast later handled by the ptep_clear_flush and the VM
-	 * handled by the anon_vma lock + PG_lock.
+	 * gup_fast later handled by the pmdp_collapse_flush and the VM
+	 * handled by the anon_vma lock + folio lock.
 	 *
 	 * UFFDIO_MOVE is prevented to race as well thanks to the
 	 * mmap_lock.
@@ -1767,12 +1767,9 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 		/*
 		 * Check if the page has any GUP (or other external) pins.
 		 *
-		 * Here the check may be racy:
-		 * it may see folio_mapcount() > folio_ref_count().
-		 * But such case is ephemeral we could always retry collapse
-		 * later.  However it may report false positive if the page
-		 * has excessive GUP pins (i.e. 512).  Anyway the same check
-		 * will be done again later the risk seems low.
+		 * Here the check is racy, but such case is ephemeral and
+		 * we could always retry collapse later. Anyway the same
+		 * check will be done again later the risk seems low.
 		 */
 		if (folio_expected_ref_count(folio) != folio_ref_count(folio)) {
 			result = SCAN_PAGE_COUNT;
-- 
2.54.0

Re: [PATCH 4/8] mm/khugepaged: fix outdated comments

[Usama Arif]

On Mon,  6 Jul 2026 09:44:51 -0600 Nico Pache <npache@redhat.com> wrote:

> Fix comment in collapse_scan_pmd() that still described the old
> folio_mapcount() > folio_ref_count() check and a "512" false-positive
> scenario. The code now uses folio_expected_ref_count() != folio_ref_count()
> which doesn't suffer from the same limitation.
> 
> Fix comment in collapse_huge_page() that referenced ptep_clear_flush,
> when the code actually uses pmdp_collapse_flush.
> 
> Fix comment in __collapse_huge_page_swapin() that referenced the old
> function name khugepaged_scan_pmd, now collapse_scan_pmd.
> 
> Also clean up some simple typos and stale terminology (mmap_sem ->
> mmap_lock, PG_lock -> folio lock, page -> folio, grammar).
> 
> Assisted-by: Cursor(claude-sonnet-4):4.6
> Signed-off-by: Nico Pache <npache@redhat.com>
> ---
>  mm/khugepaged.c | 23 ++++++++++-------------
>  1 file changed, 10 insertions(+), 13 deletions(-)
> 

Acked-by: Usama Arif <usama.arif@linux.dev>
 

[PATCH 5/8] mm/khugepaged: Refactor the PTE state checks into a helper

[Nico Pache]

For anonymous collapse, the collapse_scan_pmd() and
__collapse_huge_page_isolate() functions share a large portion of their
logic. These functions both check the state of the PTEs and verify the
following:
	- max_pte_* values are not exceeded
	- uffd is not active
	- lazyfree properties
	- non-anonymous

Merge these checks into a helper collapse_check_pte() to reduce code
duplication. We also add a helper struct for this function called
pte_check_context which allows us to pass the required parameters in a
clean and elegant manner.

A helper function is also introduced pte_check_fail() to provide a clean
interface to set the pte_check_context failure results and return
PTE_CHECK_FAIL state. This helps reduce code duplications across the new
collapse_check_pte function.

Two slight modifications are done to the original functionality. We now
warn (instead of crash) if the anon test fails, and we leverage the
vm_normal_folio function instead of page->folio, this should be
functionally equivalent.

No other functional changes intended.

This patch is heavily based off work done by Lance Yang, but modified to
deal with conflicts and feedback received during the review cycle [1].

[1] https://lore.kernel.org/all/20251008043748.45554-1-lance.yang@linux.dev/
Suggested-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Nico Pache <npache@redhat.com>
---
 mm/khugepaged.c | 295 +++++++++++++++++++++++++-----------------------
 1 file changed, 155 insertions(+), 140 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 676f75773a6c..c4ea2dc1591b 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -63,6 +63,12 @@ enum scan_result {
 	SCAN_PAGE_DIRTY_OR_WRITEBACK,
 };
 
+enum pte_check_result {
+	PTE_CHECK_SUCCEED,
+	PTE_CHECK_FAIL,
+	PTE_CHECK_CONTINUE,
+};
+
 #define CREATE_TRACE_POINTS
 #include <trace/events/huge_memory.h>
 
@@ -117,6 +123,20 @@ struct collapse_control {
 	DECLARE_BITMAP(mthp_present_ptes, MAX_PTRS_PER_PTE);
 };
 
+struct pte_check_context {
+	struct collapse_control *cc;
+	struct vm_area_struct *vma;
+	unsigned int order;
+	struct folio *folio;
+	int none_or_zero;
+	int shared;
+	int unmapped;
+	enum scan_result result;
+	unsigned int max_ptes_none;
+	unsigned int max_ptes_swap;
+	unsigned int max_ptes_shared;
+};
+
 /**
  * struct khugepaged_scan - cursor for scanning
  * @mm_head: the head of the mm list to scan
@@ -700,74 +720,130 @@ static void count_collapse_event(unsigned int order, enum vm_event_item vm_event
 	count_mthp_stat(order, mthp_event);
 }
 
+/*
+ * pte_check_fail() - A simple helper to set the pte_check_context result and
+ * return PTE_CHECK_FAIL.
+ */
+static enum pte_check_result pte_check_fail(struct pte_check_context *ctx,
+		enum scan_result result)
+{
+	ctx->result = result;
+	return PTE_CHECK_FAIL;
+}
+
+/*
+ * collapse_check_pte() - Check if a PTE is suitable for collapse
+ *
+ * Check if a PTE is suitable for collapse based on the following criteria:
+ * - max_pte_* values are not exceeded
+ * - uffd is not active
+ * - lazyfree properties are not present
+ * - only anonymous pages are present
+ *
+ * a helper struct pte_check_context is used to pass and store relevant
+ * information between the collapse_check_pte() function and the caller.
+ *
+ * Return: PTE_CHECK_SUCCEED if the PTE is suitable for collapse,
+ *         PTE_CHECK_FAIL if the PTE is not suitable for collapse,
+ *         PTE_CHECK_CONTINUE if the scan should continue to check the next PTE.
+ */
+static enum pte_check_result collapse_check_pte(pte_t pteval,
+		unsigned long addr, struct pte_check_context *ctx)
+{
+	if (pte_none_or_zero(pteval)) {
+		if (++ctx->none_or_zero > ctx->max_ptes_none) {
+			count_collapse_event(ctx->order, THP_SCAN_EXCEED_NONE_PTE,
+					MTHP_STAT_COLLAPSE_EXCEED_NONE);
+			return pte_check_fail(ctx, SCAN_EXCEED_NONE_PTE);
+		}
+		return PTE_CHECK_CONTINUE;
+	}
+	if (!pte_present(pteval)) {
+		if (ctx->unmapped == -1)
+			return pte_check_fail(ctx, SCAN_PTE_NON_PRESENT);
+		if (++ctx->unmapped > ctx->max_ptes_swap) {
+			count_collapse_event(ctx->order, THP_SCAN_EXCEED_SWAP_PTE,
+					MTHP_STAT_COLLAPSE_EXCEED_SWAP);
+			return pte_check_fail(ctx, SCAN_EXCEED_SWAP_PTE);
+		}
+		if (pte_swp_uffd_wp_any(pteval))
+			return pte_check_fail(ctx, SCAN_PTE_UFFD_WP);
+		return PTE_CHECK_CONTINUE;
+	}
+	/*
+	 * Don't collapse if any of the small PTEs are armed with uffd
+	 * write protection. Marking the new huge pmd as write protected
+	 * could bring userfault messages that fall outside of the
+	 * registered range.
+	 */
+	if (pte_uffd_wp(pteval))
+		return pte_check_fail(ctx, SCAN_PTE_UFFD_WP);
+
+	ctx->folio = vm_normal_folio(ctx->vma, addr, pteval);
+	if (unlikely(!ctx->folio) || unlikely(folio_is_zone_device(ctx->folio)))
+		return pte_check_fail(ctx, SCAN_PAGE_NULL);
+
+	/*
+	 * If the vma has the VM_DROPPABLE flag, the collapse will
+	 * preserve the lazyfree property without needing to skip.
+	 */
+	if (ctx->cc->is_khugepaged && !(ctx->vma->vm_flags & VM_DROPPABLE) &&
+	    folio_test_lazyfree(ctx->folio) && !pte_dirty(pteval))
+		return pte_check_fail(ctx, SCAN_PAGE_LAZYFREE);
+
+	if (folio_maybe_mapped_shared(ctx->folio)) {
+		/*
+		 * TODO: Support shared pages without leading to further
+		 * mTHP collapses. Currently bringing in new pages via
+		 * shared may cause a future higher order collapse on a
+		 * rescan of the same range.
+		 */
+		if (++ctx->shared > ctx->max_ptes_shared) {
+			count_collapse_event(ctx->order, THP_SCAN_EXCEED_SHARED_PTE,
+					MTHP_STAT_COLLAPSE_EXCEED_SHARED);
+			return pte_check_fail(ctx, SCAN_EXCEED_SHARED_PTE);
+		}
+	}
+
+	if (!folio_test_anon(ctx->folio)) {
+		VM_WARN_ON_FOLIO(!folio_test_anon(ctx->folio), ctx->folio);
+		return pte_check_fail(ctx, SCAN_PAGE_ANON);
+	}
+	return PTE_CHECK_SUCCEED;
+}
+
 static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
 		unsigned long start_addr, pte_t *pte, struct collapse_control *cc,
 		unsigned int order, struct list_head *compound_pagelist)
 {
-	const unsigned int max_ptes_none = collapse_max_ptes_none(cc, vma, order);
-	const unsigned int max_ptes_shared = collapse_max_ptes_shared(cc, order);
 	const unsigned long nr_pages = 1UL << order;
-	struct page *page = NULL;
 	struct folio *folio = NULL;
 	unsigned long addr = start_addr;
-	pte_t *_pte;
-	int none_or_zero = 0, shared = 0, referenced = 0;
+	pte_t *_pte, pteval;
+	int referenced = 0;
 	enum scan_result result = SCAN_FAIL;
+	enum pte_check_result pte_check;
+	struct pte_check_context ctx = {
+		.cc = cc,
+		.vma = vma,
+		.order = order,
+		.unmapped = -1, /* don't check swap PTEs */
+		.max_ptes_none = collapse_max_ptes_none(cc, vma, order),
+		.max_ptes_shared = collapse_max_ptes_shared(cc, order),
+	};
 
 	for (_pte = pte; _pte < pte + nr_pages;
 	     _pte++, addr += PAGE_SIZE) {
-		pte_t pteval = ptep_get(_pte);
-		if (pte_none_or_zero(pteval)) {
-			if (++none_or_zero > max_ptes_none) {
-				result = SCAN_EXCEED_NONE_PTE;
-				count_collapse_event(order, THP_SCAN_EXCEED_NONE_PTE,
-						MTHP_STAT_COLLAPSE_EXCEED_NONE);
-				goto out;
-			}
-			continue;
-		}
-		if (!pte_present(pteval)) {
-			result = SCAN_PTE_NON_PRESENT;
-			goto out;
-		}
-		if (pte_uffd_wp(pteval)) {
-			result = SCAN_PTE_UFFD_WP;
-			goto out;
-		}
-		page = vm_normal_page(vma, addr, pteval);
-		if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
-			result = SCAN_PAGE_NULL;
-			goto out;
-		}
-
-		folio = page_folio(page);
-		VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio);
-
-		/*
-		 * If the vma has the VM_DROPPABLE flag, the collapse will
-		 * preserve the lazyfree property without needing to skip.
-		 */
-		if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
-		    folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
-			result = SCAN_PAGE_LAZYFREE;
+		pteval = ptep_get(_pte);
+		pte_check = collapse_check_pte(pteval, addr, &ctx);
+		if (pte_check == PTE_CHECK_FAIL) {
+			result = ctx.result;
 			goto out;
 		}
+		if (pte_check == PTE_CHECK_CONTINUE)
+			continue;
+		folio = ctx.folio;
 
-		/* See collapse_scan_pmd(). */
-		if (folio_maybe_mapped_shared(folio)) {
-			/*
-			 * TODO: Support shared pages without leading to further
-			 * mTHP collapses. Currently bringing in new pages via
-			 * shared may cause a future higher order collapse on a
-			 * rescan of the same range.
-			 */
-			if (++shared > max_ptes_shared) {
-				result = SCAN_EXCEED_SHARED_PTE;
-				count_collapse_event(order, THP_SCAN_EXCEED_SHARED_PTE,
-						MTHP_STAT_COLLAPSE_EXCEED_SHARED);
-				goto out;
-			}
-		}
 		/*
 		 * TODO: In some cases of partially-mapped folios, we'd actually
 		 * want to collapse.
@@ -844,13 +920,13 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
 		result = SCAN_LACK_REFERENCED_PAGE;
 	} else {
 		result = SCAN_SUCCEED;
-		trace_mm_collapse_huge_page_isolate(folio, none_or_zero,
+		trace_mm_collapse_huge_page_isolate(folio, ctx.none_or_zero,
 						    referenced, result, order);
 		return result;
 	}
 out:
 	release_pte_pages(pte, _pte, compound_pagelist);
-	trace_mm_collapse_huge_page_isolate(folio, none_or_zero,
+	trace_mm_collapse_huge_page_isolate(folio, ctx.none_or_zero,
 					    referenced, result, order);
 	return result;
 }
@@ -1616,24 +1692,30 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 		struct vm_area_struct *vma, unsigned long start_addr,
 		bool *lock_dropped, struct collapse_control *cc)
 {
-	const unsigned int max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER);
-	const unsigned int max_ptes_swap = collapse_max_ptes_swap(cc, HPAGE_PMD_ORDER);
-	unsigned int max_ptes_none = collapse_max_ptes_none(cc, vma, HPAGE_PMD_ORDER);
 	enum tva_type tva_flags = cc->is_khugepaged ? TVA_KHUGEPAGED : TVA_FORCED_COLLAPSE;
 	pmd_t *pmd;
 	pte_t *pte, *_pte, pteval;
 	int i;
-	int none_or_zero = 0, shared = 0, referenced = 0;
-	enum scan_result result = SCAN_FAIL;
-	struct page *page = NULL;
 	struct folio *folio = NULL;
+	int referenced = 0;
+	enum scan_result result = SCAN_FAIL;
 	unsigned long addr;
 	unsigned long enabled_orders;
 	spinlock_t *ptl;
-	int node = NUMA_NO_NODE, unmapped = 0;
+	int node = NUMA_NO_NODE;
+	enum pte_check_result pte_check;
 
 	VM_BUG_ON(start_addr & ~HPAGE_PMD_MASK);
 
+	struct pte_check_context ctx = {
+		.cc = cc,
+		.vma = vma,
+		.order = HPAGE_PMD_ORDER,
+		.max_ptes_none = collapse_max_ptes_none(cc, vma, HPAGE_PMD_ORDER),
+		.max_ptes_swap = collapse_max_ptes_swap(cc, HPAGE_PMD_ORDER),
+		.max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER),
+	};
+
 	result = find_pmd_or_thp_or_none(mm, start_addr, &pmd);
 	if (result != SCAN_SUCCEED) {
 		cc->progress++;
@@ -1649,7 +1731,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 	 * scan all pages to populate the bitmap for mTHP collapse.
 	 */
 	if (enabled_orders != BIT(HPAGE_PMD_ORDER))
-		max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT;
+		ctx.max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT;
 
 	pte = pte_offset_map_lock(mm, pmd, start_addr, &ptl);
 	if (!pte) {
@@ -1665,81 +1747,14 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 
 		cc->progress++;
 
-		if (pte_none_or_zero(pteval)) {
-			if (++none_or_zero > max_ptes_none) {
-				result = SCAN_EXCEED_NONE_PTE;
-				count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_NONE_PTE,
-						MTHP_STAT_COLLAPSE_EXCEED_NONE);
-				goto out_unmap;
-			}
-			continue;
-		}
-		if (!pte_present(pteval)) {
-			if (++unmapped > max_ptes_swap) {
-				result = SCAN_EXCEED_SWAP_PTE;
-				count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_SWAP_PTE,
-						MTHP_STAT_COLLAPSE_EXCEED_SWAP);
-				goto out_unmap;
-			}
-			/*
-			 * Always be strict with uffd-wp
-			 * enabled swap entries.  Please see
-			 * comment below for pte_uffd_wp().
-			 */
-			if (pte_swp_uffd_wp_any(pteval)) {
-				result = SCAN_PTE_UFFD_WP;
-				goto out_unmap;
-			}
-			continue;
-		}
-		if (pte_uffd_wp(pteval)) {
-			/*
-			 * Don't collapse the page if any of the small
-			 * PTEs are armed with uffd write protection.
-			 * Here we can also mark the new huge pmd as
-			 * write protected if any of the small ones is
-			 * marked but that could bring unknown
-			 * userfault messages that falls outside of
-			 * the registered range.  So, just be simple.
-			 */
-			result = SCAN_PTE_UFFD_WP;
-			goto out_unmap;
-		}
-
-		page = vm_normal_page(vma, addr, pteval);
-		if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
-			result = SCAN_PAGE_NULL;
-			goto out_unmap;
-		}
-		folio = page_folio(page);
-
-		/*
-		 * If the vma has the VM_DROPPABLE flag, the collapse will
-		 * preserve the lazyfree property without needing to skip.
-		 */
-		if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
-		    folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
-			result = SCAN_PAGE_LAZYFREE;
-			goto out_unmap;
-		}
-
-		if (!folio_test_anon(folio)) {
-			result = SCAN_PAGE_ANON;
+		pte_check = collapse_check_pte(pteval, addr, &ctx);
+		if (pte_check == PTE_CHECK_FAIL) {
+			result = ctx.result;
 			goto out_unmap;
 		}
-
-		/*
-		 * We treat a single page as shared if any part of the THP
-		 * is shared.
-		 */
-		if (folio_maybe_mapped_shared(folio)) {
-			if (++shared > max_ptes_shared) {
-				result = SCAN_EXCEED_SHARED_PTE;
-				count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_SHARED_PTE,
-						MTHP_STAT_COLLAPSE_EXCEED_SHARED);
-				goto out_unmap;
-			}
-		}
+		if (pte_check == PTE_CHECK_CONTINUE)
+			continue;
+		folio = ctx.folio;
 
 		/* Set bit for occupied pages */
 		__set_bit(i, cc->mthp_present_ptes);
@@ -1781,7 +1796,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 	}
 	if (cc->is_khugepaged &&
 		   (!referenced ||
-		    (unmapped && referenced < HPAGE_PMD_NR / 2))) {
+		    (ctx.unmapped && referenced < HPAGE_PMD_NR / 2))) {
 		result = SCAN_LACK_REFERENCED_PAGE;
 	} else {
 		result = SCAN_SUCCEED;
@@ -1792,13 +1807,13 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 		/* collapse_huge_page expects the lock to be dropped before calling */
 		mmap_read_unlock(mm);
 		result = mthp_collapse(mm, start_addr, referenced,
-				       unmapped, cc, enabled_orders);
+				       ctx.unmapped, cc, enabled_orders);
 		/* mmap_lock was released above, set lock_dropped */
 		*lock_dropped = true;
 	}
 out:
 	trace_mm_khugepaged_scan_pmd(mm, folio, referenced,
-				     none_or_zero, result, unmapped);
+				     ctx.none_or_zero, result, ctx.unmapped);
 	return result;
 }
 
-- 
2.54.0

Re: [PATCH 5/8] mm/khugepaged: Refactor the PTE state checks into a helper

[Nico Pache]

On Mon, Jul 6, 2026 at 9:45 AM Nico Pache <npache@redhat.com> wrote:
>
> For anonymous collapse, the collapse_scan_pmd() and
> __collapse_huge_page_isolate() functions share a large portion of their
> logic. These functions both check the state of the PTEs and verify the
> following:
>         - max_pte_* values are not exceeded
>         - uffd is not active
>         - lazyfree properties
>         - non-anonymous
>
> Merge these checks into a helper collapse_check_pte() to reduce code
> duplication. We also add a helper struct for this function called
> pte_check_context which allows us to pass the required parameters in a
> clean and elegant manner.
>
> A helper function is also introduced pte_check_fail() to provide a clean
> interface to set the pte_check_context failure results and return
> PTE_CHECK_FAIL state. This helps reduce code duplications across the new
> collapse_check_pte function.
>
> Two slight modifications are done to the original functionality. We now
> warn (instead of crash) if the anon test fails, and we leverage the
> vm_normal_folio function instead of page->folio, this should be
> functionally equivalent.
>
> No other functional changes intended.
>
> This patch is heavily based off work done by Lance Yang, but modified to
> deal with conflicts and feedback received during the review cycle [1].
>
> [1] https://lore.kernel.org/all/20251008043748.45554-1-lance.yang@linux.dev/
> Suggested-by: David Hildenbrand <david@kernel.org>
> Signed-off-by: Nico Pache <npache@redhat.com>
> ---
>  mm/khugepaged.c | 295 +++++++++++++++++++++++++-----------------------
>  1 file changed, 155 insertions(+), 140 deletions(-)
>
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 676f75773a6c..c4ea2dc1591b 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -63,6 +63,12 @@ enum scan_result {
>         SCAN_PAGE_DIRTY_OR_WRITEBACK,
>  };
>
> +enum pte_check_result {
> +       PTE_CHECK_SUCCEED,
> +       PTE_CHECK_FAIL,
> +       PTE_CHECK_CONTINUE,
> +};
> +
>  #define CREATE_TRACE_POINTS
>  #include <trace/events/huge_memory.h>
>
> @@ -117,6 +123,20 @@ struct collapse_control {
>         DECLARE_BITMAP(mthp_present_ptes, MAX_PTRS_PER_PTE);
>  };
>
> +struct pte_check_context {
> +       struct collapse_control *cc;
> +       struct vm_area_struct *vma;
> +       unsigned int order;
> +       struct folio *folio;
> +       int none_or_zero;
> +       int shared;
> +       int unmapped;
> +       enum scan_result result;
> +       unsigned int max_ptes_none;
> +       unsigned int max_ptes_swap;
> +       unsigned int max_ptes_shared;
> +};
> +
>  /**
>   * struct khugepaged_scan - cursor for scanning
>   * @mm_head: the head of the mm list to scan
> @@ -700,74 +720,130 @@ static void count_collapse_event(unsigned int order, enum vm_event_item vm_event
>         count_mthp_stat(order, mthp_event);
>  }
>
> +/*
> + * pte_check_fail() - A simple helper to set the pte_check_context result and
> + * return PTE_CHECK_FAIL.
> + */
> +static enum pte_check_result pte_check_fail(struct pte_check_context *ctx,
> +               enum scan_result result)
> +{
> +       ctx->result = result;
> +       return PTE_CHECK_FAIL;
> +}
> +
> +/*
> + * collapse_check_pte() - Check if a PTE is suitable for collapse
> + *
> + * Check if a PTE is suitable for collapse based on the following criteria:
> + * - max_pte_* values are not exceeded
> + * - uffd is not active
> + * - lazyfree properties are not present
> + * - only anonymous pages are present
> + *
> + * a helper struct pte_check_context is used to pass and store relevant
> + * information between the collapse_check_pte() function and the caller.
> + *
> + * Return: PTE_CHECK_SUCCEED if the PTE is suitable for collapse,
> + *         PTE_CHECK_FAIL if the PTE is not suitable for collapse,
> + *         PTE_CHECK_CONTINUE if the scan should continue to check the next PTE.
> + */
> +static enum pte_check_result collapse_check_pte(pte_t pteval,
> +               unsigned long addr, struct pte_check_context *ctx)
> +{
> +       if (pte_none_or_zero(pteval)) {
> +               if (++ctx->none_or_zero > ctx->max_ptes_none) {
> +                       count_collapse_event(ctx->order, THP_SCAN_EXCEED_NONE_PTE,
> +                                       MTHP_STAT_COLLAPSE_EXCEED_NONE);
> +                       return pte_check_fail(ctx, SCAN_EXCEED_NONE_PTE);
> +               }
> +               return PTE_CHECK_CONTINUE;
> +       }
> +       if (!pte_present(pteval)) {
> +               if (ctx->unmapped == -1)
> +                       return pte_check_fail(ctx, SCAN_PTE_NON_PRESENT);
> +               if (++ctx->unmapped > ctx->max_ptes_swap) {
> +                       count_collapse_event(ctx->order, THP_SCAN_EXCEED_SWAP_PTE,
> +                                       MTHP_STAT_COLLAPSE_EXCEED_SWAP);
> +                       return pte_check_fail(ctx, SCAN_EXCEED_SWAP_PTE);
> +               }
> +               if (pte_swp_uffd_wp_any(pteval))
> +                       return pte_check_fail(ctx, SCAN_PTE_UFFD_WP);
> +               return PTE_CHECK_CONTINUE;
> +       }
> +       /*
> +        * Don't collapse if any of the small PTEs are armed with uffd
> +        * write protection. Marking the new huge pmd as write protected
> +        * could bring userfault messages that fall outside of the
> +        * registered range.
> +        */
> +       if (pte_uffd_wp(pteval))
> +               return pte_check_fail(ctx, SCAN_PTE_UFFD_WP);
> +
> +       ctx->folio = vm_normal_folio(ctx->vma, addr, pteval);
> +       if (unlikely(!ctx->folio) || unlikely(folio_is_zone_device(ctx->folio)))
> +               return pte_check_fail(ctx, SCAN_PAGE_NULL);

Sashiko brought up that by converting this to vm_normal_folio we are potentially
causing checks on incorrect metadata.

This claim is invalid due to ZONE_DEVICE pages being properly initialized.

> +
> +       /*
> +        * If the vma has the VM_DROPPABLE flag, the collapse will
> +        * preserve the lazyfree property without needing to skip.
> +        */
> +       if (ctx->cc->is_khugepaged && !(ctx->vma->vm_flags & VM_DROPPABLE) &&
> +           folio_test_lazyfree(ctx->folio) && !pte_dirty(pteval))
> +               return pte_check_fail(ctx, SCAN_PAGE_LAZYFREE);
> +
> +       if (folio_maybe_mapped_shared(ctx->folio)) {
> +               /*
> +                * TODO: Support shared pages without leading to further
> +                * mTHP collapses. Currently bringing in new pages via
> +                * shared may cause a future higher order collapse on a
> +                * rescan of the same range.
> +                */
> +               if (++ctx->shared > ctx->max_ptes_shared) {
> +                       count_collapse_event(ctx->order, THP_SCAN_EXCEED_SHARED_PTE,
> +                                       MTHP_STAT_COLLAPSE_EXCEED_SHARED);
> +                       return pte_check_fail(ctx, SCAN_EXCEED_SHARED_PTE);
> +               }
> +       }
> +
> +       if (!folio_test_anon(ctx->folio)) {
> +               VM_WARN_ON_FOLIO(!folio_test_anon(ctx->folio), ctx->folio);
> +               return pte_check_fail(ctx, SCAN_PAGE_ANON);
> +       }
> +       return PTE_CHECK_SUCCEED;
> +}
> +
>  static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
>                 unsigned long start_addr, pte_t *pte, struct collapse_control *cc,
>                 unsigned int order, struct list_head *compound_pagelist)
>  {
> -       const unsigned int max_ptes_none = collapse_max_ptes_none(cc, vma, order);
> -       const unsigned int max_ptes_shared = collapse_max_ptes_shared(cc, order);
>         const unsigned long nr_pages = 1UL << order;
> -       struct page *page = NULL;
>         struct folio *folio = NULL;
>         unsigned long addr = start_addr;
> -       pte_t *_pte;
> -       int none_or_zero = 0, shared = 0, referenced = 0;
> +       pte_t *_pte, pteval;
> +       int referenced = 0;
>         enum scan_result result = SCAN_FAIL;
> +       enum pte_check_result pte_check;
> +       struct pte_check_context ctx = {
> +               .cc = cc,
> +               .vma = vma,
> +               .order = order,
> +               .unmapped = -1, /* don't check swap PTEs */
> +               .max_ptes_none = collapse_max_ptes_none(cc, vma, order),
> +               .max_ptes_shared = collapse_max_ptes_shared(cc, order),
> +       };
>
>         for (_pte = pte; _pte < pte + nr_pages;
>              _pte++, addr += PAGE_SIZE) {
> -               pte_t pteval = ptep_get(_pte);
> -               if (pte_none_or_zero(pteval)) {
> -                       if (++none_or_zero > max_ptes_none) {
> -                               result = SCAN_EXCEED_NONE_PTE;
> -                               count_collapse_event(order, THP_SCAN_EXCEED_NONE_PTE,
> -                                               MTHP_STAT_COLLAPSE_EXCEED_NONE);
> -                               goto out;
> -                       }
> -                       continue;
> -               }
> -               if (!pte_present(pteval)) {
> -                       result = SCAN_PTE_NON_PRESENT;
> -                       goto out;
> -               }
> -               if (pte_uffd_wp(pteval)) {
> -                       result = SCAN_PTE_UFFD_WP;
> -                       goto out;
> -               }
> -               page = vm_normal_page(vma, addr, pteval);
> -               if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
> -                       result = SCAN_PAGE_NULL;
> -                       goto out;
> -               }
> -
> -               folio = page_folio(page);
> -               VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio);
> -
> -               /*
> -                * If the vma has the VM_DROPPABLE flag, the collapse will
> -                * preserve the lazyfree property without needing to skip.
> -                */
> -               if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
> -                   folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
> -                       result = SCAN_PAGE_LAZYFREE;
> +               pteval = ptep_get(_pte);
> +               pte_check = collapse_check_pte(pteval, addr, &ctx);
> +               if (pte_check == PTE_CHECK_FAIL) {
> +                       result = ctx.result;
>                         goto out;

Sashiko also brought up that we only set the local folio variable in the case of
a successful check, which can lead to tracepoints using the incorrect folio.

>                 }
> +               if (pte_check == PTE_CHECK_CONTINUE)
> +                       continue;
> +               folio = ctx.folio;
>
> -               /* See collapse_scan_pmd(). */
> -               if (folio_maybe_mapped_shared(folio)) {
> -                       /*
> -                        * TODO: Support shared pages without leading to further
> -                        * mTHP collapses. Currently bringing in new pages via
> -                        * shared may cause a future higher order collapse on a
> -                        * rescan of the same range.
> -                        */
> -                       if (++shared > max_ptes_shared) {
> -                               result = SCAN_EXCEED_SHARED_PTE;
> -                               count_collapse_event(order, THP_SCAN_EXCEED_SHARED_PTE,
> -                                               MTHP_STAT_COLLAPSE_EXCEED_SHARED);
> -                               goto out;
> -                       }
> -               }
>                 /*
>                  * TODO: In some cases of partially-mapped folios, we'd actually
>                  * want to collapse.
> @@ -844,13 +920,13 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
>                 result = SCAN_LACK_REFERENCED_PAGE;
>         } else {
>                 result = SCAN_SUCCEED;
> -               trace_mm_collapse_huge_page_isolate(folio, none_or_zero,
> +               trace_mm_collapse_huge_page_isolate(folio, ctx.none_or_zero,
>                                                     referenced, result, order);
>                 return result;
>         }
>  out:
>         release_pte_pages(pte, _pte, compound_pagelist);
> -       trace_mm_collapse_huge_page_isolate(folio, none_or_zero,
> +       trace_mm_collapse_huge_page_isolate(folio, ctx.none_or_zero,
>                                             referenced, result, order);
>         return result;
>  }
> @@ -1616,24 +1692,30 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
>                 struct vm_area_struct *vma, unsigned long start_addr,
>                 bool *lock_dropped, struct collapse_control *cc)
>  {
> -       const unsigned int max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER);
> -       const unsigned int max_ptes_swap = collapse_max_ptes_swap(cc, HPAGE_PMD_ORDER);
> -       unsigned int max_ptes_none = collapse_max_ptes_none(cc, vma, HPAGE_PMD_ORDER);
>         enum tva_type tva_flags = cc->is_khugepaged ? TVA_KHUGEPAGED : TVA_FORCED_COLLAPSE;
>         pmd_t *pmd;
>         pte_t *pte, *_pte, pteval;
>         int i;
> -       int none_or_zero = 0, shared = 0, referenced = 0;
> -       enum scan_result result = SCAN_FAIL;
> -       struct page *page = NULL;
>         struct folio *folio = NULL;
> +       int referenced = 0;
> +       enum scan_result result = SCAN_FAIL;
>         unsigned long addr;
>         unsigned long enabled_orders;
>         spinlock_t *ptl;
> -       int node = NUMA_NO_NODE, unmapped = 0;
> +       int node = NUMA_NO_NODE;
> +       enum pte_check_result pte_check;
>
>         VM_BUG_ON(start_addr & ~HPAGE_PMD_MASK);
>
> +       struct pte_check_context ctx = {
> +               .cc = cc,
> +               .vma = vma,
> +               .order = HPAGE_PMD_ORDER,
> +               .max_ptes_none = collapse_max_ptes_none(cc, vma, HPAGE_PMD_ORDER),
> +               .max_ptes_swap = collapse_max_ptes_swap(cc, HPAGE_PMD_ORDER),
> +               .max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER),
> +       };
> +
>         result = find_pmd_or_thp_or_none(mm, start_addr, &pmd);
>         if (result != SCAN_SUCCEED) {
>                 cc->progress++;
> @@ -1649,7 +1731,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
>          * scan all pages to populate the bitmap for mTHP collapse.
>          */
>         if (enabled_orders != BIT(HPAGE_PMD_ORDER))
> -               max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT;
> +               ctx.max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT;
>
>         pte = pte_offset_map_lock(mm, pmd, start_addr, &ptl);
>         if (!pte) {
> @@ -1665,81 +1747,14 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
>
>                 cc->progress++;
>
> -               if (pte_none_or_zero(pteval)) {
> -                       if (++none_or_zero > max_ptes_none) {
> -                               result = SCAN_EXCEED_NONE_PTE;
> -                               count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_NONE_PTE,
> -                                               MTHP_STAT_COLLAPSE_EXCEED_NONE);
> -                               goto out_unmap;
> -                       }
> -                       continue;
> -               }
> -               if (!pte_present(pteval)) {
> -                       if (++unmapped > max_ptes_swap) {
> -                               result = SCAN_EXCEED_SWAP_PTE;
> -                               count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_SWAP_PTE,
> -                                               MTHP_STAT_COLLAPSE_EXCEED_SWAP);
> -                               goto out_unmap;
> -                       }
> -                       /*
> -                        * Always be strict with uffd-wp
> -                        * enabled swap entries.  Please see
> -                        * comment below for pte_uffd_wp().
> -                        */
> -                       if (pte_swp_uffd_wp_any(pteval)) {
> -                               result = SCAN_PTE_UFFD_WP;
> -                               goto out_unmap;
> -                       }
> -                       continue;
> -               }
> -               if (pte_uffd_wp(pteval)) {
> -                       /*
> -                        * Don't collapse the page if any of the small
> -                        * PTEs are armed with uffd write protection.
> -                        * Here we can also mark the new huge pmd as
> -                        * write protected if any of the small ones is
> -                        * marked but that could bring unknown
> -                        * userfault messages that falls outside of
> -                        * the registered range.  So, just be simple.
> -                        */
> -                       result = SCAN_PTE_UFFD_WP;
> -                       goto out_unmap;
> -               }
> -
> -               page = vm_normal_page(vma, addr, pteval);
> -               if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
> -                       result = SCAN_PAGE_NULL;
> -                       goto out_unmap;
> -               }
> -               folio = page_folio(page);
> -
> -               /*
> -                * If the vma has the VM_DROPPABLE flag, the collapse will
> -                * preserve the lazyfree property without needing to skip.
> -                */
> -               if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
> -                   folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
> -                       result = SCAN_PAGE_LAZYFREE;
> -                       goto out_unmap;
> -               }
> -
> -               if (!folio_test_anon(folio)) {
> -                       result = SCAN_PAGE_ANON;
> +               pte_check = collapse_check_pte(pteval, addr, &ctx);
> +               if (pte_check == PTE_CHECK_FAIL) {
> +                       result = ctx.result;
>                         goto out_unmap;

Same here.

The right approach is to always set the local folio variable after
collapse_check_pte.
Here is the fixup

commit 96f7896c359973a759578d2bf300138c31b494a4
Author: Nico Pache <npache@redhat.com>
Date:   Mon Jul 6 10:32:24 2026 -0600

    fixup: always set the local folio after collapse_check_pte()

    If we dont set the local folio to the result from collapse_check_pte()
    we can end up with cases that the goto out will result in pointing to a
    stale folio from the last successful PTE check.

    Signed-off-by: Nico Pache <npache@redhat.com>

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 8dd902e546b5..1cc4d864f0e3 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -836,13 +836,13 @@ static enum scan_result
__collapse_huge_page_isolate(struct vm_area_struct *vma,
 	     _pte++, addr += PAGE_SIZE) {
 		pteval = ptep_get(_pte);
 		pte_check = collapse_check_pte(pteval, addr, &ctx);
+		folio = ctx.folio;
 		if (pte_check == PTE_CHECK_FAIL) {
 			result = ctx.result;
 			goto out;
 		}
 		if (pte_check == PTE_CHECK_CONTINUE)
 			continue;
-		folio = ctx.folio;

 		/*
 		 * TODO: In some cases of partially-mapped folios, we'd actually


>                 }
> -
> -               /*
> -                * We treat a single page as shared if any part of the THP
> -                * is shared.
> -                */
> -               if (folio_maybe_mapped_shared(folio)) {
> -                       if (++shared > max_ptes_shared) {
> -                               result = SCAN_EXCEED_SHARED_PTE;
> -                               count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_SHARED_PTE,
> -                                               MTHP_STAT_COLLAPSE_EXCEED_SHARED);
> -                               goto out_unmap;
> -                       }
> -               }
> +               if (pte_check == PTE_CHECK_CONTINUE)
> +                       continue;
> +               folio = ctx.folio;
>
>                 /* Set bit for occupied pages */
>                 __set_bit(i, cc->mthp_present_ptes);
> @@ -1781,7 +1796,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
>         }
>         if (cc->is_khugepaged &&
>                    (!referenced ||
> -                   (unmapped && referenced < HPAGE_PMD_NR / 2))) {
> +                   (ctx.unmapped && referenced < HPAGE_PMD_NR / 2))) {
>                 result = SCAN_LACK_REFERENCED_PAGE;
>         } else {
>                 result = SCAN_SUCCEED;
> @@ -1792,13 +1807,13 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
>                 /* collapse_huge_page expects the lock to be dropped before calling */
>                 mmap_read_unlock(mm);
>                 result = mthp_collapse(mm, start_addr, referenced,
> -                                      unmapped, cc, enabled_orders);
> +                                      ctx.unmapped, cc, enabled_orders);
>                 /* mmap_lock was released above, set lock_dropped */
>                 *lock_dropped = true;
>         }
>  out:
>         trace_mm_khugepaged_scan_pmd(mm, folio, referenced,
> -                                    none_or_zero, result, unmapped);
> +                                    ctx.none_or_zero, result, ctx.unmapped);
>         return result;
>  }
>
> --
> 2.54.0
>

Re: [PATCH 5/8] mm/khugepaged: Refactor the PTE state checks into a helper

[Usama Arif]

On Mon,  6 Jul 2026 09:44:52 -0600 Nico Pache <npache@redhat.com> wrote:

> For anonymous collapse, the collapse_scan_pmd() and
> __collapse_huge_page_isolate() functions share a large portion of their
> logic. These functions both check the state of the PTEs and verify the
> following:
> 	- max_pte_* values are not exceeded
> 	- uffd is not active
> 	- lazyfree properties
> 	- non-anonymous
> 
> Merge these checks into a helper collapse_check_pte() to reduce code
> duplication. We also add a helper struct for this function called
> pte_check_context which allows us to pass the required parameters in a
> clean and elegant manner.
> 
> A helper function is also introduced pte_check_fail() to provide a clean
> interface to set the pte_check_context failure results and return
> PTE_CHECK_FAIL state. This helps reduce code duplications across the new
> collapse_check_pte function.
> 
> Two slight modifications are done to the original functionality. We now
> warn (instead of crash) if the anon test fails, and we leverage the
> vm_normal_folio function instead of page->folio, this should be
> functionally equivalent.
> 
> No other functional changes intended.
> 
> This patch is heavily based off work done by Lance Yang, but modified to
> deal with conflicts and feedback received during the review cycle [1].
> 
> [1] https://lore.kernel.org/all/20251008043748.45554-1-lance.yang@linux.dev/
> Suggested-by: David Hildenbrand <david@kernel.org>
> Signed-off-by: Nico Pache <npache@redhat.com>
> ---
>  mm/khugepaged.c | 295 +++++++++++++++++++++++++-----------------------
>  1 file changed, 155 insertions(+), 140 deletions(-)
> 
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 676f75773a6c..c4ea2dc1591b 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -63,6 +63,12 @@ enum scan_result {
>  	SCAN_PAGE_DIRTY_OR_WRITEBACK,
>  };
>  
> +enum pte_check_result {
> +	PTE_CHECK_SUCCEED,
> +	PTE_CHECK_FAIL,
> +	PTE_CHECK_CONTINUE,
> +};
> +
>  #define CREATE_TRACE_POINTS
>  #include <trace/events/huge_memory.h>
>  
> @@ -117,6 +123,20 @@ struct collapse_control {
>  	DECLARE_BITMAP(mthp_present_ptes, MAX_PTRS_PER_PTE);
>  };
>  
> +struct pte_check_context {
> +	struct collapse_control *cc;
> +	struct vm_area_struct *vma;
> +	unsigned int order;
> +	struct folio *folio;
> +	int none_or_zero;
> +	int shared;
> +	int unmapped;
> +	enum scan_result result;
> +	unsigned int max_ptes_none;
> +	unsigned int max_ptes_swap;
> +	unsigned int max_ptes_shared;
> +};
> +
>  /**
>   * struct khugepaged_scan - cursor for scanning
>   * @mm_head: the head of the mm list to scan
> @@ -700,74 +720,130 @@ static void count_collapse_event(unsigned int order, enum vm_event_item vm_event
>  	count_mthp_stat(order, mthp_event);
>  }
>  
> +/*
> + * pte_check_fail() - A simple helper to set the pte_check_context result and
> + * return PTE_CHECK_FAIL.
> + */
> +static enum pte_check_result pte_check_fail(struct pte_check_context *ctx,
> +		enum scan_result result)
> +{
> +	ctx->result = result;
> +	return PTE_CHECK_FAIL;
> +}
> +
> +/*
> + * collapse_check_pte() - Check if a PTE is suitable for collapse
> + *
> + * Check if a PTE is suitable for collapse based on the following criteria:
> + * - max_pte_* values are not exceeded
> + * - uffd is not active
> + * - lazyfree properties are not present
> + * - only anonymous pages are present
> + *
> + * a helper struct pte_check_context is used to pass and store relevant
> + * information between the collapse_check_pte() function and the caller.
> + *
> + * Return: PTE_CHECK_SUCCEED if the PTE is suitable for collapse,
> + *         PTE_CHECK_FAIL if the PTE is not suitable for collapse,
> + *         PTE_CHECK_CONTINUE if the scan should continue to check the next PTE.
> + */
> +static enum pte_check_result collapse_check_pte(pte_t pteval,
> +		unsigned long addr, struct pte_check_context *ctx)
> +{
> +	if (pte_none_or_zero(pteval)) {
> +		if (++ctx->none_or_zero > ctx->max_ptes_none) {
> +			count_collapse_event(ctx->order, THP_SCAN_EXCEED_NONE_PTE,
> +					MTHP_STAT_COLLAPSE_EXCEED_NONE);
> +			return pte_check_fail(ctx, SCAN_EXCEED_NONE_PTE);
> +		}
> +		return PTE_CHECK_CONTINUE;
> +	}
> +	if (!pte_present(pteval)) {
> +		if (ctx->unmapped == -1)
> +			return pte_check_fail(ctx, SCAN_PTE_NON_PRESENT);
> +		if (++ctx->unmapped > ctx->max_ptes_swap) {
> +			count_collapse_event(ctx->order, THP_SCAN_EXCEED_SWAP_PTE,
> +					MTHP_STAT_COLLAPSE_EXCEED_SWAP);
> +			return pte_check_fail(ctx, SCAN_EXCEED_SWAP_PTE);
> +		}
> +		if (pte_swp_uffd_wp_any(pteval))
> +			return pte_check_fail(ctx, SCAN_PTE_UFFD_WP);
> +		return PTE_CHECK_CONTINUE;
> +	}
> +	/*
> +	 * Don't collapse if any of the small PTEs are armed with uffd
> +	 * write protection. Marking the new huge pmd as write protected
> +	 * could bring userfault messages that fall outside of the
> +	 * registered range.
> +	 */
> +	if (pte_uffd_wp(pteval))
> +		return pte_check_fail(ctx, SCAN_PTE_UFFD_WP);
> +
> +	ctx->folio = vm_normal_folio(ctx->vma, addr, pteval);
> +	if (unlikely(!ctx->folio) || unlikely(folio_is_zone_device(ctx->folio)))
> +		return pte_check_fail(ctx, SCAN_PAGE_NULL);
> +
> +	/*
> +	 * If the vma has the VM_DROPPABLE flag, the collapse will
> +	 * preserve the lazyfree property without needing to skip.
> +	 */
> +	if (ctx->cc->is_khugepaged && !(ctx->vma->vm_flags & VM_DROPPABLE) &&
> +	    folio_test_lazyfree(ctx->folio) && !pte_dirty(pteval))
> +		return pte_check_fail(ctx, SCAN_PAGE_LAZYFREE);
> +
> +	if (folio_maybe_mapped_shared(ctx->folio)) {
> +		/*
> +		 * TODO: Support shared pages without leading to further
> +		 * mTHP collapses. Currently bringing in new pages via
> +		 * shared may cause a future higher order collapse on a
> +		 * rescan of the same range.
> +		 */
> +		if (++ctx->shared > ctx->max_ptes_shared) {
> +			count_collapse_event(ctx->order, THP_SCAN_EXCEED_SHARED_PTE,
> +					MTHP_STAT_COLLAPSE_EXCEED_SHARED);
> +			return pte_check_fail(ctx, SCAN_EXCEED_SHARED_PTE);
> +		}
> +	}
> +
> +	if (!folio_test_anon(ctx->folio)) {
> +		VM_WARN_ON_FOLIO(!folio_test_anon(ctx->folio), ctx->folio);
> +		return pte_check_fail(ctx, SCAN_PAGE_ANON);
> +	}

The order folio_maybe_mapped_shared() and !folio_test_anon() check is flipped
from what it was. Can we keep original ordering? The commit message says
no other functional change inteded, but this could result in different collapse
even counters getting incremented.


> +	return PTE_CHECK_SUCCEED;
> +}
> +
>  static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
>  		unsigned long start_addr, pte_t *pte, struct collapse_control *cc,
>  		unsigned int order, struct list_head *compound_pagelist)
>  {
> -	const unsigned int max_ptes_none = collapse_max_ptes_none(cc, vma, order);
> -	const unsigned int max_ptes_shared = collapse_max_ptes_shared(cc, order);
>  	const unsigned long nr_pages = 1UL << order;
> -	struct page *page = NULL;
>  	struct folio *folio = NULL;
>  	unsigned long addr = start_addr;
> -	pte_t *_pte;
> -	int none_or_zero = 0, shared = 0, referenced = 0;
> +	pte_t *_pte, pteval;
> +	int referenced = 0;
>  	enum scan_result result = SCAN_FAIL;
> +	enum pte_check_result pte_check;
> +	struct pte_check_context ctx = {
> +		.cc = cc,
> +		.vma = vma,
> +		.order = order,
> +		.unmapped = -1, /* don't check swap PTEs */
> +		.max_ptes_none = collapse_max_ptes_none(cc, vma, order),
> +		.max_ptes_shared = collapse_max_ptes_shared(cc, order),
> +	};
>  
>  	for (_pte = pte; _pte < pte + nr_pages;
>  	     _pte++, addr += PAGE_SIZE) {
> -		pte_t pteval = ptep_get(_pte);
> -		if (pte_none_or_zero(pteval)) {
> -			if (++none_or_zero > max_ptes_none) {
> -				result = SCAN_EXCEED_NONE_PTE;
> -				count_collapse_event(order, THP_SCAN_EXCEED_NONE_PTE,
> -						MTHP_STAT_COLLAPSE_EXCEED_NONE);
> -				goto out;
> -			}
> -			continue;
> -		}
> -		if (!pte_present(pteval)) {
> -			result = SCAN_PTE_NON_PRESENT;
> -			goto out;
> -		}
> -		if (pte_uffd_wp(pteval)) {
> -			result = SCAN_PTE_UFFD_WP;
> -			goto out;
> -		}
> -		page = vm_normal_page(vma, addr, pteval);
> -		if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
> -			result = SCAN_PAGE_NULL;
> -			goto out;
> -		}
> -
> -		folio = page_folio(page);
> -		VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio);
> -
> -		/*
> -		 * If the vma has the VM_DROPPABLE flag, the collapse will
> -		 * preserve the lazyfree property without needing to skip.
> -		 */
> -		if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
> -		    folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
> -			result = SCAN_PAGE_LAZYFREE;
> +		pteval = ptep_get(_pte);
> +		pte_check = collapse_check_pte(pteval, addr, &ctx);
> +		if (pte_check == PTE_CHECK_FAIL) {
> +			result = ctx.result;
>  			goto out;
>  		}
> +		if (pte_check == PTE_CHECK_CONTINUE)
> +			continue;
> +		folio = ctx.folio;
>  
> -		/* See collapse_scan_pmd(). */
> -		if (folio_maybe_mapped_shared(folio)) {
> -			/*
> -			 * TODO: Support shared pages without leading to further
> -			 * mTHP collapses. Currently bringing in new pages via
> -			 * shared may cause a future higher order collapse on a
> -			 * rescan of the same range.
> -			 */
> -			if (++shared > max_ptes_shared) {
> -				result = SCAN_EXCEED_SHARED_PTE;
> -				count_collapse_event(order, THP_SCAN_EXCEED_SHARED_PTE,
> -						MTHP_STAT_COLLAPSE_EXCEED_SHARED);
> -				goto out;
> -			}
> -		}
>  		/*
>  		 * TODO: In some cases of partially-mapped folios, we'd actually
>  		 * want to collapse.
> @@ -844,13 +920,13 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
>  		result = SCAN_LACK_REFERENCED_PAGE;
>  	} else {
>  		result = SCAN_SUCCEED;
> -		trace_mm_collapse_huge_page_isolate(folio, none_or_zero,
> +		trace_mm_collapse_huge_page_isolate(folio, ctx.none_or_zero,
>  						    referenced, result, order);
>  		return result;
>  	}
>  out:
>  	release_pte_pages(pte, _pte, compound_pagelist);
> -	trace_mm_collapse_huge_page_isolate(folio, none_or_zero,
> +	trace_mm_collapse_huge_page_isolate(folio, ctx.none_or_zero,
>  					    referenced, result, order);
>  	return result;
>  }
> @@ -1616,24 +1692,30 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
>  		struct vm_area_struct *vma, unsigned long start_addr,
>  		bool *lock_dropped, struct collapse_control *cc)
>  {
> -	const unsigned int max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER);
> -	const unsigned int max_ptes_swap = collapse_max_ptes_swap(cc, HPAGE_PMD_ORDER);
> -	unsigned int max_ptes_none = collapse_max_ptes_none(cc, vma, HPAGE_PMD_ORDER);
>  	enum tva_type tva_flags = cc->is_khugepaged ? TVA_KHUGEPAGED : TVA_FORCED_COLLAPSE;
>  	pmd_t *pmd;
>  	pte_t *pte, *_pte, pteval;
>  	int i;
> -	int none_or_zero = 0, shared = 0, referenced = 0;
> -	enum scan_result result = SCAN_FAIL;
> -	struct page *page = NULL;
>  	struct folio *folio = NULL;
> +	int referenced = 0;
> +	enum scan_result result = SCAN_FAIL;
>  	unsigned long addr;
>  	unsigned long enabled_orders;
>  	spinlock_t *ptl;
> -	int node = NUMA_NO_NODE, unmapped = 0;
> +	int node = NUMA_NO_NODE;
> +	enum pte_check_result pte_check;
>  
>  	VM_BUG_ON(start_addr & ~HPAGE_PMD_MASK);
>  
> +	struct pte_check_context ctx = {
> +		.cc = cc,
> +		.vma = vma,
> +		.order = HPAGE_PMD_ORDER,
> +		.max_ptes_none = collapse_max_ptes_none(cc, vma, HPAGE_PMD_ORDER),
> +		.max_ptes_swap = collapse_max_ptes_swap(cc, HPAGE_PMD_ORDER),
> +		.max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER),
> +	};
> +
>  	result = find_pmd_or_thp_or_none(mm, start_addr, &pmd);
>  	if (result != SCAN_SUCCEED) {
>  		cc->progress++;
> @@ -1649,7 +1731,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
>  	 * scan all pages to populate the bitmap for mTHP collapse.
>  	 */
>  	if (enabled_orders != BIT(HPAGE_PMD_ORDER))
> -		max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT;
> +		ctx.max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT;
>  
>  	pte = pte_offset_map_lock(mm, pmd, start_addr, &ptl);
>  	if (!pte) {
> @@ -1665,81 +1747,14 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
>  
>  		cc->progress++;
>  
> -		if (pte_none_or_zero(pteval)) {
> -			if (++none_or_zero > max_ptes_none) {
> -				result = SCAN_EXCEED_NONE_PTE;
> -				count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_NONE_PTE,
> -						MTHP_STAT_COLLAPSE_EXCEED_NONE);
> -				goto out_unmap;
> -			}
> -			continue;
> -		}
> -		if (!pte_present(pteval)) {
> -			if (++unmapped > max_ptes_swap) {
> -				result = SCAN_EXCEED_SWAP_PTE;
> -				count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_SWAP_PTE,
> -						MTHP_STAT_COLLAPSE_EXCEED_SWAP);
> -				goto out_unmap;
> -			}
> -			/*
> -			 * Always be strict with uffd-wp
> -			 * enabled swap entries.  Please see
> -			 * comment below for pte_uffd_wp().
> -			 */
> -			if (pte_swp_uffd_wp_any(pteval)) {
> -				result = SCAN_PTE_UFFD_WP;
> -				goto out_unmap;
> -			}
> -			continue;
> -		}
> -		if (pte_uffd_wp(pteval)) {
> -			/*
> -			 * Don't collapse the page if any of the small
> -			 * PTEs are armed with uffd write protection.
> -			 * Here we can also mark the new huge pmd as
> -			 * write protected if any of the small ones is
> -			 * marked but that could bring unknown
> -			 * userfault messages that falls outside of
> -			 * the registered range.  So, just be simple.
> -			 */
> -			result = SCAN_PTE_UFFD_WP;
> -			goto out_unmap;
> -		}
> -
> -		page = vm_normal_page(vma, addr, pteval);
> -		if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
> -			result = SCAN_PAGE_NULL;
> -			goto out_unmap;
> -		}
> -		folio = page_folio(page);
> -
> -		/*
> -		 * If the vma has the VM_DROPPABLE flag, the collapse will
> -		 * preserve the lazyfree property without needing to skip.
> -		 */
> -		if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
> -		    folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
> -			result = SCAN_PAGE_LAZYFREE;
> -			goto out_unmap;
> -		}
> -
> -		if (!folio_test_anon(folio)) {
> -			result = SCAN_PAGE_ANON;
> +		pte_check = collapse_check_pte(pteval, addr, &ctx);
> +		if (pte_check == PTE_CHECK_FAIL) {
> +			result = ctx.result;
>  			goto out_unmap;
>  		}
> -
> -		/*
> -		 * We treat a single page as shared if any part of the THP
> -		 * is shared.
> -		 */
> -		if (folio_maybe_mapped_shared(folio)) {
> -			if (++shared > max_ptes_shared) {
> -				result = SCAN_EXCEED_SHARED_PTE;
> -				count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_SHARED_PTE,
> -						MTHP_STAT_COLLAPSE_EXCEED_SHARED);
> -				goto out_unmap;
> -			}
> -		}
> +		if (pte_check == PTE_CHECK_CONTINUE)
> +			continue;
> +		folio = ctx.folio;
>  
>  		/* Set bit for occupied pages */
>  		__set_bit(i, cc->mthp_present_ptes);
> @@ -1781,7 +1796,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
>  	}
>  	if (cc->is_khugepaged &&
>  		   (!referenced ||
> -		    (unmapped && referenced < HPAGE_PMD_NR / 2))) {
> +		    (ctx.unmapped && referenced < HPAGE_PMD_NR / 2))) {
>  		result = SCAN_LACK_REFERENCED_PAGE;
>  	} else {
>  		result = SCAN_SUCCEED;
> @@ -1792,13 +1807,13 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
>  		/* collapse_huge_page expects the lock to be dropped before calling */
>  		mmap_read_unlock(mm);
>  		result = mthp_collapse(mm, start_addr, referenced,
> -				       unmapped, cc, enabled_orders);
> +				       ctx.unmapped, cc, enabled_orders);
>  		/* mmap_lock was released above, set lock_dropped */
>  		*lock_dropped = true;
>  	}
>  out:
>  	trace_mm_khugepaged_scan_pmd(mm, folio, referenced,
> -				     none_or_zero, result, unmapped);
> +				     ctx.none_or_zero, result, ctx.unmapped);
>  	return result;
>  }
>  
> -- 
> 2.54.0
> 
> 

Re: [PATCH 5/8] mm/khugepaged: Refactor the PTE state checks into a helper

[Nico Pache]

On Mon, Jul 6, 2026 at 11:49 AM Usama Arif <usama.arif@linux.dev> wrote:
>
> On Mon,  6 Jul 2026 09:44:52 -0600 Nico Pache <npache@redhat.com> wrote:
>
> > For anonymous collapse, the collapse_scan_pmd() and
> > __collapse_huge_page_isolate() functions share a large portion of their
> > logic. These functions both check the state of the PTEs and verify the
> > following:
> >       - max_pte_* values are not exceeded
> >       - uffd is not active
> >       - lazyfree properties
> >       - non-anonymous
> >
> > Merge these checks into a helper collapse_check_pte() to reduce code
> > duplication. We also add a helper struct for this function called
> > pte_check_context which allows us to pass the required parameters in a
> > clean and elegant manner.
> >
> > A helper function is also introduced pte_check_fail() to provide a clean
> > interface to set the pte_check_context failure results and return
> > PTE_CHECK_FAIL state. This helps reduce code duplications across the new
> > collapse_check_pte function.
> >
> > Two slight modifications are done to the original functionality. We now
> > warn (instead of crash) if the anon test fails, and we leverage the
> > vm_normal_folio function instead of page->folio, this should be
> > functionally equivalent.
> >
> > No other functional changes intended.
> >
> > This patch is heavily based off work done by Lance Yang, but modified to
> > deal with conflicts and feedback received during the review cycle [1].
> >
> > [1] https://lore.kernel.org/all/20251008043748.45554-1-lance.yang@linux.dev/
> > Suggested-by: David Hildenbrand <david@kernel.org>
> > Signed-off-by: Nico Pache <npache@redhat.com>
> > ---
> >  mm/khugepaged.c | 295 +++++++++++++++++++++++++-----------------------
> >  1 file changed, 155 insertions(+), 140 deletions(-)
> >
> > diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> > index 676f75773a6c..c4ea2dc1591b 100644
> > --- a/mm/khugepaged.c
> > +++ b/mm/khugepaged.c
> > @@ -63,6 +63,12 @@ enum scan_result {
> >       SCAN_PAGE_DIRTY_OR_WRITEBACK,
> >  };
> >
> > +enum pte_check_result {
> > +     PTE_CHECK_SUCCEED,
> > +     PTE_CHECK_FAIL,
> > +     PTE_CHECK_CONTINUE,
> > +};
> > +
> >  #define CREATE_TRACE_POINTS
> >  #include <trace/events/huge_memory.h>
> >
> > @@ -117,6 +123,20 @@ struct collapse_control {
> >       DECLARE_BITMAP(mthp_present_ptes, MAX_PTRS_PER_PTE);
> >  };
> >
> > +struct pte_check_context {
> > +     struct collapse_control *cc;
> > +     struct vm_area_struct *vma;
> > +     unsigned int order;
> > +     struct folio *folio;
> > +     int none_or_zero;
> > +     int shared;
> > +     int unmapped;
> > +     enum scan_result result;
> > +     unsigned int max_ptes_none;
> > +     unsigned int max_ptes_swap;
> > +     unsigned int max_ptes_shared;
> > +};
> > +
> >  /**
> >   * struct khugepaged_scan - cursor for scanning
> >   * @mm_head: the head of the mm list to scan
> > @@ -700,74 +720,130 @@ static void count_collapse_event(unsigned int order, enum vm_event_item vm_event
> >       count_mthp_stat(order, mthp_event);
> >  }
> >
> > +/*
> > + * pte_check_fail() - A simple helper to set the pte_check_context result and
> > + * return PTE_CHECK_FAIL.
> > + */
> > +static enum pte_check_result pte_check_fail(struct pte_check_context *ctx,
> > +             enum scan_result result)
> > +{
> > +     ctx->result = result;
> > +     return PTE_CHECK_FAIL;
> > +}
> > +
> > +/*
> > + * collapse_check_pte() - Check if a PTE is suitable for collapse
> > + *
> > + * Check if a PTE is suitable for collapse based on the following criteria:
> > + * - max_pte_* values are not exceeded
> > + * - uffd is not active
> > + * - lazyfree properties are not present
> > + * - only anonymous pages are present
> > + *
> > + * a helper struct pte_check_context is used to pass and store relevant
> > + * information between the collapse_check_pte() function and the caller.
> > + *
> > + * Return: PTE_CHECK_SUCCEED if the PTE is suitable for collapse,
> > + *         PTE_CHECK_FAIL if the PTE is not suitable for collapse,
> > + *         PTE_CHECK_CONTINUE if the scan should continue to check the next PTE.
> > + */
> > +static enum pte_check_result collapse_check_pte(pte_t pteval,
> > +             unsigned long addr, struct pte_check_context *ctx)
> > +{
> > +     if (pte_none_or_zero(pteval)) {
> > +             if (++ctx->none_or_zero > ctx->max_ptes_none) {
> > +                     count_collapse_event(ctx->order, THP_SCAN_EXCEED_NONE_PTE,
> > +                                     MTHP_STAT_COLLAPSE_EXCEED_NONE);
> > +                     return pte_check_fail(ctx, SCAN_EXCEED_NONE_PTE);
> > +             }
> > +             return PTE_CHECK_CONTINUE;
> > +     }
> > +     if (!pte_present(pteval)) {
> > +             if (ctx->unmapped == -1)
> > +                     return pte_check_fail(ctx, SCAN_PTE_NON_PRESENT);
> > +             if (++ctx->unmapped > ctx->max_ptes_swap) {
> > +                     count_collapse_event(ctx->order, THP_SCAN_EXCEED_SWAP_PTE,
> > +                                     MTHP_STAT_COLLAPSE_EXCEED_SWAP);
> > +                     return pte_check_fail(ctx, SCAN_EXCEED_SWAP_PTE);
> > +             }
> > +             if (pte_swp_uffd_wp_any(pteval))
> > +                     return pte_check_fail(ctx, SCAN_PTE_UFFD_WP);
> > +             return PTE_CHECK_CONTINUE;
> > +     }
> > +     /*
> > +      * Don't collapse if any of the small PTEs are armed with uffd
> > +      * write protection. Marking the new huge pmd as write protected
> > +      * could bring userfault messages that fall outside of the
> > +      * registered range.
> > +      */
> > +     if (pte_uffd_wp(pteval))
> > +             return pte_check_fail(ctx, SCAN_PTE_UFFD_WP);
> > +
> > +     ctx->folio = vm_normal_folio(ctx->vma, addr, pteval);
> > +     if (unlikely(!ctx->folio) || unlikely(folio_is_zone_device(ctx->folio)))
> > +             return pte_check_fail(ctx, SCAN_PAGE_NULL);
> > +
> > +     /*
> > +      * If the vma has the VM_DROPPABLE flag, the collapse will
> > +      * preserve the lazyfree property without needing to skip.
> > +      */
> > +     if (ctx->cc->is_khugepaged && !(ctx->vma->vm_flags & VM_DROPPABLE) &&
> > +         folio_test_lazyfree(ctx->folio) && !pte_dirty(pteval))
> > +             return pte_check_fail(ctx, SCAN_PAGE_LAZYFREE);
> > +
> > +     if (folio_maybe_mapped_shared(ctx->folio)) {
> > +             /*
> > +              * TODO: Support shared pages without leading to further
> > +              * mTHP collapses. Currently bringing in new pages via
> > +              * shared may cause a future higher order collapse on a
> > +              * rescan of the same range.
> > +              */
> > +             if (++ctx->shared > ctx->max_ptes_shared) {
> > +                     count_collapse_event(ctx->order, THP_SCAN_EXCEED_SHARED_PTE,
> > +                                     MTHP_STAT_COLLAPSE_EXCEED_SHARED);
> > +                     return pte_check_fail(ctx, SCAN_EXCEED_SHARED_PTE);
> > +             }
> > +     }
> > +
> > +     if (!folio_test_anon(ctx->folio)) {
> > +             VM_WARN_ON_FOLIO(!folio_test_anon(ctx->folio), ctx->folio);
> > +             return pte_check_fail(ctx, SCAN_PAGE_ANON);
> > +     }
>
> The order folio_maybe_mapped_shared() and !folio_test_anon() check is flipped
> from what it was. Can we keep original ordering? The commit message says
> no other functional change inteded, but this could result in different collapse
> even counters getting incremented.

Good catch, I didn't realize i reordered it.

It might make even more sense to move it to right after we get the
folio. All other checks are pointless are we somehow landed on a
!(anon) page.

Does that sound better?

>
>
> > +     return PTE_CHECK_SUCCEED;
> > +}
> > +
> >  static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
> >               unsigned long start_addr, pte_t *pte, struct collapse_control *cc,
> >               unsigned int order, struct list_head *compound_pagelist)
> >  {
> > -     const unsigned int max_ptes_none = collapse_max_ptes_none(cc, vma, order);
> > -     const unsigned int max_ptes_shared = collapse_max_ptes_shared(cc, order);
> >       const unsigned long nr_pages = 1UL << order;
> > -     struct page *page = NULL;
> >       struct folio *folio = NULL;
> >       unsigned long addr = start_addr;
> > -     pte_t *_pte;
> > -     int none_or_zero = 0, shared = 0, referenced = 0;
> > +     pte_t *_pte, pteval;
> > +     int referenced = 0;
> >       enum scan_result result = SCAN_FAIL;
> > +     enum pte_check_result pte_check;
> > +     struct pte_check_context ctx = {
> > +             .cc = cc,
> > +             .vma = vma,
> > +             .order = order,
> > +             .unmapped = -1, /* don't check swap PTEs */
> > +             .max_ptes_none = collapse_max_ptes_none(cc, vma, order),
> > +             .max_ptes_shared = collapse_max_ptes_shared(cc, order),
> > +     };
> >
> >       for (_pte = pte; _pte < pte + nr_pages;
> >            _pte++, addr += PAGE_SIZE) {
> > -             pte_t pteval = ptep_get(_pte);
> > -             if (pte_none_or_zero(pteval)) {
> > -                     if (++none_or_zero > max_ptes_none) {
> > -                             result = SCAN_EXCEED_NONE_PTE;
> > -                             count_collapse_event(order, THP_SCAN_EXCEED_NONE_PTE,
> > -                                             MTHP_STAT_COLLAPSE_EXCEED_NONE);
> > -                             goto out;
> > -                     }
> > -                     continue;
> > -             }
> > -             if (!pte_present(pteval)) {
> > -                     result = SCAN_PTE_NON_PRESENT;
> > -                     goto out;
> > -             }
> > -             if (pte_uffd_wp(pteval)) {
> > -                     result = SCAN_PTE_UFFD_WP;
> > -                     goto out;
> > -             }
> > -             page = vm_normal_page(vma, addr, pteval);
> > -             if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
> > -                     result = SCAN_PAGE_NULL;
> > -                     goto out;
> > -             }
> > -
> > -             folio = page_folio(page);
> > -             VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio);
> > -
> > -             /*
> > -              * If the vma has the VM_DROPPABLE flag, the collapse will
> > -              * preserve the lazyfree property without needing to skip.
> > -              */
> > -             if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
> > -                 folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
> > -                     result = SCAN_PAGE_LAZYFREE;
> > +             pteval = ptep_get(_pte);
> > +             pte_check = collapse_check_pte(pteval, addr, &ctx);
> > +             if (pte_check == PTE_CHECK_FAIL) {
> > +                     result = ctx.result;
> >                       goto out;
> >               }
> > +             if (pte_check == PTE_CHECK_CONTINUE)
> > +                     continue;
> > +             folio = ctx.folio;
> >
> > -             /* See collapse_scan_pmd(). */
> > -             if (folio_maybe_mapped_shared(folio)) {
> > -                     /*
> > -                      * TODO: Support shared pages without leading to further
> > -                      * mTHP collapses. Currently bringing in new pages via
> > -                      * shared may cause a future higher order collapse on a
> > -                      * rescan of the same range.
> > -                      */
> > -                     if (++shared > max_ptes_shared) {
> > -                             result = SCAN_EXCEED_SHARED_PTE;
> > -                             count_collapse_event(order, THP_SCAN_EXCEED_SHARED_PTE,
> > -                                             MTHP_STAT_COLLAPSE_EXCEED_SHARED);
> > -                             goto out;
> > -                     }
> > -             }
> >               /*
> >                * TODO: In some cases of partially-mapped folios, we'd actually
> >                * want to collapse.
> > @@ -844,13 +920,13 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
> >               result = SCAN_LACK_REFERENCED_PAGE;
> >       } else {
> >               result = SCAN_SUCCEED;
> > -             trace_mm_collapse_huge_page_isolate(folio, none_or_zero,
> > +             trace_mm_collapse_huge_page_isolate(folio, ctx.none_or_zero,
> >                                                   referenced, result, order);
> >               return result;
> >       }
> >  out:
> >       release_pte_pages(pte, _pte, compound_pagelist);
> > -     trace_mm_collapse_huge_page_isolate(folio, none_or_zero,
> > +     trace_mm_collapse_huge_page_isolate(folio, ctx.none_or_zero,
> >                                           referenced, result, order);
> >       return result;
> >  }
> > @@ -1616,24 +1692,30 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
> >               struct vm_area_struct *vma, unsigned long start_addr,
> >               bool *lock_dropped, struct collapse_control *cc)
> >  {
> > -     const unsigned int max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER);
> > -     const unsigned int max_ptes_swap = collapse_max_ptes_swap(cc, HPAGE_PMD_ORDER);
> > -     unsigned int max_ptes_none = collapse_max_ptes_none(cc, vma, HPAGE_PMD_ORDER);
> >       enum tva_type tva_flags = cc->is_khugepaged ? TVA_KHUGEPAGED : TVA_FORCED_COLLAPSE;
> >       pmd_t *pmd;
> >       pte_t *pte, *_pte, pteval;
> >       int i;
> > -     int none_or_zero = 0, shared = 0, referenced = 0;
> > -     enum scan_result result = SCAN_FAIL;
> > -     struct page *page = NULL;
> >       struct folio *folio = NULL;
> > +     int referenced = 0;
> > +     enum scan_result result = SCAN_FAIL;
> >       unsigned long addr;
> >       unsigned long enabled_orders;
> >       spinlock_t *ptl;
> > -     int node = NUMA_NO_NODE, unmapped = 0;
> > +     int node = NUMA_NO_NODE;
> > +     enum pte_check_result pte_check;
> >
> >       VM_BUG_ON(start_addr & ~HPAGE_PMD_MASK);
> >
> > +     struct pte_check_context ctx = {
> > +             .cc = cc,
> > +             .vma = vma,
> > +             .order = HPAGE_PMD_ORDER,
> > +             .max_ptes_none = collapse_max_ptes_none(cc, vma, HPAGE_PMD_ORDER),
> > +             .max_ptes_swap = collapse_max_ptes_swap(cc, HPAGE_PMD_ORDER),
> > +             .max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER),
> > +     };
> > +
> >       result = find_pmd_or_thp_or_none(mm, start_addr, &pmd);
> >       if (result != SCAN_SUCCEED) {
> >               cc->progress++;
> > @@ -1649,7 +1731,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
> >        * scan all pages to populate the bitmap for mTHP collapse.
> >        */
> >       if (enabled_orders != BIT(HPAGE_PMD_ORDER))
> > -             max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT;
> > +             ctx.max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT;
> >
> >       pte = pte_offset_map_lock(mm, pmd, start_addr, &ptl);
> >       if (!pte) {
> > @@ -1665,81 +1747,14 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
> >
> >               cc->progress++;
> >
> > -             if (pte_none_or_zero(pteval)) {
> > -                     if (++none_or_zero > max_ptes_none) {
> > -                             result = SCAN_EXCEED_NONE_PTE;
> > -                             count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_NONE_PTE,
> > -                                             MTHP_STAT_COLLAPSE_EXCEED_NONE);
> > -                             goto out_unmap;
> > -                     }
> > -                     continue;
> > -             }
> > -             if (!pte_present(pteval)) {
> > -                     if (++unmapped > max_ptes_swap) {
> > -                             result = SCAN_EXCEED_SWAP_PTE;
> > -                             count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_SWAP_PTE,
> > -                                             MTHP_STAT_COLLAPSE_EXCEED_SWAP);
> > -                             goto out_unmap;
> > -                     }
> > -                     /*
> > -                      * Always be strict with uffd-wp
> > -                      * enabled swap entries.  Please see
> > -                      * comment below for pte_uffd_wp().
> > -                      */
> > -                     if (pte_swp_uffd_wp_any(pteval)) {
> > -                             result = SCAN_PTE_UFFD_WP;
> > -                             goto out_unmap;
> > -                     }
> > -                     continue;
> > -             }
> > -             if (pte_uffd_wp(pteval)) {
> > -                     /*
> > -                      * Don't collapse the page if any of the small
> > -                      * PTEs are armed with uffd write protection.
> > -                      * Here we can also mark the new huge pmd as
> > -                      * write protected if any of the small ones is
> > -                      * marked but that could bring unknown
> > -                      * userfault messages that falls outside of
> > -                      * the registered range.  So, just be simple.
> > -                      */
> > -                     result = SCAN_PTE_UFFD_WP;
> > -                     goto out_unmap;
> > -             }
> > -
> > -             page = vm_normal_page(vma, addr, pteval);
> > -             if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
> > -                     result = SCAN_PAGE_NULL;
> > -                     goto out_unmap;
> > -             }
> > -             folio = page_folio(page);
> > -
> > -             /*
> > -              * If the vma has the VM_DROPPABLE flag, the collapse will
> > -              * preserve the lazyfree property without needing to skip.
> > -              */
> > -             if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
> > -                 folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
> > -                     result = SCAN_PAGE_LAZYFREE;
> > -                     goto out_unmap;
> > -             }
> > -
> > -             if (!folio_test_anon(folio)) {
> > -                     result = SCAN_PAGE_ANON;
> > +             pte_check = collapse_check_pte(pteval, addr, &ctx);
> > +             if (pte_check == PTE_CHECK_FAIL) {
> > +                     result = ctx.result;
> >                       goto out_unmap;
> >               }
> > -
> > -             /*
> > -              * We treat a single page as shared if any part of the THP
> > -              * is shared.
> > -              */
> > -             if (folio_maybe_mapped_shared(folio)) {
> > -                     if (++shared > max_ptes_shared) {
> > -                             result = SCAN_EXCEED_SHARED_PTE;
> > -                             count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_SHARED_PTE,
> > -                                             MTHP_STAT_COLLAPSE_EXCEED_SHARED);
> > -                             goto out_unmap;
> > -                     }
> > -             }
> > +             if (pte_check == PTE_CHECK_CONTINUE)
> > +                     continue;
> > +             folio = ctx.folio;
> >
> >               /* Set bit for occupied pages */
> >               __set_bit(i, cc->mthp_present_ptes);
> > @@ -1781,7 +1796,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
> >       }
> >       if (cc->is_khugepaged &&
> >                  (!referenced ||
> > -                 (unmapped && referenced < HPAGE_PMD_NR / 2))) {
> > +                 (ctx.unmapped && referenced < HPAGE_PMD_NR / 2))) {
> >               result = SCAN_LACK_REFERENCED_PAGE;
> >       } else {
> >               result = SCAN_SUCCEED;
> > @@ -1792,13 +1807,13 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
> >               /* collapse_huge_page expects the lock to be dropped before calling */
> >               mmap_read_unlock(mm);
> >               result = mthp_collapse(mm, start_addr, referenced,
> > -                                    unmapped, cc, enabled_orders);
> > +                                    ctx.unmapped, cc, enabled_orders);
> >               /* mmap_lock was released above, set lock_dropped */
> >               *lock_dropped = true;
> >       }
> >  out:
> >       trace_mm_khugepaged_scan_pmd(mm, folio, referenced,
> > -                                  none_or_zero, result, unmapped);
> > +                                  ctx.none_or_zero, result, ctx.unmapped);
> >       return result;
> >  }
> >
> > --
> > 2.54.0
> >
> >
>

[PATCH 6/8] mm/khugepaged: unmap pte before releasing vma write lock

[Nico Pache]

We are currently dropping the anon_vma write lock before unmapping the
PTE. Although this is safe, due to us still holding the mmap_write_lock,
its safer and less confusing to switch the order of these two operations.

Suggested-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Nico Pache <npache@redhat.com>
---
 mm/khugepaged.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index c4ea2dc1591b..3c6f1254deca 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1548,10 +1548,10 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s
 
 	result = SCAN_SUCCEED;
 out_up_write:
-	if (anon_vma_locked)
-		anon_vma_unlock_write(vma->anon_vma);
 	if (pte)
 		pte_unmap(pte);
+	if (anon_vma_locked)
+		anon_vma_unlock_write(vma->anon_vma);
 	mmap_write_unlock(mm);
 out_nolock:
 	if (folio)
-- 
2.54.0

Re: [PATCH 6/8] mm/khugepaged: unmap pte before releasing vma write lock

[Usama Arif]

On Mon,  6 Jul 2026 09:44:53 -0600 Nico Pache <npache@redhat.com> wrote:

> We are currently dropping the anon_vma write lock before unmapping the
> PTE. Although this is safe, due to us still holding the mmap_write_lock,
> its safer and less confusing to switch the order of these two operations.
> 
> Suggested-by: David Hildenbrand <david@kernel.org>
> Signed-off-by: Nico Pache <npache@redhat.com>
> ---
>  mm/khugepaged.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)

Would it not be better to add a comment here why its safe to do so, instead
of extending lock hold period?

I think its more confusing to hold a lock for longer than we should.

> 
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index c4ea2dc1591b..3c6f1254deca 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -1548,10 +1548,10 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s
>  
>  	result = SCAN_SUCCEED;
>  out_up_write:
> -	if (anon_vma_locked)
> -		anon_vma_unlock_write(vma->anon_vma);
>  	if (pte)
>  		pte_unmap(pte);
> +	if (anon_vma_locked)
> +		anon_vma_unlock_write(vma->anon_vma);
>  	mmap_write_unlock(mm);
>  out_nolock:
>  	if (folio)
> -- 
> 2.54.0
> 
> 

[PATCH 7/8] mm/khugepaged: clarify a comment regarding max_ptes_none check

[Nico Pache]

While reading collapse_scan_pmd, one may be confused on where the deferred
max_ptes_none check is done. Expand on this current comment by explaining
which function the max_ptes_none check is now done.

Suggested-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Nico Pache <npache@redhat.com>
---
 mm/khugepaged.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 3c6f1254deca..388045a524a3 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1728,7 +1728,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 
 	/*
 	 * If PMD is the only enabled order, enforce max_ptes_none, otherwise
-	 * scan all pages to populate the bitmap for mTHP collapse.
+	 * scan all pages to populate the bitmap for mTHP collapse. The bitmap
+	 * is then checked again in mthp_collapse() for each attempted order.
 	 */
 	if (enabled_orders != BIT(HPAGE_PMD_ORDER))
 		ctx.max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT;
-- 
2.54.0

Re: [PATCH 7/8] mm/khugepaged: clarify a comment regarding max_ptes_none check

[Usama Arif]

On Mon,  6 Jul 2026 09:44:54 -0600 Nico Pache <npache@redhat.com> wrote:

> While reading collapse_scan_pmd, one may be confused on where the deferred
> max_ptes_none check is done. Expand on this current comment by explaining
> which function the max_ptes_none check is now done.
> 
> Suggested-by: David Hildenbrand <david@kernel.org>
> Signed-off-by: Nico Pache <npache@redhat.com>
> ---
>  mm/khugepaged.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 3c6f1254deca..388045a524a3 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -1728,7 +1728,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
>  
>  	/*
>  	 * If PMD is the only enabled order, enforce max_ptes_none, otherwise
> -	 * scan all pages to populate the bitmap for mTHP collapse.
> +	 * scan all pages to populate the bitmap for mTHP collapse. The bitmap
> +	 * is then checked again in mthp_collapse() for each attempted order.

Could be combined with patch 4?

  	 */
>  	if (enabled_orders != BIT(HPAGE_PMD_ORDER))
>  		ctx.max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT;
> -- 
> 2.54.0
> 
> 

[PATCH 8/8] mm: Documentation: clarify where the mTHP stats live

[Nico Pache]

The note about khugepaged counters references /proc/vmstat for the PMD
case, but never mentions where the mTHPs stats can be found
(i.e.: /sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/stats/)
Add a small addition to this section for clarity.

Also fix a missing period while we are at it.

Suggested-by: Lorenzo Stoakes <ljs@kernel.org>
Signed-off-by: Nico Pache <npache@redhat.com>
---
 Documentation/admin-guide/mm/transhuge.rst | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
index 23f8d13c2629..123ea281082c 100644
--- a/Documentation/admin-guide/mm/transhuge.rst
+++ b/Documentation/admin-guide/mm/transhuge.rst
@@ -224,7 +224,7 @@ khugepaged will be automatically started when any THP size is enabled
 (either of the per-size anon control or the top-level control are set
 to "always" or "madvise"), and it'll be automatically shutdown when
 all THP sizes are disabled (when both the per-size anon control and the
-top-level control are "never")
+top-level control are "never").
 
 process THP controls
 --------------------
@@ -301,7 +301,9 @@ being replaced by a PMD mapping, or (2) physical pages replaced by one
 hugepage of various sizes (PMD-sized or mTHP). Each may happen independently,
 or together, depending on the type of memory and the failures that occur.
 As such, this value should be interpreted roughly as a sign of progress,
-and counters in /proc/vmstat consulted for more accurate accounting)::
+and counters in /proc/vmstat consulted for more accurate accounting.
+Per-order mTHP collapse statistics are also available under
+/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/stats/)::
 
 	/sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed
 
-- 
2.54.0