[RFC PATCH 25/57] mm/collapse: collapse anonymous memory with the new engine

[Kiryl Shutsemau <kirill@shutemov.name>]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

Point the anonymous path at the engine.  Everything it needs is in place,
so this is the whole switch: collapse_single_pmd() calls
collapse_scan_anon_pmd() and then collapse_anon_pmd(), where it used to
call collapse_scan_pmd().  The scan runs under the mmap_read the caller
already holds; the collapse is called after dropping it, and takes the
lock itself for each round.

Both callers hand the engine a PMD-aligned address with the whole table
inside the VMA: khugepaged walks [ALIGN(vm_start), ALIGN_DOWN(vm_end)) a
table at a time, and MADV_COLLAPSE aligns its range inwards the same way.
So the range passed is always the table.  The engine accepts a narrower
one, which nothing asks for yet.

Two things userspace sees change:

 - A collapse runs under mmap_read rather than holding mmap_write for its
   duration, so faults elsewhere in the address space are no longer
   stopped while it works.
 - A table that cannot become one huge page still yields the largest
   windows inside it, where before a single disqualified PTE gave up the
   whole table.

The result the caller gets is the engine's, and it still acts on an
allocation failure by backing off.

The mechanism this replaces is left in place, now unreferenced.

Assisted-by: Claude-Code:claude-opus-5
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 mm/collapse.c   | 15 ++++++---------
 mm/collapse.h   |  5 +++++
 mm/khugepaged.c | 14 ++++++++++++--
 3 files changed, 23 insertions(+), 11 deletions(-)

diff --git a/mm/collapse.c b/mm/collapse.c
index 91ff20138a8e..df3760e3918b 100644
--- a/mm/collapse.c
+++ b/mm/collapse.c
@@ -83,9 +83,6 @@
  * consecutive pages of one folio -- so partially mapped and compound sources
  * collapse too: any order below the window's is a source, and a PMD candidate
  * takes even a PTE-mapped THP of its own order.
- *
- * Nothing calls any of this yet: the anon path still uses the mechanism it
- * replaces, and is switched over once both halves are complete.
  */
 
 /*
@@ -1926,9 +1923,9 @@ static void collapse_anon_scan_init(struct collapse_control *cc)
  * that acts on what it found hands the range to collapse_anon_pmd() afterwards,
  * without the lock.
  */
-static enum scan_result __maybe_unused
-collapse_scan_anon_pmd(struct vm_area_struct *vma, unsigned long start,
-		       unsigned long end, struct collapse_control *cc)
+enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma,
+					unsigned long start, unsigned long end,
+					struct collapse_control *cc)
 {
 	const unsigned long pmd_addr = start & HPAGE_PMD_MASK;
 	struct mm_struct *mm = vma->vm_mm;
@@ -2337,9 +2334,9 @@ static void collapse_add_candidate(struct collapse_control *cc,
  * largest order downwards.  Returns what the table yielded: a collapse, or
  * the reason it did not.
  */
-static enum scan_result __maybe_unused
-collapse_anon_pmd(struct mm_struct *mm, unsigned long start, unsigned long end,
-		  struct collapse_control *cc)
+enum scan_result collapse_anon_pmd(struct mm_struct *mm, unsigned long start,
+				   unsigned long end,
+				   struct collapse_control *cc)
 {
 	const unsigned long pmd_addr = start & HPAGE_PMD_MASK;
 	unsigned int offset, order;
diff --git a/mm/collapse.h b/mm/collapse.h
index 34de3ebb05e3..50a9d59bbf03 100644
--- a/mm/collapse.h
+++ b/mm/collapse.h
@@ -186,6 +186,11 @@ static inline int collapse_test_exit_or_disable(struct mm_struct *mm)
 		mm_flags_test(MMF_DISABLE_THP_COMPLETELY, mm);
 }
 
+enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma,
+		unsigned long start, unsigned long end,
+		struct collapse_control *cc);
+enum scan_result collapse_anon_pmd(struct mm_struct *mm, unsigned long start,
+		unsigned long end, struct collapse_control *cc);
 int collapse_control_init(struct collapse_control *cc);
 void collapse_control_release(struct collapse_control *cc);
 
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index c7c933e819e2..0662d08f7c60 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1553,7 +1553,8 @@ static enum scan_result mthp_collapse(struct mm_struct *mm,
 	return last_result;
 }
 
-static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
+static enum scan_result __maybe_unused
+collapse_scan_pmd(struct mm_struct *mm,
 		struct vm_area_struct *vma, unsigned long start_addr,
 		bool *lock_dropped, struct collapse_control *cc)
 {
@@ -2749,7 +2750,16 @@ static enum scan_result collapse_single_pmd(unsigned long addr,
 	mmap_assert_locked(mm);
 
 	if (vma_is_anonymous(vma)) {
-		result = collapse_scan_pmd(mm, vma, addr, lock_dropped, cc);
+		result = collapse_scan_anon_pmd(vma, addr, addr + HPAGE_PMD_SIZE,
+						cc);
+		if (!cc->select_orders)
+			goto end;
+
+		/* collapse_anon_pmd() takes mmap_lock itself, where it needs it */
+		mmap_read_unlock(mm);
+		*lock_dropped = true;
+
+		result = collapse_anon_pmd(mm, addr, addr + HPAGE_PMD_SIZE, cc);
 		goto end;
 	}
 
-- 
2.54.0