[RFC PATCH 0/4] mm: compaction: mTHP-friendly memory compaction

[RFC PATCH 0/4] mm: compaction: mTHP-friendly memory compaction

[Bo Zhang]

Hi all,

This series improves memory compaction to better serve mTHP (multi-size
THP) allocations, particularly for small orders like order-2 (16KB).
The changes cover the proactive compaction and kswapd-triggered compaction
paths.

Problem:

The current proactive compaction targets COMPACTION_HPAGE_ORDER (order-9,
2MB), which is unfriendly to mTHP in two ways:

1. It migrates folios that already satisfy mTHP allocation needs. For
   example, an order-2 folio is a valid mTHP page, yet compaction still
   moves it around trying to form order-9 blocks. This is unnecessary
   work and wastes energy.

2. Compaction designed for 2MB huge pages is too heavyweight for mTHP.
   mTHP allocations are frequent and only require small contiguous blocks
   (e.g., 4 pages for order-2). A lighter-weight, mTHP-aware compaction
   strategy is needed to reduce overhead and power consumption.

Approach:

This series makes four changes:

1. Generalize the fragmentation score functions to accept an order parameter
   and use the minimum always-enabled mTHP order as the compaction target.

2. During proactive compaction (compact_memory), skip isolating folios that
   already satisfy mTHP requirements, avoiding unnecessary migration overhead.

3. Allow proactive compaction to proceed concurrently with kswapd for
   non-costly mTHP orders, since kswapd reclaim alone may not produce the
   contiguous blocks needed for these allocations.

4. Introduce zone_effective_free_pages() that provides mTHP-aware free page
   accounting for watermark checks, counting only buddy blocks that can
   actually satisfy mTHP allocations.

Test setup:

Boot Ubuntu with 2700 MB of memory, with mTHP disabled
initially and the defrag mode set to defer+madvise.

Then set the 16 KB mTHP size to always and run a kernel
build with -j20.

For both cases below, we run a background script to
proactively trigger compaction as follows:
 #!/bin/bash

 while true; do
 	echo 50 > /proc/sys/vm/compaction_proactiveness
 	sleep 0.1
 done

W/o patch:

*** Executing round 0 ***

real	2m1.312s
user	25m38.968s
sys	5m16.934s
anon_fault_alloc: 6328991
anon_fault_fallback: 214553

*** Executing round 1 ***

real	1m55.370s
user	25m24.482s
sys	3m52.512s
anon_fault_alloc: 6355263
anon_fault_fallback: 108692

*** Executing round 2 ***

real	2m7.579s
user	25m11.530s
sys	3m45.456s
anon_fault_alloc: 6355816
anon_fault_fallback: 107852

*** Executing round 3 ***

real	1m53.824s
user	25m26.774s
sys	3m42.160s
anon_fault_alloc: 6355457
anon_fault_fallback: 107705

W/patch:

*** Executing round 0 ***

real	1m55.906s
user	25m16.985s
sys	4m24.480s
anon_fault_alloc: 6354486
anon_fault_fallback: 109845

*** Executing round 1 ***

real	1m51.303s
user	25m16.456s
sys	3m26.629s
anon_fault_alloc: 6392515
anon_fault_fallback: 69797

*** Executing round 2 ***

real	1m51.495s
user	25m13.075s
sys	3m28.501s
anon_fault_alloc: 6395096
anon_fault_fallback: 67510

*** Executing round 3 ***

real	1m52.980s
user	25m8.217s
sys	3m37.506s
anon_fault_alloc: 6389556
anon_fault_fallback: 72743

Before "Executing round 0", mTHP is not enabled. Therefore, both
cases show a higher anon_fault_fallback in Round 0 than in the
other rounds. With the patch, however, memory can be compacted faster
into an mTHP-friendly state, resulting in a much lower fallback rate
in Round 0. In the other rounds, the patch also consistently shows
a lower anon_fault_fallback, as well as lower sys and wall time for
the kernel build.

Open questions:

1. When multiple mTHP orders are enabled (e.g., order-2 and order-4 both
   "always"), this series only targets the minimum order. Should proactive
   compaction also independently evaluate and serve higher orders?

2. In skip_isolation_on_order(), the filter order ideally should come from
   the compaction control path. However, during proactive compaction
   target_order is always -1 (via compact_memory), and there is no clean
   way to pass the mTHP order down from upper layers. Currently we read
   huge_anon_orders_always directly, but this variable can be changed by
   userspace at any time, making the semantic fragile (the compaction may
   start with one order target and finish with another). Ideas on how to
   plumb the target order through the proactive compaction path cleanly
   are welcome.

3. In __compact_finished(), the original code skips proactive compaction
   when kswapd is running to avoid interference. Patch 3 removes this
   skip for non-costly mTHP orders (< PAGE_ALLOC_COSTLY_ORDER). The
   reason is that small-order compaction is lightweight and likely to
   succeed quickly even while kswapd is reclaiming, forming an order-2
   block requires migrating very few pages. Does this approach make sense,
   or is there a better way to coordinate proactive compaction with kswapd
   in the mTHP scenario?

4. In the direct reclaim path (__alloc_pages_slowpath), compact_first is
   only set for costly orders or non-movable allocations. For mTHP always-
   enabled non-costly orders (e.g., order-2 MIGRATE_MOVABLE), when free
   memory is sufficient (watermarks met) but fragmentation is high,
   compaction is more appropriate than reclaim. Should we also set
   compact_first for this case to avoid unnecessary reclaim?

Bo Zhang (4):
  mm: compaction: make proactive compaction mTHP-aware
  mm: compaction: skip isolating large folios that satisfy the mTHP order
  mm: compaction: don't skip proactive compaction for non-costly mTHP
  mm: adjust free_pages to make __zone_watermark_ok() mTHP-aware

 mm/compaction.c | 83 ++++++++++++++++++++++++++++++++++++++-----------
 mm/internal.h   |  3 ++
 mm/vmscan.c     | 23 +++-----------
 3 files changed, 72 insertions(+), 37 deletions(-)

--
2.34.1

[RFC PATCH 1/4] mm: compaction: make proactive compaction mTHP-aware

[Bo Zhang]

Currently, proactive compaction only evaluates fragmentation relative
to COMPACTION_HPAGE_ORDER (typically order-9 for 2MB THP). This makes
it unsuitable for systems that primarily need smaller high-order pages,
such as order-2 (16KB) for mTHP.

Generalize the fragmentation score functions to accept an order parameter:
  - fragmentation_score_zone(zone, order)
  - fragmentation_score_zone_weighted(zone, order)
  - fragmentation_score_node(pgdat, order)

Calculate the min order in huge_anon_orders_always to configure the target
order for proactive compaction.

This enables proactive compaction to maintain free page availability
at any order, which is particularly useful for mTHP-enabled systems
where order-n (n < 9) allocation pressure is high.

Signed-off-by: Bo Zhang <zhangbo56@xiaomi.com>
---
 mm/compaction.c | 36 ++++++++++++++++++++++++------------
 1 file changed, 24 insertions(+), 12 deletions(-)

diff --git a/mm/compaction.c b/mm/compaction.c
index a049415512c6..a4f87232ad80 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -24,6 +24,7 @@
 #include <linux/page_owner.h>
 #include <linux/psi.h>
 #include <linux/cpuset.h>
+#include <linux/huge_mm.h>
 #include "page_alloc.h"
 #include "internal.h"
 
@@ -81,6 +82,15 @@ static inline bool is_via_compact_memory(int order) { return false; }
 #define COMPACTION_HPAGE_ORDER	(PMD_SHIFT - PAGE_SHIFT)
 #endif
 
+static inline int compact_hpage_order(void)
+{
+	unsigned long orders = READ_ONCE(huge_anon_orders_always);
+
+	if (orders)
+		return __ffs(orders);
+	return COMPACTION_HPAGE_ORDER;
+}
+
 static struct page *mark_allocated_noprof(struct page *page, unsigned int order, gfp_t gfp_flags)
 {
 	post_alloc_hook(page, order, __GFP_MOVABLE, ALLOC_DEFAULT);
@@ -2208,16 +2218,16 @@ static bool kswapd_is_running(pg_data_t *pgdat)
 
 /*
  * A zone's fragmentation score is the external fragmentation wrt to the
- * COMPACTION_HPAGE_ORDER. It returns a value in the range [0, 100].
+ * compact_hpage_order(). It returns a value in the range [0, 100].
  */
-static unsigned int fragmentation_score_zone(struct zone *zone)
+static unsigned int fragmentation_score_zone(struct zone *zone, unsigned int order)
 {
-	return extfrag_for_order(zone, COMPACTION_HPAGE_ORDER);
+	return extfrag_for_order(zone, order);
 }
 
 /*
  * A weighted zone's fragmentation score is the external fragmentation
- * wrt to the COMPACTION_HPAGE_ORDER scaled by the zone's size. It
+ * wrt to the compact_hpage_order() scaled by the zone's size. It
  * returns a value in the range [0, 100].
  *
  * The scaling factor ensures that proactive compaction focuses on larger
@@ -2225,11 +2235,11 @@ static unsigned int fragmentation_score_zone(struct zone *zone)
  * ZONE_DMA32. For smaller zones, the score value remains close to zero,
  * and thus never exceeds the high threshold for proactive compaction.
  */
-static unsigned int fragmentation_score_zone_weighted(struct zone *zone)
+static unsigned int fragmentation_score_zone_weighted(struct zone *zone, unsigned int order)
 {
 	unsigned long score;
 
-	score = zone->present_pages * fragmentation_score_zone(zone);
+	score = zone->present_pages * fragmentation_score_zone(zone, order);
 	return div64_ul(score, zone->zone_pgdat->node_present_pages + 1);
 }
 
@@ -2240,7 +2250,7 @@ static unsigned int fragmentation_score_zone_weighted(struct zone *zone)
  * the node's score falls below the low threshold, or one of the back-off
  * conditions is met.
  */
-static unsigned int fragmentation_score_node(pg_data_t *pgdat)
+static unsigned int fragmentation_score_node(pg_data_t *pgdat, unsigned int order)
 {
 	unsigned int score = 0;
 	int zoneid;
@@ -2251,7 +2261,7 @@ static unsigned int fragmentation_score_node(pg_data_t *pgdat)
 		zone = &pgdat->node_zones[zoneid];
 		if (!populated_zone(zone))
 			continue;
-		score += fragmentation_score_zone_weighted(zone);
+		score += fragmentation_score_zone_weighted(zone, order);
 	}
 
 	return score;
@@ -2269,12 +2279,13 @@ static unsigned int fragmentation_score_wmark(bool low)
 static bool should_proactive_compact_node(pg_data_t *pgdat)
 {
 	int wmark_high;
+	unsigned int order = compact_hpage_order();
 
 	if (!sysctl_compaction_proactiveness || kswapd_is_running(pgdat))
 		return false;
 
 	wmark_high = fragmentation_score_wmark(false);
-	return fragmentation_score_node(pgdat) > wmark_high;
+	return fragmentation_score_node(pgdat, order) > wmark_high;
 }
 
 static enum compact_result __compact_finished(struct compact_control *cc)
@@ -2311,7 +2322,7 @@ static enum compact_result __compact_finished(struct compact_control *cc)
 		if (kswapd_is_running(pgdat))
 			return COMPACT_PARTIAL_SKIPPED;
 
-		score = fragmentation_score_zone(cc->zone);
+		score = fragmentation_score_zone(cc->zone, compact_hpage_order());
 		wmark_low = fragmentation_score_wmark(true);
 
 		if (score > wmark_low)
@@ -3238,10 +3249,11 @@ static int kcompactd(void *p)
 		timeout = default_timeout;
 		if (should_proactive_compact_node(pgdat)) {
 			unsigned int prev_score, score;
+			unsigned int order = compact_hpage_order();
 
-			prev_score = fragmentation_score_node(pgdat);
+			prev_score = fragmentation_score_node(pgdat, order);
 			compact_node(pgdat, true);
-			score = fragmentation_score_node(pgdat);
+			score = fragmentation_score_node(pgdat, order);
 			/*
 			 * Defer proactive compaction if the fragmentation
 			 * score did not go down i.e. no progress made.
-- 
2.34.1

Re: [RFC PATCH 1/4] mm: compaction: make proactive compaction mTHP-aware

[Bo Zhang]

Thanks for the review. Replying to the three points below.

1) Build breakage when CONFIG_TRANSPARENT_HUGEPAGE is disabled

Sashiko says

"Does this break the build when CONFIG_TRANSPARENT_HUGEPAGE is disabled?
In include/linux/huge_mm.h, the declaration of huge_anon_orders_always is
guarded by #ifdef CONFIG_TRANSPARENT_HUGEPAGE, but here it is referenced
unconditionally."

You're right (the kernel test robot reported the same on m68k). In v2 I'll
guard the access with #ifdef CONFIG_TRANSPARENT_HUGEPAGE and fall back to
COMPACTION_HPAGE_ORDER.

2) __ffs() singles out the lowest enabled mTHP order

Sashiko says

"If an administrator enables both order-2 and order-9 THPs, it seems this
will return order-2, and __compact_finished() will halt compaction early
once enough order-2 blocks are formed, potentially starving the system of
order-9 THPs."

Yes, this is a deliberate simplification, also raised as an open question
in the cover letter. I don't have a firm answer on the best policy for
multiple enabled orders yet and would welcome opinions.

3) Racing with concurrent sysfs updates

Sashiko says

"kcompactd() caches the target order ... However, __compact_finished()
dynamically re-reads it here. If the target order changes mid-compaction
... Could this mismatch falsely indicate a lack of progress and erroneously
defer proactive compaction?"

Good catch. The clean fix would be to read the order once and plumb it
through the compaction path. But during proactive compaction target_order
is -1, and I couldn't find a clean way to pass the mTHP order down - which
is why the code reads huge_anon_orders_always directly. This is the open
question I raised in the cover letter; suggestions welcome.

Bo

[RFC PATCH 2/4] mm: compaction: skip isolating large folios that satisfy the mTHP order

[Bo Zhang]

When proactive compaction is triggered via compact_memory and mTHP
always-enabled orders are configured, skip isolating folios whose order
is >= the minimum always-enabled mTHP order. These folios already satisfy
mTHP allocation requirements and migrating them is unnecessary overhead.

Signed-off-by: Bo Zhang <zhangbo56@xiaomi.com>
---
 mm/compaction.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/mm/compaction.c b/mm/compaction.c
index a4f87232ad80..a15ed2576562 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -837,6 +837,12 @@ static bool skip_isolation_on_order(int order, int target_order)
 	 */
 	if (!is_via_compact_memory(target_order) && order >= target_order)
 		return true;
+
+	/* We are compacting for multi-size THP allocation */
+	if (is_via_compact_memory(target_order) && order >= compact_hpage_order() &&
+	    READ_ONCE(huge_anon_orders_always))
+		return true;
+
 	/*
 	 * We limit memory compaction to pageblocks and won't try
 	 * creating free blocks of memory that are larger than that.
-- 
2.34.1

Re: [RFC PATCH 2/4] mm: compaction: skip isolating large folios that satisfy the mTHP order

[Bo Zhang]

Thanks for the review. Replying to the two points below.

1) Build breakage when CONFIG_TRANSPARENT_HUGEPAGE is disabled

Sashiko says

"huge_anon_orders_always is only declared in include/linux/huge_mm.h under
CONFIG_TRANSPARENT_HUGEPAGE. Without an #else fallback stub, any kernel
build configured with CONFIG_COMPACTION=y and CONFIG_TRANSPARENT_HUGEPAGE=n
might fail with an undeclared identifier error."

Correct. In v2 the huge_anon_orders_always access is confined to
compact_hpage_order() and guarded by #ifdef CONFIG_TRANSPARENT_HUGEPAGE,
so this call site no longer references it directly.

2) Skipping isolation harms manual compaction and HugeTLB pool resizing

Sashiko says

"If smaller mTHPs (like order-2) are enabled, compact_hpage_order() will
return 2. This logic will then unconditionally return true for any folio of
order 2 or higher, skipping its isolation. Could this cause administrative
commands or automated HugeTLB pool resizing scripts to inexplicably fail to
allocate large contiguous memory blocks ...?"

To clarify the intent first: when mTHP is enabled the proactive compaction
target is compact_hpage_order() (e.g. order-2), so folios already at or
above that order need not be isolated - migrating smaller fragments is
enough to assemble the target block, and isolating larger folios is just
wasted work.

That said, the problem you describe is real: compact_hpage_order() is
derived from huge_anon_orders_always, so it also affects manual
`echo 1 > compact_memory` and HugeTLB pool resizing, which legitimately
want to form order-9/order-11 blocks.

This comes back to a question I raised in the cover letter: what is the
right way to determine the compaction target order? Deriving it from
huge_anon_orders_always is convenient but conflates mTHP proactive
compaction with these other callers. An alternative would be a dedicated
knob (e.g. a sysctl compaction target order). I haven't settled on the
right approach and would welcome opinions.

Bo

[RFC PATCH 3/4] mm: compaction: don't skip proactive compaction for non-costly mTHP

[Bo Zhang]

When the minimum always-enabled mTHP order is below
PAGE_ALLOC_COSTLY_ORDER, do not skip proactive compaction even if kswapd
is running. For these non-costly mTHP orders, kswapd reclaim alone may
not produce the contiguous free blocks needed, so proactive compaction
should proceed concurrently to ensure mTHP allocation success.

Signed-off-by: Bo Zhang <zhangbo56@xiaomi.com>
---
 mm/compaction.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mm/compaction.c b/mm/compaction.c
index a15ed2576562..15b92475562a 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -2323,9 +2323,10 @@ static enum compact_result __compact_finished(struct compact_control *cc)
 	if (cc->proactive_compaction) {
 		int score, wmark_low;
 		pg_data_t *pgdat;
+		bool costly = compact_hpage_order() > PAGE_ALLOC_COSTLY_ORDER;
 
 		pgdat = cc->zone->zone_pgdat;
-		if (kswapd_is_running(pgdat))
+		if (costly && kswapd_is_running(pgdat))
 			return COMPACT_PARTIAL_SKIPPED;
 
 		score = fragmentation_score_zone(cc->zone, compact_hpage_order());
-- 
2.34.1

Re: [RFC PATCH 3/4] mm: compaction: don't skip proactive compaction for non-costly mTHP

[Bo Zhang]

Thanks for the review.

Sashiko says

"Does this change miss an entry condition check in
should_proactive_compact_node() in the same file? If kswapd is already
active when kcompactd evaluates whether to start,
should_proactive_compact_node() still unconditionally aborts ... Could this
prevent proactive compaction from even starting for non-costly orders,
defeating the updated mid-flight check entirely?"

You're right. I only relaxed the mid-flight check in __compact_finished(),
but should_proactive_compact_node() still bails out unconditionally when
kswapd is running, so for non-costly mTHP orders proactive compaction never
even starts and my change has no effect in the common case (kswapd running
under memory pressure).

In v2 I'll apply the same non-costly condition at the entry check:

	if (!sysctl_compaction_proactiveness)
		return false;
	if (costly && kswapd_is_running(pgdat))
		return false;

so the entry and mid-flight checks stay consistent.

Bo

[RFC PATCH 4/4] mm: adjust free_pages to make __zone_watermark_ok() mTHP-aware

[Bo Zhang]

To improve the mTHP allocation success rate and reduce
fragmentation over time, introduce
zone_effective_free_pages() for mTHP-aware free page accounting.

Refactor the free page counting logic used by pgdat_balanced() and
compaction_suit_allocation_order() into a shared helper function
zone_effective_free_pages().

The function selects the appropriate free page metric based on context:
- defrag_mode: use NR_FREE_PAGES_BLOCKS (whole pageblock accounting)
- mTHP always-enabled: count only free pages in buddy blocks >= the
  minimum always-enabled mTHP order, since smaller fragments cannot
  satisfy mTHP allocations
- otherwise: use NR_FREE_PAGES

Signed-off-by: Bo Zhang <zhangbo56@xiaomi.com>
---
 mm/compaction.c | 38 ++++++++++++++++++++++++++++++++++----
 mm/internal.h   |  3 +++
 mm/vmscan.c     | 23 +++++------------------
 3 files changed, 42 insertions(+), 22 deletions(-)

diff --git a/mm/compaction.c b/mm/compaction.c
index 15b92475562a..29be72597415 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -2528,6 +2528,38 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
 	return false;
 }
 
+/**
+ * zone_effective_free_pages - get free pages relevant to allocation order
+ * @zone:       target zone
+ * @order:      allocation order
+ * @use_blocks: if true, use NR_FREE_PAGES_BLOCKS
+ *
+ * In defrag_mode, watermarks must be met in whole blocks to avoid
+ * polluting allocator fallbacks. kswapd usually cannot accomplish
+ * this on its own and needs kcompactd support.
+ *
+ * When mTHP always-enabled orders are configured, count only free pages
+ * in blocks >= min mTHP order, as smaller fragments cannot satisfy mTHP
+ * allocations.
+ */
+unsigned long zone_effective_free_pages(struct zone *zone,
+					unsigned int order,
+					bool use_blocks)
+{
+	if (use_blocks)
+		return zone_page_state(zone, NR_FREE_PAGES_BLOCKS);
+
+	if (READ_ONCE(huge_anon_orders_always) && order == compact_hpage_order()) {
+		unsigned long free_pages = 0;
+
+		for (int o = order; o < NR_PAGE_ORDERS; o++)
+			free_pages += zone->free_area[o].nr_free << o;
+		return free_pages;
+	}
+
+	return zone_page_state(zone, NR_FREE_PAGES);
+}
+
 /*
  * Should we do compaction for target allocation order.
  * Return COMPACT_SUCCESS if allocation for target order can be already
@@ -2543,10 +2575,8 @@ compaction_suit_allocation_order(struct zone *zone, unsigned int order,
 	unsigned long free_pages;
 	unsigned long watermark;
 
-	if (kcompactd && defrag_mode)
-		free_pages = zone_page_state(zone, NR_FREE_PAGES_BLOCKS);
-	else
-		free_pages = zone_page_state(zone, NR_FREE_PAGES);
+	free_pages = zone_effective_free_pages(zone, order,
+					       kcompactd && defrag_mode);
 
 	watermark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK);
 	if (__zone_watermark_ok(zone, order, watermark, highest_zoneidx,
diff --git a/mm/internal.h b/mm/internal.h
index 38b1165212c9..14bb9543879b 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -1654,4 +1654,7 @@ static inline bool can_spin_trylock(void)
 	return true;
 }
 
+unsigned long zone_effective_free_pages(struct zone *zone,
+					unsigned int order,
+					bool use_blocks);
 #endif	/* __MM_INTERNAL_H */
diff --git a/mm/vmscan.c b/mm/vmscan.c
index c1404a59523d..a419a2c2fca4 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -6966,7 +6966,6 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx)
 	 * meet watermarks.
 	 */
 	for_each_managed_zone_pgdat(zone, pgdat, i, highest_zoneidx) {
-		enum zone_stat_item item;
 		unsigned long free_pages;
 
 		if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
@@ -6974,21 +6973,6 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx)
 		else
 			mark = high_wmark_pages(zone);
 
-		/*
-		 * In defrag_mode, watermarks must be met in whole
-		 * blocks to avoid polluting allocator fallbacks.
-		 *
-		 * However, kswapd usually cannot accomplish this on
-		 * its own and needs kcompactd support. Once it's
-		 * reclaimed a compaction gap, and kswapd_shrink_node
-		 * has dropped order, simply ensure there are enough
-		 * base pages for compaction, wake kcompactd & sleep.
-		 */
-		if (defrag_mode && order)
-			item = NR_FREE_PAGES_BLOCKS;
-		else
-			item = NR_FREE_PAGES;
-
 		/*
 		 * When there is a high number of CPUs in the system,
 		 * the cumulative error from the vmstat per-cpu cache
@@ -7001,9 +6985,12 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx)
 		 * counter won't actually be per-cpu cached. But keep
 		 * things simple for now; revisit when somebody cares.
 		 */
-		free_pages = zone_page_state(zone, item);
+		free_pages = zone_effective_free_pages(zone, order,
+						       defrag_mode & order);
 		if (zone->percpu_drift_mark && free_pages < zone->percpu_drift_mark)
-			free_pages = zone_page_state_snapshot(zone, item);
+			free_pages = zone_page_state_snapshot(zone,
+					defrag_mode & order ?
+					NR_FREE_PAGES_BLOCKS : NR_FREE_PAGES);
 
 		if (__zone_watermark_ok(zone, order, mark, highest_zoneidx,
 					0, free_pages))
-- 
2.34.1

Re: [RFC PATCH 4/4] mm: adjust free_pages to make __zone_watermark_ok() mTHP-aware

[Xueyuan Chen]

On Tue, Aug 25, 2026 at 12:38:33PM +0800, Bo Zhang wrote:
>To improve the mTHP allocation success rate and reduce
>fragmentation over time, introduce
>zone_effective_free_pages() for mTHP-aware free page accounting.
>
>Refactor the free page counting logic used by pgdat_balanced() and
>compaction_suit_allocation_order() into a shared helper function
>zone_effective_free_pages().
>
>The function selects the appropriate free page metric based on context:
>- defrag_mode: use NR_FREE_PAGES_BLOCKS (whole pageblock accounting)
>- mTHP always-enabled: count only free pages in buddy blocks >= the
>  minimum always-enabled mTHP order, since smaller fragments cannot
>  satisfy mTHP allocations
>- otherwise: use NR_FREE_PAGES
>
>Signed-off-by: Bo Zhang <zhangbo56@xiaomi.com>
>---
> mm/compaction.c | 38 ++++++++++++++++++++++++++++++++++----
> mm/internal.h   |  3 +++
> mm/vmscan.c     | 23 +++++------------------
> 3 files changed, 42 insertions(+), 22 deletions(-)
>
>diff --git a/mm/compaction.c b/mm/compaction.c
>index 15b92475562a..29be72597415 100644
>--- a/mm/compaction.c
>+++ b/mm/compaction.c
>@@ -2528,6 +2528,38 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
> 	return false;
> }
> 
>+/**
>+ * zone_effective_free_pages - get free pages relevant to allocation order
>+ * @zone:       target zone
>+ * @order:      allocation order
>+ * @use_blocks: if true, use NR_FREE_PAGES_BLOCKS
>+ *
>+ * In defrag_mode, watermarks must be met in whole blocks to avoid
>+ * polluting allocator fallbacks. kswapd usually cannot accomplish
>+ * this on its own and needs kcompactd support.
>+ *
>+ * When mTHP always-enabled orders are configured, count only free pages
>+ * in blocks >= min mTHP order, as smaller fragments cannot satisfy mTHP
>+ * allocations.
>+ */
>+unsigned long zone_effective_free_pages(struct zone *zone,
>+					unsigned int order,
>+					bool use_blocks)
>+{
>+	if (use_blocks)
>+		return zone_page_state(zone, NR_FREE_PAGES_BLOCKS);
>+
>+	if (READ_ONCE(huge_anon_orders_always) && order == compact_hpage_order()) {
>+		unsigned long free_pages = 0;
>+
>+		for (int o = order; o < NR_PAGE_ORDERS; o++)
>+			free_pages += zone->free_area[o].nr_free << o;
>+		return free_pages;
>+	}
>+
>+	return zone_page_state(zone, NR_FREE_PAGES);
>+}
>+
> /*
>  * Should we do compaction for target allocation order.
>  * Return COMPACT_SUCCESS if allocation for target order can be already
>@@ -2543,10 +2575,8 @@ compaction_suit_allocation_order(struct zone *zone, unsigned int order,
> 	unsigned long free_pages;
> 	unsigned long watermark;
> 
>-	if (kcompactd && defrag_mode)
>-		free_pages = zone_page_state(zone, NR_FREE_PAGES_BLOCKS);
>-	else
>-		free_pages = zone_page_state(zone, NR_FREE_PAGES);
>+	free_pages = zone_effective_free_pages(zone, order,
>+					       kcompactd && defrag_mode);
> 
> 	watermark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK);
> 	if (__zone_watermark_ok(zone, order, watermark, highest_zoneidx,
>diff --git a/mm/internal.h b/mm/internal.h
>index 38b1165212c9..14bb9543879b 100644
>--- a/mm/internal.h
>+++ b/mm/internal.h
>@@ -1654,4 +1654,7 @@ static inline bool can_spin_trylock(void)
> 	return true;
> }
> 
>+unsigned long zone_effective_free_pages(struct zone *zone,
>+					unsigned int order,
>+					bool use_blocks);
> #endif	/* __MM_INTERNAL_H */
>diff --git a/mm/vmscan.c b/mm/vmscan.c
>index c1404a59523d..a419a2c2fca4 100644
>--- a/mm/vmscan.c
>+++ b/mm/vmscan.c
>@@ -6966,7 +6966,6 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx)
> 	 * meet watermarks.
> 	 */
> 	for_each_managed_zone_pgdat(zone, pgdat, i, highest_zoneidx) {
>-		enum zone_stat_item item;
> 		unsigned long free_pages;
> 
> 		if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
>@@ -6974,21 +6973,6 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx)
> 		else
> 			mark = high_wmark_pages(zone);
> 
>-		/*
>-		 * In defrag_mode, watermarks must be met in whole
>-		 * blocks to avoid polluting allocator fallbacks.
>-		 *
>-		 * However, kswapd usually cannot accomplish this on
>-		 * its own and needs kcompactd support. Once it's
>-		 * reclaimed a compaction gap, and kswapd_shrink_node
>-		 * has dropped order, simply ensure there are enough
>-		 * base pages for compaction, wake kcompactd & sleep.
>-		 */
>-		if (defrag_mode && order)
>-			item = NR_FREE_PAGES_BLOCKS;
>-		else
>-			item = NR_FREE_PAGES;
>-
> 		/*
> 		 * When there is a high number of CPUs in the system,
> 		 * the cumulative error from the vmstat per-cpu cache
>@@ -7001,9 +6985,12 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx)
> 		 * counter won't actually be per-cpu cached. But keep
> 		 * things simple for now; revisit when somebody cares.
> 		 */
>-		free_pages = zone_page_state(zone, item);
>+		free_pages = zone_effective_free_pages(zone, order,
>+						       defrag_mode & order);

Hi Bo,

  zone_effective_free_pages(zone, order, defrag_mode & order);

"&" should be "&&". defrag_mode is 0 or 1, so this is just order & 1:
works for order-3 by accident, always 0 for order-2.

Same for the zone_page_state_snapshot() call below.

Thanks,
Xueyuan

> 		if (zone->percpu_drift_mark && free_pages < zone->percpu_drift_mark)
>-			free_pages = zone_page_state_snapshot(zone, item);
>+			free_pages = zone_page_state_snapshot(zone,
>+					defrag_mode & order ?
>+					NR_FREE_PAGES_BLOCKS : NR_FREE_PAGES);
> 
> 		if (__zone_watermark_ok(zone, order, mark, highest_zoneidx,
> 					0, free_pages))
>-- 
>2.34.1
>
>

Re: [RFC PATCH 4/4] mm: adjust free_pages to make __zone_watermark_ok() mTHP-aware

[Bo Zhang]

On Thu, Sep 03, 2026 at 10:46:03AM +0800, Xueyuan Chen wrote:
> >+		free_pages = zone_effective_free_pages(zone, order,
> >+						       defrag_mode & order);
>
> "&" should be "&&". defrag_mode is 0 or 1, so this is just order & 1:
> works for order-3 by accident, always 0 for order-2.
>
> Same for the zone_page_state_snapshot() call below.

Good catch, thanks. You're right - this should be a logical "&&".
The bitwise "&" accidentally disables the block accounting for even
orders like order-2, which is exactly the mTHP case this series targets.

I'll fix both occurrences in v2.

Thanks,
Bo

Re: [RFC PATCH 4/4] mm: adjust free_pages to make __zone_watermark_ok() mTHP-aware

[Bo Zhang]

Thanks for the review. (The "defrag_mode & order" bitwise bug was already
covered in the reply to Xueyuan Chen; it will use "&&" in v2.) Replying to
the other two points below.

1) mTHP count discarded by the percpu_drift_mark fallback

Sashiko says

"The restricted mTHP free_pages count is being compared against
zone->percpu_drift_mark, which is tuned for the total number of free pages
... won't this condition almost always evaluate to true under memory
pressure? If so, it would unconditionally overwrite free_pages with the
snapshot of NR_FREE_PAGES, bypassing the new mTHP logic entirely."

Right. percpu_drift_mark is calibrated against total free pages. Under
fragmentation or memory pressure - exactly when pgdat_balanced() matters -
the >= mTHP-order free blocks shrink relative to total free pages, so the
mTHP subset drops below percpu_drift_mark and the snapshot overwrites it,
bypassing the mTHP value. In any case the drift correction shouldn't apply
to the mTHP path: it reads free_area directly, which is already exact and
not subject to per-cpu drift.

2) Restricted subset can go negative in __zone_watermark_ok()

Sashiko says

"Since __zone_watermark_unusable_free() includes all order-0 cma and
highatomic pages across the entire zone, subtracting the global unusable
total from the heavily restricted subset can push free_pages negative. This
fails the watermark check permanently and can cause kswapd to spin
endlessly."

The operand mismatch is a pre-existing pattern, not something this patch
introduces: the existing defrag_mode path already feeds a subset
(NR_FREE_PAGES_BLOCKS, only >= pageblock_order) into __zone_watermark_ok()
the same way, and my mTHP subset (>= order-2) is actually a superset of
that, so it's larger and less likely to go negative. Going negative would
require an extreme case - near-zero free blocks of the target order plus a
non-trivial amount of free CMA/highatomic. If the existing order-9 subset
path doesn't need special handling here, the order-2 subset doesn't either,
so I don't plan to change this behavior in the series.

For point 1, the initial free_pages goes through zone_effective_free_pages()
(three paths: blocks / mTHP subset / total), but the percpu_drift snapshot
fallback only has two paths and has no mTHP case, so the mTHP subset falls
into the NR_FREE_PAGES total path and gets overwritten. In v2 I'll make the
snapshot fallback symmetric with zone_effective_free_pages() so the mTHP
path stays consistent (the mTHP count is read directly from free_area and
isn't subject to per-cpu drift anyway). Point 2 is an existing property of
the subset-based watermark check rather than something specific to this
patch. Suggestions welcome if a different overall shape is preferred.

Bo

Re: [RFC PATCH 0/4] mm: compaction: mTHP-friendly memory compaction

[Zi Yan]

On Tue Aug 25, 2026 at 12:38 AM EDT, Bo Zhang wrote:
> Hi all,
>
> This series improves memory compaction to better serve mTHP (multi-size
> THP) allocations, particularly for small orders like order-2 (16KB).
> The changes cover the proactive compaction and kswapd-triggered compaction
> paths.
>
> Problem:
>
> The current proactive compaction targets COMPACTION_HPAGE_ORDER (order-9,
> 2MB), which is unfriendly to mTHP in two ways:
>
> 1. It migrates folios that already satisfy mTHP allocation needs. For
>    example, an order-2 folio is a valid mTHP page, yet compaction still
>    moves it around trying to form order-9 blocks. This is unnecessary
>    work and wastes energy.

But skip_isolation_on_order() skips a folio with an order >= target
order.

>
> 2. Compaction designed for 2MB huge pages is too heavyweight for mTHP.

Yes.

>    mTHP allocations are frequent and only require small contiguous blocks
>    (e.g., 4 pages for order-2). A lighter-weight, mTHP-aware compaction
>    strategy is needed to reduce overhead and power consumption.
>
> Approach:
>
> This series makes four changes:
>
> 1. Generalize the fragmentation score functions to accept an order parameter
>    and use the minimum always-enabled mTHP order as the compaction target.

Makes sense.

>
> 2. During proactive compaction (compact_memory), skip isolating folios that
>    already satisfy mTHP requirements, avoiding unnecessary migration overhead.

Oh, you are targeting proactive compaction, where
skip_isolation_on_order() does not apply.

>
> 3. Allow proactive compaction to proceed concurrently with kswapd for
>    non-costly mTHP orders, since kswapd reclaim alone may not produce the
>    contiguous blocks needed for these allocations.
>
> 4. Introduce zone_effective_free_pages() that provides mTHP-aware free page
>    accounting for watermark checks, counting only buddy blocks that can
>    actually satisfy mTHP allocations.
>
> Test setup:
>
> Boot Ubuntu with 2700 MB of memory, with mTHP disabled
> initially and the defrag mode set to defer+madvise.
>
> Then set the 16 KB mTHP size to always and run a kernel
> build with -j20.
>
> For both cases below, we run a background script to
> proactively trigger compaction as follows:
>  #!/bin/bash
>
>  while true; do
>  	echo 50 > /proc/sys/vm/compaction_proactiveness
>  	sleep 0.1
>  done
>
> W/o patch:
>
> *** Executing round 0 ***
>
> real	2m1.312s
> user	25m38.968s
> sys	5m16.934s
> anon_fault_alloc: 6328991
> anon_fault_fallback: 214553
>
> *** Executing round 1 ***
>
> real	1m55.370s
> user	25m24.482s
> sys	3m52.512s
> anon_fault_alloc: 6355263
> anon_fault_fallback: 108692
>
> *** Executing round 2 ***
>
> real	2m7.579s
> user	25m11.530s
> sys	3m45.456s
> anon_fault_alloc: 6355816
> anon_fault_fallback: 107852
>
> *** Executing round 3 ***
>
> real	1m53.824s
> user	25m26.774s
> sys	3m42.160s
> anon_fault_alloc: 6355457
> anon_fault_fallback: 107705
>
> W/patch:
>
> *** Executing round 0 ***
>
> real	1m55.906s
> user	25m16.985s
> sys	4m24.480s
> anon_fault_alloc: 6354486
> anon_fault_fallback: 109845
>
> *** Executing round 1 ***
>
> real	1m51.303s
> user	25m16.456s
> sys	3m26.629s
> anon_fault_alloc: 6392515
> anon_fault_fallback: 69797
>
> *** Executing round 2 ***
>
> real	1m51.495s
> user	25m13.075s
> sys	3m28.501s
> anon_fault_alloc: 6395096
> anon_fault_fallback: 67510
>
> *** Executing round 3 ***
>
> real	1m52.980s
> user	25m8.217s
> sys	3m37.506s
> anon_fault_alloc: 6389556
> anon_fault_fallback: 72743
>
> Before "Executing round 0", mTHP is not enabled. Therefore, both
> cases show a higher anon_fault_fallback in Round 0 than in the
> other rounds. With the patch, however, memory can be compacted faster
> into an mTHP-friendly state, resulting in a much lower fallback rate
> in Round 0. In the other rounds, the patch also consistently shows
> a lower anon_fault_fallback, as well as lower sys and wall time for
> the kernel build.

What about the impact on THP compaction? How does it affect direct
compaction for both mTHP and THP?

It sounds to me that this patch series target proactive compaction. Am I
getting right?

Thanks.

>
> Open questions:
>
> 1. When multiple mTHP orders are enabled (e.g., order-2 and order-4 both
>    "always"), this series only targets the minimum order. Should proactive
>    compaction also independently evaluate and serve higher orders?
>
> 2. In skip_isolation_on_order(), the filter order ideally should come from
>    the compaction control path. However, during proactive compaction
>    target_order is always -1 (via compact_memory), and there is no clean
>    way to pass the mTHP order down from upper layers. Currently we read
>    huge_anon_orders_always directly, but this variable can be changed by
>    userspace at any time, making the semantic fragile (the compaction may
>    start with one order target and finish with another). Ideas on how to
>    plumb the target order through the proactive compaction path cleanly
>    are welcome.
>
> 3. In __compact_finished(), the original code skips proactive compaction
>    when kswapd is running to avoid interference. Patch 3 removes this
>    skip for non-costly mTHP orders (< PAGE_ALLOC_COSTLY_ORDER). The
>    reason is that small-order compaction is lightweight and likely to
>    succeed quickly even while kswapd is reclaiming, forming an order-2
>    block requires migrating very few pages. Does this approach make sense,
>    or is there a better way to coordinate proactive compaction with kswapd
>    in the mTHP scenario?
>
> 4. In the direct reclaim path (__alloc_pages_slowpath), compact_first is
>    only set for costly orders or non-movable allocations. For mTHP always-
>    enabled non-costly orders (e.g., order-2 MIGRATE_MOVABLE), when free
>    memory is sufficient (watermarks met) but fragmentation is high,
>    compaction is more appropriate than reclaim. Should we also set
>    compact_first for this case to avoid unnecessary reclaim?
>
> Bo Zhang (4):
>   mm: compaction: make proactive compaction mTHP-aware
>   mm: compaction: skip isolating large folios that satisfy the mTHP order
>   mm: compaction: don't skip proactive compaction for non-costly mTHP
>   mm: adjust free_pages to make __zone_watermark_ok() mTHP-aware
>
>  mm/compaction.c | 83 ++++++++++++++++++++++++++++++++++++++-----------
>  mm/internal.h   |  3 ++
>  mm/vmscan.c     | 23 +++-----------
>  3 files changed, 72 insertions(+), 37 deletions(-)
>
> --
> 2.34.1




-- 
Best Regards,
Yan, Zi

Re: [RFC PATCH 0/4] mm: compaction: mTHP-friendly memory compaction

[Bo Zhang]

On Sun Sep 07, 2026 at 10:51 PM EDT, Zi Yan wrote:
> But skip_isolation_on_order() skips a folio with an order >= target
> order.
> ...
> Oh, you are targeting proactive compaction, where
> skip_isolation_on_order() does not apply.

Right. To clarify the cover letter: the "migrating folios that already
satisfy mTHP" concern is specific to proactive compaction, where
target_order is -1 (via compact_memory) and the order >= target_order
check in skip_isolation_on_order() does not apply. For compaction with an
explicit target order that path already handles it.

> What about the impact on THP compaction? How does it affect direct
> compaction for both mTHP and THP?
>
> It sounds to me that this patch series target proactive compaction. Am I
> getting right?

Two things:

1) Traditional THP (order-9) is not affected. The mTHP-aware branch in
zone_effective_free_pages() only triggers when order == compact_hpage_order(),
i.e. the minimum always-enabled mTHP order (e.g. order-2). An order-9 THP
request does not match that, so it keeps its original behavior exactly
(NR_FREE_PAGES_BLOCKS under defrag_mode, NR_FREE_PAGES otherwise). We
didn't change the THP path.

2) The series isn't limited to proactive compaction. Patches 1-3 target
proactive compaction, but patch 4 also covers the kswapd -> kcompactd path
via pgdat_balanced(), so both proactive compaction and kswapd wakeup are
addressed.

For direct compaction: patch 4 does touch compaction_suit_allocation_order(),
which is shared with direct compaction, so order-2 mTHP direct compaction
would also fall into the new accounting. However, the direct compaction case
needs more testing and thought. How direct compaction should behave for mTHP
is something worth discussing together to decide the right approach.

Thanks for the review.

Bo