[PATCH v3] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[PATCH v3] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Salvatore Dipietro]

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
introduced high-order folio allocations in the iomap buffered write
path.  When memory is fragmented, each failed costly-order allocation
enters __alloc_pages_slowpath() which runs direct compaction and
drain_all_pages(), causing a 0.45x throughput drop on PostgreSQL
pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct compaction is too expensive for hot
allocation paths that have fallbacks to smaller allocations.
__filemap_get_folio_mpol() already marks higher-order allocations with
__GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle
failure.  However, the page allocator still attempts full direct
compaction for costly orders with __GFP_NORETRY, which is unnecessarily
aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, skip direct compaction
but wake kcompactd on the preferred node so that background compaction
can defragment memory for future allocations, and return failure
immediately so the caller can fall back.

This keeps compaction working for long-term system health while
removing it from the latency-critical direct allocation path.

Test environment:
  Hardware:  AWS EC2 m8g.24xlarge (96 vCPU, arm64)
             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    next-20260707
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):

  Config                   Avg TPS      % vs Baseline
  baseline (no patch)       70,389.24    -
  With this patch          154,977.02   +120.17%


Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1]
Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2]
Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
Cc: stable@vger.kernel.org
Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>
---
v3: Move to mm/page_alloc.c, wake kcompactd instead of avoiding it
v2: Move from fs/iomap/buffered-io.c to mm/filemap.c
v1: Avoid compaction in iomap folio allocation

 mm/page_alloc.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index a63733dac659..2d02703d8f0f 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4883,6 +4883,26 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	/* If allocation has taken excessively long, warn about it */
 	check_alloc_stall_warn(gfp_mask, ac->nodemask, order, alloc_start_time);
 
+	/*
+	 * Costly allocations with __GFP_NORETRY are opportunistic - Don't
+	 * stall on direct compaction or reclaim; instead, kick
+	 * kcompactd on the preferred node so large pages may become
+	 * available for future allocations and let the caller fall back now.
+	 *
+	 * Direct compaction is way too costly for hot allocation paths on
+	 * large systems: each attempt calls drain_all_pages() which IPIs
+	 * every CPU.  Only wake kcompactd on the local node to avoid
+	 * cross-NUMA interference with unrelated workloads.
+	 */
+	if (costly_order && (gfp_mask & __GFP_NORETRY)) {
+		struct zone *preferred_zone = ac->preferred_zoneref->zone;
+
+		if (preferred_zone)
+			wakeup_kcompactd(preferred_zone->zone_pgdat, order,
+					 ac->highest_zoneidx);
+		goto nopage;
+	}
+
 	/* Try direct reclaim and then allocating */
 	if (!compact_first) {
 		page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags,
-- 
2.47.3




AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

Re: [PATCH v3] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Johannes Weiner]

On Fri, Jul 10, 2026 at 02:34:37PM +0000, Salvatore Dipietro wrote:
> Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
> introduced high-order folio allocations in the iomap buffered write
> path.  When memory is fragmented, each failed costly-order allocation
> enters __alloc_pages_slowpath() which runs direct compaction and
> drain_all_pages(), causing a 0.45x throughput drop on PostgreSQL
> pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.
> 
> The root issue is that direct compaction is too expensive for hot
> allocation paths that have fallbacks to smaller allocations.
> __filemap_get_folio_mpol() already marks higher-order allocations with
> __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle
> failure.  However, the page allocator still attempts full direct
> compaction for costly orders with __GFP_NORETRY, which is unnecessarily
> aggressive when the caller will simply retry at a lower order.
> 
> For costly-order allocations with __GFP_NORETRY, skip direct compaction
> but wake kcompactd on the preferred node so that background compaction
> can defragment memory for future allocations, and return failure
> immediately so the caller can fall back.
> 
> This keeps compaction working for long-term system health while
> removing it from the latency-critical direct allocation path.
> 
> Test environment:
>   Hardware:  AWS EC2 m8g.24xlarge (96 vCPU, arm64)
>              12x 1TB IO2 32000 IOPS RAID0 XFS
>   OS:        AL2023
>   Kernel:    next-20260707
>   Database:  PostgreSQL 18.4
>   Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s
> 
> Results (average of 3 runs, TPS):
> 
>   Config                   Avg TPS      % vs Baseline
>   baseline (no patch)       70,389.24    -
>   With this patch          154,977.02   +120.17%
> 
> 
> Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1]
> Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2]
> Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
> Cc: stable@vger.kernel.org
> Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>
> ---
> v3: Move to mm/page_alloc.c, wake kcompactd instead of avoiding it
> v2: Move from fs/iomap/buffered-io.c to mm/filemap.c
> v1: Avoid compaction in iomap folio allocation
> 
>  mm/page_alloc.c | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index a63733dac659..2d02703d8f0f 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -4883,6 +4883,26 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>  	/* If allocation has taken excessively long, warn about it */
>  	check_alloc_stall_warn(gfp_mask, ac->nodemask, order, alloc_start_time);
>  
> +	/*
> +	 * Costly allocations with __GFP_NORETRY are opportunistic - Don't
> +	 * stall on direct compaction or reclaim; instead, kick
> +	 * kcompactd on the preferred node so large pages may become
> +	 * available for future allocations and let the caller fall back now.
> +	 *
> +	 * Direct compaction is way too costly for hot allocation paths on
> +	 * large systems: each attempt calls drain_all_pages() which IPIs
> +	 * every CPU.  Only wake kcompactd on the local node to avoid
> +	 * cross-NUMA interference with unrelated workloads.
> +	 */
> +	if (costly_order && (gfp_mask & __GFP_NORETRY)) {

I think that's reasonable. __GFP_NORETRY is a bad name, in practice it
just means try not too hard. One direct attempt at say order-0 is
fine; a direct attempt at order-8 is something entirely different.

The callsites COULD judiciously use __GFP_DIRECT_RECLAIM but I don't
think we want to burden them with that. And as much as I hate that
kind of arbitrary costly_order gating, having every caller make up its
own rules would be even worse.

> +		struct zone *preferred_zone = ac->preferred_zoneref->zone;
> +
> +		if (preferred_zone)
> +			wakeup_kcompactd(preferred_zone->zone_pgdat, order,
> +					 ac->highest_zoneidx);

Let's not do that, though. The page allocator doesn't wake kcompactd
directly - it's subordinate to kswapd because it needs free pages to
operate. The coordination code is in kswapd. And that's woken up
further up the function.

The problem is we're lying to it by passing __GFP_DIRECT_RECLAIM when
in fact we categorically don't do that for this request.

And we're lying to various other can_direct_reclaim and can_compact
gates inside the slowpath itself.

That's not good.

My suggestion would be to clear __GFP_DIRECT_RECLAIM first thing in
the slowpath function. Before that nofail branch. Before
can_direct_reclaim and can_compact are set. So that everything in the
function that checks these works properly. And then you'll get kswapd
-> kcompactd wakes for that request.

> +		goto nopage;
> +	}
> +
>  	/* Try direct reclaim and then allocating */
>  	if (!compact_first) {
>  		page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags,
> -- 
> 2.47.3

Re: [PATCH v3] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Matthew Wilcox]

On Fri, Jul 10, 2026 at 02:34:37PM +0000, Salvatore Dipietro wrote:
> Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
> introduced high-order folio allocations in the iomap buffered write
> path.

https://lore.kernel.org/linux-mm/aeZzP6iQel-tkZOu@casper.infradead.org/

I just had a go at implementing what I thought might be the right design
(having a folio_alloc_orders(min, max, gfp)), but that's not really what
__filemap_get_folio_mpol() wants because it needs to integrate the actual
adding of folios to the page cache into the retry loop.

So instead, let's try this.  The idea is that we want to try direct reclaim
_twice_.  Once gently (ie with NORETRY specified) when we're trying to
allocate the maximum order folio.  But now that we've tried that once,
there's no point trying direct reclaim for other sizes, we just want to
ask the page allocator if it can give us memory of any subsequent size.

Until we come to the minimum order.  Then we want to try exactly as hard
as we were originally asked to try.  So revert to the original gfp flags
and don't set the NOWARN or NORETRY flags.

diff --git a/mm/filemap.c b/mm/filemap.c
index 58eb9d240643..23eecaf9b328 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1983,6 +1983,7 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
 	if (!folio && (fgp_flags & FGP_CREAT)) {
 		unsigned int min_order = mapping_min_folio_order(mapping);
 		unsigned int order = max(min_order, FGF_GET_ORDER(fgp_flags));
+		gfp_t alloc_gfp = gfp | __GFP_NORETRY | __GFP_NOWARN;
 		int err;
 		index = mapping_align_index(mapping, index);
 
@@ -2004,12 +2005,11 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
 			order = __ffs(index);
 
 		do {
-			gfp_t alloc_gfp = gfp;
-
 			err = -ENOMEM;
-			if (order > min_order)
-				alloc_gfp |= __GFP_NORETRY | __GFP_NOWARN;
+			if (order == min_order)
+				alloc_gfp = gfp;
 			folio = filemap_alloc_folio(alloc_gfp, order, policy);
+			alloc_gfp &= ~__GFP_DIRECT_RECLAIM;
 			if (!folio)
 				continue;