[PATCH v2 0/2] mm: page_alloc: fix reserve access for non-blocking requests

[PATCH v2 0/2] mm: page_alloc: fix reserve access for non-blocking requests

[Johannes Weiner]

Hi Andrew,

Two fixes for how non-blocking allocations interact with the
highatomic reserve. v1 was posted inside Salvatore's thread on
skipping direct reclaim and compaction for costly __GFP_NORETRY
allocations [1]. This is the same code as a standalone series, rebased
to mm-hotfixes-unstable, with the changelog clarifications from
review. Salvatore's patch is parked in mm-new behind these [2]:
mm-page_alloc-avoid-direct-reclaim-and-compaction-for-costly-__gfp_noretry-allocations.patch

Patch 1 narrows which requests may physically take highatomic pages,
patch 2 narrows which requests may count them in their watermark check.

Both are prerequisites for Salvatore's patch: it clears
__GFP_DIRECT_RECLAIM for costly __GFP_NORETRY requests, which turns
every large page cache folio allocation that reaches the slowpath into
a non-blocking one. Without patch 1, those requests may fall back to
the highatomic freelists reserved for atomic requests. Without patch
2, they count the free highatomic pages in their watermark check while
being served from regular memory, so they can consume regular
high-order blocks below the min watermark by the size of the
reserve. GFP_NOWAIT requests hit both paths today, so these are
existing problems worthy of their own stable backports; but
Salvatore's patch would widen them significantly.

Patch 2 reproduction: a GFP_NOWAIT burst under memory pressure can
drain regular free memory to zero while the highatomic reserve sits
idle. Measured on a 32G box with an anonymous working set, swap, and a
full 290M reserve, three synthetic bursts drove the 28G Normal zone's
regular free memory to 28M, 0.8M and 0.6M against a 60M min
watermark. Swapout failed its swap table allocation inside reclaim,
reclaim scanned 13M pages to free 200k, and page faults stalled for
tens to hundreds of milliseconds. With the fix the burst stops at the
min watermark and nothing else happens. This was found in review and
reproduced synthetically; it has not been observed in production.

GFP_ATOMIC order-0 requests have a milder version of the same problem:
They are entitled to the reserve but are served regular memory first,
so the fix there has to change the watermark check and the freelist
order together. Salvatore's change doesn't interact with this, though;
fixing that is left for a separate patch.

Changes since v1:
- rebased to mm-hotfixes-unstable (2026-10-27)
- 2/2: changelog states the bug was found in review and reproduced
  with a synthetic burst rather than observed in production
  (Vlastimil); notes the remaining GFP_ATOMIC order-0 problem as
  follow-up work
- kept the ALLOC_MASK_ATOMIC check inline in rmqueue_buddy() rather
  than adding a helper, as discussed: it pairs with the
  ALLOC_HIGHATOMIC attempt above it, which is what makes it order-0
  by construction

[1] https://lore.kernel.org/linux-mm/arFBWvzXPYrKuIYi@cmpxchg.org/
[2] https://lore.kernel.org/linux-mm/20261001082152.2879289-1-dipiets@amazon.it/

 mm/page_alloc.c |  4 +++-
 mm/page_alloc.h | 14 ++++++++------
 2 files changed, 11 insertions(+), 7 deletions(-)

[PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

[Johannes Weiner]

281dd25c1a01 ("mm/page_alloc: let GFP_ATOMIC order-0 allocs access
highatomic reserves") accidentally gave all non-blocking allocation
requests access to highatomic reserves, which includes GFP_NOWAIT and
other sites clearing __GFP_DIRECT_RECLAIM.

While this improved atomic request success rate, it's unintentionally
broad and can actually worsen highatomic requests by squandering the
reserves on requests that don't need it.

There are concurrent efforts to clear __GFP_DIRECT_RECLAIM altogether
for costly order __GFP_NORETRY requests, which would then also fall
into this exemption and deplete reserves even faster.

What GFP_ATOMIC has over the others is __GFP_HIGH, which
alloc_flags_slowpath() translates to ALLOC_MIN_RESERVE.

Narrow the exemption to contexts that have both ALLOC_NON_BLOCK |
ALLOC_MIN_RESERVE.

Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260904115629.3993331-1-dipiets@amazon.it
Fixes: 281dd25c1a01 ("mm/page_alloc: let GFP_ATOMIC order-0 allocs access highatomic reserves")
Cc: stable@vger.kernel.org
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
---
 mm/page_alloc.c | 4 +++-
 mm/page_alloc.h | 3 +++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index a8cac5e01066..8ac2e2bf14fc 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3238,7 +3238,9 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
 			 * reserves as failing now is worse than failing a
 			 * high-order atomic allocation in the future.
 			 */
-			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
+			if (!page &&
+			    ((alloc_flags & ALLOC_OOM) ||
+			     (alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC))
 				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
 
 			if (!page) {
diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index e238ee91ed24..9c3c2830acb0 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -60,6 +60,9 @@
 /* Flags that allow allocations below the min watermark. */
 #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
 
+/* Flags that mean GFP_ATOMIC */
+#define ALLOC_MASK_ATOMIC (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE)
+
 /*
  * Structure for holding the mostly immutable allocation parameters passed
  * between functions involved in allocations, including the alloc_pages*
-- 
2.56.0

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

[Balbir Singh]

On Wed, Oct 28, 2026 at 01:15:53PM -0400, Johannes Weiner wrote:
> 281dd25c1a01 ("mm/page_alloc: let GFP_ATOMIC order-0 allocs access
> highatomic reserves") accidentally gave all non-blocking allocation
> requests access to highatomic reserves, which includes GFP_NOWAIT and
> other sites clearing __GFP_DIRECT_RECLAIM.
> 
> While this improved atomic request success rate, it's unintentionally
> broad and can actually worsen highatomic requests by squandering the
> reserves on requests that don't need it.
> 
> There are concurrent efforts to clear __GFP_DIRECT_RECLAIM altogether
> for costly order __GFP_NORETRY requests, which would then also fall
> into this exemption and deplete reserves even faster.
> 
> What GFP_ATOMIC has over the others is __GFP_HIGH, which
> alloc_flags_slowpath() translates to ALLOC_MIN_RESERVE.
> 
> Narrow the exemption to contexts that have both ALLOC_NON_BLOCK |
> ALLOC_MIN_RESERVE.
> 
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Link: https://sashiko.dev/#/patchset/20260904115629.3993331-1-dipiets@amazon.it
> Fixes: 281dd25c1a01 ("mm/page_alloc: let GFP_ATOMIC order-0 allocs access highatomic reserves")
> Cc: stable@vger.kernel.org
> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
> ---
>  mm/page_alloc.c | 4 +++-
>  mm/page_alloc.h | 3 +++
>  2 files changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index a8cac5e01066..8ac2e2bf14fc 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -3238,7 +3238,9 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
>  			 * reserves as failing now is worse than failing a
>  			 * high-order atomic allocation in the future.
>  			 */
> -			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
> +			if (!page &&
> +			    ((alloc_flags & ALLOC_OOM) ||
> +			     (alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC))
>  				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
>  
>  			if (!page) {
> diff --git a/mm/page_alloc.h b/mm/page_alloc.h
> index e238ee91ed24..9c3c2830acb0 100644
> --- a/mm/page_alloc.h
> +++ b/mm/page_alloc.h
> @@ -60,6 +60,9 @@
>  /* Flags that allow allocations below the min watermark. */
>  #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
>  
> +/* Flags that mean GFP_ATOMIC */
> +#define ALLOC_MASK_ATOMIC (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE)
> +
>  /*
>   * Structure for holding the mostly immutable allocation parameters passed
>   * between functions involved in allocations, including the alloc_pages*

I feel patch 2 needs to come first, so that
__zone_watermark_unusable_free() accounts correctly.

Acked-by: Balbir Singh <balbirs@nvidia.com>

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

[Vlastimil Babka (SUSE)]

On 10/28/26 18:15, Johannes Weiner wrote:
> 281dd25c1a01 ("mm/page_alloc: let GFP_ATOMIC order-0 allocs access
> highatomic reserves") accidentally gave all non-blocking allocation
> requests access to highatomic reserves, which includes GFP_NOWAIT and
> other sites clearing __GFP_DIRECT_RECLAIM.
> 
> While this improved atomic request success rate, it's unintentionally
> broad and can actually worsen highatomic requests by squandering the
> reserves on requests that don't need it.
> 
> There are concurrent efforts to clear __GFP_DIRECT_RECLAIM altogether
> for costly order __GFP_NORETRY requests, which would then also fall
> into this exemption and deplete reserves even faster.
> 
> What GFP_ATOMIC has over the others is __GFP_HIGH, which
> alloc_flags_slowpath() translates to ALLOC_MIN_RESERVE.
> 
> Narrow the exemption to contexts that have both ALLOC_NON_BLOCK |
> ALLOC_MIN_RESERVE.
> 
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Link: https://sashiko.dev/#/patchset/20260904115629.3993331-1-dipiets@amazon.it
> Fixes: 281dd25c1a01 ("mm/page_alloc: let GFP_ATOMIC order-0 allocs access highatomic reserves")
> Cc: stable@vger.kernel.org
> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>

Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>

> ---
>  mm/page_alloc.c | 4 +++-
>  mm/page_alloc.h | 3 +++
>  2 files changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index a8cac5e01066..8ac2e2bf14fc 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -3238,7 +3238,9 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
>  			 * reserves as failing now is worse than failing a
>  			 * high-order atomic allocation in the future.
>  			 */
> -			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
> +			if (!page &&
> +			    ((alloc_flags & ALLOC_OOM) ||
> +			     (alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC))
>  				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
>  
>  			if (!page) {
> diff --git a/mm/page_alloc.h b/mm/page_alloc.h
> index e238ee91ed24..9c3c2830acb0 100644
> --- a/mm/page_alloc.h
> +++ b/mm/page_alloc.h
> @@ -60,6 +60,9 @@
>  /* Flags that allow allocations below the min watermark. */
>  #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
>  
> +/* Flags that mean GFP_ATOMIC */
> +#define ALLOC_MASK_ATOMIC (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE)
> +
>  /*
>   * Structure for holding the mostly immutable allocation parameters passed
>   * between functions involved in allocations, including the alloc_pages*

[PATCH 2/2] mm: page_alloc: remove ALLOC_NON_BLOCK from ALLOC_RESERVES

[Johannes Weiner]

1ebbb21811b7 ("mm/page_alloc: explicitly define how __GFP_HIGH
non-blocking allocations accesses reserves") stopped handing out
reserve access for ALLOC_NON_BLOCK on its own: the extra 25% below the
min watermark is now only granted on top of ALLOC_MIN_RESERVE. But the
flag was left in ALLOC_RESERVES, which produces something odd:

With the ALLOC_RESERVES match, __zone_watermark_unusable_free() doesn't
subtract the free highatomic pages for them. So in the slowpath, they
get to consume regular blocks below the min watermark by the number of
free highatomic pages. The highatomic reserve is capped at 1% of the
zone, which on any decently sized machine is a multiple of the min
watermark: GFP_NOWAIT can drain regular memory to zero.

The user-visible result is brutal hiccups during bursts of GFP_NOWAIT
allocations under memory pressure. This was found in review and
reproduced with a synthetic burst, not observed in production, but the
sequence of events is a realistic one: On a 32G box with an anonymous
working set, swap, and a filled 290M highatomic reserve, a GFP_NOWAIT
burst drove regular free memory in the 28G Normal zone (min=60M) to
28M, 0.8M and 0.6M in three runs. Swapout failed to allocate its swap
table, reclaim scanned 13M pages to reclaim 200k, page faults stalled
for tens to hundreds of milliseconds. The machine survives it, but not
by design: direct reclaimers eventually fail and start unreserving
highatomic blocks, until the allocation succeeds or the reserve is
gone and the OOM killer runs. That reserve exists for high-order
atomic allocations; here it is destroyed to bail out a GFP_NOWAIT
consumer that was never entitled to the memory.

Remove ALLOC_NON_BLOCK from ALLOC_RESERVES. With that, the GFP_NOWAIT
burst is stopped short at the min watermark. No direct reclaim, no
stalls, no failed allocations, and the highatomic reserve stays intact
for the requests it exists for.

__zone_watermark_ok() is unaffected, since everything it keys on
ALLOC_NON_BLOCK is already nested under ALLOC_MIN_RESERVE. Update the
flag comments accordingly: ALLOC_NON_BLOCK just means the caller can't
block; the reserve math belongs with ALLOC_MIN_RESERVE.

GFP_ATOMIC order-0 requests have a milder version of the same problem.
Through ALLOC_MIN_RESERVE they also count the free highatomic pages in
their watermark check, but rmqueue_buddy() serves them regular pages
first and only falls back to the reserve once those are gone. They can
still push regular free memory to 37.5% of min minus the free
highatomic pages before the reserve is touched. Unlike GFP_NOWAIT they
are entitled to the reserve, so this needs the watermark check and the
freelist order reworked together, and is left to a separate patch.

Fixes: 1ebbb21811b7 ("mm/page_alloc: explicitly define how __GFP_HIGH non-blocking allocations accesses reserves")
Cc: stable@vger.kernel.org
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
---
 mm/page_alloc.h | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index 9c3c2830acb0..7097ea1805f6 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -32,12 +32,11 @@
 #define ALLOC_OOM		ALLOC_NO_WATERMARKS
 #endif
 
-#define ALLOC_NON_BLOCK		 0x10 /* Caller cannot block. Allow access
-				       * to 25% of the min watermark or
-				       * 62.5% if __GFP_HIGH is set.
-				       */
+#define ALLOC_NON_BLOCK		 0x10 /* Caller cannot block. */
 #define ALLOC_MIN_RESERVE	 0x20 /* __GFP_HIGH set. Allow access to 50%
-				       * of the min watermark.
+				       * of the min watermark, or 62.5% if
+				       * the caller cannot block either
+				       * (ALLOC_NON_BLOCK).
 				       */
 #define ALLOC_CPUSET		 0x40 /* check for correct cpuset */
 #define ALLOC_CMA		 0x80 /* allow allocations from CMA areas */
@@ -58,7 +57,7 @@
 #define ALLOC_NO_CODETAG       0x1000
 
 /* Flags that allow allocations below the min watermark. */
-#define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
+#define ALLOC_RESERVES (ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
 
 /* Flags that mean GFP_ATOMIC */
 #define ALLOC_MASK_ATOMIC (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE)
-- 
2.56.0

Re: [PATCH 2/2] mm: page_alloc: remove ALLOC_NON_BLOCK from ALLOC_RESERVES

[Balbir Singh]

On Wed, Oct 28, 2026 at 01:15:54PM -0400, Johannes Weiner wrote:
> 1ebbb21811b7 ("mm/page_alloc: explicitly define how __GFP_HIGH
> non-blocking allocations accesses reserves") stopped handing out
> reserve access for ALLOC_NON_BLOCK on its own: the extra 25% below the
> min watermark is now only granted on top of ALLOC_MIN_RESERVE. But the
> flag was left in ALLOC_RESERVES, which produces something odd:
> 
> With the ALLOC_RESERVES match, __zone_watermark_unusable_free() doesn't
> subtract the free highatomic pages for them. So in the slowpath, they
> get to consume regular blocks below the min watermark by the number of
> free highatomic pages. The highatomic reserve is capped at 1% of the
> zone, which on any decently sized machine is a multiple of the min
> watermark: GFP_NOWAIT can drain regular memory to zero.
> 
> The user-visible result is brutal hiccups during bursts of GFP_NOWAIT
> allocations under memory pressure. This was found in review and
> reproduced with a synthetic burst, not observed in production, but the
> sequence of events is a realistic one: On a 32G box with an anonymous
> working set, swap, and a filled 290M highatomic reserve, a GFP_NOWAIT
> burst drove regular free memory in the 28G Normal zone (min=60M) to
> 28M, 0.8M and 0.6M in three runs. Swapout failed to allocate its swap
> table, reclaim scanned 13M pages to reclaim 200k, page faults stalled
> for tens to hundreds of milliseconds. The machine survives it, but not
> by design: direct reclaimers eventually fail and start unreserving
> highatomic blocks, until the allocation succeeds or the reserve is
> gone and the OOM killer runs. That reserve exists for high-order
> atomic allocations; here it is destroyed to bail out a GFP_NOWAIT
> consumer that was never entitled to the memory.
> 
> Remove ALLOC_NON_BLOCK from ALLOC_RESERVES. With that, the GFP_NOWAIT
> burst is stopped short at the min watermark. No direct reclaim, no
> stalls, no failed allocations, and the highatomic reserve stays intact
> for the requests it exists for.
> 
> __zone_watermark_ok() is unaffected, since everything it keys on
> ALLOC_NON_BLOCK is already nested under ALLOC_MIN_RESERVE. Update the
> flag comments accordingly: ALLOC_NON_BLOCK just means the caller can't
> block; the reserve math belongs with ALLOC_MIN_RESERVE.
> 
> GFP_ATOMIC order-0 requests have a milder version of the same problem.
> Through ALLOC_MIN_RESERVE they also count the free highatomic pages in
> their watermark check, but rmqueue_buddy() serves them regular pages
> first and only falls back to the reserve once those are gone. They can
> still push regular free memory to 37.5% of min minus the free
> highatomic pages before the reserve is touched. Unlike GFP_NOWAIT they
> are entitled to the reserve, so this needs the watermark check and the
> freelist order reworked together, and is left to a separate patch.
> 
> Fixes: 1ebbb21811b7 ("mm/page_alloc: explicitly define how __GFP_HIGH non-blocking allocations accesses reserves")
> Cc: stable@vger.kernel.org
> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
> ---
>  mm/page_alloc.h | 11 +++++------
>  1 file changed, 5 insertions(+), 6 deletions(-)
> 
> diff --git a/mm/page_alloc.h b/mm/page_alloc.h
> index 9c3c2830acb0..7097ea1805f6 100644
> --- a/mm/page_alloc.h
> +++ b/mm/page_alloc.h
> @@ -32,12 +32,11 @@
>  #define ALLOC_OOM		ALLOC_NO_WATERMARKS
>  #endif
>  
> -#define ALLOC_NON_BLOCK		 0x10 /* Caller cannot block. Allow access
> -				       * to 25% of the min watermark or
> -				       * 62.5% if __GFP_HIGH is set.
> -				       */
> +#define ALLOC_NON_BLOCK		 0x10 /* Caller cannot block. */
>  #define ALLOC_MIN_RESERVE	 0x20 /* __GFP_HIGH set. Allow access to 50%
> -				       * of the min watermark.
> +				       * of the min watermark, or 62.5% if
> +				       * the caller cannot block either
> +				       * (ALLOC_NON_BLOCK).
>  				       */
>  #define ALLOC_CPUSET		 0x40 /* check for correct cpuset */
>  #define ALLOC_CMA		 0x80 /* allow allocations from CMA areas */
> @@ -58,7 +57,7 @@
>  #define ALLOC_NO_CODETAG       0x1000
>  
>  /* Flags that allow allocations below the min watermark. */
> -#define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
> +#define ALLOC_RESERVES (ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
>  
>  /* Flags that mean GFP_ATOMIC */
>  #define ALLOC_MASK_ATOMIC (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE)
>

Acked-by: Balbir Singh <balbirs@nvidia.com>

Re: [PATCH 2/2] mm: page_alloc: remove ALLOC_NON_BLOCK from ALLOC_RESERVES

[Vlastimil Babka (SUSE)]

On 10/28/26 18:15, Johannes Weiner wrote:
> 1ebbb21811b7 ("mm/page_alloc: explicitly define how __GFP_HIGH
> non-blocking allocations accesses reserves") stopped handing out
> reserve access for ALLOC_NON_BLOCK on its own: the extra 25% below the
> min watermark is now only granted on top of ALLOC_MIN_RESERVE. But the
> flag was left in ALLOC_RESERVES, which produces something odd:
> 
> With the ALLOC_RESERVES match, __zone_watermark_unusable_free() doesn't
> subtract the free highatomic pages for them. So in the slowpath, they
> get to consume regular blocks below the min watermark by the number of
> free highatomic pages. The highatomic reserve is capped at 1% of the
> zone, which on any decently sized machine is a multiple of the min
> watermark: GFP_NOWAIT can drain regular memory to zero.
> 
> The user-visible result is brutal hiccups during bursts of GFP_NOWAIT
> allocations under memory pressure. This was found in review and
> reproduced with a synthetic burst, not observed in production, but the
> sequence of events is a realistic one: On a 32G box with an anonymous
> working set, swap, and a filled 290M highatomic reserve, a GFP_NOWAIT
> burst drove regular free memory in the 28G Normal zone (min=60M) to
> 28M, 0.8M and 0.6M in three runs. Swapout failed to allocate its swap
> table, reclaim scanned 13M pages to reclaim 200k, page faults stalled
> for tens to hundreds of milliseconds. The machine survives it, but not
> by design: direct reclaimers eventually fail and start unreserving
> highatomic blocks, until the allocation succeeds or the reserve is
> gone and the OOM killer runs. That reserve exists for high-order
> atomic allocations; here it is destroyed to bail out a GFP_NOWAIT
> consumer that was never entitled to the memory.
> 
> Remove ALLOC_NON_BLOCK from ALLOC_RESERVES. With that, the GFP_NOWAIT
> burst is stopped short at the min watermark. No direct reclaim, no
> stalls, no failed allocations, and the highatomic reserve stays intact
> for the requests it exists for.
> 
> __zone_watermark_ok() is unaffected, since everything it keys on
> ALLOC_NON_BLOCK is already nested under ALLOC_MIN_RESERVE. Update the
> flag comments accordingly: ALLOC_NON_BLOCK just means the caller can't
> block; the reserve math belongs with ALLOC_MIN_RESERVE.
> 
> GFP_ATOMIC order-0 requests have a milder version of the same problem.
> Through ALLOC_MIN_RESERVE they also count the free highatomic pages in
> their watermark check, but rmqueue_buddy() serves them regular pages
> first and only falls back to the reserve once those are gone. They can
> still push regular free memory to 37.5% of min minus the free
> highatomic pages before the reserve is touched. Unlike GFP_NOWAIT they
> are entitled to the reserve, so this needs the watermark check and the
> freelist order reworked together, and is left to a separate patch.
> 
> Fixes: 1ebbb21811b7 ("mm/page_alloc: explicitly define how __GFP_HIGH non-blocking allocations accesses reserves")
> Cc: stable@vger.kernel.org
> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>

Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>

> ---
>  mm/page_alloc.h | 11 +++++------
>  1 file changed, 5 insertions(+), 6 deletions(-)
> 
> diff --git a/mm/page_alloc.h b/mm/page_alloc.h
> index 9c3c2830acb0..7097ea1805f6 100644
> --- a/mm/page_alloc.h
> +++ b/mm/page_alloc.h
> @@ -32,12 +32,11 @@
>  #define ALLOC_OOM		ALLOC_NO_WATERMARKS
>  #endif
>  
> -#define ALLOC_NON_BLOCK		 0x10 /* Caller cannot block. Allow access
> -				       * to 25% of the min watermark or
> -				       * 62.5% if __GFP_HIGH is set.
> -				       */
> +#define ALLOC_NON_BLOCK		 0x10 /* Caller cannot block. */
>  #define ALLOC_MIN_RESERVE	 0x20 /* __GFP_HIGH set. Allow access to 50%
> -				       * of the min watermark.
> +				       * of the min watermark, or 62.5% if
> +				       * the caller cannot block either
> +				       * (ALLOC_NON_BLOCK).
>  				       */
>  #define ALLOC_CPUSET		 0x40 /* check for correct cpuset */
>  #define ALLOC_CMA		 0x80 /* allow allocations from CMA areas */
> @@ -58,7 +57,7 @@
>  #define ALLOC_NO_CODETAG       0x1000
>  
>  /* Flags that allow allocations below the min watermark. */
> -#define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
> +#define ALLOC_RESERVES (ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
>  
>  /* Flags that mean GFP_ATOMIC */
>  #define ALLOC_MASK_ATOMIC (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE)

[PATCH 2/2] mm: page_alloc: remove ALLOC_NON_BLOCK from ALLOC_RESERVES

[Johannes Weiner]

1ebbb21811b7 ("mm/page_alloc: explicitly define how __GFP_HIGH
non-blocking allocations accesses reserves") stopped handing out
reserve access for ALLOC_NON_BLOCK on its own: the extra 25% below the
min watermark is now only granted on top of ALLOC_MIN_RESERVE. But the
flag was left in ALLOC_RESERVES, which produces something odd:

With the ALLOC_RESERVES match, __zone_watermark_unusable_free() doesn't
subtract the free highatomic pages for them. So in the slowpath, they
get to consume regular blocks below the min watermark by the number of
free highatomic pages. The highatomic reserve is capped at 1% of the
zone, which on any decently sized machine is a multiple of the min
watermark: GFP_NOWAIT can drain regular memory to zero.

The user-visible result is brutal hiccups during bursts of GFP_NOWAIT
allocations under memory pressure. On a 32G box with an anonymous
working set, swap, and a filled 290M highatomic reserve, a GFP_NOWAIT
burst drove regular free memory in the 28G Normal zone (min=60M) to
28M, 0.8M and 0.6M in three runs. Swapout failed to allocate its swap
table, reclaim scanned 13M pages to reclaim 200k, page faults stalled
for tens to hundreds of milliseconds. The machine survives it, but not
by design: direct reclaimers eventually fail and start unreserving
highatomic blocks, until the allocation succeeds or the reserve is
gone and the OOM killer runs. That reserve exists for high-order
atomic allocations; here it is destroyed to bail out a GFP_NOWAIT
consumer that was never entitled to the memory.

Remove ALLOC_NON_BLOCK from ALLOC_RESERVES. With that, the GFP_NOWAIT
burst is stopped short at the min watermark. No direct reclaim, no
stalls, no failed allocations, and the highatomic reserve stays intact
for the requests it exists for.

__zone_watermark_ok() is unaffected, since everything it keys on
ALLOC_NON_BLOCK is already nested under ALLOC_MIN_RESERVE. Update the
flag comments accordingly: ALLOC_NON_BLOCK just means the caller can't
block; the reserve math belongs with ALLOC_MIN_RESERVE.

Fixes: 1ebbb21811b7 ("mm/page_alloc: explicitly define how __GFP_HIGH non-blocking allocations accesses reserves")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
---
 mm/page_alloc.h | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index ad89f83d1dab..c8af79decbd0 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -32,12 +32,11 @@
 #define ALLOC_OOM		ALLOC_NO_WATERMARKS
 #endif
 
-#define ALLOC_NON_BLOCK		 0x10 /* Caller cannot block. Allow access
-				       * to 25% of the min watermark or
-				       * 62.5% if __GFP_HIGH is set.
-				       */
+#define ALLOC_NON_BLOCK		 0x10 /* Caller cannot block. */
 #define ALLOC_MIN_RESERVE	 0x20 /* __GFP_HIGH set. Allow access to 50%
-				       * of the min watermark.
+				       * of the min watermark, or 62.5% if
+				       * the caller cannot block either
+				       * (ALLOC_NON_BLOCK).
 				       */
 #define ALLOC_CPUSET		 0x40 /* check for correct cpuset */
 #define ALLOC_CMA		 0x80 /* allow allocations from CMA areas */
@@ -58,7 +57,7 @@
 #define ALLOC_NO_CODETAG       0x1000
 
 /* Flags that allow allocations below the min watermark. */
-#define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
+#define ALLOC_RESERVES (ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
 
 /* Flags that mean GFP_ATOMIC */
 #define ALLOC_MASK_ATOMIC (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE)
-- 
2.55.0

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

[Salvatore Dipietro]

Hi Johannes,

Vlastimil asked that v6 of my page_alloc patch [1] not go to mainline
before these two patches land, so Andrew parked v6 at the tail of
mm-unstable and dropped its Cc: stable until then.

Are you still planning to work on 1/2 and 2/2, and if so, do you have
a rough ETA? Or would you rather post them as a separate series as
they are? If anything is needed on my side to move them forward, I'm
happy to help.

For what it's worth, I ran both stacked under v6 on v7.3-rc1 and saw
no performance difference on my benchmark with or without them.

[1] https://lore.kernel.org/linux-mm/20261001082152.2879289-1-dipiets@amazon.it/


Thanks,
Salvatore





AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

[Christoph Hellwig]

On Tue, Oct 27, 2026 at 03:00:02PM +0000, Salvatore Dipietro wrote:
> 
> Hi Johannes,
> 
> Vlastimil asked that v6 of my page_alloc patch [1] not go to mainline
> before these two patches land, so Andrew parked v6 at the tail of
> mm-unstable and dropped its Cc: stable until then.
> 
> Are you still planning to work on 1/2 and 2/2, and if so, do you have
> a rough ETA? Or would you rather post them as a separate series as
> they are? If anything is needed on my side to move them forward, I'm
> happy to help.
> 
> For what it's worth, I ran both stacked under v6 on v7.3-rc1 and saw
> no performance difference on my benchmark with or without them.

It would be really important to get some fix for this into 7.3.
We can still perfect it later.

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

[Andrew Morton]

On Wed, 28 Oct 2026 15:41:48 +0100 Christoph Hellwig <hch@lst.de> wrote:

> On Tue, Oct 27, 2026 at 03:00:02PM +0000, Salvatore Dipietro wrote:
> > 
> > Hi Johannes,
> > 
> > Vlastimil asked that v6 of my page_alloc patch [1] not go to mainline
> > before these two patches land, so Andrew parked v6 at the tail of
> > mm-unstable and dropped its Cc: stable until then.
> > 
> > Are you still planning to work on 1/2 and 2/2, and if so, do you have
> > a rough ETA? Or would you rather post them as a separate series as
> > they are? If anything is needed on my side to move them forward, I'm
> > happy to help.
> > 
> > For what it's worth, I ran both stacked under v6 on v7.3-rc1 and saw
> > no performance difference on my benchmark with or without them.
> 
> It would be really important to get some fix for this into 7.3.
> We can still perfect it later.

What is "this"?  You're referring to the performance regression which
Salvatore's "mm/page_alloc: avoid direct reclaim and compaction for
costly __GFP_NORETRY allocations" addresses?

In
https://lore.kernel.org/all/9f415dc7-adad-4161-b20d-7c3173f50ff3@kernel.org/
vbabka indicated that Fixes: 281dd25c1a01 is needed on that patch.  The
patch I'm sitting on (below) has Fixes: 5d8edfb900d5.  Please clarify.

In
https://lore.kernel.org/all/c6e2c1e1-a285-406e-a2b6-c09e05df91d8@kernel.org/
vbabka argued that Salvatore's "mm/page_alloc: avoid direct reclaim and
compaction for costly __GFP_NORETRY allocations" should be a 7.4 thing.

So it's all clear as mud.  Can people please clarify?

I have hannes's series "mm: page_alloc: fix reserve access for
non-blocking requests, v2" in mm-hotfixes, planned for 7.4-rcX.

Should I move Salvatore's "mm/page_alloc: avoid direct reclaim and
compaction for costly __GFP_NORETRY allocations, v6" into mm-hotfixes
also, and restore its cc:stable?

My vote: "yes" - that's a pretty obnoxious perf regression.


From: Salvatore Dipietro <dipiets@amazon.it>
Subject: mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations
Date: Thu, 1 Oct 2026 08:21:51 +0000

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
introduced high-order folio allocations in the iomap buffered write path. 
When memory is fragmented, each failed costly-order allocation enters
__alloc_pages_slowpath() which runs direct reclaim, direct compaction and
drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench
(simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct reclaim and direct compaction are too
expensive for hot allocation paths that have fallbacks to smaller
allocations.  __filemap_get_folio_mpol() already marks higher-order
allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller
can handle failure.  However, the page allocator still enters the full
blocking slowpath for costly orders with __GFP_NORETRY, which is
unnecessarily aggressive when the caller will simply retry at a lower
order.

For costly-order allocations with __GFP_NORETRY, clear
__GFP_DIRECT_RECLAIM at the very start of the slowpath, before
can_direct_reclaim, can_compact and the nofail checks are evaluated.  This
makes the entire slowpath treat the request as non-blocking: no direct
reclaim, no direct compaction and no drain_all_pages() IPI across every
CPU.  Reclaim and compaction move to the background instead: kswapd is
still woken for reclaim, and it wakes kcompactd for defragmentation.  The
system keeps its long-term health while the latency-critical direct
allocation path no longer stalls.

Allocations that also request __GFP_THISNODE are exempted.  That flag
pairing identifies the local-node-first THP attempt issued by
alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to
form transparent huge pages.

Clearing __GFP_DIRECT_RECLAIM makes these allocations reach the
!can_direct_reclaim bailout, which since commit 4c0ed883e051
("mm/page_alloc: fix defrag_mode for non-reclaimable allocations") drops
ALLOC_NOFRAGMENT and retries instead of failing when defrag_mode is on. 
With vm.defrag_mode=1 a __GFP_NORETRY allocation could therefore succeed
by fragmenting another migratetype's pageblock.  That is the wrong trade
for a caller that has already communicated it can handle failure and has a
cheap lower-order fallback, so exclude __GFP_NORETRY from the relaxation. 
The exclusion is not restricted to costly orders: any __GFP_NORETRY caller
has said it can cope with failure, so failing is preferable to fragmenting
for all of them.

Test environment:
  Hardware:  AWS EC2 m8g.24xlarge (96 vCPU, arm64)
             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):

  Config                                       Avg TPS   % vs baseline
  v7.3-rc1 baseline (vm.defrag_mode=0)          58,602               -
  v7.3-rc1 + this patch (vm.defrag_mode=0)     157,246         +168.3%
  v7.3-rc1 baseline (vm.defrag_mode=1)         109,012               -
  v7.3-rc1 + this patch (vm.defrag_mode=1)     157,778          +44.7%

Link: https://lore.kernel.org/20261001082152.2879289-1-dipiets@amazon.it
Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1]
Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2]
Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@amazon.it/T/#u [v3]
Link: https://lore.kernel.org/all/20260904115629.3993331-1-dipiets@amazon.it/T/#u [v4]
Link: https://lore.kernel.org/all/20260911142102.2294202-1-dipiets@amazon.it/T/#u [v5]
Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: David Hildenbrand <david@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Dave Chinner <dgc@kernel.org>
Cc: Ritesh Harjani <ritesh.list@gmail.com>
Cc: Dmitry Ilvokhin <d@ilvokhin.com>
Cc: Hazem Mohamed Abuelfotoh <abuehaze@amazon.com>
Cc: Ali Saidi <alisaidi@amazon.com>
Cc: Geoff Blake <blakgeof@amazon.com>
Cc: Christian Brauner <brauner@kernel.org>
Cc: <dipietro.salvatore@gmail.com>
Cc: Darrick J. Wong <djwong@kernel.org>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: <rvvandan@amazon.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: "Garg, Shivank" <shivankg@amd.com>
---

 mm/page_alloc.c |   21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

--- a/mm/page_alloc.c~mm-page_alloc-avoid-direct-reclaim-and-compaction-for-costly-__gfp_noretry-allocations
+++ a/mm/page_alloc.c
@@ -4773,10 +4773,10 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
-	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
-	bool nofail = gfp_mask & __GFP_NOFAIL;
 	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	bool can_direct_reclaim;
+	bool can_compact;
+	bool nofail;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;
@@ -4791,6 +4791,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, u
 	bool can_retry_reserves = true;
 	unsigned long alloc_start_time = jiffies;
 
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
+	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
+	 */
+	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
+	    !(gfp_mask & __GFP_THISNODE))
+		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
+
+	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
+	nofail = gfp_mask & __GFP_NOFAIL;
+
 	if (unlikely(nofail)) {
 		/*
 		 * We don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,
@@ -4907,8 +4919,11 @@ retry:
 		 * Reclaim/compaction cannot run, so defrag_mode's strategy
 		 * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow
 		 * fallbacks rather than failing the allocation outright.
+		 * Not for __GFP_NORETRY: those have a cheap lower order
+		 * fallback, so failing beats fragmenting.
 		 */
 		if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT) &&
+		    !(gfp_mask & __GFP_NORETRY) &&
 		    (gfp_mask & __GFP_KSWAPD_RECLAIM)) {
 			alloc_flags &= ~ALLOC_NOFRAGMENT;
 			goto retry;
_

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

[Vlastimil Babka (SUSE)]

On 10/29/26 06:55, Andrew Morton wrote:
> On Wed, 28 Oct 2026 15:41:48 +0100 Christoph Hellwig <hch@lst.de> wrote:
> 
>> On Tue, Oct 27, 2026 at 03:00:02PM +0000, Salvatore Dipietro wrote:
>> > 
>> > Hi Johannes,
>> > 
>> > Vlastimil asked that v6 of my page_alloc patch [1] not go to mainline
>> > before these two patches land, so Andrew parked v6 at the tail of
>> > mm-unstable and dropped its Cc: stable until then.
>> > 
>> > Are you still planning to work on 1/2 and 2/2, and if so, do you have
>> > a rough ETA? Or would you rather post them as a separate series as
>> > they are? If anything is needed on my side to move them forward, I'm
>> > happy to help.
>> > 
>> > For what it's worth, I ran both stacked under v6 on v7.3-rc1 and saw
>> > no performance difference on my benchmark with or without them.
>> 
>> It would be really important to get some fix for this into 7.3.
>> We can still perfect it later.
> 
> What is "this"?  You're referring to the performance regression which
> Salvatore's "mm/page_alloc: avoid direct reclaim and compaction for
> costly __GFP_NORETRY allocations" addresses?
> 
> In
> https://lore.kernel.org/all/9f415dc7-adad-4161-b20d-7c3173f50ff3@kernel.org/
> vbabka indicated that Fixes: 281dd25c1a01 is needed on that patch.  The

"that patch" being Johannes's and the v2 you picked up has it

> patch I'm sitting on (below) has Fixes: 5d8edfb900d5.  Please clarify.

It's correct.

> In
> https://lore.kernel.org/all/c6e2c1e1-a285-406e-a2b6-c09e05df91d8@kernel.org/
> vbabka argued that Salvatore's "mm/page_alloc: avoid direct reclaim and
> compaction for costly __GFP_NORETRY allocations" should be a 7.4 thing.
> 
> So it's all clear as mud.  Can people please clarify?
> 
> I have hannes's series "mm: page_alloc: fix reserve access for
> non-blocking requests, v2" in mm-hotfixes, planned for 7.4-rcX.
> 
> Should I move Salvatore's "mm/page_alloc: avoid direct reclaim and
> compaction for costly __GFP_NORETRY allocations, v6" into mm-hotfixes
> also, and restore its cc:stable?
> 
> My vote: "yes" - that's a pretty obnoxious perf regression.

Yes please.

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

[Christoph Hellwig]

On Wed, Oct 28, 2026 at 10:55:17PM -0700, Andrew Morton wrote:
> > It would be really important to get some fix for this into 7.3.
> > We can still perfect it later.
> 
> What is "this"?  You're referring to the performance regression which
> Salvatore's "mm/page_alloc: avoid direct reclaim and compaction for
> costly __GFP_NORETRY allocations" addresses?

As in I don't really care how the xfs/iomap performance issue gets fixed,
I just want it fixed yesterday.

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

[Johannes Weiner]

On Tue, Oct 27, 2026 at 03:00:02PM +0000, Salvatore Dipietro wrote:
> 
> Hi Johannes,
> 
> Vlastimil asked that v6 of my page_alloc patch [1] not go to mainline
> before these two patches land, so Andrew parked v6 at the tail of
> mm-unstable and dropped its Cc: stable until then.
> 
> Are you still planning to work on 1/2 and 2/2, and if so, do you have
> a rough ETA? Or would you rather post them as a separate series as
> they are? If anything is needed on my side to move them forward, I'm
> happy to help.
> 
> For what it's worth, I ran both stacked under v6 on v7.3-rc1 and saw
> no performance difference on my benchmark with or without them.
> 
> [1] https://lore.kernel.org/linux-mm/20261001082152.2879289-1-dipiets@amazon.it/

I'll work on this today, will have patches before EOD. Sorry for
dropping this.

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

[Johannes Weiner]

On Wed, Oct 28, 2026 at 11:19:08AM -0400, Johannes Weiner wrote:
> On Tue, Oct 27, 2026 at 03:00:02PM +0000, Salvatore Dipietro wrote:
> > 
> > Hi Johannes,
> > 
> > Vlastimil asked that v6 of my page_alloc patch [1] not go to mainline
> > before these two patches land, so Andrew parked v6 at the tail of
> > mm-unstable and dropped its Cc: stable until then.
> > 
> > Are you still planning to work on 1/2 and 2/2, and if so, do you have
> > a rough ETA? Or would you rather post them as a separate series as
> > they are? If anything is needed on my side to move them forward, I'm
> > happy to help.
> > 
> > For what it's worth, I ran both stacked under v6 on v7.3-rc1 and saw
> > no performance difference on my benchmark with or without them.
> > 
> > [1] https://lore.kernel.org/linux-mm/20261001082152.2879289-1-dipiets@amazon.it/
> 
> I'll work on this today, will have patches before EOD. Sorry for
> dropping this.

https://lore.kernel.org/all/20261028171719.2043421-1-hannes@cmpxchg.org/

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Andrew Morton]

On Fri, 4 Sep 2026 11:56:28 +0000 Salvatore Dipietro <dipiets@amazon.it> wrote:

> Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
> introduced high-order folio allocations in the iomap buffered write
> path.  When memory is fragmented, each failed costly-order allocation
> enters __alloc_pages_slowpath() which runs direct compaction and
> drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL
> pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

Is there anything particularly unusual about this test case?

> Results (average of 3 runs, TPS):
> 
>   Config                   Avg TPS      % vs Baseline
>   baseline (no patch)       59,408       -
>   With this patch          155,409      +161.6%

Is this back to pre-5d8edfb900d5 performance?

> Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1]
> Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2]
> Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@amazon.it/T/#u [v3]
> Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")

That's three years old.  v6.10.  

This is not good.  Can you think of an exceptional reason why this took
so long to surface, or is it simply that our testing isn't good?

And thanks for fixing it.

AI review asked a few serious-looking questions:
	https://sashiko.dev/#/patchset/20260904115629.3993331-1-dipiets@amazon.it

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Salvatore Dipietro]

On Sat, 05 Sep 2026 17:42:39 -0700 Andrew Morton <akpm@linux-foundation.org> wrote:

> Is there anything particularly unusual about this test case?

It is a stock pgbench simple-update PostgreSQL workload on a large
instance (96 vCPUs), using standard PostgreSQL settings and with no huge
pages assigned to the database. We deliberately overprovision the
pgbench clients: 1024 clients over 96 threads. That keeps enough writers
in the buffered write path concurrently to hit the costly-order
allocation failure path continuously. The memory fragmentation comes from 
page tables: PostgreSQL spawns a new process per client, and those page
tables consume ~40% of memory, which significantly limits the page cache
and the free memory available.


> > Results (average of 3 runs, TPS):
> >
> >   Config                   Avg TPS      % vs Baseline
> >   baseline (no patch)       59,408       -
> >   With this patch          155,409      +161.6%
>
> Is this back to pre-5d8edfb900d5 performance?

Yes - fully recovered. Here is the summary, same host and workload
throughout, average of 3 runs:

  Config                                        Avg TPS   % vs baseline
  AL2023 stock 6.1 kernel                       136,942            n/a
  v7.3-rc1 baseline (no patch)                   59,408             -
  v7.3-rc1, 5d8edfb900d5 behaviour reverted     151,184        +154.5%
  v7.3-rc1 + v4                                 155,409        +161.6%

The 6.1 row predates 5d8edfb900d5 entirely. A different kernel version,
so not directly comparable, but it shows the performance this workload
used to get on this host.

A literal "git revert 5d8edfb900d5" does not apply to v7.3-rc1 -
iomap_write_iter() has been rewritten since - so the reverted row is a
one-line behavioural revert, forcing the write loop back to copying at
most PAGE_SIZE per iteration:

  -	size_t chunk = mapping_max_folio_size(mapping);
  +	size_t chunk = PAGE_SIZE;

which is what the pre-5d8edfb900d5 loop computed. That makes
iomap_get_folio() pass no order hint, so the path issues only order-0
allocations.


> AI review asked a few serious-looking questions:
> 	https://sashiko.dev/#/patchset/20260904115629.3993331-1-dipiets@amazon.it

Thanks for pointing that out. To address them, we can have something
like the patch below. Performance results are still similar to v4. Happy
to submit a formal v5 patch with it if you would like.

  Config                                        Avg TPS   % vs baseline
  v7.3-rc1 baseline (no patch)                   59,408             -
  v7.3-rc1 + v4                                 155,409        +161.6%
  v7.3-rc1 + proposed patch                     161,994        +172.7%

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 12fac9084c48..be8b0d72a2db 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4784,10 +4784,22 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback to a lower order,
+	 * so don't stall them in direct reclaim or direct compaction.  Exempt
+	 * __GFP_THISNODE (the THP attempt from alloc_pages_mpol() needs direct
+	 * compaction) and __GFP_NOFAIL (must not be made to fail).  Don't
+	 * clear __GFP_DIRECT_RECLAIM from gfp_mask instead: that would also
+	 * change the alloc_flags derived by alloc_flags_slowpath().
+	 */
+	const bool costly_noretry = costly_order &&
+		(gfp_mask & __GFP_NORETRY) &&
+		!(gfp_mask & (__GFP_THISNODE | __GFP_NOFAIL));
+	bool can_direct_reclaim = !costly_noretry &&
+		(gfp_mask & __GFP_DIRECT_RECLAIM);
 	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
 	bool nofail = gfp_mask & __GFP_NOFAIL;
-	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;


Thanks,
Salvatore



AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Johannes Weiner]

On Thu, Sep 10, 2026 at 11:46:02AM +0000, Salvatore Dipietro wrote:
> 
> On Sat, 05 Sep 2026 17:42:39 -0700 Andrew Morton <akpm@linux-foundation.org> wrote:
> 
> > Is there anything particularly unusual about this test case?
> 
> It is a stock pgbench simple-update PostgreSQL workload on a large
> instance (96 vCPUs), using standard PostgreSQL settings and with no huge
> pages assigned to the database. We deliberately overprovision the
> pgbench clients: 1024 clients over 96 threads. That keeps enough writers
> in the buffered write path concurrently to hit the costly-order
> allocation failure path continuously. The memory fragmentation comes from 
> page tables: PostgreSQL spawns a new process per client, and those page
> tables consume ~40% of memory, which significantly limits the page cache
> and the free memory available.
> 
> 
> > > Results (average of 3 runs, TPS):
> > >
> > >   Config                   Avg TPS      % vs Baseline
> > >   baseline (no patch)       59,408       -
> > >   With this patch          155,409      +161.6%
> >
> > Is this back to pre-5d8edfb900d5 performance?
> 
> Yes - fully recovered. Here is the summary, same host and workload
> throughout, average of 3 runs:
> 
>   Config                                        Avg TPS   % vs baseline
>   AL2023 stock 6.1 kernel                       136,942            n/a
>   v7.3-rc1 baseline (no patch)                   59,408             -
>   v7.3-rc1, 5d8edfb900d5 behaviour reverted     151,184        +154.5%
>   v7.3-rc1 + v4                                 155,409        +161.6%
> 
> The 6.1 row predates 5d8edfb900d5 entirely. A different kernel version,
> so not directly comparable, but it shows the performance this workload
> used to get on this host.
> 
> A literal "git revert 5d8edfb900d5" does not apply to v7.3-rc1 -
> iomap_write_iter() has been rewritten since - so the reverted row is a
> one-line behavioural revert, forcing the write loop back to copying at
> most PAGE_SIZE per iteration:
> 
>   -	size_t chunk = mapping_max_folio_size(mapping);
>   +	size_t chunk = PAGE_SIZE;
> 
> which is what the pre-5d8edfb900d5 loop computed. That makes
> iomap_get_folio() pass no order hint, so the path issues only order-0
> allocations.
> 
> 
> > AI review asked a few serious-looking questions:
> > 	https://sashiko.dev/#/patchset/20260904115629.3993331-1-dipiets@amazon.it
> 
> Thanks for pointing that out. To address them, we can have something
> like the patch below. Performance results are still similar to v4. Happy
> to submit a formal v5 patch with it if you would like.

The feedback looks misleading to me.

ALLOC_NON_BLOCKING itself is fine, it usually only means something in
conjunction with other flags, like __GFP_HIGH. The access to
MIGRATE_HIGHATOMIC that it points out is in itself too generous. This
seems like a real but separate bug. It will allow GFP_TRANSHUGE_LIGHT
into the highatomic reserves as well, for example.

The rt issue seems made up too. We're talking about costly_order &&
__GFP_NORETRY allocations. Why would they need access to reserves?

>   Config                                        Avg TPS   % vs baseline
>   v7.3-rc1 baseline (no patch)                   59,408             -
>   v7.3-rc1 + v4                                 155,409        +161.6%
>   v7.3-rc1 + proposed patch                     161,994        +172.7%
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 12fac9084c48..be8b0d72a2db 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -4784,10 +4784,22 @@ static inline struct page *
>  __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>  						struct alloc_context *ac)
>  {
> -	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
> +	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
> +	/*
> +	 * Costly __GFP_NORETRY callers have a cheap fallback to a lower order,
> +	 * so don't stall them in direct reclaim or direct compaction.  Exempt
> +	 * __GFP_THISNODE (the THP attempt from alloc_pages_mpol() needs direct
> +	 * compaction) and __GFP_NOFAIL (must not be made to fail).  Don't
> +	 * clear __GFP_DIRECT_RECLAIM from gfp_mask instead: that would also
> +	 * change the alloc_flags derived by alloc_flags_slowpath().
> +	 */
> +	const bool costly_noretry = costly_order &&
> +		(gfp_mask & __GFP_NORETRY) &&
> +		!(gfp_mask & (__GFP_THISNODE | __GFP_NOFAIL));
> +	bool can_direct_reclaim = !costly_noretry &&
> +		(gfp_mask & __GFP_DIRECT_RECLAIM);

IMO this is worse because now there is a disconnect between these
flags and the gfp mask again. This sets us up for subtle bugs.

I much preferred v4.

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Vlastimil Babka (SUSE)]

On 9/11/26 17:59, Johannes Weiner wrote:
> On Thu, Sep 10, 2026 at 11:46:02AM +0000, Salvatore Dipietro wrote:
>> 
>> On Sat, 05 Sep 2026 17:42:39 -0700 Andrew Morton <akpm@linux-foundation.org> wrote:
>> 
>> > Is there anything particularly unusual about this test case?
>> 
>> It is a stock pgbench simple-update PostgreSQL workload on a large
>> instance (96 vCPUs), using standard PostgreSQL settings and with no huge
>> pages assigned to the database. We deliberately overprovision the
>> pgbench clients: 1024 clients over 96 threads. That keeps enough writers
>> in the buffered write path concurrently to hit the costly-order
>> allocation failure path continuously. The memory fragmentation comes from 
>> page tables: PostgreSQL spawns a new process per client, and those page
>> tables consume ~40% of memory, which significantly limits the page cache
>> and the free memory available.
>> 
>> 
>> > > Results (average of 3 runs, TPS):
>> > >
>> > >   Config                   Avg TPS      % vs Baseline
>> > >   baseline (no patch)       59,408       -
>> > >   With this patch          155,409      +161.6%
>> >
>> > Is this back to pre-5d8edfb900d5 performance?
>> 
>> Yes - fully recovered. Here is the summary, same host and workload
>> throughout, average of 3 runs:
>> 
>>   Config                                        Avg TPS   % vs baseline
>>   AL2023 stock 6.1 kernel                       136,942            n/a
>>   v7.3-rc1 baseline (no patch)                   59,408             -
>>   v7.3-rc1, 5d8edfb900d5 behaviour reverted     151,184        +154.5%
>>   v7.3-rc1 + v4                                 155,409        +161.6%
>> 
>> The 6.1 row predates 5d8edfb900d5 entirely. A different kernel version,
>> so not directly comparable, but it shows the performance this workload
>> used to get on this host.
>> 
>> A literal "git revert 5d8edfb900d5" does not apply to v7.3-rc1 -
>> iomap_write_iter() has been rewritten since - so the reverted row is a
>> one-line behavioural revert, forcing the write loop back to copying at
>> most PAGE_SIZE per iteration:
>> 
>>   -	size_t chunk = mapping_max_folio_size(mapping);
>>   +	size_t chunk = PAGE_SIZE;
>> 
>> which is what the pre-5d8edfb900d5 loop computed. That makes
>> iomap_get_folio() pass no order hint, so the path issues only order-0
>> allocations.
>> 
>> 
>> > AI review asked a few serious-looking questions:
>> > 	https://sashiko.dev/#/patchset/20260904115629.3993331-1-dipiets@amazon.it
>> 
>> Thanks for pointing that out. To address them, we can have something
>> like the patch below. Performance results are still similar to v4. Happy
>> to submit a formal v5 patch with it if you would like.
> 
> The feedback looks misleading to me.

Agreed.

> ALLOC_NON_BLOCKING itself is fine, it usually only means something in
> conjunction with other flags, like __GFP_HIGH.

+1

> The access to
> MIGRATE_HIGHATOMIC that it points out is in itself too generous. This
> seems like a real but separate bug. It will allow GFP_TRANSHUGE_LIGHT
> into the highatomic reserves as well, for example.

I don't follow this part. For ALLOC_HIGHATOMIC you need __GFP_HIGH in
alloc_flags_nonblocking(). So GFP_TRANSHUGE_LIGHT won't get the access, no?

> The rt issue seems made up too. We're talking about costly_order &&
> __GFP_NORETRY allocations. Why would they need access to reserves?

Yep.

And the concern about warning is also moot as __GFP_NORETRY is still needed
and its users virtually always add __GFP_NOWARN as well.

>>   Config                                        Avg TPS   % vs baseline
>>   v7.3-rc1 baseline (no patch)                   59,408             -
>>   v7.3-rc1 + v4                                 155,409        +161.6%
>>   v7.3-rc1 + proposed patch                     161,994        +172.7%
>> 
>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>> index 12fac9084c48..be8b0d72a2db 100644
>> --- a/mm/page_alloc.c
>> +++ b/mm/page_alloc.c
>> @@ -4784,10 +4784,22 @@ static inline struct page *
>>  __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>>  						struct alloc_context *ac)
>>  {
>> -	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
>> +	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
>> +	/*
>> +	 * Costly __GFP_NORETRY callers have a cheap fallback to a lower order,
>> +	 * so don't stall them in direct reclaim or direct compaction.  Exempt
>> +	 * __GFP_THISNODE (the THP attempt from alloc_pages_mpol() needs direct
>> +	 * compaction) and __GFP_NOFAIL (must not be made to fail).  Don't
>> +	 * clear __GFP_DIRECT_RECLAIM from gfp_mask instead: that would also
>> +	 * change the alloc_flags derived by alloc_flags_slowpath().
>> +	 */
>> +	const bool costly_noretry = costly_order &&
>> +		(gfp_mask & __GFP_NORETRY) &&
>> +		!(gfp_mask & (__GFP_THISNODE | __GFP_NOFAIL));
>> +	bool can_direct_reclaim = !costly_noretry &&
>> +		(gfp_mask & __GFP_DIRECT_RECLAIM);
> 
> IMO this is worse because now there is a disconnect between these
> flags and the gfp mask again. This sets us up for subtle bugs.
> 
> I much preferred v4.

Unless I'm mistaken about that MIGRATE_HIGHATOMIC part, it seems all sashiko
concerns can be dismissed and then indeed v4 is the better version.

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Johannes Weiner]

On Wed, Sep 16, 2026 at 01:24:31PM +0200, Vlastimil Babka (SUSE) wrote:
> On 9/11/26 17:59, Johannes Weiner wrote:
> > The access to
> > MIGRATE_HIGHATOMIC that it points out is in itself too generous. This
> > seems like a real but separate bug. It will allow GFP_TRANSHUGE_LIGHT
> > into the highatomic reserves as well, for example.
> 
> I don't follow this part. For ALLOC_HIGHATOMIC you need __GFP_HIGH in
> alloc_flags_nonblocking(). So GFP_TRANSHUGE_LIGHT won't get the access, no?

rmqueue_buddy() has this:

			/*
			 * If the allocation fails, allow OOM handling and
			 * order-0 (atomic) allocs access to HIGHATOMIC
			 * reserves as failing now is worse than failing a
			 * high-order atomic allocation in the future.
			 */
			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);

It says "atomic", but it's checking only ALLOC_NON_BLOCK, which is
broader than GFP_ATOMIC. alloc_flags_nonblocking();

	if (gfp_mask & __GFP_DIRECT_RECLAIM)
		return 0;

	if (gfp_mask & __GFP_NOMEMALLOC)
		return 0;

	alloc_flags |= ALLOC_NON_BLOCK;

	if (order > 0 && (gfp_mask & __GFP_HIGH))
		alloc_flags |= ALLOC_HIGHATOMIC;

So this can apply to random !direct_reclaim requests, no?

I have to correct myself on GFP_TRANSHUGE_LIGHT because it happens to
include __GFP_NOMEMALLOC, and so won't actually get ALLOC_NON_BLOCK.

But what about random GFP_NOWAIT and & ~__GFP_DIRECT_RECLAIM sites?
Those explicitly don't get watermark exemptions already, and weren't
the intent of the rmqueue_buddy() exemption above.

Something like this?

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 12fac9084c48..c79cc7aa4dff 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3246,7 +3246,8 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
 			 * reserves as failing now is worse than failing a
 			 * high-order atomic allocation in the future.
 			 */
-			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
+			if (!page && ((alloc_flags & ALLOC_OOM) ||
+				      ((alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC)))
 				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
 
 			if (!page) {
diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index b9259deddb59..11714ddca254 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -60,6 +60,9 @@
 /* Flags that allow allocations below the min watermark. */
 #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
 
+/* Flag combination from GFP_ATOMIC */
+#define ALLOC_MASK_ATOMIC (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE)
+
 /*
  * Structure for holding the mostly immutable allocation parameters passed
  * between functions involved in allocations, including the alloc_pages*

> Unless I'm mistaken about that MIGRATE_HIGHATOMIC part, it seems all sashiko
> concerns can be dismissed and then indeed v4 is the better version.

It looks like a real issue to me, but one that already exists
independent of Salvatore's change.

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Vlastimil Babka (SUSE)]

On 9/16/26 17:58, Johannes Weiner wrote:
> On Wed, Sep 16, 2026 at 01:24:31PM +0200, Vlastimil Babka (SUSE) wrote:
>> On 9/11/26 17:59, Johannes Weiner wrote:
>> > The access to
>> > MIGRATE_HIGHATOMIC that it points out is in itself too generous. This
>> > seems like a real but separate bug. It will allow GFP_TRANSHUGE_LIGHT
>> > into the highatomic reserves as well, for example.
>> 
>> I don't follow this part. For ALLOC_HIGHATOMIC you need __GFP_HIGH in
>> alloc_flags_nonblocking(). So GFP_TRANSHUGE_LIGHT won't get the access, no?
> 
> rmqueue_buddy() has this:
> 
> 			/*
> 			 * If the allocation fails, allow OOM handling and
> 			 * order-0 (atomic) allocs access to HIGHATOMIC
> 			 * reserves as failing now is worse than failing a
> 			 * high-order atomic allocation in the future.
> 			 */
> 			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
> 				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);

Ah right, comes from Matt's 281dd25c1a01 ("mm/page_alloc: let GFP_ATOMIC
order-0 allocs access highatomic reserves")

> It says "atomic", but it's checking only ALLOC_NON_BLOCK, which is
> broader than GFP_ATOMIC. alloc_flags_nonblocking();
> 
> 	if (gfp_mask & __GFP_DIRECT_RECLAIM)
> 		return 0;
> 
> 	if (gfp_mask & __GFP_NOMEMALLOC)
> 		return 0;
> 
> 	alloc_flags |= ALLOC_NON_BLOCK;
> 
> 	if (order > 0 && (gfp_mask & __GFP_HIGH))
> 		alloc_flags |= ALLOC_HIGHATOMIC;
> 
> So this can apply to random !direct_reclaim requests, no?

Yes, and I agree it shouldn't.

> I have to correct myself on GFP_TRANSHUGE_LIGHT because it happens to
> include __GFP_NOMEMALLOC, and so won't actually get ALLOC_NON_BLOCK.

At least there's that.

> But what about random GFP_NOWAIT and & ~__GFP_DIRECT_RECLAIM sites?
> Those explicitly don't get watermark exemptions already, and weren't
> the intent of the rmqueue_buddy() exemption above.
> 
> Something like this?
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 12fac9084c48..c79cc7aa4dff 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -3246,7 +3246,8 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
>  			 * reserves as failing now is worse than failing a
>  			 * high-order atomic allocation in the future.
>  			 */
> -			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
> +			if (!page && ((alloc_flags & ALLOC_OOM) ||
> +				      ((alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC)))
>  				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
>  
>  			if (!page) {
> diff --git a/mm/page_alloc.h b/mm/page_alloc.h
> index b9259deddb59..11714ddca254 100644
> --- a/mm/page_alloc.h
> +++ b/mm/page_alloc.h
> @@ -60,6 +60,9 @@
>  /* Flags that allow allocations below the min watermark. */
>  #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
>  
> +/* Flag combination from GFP_ATOMIC */
> +#define ALLOC_MASK_ATOMIC (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE)
> +
>  /*
>   * Structure for holding the mostly immutable allocation parameters passed
>   * between functions involved in allocations, including the alloc_pages*

Should work.

> 
>> Unless I'm mistaken about that MIGRATE_HIGHATOMIC part, it seems all sashiko
>> concerns can be dismissed and then indeed v4 is the better version.
> 
> It looks like a real issue to me, but one that already exists
> independent of Salvatore's change.

But Salvatore's change (v4, not v5 IIUC) will make it worse because
__GFP_DIRECT_RECLAIM + __GFP_NORETRY costly order opportunistic attempts
with fallback will start eating the highatomic reserves too?

In that case the fix for this should probably be part of the same PR to
Linus and be also stable with Fixes: 281dd25c1a01

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Johannes Weiner]

On Fri, Sep 18, 2026 at 09:05:40AM +0200, Vlastimil Babka (SUSE) wrote:
> On 9/16/26 17:58, Johannes Weiner wrote:
> > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > index 12fac9084c48..c79cc7aa4dff 100644
> > --- a/mm/page_alloc.c
> > +++ b/mm/page_alloc.c
> > @@ -3246,7 +3246,8 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
> >  			 * reserves as failing now is worse than failing a
> >  			 * high-order atomic allocation in the future.
> >  			 */
> > -			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
> > +			if (!page && ((alloc_flags & ALLOC_OOM) ||
> > +				      ((alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC)))
> >  				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
> >  
> >  			if (!page) {
> > diff --git a/mm/page_alloc.h b/mm/page_alloc.h
> > index b9259deddb59..11714ddca254 100644
> > --- a/mm/page_alloc.h
> > +++ b/mm/page_alloc.h
> > @@ -60,6 +60,9 @@
> >  /* Flags that allow allocations below the min watermark. */
> >  #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
> >  
> > +/* Flag combination from GFP_ATOMIC */
> > +#define ALLOC_MASK_ATOMIC (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE)
> > +
> >  /*
> >   * Structure for holding the mostly immutable allocation parameters passed
> >   * between functions involved in allocations, including the alloc_pages*
> 
> Should work.
> 
> > 
> >> Unless I'm mistaken about that MIGRATE_HIGHATOMIC part, it seems all sashiko
> >> concerns can be dismissed and then indeed v4 is the better version.
> > 
> > It looks like a real issue to me, but one that already exists
> > independent of Salvatore's change.
> 
> But Salvatore's change (v4, not v5 IIUC) will make it worse because
> __GFP_DIRECT_RECLAIM + __GFP_NORETRY costly order opportunistic attempts
> with fallback will start eating the highatomic reserves too?
>
> In that case the fix for this should probably be part of the same PR to
> Linus and be also stable with Fixes: 281dd25c1a01

Yes, agreed on both.

Sorry for the delay - while poking around I found another oddity in
how the reserves are accessed.

I'll reply with two patches that I think we want to go in as fixes,
whether Salvatore's patch goes ahead or not.

[PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

[Johannes Weiner]

281dd25c1a01 ("mm/page_alloc: let GFP_ATOMIC order-0 allocs access
highatomic reserves") accidentally gave all non-blocking allocation
requests access to highatomic reserves, which includes GFP_NOWAIT and
other sites clearing __GFP_DIRECT_RECLAIM.

While this improved atomic request success rate, it's unintentionally
broad and can actually worsen highatomic requests by squandering the
reserves on requests that don't need it.

There are concurrent efforts to clear __GFP_DIRECT_RECLAIM altogether
for costly order __GFP_NORETRY requests, which would have then also
fall into this exemption and deplete reserves even faster.

What GFP_ATOMIC has over the others is __GFP_HIGH, which
alloc_flags_slowpath() translates to ALLOC_MIN_RESERVE.

Narrow the exemption to contexts that have both ALLOC_NON_BLOCK |
ALLOC_MIN_RESERVE.

Reported-by: Sashiko <sashiko-bot@kernel.org>
Fixes: 281dd25c1a01 ("mm/page_alloc: let GFP_ATOMIC order-0 allocs access highatomic reserves")
Cc: stable@vger.kernel.org
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
---
 mm/page_alloc.c | 4 +++-
 mm/page_alloc.h | 3 +++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 12fac9084c48..7b46d0ac3a56 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3246,7 +3246,9 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
 			 * reserves as failing now is worse than failing a
 			 * high-order atomic allocation in the future.
 			 */
-			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
+			if (!page &&
+			    ((alloc_flags & ALLOC_OOM) ||
+			     (alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC))
 				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
 
 			if (!page) {
diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index b9259deddb59..ad89f83d1dab 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -60,6 +60,9 @@
 /* Flags that allow allocations below the min watermark. */
 #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
 
+/* Flags that mean GFP_ATOMIC */
+#define ALLOC_MASK_ATOMIC (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE)
+
 /*
  * Structure for holding the mostly immutable allocation parameters passed
  * between functions involved in allocations, including the alloc_pages*
-- 
2.55.0

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

[Matthew Wilcox]

On Mon, Sep 21, 2026 at 10:38:18AM -0400, Johannes Weiner wrote:
> +++ b/mm/page_alloc.c
> @@ -3246,7 +3246,9 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
>  			 * reserves as failing now is worse than failing a
>  			 * high-order atomic allocation in the future.
>  			 */
> -			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
> +			if (!page &&
> +			    ((alloc_flags & ALLOC_OOM) ||
> +			     (alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC))
>  				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);

Would this be slightly neater?

static inline bool may_access_reserves(unsigned int alloc_flags)
{
	if (alloc_flags & ALLOC_OOM)
		return true;
	if (alloc_flags & (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)) ==
	    (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)
		return true;
	return false;
}

[...]

-			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
+			if (!page && may_access_reserves(alloc_flags))
 				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

[Johannes Weiner]

On Mon, Sep 21, 2026 at 03:54:32PM +0100, Matthew Wilcox wrote:
> On Mon, Sep 21, 2026 at 10:38:18AM -0400, Johannes Weiner wrote:
> > +++ b/mm/page_alloc.c
> > @@ -3246,7 +3246,9 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
> >  			 * reserves as failing now is worse than failing a
> >  			 * high-order atomic allocation in the future.
> >  			 */
> > -			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
> > +			if (!page &&
> > +			    ((alloc_flags & ALLOC_OOM) ||
> > +			     (alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC))
> >  				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
> 
> Would this be slightly neater?
> 
> static inline bool may_access_reserves(unsigned int alloc_flags)
> {
> 	if (alloc_flags & ALLOC_OOM)
> 		return true;
> 	if (alloc_flags & (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)) ==
> 	    (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)
> 		return true;
> 	return false;
> }

I tend to be hesitant with single-use abstractions, but no objection
if people think this is better.

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

[Vlastimil Babka (SUSE)]

On 9/21/26 5:58 PM, Johannes Weiner wrote:
> On Mon, Sep 21, 2026 at 03:54:32PM +0100, Matthew Wilcox wrote:
>> On Mon, Sep 21, 2026 at 10:38:18AM -0400, Johannes Weiner wrote:
>>> +++ b/mm/page_alloc.c
>>> @@ -3246,7 +3246,9 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
>>>  			 * reserves as failing now is worse than failing a
>>>  			 * high-order atomic allocation in the future.
>>>  			 */
>>> -			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
>>> +			if (!page &&
>>> +			    ((alloc_flags & ALLOC_OOM) ||
>>> +			     (alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC))
>>>  				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
>>
>> Would this be slightly neater?
>>
>> static inline bool may_access_reserves(unsigned int alloc_flags)
>> {
>> 	if (alloc_flags & ALLOC_OOM)
>> 		return true;
>> 	if (alloc_flags & (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)) ==
>> 	    (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)
>> 		return true;
>> 	return false;
>> }

I think it should be named e.g. may_access_highatomic_reserves() as
may_access_reserves() is rather generic and would seem to imply an
ALLOC_RESERVES match (see 2/2).

> 
> I tend to be hesitant with single-use abstractions, but no objection
> if people think this is better.

True but single-use ALLOC_MASK_ATOMIC is also not that great, and the
usage makes the code hard to decipher. And see my reply to 2/2.

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

[Johannes Weiner]

On Tue, Sep 22, 2026 at 01:52:41PM +0200, Vlastimil Babka (SUSE) wrote:
> 
> 
> On 9/21/26 5:58 PM, Johannes Weiner wrote:
> > On Mon, Sep 21, 2026 at 03:54:32PM +0100, Matthew Wilcox wrote:
> >> On Mon, Sep 21, 2026 at 10:38:18AM -0400, Johannes Weiner wrote:
> >>> +++ b/mm/page_alloc.c
> >>> @@ -3246,7 +3246,9 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
> >>>  			 * reserves as failing now is worse than failing a
> >>>  			 * high-order atomic allocation in the future.
> >>>  			 */
> >>> -			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
> >>> +			if (!page &&
> >>> +			    ((alloc_flags & ALLOC_OOM) ||
> >>> +			     (alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC))
> >>>  				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
> >>
> >> Would this be slightly neater?
> >>
> >> static inline bool may_access_reserves(unsigned int alloc_flags)
> >> {
> >> 	if (alloc_flags & ALLOC_OOM)
> >> 		return true;
> >> 	if (alloc_flags & (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)) ==
> >> 	    (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)
> >> 		return true;
> >> 	return false;
> >> }
> 
> I think it should be named e.g. may_access_highatomic_reserves() as
> may_access_reserves() is rather generic and would seem to imply an
> ALLOC_RESERVES match (see 2/2).

Note that it doesn't actually check ALLOC_HIGHATOMIC itself. It's just
the hail-mary AFTER trying the primary migratetype. So the name still
doesn't look right, and rmqueue_buddy() reads kind of awkardly:

if (alloc_flags & ALLOC_HIGHATOMIC)
	page = __rmqueue_smallest(..., MIGRATE_HIGHATOMIC);
if (!page) {
	page = __rmqueue(..., migratetype, ...);

	/* Allow OOM and order-0 atomic */
	if (!page && may_access_highatomic_reserve())
		page = __rmqueue_smallest(..., MIGRATE_HIGHATOMIC);
}

It would have to be may_access_highatomic_reserve_as_last_resort() or
something?

Hm, this is a mess. Looking closer, I think there is more breakage,
name aside.

You suggest in 2/2 to use the same helper in unusable_free. But I
think that's broken. My 2/2 does not look like the full fix either:

The fundamental problem is including the highatomics reserves in the
watermark check for any allocations that first prefer a different
migratetype - "regular memory". Any time we do this, we allow those
allocations to draw down regular memory to 0 based on the presence of
the highatomic reserves. And when reclaim, swap etc. come along there
is nothing left for them. ALLOC_NO_WATERMARKS e.g. permits ignoring
the wmarks but doesn't grant access to highatomic *freelists*.

So I think there are two choices:

(1) Let *everything* with some sort of reserve access fall back to
highatomic, or

(2) Only allow ALLOC_HIGHATOMIC to include highatomic reserves in
their watermarks check. With limited opportunistic fallback to the
highatomics *freelists*, like the order-0 atomics above.

My intuition is that (1) might weaken the highatomic reserves to the
point of uselessness, and we should probably go with (2):

watermarks:
if (alloc_flags & ALLOC_HIGHATOMIC)
	unusable_free += zone->nr_free_highatomic

freelists:
if (alloc_flags & ALLOC_HIGHATOMIC)
	page = __rmqueue_smallest(..., MIGRATE_HIGHATOMIC);
if (!page) {
	page = __rmqueue(..., migratetype, ...);
	/* Opportunistic fallback for order-0 atomics */
	if (!page && opportunistic_highatomic_fallback())
		page = __rmqueue_smallest(..., MIGRATE_HIGHATOMIC);
}

> > I tend to be hesitant with single-use abstractions, but no objection
> > if people think this is better.
> 
> True but single-use ALLOC_MASK_ATOMIC is also not that great, and the
> usage makes the code hard to decipher. And see my reply to 2/2.

There is one small upside, which is that it pairs with the
ALLOC_HIGHATOMIC check that precedes it. The comment says "order-0
atomics" get a hail mary, but there is no order check. That order-0
comes out of the sequence of events here: we first check highatomic,
which is order > 0 && atomic. If that, and the native type, fail, we
do the hail mary for atomic, which must be by definition order-0.

If you abstract that privilege into a generic "can access highatomic
reserves" without an order check, it tempts refactors that cause bugs
like the above.

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

[Vlastimil Babka (SUSE)]

On 9/22/26 15:56, Johannes Weiner wrote:
> On Tue, Sep 22, 2026 at 01:52:41PM +0200, Vlastimil Babka (SUSE) wrote:
>> 
>> 
>> On 9/21/26 5:58 PM, Johannes Weiner wrote:
>> > On Mon, Sep 21, 2026 at 03:54:32PM +0100, Matthew Wilcox wrote:
>> >> On Mon, Sep 21, 2026 at 10:38:18AM -0400, Johannes Weiner wrote:
>> >>> +++ b/mm/page_alloc.c
>> >>> @@ -3246,7 +3246,9 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
>> >>>  			 * reserves as failing now is worse than failing a
>> >>>  			 * high-order atomic allocation in the future.
>> >>>  			 */
>> >>> -			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
>> >>> +			if (!page &&
>> >>> +			    ((alloc_flags & ALLOC_OOM) ||
>> >>> +			     (alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC))
>> >>>  				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
>> >>
>> >> Would this be slightly neater?
>> >>
>> >> static inline bool may_access_reserves(unsigned int alloc_flags)
>> >> {
>> >> 	if (alloc_flags & ALLOC_OOM)
>> >> 		return true;
>> >> 	if (alloc_flags & (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)) ==
>> >> 	    (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)
>> >> 		return true;
>> >> 	return false;
>> >> }
>> 
>> I think it should be named e.g. may_access_highatomic_reserves() as
>> may_access_reserves() is rather generic and would seem to imply an
>> ALLOC_RESERVES match (see 2/2).
> 
> Note that it doesn't actually check ALLOC_HIGHATOMIC itself. It's just
> the hail-mary AFTER trying the primary migratetype. So the name still
> doesn't look right, and rmqueue_buddy() reads kind of awkardly:
> 
> if (alloc_flags & ALLOC_HIGHATOMIC)
> 	page = __rmqueue_smallest(..., MIGRATE_HIGHATOMIC);
> if (!page) {
> 	page = __rmqueue(..., migratetype, ...);
> 
> 	/* Allow OOM and order-0 atomic */
> 	if (!page && may_access_highatomic_reserve())
> 		page = __rmqueue_smallest(..., MIGRATE_HIGHATOMIC);
> }
> 
> It would have to be may_access_highatomic_reserve_as_last_resort() or
> something?
> 
> Hm, this is a mess. Looking closer, I think there is more breakage,
> name aside.
> 
> You suggest in 2/2 to use the same helper in unusable_free. But I
> think that's broken. My 2/2 does not look like the full fix either:
> 
> The fundamental problem is including the highatomics reserves in the
> watermark check for any allocations that first prefer a different
> migratetype - "regular memory". Any time we do this, we allow those
> allocations to draw down regular memory to 0 based on the presence of
> the highatomic reserves. And when reclaim, swap etc. come along there
> is nothing left for them. ALLOC_NO_WATERMARKS e.g. permits ignoring
> the wmarks but doesn't grant access to highatomic *freelists*.

Hmm yes, we'd have to be nuanced about which freelists we allow allocating
from depending on their particular watermarks. I.e. if we passed the
watermark checks only thanks to zone->nr_free_highatomic, we would have to
try those first. I assume it would be complicated and perhaps with perf
impact. But maybe if we managed to keep this complexity outside of fastpath
attempts?

The CMA handling had a similar issue and now we have that balancing
heuristic since 16867664936e ("mm,page_alloc,cma: conditionally prefer cma
pageblocks for movable allocations")

> So I think there are two choices:
> 
> (1) Let *everything* with some sort of reserve access fall back to
> highatomic, or
> 
> (2) Only allow ALLOC_HIGHATOMIC to include highatomic reserves in
> their watermarks check. With limited opportunistic fallback to the
> highatomics *freelists*, like the order-0 atomics above.
> 
> My intuition is that (1) might weaken the highatomic reserves to the
> point of uselessness, and we should probably go with (2):

I think the goal of commit 281dd25c1a01 ("mm/page_alloc: let GFP_ATOMIC
order-0 allocs access highatomic reserves") would be defeated with (2). The
order-0 atomic fallback to MIGRATE_HIGHATOMIC freelists would be there, but
the watermark check would never let such allocation through anymore, if
there wasn't a free page on the regular freelists? So it would become (some
races aside maybe) a dead code?

I guess your current patches are a compromise that can work, as patch 2/2
improves the current worse-than-(1) state to exclude the plain non-block
allocations.

> watermarks:
> if (alloc_flags & ALLOC_HIGHATOMIC)
> 	unusable_free += zone->nr_free_highatomic
> 
> freelists:
> if (alloc_flags & ALLOC_HIGHATOMIC)
> 	page = __rmqueue_smallest(..., MIGRATE_HIGHATOMIC);
> if (!page) {
> 	page = __rmqueue(..., migratetype, ...);
> 	/* Opportunistic fallback for order-0 atomics */
> 	if (!page && opportunistic_highatomic_fallback())
> 		page = __rmqueue_smallest(..., MIGRATE_HIGHATOMIC);
> }
> 
>> > I tend to be hesitant with single-use abstractions, but no objection
>> > if people think this is better.
>> 
>> True but single-use ALLOC_MASK_ATOMIC is also not that great, and the
>> usage makes the code hard to decipher. And see my reply to 2/2.
> 
> There is one small upside, which is that it pairs with the
> ALLOC_HIGHATOMIC check that precedes it. The comment says "order-0
> atomics" get a hail mary, but there is no order check. That order-0
> comes out of the sequence of events here: we first check highatomic,
> which is order > 0 && atomic. If that, and the native type, fail, we
> do the hail mary for atomic, which must be by definition order-0.
> 
> If you abstract that privilege into a generic "can access highatomic
> reserves" without an order check, it tempts refactors that cause bugs
> like the above.

Ack.