[PATCH v11] mm/page_alloc: boost watermarks on atomic allocation failure

[PATCH v11] mm/page_alloc: boost watermarks on atomic allocation failure

[Qiliang Yuan]

Atomic allocations (GFP_ATOMIC) are prone to failure under heavy memory
pressure as they cannot enter direct reclaim.

Handle these failures by introducing a watermark boost mechanism for
atomic requests. Refactor boost_watermark() using an internal helper to
support both fragmentation and atomic paths. Apply zone-proportional
boosts (~0.1% of managed pages) for atomic allocations, while
decoupling it from watermark_boost_factor.

Implement boost_zones_for_atomic() to iterate through and boost all
eligible zones in the zonelist, respecting nodemasks. Use a per-zone
1-second debounce timer via last_boost_jiffies to prevent excessive
boosting. Check and update the debounce timestamp under zone->lock,
alongside the watermark modification, so concurrent callers on other
CPUs cannot all observe a stale timestamp and pile onto the same zone
within the same window. Verify with lockdep. Integrate the mechanism
into the page allocation slowpath specifically for order-0 GFP_ATOMIC
requests.

This approach reuses existing infrastructure and ensures emergency
reserves even if fragmentation boosting is disabled.

The mechanism is bounded by construction. Each zone accepts at most
one boost per second (the debounce timer), and zone->watermark_boost
is clamped to zone->_watermark[WMARK_HIGH] / 10 regardless of how
long the atomic pressure lasts or how many zones are boosted in a
single slowpath call. Worst case, sustained atomic pressure across
all zones in a zonelist adds at most one extra kswapd wakeup per zone
per second, and the boost on any single zone cannot exceed 10% of
that zone's high watermark. It cannot run away or starve unrelated
allocations beyond that ceiling.

This failure signature keeps recurring in production: a host running
a downstream 4.19 kernel logged 144 order-0 GFP_ATOMIC failures over a
4h15m window, all through the same NIC driver receive softirq path,
across several unrelated network-facing services on the box. This
confirms the underlying problem is real and ongoing. It does not by
itself measure this patch's effect, since the fix has not been
deployed on that fleet yet.

Allocation failure logs:
[38535644.718700] node 0: slabs: 1031, objs: 43328, free: 0
[38535644.725059] node 1: slabs: 339, objs: 17616, free: 317
[38535645.428345] SLUB: Unable to allocate memory on node -1, gfp=0x480020(GFP_ATOMIC)
[38535645.436888] cache: skbuff_head_cache, object size: 232, buffer size: 256, default order: 2, min order: 0
[38535645.447664] node 0: slabs: 940, objs: 40864, free: 144
[38535645.454026] node 1: slabs: 322, objs: 19168, free: 383
[38535645.556122] SLUB: Unable to allocate memory on node -1, gfp=0x480020(GFP_ATOMIC)
[38535645.564576] cache: skbuff_head_cache, object size: 232, buffer size: 256, default order: 2, min order: 0
[38535649.655523] warn_alloc: 59 callbacks suppressed
[38535649.655527] swapper/100: page allocation failure: order:0, mode:0x480020(GFP_ATOMIC), nodemask=(null)
[38535649.671692] swapper/100 cpuset=/ mems_allowed=0-1

Signed-off-by: Qiliang Yuan <realwujing@gmail.com>
Signed-off-by: Jing Wu <realwujing@gmail.com>
Acked-by: SeongJae Park <sj@kernel.org>
---
V10 -> V11:
- Fix a debounce race: last_boost_jiffies was checked and updated
  outside zone->lock, so concurrent CPUs handling a multi-queue NIC
  RX storm could all pass the once-per-second check for the same
  zone before either updated the timestamp. Move the check-and-set
  under zone->lock alongside the watermark update.
- Add production evidence of the failure signature recurring (see
  commit message) in response to Vlastimil's repeated request (v2,
  v6, v10) for real-world data on whether this problem is real and
  whether the fix helps.
- Drop Vlastimil's Acked-by: he took it back in review of v10,
  noting the v10 changes were not trivial. Not resending solely to
  drop the tag, per his request; folding it into this update instead.
- Cc netdev and add Matthew Wilcox, per Andrew Morton's (v1, v5) and
  Wilcox's (v1) requests to involve the networking maintainers, which
  never actually happened in v1-v10.

V9 -> V10:
- Refactor watermark boosting into mechanism (__boost_watermark) and
  policy logic.
- Decouple Atomic boost from watermark_boost_factor to ensure
  emergency reserves.
- Simplify Atomic boost calculation to ~0.1% of managed pages with a
  10% high-WM cap.
- Boost all eligible zones in the zonelist while respecting
  nodemasks.

V8 -> V9:
- Use mult_frac() for boost calculation. (SJ)
- Add !can_direct_reclaim check. (Vlastimil)
- Code cleanup: naming, scope, and line limits. (SJ)
- Update tags: Add Vlastimil's Acked-by.

V7 -> V8:
- Use spin_lock_irqsave() to prevent inconsistent lock state
  (softirq-on vs in-softirq) as reported by LKP.

V6 -> V7:
- Use local variable for boost_amount to improve code readability.
- Add zone->lock protection in boost_zones_for_atomic().
- Add lockdep assertion in boost_watermark() to prevent locking
  mistakes.
- Remove redundant boost call at fail label due to 1-second
  debounce.

V5 -> V6:
- Replace magic number ">> 10" with ATOMIC_BOOST_SCALE_SHIFT define.
- Add documentation explaining 0.1% zone size boost rationale.

V4 -> V5:
- Simplify to use native boost_watermark() instead of custom logic.

V3 -> V4:
- Add watermark_scale_boost and gradual decay via balance_pgdat.

V2 -> V3:
- Move debounce timer to per-zone to avoid cross-node interference.
- Optimize candidate zone selection to reduce global reclaim
  pressure.

V1 -> V2:
- Add basic debounce logic and scale boosting strength based on zone
  size.

v10: https://lore.kernel.org/r/20260214-wujing-mm-page_alloc-v8-v10-1-bdfea431fd97@gmail.com
v9: https://lore.kernel.org/r/20260213-wujing-mm-page_alloc-v8-v9-1-cd99f3a6cb70@gmail.com
v8: https://lore.kernel.org/r/20260212-wujing-mm-page_alloc-v8-v8-1-daba38990cd3@gmail.com
v7: https://lore.kernel.org/all/20260123064231.250767-1-realwujing@gmail.com/
v5: https://lore.kernel.org/all/20260121065740.35616-1-realwujing@gmail.com/
v4: https://lore.kernel.org/all/tencent_D23BFCB69EA088C55AFAF89F926036743E0A@qq.com/
v3: https://lore.kernel.org/all/tencent_44B556221480D8371FBC534ACCF3CE2C8707@qq.com/
v2: https://lore.kernel.org/all/tencent_6FE67BA7BE8376AB038A71ACAD4FF8A90006@qq.com/
v1: https://lore.kernel.org/all/tencent_9DB6637676D639B4B7AEA09CC6A6F9E49D0A@qq.com/
---
 include/linux/mmzone.h |  1 +
 mm/page_alloc.c        | 80 ++++++++++++++++++++++++++++++++++++++++++++------
 2 files changed, 72 insertions(+), 9 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 75ef7c9f9307f..8e37e4e6765b5 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -882,6 +882,7 @@ struct zone {
 	/* zone watermarks, access with *_wmark_pages(zone) macros */
 	unsigned long _watermark[NR_WMARK];
 	unsigned long watermark_boost;
+	unsigned long last_boost_jiffies;
 
 	unsigned long nr_reserved_highatomic;
 	unsigned long nr_free_highatomic;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index c380f063e8b7b..4517e2c1310ee 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2158,12 +2158,15 @@ bool pageblock_unisolate_and_move_free_pages(struct zone *zone, struct page *pag
 
 #endif /* CONFIG_MEMORY_ISOLATION */
 
-static inline bool boost_watermark(struct zone *zone)
+/*
+ * Helper for boosting watermarks. Called with zone->lock held.
+ * Use max_boost to limit the boost to a percentage of the high watermark.
+ */
+static inline bool __boost_watermark(struct zone *zone, unsigned long amount,
+				     unsigned long max_boost)
 {
-	unsigned long max_boost;
+	lockdep_assert_held(&zone->lock);
 
-	if (!watermark_boost_factor)
-		return false;
 	/*
 	 * Don't bother in zones that are unlikely to produce results.
 	 * On small machines, including kdump capture kernels running
@@ -2173,9 +2176,6 @@ static inline bool boost_watermark(struct zone *zone)
 	if ((pageblock_nr_pages * 4) > zone_managed_pages(zone))
 		return false;
 
-	max_boost = mult_frac(zone->_watermark[WMARK_HIGH],
-			watermark_boost_factor, 10000);
-
 	/*
 	 * high watermark may be uninitialised if fragmentation occurs
 	 * very early in boot so do not boost. We do not fall
@@ -2189,12 +2189,70 @@ static inline bool boost_watermark(struct zone *zone)
 
 	max_boost = max(pageblock_nr_pages, max_boost);
 
-	zone->watermark_boost = min(zone->watermark_boost + pageblock_nr_pages,
-		max_boost);
+	zone->watermark_boost = min(zone->watermark_boost + amount,
+				    max_boost);
 
 	return true;
 }
 
+/*
+ * Boost watermarks to increase reclaim pressure when fragmentation occurs
+ * and we fall back to other migratetypes.
+ */
+static inline bool boost_watermark(struct zone *zone)
+{
+	if (!watermark_boost_factor)
+		return false;
+
+	return __boost_watermark(zone, pageblock_nr_pages,
+			mult_frac(zone->_watermark[WMARK_HIGH],
+				  watermark_boost_factor, 10000));
+}
+
+/*
+ * Boost watermarks by ~0.1% of zone size on atomic allocation pressure.
+ * This provides zone-proportional safety buffers: ~1MB per 1GB of zone
+ * size. Max boost ceiling is fixed at ~10% of high watermark.
+ *
+ * This emergency reserve is independent of watermark_boost_factor.
+ */
+static inline bool boost_watermark_atomic(struct zone *zone)
+{
+	return __boost_watermark(zone,
+			max(pageblock_nr_pages, zone_managed_pages(zone) / 1000),
+			zone->_watermark[WMARK_HIGH] / 10);
+}
+
+static void boost_zones_for_atomic(struct alloc_context *ac, gfp_t gfp_mask)
+{
+	struct zoneref *z;
+	struct zone *zone;
+	unsigned long now = jiffies;
+
+	for_each_zone_zonelist_nodemask(zone, z, ac->zonelist,
+					ac->highest_zoneidx, ac->nodemask) {
+		unsigned long flags;
+		bool should_wake = false;
+
+		/*
+		 * Check and update the per-zone debounce timestamp under
+		 * zone->lock so concurrent callers on other CPUs (e.g. a
+		 * multi-queue NIC spreading GFP_ATOMIC allocations across
+		 * several softirqs) cannot all observe a stale timestamp
+		 * and pile onto the same zone within the same window.
+		 */
+		spin_lock_irqsave(&zone->lock, flags);
+		if (time_after(now, zone->last_boost_jiffies + HZ)) {
+			zone->last_boost_jiffies = now;
+			should_wake = boost_watermark_atomic(zone);
+		}
+		spin_unlock_irqrestore(&zone->lock, flags);
+
+		if (should_wake)
+			wakeup_kswapd(zone, gfp_mask, 0, ac->highest_zoneidx);
+	}
+}
+
 /*
  * When we are falling back to another migratetype during allocation, should we
  * try to claim an entire block to satisfy further allocations, instead of
@@ -4742,6 +4800,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	if (page)
 		goto got_pg;
 
+	/* Boost watermarks for atomic requests entering slowpath */
+	if (((gfp_mask & GFP_ATOMIC) == GFP_ATOMIC) && order == 0 && !can_direct_reclaim)
+		boost_zones_for_atomic(ac, gfp_mask);
+
 	/*
 	 * For costly allocations, try direct compaction first, as it's likely
 	 * that we have enough base pages and don't need to reclaim. For non-

---
base-commit: b54345928fa1dbde534e32ecaa138678fd5d2135
change-id: 20260720-feat-mm-page_alloc-v11-5f415da4aeb0

Best regards,
-- 
Jing Wu <realwujing@gmail.com>

Re: [PATCH v11] mm/page_alloc: boost watermarks on atomic allocation failure

[Andrew Morton]

On Mon, 20 Jul 2026 16:15:48 +0800 Qiliang Yuan <realwujing@gmail.com> wrote:

> Atomic allocations (GFP_ATOMIC) are prone to failure under heavy memory
> pressure as they cannot enter direct reclaim.
> 
> Handle these failures by introducing a watermark boost mechanism for
> atomic requests. Refactor boost_watermark() using an internal helper to
> support both fragmentation and atomic paths. Apply zone-proportional
> boosts (~0.1% of managed pages) for atomic allocations, while
> decoupling it from watermark_boost_factor.

Thanks for persisting with this.

You didn't retain Vlastimil's Reviewed-by: from v8?

This is very much a networking thing - they must have considered
similar things.  But my not-very-energetic attempts to get input from
networking people have thus far failed.

> This failure signature keeps recurring in production: a host running
> a downstream 4.19 kernel logged 144 order-0 GFP_ATOMIC failures over a
> 4h15m window, all through the same NIC driver receive softirq path,
> across several unrelated network-facing services on the box. This
> confirms the underlying problem is real and ongoing. It does not by
> itself measure this patch's effect, since the fix has not been
> deployed on that fleet yet.

We'll of course be very interested in these results.  Do you know
if/when they'll be available?

Anyway, let me get this into mm.git and linux-next so we can at least
parallelize wider testing with ongoing review.

Re: [PATCH v11] mm/page_alloc: boost watermarks on atomic allocation failure

[Vlastimil Babka (SUSE)]

On 7/21/26 01:37, Andrew Morton wrote:
> On Mon, 20 Jul 2026 16:15:48 +0800 Qiliang Yuan <realwujing@gmail.com> wrote:
> 
>> Atomic allocations (GFP_ATOMIC) are prone to failure under heavy memory
>> pressure as they cannot enter direct reclaim.
>> 
>> Handle these failures by introducing a watermark boost mechanism for
>> atomic requests. Refactor boost_watermark() using an internal helper to
>> support both fragmentation and atomic paths. Apply zone-proportional
>> boosts (~0.1% of managed pages) for atomic allocations, while
>> decoupling it from watermark_boost_factor.
> 
> Thanks for persisting with this.
> 
> You didn't retain Vlastimil's Reviewed-by: from v8?

It was Acked-by: and I asked for it to be removed due [1] to significant
changes in v10, which was acknowleded [2] (thanks):

> This is very much a networking thing - they must have considered
> similar things.  But my not-very-energetic attempts to get input from
> networking people have thus far failed.

Yes it would have been useful to have their input.

>> This failure signature keeps recurring in production: a host running
>> a downstream 4.19 kernel logged 144 order-0 GFP_ATOMIC failures over a

I think first only in [2] and now here we learn it's motivated by failures
observed on a downstream 4.19 based kernel.

>> 4h15m window, all through the same NIC driver receive softirq path,
>> across several unrelated network-facing services on the box. This
>> confirms the underlying problem is real and ongoing.

... on a 4.19 (released in 2018) based kernel. There were many changes to
this area since then, some for highatomic allocations even very recently.
So it's necessary to demonstrate the problem exists today as well.

And it shouldn't exist in the form of "logged failures" anyway, thanks to
commits such as c89cca307b20 ("net: skbuff: sprinkle more __GFP_NOWARN on
ingress allocs") that use GFP_ATOMIC with __GFP_NOWARN. So it's not about
avoiding warnings anymore, but preventing fallbacks to non-irq contexts
(that those allocations AFAIK have) and probably thus rather demonstrating
how that improves performance and justifies the patch and risks that come
with it (these heurstics are unfortunately fraught with them).

> It does not by
>> itself measure this patch's effect, since the fix has not been
>> deployed on that fleet yet.

That makes the argument for this patch even worse, but also due to the
above, it wouldn't really be relevant to do that with that 4.19 based kernel
so I can advice not investing time into that.

So what we'd need is to demonstrate that current mainline has a problem and
how it's fixed. A synthetic reproducer suggested in [2] can however be
misleading in the form of apparently confirming that yes, increasing
watermarks by 10% can succeed 10% longer bursts of atomic allocations. But
that alone is not enough to justify this change.

> We'll of course be very interested in these results.  Do you know
> if/when they'll be available?
> 
> Anyway, let me get this into mm.git and linux-next so we can at least
> parallelize wider testing with ongoing review.

linux-next means mm-unstable? I don't think this should be headed for the
next merge window given the above.

[1] https://lore.kernel.org/all/e011c6a8-cda5-42ce-9d42-b23d1c81b26b@suse.cz/#t
[2] https://lore.kernel.org/all/20260720033804.3862547-1-realwujing@gmail.com/