From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f194.google.com (mail-qk1-f194.google.com [209.85.222.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0799522FE0E for ; Fri, 13 Feb 2026 19:36:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.194 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771011388; cv=none; b=oKDXPPnZiOAA84jDCRGeeYOOZPiAKKvBS64mnaksS/GRNsgrguYE6JDw5T1FK0CatDi1NHXLqESvfdShQEcKTATG9FFglxK2D4UMZ0YMhstCmL0z9gUbN02D/V1iCpKGvdwErKTtlLQOVq7u7kJycIpnIH4JLARIwtk2GfEmnUU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771011388; c=relaxed/simple; bh=7qUOn/6B71wtAinq1AYQY4GwDXqyEx7Ryj8qVUutPTY=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=N396QUYFBOrSfw9ycFNMwcyaNf84x+t4Sm/DfdNbEHmW8gN0e6y/CI0bBNOB0xF64M6aaRBYz2c5+BbfpldbtdeR/tD7Ny7yWAOnoI7sAhMaov+fqPrKlcysmxcIpBZT8l0dMWNOutF7/gBBboOal7GEzzkiPGY74FucLMFcZJE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=g8OcoRqc; arc=none smtp.client-ip=209.85.222.194 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="g8OcoRqc" Received: by mail-qk1-f194.google.com with SMTP id af79cd13be357-8cb3dfb3461so138351085a.3 for ; Fri, 13 Feb 2026 11:36:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1771011385; x=1771616185; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=llkjJOAUSGkOBuN7S3aJDNNVSVBODvl4G/+bNRBiTNs=; b=g8OcoRqcmhJp1gR9C5D9ICEISGUYqJDBr0U/T1Rmh9ZROKsUf/1vpD+KjN6PMblwyO QX8fr0bTETXSQIjcanM0/nkQvoqAVK5L1fytwD/PyScZ+Tqjw1Qx+Q8GBeDk2W3tjJZD fLkgIBhVTzMMcS6Eck9MSzXqfylBqxXrGZwIRwB052vc6aow3v/GS16nodB/9D8yC/1y iJjGYwnSm6rTHOC3Yt5vFayAkTiUHvUac9baMaYK696BJ9pebGcHR0XIxGjPHdRy+gVU q/IyHiPJNCNQR0cmxGn/zIofeinzIICBcdI1eVn+RtxSnQE0VpWF+WaXcR/EaLvv3Gh5 60oQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771011385; x=1771616185; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=llkjJOAUSGkOBuN7S3aJDNNVSVBODvl4G/+bNRBiTNs=; b=kL2n2V8MbvuQmENAEMYFmyu8qM3LNdtgKneqomRgbv0jNELieMBvJg947HNufg4i9o XuDPsHl33R2GrUV7JJ+siaqWWI+DkADJQtvrmi/CvRUczCNLzUpK78inJ96gvxOBz4rn FRYnUDw3+lrK2xhLzJpkOs/OujuQT1+87ZU/lv/bNiRpo0oc1vH/2uQyEk71WIwr/VxR L7X/CRmyb++/JCuYbSZYzNT+yBDdvI8qFMfKLwGzkketIazobnjv1tjwTQTTMRJM+Jg6 MNaBSLeWus0MWINZ5GqpH/3GDBBpERJbP9tnQn0SzvmQG5ZzHzeiiDnF3vya8lgsmUev Xapg== X-Forwarded-Encrypted: i=1; AJvYcCWjeh9vtc2RFAjr9t6CSa6fSRqglOcFEsoto53N35H36olndRSBej7poipcoEQquk6od8QBM3vOfanqWao=@vger.kernel.org X-Gm-Message-State: AOJu0Yx1b/1qewS5Fbg+NQq+3bddpHNqm89EZ5BDERp6mVukVkGa6sRD dwB8UtBG+Aw57Vj0clnaTqPzfBtgVMcnm7zeNBjZmhReXKqrS/2VsAFcYAfZa+EfA/0= X-Gm-Gg: AZuq6aIhmIwrgf8jH5nLWHlsrJgvF7jpDM3cbCrKBxBfYG4JUxLPbgvt/LfaxshXnVl /nlNRr+PlVizkZYBhU4se6Vge1dj4JiPJXuW4oAV63ZZIe1L3Q2CYznXrdjAoLS3u5KVSK7A4dH i2ZacUJSENqByCiFmBQa03Y5xnK1vwdL39DaKKz5P2U4ChOkWp9zOaS1X9ouE3xRlhg71qq4xQ5 GNJqjsWWjAn7ysr6rhYzLAs5DtewhtNlMenjx17H7Ln9+pwr/BrRd0LhmmP3uvMvM7TgC1qrBYK QqxBgyDcC+o1ojfmTlCTCIY/L8TG29NcGQnNGkeHJJDCaXBpZAhm4j6zXyBFXsNePNMhwOUZycl yp8ckN5Q+gOXA0lLr4fpGPWPvaojIgbqlpQEhkhEq3yCoydAcr/CL646Wi5jn+1ktm6j3dJo6An AnkZGemJ785cCXSIFafJ5VuA== X-Received: by 2002:a05:620a:4505:b0:8cb:2b04:85fa with SMTP id af79cd13be357-8cb4081f6c3mr456742285a.13.1771011384548; Fri, 13 Feb 2026 11:36:24 -0800 (PST) Received: from localhost ([2603:7000:c00:3a00:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8cb2b0f38afsm675777385a.22.2026.02.13.11.36.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Feb 2026 11:36:23 -0800 (PST) Date: Fri, 13 Feb 2026 14:36:19 -0500 From: Johannes Weiner To: Qiliang Yuan Cc: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Axel Rasmussen , Yuanchu Xie , Wei Xu , Brendan Jackman , Zi Yan , Lance Yang , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v9] mm/page_alloc: boost watermarks on atomic allocation failure Message-ID: References: <20260213-wujing-mm-page_alloc-v8-v9-1-cd99f3a6cb70@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260213-wujing-mm-page_alloc-v8-v9-1-cd99f3a6cb70@gmail.com> On Fri, Feb 13, 2026 at 11:17:59AM +0800, Qiliang Yuan wrote: > Atomic allocations (GFP_ATOMIC) are prone to failure under heavy memory > pressure as they cannot enter direct reclaim. This patch introduces a > watermark boost mechanism to mitigate this issue. > > When a GFP_ATOMIC request enters the slowpath, the preferred zone's > watermark_boost is increased under zone->lock protection. This triggers > kswapd to proactively reclaim memory, creating a safety buffer for > future atomic allocations. A 1-second debounce timer prevents excessive > boosts during traffic bursts. > > This approach reuses existing watermark_boost infrastructure with > minimal overhead and proper locking to ensure thread safety. > > Allocation failure logs: > [38535644.718700] node 0: slabs: 1031, objs: 43328, free: 0 > [38535644.725059] node 1: slabs: 339, objs: 17616, free: 317 > [38535645.428345] SLUB: Unable to allocate memory on node -1, gfp=0x480020(GFP_ATOMIC) > [38535645.436888] cache: skbuff_head_cache, object size: 232, buffer size: 256, default order: 2, min order: 0 > [38535645.447664] node 0: slabs: 940, objs: 40864, free: 144 > [38535645.454026] node 1: slabs: 322, objs: 19168, free: 383 > [38535645.556122] SLUB: Unable to allocate memory on node -1, gfp=0x480020(GFP_ATOMIC) > [38535645.564576] cache: skbuff_head_cache, object size: 232, buffer size: 256, default order: 2, min order: 0 > [38535649.655523] warn_alloc: 59 callbacks suppressed > [38535649.655527] swapper/100: page allocation failure: order:0, mode:0x480020(GFP_ATOMIC), nodemask=(null) > [38535649.671692] swapper/100 cpuset=/ mems_allowed=0-1 > > Acked-by: Vlastimil Babka > Signed-off-by: Qiliang Yuan > --- > v9: > - Use mult_frac() for boost calculation. (SJ) > - Add !can_direct_reclaim check. (Vlastimil) > - Code cleanup: naming, scope, and line limits. (SJ) > - Update tags: Add Vlastimil's Acked-by. > > v8: > - Use spin_lock_irqsave() to prevent inconsistent lock state. > > v7: > - Use local variable for boost_amount. > - Add zone->lock protection. > - Add lockdep assertion. > > v6: > - Use ATOMIC_BOOST_SCALE_SHIFT define. > - Add documentation for 0.1% rationale. > > v5: > - Use native boost_watermark(). > > v4: > - Add watermark_scale_boost and gradual decay. > > v3: > - Per-zone debounce timer. > > v2: > - Debounce logic and zone-proportional boosting. > > v1: > - Initial version. > --- > Link to v8: https://lore.kernel.org/r/20260212-wujing-mm-page_alloc-v8-v8-1-daba38990cd3@gmail.com > --- > include/linux/mmzone.h | 1 + > mm/page_alloc.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++-- > 2 files changed, 48 insertions(+), 2 deletions(-) > > diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h > index 75ef7c9f9307..8e37e4e6765b 100644 > --- a/include/linux/mmzone.h > +++ b/include/linux/mmzone.h > @@ -882,6 +882,7 @@ struct zone { > /* zone watermarks, access with *_wmark_pages(zone) macros */ > unsigned long _watermark[NR_WMARK]; > unsigned long watermark_boost; > + unsigned long last_boost_jiffies; > > unsigned long nr_reserved_highatomic; > unsigned long nr_free_highatomic; > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index c380f063e8b7..8af88584a8bd 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -218,6 +218,13 @@ unsigned int pageblock_order __read_mostly; > static void __free_pages_ok(struct page *page, unsigned int order, > fpi_t fpi_flags); > > +/* > + * Boost watermarks by ~0.1% of zone size on atomic allocation pressure. > + * This provides zone-proportional safety buffers: ~1MB per 1GB of zone size. > + * Larger zones under GFP_ATOMIC pressure need proportionally larger reserves. > + */ > +#define ATOMIC_BOOST_FACTOR 1 > + > /* > * results with 256, 32 in the lowmem_reserve sysctl: > * 1G machine -> (16M dma, 800M-16M normal, 1G-800M high) > @@ -2161,6 +2168,9 @@ bool pageblock_unisolate_and_move_free_pages(struct zone *zone, struct page *pag > static inline bool boost_watermark(struct zone *zone) > { > unsigned long max_boost; > + unsigned long boost_amount; > + > + lockdep_assert_held(&zone->lock); > > if (!watermark_boost_factor) > return false; watermark_boost_factor is for fragmentation management. It's valid to have this set to 0 and still want boosting for atomic. > @@ -2189,12 +2199,43 @@ static inline bool boost_watermark(struct zone *zone) > > max_boost = max(pageblock_nr_pages, max_boost); > > - zone->watermark_boost = min(zone->watermark_boost + pageblock_nr_pages, > - max_boost); > + boost_amount = max(pageblock_nr_pages, > + mult_frac(zone_managed_pages(zone), ATOMIC_BOOST_FACTOR, 1000)); > + zone->watermark_boost = min(zone->watermark_boost + boost_amount, > + max_boost); Likewise, you don't want to add the atomic boost every time there is a fragmentation event. You need to separate these paths. The mult_frac() with constants seems a bit funny to me. Just do zone_managed_pages(zone) / 1000, drop the define, and move the comment and move the comment here. > +static void boost_zone_for_atomic(struct alloc_context *ac, gfp_t gfp_mask) > +{ > + struct zoneref *z; > + struct zone *zone; > + unsigned long now = jiffies; > + > + for_each_zone_zonelist(zone, z, ac->zonelist, ac->highest_zoneidx) { for_each_zone_zonelist_nodemask() with ac->nodemask? > + /* Rate-limit boosts to once per second per zone */ > + if (time_after(now, zone->last_boost_jiffies + HZ)) { > + unsigned long flags; > + bool should_wake; > + > + zone->last_boost_jiffies = now; > + > + /* Modify watermark under lock, wake kswapd outside */ > + spin_lock_irqsave(&zone->lock, flags); > + should_wake = boost_watermark(zone); > + spin_unlock_irqrestore(&zone->lock, flags); > + > + if (should_wake) > + wakeup_kswapd(zone, gfp_mask, 0, > + ac->highest_zoneidx); > + > + /* Boost only the preferred zone */ > + break; > + } > + } This is a bit strange to me. By the time you boost, all eligible zones have been tried, and ALL their reserves were found to be inadequate for the incoming atomic requests. They all *should* be boosted. By doing them one by one, you risk additional failures even though you already KNOW at this point that these other zones are problematic too. So IMO, by the time you reach here, they should all be boosted. > @@ -4742,6 +4783,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, > if (page) > goto got_pg; > > + /* Boost watermarks for atomic requests entering slowpath */ > + if ((gfp_mask & GFP_ATOMIC) && order == 0 && !can_direct_reclaim) This is a bit weird. GFP_ATOMIC is a mask, so this check will trigger on anything that has __GFP_KSWAPD_RECLAIM set (which is most things), so in turn you then have to filter out direct reclaim again (which the real GFP_ATOMIC implies). if (gfp_has_flags(gfp_mask, GFP_ATOMIC)) > + boost_zone_for_atomic(ac, gfp_mask); > +