From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
To: Andrew Morton <akpm@linux-foundation.org>,
Vlastimil Babka <vbabka@kernel.org>,
Johannes Weiner <hannes@cmpxchg.org>,
David Hildenbrand <david@kernel.org>,
Harry Yoo <harry@kernel.org>
Cc: "Kiryl Shutsemau (Meta)" <kas@kernel.org>,
Suren Baghdasaryan <surenb@google.com>,
Michal Hocko <mhocko@suse.com>,
Brendan Jackman <brendan.jackman@linux.dev>,
Zi Yan <ziy@nvidia.com>, Lorenzo Stoakes <ljs@kernel.org>,
"Liam R. Howlett" <liam@infradead.org>,
Mike Rapoport <rppt@kernel.org>, Qi Zheng <qi.zheng@linux.dev>,
Shakeel Butt <shakeel.butt@linux.dev>,
Kairui Song <kasong@tencent.com>, Barry Song <baohua@kernel.org>,
Axel Rasmussen <axelrasmussen@google.com>,
Yuanchu Xie <yuanchu@google.com>, Wei Xu <weixugc@google.com>,
Baoquan He <baoquan.he@linux.dev>,
Baolin Wang <baolin.wang@linux.alibaba.com>,
Dev Jain <dev.jain@arm.com>, Usama Arif <usama.arif@linux.dev>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
kernel-team@meta.com
Subject: [PATCH v3 0/3] mm: make watermark checks honour ALLOC_NOFRAGMENT
Date: Mon, 16 Nov 2026 12:50:58 +0000 [thread overview]
Message-ID: <20261116125103.2894162-1-kas@kernel.org> (raw)
With defrag_mode, 64K mTHP faults under memory pressure stall in
compaction 18 times as often as without it, and get about a third of the
folios. Dev Jain reported it [1].
The high-order watermark check counts free pages of every migratetype.
With defrag_mode, every slowpath allocation is ALLOC_NOFRAGMENT, and a
request below pageblock_order cannot take free pages from another
migratetype's pageblock. Under pressure, that is where the large free
pages end up. Two callers act on the wrong answer:
- direct compaction's up-front check returns COMPACT_SUCCESS without
compacting, so compaction is never deferred;
- direct reclaim for a costly order stops early, having freed little.
Patches 1-2 give the check the request's migratetype and make it skip
the free lists the request cannot take. Patch 3 tells direct reclaim
that the allocation is ALLOC_NOFRAGMENT.
Dev's benchmark in a 7G VM with only 64K mTHP enabled, medians of 15
runs:
defrag_mode=0 defrag_mode=1 defrag_mode=1
+ series
compact_stall 1690 30669 7366
64K faults allocated 20750 7391 20129
64K faults falling back to 4K 725 13832 1339
defrag_mode=1 still compacts more than defrag_mode=0. That is by design,
as it keeps 64K faults out of unmovable and reclaimable pageblocks [2].
With the series, defrag_mode=0 is unchanged at 20698 64K faults
allocated.
Based on page_alloc/for-7.5/misc. Two trivial conflicts with other trees:
- v7.4-rc2 changed the trace_reclaim_retry_zone() call next to patch
2's change in should_reclaim_retry();
- mm-new moved struct scan_control to mm/vmscan.h, where patch 3's
new field goes.
v3:
- Patch 1: declare the watermark functions in mm/page_alloc.h; rewrite
the comment above __zone_watermark_ok() and the changelog
(Vlastimil, Harry)
- Patch 2: plainer comments in should_reclaim_retry() and
__compact_finished()
- Patch 3: pass alloc_flags to try_to_free_pages() (Vlastimil)
- Rebased onto page_alloc/for-7.5/misc
v2:
- Patch 2 changelog: describe what the fast path change does to pages
on the per-cpu lists (Andrew); drop the should_reclaim_retry()
paragraph
- Patch 3 changelog: say that the benchmark baseline already has the
high-order check honouring ALLOC_NOFRAGMENT (Andrew)
- Rebased onto v7.4-rc1
v2: https://lore.kernel.org/all/20261102093939.3340540-1-kas@kernel.org/
v1: https://lore.kernel.org/all/20261028132638.2560361-1-kirill@shutemov.name/
[1] https://lore.kernel.org/all/4aa14b41-9b89-494c-8825-55fffd5a541c@arm.com/
[2] https://lore.kernel.org/all/auCDnqJc7fsBpUW2@thinkstation/
Kiryl Shutsemau (Meta) (3):
mm: page_alloc: add a watermark check that knows the migratetype
mm: page_alloc: make the high-order watermark check honour
ALLOC_NOFRAGMENT
mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT
include/linux/mmzone.h | 3 --
mm/compaction.c | 16 +++++++----
mm/internal.h | 3 +-
mm/page_alloc.c | 62 ++++++++++++++++++++++++++++++++----------
mm/page_alloc.h | 8 ++++++
mm/vmscan.c | 24 ++++++++++++----
6 files changed, 88 insertions(+), 28 deletions(-)
Range-diff against v2:
1: ed3ef9f9fd13 ! 1: e3c2bded052a mm: page_alloc: add a watermark check that knows the migratetype
@@ Metadata
## Commit message ##
mm: page_alloc: add a watermark check that knows the migratetype
- The high-order check in __zone_watermark_ok() accepts a free page of any
- migratetype. Under ALLOC_NOFRAGMENT, a request below pageblock_order can
- only take a page of its own migratetype or claim a whole free block, so
- the check can pass when the allocation cannot succeed.
+ For a high-order request, __zone_watermark_ok() accepts a free page of
+ any migratetype. An ALLOC_NOFRAGMENT request below pageblock_order can
+ take a page of its own migratetype or claim a whole free pageblock, but
+ not steal from another migratetype's pageblock. So the check can pass
+ for an allocation that is going to fail.
- Telling those apart needs the request's migratetype, but not every check
- wants them told apart. A check whether an allocation can be served now
- has to count only the pages it can take.
+ To tell these apart, the check needs the migratetype of the request.
+ Add __zone_watermark_ok_mt(), which takes it, and use it where the
+ caller asks whether this allocation can succeed:
- A check whether there is enough free memory counts every migratetype:
- kswapd and kcompactd balancing a zone, order-0 thresholds, the
- slowpath's forecast whether reclaim and compaction can still make
- progress. Reclaim and compaction work across migratetypes to free whole
- pageblocks.
+ - zone_watermark_fast();
+ - the recheck after node reclaim in get_page_from_freelist();
+ - compaction_suit_allocation_order().
- Move the body of __zone_watermark_ok() into __zone_watermark_ok_mt(),
- which takes the migratetype as well, and make __zone_watermark_ok() a
- wrapper around it for the second kind of check. These use the new
- function:
+ Other callers keep __zone_watermark_ok(), which counts every
+ migratetype. kswapd and kcompactd balance the zone as a whole.
+ should_reclaim_retry() asks whether the allocation could succeed if all
+ reclaimable pages were freed, and compaction can then free a whole
+ pageblock for it to claim.
- - the allocator fast path, zone_watermark_fast();
- - its recheck after node reclaim in get_page_from_freelist();
- - compaction's up-front check, compaction_suit_allocation_order().
+ kcompactd passes MIGRATE_MOVABLE to compaction_suit_allocation_order().
+ It never sets ALLOC_NOFRAGMENT, so the value is not used.
- Declare both in mm/internal.h: nothing outside mm uses them.
+ Nothing outside mm/ uses __zone_watermark_ok(). Move its declaration
+ to mm/page_alloc.h, next to the new one.
- Nothing reads the migratetype yet: no functional change.
+ Nothing reads the migratetype yet, so there is no functional change.
+ Preparation for making the high-order check honour ALLOC_NOFRAGMENT.
+ Reviewed-by: Harry Yoo (Meta) <harry@kernel.org>
Tested-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
@@ mm/compaction.c: static void kcompactd_do_work(pg_data_t *pgdat)
continue;
- ## mm/internal.h ##
-@@ mm/internal.h: extern int min_free_kbytes;
- extern int defrag_mode;
-
- void setup_per_zone_wmarks(void);
-+bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
-+ int highest_zoneidx, unsigned int alloc_flags,
-+ long free_pages);
-+bool __zone_watermark_ok_mt(struct zone *z, unsigned int order,
-+ unsigned long mark, int highest_zoneidx,
-+ unsigned int alloc_flags, long free_pages,
-+ int migratetype);
- void calculate_min_free_kbytes(void);
- int __meminit init_per_zone_wmark_min(void);
-
-
## mm/page_alloc.c ##
@@ mm/page_alloc.c: static inline long __zone_watermark_unusable_free(struct zone *z,
* one free page of a suitable size. Checking now avoids taking the zone lock
@@ mm/page_alloc.c: bool __zone_watermark_ok(struct zone *z, unsigned int order, un
}
+/*
-+ * For checks that are not about serving one allocation now: kswapd and
-+ * kcompactd balancing a zone, order-0 thresholds, forecasts whether
-+ * reclaim and compaction can still make progress. They count free pages
-+ * of every migratetype: reclaim and compaction work across migratetypes
-+ * to free whole pageblocks.
++ * Return true if free base pages in the zone are above 'mark' regardless
++ * of migratetype. Use it to check whether the zone as a whole has enough
++ * free memory. Use __zone_watermark_ok_mt() instead to check whether a
++ * particular allocation can be served.
+ */
+bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
+ int highest_zoneidx, unsigned int alloc_flags,
@@ mm/page_alloc.c: get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int
continue;
}
+
+ ## mm/page_alloc.h ##
+@@ mm/page_alloc.h: extern bool free_pages_prepare(struct page *page, unsigned int order);
+
+ extern int user_min_free_kbytes;
+
++bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
++ int highest_zoneidx, unsigned int alloc_flags,
++ long free_pages);
++bool __zone_watermark_ok_mt(struct zone *z, unsigned int order,
++ unsigned long mark, int highest_zoneidx,
++ unsigned int alloc_flags, long free_pages,
++ int migratetype);
++
+ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int nid,
+ nodemask_t *nodemask, unsigned int alloc_flags);
+ #define __alloc_frozen_pages(...) \
2: 3347b0a24b0e ! 2: c991756a139a mm: page_alloc: make the high-order watermark check honour ALLOC_NOFRAGMENT
@@ Commit message
With defrag_mode, every slowpath allocation carries ALLOC_NOFRAGMENT. A
request below pageblock_order may take a page of its own migratetype or
claim a whole free block, but not steal from another migratetype's block.
- The high-order part of the watermark check still accepts a free page of
- any migratetype.
+ For a high-order request, the watermark check still accepts a free page
+ of any migratetype.
Under memory pressure, movable allocations take free movable pages as
soon as reclaim frees them. Free space in unmovable and reclaimable
@@ mm/compaction.c: static enum compact_result __compact_finished(struct compact_co
!free_area_empty(area, MIGRATE_CMA))
return COMPACT_SUCCESS;
#endif
-+ /* Under ALLOC_NOFRAGMENT it may only claim a whole block */
++ /* An ALLOC_NOFRAGMENT request may only claim a whole block */
+ if ((cc->alloc_flags & ALLOC_NOFRAGMENT) &&
+ order < pageblock_order)
+ continue;
@@ mm/page_alloc.c: should_reclaim_retry(gfp_t gfp_mask, unsigned order,
/*
* Would the allocation succeed if we reclaimed all
- * reclaimable pages?
-+ * reclaimable pages? Count free pages of every migratetype:
-+ * reclaim and compaction work across migratetypes to free a
-+ * whole pageblock, which the allocation can then claim.
++ * reclaimable pages? Count free pages of all migratetypes.
++ * Compaction can then free a whole pageblock for the
++ * allocation to claim.
*/
wmark = __zone_watermark_ok(zone, order, min_wmark,
- ac->highest_zoneidx, alloc_flags, available);
3: 0ecbcc5f5fb0 ! 3: fd8bb9835186 mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT
@@ Commit message
64K faults allocated 20750 11538 19759
64K faults falling back to 4K 725 9945 1664
- Pass ALLOC_NOFRAGMENT from the allocation down to try_to_free_pages().
- Both checks then ask __zone_watermark_ok_mt() for a free page the
- allocation can take, and an order-4 direct reclaim frees 40 pages on
- average.
+ Pass the allocation's alloc_flags down to try_to_free_pages(), and
+ record ALLOC_NOFRAGMENT in scan_control. Both checks then ask
+ __zone_watermark_ok_mt() for a free page the allocation can take, and an
+ order-4 direct reclaim frees 40 pages on average.
Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Reported-by: Dev Jain <dev.jain@arm.com>
@@ mm/internal.h: void folio_mark_lazyfree(struct folio *folio);
unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
- gfp_t gfp_mask, const nodemask_t *mask);
+ gfp_t gfp_mask, const nodemask_t *mask,
-+ bool nofragment);
++ unsigned int alloc_flags);
unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru,
int zone_idx);
@@ mm/page_alloc.c: __perform_reclaim(gfp_t gfp_mask, unsigned int order,
progress = try_to_free_pages(ac->zonelist, order, gfp_mask,
- ac->nodemask);
-+ ac->nodemask,
-+ alloc_flags & ALLOC_NOFRAGMENT);
++ ac->nodemask, alloc_flags);
memalloc_noreclaim_restore(noreclaim_flag);
fs_reclaim_release(gfp_mask);
@@ mm/vmscan.c: struct scan_control {
/* One of the zones is ready for compaction */
unsigned int compaction_ready:1;
-+ /* The allocation cannot fall back to other migratetypes' blocks */
++ /* The allocation may not steal from other migratetypes' blocks */
+ unsigned int nofragment:1;
+
/* There is easily reclaimable cold cache in the current node */
@@ mm/vmscan.c: static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist
unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
- gfp_t gfp_mask, const nodemask_t *nodemask)
+ gfp_t gfp_mask, const nodemask_t *nodemask,
-+ bool nofragment)
++ unsigned int alloc_flags)
{
unsigned long nr_reclaimed;
struct scan_control sc = {
@@ mm/vmscan.c: unsigned long try_to_free_pages(struct zonelist *zonelist, int orde
.may_writepage = 1,
.may_unmap = 1,
.may_swap = 1,
-+ .nofragment = nofragment,
++ .nofragment = !!(alloc_flags & ALLOC_NOFRAGMENT),
};
/*
base-commit: 3a99594e62fce8c646e8aac01d38af16d6833a40
--
2.54.0
next reply other threads:[~2026-11-16 12:51 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 12:50 Kiryl Shutsemau (Meta) [this message]
2026-11-16 12:50 ` [PATCH v3 1/3] mm: page_alloc: add a watermark check that knows the migratetype Kiryl Shutsemau (Meta)
2026-11-16 12:51 ` [PATCH v3 2/3] mm: page_alloc: make the high-order watermark check honour ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
2026-11-16 12:51 ` [PATCH v3 3/3] mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261116125103.2894162-1-kas@kernel.org \
--to=kas@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=axelrasmussen@google.com \
--cc=baohua@kernel.org \
--cc=baolin.wang@linux.alibaba.com \
--cc=baoquan.he@linux.dev \
--cc=brendan.jackman@linux.dev \
--cc=david@kernel.org \
--cc=dev.jain@arm.com \
--cc=hannes@cmpxchg.org \
--cc=harry@kernel.org \
--cc=kasong@tencent.com \
--cc=kernel-team@meta.com \
--cc=liam@infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ljs@kernel.org \
--cc=mhocko@suse.com \
--cc=qi.zheng@linux.dev \
--cc=rppt@kernel.org \
--cc=shakeel.butt@linux.dev \
--cc=surenb@google.com \
--cc=usama.arif@linux.dev \
--cc=vbabka@kernel.org \
--cc=weixugc@google.com \
--cc=yuanchu@google.com \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox