From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
To: Andrew Morton <akpm@linux-foundation.org>,
Vlastimil Babka <vbabka@kernel.org>,
Johannes Weiner <hannes@cmpxchg.org>,
David Hildenbrand <david@kernel.org>,
Harry Yoo <harry@kernel.org>
Cc: "Kiryl Shutsemau (Meta)" <kas@kernel.org>,
Suren Baghdasaryan <surenb@google.com>,
Michal Hocko <mhocko@suse.com>,
Brendan Jackman <brendan.jackman@linux.dev>,
Zi Yan <ziy@nvidia.com>, Lorenzo Stoakes <ljs@kernel.org>,
"Liam R. Howlett" <liam@infradead.org>,
Mike Rapoport <rppt@kernel.org>, Qi Zheng <qi.zheng@linux.dev>,
Shakeel Butt <shakeel.butt@linux.dev>,
Kairui Song <kasong@tencent.com>, Barry Song <baohua@kernel.org>,
Axel Rasmussen <axelrasmussen@google.com>,
Yuanchu Xie <yuanchu@google.com>, Wei Xu <weixugc@google.com>,
Baoquan He <baoquan.he@linux.dev>,
Baolin Wang <baolin.wang@linux.alibaba.com>,
Dev Jain <dev.jain@arm.com>, Usama Arif <usama.arif@linux.dev>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
kernel-team@meta.com
Subject: [PATCH v3 3/3] mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT
Date: Mon, 16 Nov 2026 12:51:01 +0000 [thread overview]
Message-ID: <20261116125103.2894162-4-kas@kernel.org> (raw)
In-Reply-To: <20261116125103.2894162-1-kas@kernel.org>
Direct reclaim for a costly order stops as soon as the allocation could
succeed. compaction_ready() skips a zone and should_continue_reclaim()
stops reclaiming when zone_watermark_ok() passes, and zone_watermark_ok()
accepts a free page of any migratetype.
With defrag_mode, the allocation behind the reclaim is ALLOC_NOFRAGMENT
and cannot take the free pages left in unmovable and reclaimable
pageblocks. Reclaim stops on them anyway, having freed little.
In Dev Jain's 64K mTHP benchmark, with the high-order watermark check
already honouring ALLOC_NOFRAGMENT, an order-4 direct reclaim frees 16
pages on average with defrag_mode and 45 without. 64K faults get a bit
over half as many folios as without defrag_mode (medians of 15 runs):
defrag_mode=0 defrag_mode=1 defrag_mode=1
+ this patch
64K faults allocated 20750 11538 19759
64K faults falling back to 4K 725 9945 1664
Pass the allocation's alloc_flags down to try_to_free_pages(), and
record ALLOC_NOFRAGMENT in scan_control. Both checks then ask
__zone_watermark_ok_mt() for a free page the allocation can take, and an
order-4 direct reclaim frees 40 pages on average.
Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Reported-by: Dev Jain <dev.jain@arm.com>
Closes: https://lore.kernel.org/all/4aa14b41-9b89-494c-8825-55fffd5a541c@arm.com/
Tested-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
try_to_free_pages() takes the whole alloc_flags, but reclaim uses only
ALLOC_NOFRAGMENT. Using all of them would change when costly reclaim
stops for every allocation, not only with defrag_mode. ALLOC_CMA would
count free CMA pages for movable requests, and the reserve flags would
lower the min.
Storing the whole word in scan_control and masking it would work the
same, but kswapd and memcg reclaim have no alloc_flags to put there.
So the patch stores ALLOC_NOFRAGMENT as a bit in scan_control and
reconstructs it in allocation_can_succeed().
mm/internal.h | 3 ++-
mm/page_alloc.c | 7 ++++---
mm/vmscan.c | 24 +++++++++++++++++++-----
3 files changed, 25 insertions(+), 9 deletions(-)
diff --git a/mm/internal.h b/mm/internal.h
index 566cbd2dce48..3ef9b7036afd 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -65,7 +65,8 @@ void folio_mark_lazyfree(struct folio *folio);
/* mm/vmscan.c */
unsigned long zone_reclaimable_pages(struct zone *zone);
unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
- gfp_t gfp_mask, const nodemask_t *mask);
+ gfp_t gfp_mask, const nodemask_t *mask,
+ unsigned int alloc_flags);
unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru,
int zone_idx);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 6e75c03b303b..93362e0f699b 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4459,7 +4459,7 @@ static unsigned int check_retry_zonelist(unsigned int seq)
/* Perform direct synchronous page reclaim */
static unsigned long
-__perform_reclaim(gfp_t gfp_mask, unsigned int order,
+__perform_reclaim(gfp_t gfp_mask, unsigned int order, unsigned int alloc_flags,
const struct alloc_context *ac)
{
unsigned int noreclaim_flag;
@@ -4473,7 +4473,7 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order,
noreclaim_flag = memalloc_noreclaim_save();
progress = try_to_free_pages(ac->zonelist, order, gfp_mask,
- ac->nodemask);
+ ac->nodemask, alloc_flags);
memalloc_noreclaim_restore(noreclaim_flag);
fs_reclaim_release(gfp_mask);
@@ -4495,7 +4495,8 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
int reclaim_order = ac->reclaim_compact_order;
psi_memstall_enter(&pflags);
- *did_some_progress = __perform_reclaim(gfp_mask, reclaim_order, ac);
+ *did_some_progress = __perform_reclaim(gfp_mask, reclaim_order,
+ alloc_flags, ac);
if (unlikely(!(*did_some_progress)))
goto out;
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 9959ca158ce8..3fbcdcc87a00 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -142,6 +142,9 @@ struct scan_control {
/* One of the zones is ready for compaction */
unsigned int compaction_ready:1;
+ /* The allocation may not steal from other migratetypes' blocks */
+ unsigned int nofragment:1;
+
/* There is easily reclaimable cold cache in the current node */
unsigned int cache_trim_mode:1;
@@ -6316,6 +6319,17 @@ static bool in_reclaim_compaction(struct scan_control *sc)
return false;
}
+static bool allocation_can_succeed(struct zone *zone, struct scan_control *sc,
+ unsigned long mark)
+{
+ unsigned int alloc_flags = sc->nofragment ? ALLOC_NOFRAGMENT : 0;
+ long free_pages = zone_page_state(zone, NR_FREE_PAGES);
+ int migratetype = gfp_migratetype(sc->gfp_mask);
+
+ return __zone_watermark_ok_mt(zone, sc->order, mark, sc->reclaim_idx,
+ alloc_flags, free_pages, migratetype);
+}
+
/*
* Reclaim/compaction is used for high-order allocation requests. It reclaims
* order-0 pages before compacting the zone. should_continue_reclaim() returns
@@ -6354,8 +6368,7 @@ static inline bool should_continue_reclaim(struct pglist_data *pgdat,
unsigned long watermark = min_wmark_pages(zone);
/* Allocation can already succeed, nothing to do */
- if (zone_watermark_ok(zone, sc->order, watermark,
- sc->reclaim_idx, 0))
+ if (allocation_can_succeed(zone, sc, watermark))
return false;
if (compaction_suitable(zone, sc->order, watermark,
@@ -6580,8 +6593,7 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc)
return false;
/* Allocation can already succeed, nothing to do */
- if (zone_watermark_ok(zone, sc->order, min_wmark_pages(zone),
- sc->reclaim_idx, 0))
+ if (allocation_can_succeed(zone, sc, min_wmark_pages(zone)))
return true;
/*
@@ -6965,7 +6977,8 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,
}
unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
- gfp_t gfp_mask, const nodemask_t *nodemask)
+ gfp_t gfp_mask, const nodemask_t *nodemask,
+ unsigned int alloc_flags)
{
unsigned long nr_reclaimed;
struct scan_control sc = {
@@ -6978,6 +6991,7 @@ unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
.may_writepage = 1,
.may_unmap = 1,
.may_swap = 1,
+ .nofragment = !!(alloc_flags & ALLOC_NOFRAGMENT),
};
/*
--
2.54.0
prev parent reply other threads:[~2026-11-16 12:51 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 12:50 [PATCH v3 0/3] mm: make watermark checks honour ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
2026-11-16 12:50 ` [PATCH v3 1/3] mm: page_alloc: add a watermark check that knows the migratetype Kiryl Shutsemau (Meta)
2026-11-16 12:51 ` [PATCH v3 2/3] mm: page_alloc: make the high-order watermark check honour ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
2026-11-16 12:51 ` Kiryl Shutsemau (Meta) [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261116125103.2894162-4-kas@kernel.org \
--to=kas@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=axelrasmussen@google.com \
--cc=baohua@kernel.org \
--cc=baolin.wang@linux.alibaba.com \
--cc=baoquan.he@linux.dev \
--cc=brendan.jackman@linux.dev \
--cc=david@kernel.org \
--cc=dev.jain@arm.com \
--cc=hannes@cmpxchg.org \
--cc=harry@kernel.org \
--cc=kasong@tencent.com \
--cc=kernel-team@meta.com \
--cc=liam@infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ljs@kernel.org \
--cc=mhocko@suse.com \
--cc=qi.zheng@linux.dev \
--cc=rppt@kernel.org \
--cc=shakeel.butt@linux.dev \
--cc=surenb@google.com \
--cc=usama.arif@linux.dev \
--cc=vbabka@kernel.org \
--cc=weixugc@google.com \
--cc=yuanchu@google.com \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox