From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
To: Andrew Morton <akpm@linux-foundation.org>,
Vlastimil Babka <vbabka@kernel.org>,
Johannes Weiner <hannes@cmpxchg.org>,
David Hildenbrand <david@kernel.org>,
Harry Yoo <harry@kernel.org>
Cc: "Kiryl Shutsemau (Meta)" <kas@kernel.org>,
Suren Baghdasaryan <surenb@google.com>,
Michal Hocko <mhocko@suse.com>,
Brendan Jackman <brendan.jackman@linux.dev>,
Zi Yan <ziy@nvidia.com>, Lorenzo Stoakes <ljs@kernel.org>,
"Liam R. Howlett" <liam@infradead.org>,
Mike Rapoport <rppt@kernel.org>, Qi Zheng <qi.zheng@linux.dev>,
Shakeel Butt <shakeel.butt@linux.dev>,
Kairui Song <kasong@tencent.com>, Barry Song <baohua@kernel.org>,
Axel Rasmussen <axelrasmussen@google.com>,
Yuanchu Xie <yuanchu@google.com>, Wei Xu <weixugc@google.com>,
Baoquan He <baoquan.he@linux.dev>,
Baolin Wang <baolin.wang@linux.alibaba.com>,
Dev Jain <dev.jain@arm.com>, Usama Arif <usama.arif@linux.dev>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
kernel-team@meta.com
Subject: [PATCH v3 3/3] mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT
Date: Mon, 16 Nov 2026 12:51:01 +0000 [thread overview]
Message-ID: <20261116125103.2894162-4-kas@kernel.org> (raw)
In-Reply-To: <20261116125103.2894162-1-kas@kernel.org>
Direct reclaim for a costly order stops as soon as the allocation could
succeed. compaction_ready() skips a zone and should_continue_reclaim()
stops reclaiming when zone_watermark_ok() passes, and zone_watermark_ok()
accepts a free page of any migratetype.
With defrag_mode, the allocation behind the reclaim is ALLOC_NOFRAGMENT
and cannot take the free pages left in unmovable and reclaimable
pageblocks. Reclaim stops on them anyway, having freed little.
In Dev Jain's 64K mTHP benchmark, with the high-order watermark check
already honouring ALLOC_NOFRAGMENT, an order-4 direct reclaim frees 16
pages on average with defrag_mode and 45 without. 64K faults get a bit
over half as many folios as without defrag_mode (medians of 15 runs):
defrag_mode=0 defrag_mode=1 defrag_mode=1
+ this patch
64K faults allocated 20750 11538 19759
64K faults falling back to 4K 725 9945 1664
Pass the allocation's alloc_flags down to try_to_free_pages(), and
record ALLOC_NOFRAGMENT in scan_control. Both checks then ask
__zone_watermark_ok_mt() for a free page the allocation can take, and an
order-4 direct reclaim frees 40 pages on average.
Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Reported-by: Dev Jain <dev.jain@arm.com>
Closes: https://lore.kernel.org/all/4aa14b41-9b89-494c-8825-55fffd5a541c@arm.com/
Tested-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
try_to_free_pages() takes the whole alloc_flags, but reclaim uses only
ALLOC_NOFRAGMENT. Using all of them would change when costly reclaim
stops for every allocation, not only with defrag_mode. ALLOC_CMA would
count free CMA pages for movable requests, and the reserve flags would
lower the min.
Storing the whole word in scan_control and masking it would work the
same, but kswapd and memcg reclaim have no alloc_flags to put there.
So the patch stores ALLOC_NOFRAGMENT as a bit in scan_control and
reconstructs it in allocation_can_succeed().
mm/internal.h | 3 ++-
mm/page_alloc.c | 7 ++++---
mm/vmscan.c | 24 +++++++++++++++++++-----
3 files changed, 25 insertions(+), 9 deletions(-)
diff --git a/mm/internal.h b/mm/internal.h
index 566cbd2dce48..3ef9b7036afd 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -65,7 +65,8 @@ void folio_mark_lazyfree(struct folio *folio);
/* mm/vmscan.c */
unsigned long zone_reclaimable_pages(struct zone *zone);
unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
- gfp_t gfp_mask, const nodemask_t *mask);
+ gfp_t gfp_mask, const nodemask_t *mask,
+ unsigned int alloc_flags);
unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru,
int zone_idx);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 6e75c03b303b..93362e0f699b 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4459,7 +4459,7 @@ static unsigned int check_retry_zonelist(unsigned int seq)
/* Perform direct synchronous page reclaim */
static unsigned long
-__perform_reclaim(gfp_t gfp_mask, unsigned int order,
+__perform_reclaim(gfp_t gfp_mask, unsigned int order, unsigned int alloc_flags,
const struct alloc_context *ac)
{
unsigned int noreclaim_flag;
@@ -4473,7 +4473,7 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order,
noreclaim_flag = memalloc_noreclaim_save();
progress = try_to_free_pages(ac->zonelist, order, gfp_mask,
- ac->nodemask);
+ ac->nodemask, alloc_flags);
memalloc_noreclaim_restore(noreclaim_flag);
fs_reclaim_release(gfp_mask);
@@ -4495,7 +4495,8 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
int reclaim_order = ac->reclaim_compact_order;
psi_memstall_enter(&pflags);
- *did_some_progress = __perform_reclaim(gfp_mask, reclaim_order, ac);
+ *did_some_progress = __perform_reclaim(gfp_mask, reclaim_order,
+ alloc_flags, ac);
if (unlikely(!(*did_some_progress)))
goto out;
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 9959ca158ce8..3fbcdcc87a00 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -142,6 +142,9 @@ struct scan_control {
/* One of the zones is ready for compaction */
unsigned int compaction_ready:1;
+ /* The allocation may not steal from other migratetypes' blocks */
+ unsigned int nofragment:1;
+
/* There is easily reclaimable cold cache in the current node */
unsigned int cache_trim_mode:1;
@@ -6316,6 +6319,17 @@ static bool in_reclaim_compaction(struct scan_control *sc)
return false;
}
+static bool allocation_can_succeed(struct zone *zone, struct scan_control *sc,
+ unsigned long mark)
+{
+ unsigned int alloc_flags = sc->nofragment ? ALLOC_NOFRAGMENT : 0;
+ long free_pages = zone_page_state(zone, NR_FREE_PAGES);
+ int migratetype = gfp_migratetype(sc->gfp_mask);
+
+ return __zone_watermark_ok_mt(zone, sc->order, mark, sc->reclaim_idx,
+ alloc_flags, free_pages, migratetype);
+}
+
/*
* Reclaim/compaction is used for high-order allocation requests. It reclaims
* order-0 pages before compacting the zone. should_continue_reclaim() returns
@@ -6354,8 +6368,7 @@ static inline bool should_continue_reclaim(struct pglist_data *pgdat,
unsigned long watermark = min_wmark_pages(zone);
/* Allocation can already succeed, nothing to do */
- if (zone_watermark_ok(zone, sc->order, watermark,
- sc->reclaim_idx, 0))
+ if (allocation_can_succeed(zone, sc, watermark))
return false;
if (compaction_suitable(zone, sc->order, watermark,
@@ -6580,8 +6593,7 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc)
return false;
/* Allocation can already succeed, nothing to do */
- if (zone_watermark_ok(zone, sc->order, min_wmark_pages(zone),
- sc->reclaim_idx, 0))
+ if (allocation_can_succeed(zone, sc, min_wmark_pages(zone)))
return true;
/*
@@ -6965,7 +6977,8 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,
}
unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
- gfp_t gfp_mask, const nodemask_t *nodemask)
+ gfp_t gfp_mask, const nodemask_t *nodemask,
+ unsigned int alloc_flags)
{
unsigned long nr_reclaimed;
struct scan_control sc = {
@@ -6978,6 +6991,7 @@ unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
.may_writepage = 1,
.may_unmap = 1,
.may_swap = 1,
+ .nofragment = !!(alloc_flags & ALLOC_NOFRAGMENT),
};
/*
--
2.54.0
prev parent reply other threads:[~2026-11-16 12:51 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 12:50 [PATCH v3 0/3] mm: make watermark checks honour ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
2026-11-16 12:50 ` [PATCH v3 1/3] mm: page_alloc: add a watermark check that knows the migratetype Kiryl Shutsemau (Meta)
2026-11-16 12:51 ` [PATCH v3 2/3] mm: page_alloc: make the high-order watermark check honour ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
2026-11-17 12:51 ` sashiko-bot
2026-11-18 8:10 ` Dev Jain
2026-11-16 12:51 ` Kiryl Shutsemau (Meta) [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261116125103.2894162-4-kas@kernel.org \
--to=kas@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=axelrasmussen@google.com \
--cc=baohua@kernel.org \
--cc=baolin.wang@linux.alibaba.com \
--cc=baoquan.he@linux.dev \
--cc=brendan.jackman@linux.dev \
--cc=david@kernel.org \
--cc=dev.jain@arm.com \
--cc=hannes@cmpxchg.org \
--cc=harry@kernel.org \
--cc=kasong@tencent.com \
--cc=kernel-team@meta.com \
--cc=liam@infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ljs@kernel.org \
--cc=mhocko@suse.com \
--cc=qi.zheng@linux.dev \
--cc=rppt@kernel.org \
--cc=shakeel.butt@linux.dev \
--cc=surenb@google.com \
--cc=usama.arif@linux.dev \
--cc=vbabka@kernel.org \
--cc=weixugc@google.com \
--cc=yuanchu@google.com \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.