[PATCH v2 4/4] mm: page_alloc: fix non-movable reclaim storm in defrag_mode

[PATCH v2 4/4] mm: page_alloc: fix non-movable reclaim storm in defrag_mode

[Johannes Weiner]

As we deployed defrag_mode into Meta production, pressure spikes and
excessive swapping were observed on some workloads. Tracing confirmed
that this is unmovable/reclaimable requests spinning in the allocator
and direct reclaim, causing excessive amounts of swap.

The initial plan for defrag_mode was to rely on kswapd/kcompactd to
produce blocks, and if those are overwhelmed under high pressure, let
the allocator fall back (__rmqueue_steal()) after its retry loops.
However, that retrying results in more reclaim on some of these
workloads than we'd hoped, sometimes excessively so, spurred on by the
!costly order conditions in should_reclaim_retry().

The storms are dependent on the request type. Reclaim will inevitably
make room in existing movable blocks, since that's where the LRU pages
live. So if movable requests retry on reclaim, they make progress.

When non-movable requests spin in reclaim that isn't productive. They
cannot use the individually freed pages, and the process is unlikely
to accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar.
They spin and overreclaim excessively, which tanks performance and
triggers userspace guards like swap exhaustion or pressure based OOM.

To fix this, send non-movable requests, regardless of order, into
pageblock reclaim/compaction. This way, they help move things along to
meet the ALLOC_NOFRAGMENT bar. After this patch, the reclaim storms
and excess OOM rates are no longer observed in production.

The longer-term plan is still to have all requests, including the
movable ones, help make blocks to spread the cost of defragmenting
more evenly and fairly; combined with proper watermarking to reduce
allocation latencies in the common case. However, doing this naively
unearths scaling and concurrency limitations in compaction that need
to be addressed first. Promoting just non-movables for now is the
minimally viable bug fix for the above issue.

Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Cc: <stable@vger.kernel.org>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
---
 mm/internal.h   |  6 ++++++
 mm/page_alloc.c | 31 ++++++++++++++++++++++++++-----
 2 files changed, 32 insertions(+), 5 deletions(-)

diff --git a/mm/internal.h b/mm/internal.h
index 6d3402001b93..5acba3470659 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -1061,6 +1061,12 @@ struct compact_control {
 struct capture_control {
 	struct zone *zone;
 	int migratetype;
+	/*
+	 * Allocation request order. May differ from the compaction
+	 * order: defrag_mode promotes sub-block allocations to
+	 * pageblock-order compaction; capture still matches at the
+	 * original allocation order so prep_new_page() is consistent.
+	 */
 	int order;
 	struct page *page;
 };
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 4b9686fedb71..f92055827ae9 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4152,8 +4152,24 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
 		.order = order,
 		.page = NULL,
 	};
+	int compact_order = order;
 
-	if (!order)
+	/*
+	 * If fallbacks are not permitted (defrag_mode), we either
+	 * need to reclaim space in a block of matching type, or clear
+	 * out an entire block to allow __rmqueue_claim() to convert.
+	 *
+	 * Reclaim by itself is primarily freeing space in movable
+	 * blocks, since that's where the LRU pages live. So this
+	 * works for movable requests, but not for others.
+	 *
+	 * For those, promote the order to help make blocks, instead
+	 * of spinning in reclaim alone unproductively.
+	 */
+	if ((alloc_flags & ALLOC_NOFRAGMENT) && ac->migratetype != MIGRATE_MOVABLE)
+		compact_order = max(order, pageblock_order);
+
+	if (!compact_order)
 		return NULL;
 
 	psi_memstall_enter(&pflags);
@@ -4169,8 +4185,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
 	barrier();
 	WRITE_ONCE(current->capture_control, &capc);
 
-	*compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac,
-							       prio, &capc);
+	*compact_result = try_to_compact_pages(gfp_mask, compact_order,
+					       alloc_flags, ac, prio, &capc);
 
 	/*
 	 * Make sure we hide capture control first before we read the captured
@@ -4215,7 +4231,7 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
 		struct zone *zone = page_zone(page);
 
 		zone->compact_blockskip_flush = false;
-		compaction_defer_reset(zone, order, true);
+		compaction_defer_reset(zone, compact_order, true);
 		count_vm_event(COMPACTSUCCESS);
 		return page;
 	}
@@ -4455,9 +4471,14 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
 	struct page *page = NULL;
 	unsigned long pflags;
 	bool drained = false;
+	int reclaim_order = order;
+
+	/* Match the slowpath compaction promotion in __alloc_pages_direct_compact */
+	if ((alloc_flags & ALLOC_NOFRAGMENT) && ac->migratetype != MIGRATE_MOVABLE)
+		reclaim_order = max(order, pageblock_order);
 
 	psi_memstall_enter(&pflags);
-	*did_some_progress = __perform_reclaim(gfp_mask, order, ac);
+	*did_some_progress = __perform_reclaim(gfp_mask, reclaim_order, ac);
 	if (unlikely(!(*did_some_progress)))
 		goto out;
 
-- 
2.55.0

Re: [PATCH v2 4/4] mm: page_alloc: fix non-movable reclaim storm in defrag_mode

[Brendan Jackman]

On Wed Jul 22, 2026 at 2:56 PM UTC, Johannes Weiner wrote:
> As we deployed defrag_mode into Meta production, pressure spikes and
> excessive swapping were observed on some workloads. Tracing confirmed
> that this is unmovable/reclaimable requests spinning in the allocator
> and direct reclaim, causing excessive amounts of swap.
>
> The initial plan for defrag_mode was to rely on kswapd/kcompactd to
> produce blocks, and if those are overwhelmed under high pressure, let
> the allocator fall back (__rmqueue_steal()) after its retry loops.
> However, that retrying results in more reclaim on some of these
> workloads than we'd hoped, sometimes excessively so, spurred on by the
> !costly order conditions in should_reclaim_retry().
>
> The storms are dependent on the request type. Reclaim will inevitably
> make room in existing movable blocks, since that's where the LRU pages
> live. So if movable requests retry on reclaim, they make progress.
>
> When non-movable requests spin in reclaim that isn't productive. They
> cannot use the individually freed pages, and the process is unlikely
> to accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar.
> They spin and overreclaim excessively, which tanks performance and
> triggers userspace guards like swap exhaustion or pressure based OOM.
>
> To fix this, send non-movable requests, regardless of order, into
> pageblock reclaim/compaction. This way, they help move things along to
> meet the ALLOC_NOFRAGMENT bar. After this patch, the reclaim storms
> and excess OOM rates are no longer observed in production.
>
> The longer-term plan is still to have all requests, including the
> movable ones, help make blocks to spread the cost of defragmenting
> more evenly and fairly; combined with proper watermarking to reduce
> allocation latencies in the common case. However, doing this naively
> unearths scaling and concurrency limitations in compaction that need
> to be addressed first. Promoting just non-movables for now is the
> minimally viable bug fix for the above issue.
>
> Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
> Cc: <stable@vger.kernel.org>
> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>

Just a note that I'm rebasing my ALLOC_UNMAPPED series onto this, per
the discusison here I have similar needs re making compaction run on
orders above the allocation order:

https://lore.kernel.org/all/17f2f1e7-36ee-4d90-af1c-1fbca6f94025@kernel.org/

I think the way you've done it (i.e. use the capture_control for the
"other" order) is good.

Re: [PATCH v2 4/4] mm: page_alloc: fix non-movable reclaim storm in defrag_mode

[Andrew Morton]

On Wed, 22 Jul 2026 10:56:47 -0400 Johannes Weiner <hannes@cmpxchg.org> wrote:

> As we deployed defrag_mode into Meta production, pressure spikes and
> excessive swapping were observed on some workloads. Tracing confirmed
> that this is unmovable/reclaimable requests spinning in the allocator
> and direct reclaim, causing excessive amounts of swap.
> 
> The initial plan for defrag_mode was to rely on kswapd/kcompactd to
> produce blocks, and if those are overwhelmed under high pressure, let
> the allocator fall back (__rmqueue_steal()) after its retry loops.
> However, that retrying results in more reclaim on some of these
> workloads than we'd hoped, sometimes excessively so, spurred on by the
> !costly order conditions in should_reclaim_retry().
> 
> The storms are dependent on the request type. Reclaim will inevitably
> make room in existing movable blocks, since that's where the LRU pages
> live. So if movable requests retry on reclaim, they make progress.
> 
> When non-movable requests spin in reclaim that isn't productive. They
> cannot use the individually freed pages, and the process is unlikely
> to accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar.
> They spin and overreclaim excessively, which tanks performance and
> triggers userspace guards like swap exhaustion or pressure based OOM.
> 
> To fix this, send non-movable requests, regardless of order, into
> pageblock reclaim/compaction. This way, they help move things along to
> meet the ALLOC_NOFRAGMENT bar. After this patch, the reclaim storms
> and excess OOM rates are no longer observed in production.
> 
> The longer-term plan is still to have all requests, including the
> movable ones, help make blocks to spread the cost of defragmenting
> more evenly and fairly; combined with proper watermarking to reduce
> allocation latencies in the common case. However, doing this naively
> unearths scaling and concurrency limitations in compaction that need
> to be addressed first. Promoting just non-movables for now is the
> minimally viable bug fix for the above issue.
> 
> Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
> Cc: <stable@vger.kernel.org>
> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>

This cc:stable fix doesn't apply to current mainline because the
preceding non-cc:stable patches get in the way.

This means that the -stable maintainers will have to ask you to redo
the fix to make it backportable.  And you'll then be preparing a patch
which hasn't had standalone testing in mainline.  All very cumbersome.

So is it practical to make the fix come first, as a cc:stable hotfix so
we can upstream the other patches in the next merge window?

(An alternative would be to put cc:stable on all four patches.  That's
a bit abusive but we do it rarely).

Either way, I'll get these into mm.git for testing while we await
review on [4/4], thanks.

Re: [PATCH v2 4/4] mm: page_alloc: fix non-movable reclaim storm in defrag_mode

[Vlastimil Babka (SUSE)]

+ Greg and Sasha

On 7/23/26 01:22, Andrew Morton wrote:
> On Wed, 22 Jul 2026 10:56:47 -0400 Johannes Weiner <hannes@cmpxchg.org> wrote:
> 
>> 
>> Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
>> Cc: <stable@vger.kernel.org>
>> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
> 
> This cc:stable fix doesn't apply to current mainline because the
> preceding non-cc:stable patches get in the way.
> 
> This means that the -stable maintainers will have to ask you to redo
> the fix to make it backportable.  And you'll then be preparing a patch
> which hasn't had standalone testing in mainline.  All very cumbersome.

Or they backport the whole series, even if only 4/4 has a Fixes: tag.

> So is it practical to make the fix come first, as a cc:stable hotfix so
> we can upstream the other patches in the next merge window?

I think it can't come first, it's simply a complex fix that can't be done in
a single patch that would be reviewable. 3/4 is certainly a strict
dependency. I'm not sure if 2/4 as well. Perhaps 2/4 does deserve a Fixes:
tag and Cc: stable on its own, as it also fixes a defrag_mode deficiency.

> (An alternative would be to put cc:stable on all four patches.  That's
> a bit abusive but we do it rarely).

I don't think it's necessary, the stable team are quite capable of figuring
out the deps and backporting them with a Stable-dep-off: tag. IIRC the
"fake" tagging is discouraged? Maybe they can clarify.

> Either way, I'll get these into mm.git for testing while we await
> review on [4/4], thanks.
> 

Re: [PATCH v2 4/4] mm: page_alloc: fix non-movable reclaim storm in defrag_mode

[Johannes Weiner]

On Thu, Jul 23, 2026 at 12:22:58PM +0200, Vlastimil Babka (SUSE) wrote:
> + Greg and Sasha
> 
> On 7/23/26 01:22, Andrew Morton wrote:
> > On Wed, 22 Jul 2026 10:56:47 -0400 Johannes Weiner <hannes@cmpxchg.org> wrote:
> > 
> >> 
> >> Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
> >> Cc: <stable@vger.kernel.org>
> >> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
> > 
> > This cc:stable fix doesn't apply to current mainline because the
> > preceding non-cc:stable patches get in the way.
> > 
> > This means that the -stable maintainers will have to ask you to redo
> > the fix to make it backportable.  And you'll then be preparing a patch
> > which hasn't had standalone testing in mainline.  All very cumbersome.
> 
> Or they backport the whole series, even if only 4/4 has a Fixes: tag.
> 
> > So is it practical to make the fix come first, as a cc:stable hotfix so
> > we can upstream the other patches in the next merge window?
> 
> I think it can't come first, it's simply a complex fix that can't be done in
> a single patch that would be reviewable. 3/4 is certainly a strict
> dependency. I'm not sure if 2/4 as well. Perhaps 2/4 does deserve a Fixes:
> tag and Cc: stable on its own, as it also fixes a defrag_mode deficiency.

Yes. Only 1/4 could be argued away.

2/4 actually came after 4/4 during development. Without it, compaction
cannot make sufficient forward progress on the promoted non-movable
requests. So it is a hard functional requirement.

Whether it deserves an independent Fixes: depends on how many
GFP_KERNEL pageblock_order requests we have :-) I'm guessing not many.

> > (An alternative would be to put cc:stable on all four patches.  That's
> > a bit abusive but we do it rarely).
> 
> I don't think it's necessary, the stable team are quite capable of figuring
> out the deps and backporting them with a Stable-dep-off: tag. IIRC the
> "fake" tagging is discouraged? Maybe they can clarify.

Yes, I *think* it should work with tagging just 4/4.

It won't apply cleanly or build without 3/4. So I'd assume if that
dependency is lost, the automated backport will bounce and I can send
a backport series by hand.

Re: [PATCH v2 4/4] mm: page_alloc: fix non-movable reclaim storm in defrag_mode

[Vlastimil Babka (SUSE)]

On 7/23/26 16:08, Johannes Weiner wrote:
> On Thu, Jul 23, 2026 at 12:22:58PM +0200, Vlastimil Babka (SUSE) wrote:
>> + Greg and Sasha
>> 
>> On 7/23/26 01:22, Andrew Morton wrote:
>> > On Wed, 22 Jul 2026 10:56:47 -0400 Johannes Weiner <hannes@cmpxchg.org> wrote:
>> > 
>> >> 
>> >> Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
>> >> Cc: <stable@vger.kernel.org>
>> >> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
>> > 
>> > This cc:stable fix doesn't apply to current mainline because the
>> > preceding non-cc:stable patches get in the way.
>> > 
>> > This means that the -stable maintainers will have to ask you to redo
>> > the fix to make it backportable.  And you'll then be preparing a patch
>> > which hasn't had standalone testing in mainline.  All very cumbersome.
>> 
>> Or they backport the whole series, even if only 4/4 has a Fixes: tag.
>> 
>> > So is it practical to make the fix come first, as a cc:stable hotfix so
>> > we can upstream the other patches in the next merge window?
>> 
>> I think it can't come first, it's simply a complex fix that can't be done in
>> a single patch that would be reviewable. 3/4 is certainly a strict
>> dependency. I'm not sure if 2/4 as well. Perhaps 2/4 does deserve a Fixes:
>> tag and Cc: stable on its own, as it also fixes a defrag_mode deficiency.
> 
> Yes. Only 1/4 could be argued away.
> 
> 2/4 actually came after 4/4 during development. Without it, compaction
> cannot make sufficient forward progress on the promoted non-movable
> requests. So it is a hard functional requirement.

So that's the tricky part then. If it's a hard functional requirement for
4/4 and yet leaving it out results in no conflicts when applying 3/4 and 4/4
(as it indeed seems so, based on my local test), then it's quite possible
stable folks will figure out themselves they need 3/4 as a stable-dep-off:
4/4 but won't backport 2/4, unless you act quickly enough on their mails.

> Whether it deserves an independent Fixes: depends on how many
> GFP_KERNEL pageblock_order requests we have :-) I'm guessing not many.

I don't know how else to indicate it though. It's kind of a co-fix with 4/4?

>> > (An alternative would be to put cc:stable on all four patches.  That's
>> > a bit abusive but we do it rarely).
>> 
>> I don't think it's necessary, the stable team are quite capable of figuring
>> out the deps and backporting them with a Stable-dep-off: tag. IIRC the
>> "fake" tagging is discouraged? Maybe they can clarify.
> 
> Yes, I *think* it should work with tagging just 4/4.
> 
> It won't apply cleanly or build without 3/4. So I'd assume if that
> dependency is lost, the automated backport will bounce and I can send
> a backport series by hand.

The way I see if often happen is that Greg sends a mail that it bounced and
soon after Sasha replies with fixed up version/series. Which could very
easily be 3/4 + 4/4 without 2/4.

Re: [PATCH v2 4/4] mm: page_alloc: fix non-movable reclaim storm in defrag_mode

[Andrew Morton]

On Thu, 23 Jul 2026 16:17:51 +0200 "Vlastimil Babka (SUSE)" <vbabka@kernel.org> wrote:

> > 2/4 actually came after 4/4 during development. Without it, compaction
> > cannot make sufficient forward progress on the promoted non-movable
> > requests. So it is a hard functional requirement.
> 
> So that's the tricky part then. If it's a hard functional requirement for
> 4/4 and yet leaving it out results in no conflicts when applying 3/4 and 4/4
> (as it indeed seems so, based on my local test), then it's quite possible
> stable folks will figure out themselves they need 3/4 as a stable-dep-off:
> 4/4 but won't backport 2/4, unless you act quickly enough on their mails.
> 
> > Whether it deserves an independent Fixes: depends on how many
> > GFP_KERNEL pageblock_order requests we have :-) I'm guessing not many.
> 
> I don't know how else to indicate it though. It's kind of a co-fix with 4/4?
> 
> >> > (An alternative would be to put cc:stable on all four patches.  That's
> >> > a bit abusive but we do it rarely).
> >> 
> >> I don't think it's necessary, the stable team are quite capable of figuring
> >> out the deps and backporting them with a Stable-dep-off: tag. IIRC the
> >> "fake" tagging is discouraged? Maybe they can clarify.
> > 
> > Yes, I *think* it should work with tagging just 4/4.
> > 
> > It won't apply cleanly or build without 3/4. So I'd assume if that
> > dependency is lost, the automated backport will bounce and I can send
> > a backport series by hand.
> 
> The way I see if often happen is that Greg sends a mail that it bounced and
> soon after Sasha replies with fixed up version/series. Which could very
> easily be 3/4 + 4/4 without 2/4.

My brain hurts.  I'll include

Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Cc: <stable@vger.kernel.org>

in all four patches.

Re: [PATCH v2 4/4] mm: page_alloc: fix non-movable reclaim storm in defrag_mode

[Greg Kroah-Hartman]

On Thu, Jul 23, 2026 at 05:02:49PM -0700, Andrew Morton wrote:
> On Thu, 23 Jul 2026 16:17:51 +0200 "Vlastimil Babka (SUSE)" <vbabka@kernel.org> wrote:
> 
> > > 2/4 actually came after 4/4 during development. Without it, compaction
> > > cannot make sufficient forward progress on the promoted non-movable
> > > requests. So it is a hard functional requirement.
> > 
> > So that's the tricky part then. If it's a hard functional requirement for
> > 4/4 and yet leaving it out results in no conflicts when applying 3/4 and 4/4
> > (as it indeed seems so, based on my local test), then it's quite possible
> > stable folks will figure out themselves they need 3/4 as a stable-dep-off:
> > 4/4 but won't backport 2/4, unless you act quickly enough on their mails.
> > 
> > > Whether it deserves an independent Fixes: depends on how many
> > > GFP_KERNEL pageblock_order requests we have :-) I'm guessing not many.
> > 
> > I don't know how else to indicate it though. It's kind of a co-fix with 4/4?
> > 
> > >> > (An alternative would be to put cc:stable on all four patches.  That's
> > >> > a bit abusive but we do it rarely).
> > >> 
> > >> I don't think it's necessary, the stable team are quite capable of figuring
> > >> out the deps and backporting them with a Stable-dep-off: tag. IIRC the
> > >> "fake" tagging is discouraged? Maybe they can clarify.
> > > 
> > > Yes, I *think* it should work with tagging just 4/4.
> > > 
> > > It won't apply cleanly or build without 3/4. So I'd assume if that
> > > dependency is lost, the automated backport will bounce and I can send
> > > a backport series by hand.
> > 
> > The way I see if often happen is that Greg sends a mail that it bounced and
> > soon after Sasha replies with fixed up version/series. Which could very
> > easily be 3/4 + 4/4 without 2/4.
> 
> My brain hurts.  I'll include
> 
> Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
> Cc: <stable@vger.kernel.org>
> 
> in all four patches.

Thank you!  Don't make us "figure it out" as that way lies struggles and
time delays and the huge chance I get it wrong.  Tagging them all
ensures none of this happens and it all "just works".

thanks,

greg k-h

Re: [PATCH v2 4/4] mm: page_alloc: fix non-movable reclaim storm in defrag_mode

[Andrew Morton]

On Thu, 23 Jul 2026 12:22:58 +0200 "Vlastimil Babka (SUSE)" <vbabka@kernel.org> wrote:

> + Greg and Sasha
> 
> On 7/23/26 01:22, Andrew Morton wrote:
> > On Wed, 22 Jul 2026 10:56:47 -0400 Johannes Weiner <hannes@cmpxchg.org> wrote:
> > 
> >> 
> >> Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
> >> Cc: <stable@vger.kernel.org>
> >> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
> > 
> > This cc:stable fix doesn't apply to current mainline because the
> > preceding non-cc:stable patches get in the way.
> > 
> > This means that the -stable maintainers will have to ask you to redo
> > the fix to make it backportable.  And you'll then be preparing a patch
> > which hasn't had standalone testing in mainline.  All very cumbersome.
> 
> Or they backport the whole series, even if only 4/4 has a Fixes: tag.

Sure.

> I don't think it's necessary, the stable team are quite capable of figuring
> out the deps and backporting them with a Stable-dep-off: tag.

What's that.

hp2:/usr/src/25> grep -ri stable-dep ../gitlog Documentation  
hp2:/usr/src/25> 

> IIRC the
> "fake" tagging is discouraged? Maybe they can clarify.

Re: [PATCH v2 4/4] mm: page_alloc: fix non-movable reclaim storm in defrag_mode

[Vlastimil Babka (SUSE)]

On 7/22/26 16:56, Johannes Weiner wrote:
> As we deployed defrag_mode into Meta production, pressure spikes and
> excessive swapping were observed on some workloads. Tracing confirmed
> that this is unmovable/reclaimable requests spinning in the allocator
> and direct reclaim, causing excessive amounts of swap.
> 
> The initial plan for defrag_mode was to rely on kswapd/kcompactd to
> produce blocks, and if those are overwhelmed under high pressure, let
> the allocator fall back (__rmqueue_steal()) after its retry loops.
> However, that retrying results in more reclaim on some of these
> workloads than we'd hoped, sometimes excessively so, spurred on by the
> !costly order conditions in should_reclaim_retry().
> 
> The storms are dependent on the request type. Reclaim will inevitably
> make room in existing movable blocks, since that's where the LRU pages
> live. So if movable requests retry on reclaim, they make progress.
> 
> When non-movable requests spin in reclaim that isn't productive. They
> cannot use the individually freed pages, and the process is unlikely
> to accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar.
> They spin and overreclaim excessively, which tanks performance and
> triggers userspace guards like swap exhaustion or pressure based OOM.
> 
> To fix this, send non-movable requests, regardless of order, into
> pageblock reclaim/compaction. This way, they help move things along to
> meet the ALLOC_NOFRAGMENT bar. After this patch, the reclaim storms
> and excess OOM rates are no longer observed in production.
> 
> The longer-term plan is still to have all requests, including the
> movable ones, help make blocks to spread the cost of defragmenting
> more evenly and fairly; combined with proper watermarking to reduce
> allocation latencies in the common case. However, doing this naively
> unearths scaling and concurrency limitations in compaction that need
> to be addressed first. Promoting just non-movables for now is the
> minimally viable bug fix for the above issue.
> 
> Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
> Cc: <stable@vger.kernel.org>
> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>

Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>

> ---
>  mm/internal.h   |  6 ++++++
>  mm/page_alloc.c | 31 ++++++++++++++++++++++++++-----
>  2 files changed, 32 insertions(+), 5 deletions(-)
> 
> diff --git a/mm/internal.h b/mm/internal.h
> index 6d3402001b93..5acba3470659 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -1061,6 +1061,12 @@ struct compact_control {
>  struct capture_control {
>  	struct zone *zone;
>  	int migratetype;
> +	/*
> +	 * Allocation request order. May differ from the compaction
> +	 * order: defrag_mode promotes sub-block allocations to
> +	 * pageblock-order compaction; capture still matches at the
> +	 * original allocation order so prep_new_page() is consistent.
> +	 */

Yeah, a premature capture can't jeopardize the full-pageblock clearing
because compaction_capture() disallows capturing <pageblock_order page from
MOVABLE pageblock for !MOVABLE allocations.

It's thus unlikely the capture will ever succeed in these scenarios. But
it's not wrong.

>  	int order;
>  	struct page *page;
>  };
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 4b9686fedb71..f92055827ae9 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -4152,8 +4152,24 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
>  		.order = order,
>  		.page = NULL,
>  	};
> +	int compact_order = order;
>  
> -	if (!order)
> +	/*
> +	 * If fallbacks are not permitted (defrag_mode), we either
> +	 * need to reclaim space in a block of matching type, or clear
> +	 * out an entire block to allow __rmqueue_claim() to convert.
> +	 *
> +	 * Reclaim by itself is primarily freeing space in movable
> +	 * blocks, since that's where the LRU pages live. So this
> +	 * works for movable requests, but not for others.
> +	 *
> +	 * For those, promote the order to help make blocks, instead
> +	 * of spinning in reclaim alone unproductively.
> +	 */
> +	if ((alloc_flags & ALLOC_NOFRAGMENT) && ac->migratetype != MIGRATE_MOVABLE)
> +		compact_order = max(order, pageblock_order);
> +
> +	if (!compact_order)
>  		return NULL;
>  
>  	psi_memstall_enter(&pflags);
> @@ -4169,8 +4185,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
>  	barrier();
>  	WRITE_ONCE(current->capture_control, &capc);
>  
> -	*compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac,
> -							       prio, &capc);
> +	*compact_result = try_to_compact_pages(gfp_mask, compact_order,
> +					       alloc_flags, ac, prio, &capc);
>  
>  	/*
>  	 * Make sure we hide capture control first before we read the captured
> @@ -4215,7 +4231,7 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
>  		struct zone *zone = page_zone(page);
>  
>  		zone->compact_blockskip_flush = false;
> -		compaction_defer_reset(zone, order, true);
> +		compaction_defer_reset(zone, compact_order, true);
>  		count_vm_event(COMPACTSUCCESS);
>  		return page;
>  	}
> @@ -4455,9 +4471,14 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
>  	struct page *page = NULL;
>  	unsigned long pflags;
>  	bool drained = false;
> +	int reclaim_order = order;
> +
> +	/* Match the slowpath compaction promotion in __alloc_pages_direct_compact */
> +	if ((alloc_flags & ALLOC_NOFRAGMENT) && ac->migratetype != MIGRATE_MOVABLE)
> +		reclaim_order = max(order, pageblock_order);
>  
>  	psi_memstall_enter(&pflags);
> -	*did_some_progress = __perform_reclaim(gfp_mask, order, ac);
> +	*did_some_progress = __perform_reclaim(gfp_mask, reclaim_order, ac);
>  	if (unlikely(!(*did_some_progress)))
>  		goto out;
>  

Re: [PATCH v2 4/4] mm: page_alloc: fix non-movable reclaim storm in defrag_mode

[Brendan Jackman]

On Wed Jul 22, 2026 at 2:56 PM UTC, Johannes Weiner wrote:
> As we deployed defrag_mode into Meta production, pressure spikes and
> excessive swapping were observed on some workloads. Tracing confirmed
> that this is unmovable/reclaimable requests spinning in the allocator
> and direct reclaim, causing excessive amounts of swap.
>
> The initial plan for defrag_mode was to rely on kswapd/kcompactd to
> produce blocks, and if those are overwhelmed under high pressure, let
> the allocator fall back (__rmqueue_steal()) after its retry loops.
> However, that retrying results in more reclaim on some of these
> workloads than we'd hoped, sometimes excessively so, spurred on by the
> !costly order conditions in should_reclaim_retry().
>
> The storms are dependent on the request type. Reclaim will inevitably
> make room in existing movable blocks, since that's where the LRU pages
> live. So if movable requests retry on reclaim, they make progress.
>
> When non-movable requests spin in reclaim that isn't productive. They
> cannot use the individually freed pages, and the process is unlikely
> to accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar.
> They spin and overreclaim excessively, which tanks performance and
> triggers userspace guards like swap exhaustion or pressure based OOM.
>
> To fix this, send non-movable requests, regardless of order, into
> pageblock reclaim/compaction. This way, they help move things along to
> meet the ALLOC_NOFRAGMENT bar. After this patch, the reclaim storms
> and excess OOM rates are no longer observed in production.
>
> The longer-term plan is still to have all requests, including the
> movable ones, help make blocks to spread the cost of defragmenting
> more evenly and fairly; combined with proper watermarking to reduce
> allocation latencies in the common case. However, doing this naively
> unearths scaling and concurrency limitations in compaction that need
> to be addressed first. Promoting just non-movables for now is the
> minimally viable bug fix for the above issue.

Please forgive the noob question, I'm still struggling to get a really
good mental handle on this stuff. But what about compact_first here?
When we're promoting the order for compaction does it also make sense to
promote compaction itself?

I'm aware you said "minimally viable bug fix" so it's fine if this falls
outside of that, I'm just trying to poke around to improve my
understanding.

Re: [PATCH v2 4/4] mm: page_alloc: fix non-movable reclaim storm in defrag_mode

[Johannes Weiner]

On Fri, Jul 24, 2026 at 03:09:49PM +0000, Brendan Jackman wrote:
> On Wed Jul 22, 2026 at 2:56 PM UTC, Johannes Weiner wrote:
> > As we deployed defrag_mode into Meta production, pressure spikes and
> > excessive swapping were observed on some workloads. Tracing confirmed
> > that this is unmovable/reclaimable requests spinning in the allocator
> > and direct reclaim, causing excessive amounts of swap.
> >
> > The initial plan for defrag_mode was to rely on kswapd/kcompactd to
> > produce blocks, and if those are overwhelmed under high pressure, let
> > the allocator fall back (__rmqueue_steal()) after its retry loops.
> > However, that retrying results in more reclaim on some of these
> > workloads than we'd hoped, sometimes excessively so, spurred on by the
> > !costly order conditions in should_reclaim_retry().
> >
> > The storms are dependent on the request type. Reclaim will inevitably
> > make room in existing movable blocks, since that's where the LRU pages
> > live. So if movable requests retry on reclaim, they make progress.
> >
> > When non-movable requests spin in reclaim that isn't productive. They
> > cannot use the individually freed pages, and the process is unlikely
> > to accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar.
> > They spin and overreclaim excessively, which tanks performance and
> > triggers userspace guards like swap exhaustion or pressure based OOM.
> >
> > To fix this, send non-movable requests, regardless of order, into
> > pageblock reclaim/compaction. This way, they help move things along to
> > meet the ALLOC_NOFRAGMENT bar. After this patch, the reclaim storms
> > and excess OOM rates are no longer observed in production.
> >
> > The longer-term plan is still to have all requests, including the
> > movable ones, help make blocks to spread the cost of defragmenting
> > more evenly and fairly; combined with proper watermarking to reduce
> > allocation latencies in the common case. However, doing this naively
> > unearths scaling and concurrency limitations in compaction that need
> > to be addressed first. Promoting just non-movables for now is the
> > minimally viable bug fix for the above issue.
> 
> Please forgive the noob question, I'm still struggling to get a really
> good mental handle on this stuff. But what about compact_first here?
> When we're promoting the order for compaction does it also make sense to
> promote compaction itself?

My understanding is that this is just an optimization for requests
that are more likely shape-limited (block contiguity) rather than
capacity-limited (free order-0). In that case we do a quick async
compaction attempt first and skip a potentially unnecessary but still
disruptive reclaim invocation.

However, it does seem to me there is a bit of redundancy between the
allocator and reclaim: direct reclaim also has that early bailout
condition on compaction_ready() for costly orders.

In defrag_mode, since reclaim is promoted to pageblock_order (costly),
we always bail reclaim on compaction_ready(). So I would assume there
is not much difference from which we call first - although I have not
tested it.

> I'm aware you said "minimally viable bug fix" so it's fine if this falls
> outside of that, I'm just trying to poke around to improve my
> understanding.

It works in production, but I'm not entirely happy yet with the retry
logic. I also find it somewhat difficult to understand, especially the
split between should_reclaim_retry() and should_compact_retry().

I'm working on unifying them and put the defrag_mode order promotion
directly into the slowpath function. This way we can use "work_order"
and "order" throughout the functions and its callees as appropriate.

But yeah this is follow-up work to the fix here.

Re: [PATCH v2 4/4] mm: page_alloc: fix non-movable reclaim storm in defrag_mode

[Brendan Jackman]

On Wed Jul 22, 2026 at 2:56 PM UTC, Johannes Weiner wrote:
> @@ -4169,8 +4185,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
>  	barrier();
>  	WRITE_ONCE(current->capture_control, &capc);
>  
> -	*compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac,
> -							       prio, &capc);
> +	*compact_result = try_to_compact_pages(gfp_mask, compact_order,
> +					       alloc_flags, ac, prio, &capc);
>  

Nit: the comment for try_to_compact_pages() is falsified by this,
we need something like:

diff --git i/mm/compaction.c w/mm/compaction.c
index 0568623d9384d..ca57403dc1522 100644
--- i/mm/compaction.c
+++ w/mm/compaction.c
@@ -2823,7 +2823,7 @@ static enum compact_result compact_zone_order(struct zone *zone, int order,
 /**
  * try_to_compact_pages - Direct compact to satisfy a high-order allocation
  * @gfp_mask: The GFP mask of the current allocation
- * @order: The order of the current allocation
+ * @order: The order to try and make available
  * @alloc_flags: The allocation flags of the current allocation
  * @ac: The context of current allocation
  * @prio: Determines how hard direct compaction should try to succeed

Re: [PATCH v2 4/4] mm: page_alloc: fix non-movable reclaim storm in defrag_mode

[Johannes Weiner]

On Sat, Jul 25, 2026 at 12:34:29PM +0000, Brendan Jackman wrote:
> On Wed Jul 22, 2026 at 2:56 PM UTC, Johannes Weiner wrote:
> > @@ -4169,8 +4185,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
> >  	barrier();
> >  	WRITE_ONCE(current->capture_control, &capc);
> >  
> > -	*compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac,
> > -							       prio, &capc);
> > +	*compact_result = try_to_compact_pages(gfp_mask, compact_order,
> > +					       alloc_flags, ac, prio, &capc);
> >  
> 
> Nit: the comment for try_to_compact_pages() is falsified by this,
> we need something like:

Yes, good catch. Thanks.

Andrew, would you mind folding this difflet in place?

> diff --git i/mm/compaction.c w/mm/compaction.c
> index 0568623d9384d..ca57403dc1522 100644
> --- i/mm/compaction.c
> +++ w/mm/compaction.c
> @@ -2823,7 +2823,7 @@ static enum compact_result compact_zone_order(struct zone *zone, int order,
>  /**
>   * try_to_compact_pages - Direct compact to satisfy a high-order allocation
>   * @gfp_mask: The GFP mask of the current allocation
> - * @order: The order of the current allocation
> + * @order: The order to try and make available
>   * @alloc_flags: The allocation flags of the current allocation
>   * @ac: The context of current allocation
>   * @prio: Determines how hard direct compaction should try to succeed