[PATCH] mm/mglru: skip oom killing for high order allocs in kswapd

[PATCH] mm/mglru: skip oom killing for high order allocs in kswapd

[Bingfang Guo via B4 Relay]

From: Bingfang Guo <bingfangguo@tencent.com>

We observed that when MGLRU min_ttl is set and the system is highly
fragmented, high order allocations (e.g. order == 9) will fail and wake
up kswapd. It can not make any progress, while pushing generations of
memcgs so that they all end up younger than min_ttl_ms, repeatedly
triggering the OOM killer in lru_gen_age_node(). Then the whole process
repeats until all processes are killed and the kernel panics.

Fix this by skipping OOM killer invocations in lru_gen_age_node() for
sc->order > PAGE_ALLOC_COSTLY_ORDER, which is also what the normal
allocation path will do, to fail high order allocations without OOM
killing.

==== Testing ====

Tested on a VM with 6 cores and 6GiB of memory, HZ=1000, MGLRU enabled,
min_ttl_ms set to 1000, and a workload that fragments the memory then
allocates 2M pages.

Without the patch, kswapd kills all processes and panics the system even
with plenty of memory available. With the patch, allocations fail but
the system remains stable.

Fixes: 30d77b7eef019 ("mm/mglru: fix ineffective protection calculation")
Signed-off-by: Bingfang Guo <bingfangguo@tencent.com>
---
 mm/vmscan.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 9959ca158ce83..b32777468ead8 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4407,6 +4407,12 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc)
 			reclaimable = lruvec_is_reclaimable(lruvec, sc, min_ttl);
 	} while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)));
 
+	/*
+	 * OOM killing won't help high order allocations. Skip and let them fail.
+	 */
+	if (sc->order > PAGE_ALLOC_COSTLY_ORDER)
+		return;
+
 	/*
 	 * The main goal is to OOM kill if every generation from all memcgs is
 	 * younger than min_ttl. However, another possibility is all memcgs are

---
base-commit: 39274b454e9993cea319f4ac342a8082a9ee1730
change-id: 20261026-fix-mglru-kswapd-age-98390614c746

Best regards,
--  
Bingfang Guo <bingfangguo@tencent.com>

Re: [PATCH] mm/mglru: skip oom killing for high order allocs in kswapd

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] mm/mglru: lru_gen_age_node() still OOM-kills for order 1..3 reclaim
--

commit cb44e931601f0546134f19e7185758d84dbc23fb
Author: Bingfang Guo <bingfangguo@tencent.com>

mm/mglru: skip oom killing for high order allocs in kswapd

This patch skips invoking the OOM killer in lru_gen_age_node() when
sc->order > PAGE_ALLOC_COSTLY_ORDER so that high-order allocations in
fragmented systems with MGLRU min_ttl_ms enabled do not repeatedly trigger OOM
kills from kswapd.

> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 9959ca158ce83..b32777468ead8 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -4407,6 +4407,12 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc)
>  			reclaimable = lruvec_is_reclaimable(lruvec, sc, min_ttl);
>  	} while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)));
>  
> +	/*
> +	 * OOM killing won't help high order allocations. Skip and let them fail.
> +	 */
> +	if (sc->order > PAGE_ALLOC_COSTLY_ORDER)
> +		return;

[Severity: High]
Can kswapd still prematurely OOM-kill user processes during compaction-induced
reclaim for orders 1 through PAGE_ALLOC_COSTLY_ORDER?

Prior to commit 30d77b7eef019 ("mm/mglru: fix ineffective protection
calculation"), lru_gen_age_node() skipped OOM kills for any non-zero
sc->order, matching should_abort_scan():

mm/vmscan.c:lru_gen_age_node() {
	/* check the order to exclude compaction-induced reclaim */
	if (!min_ttl || sc->order || sc->priority == DEF_PRIORITY)
		return;
}

mm/vmscan.c:should_abort_scan() {
	/* check the order to exclude compaction-induced reclaim */
	if (!current_is_kswapd() || sc->order)
		return false;
}

In __alloc_pages_slowpath(), any order 1..3 allocation with
__GFP_KSWAPD_RECLAIM calls wake_all_kswapds(), even when the caller specifies
GFP_NOWAIT, __GFP_NORETRY, or __GFP_RETRY_MAYFAIL (for example, multi-page
slab allocations in allocate_slab(), network fragment refills in
skb_page_frag_refill(), or large-folio readahead).

When kswapd runs balance_pgdat(), it sets sc.gfp_mask to GFP_KERNEL (losing
the original caller's GFP flags) and does not run compaction inside the
reclaim loop. In kswapd_shrink_node(), sc->order is only reset to 0 after
reclaiming compact_gap(sc->order) pages:

mm/vmscan.c:kswapd_shrink_node() {
	if (sc->order && sc->nr_reclaimed >= compact_gap(sc->order))
		sc->order = 0;
}

If physical memory is fragmented for orders 1..3 while order-0 watermarks are
healthy, and aging pushes all memcg generations younger than min_ttl_ms before
compact_gap(sc->order) pages are reclaimed, sc->order remains in 1..3 and
lru_gen_age_node() will still invoke out_of_memory() on each priority step
until all processes are killed.

Additionally, checking sc->order after the mem_cgroup_iter() loop means
lruvec_is_reclaimable() is evaluated across every memcg even when sc->order
causes an early return.

Would initializing reclaimable as:

	bool reclaimable = !min_ttl || sc->order;

at the top of lru_gen_age_node() restore the original exclusion for all
compaction-induced non-zero orders and avoid the unnecessary
lruvec_is_reclaimable() checks (while still allowing OOM kills if kswapd falls
back to order-0 reclaim in kswapd() when order-0 watermarks are unbalanced)?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261027-fix-mglru-kswapd-age-v1-1-b1ffdec4ae52@tencent.com?part=1

Re: [PATCH] mm/mglru: skip oom killing for high order allocs in kswapd

[Kairui Song]

On Tue, Oct 27, 2026 at 10:21 AM Bingfang Guo via B4 Relay
<devnull+bingfangguo.tencent.com@kernel.org> wrote:
>
> From: Bingfang Guo <bingfangguo@tencent.com>
>
> We observed that when MGLRU min_ttl is set and the system is highly
> fragmented, high order allocations (e.g. order == 9) will fail and wake
> up kswapd. It can not make any progress, while pushing generations of
> memcgs so that they all end up younger than min_ttl_ms, repeatedly
> triggering the OOM killer in lru_gen_age_node(). Then the whole process
> repeats until all processes are killed and the kernel panics.
>
> Fix this by skipping OOM killer invocations in lru_gen_age_node() for
> sc->order > PAGE_ALLOC_COSTLY_ORDER, which is also what the normal
> allocation path will do, to fail high order allocations without OOM
> killing.
>
> ==== Testing ====
>
> Tested on a VM with 6 cores and 6GiB of memory, HZ=1000, MGLRU enabled,
> min_ttl_ms set to 1000, and a workload that fragments the memory then
> allocates 2M pages.
>
> Without the patch, kswapd kills all processes and panics the system even
> with plenty of memory available. With the patch, allocations fail but
> the system remains stable.
>
> Fixes: 30d77b7eef019 ("mm/mglru: fix ineffective protection calculation")
> Signed-off-by: Bingfang Guo <bingfangguo@tencent.com>
> ---
>  mm/vmscan.c | 6 ++++++
>  1 file changed, 6 insertions(+)

Thanks! Really nice catch.

> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 9959ca158ce83..b32777468ead8 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -4407,6 +4407,12 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc)
>                         reclaimable = lruvec_is_reclaimable(lruvec, sc, min_ttl);
>         } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)));
>
> +       /*
> +        * OOM killing won't help high order allocations. Skip and let them fail.

A nit pick, OOM definately helps for high order allocations :-D, maybe
something like, the OOM kill is to prevent thrashing whtn all memcgs
are younger than min_ttl (and maybe merge with the comment & check
below?), so OOM for fragmentation is definitely unwanted.

> +        */
> +       if (sc->order > PAGE_ALLOC_COSTLY_ORDER)
> +               return;
> +
>         /*
>          * The main goal is to OOM kill if every generation from all memcgs is
>          * younger than min_ttl. However, another possibility is all memcgs are

This patch is good to me on its own with a nit fix above.

And checking commit 30d77b7eef019, you might need a Fixes: or maybe
even bring the check back which was removed in that commit?

But for long term, and as a follow up, the whole idea of OOM in kswapd
seems kind of wrong, maybe at least it need to check the water mark
here?

Re: [PATCH] mm/mglru: skip oom killing for high order allocs in kswapd

[Kairui Song]

On Wed, Oct 28, 2026 at 9:56 PM Kairui Song <ryncsn@gmail.com> wrote:
>
> On Tue, Oct 27, 2026 at 10:21 AM Bingfang Guo via B4 Relay
> <devnull+bingfangguo.tencent.com@kernel.org> wrote:
> >
> > From: Bingfang Guo <bingfangguo@tencent.com>
> >
> > We observed that when MGLRU min_ttl is set and the system is highly
> > fragmented, high order allocations (e.g. order == 9) will fail and wake
> > up kswapd. It can not make any progress, while pushing generations of
> > memcgs so that they all end up younger than min_ttl_ms, repeatedly
> > triggering the OOM killer in lru_gen_age_node(). Then the whole process
> > repeats until all processes are killed and the kernel panics.
> >
> > Fix this by skipping OOM killer invocations in lru_gen_age_node() for
> > sc->order > PAGE_ALLOC_COSTLY_ORDER, which is also what the normal
> > allocation path will do, to fail high order allocations without OOM
> > killing.
> >
> > ==== Testing ====
> >
> > Tested on a VM with 6 cores and 6GiB of memory, HZ=1000, MGLRU enabled,
> > min_ttl_ms set to 1000, and a workload that fragments the memory then
> > allocates 2M pages.
> >
> > Without the patch, kswapd kills all processes and panics the system even
> > with plenty of memory available. With the patch, allocations fail but
> > the system remains stable.
> >
> > Fixes: 30d77b7eef019 ("mm/mglru: fix ineffective protection calculation")
> > Signed-off-by: Bingfang Guo <bingfangguo@tencent.com>
> > ---
> >  mm/vmscan.c | 6 ++++++
> >  1 file changed, 6 insertions(+)
>
> Thanks! Really nice catch.
>
> > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > index 9959ca158ce83..b32777468ead8 100644
> > --- a/mm/vmscan.c
> > +++ b/mm/vmscan.c
> > @@ -4407,6 +4407,12 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc)
> >                         reclaimable = lruvec_is_reclaimable(lruvec, sc, min_ttl);
> >         } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)));
> >
> > +       /*
> > +        * OOM killing won't help high order allocations. Skip and let them fail.
>
> A nit pick, OOM definately helps for high order allocations :-D, maybe
> something like, the OOM kill is to prevent thrashing whtn all memcgs
> are younger than min_ttl (and maybe merge with the comment & check
> below?), so OOM for fragmentation is definitely unwanted.
>
> > +        */
> > +       if (sc->order > PAGE_ALLOC_COSTLY_ORDER)
> > +               return;
> > +
> >         /*
> >          * The main goal is to OOM kill if every generation from all memcgs is
> >          * younger than min_ttl. However, another possibility is all memcgs are
>
> This patch is good to me on its own with a nit fix above.
>
> And checking commit 30d77b7eef019, you might need a Fixes: or maybe
> even bring the check back which was removed in that commit?

Oh I noticed it's already there, please ignore this one :P

Re: [PATCH] mm/mglru: skip oom killing for high order allocs in kswapd

[Bingfang Guo]

[-- Attachment #1: Type: text/plain, Size: 3716 bytes --]

On Wed, Oct 28, 2026 at 09:56:48PM +0800, Kairui Song wrote:
> On Tue, Oct 27, 2026 at 10:21 AM Bingfang Guo via B4 Relay
> <devnull+bingfangguo.tencent.com@kernel.org> wrote:
>> 
>> From: Bingfang Guo <bingfangguo@tencent.com>
>> 
>> We observed that when MGLRU min_ttl is set and the system is highly
>> fragmented, high order allocations (e.g. order == 9) will fail and wake
>> up kswapd. It can not make any progress, while pushing generations of
>> memcgs so that they all end up younger than min_ttl_ms, repeatedly
>> triggering the OOM killer in lru_gen_age_node(). Then the whole process
>> repeats until all processes are killed and the kernel panics.
>> 
>> Fix this by skipping OOM killer invocations in lru_gen_age_node() for
>> sc->order > PAGE_ALLOC_COSTLY_ORDER, which is also what the normal
>> allocation path will do, to fail high order allocations without OOM
>> killing.
>> 
>> ==== Testing ====
>> 
>> Tested on a VM with 6 cores and 6GiB of memory, HZ=1000, MGLRU enabled,
>> min_ttl_ms set to 1000, and a workload that fragments the memory then
>> allocates 2M pages.
>> 
>> Without the patch, kswapd kills all processes and panics the system even
>> with plenty of memory available. With the patch, allocations fail but
>> the system remains stable.
>> 
>> Fixes: 30d77b7eef019 ("mm/mglru: fix ineffective protection calculation")
>> Signed-off-by: Bingfang Guo <bingfangguo@tencent.com>
>> ---
>> mm/vmscan.c | 6 ++++++
>> 1 file changed, 6 insertions(+)
> 
> Thanks! Really nice catch.
> 

Thanks for reviewing, Kairui!

>> diff --git a/mm/vmscan.c b/mm/vmscan.c
>> index 9959ca158ce83..b32777468ead8 100644
>> --- a/mm/vmscan.c
>> +++ b/mm/vmscan.c
>> @@ -4407,6 +4407,12 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc)
>>                        reclaimable = lruvec_is_reclaimable(lruvec, sc, min_ttl);
>>        } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)));
>> 
>> +       /*
>> +        * OOM killing won't help high order allocations. Skip and let them fail.
> 
> A nit pick, OOM definately helps for high order allocations :-D, maybe
> something like, the OOM kill is to prevent thrashing whtn all memcgs
> are younger than min_ttl (and maybe merge with the comment & check
> below?), so OOM for fragmentation is definitely unwanted.
> 

Right. OOM might help but not in this case. And that sounds make much more
sense. I'll update it!

>> +        */
>> +       if (sc->order > PAGE_ALLOC_COSTLY_ORDER)
>> +               return;
>> +
>>        /*
>>         * The main goal is to OOM kill if every generation from all memcgs is
>>         * younger than min_ttl. However, another possibility is all memcgs are
> 
> This patch is good to me on its own with a nit fix above.
> 
> And checking commit 30d77b7eef019, you might need a Fixes: or maybe
> even bring the check back which was removed in that commit?
> 

And after reading reviews from sashiko, I think I can just restore the original
check. Use:

   bool reclaimable = !min_ttl || sc->order || sc->priority != DEF_PRIORITY;

This prevents unexpected OOMs for 1..3 order allocs in some case as well,
although they should be rare based on my tests. And this removes redundant check
runs for every memcg in this case.

> But for long term, and as a follow up, the whole idea of OOM in kswapd
> seems kind of wrong, maybe at least it need to check the water mark
> here?
> 
> 

Agreed. And as you mentioned above, OOMs are not expected by fragmentation. So
it's best to have some ways to distinguish very young generations caused by
thrashing from something else.


--
Thanks,
Bingfang

[-- Attachment #2: Type: text/html, Size: 24130 bytes --]