[PATCH] mm/lruvec: preemptively free dead folios during lru_add drain

[PATCH] mm/lruvec: preemptively free dead folios during lru_add drain

[JP Kobryn (Meta)]

Of all observable lruvec lock contention in our fleet, we find that ~24%
occurs when dead folios are present in lru_add batches at drain time. This
is wasteful in the sense that the folio is added to the LRU just to be
immediately removed via folios_put_refs(), incurring two unnecessary lock
acquisitions.

Eliminate this overhead by preemptively cleaning up dead folios before they
make it into the LRU. Use folio_ref_freeze() to filter folios whose only
remaining refcount is the batch ref. When dead folios are found, move them
off the add batch and onto a temporary batch to be freed.

During A/B testing on one of our prod instagram workloads (high-frequency
short-lived requests), the patch intercepted almost all dead folios before
they entered the LRU. Data collected using the mm_lru_insertion tracepoint
shows the effectiveness of the patch:

Per-host LRU add averages at 95% CPU load
(60 hosts each side, 3 x 60s intervals)

            dead folios/min  total folios/min   dead %
unpatched:        1,297,785        19,341,986  6.7097%
patched:                 14        19,039,996  0.0001%

Within this workload, we save ~2.6M lock acquisitions per minute per host
as a result.

System-wide memory stats improved on the patched side also at 95% CPU load:
 - direct reclaim scanning reduced 7%
 - allocation stalls reduced 5.2%
 - compaction stalls reduced 12.3%
 - page frees reduced 4.9%

No regressions were observed in requests served per second or request tail
latency (p99). Both metrics showed directional improvement at higher CPU
utilization (comparing 85% to 95%).

Signed-off-by: JP Kobryn (Meta) <jp.kobryn@linux.dev>
---
 mm/swap.c | 36 +++++++++++++++++++++++++++++++++++-
 1 file changed, 35 insertions(+), 1 deletion(-)

diff --git a/mm/swap.c b/mm/swap.c
index 5cc44f0de9877..71607b0ce3d18 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -160,13 +160,36 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)
 	int i;
 	struct lruvec *lruvec = NULL;
 	unsigned long flags = 0;
+	struct folio_batch free_fbatch;
+	bool is_lru_add = (move_fn == lru_add);
+
+	/*
+	 * If we're adding to the LRU, preemptively filter dead folios. Use
+	 * this dedicated folio batch for temp storage and deferred cleanup.
+	 */
+	if (is_lru_add)
+		folio_batch_init(&free_fbatch);
 
 	for (i = 0; i < folio_batch_count(fbatch); i++) {
 		struct folio *folio = fbatch->folios[i];
 
 		/* block memcg migration while the folio moves between lru */
-		if (move_fn != lru_add && !folio_test_clear_lru(folio))
+		if (!is_lru_add && !folio_test_clear_lru(folio))
+			continue;
+
+		/*
+		 * Filter dead folios by moving them from the add batch to the temp
+		 * batch for freeing after this loop.
+		 *
+		 * Since the folio may be part of a huge page, unqueue from
+		 * deferred split list to avoid a dangling list entry.
+		 */
+		if (is_lru_add && folio_ref_freeze(folio, 1)) {
+			folio_unqueue_deferred_split(folio);
+			fbatch->folios[i] = NULL;
+			folio_batch_add(&free_fbatch, folio);
 			continue;
+		}
 
 		folio_lruvec_relock_irqsave(folio, &lruvec, &flags);
 		move_fn(lruvec, folio);
@@ -176,6 +199,13 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)
 
 	if (lruvec)
 		lruvec_unlock_irqrestore(lruvec, flags);
+
+	/* Cleanup filtered dead folios. */
+	if (is_lru_add) {
+		mem_cgroup_uncharge_folios(&free_fbatch);
+		free_unref_folios(&free_fbatch);
+	}
+
 	folios_put(fbatch);
 }
 
@@ -964,6 +994,10 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
 		struct folio *folio = folios->folios[i];
 		unsigned int nr_refs = refs ? refs[i] : 1;
 
+		/* Folio batch entry may have been preemptively removed during drain. */
+		if (!folio)
+			continue;
+
 		if (is_huge_zero_folio(folio))
 			continue;
 
-- 
2.52.0

Re: [PATCH] mm/lruvec: preemptively free dead folios during lru_add drain

[Matthew Wilcox]

On Thu, Apr 23, 2026 at 09:43:07AM -0700, JP Kobryn (Meta) wrote:
> Of all observable lruvec lock contention in our fleet, we find that ~24%
> occurs when dead folios are present in lru_add batches at drain time. This
> is wasteful in the sense that the folio is added to the LRU just to be
> immediately removed via folios_put_refs(), incurring two unnecessary lock
> acquisitions.

Well, this is a lovely patch with no obvious downsides.  Nicely done.

> Eliminate this overhead by preemptively cleaning up dead folios before they
> make it into the LRU. Use folio_ref_freeze() to filter folios whose only
> remaining refcount is the batch ref. When dead folios are found, move them
> off the add batch and onto a temporary batch to be freed.
> 
> During A/B testing on one of our prod instagram workloads (high-frequency
> short-lived requests), the patch intercepted almost all dead folios before
> they entered the LRU. Data collected using the mm_lru_insertion tracepoint
> shows the effectiveness of the patch:
> 
> Per-host LRU add averages at 95% CPU load
> (60 hosts each side, 3 x 60s intervals)
> 
>             dead folios/min  total folios/min   dead %
> unpatched:        1,297,785        19,341,986  6.7097%
> patched:                 14        19,039,996  0.0001%
> 
> Within this workload, we save ~2.6M lock acquisitions per minute per host
> as a result.
> 
> System-wide memory stats improved on the patched side also at 95% CPU load:
>  - direct reclaim scanning reduced 7%
>  - allocation stalls reduced 5.2%
>  - compaction stalls reduced 12.3%
>  - page frees reduced 4.9%
> 
> No regressions were observed in requests served per second or request tail
> latency (p99). Both metrics showed directional improvement at higher CPU
> utilization (comparing 85% to 95%).
> 
> Signed-off-by: JP Kobryn (Meta) <jp.kobryn@linux.dev>
> ---
>  mm/swap.c | 36 +++++++++++++++++++++++++++++++++++-
>  1 file changed, 35 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/swap.c b/mm/swap.c
> index 5cc44f0de9877..71607b0ce3d18 100644
> --- a/mm/swap.c
> +++ b/mm/swap.c
> @@ -160,13 +160,36 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)
>  	int i;
>  	struct lruvec *lruvec = NULL;
>  	unsigned long flags = 0;
> +	struct folio_batch free_fbatch;
> +	bool is_lru_add = (move_fn == lru_add);
> +
> +	/*
> +	 * If we're adding to the LRU, preemptively filter dead folios. Use
> +	 * this dedicated folio batch for temp storage and deferred cleanup.
> +	 */
> +	if (is_lru_add)
> +		folio_batch_init(&free_fbatch);
>  
>  	for (i = 0; i < folio_batch_count(fbatch); i++) {
>  		struct folio *folio = fbatch->folios[i];
>  
>  		/* block memcg migration while the folio moves between lru */
> -		if (move_fn != lru_add && !folio_test_clear_lru(folio))
> +		if (!is_lru_add && !folio_test_clear_lru(folio))
> +			continue;
> +
> +		/*
> +		 * Filter dead folios by moving them from the add batch to the temp
> +		 * batch for freeing after this loop.
> +		 *
> +		 * Since the folio may be part of a huge page, unqueue from
> +		 * deferred split list to avoid a dangling list entry.
> +		 */
> +		if (is_lru_add && folio_ref_freeze(folio, 1)) {
> +			folio_unqueue_deferred_split(folio);

Would it be better to do this outside the lru lock; it's just that we
don't have a convenient batched version to do it?  It seems like
there are a few places that could use a batched version in vmscan.c and
swap.c.  Not that I think we should hold up this patch to investigate
that micro-optimisation!  Just something you couldlook at as a
follow-up.

Reviewed-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Re: [PATCH] mm/lruvec: preemptively free dead folios during lru_add drain

[JP Kobryn (Meta)]

On 4/23/26 10:15 AM, Matthew Wilcox wrote:
> On Thu, Apr 23, 2026 at 09:43:07AM -0700, JP Kobryn (Meta) wrote:
>> Of all observable lruvec lock contention in our fleet, we find that ~24%
>> occurs when dead folios are present in lru_add batches at drain time. This
>> is wasteful in the sense that the folio is added to the LRU just to be
>> immediately removed via folios_put_refs(), incurring two unnecessary lock
>> acquisitions.
> 
> Well, this is a lovely patch with no obvious downsides.  Nicely done.

Thanks for the kind words and review :)

[...]
>> diff --git a/mm/swap.c b/mm/swap.c
>> index 5cc44f0de9877..71607b0ce3d18 100644
>> --- a/mm/swap.c
>> +++ b/mm/swap.c
>> @@ -160,13 +160,36 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)
>>   	int i;
>>   	struct lruvec *lruvec = NULL;
>>   	unsigned long flags = 0;
>> +	struct folio_batch free_fbatch;
>> +	bool is_lru_add = (move_fn == lru_add);
>> +
>> +	/*
>> +	 * If we're adding to the LRU, preemptively filter dead folios. Use
>> +	 * this dedicated folio batch for temp storage and deferred cleanup.
>> +	 */
>> +	if (is_lru_add)
>> +		folio_batch_init(&free_fbatch);
>>   
>>   	for (i = 0; i < folio_batch_count(fbatch); i++) {
>>   		struct folio *folio = fbatch->folios[i];
>>   
>>   		/* block memcg migration while the folio moves between lru */
>> -		if (move_fn != lru_add && !folio_test_clear_lru(folio))
>> +		if (!is_lru_add && !folio_test_clear_lru(folio))
>> +			continue;
>> +
>> +		/*
>> +		 * Filter dead folios by moving them from the add batch to the temp
>> +		 * batch for freeing after this loop.
>> +		 *
>> +		 * Since the folio may be part of a huge page, unqueue from
>> +		 * deferred split list to avoid a dangling list entry.
>> +		 */
>> +		if (is_lru_add && folio_ref_freeze(folio, 1)) {
>> +			folio_unqueue_deferred_split(folio);
> 
> Would it be better to do this outside the lru lock; it's just that we
> don't have a convenient batched version to do it?  It seems like
> there are a few places that could use a batched version in vmscan.c and
> swap.c.  Not that I think we should hold up this patch to investigate
> that micro-optimisation!  Just something you couldlook at as a
> follow-up.

Good call. I'll leave this patch as-is (unless other feedback), then
pursue the batched version of unqueuing the split in a separate
follow-up patch.

> 
> Reviewed-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Re: [PATCH] mm/lruvec: preemptively free dead folios during lru_add drain

[Shakeel Butt]

On Thu, Apr 23, 2026 at 09:43:07AM -0700, JP Kobryn (Meta) wrote:
> Of all observable lruvec lock contention in our fleet, we find that ~24%
> occurs when dead folios are present in lru_add batches at drain time.

So, when they were added to the percpu lru cache, they were alive but during
their stay in lru cache, they were freed (last non-lrucache ref dropped) or
somehow we are adding folio where the caller drops the reference just after
adding to percpu lru cache e.g. folio_putback_lru() ?

> This
> is wasteful in the sense that the folio is added to the LRU just to be
> immediately removed via folios_put_refs(), incurring two unnecessary lock
> acquisitions.
> 
> Eliminate this overhead by preemptively cleaning up dead folios before they
> make it into the LRU. Use folio_ref_freeze() to filter folios whose only
> remaining refcount is the batch ref. When dead folios are found, move them
> off the add batch and onto a temporary batch to be freed.
> 
> During A/B testing on one of our prod instagram workloads (high-frequency
> short-lived requests), the patch intercepted almost all dead folios before
> they entered the LRU. Data collected using the mm_lru_insertion tracepoint
> shows the effectiveness of the patch:
> 
> Per-host LRU add averages at 95% CPU load
> (60 hosts each side, 3 x 60s intervals)
> 
>             dead folios/min  total folios/min   dead %
> unpatched:        1,297,785        19,341,986  6.7097%
> patched:                 14        19,039,996  0.0001%
> 
> Within this workload, we save ~2.6M lock acquisitions per minute per host
> as a result.
> 
> System-wide memory stats improved on the patched side also at 95% CPU load:
>  - direct reclaim scanning reduced 7%
>  - allocation stalls reduced 5.2%
>  - compaction stalls reduced 12.3%
>  - page frees reduced 4.9%
> 
> No regressions were observed in requests served per second or request tail
> latency (p99). Both metrics showed directional improvement at higher CPU
> utilization (comparing 85% to 95%).
> 
> Signed-off-by: JP Kobryn (Meta) <jp.kobryn@linux.dev>

Overall the code looks good but I do wonder if we can add something similar to
folio_add_lru() and if that would be enough. 

Re: [PATCH] mm/lruvec: preemptively free dead folios during lru_add drain

[JP Kobryn (Meta)]

On 4/23/26 11:46 AM, Shakeel Butt wrote:
> On Thu, Apr 23, 2026 at 09:43:07AM -0700, JP Kobryn (Meta) wrote:
>> Of all observable lruvec lock contention in our fleet, we find that ~24%
>> occurs when dead folios are present in lru_add batches at drain time.
> 
> So, when they were added to the percpu lru cache, they were alive but during
> their stay in lru cache, they were freed (last non-lrucache ref dropped) or
> somehow we are adding folio where the caller drops the reference just after
> adding to percpu lru cache e.g. folio_putback_lru() ?

Both scenarios can occur. Whether all callers put the folio while it is
on the per-cpu batch or putback drops ref from 2 to 1, the batch ref
is what remains.

[...]
> 
> Overall the code looks good but I do wonder if we can add something similar to
> folio_add_lru() and if that would be enough.

folio_add_lru() is how it gets onto the batch. But it's still alive at
that point - at least one caller ref.

Re: [PATCH] mm/lruvec: preemptively free dead folios during lru_add drain

[Shakeel Butt]

On Thu, Apr 23, 2026 at 02:18:16PM -0700, JP Kobryn (Meta) wrote:
> On 4/23/26 11:46 AM, Shakeel Butt wrote:
> > On Thu, Apr 23, 2026 at 09:43:07AM -0700, JP Kobryn (Meta) wrote:
> > > Of all observable lruvec lock contention in our fleet, we find that ~24%
> > > occurs when dead folios are present in lru_add batches at drain time.
> > 
> > So, when they were added to the percpu lru cache, they were alive but during
> > their stay in lru cache, they were freed (last non-lrucache ref dropped) or
> > somehow we are adding folio where the caller drops the reference just after
> > adding to percpu lru cache e.g. folio_putback_lru() ?
> 
> Both scenarios can occur. Whether all callers put the folio while it is
> on the per-cpu batch or putback drops ref from 2 to 1, the batch ref
> is what remains.

I was wondering which one is dominant.

> 
> [...]
> > 
> > Overall the code looks good but I do wonder if we can add something similar to
> > folio_add_lru() and if that would be enough.
> 
> folio_add_lru() is how it gets onto the batch. But it's still alive at
> that point - at least one caller ref.

Yeah, I was thinking if we do special checking at folio_putback_lru() to avoid
folio_add_lru() at all (only if folio_putback_lru() is the one causing such
scenario most of the time).

Anyways, that analysis can be done later. So, for the path:

Acked-by: Shakeel Butt <shakeel.butt@linux.dev>

Re: [PATCH] mm/lruvec: preemptively free dead folios during lru_add drain

[Barry Song]

On Fri, Apr 24, 2026 at 12:43 AM JP Kobryn (Meta) <jp.kobryn@linux.dev> wrote:
>
> Of all observable lruvec lock contention in our fleet, we find that ~24%
> occurs when dead folios are present in lru_add batches at drain time. This
> is wasteful in the sense that the folio is added to the LRU just to be
> immediately removed via folios_put_refs(), incurring two unnecessary lock
> acquisitions.
>
> Eliminate this overhead by preemptively cleaning up dead folios before they
> make it into the LRU. Use folio_ref_freeze() to filter folios whose only
> remaining refcount is the batch ref. When dead folios are found, move them
> off the add batch and onto a temporary batch to be freed.
>
> During A/B testing on one of our prod instagram workloads (high-frequency
> short-lived requests), the patch intercepted almost all dead folios before
> they entered the LRU. Data collected using the mm_lru_insertion tracepoint
> shows the effectiveness of the patch:
>
> Per-host LRU add averages at 95% CPU load
> (60 hosts each side, 3 x 60s intervals)
>
>             dead folios/min  total folios/min   dead %
> unpatched:        1,297,785        19,341,986  6.7097%
> patched:                 14        19,039,996  0.0001%
>
> Within this workload, we save ~2.6M lock acquisitions per minute per host
> as a result.
>
> System-wide memory stats improved on the patched side also at 95% CPU load:
>  - direct reclaim scanning reduced 7%
>  - allocation stalls reduced 5.2%
>  - compaction stalls reduced 12.3%
>  - page frees reduced 4.9%
>
> No regressions were observed in requests served per second or request tail
> latency (p99). Both metrics showed directional improvement at higher CPU
> utilization (comparing 85% to 95%).
>
> Signed-off-by: JP Kobryn (Meta) <jp.kobryn@linux.dev>

Hi JP,
I’m seeing a large number of "BAD page" bugs.
Not sure if it’s related, but reverting this patch
seems to fix the issue.

[ 2869.365978] BUG: Bad page state in process uname  pfn:3a5417
[ 2869.365981] page: refcount:0 mapcount:0 mapping:0000000000000000
index:0x724884c20 pfn:0x3a5417
[ 2869.365983] flags:
0x17ffffc0020908(uptodate|active|owner_2|swapbacked|node=0|zone=2|lastcpupid=0x1fffff)
[ 2869.365985] raw: 0017ffffc0020908 0000000000000000 dead000000000122
0000000000000000
[ 2869.365986] raw: 0000000724884c20 0000000000000000 00000000ffffffff
0000000000000000
[ 2869.365986] page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) set
[ 2869.366037] CPU: 11 UID: 0 PID: 179413 Comm: uname Tainted: G S  B
             7.0.0mglruswappiness+ #401 PREEMPT(full)
[ 2869.366039] Tainted: [S]=CPU_OUT_OF_SPEC, [B]=BAD_PAGE
[ 2869.366039] Hardware name: To Be Filled By O.E.M. To Be Filled By
O.E.M./H470D4-P1, BIOS P2.00 04/01/2021
[ 2869.366040] Call Trace:
[ 2869.366041]  <TASK>
[ 2869.366041]  dump_stack_lvl+0x76/0xa0
[ 2869.366043]  dump_stack+0x10/0x20
[ 2869.366044]  bad_page+0x79/0x120
[ 2869.366046]  free_unref_folios+0x85b/0x960
[ 2869.366048]  ? __pfx_lru_add+0x10/0x10
[ 2869.366049]  folio_batch_move_lru+0x260/0x2d0
[ 2869.366051]  __folio_batch_add_and_move+0x7f/0x110
[ 2869.366052]  folio_add_lru+0x49/0x60
[ 2869.366053]  folio_add_lru_vma+0x7b/0xb0
[ 2869.366054]  map_anon_folio_pte_nopf+0xb8/0x170
[ 2869.366055]  do_anonymous_page+0x5fa/0x960
[ 2869.366056]  ? __pte_offset_map+0x1c/0x140
[ 2869.366058]  __handle_mm_fault+0xbb6/0x1010
[ 2869.366059]  ? mt_find+0xe8/0x560
[ 2869.366061]  handle_mm_fault+0x1b0/0x370
[ 2869.366063]  do_user_addr_fault+0x2c8/0x870
[ 2869.366064]  exc_page_fault+0x7d/0x1d0
[ 2869.366065]  asm_exc_page_fault+0x27/0x30
[ 2869.366066] RIP: 0010:rep_movs_alternative+0x11/0x90
[ 2869.366068] Code: c3 cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90
90 90 90 90 90 90 90 90 90 48 83 f9 40 73 44 83 f9 08 73 25 85 c9 74
0f 8a 06 <88> 07 48 ff c7 48 ff c6 48 ff c9 75 f1 c3 cc cc cc cc 66 66
2e 0f
[ 2869.366069] RSP: 0018:ffffccb46a837a78 EFLAGS: 00050202
[ 2869.366070] RAX: 0000000000000078 RBX: 00007ffee44638d0 RCX: 0000000000000007
[ 2869.366071] RDX: 0000000000000000 RSI: ffffffff8d0d1d90 RDI: 00007ffee44638c9
[ 2869.366072] RBP: ffffccb46a837a80 R08: 00007ffffffff000 R09: 0000000000000000
[ 2869.366072] R10: ffff8c7769873600 R11: 0000000000000000 R12: ffff8c7786563800
[ 2869.366073] R13: ffff8c76c6668480 R14: ffff8c76cfe3ba40 R15: ffff8c770f419c00
[ 2869.366074]  ? arch_align_stack+0x3f/0x60
[ 2869.366076]  ? _copy_to_user+0x31/0x60
[ 2869.366078]  load_elf_binary+0xda1/0x1800
[ 2869.366080]  bprm_execve+0x2d7/0x590
[ 2869.366083]  do_execveat_common.isra.0+0x15c/0x1c0
[ 2869.366084]  __x64_sys_execve+0x3e/0x70
[ 2869.366085]  x64_sys_call+0xd91/0x26e0
[ 2869.366087]  do_syscall_64+0xe7/0x560
[ 2869.366088]  ? folio_add_lru+0x49/0x60
[ 2869.366089]  ? folio_add_lru_vma+0x7b/0xb0
[ 2869.366090]  ? set_ptes.isra.0+0x3b/0x90
[ 2869.366091]  ? do_wp_page+0x85d/0xed0
[ 2869.366092]  ? __handle_mm_fault+0xafe/0x1010
[ 2869.366094]  ? do_sigaction+0x165/0x4c0
[ 2869.366096]  ? _copy_to_user+0x31/0x60
[ 2869.366098]  ? __x64_sys_rt_sigaction+0xb7/0x110
[ 2869.366100]  ? x64_sys_call+0x880/0x26e0
[ 2869.366101]  ? do_syscall_64+0x11b/0x560
[ 2869.366102]  ? do_syscall_64+0x9c/0x560
[ 2869.366103]  ? clear_bhb_loop+0x30/0x80
[ 2869.366104]  entry_SYSCALL_64_after_hwframe+0x76/0x7e

Thanks
Barry

Re: [PATCH] mm/lruvec: preemptively free dead folios during lru_add drain

[Shakeel Butt]

On Fri, Apr 24, 2026 at 07:22:30AM +0800, Barry Song wrote:
> On Fri, Apr 24, 2026 at 12:43 AM JP Kobryn (Meta) <jp.kobryn@linux.dev> wrote:
> >
> > Of all observable lruvec lock contention in our fleet, we find that ~24%
> > occurs when dead folios are present in lru_add batches at drain time. This
> > is wasteful in the sense that the folio is added to the LRU just to be
> > immediately removed via folios_put_refs(), incurring two unnecessary lock
> > acquisitions.
> >
> > Eliminate this overhead by preemptively cleaning up dead folios before they
> > make it into the LRU. Use folio_ref_freeze() to filter folios whose only
> > remaining refcount is the batch ref. When dead folios are found, move them
> > off the add batch and onto a temporary batch to be freed.
> >
> > During A/B testing on one of our prod instagram workloads (high-frequency
> > short-lived requests), the patch intercepted almost all dead folios before
> > they entered the LRU. Data collected using the mm_lru_insertion tracepoint
> > shows the effectiveness of the patch:
> >
> > Per-host LRU add averages at 95% CPU load
> > (60 hosts each side, 3 x 60s intervals)
> >
> >             dead folios/min  total folios/min   dead %
> > unpatched:        1,297,785        19,341,986  6.7097%
> > patched:                 14        19,039,996  0.0001%
> >
> > Within this workload, we save ~2.6M lock acquisitions per minute per host
> > as a result.
> >
> > System-wide memory stats improved on the patched side also at 95% CPU load:
> >  - direct reclaim scanning reduced 7%
> >  - allocation stalls reduced 5.2%
> >  - compaction stalls reduced 12.3%
> >  - page frees reduced 4.9%
> >
> > No regressions were observed in requests served per second or request tail
> > latency (p99). Both metrics showed directional improvement at higher CPU
> > utilization (comparing 85% to 95%).
> >
> > Signed-off-by: JP Kobryn (Meta) <jp.kobryn@linux.dev>
> 
> Hi JP,
> I’m seeing a large number of "BAD page" bugs.
> Not sure if it’s related, but reverting this patch
> seems to fix the issue.
> 
> [ 2869.365978] BUG: Bad page state in process uname  pfn:3a5417
> [ 2869.365981] page: refcount:0 mapcount:0 mapping:0000000000000000
> index:0x724884c20 pfn:0x3a5417
> [ 2869.365983] flags:
> 0x17ffffc0020908(uptodate|active|owner_2|swapbacked|node=0|zone=2|lastcpupid=0x1fffff)

Hi Barry, are you using MGLRU? It seems like MGLRU set active flag in
folio_add_lru().

JP, we need to clean active flag but let's check what else can be set before
folio_add_lru().

Re: [PATCH] mm/lruvec: preemptively free dead folios during lru_add drain

[Barry Song]

On Fri, Apr 24, 2026 at 7:46 AM Shakeel Butt <shakeel.butt@linux.dev> wrote:
>
> On Fri, Apr 24, 2026 at 07:22:30AM +0800, Barry Song wrote:
> > On Fri, Apr 24, 2026 at 12:43 AM JP Kobryn (Meta) <jp.kobryn@linux.dev> wrote:
> > >
> > > Of all observable lruvec lock contention in our fleet, we find that ~24%
> > > occurs when dead folios are present in lru_add batches at drain time. This
> > > is wasteful in the sense that the folio is added to the LRU just to be
> > > immediately removed via folios_put_refs(), incurring two unnecessary lock
> > > acquisitions.
> > >
> > > Eliminate this overhead by preemptively cleaning up dead folios before they
> > > make it into the LRU. Use folio_ref_freeze() to filter folios whose only
> > > remaining refcount is the batch ref. When dead folios are found, move them
> > > off the add batch and onto a temporary batch to be freed.
> > >
> > > During A/B testing on one of our prod instagram workloads (high-frequency
> > > short-lived requests), the patch intercepted almost all dead folios before
> > > they entered the LRU. Data collected using the mm_lru_insertion tracepoint
> > > shows the effectiveness of the patch:
> > >
> > > Per-host LRU add averages at 95% CPU load
> > > (60 hosts each side, 3 x 60s intervals)
> > >
> > >             dead folios/min  total folios/min   dead %
> > > unpatched:        1,297,785        19,341,986  6.7097%
> > > patched:                 14        19,039,996  0.0001%
> > >
> > > Within this workload, we save ~2.6M lock acquisitions per minute per host
> > > as a result.
> > >
> > > System-wide memory stats improved on the patched side also at 95% CPU load:
> > >  - direct reclaim scanning reduced 7%
> > >  - allocation stalls reduced 5.2%
> > >  - compaction stalls reduced 12.3%
> > >  - page frees reduced 4.9%
> > >
> > > No regressions were observed in requests served per second or request tail
> > > latency (p99). Both metrics showed directional improvement at higher CPU
> > > utilization (comparing 85% to 95%).
> > >
> > > Signed-off-by: JP Kobryn (Meta) <jp.kobryn@linux.dev>
> >
> > Hi JP,
> > I’m seeing a large number of "BAD page" bugs.
> > Not sure if it’s related, but reverting this patch
> > seems to fix the issue.
> >
> > [ 2869.365978] BUG: Bad page state in process uname  pfn:3a5417
> > [ 2869.365981] page: refcount:0 mapcount:0 mapping:0000000000000000
> > index:0x724884c20 pfn:0x3a5417
> > [ 2869.365983] flags:
> > 0x17ffffc0020908(uptodate|active|owner_2|swapbacked|node=0|zone=2|lastcpupid=0x1fffff)
>
> Hi Barry, are you using MGLRU? It seems like MGLRU set active flag in
> folio_add_lru().

Yes. If you are referring to this set_active, I think it is
incorrect, so I have fixed it here and am waiting for review:

https://lore.kernel.org/linux-mm/20260418120233.7162-1-baohua@kernel.org/

>
> JP, we need to clean active flag but let's check what else can be set before
> folio_add_lru().

Best Regards
Barry

Re: [PATCH] mm/lruvec: preemptively free dead folios during lru_add drain

[JP Kobryn (Meta)]

On 4/23/26 4:53 PM, Barry Song wrote:
> On Fri, Apr 24, 2026 at 7:46 AM Shakeel Butt <shakeel.butt@linux.dev> wrote:
>>
>> On Fri, Apr 24, 2026 at 07:22:30AM +0800, Barry Song wrote:
>>> On Fri, Apr 24, 2026 at 12:43 AM JP Kobryn (Meta) <jp.kobryn@linux.dev> wrote:
>>>>
>>>> Of all observable lruvec lock contention in our fleet, we find that ~24%
>>>> occurs when dead folios are present in lru_add batches at drain time. This
>>>> is wasteful in the sense that the folio is added to the LRU just to be
>>>> immediately removed via folios_put_refs(), incurring two unnecessary lock
>>>> acquisitions.
>>>>
>>>> Eliminate this overhead by preemptively cleaning up dead folios before they
>>>> make it into the LRU. Use folio_ref_freeze() to filter folios whose only
>>>> remaining refcount is the batch ref. When dead folios are found, move them
>>>> off the add batch and onto a temporary batch to be freed.
>>>>
>>>> During A/B testing on one of our prod instagram workloads (high-frequency
>>>> short-lived requests), the patch intercepted almost all dead folios before
>>>> they entered the LRU. Data collected using the mm_lru_insertion tracepoint
>>>> shows the effectiveness of the patch:
>>>>
>>>> Per-host LRU add averages at 95% CPU load
>>>> (60 hosts each side, 3 x 60s intervals)
>>>>
>>>>              dead folios/min  total folios/min   dead %
>>>> unpatched:        1,297,785        19,341,986  6.7097%
>>>> patched:                 14        19,039,996  0.0001%
>>>>
>>>> Within this workload, we save ~2.6M lock acquisitions per minute per host
>>>> as a result.
>>>>
>>>> System-wide memory stats improved on the patched side also at 95% CPU load:
>>>>   - direct reclaim scanning reduced 7%
>>>>   - allocation stalls reduced 5.2%
>>>>   - compaction stalls reduced 12.3%
>>>>   - page frees reduced 4.9%
>>>>
>>>> No regressions were observed in requests served per second or request tail
>>>> latency (p99). Both metrics showed directional improvement at higher CPU
>>>> utilization (comparing 85% to 95%).
>>>>
>>>> Signed-off-by: JP Kobryn (Meta) <jp.kobryn@linux.dev>
>>>
>>> Hi JP,
>>> I’m seeing a large number of "BAD page" bugs.
>>> Not sure if it’s related, but reverting this patch
>>> seems to fix the issue.

It seems this was missed since classic LRU was used in testing.

>>>
>>> [ 2869.365978] BUG: Bad page state in process uname  pfn:3a5417
>>> [ 2869.365981] page: refcount:0 mapcount:0 mapping:0000000000000000
>>> index:0x724884c20 pfn:0x3a5417
>>> [ 2869.365983] flags:
>>> 0x17ffffc0020908(uptodate|active|owner_2|swapbacked|node=0|zone=2|lastcpupid=0x1fffff)
>>
>> Hi Barry, are you using MGLRU? It seems like MGLRU set active flag in
>> folio_add_lru().
> 
> Yes. If you are referring to this set_active, I think it is
> incorrect, so I have fixed it here and am waiting for review:
> 
> https://lore.kernel.org/linux-mm/20260418120233.7162-1-baohua@kernel.org/
> 
>>
>> JP, we need to clean active flag but let's check what else can be set before
>> folio_add_lru().

Looks like only active is the problem. If we start manually clearing
flags it starts to feel messy. I get that some fix is needed though. I
don't see this patch in mm-new yet so maybe we can hold off on merging
there to avoid the MGLRU case. But if Barry's patch is accepted, could
we re-apply?

Let me know if you're thinking there are any implications beyond the
active flag.