[RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressure

[RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressure

[Ehab Ababneh]

Commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to replace
folio_set_active") introduced a regression for workloads that rely on
readahead to keep sequential file access efficient.

The problem is that MGLRU can place fault-path file folios in older
generations, so memory pressure can reclaim readahead folios before the
workload touches them. In our Cassandra read benchmark, this raised p99
latency to about 9.2-9.5 ms and cut throughput to roughly 41.8k-43.6k
op/s; the revert restored the workload to about 5.5-5.6 ms and
51.9k-53.1k op/s.

Readahead is important for sequential I/O and mmap scans, but it should
not be retained when the workload does not benefit from it. The goal is
to keep the optimization without keeping readahead pages alive forever.

This patch provides a middle ground: keep the original behavior by
default, but temporarily protect fault-path file folios when repeated
file refaults show that readahead is actually helping.

The mechanism is dynamic and self-tuning:

- add a per-lruvec readahead/refault credit
- accumulate credit on file refaults in the MGLRU refault path
- consume credit in folio_add_lru() for fault-path file folios
- keep the folio active while credit is available, and otherwise let the
  original behavior stand
- decay/reset the credit as generations advance and when an lruvec is
  initialized

This means we only protect fault-path file folios when refault pressure
shows that the workload is actively benefiting from readahead. If the
workload does not need that protection, the original optimization
remains intact and we do not keep readahead pages around unnecessarily.

Benchmark results for the Cassandra read workload
(4 nodes, 720s, 100 readers):

- with commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to
  replace folio_set_active"):
  p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/s
- with revert of commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to
  replace folio_set_active"):
  p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
- with this fix: p99 ~5.8 ms, throughput ~51.9k-52.7k op/s

The fix restores the readahead protection lost by the regression while
preserving the original intent of the optimization: do not keep
readahead pages around if the workload does not need them.

Fixes: 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to replace folio_set_active")
Signed-off-by: Ehab Ababneh <ehab.ababneh@intel.com>
---
 include/linux/mmzone.h |  2 ++
 mm/swap.c              | 82 ++++++++++++++++++++++++++++++++++++++----
 mm/vmscan.c            |  7 ++++
 mm/workingset.c        | 18 ++++++++++
 4 files changed, 102 insertions(+), 7 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index ca2712187147..c998b1e0b8a7 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -578,6 +578,8 @@ struct lru_gen_folio {
 	/* can be modified without holding the LRU lock */
 	atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
 	atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
+	/* credit: file refaults indicate fault-path file folios need protection */
+	atomic_long_t ra_refaults;
 	/* whether the multi-gen LRU is enabled */
 	bool enabled;
 	/* the memcg generation this lru_gen_folio belongs to */
diff --git a/mm/swap.c b/mm/swap.c
index 588f50d8f1a8..a31c9000868a 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -70,6 +70,70 @@ static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches) = {
 	.lock_irq = INIT_LOCAL_LOCK(lock_irq),
 };
 
+#ifdef CONFIG_LRU_GEN
+/* Refill two default readahead windows to amortize shared-counter updates. */
+#define RA_REFAULT_LOCAL_BATCH	(VM_READAHEAD_PAGES * 2)
+
+struct ra_credit_cache {
+	/* Batch shared credit per CPU to avoid a contended atomic RMW per folio. */
+	/* only compared for identity, never dereferenced */
+	struct lru_gen_folio *lrugen;
+	long credit;
+};
+
+static DEFINE_PER_CPU(struct ra_credit_cache, ra_credit_cache);
+
+/*
+ * Spend readahead protection credit from a per-CPU bucket, refilled in batches
+ * from the shared per-lruvec counter, so the fault path avoids an atomic RMW on
+ * a contended cacheline for every folio.
+ */
+static bool lru_gen_take_ra_credit(struct folio *folio)
+{
+	struct lru_gen_folio *lrugen;
+	long nr_pages = folio_nr_pages(folio);
+	struct ra_credit_cache *cache;
+	bool taken = false;
+	long old, new;
+
+	rcu_read_lock();
+	lrugen = &folio_lruvec(folio)->lrugen;
+	cache = get_cpu_ptr(&ra_credit_cache);
+
+	/* credit cached for a different lruvec is forfeited, bounded by the batch */
+	if (cache->lrugen != lrugen) {
+		cache->lrugen = lrugen;
+		cache->credit = 0;
+	}
+
+	if (cache->credit < nr_pages) {
+		old = atomic_long_read(&lrugen->ra_refaults);
+		while (old > 0) {
+			new = old - min_t(long, old, RA_REFAULT_LOCAL_BATCH);
+			if (atomic_long_try_cmpxchg(&lrugen->ra_refaults, &old, new)) {
+				cache->credit += old - new;
+				break;
+			}
+		}
+	}
+
+	if (cache->credit >= nr_pages) {
+		cache->credit -= nr_pages;
+		taken = true;
+	}
+
+	put_cpu_ptr(&ra_credit_cache);
+	rcu_read_unlock();
+
+	return taken;
+}
+#else
+static bool lru_gen_take_ra_credit(struct folio *folio)
+{
+	return false;
+}
+#endif /* CONFIG_LRU_GEN */
+
 static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp,
 		unsigned long *flagsp)
 {
@@ -545,18 +609,22 @@ void folio_add_lru(struct folio *folio)
 	VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
 
 	/*
-	 * For refaulted workingset folios, set PG_active so they
-	 * can be added to active generations.
-	 * For prefaulted file folios, folio_mark_accessed() sets
-	 * PG_referenced so lru_gen_folio_seq() places them into
-	 * the second oldest generation.
+	 * For refaulted workingset folios, set PG_active so they can be added to
+	 * active generations. For file folios in the fault path, consume refault
+	 * credit to temporarily protect folios that are likely useful readahead.
 	 */
 	if (lru_gen_enabled() && !folio_test_unevictable(folio) &&
 	    lru_gen_in_fault() && !(current->flags & PF_MEMALLOC)) {
-		if (folio_test_workingset(folio))
+		if (folio_test_workingset(folio)) {
 			folio_set_active(folio);
-		else if (!folio_test_referenced(folio))
+		} else if (folio_is_file_lru(folio)) {
+			if (lru_gen_take_ra_credit(folio))
+				folio_set_active(folio);
+			else if (!folio_test_referenced(folio))
+				folio_mark_accessed(folio);
+		} else if (!folio_test_referenced(folio)) {
 			folio_mark_accessed(folio);
+		}
 	}
 
 	folio_batch_add_and_move(folio, lru_add);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 56708d1d2dfd..767311593296 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3931,6 +3931,7 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness
 	bool success;
 	int prev, next;
 	int type, zone;
+	long old, new;
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
 restart:
 	if (seq < READ_ONCE(lrugen->max_seq))
@@ -3983,6 +3984,11 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness
 		reset_ctrl_pos(lruvec, type, false);
 
 	WRITE_ONCE(lrugen->timestamps[next], jiffies);
+	/* decay readahead protection credit so stale signal doesn't persist */
+	old = atomic_long_read(&lrugen->ra_refaults);
+	do {
+		new = (old * 3) / 4;
+	} while (!atomic_long_try_cmpxchg(&lrugen->ra_refaults, &old, new));
 	/* make sure preceding modifications appear */
 	smp_store_release(&lrugen->max_seq, lrugen->max_seq + 1);
 unlock:
@@ -5784,6 +5790,7 @@ void lru_gen_init_lruvec(struct lruvec *lruvec)
 
 	lrugen->max_seq = MIN_NR_GENS + 1;
 	lrugen->enabled = lru_gen_enabled();
+	atomic_long_set(&lrugen->ra_refaults, 0);
 
 	for (i = 0; i <= MIN_NR_GENS + 1; i++)
 		lrugen->timestamps[i] = jiffies;
diff --git a/mm/workingset.c b/mm/workingset.c
index f351798e723a..63baa7220136 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -319,6 +319,24 @@ static void lru_gen_refault(struct folio *folio, void *shadow)
 
 	atomic_long_add(delta, &lrugen->refaulted[hist][type][tier]);
 
+	if (type == LRU_GEN_FILE) {
+		/*
+		 * Cap credit at total file pages to avoid runaway while
+		 * allowing sustained protection.
+		 */
+		long cap = lruvec_page_state(lruvec, NR_LRU_BASE + LRU_INACTIVE_FILE) +
+			lruvec_page_state(lruvec, NR_LRU_BASE + LRU_ACTIVE_FILE);
+		long add = (long)delta * VM_READAHEAD_PAGES;
+		long old, new;
+
+		old = atomic_long_read(&lrugen->ra_refaults);
+		do {
+			if (old >= cap)
+				break;
+			new = min(cap, old + add);
+		} while (!atomic_long_try_cmpxchg(&lrugen->ra_refaults, &old, new));
+	}
+
 	if (workingset) {
 		/*
 		 * see folio_add_lru(), where folio_set_active() is
-- 
2.43.0

Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressure

[Kairui Song]

On Tue, Sep 01, 2026 at 11:06:43AM +0800, Ehab Ababneh wrote:
> Commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to replace
> folio_set_active") introduced a regression for workloads that rely on
> readahead to keep sequential file access efficient.
> 
> The problem is that MGLRU can place fault-path file folios in older
> generations, so memory pressure can reclaim readahead folios before the
> workload touches them. In our Cassandra read benchmark, this raised p99
> latency to about 9.2-9.5 ms and cut throughput to roughly 41.8k-43.6k
> op/s; the revert restored the workload to about 5.5-5.6 ms and
> 51.9k-53.1k op/s.
> 
> Readahead is important for sequential I/O and mmap scans, but it should
> not be retained when the workload does not benefit from it. The goal is
> to keep the optimization without keeping readahead pages alive forever.
> 
> This patch provides a middle ground: keep the original behavior by
> default, but temporarily protect fault-path file folios when repeated
> file refaults show that readahead is actually helping.
> 
> The mechanism is dynamic and self-tuning:
> 
> - add a per-lruvec readahead/refault credit
> - accumulate credit on file refaults in the MGLRU refault path
> - consume credit in folio_add_lru() for fault-path file folios
> - keep the folio active while credit is available, and otherwise let the
>   original behavior stand
> - decay/reset the credit as generations advance and when an lruvec is
>   initialized
> 
> This means we only protect fault-path file folios when refault pressure
> shows that the workload is actively benefiting from readahead. If the
> workload does not need that protection, the original optimization
> remains intact and we do not keep readahead pages around unnecessarily.
> 
> Benchmark results for the Cassandra read workload
> (4 nodes, 720s, 100 readers):
> 
> - with commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to
>   replace folio_set_active"):
>   p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/s
> - with revert of commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to
>   replace folio_set_active"):
>   p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
> - with this fix: p99 ~5.8 ms, throughput ~51.9k-52.7k op/s

Hello Ehab,

We ran into the same issue on our side too. I hesitated to report or fix
that as I'm working on MGLRU-FG which fixed the problem on my side:

https://lore.kernel.org/linux-mm/20260804-mglru-fg-v1-0-4d8dad39dad6@tencent.com/

I especially mentioned it, see the parts after:
"recent change in lru_gen_folio_seq that bumps new folios with refs == 1"

Latest version still being tested which you can use directly:
https://github.com/ryncsn/linux/commits/b4/mglru-fg-v1.8/

Do you mind have a look of that as well? I think in the long term that is
the right direction. With our test the regression is gone and performance
is even better.

And is there any easy way to reproduce the specific case you are reporting?

> The fix restores the readahead protection lost by the regression while
> preserving the original intent of the optimization: do not keep
> readahead pages around if the workload does not need them.
> 
> Fixes: 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to replace folio_set_active")
> Signed-off-by: Ehab Ababneh <ehab.ababneh@intel.com>
> ---
>  include/linux/mmzone.h |  2 ++
>  mm/swap.c              | 82 ++++++++++++++++++++++++++++++++++++++----
>  mm/vmscan.c            |  7 ++++
>  mm/workingset.c        | 18 ++++++++++
>  4 files changed, 102 insertions(+), 7 deletions(-)
> 
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index ca2712187147..c998b1e0b8a7 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -578,6 +578,8 @@ struct lru_gen_folio {
>  	/* can be modified without holding the LRU lock */
>  	atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
>  	atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
> +	/* credit: file refaults indicate fault-path file folios need protection */
> +	atomic_long_t ra_refaults;
>  	/* whether the multi-gen LRU is enabled */
>  	bool enabled;
>  	/* the memcg generation this lru_gen_folio belongs to */
> diff --git a/mm/swap.c b/mm/swap.c
> index 588f50d8f1a8..a31c9000868a 100644
> --- a/mm/swap.c
> +++ b/mm/swap.c
> @@ -70,6 +70,70 @@ static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches) = {
>  	.lock_irq = INIT_LOCAL_LOCK(lock_irq),
>  };
>  
> +#ifdef CONFIG_LRU_GEN
> +/* Refill two default readahead windows to amortize shared-counter updates. */
> +#define RA_REFAULT_LOCAL_BATCH	(VM_READAHEAD_PAGES * 2)
> +
> +struct ra_credit_cache {
> +	/* Batch shared credit per CPU to avoid a contended atomic RMW per folio. */
> +	/* only compared for identity, never dereferenced */
> +	struct lru_gen_folio *lrugen;
> +	long credit;
> +};
> +
> +static DEFINE_PER_CPU(struct ra_credit_cache, ra_credit_cache);
> +
> +/*
> + * Spend readahead protection credit from a per-CPU bucket, refilled in batches
> + * from the shared per-lruvec counter, so the fault path avoids an atomic RMW on
> + * a contended cacheline for every folio.
> + */
> +static bool lru_gen_take_ra_credit(struct folio *folio)
> +{
> +	struct lru_gen_folio *lrugen;
> +	long nr_pages = folio_nr_pages(folio);
> +	struct ra_credit_cache *cache;
> +	bool taken = false;
> +	long old, new;
> +
> +	rcu_read_lock();
> +	lrugen = &folio_lruvec(folio)->lrugen;
> +	cache = get_cpu_ptr(&ra_credit_cache);
> +
> +	/* credit cached for a different lruvec is forfeited, bounded by the batch */
> +	if (cache->lrugen != lrugen) {
> +		cache->lrugen = lrugen;
> +		cache->credit = 0;
> +	}
> +
> +	if (cache->credit < nr_pages) {
> +		old = atomic_long_read(&lrugen->ra_refaults);
> +		while (old > 0) {
> +			new = old - min_t(long, old, RA_REFAULT_LOCAL_BATCH);
> +			if (atomic_long_try_cmpxchg(&lrugen->ra_refaults, &old, new)) {
> +				cache->credit += old - new;
> +				break;
> +			}
> +		}
> +	}
> +
> +	if (cache->credit >= nr_pages) {
> +		cache->credit -= nr_pages;
> +		taken = true;
> +	}
> +
> +	put_cpu_ptr(&ra_credit_cache);
> +	rcu_read_unlock();
> +
> +	return taken;
> +}
> +#else
> +static bool lru_gen_take_ra_credit(struct folio *folio)
> +{
> +	return false;
> +}
> +#endif /* CONFIG_LRU_GEN */
> +

Just an idea. For an short term and easy fix, what if we simply revert
than, then only protect in_fault && folio_test_swapbacked folios with
PG_active?

Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressure

[Barry Song (Xiaomi)]

On Wed, Sep 2, 2026 at 2:21 AM Kairui Song <ryncsn@gmail.com> wrote:
>
> On Tue, Sep 01, 2026 at 11:06:43AM +0800, Ehab Ababneh wrote:
> > Commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to replace
> > folio_set_active") introduced a regression for workloads that rely on
> > readahead to keep sequential file access efficient.
> >
> > The problem is that MGLRU can place fault-path file folios in older
> > generations, so memory pressure can reclaim readahead folios before the
> > workload touches them. In our Cassandra read benchmark, this raised p99
> > latency to about 9.2-9.5 ms and cut throughput to roughly 41.8k-43.6k
> > op/s; the revert restored the workload to about 5.5-5.6 ms and
> > 51.9k-53.1k op/s.
> >
> > Readahead is important for sequential I/O and mmap scans, but it should
> > not be retained when the workload does not benefit from it. The goal is
> > to keep the optimization without keeping readahead pages alive forever.
> >
> > This patch provides a middle ground: keep the original behavior by
> > default, but temporarily protect fault-path file folios when repeated
> > file refaults show that readahead is actually helping.
> >
> > The mechanism is dynamic and self-tuning:
> >
> > - add a per-lruvec readahead/refault credit
> > - accumulate credit on file refaults in the MGLRU refault path
> > - consume credit in folio_add_lru() for fault-path file folios
> > - keep the folio active while credit is available, and otherwise let the
> >   original behavior stand
> > - decay/reset the credit as generations advance and when an lruvec is
> >   initialized
> >
> > This means we only protect fault-path file folios when refault pressure
> > shows that the workload is actively benefiting from readahead. If the
> > workload does not need that protection, the original optimization
> > remains intact and we do not keep readahead pages around unnecessarily.
> >
> > Benchmark results for the Cassandra read workload
> > (4 nodes, 720s, 100 readers):
> >
> > - with commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to
> >   replace folio_set_active"):
> >   p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/s
> > - with revert of commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to
> >   replace folio_set_active"):
> >   p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
> > - with this fix: p99 ~5.8 ms, throughput ~51.9k-52.7k op/s
>
[...]
>
> Just an idea. For an short term and easy fix, what if we simply revert
> than, then only protect in_fault && folio_test_swapbacked folios with
> PG_active?

Hi Ehab and Kairui,

Thanks very much for your reports and discussion.

I'm not quite sure we want to revert it entirely, as that would
immediately regress the workloads improved by the commit, such as the
kernel build.
Also, for example, Kairui's cover letter mentioned the LevelDB benchmark:
 "I also retested the LevelDB benchmark from the cache_ext paper [5].
Interestingly, mainline MGLRU already beats CLRU on this one after a
recent change in lru_gen_folio_seq that bumps new folios with refs == 1
to the second-oldest generation."

I guess we could instead try to mitigate the cases where some workloads
are negatively affected while preserving the original intention. Does the
fix below address both of your cases, or is Ehab's case actually different
from Kairui's?
(The kernel-build test on my machine looks quite positive. It not only
preserves the original optimization, but even provides further gains.)

diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 621c8653d8f7..91396c796a34 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -239,7 +239,8 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
 	 * |<---------------------------- MAX_NR_GENS ---------------------------->|
 	 */
 	if (folio_test_active(folio))
-		gen = MIN_NR_GENS - folio_test_workingset(folio);
+		gen = MIN_NR_GENS - folio_test_workingset(folio) +
+		      (type ? !folio_test_workingset(folio) : 0);
 	else if (reclaiming)
 		gen = MAX_NR_GENS;
 	else if ((!folio_is_file_lru(folio) && !folio_test_swapcache(folio)) ||
@@ -247,7 +248,7 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
 		  (folio_test_dirty(folio) || folio_test_writeback(folio))))
 		gen = MIN_NR_GENS;
 	else
-		gen = MAX_NR_GENS - (folio_test_workingset(folio) || folio_test_referenced(folio));
+		gen = MAX_NR_GENS - folio_test_workingset(folio);
 
 	return max(READ_ONCE(lrugen->max_seq) - gen + 1, READ_ONCE(lrugen->min_seq[type]));
 }
diff --git a/mm/folio.c b/mm/folio.c
index c02dcea9c03c..2fd835b3b50c 100644
--- a/mm/folio.c
+++ b/mm/folio.c
@@ -470,20 +470,10 @@ void folio_add_lru(struct folio *folio)
 			folio_test_unevictable(folio), folio);
 	VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
 
-	/*
-	 * For refaulted workingset folios, set PG_active so they
-	 * can be added to active generations.
-	 * For prefaulted file folios, folio_mark_accessed() sets
-	 * PG_referenced so lru_gen_folio_seq() places them into
-	 * the second oldest generation.
-	 */
+	/* see the comment in lru_gen_folio_seq() */
 	if (lru_gen_enabled() && !folio_test_unevictable(folio) &&
-	    lru_gen_in_fault() && !(current->flags & PF_MEMALLOC)) {
-		if (folio_test_workingset(folio))
-			folio_set_active(folio);
-		else if (!folio_test_referenced(folio))
-			folio_mark_accessed(folio);
-	}
+	    lru_gen_in_fault() && !(current->flags & PF_MEMALLOC))
+		folio_set_active(folio);
 
 	folio_batch_add_and_move(folio, lru_add);
 }
diff --git a/mm/vmscan.c b/mm/vmscan.c
index f11491ee9ed5..aa500ae9371a 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -851,11 +851,7 @@ static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags)
 		return false;
 	}
 
-	/* Promote on second access */
-	if (folio_lru_refs(folio) > 1)
-		set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
-	else
-		folio_mark_accessed(folio);
+	set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
 	return true;
 }
 #else
diff --git a/mm/workingset.c b/mm/workingset.c
index 7ac2b88c80ae..a79b73ee9762 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -319,13 +319,11 @@ static void lru_gen_refault(struct folio *folio, void *shadow)
 
 	atomic_long_add(delta, &lrugen->refaulted[hist][type][tier]);
 
+	/* see folio_add_lru() where folio_set_active() will be called */
+	if (lru_gen_in_fault())
+		mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
+
 	if (workingset) {
-		/*
-		 * see folio_add_lru(), where folio_set_active() is
-		 * called for workingset folios
-		 */
-		if (lru_gen_in_fault())
-			mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
 		folio_set_workingset(folio);
 		mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + type, delta);
 	} else
-- 
2.34.1

RE: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressure

[Ababneh, Ehab]

Hi Barry, Kairui,

Thanks for the feedback and suggestions.

> -----Original Message-----
> From: Barry Song (Xiaomi) <baohua@kernel.org>
> Sent: Tuesday, September 1, 2026 3:05 PM
> To: ryncsn@gmail.com; Ababneh, Ehab <ehab.ababneh@intel.com>
> Cc: akpm@linux-foundation.org; axelrasmussen@google.com;
> baohua@kernel.org; kasong@tencent.com; lance.yang@linux.dev; linux-
> kernel@vger.kernel.org; linux-mm@kvack.org; qi.zheng@linux.dev;
> shakeel.butt@linux.dev; weixugc@google.com; yuanchu@google.com;
> yuzhao@google.com
> Subject: Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault
> folios under refault pressure
> 
> On Wed, Sep 2, 2026 at 2:21 AM Kairui Song <ryncsn@gmail.com> wrote:
> >
> > On Tue, Sep 01, 2026 at 11:06:43AM +0800, Ehab Ababneh wrote:
> > > Commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to replace
> > > folio_set_active") introduced a regression for workloads that rely
> > > on readahead to keep sequential file access efficient.
> > >
> > > The problem is that MGLRU can place fault-path file folios in older
> > > generations, so memory pressure can reclaim readahead folios before
> > > the workload touches them. In our Cassandra read benchmark, this
> > > raised p99 latency to about 9.2-9.5 ms and cut throughput to roughly
> > > 41.8k-43.6k op/s; the revert restored the workload to about 5.5-5.6
> > > ms and 51.9k-53.1k op/s.
> > >
> > > Readahead is important for sequential I/O and mmap scans, but it
> > > should not be retained when the workload does not benefit from it.
> > > The goal is to keep the optimization without keeping readahead pages alive
> forever.
> > >
> > > This patch provides a middle ground: keep the original behavior by
> > > default, but temporarily protect fault-path file folios when
> > > repeated file refaults show that readahead is actually helping.
> > >
> > > The mechanism is dynamic and self-tuning:
> > >
> > > - add a per-lruvec readahead/refault credit
> > > - accumulate credit on file refaults in the MGLRU refault path
> > > - consume credit in folio_add_lru() for fault-path file folios
> > > - keep the folio active while credit is available, and otherwise let
> > > the
> > >   original behavior stand
> > > - decay/reset the credit as generations advance and when an lruvec
> > > is
> > >   initialized
> > >
> > > This means we only protect fault-path file folios when refault
> > > pressure shows that the workload is actively benefiting from
> > > readahead. If the workload does not need that protection, the
> > > original optimization remains intact and we do not keep readahead pages
> around unnecessarily.
> > >
> > > Benchmark results for the Cassandra read workload
> > > (4 nodes, 720s, 100 readers):
> > >
> > > - with commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to
> > >   replace folio_set_active"):
> > >   p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/s
> > > - with revert of commit 6cbdd9726fb5 ("mm/mglru: use
> > > folio_mark_accessed to
> > >   replace folio_set_active"):
> > >   p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
> > > - with this fix: p99 ~5.8 ms, throughput ~51.9k-52.7k op/s
> >
> [...]
> >
> > Just an idea. For an short term and easy fix, what if we simply revert
> > than, then only protect in_fault && folio_test_swapbacked folios with
> > PG_active?
> 
> Hi Ehab and Kairui,
> 
> Thanks very much for your reports and discussion.
> 
> I'm not quite sure we want to revert it entirely, as that would immediately
> regress the workloads improved by the commit, such as the kernel build.
> Also, for example, Kairui's cover letter mentioned the LevelDB benchmark:
>  "I also retested the LevelDB benchmark from the cache_ext paper [5].
> Interestingly, mainline MGLRU already beats CLRU on this one after a recent
> change in lru_gen_folio_seq that bumps new folios with refs == 1 to the
> second-oldest generation."
> 
> I guess we could instead try to mitigate the cases where some workloads are
> negatively affected while preserving the original intention. Does the fix below
> address both of your cases, or is Ehab's case actually different from Kairui's?
> (The kernel-build test on my machine looks quite positive. It not only
> preserves the original optimization, but even provides further gains.)
> 

I agree that reverting the commit that caused the regression is not the
optimal path. I expect there are many workloads and scenarios that
benefit from the behavior introduced by that commit, so reverting it
could unnecessarily regress those workloads.

I will run the Cassandra benchmark with Kairui's MGLRU-FG patches to see
whether they address the issue I am seeing. I will send the results when
they are ready.

Thanks,
Ehab

> diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index
> 621c8653d8f7..91396c796a34 100644
> --- a/include/linux/mm_inline.h
> +++ b/include/linux/mm_inline.h
> @@ -239,7 +239,8 @@ static inline unsigned long lru_gen_folio_seq(const
> struct lruvec *lruvec,
>  	 * |<---------------------------- MAX_NR_GENS ---------------------------->|
>  	 */
>  	if (folio_test_active(folio))
> -		gen = MIN_NR_GENS - folio_test_workingset(folio);
> +		gen = MIN_NR_GENS - folio_test_workingset(folio) +
> +		      (type ? !folio_test_workingset(folio) : 0);
>  	else if (reclaiming)
>  		gen = MAX_NR_GENS;
>  	else if ((!folio_is_file_lru(folio) && !folio_test_swapcache(folio)) ||
> @@ -247,7 +248,7 @@ static inline unsigned long lru_gen_folio_seq(const
> struct lruvec *lruvec,
>  		  (folio_test_dirty(folio) || folio_test_writeback(folio))))
>  		gen = MIN_NR_GENS;
>  	else
> -		gen = MAX_NR_GENS - (folio_test_workingset(folio) ||
> folio_test_referenced(folio));
> +		gen = MAX_NR_GENS - folio_test_workingset(folio);
> 
>  	return max(READ_ONCE(lrugen->max_seq) - gen + 1,
> READ_ONCE(lrugen->min_seq[type]));
>  }
> diff --git a/mm/folio.c b/mm/folio.c
> index c02dcea9c03c..2fd835b3b50c 100644
> --- a/mm/folio.c
> +++ b/mm/folio.c
> @@ -470,20 +470,10 @@ void folio_add_lru(struct folio *folio)
>  			folio_test_unevictable(folio), folio);
>  	VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
> 
> -	/*
> -	 * For refaulted workingset folios, set PG_active so they
> -	 * can be added to active generations.
> -	 * For prefaulted file folios, folio_mark_accessed() sets
> -	 * PG_referenced so lru_gen_folio_seq() places them into
> -	 * the second oldest generation.
> -	 */
> +	/* see the comment in lru_gen_folio_seq() */
>  	if (lru_gen_enabled() && !folio_test_unevictable(folio) &&
> -	    lru_gen_in_fault() && !(current->flags & PF_MEMALLOC)) {
> -		if (folio_test_workingset(folio))
> -			folio_set_active(folio);
> -		else if (!folio_test_referenced(folio))
> -			folio_mark_accessed(folio);
> -	}
> +	    lru_gen_in_fault() && !(current->flags & PF_MEMALLOC))
> +		folio_set_active(folio);
> 
>  	folio_batch_add_and_move(folio, lru_add);  } diff --git
> a/mm/vmscan.c b/mm/vmscan.c index f11491ee9ed5..aa500ae9371a
> 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -851,11 +851,7 @@ static bool lru_gen_set_refs(struct folio *folio, const
> vma_flags_t *vma_flags)
>  		return false;
>  	}
> 
> -	/* Promote on second access */
> -	if (folio_lru_refs(folio) > 1)
> -		set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS,
> BIT(PG_workingset));
> -	else
> -		folio_mark_accessed(folio);
> +	set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
>  	return true;
>  }
>  #else
> diff --git a/mm/workingset.c b/mm/workingset.c index
> 7ac2b88c80ae..a79b73ee9762 100644
> --- a/mm/workingset.c
> +++ b/mm/workingset.c
> @@ -319,13 +319,11 @@ static void lru_gen_refault(struct folio *folio, void
> *shadow)
> 
>  	atomic_long_add(delta, &lrugen->refaulted[hist][type][tier]);
> 
> +	/* see folio_add_lru() where folio_set_active() will be called */
> +	if (lru_gen_in_fault())
> +		mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE +
> type, delta);
> +
>  	if (workingset) {
> -		/*
> -		 * see folio_add_lru(), where folio_set_active() is
> -		 * called for workingset folios
> -		 */
> -		if (lru_gen_in_fault())
> -			mod_lruvec_state(lruvec,
> WORKINGSET_ACTIVATE_BASE + type, delta);
>  		folio_set_workingset(folio);
>  		mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE +
> type, delta);
>  	} else
> --
> 2.34.1

Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressure

[Barry Song]

On Thu, Sep 3, 2026 at 5:52 AM Ababneh, Ehab <ehab.ababneh@intel.com> wrote:
>
> Hi Barry, Kairui,
>
> Thanks for the feedback and suggestions.
>
> > -----Original Message-----
> > From: Barry Song (Xiaomi) <baohua@kernel.org>
> > Sent: Tuesday, September 1, 2026 3:05 PM
> > To: ryncsn@gmail.com; Ababneh, Ehab <ehab.ababneh@intel.com>
> > Cc: akpm@linux-foundation.org; axelrasmussen@google.com;
> > baohua@kernel.org; kasong@tencent.com; lance.yang@linux.dev; linux-
> > kernel@vger.kernel.org; linux-mm@kvack.org; qi.zheng@linux.dev;
> > shakeel.butt@linux.dev; weixugc@google.com; yuanchu@google.com;
> > yuzhao@google.com
> > Subject: Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault
> > folios under refault pressure
> >
> > On Wed, Sep 2, 2026 at 2:21 AM Kairui Song <ryncsn@gmail.com> wrote:
> > >
> > > On Tue, Sep 01, 2026 at 11:06:43AM +0800, Ehab Ababneh wrote:
> > > > Commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to replace
> > > > folio_set_active") introduced a regression for workloads that rely
> > > > on readahead to keep sequential file access efficient.
> > > >
> > > > The problem is that MGLRU can place fault-path file folios in older
> > > > generations, so memory pressure can reclaim readahead folios before
> > > > the workload touches them. In our Cassandra read benchmark, this
> > > > raised p99 latency to about 9.2-9.5 ms and cut throughput to roughly
> > > > 41.8k-43.6k op/s; the revert restored the workload to about 5.5-5.6
> > > > ms and 51.9k-53.1k op/s.
> > > >
> > > > Readahead is important for sequential I/O and mmap scans, but it
> > > > should not be retained when the workload does not benefit from it.
> > > > The goal is to keep the optimization without keeping readahead pages alive
> > forever.
> > > >
> > > > This patch provides a middle ground: keep the original behavior by
> > > > default, but temporarily protect fault-path file folios when
> > > > repeated file refaults show that readahead is actually helping.
> > > >
> > > > The mechanism is dynamic and self-tuning:
> > > >
> > > > - add a per-lruvec readahead/refault credit
> > > > - accumulate credit on file refaults in the MGLRU refault path
> > > > - consume credit in folio_add_lru() for fault-path file folios
> > > > - keep the folio active while credit is available, and otherwise let
> > > > the
> > > >   original behavior stand
> > > > - decay/reset the credit as generations advance and when an lruvec
> > > > is
> > > >   initialized
> > > >
> > > > This means we only protect fault-path file folios when refault
> > > > pressure shows that the workload is actively benefiting from
> > > > readahead. If the workload does not need that protection, the
> > > > original optimization remains intact and we do not keep readahead pages
> > around unnecessarily.
> > > >
> > > > Benchmark results for the Cassandra read workload
> > > > (4 nodes, 720s, 100 readers):
> > > >
> > > > - with commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to
> > > >   replace folio_set_active"):
> > > >   p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/s
> > > > - with revert of commit 6cbdd9726fb5 ("mm/mglru: use
> > > > folio_mark_accessed to
> > > >   replace folio_set_active"):
> > > >   p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
> > > > - with this fix: p99 ~5.8 ms, throughput ~51.9k-52.7k op/s
> > >
> > [...]
> > >
> > > Just an idea. For an short term and easy fix, what if we simply revert
> > > than, then only protect in_fault && folio_test_swapbacked folios with
> > > PG_active?
> >
> > Hi Ehab and Kairui,
> >
> > Thanks very much for your reports and discussion.
> >
> > I'm not quite sure we want to revert it entirely, as that would immediately
> > regress the workloads improved by the commit, such as the kernel build.
> > Also, for example, Kairui's cover letter mentioned the LevelDB benchmark:
> >  "I also retested the LevelDB benchmark from the cache_ext paper [5].
> > Interestingly, mainline MGLRU already beats CLRU on this one after a recent
> > change in lru_gen_folio_seq that bumps new folios with refs == 1 to the
> > second-oldest generation."
> >
> > I guess we could instead try to mitigate the cases where some workloads are
> > negatively affected while preserving the original intention. Does the fix below
> > address both of your cases, or is Ehab's case actually different from Kairui's?
> > (The kernel-build test on my machine looks quite positive. It not only
> > preserves the original optimization, but even provides further gains.)
> >
>
> I agree that reverting the commit that caused the regression is not the
> optimal path. I expect there are many workloads and scenarios that
> benefit from the behavior introduced by that commit, so reverting it
> could unnecessarily regress those workloads.
>
> I will run the Cassandra benchmark with Kairui's MGLRU-FG patches to see
> whether they address the issue I am seeing. I will send the results when
> they are ready.
>

Thanks very much, Ehab. Could you please also run my proposed quick fix:

https://lore.kernel.org/linux-mm/20260901220430.79810-1-baohua@kernel.org/

Best Regards
Barry

RE: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressure

[Ababneh, Ehab]

> -----Original Message-----
> From: Barry Song <baohua@kernel.org>
> Sent: Wednesday, September 2, 2026 2:58 PM
> To: Ababneh, Ehab <ehab.ababneh@intel.com>
> Cc: ryncsn@gmail.com; akpm@linux-foundation.org;
> axelrasmussen@google.com; kasong@tencent.com; lance.yang@linux.dev;
> linux-kernel@vger.kernel.org; linux-mm@kvack.org; qi.zheng@linux.dev;
> shakeel.butt@linux.dev; weixugc@google.com; yuanchu@google.com;
> yuzhao@google.com
> Subject: Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault
> folios under refault pressure
> 
> On Thu, Sep 3, 2026 at 5:52 AM Ababneh, Ehab <ehab.ababneh@intel.com>
> wrote:
> >
> > Hi Barry, Kairui,
> >
> > Thanks for the feedback and suggestions.
> >
> > > -----Original Message-----
> > > From: Barry Song (Xiaomi) <baohua@kernel.org>
> > > Sent: Tuesday, September 1, 2026 3:05 PM
> > > To: ryncsn@gmail.com; Ababneh, Ehab <ehab.ababneh@intel.com>
> > > Cc: akpm@linux-foundation.org; axelrasmussen@google.com;
> > > baohua@kernel.org; kasong@tencent.com; lance.yang@linux.dev; linux-
> > > kernel@vger.kernel.org; linux-mm@kvack.org; qi.zheng@linux.dev;
> > > shakeel.butt@linux.dev; weixugc@google.com; yuanchu@google.com;
> > > yuzhao@google.com
> > > Subject: Re: [RFC PATCH] mm/mglru: dynamically protect readahead
> > > fault folios under refault pressure
> > >
> > > On Wed, Sep 2, 2026 at 2:21 AM Kairui Song <ryncsn@gmail.com> wrote:
> > > >
> > > > On Tue, Sep 01, 2026 at 11:06:43AM +0800, Ehab Ababneh wrote:
> > > > > Commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to
> > > > > replace
> > > > > folio_set_active") introduced a regression for workloads that
> > > > > rely on readahead to keep sequential file access efficient.
> > > > >
> > > > > The problem is that MGLRU can place fault-path file folios in
> > > > > older generations, so memory pressure can reclaim readahead
> > > > > folios before the workload touches them. In our Cassandra read
> > > > > benchmark, this raised p99 latency to about 9.2-9.5 ms and cut
> > > > > throughput to roughly 41.8k-43.6k op/s; the revert restored the
> > > > > workload to about 5.5-5.6 ms and 51.9k-53.1k op/s.
> > > > >
> > > > > Readahead is important for sequential I/O and mmap scans, but it
> > > > > should not be retained when the workload does not benefit from it.
> > > > > The goal is to keep the optimization without keeping readahead
> > > > > pages alive
> > > forever.
> > > > >
> > > > > This patch provides a middle ground: keep the original behavior
> > > > > by default, but temporarily protect fault-path file folios when
> > > > > repeated file refaults show that readahead is actually helping.
> > > > >
> > > > > The mechanism is dynamic and self-tuning:
> > > > >
> > > > > - add a per-lruvec readahead/refault credit
> > > > > - accumulate credit on file refaults in the MGLRU refault path
> > > > > - consume credit in folio_add_lru() for fault-path file folios
> > > > > - keep the folio active while credit is available, and otherwise
> > > > > let the
> > > > >   original behavior stand
> > > > > - decay/reset the credit as generations advance and when an
> > > > > lruvec is
> > > > >   initialized
> > > > >
> > > > > This means we only protect fault-path file folios when refault
> > > > > pressure shows that the workload is actively benefiting from
> > > > > readahead. If the workload does not need that protection, the
> > > > > original optimization remains intact and we do not keep
> > > > > readahead pages
> > > around unnecessarily.
> > > > >
> > > > > Benchmark results for the Cassandra read workload
> > > > > (4 nodes, 720s, 100 readers):
> > > > >
> > > > > - with commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to
> > > > >   replace folio_set_active"):
> > > > >   p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/s
> > > > > - with revert of commit 6cbdd9726fb5 ("mm/mglru: use
> > > > > folio_mark_accessed to
> > > > >   replace folio_set_active"):
> > > > >   p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
> > > > > - with this fix: p99 ~5.8 ms, throughput ~51.9k-52.7k op/s
> > > >
> > > [...]
> > > >
> > > > Just an idea. For an short term and easy fix, what if we simply
> > > > revert than, then only protect in_fault && folio_test_swapbacked
> > > > folios with PG_active?
> > >
> > > Hi Ehab and Kairui,
> > >
> > > Thanks very much for your reports and discussion.
> > >
> > > I'm not quite sure we want to revert it entirely, as that would
> > > immediately regress the workloads improved by the commit, such as the
> kernel build.
> > > Also, for example, Kairui's cover letter mentioned the LevelDB benchmark:
> > >  "I also retested the LevelDB benchmark from the cache_ext paper [5].
> > > Interestingly, mainline MGLRU already beats CLRU on this one after a
> > > recent change in lru_gen_folio_seq that bumps new folios with refs
> > > == 1 to the second-oldest generation."
> > >
> > > I guess we could instead try to mitigate the cases where some
> > > workloads are negatively affected while preserving the original
> > > intention. Does the fix below address both of your cases, or is Ehab's case
> actually different from Kairui's?
> > > (The kernel-build test on my machine looks quite positive. It not
> > > only preserves the original optimization, but even provides further
> > > gains.)
> > >
> >
> > I agree that reverting the commit that caused the regression is not
> > the optimal path. I expect there are many workloads and scenarios that
> > benefit from the behavior introduced by that commit, so reverting it
> > could unnecessarily regress those workloads.
> >
> > I will run the Cassandra benchmark with Kairui's MGLRU-FG patches to
> > see whether they address the issue I am seeing. I will send the
> > results when they are ready.
> >
> 
> Thanks very much, Ehab. Could you please also run my proposed quick fix:
> 
> https://lore.kernel.org/linux-mm/20260901220430.79810-1-
> baohua@kernel.org/
> 

Sure, I will do. I will report back when I have the results.

Thanks,
Ehab

> Best Regards
> Barry