[PATCH RFC 00/15] mm/mglru: frequency guided promotion (MGLRU-FG) and flag cleanup

[PATCH RFC 00/15] mm/mglru: frequency guided promotion (MGLRU-FG) and flag cleanup

[Kairui Song via B4 Relay]

This is the updated RFC following the idea proposed at LSF/MM/BPF [1] this
year. It's very usable, stable, and performing well, but I'll keep it RFC
for V1 as some tests are still ongoing and results can be more accurate
with further auditing.

With this series, I'm seeing an obvious performance gain across all kinds
of tests, and it reduces MGLRU's flag usage by one. It also fixes several
long-standing issues including under-accounted PSI and poor workingset
tracking (especially for page cache).

Some test results (CLRU means classical LRU):

Build kernel test, running make -j48 in a 3G memcg, using disk swap and
holding the kernel and build output on the same NVMe drive, 16 runs using
different swappiness configurations [2]; the patched version is better than
mainline at almost every swappiness value, measuring the total average:

        real     sys  pgpgin  pswpin  pswpout  refault_file  refault_anon
CLRU   6m06s  31m01s   50.3M   3.20M    13.8M         10.3M         3.35M
Before 2m56s  11m06s   10.6M   1.59M    5.40M          414k         1.03M
After  2m49s  10m39s    9.0M   1.36M    4.82M          280k          861k
delta    -7s    -27s    -15%    -14%     -11%          -32%          -16%

MongoDB YCSB workloadb (recordcount:20000000 operationcount:6000000,
threads:48, in a 16G memcg), 3 runs [3]:
CLRU:          98389.94 ops/s
MGLRU Before:  86421.44 ops/s
MGLRU After:   95378.50 ops/s (+10.3%)

Chromium & Node.js test, using ZRAM as swap, on a 48c96t machine with
128G memory, 64 workers, run for 1 hour [4]:
                 Total requests:
CLRU:                     63822
MGLRU Before:            153029
MGLRU After:             225664 (+47.4%)

(NOTE: It seems some recent change broken MGLRU's fainress guarteen and
also made this test dramatically faster than a few months ago, which isn't
related to this series and reading are even better now, but I'll take a
deeper look later.)

FIO, zipf 0.9 distribution on an NVMe disk, in a 16G cgroup, total file
size 40G; this measures the LRU's theoretical ability to distinguish the
hotter portion:

fio --name=fg --numjobs=16 --nrfiles=1 \
    --filename_format="$testdir/rnvmedk.\$jobnum.img" \
    --size=${FILE_MIB}M \
    --buffered=1 --ioengine=sync --rw=randread \
    --random_distribution=zipf:$ZIPF --bs=$BS --time_based \
    --ramp_time=45s --runtime=600s \
    --group_reporting

CLRU:   Avg: 2454.37 MB/s
Before: Avg: 2350.40 MB/s
After:  Avg: 2611.37 MB/s (+11.1%)

I also retested the LevelDB benchmark from the cache_ext paper [5].
Interestingly, mainline MGLRU already beats CLRU on this one after a
recent change in lru_gen_folio_seq that bumps new folios with refs == 1
to the second-oldest generation. That change accidentally gave random
reads a higher hotness level while making sequential reads much colder:
sequential reads involve many readahead hits, and readahead folios start
with refs == 0, so they're already in the oldest generation, and
folio_mark_accessed() on a readahead hit has almost no effect on generations
in mainline MGLRU. Meanwhile, all direct-hit (random read) folios start
with refs == 1 in the second-oldest generation. As a result, the scan-get
test natively protects the "get" part and sacrifices the "scan" part.
That's not the best solution though. It's unreliable because it depends
on LRU drain timing, and it hurts workloads where the sequential part is
actually hotter (any workload involving a hotter large file and many
small cold files will be affected).

This series improves on that base: it covers ordinary workloads without
hurting the scan-get workload and without relying on that initial bump.

LevelDB Scan / Get, Throughput Total:
CLRU:         4668.8 ops/s
MGLRU:        5026.9 ops/s (faster than CLRU, but hurts other workloads)
MGLRU After:  5029.7 ops/s (fastest in all cases, and no regression)

The hot-sequential and cold-random workload can be easily reproduced with
SQLite and grep. SQLite continuously scans and looks up a small hot
portion of a DB file, while grep iterates over a set of small files much
larger than RAM [6]:

         SQLite scan & lookup time:       Grep iterate time:
CLRU:                       14.51ms               13281.37ms
MGLRU mainline:            567.05ms               13694.47ms
MGLRU After this series:    10.58ms               12930.43ms

The grep cold portion is larger than RAM and accessed only once per
iteration, so there's no promotion of any of it. CLRU handles
this reasonably; mainline MGLRU has a clear regression; MGLRU-FG now
not only recovers but is able to catch some hot parts from the cold grep
workload. This test is somewhat subjective, but the signal is clear.

Additionally, PSI, smaps, and readahead all benefit from better accuracy
since this series unifies the flag usage between classical LRU and MGLRU.

Other tests such as MySQL are looking fine, with no regressions.

Refault distance is not included yet, so MGLRU may respond more slowly to
workingset shifts. That can be added later, as previously demonstrated
[7], [8].

Extra note about future development: this series is highly compatible with
ideas like workingset reporting [9]. The "gen climbing folio" design may
appear to conflict with workingset reporting's idea of using generations
as access-gap identifiers, but it doesn't — the solution is
straightforward: once we can extend the generation number to a larger
value (e.g. 64 or 128), the refs-driven promotion can stop at a lower
gen (e.g. oldest_gen + 16), leaving the remaining newer generations as
perfectly time-gap-separated bins.

The tier count is not fixed either; we'll need to find a way to tune it
if tiers go beyond 4, but that shouldn't be hard.

More details are in the individual commit messages.

Link: https://lore.kernel.org/linux-mm/CAMgjq7BoekNjg-Ra3C8M7=8=75su38w=HD782T5E_cxyeCeH_g@mail.gmail.com/ [1]
Link: https://lore.kernel.org/linux-mm/CAGsJ_4xre-x0e+qNVm=KLFnO1dbPkPX5RuecqwvTZu-vS+o8yQ@mail.gmail.com/ [2]
Link: https://github.com/brianfrankcooper/YCSB/blob/master/workloads/workloadb [3]
Link: https://lore.kernel.org/all/20221220214923.1229538-1-yuzhao@google.com/ [4]
Link: https://dl.acm.org/doi/10.1145/3731569.3764820 [5]
Link: https://github.com/ryncsn/emm-test-project/tree/master/sqlite-grep [6]
Link: https://lwn.net/Articles/945266/ [7]
Link: https://lore.kernel.org/linux-mm/20260502-mglru-fg-v1-0-913619b014d9@tencent.com/ [8]
Link: https://lwn.net/Articles/976985/ [9]

Signed-off-by: Kairui Song <kasong@tencent.com>
---
Kairui Song (15):
      mm/memcontrol: make lru_zone_size atomic and simplify sanity check
      mm/memcontrol: allow update of LRU statistic without holding LRU lock
      mm/mglru: introduce and always use helpers for manipulating page flags
      mm/mglru: make generation page counters atomic
      mm/mglru: move max_seq read into walk_update_folio
      mm/mglru: use explicit tier range in read_ctrl_pos()
      mm/mglru: move refault workingset activation into lru_gen_refault
      mm/memcg: add folio-based lruvec live helper
      mm/mglru: frequency guided workingset promotion (MGLRU-FG)
      mm/mglru: make folio lru referenced times count a generic API
      mm/mglru: replace folio workinset check and update with new helper
      mm/smap: report workingset folios as referenced
      mm/huge_memory: mark file folio as accessed more accurately on split
      mm/khugepaged: consider workingset folios as referenced
      mm/madvise: convert to new lru refs API and better support for MGLRU

 fs/btrfs/compression.c     |   3 +-
 fs/proc/task_mmu.c         |  22 ++-
 include/linux/memcontrol.h |  47 +++++-
 include/linux/mm_inline.h  | 256 ++++++++++++++++++++++++++------
 include/linux/mmzone.h     | 137 ++++++++++++-----
 kernel/bounds.c            |   2 +-
 mm/filemap.c               |   8 +-
 mm/folio.c                 |  49 +-----
 mm/huge_memory.c           |   8 +-
 mm/khugepaged.c            |   6 +-
 mm/madvise.c               |  37 +++--
 mm/memcontrol.c            |  22 +--
 mm/migrate.c               |   4 -
 mm/page_io.c               |   3 +-
 mm/readahead.c             |   8 +-
 mm/vmscan.c                | 360 ++++++++++++++++++++++++++++-----------------
 mm/workingset.c            |  66 ++++++---
 17 files changed, 689 insertions(+), 349 deletions(-)
---
base-commit: 94f9b3980dd446b56acf1dfed649e9b32a9f3813
change-id: 20260722-mglru-fg-3a2c8574725b

Best regards,
--  
Kairui Song <kasong@tencent.com>

[PATCH RFC 01/15] mm/memcontrol: make lru_zone_size atomic and simplify sanity check

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

commit ca707239e8a7 ("mm: update_lru_size warn and reset bad lru_size")
introduced a sanity check to catch memcg counter underflow, which is
more like a workaround for another bug: lru_zone_size is unsigned, so
underflow will wrap it around and return an enormously large number,
then the memcg shrinker will loop almost forever as the calculated
number of folios to shrink is huge. That commit also checks if a zero
value matches the empty LRU list, so we have to hold the LRU lock, and
do the counter adding differently depending on whether the nr_pages is
negative.

But later commit b4536f0c829c ("mm, memcg: fix the active list aging for
lowmem requests when memcg is enabled") already removed the LRU
emptiness check, doing the adding differently is meaningless now. And if
we just turn it into an atomic long, underflow isn't a big issue either,
and can be checked at the reader side. The reader size is much less
frequently called than the updater.

So let's turn the counter into an atomic long and check at the
reader side instead, which has a smaller overhead. Use atomic to avoid
potential locking issue. The underflow correction is removed, which
should be fine as if there is a mass leaking of the LRU size counter,
something else may also have gone very wrong, and one should fix that
leaking site instead. Besides, doing the sanity check in updater is
unlikely to catch the leaking site, e.g. a folio was removed minutes ago
without updating the counter, while there are still other folios on the
LRU, the WARN won't be triggered until other folios are removed from a
likely correct callsite.

For now still keep the LRU lock context, in theory that can be removed
too since the update is atomic, if we can tolerate a temporary
inaccurate reading, but currently there is no benefit doing so yet.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/memcontrol.h |  9 +++++++--
 mm/memcontrol.c            | 18 +-----------------
 mm/vmscan.c                |  5 -----
 3 files changed, 8 insertions(+), 24 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index e78bc98ab229..b13e3f056319 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -113,7 +113,7 @@ struct mem_cgroup_per_node {
 	/* Fields which get updated often at the end. */
 	struct lruvec		lruvec;
 	CACHELINE_PADDING(_pad2_);
-	unsigned long		lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS];
+	atomic_long_t		lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS];
 	struct mem_cgroup_reclaim_iter	iter;
 
 	/*
@@ -897,10 +897,15 @@ static inline
 unsigned long mem_cgroup_get_zone_lru_size(struct lruvec *lruvec,
 		enum lru_list lru, int zone_idx)
 {
+	long val;
 	struct mem_cgroup_per_node *mz;
 
 	mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
-	return READ_ONCE(mz->lru_zone_size[zone_idx][lru]);
+	val = atomic_long_read(&mz->lru_zone_size[zone_idx][lru]);
+	if (WARN_ON_ONCE(val < 0))
+		return 0;
+
+	return val;
 }
 
 void __mem_cgroup_handle_over_high(gfp_t gfp_mask);
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 1bde9d5af88a..7511773b3491 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1529,28 +1529,12 @@ void mem_cgroup_update_lru_size(struct lruvec *lruvec, enum lru_list lru,
 				int zid, long nr_pages)
 {
 	struct mem_cgroup_per_node *mz;
-	unsigned long *lru_size;
-	long size;
 
 	if (mem_cgroup_disabled())
 		return;
 
 	mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
-	lru_size = &mz->lru_zone_size[zid][lru];
-
-	if (nr_pages < 0)
-		*lru_size += nr_pages;
-
-	size = *lru_size;
-	if (WARN_ONCE(size < 0,
-		"%s(%p, %d, %ld): lru_size %ld\n",
-		__func__, lruvec, lru, nr_pages, size)) {
-		VM_BUG_ON(1);
-		*lru_size = 0;
-	}
-
-	if (nr_pages > 0)
-		*lru_size += nr_pages;
+	atomic_long_add(nr_pages, &mz->lru_zone_size[zid][lru]);
 }
 
 /**
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 17d2b793cbfc..fe7cf5d42e0c 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1636,10 +1636,6 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,
 	return nr_reclaimed;
 }
 
-/*
- * Update LRU sizes after isolating pages. The LRU size updates must
- * be complete before mem_cgroup_update_lru_size due to a sanity check.
- */
 static __always_inline void update_lru_sizes(struct lruvec *lruvec,
 			enum lru_list lru, unsigned long *nr_zone_taken)
 {
@@ -1651,7 +1647,6 @@ static __always_inline void update_lru_sizes(struct lruvec *lruvec,
 
 		update_lru_size(lruvec, lru, zid, -nr_zone_taken[zid]);
 	}
-
 }
 
 /*

-- 
2.55.0

[PATCH RFC 02/15] mm/memcontrol: allow update of LRU statistic without holding LRU lock

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

To enable moving file pages in folio_mark_accessed directly and lazily
for MGLRU, allow updating the LRU statistic atomically without holding a
lock. It may cause temporary counter underflow, which should be fine as
we still follow final consistency of the counter, and it only serves as
a factor for calculating the reclaim budget in vmscan. A little
inaccuracy has no visible effect.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/memcontrol.h | 2 +-
 include/linux/mm_inline.h  | 3 +--
 mm/memcontrol.c            | 4 ++--
 3 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index b13e3f056319..68f363000d7f 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -902,7 +902,7 @@ unsigned long mem_cgroup_get_zone_lru_size(struct lruvec *lruvec,
 
 	mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
 	val = atomic_long_read(&mz->lru_zone_size[zone_idx][lru]);
-	if (WARN_ON_ONCE(val < 0))
+	if (val < 0)
 		return 0;
 
 	return val;
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 621c8653d8f7..3ad1619ce8ca 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -36,11 +36,10 @@ static __always_inline void __update_lru_size(struct lruvec *lruvec,
 {
 	struct pglist_data *pgdat = lruvec_pgdat(lruvec);
 
-	lockdep_assert_held(&lruvec->lru_lock);
 	WARN_ON_ONCE(nr_pages != (int)nr_pages);
 
 	mod_lruvec_state(lruvec, NR_LRU_BASE + lru, nr_pages);
-	__mod_zone_page_state(&pgdat->node_zones[zid],
+	mod_zone_page_state(&pgdat->node_zones[zid],
 				NR_ZONE_LRU_BASE + lru, nr_pages);
 }
 
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 7511773b3491..3fbd7a6f6650 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1522,8 +1522,8 @@ struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
  * @zid: zone id of the accounted pages
  * @nr_pages: positive when adding or negative when removing
  *
- * This function must be called under lru_lock, just before a page is added
- * to or just after a page is removed from an lru list.
+ * This function must be called when a page is added to or removed from
+ * an lru list. Caller need to protect the lruvec from being freed.
  */
 void mem_cgroup_update_lru_size(struct lruvec *lruvec, enum lru_list lru,
 				int zid, long nr_pages)

-- 
2.55.0

[PATCH RFC 03/15] mm/mglru: introduce and always use helpers for manipulating page flags

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

Instead of doing bit ops on folio->flags.f, introduce helpers for
adjusting folio's refs and gen info, make the code easier to debug and
understand.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/mm_inline.h | 83 ++++++++++++++++++++++++++++++++++++++++-------
 include/linux/mmzone.h    |  2 ++
 mm/folio.c                | 19 ++++++-----
 mm/migrate.c              |  2 --
 mm/vmscan.c               | 68 +++++++++++++++++++++-----------------
 5 files changed, 122 insertions(+), 52 deletions(-)

diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 3ad1619ce8ca..4076e3f7dcc8 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -141,10 +141,42 @@ static inline int lru_tier_from_refs(int refs, bool workingset)
 	return workingset ? MAX_NR_TIERS - 1 : order_base_2(refs);
 }
 
-static inline int folio_lru_refs(const struct folio *folio)
+/**
+ * lru_gen_from_flags - Return the LRU generation number from folio flags.
+ * @flags: folio flags
+ *
+ * Returns: A number between 0 and LRU_GEN_MAX, inclusive. Returns -1 if the
+ * flags indicate the folio is off the list (e.g., isolated).
+ */
+static inline int lru_gen_from_flags(unsigned long flags)
 {
-	unsigned long flags = READ_ONCE(folio->flags.f);
+	int gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF);
 
+	gen -= 1;
+	VM_WARN_ON_ONCE(gen != -1 && gen > LRU_GEN_MAX);
+	return gen;
+}
+
+/**
+ * lru_gen_set_flags - Set the LRU generation number to specified folio flags.
+ * @flags: pointer to the folio flags
+ * @gen: generation number, between 0 and LRU_GEN_MAX, inclusive.
+ */
+static inline void lru_gen_set_flags(unsigned long *flags, int gen)
+{
+	VM_WARN_ON_ONCE(gen > LRU_GEN_MAX || gen < 0);
+	BUILD_BUG_ON((LRU_GEN_MAX + 1) != MAX_NR_GENS);
+
+	*flags &= ~LRU_GEN_MASK;
+	*flags |= (gen + 1UL) << LRU_GEN_PGOFF;
+}
+
+/**
+ * lru_refs_from_flags - Return LRU referenced / access count from folio flags.
+ * @flags: folio flags
+ */
+static inline int lru_refs_from_flags(unsigned long flags)
+{
 	if (!(flags & BIT(PG_referenced)))
 		return 0;
 	/*
@@ -154,18 +186,46 @@ static inline int folio_lru_refs(const struct folio *folio)
 	return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + 1;
 }
 
-static inline int folio_lru_gen(const struct folio *folio)
+/**
+ * lru_refs_set_flags - Set the LRU referenced / access count to specified folio flags.
+ * @flags: pointer to the folio flags
+ * @refs: referenced / access count number, between 0 and LRU_REFS_MAX, inclusive.
+ */
+static inline void lru_refs_set_flags(unsigned long *flags, unsigned int refs)
+{
+	VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
+
+	*flags &= ~LRU_REFS_FLAGS;
+	if (!refs)
+		return;
+	*flags |= (BIT(PG_referenced) | ((refs - 1UL) << LRU_REFS_PGOFF));
+}
+
+static inline int folio_lru_refs(const struct folio *folio)
+{
+	return lru_refs_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
+}
+
+static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
 {
-	unsigned long flags = READ_ONCE(folio->flags.f);
+	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
+
+	do {
+		new_flags = old_flags;
+		lru_refs_set_flags(&new_flags, refs);
+	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
+}
 
-	return ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
+static inline int folio_lru_gen(const struct folio *folio)
+{
+	return lru_gen_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
 }
 
 static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen)
 {
 	unsigned long max_seq = lruvec->lrugen.max_seq;
 
-	VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
+	VM_WARN_ON_ONCE(gen > LRU_GEN_MAX);
 
 	/* see the comment on MIN_NR_GENS */
 	return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1);
@@ -269,7 +329,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
 	gen = lru_gen_from_seq(seq);
 	flags = (gen + 1UL) << LRU_GEN_PGOFF;
 	/* see the comment on MIN_NR_GENS about PG_active */
-	set_mask_bits(&folio->flags.f, LRU_GEN_MASK | BIT(PG_active), flags);
+	set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
 
 	lru_gen_update_size(lruvec, folio, -1, gen);
 	/* for folio_rotate_reclaimable() */
@@ -294,7 +354,7 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
 
 	/* for folio_migrate_flags() */
 	flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
-	flags = set_mask_bits(&folio->flags.f, LRU_GEN_MASK, flags);
+	flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, flags);
 	gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
 
 	lru_gen_update_size(lruvec, folio, gen, -1);
@@ -305,9 +365,7 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
 
 static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
 {
-	unsigned long refs = READ_ONCE(old->flags.f) & LRU_REFS_MASK;
-
-	set_mask_bits(&new->flags.f, LRU_REFS_MASK, refs);
+	folio_set_lru_refs(new, folio_lru_refs(old));
 }
 #else /* !CONFIG_LRU_GEN */
 
@@ -338,7 +396,8 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
 
 static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
 {
-
+	if (folio_test_referenced(old))
+		folio_set_referenced(new);
 }
 #endif /* CONFIG_LRU_GEN */
 
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index a26c8b855222..8048c6b0544d 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -496,7 +496,9 @@ enum lruvec_flags {
 #ifndef __GENERATING_BOUNDS_H
 
 #define LRU_GEN_MASK		((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)
+#define LRU_GEN_MAX		(BIT(LRU_GEN_WIDTH - 1) - 1)
 #define LRU_REFS_MASK		((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF)
+#define LRU_REFS_MAX		BIT(LRU_REFS_WIDTH)
 
 /*
  * For folios accessed multiple times through file descriptors,
diff --git a/mm/folio.c b/mm/folio.c
index a9e328c3f21b..f90b7f86dbe3 100644
--- a/mm/folio.c
+++ b/mm/folio.c
@@ -353,26 +353,28 @@ static void __lru_cache_activate_folio(struct folio *folio)
 
 static void lru_gen_inc_refs(struct folio *folio)
 {
-	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
+	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
+	int refs;
 
 	if (folio_test_unevictable(folio))
 		return;
 
 	/* see the comment on LRU_REFS_FLAGS */
-	if (!folio_test_referenced(folio)) {
-		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
+	if (!folio_lru_refs(folio)) {
+		folio_set_lru_refs(folio, 1);
 		return;
 	}
 
 	do {
-		if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
+		new_flags = old_flags;
+		refs = lru_refs_from_flags(old_flags);
+		if (refs == LRU_REFS_MAX) {
 			if (!folio_test_workingset(folio))
 				folio_set_workingset(folio);
 			return;
 		}
-
-		new_flags = old_flags + BIT(LRU_REFS_PGOFF);
-	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
+		lru_refs_set_flags(&new_flags, refs + 1);
+	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
 }
 
 static bool lru_gen_clear_refs(struct folio *folio)
@@ -384,7 +386,8 @@ static bool lru_gen_clear_refs(struct folio *folio)
 	if (gen < 0)
 		return true;
 
-	set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS | BIT(PG_workingset), 0);
+	folio_set_lru_refs(folio, 0);
+	folio_clear_workingset(folio);
 
 	rcu_read_lock();
 	seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
diff --git a/mm/migrate.c b/mm/migrate.c
index b937cbd76480..c737d0682fa4 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -779,8 +779,6 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
 {
 	int cpupid;
 
-	if (folio_test_referenced(folio))
-		folio_set_referenced(newfolio);
 	if (folio_test_uptodate(folio))
 		folio_mark_uptodate(newfolio);
 	if (folio_test_clear_active(folio)) {
diff --git a/mm/vmscan.c b/mm/vmscan.c
index fe7cf5d42e0c..f5b0a7c63a3a 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -842,19 +842,22 @@ static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags)
 	if (!folio_test_referenced(folio) && !folio_test_workingset(folio)) {
 		/* Activate file-backed executable folios after first usage. */
 		if (is_exec_file_folio(folio, vma_flags)) {
-			set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
+			folio_set_lru_refs(folio, 0);
+			folio_set_workingset(folio);
 			return true;
 		}
 
-		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
+		folio_set_lru_refs(folio, 1);
 		return false;
 	}
 
 	/* Promote on second access */
-	if (folio_lru_refs(folio) > 1)
-		set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
-	else
+	if (folio_lru_refs(folio) > 1) {
+		folio_set_lru_refs(folio, 0);
+		folio_set_workingset(folio);
+	} else {
 		folio_mark_accessed(folio);
+	}
 	return true;
 }
 #else
@@ -3260,11 +3263,10 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv)
  ******************************************************************************/
 
 /* promote pages accessed through page tables */
-static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma_flags)
+static int folio_update_gen(struct folio *folio, int new_gen, const vma_flags_t *vma_flags)
 {
-	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
-
-	VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
+	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
+	int old_gen;
 
 	/*
 	 * See the comment on LRU_REFS_FLAGS, and activate file-backed
@@ -3273,20 +3275,24 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma
 	 */
 	if (!folio_test_referenced(folio) && !folio_test_workingset(folio) &&
 	    !is_exec_file_folio(folio, vma_flags)) {
-		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
+		folio_set_lru_refs(folio, 1);
 		return -1;
 	}
 
 	do {
+		old_gen = lru_gen_from_flags(old_flags);
+		new_flags = old_flags;
+
 		/* lru_gen_del_folio() has isolated this page? */
-		if (!(old_flags & LRU_GEN_MASK))
-			return -1;
+		if (old_gen < 0)
+			break;
 
-		new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);
-		new_flags |= ((gen + 1UL) << LRU_GEN_PGOFF) | BIT(PG_workingset);
-	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
+		lru_gen_set_flags(&new_flags, new_gen);
+		lru_refs_set_flags(&new_flags, 0);
+		new_flags |= BIT(PG_workingset);
+	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
 
-	return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
+	return old_gen;
 }
 
 /* protect pages accessed multiple times through file descriptors */
@@ -3294,22 +3300,22 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
 {
 	int type = folio_is_file_lru(folio);
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
-	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
-	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
-
-	VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio);
+	int old_gen, new_gen, min_gen = lru_gen_from_seq(lrugen->min_seq[type]);
+	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
 
 	do {
-		new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
+		old_gen = lru_gen_from_flags(old_flags);
+		VM_WARN_ON_ONCE_FOLIO(old_gen < 0, folio);
+
 		/* folio_update_gen() has promoted this page? */
-		if (new_gen >= 0 && new_gen != old_gen)
-			return new_gen;
+		if (old_gen >= 0 && old_gen != min_gen)
+			return old_gen;
 
+		new_flags = old_flags;
 		new_gen = (old_gen + 1) % MAX_NR_GENS;
-
-		new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);
-		new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF;
-	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
+		lru_gen_set_flags(&new_flags, new_gen);
+		lru_refs_set_flags(&new_flags, 0);
+	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
 
 	lru_gen_update_size(lruvec, folio, old_gen, new_gen);
 
@@ -4712,7 +4718,7 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca
 
 	/* see the comment on LRU_REFS_FLAGS */
 	if (!folio_test_referenced(folio))
-		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, 0);
+		folio_set_lru_refs(folio, 0);
 
 	success = lru_gen_del_folio(lruvec, folio, true);
 	VM_WARN_ON_ONCE_FOLIO(!success, folio);
@@ -4930,8 +4936,10 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 		}
 
 		/* don't add rejected folios to the oldest generation */
-		if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type])
-			set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_active));
+		if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type]) {
+			folio_set_lru_refs(folio, 0);
+			folio_set_active(folio);
+		}
 	}
 
 	move_folios_to_lru(&list);

-- 
2.55.0

[PATCH RFC 04/15] mm/mglru: make generation page counters atomic

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

No feature change, convert them to atomic so we can update them without
holding the LRU lock. There is no risk of overflow. The reader always
compares and uses zero instead if the counter values are negative. It
follows final consistency.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/mm_inline.h |  6 ++----
 include/linux/mmzone.h    |  2 +-
 mm/vmscan.c               | 20 ++++++++++----------
 3 files changed, 13 insertions(+), 15 deletions(-)

diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 4076e3f7dcc8..018a2f54a5c9 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -245,11 +245,9 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli
 	VM_WARN_ON_ONCE(old_gen == -1 && new_gen == -1);
 
 	if (old_gen >= 0)
-		WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
-			   lrugen->nr_pages[old_gen][type][zone] - delta);
+		atomic_long_sub(delta, &lrugen->nr_pages[old_gen][type][zone]);
 	if (new_gen >= 0)
-		WRITE_ONCE(lrugen->nr_pages[new_gen][type][zone],
-			   lrugen->nr_pages[new_gen][type][zone] + delta);
+		atomic_long_add(delta, &lrugen->nr_pages[new_gen][type][zone]);
 
 	/* addition */
 	if (old_gen < 0) {
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 8048c6b0544d..4225dab760ba 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -572,7 +572,7 @@ struct lru_gen_folio {
 	/* the multi-gen LRU lists, lazily sorted on eviction */
 	struct list_head folios[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
 	/* the multi-gen LRU sizes, eventually consistent */
-	long nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
+	atomic_long_t nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
 	/* the exponential moving average of refaulted */
 	unsigned long avg_refaulted[ANON_AND_FILE][MAX_NR_TIERS];
 	/* the exponential moving average of evicted+protected */
diff --git a/mm/vmscan.c b/mm/vmscan.c
index f5b0a7c63a3a..b02d2ec8ff4b 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3354,8 +3354,7 @@ static void reset_batch_size(struct lru_gen_mm_walk *walk)
 			continue;
 
 		walk->nr_pages[gen][type][zone] = 0;
-		WRITE_ONCE(lrugen->nr_pages[gen][type][zone],
-			   lrugen->nr_pages[gen][type][zone] + delta);
+		atomic_long_add(delta, &lrugen->nr_pages[gen][type][zone]);
 
 		if (lru_gen_is_active(lruvec, gen))
 			lru += LRU_ACTIVE;
@@ -4044,8 +4043,8 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness
 	for (type = 0; type < ANON_AND_FILE; type++) {
 		for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 			enum lru_list lru = type * LRU_INACTIVE_FILE;
-			long delta = lrugen->nr_pages[prev][type][zone] -
-				     lrugen->nr_pages[next][type][zone];
+			long delta = atomic_long_read(&lrugen->nr_pages[prev][type][zone]) -
+				     atomic_long_read(&lrugen->nr_pages[next][type][zone]);
 
 			if (!delta)
 				continue;
@@ -4163,7 +4162,8 @@ static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness
 		for (seq = min_seq[type]; seq <= max_seq; seq++) {
 			gen = lru_gen_from_seq(seq);
 			for (zone = 0; zone < MAX_NR_ZONES; zone++)
-				total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
+				total += max(atomic_long_read(&lrugen->nr_pages[gen][type][zone]),
+					     0L);
 		}
 	}
 
@@ -4598,7 +4598,7 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
 
 	for (i = 0; i < get_nr_gens(child_lruvec, type); i++) {
 		int gen = lru_gen_from_seq(child_lrugen->max_seq - i);
-		long nr_pages = child_lrugen->nr_pages[gen][type][zone];
+		long nr_pages = atomic_long_read(&child_lrugen->nr_pages[gen][type][zone]);
 		int child_lru_active = lru_gen_is_active(child_lruvec, gen) ? LRU_ACTIVE : 0;
 		int parent_lru_active = lru_gen_is_active(parent_lruvec, gen) ? LRU_ACTIVE : 0;
 
@@ -4606,9 +4606,8 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
 		list_splice_tail_init(&child_lrugen->folios[gen][type][zone],
 				      &parent_lrugen->folios[gen][type][zone]);
 
-		WRITE_ONCE(child_lrugen->nr_pages[gen][type][zone], 0);
-		WRITE_ONCE(parent_lrugen->nr_pages[gen][type][zone],
-			   parent_lrugen->nr_pages[gen][type][zone] + nr_pages);
+		atomic_long_set(&child_lrugen->nr_pages[gen][type][zone], 0);
+		atomic_long_add(nr_pages, &parent_lrugen->nr_pages[gen][type][zone]);
 
 		if (lru_gen_is_active(child_lruvec, gen) != lru_gen_is_active(parent_lruvec, gen)) {
 			__update_lru_size(child_lruvec, lru + child_lru_active, zone, -nr_pages);
@@ -5650,7 +5649,8 @@ static int lru_gen_seq_show(struct seq_file *m, void *v)
 			char mark = full && seq < min_seq[type] ? 'x' : ' ';
 
 			for (zone = 0; zone < MAX_NR_ZONES; zone++)
-				size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
+				size += max(atomic_long_read(&lrugen->nr_pages[gen][type][zone]),
+					    0L);
 
 			seq_printf(m, " %10lu%c", size, mark);
 		}

-- 
2.55.0

[PATCH RFC 05/15] mm/mglru: move max_seq read into walk_update_folio

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

walk_pte_range(), walk_pmd_range_locked(), and lru_gen_look_around()
each read lrugen->max_seq to compute the target generation for
folio_update_gen(), then pass it as a parameter to
walk_update_folio().  Move the read into walk_update_folio() itself
so the callers no longer need to compute or pass the value.

This is a pure refactoring: no functional change.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 mm/vmscan.c | 29 ++++++++++++-----------------
 1 file changed, 12 insertions(+), 17 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index b02d2ec8ff4b..c2ea92c2b69e 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3511,13 +3511,15 @@ static bool suitable_to_scan(int total, int young)
 }
 
 static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struct *vma,
-		struct folio *folio, int new_gen, bool dirty)
+			      struct lruvec *lruvec, struct folio *folio, bool dirty)
 {
-	int old_gen;
+	int new_gen, old_gen;
 
 	if (!folio)
 		return;
 
+	new_gen = lru_gen_from_seq(READ_ONCE(lruvec->lrugen.max_seq));
+
 	if (dirty && !folio_test_dirty(folio) &&
 	    !(folio_test_anon(folio) && folio_test_swapbacked(folio) &&
 	      !folio_test_swapcache(folio)))
@@ -3548,8 +3550,6 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
 	struct lru_gen_mm_walk *walk = args->private;
 	struct mem_cgroup *memcg = lruvec_memcg(walk->lruvec);
 	struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec);
-	DEFINE_MAX_SEQ(walk->lruvec);
-	int gen = lru_gen_from_seq(max_seq);
 	unsigned int nr;
 	pmd_t pmdval;
 
@@ -3600,7 +3600,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
 			continue;
 
 		if (last != folio) {
-			walk_update_folio(walk, args->vma, last, gen, dirty);
+			walk_update_folio(walk, args->vma, walk->lruvec, last, dirty);
 
 			last = folio;
 			dirty = false;
@@ -3613,7 +3613,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
 		walk->mm_stats[MM_LEAF_YOUNG] += nr;
 	}
 
-	walk_update_folio(walk, args->vma, last, gen, dirty);
+	walk_update_folio(walk, args->vma, walk->lruvec, last, dirty);
 	last = NULL;
 
 	if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &end))
@@ -3636,8 +3636,6 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
 	struct lru_gen_mm_walk *walk = args->private;
 	struct mem_cgroup *memcg = lruvec_memcg(walk->lruvec);
 	struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec);
-	DEFINE_MAX_SEQ(walk->lruvec);
-	int gen = lru_gen_from_seq(max_seq);
 
 	VM_WARN_ON_ONCE(pud_leaf(*pud));
 
@@ -3691,7 +3689,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
 			goto next;
 
 		if (last != folio) {
-			walk_update_folio(walk, vma, last, gen, dirty);
+			walk_update_folio(walk, vma, walk->lruvec, last, dirty);
 
 			last = folio;
 			dirty = false;
@@ -3705,7 +3703,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
 		i = i > MIN_LRU_BATCH ? 0 : find_next_bit(bitmap, MIN_LRU_BATCH, i) + 1;
 	} while (i <= MIN_LRU_BATCH);
 
-	walk_update_folio(walk, vma, last, gen, dirty);
+	walk_update_folio(walk, vma, walk->lruvec, last, dirty);
 
 	lazy_mmu_mode_disable();
 	spin_unlock(ptl);
@@ -4270,8 +4268,6 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 	struct pglist_data *pgdat = folio_pgdat(folio);
 	struct lruvec *lruvec;
 	struct lru_gen_mm_state *mm_state;
-	unsigned long max_seq;
-	int gen;
 
 	lockdep_assert_held(pvmw->ptl);
 	VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio);
@@ -4308,8 +4304,6 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 
 	memcg = get_mem_cgroup_from_folio(folio);
 	lruvec = mem_cgroup_lruvec(memcg, pgdat);
-	max_seq = READ_ONCE((lruvec)->lrugen.max_seq);
-	gen = lru_gen_from_seq(max_seq);
 	mm_state = get_mm_state(lruvec);
 
 	lazy_mmu_mode_enable();
@@ -4341,7 +4335,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 			continue;
 
 		if (last != folio) {
-			walk_update_folio(walk, vma, last, gen, dirty);
+			walk_update_folio(walk, vma, lruvec, last, dirty);
 
 			last = folio;
 			dirty = false;
@@ -4353,13 +4347,14 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 		young += nr;
 	}
 
-	walk_update_folio(walk, vma, last, gen, dirty);
+	walk_update_folio(walk, vma, lruvec, last, dirty);
 
 	lazy_mmu_mode_disable();
 
 	/* feedback from rmap walkers to page table walkers */
 	if (mm_state && suitable_to_scan(i, young))
-		update_bloom_filter(mm_state, max_seq, pvmw->pmd);
+		update_bloom_filter(mm_state, READ_ONCE(lruvec->lrugen.max_seq),
+				    pvmw->pmd);
 
 	mem_cgroup_put(memcg);
 

-- 
2.55.0

[PATCH RFC 06/15] mm/mglru: use explicit tier range in read_ctrl_pos()

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

read_ctrl_pos() encodes the tier range in a single "tier" parameter
via "tier % MAX_NR_TIERS" as the start and "min(tier, MAX_NR_TIERS-1)"
as the end.  This is hard to follow or maintain or extend. Tier
values 0..3 select a single tier, while tier == MAX_NR_TIERS selects
the full range.

Replace it with explicit (tier_min, tier_max) parameters using a
half-open [tier_min, tier_max) interval, which is the conventional C
idiom.  The call sites become self-documenting:

  - get_tier_idx:   (0, 1) for tier 0, (tier, tier+1) for each tier
  - get_type_to_scan: (0, MAX_NR_TIERS) for the full range

No functional change.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 mm/vmscan.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index c2ea92c2b69e..a359d5a1ff41 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3192,8 +3192,8 @@ struct ctrl_pos {
 	int gain;
 };
 
-static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain,
-			  struct ctrl_pos *pos)
+static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier_min,
+			  int tier_max, int gain, struct ctrl_pos *pos)
 {
 	int i;
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
@@ -3202,7 +3202,7 @@ static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain,
 	pos->gain = gain;
 	pos->refaulted = pos->total = 0;
 
-	for (i = tier % MAX_NR_TIERS; i <= min(tier, MAX_NR_TIERS - 1); i++) {
+	for (i = tier_min; i < tier_max; i++) {
 		pos->refaulted += lrugen->avg_refaulted[type][i] +
 				  atomic_long_read(&lrugen->refaulted[hist][type][i]);
 		pos->total += lrugen->avg_total[type][i] +
@@ -4805,9 +4805,9 @@ static int get_tier_idx(struct lruvec *lruvec, int type)
 	 * This value is chosen because any other tier would have at least twice
 	 * as many refaults as the first tier.
 	 */
-	read_ctrl_pos(lruvec, type, 0, 2, &sp);
+	read_ctrl_pos(lruvec, type, 0, 1, 2, &sp);
 	for (tier = 1; tier < MAX_NR_TIERS; tier++) {
-		read_ctrl_pos(lruvec, type, tier, 3, &pv);
+		read_ctrl_pos(lruvec, type, tier, tier + 1, 3, &pv);
 		if (!positive_ctrl_err(&sp, &pv))
 			break;
 	}
@@ -4828,8 +4828,8 @@ static int get_type_to_scan(struct lruvec *lruvec, int swappiness)
 	 * Compare the sum of all tiers of anon with that of file to determine
 	 * which type to scan.
 	 */
-	read_ctrl_pos(lruvec, LRU_GEN_ANON, MAX_NR_TIERS, swappiness, &sp);
-	read_ctrl_pos(lruvec, LRU_GEN_FILE, MAX_NR_TIERS, MAX_SWAPPINESS - swappiness, &pv);
+	read_ctrl_pos(lruvec, LRU_GEN_ANON, 0, MAX_NR_TIERS, swappiness, &sp);
+	read_ctrl_pos(lruvec, LRU_GEN_FILE, 0, MAX_NR_TIERS, MAX_SWAPPINESS - swappiness, &pv);
 
 	return positive_ctrl_err(&sp, &pv);
 }

-- 
2.55.0

[PATCH RFC 07/15] mm/mglru: move refault workingset activation into lru_gen_refault

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

Move the folio_set_active() for refaulted workingset folios from
folio_add_lru() into lru_gen_refault(), where the refault detection
already happens.  No functional change: the ordering and logic are
preserved in all cases, and no other paths reach the removed branch.

This is a preparatory cleanup for MGLRU-FG.
---
 mm/folio.c      | 6 +-----
 mm/workingset.c | 9 ++++-----
 2 files changed, 5 insertions(+), 10 deletions(-)

diff --git a/mm/folio.c b/mm/folio.c
index f90b7f86dbe3..fab00cb02970 100644
--- a/mm/folio.c
+++ b/mm/folio.c
@@ -474,17 +474,13 @@ void folio_add_lru(struct folio *folio)
 	VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
 
 	/*
-	 * For refaulted workingset folios, set PG_active so they
-	 * can be added to active generations.
 	 * For prefaulted file folios, folio_mark_accessed() sets
 	 * PG_referenced so lru_gen_folio_seq() places them into
 	 * the second oldest generation.
 	 */
 	if (lru_gen_enabled() && !folio_test_unevictable(folio) &&
 	    lru_gen_in_fault() && !(current->flags & PF_MEMALLOC)) {
-		if (folio_test_workingset(folio))
-			folio_set_active(folio);
-		else if (!folio_test_referenced(folio))
+		if (!folio_test_referenced(folio) && !folio_test_workingset(folio))
 			folio_mark_accessed(folio);
 	}
 
diff --git a/mm/workingset.c b/mm/workingset.c
index 7ac2b88c80ae..5438e9390011 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -320,12 +320,11 @@ static void lru_gen_refault(struct folio *folio, void *shadow)
 	atomic_long_add(delta, &lrugen->refaulted[hist][type][tier]);
 
 	if (workingset) {
-		/*
-		 * see folio_add_lru(), where folio_set_active() is
-		 * called for workingset folios
-		 */
-		if (lru_gen_in_fault())
+		/* Send refaulted workingset folios to active generations. */
+		if (lru_gen_in_fault()) {
+			folio_set_active(folio);
 			mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
+		}
 		folio_set_workingset(folio);
 		mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + type, delta);
 	} else

-- 
2.55.0

[PATCH RFC 08/15] mm/memcg: add folio-based lruvec live helper

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

Add a helper that resolves a stable lruvec for a folio under RCU
without taking the lruvec lock.  It takes a folio directly so the
lruvec lookup happens inside the RCU read-side critical section,
which a lruvec-based interface cannot guarantee.

The lock-taking variant now inlines the ancestor walk instead of
calling a separate helper.

No functional change.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/memcontrol.h | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 68f363000d7f..ea0111392b9b 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1506,6 +1506,44 @@ static inline void lruvec_lock_irq(struct lruvec *lruvec)
 	spin_lock_irq(&lruvec->lru_lock);
 }
 
+/**
+ * folio_lruvec_live_get - get a live lruvec for a folio under RCU
+ * @folio: the folio
+ *
+ * Computes @folio's lruvec and walks up to the nearest live ancestor
+ * if the folio's memcg is dying.  Must be paired with
+ * folio_lruvec_live_put().
+ *
+ * Return: the live lruvec, with rcu_read_lock held.
+ */
+static inline struct lruvec *folio_lruvec_live_get(struct folio *folio)
+{
+#ifdef CONFIG_MEMCG
+	struct lruvec *lruvec;
+	struct pglist_data *pgdat;
+	struct mem_cgroup *memcg;
+
+	rcu_read_lock();
+	lruvec = folio_lruvec(folio);
+	pgdat = lruvec_pgdat(lruvec);
+	memcg = lruvec_memcg(lruvec);
+	while (unlikely(memcg && css_is_dying(&memcg->css))) {
+		memcg = parent_mem_cgroup(memcg);
+		lruvec = mem_cgroup_lruvec(memcg, pgdat);
+	}
+	return lruvec;
+#else
+	return folio_lruvec(folio);
+#endif
+}
+
+static inline void folio_lruvec_live_put(struct lruvec *lruvec)
+{
+#ifdef CONFIG_MEMCG
+	rcu_read_unlock();
+#endif
+}
+
 static inline struct lruvec *lruvec_live_lock_irq(struct lruvec *lruvec)
 {
 #ifdef CONFIG_MEMCG

-- 
2.55.0

[PATCH RFC 09/15] mm/mglru: frequency guided workingset promotion (MGLRU-FG)

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

Complement MGLRU's eviction-time tier-PID protection with access-time
frequency-guided promotion.  Introduce a unified set of helpers built based
on referenced (access) count of a folio.

Each access increments a folio's referenced count stored in folio flags
(refs), refs still mappes to a logarithmic tier just like before, but with
more formal bit definitions, a few special thresholds are introduced:
LRU_REFS_REFERENCED (1), LRU_REFS_WORKINGSET (2), LRU_REFS_PROTECTED (3),
and LRU_REFS_MAX(7). When it reaches certain threshold, the folio is
promoted proactively instead of wait for the PID controller to kick in.

Also simplify MGLRU's usage of PG_workingset and PG_referenced, now
these 2 flags are purely used as the lower 2 bit of refs for MGLRU. This
doesn't effect classical LRU in any way. This will actually simplify and
make MGLRU's certain metric reading more accurate, and reduced MGLRU's
original tier / referenced count bit by one since only one extra bit is
now needed to record a max referenced count of 7 (previously 2 extra bits
are needed). This changes make sense because MGLRU doesn't have demotion
so these 2 flags are never separately useful for MGLRU.

This addresses several shortcomings of the old model:

- Long feedback loop: protection only activated after enough
  re-faults, by which time the folio is often no longer hot.

- Limited tier resolution: once referenced count exceeded the bits
  limit (8 previously), MGLRU could no longer distinguish hotter folios as
  they are capped by the tier.  And what's worse, PG_workingset
  forces a folio to stay on tier 3.

- Eviction-time bias: because PID protection activates upon eviction
  and always targets the LRU tail, it tends to protect cold tail
  folios at the expense of hotter head folios.  Once the tail folios
  consume the PID protection budget, head folios lose their
  protection.  Additionally, the PID cannot distinguish the access
  time of folios that share the same reference count.

Besides reworking the LRU_REFS related helpers and definitions, most of
the work is done by the helpers below; the implementation details are
described in their inline comments.

- folio_inc_lru_refs(): Used by both cache access (folio_mark_accessed)
  and page table access.  The folio could be off-list (isolated),
  unlocked, or unmapped.  This helper uses PG_lru to stabilize the
  folio and performs a speculative and lazy promotion.

- folio_inc_lru_refs_walk(): Used by the PTE walk path during aging,
  where generations are stable; performs lazy promotion.

- folio_inc_lru_refs_isolated(): Used by the rmap check before
  eviction.  The folio is isolated and hence this doesn't perform
  promotion by itself; the folio will be added back to the right gen
  upon return.

The eviction-time folio_inc_gen() still handles PID protection, but the
protection ratio is softer than before, and it caps refs at WORKINGSET
so the folio retains enough history to stay above the cold tier.

The PID controller gain factors in get_tier_idx() are also relaxed from
(2:3) to (1:2).  Since the new folio gen bump paths already proactively
protect hot folios, PID protection can afford to be more permissive
without increasing the refault rate.

PG_workingset and PG_referenced are repurposed as the low two bits
of the unified LRU reference count.  LRU_REFS_MASK provides the
higher bits.  This eliminates the old restriction where LRU_REFS_MASK
was only valid when PG_referenced was set, and allows all paths to use
the same encoding consistently.

Hence, a workingset folio is now defined as refs >= LRU_REFS_WORKINGSET
(2), matching the active/inactive LRU's definition and giving in-kernel
consumers (PSI, readahead) consistent behavior on MGLRU, which will be
done in later commits.

Note that PG_workingset and PG_referenced are no longer independent
flags under MGLRU.  Adjusting existing raw folio_test_*() callers
to the new semantics is left as follow-ups.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/mm_inline.h |  83 ++++++++-----
 include/linux/mmzone.h    | 135 ++++++++++++++-------
 kernel/bounds.c           |   2 +-
 mm/folio.c                |  46 +-------
 mm/vmscan.c               | 290 ++++++++++++++++++++++++++++++----------------
 mm/workingset.c           |  55 ++++++---
 6 files changed, 385 insertions(+), 226 deletions(-)

diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 018a2f54a5c9..944baa91bf18 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -133,12 +133,13 @@ static inline int lru_hist_from_seq(unsigned long seq)
 	return seq % NR_HIST_GENS;
 }
 
-static inline int lru_tier_from_refs(int refs, bool workingset)
+static inline int lru_tier_from_refs(unsigned int refs)
 {
-	VM_WARN_ON_ONCE(refs > BIT(LRU_REFS_WIDTH));
-
-	/* see the comment on MAX_NR_TIERS */
-	return workingset ? MAX_NR_TIERS - 1 : order_base_2(refs);
+	BUILD_BUG_ON(fls(LRU_REFS_MAX - 1) > MAX_NR_TIERS - 1);
+	VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
+	if (refs < LRU_REFS_WORKINGSET)
+		return 0;
+	return fls(refs - 1);
 }
 
 /**
@@ -164,9 +165,8 @@ static inline int lru_gen_from_flags(unsigned long flags)
  */
 static inline void lru_gen_set_flags(unsigned long *flags, int gen)
 {
-	VM_WARN_ON_ONCE(gen > LRU_GEN_MAX || gen < 0);
 	BUILD_BUG_ON((LRU_GEN_MAX + 1) != MAX_NR_GENS);
-
+	VM_WARN_ON_ONCE(gen > LRU_GEN_MAX || gen < 0);
 	*flags &= ~LRU_GEN_MASK;
 	*flags |= (gen + 1UL) << LRU_GEN_PGOFF;
 }
@@ -177,13 +177,16 @@ static inline void lru_gen_set_flags(unsigned long *flags, int gen)
  */
 static inline int lru_refs_from_flags(unsigned long flags)
 {
-	if (!(flags & BIT(PG_referenced)))
-		return 0;
+	int refs;
+
 	/*
-	 * Return the total number of accesses including PG_referenced. Also see
-	 * the comment on LRU_REFS_FLAGS.
+	 * Return the total number of accesses. Also see the comment on
+	 * LRU_REFS_FLAGS.
 	 */
-	return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + 1;
+	refs = (flags & BIT(PG_referenced)) ? BIT(0) : 0;
+	refs += (flags & BIT(PG_workingset)) ? BIT(1) : 0;
+	refs += ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) << 2;
+	return refs;
 }
 
 /**
@@ -194,11 +197,13 @@ static inline int lru_refs_from_flags(unsigned long flags)
 static inline void lru_refs_set_flags(unsigned long *flags, unsigned int refs)
 {
 	VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
-
+	BUILD_BUG_ON((LRU_REFS_MAX >> 2) > (BIT(LRU_REFS_WIDTH) - 1));
 	*flags &= ~LRU_REFS_FLAGS;
-	if (!refs)
-		return;
-	*flags |= (BIT(PG_referenced) | ((refs - 1UL) << LRU_REFS_PGOFF));
+	if (refs & BIT(0))
+		*flags |= BIT(PG_referenced);
+	if (refs & BIT(1))
+		*flags |= BIT(PG_workingset);
+	*flags |= (((unsigned long)refs) >> 2) << LRU_REFS_PGOFF;
 }
 
 static inline int folio_lru_refs(const struct folio *folio)
@@ -216,6 +221,8 @@ static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
 	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
 }
 
+int folio_inc_lru_refs(struct folio *folio, bool is_fault, bool is_exec);
+
 static inline int folio_lru_gen(const struct folio *folio)
 {
 	return lru_gen_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
@@ -223,7 +230,7 @@ static inline int folio_lru_gen(const struct folio *folio)
 
 static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen)
 {
-	unsigned long max_seq = lruvec->lrugen.max_seq;
+	unsigned long max_seq = READ_ONCE(lruvec->lrugen.max_seq);
 
 	VM_WARN_ON_ONCE(gen > LRU_GEN_MAX);
 
@@ -280,23 +287,24 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
 					      bool reclaiming)
 {
 	int gen;
+	int refs = folio_lru_refs(folio);
 	int type = folio_is_file_lru(folio);
 	const struct lru_gen_folio *lrugen = &lruvec->lrugen;
 
 	/*
-	 * +-----------------------------------+-----------------------------------+
-	 * | Accessed through page tables and  | Accessed through file descriptors |
-	 * | promoted by folio_update_gen()    | and protected by folio_inc_gen()  |
-	 * +-----------------------------------+-----------------------------------+
-	 * | PG_active (set while isolated)    |                                   |
-	 * +-----------------+-----------------+-----------------+-----------------+
-	 * |  PG_workingset  |  PG_referenced  |  PG_workingset  |  LRU_REFS_FLAGS |
-	 * +-----------------------------------+-----------------------------------+
-	 * |<---------- MIN_NR_GENS ---------->|                                   |
-	 * |<---------------------------- MAX_NR_GENS ---------------------------->|
+	 * +------------------------------------------+------------------------------------------+
+	 * |     Accessed through page tables and     |     Accessed through file descriptors    |
+	 * | promoted by folio_inc_lru_refs_walk()    | protected by folio_inc_lru_refs/inc_gen  |
+	 * +------------------------------------------+------------------------------------------+
+	 * | PG_active (set at isolation or refault)  |                                          |
+	 * +--------------------+---------------------+--------------------+---------------------+
+	 * |     LRU_REFS_MAX   | LRU_REFS_WORKINGSET |    LRU_REFS_MAX    | LRU_REFS_WORKINGSET |
+	 * +------------------------------------------+------------------------------------------+
+	 * |<-------------- MIN_NR_GENS ------------->|                                          |
+	 * |<----------------------------------- MAX_NR_GENS ----------------------------------->|
 	 */
 	if (folio_test_active(folio))
-		gen = MIN_NR_GENS - folio_test_workingset(folio);
+		gen = MIN_NR_GENS - (refs >= LRU_REFS_WORKINGSET);
 	else if (reclaiming)
 		gen = MAX_NR_GENS;
 	else if ((!folio_is_file_lru(folio) && !folio_test_swapcache(folio)) ||
@@ -304,7 +312,7 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
 		  (folio_test_dirty(folio) || folio_test_writeback(folio))))
 		gen = MIN_NR_GENS;
 	else
-		gen = MAX_NR_GENS - (folio_test_workingset(folio) || folio_test_referenced(folio));
+		gen = MAX_NR_GENS - (refs >= LRU_REFS_WORKINGSET);
 
 	return max(READ_ONCE(lrugen->max_seq) - gen + 1, READ_ONCE(lrugen->min_seq[type]));
 }
@@ -365,6 +373,7 @@ static inline void folio_migrate_refs(struct folio *new, const struct folio *old
 {
 	folio_set_lru_refs(new, folio_lru_refs(old));
 }
+
 #else /* !CONFIG_LRU_GEN */
 
 static inline bool lru_gen_enabled(void)
@@ -392,10 +401,26 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
 	return false;
 }
 
+static inline int folio_lru_refs(const struct folio *folio)
+{
+	return 0;
+}
+
+static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
+{
+}
+
+static inline int folio_inc_lru_refs(struct folio *folio, bool promote, bool is_exec)
+{
+	return 0;
+}
+
 static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
 {
 	if (folio_test_referenced(old))
 		folio_set_referenced(new);
+	if (folio_test_workingset(old))
+		folio_set_workingset(new);
 }
 #endif /* CONFIG_LRU_GEN */
 
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 4225dab760ba..e4f7efc02e50 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -472,56 +472,111 @@ enum lruvec_flags {
 #define MAX_NR_GENS		4U
 
 /*
- * Each generation is divided into multiple tiers. A folio accessed N times
- * through file descriptors is in tier order_base_2(N). A folio in the first
- * tier (N=0,1) is marked by PG_referenced unless it was faulted in through page
- * tables or read ahead. A folio in the last tier (MAX_NR_TIERS-1) is marked by
- * PG_workingset. A folio in any other tier (1<N<5) between the first and last
- * is marked by additional bits of LRU_REFS_WIDTH in folio->flags.
+ * Each generation is divided into multiple tiers. A folio's referenced
+ * count maps to a tier as shown below:
  *
- * In contrast to moving across generations which requires the LRU lock, moving
- * across tiers only involves atomic operations on folio->flags and therefore
- * has a negligible cost in the buffered access path. In the eviction path,
- * comparisons of refaulted/(evicted+protected) from the first tier and the rest
- * infer whether folios accessed multiple times through file descriptors are
- * statistically hot and thus worth protecting.
+ * MGLRU (frequency guidance)
+ *  Refs  Tier  |- Refs: how many times (at least) a folio has been referenced.
+ *   0      0   |- Mostly cold pages, readahead, etc. [1]
+ *   1      0   |= LRU_REFS_REFERENCED: Used at least once. [2]
+ * -WORKINGSET-+|- Pages beyond are workingset and never fall below this floor. [3]
+ *   2      1<-+|= LRU_REFS_WORKINGSET: Classical workingset, accessed twice, protected. [4]
+ *   3      2   |- LRU_REFS_PROTECTED: Protected workingset, promoted pages capped at here. [5]
+ *   4*     2   |
+ *   5*     3   |- The tier here is MAX_NR_TIERS - 1
+ *   6*     3   |
+ *   7*     3   |= LRU_REFS_MAX: Promotion candidate. [6]
+ * -PROMOTION->-/
  *
- * MAX_NR_TIERS is set to 4 so that the multi-gen LRU can support twice the
- * number of categories of the active/inactive LRU when keeping track of
- * accesses through file descriptors. This uses MAX_NR_TIERS-2 spare bits in
- * folio->flags, masked by LRU_REFS_MASK.
+ * Ideally each tier holds folios of similar access patterns: lower tiers
+ * are less important and evicted faster.  A page's reference count and
+ * tier are capped when it changes generation, preventing it from
+ * dominating the new generation based on old-generation access history.
+ * Generation ordering already ensures a newer-gen page is hotter than an
+ * older-gen one regardless of tier.
+ *
+ * Refs tracks accesses from two sources: page table (lazily collected by
+ * the page table aging walk or rmap eviction lookup) and file descriptors
+ * (by folio_mark_accessed).  Page table accesses are weighted heavier
+ * because the accessed bit is sticky (undercounts repeated accesses),
+ * passively collected, and page faults are generally more important as
+ * userspace does not expect a memory access to block on reclaim.  Both
+ * access types increment refs by one; the result is capped at
+ * LRU_REFS_PROTECTED on promotion or deferral, or LRU_REFS_MAX otherwise.
+ *
+ * 1. Tier is fls(N-1) for N > 1, 0 otherwise.  Folios with zero
+ *    accesses (refs == 0) are generally cold, e.g. readahead folios.
+ *
+ *    Page table access advances a folio by one generation even at the
+ *    lowest refs or tier.  Freshly allocated folios start with refs == 0;
+ *    faulted and mapped folios have their page table access bit set, so
+ *    the first page table access check always sets LRU_REFS_REFERENCED and
+ *    moves them one generation forward, driving aging and workingset shift.
+ *
+ * 2. Folios accessed once stay on tier 0: one-time usage does not
+ *    qualify for protection.  A second access advances the folio,
+ *    aligning with classical LRU's use-twice threshold.  A second page
+ *    table access promotes to the latest gen; file access only defers
+ *    eviction from the oldest gen.
+ *
+ * 3. Folios accessed at least twice are considered workingset.  This
+ *    mostly aligns with classical LRU: at least one I/O is saved by
+ *    keeping them in memory.  Folios at or above this level never fall
+ *    below tier 1 (the workingset floor), so tier 0 stays a clean tier
+ *    for cold cache while tier 1 serves as the fallback line for
+ *    actually reused or historically hot folios.
+ *
+ *    Folios refaulted through a page fault at refs 1 will enter the second
+ *    newest gen, so faulting will be protected better.
+ *
+ * 4. Starting from tier 1, PID protection sacrifices lower tiers to
+ *    protect higher tiers by comparing refault rates for long-term
+ *    accuracy, and caps higher refs to this value.  Since PID protection
+ *    bypasses page table lookup and clearing, when a further eviction
+ *    attempt occurs after PID loosens, the folio's page table access is
+ *    rechecked and the folio is sent back to LRU_REFS_PROTECTED.  This
+ *    also gives folios a fair opportunity to be promoted by file access
+ *    again.
+ *
+ *    Folios refaulted through a page fault at tier 1 or above are activated
+ *    and enter the newest gen. Non fault page will enter second oldest gen,
+ *    driving aging and workingset shifting.
+ *
+ * 5. Pages beyond the ordinary workingset tier form new tiers for the
+ *    PID controller to protect differently.  Folios at or above this
+ *    level are capped at LRU_REFS_PROTECTED on promotion or deferral,
+ *    and at LRU_REFS_WORKINGSET under PID protection in the oldest
+ *    generation, where they represent a historical workingset.
+ *
+ * 6. Folios that reach LRU_REFS_MAX are advanced to the next generation
+ *    on further access, with refs capped to LRU_REFS_PROTECTED.  This
+ *    gives them a fair start for advancement to an even newer generation
+ *    while keeping hot folios distinguishable.
+ *
+ * Tiering uses PG_referenced and PG_workingset as the lower two bits,
+ * and the bits masked by LRU_REFS_MASK as the higher bits.
+ *
+ * A folio's referenced count never goes backwards except upon gen
+ * increase as described above.  Refault of a reclaimed folio restores
+ * its referenced count, capped at LRU_REFS_PROTECTED, which aligns with
+ * promotion.  Page table refaults of previous workingset folios send
+ * them to the latest gen, driving aging faster.
+ *
+ * MAX_NR_TIERS is set to 4 so that the multi-gen LRU can support twice
+ * the number of categories of the active/inactive LRU.
  */
 #define MAX_NR_TIERS		4U
+#define LRU_REFS_REFERENCED	0x1
+#define LRU_REFS_WORKINGSET	0x2
+#define LRU_REFS_PROTECTED	0x3
 
 #ifndef __GENERATING_BOUNDS_H
 
 #define LRU_GEN_MASK		((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)
 #define LRU_GEN_MAX		(BIT(LRU_GEN_WIDTH - 1) - 1)
 #define LRU_REFS_MASK		((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF)
-#define LRU_REFS_MAX		BIT(LRU_REFS_WIDTH)
-
-/*
- * For folios accessed multiple times through file descriptors,
- * lru_gen_inc_refs() sets additional bits of LRU_REFS_WIDTH in folio->flags
- * after PG_referenced, then PG_workingset after LRU_REFS_WIDTH. After all its
- * bits are set, i.e., LRU_REFS_FLAGS|BIT(PG_workingset), a folio is lazily
- * promoted into the second oldest generation in the eviction path. And when
- * folio_inc_gen() does that, it clears LRU_REFS_FLAGS so that
- * lru_gen_inc_refs() can start over. Note that for this case, LRU_REFS_MASK is
- * only valid when PG_referenced is set.
- *
- * For folios accessed multiple times through page tables, folio_update_gen()
- * from a page table walk or lru_gen_set_refs() from a rmap walk sets
- * PG_referenced after the accessed bit is cleared for the first time.
- * Thereafter, those two paths set PG_workingset and promote folios to the
- * youngest generation. Like folio_inc_gen(), folio_update_gen() also clears
- * PG_referenced. Note that for this case, LRU_REFS_MASK is not used.
- *
- * For both cases above, after PG_workingset is set on a folio, it remains until
- * this folio is either reclaimed, or "deactivated" by lru_gen_clear_refs(). It
- * can be set again if lru_gen_test_recent() returns true upon a refault.
- */
-#define LRU_REFS_FLAGS		(LRU_REFS_MASK | BIT(PG_referenced))
+#define LRU_REFS_FLAGS		(LRU_REFS_MASK | BIT(PG_referenced) | BIT(PG_workingset))
+#define LRU_REFS_MAX		(BIT(LRU_REFS_WIDTH + 2) - 1)
 
 struct lruvec;
 struct page_vma_mapped_walk;
diff --git a/kernel/bounds.c b/kernel/bounds.c
index 02b619eb6106..06a034713b5d 100644
--- a/kernel/bounds.c
+++ b/kernel/bounds.c
@@ -25,7 +25,7 @@ int main(void)
 	DEFINE(SPINLOCK_SIZE, sizeof(spinlock_t));
 #ifdef CONFIG_LRU_GEN
 	DEFINE(LRU_GEN_WIDTH, order_base_2(MAX_NR_GENS + 1));
-	DEFINE(__LRU_REFS_WIDTH, MAX_NR_TIERS - 2);
+	DEFINE(__LRU_REFS_WIDTH, MAX_NR_TIERS - 3);
 #else
 	DEFINE(LRU_GEN_WIDTH, 0);
 	DEFINE(__LRU_REFS_WIDTH, 0);
diff --git a/mm/folio.c b/mm/folio.c
index fab00cb02970..a326602a59fe 100644
--- a/mm/folio.c
+++ b/mm/folio.c
@@ -272,7 +272,6 @@ static void lru_activate(struct lruvec *lruvec, struct folio *folio)
 	if (folio_test_active(folio) || folio_test_unevictable(folio))
 		return;
 
-
 	lruvec_del_folio(lruvec, folio);
 	folio_set_active(folio);
 	lruvec_add_folio(lruvec, folio);
@@ -351,32 +350,6 @@ static void __lru_cache_activate_folio(struct folio *folio)
 
 #ifdef CONFIG_LRU_GEN
 
-static void lru_gen_inc_refs(struct folio *folio)
-{
-	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
-	int refs;
-
-	if (folio_test_unevictable(folio))
-		return;
-
-	/* see the comment on LRU_REFS_FLAGS */
-	if (!folio_lru_refs(folio)) {
-		folio_set_lru_refs(folio, 1);
-		return;
-	}
-
-	do {
-		new_flags = old_flags;
-		refs = lru_refs_from_flags(old_flags);
-		if (refs == LRU_REFS_MAX) {
-			if (!folio_test_workingset(folio))
-				folio_set_workingset(folio);
-			return;
-		}
-		lru_refs_set_flags(&new_flags, refs + 1);
-	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
-}
-
 static bool lru_gen_clear_refs(struct folio *folio)
 {
 	int gen = folio_lru_gen(folio);
@@ -387,7 +360,6 @@ static bool lru_gen_clear_refs(struct folio *folio)
 		return true;
 
 	folio_set_lru_refs(folio, 0);
-	folio_clear_workingset(folio);
 
 	rcu_read_lock();
 	seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
@@ -398,10 +370,6 @@ static bool lru_gen_clear_refs(struct folio *folio)
 
 #else /* !CONFIG_LRU_GEN */
 
-static void lru_gen_inc_refs(struct folio *folio)
-{
-}
-
 static bool lru_gen_clear_refs(struct folio *folio)
 {
 	return false;
@@ -427,7 +395,8 @@ void folio_mark_accessed(struct folio *folio)
 	if (folio_test_dropbehind(folio))
 		return;
 	if (lru_gen_enabled()) {
-		lru_gen_inc_refs(folio);
+		if (!folio_test_unevictable(folio))
+			folio_inc_lru_refs(folio, false, false);
 		return;
 	}
 
@@ -473,17 +442,6 @@ void folio_add_lru(struct folio *folio)
 			folio_test_unevictable(folio), folio);
 	VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
 
-	/*
-	 * For prefaulted file folios, folio_mark_accessed() sets
-	 * PG_referenced so lru_gen_folio_seq() places them into
-	 * the second oldest generation.
-	 */
-	if (lru_gen_enabled() && !folio_test_unevictable(folio) &&
-	    lru_gen_in_fault() && !(current->flags & PF_MEMALLOC)) {
-		if (!folio_test_referenced(folio) && !folio_test_workingset(folio))
-			folio_mark_accessed(folio);
-	}
-
 	folio_batch_add_and_move(folio, lru_add);
 }
 EXPORT_SYMBOL(folio_add_lru);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index a359d5a1ff41..9c8d9e3af375 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -830,38 +830,177 @@ enum folio_references {
 };
 
 #ifdef CONFIG_LRU_GEN
+/******************************************************************************
+ *                     Referenced count feedback
+ ******************************************************************************/
+
 /*
- * Only used on a mapped folio in the eviction (rmap walk) path, where promotion
- * needs to be done by taking the folio off the LRU list and then adding it back
- * with PG_active set. In contrast, the aging (page table walk) path uses
- * folio_update_gen().
+ * The folio_inc_lru_refs{_*} helpers below collect the referenced info
+ * (hotness) from other parts, including the page table walker, the rmap walk
+ * upon eviction, the rmap lookaround, and file descriptors
+ * (folio_mark_accessed).
+ *
+ * Page table accesses escalate a folio in two steps.  The first access
+ * advances it one generation; a second access sends it to the newest
+ * generation.  Executable file folios skip the first step and are promoted
+ * immediately, as reclaiming them causes IO thrashing.
+ *
+ * File descriptor accesses do not promote.  They only defer eviction from
+ * the oldest generation, and only once the folio is a workingset folio
+ * (LRU_REFS_WORKINGSET), leaving the rest to PID protection.  Page table
+ * accesses are treated more generously because the accessed bit is sticky
+ * (it under-counts repeated accesses) and because a page fault is more
+ * costly than file descriptor I/O.
+ *
+ * PID protection operates on tier > 0 folios.  The one proactive promotion
+ * outside of it and the page table path is the overflow case where the
+ * referenced count exceeds LRU_REFS_MAX, which means the folio is hotter
+ * than everything else in its generation.
+ *
+ * Whenever a folio changes generation here its referenced count is capped at
+ * LRU_REFS_PROTECTED, so it starts at or below the protected tier regardless
+ * of its old-generation access history.  PID protection (folio_inc_gen) caps
+ * at LRU_REFS_WORKINGSET independently.
  */
-static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags)
-{
-	/* see the comment on LRU_REFS_FLAGS */
-	if (!folio_test_referenced(folio) && !folio_test_workingset(folio)) {
-		/* Activate file-backed executable folios after first usage. */
-		if (is_exec_file_folio(folio, vma_flags)) {
-			folio_set_lru_refs(folio, 0);
-			folio_set_workingset(folio);
-			return true;
+
+/*
+ * Update the folio's lru refs indicator without taking the folio lock,
+ * isolation, or lruvec lock. Used by both page table access (@is_fault=true)
+ * and by file access (@is_fault=false).
+ */
+int folio_inc_lru_refs(struct folio *folio, bool is_fault, bool is_exec)
+{
+	int max_gen, min_gen;
+	int type, refs, gen, new_gen;
+	unsigned long new_flags, old_flags, max_seq;
+	struct lru_gen_folio *lrugen;
+	struct lruvec *lruvec;
+
+	type = folio_is_file_lru(folio);
+	lruvec = folio_lruvec_live_get(folio);
+	lrugen = &lruvec->lrugen;
+
+	old_flags = READ_ONCE(*folio_flags(folio, 0));
+	do {
+		new_flags = old_flags;
+		gen = lru_gen_from_flags(old_flags);
+		refs = lru_refs_from_flags(old_flags) + 1;
+		new_gen = gen;
+		if (!(old_flags & BIT(PG_lru)) || gen < 0)
+			goto out;
+
+		max_seq = READ_ONCE(lrugen->max_seq);
+		max_gen = lru_gen_from_seq(max_seq);
+		min_gen = lru_gen_from_seq(READ_ONCE(lrugen->min_seq[type]));
+		if (gen == max_gen)
+			goto out;
+
+		if (is_fault || is_exec) {
+			/* Promote second page table access or executable */
+			if (refs > LRU_REFS_REFERENCED || is_exec)
+				new_gen = max_gen;
+			else
+				new_gen = (gen + 1UL) % MAX_NR_GENS;
+			refs = min(refs, LRU_REFS_PROTECTED);
+		} else if (refs > LRU_REFS_MAX) {
+			/* LRU refs counting overflow, bump the gen */
+			new_gen = (gen + 1UL) % MAX_NR_GENS;
+			refs = LRU_REFS_PROTECTED;
+		} else if (gen == min_gen && refs >= LRU_REFS_WORKINGSET) {
+			/* Defer eviction of just accessed workingset */
+			new_gen = (gen + 1UL) % MAX_NR_GENS;
+			refs = min(refs, LRU_REFS_PROTECTED);
 		}
+out:
+		refs = min(refs, LRU_REFS_MAX);
+		lru_refs_set_flags(&new_flags, refs);
+		if (new_gen >= 0)
+			lru_gen_set_flags(&new_flags, new_gen);
+	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
 
-		folio_set_lru_refs(folio, 1);
-		return false;
+	if (new_gen != gen) {
+		/*
+		 * Gen can only go forward, so concurrent aging is
+		 * usually fine, except when multiple aging increase
+		 * max_seq multiple times, new_gen may have go beyond
+		 * the new max_seq's current gen border and causes
+		 * hotness inversion. In that very unlikely case,
+		 * just activate the folio.
+		 */
+		lru_gen_update_size(lruvec, folio, gen, new_gen);
+		if (unlikely(READ_ONCE(lrugen->max_seq) - max_seq > MIN_NR_GENS))
+			folio_activate(folio);
 	}
 
-	/* Promote on second access */
-	if (folio_lru_refs(folio) > 1) {
-		folio_set_lru_refs(folio, 0);
-		folio_set_workingset(folio);
-	} else {
-		folio_mark_accessed(folio);
-	}
-	return true;
+	folio_lruvec_live_put(lruvec);
+	return refs;
+}
+
+/*
+ * Update the folio's lru refs indicator during a page table walk.
+ * max_seq is stable since this runs inside the aging process.
+ *
+ * Returns the old generation and stores the new generation in @new_gen when
+ * the folio is promoted (to max_gen) or advanced by one generation.
+ * Returns -1 if no gen change occurred.
+ */
+static int folio_inc_lru_refs_walk(struct folio *folio, struct lruvec *lruvec,
+				   const vma_flags_t *vma_flags, int *new_gen)
+{
+	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
+	unsigned long max_seq = READ_ONCE(lruvec->lrugen.max_seq);
+	int refs, gen, max_gen, ret;
+
+	max_gen = lru_gen_from_seq(max_seq);
+
+	do {
+		gen = lru_gen_from_flags(old_flags);
+		refs = lru_refs_from_flags(old_flags) + 1;
+		new_flags = old_flags;
+
+		if (gen >= 0 && gen != max_gen) {
+			ret = gen;
+			/* Promote second page table access or executable */
+			if (refs > LRU_REFS_REFERENCED || is_exec_file_folio(folio, vma_flags))
+				*new_gen = max_gen;
+			else
+				*new_gen = (gen + 1) % MAX_NR_GENS;
+			lru_gen_set_flags(&new_flags, *new_gen);
+			lru_refs_set_flags(&new_flags, min(refs, LRU_REFS_PROTECTED));
+		} else {
+			ret = -1;
+			lru_refs_set_flags(&new_flags, min(refs, LRU_REFS_MAX));
+		}
+	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
+
+	return ret;
+}
+
+/*
+ * Update the folio's lru refs indicator while the folio is isolated.
+ * Only used on mapped folios upon the final eviction, when the folio is
+ * off the LRU list (isolated).
+ *
+ * Increments the refs count (capped at LRU_REFS_PROTECTED).  Returns true
+ * if the caller should activate the folio (second access or
+ * executable), false to keep it in the eviction list.
+ */
+static bool folio_inc_lru_refs_isolated(struct folio *folio, const vma_flags_t *vma_flags)
+{
+	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
+	int refs;
+
+	do {
+		new_flags = old_flags;
+		refs = lru_refs_from_flags(old_flags) + 1;
+		lru_refs_set_flags(&new_flags, min(refs, LRU_REFS_PROTECTED));
+	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
+
+	/* Promote second page table access or executable */
+	return refs > LRU_REFS_REFERENCED || is_exec_file_folio(folio, vma_flags);
 }
 #else
-static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags)
+static bool folio_inc_lru_refs_isolated(struct folio *folio, const vma_flags_t *vma_flags)
 {
 	return false;
 }
@@ -896,7 +1035,8 @@ static enum folio_references folio_check_references(struct folio *folio,
 		if (!referenced_ptes)
 			return FOLIOREF_RECLAIM;
 
-		return lru_gen_set_refs(folio, &vma_flags) ? FOLIOREF_ACTIVATE : FOLIOREF_KEEP;
+		return folio_inc_lru_refs_isolated(folio, &vma_flags) ?
+		       FOLIOREF_ACTIVATE : FOLIOREF_KEEP;
 	}
 
 	referenced_folio = folio_test_clear_referenced(folio);
@@ -3262,59 +3402,31 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv)
  *                          the aging
  ******************************************************************************/
 
-/* promote pages accessed through page tables */
-static int folio_update_gen(struct folio *folio, int new_gen, const vma_flags_t *vma_flags)
-{
-	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
-	int old_gen;
-
-	/*
-	 * See the comment on LRU_REFS_FLAGS, and activate file-backed
-	 * executable folios after first usage to avoid typical IO
-	 * thrashing from reclaiming.
-	 */
-	if (!folio_test_referenced(folio) && !folio_test_workingset(folio) &&
-	    !is_exec_file_folio(folio, vma_flags)) {
-		folio_set_lru_refs(folio, 1);
-		return -1;
-	}
-
-	do {
-		old_gen = lru_gen_from_flags(old_flags);
-		new_flags = old_flags;
-
-		/* lru_gen_del_folio() has isolated this page? */
-		if (old_gen < 0)
-			break;
-
-		lru_gen_set_flags(&new_flags, new_gen);
-		lru_refs_set_flags(&new_flags, 0);
-		new_flags |= BIT(PG_workingset);
-	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
-
-	return old_gen;
-}
-
-/* protect pages accessed multiple times through file descriptors */
+/*
+ * Force bump a folio's generation. Used for PID protection or defer the
+ * eviction of temporarily unevictable folio.
+ */
 static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
 {
+	int refs;
 	int type = folio_is_file_lru(folio);
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
 	int old_gen, new_gen, min_gen = lru_gen_from_seq(lrugen->min_seq[type]);
 	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
 
 	do {
+		new_flags = old_flags;
+		refs = lru_refs_from_flags(old_flags);
 		old_gen = lru_gen_from_flags(old_flags);
 		VM_WARN_ON_ONCE_FOLIO(old_gen < 0, folio);
 
-		/* folio_update_gen() has promoted this page? */
+		/* folio has been promoted? */
 		if (old_gen >= 0 && old_gen != min_gen)
 			return old_gen;
 
-		new_flags = old_flags;
 		new_gen = (old_gen + 1) % MAX_NR_GENS;
 		lru_gen_set_flags(&new_flags, new_gen);
-		lru_refs_set_flags(&new_flags, 0);
+		lru_refs_set_flags(&new_flags, min(refs, LRU_REFS_WORKINGSET));
 	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
 
 	lru_gen_update_size(lruvec, folio, old_gen, new_gen);
@@ -3518,21 +3630,17 @@ static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struc
 	if (!folio)
 		return;
 
-	new_gen = lru_gen_from_seq(READ_ONCE(lruvec->lrugen.max_seq));
-
 	if (dirty && !folio_test_dirty(folio) &&
 	    !(folio_test_anon(folio) && folio_test_swapbacked(folio) &&
 	      !folio_test_swapcache(folio)))
 		folio_mark_dirty(folio);
 
 	if (walk) {
-		old_gen = folio_update_gen(folio, new_gen, &vma->flags);
-		if (old_gen >= 0 && old_gen != new_gen)
+		old_gen = folio_inc_lru_refs_walk(folio, lruvec, &vma->flags, &new_gen);
+		if (old_gen >= 0)
 			update_batch_size(walk, folio, old_gen, new_gen);
-	} else if (lru_gen_set_refs(folio, &vma->flags)) {
-		old_gen = folio_lru_gen(folio);
-		if (old_gen >= 0 && old_gen != new_gen)
-			folio_activate(folio);
+	} else {
+		folio_inc_lru_refs(folio, true, is_exec_file_folio(folio, &vma->flags));
 	}
 }
 
@@ -3917,7 +4025,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 		while (!list_empty(head)) {
 			struct folio *folio = lru_to_folio(head);
 			int refs = folio_lru_refs(folio);
-			bool workingset = folio_test_workingset(folio);
+			int delta = folio_nr_pages(folio);
+			int tier = lru_tier_from_refs(refs);
 
 			VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio);
@@ -3927,14 +4036,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			new_gen = folio_inc_gen(lruvec, folio);
 			list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
 
-			/* don't count the workingset being lazily promoted */
-			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
-				int tier = lru_tier_from_refs(refs, workingset);
-				int delta = folio_nr_pages(folio);
-
-				WRITE_ONCE(lrugen->protected[hist][type][tier],
-					   lrugen->protected[hist][type][tier] + delta);
-			}
+			WRITE_ONCE(lrugen->protected[hist][type][tier],
+				   lrugen->protected[hist][type][tier] + delta);
 
 			if (!--remaining)
 				return false;
@@ -4649,8 +4752,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c
 	int zone = folio_zonenum(folio);
 	int delta = folio_nr_pages(folio);
 	int refs = folio_lru_refs(folio);
-	bool workingset = folio_test_workingset(folio);
-	int tier = lru_tier_from_refs(refs, workingset);
+	int tier = lru_tier_from_refs(refs);
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
 
 	VM_WARN_ON_ONCE_FOLIO(gen >= MAX_NR_GENS, folio);
@@ -4672,17 +4774,15 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c
 	}
 
 	/* protected */
-	if (tier > tier_idx || refs + workingset == BIT(LRU_REFS_WIDTH) + 1) {
+	if (tier > tier_idx) {
+		int hist = lru_hist_from_seq(lrugen->min_seq[type]);
+
 		gen = folio_inc_gen(lruvec, folio);
 		list_move(&folio->lru, &lrugen->folios[gen][type][zone]);
 
-		/* don't count the workingset being lazily promoted */
-		if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
-			int hist = lru_hist_from_seq(lrugen->min_seq[type]);
+		WRITE_ONCE(lrugen->protected[hist][type][tier],
+			   lrugen->protected[hist][type][tier] + delta);
 
-			WRITE_ONCE(lrugen->protected[hist][type][tier],
-				   lrugen->protected[hist][type][tier] + delta);
-		}
 		return true;
 	}
 
@@ -4710,10 +4810,6 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca
 		return false;
 	}
 
-	/* see the comment on LRU_REFS_FLAGS */
-	if (!folio_test_referenced(folio))
-		folio_set_lru_refs(folio, 0);
-
 	success = lru_gen_del_folio(lruvec, folio, true);
 	VM_WARN_ON_ONCE_FOLIO(!success, folio);
 
@@ -4801,13 +4897,13 @@ static int get_tier_idx(struct lruvec *lruvec, int type)
 	struct ctrl_pos sp, pv = {};
 
 	/*
-	 * To leave a margin for fluctuations, use a larger gain factor (2:3).
+	 * To leave a margin for fluctuations, use a larger gain factor (1:2).
 	 * This value is chosen because any other tier would have at least twice
 	 * as many refaults as the first tier.
 	 */
-	read_ctrl_pos(lruvec, type, 0, 1, 2, &sp);
 	for (tier = 1; tier < MAX_NR_TIERS; tier++) {
-		read_ctrl_pos(lruvec, type, tier, tier + 1, 3, &pv);
+		read_ctrl_pos(lruvec, type, 0, tier, 1, &sp);
+		read_ctrl_pos(lruvec, type, tier, tier + 1, 2, &pv);
 		if (!positive_ctrl_err(&sp, &pv))
 			break;
 	}
@@ -4930,10 +5026,8 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 		}
 
 		/* don't add rejected folios to the oldest generation */
-		if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type]) {
-			folio_set_lru_refs(folio, 0);
+		if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type])
 			folio_set_active(folio);
-		}
 	}
 
 	move_folios_to_lru(&list);
diff --git a/mm/workingset.c b/mm/workingset.c
index 5438e9390011..452fe8554990 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -189,6 +189,13 @@
 #define EVICTION_MASK	(~0UL >> EVICTION_SHIFT)
 #define EVICTION_MASK_ANON	(~0UL >> EVICTION_SHIFT_ANON)
 
+/*
+ * LRU refs uses LRU_REFS_WIDTH + 2 bits, the 2 bits are PG_workingset and
+ * PG_referenced. But here we record PG_workingset separately (to reuse
+ * pack_shadow).
+ */
+#define LRU_REFS_BITS ((LRU_REFS_WIDTH + 2) - 1)
+
 /*
  * Eviction timestamps need to be able to cover the full range of
  * actionable refaults. However, bits are tight in the xarray
@@ -242,13 +249,12 @@ static void *lru_gen_eviction(struct folio *folio)
 	int type = folio_is_file_lru(folio);
 	int delta = folio_nr_pages(folio);
 	int refs = folio_lru_refs(folio);
-	bool workingset = folio_test_workingset(folio);
-	int tier = lru_tier_from_refs(refs, workingset);
+	int tier = lru_tier_from_refs(refs);
 	struct mem_cgroup *memcg;
 	struct pglist_data *pgdat = folio_pgdat(folio);
 	unsigned short memcg_id;
 
-	BUILD_BUG_ON(LRU_GEN_WIDTH + LRU_REFS_WIDTH >
+	BUILD_BUG_ON(LRU_GEN_WIDTH + LRU_REFS_BITS >
 		     BITS_PER_LONG - max(EVICTION_SHIFT, EVICTION_SHIFT_ANON));
 
 	rcu_read_lock();
@@ -256,14 +262,14 @@ static void *lru_gen_eviction(struct folio *folio)
 	lruvec = mem_cgroup_lruvec(memcg, pgdat);
 	lrugen = &lruvec->lrugen;
 	min_seq = READ_ONCE(lrugen->min_seq[type]);
-	token = (min_seq << LRU_REFS_WIDTH) | max(refs - 1, 0);
+	token = (min_seq << LRU_REFS_BITS) | refs >> 1;
 
 	hist = lru_hist_from_seq(min_seq);
 	atomic_long_add(delta, &lrugen->evicted[hist][type][tier]);
 	memcg_id = mem_cgroup_private_id(memcg);
 	rcu_read_unlock();
 
-	return pack_shadow(memcg_id, pgdat, token, workingset, type);
+	return pack_shadow(memcg_id, pgdat, token, refs & 1, type);
 }
 
 /*
@@ -284,11 +290,24 @@ static bool lru_gen_test_recent(void *shadow, struct lruvec **lruvec,
 	*lruvec = mem_cgroup_lruvec(memcg, pgdat);
 
 	max_seq = READ_ONCE((*lruvec)->lrugen.max_seq);
-	max_seq &= (file ? EVICTION_MASK : EVICTION_MASK_ANON) >> LRU_REFS_WIDTH;
+	max_seq &= (file ? EVICTION_MASK : EVICTION_MASK_ANON) >> LRU_REFS_BITS;
 
-	return abs_diff(max_seq, *token >> LRU_REFS_WIDTH) < MAX_NR_GENS;
+	return abs_diff(max_seq, *token >> LRU_REFS_BITS) < MAX_NR_GENS;
 }
 
+/*
+ * Restore the refs of a refaulted folio from its shadow entry.
+ *
+ * Any folio that was accessed at least once before eviction (refs >=
+ * LRU_REFS_REFERENCED) is activated on a fault-driven refault, giving it a
+ * strong gen placement. Non-fault refaults (e.g. readahead) are not
+ * activated regardless of refs.
+ *
+ * The restored refs is capped at LRU_REFS_PROTECTED to prevent stale
+ * high-tier history from carrying over across eviction cycles. The
+ * WORKINGSET_RESTORE stat is bumped only for refs >= LRU_REFS_WORKINGSET
+ * to track genuine workingset restoration.
+ */
 static void lru_gen_refault(struct folio *folio, void *shadow)
 {
 	bool recent;
@@ -314,21 +333,29 @@ static void lru_gen_refault(struct folio *folio, void *shadow)
 	lrugen = &lruvec->lrugen;
 
 	hist = lru_hist_from_seq(READ_ONCE(lrugen->min_seq[type]));
-	refs = (token & (BIT(LRU_REFS_WIDTH) - 1)) + 1;
-	tier = lru_tier_from_refs(refs, workingset);
+	refs = ((token & (BIT(LRU_REFS_BITS) - 1)) << 1) + workingset;
+	tier = lru_tier_from_refs(refs);
 
 	atomic_long_add(delta, &lrugen->refaulted[hist][type][tier]);
 
-	if (workingset) {
-		/* Send refaulted workingset folios to active generations. */
+	/*
+	 * Activate a fault-driven refault: the folio was accessed at
+	 * least once before eviction and would have been promoted had
+	 * it stayed in memory.
+	 */
+	if (refs >= LRU_REFS_REFERENCED) {
 		if (lru_gen_in_fault()) {
 			folio_set_active(folio);
 			mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
 		}
-		folio_set_workingset(folio);
+		/* Cap restored refs to prevent stale high-tier carry-over */
+		folio_set_lru_refs(folio, min(refs, LRU_REFS_PROTECTED));
+	}
+
+	/* WORKINGSET_RESTORE tracks genuine workingset-level refaults */
+	if (refs >= LRU_REFS_WORKINGSET)
 		mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + type, delta);
-	} else
-		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, (refs - 1UL) << LRU_REFS_PGOFF);
+
 unlock:
 	rcu_read_unlock();
 }

-- 
2.55.0

[PATCH RFC 10/15] mm/mglru: make folio lru referenced times count a generic API

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

To prepare for unifying the API for checking folio referenced status,
expose the referenced times counting as a generic API. For MGLRU this
helps to adapt other subsystem based on the referenced times counting,
for non-MGLRU this is still bitwise compatible and there won't be
major behavior change.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/mm_inline.h | 233 ++++++++++++++++++++++++++++++----------------
 mm/migrate.c              |   2 -
 2 files changed, 155 insertions(+), 80 deletions(-)

diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 944baa91bf18..a13b7d3c033a 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -94,6 +94,161 @@ static __always_inline enum lru_list folio_lru_list(const struct folio *folio)
 	return lru;
 }
 
+/**
+ * lru_refs_from_flags - Return LRU referenced / access count from folio flags.
+ * @flags: folio flags
+ */
+static inline int lru_refs_from_flags(unsigned long flags)
+{
+	int refs;
+
+	/*
+	 * Return the total number of accesses. Also see the comment on
+	 * LRU_REFS_FLAGS.
+	 */
+	refs = (flags & BIT(PG_referenced)) ? BIT(0) : 0;
+	refs += (flags & BIT(PG_workingset)) ? BIT(1) : 0;
+	refs += ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) << 2;
+	return refs;
+}
+
+/**
+ * lru_refs_set_flags - Set the LRU referenced / access count to specified folio flags.
+ * @flags: pointer to the folio flags
+ * @refs: referenced / access count number, between 0 and LRU_REFS_MAX, inclusive.
+ */
+static inline void lru_refs_set_flags(unsigned long *flags, unsigned int refs)
+{
+	VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
+	BUILD_BUG_ON((LRU_REFS_MAX >> 2) > (BIT(LRU_REFS_WIDTH) - 1));
+	*flags &= ~LRU_REFS_FLAGS;
+	if (refs & BIT(0))
+		*flags |= BIT(PG_referenced);
+	if (refs & BIT(1))
+		*flags |= BIT(PG_workingset);
+	*flags |= (((unsigned long)refs) >> 2) << LRU_REFS_PGOFF;
+}
+
+static inline int folio_lru_refs(const struct folio *folio)
+{
+	return lru_refs_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
+}
+
+static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
+{
+	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
+
+	do {
+		new_flags = old_flags;
+		lru_refs_set_flags(&new_flags, refs);
+	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
+}
+
+int folio_inc_lru_refs(struct folio *folio, bool is_fault, bool is_exec);
+
+/**
+ * folio_is_referenced - Tell if a folio was accessed before.
+ * @folio: the folio.
+ *
+ * This helper currently only works as intended for MGLRU, as it checks
+ * all LRU_REFS_FLAGS. It might be fine for non-MGLRU to replace
+ * folio_test_referenced in some cases but the user should be careful.
+ *
+ * Returns: true if the folio's LRU referenced / accessed count > 0.
+ */
+static inline bool folio_is_referenced(const struct folio *folio)
+{
+	return folio_lru_refs(folio) >= LRU_REFS_REFERENCED;
+}
+
+/**
+ * folio_mark_referenced - Mark a folio as referenced.
+ * @folio: the folio.
+ *
+ * Ensures the folio's LRU referenced count is at least
+ * LRU_REFS_REFERENCED. Won't do anything if the count is already larger
+ * than that. This helper currently only works as intended for MGLRU.
+ * Not a drop-in replacement, but should be fine for non-MGLRU to replace
+ * folio_set_referenced with this after audit.
+ */
+static inline void folio_mark_referenced(struct folio *folio)
+{
+	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
+
+	do {
+		new_flags = old_flags;
+		if (lru_refs_from_flags(new_flags) >= LRU_REFS_REFERENCED)
+			return;
+		lru_refs_set_flags(&new_flags, LRU_REFS_REFERENCED);
+	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
+}
+
+/**
+ * folio_mark_referenced_by_bit - Mark a folio as referenced by bit.
+ * @folio: the folio.
+ *
+ * non-MGLRU may want to make use of the lowest LRU referenced count bit
+ * explicitly as a referenced mark.
+ */
+static inline void folio_mark_referenced_by_bit(struct folio *folio)
+{
+	set_bit(PG_referenced, folio_flags(folio, 0));
+}
+
+/**
+ * folio_clear_referenced_by_bit - Clear the referenced bit of a folio.
+ * @folio: the folio.
+ */
+static inline void folio_clear_referenced_by_bit(struct folio *folio)
+{
+	clear_bit(PG_referenced, folio_flags(folio, 0));
+}
+
+/**
+ * folio_test_clear_referenced_by_bit - Test and clear the referenced bit
+ * @folio: the folio.
+ */
+static inline bool folio_test_clear_referenced_by_bit(struct folio *folio)
+{
+	return test_and_clear_bit(PG_referenced, folio_flags(folio, 0));
+}
+
+/**
+ * folio_is_referenced_by_bit - Test if the referenced bit of a folio is set.
+ * @folio: the folio.
+ */
+static inline bool folio_is_referenced_by_bit(const struct folio *folio)
+{
+	return test_bit(PG_referenced, const_folio_flags(folio, 0));
+}
+
+/**
+ * folio_is_workingset - Tell if a folio is part of the workingset.
+ * @folio: the folio.
+ *
+ * Can be used to replace folio_test_workingset safely. For MGLRU the LRU
+ * referenced count tells if a folio is a workingset as intended. For non-MGLRU,
+ * the check below only holds true if the PG_workingset bit is set.
+ */
+static inline bool folio_is_workingset(const struct folio *folio)
+{
+	return folio_lru_refs(folio) >= LRU_REFS_WORKINGSET;
+}
+
+/**
+ * folio_mark_workingset_by_bit - Set the workingset bit of a folio.
+ * @folio: the folio.
+ */
+static inline void folio_mark_workingset_by_bit(struct folio *folio)
+{
+	set_bit(PG_workingset, folio_flags(folio, 0));
+}
+
+static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
+{
+	folio_set_lru_refs(new, folio_lru_refs(old));
+}
+
 #ifdef CONFIG_LRU_GEN
 
 static inline bool lru_gen_switching(void)
@@ -171,58 +326,6 @@ static inline void lru_gen_set_flags(unsigned long *flags, int gen)
 	*flags |= (gen + 1UL) << LRU_GEN_PGOFF;
 }
 
-/**
- * lru_refs_from_flags - Return LRU referenced / access count from folio flags.
- * @flags: folio flags
- */
-static inline int lru_refs_from_flags(unsigned long flags)
-{
-	int refs;
-
-	/*
-	 * Return the total number of accesses. Also see the comment on
-	 * LRU_REFS_FLAGS.
-	 */
-	refs = (flags & BIT(PG_referenced)) ? BIT(0) : 0;
-	refs += (flags & BIT(PG_workingset)) ? BIT(1) : 0;
-	refs += ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) << 2;
-	return refs;
-}
-
-/**
- * lru_refs_set_flags - Set the LRU referenced / access count to specified folio flags.
- * @flags: pointer to the folio flags
- * @refs: referenced / access count number, between 0 and LRU_REFS_MAX, inclusive.
- */
-static inline void lru_refs_set_flags(unsigned long *flags, unsigned int refs)
-{
-	VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
-	BUILD_BUG_ON((LRU_REFS_MAX >> 2) > (BIT(LRU_REFS_WIDTH) - 1));
-	*flags &= ~LRU_REFS_FLAGS;
-	if (refs & BIT(0))
-		*flags |= BIT(PG_referenced);
-	if (refs & BIT(1))
-		*flags |= BIT(PG_workingset);
-	*flags |= (((unsigned long)refs) >> 2) << LRU_REFS_PGOFF;
-}
-
-static inline int folio_lru_refs(const struct folio *folio)
-{
-	return lru_refs_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
-}
-
-static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
-{
-	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
-
-	do {
-		new_flags = old_flags;
-		lru_refs_set_flags(&new_flags, refs);
-	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
-}
-
-int folio_inc_lru_refs(struct folio *folio, bool is_fault, bool is_exec);
-
 static inline int folio_lru_gen(const struct folio *folio)
 {
 	return lru_gen_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
@@ -369,11 +472,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
 	return true;
 }
 
-static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
-{
-	folio_set_lru_refs(new, folio_lru_refs(old));
-}
-
 #else /* !CONFIG_LRU_GEN */
 
 static inline bool lru_gen_enabled(void)
@@ -401,27 +499,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
 	return false;
 }
 
-static inline int folio_lru_refs(const struct folio *folio)
-{
-	return 0;
-}
-
-static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
-{
-}
-
-static inline int folio_inc_lru_refs(struct folio *folio, bool promote, bool is_exec)
-{
-	return 0;
-}
-
-static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
-{
-	if (folio_test_referenced(old))
-		folio_set_referenced(new);
-	if (folio_test_workingset(old))
-		folio_set_workingset(new);
-}
 #endif /* CONFIG_LRU_GEN */
 
 static __always_inline
diff --git a/mm/migrate.c b/mm/migrate.c
index c737d0682fa4..806f1e913a38 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -786,8 +786,6 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
 		folio_set_active(newfolio);
 	} else if (folio_test_clear_unevictable(folio))
 		folio_set_unevictable(newfolio);
-	if (folio_test_workingset(folio))
-		folio_set_workingset(newfolio);
 	if (folio_test_checked(folio))
 		folio_set_checked(newfolio);
 	/*

-- 
2.55.0

[PATCH RFC 11/15] mm/mglru: replace folio workinset check and update with new helper

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

With the new folio LRU refs tracking API, when MGLRU enabled, a folio
is considered a workingset folio if its referenced count > 1. This is
compatible with classical LRU and reasonable in many ways:

The PG_referenced and PG_workingset are used as the lower bits of the
LRU refs counter, and MGLRU will make use of extra bits as higher bits.

So when MGLRU is disabled, all higher bits are always 0, making the
check bit-wise equal to the old behavior. Active/inactive LRU sets
PG_workingset explicitly for folios moved from active list to inactive
list, and that makes the LRU refs tracking API (folio_is_workingset)
report a referenced number > 1. Clearing PG_workingset will always
return a value <= 1.

When MGLRU is enabled, a folio referenced twice is considered a
workingset folio, which is basically the same as how active/inactive LRU
used to promote a file page to the active list. Note for
active/inactive, the folio has to be marked inactive and PG_workingset
before eviction, but MGLRU doesn't have a demotion process, so this
simplified check ensures we have a stable definition and accurate
readings in PSI and readaheads just like before.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 fs/btrfs/compression.c | 3 ++-
 mm/filemap.c           | 8 ++++----
 mm/madvise.c           | 4 ++--
 mm/page_io.c           | 3 ++-
 mm/readahead.c         | 8 ++++----
 mm/vmscan.c            | 2 +-
 mm/workingset.c        | 4 ++--
 7 files changed, 17 insertions(+), 15 deletions(-)

diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c
index ffb6b52863a7..e756403e8bd5 100644
--- a/fs/btrfs/compression.c
+++ b/fs/btrfs/compression.c
@@ -21,6 +21,7 @@
 #include <linux/sched/mm.h>
 #include <linux/log2.h>
 #include <linux/shrinker.h>
+#include <linux/mm_inline.h>
 #include "misc.h"
 #include "ctree.h"
 #include "fs.h"
@@ -448,7 +449,7 @@ static noinline int add_ra_bio_folios(struct inode *inode, u64 compressed_end,
 			continue;
 		}
 
-		if (!*memstall && folio_test_workingset(folio)) {
+		if (!*memstall && folio_is_workingset(folio)) {
 			psi_memstall_enter(pflags);
 			*memstall = 1;
 		}
diff --git a/mm/filemap.c b/mm/filemap.c
index 6afec636881f..a88a6140ed09 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1259,7 +1259,7 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr,
 	bool in_thrashing;
 
 	if (bit_nr == PG_locked &&
-	    !folio_test_uptodate(folio) && folio_test_workingset(folio)) {
+	    !folio_test_uptodate(folio) && folio_is_workingset(folio)) {
 		delayacct_thrashing_start(&in_thrashing);
 		psi_memstall_enter(&pflags);
 		thrashing = true;
@@ -1414,7 +1414,7 @@ void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *ptl)
 	struct folio *folio = softleaf_to_folio(entry);
 
 	q = folio_waitqueue(folio);
-	if (!folio_test_uptodate(folio) && folio_test_workingset(folio)) {
+	if (!folio_test_uptodate(folio) && folio_is_workingset(folio)) {
 		delayacct_thrashing_start(&in_thrashing);
 		psi_memstall_enter(&pflags);
 		thrashing = true;
@@ -2510,7 +2510,7 @@ static void filemap_get_read_batch(struct address_space *mapping,
 static int filemap_read_folio(struct file *file, filler_t filler,
 		struct folio *folio)
 {
-	bool workingset = folio_test_workingset(folio);
+	bool workingset = folio_is_workingset(folio);
 	unsigned long pflags;
 	int error;
 
@@ -3981,7 +3981,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,
 		 */
 		if ((map_ret & VM_FAULT_NOPAGE) &&
 		    !(vmf->flags & FAULT_FLAG_TRIED) &&
-		    !folio_test_workingset(folio) &&
+		    !folio_is_workingset(folio) &&
 		    !(vma->vm_flags & (VM_SEQ_READ | VM_EXEC))) {
 			unsigned short mmap_miss;
 
diff --git a/mm/madvise.c b/mm/madvise.c
index 07a21ca31bad..abb17760b8b5 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -427,7 +427,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 		folio_clear_referenced(folio);
 		folio_test_clear_young(folio);
 		if (folio_test_active(folio))
-			folio_set_workingset(folio);
+			folio_mark_workingset_by_bit(folio);
 		if (pageout) {
 			if (folio_isolate_lru(folio)) {
 				if (folio_test_unevictable(folio))
@@ -542,7 +542,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 		folio_clear_referenced(folio);
 		folio_test_clear_young(folio);
 		if (folio_test_active(folio))
-			folio_set_workingset(folio);
+			folio_mark_workingset_by_bit(folio);
 		if (pageout) {
 			if (folio_isolate_lru(folio)) {
 				if (folio_test_unevictable(folio))
diff --git a/mm/page_io.c b/mm/page_io.c
index e4fa7ffffe8b..e7efc5bff668 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -25,6 +25,7 @@
 #include <linux/sched/task.h>
 #include <linux/delayacct.h>
 #include <linux/zswap.h>
+#include <linux/mm_inline.h>
 #include "swap.h"
 #include "swap_table.h"
 
@@ -452,7 +453,7 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
 {
 	struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
 	bool synchronous = sis->flags & SWP_SYNCHRONOUS_IO;
-	bool workingset = folio_test_workingset(folio);
+	bool workingset = folio_is_workingset(folio);
 	unsigned long pflags;
 	bool in_thrashing;
 
diff --git a/mm/readahead.c b/mm/readahead.c
index 558c92957518..3ab796af6490 100644
--- a/mm/readahead.c
+++ b/mm/readahead.c
@@ -302,7 +302,7 @@ void page_cache_ra_unbounded(struct readahead_control *ractl,
 		}
 		if (i == mark)
 			folio_set_readahead(folio);
-		ractl->_workingset |= folio_test_workingset(folio);
+		ractl->_workingset |= folio_is_workingset(folio);
 		ractl->_nr_pages += min_nrpages;
 		i += min_nrpages;
 	}
@@ -474,7 +474,7 @@ static inline int ra_alloc_folio(struct readahead_control *ractl, pgoff_t index,
 	}
 
 	ractl->_nr_pages += 1UL << order;
-	ractl->_workingset |= folio_test_workingset(folio);
+	ractl->_workingset |= folio_is_workingset(folio);
 	return 0;
 }
 
@@ -817,7 +817,7 @@ void readahead_expand(struct readahead_control *ractl,
 			folio_put(folio);
 			return;
 		}
-		if (unlikely(folio_test_workingset(folio)) &&
+		if (unlikely(folio_is_workingset(folio)) &&
 				!ractl->_workingset) {
 			ractl->_workingset = true;
 			psi_memstall_enter(&ractl->_pflags);
@@ -846,7 +846,7 @@ void readahead_expand(struct readahead_control *ractl,
 			folio_put(folio);
 			return;
 		}
-		if (unlikely(folio_test_workingset(folio)) &&
+		if (unlikely(folio_is_workingset(folio)) &&
 				!ractl->_workingset) {
 			ractl->_workingset = true;
 			psi_memstall_enter(&ractl->_pflags);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 9c8d9e3af375..913e69eae534 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2268,7 +2268,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
 		}
 
 		folio_clear_active(folio);	/* we are de-activating */
-		folio_set_workingset(folio);
+		folio_mark_workingset_by_bit(folio);
 		list_add(&folio->lru, &l_inactive);
 	}
 
diff --git a/mm/workingset.c b/mm/workingset.c
index 452fe8554990..568a3e44cd5e 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -438,7 +438,7 @@ void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg)
 	eviction >>= bucket_order[file];
 	workingset_age_nonresident(lruvec, folio_nr_pages(folio));
 	return pack_shadow(memcgid, pgdat, eviction,
-			   folio_test_workingset(folio), file);
+			   folio_is_workingset(folio), file);
 }
 
 /**
@@ -609,7 +609,7 @@ void workingset_refault(struct folio *folio, void *shadow)
 
 	/* Folio was active prior to eviction */
 	if (workingset) {
-		folio_set_workingset(folio);
+		folio_mark_workingset_by_bit(folio);
 		mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr);
 	}
 out:

-- 
2.55.0

[PATCH RFC 12/15] mm/smap: report workingset folios as referenced

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

For MGLRU, switch smap to use the folio refs count API so smap will
report all folio with referenced count >= 1 as "Referenced". Current
smap checking PG_referenced is causing folios to flick between
referenced and not-reference status, because for both MGLRU and
active/inactive LRU, PG_referenced may got cleared on second access.
(Increase of LRU referenced times count for MGLRU, and movig to active
list active/inactive all clears that bit).

After this, we will have a more reliable and useful reading for MGLRU.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 fs/proc/task_mmu.c | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 817e3e0f9194..c5c96c523291 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -944,6 +944,22 @@ static void smaps_page_accumulate(struct mem_size_stats *mss,
 	}
 }
 
+static bool smap_check_folio_referenced(struct folio *folio)
+{
+	if (lru_gen_enabled())
+		return folio_is_referenced(folio);
+	else
+		return folio_is_referenced_by_bit(folio);
+}
+
+static void smap_clear_folio_referenced(struct folio *folio)
+{
+	if (lru_gen_enabled())
+		folio_set_lru_refs(folio, 0);
+	else
+		folio_clear_referenced_by_bit(folio);
+}
+
 static void smaps_account(struct mem_size_stats *mss, struct page *page,
 		bool compound, bool young, bool dirty, bool locked,
 		bool present)
@@ -970,7 +986,7 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,
 
 	mss->resident += size;
 	/* Accumulate the size in pages that have been accessed. */
-	if (young || folio_test_young(folio) || folio_test_referenced(folio))
+	if (young || folio_test_young(folio) || smap_check_folio_referenced(folio))
 		mss->referenced += size;
 
 	/*
@@ -1791,7 +1807,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,
 		/* Clear accessed and referenced bits. */
 		pmdp_test_and_clear_young(vma, addr, pmd);
 		folio_test_clear_young(folio);
-		folio_clear_referenced(folio);
+		smap_clear_folio_referenced(folio);
 out:
 		spin_unlock(ptl);
 		return 0;
@@ -1820,7 +1836,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,
 		/* Clear accessed and referenced bits. */
 		ptep_test_and_clear_young(vma, addr, pte);
 		folio_test_clear_young(folio);
-		folio_clear_referenced(folio);
+		smap_clear_folio_referenced(folio);
 	}
 	pte_unmap_unlock(pte - 1, ptl);
 	cond_resched();

-- 
2.55.0

[PATCH RFC 13/15] mm/huge_memory: mark file folio as accessed more accurately on split

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

The behavior of updating the folio's access info isn't consistent for
huge mapping splitting or ordinary unmapping. The page table's young
flag has to be translated into folio's access info.

Right now it only check and set folio's referenced flag, which isn't
enough since folio flags update on access have its rules. Ordinary
unmapping (zapping) calls folio_mark_accessed(), and it also checks
if the VMA has recency to avoid false updates.

So first just use the right helper here to be more consistent.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 mm/huge_memory.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 21c92ee48e46..043c9ac963b4 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3058,8 +3058,8 @@ static void __split_huge_pud_locked(struct vm_area_struct *vma, pud_t *pud,
 
 	if (!folio_test_dirty(folio) && pud_dirty(old_pud))
 		folio_mark_dirty(folio);
-	if (!folio_test_referenced(folio) && pud_young(old_pud))
-		folio_set_referenced(folio);
+	if (pud_young(old_pud) && vma_has_recency(vma))
+		folio_mark_accessed(folio);
 	folio_remove_rmap_pud(folio, page, vma);
 	add_mm_counter(vma->vm_mm, mm_counter_file(folio),
 		-HPAGE_PUD_NR);
@@ -3181,8 +3181,8 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
 			folio = page_folio(page);
 			if (!folio_test_dirty(folio) && pmd_dirty(old_pmd))
 				folio_mark_dirty(folio);
-			if (!folio_test_referenced(folio) && pmd_young(old_pmd))
-				folio_set_referenced(folio);
+			if (pmd_young(old_pmd) && vma_has_recency(vma))
+				folio_mark_accessed(folio);
 			folio_remove_rmap_pmd(folio, page, vma);
 			add_mm_counter(mm, mm_counter_file(folio), -HPAGE_PMD_NR);
 			folio_put(folio);

-- 
2.55.0

[PATCH RFC 14/15] mm/khugepaged: consider workingset folios as referenced

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

The folio_test_referenced check here is clearly trying to test if the
folio was ever referenced. It was first introduced by commit
8ee53820edfd ("thp: mmu_notifier_test_young") as an supplement of the
young bit check.

Folios are marked as PG_referenced on first access, but following access
will clear their PG_referenced on second access. So checking only the
referenced flag is not accurate enough.

Switch to use the new helper, so we can cover the secondary and following
access from MGLRU side. For non-MGLRU, this will make it return positve
for workingset folios too though, which should be OK.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 mm/khugepaged.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index b237f6e7662a..86c9b07dece6 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -809,7 +809,7 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
 		 */
 		if (cc->is_khugepaged &&
 		    (pte_young(pteval) || folio_test_young(folio) ||
-		     folio_test_referenced(folio) ||
+		     folio_is_referenced(folio) ||
 		     mmu_notifier_test_young(vma->vm_mm, addr)))
 			referenced++;
 	}
@@ -1767,7 +1767,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 		 */
 		if (cc->is_khugepaged &&
 		    (pte_young(pteval) || folio_test_young(folio) ||
-		     folio_test_referenced(folio) ||
+		     folio_is_referenced(folio) ||
 		     mmu_notifier_test_young(vma->vm_mm, addr)))
 			referenced++;
 	}
@@ -2752,7 +2752,7 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,
 		/*
 		 * We probably should check if the folio is referenced
 		 * here, but nobody would transfer pte_young() to
-		 * folio_test_referenced() for us.  And rmap walk here
+		 * folio_is_referenced() for us.  And rmap walk here
 		 * is just too costly...
 		 */
 

-- 
2.55.0

[PATCH RFC 15/15] mm/madvise: convert to new lru refs API and better support for MGLRU

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

For active/inactive LRU, madvise wants evicted folios from active LRU to
be considered for PSI too, so some special handling are added. But MGLRU
doesn't really need this, as it has a different activation logic.

Switch to new helpers and improve the support for MGLRU here.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 mm/madvise.c | 37 +++++++++++++++++++++++--------------
 1 file changed, 23 insertions(+), 14 deletions(-)

diff --git a/mm/madvise.c b/mm/madvise.c
index abb17760b8b5..f132dd7418f5 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -350,6 +350,27 @@ static inline int madvise_folio_pte_batch(unsigned long addr, unsigned long end,
 				     FPB_MERGE_YOUNG_DIRTY);
 }
 
+/*
+ * We are deactivating a folio for accelerating reclaiming.
+ * VM couldn't reclaim the folio unless we clear PG_young.
+ * As a side effect, it makes confuse idle-page tracking
+ * because they will miss recent referenced history.
+ */
+static void madvise_cold_or_pageout_prep_folio(struct folio *folio)
+{
+	folio_test_clear_young(folio);
+
+	/*
+	 * MGLRU clears all reference flags in folio_deactivate,
+	 * no need to touch it here.
+	 */
+	if (!lru_gen_enabled()) {
+		folio_clear_referenced_by_bit(folio);
+		if (folio_test_active(folio))
+			folio_mark_workingset_by_bit(folio);
+	}
+}
+
 static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 				unsigned long addr, unsigned long end,
 				struct mm_walk *walk)
@@ -424,10 +445,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 			tlb_remove_pmd_tlb_entry(tlb, pmd, addr);
 		}
 
-		folio_clear_referenced(folio);
-		folio_test_clear_young(folio);
-		if (folio_test_active(folio))
-			folio_mark_workingset_by_bit(folio);
+		madvise_cold_or_pageout_prep_folio(folio);
 		if (pageout) {
 			if (folio_isolate_lru(folio)) {
 				if (folio_test_unevictable(folio))
@@ -533,16 +551,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 			tlb_remove_tlb_entries(tlb, pte, nr, addr);
 		}
 
-		/*
-		 * We are deactivating a folio for accelerating reclaiming.
-		 * VM couldn't reclaim the folio unless we clear PG_young.
-		 * As a side effect, it makes confuse idle-page tracking
-		 * because they will miss recent referenced history.
-		 */
-		folio_clear_referenced(folio);
-		folio_test_clear_young(folio);
-		if (folio_test_active(folio))
-			folio_mark_workingset_by_bit(folio);
+		madvise_cold_or_pageout_prep_folio(folio);
 		if (pageout) {
 			if (folio_isolate_lru(folio)) {
 				if (folio_test_unevictable(folio))

-- 
2.55.0

Re: [PATCH RFC 12/15] mm/smap: report workingset folios as referenced

[Johannes Weiner]

On Tue, Aug 04, 2026 at 03:47:08AM +0800, Kairui Song via B4 Relay wrote:
> @@ -944,6 +944,22 @@ static void smaps_page_accumulate(struct mem_size_stats *mss,
>  	}
>  }
>  
> +static bool smap_check_folio_referenced(struct folio *folio)
> +{
> +	if (lru_gen_enabled())
> +		return folio_is_referenced(folio);
> +	else
> +		return folio_is_referenced_by_bit(folio);
> +}
> +
> +static void smap_clear_folio_referenced(struct folio *folio)
> +{
> +	if (lru_gen_enabled())
> +		folio_set_lru_refs(folio, 0);
> +	else
> +		folio_clear_referenced_by_bit(folio);
> +}

I don't really understand this and the preceding 2 patches.

What is the benefit of a "shared" ref/workingset state implementation
when you then still have separate APIs for querying & modifying?

Switching some these callers from folio_test_workingset() to
folio_is_workingset()? But both functions continue to exist?

Why not make versions of folio_test_workingset(),
folio_test_referenced(), folio_clear_referenced() etc. that do the
arbitration (bit ops on classic, refs ops on mglru) and leave callers
that are only interested in boolean states, like here, alone?

folio_test_anon(), folio_test_lazyfree(), folio_test_swapcache() -
there are many examples where they don't just test a raw bit but
assess more complex, composed state.

Re: [PATCH RFC 12/15] mm/smap: report workingset folios as referenced

[Kairui Song]

On Mon, Aug 03, 2026 at 09:21:33PM +0800, Johannes Weiner wrote:
> On Tue, Aug 04, 2026 at 03:47:08AM +0800, Kairui Song via B4 Relay wrote:
> > @@ -944,6 +944,22 @@ static void smaps_page_accumulate(struct mem_size_stats *mss,
> >  	}
> >  }
> >  
> > +static bool smap_check_folio_referenced(struct folio *folio)
> > +{
> > +	if (lru_gen_enabled())
> > +		return folio_is_referenced(folio);
> > +	else
> > +		return folio_is_referenced_by_bit(folio);
> > +}
> > +
> > +static void smap_clear_folio_referenced(struct folio *folio)
> > +{
> > +	if (lru_gen_enabled())
> > +		folio_set_lru_refs(folio, 0);
> > +	else
> > +		folio_clear_referenced_by_bit(folio);
> > +}
> 
> I don't really understand this and the preceding 2 patches.
> 
> What is the benefit of a "shared" ref/workingset state implementation
> when you then still have separate APIs for querying & modifying?
> 
> Switching some these callers from folio_test_workingset() to
> folio_is_workingset()? But both functions continue to exist?
> 
> Why not make versions of folio_test_workingset(),
> folio_test_referenced(), folio_clear_referenced() etc. that do the
> arbitration (bit ops on classic, refs ops on mglru) and leave callers
> that are only interested in boolean states, like here, alone?
> 
> folio_test_anon(), folio_test_lazyfree(), folio_test_swapcache() -
> there are many examples where they don't just test a raw bit but
> assess more complex, composed state.
> 

Sound good, I just copied the folio_is/mark_workingset helper from
my previous combined RFC, where I tried to drop PG_workingset hence
used a standalone helper to convert the users one by one for easier
audit and tracking.

I think I can just replace the default helpers and have zero effect
for CLRU. There are actually a few more helpers not used but introduced
here, will drop these in V2. Thanks for the review!

Re: [PATCH RFC 09/15] mm/mglru: frequency guided workingset promotion (MGLRU-FG)

[Kairui Song]

On Tue, Aug 4, 2026 at 3:47 AM Kairui Song via B4 Relay
<devnull+kasong.tencent.com@kernel.org> wrote:
>
> From: Kairui Song <kasong@tencent.com>
>
> Complement MGLRU's eviction-time tier-PID protection with access-time
> frequency-guided promotion.  Introduce a unified set of helpers built based
> on referenced (access) count of a folio.
>
> Each access increments a folio's referenced count stored in folio flags
> (refs), refs still mappes to a logarithmic tier just like before, but with
> more formal bit definitions, a few special thresholds are introduced:
> LRU_REFS_REFERENCED (1), LRU_REFS_WORKINGSET (2), LRU_REFS_PROTECTED (3),
> and LRU_REFS_MAX(7). When it reaches certain threshold, the folio is
> promoted proactively instead of wait for the PID controller to kick in.
>
> Also simplify MGLRU's usage of PG_workingset and PG_referenced, now
> these 2 flags are purely used as the lower 2 bit of refs for MGLRU. This
> doesn't effect classical LRU in any way. This will actually simplify and
> make MGLRU's certain metric reading more accurate, and reduced MGLRU's
> original tier / referenced count bit by one since only one extra bit is
> now needed to record a max referenced count of 7 (previously 2 extra bits
> are needed). This changes make sense because MGLRU doesn't have demotion
> so these 2 flags are never separately useful for MGLRU.
>
> This addresses several shortcomings of the old model:
>
> - Long feedback loop: protection only activated after enough
>   re-faults, by which time the folio is often no longer hot.
>
> - Limited tier resolution: once referenced count exceeded the bits
>   limit (8 previously), MGLRU could no longer distinguish hotter folios as
>   they are capped by the tier.  And what's worse, PG_workingset
>   forces a folio to stay on tier 3.
>
> - Eviction-time bias: because PID protection activates upon eviction
>   and always targets the LRU tail, it tends to protect cold tail
>   folios at the expense of hotter head folios.  Once the tail folios
>   consume the PID protection budget, head folios lose their
>   protection.  Additionally, the PID cannot distinguish the access
>   time of folios that share the same reference count.
>
> Besides reworking the LRU_REFS related helpers and definitions, most of
> the work is done by the helpers below; the implementation details are
> described in their inline comments.
>
> - folio_inc_lru_refs(): Used by both cache access (folio_mark_accessed)
>   and page table access.  The folio could be off-list (isolated),
>   unlocked, or unmapped.  This helper uses PG_lru to stabilize the
>   folio and performs a speculative and lazy promotion.
>
> - folio_inc_lru_refs_walk(): Used by the PTE walk path during aging,
>   where generations are stable; performs lazy promotion.
>
> - folio_inc_lru_refs_isolated(): Used by the rmap check before
>   eviction.  The folio is isolated and hence this doesn't perform
>   promotion by itself; the folio will be added back to the right gen
>   upon return.
>
> The eviction-time folio_inc_gen() still handles PID protection, but the
> protection ratio is softer than before, and it caps refs at WORKINGSET
> so the folio retains enough history to stay above the cold tier.
>
> The PID controller gain factors in get_tier_idx() are also relaxed from
> (2:3) to (1:2).  Since the new folio gen bump paths already proactively
> protect hot folios, PID protection can afford to be more permissive
> without increasing the refault rate.
>
> PG_workingset and PG_referenced are repurposed as the low two bits
> of the unified LRU reference count.  LRU_REFS_MASK provides the
> higher bits.  This eliminates the old restriction where LRU_REFS_MASK
> was only valid when PG_referenced was set, and allows all paths to use
> the same encoding consistently.
>
> Hence, a workingset folio is now defined as refs >= LRU_REFS_WORKINGSET
> (2), matching the active/inactive LRU's definition and giving in-kernel
> consumers (PSI, readahead) consistent behavior on MGLRU, which will be
> done in later commits.
>
> Note that PG_workingset and PG_referenced are no longer independent
> flags under MGLRU.  Adjusting existing raw folio_test_*() callers
> to the new semantics is left as follow-ups.
>
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
>  include/linux/mm_inline.h |  83 ++++++++-----
>  include/linux/mmzone.h    | 135 ++++++++++++++-------
>  kernel/bounds.c           |   2 +-
>  mm/folio.c                |  46 +-------
>  mm/vmscan.c               | 290 ++++++++++++++++++++++++++++++----------------
>  mm/workingset.c           |  55 ++++++---
>  6 files changed, 385 insertions(+), 226 deletions(-)
>


> +/*
> + * Update the folio's lru refs indicator without taking the folio lock,
> + * isolation, or lruvec lock. Used by both page table access (@is_fault=true)
> + * and by file access (@is_fault=false).
> + */
> +int folio_inc_lru_refs(struct folio *folio, bool is_fault, bool is_exec)
> +{
> +       int max_gen, min_gen;
> +       int type, refs, gen, new_gen;
> +       unsigned long new_flags, old_flags, max_seq;
> +       struct lru_gen_folio *lrugen;
> +       struct lruvec *lruvec;
> +
> +       type = folio_is_file_lru(folio);
> +       lruvec = folio_lruvec_live_get(folio);
> +       lrugen = &lruvec->lrugen;
> +
> +       old_flags = READ_ONCE(*folio_flags(folio, 0));
> +       do {
> +               new_flags = old_flags;
> +               gen = lru_gen_from_flags(old_flags);
> +               refs = lru_refs_from_flags(old_flags) + 1;
> +               new_gen = gen;
> +               if (!(old_flags & BIT(PG_lru)) || gen < 0)
> +                       goto out;
> +
> +               max_seq = READ_ONCE(lrugen->max_seq);
> +               max_gen = lru_gen_from_seq(max_seq);
> +               min_gen = lru_gen_from_seq(READ_ONCE(lrugen->min_seq[type]));
> +               if (gen == max_gen)
> +                       goto out;
> +
> +               if (is_fault || is_exec) {
> +                       /* Promote second page table access or executable */
> +                       if (refs > LRU_REFS_REFERENCED || is_exec)
> +                               new_gen = max_gen;
> +                       else
> +                               new_gen = (gen + 1UL) % MAX_NR_GENS;
> +                       refs = min(refs, LRU_REFS_PROTECTED);
> +               } else if (refs > LRU_REFS_MAX) {
> +                       /* LRU refs counting overflow, bump the gen */
> +                       new_gen = (gen + 1UL) % MAX_NR_GENS;
> +                       refs = LRU_REFS_PROTECTED;
> +               } else if (gen == min_gen && refs >= LRU_REFS_WORKINGSET) {
> +                       /* Defer eviction of just accessed workingset */
> +                       new_gen = (gen + 1UL) % MAX_NR_GENS;
> +                       refs = min(refs, LRU_REFS_PROTECTED);
>                 }
> +out:
> +               refs = min(refs, LRU_REFS_MAX);
> +               lru_refs_set_flags(&new_flags, refs);
> +               if (new_gen >= 0)
> +                       lru_gen_set_flags(&new_flags, new_gen);
> +       } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>
> -               folio_set_lru_refs(folio, 1);
> -               return false;
> +       if (new_gen != gen) {
> +               /*
> +                * Gen can only go forward, so concurrent aging is
> +                * usually fine, except when multiple aging increase
> +                * max_seq multiple times, new_gen may have go beyond
> +                * the new max_seq's current gen border and causes
> +                * hotness inversion. In that very unlikely case,
> +                * just activate the folio.
> +                */
> +               lru_gen_update_size(lruvec, folio, gen, new_gen);
> +               if (unlikely(READ_ONCE(lrugen->max_seq) - max_seq > MIN_NR_GENS))
> +                       folio_activate(folio);
>         }

So sashiko complained about a potential race here, which I suspected
but let through for this RFC:

lru_gen_update_size is lockless here, so if the folio is move between
active and inactive gen and a concurrent aging occurs, we might lost a
active / inactive counter update and drift the reading. The chance is
tiny and active/inactive counts are just metrics reading for MGLRU so
this should have no effect on performance.

On Tue, Aug 4, 2026 at 3:47 AM Kairui Song via B4 Relay
<devnull+kasong.tencent.com@kernel.org> wrote:
>
> From: Kairui Song <kasong@tencent.com>
>
> Complement MGLRU's eviction-time tier-PID protection with access-time
> frequency-guided promotion.  Introduce a unified set of helpers built based
> on referenced (access) count of a folio.
>
> Each access increments a folio's referenced count stored in folio flags
> (refs), refs still mappes to a logarithmic tier just like before, but with
> more formal bit definitions, a few special thresholds are introduced:
> LRU_REFS_REFERENCED (1), LRU_REFS_WORKINGSET (2), LRU_REFS_PROTECTED (3),
> and LRU_REFS_MAX(7). When it reaches certain threshold, the folio is
> promoted proactively instead of wait for the PID controller to kick in.
>
> Also simplify MGLRU's usage of PG_workingset and PG_referenced, now
> these 2 flags are purely used as the lower 2 bit of refs for MGLRU. This
> doesn't effect classical LRU in any way. This will actually simplify and
> make MGLRU's certain metric reading more accurate, and reduced MGLRU's
> original tier / referenced count bit by one since only one extra bit is
> now needed to record a max referenced count of 7 (previously 2 extra bits
> are needed). This changes make sense because MGLRU doesn't have demotion
> so these 2 flags are never separately useful for MGLRU.
>
> This addresses several shortcomings of the old model:
>
> - Long feedback loop: protection only activated after enough
>   re-faults, by which time the folio is often no longer hot.
>
> - Limited tier resolution: once referenced count exceeded the bits
>   limit (8 previously), MGLRU could no longer distinguish hotter folios as
>   they are capped by the tier.  And what's worse, PG_workingset
>   forces a folio to stay on tier 3.
>
> - Eviction-time bias: because PID protection activates upon eviction
>   and always targets the LRU tail, it tends to protect cold tail
>   folios at the expense of hotter head folios.  Once the tail folios
>   consume the PID protection budget, head folios lose their
>   protection.  Additionally, the PID cannot distinguish the access
>   time of folios that share the same reference count.
>
> Besides reworking the LRU_REFS related helpers and definitions, most of
> the work is done by the helpers below; the implementation details are
> described in their inline comments.
>
> - folio_inc_lru_refs(): Used by both cache access (folio_mark_accessed)
>   and page table access.  The folio could be off-list (isolated),
>   unlocked, or unmapped.  This helper uses PG_lru to stabilize the
>   folio and performs a speculative and lazy promotion.
>
> - folio_inc_lru_refs_walk(): Used by the PTE walk path during aging,
>   where generations are stable; performs lazy promotion.
>
> - folio_inc_lru_refs_isolated(): Used by the rmap check before
>   eviction.  The folio is isolated and hence this doesn't perform
>   promotion by itself; the folio will be added back to the right gen
>   upon return.
>
> The eviction-time folio_inc_gen() still handles PID protection, but the
> protection ratio is softer than before, and it caps refs at WORKINGSET
> so the folio retains enough history to stay above the cold tier.
>
> The PID controller gain factors in get_tier_idx() are also relaxed from
> (2:3) to (1:2).  Since the new folio gen bump paths already proactively
> protect hot folios, PID protection can afford to be more permissive
> without increasing the refault rate.
>
> PG_workingset and PG_referenced are repurposed as the low two bits
> of the unified LRU reference count.  LRU_REFS_MASK provides the
> higher bits.  This eliminates the old restriction where LRU_REFS_MASK
> was only valid when PG_referenced was set, and allows all paths to use
> the same encoding consistently.
>
> Hence, a workingset folio is now defined as refs >= LRU_REFS_WORKINGSET
> (2), matching the active/inactive LRU's definition and giving in-kernel
> consumers (PSI, readahead) consistent behavior on MGLRU, which will be
> done in later commits.
>
> Note that PG_workingset and PG_referenced are no longer independent
> flags under MGLRU.  Adjusting existing raw folio_test_*() callers
> to the new semantics is left as follow-ups.
>
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
>  include/linux/mm_inline.h |  83 ++++++++-----
>  include/linux/mmzone.h    | 135 ++++++++++++++-------
>  kernel/bounds.c           |   2 +-
>  mm/folio.c                |  46 +-------
>  mm/vmscan.c               | 290 ++++++++++++++++++++++++++++++----------------
>  mm/workingset.c           |  55 ++++++---
>  6 files changed, 385 insertions(+), 226 deletions(-)
>


> +/*
> + * Update the folio's lru refs indicator without taking the folio lock,
> + * isolation, or lruvec lock. Used by both page table access (@is_fault=true)
> + * and by file access (@is_fault=false).
> + */
> +int folio_inc_lru_refs(struct folio *folio, bool is_fault, bool is_exec)
> +{
> +       int max_gen, min_gen;
> +       int type, refs, gen, new_gen;
> +       unsigned long new_flags, old_flags, max_seq;
> +       struct lru_gen_folio *lrugen;
> +       struct lruvec *lruvec;
> +
> +       type = folio_is_file_lru(folio);
> +       lruvec = folio_lruvec_live_get(folio);
> +       lrugen = &lruvec->lrugen;
> +
> +       old_flags = READ_ONCE(*folio_flags(folio, 0));
> +       do {
> +               new_flags = old_flags;
> +               gen = lru_gen_from_flags(old_flags);
> +               refs = lru_refs_from_flags(old_flags) + 1;
> +               new_gen = gen;
> +               if (!(old_flags & BIT(PG_lru)) || gen < 0)
> +                       goto out;
> +
> +               max_seq = READ_ONCE(lrugen->max_seq);
> +               max_gen = lru_gen_from_seq(max_seq);
> +               min_gen = lru_gen_from_seq(READ_ONCE(lrugen->min_seq[type]));
> +               if (gen == max_gen)
> +                       goto out;
> +
> +               if (is_fault || is_exec) {
> +                       /* Promote second page table access or executable */
> +                       if (refs > LRU_REFS_REFERENCED || is_exec)
> +                               new_gen = max_gen;
> +                       else
> +                               new_gen = (gen + 1UL) % MAX_NR_GENS;
> +                       refs = min(refs, LRU_REFS_PROTECTED);
> +               } else if (refs > LRU_REFS_MAX) {
> +                       /* LRU refs counting overflow, bump the gen */
> +                       new_gen = (gen + 1UL) % MAX_NR_GENS;
> +                       refs = LRU_REFS_PROTECTED;
> +               } else if (gen == min_gen && refs >= LRU_REFS_WORKINGSET) {
> +                       /* Defer eviction of just accessed workingset */
> +                       new_gen = (gen + 1UL) % MAX_NR_GENS;
> +                       refs = min(refs, LRU_REFS_PROTECTED);
>                 }
> +out:
> +               refs = min(refs, LRU_REFS_MAX);
> +               lru_refs_set_flags(&new_flags, refs);
> +               if (new_gen >= 0)
> +                       lru_gen_set_flags(&new_flags, new_gen);
> +       } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>
> -               folio_set_lru_refs(folio, 1);
> -               return false;
> +       if (new_gen != gen) {
> +               /*
> +                * Gen can only go forward, so concurrent aging is
> +                * usually fine, except when multiple aging increase
> +                * max_seq multiple times, new_gen may have go beyond
> +                * the new max_seq's current gen border and causes
> +                * hotness inversion. In that very unlikely case,
> +                * just activate the folio.
> +                */
> +               lru_gen_update_size(lruvec, folio, gen, new_gen);
> +               if (unlikely(READ_ONCE(lrugen->max_seq) - max_seq > MIN_NR_GENS))
> +                       folio_activate(folio);
>         }

So sashiko complained about a potential race here, which I suspected
but let through for this RFC:

lru_gen_update_size is lockless here, so if the folio is move between
active and inactive gen and a concurrent aging occurs, we might lost a
active / inactive counter update and drift the reading. The chance is
tiny and active/inactive counts are just metrics reading for MGLRU so
this should have no effect on performance.

I'll need to figure out a way to fix this, hopefully without any lock
because this is a hot path where overheads are critical.

One idea is that we directly jump to the part where we decouple the
active/inactive counter with MGLRU's gens (I mentiones this before
just didn't post any code about it yet)? If you use MGLRU in a fleet
you should already have noticed that the active / inactive reading is
almost unusable, because MGLRU's aging moves folios in whole
generations. Because the active/inactive count is bounded by
generations, you will see the numbers are very "jumpy," and almost all
page cache is always reported as inactive since the PID can only
promote the page cache to the second oldest generation, which is
considered "inactive." This is incorrect and breaks things like
cAdvisor in K8s, which uses the inactive reading to determine how much
of the page cache is actually used.

So instead, if we bound active / inactive with tiers instead of gens,
all the issues are gone. I planned to do that later, but if everyone
is OK with it, we can do this earlier.

Or else, some fancy atomic / barrier might be needed, I think that's
still doable though but kind of complex and maybe ugly.

[syzbot ci] Re: mm/mglru: frequency guided promotion (MGLRU-FG) and flag cleanup

[syzbot ci]

syzbot ci has tested the following series

[v1] mm/mglru: frequency guided promotion (MGLRU-FG) and flag cleanup
https://lore.kernel.org/all/20260804-mglru-fg-v1-0-4d8dad39dad6@tencent.com
* [PATCH RFC 01/15] mm/memcontrol: make lru_zone_size atomic and simplify sanity check
* [PATCH RFC 02/15] mm/memcontrol: allow update of LRU statistic without holding LRU lock
* [PATCH RFC 03/15] mm/mglru: introduce and always use helpers for manipulating page flags
* [PATCH RFC 04/15] mm/mglru: make generation page counters atomic
* [PATCH RFC 05/15] mm/mglru: move max_seq read into walk_update_folio
* [PATCH RFC 06/15] mm/mglru: use explicit tier range in read_ctrl_pos()
* [PATCH RFC 07/15] mm/mglru: move refault workingset activation into lru_gen_refault
* [PATCH RFC 08/15] mm/memcg: add folio-based lruvec live helper
* [PATCH RFC 09/15] mm/mglru: frequency guided workingset promotion (MGLRU-FG)
* [PATCH RFC 10/15] mm/mglru: make folio lru referenced times count a generic API
* [PATCH RFC 11/15] mm/mglru: replace folio workinset check and update with new helper
* [PATCH RFC 12/15] mm/smap: report workingset folios as referenced
* [PATCH RFC 13/15] mm/huge_memory: mark file folio as accessed more accurately on split
* [PATCH RFC 14/15] mm/khugepaged: consider workingset folios as referenced
* [PATCH RFC 15/15] mm/madvise: convert to new lru refs API and better support for MGLRU

and found the following issue:
WARNING in folio_inc_lru_refs

Full report is available here:
https://ci.syzbot.org/series/5db36d1d-9faa-4882-9f0d-1a8f52132274

***

WARNING in folio_inc_lru_refs

tree:      mm-new
URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/akpm/mm.git
base:      94f9b3980dd446b56acf1dfed649e9b32a9f3813
arch:      amd64
compiler:  Debian clang version 22.1.8 (++20260613092233+e80beda6e255-1~exp1~20260613092250.77), Debian LLD 22.1.8
config:    https://ci.syzbot.org/builds/9f324367-2b95-4fd0-9025-fef1ff1f605a/config

page: refcount:3 mapcount:2 mapping:0000000000000000 index:0x0 pfn:0xe4ee
flags: 0xfff00000002000(reserved|node=0|zone=1|lastcpupid=0x7ff)
raw: 00fff00000002000 ffffea0000393b88 ffffea0000393b88 0000000000000000
raw: 0000000000000000 0000000000000000 0000000300000001 0000000000000000
page dumped because: VM_WARN_ON_ONCE_FOLIO(!memcg && !mem_cgroup_disabled())
page_owner info is not present (never set?)
------------[ cut here ]------------
1
WARNING: ./include/linux/memcontrol.h:745 at folio_inc_lru_refs+0xb4f/0xc10, CPU#0: mount/5025
Modules linked in:
CPU: 0 UID: 0 PID: 5025 Comm: mount Not tainted syzkaller #0 PREEMPT(full) 
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
RIP: 0010:folio_inc_lru_refs+0xb4f/0xc10
Code: ff 4c 89 e7 e8 c2 ae fd ff e9 e9 fc ff ff e8 18 91 ba ff 4c 89 e7 48 c7 c6 20 90 f8 8b e8 99 b0 1b ff c6 05 a6 a9 34 0e 01 90 <0f> 0b 90 e9 85 f6 ff ff e8 f4 90 ba ff e9 29 f8 ff ff 44 89 f1 80
RSP: 0018:ffffc9000324f4c0 EFLAGS: 00010246
RAX: 8e914a919570c700 RBX: 0000000000000000 RCX: 0000000000000001
RDX: 0000000000000000 RSI: ffffffff8e4b4187 RDI: ffff888174f53c00
RBP: ffffc9000324f5d0 R08: 0000000000000003 R09: 0000000000000004
R10: dffffc0000000000 R11: fffffbfff1d3ca24 R12: ffffea0000393b80
R13: 1ffffd4000072770 R14: 1ffff92000649ea8 R15: dffffc0000000000
FS:  0000000000000000(0000) GS:ffff88818d949000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb77b215440 CR3: 000000000e946000 CR4: 00000000000006f0
Call Trace:
 <TASK>
 __zap_vma_range+0x20f5/0x4f70
 unmap_vmas+0x390/0x550
 exit_mmap+0x293/0x9f0
 __mmput+0x118/0x420
 exit_mm+0x221/0x2d0
 do_exit+0x6cd/0x2360
 do_group_exit+0x22d/0x2f0
 __x64_sys_exit_group+0x3f/0x40
 x64_sys_call+0x221a/0x2240
 do_syscall_64+0x174/0x580
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7fb77b2d3a90
Code: Unable to access opcode bytes at 0x7fb77b2d3a66.
RSP: 002b:00007ffe32d58618 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 00007fb77b3c4860 RCX: 00007fb77b2d3a90
RDX: 00000000000000e7 RSI: 000000000000003c RDI: 0000000000000000
RBP: 00007fb77b3c4860 R08: 00007ffe32d58490 R09: 00007ffe32d58570
R10: 00007ffe32d584d0 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007fb77b3c8658 R15: 0000000000000001
 </TASK>


***

If these findings have caused you to resend the series or submit a
separate fix, please add the following tag to your commit message:
  Tested-by: syzbot@syzkaller.appspotmail.com

---
This report is generated by a bot. It may contain errors.
syzbot ci engineers can be reached at syzkaller@googlegroups.com.

To test a patch for this bug, please reply with `#syz test`
(should be on a separate line).

The patch should be attached to the email.
Note: arguments like custom git repos and branches are not supported.

Re: [syzbot ci] Re: mm/mglru: frequency guided promotion (MGLRU-FG) and flag cleanup

[Kairui Song]

On Mon, Aug 03, 2026 at 10:26:17PM +0800, syzbot ci wrote:
> syzbot ci has tested the following series
> 
> [v1] mm/mglru: frequency guided promotion (MGLRU-FG) and flag cleanup
> https://lore.kernel.org/all/20260804-mglru-fg-v1-0-4d8dad39dad6@tencent.com
> * [PATCH RFC 01/15] mm/memcontrol: make lru_zone_size atomic and simplify sanity check
> * [PATCH RFC 02/15] mm/memcontrol: allow update of LRU statistic without holding LRU lock
> * [PATCH RFC 03/15] mm/mglru: introduce and always use helpers for manipulating page flags
> * [PATCH RFC 04/15] mm/mglru: make generation page counters atomic
> * [PATCH RFC 05/15] mm/mglru: move max_seq read into walk_update_folio
> * [PATCH RFC 06/15] mm/mglru: use explicit tier range in read_ctrl_pos()
> * [PATCH RFC 07/15] mm/mglru: move refault workingset activation into lru_gen_refault
> * [PATCH RFC 08/15] mm/memcg: add folio-based lruvec live helper
> * [PATCH RFC 09/15] mm/mglru: frequency guided workingset promotion (MGLRU-FG)
> * [PATCH RFC 10/15] mm/mglru: make folio lru referenced times count a generic API
> * [PATCH RFC 11/15] mm/mglru: replace folio workinset check and update with new helper
> * [PATCH RFC 12/15] mm/smap: report workingset folios as referenced
> * [PATCH RFC 13/15] mm/huge_memory: mark file folio as accessed more accurately on split
> * [PATCH RFC 14/15] mm/khugepaged: consider workingset folios as referenced
> * [PATCH RFC 15/15] mm/madvise: convert to new lru refs API and better support for MGLRU
> 
> and found the following issue:
> WARNING in folio_inc_lru_refs
> 
> Full report is available here:
> https://ci.syzbot.org/series/5db36d1d-9faa-4882-9f0d-1a8f52132274
> 
> ***
> 
> WARNING in folio_inc_lru_refs
> 
> tree:      mm-new
> URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/akpm/mm.git
> base:      94f9b3980dd446b56acf1dfed649e9b32a9f3813
> arch:      amd64
> compiler:  Debian clang version 22.1.8 (++20260613092233+e80beda6e255-1~exp1~20260613092250.77), Debian LLD 22.1.8
> config:    https://ci.syzbot.org/builds/9f324367-2b95-4fd0-9025-fef1ff1f605a/config
> 
> page: refcount:3 mapcount:2 mapping:0000000000000000 index:0x0 pfn:0xe4ee
> flags: 0xfff00000002000(reserved|node=0|zone=1|lastcpupid=0x7ff)
> raw: 00fff00000002000 ffffea0000393b88 ffffea0000393b88 0000000000000000
> raw: 0000000000000000 0000000000000000 0000000300000001 0000000000000000
> page dumped because: VM_WARN_ON_ONCE_FOLIO(!memcg && !mem_cgroup_disabled())
> page_owner info is not present (never set?)
> ------------[ cut here ]------------
> 1
> WARNING: ./include/linux/memcontrol.h:745 at folio_inc_lru_refs+0xb4f/0xc10, CPU#0: mount/5025
> Modules linked in:
> CPU: 0 UID: 0 PID: 5025 Comm: mount Not tainted syzkaller #0 PREEMPT(full) 
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
> RIP: 0010:folio_inc_lru_refs+0xb4f/0xc10
> Code: ff 4c 89 e7 e8 c2 ae fd ff e9 e9 fc ff ff e8 18 91 ba ff 4c 89 e7 48 c7 c6 20 90 f8 8b e8 99 b0 1b ff c6 05 a6 a9 34 0e 01 90 <0f> 0b 90 e9 85 f6 ff ff e8 f4 90 ba ff e9 29 f8 ff ff 44 89 f1 80
> RSP: 0018:ffffc9000324f4c0 EFLAGS: 00010246
> RAX: 8e914a919570c700 RBX: 0000000000000000 RCX: 0000000000000001
> RDX: 0000000000000000 RSI: ffffffff8e4b4187 RDI: ffff888174f53c00
> RBP: ffffc9000324f5d0 R08: 0000000000000003 R09: 0000000000000004
> R10: dffffc0000000000 R11: fffffbfff1d3ca24 R12: ffffea0000393b80
> R13: 1ffffd4000072770 R14: 1ffff92000649ea8 R15: dffffc0000000000
> FS:  0000000000000000(0000) GS:ffff88818d949000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007fb77b215440 CR3: 000000000e946000 CR4: 00000000000006f0
> Call Trace:
>  <TASK>
>  __zap_vma_range+0x20f5/0x4f70
>  unmap_vmas+0x390/0x550
>  exit_mmap+0x293/0x9f0
>  __mmput+0x118/0x420
>  exit_mm+0x221/0x2d0
>  do_exit+0x6cd/0x2360
>  do_group_exit+0x22d/0x2f0
>  __x64_sys_exit_group+0x3f/0x40
>  x64_sys_call+0x221a/0x2240
>  do_syscall_64+0x174/0x580
>  entry_SYSCALL_64_after_hwframe+0x77/0x7f
> RIP: 0033:0x7fb77b2d3a90
> Code: Unable to access opcode bytes at 0x7fb77b2d3a66.
> RSP: 002b:00007ffe32d58618 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
> RAX: ffffffffffffffda RBX: 00007fb77b3c4860 RCX: 00007fb77b2d3a90
> RDX: 00000000000000e7 RSI: 000000000000003c RDI: 0000000000000000
> RBP: 00007fb77b3c4860 R08: 00007ffe32d58490 R09: 00007ffe32d58570
> R10: 00007ffe32d584d0 R11: 0000000000000246 R12: 0000000000000000
> R13: 0000000000000000 R14: 00007fb77b3c8658 R15: 0000000000000001
>  </TASK>

OK, so we might hit a uncharged folio in folio_mark_access, which isn't
strange, right now it will just skip the gen bump and work as expected,
problem is it's triggering this warning, and doing redundant work for
lruvec lookup. To fix that, checking flags first then the lruvec should
be good.

Will do in V2.

Re: [PATCH RFC 08/15] mm/memcg: add folio-based lruvec live helper

[Lian Wang]

From: "Lian Wang (ProcessMission)" <lianux.mm@gmail.com>

Hi Kairui,

I am trying to understand the lifetime and accounting guarantee here, and
would appreciate your guidance. My understanding is that RCU protects the
lruvec lifetime, but by itself does not stabilize the folio->lruvec
association across memcg deletion and reparenting.

Could folio_inc_lru_refs() obtain the child lruvec here, then race with
__lru_gen_reparent_memcg(), and finally account the generation move to the old
child after the folio and its counters have moved to the parent? The opposite
ordering also seems possible: this helper observes css_is_dying() and selects
the parent while the folio is still accounted to the child.

Is there another invariant that closes these races? If my understanding is
correct, it seems the helper guarantees a live object, but not a stable
binding, and the lockless promotion path may need validation/retry or explicit
synchronization with reparenting.

If I have misunderstood the intended synchronization here, please feel free
to ignore this concern.

Thanks,
Lian

On Tue, 04 Aug 2026 03:47:04 +0800 Kairui Song via B4 Relay <devnull+kasong.tencent.com@kernel.org> wrote:

> From: Kairui Song <kasong@tencent.com>
> 
> Add a helper that resolves a stable lruvec for a folio under RCU
> without taking the lruvec lock.  It takes a folio directly so the
> lruvec lookup happens inside the RCU read-side critical section,
> which a lruvec-based interface cannot guarantee.
> 
> The lock-taking variant now inlines the ancestor walk instead of
> calling a separate helper.
> 
> No functional change.
> 
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
>  include/linux/memcontrol.h | 38 ++++++++++++++++++++++++++++++++++++++
>  1 file changed, 38 insertions(+)
> 
> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> index 68f363000d7f..ea0111392b9b 100644
> --- a/include/linux/memcontrol.h
> +++ b/include/linux/memcontrol.h
> @@ -1506,6 +1506,44 @@ static inline void lruvec_lock_irq(struct lruvec *lruvec)
>  	spin_lock_irq(&lruvec->lru_lock);
>  }
>  
> +/**
> + * folio_lruvec_live_get - get a live lruvec for a folio under RCU
> + * @folio: the folio
> + *
> + * Computes @folio's lruvec and walks up to the nearest live ancestor
> + * if the folio's memcg is dying.  Must be paired with
> + * folio_lruvec_live_put().
> + *
> + * Return: the live lruvec, with rcu_read_lock held.
> + */
> +static inline struct lruvec *folio_lruvec_live_get(struct folio *folio)
> +{
> +#ifdef CONFIG_MEMCG
> +	struct lruvec *lruvec;
> +	struct pglist_data *pgdat;
> +	struct mem_cgroup *memcg;
> +
> +	rcu_read_lock();
> +	lruvec = folio_lruvec(folio);
> +	pgdat = lruvec_pgdat(lruvec);
> +	memcg = lruvec_memcg(lruvec);
> +	while (unlikely(memcg && css_is_dying(&memcg->css))) {
> +		memcg = parent_mem_cgroup(memcg);
> +		lruvec = mem_cgroup_lruvec(memcg, pgdat);
> +	}
> +	return lruvec;
> +#else
> +	return folio_lruvec(folio);
> +#endif
> +}
> +
> +static inline void folio_lruvec_live_put(struct lruvec *lruvec)
> +{
> +#ifdef CONFIG_MEMCG
> +	rcu_read_unlock();
> +#endif
> +}
> +
>  static inline struct lruvec *lruvec_live_lock_irq(struct lruvec *lruvec)
>  {
>  #ifdef CONFIG_MEMCG
> 
> -- 
> 2.55.0
> 
> 
> 

Sent using hkml (https://github.com/sjp38/hackermail)

Re: [PATCH RFC 10/15] mm/mglru: make folio lru referenced times count a generic API

[Lian Wang]

From: "Lian Wang (ProcessMission)" <lianux.mm@gmail.com>

Hi Kairui,

I am trying to understand the intended semantics of making the referenced
count a generic API, and would appreciate your guidance. My understanding is
that, with the new encoding, raw PG_referenced and PG_workingset users no
longer see simple boolean states for every reference count.

A few examples I found:

- damon_pa_pageout() still calls folio_clear_referenced(). With refs == 2 it
  clears nothing, and with refs == 3 it leaves refs == 2. Thus DAMOS pageout
  may retain workingset history instead of clearing the MGLRU reference state.
- EROFS zdata uses PageWorkingset() for PSI accounting. With the new encoding,
  the PG_workingset bit is clear for refs == 4 or 5 even though the folio is
  hot.
- /proc/kpageflags exports PG_referenced directly, so KPF_REFERENCED appears
  to become the parity of refs rather than a boolean referenced state.

Are these semantics intended? The DAMON case in particular looks similar to
the madvise conversion in patch 15. If my understanding is correct, would the
remaining raw-bit users need a tree-wide audit together with the API
conversion?

If I have misunderstood how these users are expected to behave, please feel
free to ignore these concerns.

Thanks,
Lian

On Tue, 04 Aug 2026 03:47:06 +0800 Kairui Song via B4 Relay <devnull+kasong.tencent.com@kernel.org> wrote:

> From: Kairui Song <kasong@tencent.com>
> 
> To prepare for unifying the API for checking folio referenced status,
> expose the referenced times counting as a generic API. For MGLRU this
> helps to adapt other subsystem based on the referenced times counting,
> for non-MGLRU this is still bitwise compatible and there won't be
> major behavior change.
> 
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
>  include/linux/mm_inline.h | 233 ++++++++++++++++++++++++++++++----------------
>  mm/migrate.c              |   2 -
>  2 files changed, 155 insertions(+), 80 deletions(-)
> 
> diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
> index 944baa91bf18..a13b7d3c033a 100644
> --- a/include/linux/mm_inline.h
> +++ b/include/linux/mm_inline.h
> @@ -94,6 +94,161 @@ static __always_inline enum lru_list folio_lru_list(const struct folio *folio)
>  	return lru;
>  }
>  
> +/**
> + * lru_refs_from_flags - Return LRU referenced / access count from folio flags.
> + * @flags: folio flags
> + */
> +static inline int lru_refs_from_flags(unsigned long flags)
> +{
> +	int refs;
> +
> +	/*
> +	 * Return the total number of accesses. Also see the comment on
> +	 * LRU_REFS_FLAGS.
> +	 */
> +	refs = (flags & BIT(PG_referenced)) ? BIT(0) : 0;
> +	refs += (flags & BIT(PG_workingset)) ? BIT(1) : 0;
> +	refs += ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) << 2;
> +	return refs;
> +}
> +
> +/**
> + * lru_refs_set_flags - Set the LRU referenced / access count to specified folio flags.
> + * @flags: pointer to the folio flags
> + * @refs: referenced / access count number, between 0 and LRU_REFS_MAX, inclusive.
> + */
> +static inline void lru_refs_set_flags(unsigned long *flags, unsigned int refs)
> +{
> +	VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
> +	BUILD_BUG_ON((LRU_REFS_MAX >> 2) > (BIT(LRU_REFS_WIDTH) - 1));
> +	*flags &= ~LRU_REFS_FLAGS;
> +	if (refs & BIT(0))
> +		*flags |= BIT(PG_referenced);
> +	if (refs & BIT(1))
> +		*flags |= BIT(PG_workingset);
> +	*flags |= (((unsigned long)refs) >> 2) << LRU_REFS_PGOFF;
> +}
> +
> +static inline int folio_lru_refs(const struct folio *folio)
> +{
> +	return lru_refs_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
> +}
> +
> +static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
> +{
> +	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
> +
> +	do {
> +		new_flags = old_flags;
> +		lru_refs_set_flags(&new_flags, refs);
> +	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
> +}
> +
> +int folio_inc_lru_refs(struct folio *folio, bool is_fault, bool is_exec);
> +
> +/**
> + * folio_is_referenced - Tell if a folio was accessed before.
> + * @folio: the folio.
> + *
> + * This helper currently only works as intended for MGLRU, as it checks
> + * all LRU_REFS_FLAGS. It might be fine for non-MGLRU to replace
> + * folio_test_referenced in some cases but the user should be careful.
> + *
> + * Returns: true if the folio's LRU referenced / accessed count > 0.
> + */
> +static inline bool folio_is_referenced(const struct folio *folio)
> +{
> +	return folio_lru_refs(folio) >= LRU_REFS_REFERENCED;
> +}
> +
> +/**
> + * folio_mark_referenced - Mark a folio as referenced.
> + * @folio: the folio.
> + *
> + * Ensures the folio's LRU referenced count is at least
> + * LRU_REFS_REFERENCED. Won't do anything if the count is already larger
> + * than that. This helper currently only works as intended for MGLRU.
> + * Not a drop-in replacement, but should be fine for non-MGLRU to replace
> + * folio_set_referenced with this after audit.
> + */
> +static inline void folio_mark_referenced(struct folio *folio)
> +{
> +	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
> +
> +	do {
> +		new_flags = old_flags;
> +		if (lru_refs_from_flags(new_flags) >= LRU_REFS_REFERENCED)
> +			return;
> +		lru_refs_set_flags(&new_flags, LRU_REFS_REFERENCED);
> +	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
> +}
> +
> +/**
> + * folio_mark_referenced_by_bit - Mark a folio as referenced by bit.
> + * @folio: the folio.
> + *
> + * non-MGLRU may want to make use of the lowest LRU referenced count bit
> + * explicitly as a referenced mark.
> + */
> +static inline void folio_mark_referenced_by_bit(struct folio *folio)
> +{
> +	set_bit(PG_referenced, folio_flags(folio, 0));
> +}
> +
> +/**
> + * folio_clear_referenced_by_bit - Clear the referenced bit of a folio.
> + * @folio: the folio.
> + */
> +static inline void folio_clear_referenced_by_bit(struct folio *folio)
> +{
> +	clear_bit(PG_referenced, folio_flags(folio, 0));
> +}
> +
> +/**
> + * folio_test_clear_referenced_by_bit - Test and clear the referenced bit
> + * @folio: the folio.
> + */
> +static inline bool folio_test_clear_referenced_by_bit(struct folio *folio)
> +{
> +	return test_and_clear_bit(PG_referenced, folio_flags(folio, 0));
> +}
> +
> +/**
> + * folio_is_referenced_by_bit - Test if the referenced bit of a folio is set.
> + * @folio: the folio.
> + */
> +static inline bool folio_is_referenced_by_bit(const struct folio *folio)
> +{
> +	return test_bit(PG_referenced, const_folio_flags(folio, 0));
> +}
> +
> +/**
> + * folio_is_workingset - Tell if a folio is part of the workingset.
> + * @folio: the folio.
> + *
> + * Can be used to replace folio_test_workingset safely. For MGLRU the LRU
> + * referenced count tells if a folio is a workingset as intended. For non-MGLRU,
> + * the check below only holds true if the PG_workingset bit is set.
> + */
> +static inline bool folio_is_workingset(const struct folio *folio)
> +{
> +	return folio_lru_refs(folio) >= LRU_REFS_WORKINGSET;
> +}
> +
> +/**
> + * folio_mark_workingset_by_bit - Set the workingset bit of a folio.
> + * @folio: the folio.
> + */
> +static inline void folio_mark_workingset_by_bit(struct folio *folio)
> +{
> +	set_bit(PG_workingset, folio_flags(folio, 0));
> +}
> +
> +static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
> +{
> +	folio_set_lru_refs(new, folio_lru_refs(old));
> +}
> +
>  #ifdef CONFIG_LRU_GEN
>  
>  static inline bool lru_gen_switching(void)
> @@ -171,58 +326,6 @@ static inline void lru_gen_set_flags(unsigned long *flags, int gen)
>  	*flags |= (gen + 1UL) << LRU_GEN_PGOFF;
>  }
>  
> -/**
> - * lru_refs_from_flags - Return LRU referenced / access count from folio flags.
> - * @flags: folio flags
> - */
> -static inline int lru_refs_from_flags(unsigned long flags)
> -{
> -	int refs;
> -
> -	/*
> -	 * Return the total number of accesses. Also see the comment on
> -	 * LRU_REFS_FLAGS.
> -	 */
> -	refs = (flags & BIT(PG_referenced)) ? BIT(0) : 0;
> -	refs += (flags & BIT(PG_workingset)) ? BIT(1) : 0;
> -	refs += ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) << 2;
> -	return refs;
> -}
> -
> -/**
> - * lru_refs_set_flags - Set the LRU referenced / access count to specified folio flags.
> - * @flags: pointer to the folio flags
> - * @refs: referenced / access count number, between 0 and LRU_REFS_MAX, inclusive.
> - */
> -static inline void lru_refs_set_flags(unsigned long *flags, unsigned int refs)
> -{
> -	VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
> -	BUILD_BUG_ON((LRU_REFS_MAX >> 2) > (BIT(LRU_REFS_WIDTH) - 1));
> -	*flags &= ~LRU_REFS_FLAGS;
> -	if (refs & BIT(0))
> -		*flags |= BIT(PG_referenced);
> -	if (refs & BIT(1))
> -		*flags |= BIT(PG_workingset);
> -	*flags |= (((unsigned long)refs) >> 2) << LRU_REFS_PGOFF;
> -}
> -
> -static inline int folio_lru_refs(const struct folio *folio)
> -{
> -	return lru_refs_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
> -}
> -
> -static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
> -{
> -	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
> -
> -	do {
> -		new_flags = old_flags;
> -		lru_refs_set_flags(&new_flags, refs);
> -	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
> -}
> -
> -int folio_inc_lru_refs(struct folio *folio, bool is_fault, bool is_exec);
> -
>  static inline int folio_lru_gen(const struct folio *folio)
>  {
>  	return lru_gen_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
> @@ -369,11 +472,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
>  	return true;
>  }
>  
> -static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
> -{
> -	folio_set_lru_refs(new, folio_lru_refs(old));
> -}
> -
>  #else /* !CONFIG_LRU_GEN */
>  
>  static inline bool lru_gen_enabled(void)
> @@ -401,27 +499,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
>  	return false;
>  }
>  
> -static inline int folio_lru_refs(const struct folio *folio)
> -{
> -	return 0;
> -}
> -
> -static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
> -{
> -}
> -
> -static inline int folio_inc_lru_refs(struct folio *folio, bool promote, bool is_exec)
> -{
> -	return 0;
> -}
> -
> -static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
> -{
> -	if (folio_test_referenced(old))
> -		folio_set_referenced(new);
> -	if (folio_test_workingset(old))
> -		folio_set_workingset(new);
> -}
>  #endif /* CONFIG_LRU_GEN */
>  
>  static __always_inline
> diff --git a/mm/migrate.c b/mm/migrate.c
> index c737d0682fa4..806f1e913a38 100644
> --- a/mm/migrate.c
> +++ b/mm/migrate.c
> @@ -786,8 +786,6 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
>  		folio_set_active(newfolio);
>  	} else if (folio_test_clear_unevictable(folio))
>  		folio_set_unevictable(newfolio);
> -	if (folio_test_workingset(folio))
> -		folio_set_workingset(newfolio);
>  	if (folio_test_checked(folio))
>  		folio_set_checked(newfolio);
>  	/*
> 
> -- 
> 2.55.0
> 
> 
> 

Sent using hkml (https://github.com/sjp38/hackermail)

Re: [PATCH RFC 08/15] mm/memcg: add folio-based lruvec live helper

[Kairui Song]

On Tue, Aug 4, 2026 at 3:49 PM Lian Wang <lianux.mm@gmail.com> wrote:
>
> From: "Lian Wang (ProcessMission)" <lianux.mm@gmail.com>
>
> Hi Kairui,
>
> I am trying to understand the lifetime and accounting guarantee here, and
> would appreciate your guidance. My understanding is that RCU protects the
> lruvec lifetime, but by itself does not stabilize the folio->lruvec
> association across memcg deletion and reparenting.

Hello,

Thanks for the coment and review!

Yes that's correct. And this helper is just a copy of
lruvec_live_lock_irq without the lock.

See https://lore.kernel.org/linux-mm/20260710154318.75388-1-qi.zheng@linux.dev/

> Could folio_inc_lru_refs() obtain the child lruvec here, then race with
> __lru_gen_reparent_memcg(), and finally account the generation move to the old
> child after the folio and its counters have moved to the parent? The opposite

Just live the patch above, we are fine as reparent of the
non-hierarchical counters itself will fix it.

> ordering also seems possible: this helper observes css_is_dying() and selects
> the parent while the folio is still accounted to the child.
>
> Is there another invariant that closes these races? If my understanding is
> correct, it seems the helper guarantees a live object, but not a stable
> binding, and the lockless promotion path may need validation/retry or explicit
> synchronization with reparenting.
>
> If I have misunderstood the intended synchronization here, please feel free
> to ignore this concern.

See the example and explanation above, I think there is no issue as
long as the final counter is consistent. Some comment could be
helpful; I will add it in V2, thanks again!

Re: [PATCH RFC 10/15] mm/mglru: make folio lru referenced times count a generic API

[Kairui Song]

On Tue, Aug 4, 2026 at 3:49 PM Lian Wang <lianux.mm@gmail.com> wrote:
>
> From: "Lian Wang (ProcessMission)" <lianux.mm@gmail.com>
>
> Hi Kairui,
>
> I am trying to understand the intended semantics of making the referenced
> count a generic API, and would appreciate your guidance. My understanding is
> that, with the new encoding, raw PG_referenced and PG_workingset users no
> longer see simple boolean states for every reference count.

Hello Lian

Yes exactly, actually the first thing come to my mind is DAMON. Now
folio_inc_lru_refs seems a good way for DAMON to feedback hotness info
to MGLRU? It track the referenced total number count, and handles
promotion and refs overflow internally, and affects PID nicely. We
don't have a similiar interface before, folio_mark_access doesn't work
well for MGLRU for this case as it has very limited effect, only work
through PID feedback.

>
> A few examples I found:
>
> - damon_pa_pageout() still calls folio_clear_referenced(). With refs == 2 it
>   clears nothing, and with refs == 3 it leaves refs == 2. Thus DAMOS pageout
>   may retain workingset history instead of clearing the MGLRU reference state.

Right I have a patch before:
https://lore.kernel.org/linux-mm/20260502-mglru-fg-v1-20-913619b014d9@tencent.com/

Just didn't include it this time, series is getting too long...

> - EROFS zdata uses PageWorkingset() for PSI accounting. With the new encoding,
>   the PG_workingset bit is clear for refs == 4 or 5 even though the folio is
>   hot.
> - /proc/kpageflags exports PG_referenced directly, so KPF_REFERENCED appears
>   to become the parity of refs rather than a boolean referenced state.
>
> Are these semantics intended? The DAMON case in particular looks similar to
> the madvise conversion in patch 15. If my understanding is correct, would the
> remaining raw-bit users need a tree-wide audit together with the API
> conversion?

I think most existing workingset / referenced bit user can just
transparently transfer to the new layout if I change
folio_test_workingset itself instead of conver the users one by one,
and my bad I forgot the EROFS case this time :), will fix in v2, I did
convert BTRFS though:
https://lore.kernel.org/linux-mm/20260804-mglru-fg-v1-11-4d8dad39dad6@tencent.com/T/#Z2e.:..:20260804-mglru-fg-v1-11-4d8dad39dad6::40tencent.com:1fs:btrfs:compression.c

And MGLRU's previous PG_workinset certerial is very different from
CLRU and is already causing inaccurate reading of PSI, the new design
actually aligned the PG_workingset defination with CLRU. For
PG_referenced, I actually don't think that flag was really useful
before, because it constantly gets unset / set as the folio is moved
betweet active / inactive for CLRU. The new layout still mostly aligns
with the existing MGLRU design for that referenced flag, and I think
it's clearer now: once a folio is ever seen referenced, the flag
remains and won't disappear (unless some user explicitly needs to
reset refs for some reason). And refaulted folios are definitely
referenced unless they are readahead, and we can try reporting
readahead folios as not referenced if necessary.

>
> If I have misunderstood how these users are expected to behave, please feel
> free to ignore these concerns.
>
> Thanks,
> Lian
>
> On Tue, 04 Aug 2026 03:47:06 +0800 Kairui Song via B4 Relay <devnull+kasong.tencent.com@kernel.org> wrote:
>
...

BTW I suggest you to trim the tailing part on reply, that will make it
easier for others to read the mail :)