[PATCH 0/7] mm/mglru: clean up folio counters and flag usage

[PATCH 0/7] mm/mglru: clean up folio counters and flag usage

[Kairui Song via B4 Relay]

This is a cleanup series separated out from the MGLRU-FG series [1]. As
that series is getting too long in following updates, seperate out the
clean up part for easier review and merge.

No feature change is intended, except one bugfix. It mostly replaces
the open-coded bit operations scattered throughout the MGLRU code with
new helpers, with proper kdocs, sanity debug checks, and hardens a few
MGLRU functions.

A subtle generation counter leak is also found during the refactoring
and the fix is included.

Also collected review feedbacks on the cleanup part from the posted
series, thanks to Barry and Baoquan.

Link: https://lore.kernel.org/linux-mm/20260804-mglru-fg-v1-0-4d8dad39dad6@tencent.com/ [1]
Signed-off-by: Kairui Song <kasong@tencent.com>
---
Kairui Song (7):
      mm/memcontrol: make lru_zone_size atomic and simplify sanity check
      mm/mglru: introduce helpers for manipulating gen and refs flags
      mm/migrate: copy the referenced state via folio_migrate_refs()
      mm/mglru: move max_seq read into walk_update_folio
      mm/mglru: use explicit tier range in read_ctrl_pos()
      mm/mglru: fix potential generation folio number leak
      mm/mglru: improve code readability and harden folio_inc_gen

 include/linux/memcontrol.h |   9 +++-
 include/linux/mm_inline.h  | 102 ++++++++++++++++++++++++++++++-----
 include/linux/mmzone.h     |   4 ++
 mm/folio.c                 |  19 ++++---
 mm/memcontrol.c            |  18 +------
 mm/migrate.c               |   4 +-
 mm/vmscan.c                | 129 ++++++++++++++++++++++++---------------------
 7 files changed, 183 insertions(+), 102 deletions(-)
---
base-commit: 3b1d6bd7bb11fd040bfa7b712486f5bd41a276cf
change-id: 20260818-mglru-flags-cleanup-cc49cfad654f

Best regards,
--  
Kairui Song <kasong@tencent.com>

[PATCH 0/7] mm/mglru: clean up folio counters and flag usage

[Kairui Song]

This is a cleanup series separated out from the MGLRU-FG series [1]. As
that series is getting too long in following updates, seperate out the
clean up part for easier review and merge.

No feature change is intended, except one bugfix. It mostly replaces
the open-coded bit operations scattered throughout the MGLRU code with
new helpers, with proper kdocs, sanity debug checks, and hardens a few
MGLRU functions.

A subtle generation counter leak is also found during the refactoring
and the fix is included.

Also collected review feedbacks on the cleanup part from the posted
series, thanks to Barry and Baoquan.

Link: https://lore.kernel.org/linux-mm/20260804-mglru-fg-v1-0-4d8dad39dad6@tencent.com/ [1]
Signed-off-by: Kairui Song <kasong@tencent.com>
---
Kairui Song (7):
      mm/memcontrol: make lru_zone_size atomic and simplify sanity check
      mm/mglru: introduce helpers for manipulating gen and refs flags
      mm/migrate: copy the referenced state via folio_migrate_refs()
      mm/mglru: move max_seq read into walk_update_folio
      mm/mglru: use explicit tier range in read_ctrl_pos()
      mm/mglru: fix potential generation folio number leak
      mm/mglru: improve code readability and harden folio_inc_gen

 include/linux/memcontrol.h |   9 +++-
 include/linux/mm_inline.h  | 102 ++++++++++++++++++++++++++++++-----
 include/linux/mmzone.h     |   4 ++
 mm/folio.c                 |  19 ++++---
 mm/memcontrol.c            |  18 +------
 mm/migrate.c               |   4 +-
 mm/vmscan.c                | 129 ++++++++++++++++++++++++---------------------
 7 files changed, 183 insertions(+), 102 deletions(-)
---
base-commit: 3b1d6bd7bb11fd040bfa7b712486f5bd41a276cf
change-id: 20260818-mglru-flags-cleanup-cc49cfad654f

Best regards,
--  
Kairui Song <kasong@tencent.com>

[PATCH 1/7] mm/memcontrol: make lru_zone_size atomic and simplify sanity check

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

commit ca707239e8a7 ("mm: update_lru_size warn and reset bad lru_size")
introduced a sanity check to catch memcg counter underflow, which was
more of a workaround for another bug: lru_zone_size is unsigned, so
underflow wraps it around and returns an enormously large number, then
the memcg shrinker loops almost forever as the calculated number of
folios to shrink is huge. That commit also checked if a zero value
matches the empty LRU list, so we have to hold the LRU lock, and
handle the positive and negative deltas separately.

But later commit b4536f0c829c ("mm, memcg: fix the active list aging
for lowmem requests when memcg is enabled") already removed the LRU
emptiness check, so handling the deltas separately is no longer
needed. And if we just turn it into an atomic long, underflow isn't a
big issue either, and can be checked at the reader side, which is
called much less frequently than the updater.

So let's turn the counter into an atomic long and check at the reader
side instead, which has a smaller overhead. The underflow correction
is removed: a massive leak of the LRU size counter would indicate
that something else has gone very wrong, and one should fix that
leaking site instead. Besides, the updater-side sanity check is
unlikely to catch the leaking site anyway: if a folio was removed
without updating the counter while other folios remain on the LRU,
the WARN only triggers much later, from a likely innocent callsite.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/memcontrol.h |  9 +++++++--
 mm/memcontrol.c            | 18 +-----------------
 2 files changed, 8 insertions(+), 19 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index e78bc98ab229..b13e3f056319 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -113,7 +113,7 @@ struct mem_cgroup_per_node {
 	/* Fields which get updated often at the end. */
 	struct lruvec		lruvec;
 	CACHELINE_PADDING(_pad2_);
-	unsigned long		lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS];
+	atomic_long_t		lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS];
 	struct mem_cgroup_reclaim_iter	iter;
 
 	/*
@@ -897,10 +897,15 @@ static inline
 unsigned long mem_cgroup_get_zone_lru_size(struct lruvec *lruvec,
 		enum lru_list lru, int zone_idx)
 {
+	long val;
 	struct mem_cgroup_per_node *mz;
 
 	mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
-	return READ_ONCE(mz->lru_zone_size[zone_idx][lru]);
+	val = atomic_long_read(&mz->lru_zone_size[zone_idx][lru]);
+	if (WARN_ON_ONCE(val < 0))
+		return 0;
+
+	return val;
 }
 
 void __mem_cgroup_handle_over_high(gfp_t gfp_mask);
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 1d3339520809..9d0ee3d3bda7 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1529,28 +1529,12 @@ void mem_cgroup_update_lru_size(struct lruvec *lruvec, enum lru_list lru,
 				int zid, long nr_pages)
 {
 	struct mem_cgroup_per_node *mz;
-	unsigned long *lru_size;
-	long size;
 
 	if (mem_cgroup_disabled())
 		return;
 
 	mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
-	lru_size = &mz->lru_zone_size[zid][lru];
-
-	if (nr_pages < 0)
-		*lru_size += nr_pages;
-
-	size = *lru_size;
-	if (WARN_ONCE(size < 0,
-		"%s(%p, %d, %ld): lru_size %ld\n",
-		__func__, lruvec, lru, nr_pages, size)) {
-		VM_BUG_ON(1);
-		*lru_size = 0;
-	}
-
-	if (nr_pages > 0)
-		*lru_size += nr_pages;
+	atomic_long_add(nr_pages, &mz->lru_zone_size[zid][lru]);
 }
 
 /**

-- 
2.55.0

[PATCH 1/7] mm/memcontrol: make lru_zone_size atomic and simplify sanity check

[Kairui Song]

commit ca707239e8a7 ("mm: update_lru_size warn and reset bad lru_size")
introduced a sanity check to catch memcg counter underflow, which was
more of a workaround for another bug: lru_zone_size is unsigned, so
underflow wraps it around and returns an enormously large number, then
the memcg shrinker loops almost forever as the calculated number of
folios to shrink is huge. That commit also checked if a zero value
matches the empty LRU list, so we have to hold the LRU lock, and
handle the positive and negative deltas separately.

But later commit b4536f0c829c ("mm, memcg: fix the active list aging
for lowmem requests when memcg is enabled") already removed the LRU
emptiness check, so handling the deltas separately is no longer
needed. And if we just turn it into an atomic long, underflow isn't a
big issue either, and can be checked at the reader side, which is
called much less frequently than the updater.

So let's turn the counter into an atomic long and check at the reader
side instead, which has a smaller overhead. The underflow correction
is removed: a massive leak of the LRU size counter would indicate
that something else has gone very wrong, and one should fix that
leaking site instead. Besides, the updater-side sanity check is
unlikely to catch the leaking site anyway: if a folio was removed
without updating the counter while other folios remain on the LRU,
the WARN only triggers much later, from a likely innocent callsite.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/memcontrol.h |  9 +++++++--
 mm/memcontrol.c            | 18 +-----------------
 2 files changed, 8 insertions(+), 19 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index e78bc98ab229..b13e3f056319 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -113,7 +113,7 @@ struct mem_cgroup_per_node {
 	/* Fields which get updated often at the end. */
 	struct lruvec		lruvec;
 	CACHELINE_PADDING(_pad2_);
-	unsigned long		lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS];
+	atomic_long_t		lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS];
 	struct mem_cgroup_reclaim_iter	iter;
 
 	/*
@@ -897,10 +897,15 @@ static inline
 unsigned long mem_cgroup_get_zone_lru_size(struct lruvec *lruvec,
 		enum lru_list lru, int zone_idx)
 {
+	long val;
 	struct mem_cgroup_per_node *mz;
 
 	mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
-	return READ_ONCE(mz->lru_zone_size[zone_idx][lru]);
+	val = atomic_long_read(&mz->lru_zone_size[zone_idx][lru]);
+	if (WARN_ON_ONCE(val < 0))
+		return 0;
+
+	return val;
 }
 
 void __mem_cgroup_handle_over_high(gfp_t gfp_mask);
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 1d3339520809..9d0ee3d3bda7 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1529,28 +1529,12 @@ void mem_cgroup_update_lru_size(struct lruvec *lruvec, enum lru_list lru,
 				int zid, long nr_pages)
 {
 	struct mem_cgroup_per_node *mz;
-	unsigned long *lru_size;
-	long size;
 
 	if (mem_cgroup_disabled())
 		return;
 
 	mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
-	lru_size = &mz->lru_zone_size[zid][lru];
-
-	if (nr_pages < 0)
-		*lru_size += nr_pages;
-
-	size = *lru_size;
-	if (WARN_ONCE(size < 0,
-		"%s(%p, %d, %ld): lru_size %ld\n",
-		__func__, lruvec, lru, nr_pages, size)) {
-		VM_BUG_ON(1);
-		*lru_size = 0;
-	}
-
-	if (nr_pages > 0)
-		*lru_size += nr_pages;
+	atomic_long_add(nr_pages, &mz->lru_zone_size[zid][lru]);
 }
 
 /**

-- 
2.55.0

[PATCH 2/7] mm/mglru: introduce helpers for manipulating gen and refs flags

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

Instead of doing bit ops on folio->flags.f, introduce helpers for
adjusting folio's refs and gen info, make the code easier to debug and
understand.

No functional change is intended: some combined atomic operations are
split into two, which only creates harmless transient states.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/mm_inline.h | 79 +++++++++++++++++++++++++++++++++++++++++------
 include/linux/mmzone.h    |  2 ++
 mm/folio.c                | 19 +++++++-----
 mm/vmscan.c               | 61 ++++++++++++++++++++----------------
 4 files changed, 117 insertions(+), 44 deletions(-)

diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 621c8653d8f7..93bf3fa221f8 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -142,10 +142,43 @@ static inline int lru_tier_from_refs(int refs, bool workingset)
 	return workingset ? MAX_NR_TIERS - 1 : order_base_2(refs);
 }
 
-static inline int folio_lru_refs(const struct folio *folio)
+/**
+ * lru_gen_from_flags - Return the LRU generation number from folio flags.
+ * @flags: folio flags
+ *
+ * Returns: A number between 0 and LRU_GEN_MAX, inclusive. Returns -1 if the
+ * flags indicate the folio is off the list (e.g., isolated).
+ */
+static inline int lru_gen_from_flags(unsigned long flags)
+{
+	int gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF);
+
+	BUILD_BUG_ON(LRU_GEN_MASK & LRU_REFS_MASK);
+	gen -= 1;
+	VM_WARN_ON_ONCE(gen != -1 && gen > LRU_GEN_MAX);
+	return gen;
+}
+
+/**
+ * lru_gen_set_flags - Set the LRU generation number to specified folio flags.
+ * @flags: pointer to the folio flags
+ * @gen: generation number, between 0 and LRU_GEN_MAX, inclusive.
+ */
+static inline void lru_gen_set_flags(unsigned long *flags, int gen)
 {
-	unsigned long flags = READ_ONCE(folio->flags.f);
+	VM_WARN_ON_ONCE(gen > LRU_GEN_MAX || gen < 0);
+	BUILD_BUG_ON((LRU_GEN_MAX + 1) != MAX_NR_GENS);
+
+	*flags &= ~LRU_GEN_MASK;
+	*flags |= (gen + 1UL) << LRU_GEN_PGOFF;
+}
 
+/**
+ * lru_refs_from_flags - Return LRU referenced / access count from folio flags.
+ * @flags: folio flags
+ */
+static inline int lru_refs_from_flags(unsigned long flags)
+{
 	if (!(flags & BIT(PG_referenced)))
 		return 0;
 	/*
@@ -155,18 +188,47 @@ static inline int folio_lru_refs(const struct folio *folio)
 	return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + 1;
 }
 
-static inline int folio_lru_gen(const struct folio *folio)
+/**
+ * lru_refs_set_flags - Set the LRU referenced / access count to specified folio flags.
+ * @flags: pointer to the folio flags
+ * @refs: referenced / access count number, between 0 and LRU_REFS_MAX, inclusive.
+ */
+static inline void lru_refs_set_flags(unsigned long *flags, unsigned int refs)
+{
+	VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
+	BUILD_BUG_ON(LRU_REFS_MAX != (LRU_REFS_MASK >> LRU_REFS_PGOFF) + 1);
+
+	*flags &= ~LRU_REFS_FLAGS;
+	if (!refs)
+		return;
+	*flags |= (BIT(PG_referenced) | ((refs - 1UL) << LRU_REFS_PGOFF));
+}
+
+static inline int folio_lru_refs(const struct folio *folio)
 {
-	unsigned long flags = READ_ONCE(folio->flags.f);
+	return lru_refs_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
+}
+
+static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
+{
+	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
+
+	do {
+		new_flags = old_flags;
+		lru_refs_set_flags(&new_flags, refs);
+	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
+}
 
-	return ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
+static inline int folio_lru_gen(const struct folio *folio)
+{
+	return lru_gen_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
 }
 
 static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen)
 {
 	unsigned long max_seq = lruvec->lrugen.max_seq;
 
-	VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
+	VM_WARN_ON_ONCE(gen > LRU_GEN_MAX);
 
 	/* see the comment on MIN_NR_GENS */
 	return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1);
@@ -270,7 +332,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
 	gen = lru_gen_from_seq(seq);
 	flags = (gen + 1UL) << LRU_GEN_PGOFF;
 	/* see the comment on MIN_NR_GENS about PG_active */
-	set_mask_bits(&folio->flags.f, LRU_GEN_MASK | BIT(PG_active), flags);
+	set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
 
 	lru_gen_update_size(lruvec, folio, -1, gen);
 	/* for folio_rotate_reclaimable() */
@@ -295,7 +357,7 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
 
 	/* for folio_migrate_flags() */
 	flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
-	flags = set_mask_bits(&folio->flags.f, LRU_GEN_MASK, flags);
+	flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, flags);
 	gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
 
 	lru_gen_update_size(lruvec, folio, gen, -1);
@@ -339,7 +401,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
 
 static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
 {
-
 }
 #endif /* CONFIG_LRU_GEN */
 
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 94f9c3ff5416..32d9354a754f 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -496,7 +496,9 @@ enum lruvec_flags {
 #ifndef __GENERATING_BOUNDS_H
 
 #define LRU_GEN_MASK		((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)
+#define LRU_GEN_MAX		(BIT(LRU_GEN_WIDTH - 1) - 1)
 #define LRU_REFS_MASK		((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF)
+#define LRU_REFS_MAX		BIT(LRU_REFS_WIDTH)
 
 /*
  * For folios accessed multiple times through file descriptors,
diff --git a/mm/folio.c b/mm/folio.c
index 59c477120b9a..0adfe4f5ef72 100644
--- a/mm/folio.c
+++ b/mm/folio.c
@@ -353,26 +353,28 @@ static void __lru_cache_activate_folio(struct folio *folio)
 
 static void lru_gen_inc_refs(struct folio *folio)
 {
-	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
+	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
+	int refs;
 
 	if (folio_test_unevictable(folio))
 		return;
 
 	/* see the comment on LRU_REFS_FLAGS */
-	if (!folio_test_referenced(folio)) {
-		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
+	if (!folio_lru_refs(folio)) {
+		folio_set_lru_refs(folio, 1);
 		return;
 	}
 
 	do {
-		if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
+		new_flags = old_flags;
+		refs = lru_refs_from_flags(old_flags);
+		if (refs == LRU_REFS_MAX) {
 			if (!folio_test_workingset(folio))
 				folio_set_workingset(folio);
 			return;
 		}
-
-		new_flags = old_flags + BIT(LRU_REFS_PGOFF);
-	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
+		lru_refs_set_flags(&new_flags, refs + 1);
+	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
 }
 
 static bool lru_gen_clear_refs(struct folio *folio)
@@ -384,7 +386,8 @@ static bool lru_gen_clear_refs(struct folio *folio)
 	if (gen < 0)
 		return true;
 
-	set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS | BIT(PG_workingset), 0);
+	folio_set_lru_refs(folio, 0);
+	folio_clear_workingset(folio);
 
 	rcu_read_lock();
 	seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index c1404a59523d..080132997d87 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -843,19 +843,22 @@ static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags)
 	if (!folio_test_referenced(folio) && !folio_test_workingset(folio)) {
 		/* Activate file-backed executable folios after first usage. */
 		if (is_exec_file_folio(folio, vma_flags)) {
-			set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
+			folio_set_lru_refs(folio, 0);
+			folio_set_workingset(folio);
 			return true;
 		}
 
-		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
+		folio_set_lru_refs(folio, 1);
 		return false;
 	}
 
 	/* Promote on second access */
-	if (folio_lru_refs(folio) > 1)
-		set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
-	else
+	if (folio_lru_refs(folio) > 1) {
+		folio_set_lru_refs(folio, 0);
+		folio_set_workingset(folio);
+	} else {
 		folio_mark_accessed(folio);
+	}
 	return true;
 }
 #else
@@ -3266,11 +3269,10 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv)
  ******************************************************************************/
 
 /* promote pages accessed through page tables */
-static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma_flags)
+static int folio_update_gen(struct folio *folio, int new_gen, const vma_flags_t *vma_flags)
 {
-	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
-
-	VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
+	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
+	int old_gen;
 
 	/*
 	 * See the comment on LRU_REFS_FLAGS, and activate file-backed
@@ -3279,20 +3281,24 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma
 	 */
 	if (!folio_test_referenced(folio) && !folio_test_workingset(folio) &&
 	    !is_exec_file_folio(folio, vma_flags)) {
-		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
+		folio_set_lru_refs(folio, 1);
 		return -1;
 	}
 
 	do {
+		old_gen = lru_gen_from_flags(old_flags);
+		new_flags = old_flags;
+
 		/* lru_gen_del_folio() has isolated this page? */
-		if (!(old_flags & LRU_GEN_MASK))
-			return -1;
+		if (old_gen < 0)
+			break;
 
-		new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);
-		new_flags |= ((gen + 1UL) << LRU_GEN_PGOFF) | BIT(PG_workingset);
-	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
+		lru_gen_set_flags(&new_flags, new_gen);
+		lru_refs_set_flags(&new_flags, 0);
+		new_flags |= BIT(PG_workingset);
+	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
 
-	return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
+	return old_gen;
 }
 
 /* protect pages accessed multiple times through file descriptors */
@@ -3301,21 +3307,20 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
 	int type = folio_is_file_lru(folio);
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
 	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
-	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
-
-	VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio);
+	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
 
 	do {
-		new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
+		new_gen = lru_gen_from_flags(old_flags);
+
 		/* folio_update_gen() has promoted this page? */
 		if (new_gen >= 0 && new_gen != old_gen)
 			return new_gen;
 
+		new_flags = old_flags;
 		new_gen = (old_gen + 1) % MAX_NR_GENS;
-
-		new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);
-		new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF;
-	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
+		lru_gen_set_flags(&new_flags, new_gen);
+		lru_refs_set_flags(&new_flags, 0);
+	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
 
 	lru_gen_update_size(lruvec, folio, old_gen, new_gen);
 
@@ -4711,7 +4716,7 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca
 
 	/* see the comment on LRU_REFS_FLAGS */
 	if (!folio_test_referenced(folio))
-		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, 0);
+		folio_set_lru_refs(folio, 0);
 
 	success = lru_gen_del_folio(lruvec, folio, true);
 	VM_WARN_ON_ONCE_FOLIO(!success, folio);
@@ -4927,8 +4932,10 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 		}
 
 		/* don't add rejected folios to the oldest generation */
-		if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type])
-			set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_active));
+		if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type]) {
+			folio_set_lru_refs(folio, 0);
+			folio_set_active(folio);
+		}
 	}
 
 	move_folios_to_lru(&list);

-- 
2.55.0

[PATCH 2/7] mm/mglru: introduce helpers for manipulating gen and refs flags

[Kairui Song]

Instead of doing bit ops on folio->flags.f, introduce helpers for
adjusting folio's refs and gen info, make the code easier to debug and
understand.

No functional change is intended: some combined atomic operations are
split into two, which only creates harmless transient states.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/mm_inline.h | 79 +++++++++++++++++++++++++++++++++++++++++------
 include/linux/mmzone.h    |  2 ++
 mm/folio.c                | 19 +++++++-----
 mm/vmscan.c               | 61 ++++++++++++++++++++----------------
 4 files changed, 117 insertions(+), 44 deletions(-)

diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 621c8653d8f7..93bf3fa221f8 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -142,10 +142,43 @@ static inline int lru_tier_from_refs(int refs, bool workingset)
 	return workingset ? MAX_NR_TIERS - 1 : order_base_2(refs);
 }
 
-static inline int folio_lru_refs(const struct folio *folio)
+/**
+ * lru_gen_from_flags - Return the LRU generation number from folio flags.
+ * @flags: folio flags
+ *
+ * Returns: A number between 0 and LRU_GEN_MAX, inclusive. Returns -1 if the
+ * flags indicate the folio is off the list (e.g., isolated).
+ */
+static inline int lru_gen_from_flags(unsigned long flags)
+{
+	int gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF);
+
+	BUILD_BUG_ON(LRU_GEN_MASK & LRU_REFS_MASK);
+	gen -= 1;
+	VM_WARN_ON_ONCE(gen != -1 && gen > LRU_GEN_MAX);
+	return gen;
+}
+
+/**
+ * lru_gen_set_flags - Set the LRU generation number to specified folio flags.
+ * @flags: pointer to the folio flags
+ * @gen: generation number, between 0 and LRU_GEN_MAX, inclusive.
+ */
+static inline void lru_gen_set_flags(unsigned long *flags, int gen)
 {
-	unsigned long flags = READ_ONCE(folio->flags.f);
+	VM_WARN_ON_ONCE(gen > LRU_GEN_MAX || gen < 0);
+	BUILD_BUG_ON((LRU_GEN_MAX + 1) != MAX_NR_GENS);
+
+	*flags &= ~LRU_GEN_MASK;
+	*flags |= (gen + 1UL) << LRU_GEN_PGOFF;
+}
 
+/**
+ * lru_refs_from_flags - Return LRU referenced / access count from folio flags.
+ * @flags: folio flags
+ */
+static inline int lru_refs_from_flags(unsigned long flags)
+{
 	if (!(flags & BIT(PG_referenced)))
 		return 0;
 	/*
@@ -155,18 +188,47 @@ static inline int folio_lru_refs(const struct folio *folio)
 	return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + 1;
 }
 
-static inline int folio_lru_gen(const struct folio *folio)
+/**
+ * lru_refs_set_flags - Set the LRU referenced / access count to specified folio flags.
+ * @flags: pointer to the folio flags
+ * @refs: referenced / access count number, between 0 and LRU_REFS_MAX, inclusive.
+ */
+static inline void lru_refs_set_flags(unsigned long *flags, unsigned int refs)
+{
+	VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
+	BUILD_BUG_ON(LRU_REFS_MAX != (LRU_REFS_MASK >> LRU_REFS_PGOFF) + 1);
+
+	*flags &= ~LRU_REFS_FLAGS;
+	if (!refs)
+		return;
+	*flags |= (BIT(PG_referenced) | ((refs - 1UL) << LRU_REFS_PGOFF));
+}
+
+static inline int folio_lru_refs(const struct folio *folio)
 {
-	unsigned long flags = READ_ONCE(folio->flags.f);
+	return lru_refs_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
+}
+
+static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
+{
+	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
+
+	do {
+		new_flags = old_flags;
+		lru_refs_set_flags(&new_flags, refs);
+	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
+}
 
-	return ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
+static inline int folio_lru_gen(const struct folio *folio)
+{
+	return lru_gen_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
 }
 
 static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen)
 {
 	unsigned long max_seq = lruvec->lrugen.max_seq;
 
-	VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
+	VM_WARN_ON_ONCE(gen > LRU_GEN_MAX);
 
 	/* see the comment on MIN_NR_GENS */
 	return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1);
@@ -270,7 +332,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
 	gen = lru_gen_from_seq(seq);
 	flags = (gen + 1UL) << LRU_GEN_PGOFF;
 	/* see the comment on MIN_NR_GENS about PG_active */
-	set_mask_bits(&folio->flags.f, LRU_GEN_MASK | BIT(PG_active), flags);
+	set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
 
 	lru_gen_update_size(lruvec, folio, -1, gen);
 	/* for folio_rotate_reclaimable() */
@@ -295,7 +357,7 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
 
 	/* for folio_migrate_flags() */
 	flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
-	flags = set_mask_bits(&folio->flags.f, LRU_GEN_MASK, flags);
+	flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, flags);
 	gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
 
 	lru_gen_update_size(lruvec, folio, gen, -1);
@@ -339,7 +401,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
 
 static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
 {
-
 }
 #endif /* CONFIG_LRU_GEN */
 
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 94f9c3ff5416..32d9354a754f 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -496,7 +496,9 @@ enum lruvec_flags {
 #ifndef __GENERATING_BOUNDS_H
 
 #define LRU_GEN_MASK		((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)
+#define LRU_GEN_MAX		(BIT(LRU_GEN_WIDTH - 1) - 1)
 #define LRU_REFS_MASK		((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF)
+#define LRU_REFS_MAX		BIT(LRU_REFS_WIDTH)
 
 /*
  * For folios accessed multiple times through file descriptors,
diff --git a/mm/folio.c b/mm/folio.c
index 59c477120b9a..0adfe4f5ef72 100644
--- a/mm/folio.c
+++ b/mm/folio.c
@@ -353,26 +353,28 @@ static void __lru_cache_activate_folio(struct folio *folio)
 
 static void lru_gen_inc_refs(struct folio *folio)
 {
-	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
+	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
+	int refs;
 
 	if (folio_test_unevictable(folio))
 		return;
 
 	/* see the comment on LRU_REFS_FLAGS */
-	if (!folio_test_referenced(folio)) {
-		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
+	if (!folio_lru_refs(folio)) {
+		folio_set_lru_refs(folio, 1);
 		return;
 	}
 
 	do {
-		if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
+		new_flags = old_flags;
+		refs = lru_refs_from_flags(old_flags);
+		if (refs == LRU_REFS_MAX) {
 			if (!folio_test_workingset(folio))
 				folio_set_workingset(folio);
 			return;
 		}
-
-		new_flags = old_flags + BIT(LRU_REFS_PGOFF);
-	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
+		lru_refs_set_flags(&new_flags, refs + 1);
+	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
 }
 
 static bool lru_gen_clear_refs(struct folio *folio)
@@ -384,7 +386,8 @@ static bool lru_gen_clear_refs(struct folio *folio)
 	if (gen < 0)
 		return true;
 
-	set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS | BIT(PG_workingset), 0);
+	folio_set_lru_refs(folio, 0);
+	folio_clear_workingset(folio);
 
 	rcu_read_lock();
 	seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index c1404a59523d..080132997d87 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -843,19 +843,22 @@ static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags)
 	if (!folio_test_referenced(folio) && !folio_test_workingset(folio)) {
 		/* Activate file-backed executable folios after first usage. */
 		if (is_exec_file_folio(folio, vma_flags)) {
-			set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
+			folio_set_lru_refs(folio, 0);
+			folio_set_workingset(folio);
 			return true;
 		}
 
-		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
+		folio_set_lru_refs(folio, 1);
 		return false;
 	}
 
 	/* Promote on second access */
-	if (folio_lru_refs(folio) > 1)
-		set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
-	else
+	if (folio_lru_refs(folio) > 1) {
+		folio_set_lru_refs(folio, 0);
+		folio_set_workingset(folio);
+	} else {
 		folio_mark_accessed(folio);
+	}
 	return true;
 }
 #else
@@ -3266,11 +3269,10 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv)
  ******************************************************************************/
 
 /* promote pages accessed through page tables */
-static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma_flags)
+static int folio_update_gen(struct folio *folio, int new_gen, const vma_flags_t *vma_flags)
 {
-	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
-
-	VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
+	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
+	int old_gen;
 
 	/*
 	 * See the comment on LRU_REFS_FLAGS, and activate file-backed
@@ -3279,20 +3281,24 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma
 	 */
 	if (!folio_test_referenced(folio) && !folio_test_workingset(folio) &&
 	    !is_exec_file_folio(folio, vma_flags)) {
-		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
+		folio_set_lru_refs(folio, 1);
 		return -1;
 	}
 
 	do {
+		old_gen = lru_gen_from_flags(old_flags);
+		new_flags = old_flags;
+
 		/* lru_gen_del_folio() has isolated this page? */
-		if (!(old_flags & LRU_GEN_MASK))
-			return -1;
+		if (old_gen < 0)
+			break;
 
-		new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);
-		new_flags |= ((gen + 1UL) << LRU_GEN_PGOFF) | BIT(PG_workingset);
-	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
+		lru_gen_set_flags(&new_flags, new_gen);
+		lru_refs_set_flags(&new_flags, 0);
+		new_flags |= BIT(PG_workingset);
+	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
 
-	return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
+	return old_gen;
 }
 
 /* protect pages accessed multiple times through file descriptors */
@@ -3301,21 +3307,20 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
 	int type = folio_is_file_lru(folio);
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
 	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
-	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
-
-	VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio);
+	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
 
 	do {
-		new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
+		new_gen = lru_gen_from_flags(old_flags);
+
 		/* folio_update_gen() has promoted this page? */
 		if (new_gen >= 0 && new_gen != old_gen)
 			return new_gen;
 
+		new_flags = old_flags;
 		new_gen = (old_gen + 1) % MAX_NR_GENS;
-
-		new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);
-		new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF;
-	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
+		lru_gen_set_flags(&new_flags, new_gen);
+		lru_refs_set_flags(&new_flags, 0);
+	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
 
 	lru_gen_update_size(lruvec, folio, old_gen, new_gen);
 
@@ -4711,7 +4716,7 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca
 
 	/* see the comment on LRU_REFS_FLAGS */
 	if (!folio_test_referenced(folio))
-		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, 0);
+		folio_set_lru_refs(folio, 0);
 
 	success = lru_gen_del_folio(lruvec, folio, true);
 	VM_WARN_ON_ONCE_FOLIO(!success, folio);
@@ -4927,8 +4932,10 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 		}
 
 		/* don't add rejected folios to the oldest generation */
-		if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type])
-			set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_active));
+		if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type]) {
+			folio_set_lru_refs(folio, 0);
+			folio_set_active(folio);
+		}
 	}
 
 	move_folios_to_lru(&list);

-- 
2.55.0

[PATCH 3/7] mm/migrate: copy the referenced state via folio_migrate_refs()

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

folio_migrate_flags() copies PG_referenced separately, while
folio_migrate_refs() copies the rest of the reference state. Make
folio_migrate_refs() copy the complete state, i.e., PG_referenced plus
the MGLRU refs counter, in both LRU implementations, and drop the
open-coded copy so the reference state is transferred in one place.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/mm_inline.h | 16 +++++++++++++---
 mm/migrate.c              |  4 ++--
 2 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 93bf3fa221f8..df62daaa2ee7 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -366,11 +366,19 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
 	return true;
 }
 
+/**
+ * folio_migrate_refs - copy the reference state to a new folio
+ * @new: the destination folio
+ * @old: the source folio
+ *
+ * Transfer the reference state to @new during migration: the MGLRU
+ * refs count, including PG_referenced, or just PG_referenced for the
+ * active/inactive LRU.
+ */
 static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
 {
-	unsigned long refs = READ_ONCE(old->flags.f) & LRU_REFS_MASK;
-
-	set_mask_bits(&new->flags.f, LRU_REFS_MASK, refs);
+	BUILD_BUG_ON(LRU_REFS_MASK & BIT(PG_referenced));
+	folio_set_lru_refs(new, folio_lru_refs(old));
 }
 #else /* !CONFIG_LRU_GEN */
 
@@ -401,6 +409,8 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
 
 static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
 {
+	if (folio_test_referenced(old))
+		folio_set_referenced(new);
 }
 #endif /* CONFIG_LRU_GEN */
 
diff --git a/mm/migrate.c b/mm/migrate.c
index 15b45832bcfa..82307332711f 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -776,8 +776,6 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
 {
 	int cpupid;
 
-	if (folio_test_referenced(folio))
-		folio_set_referenced(newfolio);
 	if (folio_test_uptodate(folio))
 		folio_mark_uptodate(newfolio);
 	if (folio_test_clear_active(folio)) {
@@ -807,7 +805,9 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
 	if (folio_test_idle(folio))
 		folio_set_idle(newfolio);
 
+	/* Copy the reference state, including PG_referenced */
 	folio_migrate_refs(newfolio, folio);
+
 	/*
 	 * Copy NUMA information to the new page, to prevent over-eager
 	 * future migrations of this same page.

-- 
2.55.0

[PATCH 3/7] mm/migrate: copy the referenced state via folio_migrate_refs()

[Kairui Song]

folio_migrate_flags() copies PG_referenced separately, while
folio_migrate_refs() copies the rest of the reference state. Make
folio_migrate_refs() copy the complete state, i.e., PG_referenced plus
the MGLRU refs counter, in both LRU implementations, and drop the
open-coded copy so the reference state is transferred in one place.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/mm_inline.h | 16 +++++++++++++---
 mm/migrate.c              |  4 ++--
 2 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 93bf3fa221f8..df62daaa2ee7 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -366,11 +366,19 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
 	return true;
 }
 
+/**
+ * folio_migrate_refs - copy the reference state to a new folio
+ * @new: the destination folio
+ * @old: the source folio
+ *
+ * Transfer the reference state to @new during migration: the MGLRU
+ * refs count, including PG_referenced, or just PG_referenced for the
+ * active/inactive LRU.
+ */
 static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
 {
-	unsigned long refs = READ_ONCE(old->flags.f) & LRU_REFS_MASK;
-
-	set_mask_bits(&new->flags.f, LRU_REFS_MASK, refs);
+	BUILD_BUG_ON(LRU_REFS_MASK & BIT(PG_referenced));
+	folio_set_lru_refs(new, folio_lru_refs(old));
 }
 #else /* !CONFIG_LRU_GEN */
 
@@ -401,6 +409,8 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
 
 static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
 {
+	if (folio_test_referenced(old))
+		folio_set_referenced(new);
 }
 #endif /* CONFIG_LRU_GEN */
 
diff --git a/mm/migrate.c b/mm/migrate.c
index 15b45832bcfa..82307332711f 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -776,8 +776,6 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
 {
 	int cpupid;
 
-	if (folio_test_referenced(folio))
-		folio_set_referenced(newfolio);
 	if (folio_test_uptodate(folio))
 		folio_mark_uptodate(newfolio);
 	if (folio_test_clear_active(folio)) {
@@ -807,7 +805,9 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
 	if (folio_test_idle(folio))
 		folio_set_idle(newfolio);
 
+	/* Copy the reference state, including PG_referenced */
 	folio_migrate_refs(newfolio, folio);
+
 	/*
 	 * Copy NUMA information to the new page, to prevent over-eager
 	 * future migrations of this same page.

-- 
2.55.0

[PATCH 4/7] mm/mglru: move max_seq read into walk_update_folio

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

walk_pte_range(), walk_pmd_range_locked(), and lru_gen_look_around()
each read lrugen->max_seq to compute the target generation used by
walk_update_folio(), then pass it as a parameter. Move the read into
walk_update_folio() itself so the callers no longer need to compute
or pass the value.

The max_seq read now happens once per folio update rather than once
per walk range, so folios always get promoted to the current youngest
generation.

Signed-off-by: Kairui Song <kasong@tencent.com>
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
---
 mm/vmscan.c | 29 ++++++++++++-----------------
 1 file changed, 12 insertions(+), 17 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 080132997d87..a819be6b7ae9 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3517,13 +3517,15 @@ static bool suitable_to_scan(int total, int young)
 }
 
 static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struct *vma,
-		struct folio *folio, int new_gen, bool dirty)
+			      struct lruvec *lruvec, struct folio *folio, bool dirty)
 {
-	int old_gen;
+	int new_gen, old_gen;
 
 	if (!folio)
 		return;
 
+	new_gen = lru_gen_from_seq(READ_ONCE(lruvec->lrugen.max_seq));
+
 	if (dirty && !folio_test_dirty(folio) &&
 	    !(folio_test_anon(folio) && folio_test_swapbacked(folio) &&
 	      !folio_test_swapcache(folio)))
@@ -3554,8 +3556,6 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
 	struct lru_gen_mm_walk *walk = args->private;
 	struct mem_cgroup *memcg = lruvec_memcg(walk->lruvec);
 	struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec);
-	DEFINE_MAX_SEQ(walk->lruvec);
-	int gen = lru_gen_from_seq(max_seq);
 	unsigned int nr;
 	pmd_t pmdval;
 
@@ -3606,7 +3606,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
 			continue;
 
 		if (last != folio) {
-			walk_update_folio(walk, args->vma, last, gen, dirty);
+			walk_update_folio(walk, args->vma, walk->lruvec, last, dirty);
 
 			last = folio;
 			dirty = false;
@@ -3619,7 +3619,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
 		walk->mm_stats[MM_LEAF_YOUNG] += nr;
 	}
 
-	walk_update_folio(walk, args->vma, last, gen, dirty);
+	walk_update_folio(walk, args->vma, walk->lruvec, last, dirty);
 	last = NULL;
 
 	if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &end))
@@ -3642,8 +3642,6 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
 	struct lru_gen_mm_walk *walk = args->private;
 	struct mem_cgroup *memcg = lruvec_memcg(walk->lruvec);
 	struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec);
-	DEFINE_MAX_SEQ(walk->lruvec);
-	int gen = lru_gen_from_seq(max_seq);
 
 	VM_WARN_ON_ONCE(pud_leaf(*pud));
 
@@ -3697,7 +3695,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
 			goto next;
 
 		if (last != folio) {
-			walk_update_folio(walk, vma, last, gen, dirty);
+			walk_update_folio(walk, vma, walk->lruvec, last, dirty);
 
 			last = folio;
 			dirty = false;
@@ -3711,7 +3709,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
 		i = i > MIN_LRU_BATCH ? 0 : find_next_bit(bitmap, MIN_LRU_BATCH, i) + 1;
 	} while (i <= MIN_LRU_BATCH);
 
-	walk_update_folio(walk, vma, last, gen, dirty);
+	walk_update_folio(walk, vma, walk->lruvec, last, dirty);
 
 	lazy_mmu_mode_disable();
 	spin_unlock(ptl);
@@ -4275,8 +4273,6 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 	struct pglist_data *pgdat = folio_pgdat(folio);
 	struct lruvec *lruvec;
 	struct lru_gen_mm_state *mm_state;
-	unsigned long max_seq;
-	int gen;
 
 	lockdep_assert_held(pvmw->ptl);
 	VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio);
@@ -4313,8 +4309,6 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 
 	memcg = get_mem_cgroup_from_folio(folio);
 	lruvec = mem_cgroup_lruvec(memcg, pgdat);
-	max_seq = READ_ONCE((lruvec)->lrugen.max_seq);
-	gen = lru_gen_from_seq(max_seq);
 	mm_state = get_mm_state(lruvec);
 
 	lazy_mmu_mode_enable();
@@ -4346,7 +4340,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 			continue;
 
 		if (last != folio) {
-			walk_update_folio(walk, vma, last, gen, dirty);
+			walk_update_folio(walk, vma, lruvec, last, dirty);
 
 			last = folio;
 			dirty = false;
@@ -4358,13 +4352,14 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 		young += nr;
 	}
 
-	walk_update_folio(walk, vma, last, gen, dirty);
+	walk_update_folio(walk, vma, lruvec, last, dirty);
 
 	lazy_mmu_mode_disable();
 
 	/* feedback from rmap walkers to page table walkers */
 	if (mm_state && suitable_to_scan(i, young))
-		update_bloom_filter(mm_state, max_seq, pvmw->pmd);
+		update_bloom_filter(mm_state, READ_ONCE(lruvec->lrugen.max_seq),
+				    pvmw->pmd);
 
 	mem_cgroup_put(memcg);
 

-- 
2.55.0

[PATCH 4/7] mm/mglru: move max_seq read into walk_update_folio

[Kairui Song]

walk_pte_range(), walk_pmd_range_locked(), and lru_gen_look_around()
each read lrugen->max_seq to compute the target generation used by
walk_update_folio(), then pass it as a parameter. Move the read into
walk_update_folio() itself so the callers no longer need to compute
or pass the value.

The max_seq read now happens once per folio update rather than once
per walk range, so folios always get promoted to the current youngest
generation.

Signed-off-by: Kairui Song <kasong@tencent.com>
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
---
 mm/vmscan.c | 29 ++++++++++++-----------------
 1 file changed, 12 insertions(+), 17 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 080132997d87..a819be6b7ae9 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3517,13 +3517,15 @@ static bool suitable_to_scan(int total, int young)
 }
 
 static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struct *vma,
-		struct folio *folio, int new_gen, bool dirty)
+			      struct lruvec *lruvec, struct folio *folio, bool dirty)
 {
-	int old_gen;
+	int new_gen, old_gen;
 
 	if (!folio)
 		return;
 
+	new_gen = lru_gen_from_seq(READ_ONCE(lruvec->lrugen.max_seq));
+
 	if (dirty && !folio_test_dirty(folio) &&
 	    !(folio_test_anon(folio) && folio_test_swapbacked(folio) &&
 	      !folio_test_swapcache(folio)))
@@ -3554,8 +3556,6 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
 	struct lru_gen_mm_walk *walk = args->private;
 	struct mem_cgroup *memcg = lruvec_memcg(walk->lruvec);
 	struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec);
-	DEFINE_MAX_SEQ(walk->lruvec);
-	int gen = lru_gen_from_seq(max_seq);
 	unsigned int nr;
 	pmd_t pmdval;
 
@@ -3606,7 +3606,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
 			continue;
 
 		if (last != folio) {
-			walk_update_folio(walk, args->vma, last, gen, dirty);
+			walk_update_folio(walk, args->vma, walk->lruvec, last, dirty);
 
 			last = folio;
 			dirty = false;
@@ -3619,7 +3619,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
 		walk->mm_stats[MM_LEAF_YOUNG] += nr;
 	}
 
-	walk_update_folio(walk, args->vma, last, gen, dirty);
+	walk_update_folio(walk, args->vma, walk->lruvec, last, dirty);
 	last = NULL;
 
 	if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &end))
@@ -3642,8 +3642,6 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
 	struct lru_gen_mm_walk *walk = args->private;
 	struct mem_cgroup *memcg = lruvec_memcg(walk->lruvec);
 	struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec);
-	DEFINE_MAX_SEQ(walk->lruvec);
-	int gen = lru_gen_from_seq(max_seq);
 
 	VM_WARN_ON_ONCE(pud_leaf(*pud));
 
@@ -3697,7 +3695,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
 			goto next;
 
 		if (last != folio) {
-			walk_update_folio(walk, vma, last, gen, dirty);
+			walk_update_folio(walk, vma, walk->lruvec, last, dirty);
 
 			last = folio;
 			dirty = false;
@@ -3711,7 +3709,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
 		i = i > MIN_LRU_BATCH ? 0 : find_next_bit(bitmap, MIN_LRU_BATCH, i) + 1;
 	} while (i <= MIN_LRU_BATCH);
 
-	walk_update_folio(walk, vma, last, gen, dirty);
+	walk_update_folio(walk, vma, walk->lruvec, last, dirty);
 
 	lazy_mmu_mode_disable();
 	spin_unlock(ptl);
@@ -4275,8 +4273,6 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 	struct pglist_data *pgdat = folio_pgdat(folio);
 	struct lruvec *lruvec;
 	struct lru_gen_mm_state *mm_state;
-	unsigned long max_seq;
-	int gen;
 
 	lockdep_assert_held(pvmw->ptl);
 	VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio);
@@ -4313,8 +4309,6 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 
 	memcg = get_mem_cgroup_from_folio(folio);
 	lruvec = mem_cgroup_lruvec(memcg, pgdat);
-	max_seq = READ_ONCE((lruvec)->lrugen.max_seq);
-	gen = lru_gen_from_seq(max_seq);
 	mm_state = get_mm_state(lruvec);
 
 	lazy_mmu_mode_enable();
@@ -4346,7 +4340,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 			continue;
 
 		if (last != folio) {
-			walk_update_folio(walk, vma, last, gen, dirty);
+			walk_update_folio(walk, vma, lruvec, last, dirty);
 
 			last = folio;
 			dirty = false;
@@ -4358,13 +4352,14 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 		young += nr;
 	}
 
-	walk_update_folio(walk, vma, last, gen, dirty);
+	walk_update_folio(walk, vma, lruvec, last, dirty);
 
 	lazy_mmu_mode_disable();
 
 	/* feedback from rmap walkers to page table walkers */
 	if (mm_state && suitable_to_scan(i, young))
-		update_bloom_filter(mm_state, max_seq, pvmw->pmd);
+		update_bloom_filter(mm_state, READ_ONCE(lruvec->lrugen.max_seq),
+				    pvmw->pmd);
 
 	mem_cgroup_put(memcg);
 

-- 
2.55.0

[PATCH 5/7] mm/mglru: use explicit tier range in read_ctrl_pos()

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

read_ctrl_pos() encodes the tier range in a single "tier" parameter
via "tier % MAX_NR_TIERS" as the start and "min(tier, MAX_NR_TIERS-1)"
as the end. This is hard to follow, maintain, or extend. Tier values
0..3 select a single tier, while tier == MAX_NR_TIERS selects the
full range.

Replace it with explicit (tier_min, tier_max) parameters using a
closed [tier_min, tier_max] interval, and add LRU_TIER_MIN and
LRU_TIER_MAX for the tier bounds. The call sites now become
self-documenting:

  - get_tier_idx:   (LRU_TIER_MIN, LRU_TIER_MIN) for the first tier,
                    (tier, tier) for each subsequent tier
  - get_type_to_scan: (LRU_TIER_MIN, LRU_TIER_MAX) for the full range

No functional change.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/mmzone.h |  2 ++
 mm/vmscan.c            | 18 ++++++++++--------
 2 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 32d9354a754f..d0b5c6217d25 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -492,6 +492,8 @@ enum lruvec_flags {
  * folio->flags, masked by LRU_REFS_MASK.
  */
 #define MAX_NR_TIERS		4U
+#define LRU_TIER_MIN		0U
+#define LRU_TIER_MAX		(MAX_NR_TIERS - 1)
 
 #ifndef __GENERATING_BOUNDS_H
 
diff --git a/mm/vmscan.c b/mm/vmscan.c
index a819be6b7ae9..a613bb8d7271 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3198,8 +3198,8 @@ struct ctrl_pos {
 	int gain;
 };
 
-static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain,
-			  struct ctrl_pos *pos)
+static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier_min,
+			  int tier_max, int gain, struct ctrl_pos *pos)
 {
 	int i;
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
@@ -3208,7 +3208,7 @@ static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain,
 	pos->gain = gain;
 	pos->refaulted = pos->total = 0;
 
-	for (i = tier % MAX_NR_TIERS; i <= min(tier, MAX_NR_TIERS - 1); i++) {
+	for (i = tier_min; i <= tier_max; i++) {
 		pos->refaulted += lrugen->avg_refaulted[type][i] +
 				  atomic_long_read(&lrugen->refaulted[hist][type][i]);
 		pos->total += lrugen->avg_total[type][i] +
@@ -4804,9 +4804,9 @@ static int get_tier_idx(struct lruvec *lruvec, int type)
 	 * This value is chosen because any other tier would have at least twice
 	 * as many refaults as the first tier.
 	 */
-	read_ctrl_pos(lruvec, type, 0, 2, &sp);
-	for (tier = 1; tier < MAX_NR_TIERS; tier++) {
-		read_ctrl_pos(lruvec, type, tier, 3, &pv);
+	read_ctrl_pos(lruvec, type, LRU_TIER_MIN, LRU_TIER_MIN, 2, &sp);
+	for (tier = LRU_TIER_MIN + 1; tier <= LRU_TIER_MAX; tier++) {
+		read_ctrl_pos(lruvec, type, tier, tier, 3, &pv);
 		if (!positive_ctrl_err(&sp, &pv))
 			break;
 	}
@@ -4827,8 +4827,10 @@ static int get_type_to_scan(struct lruvec *lruvec, int swappiness)
 	 * Compare the sum of all tiers of anon with that of file to determine
 	 * which type to scan.
 	 */
-	read_ctrl_pos(lruvec, LRU_GEN_ANON, MAX_NR_TIERS, swappiness, &sp);
-	read_ctrl_pos(lruvec, LRU_GEN_FILE, MAX_NR_TIERS, MAX_SWAPPINESS - swappiness, &pv);
+	read_ctrl_pos(lruvec, LRU_GEN_ANON, LRU_TIER_MIN, LRU_TIER_MAX,
+		      swappiness, &sp);
+	read_ctrl_pos(lruvec, LRU_GEN_FILE, LRU_TIER_MIN, LRU_TIER_MAX,
+		      MAX_SWAPPINESS - swappiness, &pv);
 
 	return positive_ctrl_err(&sp, &pv);
 }

-- 
2.55.0

[PATCH 5/7] mm/mglru: use explicit tier range in read_ctrl_pos()

[Kairui Song]

read_ctrl_pos() encodes the tier range in a single "tier" parameter
via "tier % MAX_NR_TIERS" as the start and "min(tier, MAX_NR_TIERS-1)"
as the end. This is hard to follow, maintain, or extend. Tier values
0..3 select a single tier, while tier == MAX_NR_TIERS selects the
full range.

Replace it with explicit (tier_min, tier_max) parameters using a
closed [tier_min, tier_max] interval, and add LRU_TIER_MIN and
LRU_TIER_MAX for the tier bounds. The call sites now become
self-documenting:

  - get_tier_idx:   (LRU_TIER_MIN, LRU_TIER_MIN) for the first tier,
                    (tier, tier) for each subsequent tier
  - get_type_to_scan: (LRU_TIER_MIN, LRU_TIER_MAX) for the full range

No functional change.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/mmzone.h |  2 ++
 mm/vmscan.c            | 18 ++++++++++--------
 2 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 32d9354a754f..d0b5c6217d25 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -492,6 +492,8 @@ enum lruvec_flags {
  * folio->flags, masked by LRU_REFS_MASK.
  */
 #define MAX_NR_TIERS		4U
+#define LRU_TIER_MIN		0U
+#define LRU_TIER_MAX		(MAX_NR_TIERS - 1)
 
 #ifndef __GENERATING_BOUNDS_H
 
diff --git a/mm/vmscan.c b/mm/vmscan.c
index a819be6b7ae9..a613bb8d7271 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3198,8 +3198,8 @@ struct ctrl_pos {
 	int gain;
 };
 
-static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain,
-			  struct ctrl_pos *pos)
+static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier_min,
+			  int tier_max, int gain, struct ctrl_pos *pos)
 {
 	int i;
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
@@ -3208,7 +3208,7 @@ static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain,
 	pos->gain = gain;
 	pos->refaulted = pos->total = 0;
 
-	for (i = tier % MAX_NR_TIERS; i <= min(tier, MAX_NR_TIERS - 1); i++) {
+	for (i = tier_min; i <= tier_max; i++) {
 		pos->refaulted += lrugen->avg_refaulted[type][i] +
 				  atomic_long_read(&lrugen->refaulted[hist][type][i]);
 		pos->total += lrugen->avg_total[type][i] +
@@ -4804,9 +4804,9 @@ static int get_tier_idx(struct lruvec *lruvec, int type)
 	 * This value is chosen because any other tier would have at least twice
 	 * as many refaults as the first tier.
 	 */
-	read_ctrl_pos(lruvec, type, 0, 2, &sp);
-	for (tier = 1; tier < MAX_NR_TIERS; tier++) {
-		read_ctrl_pos(lruvec, type, tier, 3, &pv);
+	read_ctrl_pos(lruvec, type, LRU_TIER_MIN, LRU_TIER_MIN, 2, &sp);
+	for (tier = LRU_TIER_MIN + 1; tier <= LRU_TIER_MAX; tier++) {
+		read_ctrl_pos(lruvec, type, tier, tier, 3, &pv);
 		if (!positive_ctrl_err(&sp, &pv))
 			break;
 	}
@@ -4827,8 +4827,10 @@ static int get_type_to_scan(struct lruvec *lruvec, int swappiness)
 	 * Compare the sum of all tiers of anon with that of file to determine
 	 * which type to scan.
 	 */
-	read_ctrl_pos(lruvec, LRU_GEN_ANON, MAX_NR_TIERS, swappiness, &sp);
-	read_ctrl_pos(lruvec, LRU_GEN_FILE, MAX_NR_TIERS, MAX_SWAPPINESS - swappiness, &pv);
+	read_ctrl_pos(lruvec, LRU_GEN_ANON, LRU_TIER_MIN, LRU_TIER_MAX,
+		      swappiness, &sp);
+	read_ctrl_pos(lruvec, LRU_GEN_FILE, LRU_TIER_MIN, LRU_TIER_MAX,
+		      MAX_SWAPPINESS - swappiness, &pv);
 
 	return positive_ctrl_err(&sp, &pv);
 }

-- 
2.55.0

[PATCH 6/7] mm/mglru: fix potential generation folio number leak

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

Each generation of MGLRU accounts anon and file folio numbers
separately. The page table walker's update_batch_size() derives the
anon / file type of a folio from its current flags, but the page table
walk holds neither the lruvec lock nor the folio lock, so the type can
change during that period.

MADV_FREE's lazyfree path clears PG_swapbacked under the lruvec lock,
so the folio is no longer considered on the anon LRU list. Lazyfreed
folios can also be changed back to the anon list again. If the flip
lands between folio_update_gen()'s cmpxchg and the type read in
update_batch_size(), the batched delta pair is applied to the wrong
type. The anon and file generation counters then carry phantom deltas
that nothing reconciles, permanently skewing lrugen->nr_pages and the
reclaim budgets derived from it.

Fix it by capturing the type from the flags snapshot the cmpxchg
linearized against: folio_update_gen() returns the type of the state
it transitioned from, and update_batch_size() accounts with it instead
of re-reading the live flags. The batched deltas then always match the
type of the state the cmpxchg transitioned from.

Fixes: 018ee47f1489 ("mm: multi-gen LRU: exploit locality in rmap")
Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/mm_inline.h |  7 ++++++-
 mm/vmscan.c               | 13 +++++++------
 2 files changed, 13 insertions(+), 7 deletions(-)

diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index df62daaa2ee7..4bb390d9516e 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -10,6 +10,11 @@
 #include <linux/userfaultfd_k.h>
 #include <linux/leafops.h>
 
+static inline int folio_flags_is_file_lru(const unsigned long *flags)
+{
+	return !test_bit(PG_swapbacked, flags);
+}
+
 /**
  * folio_is_file_lru - Should the folio be on a file LRU or anon LRU?
  * @folio: The folio to test.
@@ -27,7 +32,7 @@
  */
 static inline int folio_is_file_lru(const struct folio *folio)
 {
-	return !folio_test_swapbacked(folio);
+	return folio_flags_is_file_lru(const_folio_flags(folio, 0));
 }
 
 static __always_inline void __update_lru_size(struct lruvec *lruvec,
diff --git a/mm/vmscan.c b/mm/vmscan.c
index a613bb8d7271..7169cac60869 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3269,7 +3269,8 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv)
  ******************************************************************************/
 
 /* promote pages accessed through page tables */
-static int folio_update_gen(struct folio *folio, int new_gen, const vma_flags_t *vma_flags)
+static int folio_update_gen(struct folio *folio, int new_gen, int *is_file,
+			    const vma_flags_t *vma_flags)
 {
 	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
 	int old_gen;
@@ -3298,6 +3299,7 @@ static int folio_update_gen(struct folio *folio, int new_gen, const vma_flags_t
 		new_flags |= BIT(PG_workingset);
 	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
 
+	*is_file = folio_flags_is_file_lru(&old_flags);
 	return old_gen;
 }
 
@@ -3328,9 +3330,8 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
 }
 
 static void update_batch_size(struct lru_gen_mm_walk *walk, struct folio *folio,
-			      int old_gen, int new_gen)
+			      int old_gen, int new_gen, int type)
 {
-	int type = folio_is_file_lru(folio);
 	int zone = folio_zonenum(folio);
 	int delta = folio_nr_pages(folio);
 
@@ -3519,7 +3520,7 @@ static bool suitable_to_scan(int total, int young)
 static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struct *vma,
 			      struct lruvec *lruvec, struct folio *folio, bool dirty)
 {
-	int new_gen, old_gen;
+	int new_gen, old_gen, file;
 
 	if (!folio)
 		return;
@@ -3532,9 +3533,9 @@ static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struc
 		folio_mark_dirty(folio);
 
 	if (walk) {
-		old_gen = folio_update_gen(folio, new_gen, &vma->flags);
+		old_gen = folio_update_gen(folio, new_gen, &file, &vma->flags);
 		if (old_gen >= 0 && old_gen != new_gen)
-			update_batch_size(walk, folio, old_gen, new_gen);
+			update_batch_size(walk, folio, old_gen, new_gen, file);
 	} else if (lru_gen_set_refs(folio, &vma->flags)) {
 		old_gen = folio_lru_gen(folio);
 		if (old_gen >= 0 && old_gen != new_gen)

-- 
2.55.0

[PATCH 6/7] mm/mglru: fix potential generation folio number leak

[Kairui Song]

Each generation of MGLRU accounts anon and file folio numbers
separately. The page table walker's update_batch_size() derives the
anon / file type of a folio from its current flags, but the page table
walk holds neither the lruvec lock nor the folio lock, so the type can
change during that period.

MADV_FREE's lazyfree path clears PG_swapbacked under the lruvec lock,
so the folio is no longer considered on the anon LRU list. Lazyfreed
folios can also be changed back to the anon list again. If the flip
lands between folio_update_gen()'s cmpxchg and the type read in
update_batch_size(), the batched delta pair is applied to the wrong
type. The anon and file generation counters then carry phantom deltas
that nothing reconciles, permanently skewing lrugen->nr_pages and the
reclaim budgets derived from it.

Fix it by capturing the type from the flags snapshot the cmpxchg
linearized against: folio_update_gen() returns the type of the state
it transitioned from, and update_batch_size() accounts with it instead
of re-reading the live flags. The batched deltas then always match the
type of the state the cmpxchg transitioned from.

Fixes: 018ee47f1489 ("mm: multi-gen LRU: exploit locality in rmap")
Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/mm_inline.h |  7 ++++++-
 mm/vmscan.c               | 13 +++++++------
 2 files changed, 13 insertions(+), 7 deletions(-)

diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index df62daaa2ee7..4bb390d9516e 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -10,6 +10,11 @@
 #include <linux/userfaultfd_k.h>
 #include <linux/leafops.h>
 
+static inline int folio_flags_is_file_lru(const unsigned long *flags)
+{
+	return !test_bit(PG_swapbacked, flags);
+}
+
 /**
  * folio_is_file_lru - Should the folio be on a file LRU or anon LRU?
  * @folio: The folio to test.
@@ -27,7 +32,7 @@
  */
 static inline int folio_is_file_lru(const struct folio *folio)
 {
-	return !folio_test_swapbacked(folio);
+	return folio_flags_is_file_lru(const_folio_flags(folio, 0));
 }
 
 static __always_inline void __update_lru_size(struct lruvec *lruvec,
diff --git a/mm/vmscan.c b/mm/vmscan.c
index a613bb8d7271..7169cac60869 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3269,7 +3269,8 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv)
  ******************************************************************************/
 
 /* promote pages accessed through page tables */
-static int folio_update_gen(struct folio *folio, int new_gen, const vma_flags_t *vma_flags)
+static int folio_update_gen(struct folio *folio, int new_gen, int *is_file,
+			    const vma_flags_t *vma_flags)
 {
 	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
 	int old_gen;
@@ -3298,6 +3299,7 @@ static int folio_update_gen(struct folio *folio, int new_gen, const vma_flags_t
 		new_flags |= BIT(PG_workingset);
 	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
 
+	*is_file = folio_flags_is_file_lru(&old_flags);
 	return old_gen;
 }
 
@@ -3328,9 +3330,8 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
 }
 
 static void update_batch_size(struct lru_gen_mm_walk *walk, struct folio *folio,
-			      int old_gen, int new_gen)
+			      int old_gen, int new_gen, int type)
 {
-	int type = folio_is_file_lru(folio);
 	int zone = folio_zonenum(folio);
 	int delta = folio_nr_pages(folio);
 
@@ -3519,7 +3520,7 @@ static bool suitable_to_scan(int total, int young)
 static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struct *vma,
 			      struct lruvec *lruvec, struct folio *folio, bool dirty)
 {
-	int new_gen, old_gen;
+	int new_gen, old_gen, file;
 
 	if (!folio)
 		return;
@@ -3532,9 +3533,9 @@ static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struc
 		folio_mark_dirty(folio);
 
 	if (walk) {
-		old_gen = folio_update_gen(folio, new_gen, &vma->flags);
+		old_gen = folio_update_gen(folio, new_gen, &file, &vma->flags);
 		if (old_gen >= 0 && old_gen != new_gen)
-			update_batch_size(walk, folio, old_gen, new_gen);
+			update_batch_size(walk, folio, old_gen, new_gen, file);
 	} else if (lru_gen_set_refs(folio, &vma->flags)) {
 		old_gen = folio_lru_gen(folio);
 		if (old_gen >= 0 && old_gen != new_gen)

-- 
2.55.0

[PATCH 7/7] mm/mglru: improve code readability and harden folio_inc_gen

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

The helper should never be called for an off-list folio, and it always
expects the folio to be in the oldest generation before doing any
cmpxchg. Add a sanity check for the off-list case: if it is ever
violated, bail out and keep the folio flags untouched to minimize the
damage, instead of silently treating the folio as if it were in the
oldest generation and promoting it updating the flags to an unexpected
status.

Also rename the variables to clearly distinguish the folio's current
gen from the oldest gen.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 mm/vmscan.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 7169cac60869..7e3ae0c6cba3 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3308,18 +3308,22 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
 {
 	int type = folio_is_file_lru(folio);
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
-	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
+	int new_gen, old_gen, min_gen = lru_gen_from_seq(lrugen->min_seq[type]);
 	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
 
 	do {
-		new_gen = lru_gen_from_flags(old_flags);
+		old_gen = lru_gen_from_flags(old_flags);
+		/* This helper should never be called for off-list folios */
+		VM_WARN_ON_ONCE(old_gen < 0);
+		if (old_gen < 0)
+			return min_gen;
 
 		/* folio_update_gen() has promoted this page? */
-		if (new_gen >= 0 && new_gen != old_gen)
-			return new_gen;
+		if (old_gen != min_gen)
+			return old_gen;
 
 		new_flags = old_flags;
-		new_gen = (old_gen + 1) % MAX_NR_GENS;
+		new_gen = (min_gen + 1) % MAX_NR_GENS;
 		lru_gen_set_flags(&new_flags, new_gen);
 		lru_refs_set_flags(&new_flags, 0);
 	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));

-- 
2.55.0

[PATCH 7/7] mm/mglru: improve code readability and harden folio_inc_gen

[Kairui Song]

The helper should never be called for an off-list folio, and it always
expects the folio to be in the oldest generation before doing any
cmpxchg. Add a sanity check for the off-list case: if it is ever
violated, bail out and keep the folio flags untouched to minimize the
damage, instead of silently treating the folio as if it were in the
oldest generation and promoting it updating the flags to an unexpected
status.

Also rename the variables to clearly distinguish the folio's current
gen from the oldest gen.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 mm/vmscan.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 7169cac60869..7e3ae0c6cba3 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3308,18 +3308,22 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
 {
 	int type = folio_is_file_lru(folio);
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
-	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
+	int new_gen, old_gen, min_gen = lru_gen_from_seq(lrugen->min_seq[type]);
 	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
 
 	do {
-		new_gen = lru_gen_from_flags(old_flags);
+		old_gen = lru_gen_from_flags(old_flags);
+		/* This helper should never be called for off-list folios */
+		VM_WARN_ON_ONCE(old_gen < 0);
+		if (old_gen < 0)
+			return min_gen;
 
 		/* folio_update_gen() has promoted this page? */
-		if (new_gen >= 0 && new_gen != old_gen)
-			return new_gen;
+		if (old_gen != min_gen)
+			return old_gen;
 
 		new_flags = old_flags;
-		new_gen = (old_gen + 1) % MAX_NR_GENS;
+		new_gen = (min_gen + 1) % MAX_NR_GENS;
 		lru_gen_set_flags(&new_flags, new_gen);
 		lru_refs_set_flags(&new_flags, 0);
 	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));

-- 
2.55.0

Re: [PATCH 1/7] mm/memcontrol: make lru_zone_size atomic and simplify sanity check

[Ridong Chen]

On 8/18/2026 1:38 PM, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
> 
> commit ca707239e8a7 ("mm: update_lru_size warn and reset bad lru_size")
> introduced a sanity check to catch memcg counter underflow, which was
> more of a workaround for another bug: lru_zone_size is unsigned, so
> underflow wraps it around and returns an enormously large number, then
> the memcg shrinker loops almost forever as the calculated number of
> folios to shrink is huge. That commit also checked if a zero value
> matches the empty LRU list, so we have to hold the LRU lock, and
> handle the positive and negative deltas separately.
> 
> But later commit b4536f0c829c ("mm, memcg: fix the active list aging
> for lowmem requests when memcg is enabled") already removed the LRU
> emptiness check, so handling the deltas separately is no longer
> needed. And if we just turn it into an atomic long, underflow isn't a
> big issue either, and can be checked at the reader side, which is
> called much less frequently than the updater.
> 
> So let's turn the counter into an atomic long and check at the reader
> side instead, which has a smaller overhead. The underflow correction
> is removed: a massive leak of the LRU size counter would indicate
> that something else has gone very wrong, and one should fix that
> leaking site instead. Besides, the updater-side sanity check is
> unlikely to catch the leaking site anyway: if a folio was removed
> without updating the counter while other folios remain on the LRU,
> the WARN only triggers much later, from a likely innocent callsite.
> 
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
>   include/linux/memcontrol.h |  9 +++++++--
>   mm/memcontrol.c            | 18 +-----------------
>   2 files changed, 8 insertions(+), 19 deletions(-)
> 
> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> index e78bc98ab229..b13e3f056319 100644
> --- a/include/linux/memcontrol.h
> +++ b/include/linux/memcontrol.h
> @@ -113,7 +113,7 @@ struct mem_cgroup_per_node {
>   	/* Fields which get updated often at the end. */
>   	struct lruvec		lruvec;
>   	CACHELINE_PADDING(_pad2_);
> -	unsigned long		lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS];
> +	atomic_long_t		lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS];
>   	struct mem_cgroup_reclaim_iter	iter;
>   
>   	/*
> @@ -897,10 +897,15 @@ static inline
>   unsigned long mem_cgroup_get_zone_lru_size(struct lruvec *lruvec,
>   		enum lru_list lru, int zone_idx)
>   {
> +	long val;
>   	struct mem_cgroup_per_node *mz;
>   
>   	mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
> -	return READ_ONCE(mz->lru_zone_size[zone_idx][lru]);
> +	val = atomic_long_read(&mz->lru_zone_size[zone_idx][lru]);
> +	if (WARN_ON_ONCE(val < 0))
> +		return 0;
> +
> +	return val;
>   }
>   
>   void __mem_cgroup_handle_over_high(gfp_t gfp_mask);
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 1d3339520809..9d0ee3d3bda7 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -1529,28 +1529,12 @@ void mem_cgroup_update_lru_size(struct lruvec *lruvec, enum lru_list lru,
>   				int zid, long nr_pages)
>   {
>   	struct mem_cgroup_per_node *mz;
> -	unsigned long *lru_size;
> -	long size;
>   
>   	if (mem_cgroup_disabled())
>   		return;
>   
>   	mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
> -	lru_size = &mz->lru_zone_size[zid][lru];
> -
> -	if (nr_pages < 0)
> -		*lru_size += nr_pages;
> -
> -	size = *lru_size;
> -	if (WARN_ONCE(size < 0,
> -		"%s(%p, %d, %ld): lru_size %ld\n",
> -		__func__, lruvec, lru, nr_pages, size)) {
> -		VM_BUG_ON(1);
> -		*lru_size = 0;
> -	}
> -

This code is confusing, I used to try to remove it.

Well done.


> -	if (nr_pages > 0)
> -		*lru_size += nr_pages;
> +	atomic_long_add(nr_pages, &mz->lru_zone_size[zid][lru]);
>   }
>   
>   /**
> 

Looks good to me.

Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
-- 
Best regards
Ridong

Re: [PATCH 2/7] mm/mglru: introduce helpers for manipulating gen and refs flags

[Baolin Wang]

On 8/18/26 1:38 PM, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
> 
> Instead of doing bit ops on folio->flags.f, introduce helpers for
> adjusting folio's refs and gen info, make the code easier to debug and
> understand.
> 
> No functional change is intended: some combined atomic operations are
> split into two, which only creates harmless transient states.
> 
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---

Thanks for the cleanups. One comment below.

>   include/linux/mm_inline.h | 79 +++++++++++++++++++++++++++++++++++++++++------
>   include/linux/mmzone.h    |  2 ++
>   mm/folio.c                | 19 +++++++-----
>   mm/vmscan.c               | 61 ++++++++++++++++++++----------------
>   4 files changed, 117 insertions(+), 44 deletions(-)
> 
> diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
> index 621c8653d8f7..93bf3fa221f8 100644
> --- a/include/linux/mm_inline.h
> +++ b/include/linux/mm_inline.h
> @@ -142,10 +142,43 @@ static inline int lru_tier_from_refs(int refs, bool workingset)
>   	return workingset ? MAX_NR_TIERS - 1 : order_base_2(refs);
>   }
>   
> -static inline int folio_lru_refs(const struct folio *folio)
> +/**
> + * lru_gen_from_flags - Return the LRU generation number from folio flags.
> + * @flags: folio flags
> + *
> + * Returns: A number between 0 and LRU_GEN_MAX, inclusive. Returns -1 if the
> + * flags indicate the folio is off the list (e.g., isolated).
> + */
> +static inline int lru_gen_from_flags(unsigned long flags)
> +{
> +	int gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF);
> +
> +	BUILD_BUG_ON(LRU_GEN_MASK & LRU_REFS_MASK);
> +	gen -= 1;
> +	VM_WARN_ON_ONCE(gen != -1 && gen > LRU_GEN_MAX);
> +	return gen;
> +}
> +
> +/**
> + * lru_gen_set_flags - Set the LRU generation number to specified folio flags.
> + * @flags: pointer to the folio flags
> + * @gen: generation number, between 0 and LRU_GEN_MAX, inclusive.
> + */
> +static inline void lru_gen_set_flags(unsigned long *flags, int gen)
>   {
> -	unsigned long flags = READ_ONCE(folio->flags.f);
> +	VM_WARN_ON_ONCE(gen > LRU_GEN_MAX || gen < 0);
> +	BUILD_BUG_ON((LRU_GEN_MAX + 1) != MAX_NR_GENS);
> +
> +	*flags &= ~LRU_GEN_MASK;
> +	*flags |= (gen + 1UL) << LRU_GEN_PGOFF;
> +}
>   
> +/**
> + * lru_refs_from_flags - Return LRU referenced / access count from folio flags.
> + * @flags: folio flags
> + */
> +static inline int lru_refs_from_flags(unsigned long flags)
> +{
>   	if (!(flags & BIT(PG_referenced)))
>   		return 0;
>   	/*
> @@ -155,18 +188,47 @@ static inline int folio_lru_refs(const struct folio *folio)
>   	return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + 1;
>   }
>   
> -static inline int folio_lru_gen(const struct folio *folio)
> +/**
> + * lru_refs_set_flags - Set the LRU referenced / access count to specified folio flags.
> + * @flags: pointer to the folio flags
> + * @refs: referenced / access count number, between 0 and LRU_REFS_MAX, inclusive.
> + */
> +static inline void lru_refs_set_flags(unsigned long *flags, unsigned int refs)
> +{
> +	VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
> +	BUILD_BUG_ON(LRU_REFS_MAX != (LRU_REFS_MASK >> LRU_REFS_PGOFF) + 1);
> +
> +	*flags &= ~LRU_REFS_FLAGS;
> +	if (!refs)
> +		return;
> +	*flags |= (BIT(PG_referenced) | ((refs - 1UL) << LRU_REFS_PGOFF));
> +}
> +
> +static inline int folio_lru_refs(const struct folio *folio)
>   {
> -	unsigned long flags = READ_ONCE(folio->flags.f);
> +	return lru_refs_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
> +}
> +
> +static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
> +{
> +	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
> +
> +	do {
> +		new_flags = old_flags;
> +		lru_refs_set_flags(&new_flags, refs);
> +	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
> +}
>   
> -	return ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
> +static inline int folio_lru_gen(const struct folio *folio)
> +{
> +	return lru_gen_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
>   }
>   
>   static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen)
>   {
>   	unsigned long max_seq = lruvec->lrugen.max_seq;
>   
> -	VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
> +	VM_WARN_ON_ONCE(gen > LRU_GEN_MAX);
>   
>   	/* see the comment on MIN_NR_GENS */
>   	return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1);
> @@ -270,7 +332,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
>   	gen = lru_gen_from_seq(seq);
>   	flags = (gen + 1UL) << LRU_GEN_PGOFF;
>   	/* see the comment on MIN_NR_GENS about PG_active */
> -	set_mask_bits(&folio->flags.f, LRU_GEN_MASK | BIT(PG_active), flags);
> +	set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
>   
>   	lru_gen_update_size(lruvec, folio, -1, gen);
>   	/* for folio_rotate_reclaimable() */
> @@ -295,7 +357,7 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
>   
>   	/* for folio_migrate_flags() */
>   	flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
> -	flags = set_mask_bits(&folio->flags.f, LRU_GEN_MASK, flags);
> +	flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, flags);
>   	gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
>   
>   	lru_gen_update_size(lruvec, folio, gen, -1);
> @@ -339,7 +401,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
>   
>   static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
>   {
> -
>   }
>   #endif /* CONFIG_LRU_GEN */
>   
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index 94f9c3ff5416..32d9354a754f 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -496,7 +496,9 @@ enum lruvec_flags {
>   #ifndef __GENERATING_BOUNDS_H
>   
>   #define LRU_GEN_MASK		((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)
> +#define LRU_GEN_MAX		(BIT(LRU_GEN_WIDTH - 1) - 1)

There are still many places using MAX_NR_GENS to compare gen counters, 
so I'm concerned that introducing a new macro would make the use of the 
max gen counter even more confusing.

Re: [PATCH 4/7] mm/mglru: move max_seq read into walk_update_folio

[Baolin Wang]

On 8/18/26 1:38 PM, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
> 
> walk_pte_range(), walk_pmd_range_locked(), and lru_gen_look_around()
> each read lrugen->max_seq to compute the target generation used by
> walk_update_folio(), then pass it as a parameter. Move the read into
> walk_update_folio() itself so the callers no longer need to compute
> or pass the value.
> 
> The max_seq read now happens once per folio update rather than once
> per walk range, so folios always get promoted to the current youngest
> generation.
> 
> Signed-off-by: Kairui Song <kasong@tencent.com>
> Reviewed-by: Baoquan He <baoquan.he@linux.dev>
> ---

LGTM. One nit below.
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>

>   mm/vmscan.c | 29 ++++++++++++-----------------
>   1 file changed, 12 insertions(+), 17 deletions(-)
> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 080132997d87..a819be6b7ae9 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3517,13 +3517,15 @@ static bool suitable_to_scan(int total, int young)
>   }
>   
>   static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struct *vma,
> -		struct folio *folio, int new_gen, bool dirty)
> +			      struct lruvec *lruvec, struct folio *folio, bool dirty)

IIRC, MM maintainers prefer 2 tabs, which is why I changed this earlier:)

Re: [PATCH 5/7] mm/mglru: use explicit tier range in read_ctrl_pos()

[Baolin Wang]

On 8/18/26 1:38 PM, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
> 
> read_ctrl_pos() encodes the tier range in a single "tier" parameter
> via "tier % MAX_NR_TIERS" as the start and "min(tier, MAX_NR_TIERS-1)"
> as the end. This is hard to follow, maintain, or extend. Tier values
> 0..3 select a single tier, while tier == MAX_NR_TIERS selects the
> full range.
> 
> Replace it with explicit (tier_min, tier_max) parameters using a
> closed [tier_min, tier_max] interval, and add LRU_TIER_MIN and
> LRU_TIER_MAX for the tier bounds. The call sites now become
> self-documenting:
> 
>    - get_tier_idx:   (LRU_TIER_MIN, LRU_TIER_MIN) for the first tier,
>                      (tier, tier) for each subsequent tier
>    - get_type_to_scan: (LRU_TIER_MIN, LRU_TIER_MAX) for the full range
> 
> No functional change.
> 
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---

Yes. More readable.
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>

Re: [PATCH 2/7] mm/mglru: introduce helpers for manipulating gen and refs flags

[Kairui Song]

On Wed, Aug 19, 2026 at 5:03 PM Baolin Wang
<baolin.wang@linux.alibaba.com> wrote:
> On 8/18/26 1:38 PM, Kairui Song via B4 Relay wrote:
> > From: Kairui Song <kasong@tencent.com>
> >
> > Instead of doing bit ops on folio->flags.f, introduce helpers for
> > adjusting folio's refs and gen info, make the code easier to debug and
> > understand.
> >
> > No functional change is intended: some combined atomic operations are
> > split into two, which only creates harmless transient states.
> >
> > Signed-off-by: Kairui Song <kasong@tencent.com>
> > ---
>
> Thanks for the cleanups. One comment below.
>

Thanks for the review!

> > diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> > index 94f9c3ff5416..32d9354a754f 100644
> > --- a/include/linux/mmzone.h
> > +++ b/include/linux/mmzone.h
> > @@ -496,7 +496,9 @@ enum lruvec_flags {
> >   #ifndef __GENERATING_BOUNDS_H
> >
> >   #define LRU_GEN_MASK                ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)
> > +#define LRU_GEN_MAX          (BIT(LRU_GEN_WIDTH - 1) - 1)
>
> There are still many places using MAX_NR_GENS to compare gen counters,
> so I'm concerned that introducing a new macro would make the use of the
> max gen counter even more confusing.

I think I'd like to get rid of all LRU_GEN_MASK and LRU_REFS_MASK
users, MAX_NR_GENS is fine I think? Manipulating the bitmask looks a
bit hard to follow for me since:

set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, <number>);
set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_active));
set_mask_bits(&folio->flags.f, LRU_REFS_MASK, <number>);
new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);
new_flags |= ((gen + 1UL) << LRU_GEN_PGOFF) | BIT(PG_workingset);

Things like this are everywhere and I found it really hard to follow
what is actually going on. And adding more operations to refs/gen
looks ugly and painful to do, and, it's impossible to add any sanity
check. With proper gen and refs helpers, we can ensure every time
gen/refs is modified, the value is valid (in debug builds).

Mean while, a "% MAX_NR_GENS" or "/ MIN_NR_GEN" seems not hard to understand.

Oh and this LRU_GEN_MAX is only used for sanity check for now.

Re: [PATCH 2/7] mm/mglru: introduce helpers for manipulating gen and refs flags

[Baolin Wang]

On 8/19/26 5:37 PM, Kairui Song wrote:
> On Wed, Aug 19, 2026 at 5:03 PM Baolin Wang
> <baolin.wang@linux.alibaba.com> wrote:
>> On 8/18/26 1:38 PM, Kairui Song via B4 Relay wrote:
>>> From: Kairui Song <kasong@tencent.com>
>>>
>>> Instead of doing bit ops on folio->flags.f, introduce helpers for
>>> adjusting folio's refs and gen info, make the code easier to debug and
>>> understand.
>>>
>>> No functional change is intended: some combined atomic operations are
>>> split into two, which only creates harmless transient states.
>>>
>>> Signed-off-by: Kairui Song <kasong@tencent.com>
>>> ---
>>
>> Thanks for the cleanups. One comment below.
>>
> 
> Thanks for the review!
> 
>>> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
>>> index 94f9c3ff5416..32d9354a754f 100644
>>> --- a/include/linux/mmzone.h
>>> +++ b/include/linux/mmzone.h
>>> @@ -496,7 +496,9 @@ enum lruvec_flags {
>>>    #ifndef __GENERATING_BOUNDS_H
>>>
>>>    #define LRU_GEN_MASK                ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)
>>> +#define LRU_GEN_MAX          (BIT(LRU_GEN_WIDTH - 1) - 1)
>>
>> There are still many places using MAX_NR_GENS to compare gen counters,
>> so I'm concerned that introducing a new macro would make the use of the
>> max gen counter even more confusing.
> 
> I think I'd like to get rid of all LRU_GEN_MASK and LRU_REFS_MASK
> users, MAX_NR_GENS is fine I think? Manipulating the bitmask looks a
> bit hard to follow for me since:
> 
> set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
> set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
> set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, <number>);
> set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_active));
> set_mask_bits(&folio->flags.f, LRU_REFS_MASK, <number>);
> new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);
> new_flags |= ((gen + 1UL) << LRU_GEN_PGOFF) | BIT(PG_workingset);
> 
> Things like this are everywhere and I found it really hard to follow
> what is actually going on. And adding more operations to refs/gen
> looks ugly and painful to do, and, it's impossible to add any sanity
> check. With proper gen and refs helpers, we can ensure every time
> gen/refs is modified, the value is valid (in debug builds).

I agree LRU_GEN_MASK and LRU_REFS_MASK can be cleaned up further.

> 
> Mean while, a "% MAX_NR_GENS" or "/ MIN_NR_GEN" seems not hard to understand.
> 
> Oh and this LRU_GEN_MAX is only used for sanity check for now.

But my point is that people may not know when to use LRU_GEN_MAX vs. 
MAX_NR_GENS, since they seem to have the same semantics.

Re: [PATCH 2/7] mm/mglru: introduce helpers for manipulating gen and refs flags

[Kairui Song]

On Wed, Aug 19, 2026 at 5:46 PM Baolin Wang
<baolin.wang@linux.alibaba.com> wrote:
> On 8/19/26 5:37 PM, Kairui Song wrote:
> > On Wed, Aug 19, 2026 at 5:03 PM Baolin Wang
> > <baolin.wang@linux.alibaba.com> wrote:
> >> On 8/18/26 1:38 PM, Kairui Song via B4 Relay wrote:
> >>> From: Kairui Song <kasong@tencent.com>
> >>>
> >>> Instead of doing bit ops on folio->flags.f, introduce helpers for
> >>> adjusting folio's refs and gen info, make the code easier to debug and
> >>> understand.
> >>>
> >>> No functional change is intended: some combined atomic operations are
> >>> split into two, which only creates harmless transient states.
> >>>
> >>> Signed-off-by: Kairui Song <kasong@tencent.com>
> >>> ---
> >>
> >> Thanks for the cleanups. One comment below.
> >>
> >
> > Thanks for the review!
> >
> >>> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> >>> index 94f9c3ff5416..32d9354a754f 100644
> >>> --- a/include/linux/mmzone.h
> >>> +++ b/include/linux/mmzone.h
> >>> @@ -496,7 +496,9 @@ enum lruvec_flags {
> >>>    #ifndef __GENERATING_BOUNDS_H
> >>>
> >>>    #define LRU_GEN_MASK                ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)
> >>> +#define LRU_GEN_MAX          (BIT(LRU_GEN_WIDTH - 1) - 1)
> >>
> >> There are still many places using MAX_NR_GENS to compare gen counters,
> >> so I'm concerned that introducing a new macro would make the use of the
> >> max gen counter even more confusing.
> >
> > I think I'd like to get rid of all LRU_GEN_MASK and LRU_REFS_MASK
> > users, MAX_NR_GENS is fine I think? Manipulating the bitmask looks a
> > bit hard to follow for me since:
> >
> > set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
> > set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
> > set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, <number>);
> > set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_active));
> > set_mask_bits(&folio->flags.f, LRU_REFS_MASK, <number>);
> > new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);
> > new_flags |= ((gen + 1UL) << LRU_GEN_PGOFF) | BIT(PG_workingset);
> >
> > Things like this are everywhere and I found it really hard to follow
> > what is actually going on. And adding more operations to refs/gen
> > looks ugly and painful to do, and, it's impossible to add any sanity
> > check. With proper gen and refs helpers, we can ensure every time
> > gen/refs is modified, the value is valid (in debug builds).
>
> I agree LRU_GEN_MASK and LRU_REFS_MASK can be cleaned up further.
>
> >
> > Mean while, a "% MAX_NR_GENS" or "/ MIN_NR_GEN" seems not hard to understand.
> >
> > Oh and this LRU_GEN_MAX is only used for sanity check for now.
>
> But my point is that people may not know when to use LRU_GEN_MAX vs.
> MAX_NR_GENS, since they seem to have the same semantics.

Right, I think let me just drop LRU_GEN_MAX then, there is no real
user, it's just sanity check macro.

Re: [PATCH 3/7] mm/migrate: copy the referenced state via folio_migrate_refs()

[Baoquan He]

On 08/18/26 at 01:38pm, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
> 
> folio_migrate_flags() copies PG_referenced separately, while
> folio_migrate_refs() copies the rest of the reference state. Make
> folio_migrate_refs() copy the complete state, i.e., PG_referenced plus
> the MGLRU refs counter, in both LRU implementations, and drop the
> open-coded copy so the reference state is transferred in one place.
> 
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
>  include/linux/mm_inline.h | 16 +++++++++++++---
>  mm/migrate.c              |  4 ++--
>  2 files changed, 15 insertions(+), 5 deletions(-)

LGTM,

Reviewed-by: Baoquan He <baoquan.he@linux.dev>

> 
> diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
> index 93bf3fa221f8..df62daaa2ee7 100644
> --- a/include/linux/mm_inline.h
> +++ b/include/linux/mm_inline.h
> @@ -366,11 +366,19 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
>  	return true;
>  }
>  
> +/**
> + * folio_migrate_refs - copy the reference state to a new folio
> + * @new: the destination folio
> + * @old: the source folio
> + *
> + * Transfer the reference state to @new during migration: the MGLRU
> + * refs count, including PG_referenced, or just PG_referenced for the
> + * active/inactive LRU.
> + */
>  static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
>  {
> -	unsigned long refs = READ_ONCE(old->flags.f) & LRU_REFS_MASK;
> -
> -	set_mask_bits(&new->flags.f, LRU_REFS_MASK, refs);
> +	BUILD_BUG_ON(LRU_REFS_MASK & BIT(PG_referenced));
> +	folio_set_lru_refs(new, folio_lru_refs(old));
>  }
>  #else /* !CONFIG_LRU_GEN */
>  
> @@ -401,6 +409,8 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
>  
>  static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
>  {
> +	if (folio_test_referenced(old))
> +		folio_set_referenced(new);
>  }
>  #endif /* CONFIG_LRU_GEN */
>  
> diff --git a/mm/migrate.c b/mm/migrate.c
> index 15b45832bcfa..82307332711f 100644
> --- a/mm/migrate.c
> +++ b/mm/migrate.c
> @@ -776,8 +776,6 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
>  {
>  	int cpupid;
>  
> -	if (folio_test_referenced(folio))
> -		folio_set_referenced(newfolio);
>  	if (folio_test_uptodate(folio))
>  		folio_mark_uptodate(newfolio);
>  	if (folio_test_clear_active(folio)) {
> @@ -807,7 +805,9 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
>  	if (folio_test_idle(folio))
>  		folio_set_idle(newfolio);
>  
> +	/* Copy the reference state, including PG_referenced */
>  	folio_migrate_refs(newfolio, folio);
> +
>  	/*
>  	 * Copy NUMA information to the new page, to prevent over-eager
>  	 * future migrations of this same page.
> 
> -- 
> 2.55.0
> 
> 

Re: [PATCH 5/7] mm/mglru: use explicit tier range in read_ctrl_pos()

[Baoquan He]

On 08/18/26 at 01:38pm, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
> 
> read_ctrl_pos() encodes the tier range in a single "tier" parameter
> via "tier % MAX_NR_TIERS" as the start and "min(tier, MAX_NR_TIERS-1)"
> as the end. This is hard to follow, maintain, or extend. Tier values
> 0..3 select a single tier, while tier == MAX_NR_TIERS selects the
> full range.
> 
> Replace it with explicit (tier_min, tier_max) parameters using a
> closed [tier_min, tier_max] interval, and add LRU_TIER_MIN and
> LRU_TIER_MAX for the tier bounds. The call sites now become
> self-documenting:
> 
>   - get_tier_idx:   (LRU_TIER_MIN, LRU_TIER_MIN) for the first tier,
>                     (tier, tier) for each subsequent tier
>   - get_type_to_scan: (LRU_TIER_MIN, LRU_TIER_MAX) for the full range
> 
> No functional change.
> 
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
>  include/linux/mmzone.h |  2 ++
>  mm/vmscan.c            | 18 ++++++++++--------
>  2 files changed, 12 insertions(+), 8 deletions(-)

To be honest, I like the old version better. The complexity is
encapsulated inside the function, and tier_idx can reflect if it's
operating on a specific tier or the whole tier. Anyway, this is also
good:

Reviewed-by: Baoquan He <baoquan.he@linux.dev>

> 
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index 32d9354a754f..d0b5c6217d25 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -492,6 +492,8 @@ enum lruvec_flags {
>   * folio->flags, masked by LRU_REFS_MASK.
>   */
>  #define MAX_NR_TIERS		4U
> +#define LRU_TIER_MIN		0U
> +#define LRU_TIER_MAX		(MAX_NR_TIERS - 1)
>  
>  #ifndef __GENERATING_BOUNDS_H
>  
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index a819be6b7ae9..a613bb8d7271 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3198,8 +3198,8 @@ struct ctrl_pos {
>  	int gain;
>  };
>  
> -static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain,
> -			  struct ctrl_pos *pos)
> +static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier_min,
> +			  int tier_max, int gain, struct ctrl_pos *pos)
>  {
>  	int i;
>  	struct lru_gen_folio *lrugen = &lruvec->lrugen;
> @@ -3208,7 +3208,7 @@ static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain,
>  	pos->gain = gain;
>  	pos->refaulted = pos->total = 0;
>  
> -	for (i = tier % MAX_NR_TIERS; i <= min(tier, MAX_NR_TIERS - 1); i++) {
> +	for (i = tier_min; i <= tier_max; i++) {
>  		pos->refaulted += lrugen->avg_refaulted[type][i] +
>  				  atomic_long_read(&lrugen->refaulted[hist][type][i]);
>  		pos->total += lrugen->avg_total[type][i] +
> @@ -4804,9 +4804,9 @@ static int get_tier_idx(struct lruvec *lruvec, int type)
>  	 * This value is chosen because any other tier would have at least twice
>  	 * as many refaults as the first tier.
>  	 */
> -	read_ctrl_pos(lruvec, type, 0, 2, &sp);
> -	for (tier = 1; tier < MAX_NR_TIERS; tier++) {
> -		read_ctrl_pos(lruvec, type, tier, 3, &pv);
> +	read_ctrl_pos(lruvec, type, LRU_TIER_MIN, LRU_TIER_MIN, 2, &sp);
> +	for (tier = LRU_TIER_MIN + 1; tier <= LRU_TIER_MAX; tier++) {
> +		read_ctrl_pos(lruvec, type, tier, tier, 3, &pv);
>  		if (!positive_ctrl_err(&sp, &pv))
>  			break;
>  	}
> @@ -4827,8 +4827,10 @@ static int get_type_to_scan(struct lruvec *lruvec, int swappiness)
>  	 * Compare the sum of all tiers of anon with that of file to determine
>  	 * which type to scan.
>  	 */
> -	read_ctrl_pos(lruvec, LRU_GEN_ANON, MAX_NR_TIERS, swappiness, &sp);
> -	read_ctrl_pos(lruvec, LRU_GEN_FILE, MAX_NR_TIERS, MAX_SWAPPINESS - swappiness, &pv);
> +	read_ctrl_pos(lruvec, LRU_GEN_ANON, LRU_TIER_MIN, LRU_TIER_MAX,
> +		      swappiness, &sp);
> +	read_ctrl_pos(lruvec, LRU_GEN_FILE, LRU_TIER_MIN, LRU_TIER_MAX,
> +		      MAX_SWAPPINESS - swappiness, &pv);
>  
>  	return positive_ctrl_err(&sp, &pv);
>  }
> 
> -- 
> 2.55.0
> 
> 

Re: [PATCH 5/7] mm/mglru: use explicit tier range in read_ctrl_pos()

[Barry Song]

On Tue, Aug 18, 2026 at 1:38 PM Kairui Song via B4 Relay
<devnull+kasong.tencent.com@kernel.org> wrote:
>
> From: Kairui Song <kasong@tencent.com>
>
> read_ctrl_pos() encodes the tier range in a single "tier" parameter
> via "tier % MAX_NR_TIERS" as the start and "min(tier, MAX_NR_TIERS-1)"
> as the end. This is hard to follow, maintain, or extend. Tier values
> 0..3 select a single tier, while tier == MAX_NR_TIERS selects the
> full range.
>
> Replace it with explicit (tier_min, tier_max) parameters using a
> closed [tier_min, tier_max] interval, and add LRU_TIER_MIN and
> LRU_TIER_MAX for the tier bounds. The call sites now become
> self-documenting:
>
>   - get_tier_idx:   (LRU_TIER_MIN, LRU_TIER_MIN) for the first tier,
>                     (tier, tier) for each subsequent tier
>   - get_type_to_scan: (LRU_TIER_MIN, LRU_TIER_MAX) for the full range
>
> No functional change.
>
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---

Reviewed-by: Barry Song <baohua@kernel.org>

Re: [PATCH 7/7] mm/mglru: improve code readability and harden folio_inc_gen

[Barry Song]

On Tue, Aug 18, 2026 at 1:38 PM Kairui Song via B4 Relay
<devnull+kasong.tencent.com@kernel.org> wrote:
>
> From: Kairui Song <kasong@tencent.com>
>
> The helper should never be called for an off-list folio, and it always
> expects the folio to be in the oldest generation before doing any

I'm really not sure whether that's true. Could the folios have been
found to be young more than once during the last two or three (or more)
aging cycles?

> cmpxchg. Add a sanity check for the off-list case: if it is ever
> violated, bail out and keep the folio flags untouched to minimize the
> damage, instead of silently treating the folio as if it were in the
> oldest generation and promoting it updating the flags to an unexpected
> status.

My printk shows that a folio can be promoted two or three times.
If a folio is promoted too many times, is that the cold/hot inversion
you are trying to fix?

>
> Also rename the variables to clearly distinguish the folio's current
> gen from the oldest gen.
>
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---

Best Regards
Barry

Re: [PATCH 7/7] mm/mglru: improve code readability and harden folio_inc_gen

[Baoquan He]

On 08/18/26 at 01:38pm, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
> 
> The helper should never be called for an off-list folio, and it always
> expects the folio to be in the oldest generation before doing any
> cmpxchg. Add a sanity check for the off-list case: if it is ever
> violated, bail out and keep the folio flags untouched to minimize the
> damage, instead of silently treating the folio as if it were in the
> oldest generation and promoting it updating the flags to an unexpected
> status.
> 
> Also rename the variables to clearly distinguish the folio's current
> gen from the oldest gen.
> 
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
>  mm/vmscan.c | 14 +++++++++-----
>  1 file changed, 9 insertions(+), 5 deletions(-)
> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 7169cac60869..7e3ae0c6cba3 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3308,18 +3308,22 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
>  {
>  	int type = folio_is_file_lru(folio);
>  	struct lru_gen_folio *lrugen = &lruvec->lrugen;
> -	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
> +	int new_gen, old_gen, min_gen = lru_gen_from_seq(lrugen->min_seq[type]);
>  	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
>  
>  	do {
> -		new_gen = lru_gen_from_flags(old_flags);
> +		old_gen = lru_gen_from_flags(old_flags);
> +		/* This helper should never be called for off-list folios */
> +		VM_WARN_ON_ONCE(old_gen < 0);
> +		if (old_gen < 0)
> +			return min_gen;

As Barry doubted, I think this change is wrong. old_gen < 0 in folio_inc_gen()
could only happen inc_min_seq() call it. While inc_min_seq() call it
because inc_max_seq() need increase max_gen to max_gen + 1 and found
get_nr_gens(lruvec, type) == MAX_NR_GENS, it has to move the oldest gen to
2nd old oldest gen. Here returning min_gen for old_gen < 0 means it will
be put in the lastest max_gen. It may not be expected.

>  
>  		/* folio_update_gen() has promoted this page? */
> -		if (new_gen >= 0 && new_gen != old_gen)
> -			return new_gen;
> +		if (old_gen != min_gen)
> +			return old_gen;
>  
>  		new_flags = old_flags;
> -		new_gen = (old_gen + 1) % MAX_NR_GENS;
> +		new_gen = (min_gen + 1) % MAX_NR_GENS;
>  		lru_gen_set_flags(&new_flags, new_gen);
>  		lru_refs_set_flags(&new_flags, 0);
>  	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
> 
> -- 
> 2.55.0
> 
> 

Re: [PATCH 7/7] mm/mglru: improve code readability and harden folio_inc_gen

[Baoquan He]

On 08/20/26 at 08:53am, Baoquan He wrote:
> On 08/18/26 at 01:38pm, Kairui Song via B4 Relay wrote:
> > From: Kairui Song <kasong@tencent.com>
> > 
> > The helper should never be called for an off-list folio, and it always
> > expects the folio to be in the oldest generation before doing any
> > cmpxchg. Add a sanity check for the off-list case: if it is ever
> > violated, bail out and keep the folio flags untouched to minimize the
> > damage, instead of silently treating the folio as if it were in the
> > oldest generation and promoting it updating the flags to an unexpected
> > status.
> > 
> > Also rename the variables to clearly distinguish the folio's current
> > gen from the oldest gen.
> > 
> > Signed-off-by: Kairui Song <kasong@tencent.com>
> > ---
> >  mm/vmscan.c | 14 +++++++++-----
> >  1 file changed, 9 insertions(+), 5 deletions(-)
> > 
> > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > index 7169cac60869..7e3ae0c6cba3 100644
> > --- a/mm/vmscan.c
> > +++ b/mm/vmscan.c
> > @@ -3308,18 +3308,22 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
> >  {
> >  	int type = folio_is_file_lru(folio);
> >  	struct lru_gen_folio *lrugen = &lruvec->lrugen;
> > -	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
> > +	int new_gen, old_gen, min_gen = lru_gen_from_seq(lrugen->min_seq[type]);
> >  	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
> >  
> >  	do {
> > -		new_gen = lru_gen_from_flags(old_flags);
> > +		old_gen = lru_gen_from_flags(old_flags);
> > +		/* This helper should never be called for off-list folios */
> > +		VM_WARN_ON_ONCE(old_gen < 0);
> > +		if (old_gen < 0)
> > +			return min_gen;
> 
> As Barry doubted, I think this change is wrong. old_gen < 0 in folio_inc_gen()
> could only happen inc_min_seq() call it. While inc_min_seq() call it
> because inc_max_seq() need increase max_gen to max_gen + 1 and found
> get_nr_gens(lruvec, type) == MAX_NR_GENS, it has to move the oldest gen to
                                            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 2nd old oldest gen. Here returning min_gen for old_gen < 0 means it will
  ~~~~~~~~~~~~~~~~
Here, I mean it has to move folios from the oldest gen (min_gen) to the 2nd
oldest gen (min_gen + 1). The empty min_gen will become the new max_gen.

> be put in the lastest max_gen. It may not be expected.
> 
> >  
> >  		/* folio_update_gen() has promoted this page? */
> > -		if (new_gen >= 0 && new_gen != old_gen)
> > -			return new_gen;
> > +		if (old_gen != min_gen)
> > +			return old_gen;
> >  
> >  		new_flags = old_flags;
> > -		new_gen = (old_gen + 1) % MAX_NR_GENS;
> > +		new_gen = (min_gen + 1) % MAX_NR_GENS;
> >  		lru_gen_set_flags(&new_flags, new_gen);
> >  		lru_refs_set_flags(&new_flags, 0);
> >  	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
> > 
> > -- 
> > 2.55.0
> > 
> > 

Re: [PATCH 7/7] mm/mglru: improve code readability and harden folio_inc_gen

[Baolin Wang]

On 8/20/26 8:57 AM, Baoquan He wrote:
> On 08/20/26 at 08:53am, Baoquan He wrote:
>> On 08/18/26 at 01:38pm, Kairui Song via B4 Relay wrote:
>>> From: Kairui Song <kasong@tencent.com>
>>>
>>> The helper should never be called for an off-list folio, and it always
>>> expects the folio to be in the oldest generation before doing any
>>> cmpxchg. Add a sanity check for the off-list case: if it is ever
>>> violated, bail out and keep the folio flags untouched to minimize the
>>> damage, instead of silently treating the folio as if it were in the
>>> oldest generation and promoting it updating the flags to an unexpected
>>> status.
>>>
>>> Also rename the variables to clearly distinguish the folio's current
>>> gen from the oldest gen.
>>>
>>> Signed-off-by: Kairui Song <kasong@tencent.com>
>>> ---
>>>   mm/vmscan.c | 14 +++++++++-----
>>>   1 file changed, 9 insertions(+), 5 deletions(-)
>>>
>>> diff --git a/mm/vmscan.c b/mm/vmscan.c
>>> index 7169cac60869..7e3ae0c6cba3 100644
>>> --- a/mm/vmscan.c
>>> +++ b/mm/vmscan.c
>>> @@ -3308,18 +3308,22 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
>>>   {
>>>   	int type = folio_is_file_lru(folio);
>>>   	struct lru_gen_folio *lrugen = &lruvec->lrugen;
>>> -	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
>>> +	int new_gen, old_gen, min_gen = lru_gen_from_seq(lrugen->min_seq[type]);
>>>   	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
>>>   
>>>   	do {
>>> -		new_gen = lru_gen_from_flags(old_flags);
>>> +		old_gen = lru_gen_from_flags(old_flags);
>>> +		/* This helper should never be called for off-list folios */
>>> +		VM_WARN_ON_ONCE(old_gen < 0);
>>> +		if (old_gen < 0)
>>> +			return min_gen;
>>
>> As Barry doubted, I think this change is wrong. old_gen < 0 in folio_inc_gen()
>> could only happen inc_min_seq() call it. While inc_min_seq() call it
>> because inc_max_seq() need increase max_gen to max_gen + 1 and found
>> get_nr_gens(lruvec, type) == MAX_NR_GENS, it has to move the oldest gen to
>                                              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> 2nd old oldest gen. Here returning min_gen for old_gen < 0 means it will
>    ~~~~~~~~~~~~~~~~
> Here, I mean it has to move folios from the oldest gen (min_gen) to the 2nd
> oldest gen (min_gen + 1). The empty min_gen will become the new max_gen.
> 
>> be put in the lastest max_gen. It may not be expected.

But how does old_gen < 0 actually happen? folio_inc_gen() is called 
under the lru lock, so how can a folio listed in MGLRU have a gen 
counter < 0? If this can happen in any case, we should fix this bug first.

Re: [PATCH 2/7] mm/mglru: introduce helpers for manipulating gen and refs flags

[Ridong Chen]

On 8/18/2026 1:38 PM, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
> 
> Instead of doing bit ops on folio->flags.f, introduce helpers for
> adjusting folio's refs and gen info, make the code easier to debug and
> understand.
> 
> No functional change is intended: some combined atomic operations are
> split into two, which only creates harmless transient states.
> 
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
>   include/linux/mm_inline.h | 79 +++++++++++++++++++++++++++++++++++++++++------
>   include/linux/mmzone.h    |  2 ++
>   mm/folio.c                | 19 +++++++-----
>   mm/vmscan.c               | 61 ++++++++++++++++++++----------------
>   4 files changed, 117 insertions(+), 44 deletions(-)
> 
> diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
> index 621c8653d8f7..93bf3fa221f8 100644
> --- a/include/linux/mm_inline.h
> +++ b/include/linux/mm_inline.h
> @@ -142,10 +142,43 @@ static inline int lru_tier_from_refs(int refs, bool workingset)
>   	return workingset ? MAX_NR_TIERS - 1 : order_base_2(refs);
>   }
>   
> -static inline int folio_lru_refs(const struct folio *folio)
> +/**
> + * lru_gen_from_flags - Return the LRU generation number from folio flags.
> + * @flags: folio flags
> + *
> + * Returns: A number between 0 and LRU_GEN_MAX, inclusive. Returns -1 if the
> + * flags indicate the folio is off the list (e.g., isolated).
> + */
> +static inline int lru_gen_from_flags(unsigned long flags)
> +{
> +	int gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF);
> +
> +	BUILD_BUG_ON(LRU_GEN_MASK & LRU_REFS_MASK);
> +	gen -= 1;
> +	VM_WARN_ON_ONCE(gen != -1 && gen > LRU_GEN_MAX);

Perhaps we could define a macro such as GEN_OFF = -1 to make the code more 
self-explanatory, I found this warning a bit confusing at first glance.

LRU_GEN_MAX already bears some resemblance to MAX_NR_GENS, so introducing yet 
another macro may add some clutter.

Just my two cents.

> +	return gen;
> +}
> +
> +/**
> + * lru_gen_set_flags - Set the LRU generation number to specified folio flags.
> + * @flags: pointer to the folio flags
> + * @gen: generation number, between 0 and LRU_GEN_MAX, inclusive.
> + */
> +static inline void lru_gen_set_flags(unsigned long *flags, int gen)
>   {
> -	unsigned long flags = READ_ONCE(folio->flags.f);
> +	VM_WARN_ON_ONCE(gen > LRU_GEN_MAX || gen < 0);
> +	BUILD_BUG_ON((LRU_GEN_MAX + 1) != MAX_NR_GENS);
> +
> +	*flags &= ~LRU_GEN_MASK;
> +	*flags |= (gen + 1UL) << LRU_GEN_PGOFF;
> +}
>   
> +/**
> + * lru_refs_from_flags - Return LRU referenced / access count from folio flags.
> + * @flags: folio flags
> + */
> +static inline int lru_refs_from_flags(unsigned long flags)
> +{
>   	if (!(flags & BIT(PG_referenced)))
>   		return 0;
>   	/*
> @@ -155,18 +188,47 @@ static inline int folio_lru_refs(const struct folio *folio)
>   	return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + 1;
>   }
>   
> -static inline int folio_lru_gen(const struct folio *folio)
> +/**
> + * lru_refs_set_flags - Set the LRU referenced / access count to specified folio flags.
> + * @flags: pointer to the folio flags
> + * @refs: referenced / access count number, between 0 and LRU_REFS_MAX, inclusive.
> + */
> +static inline void lru_refs_set_flags(unsigned long *flags, unsigned int refs)
> +{
> +	VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
> +	BUILD_BUG_ON(LRU_REFS_MAX != (LRU_REFS_MASK >> LRU_REFS_PGOFF) + 1);
> +
> +	*flags &= ~LRU_REFS_FLAGS;
> +	if (!refs)
> +		return;
> +	*flags |= (BIT(PG_referenced) | ((refs - 1UL) << LRU_REFS_PGOFF));
> +}
> +
> +static inline int folio_lru_refs(const struct folio *folio)
>   {
> -	unsigned long flags = READ_ONCE(folio->flags.f);
> +	return lru_refs_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
> +}
> +
> +static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
> +{
> +	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
> +
> +	do {
> +		new_flags = old_flags;
> +		lru_refs_set_flags(&new_flags, refs);
> +	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
> +}
>   
> -	return ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
> +static inline int folio_lru_gen(const struct folio *folio)
> +{
> +	return lru_gen_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
>   }
>   
>   static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen)
>   {
>   	unsigned long max_seq = lruvec->lrugen.max_seq;
>   
> -	VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
> +	VM_WARN_ON_ONCE(gen > LRU_GEN_MAX);
>   
>   	/* see the comment on MIN_NR_GENS */
>   	return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1);
> @@ -270,7 +332,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
>   	gen = lru_gen_from_seq(seq);
>   	flags = (gen + 1UL) << LRU_GEN_PGOFF;
>   	/* see the comment on MIN_NR_GENS about PG_active */
> -	set_mask_bits(&folio->flags.f, LRU_GEN_MASK | BIT(PG_active), flags);
> +	set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
>   
>   	lru_gen_update_size(lruvec, folio, -1, gen);
>   	/* for folio_rotate_reclaimable() */
> @@ -295,7 +357,7 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
>   
>   	/* for folio_migrate_flags() */
>   	flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
> -	flags = set_mask_bits(&folio->flags.f, LRU_GEN_MASK, flags);
> +	flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, flags);
>   	gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
>   
>   	lru_gen_update_size(lruvec, folio, gen, -1);
> @@ -339,7 +401,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
>   
>   static inline void folio_migrate_refs(struct folio *new, const struct folio *old)
>   {
> -
>   }
>   #endif /* CONFIG_LRU_GEN */
>   
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index 94f9c3ff5416..32d9354a754f 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -496,7 +496,9 @@ enum lruvec_flags {
>   #ifndef __GENERATING_BOUNDS_H
>   
>   #define LRU_GEN_MASK		((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)
> +#define LRU_GEN_MAX		(BIT(LRU_GEN_WIDTH - 1) - 1)
>   #define LRU_REFS_MASK		((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF)
> +#define LRU_REFS_MAX		BIT(LRU_REFS_WIDTH)
>   
>   /*
>    * For folios accessed multiple times through file descriptors,
> diff --git a/mm/folio.c b/mm/folio.c
> index 59c477120b9a..0adfe4f5ef72 100644
> --- a/mm/folio.c
> +++ b/mm/folio.c
> @@ -353,26 +353,28 @@ static void __lru_cache_activate_folio(struct folio *folio)
>   
>   static void lru_gen_inc_refs(struct folio *folio)
>   {
> -	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
> +	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
> +	int refs;
>   
>   	if (folio_test_unevictable(folio))
>   		return;
>   
>   	/* see the comment on LRU_REFS_FLAGS */
> -	if (!folio_test_referenced(folio)) {
> -		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
> +	if (!folio_lru_refs(folio)) {
> +		folio_set_lru_refs(folio, 1);
>   		return;
>   	}
>   
>   	do {
> -		if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
> +		new_flags = old_flags;
> +		refs = lru_refs_from_flags(old_flags);
> +		if (refs == LRU_REFS_MAX) {
>   			if (!folio_test_workingset(folio))
>   				folio_set_workingset(folio);
>   			return;
>   		}
> -
> -		new_flags = old_flags + BIT(LRU_REFS_PGOFF);
> -	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
> +		lru_refs_set_flags(&new_flags, refs + 1);
> +	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>   }
>   
>   static bool lru_gen_clear_refs(struct folio *folio)
> @@ -384,7 +386,8 @@ static bool lru_gen_clear_refs(struct folio *folio)
>   	if (gen < 0)
>   		return true;
>   
> -	set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS | BIT(PG_workingset), 0);
> +	folio_set_lru_refs(folio, 0);
> +	folio_clear_workingset(folio);
>   
>   	rcu_read_lock();
>   	seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index c1404a59523d..080132997d87 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -843,19 +843,22 @@ static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags)
>   	if (!folio_test_referenced(folio) && !folio_test_workingset(folio)) {
>   		/* Activate file-backed executable folios after first usage. */
>   		if (is_exec_file_folio(folio, vma_flags)) {
> -			set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
> +			folio_set_lru_refs(folio, 0);
> +			folio_set_workingset(folio);
>   			return true;
>   		}
>   
> -		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
> +		folio_set_lru_refs(folio, 1);
>   		return false;
>   	}
>   
>   	/* Promote on second access */
> -	if (folio_lru_refs(folio) > 1)
> -		set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
> -	else
> +	if (folio_lru_refs(folio) > 1) {
> +		folio_set_lru_refs(folio, 0);
> +		folio_set_workingset(folio);
> +	} else {
>   		folio_mark_accessed(folio);
> +	}
>   	return true;
>   }
>   #else
> @@ -3266,11 +3269,10 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv)
>    ******************************************************************************/
>   
>   /* promote pages accessed through page tables */
> -static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma_flags)
> +static int folio_update_gen(struct folio *folio, int new_gen, const vma_flags_t *vma_flags)
>   {
> -	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
> -
> -	VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
> +	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
> +	int old_gen;
>   
>   	/*
>   	 * See the comment on LRU_REFS_FLAGS, and activate file-backed
> @@ -3279,20 +3281,24 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma
>   	 */
>   	if (!folio_test_referenced(folio) && !folio_test_workingset(folio) &&
>   	    !is_exec_file_folio(folio, vma_flags)) {
> -		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
> +		folio_set_lru_refs(folio, 1);
>   		return -1;
>   	}
>   
>   	do {
> +		old_gen = lru_gen_from_flags(old_flags);
> +		new_flags = old_flags;
> +
>   		/* lru_gen_del_folio() has isolated this page? */
> -		if (!(old_flags & LRU_GEN_MASK))
> -			return -1;
> +		if (old_gen < 0)
> +			break;
>   
> -		new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);
> -		new_flags |= ((gen + 1UL) << LRU_GEN_PGOFF) | BIT(PG_workingset);
> -	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
> +		lru_gen_set_flags(&new_flags, new_gen);
> +		lru_refs_set_flags(&new_flags, 0);
> +		new_flags |= BIT(PG_workingset);
> +	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>   
> -	return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
> +	return old_gen;
>   }
>   
>   /* protect pages accessed multiple times through file descriptors */
> @@ -3301,21 +3307,20 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
>   	int type = folio_is_file_lru(folio);
>   	struct lru_gen_folio *lrugen = &lruvec->lrugen;
>   	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
> -	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
> -
> -	VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio);
> +	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
>   
>   	do {
> -		new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
> +		new_gen = lru_gen_from_flags(old_flags);
> +
>   		/* folio_update_gen() has promoted this page? */
>   		if (new_gen >= 0 && new_gen != old_gen)
>   			return new_gen;
>   
> +		new_flags = old_flags;
>   		new_gen = (old_gen + 1) % MAX_NR_GENS;
> -
> -		new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);
> -		new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF;
> -	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
> +		lru_gen_set_flags(&new_flags, new_gen);
> +		lru_refs_set_flags(&new_flags, 0);
> +	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>   
>   	lru_gen_update_size(lruvec, folio, old_gen, new_gen);
>   
> @@ -4711,7 +4716,7 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca
>   
>   	/* see the comment on LRU_REFS_FLAGS */
>   	if (!folio_test_referenced(folio))
> -		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, 0);
> +		folio_set_lru_refs(folio, 0);
>   
>   	success = lru_gen_del_folio(lruvec, folio, true);
>   	VM_WARN_ON_ONCE_FOLIO(!success, folio);
> @@ -4927,8 +4932,10 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
>   		}
>   
>   		/* don't add rejected folios to the oldest generation */
> -		if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type])
> -			set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_active));
> +		if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type]) {
> +			folio_set_lru_refs(folio, 0);
> +			folio_set_active(folio);
> +		}
>   	}
>   
>   	move_folios_to_lru(&list);
> 

-- 
Best regards
Ridong

Re: [PATCH 6/7] mm/mglru: fix potential generation folio number leak

[Baolin Wang]

On 8/18/26 1:38 PM, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
> 
> Each generation of MGLRU accounts anon and file folio numbers
> separately. The page table walker's update_batch_size() derives the
> anon / file type of a folio from its current flags, but the page table
> walk holds neither the lruvec lock nor the folio lock, so the type can
> change during that period.

Right.

> MADV_FREE's lazyfree path clears PG_swapbacked under the lruvec lock,
> so the folio is no longer considered on the anon LRU list. Lazyfreed
> folios can also be changed back to the anon list again. If the flip
> lands between folio_update_gen()'s cmpxchg and the type read in
> update_batch_size(), the batched delta pair is applied to the wrong
> type. The anon and file generation counters then carry phantom deltas
> that nothing reconciles, permanently skewing lrugen->nr_pages and the
> reclaim budgets derived from it.

But I think the problem occurs between update_batch_size() and 
sort_folio(). update_batch_size() only updates the anon or file folio 
statistics, while sort_folio() moves promoted folios to the 
corresponding type's list:

/* promoted */
if (gen != lru_gen_from_seq(lrugen->min_seq[type])) {
	list_move(&folio->lru, &lrugen->folios[gen][type][zone]);
	return true;
}

If the folio's anon/file type changes between these two steps (e.g., a 
lazyfree folio), it would lead to what you described: "The anon and file 
generation counters then carry phantom deltas that nothing reconciles, 
permanently skewing lrugen->nr_pages and the reclaim budgets derived 
from it."

If you agree that this is where the problem lies, I don't see a good way 
to fix it, since the state of a lazyfree folio can change between 
update_batch_size() and sort_folio().

A simple approach would be to skip checking the access flag for lazyfree 
folios during the page table walk, and let shrink_folio_list() 
reactivate accessed lazyfree folios instead. What do you think?

> Fix it by capturing the type from the flags snapshot the cmpxchg
> linearized against: folio_update_gen() returns the type of the state
> it transitioned from, and update_batch_size() accounts with it instead
> of re-reading the live flags. The batched deltas then always match the
> type of the state the cmpxchg transitioned from.
> 
> Fixes: 018ee47f1489 ("mm: multi-gen LRU: exploit locality in rmap")
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
>   include/linux/mm_inline.h |  7 ++++++-
>   mm/vmscan.c               | 13 +++++++------
>   2 files changed, 13 insertions(+), 7 deletions(-)
> 
> diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
> index df62daaa2ee7..4bb390d9516e 100644
> --- a/include/linux/mm_inline.h
> +++ b/include/linux/mm_inline.h
> @@ -10,6 +10,11 @@
>   #include <linux/userfaultfd_k.h>
>   #include <linux/leafops.h>
>   
> +static inline int folio_flags_is_file_lru(const unsigned long *flags)
> +{
> +	return !test_bit(PG_swapbacked, flags);
> +}
> +
>   /**
>    * folio_is_file_lru - Should the folio be on a file LRU or anon LRU?
>    * @folio: The folio to test.
> @@ -27,7 +32,7 @@
>    */
>   static inline int folio_is_file_lru(const struct folio *folio)
>   {
> -	return !folio_test_swapbacked(folio);
> +	return folio_flags_is_file_lru(const_folio_flags(folio, 0));
>   }
>   
>   static __always_inline void __update_lru_size(struct lruvec *lruvec,
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index a613bb8d7271..7169cac60869 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3269,7 +3269,8 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv)
>    ******************************************************************************/
>   
>   /* promote pages accessed through page tables */
> -static int folio_update_gen(struct folio *folio, int new_gen, const vma_flags_t *vma_flags)
> +static int folio_update_gen(struct folio *folio, int new_gen, int *is_file,
> +			    const vma_flags_t *vma_flags)
>   {
>   	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
>   	int old_gen;
> @@ -3298,6 +3299,7 @@ static int folio_update_gen(struct folio *folio, int new_gen, const vma_flags_t
>   		new_flags |= BIT(PG_workingset);
>   	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>   
> +	*is_file = folio_flags_is_file_lru(&old_flags);
>   	return old_gen;
>   }
>   
> @@ -3328,9 +3330,8 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
>   }
>   
>   static void update_batch_size(struct lru_gen_mm_walk *walk, struct folio *folio,
> -			      int old_gen, int new_gen)
> +			      int old_gen, int new_gen, int type)
>   {
> -	int type = folio_is_file_lru(folio);
>   	int zone = folio_zonenum(folio);
>   	int delta = folio_nr_pages(folio);
>   
> @@ -3519,7 +3520,7 @@ static bool suitable_to_scan(int total, int young)
>   static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struct *vma,
>   			      struct lruvec *lruvec, struct folio *folio, bool dirty)
>   {
> -	int new_gen, old_gen;
> +	int new_gen, old_gen, file;
>   
>   	if (!folio)
>   		return;
> @@ -3532,9 +3533,9 @@ static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struc
>   		folio_mark_dirty(folio);
>   
>   	if (walk) {
> -		old_gen = folio_update_gen(folio, new_gen, &vma->flags);
> +		old_gen = folio_update_gen(folio, new_gen, &file, &vma->flags);
>   		if (old_gen >= 0 && old_gen != new_gen)
> -			update_batch_size(walk, folio, old_gen, new_gen);
> +			update_batch_size(walk, folio, old_gen, new_gen, file);
>   	} else if (lru_gen_set_refs(folio, &vma->flags)) {
>   		old_gen = folio_lru_gen(folio);
>   		if (old_gen >= 0 && old_gen != new_gen)
> 

Re: [PATCH 2/7] mm/mglru: introduce helpers for manipulating gen and refs flags

[Ridong Chen]

On 8/20/2026 9:43 AM, Ridong Chen wrote:
> 
> 
> On 8/18/2026 1:38 PM, Kairui Song via B4 Relay wrote:
>> From: Kairui Song <kasong@tencent.com>
>>
>> Instead of doing bit ops on folio->flags.f, introduce helpers for
>> adjusting folio's refs and gen info, make the code easier to debug and
>> understand.
>>
>> No functional change is intended: some combined atomic operations are
>> split into two, which only creates harmless transient states.
>>
>> Signed-off-by: Kairui Song <kasong@tencent.com>
>> ---
>>   include/linux/mm_inline.h | 79 +++++++++++++++++++++++++++++++++++++++++------
>>   include/linux/mmzone.h    |  2 ++
>>   mm/folio.c                | 19 +++++++-----
>>   mm/vmscan.c               | 61 ++++++++++++++++++++----------------
>>   4 files changed, 117 insertions(+), 44 deletions(-)
>>
>> diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
>> index 621c8653d8f7..93bf3fa221f8 100644
>> --- a/include/linux/mm_inline.h
>> +++ b/include/linux/mm_inline.h
>> @@ -142,10 +142,43 @@ static inline int lru_tier_from_refs(int refs, bool 
>> workingset)
>>       return workingset ? MAX_NR_TIERS - 1 : order_base_2(refs);
>>   }
>> -static inline int folio_lru_refs(const struct folio *folio)
>> +/**
>> + * lru_gen_from_flags - Return the LRU generation number from folio flags.
>> + * @flags: folio flags
>> + *
>> + * Returns: A number between 0 and LRU_GEN_MAX, inclusive. Returns -1 if the
>> + * flags indicate the folio is off the list (e.g., isolated).
>> + */
>> +static inline int lru_gen_from_flags(unsigned long flags)
>> +{
>> +    int gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF);
>> +
>> +    BUILD_BUG_ON(LRU_GEN_MASK & LRU_REFS_MASK);
>> +    gen -= 1;
>> +    VM_WARN_ON_ONCE(gen != -1 && gen > LRU_GEN_MAX);
> 
> Perhaps we could define a macro such as GEN_OFF = -1 to make the code more self- 
> explanatory, I found this warning a bit confusing at first glance.
> 
> LRU_GEN_MAX already bears some resemblance to MAX_NR_GENS, so introducing yet 
> another macro may add some clutter.
> 

Ah, I just noticed Baolin already mentioned that, sorry for the noise.

> Just my two cents.
> 
>> +    return gen;
>> +}
>> +
>> +/**
>> + * lru_gen_set_flags - Set the LRU generation number to specified folio flags.
>> + * @flags: pointer to the folio flags
>> + * @gen: generation number, between 0 and LRU_GEN_MAX, inclusive.
>> + */
>> +static inline void lru_gen_set_flags(unsigned long *flags, int gen)
>>   {
>> -    unsigned long flags = READ_ONCE(folio->flags.f);
>> +    VM_WARN_ON_ONCE(gen > LRU_GEN_MAX || gen < 0);
>> +    BUILD_BUG_ON((LRU_GEN_MAX + 1) != MAX_NR_GENS);
>> +
>> +    *flags &= ~LRU_GEN_MASK;
>> +    *flags |= (gen + 1UL) << LRU_GEN_PGOFF;
>> +}
>> +/**
>> + * lru_refs_from_flags - Return LRU referenced / access count from folio flags.
>> + * @flags: folio flags
>> + */
>> +static inline int lru_refs_from_flags(unsigned long flags)
>> +{
>>       if (!(flags & BIT(PG_referenced)))
>>           return 0;
>>       /*
>> @@ -155,18 +188,47 @@ static inline int folio_lru_refs(const struct folio *folio)
>>       return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + 1;
>>   }
>> -static inline int folio_lru_gen(const struct folio *folio)
>> +/**
>> + * lru_refs_set_flags - Set the LRU referenced / access count to specified 
>> folio flags.
>> + * @flags: pointer to the folio flags
>> + * @refs: referenced / access count number, between 0 and LRU_REFS_MAX, 
>> inclusive.
>> + */
>> +static inline void lru_refs_set_flags(unsigned long *flags, unsigned int refs)
>> +{
>> +    VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
>> +    BUILD_BUG_ON(LRU_REFS_MAX != (LRU_REFS_MASK >> LRU_REFS_PGOFF) + 1);
>> +
>> +    *flags &= ~LRU_REFS_FLAGS;
>> +    if (!refs)
>> +        return;
>> +    *flags |= (BIT(PG_referenced) | ((refs - 1UL) << LRU_REFS_PGOFF));
>> +}
>> +
>> +static inline int folio_lru_refs(const struct folio *folio)
>>   {
>> -    unsigned long flags = READ_ONCE(folio->flags.f);
>> +    return lru_refs_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
>> +}
>> +
>> +static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
>> +{
>> +    unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
>> +
>> +    do {
>> +        new_flags = old_flags;
>> +        lru_refs_set_flags(&new_flags, refs);
>> +    } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>> +}
>> -    return ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
>> +static inline int folio_lru_gen(const struct folio *folio)
>> +{
>> +    return lru_gen_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
>>   }
>>   static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen)
>>   {
>>       unsigned long max_seq = lruvec->lrugen.max_seq;
>> -    VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
>> +    VM_WARN_ON_ONCE(gen > LRU_GEN_MAX);
>>       /* see the comment on MIN_NR_GENS */
>>       return gen == lru_gen_from_seq(max_seq) || gen == 
>> lru_gen_from_seq(max_seq - 1);
>> @@ -270,7 +332,7 @@ static inline bool lru_gen_add_folio(struct lruvec 
>> *lruvec, struct folio *folio,
>>       gen = lru_gen_from_seq(seq);
>>       flags = (gen + 1UL) << LRU_GEN_PGOFF;
>>       /* see the comment on MIN_NR_GENS about PG_active */
>> -    set_mask_bits(&folio->flags.f, LRU_GEN_MASK | BIT(PG_active), flags);
>> +    set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
>>       lru_gen_update_size(lruvec, folio, -1, gen);
>>       /* for folio_rotate_reclaimable() */
>> @@ -295,7 +357,7 @@ static inline bool lru_gen_del_folio(struct lruvec 
>> *lruvec, struct folio *folio,
>>       /* for folio_migrate_flags() */
>>       flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
>> -    flags = set_mask_bits(&folio->flags.f, LRU_GEN_MASK, flags);
>> +    flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, flags);
>>       gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
>>       lru_gen_update_size(lruvec, folio, gen, -1);
>> @@ -339,7 +401,6 @@ static inline bool lru_gen_del_folio(struct lruvec 
>> *lruvec, struct folio *folio,
>>   static inline void folio_migrate_refs(struct folio *new, const struct folio 
>> *old)
>>   {
>> -
>>   }
>>   #endif /* CONFIG_LRU_GEN */
>> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
>> index 94f9c3ff5416..32d9354a754f 100644
>> --- a/include/linux/mmzone.h
>> +++ b/include/linux/mmzone.h
>> @@ -496,7 +496,9 @@ enum lruvec_flags {
>>   #ifndef __GENERATING_BOUNDS_H
>>   #define LRU_GEN_MASK        ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)
>> +#define LRU_GEN_MAX        (BIT(LRU_GEN_WIDTH - 1) - 1)
>>   #define LRU_REFS_MASK        ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF)
>> +#define LRU_REFS_MAX        BIT(LRU_REFS_WIDTH)
>>   /*
>>    * For folios accessed multiple times through file descriptors,
>> diff --git a/mm/folio.c b/mm/folio.c
>> index 59c477120b9a..0adfe4f5ef72 100644
>> --- a/mm/folio.c
>> +++ b/mm/folio.c
>> @@ -353,26 +353,28 @@ static void __lru_cache_activate_folio(struct folio *folio)
>>   static void lru_gen_inc_refs(struct folio *folio)
>>   {
>> -    unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
>> +    unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
>> +    int refs;
>>       if (folio_test_unevictable(folio))
>>           return;
>>       /* see the comment on LRU_REFS_FLAGS */
>> -    if (!folio_test_referenced(folio)) {
>> -        set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
>> +    if (!folio_lru_refs(folio)) {
>> +        folio_set_lru_refs(folio, 1);
>>           return;
>>       }
>>       do {
>> -        if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
>> +        new_flags = old_flags;
>> +        refs = lru_refs_from_flags(old_flags);
>> +        if (refs == LRU_REFS_MAX) {
>>               if (!folio_test_workingset(folio))
>>                   folio_set_workingset(folio);
>>               return;
>>           }
>> -
>> -        new_flags = old_flags + BIT(LRU_REFS_PGOFF);
>> -    } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
>> +        lru_refs_set_flags(&new_flags, refs + 1);
>> +    } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>>   }
>>   static bool lru_gen_clear_refs(struct folio *folio)
>> @@ -384,7 +386,8 @@ static bool lru_gen_clear_refs(struct folio *folio)
>>       if (gen < 0)
>>           return true;
>> -    set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS | BIT(PG_workingset), 0);
>> +    folio_set_lru_refs(folio, 0);
>> +    folio_clear_workingset(folio);
>>       rcu_read_lock();
>>       seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
>> diff --git a/mm/vmscan.c b/mm/vmscan.c
>> index c1404a59523d..080132997d87 100644
>> --- a/mm/vmscan.c
>> +++ b/mm/vmscan.c
>> @@ -843,19 +843,22 @@ static bool lru_gen_set_refs(struct folio *folio, const 
>> vma_flags_t *vma_flags)
>>       if (!folio_test_referenced(folio) && !folio_test_workingset(folio)) {
>>           /* Activate file-backed executable folios after first usage. */
>>           if (is_exec_file_folio(folio, vma_flags)) {
>> -            set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
>> +            folio_set_lru_refs(folio, 0);
>> +            folio_set_workingset(folio);
>>               return true;
>>           }
>> -        set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
>> +        folio_set_lru_refs(folio, 1);
>>           return false;
>>       }
>>       /* Promote on second access */
>> -    if (folio_lru_refs(folio) > 1)
>> -        set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
>> -    else
>> +    if (folio_lru_refs(folio) > 1) {
>> +        folio_set_lru_refs(folio, 0);
>> +        folio_set_workingset(folio);
>> +    } else {
>>           folio_mark_accessed(folio);
>> +    }
>>       return true;
>>   }
>>   #else
>> @@ -3266,11 +3269,10 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, 
>> struct ctrl_pos *pv)
>>    
>> ******************************************************************************/
>>   /* promote pages accessed through page tables */
>> -static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t 
>> *vma_flags)
>> +static int folio_update_gen(struct folio *folio, int new_gen, const 
>> vma_flags_t *vma_flags)
>>   {
>> -    unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
>> -
>> -    VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
>> +    unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
>> +    int old_gen;
>>       /*
>>        * See the comment on LRU_REFS_FLAGS, and activate file-backed
>> @@ -3279,20 +3281,24 @@ static int folio_update_gen(struct folio *folio, int 
>> gen, const vma_flags_t *vma
>>        */
>>       if (!folio_test_referenced(folio) && !folio_test_workingset(folio) &&
>>           !is_exec_file_folio(folio, vma_flags)) {
>> -        set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
>> +        folio_set_lru_refs(folio, 1);
>>           return -1;
>>       }
>>       do {
>> +        old_gen = lru_gen_from_flags(old_flags);
>> +        new_flags = old_flags;
>> +
>>           /* lru_gen_del_folio() has isolated this page? */
>> -        if (!(old_flags & LRU_GEN_MASK))
>> -            return -1;
>> +        if (old_gen < 0)
>> +            break;
>> -        new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);
>> -        new_flags |= ((gen + 1UL) << LRU_GEN_PGOFF) | BIT(PG_workingset);
>> -    } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
>> +        lru_gen_set_flags(&new_flags, new_gen);
>> +        lru_refs_set_flags(&new_flags, 0);
>> +        new_flags |= BIT(PG_workingset);
>> +    } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>> -    return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
>> +    return old_gen;
>>   }
>>   /* protect pages accessed multiple times through file descriptors */
>> @@ -3301,21 +3307,20 @@ static int folio_inc_gen(struct lruvec *lruvec, struct 
>> folio *folio)
>>       int type = folio_is_file_lru(folio);
>>       struct lru_gen_folio *lrugen = &lruvec->lrugen;
>>       int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
>> -    unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
>> -
>> -    VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio);
>> +    unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
>>       do {
>> -        new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
>> +        new_gen = lru_gen_from_flags(old_flags);
>> +
>>           /* folio_update_gen() has promoted this page? */
>>           if (new_gen >= 0 && new_gen != old_gen)
>>               return new_gen;
>> +        new_flags = old_flags;
>>           new_gen = (old_gen + 1) % MAX_NR_GENS;
>> -
>> -        new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);
>> -        new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF;
>> -    } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
>> +        lru_gen_set_flags(&new_flags, new_gen);
>> +        lru_refs_set_flags(&new_flags, 0);
>> +    } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>>       lru_gen_update_size(lruvec, folio, old_gen, new_gen);
>> @@ -4711,7 +4716,7 @@ static bool isolate_folio(struct lruvec *lruvec, struct 
>> folio *folio, struct sca
>>       /* see the comment on LRU_REFS_FLAGS */
>>       if (!folio_test_referenced(folio))
>> -        set_mask_bits(&folio->flags.f, LRU_REFS_MASK, 0);
>> +        folio_set_lru_refs(folio, 0);
>>       success = lru_gen_del_folio(lruvec, folio, true);
>>       VM_WARN_ON_ONCE_FOLIO(!success, folio);
>> @@ -4927,8 +4932,10 @@ static int evict_folios(unsigned long nr_to_scan, 
>> struct lruvec *lruvec,
>>           }
>>           /* don't add rejected folios to the oldest generation */
>> -        if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type])
>> -            set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_active));
>> +        if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type]) {
>> +            folio_set_lru_refs(folio, 0);
>> +            folio_set_active(folio);
>> +        }
>>       }
>>       move_folios_to_lru(&list);
>>
> 

-- 
Best regards
Ridong

Re: [PATCH 2/7] mm/mglru: introduce helpers for manipulating gen and refs flags

[Ridong Chen]

On 8/20/2026 9:43 AM, Ridong Chen wrote:
> 
> 
> On 8/18/2026 1:38 PM, Kairui Song via B4 Relay wrote:
>> From: Kairui Song <kasong@tencent.com>
>>
>> Instead of doing bit ops on folio->flags.f, introduce helpers for
>> adjusting folio's refs and gen info, make the code easier to debug and
>> understand.
>>
>> No functional change is intended: some combined atomic operations are
>> split into two, which only creates harmless transient states.
>>
>> Signed-off-by: Kairui Song <kasong@tencent.com>
>> ---
>>   include/linux/mm_inline.h | 79 +++++++++++++++++++++++++++++++++++++++++------
>>   include/linux/mmzone.h    |  2 ++
>>   mm/folio.c                | 19 +++++++-----
>>   mm/vmscan.c               | 61 ++++++++++++++++++++----------------
>>   4 files changed, 117 insertions(+), 44 deletions(-)
>>
>> diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
>> index 621c8653d8f7..93bf3fa221f8 100644
>> --- a/include/linux/mm_inline.h
>> +++ b/include/linux/mm_inline.h
>> @@ -142,10 +142,43 @@ static inline int lru_tier_from_refs(int refs, bool 
>> workingset)
>>       return workingset ? MAX_NR_TIERS - 1 : order_base_2(refs);
>>   }
>> -static inline int folio_lru_refs(const struct folio *folio)
>> +/**
>> + * lru_gen_from_flags - Return the LRU generation number from folio flags.
>> + * @flags: folio flags
>> + *
>> + * Returns: A number between 0 and LRU_GEN_MAX, inclusive. Returns -1 if the
>> + * flags indicate the folio is off the list (e.g., isolated).
>> + */
>> +static inline int lru_gen_from_flags(unsigned long flags)
>> +{
>> +    int gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF);
>> +
>> +    BUILD_BUG_ON(LRU_GEN_MASK & LRU_REFS_MASK);
>> +    gen -= 1;
>> +    VM_WARN_ON_ONCE(gen != -1 && gen > LRU_GEN_MAX);
> 
> Perhaps we could define a macro such as GEN_OFF = -1 to make the code more self- 
> explanatory, I found this warning a bit confusing at first glance.
> 
> LRU_GEN_MAX already bears some resemblance to MAX_NR_GENS, so introducing yet 
> another macro may add some clutter.
> 

Ah, I just noticed Baolin already mentioned that, sorry for the noise.

> Just my two cents.
> 
>> +    return gen;
>> +}
>> +
>> +/**
>> + * lru_gen_set_flags - Set the LRU generation number to specified folio flags.
>> + * @flags: pointer to the folio flags
>> + * @gen: generation number, between 0 and LRU_GEN_MAX, inclusive.
>> + */
>> +static inline void lru_gen_set_flags(unsigned long *flags, int gen)
>>   {
>> -    unsigned long flags = READ_ONCE(folio->flags.f);
>> +    VM_WARN_ON_ONCE(gen > LRU_GEN_MAX || gen < 0);
>> +    BUILD_BUG_ON((LRU_GEN_MAX + 1) != MAX_NR_GENS);
>> +
>> +    *flags &= ~LRU_GEN_MASK;
>> +    *flags |= (gen + 1UL) << LRU_GEN_PGOFF;
>> +}
>> +/**
>> + * lru_refs_from_flags - Return LRU referenced / access count from folio flags.
>> + * @flags: folio flags
>> + */
>> +static inline int lru_refs_from_flags(unsigned long flags)
>> +{
>>       if (!(flags & BIT(PG_referenced)))
>>           return 0;
>>       /*
>> @@ -155,18 +188,47 @@ static inline int folio_lru_refs(const struct folio *folio)
>>       return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + 1;
>>   }
>> -static inline int folio_lru_gen(const struct folio *folio)
>> +/**
>> + * lru_refs_set_flags - Set the LRU referenced / access count to specified 
>> folio flags.
>> + * @flags: pointer to the folio flags
>> + * @refs: referenced / access count number, between 0 and LRU_REFS_MAX, 
>> inclusive.
>> + */
>> +static inline void lru_refs_set_flags(unsigned long *flags, unsigned int refs)
>> +{
>> +    VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
>> +    BUILD_BUG_ON(LRU_REFS_MAX != (LRU_REFS_MASK >> LRU_REFS_PGOFF) + 1);
>> +
>> +    *flags &= ~LRU_REFS_FLAGS;
>> +    if (!refs)
>> +        return;
>> +    *flags |= (BIT(PG_referenced) | ((refs - 1UL) << LRU_REFS_PGOFF));
>> +}
>> +
>> +static inline int folio_lru_refs(const struct folio *folio)
>>   {
>> -    unsigned long flags = READ_ONCE(folio->flags.f);
>> +    return lru_refs_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
>> +}
>> +
>> +static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
>> +{
>> +    unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
>> +
>> +    do {
>> +        new_flags = old_flags;
>> +        lru_refs_set_flags(&new_flags, refs);
>> +    } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>> +}
>> -    return ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
>> +static inline int folio_lru_gen(const struct folio *folio)
>> +{
>> +    return lru_gen_from_flags(READ_ONCE(*const_folio_flags(folio, 0)));
>>   }
>>   static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen)
>>   {
>>       unsigned long max_seq = lruvec->lrugen.max_seq;
>> -    VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
>> +    VM_WARN_ON_ONCE(gen > LRU_GEN_MAX);
>>       /* see the comment on MIN_NR_GENS */
>>       return gen == lru_gen_from_seq(max_seq) || gen == 
>> lru_gen_from_seq(max_seq - 1);
>> @@ -270,7 +332,7 @@ static inline bool lru_gen_add_folio(struct lruvec 
>> *lruvec, struct folio *folio,
>>       gen = lru_gen_from_seq(seq);
>>       flags = (gen + 1UL) << LRU_GEN_PGOFF;
>>       /* see the comment on MIN_NR_GENS about PG_active */
>> -    set_mask_bits(&folio->flags.f, LRU_GEN_MASK | BIT(PG_active), flags);
>> +    set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
>>       lru_gen_update_size(lruvec, folio, -1, gen);
>>       /* for folio_rotate_reclaimable() */
>> @@ -295,7 +357,7 @@ static inline bool lru_gen_del_folio(struct lruvec 
>> *lruvec, struct folio *folio,
>>       /* for folio_migrate_flags() */
>>       flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
>> -    flags = set_mask_bits(&folio->flags.f, LRU_GEN_MASK, flags);
>> +    flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, flags);
>>       gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
>>       lru_gen_update_size(lruvec, folio, gen, -1);
>> @@ -339,7 +401,6 @@ static inline bool lru_gen_del_folio(struct lruvec 
>> *lruvec, struct folio *folio,
>>   static inline void folio_migrate_refs(struct folio *new, const struct folio 
>> *old)
>>   {
>> -
>>   }
>>   #endif /* CONFIG_LRU_GEN */
>> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
>> index 94f9c3ff5416..32d9354a754f 100644
>> --- a/include/linux/mmzone.h
>> +++ b/include/linux/mmzone.h
>> @@ -496,7 +496,9 @@ enum lruvec_flags {
>>   #ifndef __GENERATING_BOUNDS_H
>>   #define LRU_GEN_MASK        ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)
>> +#define LRU_GEN_MAX        (BIT(LRU_GEN_WIDTH - 1) - 1)
>>   #define LRU_REFS_MASK        ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF)
>> +#define LRU_REFS_MAX        BIT(LRU_REFS_WIDTH)
>>   /*
>>    * For folios accessed multiple times through file descriptors,
>> diff --git a/mm/folio.c b/mm/folio.c
>> index 59c477120b9a..0adfe4f5ef72 100644
>> --- a/mm/folio.c
>> +++ b/mm/folio.c
>> @@ -353,26 +353,28 @@ static void __lru_cache_activate_folio(struct folio *folio)
>>   static void lru_gen_inc_refs(struct folio *folio)
>>   {
>> -    unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
>> +    unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
>> +    int refs;
>>       if (folio_test_unevictable(folio))
>>           return;
>>       /* see the comment on LRU_REFS_FLAGS */
>> -    if (!folio_test_referenced(folio)) {
>> -        set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
>> +    if (!folio_lru_refs(folio)) {
>> +        folio_set_lru_refs(folio, 1);
>>           return;
>>       }
>>       do {
>> -        if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
>> +        new_flags = old_flags;
>> +        refs = lru_refs_from_flags(old_flags);
>> +        if (refs == LRU_REFS_MAX) {
>>               if (!folio_test_workingset(folio))
>>                   folio_set_workingset(folio);
>>               return;
>>           }
>> -
>> -        new_flags = old_flags + BIT(LRU_REFS_PGOFF);
>> -    } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
>> +        lru_refs_set_flags(&new_flags, refs + 1);
>> +    } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>>   }
>>   static bool lru_gen_clear_refs(struct folio *folio)
>> @@ -384,7 +386,8 @@ static bool lru_gen_clear_refs(struct folio *folio)
>>       if (gen < 0)
>>           return true;
>> -    set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS | BIT(PG_workingset), 0);
>> +    folio_set_lru_refs(folio, 0);
>> +    folio_clear_workingset(folio);
>>       rcu_read_lock();
>>       seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
>> diff --git a/mm/vmscan.c b/mm/vmscan.c
>> index c1404a59523d..080132997d87 100644
>> --- a/mm/vmscan.c
>> +++ b/mm/vmscan.c
>> @@ -843,19 +843,22 @@ static bool lru_gen_set_refs(struct folio *folio, const 
>> vma_flags_t *vma_flags)
>>       if (!folio_test_referenced(folio) && !folio_test_workingset(folio)) {
>>           /* Activate file-backed executable folios after first usage. */
>>           if (is_exec_file_folio(folio, vma_flags)) {
>> -            set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
>> +            folio_set_lru_refs(folio, 0);
>> +            folio_set_workingset(folio);
>>               return true;
>>           }
>> -        set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
>> +        folio_set_lru_refs(folio, 1);
>>           return false;
>>       }
>>       /* Promote on second access */
>> -    if (folio_lru_refs(folio) > 1)
>> -        set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
>> -    else
>> +    if (folio_lru_refs(folio) > 1) {
>> +        folio_set_lru_refs(folio, 0);
>> +        folio_set_workingset(folio);
>> +    } else {
>>           folio_mark_accessed(folio);
>> +    }
>>       return true;
>>   }
>>   #else
>> @@ -3266,11 +3269,10 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, 
>> struct ctrl_pos *pv)
>>    
>> ******************************************************************************/
>>   /* promote pages accessed through page tables */
>> -static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t 
>> *vma_flags)
>> +static int folio_update_gen(struct folio *folio, int new_gen, const 
>> vma_flags_t *vma_flags)
>>   {
>> -    unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
>> -
>> -    VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
>> +    unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
>> +    int old_gen;
>>       /*
>>        * See the comment on LRU_REFS_FLAGS, and activate file-backed
>> @@ -3279,20 +3281,24 @@ static int folio_update_gen(struct folio *folio, int 
>> gen, const vma_flags_t *vma
>>        */
>>       if (!folio_test_referenced(folio) && !folio_test_workingset(folio) &&
>>           !is_exec_file_folio(folio, vma_flags)) {
>> -        set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
>> +        folio_set_lru_refs(folio, 1);
>>           return -1;
>>       }
>>       do {
>> +        old_gen = lru_gen_from_flags(old_flags);
>> +        new_flags = old_flags;
>> +
>>           /* lru_gen_del_folio() has isolated this page? */
>> -        if (!(old_flags & LRU_GEN_MASK))
>> -            return -1;
>> +        if (old_gen < 0)
>> +            break;
>> -        new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);
>> -        new_flags |= ((gen + 1UL) << LRU_GEN_PGOFF) | BIT(PG_workingset);
>> -    } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
>> +        lru_gen_set_flags(&new_flags, new_gen);
>> +        lru_refs_set_flags(&new_flags, 0);
>> +        new_flags |= BIT(PG_workingset);
>> +    } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>> -    return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
>> +    return old_gen;
>>   }
>>   /* protect pages accessed multiple times through file descriptors */
>> @@ -3301,21 +3307,20 @@ static int folio_inc_gen(struct lruvec *lruvec, struct 
>> folio *folio)
>>       int type = folio_is_file_lru(folio);
>>       struct lru_gen_folio *lrugen = &lruvec->lrugen;
>>       int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
>> -    unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
>> -
>> -    VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio);
>> +    unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
>>       do {
>> -        new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
>> +        new_gen = lru_gen_from_flags(old_flags);
>> +
>>           /* folio_update_gen() has promoted this page? */
>>           if (new_gen >= 0 && new_gen != old_gen)
>>               return new_gen;
>> +        new_flags = old_flags;
>>           new_gen = (old_gen + 1) % MAX_NR_GENS;
>> -
>> -        new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);
>> -        new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF;
>> -    } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
>> +        lru_gen_set_flags(&new_flags, new_gen);
>> +        lru_refs_set_flags(&new_flags, 0);
>> +    } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>>       lru_gen_update_size(lruvec, folio, old_gen, new_gen);
>> @@ -4711,7 +4716,7 @@ static bool isolate_folio(struct lruvec *lruvec, struct 
>> folio *folio, struct sca
>>       /* see the comment on LRU_REFS_FLAGS */
>>       if (!folio_test_referenced(folio))
>> -        set_mask_bits(&folio->flags.f, LRU_REFS_MASK, 0);
>> +        folio_set_lru_refs(folio, 0);
>>       success = lru_gen_del_folio(lruvec, folio, true);
>>       VM_WARN_ON_ONCE_FOLIO(!success, folio);
>> @@ -4927,8 +4932,10 @@ static int evict_folios(unsigned long nr_to_scan, 
>> struct lruvec *lruvec,
>>           }
>>           /* don't add rejected folios to the oldest generation */
>> -        if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type])
>> -            set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_active));
>> +        if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type]) {
>> +            folio_set_lru_refs(folio, 0);
>> +            folio_set_active(folio);
>> +        }
>>       }
>>       move_folios_to_lru(&list);
>>
> 

-- 
Best regards
Ridong

Re: [PATCH 7/7] mm/mglru: improve code readability and harden folio_inc_gen

[Kairui Song]

On Thu, Aug 20, 2026 at 9:02 AM Baolin Wang
<baolin.wang@linux.alibaba.com> wrote:
> On 8/20/26 8:57 AM, Baoquan He wrote:
> > On 08/20/26 at 08:53am, Baoquan He wrote:
> >> On 08/18/26 at 01:38pm, Kairui Song via B4 Relay wrote:
> >>> From: Kairui Song <kasong@tencent.com>
> >>>
> >>> The helper should never be called for an off-list folio, and it always
> >>> expects the folio to be in the oldest generation before doing any
> >>> cmpxchg. Add a sanity check for the off-list case: if it is ever
> >>> violated, bail out and keep the folio flags untouched to minimize the
> >>> damage, instead of silently treating the folio as if it were in the
> >>> oldest generation and promoting it updating the flags to an unexpected
> >>> status.
> >>>
> >>> Also rename the variables to clearly distinguish the folio's current
> >>> gen from the oldest gen.
> >>>
> >>> Signed-off-by: Kairui Song <kasong@tencent.com>
> >>> ---
> >>>   mm/vmscan.c | 14 +++++++++-----
> >>>   1 file changed, 9 insertions(+), 5 deletions(-)
> >>>
> >>> diff --git a/mm/vmscan.c b/mm/vmscan.c
> >>> index 7169cac60869..7e3ae0c6cba3 100644
> >>> --- a/mm/vmscan.c
> >>> +++ b/mm/vmscan.c
> >>> @@ -3308,18 +3308,22 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
> >>>   {
> >>>     int type = folio_is_file_lru(folio);
> >>>     struct lru_gen_folio *lrugen = &lruvec->lrugen;
> >>> -   int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
> >>> +   int new_gen, old_gen, min_gen = lru_gen_from_seq(lrugen->min_seq[type]);
> >>>     unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
> >>>
> >>>     do {
> >>> -           new_gen = lru_gen_from_flags(old_flags);
> >>> +           old_gen = lru_gen_from_flags(old_flags);
> >>> +           /* This helper should never be called for off-list folios */
> >>> +           VM_WARN_ON_ONCE(old_gen < 0);
> >>> +           if (old_gen < 0)
> >>> +                   return min_gen;
> >>
> >> As Barry doubted, I think this change is wrong. old_gen < 0 in folio_inc_gen()
> >> could only happen inc_min_seq() call it. While inc_min_seq() call it
> >> because inc_max_seq() need increase max_gen to max_gen + 1 and found
> >> get_nr_gens(lruvec, type) == MAX_NR_GENS, it has to move the oldest gen to
> >                                              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >> 2nd old oldest gen. Here returning min_gen for old_gen < 0 means it will
> >    ~~~~~~~~~~~~~~~~
> > Here, I mean it has to move folios from the oldest gen (min_gen) to the 2nd
> > oldest gen (min_gen + 1). The empty min_gen will become the new max_gen.
> >
> >> be put in the lastest max_gen. It may not be expected.
>
> But how does old_gen < 0 actually happen? folio_inc_gen() is called
> under the lru lock, so how can a folio listed in MGLRU have a gen
> counter < 0? If this can happen in any case, we should fix this bug first.

Hi All,

Actually that's the confusing part, the "new_gen" variable here is
actually the old gen (the gen number before the CAS here) of the
folio, and the "old_gen" here is actually the min_seq gen of the
lruvec, and that's why I'm renaming it.

The current new_gen (which is actually the folio's current gen, the
old gen number) or old_gen (the lruvec's oldest gen) should both never
be < 0 in any case, if it happens, folio_inc_gen will corrupt the gen
counters or page flags.

This change isn't fixing anything, this commit just make old_gen to
hold the folio's current gen number (before the CAS), and if that is <
0 (the folio is off-list), don't touch the folio at all which will
further corrupt a already buggy satuation (this should never happen,
so I added a VM_WARN_ON). And now min_gen will be meaning the lruvec's
oldest gen.

There is no functional change, except one more sanity check and defensive check.

Re: [PATCH 4/7] mm/mglru: move max_seq read into walk_update_folio

[Ridong Chen]

On 8/18/2026 1:38 PM, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
> 
> walk_pte_range(), walk_pmd_range_locked(), and lru_gen_look_around()
> each read lrugen->max_seq to compute the target generation used by
> walk_update_folio(), then pass it as a parameter. Move the read into
> walk_update_folio() itself so the callers no longer need to compute
> or pass the value.
> 
> The max_seq read now happens once per folio update rather than once
> per walk range, so folios always get promoted to the current youngest
> generation.
> 
> Signed-off-by: Kairui Song <kasong@tencent.com>
> Reviewed-by: Baoquan He <baoquan.he@linux.dev>
> ---
>   mm/vmscan.c | 29 ++++++++++++-----------------
>   1 file changed, 12 insertions(+), 17 deletions(-)
> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 080132997d87..a819be6b7ae9 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3517,13 +3517,15 @@ static bool suitable_to_scan(int total, int young)
>   }
>   
>   static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struct *vma,
> -		struct folio *folio, int new_gen, bool dirty)
> +			      struct lruvec *lruvec, struct folio *folio, bool dirty)
>   {
> -	int old_gen;
> +	int new_gen, old_gen;
>   
>   	if (!folio)
>   		return;
>   
> +	new_gen = lru_gen_from_seq(READ_ONCE(lruvec->lrugen.max_seq));
> +
>   	if (dirty && !folio_test_dirty(folio) &&
>   	    !(folio_test_anon(folio) && folio_test_swapbacked(folio) &&
>   	      !folio_test_swapcache(folio)))
> @@ -3554,8 +3556,6 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
>   	struct lru_gen_mm_walk *walk = args->private;
>   	struct mem_cgroup *memcg = lruvec_memcg(walk->lruvec);
>   	struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec);
> -	DEFINE_MAX_SEQ(walk->lruvec);
> -	int gen = lru_gen_from_seq(max_seq);
>   	unsigned int nr;
>   	pmd_t pmdval;
>   
> @@ -3606,7 +3606,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
>   			continue;
>   
>   		if (last != folio) {
> -			walk_update_folio(walk, args->vma, last, gen, dirty);
> +			walk_update_folio(walk, args->vma, walk->lruvec, last, dirty);
>   
>   			last = folio;
>   			dirty = false;
> @@ -3619,7 +3619,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
>   		walk->mm_stats[MM_LEAF_YOUNG] += nr;
>   	}
>   
> -	walk_update_folio(walk, args->vma, last, gen, dirty);
> +	walk_update_folio(walk, args->vma, walk->lruvec, last, dirty);
>   	last = NULL;
>   
>   	if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &end))
> @@ -3642,8 +3642,6 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
>   	struct lru_gen_mm_walk *walk = args->private;
>   	struct mem_cgroup *memcg = lruvec_memcg(walk->lruvec);
>   	struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec);
> -	DEFINE_MAX_SEQ(walk->lruvec);
> -	int gen = lru_gen_from_seq(max_seq);
>   
>   	VM_WARN_ON_ONCE(pud_leaf(*pud));
>   
> @@ -3697,7 +3695,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
>   			goto next;
>   
>   		if (last != folio) {
> -			walk_update_folio(walk, vma, last, gen, dirty);
> +			walk_update_folio(walk, vma, walk->lruvec, last, dirty);
>   
>   			last = folio;
>   			dirty = false;
> @@ -3711,7 +3709,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
>   		i = i > MIN_LRU_BATCH ? 0 : find_next_bit(bitmap, MIN_LRU_BATCH, i) + 1;
>   	} while (i <= MIN_LRU_BATCH);
>   
> -	walk_update_folio(walk, vma, last, gen, dirty);
> +	walk_update_folio(walk, vma, walk->lruvec, last, dirty);
>   
>   	lazy_mmu_mode_disable();
>   	spin_unlock(ptl);
> @@ -4275,8 +4273,6 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
>   	struct pglist_data *pgdat = folio_pgdat(folio);
>   	struct lruvec *lruvec;
>   	struct lru_gen_mm_state *mm_state;
> -	unsigned long max_seq;
> -	int gen;
>   
>   	lockdep_assert_held(pvmw->ptl);
>   	VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio);
> @@ -4313,8 +4309,6 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
>   
>   	memcg = get_mem_cgroup_from_folio(folio);
>   	lruvec = mem_cgroup_lruvec(memcg, pgdat);
> -	max_seq = READ_ONCE((lruvec)->lrugen.max_seq);
> -	gen = lru_gen_from_seq(max_seq);
>   	mm_state = get_mm_state(lruvec);
>   
>   	lazy_mmu_mode_enable();
> @@ -4346,7 +4340,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
>   			continue;
>   
>   		if (last != folio) {
> -			walk_update_folio(walk, vma, last, gen, dirty);
> +			walk_update_folio(walk, vma, lruvec, last, dirty);
>   
>   			last = folio;
>   			dirty = false;
> @@ -4358,13 +4352,14 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
>   		young += nr;
>   	}
>   
> -	walk_update_folio(walk, vma, last, gen, dirty);
> +	walk_update_folio(walk, vma, lruvec, last, dirty);
>   
>   	lazy_mmu_mode_disable();
>   
>   	/* feedback from rmap walkers to page table walkers */
>   	if (mm_state && suitable_to_scan(i, young))
> -		update_bloom_filter(mm_state, max_seq, pvmw->pmd);
> +		update_bloom_filter(mm_state, READ_ONCE(lruvec->lrugen.max_seq),
> +				    pvmw->pmd);
>   
>   	mem_cgroup_put(memcg);
>   
> 

LGTM.

Reviewed-by: Ridong Chen <ridong.chen@linux.dev>

-- 
Best regards
Ridong

Re: [PATCH 7/7] mm/mglru: improve code readability and harden folio_inc_gen

[Baoquan He]

On 08/20/26 at 09:02am, Baolin Wang wrote:
> 
> 
> On 8/20/26 8:57 AM, Baoquan He wrote:
> > On 08/20/26 at 08:53am, Baoquan He wrote:
> > > On 08/18/26 at 01:38pm, Kairui Song via B4 Relay wrote:
> > > > From: Kairui Song <kasong@tencent.com>
> > > > 
> > > > The helper should never be called for an off-list folio, and it always
> > > > expects the folio to be in the oldest generation before doing any
> > > > cmpxchg. Add a sanity check for the off-list case: if it is ever
> > > > violated, bail out and keep the folio flags untouched to minimize the
> > > > damage, instead of silently treating the folio as if it were in the
> > > > oldest generation and promoting it updating the flags to an unexpected
> > > > status.
> > > > 
> > > > Also rename the variables to clearly distinguish the folio's current
> > > > gen from the oldest gen.
> > > > 
> > > > Signed-off-by: Kairui Song <kasong@tencent.com>
> > > > ---
> > > >   mm/vmscan.c | 14 +++++++++-----
> > > >   1 file changed, 9 insertions(+), 5 deletions(-)
> > > > 
> > > > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > > > index 7169cac60869..7e3ae0c6cba3 100644
> > > > --- a/mm/vmscan.c
> > > > +++ b/mm/vmscan.c
> > > > @@ -3308,18 +3308,22 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
> > > >   {
> > > >   	int type = folio_is_file_lru(folio);
> > > >   	struct lru_gen_folio *lrugen = &lruvec->lrugen;
> > > > -	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
> > > > +	int new_gen, old_gen, min_gen = lru_gen_from_seq(lrugen->min_seq[type]);
> > > >   	unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
> > > >   	do {
> > > > -		new_gen = lru_gen_from_flags(old_flags);
> > > > +		old_gen = lru_gen_from_flags(old_flags);
> > > > +		/* This helper should never be called for off-list folios */
> > > > +		VM_WARN_ON_ONCE(old_gen < 0);
> > > > +		if (old_gen < 0)
> > > > +			return min_gen;
> > > 
> > > As Barry doubted, I think this change is wrong. old_gen < 0 in folio_inc_gen()
> > > could only happen inc_min_seq() call it. While inc_min_seq() call it
> > > because inc_max_seq() need increase max_gen to max_gen + 1 and found
> > > get_nr_gens(lruvec, type) == MAX_NR_GENS, it has to move the oldest gen to
> >                                              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > 2nd old oldest gen. Here returning min_gen for old_gen < 0 means it will
> >    ~~~~~~~~~~~~~~~~
> > Here, I mean it has to move folios from the oldest gen (min_gen) to the 2nd
> > oldest gen (min_gen + 1). The empty min_gen will become the new max_gen.
> > 
> > > be put in the lastest max_gen. It may not be expected.
> 
> But how does old_gen < 0 actually happen? folio_inc_gen() is called under
> the lru lock, so how can a folio listed in MGLRU have a gen counter < 0? If
> this can happen in any case, we should fix this bug first.

That's a good question, and I agree with you that folio_inc_gen() is
called under lru lock, and lru_gen_del_folio() which clears the gen
and take folio off lru is also called under lru lock. So old_gen < 0
here is a "should never happen" defensive branch (plus the
WARN_ON_ONCE), and if it really occurs there is a real bug to fix.
Wondering how Barry trigger his printk debugging and observed it.

However, the defensive branch itself is incorrect. Not only folio of
old_gen < 0 are put in the new max_gen, but what is worse, it doesn't
clear the old_gen <0 in folio->flags even though it's put back in the
min_gen list, next time aging comes to next round of min_gen and sort_folio()
will get a lrugen->folios[-1] out of bound accessing

static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_control *sc,
                       int tier_idx)
{
	...
        /* promoted */
        if (gen != lru_gen_from_seq(lrugen->min_seq[type])) {
                list_move(&folio->lru, &lrugen->folios[gen][type][zone]);
                return true;
        }
	...
}

So I think we should keep the old code unchanged, fix any warning report 
triggered by VM_WARN_ON_ONCE(old_gen < 0).

Re: [PATCH 5/7] mm/mglru: use explicit tier range in read_ctrl_pos()

[Ridong Chen]

On 8/18/2026 1:38 PM, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
> 
> read_ctrl_pos() encodes the tier range in a single "tier" parameter
> via "tier % MAX_NR_TIERS" as the start and "min(tier, MAX_NR_TIERS-1)"
> as the end. This is hard to follow, maintain, or extend. Tier values
> 0..3 select a single tier, while tier == MAX_NR_TIERS selects the
> full range.
> 
> Replace it with explicit (tier_min, tier_max) parameters using a
> closed [tier_min, tier_max] interval, and add LRU_TIER_MIN and
> LRU_TIER_MAX for the tier bounds. The call sites now become
> self-documenting:
> 
>    - get_tier_idx:   (LRU_TIER_MIN, LRU_TIER_MIN) for the first tier,
>                      (tier, tier) for each subsequent tier
>    - get_type_to_scan: (LRU_TIER_MIN, LRU_TIER_MAX) for the full range
> 
> No functional change.
> 

Just like LRU_GEN_MAX and MAX_NR_GENS issue.
Why do we have to add LRU_TIER_MAX, can we just use [tier_min, MAX_NR_TIERS)?


> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
>   include/linux/mmzone.h |  2 ++
>   mm/vmscan.c            | 18 ++++++++++--------
>   2 files changed, 12 insertions(+), 8 deletions(-)
> 
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index 32d9354a754f..d0b5c6217d25 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -492,6 +492,8 @@ enum lruvec_flags {
>    * folio->flags, masked by LRU_REFS_MASK.
>    */
>   #define MAX_NR_TIERS		4U
> +#define LRU_TIER_MIN		0U
> +#define LRU_TIER_MAX		(MAX_NR_TIERS - 1)
>   
>   #ifndef __GENERATING_BOUNDS_H
>   
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index a819be6b7ae9..a613bb8d7271 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3198,8 +3198,8 @@ struct ctrl_pos {
>   	int gain;
>   };
>   
> -static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain,
> -			  struct ctrl_pos *pos)
> +static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier_min,
> +			  int tier_max, int gain, struct ctrl_pos *pos)
>   {
>   	int i;
>   	struct lru_gen_folio *lrugen = &lruvec->lrugen;
> @@ -3208,7 +3208,7 @@ static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain,
>   	pos->gain = gain;
>   	pos->refaulted = pos->total = 0;
>   
> -	for (i = tier % MAX_NR_TIERS; i <= min(tier, MAX_NR_TIERS - 1); i++) {
> +	for (i = tier_min; i <= tier_max; i++) {
>   		pos->refaulted += lrugen->avg_refaulted[type][i] +
>   				  atomic_long_read(&lrugen->refaulted[hist][type][i]);
>   		pos->total += lrugen->avg_total[type][i] +
> @@ -4804,9 +4804,9 @@ static int get_tier_idx(struct lruvec *lruvec, int type)
>   	 * This value is chosen because any other tier would have at least twice
>   	 * as many refaults as the first tier.
>   	 */
> -	read_ctrl_pos(lruvec, type, 0, 2, &sp);
> -	for (tier = 1; tier < MAX_NR_TIERS; tier++) {
> -		read_ctrl_pos(lruvec, type, tier, 3, &pv);
> +	read_ctrl_pos(lruvec, type, LRU_TIER_MIN, LRU_TIER_MIN, 2, &sp);
> +	for (tier = LRU_TIER_MIN + 1; tier <= LRU_TIER_MAX; tier++) {
> +		read_ctrl_pos(lruvec, type, tier, tier, 3, &pv);
>   		if (!positive_ctrl_err(&sp, &pv))
>   			break;
>   	}
> @@ -4827,8 +4827,10 @@ static int get_type_to_scan(struct lruvec *lruvec, int swappiness)
>   	 * Compare the sum of all tiers of anon with that of file to determine
>   	 * which type to scan.
>   	 */
> -	read_ctrl_pos(lruvec, LRU_GEN_ANON, MAX_NR_TIERS, swappiness, &sp);
> -	read_ctrl_pos(lruvec, LRU_GEN_FILE, MAX_NR_TIERS, MAX_SWAPPINESS - swappiness, &pv);
> +	read_ctrl_pos(lruvec, LRU_GEN_ANON, LRU_TIER_MIN, LRU_TIER_MAX,
> +		      swappiness, &sp);
> +	read_ctrl_pos(lruvec, LRU_GEN_FILE, LRU_TIER_MIN, LRU_TIER_MAX,
> +		      MAX_SWAPPINESS - swappiness, &pv);
>   
>   	return positive_ctrl_err(&sp, &pv);
>   }
> 

-- 
Best regards
Ridong

Re: [PATCH 5/7] mm/mglru: use explicit tier range in read_ctrl_pos()

[Kairui Song]

On Thu, Aug 20, 2026 at 10:34 AM Ridong Chen <ridong.chen@linux.dev> wrote:
> On 8/18/2026 1:38 PM, Kairui Song via B4 Relay wrote:
> > From: Kairui Song <kasong@tencent.com>
> >
> > read_ctrl_pos() encodes the tier range in a single "tier" parameter
> > via "tier % MAX_NR_TIERS" as the start and "min(tier, MAX_NR_TIERS-1)"
> > as the end. This is hard to follow, maintain, or extend. Tier values
> > 0..3 select a single tier, while tier == MAX_NR_TIERS selects the
> > full range.
> >
> > Replace it with explicit (tier_min, tier_max) parameters using a
> > closed [tier_min, tier_max] interval, and add LRU_TIER_MIN and
> > LRU_TIER_MAX for the tier bounds. The call sites now become
> > self-documenting:
> >
> >    - get_tier_idx:   (LRU_TIER_MIN, LRU_TIER_MIN) for the first tier,
> >                      (tier, tier) for each subsequent tier
> >    - get_type_to_scan: (LRU_TIER_MIN, LRU_TIER_MAX) for the full range
> >
> > No functional change.
> >
>
> Just like LRU_GEN_MAX and MAX_NR_GENS issue.
> Why do we have to add LRU_TIER_MAX, can we just use [tier_min, MAX_NR_TIERS)?
>

I did that previously, but reviewer suggested LRU_TIER_MAX, :D

I'm fine either way, it's really trivial.

Re: [PATCH 6/7] mm/mglru: fix potential generation folio number leak

[Kairui Song]

On Thu, Aug 20, 2026 at 9:52 AM Baolin Wang
<baolin.wang@linux.alibaba.com> wrote:
> On 8/18/26 1:38 PM, Kairui Song via B4 Relay wrote:
> > From: Kairui Song <kasong@tencent.com>
> >
> > Each generation of MGLRU accounts anon and file folio numbers
> > separately. The page table walker's update_batch_size() derives the
> > anon / file type of a folio from its current flags, but the page table
> > walk holds neither the lruvec lock nor the folio lock, so the type can
> > change during that period.
>
> Right.
>
> > MADV_FREE's lazyfree path clears PG_swapbacked under the lruvec lock,
> > so the folio is no longer considered on the anon LRU list. Lazyfreed
> > folios can also be changed back to the anon list again. If the flip
> > lands between folio_update_gen()'s cmpxchg and the type read in
> > update_batch_size(), the batched delta pair is applied to the wrong
> > type. The anon and file generation counters then carry phantom deltas
> > that nothing reconciles, permanently skewing lrugen->nr_pages and the
> > reclaim budgets derived from it.
>
> But I think the problem occurs between update_batch_size() and
> sort_folio(). update_batch_size() only updates the anon or file folio
> statistics, while sort_folio() moves promoted folios to the
> corresponding type's list:
>
> /* promoted */
> if (gen != lru_gen_from_seq(lrugen->min_seq[type])) {
>         list_move(&folio->lru, &lrugen->folios[gen][type][zone]);
>         return true;
> }
>
> If the folio's anon/file type changes between these two steps (e.g., a
> lazyfree folio), it would lead to what you described: "The anon and file
> generation counters then carry phantom deltas that nothing reconciles,
> permanently skewing lrugen->nr_pages and the reclaim budgets derived
> from it."

Actually no, the counters follow eventual consistency (note the word
"permanently"), we are fine with a drift as long as it will eventually
be corrected. Lazy promotions creates counter drift from the physical
location, but that is actually fixed by the sort_folio.

Now, for the type issue, use the lazyfree case as example (I think
that's actually the only place,), lru_lazyfree will remove the folio
form lruvec before marking it !PG_swapbacked, so during that removal
period, the gen bits are zero (folio's gen == -1), so any lazy
promotion CAS won't touch the counter, and only folio_update_gen will
do it since folio_inc_gen only handles on list folios. The
PG_swapbacked clearing in lazyfree only happens on folio wich has gen
== -1 (off-list). And it makes sense since update PG_swapbacked need
to update the counter and move the folio.

And if the CAS happends before the list removal, the list removal, the
folio is on the anon list, so the CAS is moving a folio in the anon
list, folio_update_gen will call update_batch_size asking it to update
the anon counters, we are fine after this commit. (Before this commit,
the CAS is moving a folio in the anon list but update_batch_size may
occur on file coutners). The list removal will update the anon
counter, and the subsequent list addition will account for the right
file counter.

And if the CAS happens after the list add, we are still fine since the
file counter is charged and sees a file type here.

>
> If you agree that this is where the problem lies, I don't see a good way
> to fix it, since the state of a lazyfree folio can change between
> update_batch_size() and sort_folio().

That's the tricky part and I struggled a lot with it since MGLRU-FG
will rely on this kind of lazy promotion very aggressively :)

I plan to do a proper cleanup of MGLRU's gen bits and PG_lru flag, and
introduce a cleaner convention later for all of this, and I think
maybe PG_lru can be merged with the GEN bits. But for now we are still
fine. The CAS change always explicitly checks for gen > 0: so as long
as we don't touch PG_swapbacked for gen >= 0 folios, we are fine. So
far every user is following this. We can fix that if an offending user
appears, and it's not hard to add a sanity check to catch
folio_clear/set_swapbacked that happens for gen >= 0 folios (maybe
check PG_lru too).

>
> A simple approach would be to skip checking the access flag for lazyfree
> folios during the page table walk, and let shrink_folio_list()
> reactivate accessed lazyfree folios instead. What do you think?

The thing is it can go from !PG_swapbacked to PG_swapbacked, or
PG_swapbacked to !PG_swapbacked. Or maybe lazy free isn't set the
moment you look at it but it got set the next moment?