[RFC PATCH v3 0/6] mm: mglru: fix swappiness behavior

[RFC PATCH v3 0/6] mm: mglru: fix swappiness behavior

[Barry Song (Xiaomi)]

RFC v3:
 - Added Ridong's patch of improving isolate_folios() readability;
 - for lru_gen_imbalanced(), apply some suggestions from Zicheng
   and Kairui, considering lruvec size and use macro;
 - drop "mm: mglru: only fall back when reclaim is running at high
   priority"
 - mitigate aging for balanced swappiess value(near middle 100)
   as swappiness bias is less a concern for them; but slightly
   increased aging overhead might not be good.
 *  RFC v2 link:
   https://lore.kernel.org/linux-mm/20260726122123.7614-1-baohua@kernel.org/

RFC v2:
 - Quickly address a few issues raised in the Sashiko comments so
   reviewers can ignore v1 and review a cleaner version instead.
   https://sashiko.dev/#/patchset/20260726012946.18684-1-baohua@kernel.org
   Thanks, Sashiko!

The active/inactive LRU respects swappiness well. Anonymous page
scanning and reclamation increase roughly linearly with
swappiness, while file page scanning and reclamation decrease
accordingly.
For example, when swappiness reaches 200, both pgsteal_file and
pgscan_file drop to zero while building the kernel in a 1 GB
memcg. In contrast, MGLRU shows almost no change across different
swappiness values.

                   pgsteal_file

Swappiness    LRU         MGLRU
--------------------------------
1             10567455      763612
36              990706      480205
71              688170      415848
106             446294      386164
141             286307      359196
176             201733      351686
200                  0      330093


                   pgsteal_anon

Swappiness    LRU         MGLRU
--------------------------------
1              4410548     2726362
36             2465268     2762859
71             2677908     2885124
106            2737227     2841796
141            2984276     3035015
176            3381338     2938302
200           13116359     3113499


                   pgscan_file

Swappiness    LRU         MGLRU
--------------------------------
1             17997223      923094
36             1325674      539571
71              852345      464222
106             538207      464477
141             357253      412277
176             217536      399446
200                  0      375902


                   pgscan_anon

Swappiness    LRU         MGLRU
--------------------------------
1             31639423     5987136
36            23441521     5753224
71            26067110     6101780
106           25619448     5782919
141           26842088     6234264
176           29200021     5980292
200           62193924     6413125

This patchset respects the type selected by positive_ctrl_err(),
which uses swappiness as its gain. It does so by running aging when
the preferred type has few or no reclaimable folios, allowing more
folios of that type to become reclaimable.

For balanced swappiness values near the middle (for example,
60~140), the swappiness bias is less of a concern. We still avoid
aggressive aging and allow fallback to reduce any potential aging
overhead.

In other words, this patch strengthens reclaim type preference
for extreme swappiness values while preserving the existing
MGLRU behavior for balanced swappiness values.

With this patchset, swappiness starts to behave similarly to the
active/inactive LRU.

                         pgsteal_file

Swappiness      LRU        MGLRU      MGLRU+Patch
-------------------------------------------------
1             10567455      763612        1938714
36              990706      480205         427934
71              688170      415848         373450
106             446294      386164         368160
141             286307      359196         334467
176             201733      351686         293716
200                  0      330093             25


                         pgsteal_anon

Swappiness      LRU        MGLRU      MGLRU+Patch
-------------------------------------------------
1              4410548     2726362        3284272
36             2465268     2762859        3315112
71             2677908     2885124        2942357
106            2737227     2841796        2941288
141            2984276     3035015        3345933
176            3381338     2938302        3409531
200           13116359     3113499       14414151


                         pgscan_file

Swappiness      LRU        MGLRU      MGLRU+Patch
-------------------------------------------------
1             17997223      923094        2612649
36             1325674      539571         514506
71              852345      464222         423155
106             538207      464477         436573
141             357253      412277         389644
176             217536      399446         333973
200                  0      375902             25


                         pgscan_anon

Swappiness      LRU        MGLRU      MGLRU+Patch
-------------------------------------------------
1             31639423     5987136        6191482
36            23441521     5753224        6097925
71            26067110     6101780        6157755
106           25619448     5782919        6203221
141           26842088     6234264        6599502
176           29200021     5980292        6783074
200           62193924     6413125       22814377

Barry Song (Xiaomi) (4):
  mm: mglru: prevent min_seq[type] from pointing to an empty generation
  mm: mglru: let scan_folios() scan both reclaimable generations
  mm: mglru: improve scan_folios() exhaustion detection
  mm: mglru: run aging if the preferred type has no folios in
    reclaimable gens

Ridong Chen (1):
  mm/mglru: improve readability of isolate_folios()

lyugaofei (1):
  mm: mglru: run aging when pages are severely imbalanced across gens

 include/linux/mmzone.h |   6 +-
 mm/vmscan.c            | 145 +++++++++++++++++++++++++++++++----------
 2 files changed, 112 insertions(+), 39 deletions(-)

-- 
2.34.1

[RFC PATCH v3 1/6] mm: mglru: prevent min_seq[type] from pointing to an empty generation

[Barry Song (Xiaomi)]

In try_to_inc_min_seq(), min_seq[LRU_GEN_ANON] and
min_seq[LRU_GEN_FILE] can be adjusted to point to empty generations
to keep their gap within one. As a result, scan_folios() may
repeatedly scan empty generations because it assumes the min_seq
generation still contains folios. Likewise,
should_run_aging() has no way to tell that the min_seq generation
has already been drained.

This is quite confusing. I observed scan_folios() returning 0 even
though the following check in scan_folios() doesn't take effect:

        if (get_nr_gens(lruvec, type) == MIN_NR_GENS)
                return 0;

There is no need to adjust min_seq[], since the reclaim logic already
triggers aging when the number of generations reaches
MIN_NR_GENS, and reclaim never reduces it below MIN_NR_GENS.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 include/linux/mmzone.h |  6 ++----
 mm/vmscan.c            | 10 ----------
 2 files changed, 2 insertions(+), 14 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index a26c8b855222..233d2006a541 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -552,10 +552,8 @@ enum {
  * The youngest generation number is stored in max_seq for both anon and file
  * types as they are aged on an equal footing. The oldest generation numbers are
  * stored in min_seq[] separately for anon and file types so that they can be
- * incremented independently. Ideally min_seq[] are kept in sync when both anon
- * and file types are evictable. However, to adapt to situations like extreme
- * swappiness, they are allowed to be out of sync by at most
- * MAX_NR_GENS-MIN_NR_GENS-1.
+ * incremented independently. For both file and anonymous memory, the minimum
+ * generation must be at least MIN_NR_GENS.
  *
  * The number of pages in each generation is eventually consistent and therefore
  * can be transiently negative when reset_batch_size() is pending.
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 566c4e837c7d..ce027c271e9b 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3972,16 +3972,6 @@ static void try_to_inc_min_seq(struct lruvec *lruvec, int swappiness)
 	if (!seq_inc_flag)
 		return;
 
-	/* see the comment on lru_gen_folio */
-	if (swappiness && swappiness <= MAX_SWAPPINESS) {
-		unsigned long seq = lrugen->max_seq - MIN_NR_GENS;
-
-		if (min_seq[LRU_GEN_ANON] > seq && min_seq[LRU_GEN_FILE] < seq)
-			min_seq[LRU_GEN_ANON] = seq;
-		else if (min_seq[LRU_GEN_FILE] > seq && min_seq[LRU_GEN_ANON] < seq)
-			min_seq[LRU_GEN_FILE] = seq;
-	}
-
 	for_each_evictable_type(type, swappiness) {
 		if (min_seq[type] <= lrugen->min_seq[type])
 			continue;
-- 
2.34.1

[RFC PATCH v3 2/6] mm: mglru: let scan_folios() scan both reclaimable generations

[Barry Song (Xiaomi)]

When we have four generations, and scan_folios() exhausts the oldest
one while the second-oldest still contains reclaimable folios, the
current implementation doesn't move on to scan the second-oldest
generation. Instead, it returns, leaving that generation with no
chance to be scanned at the current sc->priority.

This doesn't seem right. Rather than breaking out and starting a
larger next iteration, let's let scan_folios() continue scanning the
second-oldest generation directly.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index ce027c271e9b..31947fa60f18 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4718,6 +4718,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	int skipped = 0;
 	unsigned long remaining = nr_to_scan;
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
+	unsigned long min_seq = lrugen->min_seq[type];
 
 	VM_WARN_ON_ONCE(nr_to_scan > MAX_LRU_BATCH);
 	VM_WARN_ON_ONCE(!list_empty(list));
@@ -4725,8 +4726,8 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	if (get_nr_gens(lruvec, type) == MIN_NR_GENS)
 		return 0;
 
-	gen = lru_gen_from_seq(lrugen->min_seq[type]);
-
+next_gen:
+	gen = lru_gen_from_seq(min_seq);
 	for (i = MAX_NR_ZONES; i > 0; i--) {
 		LIST_HEAD(moved);
 		int skipped_zone = 0;
@@ -4768,6 +4769,14 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 			break;
 	}
 
+	/*
+	 * This generation is exhausted across all zones, but the scan
+	 * target has not been reached yet. Continue with the next
+	 * reclaimable generation.
+	 */
+	if (i == 0 && ++min_seq + MIN_NR_GENS <= lrugen->max_seq)
+		goto next_gen;
+
 	item = PGSCAN_KSWAPD + reclaimer_offset(sc);
 	mod_lruvec_state(lruvec, item, isolated);
 	mod_lruvec_state(lruvec, PGREFILL, sorted);
-- 
2.34.1

[RFC PATCH v3 3/6] mm/mglru: improve readability of isolate_folios()

[Barry Song (Xiaomi)]

From: Ridong Chen <chenridong@xiaomi.com>

The for_each_evictable_type() loop in isolate_folios()
is misleading: it does not actually iterate over each
evictable type. Instead, get_type_to_scan() selects the
type to scan, while the iterator `i` merely bounds the
number of attempts.

Signed-off-by: Ridong Chen <chenridong@xiaomi.com>
Co-developed-by: Barry Song (Xiaomi) <baohua@kernel.org>
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 47 +++++++++++++++++++++++++++--------------------
 1 file changed, 27 insertions(+), 20 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 31947fa60f18..0038f33aa318 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4828,35 +4828,42 @@ static int get_type_to_scan(struct lruvec *lruvec, int swappiness)
 	return positive_ctrl_err(&sp, &pv);
 }
 
+static inline bool is_single_type_reclaim(int swappiness)
+{
+	return swappiness == MIN_SWAPPINESS ||
+	       swappiness == SWAPPINESS_ANON_ONLY;
+}
+
 static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 			  struct scan_control *sc, int swappiness,
 			  struct list_head *list, int *isolated,
 			  int *isolate_type, int *isolate_scanned)
 {
-	int i;
-	int total_scanned = 0;
+	bool single_type = is_single_type_reclaim(swappiness);
 	int type = get_type_to_scan(lruvec, swappiness);
+	int total_scanned = 0, scanned, tier;
+	bool tried = false;
 
-	for_each_evictable_type(i, swappiness) {
-		int scanned;
-		int tier = get_tier_idx(lruvec, type);
+retry:
+	tier = get_tier_idx(lruvec, type);
+	scanned = scan_folios(nr_to_scan, lruvec, sc,
+			      type, tier, list, isolated);
 
-		scanned = scan_folios(nr_to_scan, lruvec, sc,
-				      type, tier, list, isolated);
+	total_scanned += scanned;
+	if (*isolated) {
+		*isolate_type = type;
+		*isolate_scanned = scanned;
+		return total_scanned;
+	}
 
-		total_scanned += scanned;
-		if (*isolated) {
-			*isolate_type = type;
-			*isolate_scanned = scanned;
-			break;
-		}
-		/*
-		 * If scanned > 0 and isolated == 0, avoid falling back to the
-		 * other type, as this type remains sufficient. Falling back
-		 * too readily can disrupt the positive_ctrl_err() bias.
-		 */
-		if (!scanned)
-			type = !type;
+	/*
+	 * We are running out of the current reclaim type. Fall back to
+	 * the other type if allowed.
+	 */
+	if (!tried && !scanned && !single_type) {
+		type = !type;
+		tried = true;
+		goto retry;
 	}
 
 	return total_scanned;
-- 
2.34.1

[RFC PATCH v3 4/6] mm: mglru: improve scan_folios() exhaustion detection

[Barry Song (Xiaomi)]

Commit 16b475d2ac3c ("mm/mglru: avoid reclaim type fall back when
isolation makes no progress") uses scanned == 0 to determine
whether scan_folios() has exhausted a reclaim type. However,
this is not always sufficient. It is possible for scanned > 0,
while the oldest reclaimable generation is exhausted after the
first scan_folios() call.

Add an exhausted output argument to scan_folios() so it can
explicitly report whether the reclaimable lists for the current
type have been exhausted.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 0038f33aa318..2e7fef6975b6 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4707,7 +4707,8 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca
 
 static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 		       struct scan_control *sc, int type, int tier,
-		       struct list_head *list, int *isolatedp)
+		       struct list_head *list, int *isolatedp,
+		       bool *exhausted)
 {
 	int i;
 	int gen;
@@ -4723,8 +4724,10 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	VM_WARN_ON_ONCE(nr_to_scan > MAX_LRU_BATCH);
 	VM_WARN_ON_ONCE(!list_empty(list));
 
-	if (get_nr_gens(lruvec, type) == MIN_NR_GENS)
+	if (get_nr_gens(lruvec, type) == MIN_NR_GENS) {
+		*exhausted = true;
 		return 0;
+	}
 
 next_gen:
 	gen = lru_gen_from_seq(min_seq);
@@ -4785,6 +4788,11 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 				scanned, skipped, isolated,
 				type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON);
 
+	/*
+	 * This scan exhausted the reclaimable lists before reaching the
+	 * scan target or accumulating enough isolated folios.
+	 */
+	*exhausted = remaining > 0 && isolated < MIN_LRU_BATCH;
 	*isolatedp = isolated;
 	return scanned;
 }
@@ -4842,12 +4850,12 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	bool single_type = is_single_type_reclaim(swappiness);
 	int type = get_type_to_scan(lruvec, swappiness);
 	int total_scanned = 0, scanned, tier;
-	bool tried = false;
+	bool tried = false, exhausted;
 
 retry:
 	tier = get_tier_idx(lruvec, type);
 	scanned = scan_folios(nr_to_scan, lruvec, sc,
-			      type, tier, list, isolated);
+			      type, tier, list, isolated, &exhausted);
 
 	total_scanned += scanned;
 	if (*isolated) {
@@ -4860,7 +4868,7 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	 * We are running out of the current reclaim type. Fall back to
 	 * the other type if allowed.
 	 */
-	if (!tried && !scanned && !single_type) {
+	if (!tried && exhausted && !single_type) {
 		type = !type;
 		tried = true;
 		goto retry;
-- 
2.34.1

[RFC PATCH v3 5/6] mm: mglru: run aging when pages are severely imbalanced across gens

[Barry Song (Xiaomi)]

From: lyugaofei <lyugaofei@xiaomi.com>

This partially restores the reclaim behavior introduced in Yu
Zhao's initial MGLRU commit, ac35a4902370 ("mm: multi-gen LRU:
minimal implementation"):
	/*
	 * It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1)
	 * of the total number of pages for each generation. A reasonable range
	 * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The
	 * aging cares about the upper bound of hot pages, while the eviction
	 * cares about the lower bound of cold pages.
	 */
	if (young * MIN_NR_GENS > total)
		return true;
	if (old * (MIN_NR_GENS + 2) < total)
		return true;

But with a stricter condition: the younger generations must
contain at least old_ratio times as many folios as the older
generations. The old_ratio is derived from the lruvec size, so
larger lruvecs use a higher old_ratio. This allows aging to
keep folios of the preferred type distributed across the
reclaimable generations.

Also, imbalanced aging is only applied to non-balanced swappiness
values (for example, < 60 or > 140). For balanced swappiness values
near the middle, the swappiness bias is less of a concern, while the
slightly increased aging overhead might be unacceptable.

Signed-off-by: lyugaofei <lyugaofei@xiaomi.com>
Co-developed-by: Barry Song (Xiaomi) <baohua@kernel.org>
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 58 insertions(+), 1 deletion(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 2e7fef6975b6..98fd2fac90ac 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4973,9 +4973,62 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	return scanned;
 }
 
+static inline bool swappiness_is_balanced(int swappiness)
+{
+	int range = MAX_SWAPPINESS - MIN_SWAPPINESS;
+	int middle = MIN_SWAPPINESS + range / 2;
+
+	return swappiness >= middle - range / 5 &&
+	       swappiness <= middle + range / 5;
+}
+
+static bool lru_gen_imbalanced(struct lruvec *lruvec, int type,
+		unsigned long max_seq, unsigned long min_seq,
+		struct scan_control *sc, int swappiness)
+{
+	struct lru_gen_folio *lrugen = &lruvec->lrugen;
+	unsigned long young = 0, old = 0, seq;
+	unsigned long old_ratio, gb;
+
+	/* Skip swappiness bias for single-type reclaim or balanced swappiness */
+	if (is_single_type_reclaim(swappiness) || swappiness_is_balanced(swappiness))
+		return false;
+
+	/* More than two generations remain to reclaim */
+	if (min_seq + MIN_NR_GENS < max_seq)
+		return false;
+
+	/*
+	 * Run aging if the only remaining reclaimable generation
+	 * has few folios.
+	 */
+	for (seq = min_seq; seq <= max_seq; seq++) {
+		int gen = lru_gen_from_seq(seq);
+		unsigned long size = 0;
+		int zone;
+
+		for (zone = 0; zone < MAX_NR_ZONES; zone++)
+			size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
+
+		if (seq + MIN_NR_GENS > max_seq)
+			young += size;
+		else
+			old += size;
+	}
+	/*
+	 * Copied from inactive_is_low(), but uses a higher old_ratio to
+	 * make aging less aggressive.
+	 */
+	gb = (young + old) >> (30 - PAGE_SHIFT);
+	old_ratio = gb ? int_sqrt(10 * gb) : 1;
+	old_ratio *= MAX_NR_GENS;
+	return young > old * old_ratio;
+}
+
 static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,
 			     struct scan_control *sc, int swappiness)
 {
+	int type = get_type_to_scan(lruvec, swappiness);
 	DEFINE_MIN_SEQ(lruvec);
 
 	/* have to run aging, since eviction is not possible anymore */
@@ -4987,7 +5040,11 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,
 		return false;
 
 	/* better to run aging even though eviction is still possible */
-	return evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS == max_seq;
+	if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS == max_seq)
+		return true;
+
+	/* Run aging if the preferred type is severely imbalanced across gens */
+	return lru_gen_imbalanced(lruvec, type, max_seq, min_seq[type], sc, swappiness);
 }
 
 static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc,
-- 
2.34.1

[RFC PATCH v3 6/6] mm: mglru: run aging if the preferred type has no folios in reclaimable gens

[Barry Song (Xiaomi)]

Respect the type selected by positive_ctrl_err(). If there are no
reclaimable generations left for that type, run aging.

For balanced swappiness values near the middle, still allow gentle
reclaim since the swappiness bias is less of a concern, while the
slightly increased aging overhead might become a problem.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 98fd2fac90ac..902d85afc5f6 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -5035,6 +5035,10 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,
 	if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq)
 		return true;
 
+	/* run aging if the preferred type is exhausted for extreme swappiness */
+	if (min_seq[type] + MIN_NR_GENS > max_seq && !swappiness_is_balanced(swappiness))
+		return true;
+
 	/* try to avoid aging, do gentle reclaim at the default priority */
 	if (sc->priority == DEF_PRIORITY)
 		return false;
-- 
2.34.1