[RFC PATCH v4 00/16] mm: mglru: fix swappiness behavior

[RFC PATCH v4 00/16] mm: mglru: fix swappiness behavior

[Barry Song (Xiaomi)]

The active/inactive LRU respects swappiness well. Anonymous page
scanning and reclamation increase roughly linearly with swappiness,
while file page scanning and reclamation decrease accordingly.

For example, with extreme swappiness values such as 1, the
active/inactive LRU performs much more file reclaim, while at 200 it
performs much more anonymous reclaim with little or even no file
reclaim.

In contrast, MGLRU shows little change across different swappiness
values, especially at the extremes of 1 and 200.
The data below was collected while building the kernel in a 1 GB
MEMCG with -j20 and zRAM configured as swap.

                         pgsteal_file

Swappiness      LRU        MGLRU
--------------------------------
1             10788341      760931
36             1101082      526479
71              683084      406540
106             465165      380845
141             292018      361058
176             207810      332329
200                  0      349617


                         pgsteal_anon

Swappiness      LRU        MGLRU
--------------------------------
1              4305108     2594416
36             2790004     2984411
71             2692223     2965396
106            2844429     3039293
141            3101425     3087067
176            3394419     3110359
200           13210927     3080592


                         pgscan_file

Swappiness      LRU        MGLRU
--------------------------------
1             18718146      887478
36             1485019      610943
71              849670      460520
106             561680      433057
141             368688      404839
176             234273      375214
200                  0      400190


                         pgscan_anon

Swappiness      LRU        MGLRU
--------------------------------
1             29683174     6008050
36            27142699     5960053
71            25804052     6278323
106           26582642     6416301
141           28199275     6349025
176           29360048     6277269
200           62517850     6295609


                         refault_file

Swappiness      LRU        MGLRU
--------------------------------
1             10693128      587183
36             1018405      343244
71              602508      247697
106             388580      224246
141             216110      200426
176             127306      175176
200                  0      189009


                         refault_anon

Swappiness      LRU        MGLRU
--------------------------------
1               593074      540688
36              231609      654679
71              263264      642920
106             286477      640700
141             331410      634729
176             390750      656967
200            4120283      642881

This patchset respects the type selected by get_type_to_scan(),
which uses swappiness as its gain. It does so by running aging when
the preferred type has few or no reclaimable folios, allowing more
folios of that type to become reclaimable.

As we begin to respect swappiness more strictly, inc_min_seq()
may be called more frequently. While this function resolves cold/hot
inversion issues, it may itself introduce additional cold/hot
inversions, which are addressed by this patchset. In addition, we use
batching where possible to reduce the overhead of inc_min_seq().

Patches 1–3 improve readability and enhance the exhausted checks in
isolate_folios() and scan_folios().

Patch 4 increases the impact of swappiness on type selection.

Patches 5–10 fix potential cold/hot inversions and use batching where
possible to improve the performance of inc_min_seq().

Patches 11–15 improve should_run_aging() with additional generation
and imbalance checks, particularly when few folios remain in the
reclaimable generations.

Patch 16 reduces the batch size when moving folios from the oldest
generation to the second-oldest generation in inc_min_seq() to reduce
lock contention.

With this patchset, swappiness starts to behave more like it does
with the active/inactive LRU.

                         pgsteal_file

Swappiness      LRU        MGLRU      MGLRU+Patch
-------------------------------------------------
1             10788341      760931        1974207
36             1101082      526479         620248
71              683084      406540         414207
106             465165      380845         365887
141             292018      361058         335876
176             207810      332329         306476
200                  0      349617          10569


                         pgsteal_anon

Swappiness      LRU        MGLRU      MGLRU+Patch
-------------------------------------------------
1              4305108     2594416        3204425
36             2790004     2984411        2895280
71             2692223     2965396        2766936
106            2844429     3039293        3105011
141            3101425     3087067        2989129
176            3394419     3110359        3183651
200           13210927     3080592       13354629


                         pgscan_file

Swappiness      LRU        MGLRU      MGLRU+Patch
-------------------------------------------------
1             18718146      887478        2753870
36             1485019      610943         712969
71              849670      460520         471135
106             561680      433057         419504
141             368688      404839         374698
176             234273      375214         340775
200                  0      400190          10880


                         pgscan_anon

Swappiness      LRU        MGLRU      MGLRU+Patch
-------------------------------------------------
1             29683174     6008050        5645754
36            27142699     5960053        6056810
71            25804052     6278323        5877296
106           26582642     6416301        6397449
141           28199275     6349025        5951263
176           29360048     6277269        6076458
200           62517850     6295609       21506939


                         refault_file

Swappiness      LRU        MGLRU      MGLRU+Patch
-------------------------------------------------
1             10693128      587183        1836987
36             1018405      343244         443893
71              602508      247697         245171
106             388580      224246         202345
141             216110      200426         178638
176             127306      175176         141618
200                  0      189009           5003


                         refault_anon

Swappiness      LRU        MGLRU      MGLRU+Patch
-------------------------------------------------
1               593074      540688         800149
36              231609      654679         609397
71              263264      642920         582514
106             286477      640700         653094
141             331410      634729         636040
176             390750      656967         667461
200            4120283      642881        4126464

It could be brought even closer to the active/inactive LRU by
ignoring the potential cost of inc_min_seq() and allowing more
aggressive aging. For now, however, we aim to balance the benefits of
respecting swappiness against the overall system time. Across almost
all swappiness levels, the kernel build workload shows that MGLRU with
the patch achieves lower system and wall-clock times than the
active/inactive LRU.

                         sys time

Swappiness      LRU        MGLRU+Patch
---------------------------------------
1              7m42.831s     6m15.160s
36             4m51.991s     3m42.020s
71             4m29.953s     3m39.259s
106            4m34.790s     3m48.321s
141            4m43.026s     3m42.552s
176            4m54.383s     3m46.840s
200            8m11.063s     7m28.910s

                     wall time
## Swappiness      LRU        MGLRU+Patch

1              3m08.386s     2m01.631s
36             1m54.989s     1m49.887s
71             1m52.556s     1m48.879s
106            1m51.885s     1m49.942s
141            1m52.143s     1m48.916s
176            1m51.925s     1m48.703s
200            2m02.300s     2m00.487s

RFC v4:
* Clean up isolate_folios() based on Baolin's feedback. Thanks!
* Further improve isolate_folios() and scan_folios(), including
  handling of exhausted reclaim types.
* Add fixes and performance improvements to inc_min_seq(),
  and reduce lock contention.
* Improve the handling of imbalanced aging and remaining
  generation-based aging.
*  RFC v3 link:
 https://lore.kernel.org/linux-mm/20260731083843.37811-1-baohua@kernel.org/

RFC v3:
 - Added Ridong's patch of improving isolate_folios() readability;
 - for lru_gen_imbalanced(), apply some suggestions from Zicheng
   and Kairui, considering lruvec size and use macro;
 - drop "mm: mglru: only fall back when reclaim is running at high
   priority"
 - mitigate aging for balanced swappiess value(near middle 100)
   as swappiness bias is less a concern for them; but slightly
   increased aging overhead might not be good.
 *  RFC v2 link:
   https://lore.kernel.org/linux-mm/20260726122123.7614-1-baohua@kernel.org/

RFC v2:
 - Quickly address a few issues raised in the Sashiko comments so
   reviewers can ignore v1 and review a cleaner version instead.
   https://sashiko.dev/#/patchset/20260726012946.18684-1-baohua@kernel.org
   Thanks, Sashiko!

Barry Song (Xiaomi) (13):
  mm/mglru: improve scan_folios() exhaustion detection
  mm/mglru: retry the same type once if isolation fails due to races
  mm/mglru: boost swappiness responsiveness in get_type_to_scan()
  mm/mglru: batch update lrugen->nr_pages in inc_min_seq()
  mm/mglru: batch update lrugen->protected in inc_min_seq()
  mm/mglru: enhance cold/hot inversion handling in inc_min_seq()
  mm/mglru: exclude folios promoted by aging from protected in
    inc_min_seq()
  mm/mglru: move folios from oldest gen to second-oldest gen from head
    to tail
  mm/mglru: batch move folios to the second-oldest gen's LRU
  mm/mglru: skip gentle reclaim at DEF_PRIORITY for extreme swappiness
  mm/mglru: run aging if the preferred type has no reclaimable gens
  mm/mglru: dynamically scale aging threshold in lru_gen_imbalanced()
  mm/mglru: reduce folios pulled from the oldest gen in inc_min_seq()

Bo Zhang (1):
  mm/mglru: remove redundant gens <= MIN_NR_GENS check in
    should_run_aging()

Ridong Chen (1):
  mm/mglru: improve readability of isolate_folios()

lyugaofei (1):
  mm/mglru: run aging when pages are severely imbalanced across gens

 mm/vmscan.c | 289 ++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 233 insertions(+), 56 deletions(-)

-- 
2.34.1

[RFC PATCH v4 01/16] mm/mglru: improve readability of isolate_folios()

[Barry Song (Xiaomi)]

From: Ridong Chen <chenridong@xiaomi.com>

The for_each_evictable_type() loop in isolate_folios()
is misleading: it does not actually iterate over each
evictable type. Instead, get_type_to_scan() selects the
type to scan, while the iterator `i` merely bounds the
number of attempts.

Signed-off-by: Ridong Chen <chenridong@xiaomi.com>
Co-developed-by: Barry Song (Xiaomi) <baohua@kernel.org>
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 46 ++++++++++++++++++++++++++--------------------
 1 file changed, 26 insertions(+), 20 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 17d2b793cbfc..ea058692b9a5 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4839,35 +4839,41 @@ static int get_type_to_scan(struct lruvec *lruvec, int swappiness)
 	return positive_ctrl_err(&sp, &pv);
 }
 
+static inline bool is_single_type_reclaim(int swappiness)
+{
+	return swappiness == MIN_SWAPPINESS ||
+	       swappiness == SWAPPINESS_ANON_ONLY;
+}
+
 static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 			  struct scan_control *sc, int swappiness,
 			  struct list_head *list, int *isolated,
 			  int *isolate_type, int *isolate_scanned)
 {
-	int i;
-	int total_scanned = 0;
+	bool type_fallback_allowed = !is_single_type_reclaim(swappiness);
 	int type = get_type_to_scan(lruvec, swappiness);
+	int total_scanned = 0, scanned, tier;
 
-	for_each_evictable_type(i, swappiness) {
-		int scanned;
-		int tier = get_tier_idx(lruvec, type);
+retry:
+	tier = get_tier_idx(lruvec, type);
+	scanned = scan_folios(nr_to_scan, lruvec, sc,
+			      type, tier, list, isolated);
 
-		scanned = scan_folios(nr_to_scan, lruvec, sc,
-				      type, tier, list, isolated);
+	total_scanned += scanned;
+	if (*isolated) {
+		*isolate_type = type;
+		*isolate_scanned = scanned;
+		return total_scanned;
+	}
 
-		total_scanned += scanned;
-		if (*isolated) {
-			*isolate_type = type;
-			*isolate_scanned = scanned;
-			break;
-		}
-		/*
-		 * If scanned > 0 and isolated == 0, avoid falling back to the
-		 * other type, as this type remains sufficient. Falling back
-		 * too readily can disrupt the positive_ctrl_err() bias.
-		 */
-		if (!scanned)
-			type = !type;
+	/*
+	 * We are running out of the current reclaim type. Fall back to
+	 * the other type if allowed.
+	 */
+	if (!scanned && type_fallback_allowed) {
+		type = !type;
+		type_fallback_allowed = false;
+		goto retry;
 	}
 
 	return total_scanned;
-- 
2.34.1

[RFC PATCH v4 02/16] mm/mglru: improve scan_folios() exhaustion detection

[Barry Song (Xiaomi)]

Commit 16b475d2ac3c ("mm/mglru: avoid reclaim type fall back when
isolation makes no progress") uses scanned == 0 to determine
whether scan_folios() has exhausted a reclaim type. However,
this is not always sufficient. It is possible for scanned > 0,
while the oldest reclaimable generation is exhausted after the
first scan_folios() call.

We detect early_stop in scan_folios(). If we stop early for any reason,
it means the current reclaim type is not exhausted yet. If early_stop is
never reached, it means we have exhausted the current oldest generation
without hitting any scanning limit.

Another issue is that if the lruvec has 4 generations, we might have
exhausted the oldest generation while the second oldest generation is
still reclaimable. In that case, this type is not exhausted yet.

Add an exhausted output argument to scan_folios() so it can
explicitly report whether the reclaimable lists for the current
type have been exhausted.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 28 ++++++++++++++++++++++------
 1 file changed, 22 insertions(+), 6 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index ea058692b9a5..0670a25d3a7e 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4727,7 +4727,8 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca
 
 static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 		       struct scan_control *sc, int type, int tier,
-		       struct list_head *list, int *isolatedp)
+		       struct list_head *list, int *isolatedp,
+		       bool *exhausted)
 {
 	int i;
 	int gen;
@@ -4738,12 +4739,15 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	int skipped = 0;
 	unsigned long remaining = nr_to_scan;
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
+	bool early_stop = false;
 
 	VM_WARN_ON_ONCE(nr_to_scan > MAX_LRU_BATCH);
 	VM_WARN_ON_ONCE(!list_empty(list));
 
-	if (get_nr_gens(lruvec, type) == MIN_NR_GENS)
+	if (get_nr_gens(lruvec, type) == MIN_NR_GENS) {
+		*exhausted = true;
 		return 0;
+	}
 
 	gen = lru_gen_from_seq(lrugen->min_seq[type]);
 
@@ -4774,8 +4778,10 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 				skipped_zone += delta;
 			}
 
-			if (!--remaining || max(isolated, skipped_zone) >= MIN_LRU_BATCH)
+			if (!--remaining || max(isolated, skipped_zone) >= MIN_LRU_BATCH) {
+				early_stop = true;
 				break;
+			}
 		}
 
 		if (skipped_zone) {
@@ -4784,8 +4790,10 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 			skipped += skipped_zone;
 		}
 
-		if (!remaining || isolated >= MIN_LRU_BATCH)
+		if (!remaining || isolated >= MIN_LRU_BATCH) {
+			early_stop = true;
 			break;
+		}
 	}
 
 	item = PGSCAN_KSWAPD + reclaimer_offset(sc);
@@ -4796,6 +4804,13 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 				scanned, skipped, isolated,
 				type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON);
 
+	/*
+	 * If we didn't stop early, all reclaimable folios in the current
+	 * generation have been scanned. We are exhausted if this is the last
+	 * reclaimable generation.
+	 */
+	*exhausted = !early_stop &&
+		     lrugen->min_seq[type] + MIN_NR_GENS == lrugen->max_seq;
 	*isolatedp = isolated;
 	return scanned;
 }
@@ -4853,11 +4868,12 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	bool type_fallback_allowed = !is_single_type_reclaim(swappiness);
 	int type = get_type_to_scan(lruvec, swappiness);
 	int total_scanned = 0, scanned, tier;
+	bool exhausted;
 
 retry:
 	tier = get_tier_idx(lruvec, type);
 	scanned = scan_folios(nr_to_scan, lruvec, sc,
-			      type, tier, list, isolated);
+			      type, tier, list, isolated, &exhausted);
 
 	total_scanned += scanned;
 	if (*isolated) {
@@ -4870,7 +4886,7 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	 * We are running out of the current reclaim type. Fall back to
 	 * the other type if allowed.
 	 */
-	if (!scanned && type_fallback_allowed) {
+	if (exhausted && type_fallback_allowed) {
 		type = !type;
 		type_fallback_allowed = false;
 		goto retry;
-- 
2.34.1

[RFC PATCH v4 03/16] mm/mglru: retry the same type once if isolation fails due to races

[Barry Song (Xiaomi)]

If we are not exhausted (i.e., there are still folios in the
reclaimable generations) but fail to isolate any folios due to
promotions, protection, or races, give this type one more chance to
avoid going through the outer loop again.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 0670a25d3a7e..dddd2ecf970a 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4868,7 +4868,7 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	bool type_fallback_allowed = !is_single_type_reclaim(swappiness);
 	int type = get_type_to_scan(lruvec, swappiness);
 	int total_scanned = 0, scanned, tier;
-	bool exhausted;
+	bool exhausted, tried = false;
 
 retry:
 	tier = get_tier_idx(lruvec, type);
@@ -4891,6 +4891,14 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 		type_fallback_allowed = false;
 		goto retry;
 	}
+	/*
+	 * We are not exhausted, but failed to isolate any folios due to
+	 * races. Give this type one more chance to avoid a larger loop.
+	 */
+	if (!exhausted && !tried) {
+		tried = true;
+		goto retry;
+	}
 
 	return total_scanned;
 }
-- 
2.34.1

[RFC PATCH v4 04/16] mm/mglru: boost swappiness responsiveness in get_type_to_scan()

[Barry Song (Xiaomi)]

The linear weight in get_type_to_scan() can easily be overwhelmed
by historical refault costs when swappiness deviates from neutral (100).
As a result, user-configured swappiness often feels unresponsive under
memory pressure.

Apply a smooth quadratic boost based on the distance from the neutral
balance point (MAX_SWAPPINESS / 2). This amplifies the weight of the
preferred scanning type (anon vs file) while avoiding hardware division
via bitwise shift.

Assisted-by: gemini:gemini-3.6-flash
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index dddd2ecf970a..2fd1e06eb192 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4838,18 +4838,42 @@ static int get_tier_idx(struct lruvec *lruvec, int type)
 static int get_type_to_scan(struct lruvec *lruvec, int swappiness)
 {
 	struct ctrl_pos sp, pv = {};
+	int anon_gain, file_gain;
 
 	if (swappiness <= MIN_SWAPPINESS + 1)
 		return LRU_GEN_FILE;
 
 	if (swappiness >= MAX_SWAPPINESS)
 		return LRU_GEN_ANON;
+
+	/*
+	 * Apply a quadratic boost based on the distance from the neutral
+	 * balance point (swappiness = MAX_SWAPPINESS / 2).
+	 *
+	 * A linear weight is easily overwhelmed by historical refault cost
+	 * when swappiness deviates from neutral. The quadratic scaling
+	 * amplifies the weight of the preferred type smoothly.
+	 */
+	if (swappiness < MAX_SWAPPINESS / 2) {
+		int delta = (MAX_SWAPPINESS / 2) - swappiness;
+		int boost = (delta * delta) >> 4;
+
+		anon_gain = swappiness;
+		file_gain = (MAX_SWAPPINESS - swappiness) + boost;
+	} else {
+		int delta = swappiness - (MAX_SWAPPINESS / 2);
+		int boost = (delta * delta) >> 4;
+
+		anon_gain = swappiness + boost;
+		file_gain = MAX_SWAPPINESS - swappiness;
+	}
+
 	/*
 	 * Compare the sum of all tiers of anon with that of file to determine
 	 * which type to scan.
 	 */
-	read_ctrl_pos(lruvec, LRU_GEN_ANON, MAX_NR_TIERS, swappiness, &sp);
-	read_ctrl_pos(lruvec, LRU_GEN_FILE, MAX_NR_TIERS, MAX_SWAPPINESS - swappiness, &pv);
+	read_ctrl_pos(lruvec, LRU_GEN_ANON, MAX_NR_TIERS, anon_gain, &sp);
+	read_ctrl_pos(lruvec, LRU_GEN_FILE, MAX_NR_TIERS, file_gain, &pv);
 
 	return positive_ctrl_err(&sp, &pv);
 }
-- 
2.34.1

[RFC PATCH v4 05/16] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

[Barry Song (Xiaomi)]

Currently, folio_inc_gen() updates lrugen->nr_pages for every folio
as it advances generations. Instead, accumulate the size changes
and update lrugen->nr_pages in a batch after scanning the entire
oldest generation, or when the scan stops because remaining reaches
zero.

Since we only move folios from the oldest generation to the second
oldest generation, the active/inactive state cannot change. We can
therefore skip __lru_update_size().

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++-----------
 1 file changed, 35 insertions(+), 11 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 2fd1e06eb192..c6e3b92c8cae 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3295,20 +3295,21 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma
 }
 
 /* protect pages accessed multiple times through file descriptors */
-static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
+static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
 {
-	int type = folio_is_file_lru(folio);
-	struct lru_gen_folio *lrugen = &lruvec->lrugen;
-	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
 	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
+	int new_gen;
 
 	VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio);
 
 	do {
 		new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
 		/* folio_update_gen() has promoted this page? */
-		if (new_gen >= 0 && new_gen != old_gen)
+		if (new_gen >= 0 && new_gen != old_gen) {
+			if (increased)
+				*increased = false;
 			return new_gen;
+		}
 
 		new_gen = (old_gen + 1) % MAX_NR_GENS;
 
@@ -3316,8 +3317,21 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
 		new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF;
 	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
 
-	lru_gen_update_size(lruvec, folio, old_gen, new_gen);
+	if (increased)
+		*increased = true;
+	return new_gen;
+}
 
+static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
+{
+	int type = folio_is_file_lru(folio);
+	struct lru_gen_folio *lrugen = &lruvec->lrugen;
+	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
+	bool gen_increased;
+
+	new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
+	if (gen_increased)
+		lru_gen_update_size(lruvec, folio, old_gen, new_gen);
 	return new_gen;
 }
 
@@ -3903,6 +3917,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
 	int hist = lru_hist_from_seq(lrugen->min_seq[type]);
 	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
+	int target_gen = (old_gen + 1) % MAX_NR_GENS;
 
 	/* For file type, skip the check if swappiness is anon only */
 	if (type && (swappiness == SWAPPINESS_ANON_ONLY))
@@ -3915,32 +3930,41 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
+		unsigned long delta = 0;
 
 		while (!list_empty(head)) {
 			struct folio *folio = lru_to_folio(head);
+			long nr_pages = folio_nr_pages(folio);
 			int refs = folio_lru_refs(folio);
 			bool workingset = folio_test_workingset(folio);
+			bool gen_increased;
 
 			VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
-			new_gen = folio_inc_gen(lruvec, folio);
+			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
-
+			if (gen_increased)
+				delta += nr_pages;
 			/* don't count the workingset being lazily promoted */
 			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 				int tier = lru_tier_from_refs(refs, workingset);
-				int delta = folio_nr_pages(folio);
 
 				WRITE_ONCE(lrugen->protected[hist][type][tier],
-					   lrugen->protected[hist][type][tier] + delta);
+					   lrugen->protected[hist][type][tier] + nr_pages);
 			}
 
 			if (!--remaining)
-				return false;
+				break;
 		}
+		WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
+			   lrugen->nr_pages[old_gen][type][zone] - delta);
+		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
+			   lrugen->nr_pages[target_gen][type][zone] + delta);
+		if (!remaining)
+			return false;
 	}
 done:
 	reset_ctrl_pos(lruvec, type, true);
-- 
2.34.1

[RFC PATCH v4 06/16] mm/mglru: batch update lrugen->protected in inc_min_seq()

[Barry Song (Xiaomi)]

Avoid updating lrugen->protected with WRITE_ONCE() for each folio,
which may prevent potential compiler optimizations. Accumulate the
updates locally and apply them in a batch instead.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index c6e3b92c8cae..10f690389f5d 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3930,7 +3930,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
-		unsigned long delta = 0;
+		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
 
 		while (!list_empty(head)) {
 			struct folio *folio = lru_to_folio(head);
@@ -3952,8 +3952,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 				int tier = lru_tier_from_refs(refs, workingset);
 
-				WRITE_ONCE(lrugen->protected[hist][type][tier],
-					   lrugen->protected[hist][type][tier] + nr_pages);
+				protected[tier] += nr_pages;
 			}
 
 			if (!--remaining)
@@ -3963,6 +3962,9 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			   lrugen->nr_pages[old_gen][type][zone] - delta);
 		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
 			   lrugen->nr_pages[target_gen][type][zone] + delta);
+		for (int tier = 0; tier < MAX_NR_TIERS; tier++)
+			WRITE_ONCE(lrugen->protected[hist][type][tier],
+				   lrugen->protected[hist][type][tier] + protected[tier]);
 		if (!remaining)
 			return false;
 	}
-- 
2.34.1

[RFC PATCH v4 07/16] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

[Barry Song (Xiaomi)]

During aging, a folio's generation may already have been updated by
folio_update_gen(), even though it has not yet been moved to the
corresponding generation list. Such folios are hotter than those
already in that generation.

It makes sense for inc_min_seq() to increment the generation of
folios that were never promoted during aging and move them to the
tail of the new oldest generation. However, folios that were already
promoted should instead be moved to the head of their updated
generation, just as sort_folio() does in scan_folios().

Otherwise, promoted folios could end up behind folios that were
never promoted, effectively inverting their hot/cold ordering.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 10f690389f5d..e0625e6ec919 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3945,9 +3945,12 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
 			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
-			list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
-			if (gen_increased)
+			if (gen_increased) {
 				delta += nr_pages;
+				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+			} else {
+				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+			}
 			/* don't count the workingset being lazily promoted */
 			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 				int tier = lru_tier_from_refs(refs, workingset);
-- 
2.34.1

[RFC PATCH v4 08/16] mm/mglru: exclude folios promoted by aging from protected in inc_min_seq()

[Barry Song (Xiaomi)]

Some folios may have been promoted during aging, so don't count them
as protected, similar to sort_folio().

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index e0625e6ec919..27623d3bad95 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3948,16 +3948,15 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			if (gen_increased) {
 				delta += nr_pages;
 				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+				/* don't count the workingset being lazily promoted */
+				if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
+					int tier = lru_tier_from_refs(refs, workingset);
+
+					protected[tier] += nr_pages;
+				}
 			} else {
 				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
 			}
-			/* don't count the workingset being lazily promoted */
-			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
-				int tier = lru_tier_from_refs(refs, workingset);
-
-				protected[tier] += nr_pages;
-			}
-
 			if (!--remaining)
 				break;
 		}
-- 
2.34.1

[RFC PATCH v4 09/16] mm/mglru: move folios from oldest gen to second-oldest gen from head to tail

[Barry Song (Xiaomi)]

For reclamation, it makes sense to reclaim folios from tail to
head, as folios near the head are relatively hot. However, when
moving folios from the oldest generation to the second-oldest
generation, using the tail-to-head order would effectively cause
a cold/hot inversion.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 27623d3bad95..17357b16d1a7 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -190,8 +190,20 @@ struct scan_control {
 			prefetchw(&prev->_field);			\
 		}							\
 	} while (0)
+#define prefetchw_next_lru_folio(_folio, _base, _field)			\
+	do {								\
+		if ((_folio)->lru.next != _base) {			\
+			struct folio *next;				\
+									\
+			next = list_entry((_folio)->lru.next,		\
+					struct folio, lru);		\
+			prefetchw(&next->_field);			\
+		}							\
+	} while (0)
+
 #else
 #define prefetchw_prev_lru_folio(_folio, _base, _field) do { } while (0)
+#define prefetchw_next_lru_folio(_folio, _base, _field) do { } while (0)
 #endif
 
 /*
@@ -3931,9 +3943,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
 		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
+		struct list_head *pos = head->next;
 
-		while (!list_empty(head)) {
-			struct folio *folio = lru_to_folio(head);
+		while (pos != head) {
+			struct folio *folio = list_entry(pos, struct folio, lru);
 			long nr_pages = folio_nr_pages(folio);
 			int refs = folio_lru_refs(folio);
 			bool workingset = folio_test_workingset(folio);
@@ -3944,6 +3957,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
+			prefetchw_next_lru_folio(folio, head, flags);
+			pos = pos->next;
 			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			if (gen_increased) {
 				delta += nr_pages;
-- 
2.34.1

[RFC PATCH v4 10/16] mm/mglru: batch move folios to the second-oldest gen's LRU

[Barry Song (Xiaomi)]

Detect folios that need to move from the oldest generation to
the second-oldest generation, and batch-move them together.
This can significantly reduce the sys time of inc_min_seq(),
especially when the other type is significantly behind the
preferred type.

Assisted-by: gemini:gemini-3.6-flash
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 17357b16d1a7..b6c17ece3b3f 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3922,6 +3922,19 @@ static void clear_mm_walk(void)
 		kfree(walk);
 }
 
+static inline void flush_lru_batch(struct list_head *head, struct list_head **batch_end,
+				   struct list_head *dst)
+{
+	LIST_HEAD(movable);
+
+	if (!*batch_end)
+		return;
+
+	list_cut_position(&movable, head, *batch_end);
+	list_splice_tail_init(&movable, dst);
+	*batch_end = NULL;
+}
+
 static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 {
 	int zone;
@@ -3941,9 +3954,11 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
+		struct list_head *target_list = &lrugen->folios[target_gen][type][zone];
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
 		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
 		struct list_head *pos = head->next;
+		struct list_head *batch_end = NULL;
 
 		while (pos != head) {
 			struct folio *folio = list_entry(pos, struct folio, lru);
@@ -3962,7 +3977,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			if (gen_increased) {
 				delta += nr_pages;
-				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+				batch_end = &folio->lru;
+
 				/* don't count the workingset being lazily promoted */
 				if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 					int tier = lru_tier_from_refs(refs, workingset);
@@ -3970,11 +3986,14 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 					protected[tier] += nr_pages;
 				}
 			} else {
+				flush_lru_batch(head, &batch_end, target_list);
 				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
 			}
 			if (!--remaining)
 				break;
 		}
+		flush_lru_batch(head, &batch_end, target_list);
+
 		WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
 			   lrugen->nr_pages[old_gen][type][zone] - delta);
 		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
-- 
2.34.1

[RFC PATCH v4 11/16] mm/mglru: skip gentle reclaim at DEF_PRIORITY for extreme swappiness

[Barry Song (Xiaomi)]

For extreme swappiness (<= MIN_SWAPPINESS + 1 or >= MAX_SWAPPINESS),
skip gentle reclaim.
Falling back too easily, even at DEF_PRIORITY, undermines the effect
of extreme swappiness.

This matches the logic in get_type_to_scan(), where we have:
        if (swappiness <= MIN_SWAPPINESS + 1)
                return LRU_GEN_FILE;

        if (swappiness >= MAX_SWAPPINESS)
                return LRU_GEN_ANON;

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index b6c17ece3b3f..48a068b50678 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -281,6 +281,13 @@ static inline bool is_exec_file_folio(const struct folio *folio,
 	return vma_flags_test(vma_flags, VMA_EXEC_BIT) && folio_is_file_lru(folio);
 }
 
+/* See get_type_to_scan(): these values always select FILE or ANON */
+static inline bool is_extreme_swappiness(int swappiness)
+{
+	return swappiness <= MIN_SWAPPINESS + 1 ||
+	       swappiness >= MAX_SWAPPINESS;
+}
+
 static void set_task_reclaim_state(struct task_struct *task,
 				   struct reclaim_state *rs)
 {
@@ -5094,8 +5101,11 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,
 	if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq)
 		return true;
 
-	/* try to avoid aging, do gentle reclaim at the default priority */
-	if (sc->priority == DEF_PRIORITY)
+	/*
+	 * Try to avoid aging by doing gentle reclaim at the default
+	 * priority. Skip gentle reclaim for extreme swappiness.
+	 */
+	if (sc->priority == DEF_PRIORITY && !is_extreme_swappiness(swappiness))
 		return false;
 
 	/* better to run aging even though eviction is still possible */
-- 
2.34.1

[RFC PATCH v4 12/16] mm/mglru: run aging if the preferred type has no reclaimable gens

[Barry Song (Xiaomi)]

Respect the type selected by get_type_to_scan(). If there are no
reclaimable gens left for that type, run aging.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 48a068b50678..50f0bcf9576c 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -5095,12 +5095,17 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,
 			     struct scan_control *sc, int swappiness)
 {
+	int type = get_type_to_scan(lruvec, swappiness);
 	DEFINE_MIN_SEQ(lruvec);
 
 	/* have to run aging, since eviction is not possible anymore */
 	if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq)
 		return true;
 
+	/* run aging if the preferred type is exhausted */
+	if (min_seq[type] + MIN_NR_GENS > max_seq)
+		return true;
+
 	/*
 	 * Try to avoid aging by doing gentle reclaim at the default
 	 * priority. Skip gentle reclaim for extreme swappiness.
-- 
2.34.1

[RFC PATCH v4 13/16] mm/mglru: remove redundant gens <= MIN_NR_GENS check in should_run_aging()

[Barry Song (Xiaomi)]

From: Bo Zhang <zhangbo56@xiaomi.com>

If the following condition is true,

        if (min_seq[type] + MIN_NR_GENS > max_seq)
                return true;

then this one must also be true,

        if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq)
                return true;

The converse is not true, so the evictable_min_seq() check is
redundant and can be removed.

Signed-off-by: Bo Zhang <zhangbo56@xiaomi.com>
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 50f0bcf9576c..a276560bc66b 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -5098,10 +5098,6 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,
 	int type = get_type_to_scan(lruvec, swappiness);
 	DEFINE_MIN_SEQ(lruvec);
 
-	/* have to run aging, since eviction is not possible anymore */
-	if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq)
-		return true;
-
 	/* run aging if the preferred type is exhausted */
 	if (min_seq[type] + MIN_NR_GENS > max_seq)
 		return true;
-- 
2.34.1

[RFC PATCH v4 14/16] mm/mglru: run aging when pages are severely imbalanced across gens

[Barry Song (Xiaomi)]

From: lyugaofei <lyugaofei@xiaomi.com>

This partially restores the reclaim behavior introduced in Yu
Zhao's initial MGLRU commit, ac35a4902370 ("mm: multi-gen LRU:
minimal implementation"):
	/*
	 * It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1)
	 * of the total number of pages for each generation. A reasonable range
	 * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The
	 * aging cares about the upper bound of hot pages, while the eviction
	 * cares about the lower bound of cold pages.
	 */
	if (young * MIN_NR_GENS > total)
		return true;
	if (old * (MIN_NR_GENS + 2) < total)
		return true;

But with a stricter condition: the younger generations must
contain at least MAX_NR_GENS times as many folios as the older
generations.
We also consider the cost of inc_min_seq(). If the oldest generation
of the other type has fallen significantly behind, pulling those
folios from the oldest generation to the second oldest generation
can be very expensive. In this case, skip imbalance aging unless
extreme swappiness is in use.

Signed-off-by: lyugaofei <lyugaofei@xiaomi.com>
Co-developed-by: Barry Song (Xiaomi) <baohua@kernel.org>
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 50 insertions(+), 7 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index a276560bc66b..d6fac5b91ac1 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4219,20 +4219,28 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control
 	sc->priority = clamp(priority, DEF_PRIORITY / 2, DEF_PRIORITY);
 }
 
+static inline unsigned long lruvec_gen_size(struct lru_gen_folio *lrugen,
+		int type, unsigned long seq)
+{
+	int gen = lru_gen_from_seq(seq);
+	unsigned long size = 0;
+
+	for (int zone = 0; zone < MAX_NR_ZONES; zone++)
+		size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
+	return size;
+}
+
 static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness)
 {
-	int gen, type, zone;
+	int type;
 	unsigned long seq, total = 0;
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
 	DEFINE_MAX_SEQ(lruvec);
 	DEFINE_MIN_SEQ(lruvec);
 
 	for_each_evictable_type(type, swappiness) {
-		for (seq = min_seq[type]; seq <= max_seq; seq++) {
-			gen = lru_gen_from_seq(seq);
-			for (zone = 0; zone < MAX_NR_ZONES; zone++)
-				total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
-		}
+		for (seq = min_seq[type]; seq <= max_seq; seq++)
+			total += lruvec_gen_size(lrugen, type, seq);
 	}
 
 	return total;
@@ -5092,6 +5100,37 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	return scanned;
 }
 
+static bool lru_gen_imbalanced(struct lruvec *lruvec, unsigned long max_seq,
+		struct scan_control *sc, int type, int swappiness)
+{
+	struct lru_gen_folio *lrugen = &lruvec->lrugen;
+	unsigned long young = 0, old = 0, lag = 0;
+	DEFINE_MIN_SEQ(lruvec);
+
+	/* we still have enough generations to reclaim */
+	if (min_seq[type] + MIN_NR_GENS < max_seq)
+		return false;
+
+	/*
+	 * Trigger aging if the preferred type is running low on reclaimable
+	 * folios, provided the generation lag of the other type remains small
+	 * enough that inc_min_seq() introduces negligible overhead
+	 */
+	for (unsigned long seq = min_seq[type]; seq <= max_seq; seq++) {
+		unsigned long size = lruvec_gen_size(lrugen, type, seq);
+
+		if (seq + MIN_NR_GENS > max_seq)
+			young += size;
+		else
+			old += size;
+	}
+	if (min_seq[!type] + MAX_NR_GENS == max_seq + 1)
+		lag += lruvec_gen_size(lrugen, !type, min_seq[!type]);
+
+	return young > old * MAX_NR_GENS && (lag < MAX_LRU_BATCH ||
+	       (is_extreme_swappiness(swappiness) && sc->priority > 2));
+}
+
 static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,
 			     struct scan_control *sc, int swappiness)
 {
@@ -5110,7 +5149,11 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,
 		return false;
 
 	/* better to run aging even though eviction is still possible */
-	return evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS == max_seq;
+	if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS == max_seq)
+		return true;
+
+	/* Run aging if the preferred type is severely imbalanced across gens */
+	return lru_gen_imbalanced(lruvec, max_seq, sc, type, swappiness);
 }
 
 static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc,
-- 
2.34.1

[RFC PATCH v4 15/16] mm/mglru: dynamically scale aging threshold in lru_gen_imbalanced()

[Barry Song (Xiaomi)]

Currently, lru_gen_imbalanced() uses a fixed threshold (MAX_NR_GENS)
to determine whether the young-to-old folio ratio warrants aging the
preferred LRU type.

This fixed threshold may not scale well with large memory systems.
Borrow the adaptive ratio calculation from inactive_is_low() by
scaling the threshold with the square root of the memory size in GB.

Suggested-by: Zicheng Wang <wangzicheng@honor.com>
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index d6fac5b91ac1..c0350a8b61d1 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -5105,6 +5105,7 @@ static bool lru_gen_imbalanced(struct lruvec *lruvec, unsigned long max_seq,
 {
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
 	unsigned long young = 0, old = 0, lag = 0;
+	unsigned long inactive_ratio, gb;
 	DEFINE_MIN_SEQ(lruvec);
 
 	/* we still have enough generations to reclaim */
@@ -5127,7 +5128,13 @@ static bool lru_gen_imbalanced(struct lruvec *lruvec, unsigned long max_seq,
 	if (min_seq[!type] + MAX_NR_GENS == max_seq + 1)
 		lag += lruvec_gen_size(lrugen, !type, min_seq[!type]);
 
-	return young > old * MAX_NR_GENS && (lag < MAX_LRU_BATCH ||
+	/*
+	 * Borrow the adaptive ratio from inactive_is_low(), and scale
+	 * it by sqrt(MAX_NR_GENS) to make aging less aggressive
+	 */
+	gb = (young + old) >> (30 - PAGE_SHIFT);
+	inactive_ratio = gb ? int_sqrt(10 * gb * MAX_NR_GENS) : MAX_NR_GENS;
+	return young > old * inactive_ratio && (lag < MAX_LRU_BATCH ||
 	       (is_extreme_swappiness(swappiness) && sc->priority > 2));
 }
 
-- 
2.34.1

[RFC PATCH v4 16/16] mm/mglru: reduce folios pulled from the oldest gen in inc_min_seq()

[Barry Song (Xiaomi)]

Pulling folios from the oldest generation to the second-oldest
generation with MAX_LRU_BATCH can take a long time, as the spinlock
is held throughout the operation and blocks other users.

Reduce the remaining batch size significantly. This reduces the lock
hold time, allowing concurrent reclaim, lru_cache_drain, and other
operations to make progress.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index c0350a8b61d1..8e6fcefe0353 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3945,7 +3945,7 @@ static inline void flush_lru_batch(struct list_head *head, struct list_head **ba
 static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 {
 	int zone;
-	int remaining = MAX_LRU_BATCH;
+	int remaining = MAX_LRU_BATCH / (is_extreme_swappiness(swappiness) ? 2 : 8);
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
 	int hist = lru_hist_from_seq(lrugen->min_seq[type]);
 	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
-- 
2.34.1