[RFC PATCH 0/4] mm: mglru: fix swappiness behavior

[RFC PATCH 0/4] mm: mglru: fix swappiness behavior

[Barry Song (Xiaomi)]

The active/inactive LRU respects swappiness well. Anonymous page
scanning and reclamation increase roughly linearly with
swappiness, while file page scanning and reclamation decrease
accordingly.
For example, when swappiness reaches 200, both pgsteal_file and
pgscan_file drop to zero while building the kernel in a 1 GB
memcg. In contrast, MGLRU shows almost no change across different
swappiness values.

                   pgsteal_file

Swappiness    LRU         MGLRU
--------------------------------
1             10567455      763612
36              990706      480205
71              688170      415848
106             446294      386164
141             286307      359196
176             201733      351686
200                  0      330093


                   pgsteal_anon

Swappiness    LRU         MGLRU
--------------------------------
1              4410548     2726362
36             2465268     2762859
71             2677908     2885124
106            2737227     2841796
141            2984276     3035015
176            3381338     2938302
200           13116359     3113499


                   pgscan_file

Swappiness    LRU         MGLRU
--------------------------------
1             17997223      923094
36             1325674      539571
71              852345      464222
106             538207      464477
141             357253      412277
176             217536      399446
200                  0      375902


                   pgscan_anon

Swappiness    LRU         MGLRU
--------------------------------
1             31639423     5987136
36            23441521     5753224
71            26067110     6101780
106           25619448     5782919
141           26842088     6234264
176           29200021     5980292
200           62193924     6413125

This patchset respects the type selected by positive_ctrl_err(),
which uses swappiness as its gain. It does so by:

1. Avoiding premature fallback to the other type. Only fall back
   when reclaim is running at high priority.

2. Running aging when the preferred type has few or no
   reclaimable folios, so more folios of that type become
   reclaimable.

With this patchset, swappiness starts to behave similarly to the
active/inactive LRU.

                         pgsteal_file

Swappiness      LRU        MGLRU      MGLRU+Patch
-------------------------------------------------
1             10567455      763612        3507619
36              990706      480205         528781
71              688170      415848         478478
106             446294      386164         411632
141             286307      359196         375137
176             201733      351686         298586
200                  0      330093              0


                         pgsteal_anon

Swappiness      LRU        MGLRU      MGLRU+Patch
-------------------------------------------------
1              4410548     2726362        2876888
36             2465268     2762859        3022062
71             2677908     2885124        3129555
106            2737227     2841796        2974192
141            2984276     3035015        3092280
176            3381338     2938302        3353395
200           13116359     3113499       14348518


                         pgscan_file

Swappiness      LRU        MGLRU      MGLRU+Patch
-------------------------------------------------
1             17997223      923094        5842416
36             1325674      539571         648111
71              852345      464222         569930
106             538207      464477         480767
141             357253      412277         431911
176             217536      399446         337768
200                  0      375902              0


                         pgscan_anon

Swappiness      LRU        MGLRU      MGLRU+Patch
-------------------------------------------------
1             31639423     5987136        4658593
36            23441521     5753224        5444726
71            26067110     6101780        5755804
106           25619448     5782919        5872415
141           26842088     6234264        6144096
176           29200021     5980292        6615500
200           62193924     6413125       22984986

Another possible approach is to decouple anonymous and file-backed
aging by maintaining separate max_seq values for each type. This
allows anonymous and file-backed memory to age and be reclaimed
independently, enabling the swappiness-preferred type to be reclaimed
more aggressively while allowing the other type to lag behind. This
approach has already been adopted by projects such as CachyOS [1] and
Chromium [2].

However, this approach requires substantial changes to MGLRU and
fundamentally alters its design by breaking the shared aging timeline
between anonymous and file-backed memory. This timeline is the
foundation for mechanisms such as the PID controller and the
min_ttl_ms thrashing protection.

That is why this patchset aims to fix the swappiness behavior
without fundamentally changing MGLRU's design, with minimal changes.

[1] https://github.com/firelzrd/re-swappiness
[2] https://chromium.googlesource.com/chromiumos/third_party/kernel/+log/929932351492d01f0aee37a0ac3be8c7bd88f80d

Barry Song (Xiaomi) (3):
  mm: mglru: only fall back when reclaim is running at high priority
  mm: mglru: do try_to_inc_min_seq if scanned==0
  mm: mglru: run aging if the preferred type has no folios in
    reclaimable gens

lyugaofei (1):
  mm: mglru: run aging when pages are severely imbalanced across gens

 mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 43 insertions(+), 3 deletions(-)

-- 
2.39.3 (Apple Git-146)

[RFC PATCH 1/4] mm: mglru: only fall back when reclaim is running at high priority

[Barry Song (Xiaomi)]

Respect the type selected by positive_ctrl_err(), where swappiness
controls the relative weights of SP and PV. Falling back too readily
violates that bias.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 566c4e837c7d..f79402760776 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4855,9 +4855,12 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 		 * If scanned > 0 and isolated == 0, avoid falling back to the
 		 * other type, as this type remains sufficient. Falling back
 		 * too readily can disrupt the positive_ctrl_err() bias.
+		 * Only fall back when reclaim is running at high priority.
 		 */
-		if (!scanned)
-			type = !type;
+		if (!scanned) {
+			if (!sc->priority)
+				type = !type;
+		}
 	}
 
 	return total_scanned;
-- 
2.39.3 (Apple Git-146)

[RFC PATCH 2/4] mm: mglru: do try_to_inc_min_seq if scanned==0

[Barry Song (Xiaomi)]

If scanned == 0, we are likely running out of folios in the
reclaimable gens. Advance min_seq to keep it in sync with the
current reclaimable state.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index f79402760776..4f3a375de86e 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4858,6 +4858,7 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 		 * Only fall back when reclaim is running at high priority.
 		 */
 		if (!scanned) {
+			try_to_inc_min_seq(lruvec, swappiness);
 			if (!sc->priority)
 				type = !type;
 		}
-- 
2.39.3 (Apple Git-146)

[RFC PATCH 3/4] mm: mglru: run aging when pages are severely imbalanced across gens

[Barry Song (Xiaomi)]

From: lyugaofei <lyugaofei@xiaomi.com>

This partially restores the reclaim behavior introduced in Yu
Zhao's initial MGLRU commit, ac35a4902370 ("mm: multi-gen LRU:
minimal implementation"):
	/*
	 * It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1)
	 * of the total number of pages for each generation. A reasonable range
	 * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The
	 * aging cares about the upper bound of hot pages, while the eviction
	 * cares about the lower bound of cold pages.
	 */
	if (young * MIN_NR_GENS > total)
		return true;
	if (old * (MIN_NR_GENS + 2) < total)
		return true;

But with a stricter condition: the youngest gen must have at
least four times as many folios as the oldest. This helps keep
folios of the preferred type distributed across the reclaimable
gens.

Signed-off-by: lyugaofei <lyugaofei@xiaomi.com>
Co-developed-by: Barry Song (Xiaomi) <baohua@kernel.org>
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 34 +++++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 4f3a375de86e..9aac4febc206 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4963,9 +4963,37 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	return scanned;
 }
 
+static bool lru_gen_imbalanced(struct lruvec *lruvec, int type,
+		unsigned long max_seq, unsigned long min_seq,
+		int swappiness)
+{
+	struct lru_gen_folio *lrugen = &lruvec->lrugen;
+	unsigned long young = 0, old = 0, seq;
+
+	if (swappiness == MIN_SWAPPINESS || swappiness > MAX_SWAPPINESS)
+		return false;
+
+	for (seq = min_seq; seq <= max_seq; seq++) {
+		int gen = lru_gen_from_seq(seq);
+		unsigned long size = 0;
+		int zone;
+
+		gen = lru_gen_from_seq(seq);
+		for (zone = 0; zone < MAX_NR_ZONES; zone++)
+			size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
+
+		if (seq + MIN_NR_GENS > max_seq)
+			young += size;
+		else
+			old += size;
+	}
+	return young > old * 4;
+}
+
 static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,
 			     struct scan_control *sc, int swappiness)
 {
+	int type = get_type_to_scan(lruvec, swappiness);
 	DEFINE_MIN_SEQ(lruvec);
 
 	/* have to run aging, since eviction is not possible anymore */
@@ -4977,7 +5005,11 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,
 		return false;
 
 	/* better to run aging even though eviction is still possible */
-	return evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS == max_seq;
+	if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS == max_seq)
+		return true;
+
+	/* Run aging if the preferred type is severely imbalanced across gens */
+	return lru_gen_imbalanced(lruvec, type, max_seq, min_seq[type], swappiness);
 }
 
 static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc,
-- 
2.39.3 (Apple Git-146)

[RFC PATCH 4/4] mm: mglru: run aging if the preferred type has no folios in reclaimable gens

[Barry Song (Xiaomi)]

Respect the type selected by positive_ctrl_err(). If there are no
reclaimable gens left for that type, run aging.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 9aac4febc206..be34b8b946ad 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -5000,6 +5000,10 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,
 	if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq)
 		return true;
 
+	/* run aging if the preferred type is exhausted */
+	if (min_seq[type] + MIN_NR_GENS > max_seq)
+		return true;
+
 	/* try to avoid aging, do gentle reclaim at the default priority */
 	if (sc->priority == DEF_PRIORITY)
 		return false;
-- 
2.39.3 (Apple Git-146)