[PATCH 0/6] mm/mglru: speed up inc_min_seq() and fix cold/hot inversions

[PATCH 0/6] mm/mglru: speed up inc_min_seq() and fix cold/hot inversions

[Barry Song (Xiaomi)]

This is an aging speedup series split out from the MGLRU swappiness
series [1], with the inc_min_seq changes separated to make them
easier to review.

Currently, inc_min_seq performance is crucial to both the swappiness
fix and proactive aging. There are two problems with it:

1. It processes each folio one by one, while many operations can be
batched or skipped. For example, a batch of folios can be moved together
from the oldest generation to the second-oldest generation, and the
associated counting can also be done in batches.

2. It may cause potential cold/hot inversion by placing promoted folios
(which have been scanned and found to have young PTEs) behind
non-promoted folios. A similar inversion can also occur among
non-promoted folios, as tail folios from the oldest generation are
placed before head folios when moving them to the second-oldest
generation.

This series tries to batch operations as much as possible and fix the
potential cold/hot inversion by keeping promoted folios ahead of
non-promoted folios, while also preserving the order of non-promoted
folios when moving them from the oldest generation to the second-oldest
generation.

Minor issue: inc_min_seq() also counts protected folios
improperly, as promoted folios should be skipped, as in
sort_folio().

We need a stable workload with a stable number of folios to measure
aging and evaluate the speedup in inc_min_seq(). So I asked ChatGPT
to generate the microbenchmark below. It ages an LRU vec containing
512 MB of memory 100 times:

 #define _GNU_SOURCE
 
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 #include <stdint.h>
 #include <unistd.h>
 #include <fcntl.h>
 #include <errno.h>
 #include <time.h>
 #include <sys/mman.h>
 
 #define SIZE		(512UL * 1024 * 1024)
 #define LRU_GEN		"/sys/kernel/debug/lru_gen"
 #define TARGET_CGROUP	"/system.slice/agetest.scope"
 #define START_GEN	3
 #define END_GEN	103
 
 static long long nsec_diff(const struct timespec *start,
 			   const struct timespec *end)
 {
 	return (end->tv_sec - start->tv_sec) * 1000000000LL +
 	       (end->tv_nsec - start->tv_nsec);
 }
 
 static int find_memcg_id(void)
 {
 	FILE *fp;
 	char line[4096];
 	int memcg_id;
 
 	fp = fopen(LRU_GEN, "r");
 	if (!fp) {
 		perror("fopen lru_gen");
 		return -1;
 	}
 
 	while (fgets(line, sizeof(line), fp)) {
 		char *p;
 
 		if (strncmp(line, "memcg ", 6))
 			continue;
 
 		p = line + 6;
 
 		if (sscanf(p, "%d", &memcg_id) != 1)
 			continue;
 
 		/*
 		 * The memcg path follows the numeric ID.
 		 */
 		p = strchr(p, ' ');
 		if (!p)
 			continue;
 
 		if (strstr(p, TARGET_CGROUP)) {
 			fclose(fp);
 			return memcg_id;
 		}
 	}
 
 	fclose(fp);
 
 	fprintf(stderr, "Cannot find %s\n", TARGET_CGROUP);
 	return -1;
 }
 
 int main(void)
 {
 	void *addr;
 	int memcg_id;
 	int fd;
 	long long total_ns = 0;
 
 	/*
 	 * mmap 512 MB and touch every page.
 	 */
 	addr = mmap(NULL, SIZE, PROT_READ | PROT_WRITE,
 		    MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
 	if (addr == MAP_FAILED) {
 		perror("mmap");
 		return 1;
 	}
 
 	memset(addr, 0x55, SIZE);
 
 	printf("mmap: %p, size: %lu MB\n",
 	       addr, SIZE / 1024 / 1024);
 
 	/*
 	 * Find the memcg ID automatically.
 	 */
 	memcg_id = find_memcg_id();
 	if (memcg_id < 0)
 		return 1;
 
 	printf("memcg: %d (%s)\n", memcg_id, TARGET_CGROUP);
 	printf("aging generation %d -> %d\n",
 	       START_GEN, END_GEN);
 
 	fd = open(LRU_GEN, O_WRONLY);
 	if (fd < 0) {
 		perror("open lru_gen");
 		return 1;
 	}
 
 	for (int gen = START_GEN; gen <= END_GEN; gen++) {
 		char buf[128];
 		int len;
 		struct timespec start, end;
 		long long ns;
 
 		len = snprintf(buf, sizeof(buf),
 			       "+ %d 0 %d\n", memcg_id, gen);
 
 		clock_gettime(CLOCK_MONOTONIC, &start);
 
 		if (write(fd, buf, len) != len) {
 			perror("write lru_gen");
 			close(fd);
 			return 1;
 		}
 
 		clock_gettime(CLOCK_MONOTONIC, &end);
 
 		ns = nsec_diff(&start, &end);
 		total_ns += ns;
 
 		printf("gen %3d: %8.3f ms\n",
 		       gen, ns / 1000000.0);
 		fflush(stdout);
 	}
 
 	close(fd);
 
 	printf("\nTotal:   %.3f ms\n",
 	       total_ns / 1000000.0);
 	printf("Average: %.3f ms\n",
 	       total_ns / (double)(END_GEN - START_GEN + 1) /
 	       1000000.0);
 
 	while (1)
 		sleep(1);
 
 	return 0;
 }

Run the above microbenchmark with:
systemd-run --scope --unit=agetest -p MemoryMax=1024M ./agetest

I’m seeing inc_min_seq() become significantly faster:

W/o patch:

Running scope as unit: agetest.scope
mmap: 0x72c1b5a00000, size: 512 MB
memcg: 12673 (/system.slice/agetest.scope)
aging generation 3 -> 103
gen   3:    7.433 ms
gen   4:    0.949 ms
gen   5:    2.535 ms
gen   6:    5.043 ms
gen   7:    5.041 ms
gen   8:    5.027 ms
...
gen 100:    5.035 ms
gen 101:    5.011 ms
gen 102:    5.029 ms
gen 103:    5.056 ms

Total:   503.946 ms
Average: 4.990 ms

W/ patch:

Running scope as unit: agetest.scope
mmap: 0x7c4b1d200000, size: 512 MB
memcg: 12893 (/system.slice/agetest.scope)
aging generation 3 -> 103
gen   3:    7.538 ms
gen   4:    0.937 ms
gen   5:    2.348 ms
gen   6:    2.300 ms
gen   7:    2.302 ms
gen   8:    2.294 ms
gen   9:    2.296 ms
...
gen 100:    2.292 ms
gen 101:    2.307 ms
gen 102:    2.293 ms
gen 103:    2.293 ms

Total:   235.718 ms
Average: 2.334 ms

The average aging time drops from 4.990 ms to 2.334 ms!

[1] https://lore.kernel.org/linux-mm/20260812121658.69965-1-baohua@kernel.org/

Barry Song (Xiaomi) (6):
  mm/mglru: batch update lrugen->nr_pages in inc_min_seq()
  mm/mglru: batch update lrugen->protected in inc_min_seq()
  mm/mglru: enhance cold/hot inversion handling in inc_min_seq()
  mm/mglru: exclude folios promoted by aging from protected in
    inc_min_seq()
  mm/mglru: move folios from oldest gen to second-oldest gen from head
    to tail
  mm/mglru: batch move folios to the second-oldest gen's LRU

 mm/vmscan.c | 98 +++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 80 insertions(+), 18 deletions(-)

-- 
2.34.1

[PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

[Barry Song (Xiaomi)]

Currently, folio_inc_gen() updates lrugen->nr_pages for every folio
as it advances generations. Instead, accumulate the size changes
and update lrugen->nr_pages in a batch after scanning the entire
oldest generation, or when the scan stops because remaining reaches
zero.

Since we only move folios from the oldest generation to the second
oldest generation, the active/inactive state cannot change. We can
therefore skip __lru_update_size().

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++-----------
 1 file changed, 35 insertions(+), 11 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index c1404a59523d..0d74fc00abd3 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3296,20 +3296,21 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma
 }
 
 /* protect pages accessed multiple times through file descriptors */
-static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
+static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
 {
-	int type = folio_is_file_lru(folio);
-	struct lru_gen_folio *lrugen = &lruvec->lrugen;
-	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
 	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
+	int new_gen;
 
 	VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio);
 
 	do {
 		new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
 		/* folio_update_gen() has promoted this page? */
-		if (new_gen >= 0 && new_gen != old_gen)
+		if (new_gen >= 0 && new_gen != old_gen) {
+			if (increased)
+				*increased = false;
 			return new_gen;
+		}
 
 		new_gen = (old_gen + 1) % MAX_NR_GENS;
 
@@ -3317,8 +3318,21 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
 		new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF;
 	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
 
-	lru_gen_update_size(lruvec, folio, old_gen, new_gen);
+	if (increased)
+		*increased = true;
+	return new_gen;
+}
 
+static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
+{
+	int type = folio_is_file_lru(folio);
+	struct lru_gen_folio *lrugen = &lruvec->lrugen;
+	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
+	bool gen_increased;
+
+	new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
+	if (gen_increased)
+		lru_gen_update_size(lruvec, folio, old_gen, new_gen);
 	return new_gen;
 }
 
@@ -3904,6 +3918,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
 	int hist = lru_hist_from_seq(lrugen->min_seq[type]);
 	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
+	int target_gen = (old_gen + 1) % MAX_NR_GENS;
 
 	/* For file type, skip the check if swappiness is anon only */
 	if (type && (swappiness == SWAPPINESS_ANON_ONLY))
@@ -3916,32 +3931,41 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
+		unsigned long delta = 0;
 
 		while (!list_empty(head)) {
 			struct folio *folio = lru_to_folio(head);
+			long nr_pages = folio_nr_pages(folio);
 			int refs = folio_lru_refs(folio);
 			bool workingset = folio_test_workingset(folio);
+			bool gen_increased;
 
 			VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
-			new_gen = folio_inc_gen(lruvec, folio);
+			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
-
+			if (gen_increased)
+				delta += nr_pages;
 			/* don't count the workingset being lazily promoted */
 			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 				int tier = lru_tier_from_refs(refs, workingset);
-				int delta = folio_nr_pages(folio);
 
 				WRITE_ONCE(lrugen->protected[hist][type][tier],
-					   lrugen->protected[hist][type][tier] + delta);
+					   lrugen->protected[hist][type][tier] + nr_pages);
 			}
 
 			if (!--remaining)
-				return false;
+				break;
 		}
+		WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
+			   lrugen->nr_pages[old_gen][type][zone] - delta);
+		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
+			   lrugen->nr_pages[target_gen][type][zone] + delta);
+		if (!remaining)
+			return false;
 	}
 done:
 	reset_ctrl_pos(lruvec, type, true);
-- 
2.34.1

[PATCH 2/6] mm/mglru: batch update lrugen->protected in inc_min_seq()

[Barry Song (Xiaomi)]

Avoid updating lrugen->protected with WRITE_ONCE() for each folio,
which may prevent potential compiler optimizations. Accumulate the
updates locally and apply them in a batch instead.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 0d74fc00abd3..99ee3c833d54 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3931,7 +3931,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
-		unsigned long delta = 0;
+		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
 
 		while (!list_empty(head)) {
 			struct folio *folio = lru_to_folio(head);
@@ -3953,8 +3953,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 				int tier = lru_tier_from_refs(refs, workingset);
 
-				WRITE_ONCE(lrugen->protected[hist][type][tier],
-					   lrugen->protected[hist][type][tier] + nr_pages);
+				protected[tier] += nr_pages;
 			}
 
 			if (!--remaining)
@@ -3964,6 +3963,9 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			   lrugen->nr_pages[old_gen][type][zone] - delta);
 		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
 			   lrugen->nr_pages[target_gen][type][zone] + delta);
+		for (int tier = 0; tier < MAX_NR_TIERS; tier++)
+			WRITE_ONCE(lrugen->protected[hist][type][tier],
+				   lrugen->protected[hist][type][tier] + protected[tier]);
 		if (!remaining)
 			return false;
 	}
-- 
2.34.1

[PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

[Barry Song (Xiaomi)]

During aging, a folio's generation may already have been updated by
folio_update_gen(), even though it has not yet been moved to the
corresponding generation list. Such folios are hotter than those
already in that generation.

It makes sense for inc_min_seq() to increment the generation of
folios that were never promoted during aging and move them to the
tail of the new oldest generation. However, folios that were already
promoted should instead be moved to the head of their updated
generation, just as sort_folio() does in scan_folios().

Otherwise, promoted folios could end up behind folios that were
never promoted, effectively inverting their hot/cold ordering.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 99ee3c833d54..3b618a51cde2 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3946,9 +3946,12 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
 			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
-			list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
-			if (gen_increased)
+			if (gen_increased) {
 				delta += nr_pages;
+				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+			} else {
+				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+			}
 			/* don't count the workingset being lazily promoted */
 			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 				int tier = lru_tier_from_refs(refs, workingset);
-- 
2.34.1

[PATCH 4/6] mm/mglru: exclude folios promoted by aging from protected in inc_min_seq()

[Barry Song (Xiaomi)]

Some folios may have been promoted during aging, so don't count them
as protected, similar to sort_folio().

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 3b618a51cde2..7bd01875fade 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3949,16 +3949,15 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			if (gen_increased) {
 				delta += nr_pages;
 				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+				/* don't count the workingset being lazily promoted */
+				if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
+					int tier = lru_tier_from_refs(refs, workingset);
+
+					protected[tier] += nr_pages;
+				}
 			} else {
 				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
 			}
-			/* don't count the workingset being lazily promoted */
-			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
-				int tier = lru_tier_from_refs(refs, workingset);
-
-				protected[tier] += nr_pages;
-			}
-
 			if (!--remaining)
 				break;
 		}
-- 
2.34.1

[PATCH 5/6] mm/mglru: move folios from oldest gen to second-oldest gen from head to tail

[Barry Song (Xiaomi)]

For reclamation, it makes sense to reclaim folios from tail to
head, as folios near the head are relatively hot. However, when
moving folios from the oldest generation to the second-oldest
generation, using the tail-to-head order would effectively cause
a cold/hot inversion.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 7bd01875fade..2fd82b2ca4d1 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -191,8 +191,20 @@ struct scan_control {
 			prefetchw(&prev->_field);			\
 		}							\
 	} while (0)
+#define prefetchw_next_lru_folio(_folio, _base, _field)			\
+	do {								\
+		if ((_folio)->lru.next != _base) {			\
+			struct folio *next;				\
+									\
+			next = list_entry((_folio)->lru.next,		\
+					struct folio, lru);		\
+			prefetchw(&next->_field);			\
+		}							\
+	} while (0)
+
 #else
 #define prefetchw_prev_lru_folio(_folio, _base, _field) do { } while (0)
+#define prefetchw_next_lru_folio(_folio, _base, _field) do { } while (0)
 #endif
 
 /*
@@ -3932,9 +3944,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
 		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
+		struct list_head *pos = head->next;
 
-		while (!list_empty(head)) {
-			struct folio *folio = lru_to_folio(head);
+		while (pos != head) {
+			struct folio *folio = list_entry(pos, struct folio, lru);
 			long nr_pages = folio_nr_pages(folio);
 			int refs = folio_lru_refs(folio);
 			bool workingset = folio_test_workingset(folio);
@@ -3945,6 +3958,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
+			prefetchw_next_lru_folio(folio, head, flags);
+			pos = pos->next;
 			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			if (gen_increased) {
 				delta += nr_pages;
-- 
2.34.1

[PATCH 6/6] mm/mglru: batch move folios to the second-oldest gen's LRU

[Barry Song (Xiaomi)]

Detect folios that need to move from the oldest generation to
the second-oldest generation, and batch-move them together.
This can significantly reduce the sys time of inc_min_seq(),
especially when the other type is significantly behind the
preferred type.

Assisted-by: gemini:gemini-3.6-flash
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 2fd82b2ca4d1..996b48344ed0 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3923,6 +3923,19 @@ static void clear_mm_walk(void)
 		kfree(walk);
 }
 
+static inline void flush_lru_batch(struct list_head *head, struct list_head **batch_end,
+				   struct list_head *dst)
+{
+	LIST_HEAD(movable);
+
+	if (!*batch_end)
+		return;
+
+	list_cut_position(&movable, head, *batch_end);
+	list_splice_tail_init(&movable, dst);
+	*batch_end = NULL;
+}
+
 static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 {
 	int zone;
@@ -3942,9 +3955,11 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
+		struct list_head *target_list = &lrugen->folios[target_gen][type][zone];
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
 		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
 		struct list_head *pos = head->next;
+		struct list_head *batch_end = NULL;
 
 		while (pos != head) {
 			struct folio *folio = list_entry(pos, struct folio, lru);
@@ -3963,7 +3978,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			if (gen_increased) {
 				delta += nr_pages;
-				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+				batch_end = &folio->lru;
+
 				/* don't count the workingset being lazily promoted */
 				if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 					int tier = lru_tier_from_refs(refs, workingset);
@@ -3971,11 +3987,14 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 					protected[tier] += nr_pages;
 				}
 			} else {
+				flush_lru_batch(head, &batch_end, target_list);
 				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
 			}
 			if (!--remaining)
 				break;
 		}
+		flush_lru_batch(head, &batch_end, target_list);
+
 		WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
 			   lrugen->nr_pages[old_gen][type][zone] - delta);
 		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
-- 
2.34.1

Re: [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

[Lian Wang (ProcessMission)]

From: Lian Wang <lianux.mm@gmail.com>

Hi Barry,

A small nit:

> +		unsigned long delta = 0;

lrugen->nr_pages is a signed long and is documented as being able to
transiently go negative while reset_batch_size() is pending. Could delta
remain a long as well, so this arithmetic stays signed?

In 2/6, where the declarations are combined, this could be:

	unsigned long protected[MAX_NR_TIERS] = {};
	long delta = 0;

Thanks,
Lian

Re: [PATCH 5/6] mm/mglru: move folios from oldest gen to second-oldest gen from head to tail

[Kairui Song]

On Fri, Aug 21, 2026 at 6:38 PM Barry Song (Xiaomi) <baohua@kernel.org> wrote:
>
> For reclamation, it makes sense to reclaim folios from tail to
> head, as folios near the head are relatively hot. However, when
> moving folios from the oldest generation to the second-oldest
> generation, using the tail-to-head order would effectively cause
> a cold/hot inversion.
>
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>

Hi Barry

This looks a really good idea, thanks!

> ---
>  mm/vmscan.c | 19 +++++++++++++++++--
>  1 file changed, 17 insertions(+), 2 deletions(-)
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 7bd01875fade..2fd82b2ca4d1 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -191,8 +191,20 @@ struct scan_control {
>                         prefetchw(&prev->_field);                       \
>                 }                                                       \
>         } while (0)
> +#define prefetchw_next_lru_folio(_folio, _base, _field)                        \
> +       do {                                                            \
> +               if ((_folio)->lru.next != _base) {                      \
> +                       struct folio *next;                             \
> +                                                                       \
> +                       next = list_entry((_folio)->lru.next,           \
> +                                       struct folio, lru);             \
> +                       prefetchw(&next->_field);                       \
> +               }                                                       \
> +       } while (0)
> +

I got following warning from checkpatch:

    ● checkpatch.pl: 92: WARNING: Argument '_folio' is not used in
function-like macro
    ● checkpatch.pl: 92: WARNING: Argument '_base' is not used in
function-like macro
    ● checkpatch.pl: 92: WARNING: Argument '_field' is not used in
function-like macro

Maybe you could try b4; it helps run these checks automatically. Feel
free to ignore if you think these warning at pointless.

>  #else
>  #define prefetchw_prev_lru_folio(_folio, _base, _field) do { } while (0)
> +#define prefetchw_next_lru_folio(_folio, _base, _field) do { } while (0)
>  #endif
>
>  /*
> @@ -3932,9 +3944,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>         for (zone = 0; zone < MAX_NR_ZONES; zone++) {
>                 struct list_head *head = &lrugen->folios[old_gen][type][zone];
>                 unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
> +               struct list_head *pos = head->next;
>
> -               while (!list_empty(head)) {
> -                       struct folio *folio = lru_to_folio(head);
> +               while (pos != head) {

Do we need to change the while condition now? Since this commit still
moves folios one by one, will it stop when the list is empty?

> +                       prefetchw_next_lru_folio(folio, head, flags);
> +                       pos = pos->next;

I tried prefetching in MGLRU previously, and it didn't look very good
but there is no regression either; perhaps it's very arch-dependent.
Anyway, I think we can keep it here, maybe further optimize the
prefetch later.

>                         new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
>                         if (gen_increased) {
>                                 delta += nr_pages;

The rest looks good to me, thanks!

Re: [PATCH 5/6] mm/mglru: move folios from oldest gen to second-oldest gen from head to tail

[Barry Song]

On Sat, Aug 22, 2026 at 1:45 PM Kairui Song <ryncsn@gmail.com> wrote:
>
> On Fri, Aug 21, 2026 at 6:38 PM Barry Song (Xiaomi) <baohua@kernel.org> wrote:
> >
> > For reclamation, it makes sense to reclaim folios from tail to
> > head, as folios near the head are relatively hot. However, when
> > moving folios from the oldest generation to the second-oldest
> > generation, using the tail-to-head order would effectively cause
> > a cold/hot inversion.
> >
> > Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
>
> Hi Barry
>
> This looks a really good idea, thanks!
>
> > ---
> >  mm/vmscan.c | 19 +++++++++++++++++--
> >  1 file changed, 17 insertions(+), 2 deletions(-)
> >
> > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > index 7bd01875fade..2fd82b2ca4d1 100644
> > --- a/mm/vmscan.c
> > +++ b/mm/vmscan.c
> > @@ -191,8 +191,20 @@ struct scan_control {
> >                         prefetchw(&prev->_field);                       \
> >                 }                                                       \
> >         } while (0)
> > +#define prefetchw_next_lru_folio(_folio, _base, _field)                        \
> > +       do {                                                            \
> > +               if ((_folio)->lru.next != _base) {                      \
> > +                       struct folio *next;                             \
> > +                                                                       \
> > +                       next = list_entry((_folio)->lru.next,           \
> > +                                       struct folio, lru);             \
> > +                       prefetchw(&next->_field);                       \
> > +               }                                                       \
> > +       } while (0)
> > +
>
> I got following warning from checkpatch:
>
>     ● checkpatch.pl: 92: WARNING: Argument '_folio' is not used in
> function-like macro
>     ● checkpatch.pl: 92: WARNING: Argument '_base' is not used in
> function-like macro
>     ● checkpatch.pl: 92: WARNING: Argument '_field' is not used in
> function-like macro
>
> Maybe you could try b4; it helps run these checks automatically. Feel
> free to ignore if you think these warning at pointless.

Yep. I vividly remember adding this rule to the coding style, and Xining
added the corresponding `checkpatch.pl` change:

commit 6813216bbdba1 ("Documentation: coding-style: ask function-like
macros to evaluate parameters")
commit  b1be5844c1a01 ("scripts: checkpatch: check unused parameters
for function-like macro")

But once we started writing the code, I was influenced by its context and
ended up following that context instead.

>
> >  #else
> >  #define prefetchw_prev_lru_folio(_folio, _base, _field) do { } while (0)
> > +#define prefetchw_next_lru_folio(_folio, _base, _field) do { } while (0)
> >  #endif
> >
> >  /*
> > @@ -3932,9 +3944,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
> >         for (zone = 0; zone < MAX_NR_ZONES; zone++) {
> >                 struct list_head *head = &lrugen->folios[old_gen][type][zone];
> >                 unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
> > +               struct list_head *pos = head->next;
> >
> > -               while (!list_empty(head)) {
> > -                       struct folio *folio = lru_to_folio(head);
> > +               while (pos != head) {
>
> Do we need to change the while condition now? Since this commit still
> moves folios one by one, will it stop when the list is empty?

For this patch, we don't need to change the `while` condition. That change
is more relevant to the next patch:
https://lore.kernel.org/linux-mm/20260821102538.22642-7-baohua@kernel.org/

So I guess I could move this while (pos != head) change to the next patch
if that makes the review easier.

>
> > +                       prefetchw_next_lru_folio(folio, head, flags);
> > +                       pos = pos->next;
>
> I tried prefetching in MGLRU previously, and it didn't look very good
> but there is no regression either; perhaps it's very arch-dependent.
> Anyway, I think we can keep it here, maybe further optimize the
> prefetch later.

Yep. I guess we may want to keep it to help architectures that are
sensitive to prefetching.

>
> >                         new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
> >                         if (gen_increased) {
> >                                 delta += nr_pages;
>
> The rest looks good to me, thanks!

Thanks for the review!

Best Regards
Barry

Re: [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

[Barry Song]

On Sat, Aug 22, 2026 at 9:42 AM Lian Wang (ProcessMission)
<lianux.mm@gmail.com> wrote:
>
> From: Lian Wang <lianux.mm@gmail.com>
>
> Hi Barry,
>
> A small nit:
>
> > +             unsigned long delta = 0;
>
> lrugen->nr_pages is a signed long and is documented as being able to
> transiently go negative while reset_batch_size() is pending. Could delta
> remain a long as well, so this arithmetic stays signed?

Thanks, Lian, for the review. Changing it from unsigned long to long
makes sense to me.

>
> In 2/6, where the declarations are combined, this could be:
>
>         unsigned long protected[MAX_NR_TIERS] = {};
>         long delta = 0;
>

Best Regards
Barry

Re: [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

[Baoquan He]

Hi Barry,

On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> Currently, folio_inc_gen() updates lrugen->nr_pages for every folio
> as it advances generations. Instead, accumulate the size changes
> and update lrugen->nr_pages in a batch after scanning the entire
> oldest generation, or when the scan stops because remaining reaches
> zero.

This patch refactor code to split out __folio_inc_gen() and introduce 
gen_increased, this is the base of later patches. You seem to only
mention the minor optimization of lrugen->nr_pages. IMHO, this
refactoring can be split out to an independent patch. The lrugen->nr_pages
can be integrated with other patch, e.g patch 2 or patch 6.

> 
> Since we only move folios from the oldest generation to the second
> oldest generation, the active/inactive state cannot change. We can
> therefore skip __lru_update_size().

For the justification of skipping __lru_update_size(), there's a
precondition: get_nr_gens(lruvec, type) == MAX_NR_GENS; With this, the
oldest gen and 2nd oldest gen are both inactive. Calling
__lru_update_size() may waste tiny cpu, while skipping it may cause
issue in future if the precondition is changed. Do you think adding
a VM_WARN_ON_ONCE is necessary?

VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) ||
		lru_gen_is_active(lruvec, target_gen));


Thanks
Baoquan
> 
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
>  mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++-----------
>  1 file changed, 35 insertions(+), 11 deletions(-)
> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index c1404a59523d..0d74fc00abd3 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3296,20 +3296,21 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma
>  }
>  
>  /* protect pages accessed multiple times through file descriptors */
> -static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
> +static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
>  {
> -	int type = folio_is_file_lru(folio);
> -	struct lru_gen_folio *lrugen = &lruvec->lrugen;
> -	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
>  	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
> +	int new_gen;
>  
>  	VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio);
>  
>  	do {
>  		new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
>  		/* folio_update_gen() has promoted this page? */
> -		if (new_gen >= 0 && new_gen != old_gen)
> +		if (new_gen >= 0 && new_gen != old_gen) {
> +			if (increased)
> +				*increased = false;
>  			return new_gen;
> +		}
>  
>  		new_gen = (old_gen + 1) % MAX_NR_GENS;
>  
> @@ -3317,8 +3318,21 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
>  		new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF;
>  	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
>  
> -	lru_gen_update_size(lruvec, folio, old_gen, new_gen);
> +	if (increased)
> +		*increased = true;
> +	return new_gen;
> +}
>  
> +static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
> +{
> +	int type = folio_is_file_lru(folio);
> +	struct lru_gen_folio *lrugen = &lruvec->lrugen;
> +	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
> +	bool gen_increased;
> +
> +	new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
> +	if (gen_increased)
> +		lru_gen_update_size(lruvec, folio, old_gen, new_gen);
>  	return new_gen;
>  }
>  
> @@ -3904,6 +3918,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  	struct lru_gen_folio *lrugen = &lruvec->lrugen;
>  	int hist = lru_hist_from_seq(lrugen->min_seq[type]);
>  	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
> +	int target_gen = (old_gen + 1) % MAX_NR_GENS;
>  
>  	/* For file type, skip the check if swappiness is anon only */
>  	if (type && (swappiness == SWAPPINESS_ANON_ONLY))
> @@ -3916,32 +3931,41 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  	/* prevent cold/hot inversion if the type is evictable */
>  	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
>  		struct list_head *head = &lrugen->folios[old_gen][type][zone];
> +		unsigned long delta = 0;
>  
>  		while (!list_empty(head)) {
>  			struct folio *folio = lru_to_folio(head);
> +			long nr_pages = folio_nr_pages(folio);
>  			int refs = folio_lru_refs(folio);
>  			bool workingset = folio_test_workingset(folio);
> +			bool gen_increased;
>  
>  			VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio);
>  			VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio);
>  			VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);
>  			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
>  
> -			new_gen = folio_inc_gen(lruvec, folio);
> +			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
>  			list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
> -
> +			if (gen_increased)
> +				delta += nr_pages;
>  			/* don't count the workingset being lazily promoted */
>  			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
>  				int tier = lru_tier_from_refs(refs, workingset);
> -				int delta = folio_nr_pages(folio);
>  
>  				WRITE_ONCE(lrugen->protected[hist][type][tier],
> -					   lrugen->protected[hist][type][tier] + delta);
> +					   lrugen->protected[hist][type][tier] + nr_pages);
>  			}
>  
>  			if (!--remaining)
> -				return false;
> +				break;
>  		}
> +		WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
> +			   lrugen->nr_pages[old_gen][type][zone] - delta);
> +		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
> +			   lrugen->nr_pages[target_gen][type][zone] + delta);
> +		if (!remaining)
> +			return false;
>  	}
>  done:
>  	reset_ctrl_pos(lruvec, type, true);
> -- 
> 2.34.1
> 

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

[Baoquan He]

On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> During aging, a folio's generation may already have been updated by
> folio_update_gen(), even though it has not yet been moved to the
> corresponding generation list. Such folios are hotter than those
> already in that generation.
> 
> It makes sense for inc_min_seq() to increment the generation of
> folios that were never promoted during aging and move them to the
> tail of the new oldest generation. However, folios that were already
> promoted should instead be moved to the head of their updated
> generation, just as sort_folio() does in scan_folios().

While sort_folio() move protected folio to the head of next gen too.
It only moves ineligible folios to the tail of next gen.

> 
> Otherwise, promoted folios could end up behind folios that were
> never promoted, effectively inverting their hot/cold ordering.
> 
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
>  mm/vmscan.c | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 99ee3c833d54..3b618a51cde2 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3946,9 +3946,12 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
>  
>  			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
> -			list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
> -			if (gen_increased)
> +			if (gen_increased) {
>  				delta += nr_pages;
> +				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
> +			} else {
> +				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
> +			}
>  			/* don't count the workingset being lazily promoted */
>  			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
>  				int tier = lru_tier_from_refs(refs, workingset);
> -- 
> 2.34.1
> 

Re: [PATCH 4/6] mm/mglru: exclude folios promoted by aging from protected in inc_min_seq()

[Baoquan He]

On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> Some folios may have been promoted during aging, so don't count them
> as protected, similar to sort_folio().

Makes sense, the change makes it consistent with sort_filio.

Reviewed-by: Baoquan He <baoquan.he@linux.dev>

> 
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
>  mm/vmscan.c | 13 ++++++-------
>  1 file changed, 6 insertions(+), 7 deletions(-)
> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 3b618a51cde2..7bd01875fade 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3949,16 +3949,15 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  			if (gen_increased) {
>  				delta += nr_pages;
>  				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
> +				/* don't count the workingset being lazily promoted */
> +				if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
> +					int tier = lru_tier_from_refs(refs, workingset);
> +
> +					protected[tier] += nr_pages;
> +				}
>  			} else {
>  				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
>  			}
> -			/* don't count the workingset being lazily promoted */
> -			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
> -				int tier = lru_tier_from_refs(refs, workingset);
> -
> -				protected[tier] += nr_pages;
> -			}
> -
>  			if (!--remaining)
>  				break;
>  		}
> -- 
> 2.34.1
> 

Re: [PATCH 5/6] mm/mglru: move folios from oldest gen to second-oldest gen from head to tail

[Baoquan He]

On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> For reclamation, it makes sense to reclaim folios from tail to
> head, as folios near the head are relatively hot. However, when
> moving folios from the oldest generation to the second-oldest
> generation, using the tail-to-head order would effectively cause
> a cold/hot inversion.

This is great, do we need do the similar thing on scan_folios()?
While it's not relevant to this patch.

Reviewed-by: Baoquan He <baoquan.he@linux.dev>

> 
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
>  mm/vmscan.c | 19 +++++++++++++++++--
>  1 file changed, 17 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 7bd01875fade..2fd82b2ca4d1 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -191,8 +191,20 @@ struct scan_control {
>  			prefetchw(&prev->_field);			\
>  		}							\
>  	} while (0)
> +#define prefetchw_next_lru_folio(_folio, _base, _field)			\
> +	do {								\
> +		if ((_folio)->lru.next != _base) {			\
> +			struct folio *next;				\
> +									\
> +			next = list_entry((_folio)->lru.next,		\
> +					struct folio, lru);		\
> +			prefetchw(&next->_field);			\
> +		}							\
> +	} while (0)
> +
>  #else
>  #define prefetchw_prev_lru_folio(_folio, _base, _field) do { } while (0)
> +#define prefetchw_next_lru_folio(_folio, _base, _field) do { } while (0)
>  #endif
>  
>  /*
> @@ -3932,9 +3944,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
>  		struct list_head *head = &lrugen->folios[old_gen][type][zone];
>  		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
> +		struct list_head *pos = head->next;
>  
> -		while (!list_empty(head)) {
> -			struct folio *folio = lru_to_folio(head);
> +		while (pos != head) {
> +			struct folio *folio = list_entry(pos, struct folio, lru);
>  			long nr_pages = folio_nr_pages(folio);
>  			int refs = folio_lru_refs(folio);
>  			bool workingset = folio_test_workingset(folio);
> @@ -3945,6 +3958,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  			VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);
>  			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
>  
> +			prefetchw_next_lru_folio(folio, head, flags);
> +			pos = pos->next;
>  			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
>  			if (gen_increased) {
>  				delta += nr_pages;
> -- 
> 2.34.1
> 

Re: [PATCH 2/6] mm/mglru: batch update lrugen->protected in inc_min_seq()

[Baoquan He]

On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> Avoid updating lrugen->protected with WRITE_ONCE() for each folio,
> which may prevent potential compiler optimizations. Accumulate the
> updates locally and apply them in a batch instead.

Wondering how much efficiency this can bring, is there a number for this
standalone patch?

> 
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
>  mm/vmscan.c | 8 +++++---
>  1 file changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 0d74fc00abd3..99ee3c833d54 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3931,7 +3931,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  	/* prevent cold/hot inversion if the type is evictable */
>  	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
>  		struct list_head *head = &lrugen->folios[old_gen][type][zone];
> -		unsigned long delta = 0;
> +		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
>  
>  		while (!list_empty(head)) {
>  			struct folio *folio = lru_to_folio(head);
> @@ -3953,8 +3953,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
>  				int tier = lru_tier_from_refs(refs, workingset);
>  
> -				WRITE_ONCE(lrugen->protected[hist][type][tier],
> -					   lrugen->protected[hist][type][tier] + nr_pages);
> +				protected[tier] += nr_pages;
>  			}
>  
>  			if (!--remaining)
> @@ -3964,6 +3963,9 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  			   lrugen->nr_pages[old_gen][type][zone] - delta);
>  		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
>  			   lrugen->nr_pages[target_gen][type][zone] + delta);
> +		for (int tier = 0; tier < MAX_NR_TIERS; tier++)
> +			WRITE_ONCE(lrugen->protected[hist][type][tier],
> +				   lrugen->protected[hist][type][tier] + protected[tier]);
>  		if (!remaining)
>  			return false;
>  	}
> -- 
> 2.34.1
> 

Re: [PATCH 6/6] mm/mglru: batch move folios to the second-oldest gen's LRU

[Baoquan He]

On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> Detect folios that need to move from the oldest generation to
> the second-oldest generation, and batch-move them together.
> This can significantly reduce the sys time of inc_min_seq(),
> especially when the other type is significantly behind the
> preferred type.
> 
> Assisted-by: gemini:gemini-3.6-flash
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
>  mm/vmscan.c | 21 ++++++++++++++++++++-
>  1 file changed, 20 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 2fd82b2ca4d1..996b48344ed0 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3923,6 +3923,19 @@ static void clear_mm_walk(void)
>  		kfree(walk);
>  }
>  
> +static inline void flush_lru_batch(struct list_head *head, struct list_head **batch_end,
> +				   struct list_head *dst)
> +{
> +	LIST_HEAD(movable);
> +
> +	if (!*batch_end)
> +		return;
> +
> +	list_cut_position(&movable, head, *batch_end);
> +	list_splice_tail_init(&movable, dst);
> +	*batch_end = NULL;
> +}
> +
>  static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  {
>  	int zone;
> @@ -3942,9 +3955,11 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  
>  	/* prevent cold/hot inversion if the type is evictable */
>  	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
> +		struct list_head *target_list = &lrugen->folios[target_gen][type][zone];
>  		struct list_head *head = &lrugen->folios[old_gen][type][zone];
>  		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
>  		struct list_head *pos = head->next;
> +		struct list_head *batch_end = NULL;
>  
>  		while (pos != head) {
>  			struct folio *folio = list_entry(pos, struct folio, lru);
> @@ -3963,7 +3978,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
>  			if (gen_increased) {
>  				delta += nr_pages;
> -				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
> +				batch_end = &folio->lru;
> +
>  				/* don't count the workingset being lazily promoted */
>  				if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
>  					int tier = lru_tier_from_refs(refs, workingset);
> @@ -3971,11 +3987,14 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  					protected[tier] += nr_pages;
>  				}
>  			} else {
> +				flush_lru_batch(head, &batch_end, target_list);

How about this? then you only have "struct list_head *batch_end" in 2nd
parameter. And explicit resetting batch_end makes code more readable?

				flush_lru_batch(head, batch_end, target_list);
				batch_end = NULL;

Anyway, personal preference, not strong opinion.

Other than the nit,

Reviewed-by: Baoquan He <baoquan.he@linux.dev> 

>  				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
>  			}
>  			if (!--remaining)
>  				break;
>  		}
> +		flush_lru_batch(head, &batch_end, target_list);
> +
>  		WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
>  			   lrugen->nr_pages[old_gen][type][zone] - delta);
>  		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
> -- 
> 2.34.1
> 

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

[Barry Song]

On Wed, Aug 26, 2026 at 4:56 PM Baoquan He <baoquan.he@linux.dev> wrote:
>
> On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> > During aging, a folio's generation may already have been updated by
> > folio_update_gen(), even though it has not yet been moved to the
> > corresponding generation list. Such folios are hotter than those
> > already in that generation.
> >
> > It makes sense for inc_min_seq() to increment the generation of
> > folios that were never promoted during aging and move them to the
> > tail of the new oldest generation. However, folios that were already
> > promoted should instead be moved to the head of their updated
> > generation, just as sort_folio() does in scan_folios().
>
> While sort_folio() move protected folio to the head of next gen too.
> It only moves ineligible folios to the tail of next gen.
>

Hi Baoquan,

Thanks for the review! I’m not quite sure I understand what you mean :-)
Could you please clarify what you’re suggesting?

I can explain the design in more detail.

Yes. For both promoted and protected folios, `sort_folio()` moves them
to the head of the corresponding generation.

Here, `inc_min_seq()` is a bit different. We are overlapping `max_seq`
and `min_seq`, so the `min_seq` generation should be moved to the
second-oldest generation. Therefore, I think non-promoted folios should
be placed at the tail.
They are genuinely not promoted, so they shouldn't be at the head.

For example, suppose we have the following folios:

Second-oldest gen: f1, f2, f3, f4

Oldest gen:        f5 (promoted), f6 (not promoted),
f7 (promoted), f8 (not promoted)

Without my patchset, the result is:

Second-oldest:

f1, f2, f3, f4, f8 (promoted), f7 (not promoted),
f6 (not promoted), f5 (promoted)

So you can see that both promoted and non-promoted folios are at the tail
of the second-oldest generation?

With my patchset, the result is:

Second-oldest:

f5 (promoted), f8 (promoted), f1, f2, f3, f4,
f7 (not promoted), f6 (not promoted)

Best Regards
Barry

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

[Baoquan He]

On 08/27/26 at 05:43am, Barry Song wrote:
> On Wed, Aug 26, 2026 at 4:56 PM Baoquan He <baoquan.he@linux.dev> wrote:
> >
> > On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> > > During aging, a folio's generation may already have been updated by
> > > folio_update_gen(), even though it has not yet been moved to the
> > > corresponding generation list. Such folios are hotter than those
> > > already in that generation.
> > >
> > > It makes sense for inc_min_seq() to increment the generation of
> > > folios that were never promoted during aging and move them to the
> > > tail of the new oldest generation. However, folios that were already
> > > promoted should instead be moved to the head of their updated
> > > generation, just as sort_folio() does in scan_folios().
> >
> > While sort_folio() move protected folio to the head of next gen too.
> > It only moves ineligible folios to the tail of next gen.
> >
> 
> Hi Baoquan,
> 
> Thanks for the review! I’m not quite sure I understand what you mean :-)
> Could you please clarify what you’re suggesting?

Sorry for the confusion, Barry. I meant this is a good one, and
sort_folio() has the similar issue in which the protected folios are
moved to the head, wondering if that need be adjusted too. One consistent
rule for both is better.

> 
> I can explain the design in more detail.
> 
> Yes. For both promoted and protected folios, `sort_folio()` moves them
> to the head of the corresponding generation.
> 
> Here, `inc_min_seq()` is a bit different. We are overlapping `max_seq`
> and `min_seq`, so the `min_seq` generation should be moved to the
> second-oldest generation. Therefore, I think non-promoted folios should
> be placed at the tail.
> They are genuinely not promoted, so they shouldn't be at the head.
> 
> For example, suppose we have the following folios:
> 
> Second-oldest gen: f1, f2, f3, f4
> 
> Oldest gen:        f5 (promoted), f6 (not promoted),
> f7 (promoted), f8 (not promoted)
> 
> Without my patchset, the result is:
> 
> Second-oldest:
> 
> f1, f2, f3, f4, f8 (promoted), f7 (not promoted),
> f6 (not promoted), f5 (promoted)
> 
> So you can see that both promoted and non-promoted folios are at the tail
> of the second-oldest generation?
> 
> With my patchset, the result is:
> 
> Second-oldest:
> 
> f5 (promoted), f8 (promoted), f1, f2, f3, f4,
> f7 (not promoted), f6 (not promoted)
> 
> Best Regards
> Barry

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

[Barry Song]

On Thu, Aug 27, 2026 at 8:46 AM Baoquan He <baoquan.he@linux.dev> wrote:
>
> On 08/27/26 at 05:43am, Barry Song wrote:
> > On Wed, Aug 26, 2026 at 4:56 PM Baoquan He <baoquan.he@linux.dev> wrote:
> > >
> > > On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> > > > During aging, a folio's generation may already have been updated by
> > > > folio_update_gen(), even though it has not yet been moved to the
> > > > corresponding generation list. Such folios are hotter than those
> > > > already in that generation.
> > > >
> > > > It makes sense for inc_min_seq() to increment the generation of
> > > > folios that were never promoted during aging and move them to the
> > > > tail of the new oldest generation. However, folios that were already
> > > > promoted should instead be moved to the head of their updated
> > > > generation, just as sort_folio() does in scan_folios().
> > >
> > > While sort_folio() move protected folio to the head of next gen too.
> > > It only moves ineligible folios to the tail of next gen.
> > >
> >
> > Hi Baoquan,
> >
> > Thanks for the review! I’m not quite sure I understand what you mean :-)
> > Could you please clarify what you’re suggesting?
>
> Sorry for the confusion, Barry. I meant this is a good one, and
> sort_folio() has the similar issue in which the protected folios are
> moved to the head, wondering if that need be adjusted too. One consistent
> rule for both is better.

I think it might be fine for sort_folio() to move protected folios to the
head, since those folios have either been accessed multiple times or have
reached a tier higher than tier_idx. They are sort of hot in theory, right?

if (tier > tier_idx || refs + workingset == BIT(LRU_REFS_WIDTH) + 1)

But for inc_min_seq(), it is just catching up to make sure the newest
generation doesn't overlap with the oldest generation. Those non-promoted
folios themselves aren't hot , so I feel these are actually different?

Best Regards
Barry

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

[Baoquan He]

On 08/27/26 at 09:24am, Barry Song wrote:
> On Thu, Aug 27, 2026 at 8:46 AM Baoquan He <baoquan.he@linux.dev> wrote:
> >
> > On 08/27/26 at 05:43am, Barry Song wrote:
> > > On Wed, Aug 26, 2026 at 4:56 PM Baoquan He <baoquan.he@linux.dev> wrote:
> > > >
> > > > On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> > > > > During aging, a folio's generation may already have been updated by
> > > > > folio_update_gen(), even though it has not yet been moved to the
> > > > > corresponding generation list. Such folios are hotter than those
> > > > > already in that generation.
> > > > >
> > > > > It makes sense for inc_min_seq() to increment the generation of
> > > > > folios that were never promoted during aging and move them to the
> > > > > tail of the new oldest generation. However, folios that were already
> > > > > promoted should instead be moved to the head of their updated
> > > > > generation, just as sort_folio() does in scan_folios().
> > > >
> > > > While sort_folio() move protected folio to the head of next gen too.
> > > > It only moves ineligible folios to the tail of next gen.
> > > >
> > >
> > > Hi Baoquan,
> > >
> > > Thanks for the review! I’m not quite sure I understand what you mean :-)
> > > Could you please clarify what you’re suggesting?
> >
> > Sorry for the confusion, Barry. I meant this is a good one, and
> > sort_folio() has the similar issue in which the protected folios are
> > moved to the head, wondering if that need be adjusted too. One consistent
> > rule for both is better.
> 
> I think it might be fine for sort_folio() to move protected folios to the
> head, since those folios have either been accessed multiple times or have
> reached a tier higher than tier_idx. They are sort of hot in theory, right?
> 
> if (tier > tier_idx || refs + workingset == BIT(LRU_REFS_WIDTH) + 1)
> 
> But for inc_min_seq(), it is just catching up to make sure the newest
> generation doesn't overlap with the oldest generation. Those non-promoted
> folios themselves aren't hot , so I feel these are actually different?

I got your point, sort_folio() considers the hottness, inc_min_seq()
doesn't. I agree with you now. Thanks for the explanation.

BUT no matter what it is, protected folios, lazily promoted folios,
and no matter where it is, put in head of next gen or tail of next gen,
their refs are cleared by folio_inc_gen(). Then in sort_folio(), they
are all tier 0 of the oldest gen and must be reclaimed.

So here, I think differentiating them and moving them into head or tail
doesn't make sense, the thing is whether if we need do something to
retain refs of folios when gen_increased. At least, for lazily promoted
folios, it should not be put in the tail of next gen and refs cleared.
What do you think?

Thanks
Baoquan

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

[Baoquan He]

On 08/27/26 at 10:14am, Baoquan He wrote:
> On 08/27/26 at 09:24am, Barry Song wrote:
> > On Thu, Aug 27, 2026 at 8:46 AM Baoquan He <baoquan.he@linux.dev> wrote:
> > >
> > > On 08/27/26 at 05:43am, Barry Song wrote:
> > > > On Wed, Aug 26, 2026 at 4:56 PM Baoquan He <baoquan.he@linux.dev> wrote:
> > > > >
> > > > > On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> > > > > > During aging, a folio's generation may already have been updated by
> > > > > > folio_update_gen(), even though it has not yet been moved to the
> > > > > > corresponding generation list. Such folios are hotter than those
> > > > > > already in that generation.
> > > > > >
> > > > > > It makes sense for inc_min_seq() to increment the generation of
> > > > > > folios that were never promoted during aging and move them to the
> > > > > > tail of the new oldest generation. However, folios that were already
> > > > > > promoted should instead be moved to the head of their updated
> > > > > > generation, just as sort_folio() does in scan_folios().
> > > > >
> > > > > While sort_folio() move protected folio to the head of next gen too.
> > > > > It only moves ineligible folios to the tail of next gen.
> > > > >
> > > >
> > > > Hi Baoquan,
> > > >
> > > > Thanks for the review! I’m not quite sure I understand what you mean :-)
> > > > Could you please clarify what you’re suggesting?
> > >
> > > Sorry for the confusion, Barry. I meant this is a good one, and
> > > sort_folio() has the similar issue in which the protected folios are
> > > moved to the head, wondering if that need be adjusted too. One consistent
> > > rule for both is better.
> > 
> > I think it might be fine for sort_folio() to move protected folios to the
> > head, since those folios have either been accessed multiple times or have
> > reached a tier higher than tier_idx. They are sort of hot in theory, right?
> > 
> > if (tier > tier_idx || refs + workingset == BIT(LRU_REFS_WIDTH) + 1)
> > 
> > But for inc_min_seq(), it is just catching up to make sure the newest
> > generation doesn't overlap with the oldest generation. Those non-promoted
> > folios themselves aren't hot , so I feel these are actually different?
> 
> I got your point, sort_folio() considers the hottness, inc_min_seq()
> doesn't. I agree with you now. Thanks for the explanation.
> 
> BUT no matter what it is, protected folios, lazily promoted folios,
> and no matter where it is, put in head of next gen or tail of next gen,
> their refs are cleared by folio_inc_gen(). Then in sort_folio(), they
> are all tier 0 of the oldest gen and must be reclaimed.

Or mm walking will take a long time, it doesn't matter much about the
refs in next gen in inc_min_seq() because there are a lot of chances
refs are updated when it comes to sort_folio()?

> 
> So here, I think differentiating them and moving them into head or tail
> doesn't make sense, the thing is whether if we need do something to
> retain refs of folios when gen_increased. At least, for lazily promoted
> folios, it should not be put in the tail of next gen and refs cleared.
> What do you think?
> 
> Thanks
> Baoquan

Re: [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

[Kairui Song]

On Fri, Aug 21, 2026 at 7:09 PM Barry Song (Xiaomi) <baohua@kernel.org> wrote:
>
> Currently, folio_inc_gen() updates lrugen->nr_pages for every folio
> as it advances generations. Instead, accumulate the size changes
> and update lrugen->nr_pages in a batch after scanning the entire
> oldest generation, or when the scan stops because remaining reaches
> zero.
>
> Since we only move folios from the oldest generation to the second
> oldest generation, the active/inactive state cannot change. We can
> therefore skip __lru_update_size().
>
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
>  mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++-----------
>  1 file changed, 35 insertions(+), 11 deletions(-)

Hello Barry

Thanks for the patch!

>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index c1404a59523d..0d74fc00abd3 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3296,20 +3296,21 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma
>  }
>
>  /* protect pages accessed multiple times through file descriptors */
> -static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
> +static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)

I feel the naming is a bit confusing, the __ prefix doesn't tell how
it differs from folio_inc_gen very well, maybe just name one
folio_inc_gen (the old gen could be any gen), another one is
folio_inc_min_gen (the old gen can only be min_seq), and with sanity
check in folio_inc_min_gen that expects get_nr_gens == 4, and
lru_gen_is_active(min_seq) == lru_gen_is_active(min_seq + 1)? This
could be a build-time sanity check instead of a runtime debug check.

I saw baoquan also mentioned the debug check on
lru_gen_is_active(min_seq) == lru_gen_is_active(min_seq + 1), which I
agree.

And BTW I'm suggesting changing the definition of active/inactive for
MGLRU from gen-based to refs-based. The gen-based active/inactive
reading has been giving us headaches for years and is a main blocker
for server production, and I think no one cares about the
active/inactive reading for MGLRU for desktop / mobile right now
(especially anon active / inactive, which is basiclaly random number
at this point :P, and files are almost always stuck at inactive).

However that contradicts this change by a lot, folio_inc_gen will
always have to update the statistic, see (was planning to send RFC
after more test on Android):

https://github.com/ryncsn/linux/commit/d13a3abdd22a97eca32a7207fce56cc6030a3bd7

This is another reason why I'm trying to avoid aging rather than
optimizing it. I once tried some aging optimization two year ago, the
result was good by then but I gave up due to the added complexity:
https://lore.kernel.org/linux-mm/20240123184552.59758-1-ryncsn@gmail.com/

I realized the main blocker was cmpxchg and maybe async aging with a
refined reclaim cycle was the better solution rather than making
things complex.

But anyway, let's process with this; perhaps we'll need to find a
different way to batch aging later to fit the implementation of the
FG, OOM prevention or refs-based idea.

Re: [PATCH 0/6] mm/mglru: speed up inc_min_seq() and fix cold/hot inversions

[Xueyuan Chen]

On Fri, Aug 21, 2026 at 06:25:32PM +0800, Barry Song (Xiaomi) wrote:
>This is an aging speedup series split out from the MGLRU swappiness
>series [1], with the inc_min_seq changes separated to make them
>easier to review.
>
>Currently, inc_min_seq performance is crucial to both the swappiness
>fix and proactive aging. There are two problems with it:
>
>1. It processes each folio one by one, while many operations can be
>batched or skipped. For example, a batch of folios can be moved together
>from the oldest generation to the second-oldest generation, and the
>associated counting can also be done in batches.
>
>2. It may cause potential cold/hot inversion by placing promoted folios
>(which have been scanned and found to have young PTEs) behind
>non-promoted folios. A similar inversion can also occur among
>non-promoted folios, as tail folios from the oldest generation are
>placed before head folios when moving them to the second-oldest
>generation.
>
>This series tries to batch operations as much as possible and fix the
>potential cold/hot inversion by keeping promoted folios ahead of
>non-promoted folios, while also preserving the order of non-promoted
>folios when moving them from the oldest generation to the second-oldest
>generation.
>
>Minor issue: inc_min_seq() also counts protected folios
>improperly, as promoted folios should be skipped, as in
>sort_folio().
>
>We need a stable workload with a stable number of folios to measure
>aging and evaluate the speedup in inc_min_seq(). So I asked ChatGPT
>to generate the microbenchmark below. It ages an LRU vec containing
>512 MB of memory 100 times:
>
> #define _GNU_SOURCE
> 
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> #include <stdint.h>
> #include <unistd.h>
> #include <fcntl.h>
> #include <errno.h>
> #include <time.h>
> #include <sys/mman.h>
> 
> #define SIZE		(512UL * 1024 * 1024)
> #define LRU_GEN		"/sys/kernel/debug/lru_gen"
> #define TARGET_CGROUP	"/system.slice/agetest.scope"
> #define START_GEN	3
> #define END_GEN	103
> 
> static long long nsec_diff(const struct timespec *start,
> 			   const struct timespec *end)
> {
> 	return (end->tv_sec - start->tv_sec) * 1000000000LL +
> 	       (end->tv_nsec - start->tv_nsec);
> }
> 
> static int find_memcg_id(void)
> {
> 	FILE *fp;
> 	char line[4096];
> 	int memcg_id;
> 
> 	fp = fopen(LRU_GEN, "r");
> 	if (!fp) {
> 		perror("fopen lru_gen");
> 		return -1;
> 	}
> 
> 	while (fgets(line, sizeof(line), fp)) {
> 		char *p;
> 
> 		if (strncmp(line, "memcg ", 6))
> 			continue;
> 
> 		p = line + 6;
> 
> 		if (sscanf(p, "%d", &memcg_id) != 1)
> 			continue;
> 
> 		/*
> 		 * The memcg path follows the numeric ID.
> 		 */
> 		p = strchr(p, ' ');
> 		if (!p)
> 			continue;
> 
> 		if (strstr(p, TARGET_CGROUP)) {
> 			fclose(fp);
> 			return memcg_id;
> 		}
> 	}
> 
> 	fclose(fp);
> 
> 	fprintf(stderr, "Cannot find %s\n", TARGET_CGROUP);
> 	return -1;
> }
> 
> int main(void)
> {
> 	void *addr;
> 	int memcg_id;
> 	int fd;
> 	long long total_ns = 0;
> 
> 	/*
> 	 * mmap 512 MB and touch every page.
> 	 */
> 	addr = mmap(NULL, SIZE, PROT_READ | PROT_WRITE,
> 		    MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> 	if (addr == MAP_FAILED) {
> 		perror("mmap");
> 		return 1;
> 	}
> 
> 	memset(addr, 0x55, SIZE);
> 
> 	printf("mmap: %p, size: %lu MB\n",
> 	       addr, SIZE / 1024 / 1024);
> 
> 	/*
> 	 * Find the memcg ID automatically.
> 	 */
> 	memcg_id = find_memcg_id();
> 	if (memcg_id < 0)
> 		return 1;
> 
> 	printf("memcg: %d (%s)\n", memcg_id, TARGET_CGROUP);
> 	printf("aging generation %d -> %d\n",
> 	       START_GEN, END_GEN);
> 
> 	fd = open(LRU_GEN, O_WRONLY);
> 	if (fd < 0) {
> 		perror("open lru_gen");
> 		return 1;
> 	}
> 
> 	for (int gen = START_GEN; gen <= END_GEN; gen++) {
> 		char buf[128];
> 		int len;
> 		struct timespec start, end;
> 		long long ns;
> 
> 		len = snprintf(buf, sizeof(buf),
> 			       "+ %d 0 %d\n", memcg_id, gen);
> 
> 		clock_gettime(CLOCK_MONOTONIC, &start);
> 
> 		if (write(fd, buf, len) != len) {
> 			perror("write lru_gen");
> 			close(fd);
> 			return 1;
> 		}
> 
> 		clock_gettime(CLOCK_MONOTONIC, &end);
> 
> 		ns = nsec_diff(&start, &end);
> 		total_ns += ns;
> 
> 		printf("gen %3d: %8.3f ms\n",
> 		       gen, ns / 1000000.0);
> 		fflush(stdout);
> 	}
> 
> 	close(fd);
> 
> 	printf("\nTotal:   %.3f ms\n",
> 	       total_ns / 1000000.0);
> 	printf("Average: %.3f ms\n",
> 	       total_ns / (double)(END_GEN - START_GEN + 1) /
> 	       1000000.0);
> 
> 	while (1)
> 		sleep(1);
> 
> 	return 0;
> }
>
>Run the above microbenchmark with:
>systemd-run --scope --unit=agetest -p MemoryMax=1024M ./agetest
>
>I’m seeing inc_min_seq() become significantly faster:
>
>W/o patch:
>
>Running scope as unit: agetest.scope
>mmap: 0x72c1b5a00000, size: 512 MB
>memcg: 12673 (/system.slice/agetest.scope)
>aging generation 3 -> 103
>gen   3:    7.433 ms
>gen   4:    0.949 ms
>gen   5:    2.535 ms
>gen   6:    5.043 ms
>gen   7:    5.041 ms
>gen   8:    5.027 ms
>...
>gen 100:    5.035 ms
>gen 101:    5.011 ms
>gen 102:    5.029 ms
>gen 103:    5.056 ms
>
>Total:   503.946 ms
>Average: 4.990 ms
>
>W/ patch:
>
>Running scope as unit: agetest.scope
>mmap: 0x7c4b1d200000, size: 512 MB
>memcg: 12893 (/system.slice/agetest.scope)
>aging generation 3 -> 103
>gen   3:    7.538 ms
>gen   4:    0.937 ms
>gen   5:    2.348 ms
>gen   6:    2.300 ms
>gen   7:    2.302 ms
>gen   8:    2.294 ms
>gen   9:    2.296 ms
>...
>gen 100:    2.292 ms
>gen 101:    2.307 ms
>gen 102:    2.293 ms
>gen 103:    2.293 ms
>
>Total:   235.718 ms
>Average: 2.334 ms
>
>The average aging time drops from 4.990 ms to 2.334 ms!

Hi Barry,

I tested this series on my arm64 machine (24 cores, 4K base pages)
and reproduced the improvement with the microbenchmark from the
cover letter.

THP=never (PTE):
  baseline: 7.644 ms
  patched:  2.964 ms (-61.2%)

THP=always (PMD):
  baseline: 0.0373 ms
  patched:  0.0292 ms (-21.8%)

The PTE-level gain is larger than your x86 numbers (-61.2% vs
-53.2%). I suspect the per-folio cost of inc_min_seq() is higher
on my arm64 machine (or on arm64 in general).

The gain is also larger at the PTE level than at the PMD level
(-61.2% vs -21.8%), matching the much higher folio count
(131072 vs 256).

Tested-by: Xueyuan Chen <xueyuan.chen21@gmail.com>

thanks,
Xueyuan

>[1] https://lore.kernel.org/linux-mm/20260812121658.69965-1-baohua@kernel.org/
>
>Barry Song (Xiaomi) (6):
>  mm/mglru: batch update lrugen->nr_pages in inc_min_seq()
>  mm/mglru: batch update lrugen->protected in inc_min_seq()
>  mm/mglru: enhance cold/hot inversion handling in inc_min_seq()
>  mm/mglru: exclude folios promoted by aging from protected in
>    inc_min_seq()
>  mm/mglru: move folios from oldest gen to second-oldest gen from head
>    to tail
>  mm/mglru: batch move folios to the second-oldest gen's LRU
>
> mm/vmscan.c | 98 +++++++++++++++++++++++++++++++++++++++++++----------
> 1 file changed, 80 insertions(+), 18 deletions(-)
>
>-- 
>2.34.1
>
>

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

[Kairui Song]

On Thu, Aug 27, 2026 at 10:14 AM Baoquan He <baoquan.he@linux.dev> wrote:
>
> On 08/27/26 at 09:24am, Barry Song wrote:
> > On Thu, Aug 27, 2026 at 8:46 AM Baoquan He <baoquan.he@linux.dev> wrote:
> > >
> > > On 08/27/26 at 05:43am, Barry Song wrote:
> > > > On Wed, Aug 26, 2026 at 4:56 PM Baoquan He <baoquan.he@linux.dev> wrote:
> > > > >
> > > > > On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> > > > > > During aging, a folio's generation may already have been updated by
> > > > > > folio_update_gen(), even though it has not yet been moved to the
> > > > > > corresponding generation list. Such folios are hotter than those
> > > > > > already in that generation.
> > > > > >
> > > > > > It makes sense for inc_min_seq() to increment the generation of
> > > > > > folios that were never promoted during aging and move them to the
> > > > > > tail of the new oldest generation. However, folios that were already
> > > > > > promoted should instead be moved to the head of their updated
> > > > > > generation, just as sort_folio() does in scan_folios().
> > > > >
> > > > > While sort_folio() move protected folio to the head of next gen too.
> > > > > It only moves ineligible folios to the tail of next gen.
> > > > >
> > > >
> > > > Hi Baoquan,
> > > >
> > > > Thanks for the review! I’m not quite sure I understand what you mean :-)
> > > > Could you please clarify what you’re suggesting?
> > >
> > > Sorry for the confusion, Barry. I meant this is a good one, and
> > > sort_folio() has the similar issue in which the protected folios are
> > > moved to the head, wondering if that need be adjusted too. One consistent
> > > rule for both is better.
> >
> > I think it might be fine for sort_folio() to move protected folios to the
> > head, since those folios have either been accessed multiple times or have
> > reached a tier higher than tier_idx. They are sort of hot in theory, right?
> >
> > if (tier > tier_idx || refs + workingset == BIT(LRU_REFS_WIDTH) + 1)
> >
> > But for inc_min_seq(), it is just catching up to make sure the newest
> > generation doesn't overlap with the oldest generation. Those non-promoted
> > folios themselves aren't hot , so I feel these are actually different?
>
> I got your point, sort_folio() considers the hottness, inc_min_seq()
> doesn't. I agree with you now. Thanks for the explanation.
>
> BUT no matter what it is, protected folios, lazily promoted folios,
> and no matter where it is, put in head of next gen or tail of next gen,
> their refs are cleared by folio_inc_gen(). Then in sort_folio(), they
> are all tier 0 of the oldest gen and must be reclaimed.

Hi all,

I think this part is not true? a folio's PG_workingset will never
be gone after set, that pinns the folio to tier 3 through folio_inc_gen.

In fact, I consider this a pitfall rather than a gain; many workloads
have seen regression due to the over protection of such folios.
Especially with that "refs + workingset == BIT(LRU_REFS_WIDTH) + 1"

Resetting the tier to a lower position is better if the folio is
promoted by one gen due to protection, to avoid over protection IMO.

Changing that semantic will require many other adjustments; I suggest
we just ignore that here.

>
> So here, I think differentiating them and moving them into head or tail
> doesn't make sense, the thing is whether if we need do something to
> retain refs of folios when gen_increased. At least, for lazily promoted
> folios, it should not be put in the tail of next gen and refs cleared.
> What do you think?

Actually, retaining refs has a very limited effect on the current
MGLRU implementation.

Basically I agree we should keep the refs if the folios are not lazy
promoted or protected, e.g. in inc_min_seq. But somehow we need a way
to "soft reset" the refs to a slight lower value so the higher tier in
lower gen won't looks hotter than lower tiers in higher gen.

This is not very doable right now, the closest thing we can archive is
just reset the refs field and not touch the PG_workingset bit, which
already done right now. (BTW there is a LRU_REFS_WORKIGNSET reset in
the FG series for this :-)

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

[Kairui Song]

On Fri, Aug 21, 2026 at 7:09 PM Barry Song (Xiaomi) <baohua@kernel.org> wrote:
>
> During aging, a folio's generation may already have been updated by
> folio_update_gen(), even though it has not yet been moved to the
> corresponding generation list. Such folios are hotter than those
> already in that generation.
>
> It makes sense for inc_min_seq() to increment the generation of
> folios that were never promoted during aging and move them to the
> tail of the new oldest generation. However, folios that were already
> promoted should instead be moved to the head of their updated
> generation, just as sort_folio() does in scan_folios().
>
> Otherwise, promoted folios could end up behind folios that were
> never promoted, effectively inverting their hot/cold ordering.
>
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
>  mm/vmscan.c | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 99ee3c833d54..3b618a51cde2 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3946,9 +3946,12 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>                         VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
>
>                         new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
> -                       list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
> -                       if (gen_increased)
> +                       if (gen_increased) {
>                                 delta += nr_pages;
> +                               list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
> +                       } else {
> +                               list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
> +                       }
>                         /* don't count the workingset being lazily promoted */
>                         if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
>                                 int tier = lru_tier_from_refs(refs, workingset);
> --
> 2.34.1
>
>

It looks more consistent with the one in sort_folio, the idea LGTM:

Reviewed-by: Kairui Song <kasong@tencent.com>

Re: [PATCH 2/6] mm/mglru: batch update lrugen->protected in inc_min_seq()

[Barry Song]

On Wed, Aug 26, 2026 at 5:10 PM Baoquan He <baoquan.he@linux.dev> wrote:
>
> On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> > Avoid updating lrugen->protected with WRITE_ONCE() for each folio,
> > which may prevent potential compiler optimizations. Accumulate the
> > updates locally and apply them in a batch instead.
>
> Wondering how much efficiency this can bring, is there a number for this
> standalone patch?

This is a good question. I reverted the protection batching to check its
impact:

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 996b48344ed0..c9a2fd9ad844 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3957,7 +3957,7 @@ static bool inc_min_seq(struct lruvec *lruvec,
int type, int swappiness)
        for (zone = 0; zone < MAX_NR_ZONES; zone++) {
                struct list_head *target_list =
&lrugen->folios[target_gen][type][zone];
                struct list_head *head = &lrugen->folios[old_gen][type][zone];
-               unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
+               unsigned long delta = 0;
                struct list_head *pos = head->next;
                struct list_head *batch_end = NULL;

@@ -3984,7 +3984,8 @@ static bool inc_min_seq(struct lruvec *lruvec,
int type, int swappiness)
                                if (refs + workingset !=
BIT(LRU_REFS_WIDTH) + 1) {
                                        int tier =
lru_tier_from_refs(refs, workingset);

-                                       protected[tier] += nr_pages;
+
WRITE_ONCE(lrugen->protected[hist][type][tier],
+
lrugen->protected[hist][type][tier] + nr_pages);
                                }
                        } else {
                                flush_lru_batch(head, &batch_end, target_list);
@@ -3999,9 +4000,6 @@ static bool inc_min_seq(struct lruvec *lruvec,
int type, int swappiness)
                           lrugen->nr_pages[old_gen][type][zone] - delta);
                WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
                           lrugen->nr_pages[target_gen][type][zone] + delta);
-               for (int tier = 0; tier < MAX_NR_TIERS; tier++)
-                       WRITE_ONCE(lrugen->protected[hist][type][tier],
-                                  lrugen->protected[hist][type][tier]
+ protected[tier]);
                if (!remaining)
                        return false;
        }


And I see:

With protection batching:

Running scope as unit: agetest.scope
mmap: 0x7c4b1d200000, size: 512 MB
memcg: 12893 (/system.slice/agetest.scope)
aging generation 3 -> 103
gen   3:    7.538 ms
gen   4:    0.937 ms
gen   5:    2.348 ms
gen   6:    2.300 ms
gen   7:    2.302 ms
gen   8:    2.294 ms
gen   9:    2.296 ms
...
gen 100:    2.292 ms
gen 101:    2.307 ms
gen 102:    2.293 ms
gen 103:    2.293 ms

Total:   235.718 ms
Average: 2.334 ms

Without protection batching:

Running scope as unit: agetest.scope
mmap: 0x775682e00000, size: 512 MB
memcg: 12823 (/system.slice/agetest.scope)
aging generation 3 -> 103
gen   3:    7.542 ms
gen   4:    0.948 ms
gen   5:    2.338 ms
...
gen 101:    2.340 ms
gen 102:    2.342 ms
gen 103:    2.347 ms

Total:   240.505 ms
Average: 2.381 ms

It’s 2.334 vs. 2.381, which is really minor.

I’m curious to see what the data looks like on ARM. If there’s no
significant difference, we may drop this patch in v2 to reduce the amount
of code change.

Xueyuan, since you’ve been testing this patchset on ARM[1], would you mind
checking this on ARM as well?

[1] https://lore.kernel.org/linux-mm/20260827035416.3012015-1-xueyuan.chen21@gmail.com/

>
> >
> > Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> > ---
> >  mm/vmscan.c | 8 +++++---
> >  1 file changed, 5 insertions(+), 3 deletions(-)
> >
> > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > index 0d74fc00abd3..99ee3c833d54 100644
> > --- a/mm/vmscan.c
> > +++ b/mm/vmscan.c
> > @@ -3931,7 +3931,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
> >       /* prevent cold/hot inversion if the type is evictable */
> >       for (zone = 0; zone < MAX_NR_ZONES; zone++) {
> >               struct list_head *head = &lrugen->folios[old_gen][type][zone];
> > -             unsigned long delta = 0;
> > +             unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
> >
> >               while (!list_empty(head)) {
> >                       struct folio *folio = lru_to_folio(head);
> > @@ -3953,8 +3953,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
> >                       if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
> >                               int tier = lru_tier_from_refs(refs, workingset);
> >
> > -                             WRITE_ONCE(lrugen->protected[hist][type][tier],
> > -                                        lrugen->protected[hist][type][tier] + nr_pages);
> > +                             protected[tier] += nr_pages;
> >                       }
> >
> >                       if (!--remaining)
> > @@ -3964,6 +3963,9 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
> >                          lrugen->nr_pages[old_gen][type][zone] - delta);
> >               WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
> >                          lrugen->nr_pages[target_gen][type][zone] + delta);
> > +             for (int tier = 0; tier < MAX_NR_TIERS; tier++)
> > +                     WRITE_ONCE(lrugen->protected[hist][type][tier],
> > +                                lrugen->protected[hist][type][tier] + protected[tier]);
> >               if (!remaining)
> >                       return false;
> >       }
> > --
> > 2.34.1
> >

Thanks
Barry

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

[Baoquan He]

On 08/27/26 at 12:30pm, Kairui Song wrote:
> On Thu, Aug 27, 2026 at 10:14 AM Baoquan He <baoquan.he@linux.dev> wrote:
> >
> > On 08/27/26 at 09:24am, Barry Song wrote:
> > > On Thu, Aug 27, 2026 at 8:46 AM Baoquan He <baoquan.he@linux.dev> wrote:
> > > >
> > > > On 08/27/26 at 05:43am, Barry Song wrote:
> > > > > On Wed, Aug 26, 2026 at 4:56 PM Baoquan He <baoquan.he@linux.dev> wrote:
> > > > > >
> > > > > > On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> > > > > > > During aging, a folio's generation may already have been updated by
> > > > > > > folio_update_gen(), even though it has not yet been moved to the
> > > > > > > corresponding generation list. Such folios are hotter than those
> > > > > > > already in that generation.
> > > > > > >
> > > > > > > It makes sense for inc_min_seq() to increment the generation of
> > > > > > > folios that were never promoted during aging and move them to the
> > > > > > > tail of the new oldest generation. However, folios that were already
> > > > > > > promoted should instead be moved to the head of their updated
> > > > > > > generation, just as sort_folio() does in scan_folios().
> > > > > >
> > > > > > While sort_folio() move protected folio to the head of next gen too.
> > > > > > It only moves ineligible folios to the tail of next gen.
> > > > > >
> > > > >
> > > > > Hi Baoquan,
> > > > >
> > > > > Thanks for the review! I’m not quite sure I understand what you mean :-)
> > > > > Could you please clarify what you’re suggesting?
> > > >
> > > > Sorry for the confusion, Barry. I meant this is a good one, and
> > > > sort_folio() has the similar issue in which the protected folios are
> > > > moved to the head, wondering if that need be adjusted too. One consistent
> > > > rule for both is better.
> > >
> > > I think it might be fine for sort_folio() to move protected folios to the
> > > head, since those folios have either been accessed multiple times or have
> > > reached a tier higher than tier_idx. They are sort of hot in theory, right?
> > >
> > > if (tier > tier_idx || refs + workingset == BIT(LRU_REFS_WIDTH) + 1)
> > >
> > > But for inc_min_seq(), it is just catching up to make sure the newest
> > > generation doesn't overlap with the oldest generation. Those non-promoted
> > > folios themselves aren't hot , so I feel these are actually different?
> >
> > I got your point, sort_folio() considers the hottness, inc_min_seq()
> > doesn't. I agree with you now. Thanks for the explanation.
> >
> > BUT no matter what it is, protected folios, lazily promoted folios,
> > and no matter where it is, put in head of next gen or tail of next gen,
> > their refs are cleared by folio_inc_gen(). Then in sort_folio(), they
> > are all tier 0 of the oldest gen and must be reclaimed.
> 
> Hi all,
> 
> I think this part is not true? a folio's PG_workingset will never
> be gone after set, that pinns the folio to tier 3 through folio_inc_gen.

You are right, I missed the PG_workingset part. 

But then folios of refs 0 will get the same treatment as folios of
refs 4, even though later the tier_idx == 1 or 2 in sort_folio(). This
sounds not reasonable.

> 
> In fact, I consider this a pitfall rather than a gain; many workloads
> have seen regression due to the over protection of such folios.
> Especially with that "refs + workingset == BIT(LRU_REFS_WIDTH) + 1"
> 
> Resetting the tier to a lower position is better if the folio is
> promoted by one gen due to protection, to avoid over protection IMO.

Yeah, I got your point and agree. I am thinking of this too, agree we
should respect more on gen than tier for protected folio moving, while
it's not easy to implement.

> .
> Changing that semantic will require many other adjustments; I suggest
> we just ignore that here.

Hmm, my thinking is if we should differentiate folios w/ refs with
folios w/o refs or folios w/ a certain lower refs. Moving them all to
the head of next gen, this at least giving them more time to live and
update refs because eviction pick folios from the tail; or moving them
to tail of next gen, retain limited refs.

> 
> >
> > So here, I think differentiating them and moving them into head or tail
> > doesn't make sense, the thing is whether if we need do something to
> > retain refs of folios when gen_increased. At least, for lazily promoted
> > folios, it should not be put in the tail of next gen and refs cleared.
> > What do you think?
> 
> Actually, retaining refs has a very limited effect on the current
> MGLRU implementation.
> 
> Basically I agree we should keep the refs if the folios are not lazy
> promoted or protected, e.g. in inc_min_seq. But somehow we need a way
> to "soft reset" the refs to a slight lower value so the higher tier in
> lower gen won't looks hotter than lower tiers in higher gen.
> 
> This is not very doable right now, the closest thing we can archive is
> just reset the refs field and not touch the PG_workingset bit, which
> already done right now. (BTW there is a LRU_REFS_WORKIGNSET reset in
> the FG series for this :-)

Yeah. I rechecked your patchset and saw the changing, as I said
privately, the big patch better be split into smaller ones like Barry
has done in this patchset according to logic unit, then reviewing and
discussing will be much easier.

Re: [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

[Barry Song]

On Thu, Aug 27, 2026 at 11:21 AM Kairui Song <ryncsn@gmail.com> wrote:
>
> On Fri, Aug 21, 2026 at 7:09 PM Barry Song (Xiaomi) <baohua@kernel.org> wrote:
> >
> > Currently, folio_inc_gen() updates lrugen->nr_pages for every folio
> > as it advances generations. Instead, accumulate the size changes
> > and update lrugen->nr_pages in a batch after scanning the entire
> > oldest generation, or when the scan stops because remaining reaches
> > zero.
> >
> > Since we only move folios from the oldest generation to the second
> > oldest generation, the active/inactive state cannot change. We can
> > therefore skip __lru_update_size().
> >
> > Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> > ---
> >  mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++-----------
> >  1 file changed, 35 insertions(+), 11 deletions(-)
>
> Hello Barry
>
> Thanks for the patch!
>
> >
> > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > index c1404a59523d..0d74fc00abd3 100644
> > --- a/mm/vmscan.c
> > +++ b/mm/vmscan.c
> > @@ -3296,20 +3296,21 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma
> >  }
> >
> >  /* protect pages accessed multiple times through file descriptors */
> > -static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
> > +static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
>
> I feel the naming is a bit confusing, the __ prefix doesn't tell how
> it differs from folio_inc_gen very well, maybe just name one
> folio_inc_gen (the old gen could be any gen), another one is
> folio_inc_min_gen (the old gen can only be min_seq), and with sanity
> check in folio_inc_min_gen that expects get_nr_gens == 4, and
> lru_gen_is_active(min_seq) == lru_gen_is_active(min_seq + 1)? This
> could be a build-time sanity check instead of a runtime debug check.

Thanks very much for your suggestion, Kairui.

I tried splitting this into two functions, folio_inc_min_gen() and
folio_inc_gen(). I tried a couple of approaches, but I think I’ll give up
on this direction.
folio_inc_min_gen() and folio_inc_gen() share some common code while
also having some differences. It’s hard to avoid ugly code duplication
because folio_inc_gen() cannot call folio_inc_min_gen(). However,
folio_inc_gen() can call __folio_inc_gen().

The __ prefix currently indicates that we don’t update the LRU sizes in
this case. The sizes can either be updated in a batch later or updated once
by folio_inc_gen().
Maybe we could add a comment to make this easier to read, or use a better
name for __folio_inc_gen()?

>
> I saw baoquan also mentioned the debug check on
> lru_gen_is_active(min_seq) == lru_gen_is_active(min_seq + 1), which I
> agree.
>

Agreed. But I feel we only need to do this once in inc_min_seq(),
rather than for each folio.

Best Regards
Barry

Re: [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

[Kairui Song]

On Thu, Aug 27, 2026 at 7:21 PM Barry Song <baohua@kernel.org> wrote:
>
> On Thu, Aug 27, 2026 at 11:21 AM Kairui Song <ryncsn@gmail.com> wrote:
> >
> > On Fri, Aug 21, 2026 at 7:09 PM Barry Song (Xiaomi) <baohua@kernel.org> wrote:
> > >
> > > Currently, folio_inc_gen() updates lrugen->nr_pages for every folio
> > > as it advances generations. Instead, accumulate the size changes
> > > and update lrugen->nr_pages in a batch after scanning the entire
> > > oldest generation, or when the scan stops because remaining reaches
> > > zero.
> > >
> > > Since we only move folios from the oldest generation to the second
> > > oldest generation, the active/inactive state cannot change. We can
> > > therefore skip __lru_update_size().
> > >
> > > Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> > > ---
> > >  mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++-----------
> > >  1 file changed, 35 insertions(+), 11 deletions(-)
> >
> > Hello Barry
> >
> > Thanks for the patch!
> >
> > >
> > > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > > index c1404a59523d..0d74fc00abd3 100644
> > > --- a/mm/vmscan.c
> > > +++ b/mm/vmscan.c
> > > @@ -3296,20 +3296,21 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma
> > >  }
> > >
> > >  /* protect pages accessed multiple times through file descriptors */
> > > -static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
> > > +static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
> >
> > I feel the naming is a bit confusing, the __ prefix doesn't tell how
> > it differs from folio_inc_gen very well, maybe just name one
> > folio_inc_gen (the old gen could be any gen), another one is
> > folio_inc_min_gen (the old gen can only be min_seq), and with sanity
> > check in folio_inc_min_gen that expects get_nr_gens == 4, and
> > lru_gen_is_active(min_seq) == lru_gen_is_active(min_seq + 1)? This
> > could be a build-time sanity check instead of a runtime debug check.
>
> Thanks very much for your suggestion, Kairui.
>
> I tried splitting this into two functions, folio_inc_min_gen() and
> folio_inc_gen(). I tried a couple of approaches, but I think I’ll give up
> on this direction.
> folio_inc_min_gen() and folio_inc_gen() share some common code while
> also having some differences. It’s hard to avoid ugly code duplication
> because folio_inc_gen() cannot call folio_inc_min_gen(). However,
> folio_inc_gen() can call __folio_inc_gen().
>
> The __ prefix currently indicates that we don’t update the LRU sizes in
> this case. The sizes can either be updated in a batch later or updated once
> by folio_inc_gen().
> Maybe we could add a comment to make this easier to read, or use a better
> name for __folio_inc_gen()?

OK, thanks for the explanation. Let's just keep the name then.

Re: [PATCH 2/6] mm/mglru: batch update lrugen->protected in inc_min_seq()

[Xueyuan Chen]

On Thu, Aug 27, 2026 at 1:09 PM Barry Song <baohua@kernel.org> wrote:
>
> On Wed, Aug 26, 2026 at 5:10 PM Baoquan He <baoquan.he@linux.dev> wrote:
> >
> > On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> > > Avoid updating lrugen->protected with WRITE_ONCE() for each folio,
> > > which may prevent potential compiler optimizations. Accumulate the
> > > updates locally and apply them in a batch instead.
> >
> > Wondering how much efficiency this can bring, is there a number for this
> > standalone patch?
>
> This is a good question. I reverted the protection batching to check its
> impact:
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 996b48344ed0..c9a2fd9ad844 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3957,7 +3957,7 @@ static bool inc_min_seq(struct lruvec *lruvec,
> int type, int swappiness)
>         for (zone = 0; zone < MAX_NR_ZONES; zone++) {
>                 struct list_head *target_list =
> &lrugen->folios[target_gen][type][zone];
>                 struct list_head *head = &lrugen->folios[old_gen][type][zone];
> -               unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
> +               unsigned long delta = 0;
>                 struct list_head *pos = head->next;
>                 struct list_head *batch_end = NULL;
>
> @@ -3984,7 +3984,8 @@ static bool inc_min_seq(struct lruvec *lruvec,
> int type, int swappiness)
>                                 if (refs + workingset !=
> BIT(LRU_REFS_WIDTH) + 1) {
>                                         int tier =
> lru_tier_from_refs(refs, workingset);
>
> -                                       protected[tier] += nr_pages;
> +
> WRITE_ONCE(lrugen->protected[hist][type][tier],
> +
> lrugen->protected[hist][type][tier] + nr_pages);
>                                 }
>                         } else {
>                                 flush_lru_batch(head, &batch_end, target_list);
> @@ -3999,9 +4000,6 @@ static bool inc_min_seq(struct lruvec *lruvec,
> int type, int swappiness)
>                            lrugen->nr_pages[old_gen][type][zone] - delta);
>                 WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
>                            lrugen->nr_pages[target_gen][type][zone] + delta);
> -               for (int tier = 0; tier < MAX_NR_TIERS; tier++)
> -                       WRITE_ONCE(lrugen->protected[hist][type][tier],
> -                                  lrugen->protected[hist][type][tier]
> + protected[tier]);
>                 if (!remaining)
>                         return false;
>         }
>
>
> And I see:
>
> With protection batching:
>
> Running scope as unit: agetest.scope
> mmap: 0x7c4b1d200000, size: 512 MB
> memcg: 12893 (/system.slice/agetest.scope)
> aging generation 3 -> 103
> gen   3:    7.538 ms
> gen   4:    0.937 ms
> gen   5:    2.348 ms
> gen   6:    2.300 ms
> gen   7:    2.302 ms
> gen   8:    2.294 ms
> gen   9:    2.296 ms
> ...
> gen 100:    2.292 ms
> gen 101:    2.307 ms
> gen 102:    2.293 ms
> gen 103:    2.293 ms
>
> Total:   235.718 ms
> Average: 2.334 ms
>
> Without protection batching:
>
> Running scope as unit: agetest.scope
> mmap: 0x775682e00000, size: 512 MB
> memcg: 12823 (/system.slice/agetest.scope)
> aging generation 3 -> 103
> gen   3:    7.542 ms
> gen   4:    0.948 ms
> gen   5:    2.338 ms
> ...
> gen 101:    2.340 ms
> gen 102:    2.342 ms
> gen 103:    2.347 ms
>
> Total:   240.505 ms
> Average: 2.381 ms
>
> It’s 2.334 vs. 2.381, which is really minor.
>
> I’m curious to see what the data looks like on ARM. If there’s no
> significant difference, we may drop this patch in v2 to reduce the amount
> of code change.
>
> Xueyuan, since you’ve been testing this patchset on ARM[1], would you mind
> checking this on ARM as well?
>

Hi Barry,

I reverted patch 2 and ran the test again on arm64, with THP=never
like yours:

  full series:  2.964 ms
  w/o patch 2:  2.996 ms

So it's 2.964 vs. 2.996, about 1.1% here too. Consistent with your
x86 data.

Thanks,
Xueyuan

> [1] https://lore.kernel.org/linux-mm/20260827035416.3012015-1-xueyuan.chen21@gmail.com/
>
> >
> > >
> > > Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> > > ---
> > >  mm/vmscan.c | 8 +++++---
> > >  1 file changed, 5 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > > index 0d74fc00abd3..99ee3c833d54 100644
> > > --- a/mm/vmscan.c
> > > +++ b/mm/vmscan.c
> > > @@ -3931,7 +3931,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
> > >       /* prevent cold/hot inversion if the type is evictable */
> > >       for (zone = 0; zone < MAX_NR_ZONES; zone++) {
> > >               struct list_head *head = &lrugen->folios[old_gen][type][zone];
> > > -             unsigned long delta = 0;
> > > +             unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
> > >
> > >               while (!list_empty(head)) {
> > >                       struct folio *folio = lru_to_folio(head);
> > > @@ -3953,8 +3953,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
> > >                       if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
> > >                               int tier = lru_tier_from_refs(refs, workingset);
> > >
> > > -                             WRITE_ONCE(lrugen->protected[hist][type][tier],
> > > -                                        lrugen->protected[hist][type][tier] + nr_pages);
> > > +                             protected[tier] += nr_pages;
> > >                       }
> > >
> > >                       if (!--remaining)
> > > @@ -3964,6 +3963,9 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
> > >                          lrugen->nr_pages[old_gen][type][zone] - delta);
> > >               WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
> > >                          lrugen->nr_pages[target_gen][type][zone] + delta);
> > > +             for (int tier = 0; tier < MAX_NR_TIERS; tier++)
> > > +                     WRITE_ONCE(lrugen->protected[hist][type][tier],
> > > +                                lrugen->protected[hist][type][tier] + protected[tier]);
> > >               if (!remaining)
> > >                       return false;
> > >       }
> > > --
> > > 2.34.1
> > >
>
> Thanks
> Barry