[PATCH 0/6] mm/mglru: speed up inc_min_seq() and fix cold/hot inversions

[PATCH 0/6] mm/mglru: speed up inc_min_seq() and fix cold/hot inversions

[Barry Song (Xiaomi)]

This is an aging speedup series split out from the MGLRU swappiness
series [1], with the inc_min_seq changes separated to make them
easier to review.

Currently, inc_min_seq performance is crucial to both the swappiness
fix and proactive aging. There are two problems with it:

1. It processes each folio one by one, while many operations can be
batched or skipped. For example, a batch of folios can be moved together
from the oldest generation to the second-oldest generation, and the
associated counting can also be done in batches.

2. It may cause potential cold/hot inversion by placing promoted folios
(which have been scanned and found to have young PTEs) behind
non-promoted folios. A similar inversion can also occur among
non-promoted folios, as tail folios from the oldest generation are
placed before head folios when moving them to the second-oldest
generation.

This series tries to batch operations as much as possible and fix the
potential cold/hot inversion by keeping promoted folios ahead of
non-promoted folios, while also preserving the order of non-promoted
folios when moving them from the oldest generation to the second-oldest
generation.

Minor issue: inc_min_seq() also counts protected folios
improperly, as promoted folios should be skipped, as in
sort_folio().

We need a stable workload with a stable number of folios to measure
aging and evaluate the speedup in inc_min_seq(). So I asked ChatGPT
to generate the microbenchmark below. It ages an LRU vec containing
512 MB of memory 100 times:

 #define _GNU_SOURCE
 
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 #include <stdint.h>
 #include <unistd.h>
 #include <fcntl.h>
 #include <errno.h>
 #include <time.h>
 #include <sys/mman.h>
 
 #define SIZE		(512UL * 1024 * 1024)
 #define LRU_GEN		"/sys/kernel/debug/lru_gen"
 #define TARGET_CGROUP	"/system.slice/agetest.scope"
 #define START_GEN	3
 #define END_GEN	103
 
 static long long nsec_diff(const struct timespec *start,
 			   const struct timespec *end)
 {
 	return (end->tv_sec - start->tv_sec) * 1000000000LL +
 	       (end->tv_nsec - start->tv_nsec);
 }
 
 static int find_memcg_id(void)
 {
 	FILE *fp;
 	char line[4096];
 	int memcg_id;
 
 	fp = fopen(LRU_GEN, "r");
 	if (!fp) {
 		perror("fopen lru_gen");
 		return -1;
 	}
 
 	while (fgets(line, sizeof(line), fp)) {
 		char *p;
 
 		if (strncmp(line, "memcg ", 6))
 			continue;
 
 		p = line + 6;
 
 		if (sscanf(p, "%d", &memcg_id) != 1)
 			continue;
 
 		/*
 		 * The memcg path follows the numeric ID.
 		 */
 		p = strchr(p, ' ');
 		if (!p)
 			continue;
 
 		if (strstr(p, TARGET_CGROUP)) {
 			fclose(fp);
 			return memcg_id;
 		}
 	}
 
 	fclose(fp);
 
 	fprintf(stderr, "Cannot find %s\n", TARGET_CGROUP);
 	return -1;
 }
 
 int main(void)
 {
 	void *addr;
 	int memcg_id;
 	int fd;
 	long long total_ns = 0;
 
 	/*
 	 * mmap 512 MB and touch every page.
 	 */
 	addr = mmap(NULL, SIZE, PROT_READ | PROT_WRITE,
 		    MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
 	if (addr == MAP_FAILED) {
 		perror("mmap");
 		return 1;
 	}
 
 	memset(addr, 0x55, SIZE);
 
 	printf("mmap: %p, size: %lu MB\n",
 	       addr, SIZE / 1024 / 1024);
 
 	/*
 	 * Find the memcg ID automatically.
 	 */
 	memcg_id = find_memcg_id();
 	if (memcg_id < 0)
 		return 1;
 
 	printf("memcg: %d (%s)\n", memcg_id, TARGET_CGROUP);
 	printf("aging generation %d -> %d\n",
 	       START_GEN, END_GEN);
 
 	fd = open(LRU_GEN, O_WRONLY);
 	if (fd < 0) {
 		perror("open lru_gen");
 		return 1;
 	}
 
 	for (int gen = START_GEN; gen <= END_GEN; gen++) {
 		char buf[128];
 		int len;
 		struct timespec start, end;
 		long long ns;
 
 		len = snprintf(buf, sizeof(buf),
 			       "+ %d 0 %d\n", memcg_id, gen);
 
 		clock_gettime(CLOCK_MONOTONIC, &start);
 
 		if (write(fd, buf, len) != len) {
 			perror("write lru_gen");
 			close(fd);
 			return 1;
 		}
 
 		clock_gettime(CLOCK_MONOTONIC, &end);
 
 		ns = nsec_diff(&start, &end);
 		total_ns += ns;
 
 		printf("gen %3d: %8.3f ms\n",
 		       gen, ns / 1000000.0);
 		fflush(stdout);
 	}
 
 	close(fd);
 
 	printf("\nTotal:   %.3f ms\n",
 	       total_ns / 1000000.0);
 	printf("Average: %.3f ms\n",
 	       total_ns / (double)(END_GEN - START_GEN + 1) /
 	       1000000.0);
 
 	while (1)
 		sleep(1);
 
 	return 0;
 }

Run the above microbenchmark with:
systemd-run --scope --unit=agetest -p MemoryMax=1024M ./agetest

I’m seeing inc_min_seq() become significantly faster:

W/o patch:

Running scope as unit: agetest.scope
mmap: 0x72c1b5a00000, size: 512 MB
memcg: 12673 (/system.slice/agetest.scope)
aging generation 3 -> 103
gen   3:    7.433 ms
gen   4:    0.949 ms
gen   5:    2.535 ms
gen   6:    5.043 ms
gen   7:    5.041 ms
gen   8:    5.027 ms
...
gen 100:    5.035 ms
gen 101:    5.011 ms
gen 102:    5.029 ms
gen 103:    5.056 ms

Total:   503.946 ms
Average: 4.990 ms

W/ patch:

Running scope as unit: agetest.scope
mmap: 0x7c4b1d200000, size: 512 MB
memcg: 12893 (/system.slice/agetest.scope)
aging generation 3 -> 103
gen   3:    7.538 ms
gen   4:    0.937 ms
gen   5:    2.348 ms
gen   6:    2.300 ms
gen   7:    2.302 ms
gen   8:    2.294 ms
gen   9:    2.296 ms
...
gen 100:    2.292 ms
gen 101:    2.307 ms
gen 102:    2.293 ms
gen 103:    2.293 ms

Total:   235.718 ms
Average: 2.334 ms

The average aging time drops from 4.990 ms to 2.334 ms!

[1] https://lore.kernel.org/linux-mm/20260812121658.69965-1-baohua@kernel.org/

Barry Song (Xiaomi) (6):
  mm/mglru: batch update lrugen->nr_pages in inc_min_seq()
  mm/mglru: batch update lrugen->protected in inc_min_seq()
  mm/mglru: enhance cold/hot inversion handling in inc_min_seq()
  mm/mglru: exclude folios promoted by aging from protected in
    inc_min_seq()
  mm/mglru: move folios from oldest gen to second-oldest gen from head
    to tail
  mm/mglru: batch move folios to the second-oldest gen's LRU

 mm/vmscan.c | 98 +++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 80 insertions(+), 18 deletions(-)

-- 
2.34.1

[PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

[Barry Song (Xiaomi)]

Currently, folio_inc_gen() updates lrugen->nr_pages for every folio
as it advances generations. Instead, accumulate the size changes
and update lrugen->nr_pages in a batch after scanning the entire
oldest generation, or when the scan stops because remaining reaches
zero.

Since we only move folios from the oldest generation to the second
oldest generation, the active/inactive state cannot change. We can
therefore skip __lru_update_size().

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++-----------
 1 file changed, 35 insertions(+), 11 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index c1404a59523d..0d74fc00abd3 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3296,20 +3296,21 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma
 }
 
 /* protect pages accessed multiple times through file descriptors */
-static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
+static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
 {
-	int type = folio_is_file_lru(folio);
-	struct lru_gen_folio *lrugen = &lruvec->lrugen;
-	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
 	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
+	int new_gen;
 
 	VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio);
 
 	do {
 		new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
 		/* folio_update_gen() has promoted this page? */
-		if (new_gen >= 0 && new_gen != old_gen)
+		if (new_gen >= 0 && new_gen != old_gen) {
+			if (increased)
+				*increased = false;
 			return new_gen;
+		}
 
 		new_gen = (old_gen + 1) % MAX_NR_GENS;
 
@@ -3317,8 +3318,21 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
 		new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF;
 	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
 
-	lru_gen_update_size(lruvec, folio, old_gen, new_gen);
+	if (increased)
+		*increased = true;
+	return new_gen;
+}
 
+static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
+{
+	int type = folio_is_file_lru(folio);
+	struct lru_gen_folio *lrugen = &lruvec->lrugen;
+	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
+	bool gen_increased;
+
+	new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
+	if (gen_increased)
+		lru_gen_update_size(lruvec, folio, old_gen, new_gen);
 	return new_gen;
 }
 
@@ -3904,6 +3918,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
 	int hist = lru_hist_from_seq(lrugen->min_seq[type]);
 	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
+	int target_gen = (old_gen + 1) % MAX_NR_GENS;
 
 	/* For file type, skip the check if swappiness is anon only */
 	if (type && (swappiness == SWAPPINESS_ANON_ONLY))
@@ -3916,32 +3931,41 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
+		unsigned long delta = 0;
 
 		while (!list_empty(head)) {
 			struct folio *folio = lru_to_folio(head);
+			long nr_pages = folio_nr_pages(folio);
 			int refs = folio_lru_refs(folio);
 			bool workingset = folio_test_workingset(folio);
+			bool gen_increased;
 
 			VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
-			new_gen = folio_inc_gen(lruvec, folio);
+			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
-
+			if (gen_increased)
+				delta += nr_pages;
 			/* don't count the workingset being lazily promoted */
 			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 				int tier = lru_tier_from_refs(refs, workingset);
-				int delta = folio_nr_pages(folio);
 
 				WRITE_ONCE(lrugen->protected[hist][type][tier],
-					   lrugen->protected[hist][type][tier] + delta);
+					   lrugen->protected[hist][type][tier] + nr_pages);
 			}
 
 			if (!--remaining)
-				return false;
+				break;
 		}
+		WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
+			   lrugen->nr_pages[old_gen][type][zone] - delta);
+		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
+			   lrugen->nr_pages[target_gen][type][zone] + delta);
+		if (!remaining)
+			return false;
 	}
 done:
 	reset_ctrl_pos(lruvec, type, true);
-- 
2.34.1

[PATCH 2/6] mm/mglru: batch update lrugen->protected in inc_min_seq()

[Barry Song (Xiaomi)]

Avoid updating lrugen->protected with WRITE_ONCE() for each folio,
which may prevent potential compiler optimizations. Accumulate the
updates locally and apply them in a batch instead.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 0d74fc00abd3..99ee3c833d54 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3931,7 +3931,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
-		unsigned long delta = 0;
+		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
 
 		while (!list_empty(head)) {
 			struct folio *folio = lru_to_folio(head);
@@ -3953,8 +3953,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 				int tier = lru_tier_from_refs(refs, workingset);
 
-				WRITE_ONCE(lrugen->protected[hist][type][tier],
-					   lrugen->protected[hist][type][tier] + nr_pages);
+				protected[tier] += nr_pages;
 			}
 
 			if (!--remaining)
@@ -3964,6 +3963,9 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			   lrugen->nr_pages[old_gen][type][zone] - delta);
 		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
 			   lrugen->nr_pages[target_gen][type][zone] + delta);
+		for (int tier = 0; tier < MAX_NR_TIERS; tier++)
+			WRITE_ONCE(lrugen->protected[hist][type][tier],
+				   lrugen->protected[hist][type][tier] + protected[tier]);
 		if (!remaining)
 			return false;
 	}
-- 
2.34.1

[PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

[Barry Song (Xiaomi)]

During aging, a folio's generation may already have been updated by
folio_update_gen(), even though it has not yet been moved to the
corresponding generation list. Such folios are hotter than those
already in that generation.

It makes sense for inc_min_seq() to increment the generation of
folios that were never promoted during aging and move them to the
tail of the new oldest generation. However, folios that were already
promoted should instead be moved to the head of their updated
generation, just as sort_folio() does in scan_folios().

Otherwise, promoted folios could end up behind folios that were
never promoted, effectively inverting their hot/cold ordering.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 99ee3c833d54..3b618a51cde2 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3946,9 +3946,12 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
 			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
-			list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
-			if (gen_increased)
+			if (gen_increased) {
 				delta += nr_pages;
+				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+			} else {
+				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+			}
 			/* don't count the workingset being lazily promoted */
 			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 				int tier = lru_tier_from_refs(refs, workingset);
-- 
2.34.1

[PATCH 4/6] mm/mglru: exclude folios promoted by aging from protected in inc_min_seq()

[Barry Song (Xiaomi)]

Some folios may have been promoted during aging, so don't count them
as protected, similar to sort_folio().

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 3b618a51cde2..7bd01875fade 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3949,16 +3949,15 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			if (gen_increased) {
 				delta += nr_pages;
 				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+				/* don't count the workingset being lazily promoted */
+				if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
+					int tier = lru_tier_from_refs(refs, workingset);
+
+					protected[tier] += nr_pages;
+				}
 			} else {
 				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
 			}
-			/* don't count the workingset being lazily promoted */
-			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
-				int tier = lru_tier_from_refs(refs, workingset);
-
-				protected[tier] += nr_pages;
-			}
-
 			if (!--remaining)
 				break;
 		}
-- 
2.34.1

[PATCH 5/6] mm/mglru: move folios from oldest gen to second-oldest gen from head to tail

[Barry Song (Xiaomi)]

For reclamation, it makes sense to reclaim folios from tail to
head, as folios near the head are relatively hot. However, when
moving folios from the oldest generation to the second-oldest
generation, using the tail-to-head order would effectively cause
a cold/hot inversion.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 7bd01875fade..2fd82b2ca4d1 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -191,8 +191,20 @@ struct scan_control {
 			prefetchw(&prev->_field);			\
 		}							\
 	} while (0)
+#define prefetchw_next_lru_folio(_folio, _base, _field)			\
+	do {								\
+		if ((_folio)->lru.next != _base) {			\
+			struct folio *next;				\
+									\
+			next = list_entry((_folio)->lru.next,		\
+					struct folio, lru);		\
+			prefetchw(&next->_field);			\
+		}							\
+	} while (0)
+
 #else
 #define prefetchw_prev_lru_folio(_folio, _base, _field) do { } while (0)
+#define prefetchw_next_lru_folio(_folio, _base, _field) do { } while (0)
 #endif
 
 /*
@@ -3932,9 +3944,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
 		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
+		struct list_head *pos = head->next;
 
-		while (!list_empty(head)) {
-			struct folio *folio = lru_to_folio(head);
+		while (pos != head) {
+			struct folio *folio = list_entry(pos, struct folio, lru);
 			long nr_pages = folio_nr_pages(folio);
 			int refs = folio_lru_refs(folio);
 			bool workingset = folio_test_workingset(folio);
@@ -3945,6 +3958,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
+			prefetchw_next_lru_folio(folio, head, flags);
+			pos = pos->next;
 			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			if (gen_increased) {
 				delta += nr_pages;
-- 
2.34.1

[PATCH 6/6] mm/mglru: batch move folios to the second-oldest gen's LRU

[Barry Song (Xiaomi)]

Detect folios that need to move from the oldest generation to
the second-oldest generation, and batch-move them together.
This can significantly reduce the sys time of inc_min_seq(),
especially when the other type is significantly behind the
preferred type.

Assisted-by: gemini:gemini-3.6-flash
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/vmscan.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 2fd82b2ca4d1..996b48344ed0 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3923,6 +3923,19 @@ static void clear_mm_walk(void)
 		kfree(walk);
 }
 
+static inline void flush_lru_batch(struct list_head *head, struct list_head **batch_end,
+				   struct list_head *dst)
+{
+	LIST_HEAD(movable);
+
+	if (!*batch_end)
+		return;
+
+	list_cut_position(&movable, head, *batch_end);
+	list_splice_tail_init(&movable, dst);
+	*batch_end = NULL;
+}
+
 static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 {
 	int zone;
@@ -3942,9 +3955,11 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
+		struct list_head *target_list = &lrugen->folios[target_gen][type][zone];
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
 		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
 		struct list_head *pos = head->next;
+		struct list_head *batch_end = NULL;
 
 		while (pos != head) {
 			struct folio *folio = list_entry(pos, struct folio, lru);
@@ -3963,7 +3978,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			if (gen_increased) {
 				delta += nr_pages;
-				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+				batch_end = &folio->lru;
+
 				/* don't count the workingset being lazily promoted */
 				if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 					int tier = lru_tier_from_refs(refs, workingset);
@@ -3971,11 +3987,14 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 					protected[tier] += nr_pages;
 				}
 			} else {
+				flush_lru_batch(head, &batch_end, target_list);
 				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
 			}
 			if (!--remaining)
 				break;
 		}
+		flush_lru_batch(head, &batch_end, target_list);
+
 		WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
 			   lrugen->nr_pages[old_gen][type][zone] - delta);
 		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
-- 
2.34.1

Re: [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

[Lian Wang (ProcessMission)]

From: Lian Wang <lianux.mm@gmail.com>

Hi Barry,

A small nit:

> +		unsigned long delta = 0;

lrugen->nr_pages is a signed long and is documented as being able to
transiently go negative while reset_batch_size() is pending. Could delta
remain a long as well, so this arithmetic stays signed?

In 2/6, where the declarations are combined, this could be:

	unsigned long protected[MAX_NR_TIERS] = {};
	long delta = 0;

Thanks,
Lian

Re: [PATCH 5/6] mm/mglru: move folios from oldest gen to second-oldest gen from head to tail

[Kairui Song]

On Fri, Aug 21, 2026 at 6:38 PM Barry Song (Xiaomi) <baohua@kernel.org> wrote:
>
> For reclamation, it makes sense to reclaim folios from tail to
> head, as folios near the head are relatively hot. However, when
> moving folios from the oldest generation to the second-oldest
> generation, using the tail-to-head order would effectively cause
> a cold/hot inversion.
>
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>

Hi Barry

This looks a really good idea, thanks!

> ---
>  mm/vmscan.c | 19 +++++++++++++++++--
>  1 file changed, 17 insertions(+), 2 deletions(-)
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 7bd01875fade..2fd82b2ca4d1 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -191,8 +191,20 @@ struct scan_control {
>                         prefetchw(&prev->_field);                       \
>                 }                                                       \
>         } while (0)
> +#define prefetchw_next_lru_folio(_folio, _base, _field)                        \
> +       do {                                                            \
> +               if ((_folio)->lru.next != _base) {                      \
> +                       struct folio *next;                             \
> +                                                                       \
> +                       next = list_entry((_folio)->lru.next,           \
> +                                       struct folio, lru);             \
> +                       prefetchw(&next->_field);                       \
> +               }                                                       \
> +       } while (0)
> +

I got following warning from checkpatch:

    ● checkpatch.pl: 92: WARNING: Argument '_folio' is not used in
function-like macro
    ● checkpatch.pl: 92: WARNING: Argument '_base' is not used in
function-like macro
    ● checkpatch.pl: 92: WARNING: Argument '_field' is not used in
function-like macro

Maybe you could try b4; it helps run these checks automatically. Feel
free to ignore if you think these warning at pointless.

>  #else
>  #define prefetchw_prev_lru_folio(_folio, _base, _field) do { } while (0)
> +#define prefetchw_next_lru_folio(_folio, _base, _field) do { } while (0)
>  #endif
>
>  /*
> @@ -3932,9 +3944,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>         for (zone = 0; zone < MAX_NR_ZONES; zone++) {
>                 struct list_head *head = &lrugen->folios[old_gen][type][zone];
>                 unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
> +               struct list_head *pos = head->next;
>
> -               while (!list_empty(head)) {
> -                       struct folio *folio = lru_to_folio(head);
> +               while (pos != head) {

Do we need to change the while condition now? Since this commit still
moves folios one by one, will it stop when the list is empty?

> +                       prefetchw_next_lru_folio(folio, head, flags);
> +                       pos = pos->next;

I tried prefetching in MGLRU previously, and it didn't look very good
but there is no regression either; perhaps it's very arch-dependent.
Anyway, I think we can keep it here, maybe further optimize the
prefetch later.

>                         new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
>                         if (gen_increased) {
>                                 delta += nr_pages;

The rest looks good to me, thanks!

Re: [PATCH 5/6] mm/mglru: move folios from oldest gen to second-oldest gen from head to tail

[Barry Song]

On Sat, Aug 22, 2026 at 1:45 PM Kairui Song <ryncsn@gmail.com> wrote:
>
> On Fri, Aug 21, 2026 at 6:38 PM Barry Song (Xiaomi) <baohua@kernel.org> wrote:
> >
> > For reclamation, it makes sense to reclaim folios from tail to
> > head, as folios near the head are relatively hot. However, when
> > moving folios from the oldest generation to the second-oldest
> > generation, using the tail-to-head order would effectively cause
> > a cold/hot inversion.
> >
> > Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
>
> Hi Barry
>
> This looks a really good idea, thanks!
>
> > ---
> >  mm/vmscan.c | 19 +++++++++++++++++--
> >  1 file changed, 17 insertions(+), 2 deletions(-)
> >
> > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > index 7bd01875fade..2fd82b2ca4d1 100644
> > --- a/mm/vmscan.c
> > +++ b/mm/vmscan.c
> > @@ -191,8 +191,20 @@ struct scan_control {
> >                         prefetchw(&prev->_field);                       \
> >                 }                                                       \
> >         } while (0)
> > +#define prefetchw_next_lru_folio(_folio, _base, _field)                        \
> > +       do {                                                            \
> > +               if ((_folio)->lru.next != _base) {                      \
> > +                       struct folio *next;                             \
> > +                                                                       \
> > +                       next = list_entry((_folio)->lru.next,           \
> > +                                       struct folio, lru);             \
> > +                       prefetchw(&next->_field);                       \
> > +               }                                                       \
> > +       } while (0)
> > +
>
> I got following warning from checkpatch:
>
>     ● checkpatch.pl: 92: WARNING: Argument '_folio' is not used in
> function-like macro
>     ● checkpatch.pl: 92: WARNING: Argument '_base' is not used in
> function-like macro
>     ● checkpatch.pl: 92: WARNING: Argument '_field' is not used in
> function-like macro
>
> Maybe you could try b4; it helps run these checks automatically. Feel
> free to ignore if you think these warning at pointless.

Yep. I vividly remember adding this rule to the coding style, and Xining
added the corresponding `checkpatch.pl` change:

commit 6813216bbdba1 ("Documentation: coding-style: ask function-like
macros to evaluate parameters")
commit  b1be5844c1a01 ("scripts: checkpatch: check unused parameters
for function-like macro")

But once we started writing the code, I was influenced by its context and
ended up following that context instead.

>
> >  #else
> >  #define prefetchw_prev_lru_folio(_folio, _base, _field) do { } while (0)
> > +#define prefetchw_next_lru_folio(_folio, _base, _field) do { } while (0)
> >  #endif
> >
> >  /*
> > @@ -3932,9 +3944,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
> >         for (zone = 0; zone < MAX_NR_ZONES; zone++) {
> >                 struct list_head *head = &lrugen->folios[old_gen][type][zone];
> >                 unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
> > +               struct list_head *pos = head->next;
> >
> > -               while (!list_empty(head)) {
> > -                       struct folio *folio = lru_to_folio(head);
> > +               while (pos != head) {
>
> Do we need to change the while condition now? Since this commit still
> moves folios one by one, will it stop when the list is empty?

For this patch, we don't need to change the `while` condition. That change
is more relevant to the next patch:
https://lore.kernel.org/linux-mm/20260821102538.22642-7-baohua@kernel.org/

So I guess I could move this while (pos != head) change to the next patch
if that makes the review easier.

>
> > +                       prefetchw_next_lru_folio(folio, head, flags);
> > +                       pos = pos->next;
>
> I tried prefetching in MGLRU previously, and it didn't look very good
> but there is no regression either; perhaps it's very arch-dependent.
> Anyway, I think we can keep it here, maybe further optimize the
> prefetch later.

Yep. I guess we may want to keep it to help architectures that are
sensitive to prefetching.

>
> >                         new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
> >                         if (gen_increased) {
> >                                 delta += nr_pages;
>
> The rest looks good to me, thanks!

Thanks for the review!

Best Regards
Barry

Re: [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

[Barry Song]

On Sat, Aug 22, 2026 at 9:42 AM Lian Wang (ProcessMission)
<lianux.mm@gmail.com> wrote:
>
> From: Lian Wang <lianux.mm@gmail.com>
>
> Hi Barry,
>
> A small nit:
>
> > +             unsigned long delta = 0;
>
> lrugen->nr_pages is a signed long and is documented as being able to
> transiently go negative while reset_batch_size() is pending. Could delta
> remain a long as well, so this arithmetic stays signed?

Thanks, Lian, for the review. Changing it from unsigned long to long
makes sense to me.

>
> In 2/6, where the declarations are combined, this could be:
>
>         unsigned long protected[MAX_NR_TIERS] = {};
>         long delta = 0;
>

Best Regards
Barry

Re: [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

[Baoquan He]

Hi Barry,

On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> Currently, folio_inc_gen() updates lrugen->nr_pages for every folio
> as it advances generations. Instead, accumulate the size changes
> and update lrugen->nr_pages in a batch after scanning the entire
> oldest generation, or when the scan stops because remaining reaches
> zero.

This patch refactor code to split out __folio_inc_gen() and introduce 
gen_increased, this is the base of later patches. You seem to only
mention the minor optimization of lrugen->nr_pages. IMHO, this
refactoring can be split out to an independent patch. The lrugen->nr_pages
can be integrated with other patch, e.g patch 2 or patch 6.

> 
> Since we only move folios from the oldest generation to the second
> oldest generation, the active/inactive state cannot change. We can
> therefore skip __lru_update_size().

For the justification of skipping __lru_update_size(), there's a
precondition: get_nr_gens(lruvec, type) == MAX_NR_GENS; With this, the
oldest gen and 2nd oldest gen are both inactive. Calling
__lru_update_size() may waste tiny cpu, while skipping it may cause
issue in future if the precondition is changed. Do you think adding
a VM_WARN_ON_ONCE is necessary?

VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) ||
		lru_gen_is_active(lruvec, target_gen));


Thanks
Baoquan
> 
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
>  mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++-----------
>  1 file changed, 35 insertions(+), 11 deletions(-)
> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index c1404a59523d..0d74fc00abd3 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3296,20 +3296,21 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma
>  }
>  
>  /* protect pages accessed multiple times through file descriptors */
> -static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
> +static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
>  {
> -	int type = folio_is_file_lru(folio);
> -	struct lru_gen_folio *lrugen = &lruvec->lrugen;
> -	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
>  	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
> +	int new_gen;
>  
>  	VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio);
>  
>  	do {
>  		new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
>  		/* folio_update_gen() has promoted this page? */
> -		if (new_gen >= 0 && new_gen != old_gen)
> +		if (new_gen >= 0 && new_gen != old_gen) {
> +			if (increased)
> +				*increased = false;
>  			return new_gen;
> +		}
>  
>  		new_gen = (old_gen + 1) % MAX_NR_GENS;
>  
> @@ -3317,8 +3318,21 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
>  		new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF;
>  	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
>  
> -	lru_gen_update_size(lruvec, folio, old_gen, new_gen);
> +	if (increased)
> +		*increased = true;
> +	return new_gen;
> +}
>  
> +static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
> +{
> +	int type = folio_is_file_lru(folio);
> +	struct lru_gen_folio *lrugen = &lruvec->lrugen;
> +	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
> +	bool gen_increased;
> +
> +	new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
> +	if (gen_increased)
> +		lru_gen_update_size(lruvec, folio, old_gen, new_gen);
>  	return new_gen;
>  }
>  
> @@ -3904,6 +3918,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  	struct lru_gen_folio *lrugen = &lruvec->lrugen;
>  	int hist = lru_hist_from_seq(lrugen->min_seq[type]);
>  	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
> +	int target_gen = (old_gen + 1) % MAX_NR_GENS;
>  
>  	/* For file type, skip the check if swappiness is anon only */
>  	if (type && (swappiness == SWAPPINESS_ANON_ONLY))
> @@ -3916,32 +3931,41 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  	/* prevent cold/hot inversion if the type is evictable */
>  	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
>  		struct list_head *head = &lrugen->folios[old_gen][type][zone];
> +		unsigned long delta = 0;
>  
>  		while (!list_empty(head)) {
>  			struct folio *folio = lru_to_folio(head);
> +			long nr_pages = folio_nr_pages(folio);
>  			int refs = folio_lru_refs(folio);
>  			bool workingset = folio_test_workingset(folio);
> +			bool gen_increased;
>  
>  			VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio);
>  			VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio);
>  			VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);
>  			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
>  
> -			new_gen = folio_inc_gen(lruvec, folio);
> +			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
>  			list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
> -
> +			if (gen_increased)
> +				delta += nr_pages;
>  			/* don't count the workingset being lazily promoted */
>  			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
>  				int tier = lru_tier_from_refs(refs, workingset);
> -				int delta = folio_nr_pages(folio);
>  
>  				WRITE_ONCE(lrugen->protected[hist][type][tier],
> -					   lrugen->protected[hist][type][tier] + delta);
> +					   lrugen->protected[hist][type][tier] + nr_pages);
>  			}
>  
>  			if (!--remaining)
> -				return false;
> +				break;
>  		}
> +		WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
> +			   lrugen->nr_pages[old_gen][type][zone] - delta);
> +		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
> +			   lrugen->nr_pages[target_gen][type][zone] + delta);
> +		if (!remaining)
> +			return false;
>  	}
>  done:
>  	reset_ctrl_pos(lruvec, type, true);
> -- 
> 2.34.1
> 

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

[Baoquan He]

On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> During aging, a folio's generation may already have been updated by
> folio_update_gen(), even though it has not yet been moved to the
> corresponding generation list. Such folios are hotter than those
> already in that generation.
> 
> It makes sense for inc_min_seq() to increment the generation of
> folios that were never promoted during aging and move them to the
> tail of the new oldest generation. However, folios that were already
> promoted should instead be moved to the head of their updated
> generation, just as sort_folio() does in scan_folios().

While sort_folio() move protected folio to the head of next gen too.
It only moves ineligible folios to the tail of next gen.

> 
> Otherwise, promoted folios could end up behind folios that were
> never promoted, effectively inverting their hot/cold ordering.
> 
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
>  mm/vmscan.c | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 99ee3c833d54..3b618a51cde2 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3946,9 +3946,12 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
>  
>  			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
> -			list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
> -			if (gen_increased)
> +			if (gen_increased) {
>  				delta += nr_pages;
> +				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
> +			} else {
> +				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
> +			}
>  			/* don't count the workingset being lazily promoted */
>  			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
>  				int tier = lru_tier_from_refs(refs, workingset);
> -- 
> 2.34.1
> 

Re: [PATCH 4/6] mm/mglru: exclude folios promoted by aging from protected in inc_min_seq()

[Baoquan He]

On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> Some folios may have been promoted during aging, so don't count them
> as protected, similar to sort_folio().

Makes sense, the change makes it consistent with sort_filio.

Reviewed-by: Baoquan He <baoquan.he@linux.dev>

> 
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
>  mm/vmscan.c | 13 ++++++-------
>  1 file changed, 6 insertions(+), 7 deletions(-)
> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 3b618a51cde2..7bd01875fade 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3949,16 +3949,15 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  			if (gen_increased) {
>  				delta += nr_pages;
>  				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
> +				/* don't count the workingset being lazily promoted */
> +				if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
> +					int tier = lru_tier_from_refs(refs, workingset);
> +
> +					protected[tier] += nr_pages;
> +				}
>  			} else {
>  				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
>  			}
> -			/* don't count the workingset being lazily promoted */
> -			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
> -				int tier = lru_tier_from_refs(refs, workingset);
> -
> -				protected[tier] += nr_pages;
> -			}
> -
>  			if (!--remaining)
>  				break;
>  		}
> -- 
> 2.34.1
> 

Re: [PATCH 5/6] mm/mglru: move folios from oldest gen to second-oldest gen from head to tail

[Baoquan He]

On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> For reclamation, it makes sense to reclaim folios from tail to
> head, as folios near the head are relatively hot. However, when
> moving folios from the oldest generation to the second-oldest
> generation, using the tail-to-head order would effectively cause
> a cold/hot inversion.

This is great, do we need do the similar thing on scan_folios()?
While it's not relevant to this patch.

Reviewed-by: Baoquan He <baoquan.he@linux.dev>

> 
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
>  mm/vmscan.c | 19 +++++++++++++++++--
>  1 file changed, 17 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 7bd01875fade..2fd82b2ca4d1 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -191,8 +191,20 @@ struct scan_control {
>  			prefetchw(&prev->_field);			\
>  		}							\
>  	} while (0)
> +#define prefetchw_next_lru_folio(_folio, _base, _field)			\
> +	do {								\
> +		if ((_folio)->lru.next != _base) {			\
> +			struct folio *next;				\
> +									\
> +			next = list_entry((_folio)->lru.next,		\
> +					struct folio, lru);		\
> +			prefetchw(&next->_field);			\
> +		}							\
> +	} while (0)
> +
>  #else
>  #define prefetchw_prev_lru_folio(_folio, _base, _field) do { } while (0)
> +#define prefetchw_next_lru_folio(_folio, _base, _field) do { } while (0)
>  #endif
>  
>  /*
> @@ -3932,9 +3944,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
>  		struct list_head *head = &lrugen->folios[old_gen][type][zone];
>  		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
> +		struct list_head *pos = head->next;
>  
> -		while (!list_empty(head)) {
> -			struct folio *folio = lru_to_folio(head);
> +		while (pos != head) {
> +			struct folio *folio = list_entry(pos, struct folio, lru);
>  			long nr_pages = folio_nr_pages(folio);
>  			int refs = folio_lru_refs(folio);
>  			bool workingset = folio_test_workingset(folio);
> @@ -3945,6 +3958,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  			VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);
>  			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
>  
> +			prefetchw_next_lru_folio(folio, head, flags);
> +			pos = pos->next;
>  			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
>  			if (gen_increased) {
>  				delta += nr_pages;
> -- 
> 2.34.1
> 

Re: [PATCH 2/6] mm/mglru: batch update lrugen->protected in inc_min_seq()

[Baoquan He]

On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> Avoid updating lrugen->protected with WRITE_ONCE() for each folio,
> which may prevent potential compiler optimizations. Accumulate the
> updates locally and apply them in a batch instead.

Wondering how much efficiency this can bring, is there a number for this
standalone patch?

> 
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
>  mm/vmscan.c | 8 +++++---
>  1 file changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 0d74fc00abd3..99ee3c833d54 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3931,7 +3931,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  	/* prevent cold/hot inversion if the type is evictable */
>  	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
>  		struct list_head *head = &lrugen->folios[old_gen][type][zone];
> -		unsigned long delta = 0;
> +		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
>  
>  		while (!list_empty(head)) {
>  			struct folio *folio = lru_to_folio(head);
> @@ -3953,8 +3953,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
>  				int tier = lru_tier_from_refs(refs, workingset);
>  
> -				WRITE_ONCE(lrugen->protected[hist][type][tier],
> -					   lrugen->protected[hist][type][tier] + nr_pages);
> +				protected[tier] += nr_pages;
>  			}
>  
>  			if (!--remaining)
> @@ -3964,6 +3963,9 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  			   lrugen->nr_pages[old_gen][type][zone] - delta);
>  		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
>  			   lrugen->nr_pages[target_gen][type][zone] + delta);
> +		for (int tier = 0; tier < MAX_NR_TIERS; tier++)
> +			WRITE_ONCE(lrugen->protected[hist][type][tier],
> +				   lrugen->protected[hist][type][tier] + protected[tier]);
>  		if (!remaining)
>  			return false;
>  	}
> -- 
> 2.34.1
> 

Re: [PATCH 6/6] mm/mglru: batch move folios to the second-oldest gen's LRU

[Baoquan He]

On 08/21/26 at 06:25pm, Barry Song (Xiaomi) wrote:
> Detect folios that need to move from the oldest generation to
> the second-oldest generation, and batch-move them together.
> This can significantly reduce the sys time of inc_min_seq(),
> especially when the other type is significantly behind the
> preferred type.
> 
> Assisted-by: gemini:gemini-3.6-flash
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
>  mm/vmscan.c | 21 ++++++++++++++++++++-
>  1 file changed, 20 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 2fd82b2ca4d1..996b48344ed0 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3923,6 +3923,19 @@ static void clear_mm_walk(void)
>  		kfree(walk);
>  }
>  
> +static inline void flush_lru_batch(struct list_head *head, struct list_head **batch_end,
> +				   struct list_head *dst)
> +{
> +	LIST_HEAD(movable);
> +
> +	if (!*batch_end)
> +		return;
> +
> +	list_cut_position(&movable, head, *batch_end);
> +	list_splice_tail_init(&movable, dst);
> +	*batch_end = NULL;
> +}
> +
>  static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  {
>  	int zone;
> @@ -3942,9 +3955,11 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  
>  	/* prevent cold/hot inversion if the type is evictable */
>  	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
> +		struct list_head *target_list = &lrugen->folios[target_gen][type][zone];
>  		struct list_head *head = &lrugen->folios[old_gen][type][zone];
>  		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
>  		struct list_head *pos = head->next;
> +		struct list_head *batch_end = NULL;
>  
>  		while (pos != head) {
>  			struct folio *folio = list_entry(pos, struct folio, lru);
> @@ -3963,7 +3978,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
>  			if (gen_increased) {
>  				delta += nr_pages;
> -				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
> +				batch_end = &folio->lru;
> +
>  				/* don't count the workingset being lazily promoted */
>  				if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
>  					int tier = lru_tier_from_refs(refs, workingset);
> @@ -3971,11 +3987,14 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
>  					protected[tier] += nr_pages;
>  				}
>  			} else {
> +				flush_lru_batch(head, &batch_end, target_list);

How about this? then you only have "struct list_head *batch_end" in 2nd
parameter. And explicit resetting batch_end makes code more readable?

				flush_lru_batch(head, batch_end, target_list);
				batch_end = NULL;

Anyway, personal preference, not strong opinion.

Other than the nit,

Reviewed-by: Baoquan He <baoquan.he@linux.dev> 

>  				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
>  			}
>  			if (!--remaining)
>  				break;
>  		}
> +		flush_lru_batch(head, &batch_end, target_list);
> +
>  		WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
>  			   lrugen->nr_pages[old_gen][type][zone] - delta);
>  		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
> -- 
> 2.34.1
>