All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 0/6] mm/mglru: speed up inc_min_seq() and fix cold/hot inversions
@ 2026-08-21 10:25 Barry Song (Xiaomi)
  2026-08-21 10:25 ` [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq() Barry Song (Xiaomi)
                   ` (5 more replies)
  0 siblings, 6 replies; 9+ messages in thread
From: Barry Song (Xiaomi) @ 2026-08-21 10:25 UTC (permalink / raw)
  To: akpm, linux-mm
  Cc: axelrasmussen, baolin.wang, baoquan.he, chenridong, david, hannes,
	kasong, lianux.mm, linux-kernel, ljs, lyugaofei, mhocko, qi.zheng,
	shakeel.butt, stevensd, wangzicheng, weixugc, yuanchu, zhangbo56,
	Barry Song (Xiaomi)

This is an aging speedup series split out from the MGLRU swappiness
series [1], with the inc_min_seq changes separated to make them
easier to review.

Currently, inc_min_seq performance is crucial to both the swappiness
fix and proactive aging. There are two problems with it:

1. It processes each folio one by one, while many operations can be
batched or skipped. For example, a batch of folios can be moved together
from the oldest generation to the second-oldest generation, and the
associated counting can also be done in batches.

2. It may cause potential cold/hot inversion by placing promoted folios
(which have been scanned and found to have young PTEs) behind
non-promoted folios. A similar inversion can also occur among
non-promoted folios, as tail folios from the oldest generation are
placed before head folios when moving them to the second-oldest
generation.

This series tries to batch operations as much as possible and fix the
potential cold/hot inversion by keeping promoted folios ahead of
non-promoted folios, while also preserving the order of non-promoted
folios when moving them from the oldest generation to the second-oldest
generation.

Minor issue: inc_min_seq() also counts protected folios
improperly, as promoted folios should be skipped, as in
sort_folio().

We need a stable workload with a stable number of folios to measure
aging and evaluate the speedup in inc_min_seq(). So I asked ChatGPT
to generate the microbenchmark below. It ages an LRU vec containing
512 MB of memory 100 times:

 #define _GNU_SOURCE
 
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 #include <stdint.h>
 #include <unistd.h>
 #include <fcntl.h>
 #include <errno.h>
 #include <time.h>
 #include <sys/mman.h>
 
 #define SIZE		(512UL * 1024 * 1024)
 #define LRU_GEN		"/sys/kernel/debug/lru_gen"
 #define TARGET_CGROUP	"/system.slice/agetest.scope"
 #define START_GEN	3
 #define END_GEN	103
 
 static long long nsec_diff(const struct timespec *start,
 			   const struct timespec *end)
 {
 	return (end->tv_sec - start->tv_sec) * 1000000000LL +
 	       (end->tv_nsec - start->tv_nsec);
 }
 
 static int find_memcg_id(void)
 {
 	FILE *fp;
 	char line[4096];
 	int memcg_id;
 
 	fp = fopen(LRU_GEN, "r");
 	if (!fp) {
 		perror("fopen lru_gen");
 		return -1;
 	}
 
 	while (fgets(line, sizeof(line), fp)) {
 		char *p;
 
 		if (strncmp(line, "memcg ", 6))
 			continue;
 
 		p = line + 6;
 
 		if (sscanf(p, "%d", &memcg_id) != 1)
 			continue;
 
 		/*
 		 * The memcg path follows the numeric ID.
 		 */
 		p = strchr(p, ' ');
 		if (!p)
 			continue;
 
 		if (strstr(p, TARGET_CGROUP)) {
 			fclose(fp);
 			return memcg_id;
 		}
 	}
 
 	fclose(fp);
 
 	fprintf(stderr, "Cannot find %s\n", TARGET_CGROUP);
 	return -1;
 }
 
 int main(void)
 {
 	void *addr;
 	int memcg_id;
 	int fd;
 	long long total_ns = 0;
 
 	/*
 	 * mmap 512 MB and touch every page.
 	 */
 	addr = mmap(NULL, SIZE, PROT_READ | PROT_WRITE,
 		    MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
 	if (addr == MAP_FAILED) {
 		perror("mmap");
 		return 1;
 	}
 
 	memset(addr, 0x55, SIZE);
 
 	printf("mmap: %p, size: %lu MB\n",
 	       addr, SIZE / 1024 / 1024);
 
 	/*
 	 * Find the memcg ID automatically.
 	 */
 	memcg_id = find_memcg_id();
 	if (memcg_id < 0)
 		return 1;
 
 	printf("memcg: %d (%s)\n", memcg_id, TARGET_CGROUP);
 	printf("aging generation %d -> %d\n",
 	       START_GEN, END_GEN);
 
 	fd = open(LRU_GEN, O_WRONLY);
 	if (fd < 0) {
 		perror("open lru_gen");
 		return 1;
 	}
 
 	for (int gen = START_GEN; gen <= END_GEN; gen++) {
 		char buf[128];
 		int len;
 		struct timespec start, end;
 		long long ns;
 
 		len = snprintf(buf, sizeof(buf),
 			       "+ %d 0 %d\n", memcg_id, gen);
 
 		clock_gettime(CLOCK_MONOTONIC, &start);
 
 		if (write(fd, buf, len) != len) {
 			perror("write lru_gen");
 			close(fd);
 			return 1;
 		}
 
 		clock_gettime(CLOCK_MONOTONIC, &end);
 
 		ns = nsec_diff(&start, &end);
 		total_ns += ns;
 
 		printf("gen %3d: %8.3f ms\n",
 		       gen, ns / 1000000.0);
 		fflush(stdout);
 	}
 
 	close(fd);
 
 	printf("\nTotal:   %.3f ms\n",
 	       total_ns / 1000000.0);
 	printf("Average: %.3f ms\n",
 	       total_ns / (double)(END_GEN - START_GEN + 1) /
 	       1000000.0);
 
 	while (1)
 		sleep(1);
 
 	return 0;
 }

Run the above microbenchmark with:
systemd-run --scope --unit=agetest -p MemoryMax=1024M ./agetest

I’m seeing inc_min_seq() become significantly faster:

W/o patch:

Running scope as unit: agetest.scope
mmap: 0x72c1b5a00000, size: 512 MB
memcg: 12673 (/system.slice/agetest.scope)
aging generation 3 -> 103
gen   3:    7.433 ms
gen   4:    0.949 ms
gen   5:    2.535 ms
gen   6:    5.043 ms
gen   7:    5.041 ms
gen   8:    5.027 ms
...
gen 100:    5.035 ms
gen 101:    5.011 ms
gen 102:    5.029 ms
gen 103:    5.056 ms

Total:   503.946 ms
Average: 4.990 ms

W/ patch:

Running scope as unit: agetest.scope
mmap: 0x7c4b1d200000, size: 512 MB
memcg: 12893 (/system.slice/agetest.scope)
aging generation 3 -> 103
gen   3:    7.538 ms
gen   4:    0.937 ms
gen   5:    2.348 ms
gen   6:    2.300 ms
gen   7:    2.302 ms
gen   8:    2.294 ms
gen   9:    2.296 ms
...
gen 100:    2.292 ms
gen 101:    2.307 ms
gen 102:    2.293 ms
gen 103:    2.293 ms

Total:   235.718 ms
Average: 2.334 ms

The average aging time drops from 4.990 ms to 2.334 ms!

[1] https://lore.kernel.org/linux-mm/20260812121658.69965-1-baohua@kernel.org/

Barry Song (Xiaomi) (6):
  mm/mglru: batch update lrugen->nr_pages in inc_min_seq()
  mm/mglru: batch update lrugen->protected in inc_min_seq()
  mm/mglru: enhance cold/hot inversion handling in inc_min_seq()
  mm/mglru: exclude folios promoted by aging from protected in
    inc_min_seq()
  mm/mglru: move folios from oldest gen to second-oldest gen from head
    to tail
  mm/mglru: batch move folios to the second-oldest gen's LRU

 mm/vmscan.c | 98 +++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 80 insertions(+), 18 deletions(-)

-- 
2.34.1



^ permalink raw reply	[flat|nested] 9+ messages in thread
end of thread, other threads:[~2026-08-22  5:45 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-21 10:25 [PATCH 0/6] mm/mglru: speed up inc_min_seq() and fix cold/hot inversions Barry Song (Xiaomi)
2026-08-21 10:25 ` [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq() Barry Song (Xiaomi)
2026-08-22  1:42   ` Lian Wang (ProcessMission)
2026-08-21 10:25 ` [PATCH 2/6] mm/mglru: batch update lrugen->protected " Barry Song (Xiaomi)
2026-08-21 10:25 ` [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling " Barry Song (Xiaomi)
2026-08-21 10:25 ` [PATCH 4/6] mm/mglru: exclude folios promoted by aging from protected " Barry Song (Xiaomi)
2026-08-21 10:25 ` [PATCH 5/6] mm/mglru: move folios from oldest gen to second-oldest gen from head to tail Barry Song (Xiaomi)
2026-08-22  5:45   ` Kairui Song
2026-08-21 10:25 ` [PATCH 6/6] mm/mglru: batch move folios to the second-oldest gen's LRU Barry Song (Xiaomi)

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.