[PATCH 0/9] mm/mglru: suppress empty page table walks during aging

[PATCH 0/9] mm/mglru: suppress empty page table walks during aging

[Baoquan He]

In the current MGLRU, lru_gen_use_mm() will mark an mm for all nodes at
each context switch, so on each node aging will walk into each mm's page
table independently. On a multi-NUMA node system, one process launched
on one or a subset of nodes, its mm is walked by all other nodes's aging
while finds on pages for their lruvec. This is pure waste (100% walks
are empty on those other nodes)

This patch series suppresses these empty page table walks with two
complementary mechanisms at different levels:

  - empty_map (cross-node specifc). At mm granularity, one process's mm
    whose walk found no pages for one node's lruvec is skipped on that
    node between re-scan passes.

  - PUD-level Bloom filter (general). One level up from the existing PMD
    filters, it skips any 1GB PUD that had no young entries last
    generation - namely whose 512 PMDs all failed PMD test. Cross-node
    empty walks can prove its effect the best, but it also suppresses
    purely code PUD inside local mms. So it reduces unnecessary walking
    in any workload with cold areas.

The two complement each other: empty_map reduces the *number* of
cross-node walks, the PUD-level filter reduces the *cost* of the walks
that remain.

Measurement test (3-socket Xeon 6766E, 32 x 256MB workers pinned to node 0):

  node 1 (foreign mm walks):
    empty_map off (PUD filter on): 3300 walks, all empty, ~1238 PTE scans/walk
    both on:                       900 walks (-73%), ~0 PTE scans/walk

  - empty_map cuts node 1's walk count by ~73% (every 4th pass re-scans
    to close migration/fault windows).
  - the PUD-level filter makes the remaining walks nearly free: the 1GB
    subtrees are skipped instead of iterating their 512 PMDs
    (leaf_total drops from ~1238 to ~0 per walk).
  - worker RSS is unchanged in both modes - no premature reclaim. A
    force_scan pass confirms the local pages are found young and eligible
    (leaf_eligible == young == 21320 on node 0) when the filters are
    bypassed, i.e. the pages are hot and the aging walker can find them.

Performance regression test (make -j4 in a 3G memory cgroup, 4 vCPU / 8GB
2-NUMA VM, median of 3 runs):

                  baseline   patch    diff
    build time     10m01s    9m41s    -3.4%
    pgpgin        136868    138080   +0.9%
    pgmajfault      1832      1862   +1.6%

Build time, page-in and major-fault counts are within run-to-run
variance of a no-patch baseline - no measurable regression. The build
time is if anything slightly lower, consistent with the suppression
reducing reclaim overhead (fewer cross-node empty walks) during a real
memory-pressure workload.

The test codes/scripts (numa_workload + run_test_v4.sh) are available at:
https://github.com/baoquan-he/mglru-empty-walk-test

RFC-v1:
  - Bloom filter helpers are named symmetrically -
    test/update/reset_pmd_bloom_filter() alongside the PUD-level ones, and
    the struct field filters -> pmd_filters (Barry).

  - An mm is marked empty only when its page tables were actually walked
    (a failed mmap_read_trylock() or a stale seq is not empty), and the skip
    is invalidated on the major page-fault and migration paths. (Sashiko)

  - Reworked the re-scan: the old shared counter across nodes got stuck at
    the slowest node, so the skip never really engaged. Now each node re-scans
    its empty mms every N passes on its own clock. N is the skip_empty knob
    (default 4, proper read/write with input validation). (Sashiko)

  - Added measurement counters/tracepoint, and a kernel-build regression
    test (no measurable impact).


Baoquan He (9):
  mm/mglru: add MM_WALK_EMPTY stats and tracepoint
  mm/mglru: suppress cross-node empty page table walks
  mm/mglru: add debugfs knob for the empty-walk skip threshold
  mm/mglru: invalidate empty-walk skip on page fault and migration
  mm/mglru: add PUD-level Bloom filter state
  mm/mglru: refactor Bloom filter helpers for two filter levels
  mm/mglru: skip PUD subtrees during aging
  mm/mglru: report hot PUDs from the rmap feedback path
  mm/mglru: count PUD subtrees skipped by the PUD-level filter

 include/linux/mm_types.h      |  22 ++++
 include/linux/mmzone.h        |  13 ++-
 include/trace/events/vmscan.h |  30 +++++
 mm/huge_memory.c              |   3 +
 mm/memory.c                   |  15 +++
 mm/migrate.c                  |   4 +
 mm/vmscan.c                   | 205 +++++++++++++++++++++++++++++-----
 7 files changed, 264 insertions(+), 28 deletions(-)


base-commit: efecab401cb15fd3bb9bc05990609acb6b267ff2
-- 
2.54.0

[PATCH 1/9] mm/mglru: add MM_WALK_EMPTY stats and tracepoint

[Baoquan He]

Add per-walk counters to measure empty aging walks which traverse
an mm's page tables but find no folio for the current lruvec (node+memcg).
These are common on multi-NUMA node systems because lru_gen_use_mm() marks
an mm for all nodes at every context switch.

New counters (accumulated in mm_state->stats[]):

  MM_LEAF_ELIGIBLE    - folios belonging to this lruvec
  MM_WALK_TOTAL       - page-table walks completed
  MM_WALK_EMPTY       - walks that found no eligible folio
  MM_LEAF_TOTAL_EMPTY - leaf entries scanned by empty walks

A new tracepoint, mm_vmscan_lru_gen_walk(), fires after each walk, and the
debugfs lru_gen output ("TYFALWEE") exposes the new counters.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 include/linux/mmzone.h        |  4 ++++
 include/trace/events/vmscan.h | 28 ++++++++++++++++++++++++++++
 mm/vmscan.c                   | 30 ++++++++++++++++++++++++++----
 3 files changed, 58 insertions(+), 4 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 94f9c3ff5416..229d27fbfb54 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -595,6 +595,10 @@ enum {
 	MM_LEAF_YOUNG,		/* young leaf entries */
 	MM_NONLEAF_FOUND,	/* non-leaf entries found in Bloom filters */
 	MM_NONLEAF_ADDED,	/* non-leaf entries added to Bloom filters */
+	MM_LEAF_ELIGIBLE,	/* folios belonging to this lruvec (node+memcg) */
+	MM_WALK_TOTAL,		/* page-table walks completed */
+	MM_WALK_EMPTY,		/* walks that found no eligible folio */
+	MM_LEAF_TOTAL_EMPTY,	/* leaf entries scanned by empty walks */
 	NR_MM_STATS
 };
 
diff --git a/include/trace/events/vmscan.h b/include/trace/events/vmscan.h
index b4bf7b8def1f..c7c2034715b6 100644
--- a/include/trace/events/vmscan.h
+++ b/include/trace/events/vmscan.h
@@ -659,6 +659,34 @@ TRACE_EVENT(mm_vmscan_kswapd_clear_hopeless,
 		__entry->nid,
 		__print_symbolic(__entry->reason, kswapd_clear_hopeless_reason_ops))
 );
+TRACE_EVENT(mm_vmscan_lru_gen_walk,
+
+	TP_PROTO(int nid, unsigned long seq, int leaf_total,
+		 int leaf_eligible, bool empty),
+
+	TP_ARGS(nid, seq, leaf_total, leaf_eligible, empty),
+
+	TP_STRUCT__entry(
+		__field(int, nid)
+		__field(unsigned long, seq)
+		__field(int, leaf_total)
+		__field(int, leaf_eligible)
+		__field(bool, empty)
+	),
+
+	TP_fast_assign(
+		__entry->nid = nid;
+		__entry->seq = seq;
+		__entry->leaf_total = leaf_total;
+		__entry->leaf_eligible = leaf_eligible;
+		__entry->empty = empty;
+	),
+
+	TP_printk("nid=%d seq=%lu leaf_total=%d leaf_eligible=%d empty=%d",
+		__entry->nid, __entry->seq, __entry->leaf_total,
+		__entry->leaf_eligible, __entry->empty)
+);
+
 #endif /* _TRACE_VMSCAN_H */
 
 /* This part must be outside protection */
diff --git a/mm/vmscan.c b/mm/vmscan.c
index c1404a59523d..92cb83a78971 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3588,6 +3588,8 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
 		if (!folio)
 			continue;
 
+		walk->mm_stats[MM_LEAF_ELIGIBLE]++;
+
 		if (folio_test_large(folio)) {
 			const unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
 
@@ -3688,6 +3690,8 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
 		if (!folio)
 			goto next;
 
+		walk->mm_stats[MM_LEAF_ELIGIBLE]++;
+
 		if (!pmdp_test_and_clear_young_notify(vma, addr, pmd + i))
 			goto next;
 
@@ -4109,8 +4113,26 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq,
 
 	do {
 		success = iterate_mm_list(walk, &mm);
-		if (mm)
+		if (mm) {
+			bool empty = false;
+
 			walk_mm(mm, walk);
+			/* A walk that traversed page tables but found no folio
+			 * belonging to this lruvec (node+memcg) is pure waste. */
+			if (walk->mm_stats[MM_LEAF_TOTAL]) {
+				walk->mm_stats[MM_WALK_TOTAL]++;
+				if (walk->mm_stats[MM_LEAF_ELIGIBLE] == 0) {
+					walk->mm_stats[MM_WALK_EMPTY]++;
+					walk->mm_stats[MM_LEAF_TOTAL_EMPTY] +=
+						walk->mm_stats[MM_LEAF_TOTAL];
+					empty = true;
+				}
+			}
+			trace_mm_vmscan_lru_gen_walk(
+					lruvec_pgdat(lruvec)->node_id, walk->seq,
+					walk->mm_stats[MM_LEAF_TOTAL],
+					walk->mm_stats[MM_LEAF_ELIGIBLE], empty);
+		}
 	} while (mm);
 done:
 	if (success) {
@@ -5580,14 +5602,14 @@ static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruvec,
 
 	seq_puts(m, "                      ");
 	for (i = 0; i < NR_MM_STATS; i++) {
-		const char *s = "xxxx";
+		const char *s = "xxxxxxxx";
 		unsigned long n = 0;
 
 		if (seq == max_seq && NR_HIST_GENS == 1) {
-			s = "TYFA";
+			s = "TYFALWEE";
 			n = READ_ONCE(mm_state->stats[hist][i]);
 		} else if (seq != max_seq && NR_HIST_GENS > 1) {
-			s = "tyfa";
+			s = "tyfalwee";
 			n = READ_ONCE(mm_state->stats[hist][i]);
 		}
 
-- 
2.54.0

[PATCH 2/9] mm/mglru: suppress cross-node empty page table walks

[Baoquan He]

In the current MGLRU, lru_gen_use_mm() will mark one process's mm used
on all nodes at each context switch. So each nodes's aging walks into
each mm's page tables. For an mm with memory on one or only a subset of
nodes, the other nodes' walks find no pages for one lruvec. While these
empty walks are pure waste.

Track per-mm, per-node empty-walk marks: bit N on mm->lru_gen.empty_map is
set when node N's walk of the mm found no page for this lruvec, and
get_next_mm() will skip the mm on node N between re-scan passes. The re-scan
is driven by each node's own pass count (mm_state->seq), so every
mglru_empty_skip_gens-th (default 4) pass re-walks all empty-marked mms to
close migration/NUMA-balancing windows; keeping it on the node's own clock
avoids a shared "oldest marking" sequence latching at the slowest node.

A walk is "empty" when it traversed the page tables and found no folio for
this lruvec.

A page that appears on the node during the skip (fault or migration) is not
aged until the re-scan; a later patch invalidates the skip on those
paths. mm_struct grows by 8 bytes per process.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 include/linux/mm_types.h |  3 +++
 include/linux/mmzone.h   |  2 ++
 mm/vmscan.c              | 52 ++++++++++++++++++++++++++++++++++------
 3 files changed, 50 insertions(+), 7 deletions(-)

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 6d815f6440c9..3738e8877b73 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -1410,6 +1410,8 @@ struct mm_struct {
 			 * page table walkers cleared the corresponding bits.
 			 */
 			unsigned long bitmap;
+			/* bit N: node N's last walk found no folio; skip until re-scan */
+			unsigned long empty_map;
 #ifdef CONFIG_MEMCG
 			/* points to the memcg of "owner" above */
 			struct mem_cgroup *memcg;
@@ -1503,6 +1505,7 @@ static inline void lru_gen_init_mm(struct mm_struct *mm)
 {
 	INIT_LIST_HEAD(&mm->lru_gen.list);
 	mm->lru_gen.bitmap = 0;
+	mm->lru_gen.empty_map = 0;
 #ifdef CONFIG_MEMCG
 	mm->lru_gen.memcg = NULL;
 #endif
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 229d27fbfb54..30213a880db0 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -633,6 +633,8 @@ struct lru_gen_mm_walk {
 	int batched;
 	int swappiness;
 	bool force_scan;
+	/* this aging pass is an empty-walk re-scan pass (every K-th) */
+	bool rescan_pass;
 };
 
 /*
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 92cb83a78971..e8ba49683b28 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2710,6 +2710,14 @@ static bool should_clear_pmd_young(void)
 	return arch_has_hw_nonleaf_pmd_young() && get_cap(LRU_GEN_NONLEAF_YOUNG);
 }
 
+/*
+ * Cross-node empty walk suppression. lru_gen_use_mm() marks an mm used on all
+ * nodes, so aging on a node where the mm has no memory wastes a full page table
+ * walk. Skip such an mm for up to MGLRU_EMPTY_SKIP_GENS generations after an
+ * empty walk, then force-rescan to close migration/mlock/NUMA-balancing windows.
+ */
+#define MGLRU_EMPTY_SKIP_GENS 4
+
 /******************************************************************************
  *                          shorthand helpers
  ******************************************************************************/
@@ -2930,9 +2938,17 @@ static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk)
 	mm = list_entry(mm_state->head, struct mm_struct, lru_gen.list);
 	key = pgdat->node_id % BITS_PER_TYPE(mm->lru_gen.bitmap);
 
+	/* skip if not used on this node since the last walk */
 	if (!walk->force_scan && !test_bit(key, &mm->lru_gen.bitmap))
 		return NULL;
 
+	/* Skip empty-marked mms except on a re-scan pass; off on node-id alias */
+	if (!walk->force_scan &&
+	    nr_node_ids <= BITS_PER_TYPE(mm->lru_gen.bitmap) &&
+	    !walk->rescan_pass &&
+	    test_bit(key, &mm->lru_gen.empty_map))
+		return NULL;
+
 	clear_bit(key, &mm->lru_gen.bitmap);
 	mmgrab(mm);
 
@@ -3833,7 +3849,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,
 	return -EAGAIN;
 }
 
-static void walk_mm(struct mm_struct *mm, struct lru_gen_mm_walk *walk)
+static bool walk_mm(struct mm_struct *mm, struct lru_gen_mm_walk *walk)
 {
 	static const struct mm_walk_ops mm_walk_ops = {
 		.test_walk = should_skip_vma,
@@ -3841,6 +3857,7 @@ static void walk_mm(struct mm_struct *mm, struct lru_gen_mm_walk *walk)
 		.walk_lock = PGWALK_RDLOCK,
 	};
 	int err;
+	bool walked = false;
 	struct lruvec *lruvec = walk->lruvec;
 
 	walk->next_addr = FIRST_USER_ADDRESS;
@@ -3859,6 +3876,7 @@ static void walk_mm(struct mm_struct *mm, struct lru_gen_mm_walk *walk)
 			err = walk_page_range(mm, walk->next_addr, ULONG_MAX, &mm_walk_ops, walk);
 
 			mmap_read_unlock(mm);
+			walked = true;
 		}
 
 		if (walk->batched)
@@ -3866,6 +3884,9 @@ static void walk_mm(struct mm_struct *mm, struct lru_gen_mm_walk *walk)
 
 		cond_resched();
 	} while (err == -EAGAIN);
+
+	/* true only if the page tables were traversed - a failed trylock/stale seq is not */
+	return walked;
 }
 
 static struct lru_gen_mm_walk *set_mm_walk(struct pglist_data *pgdat, bool force_alloc)
@@ -4110,22 +4131,39 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq,
 	walk->seq = seq;
 	walk->swappiness = swappiness;
 	walk->force_scan = force_scan;
+	/* every skip_gens-th pass re-scans empty-marked mms; 0 disables */
+	walk->rescan_pass = mglru_empty_skip_gens == 0 ||
+			    mm_state->seq % READ_ONCE(mglru_empty_skip_gens) == 0;
 
 	do {
 		success = iterate_mm_list(walk, &mm);
 		if (mm) {
+			int nid = lruvec_pgdat(lruvec)->node_id;
+			int key = nid % BITS_PER_TYPE(mm->lru_gen.bitmap);
+			bool walked;
 			bool empty = false;
 
-			walk_mm(mm, walk);
-			/* A walk that traversed page tables but found no folio
-			 * belonging to this lruvec (node+memcg) is pure waste. */
-			if (walk->mm_stats[MM_LEAF_TOTAL]) {
+			walked = walk_mm(mm, walk);
+
+			/* Empty = walked, no eligible folio; failed trylock/stale seq is not */
+			empty = walked && walk->mm_stats[MM_LEAF_ELIGIBLE] == 0;
+			/* skip_gens == 0: stop maintaining marks */
+			if (mglru_empty_skip_gens &&
+			    nr_node_ids <= BITS_PER_TYPE(mm->lru_gen.bitmap)) {
+				if (empty)
+					set_bit(key, &mm->lru_gen.empty_map);
+				else
+					/* found eligible folios: clear the marking */
+					clear_bit(key, &mm->lru_gen.empty_map);
+			}
+
+			/* Count completed walks with the same "empty" definition */
+			if (walked) {
 				walk->mm_stats[MM_WALK_TOTAL]++;
-				if (walk->mm_stats[MM_LEAF_ELIGIBLE] == 0) {
+				if (empty) {
 					walk->mm_stats[MM_WALK_EMPTY]++;
 					walk->mm_stats[MM_LEAF_TOTAL_EMPTY] +=
 						walk->mm_stats[MM_LEAF_TOTAL];
-					empty = true;
 				}
 			}
 			trace_mm_vmscan_lru_gen_walk(
-- 
2.54.0

[PATCH 3/9] mm/mglru: add debugfs knob for the empty-walk skip threshold

[Baoquan He]

Change the hardcoded MGLRU_EMPTY_SKIP_GENS to a runtime-tunable
mglru_empty_skip_gens (default 4), set by:

  echo "skip_empty <N>" > /sys/kernel/debug/lru_gen   (0 = disable)

The current value is shown as "empty_skip" in the lru_gen read output.
N=0 disables the empty-walk skip, for easy A/B testing.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 mm/vmscan.c | 49 +++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 41 insertions(+), 8 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index e8ba49683b28..f592f04fe1bd 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2710,14 +2710,6 @@ static bool should_clear_pmd_young(void)
 	return arch_has_hw_nonleaf_pmd_young() && get_cap(LRU_GEN_NONLEAF_YOUNG);
 }
 
-/*
- * Cross-node empty walk suppression. lru_gen_use_mm() marks an mm used on all
- * nodes, so aging on a node where the mm has no memory wastes a full page table
- * walk. Skip such an mm for up to MGLRU_EMPTY_SKIP_GENS generations after an
- * empty walk, then force-rescan to close migration/mlock/NUMA-balancing windows.
- */
-#define MGLRU_EMPTY_SKIP_GENS 4
-
 /******************************************************************************
  *                          shorthand helpers
  ******************************************************************************/
@@ -2928,6 +2920,9 @@ static struct lru_gen_mm_state *get_mm_state(struct lruvec *lruvec)
 	return &lruvec->mm_state;
 }
 
+/* tunable empty-walk skip threshold; defined later, get_next_mm() needs it */
+static unsigned long mglru_empty_skip_gens __read_mostly;
+
 static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk)
 {
 	int key;
@@ -4266,6 +4261,14 @@ static bool lruvec_is_reclaimable(struct lruvec *lruvec, struct scan_control *sc
 /* to protect the working set of the last N jiffies */
 static unsigned long lru_gen_min_ttl __read_mostly;
 
+/*
+ * Skip an mm on node N between re-scan passes: every
+ * @mglru_empty_skip_gens-th aging pass the node re-scans all empty-marked
+ * mms and re-marks them if still empty. Default 4, 0 disables. Tunable via:
+ * echo "skip_empty <N>" > /sys/kernel/debug/lru_gen
+ */
+static unsigned long mglru_empty_skip_gens __read_mostly = 4;
+
 static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc)
 {
 	struct mem_cgroup *memcg;
@@ -5676,6 +5679,7 @@ static int lru_gen_seq_show(struct seq_file *m, void *v)
 			cgroup_path(memcg->css.cgroup, m->private, PATH_MAX);
 #endif
 		seq_printf(m, "memcg %llu %s\n", mem_cgroup_id(memcg), path);
+		seq_printf(m, "empty_skip %lu\n", READ_ONCE(mglru_empty_skip_gens));
 	}
 
 	seq_printf(m, " node %5d\n", nid);
@@ -5854,6 +5858,35 @@ static ssize_t lru_gen_seq_write(struct file *file, const char __user *src,
 		if (!*cur)
 			continue;
 
+		/*
+		 * set the empty-walk skip threshold: "skip_empty <N>"
+		 * (the current value is readable via /sys/kernel/debug/lru_gen)
+		 */
+		if (!strncmp(cur, "skip_empty", 10)) {
+			unsigned long val;
+
+			cur += 10;
+			/* require a space before the value: reject "skip_empty123" */
+			if (*cur && !isspace(*cur)) {
+				err = -EINVAL;
+				break;
+			}
+			cur = skip_spaces(cur);
+			if (!*cur) {
+				/* no value: read the threshold via lru_gen */
+				err = -EINVAL;
+				break;
+			}
+			/* kstrtoul() rejects negatives and trailing garbage */
+			if (kstrtoul(cur, 10, &val)) {
+				err = -EINVAL;
+				break;
+			}
+			WRITE_ONCE(mglru_empty_skip_gens, val);
+			err = 0;
+			continue;
+		}
+
 		n = sscanf(cur, "%c %llu %u %lu %n %4s %n %lu %n", &cmd, &memcg_id, &nid,
 			   &seq, &end, swap_string, &end, &opt, &end);
 		if (n < 4 || cur[end]) {
-- 
2.54.0

[PATCH 4/9] mm/mglru: invalidate empty-walk skip on page fault and migration

[Baoquan He]

empty_map skips an mm on a node for up to K generations after an empty walk.
Notify MGLRU when a page of the mm appears: set the node's bitmap
bit and clear its empty_map bit, so the next aging pass walks it again.

Major paths that map a page into the mm:
- mm/memory.c: do_anonymous_page()/finish_fault() (anon and file/COW faults),
  wp_page_copy() (anon COW), do_swap_page() (swap-in).
- mm/huge_memory.c: __do_huge_pmd_anonymous_page() (THP anon faults).
- mm/migrate.c: remove_migration_pte() (folio destination node, incl.
  NUMA-balancing migration).

The transitions are rare (the bitmap bit is usually already set and the
empty-marking bit usually clear), so each is guarded by test_bit() to avoid a
locked RMW bouncing the mm cache line on every fault; the periodic re-scan
covers any path not listed here.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 include/linux/mm_types.h | 19 +++++++++++++++++++
 mm/huge_memory.c         |  3 +++
 mm/memory.c              | 15 +++++++++++++++
 mm/migrate.c             |  4 ++++
 4 files changed, 41 insertions(+)

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 3738e8877b73..94c1d1fedc9c 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -1521,6 +1521,21 @@ static inline void lru_gen_use_mm(struct mm_struct *mm)
 	WRITE_ONCE(mm->lru_gen.bitmap, -1);
 }
 
+/*
+ * A page of this mm appeared on node @nid (fault or migration): set the
+ * node's bitmap bit and clear the empty-walk skip. Probe first - the
+ * transitions are rare and unconditional locked RMWs would bounce the
+ * mm cache line.
+ */
+static inline void lru_gen_mm_accessed(struct mm_struct *mm, int nid)
+{
+	unsigned long key = nid % BITS_PER_TYPE(mm->lru_gen.bitmap);
+
+	if (!test_bit(key, &mm->lru_gen.bitmap))
+		set_bit(key, &mm->lru_gen.bitmap);
+	if (test_bit(key, &mm->lru_gen.empty_map))
+		clear_bit(key, &mm->lru_gen.empty_map);
+}
 #else /* !CONFIG_LRU_GEN_WALKS_MMU */
 
 static inline void lru_gen_add_mm(struct mm_struct *mm)
@@ -1543,6 +1558,10 @@ static inline void lru_gen_use_mm(struct mm_struct *mm)
 {
 }
 
+static inline void lru_gen_mm_accessed(struct mm_struct *mm, int nid)
+{
+}
+
 #endif /* CONFIG_LRU_GEN_WALKS_MMU */
 
 struct vma_iterator {
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index afbb5974bd22..23ae4626a980 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1418,6 +1418,9 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf)
 		map_anon_folio_pmd_pf(folio, vmf->pmd, vma, haddr);
 		mm_inc_nr_ptes(vma->vm_mm);
 		spin_unlock(vmf->ptl);
+		/* a new THP of this mm lands on this node */
+		if (lru_gen_enabled())
+			lru_gen_mm_accessed(vma->vm_mm, folio_nid(folio));
 	}
 
 	return 0;
diff --git a/mm/memory.c b/mm/memory.c
index c54943302553..1d7ec0110ef2 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4076,6 +4076,9 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)
 		BUG_ON(unshare && pte_write(entry));
 		set_pte_at(mm, vmf->address, vmf->pte, entry);
 		update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1);
+		/* COW mapped a fresh folio of this mm on this node */
+		if (lru_gen_enabled())
+			lru_gen_mm_accessed(vma->vm_mm, folio_nid(new_folio));
 		if (old_folio) {
 			/*
 			 * Only after switching the pte to the new page may
@@ -5224,6 +5227,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 	set_ptes(vma->vm_mm, address, ptep, pte, nr_pages);
 	arch_do_swap_page_nr(vma->vm_mm, vma, address,
 			pte, pte, nr_pages);
+	/* a swapped-in page of this mm lands on this node */
+	if (lru_gen_enabled())
+		lru_gen_mm_accessed(vma->vm_mm, folio_nid(folio));
 
 	/*
 	 * Remove the swap entry and conditionally try to free up the swapcache.
@@ -5516,6 +5522,10 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 		folio_put(folio);
 		return handle_userfault(vmf, VM_UFFD_MISSING);
 	}
+	/* a new page of this mm lands on this node: invalidate any empty skip */
+	if (lru_gen_enabled())
+		lru_gen_mm_accessed(vma->vm_mm, folio_nid(folio));
+
 	map_anon_folio_pte_pf(folio, vmf->pte, vma, addr,
 			      vmf_orig_pte_uffd_wp(vmf));
 unlock:
@@ -5776,6 +5786,11 @@ vm_fault_t finish_fault(struct vm_fault *vmf)
 		page = vmf->page;
 
 	folio = page_folio(page);
+
+	/* mapping a page of this mm on this node: invalidate any empty skip */
+	if (lru_gen_enabled())
+		lru_gen_mm_accessed(vma->vm_mm, folio_nid(folio));
+
 	/*
 	 * check even for read faults because we might have lost our CoWed
 	 * page
diff --git a/mm/migrate.c b/mm/migrate.c
index 15b45832bcfa..26b8b34988d2 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -354,6 +354,10 @@ static bool remove_migration_pte(struct folio *folio,
 	struct rmap_walk_arg *rmap_walk_arg = arg;
 	DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr, PVMW_SYNC | PVMW_MIGRATION);
 
+	/* the folio ends up on folio_nid(): notify MGLRU for this mm */
+	if (lru_gen_enabled())
+		lru_gen_mm_accessed(vma->vm_mm, folio_nid(folio));
+
 	while (page_vma_mapped_walk(&pvmw)) {
 		rmap_t rmap_flags = RMAP_NONE;
 		unsigned long idx = 0;
-- 
2.54.0

[PATCH 5/9] mm/mglru: add PUD-level Bloom filter state

[Baoquan He]

Add a coarser double-buffered Bloom filters to struct lru_gen_mm_state,
one level up from the PMD-level filters. They operate at 1GB (PUD)
granularity, whose 512 PMDs would all fail the PMD-level filter - a general
suppression of unnecessary page table walks. Cross-node empty walks are the
most visible case: a foreign mm's PUDs have no young entries for this lruvec,
so the entire PMD iteration is pure waste.

No behavior change yet; the filters are populated and used by later
patches.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 include/linux/mmzone.h |  6 ++++--
 mm/vmscan.c            | 12 ++++++------
 2 files changed, 10 insertions(+), 8 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 30213a880db0..64eb010624b2 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -612,8 +612,10 @@ struct lru_gen_mm_state {
 	struct list_head *head;
 	/* where the last iteration ended before */
 	struct list_head *tail;
-	/* Bloom filters flip after each iteration */
-	unsigned long *filters[NR_BLOOM_FILTERS];
+	/* PMD-level Bloom filters flip after each iteration */
+	unsigned long *pmd_filters[NR_BLOOM_FILTERS];
+	/* PUD-level Bloom filters flip after each iteration */
+	unsigned long *pud_filters[NR_BLOOM_FILTERS];
 	/* the mm stats for debugging */
 	unsigned long stats[NR_HIST_GENS][NR_MM_STATS];
 };
diff --git a/mm/vmscan.c b/mm/vmscan.c
index f592f04fe1bd..912cbf86c7ff 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2849,7 +2849,7 @@ static bool test_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long s
 	unsigned long *filter;
 	int gen = filter_gen_from_seq(seq);
 
-	filter = READ_ONCE(mm_state->filters[gen]);
+	filter = READ_ONCE(mm_state->pmd_filters[gen]);
 	if (!filter)
 		return true;
 
@@ -2865,7 +2865,7 @@ static void update_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long
 	unsigned long *filter;
 	int gen = filter_gen_from_seq(seq);
 
-	filter = READ_ONCE(mm_state->filters[gen]);
+	filter = READ_ONCE(mm_state->pmd_filters[gen]);
 	if (!filter)
 		return;
 
@@ -2882,7 +2882,7 @@ static void reset_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long
 	unsigned long *filter;
 	int gen = filter_gen_from_seq(seq);
 
-	filter = mm_state->filters[gen];
+	filter = mm_state->pmd_filters[gen];
 	if (filter) {
 		bitmap_clear(filter, 0, BIT(BLOOM_FILTER_SHIFT));
 		return;
@@ -2890,7 +2890,7 @@ static void reset_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long
 
 	filter = bitmap_zalloc(BIT(BLOOM_FILTER_SHIFT),
 			       __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN);
-	WRITE_ONCE(mm_state->filters[gen], filter);
+	WRITE_ONCE(mm_state->pmd_filters[gen], filter);
 }
 
 /******************************************************************************
@@ -6010,8 +6010,8 @@ void lru_gen_exit_memcg(struct mem_cgroup *memcg)
 			continue;
 
 		for (i = 0; i < NR_BLOOM_FILTERS; i++) {
-			bitmap_free(mm_state->filters[i]);
-			mm_state->filters[i] = NULL;
+			bitmap_free(mm_state->pmd_filters[i]);
+			mm_state->pmd_filters[i] = NULL;
 		}
 	}
 }
-- 
2.54.0

[PATCH 6/9] mm/mglru: refactor Bloom filter helpers for two filter levels

[Baoquan He]

Split functions test/update/reset_bloom_filter() into __ prefixed helpers
that operate on a generic filters array, and wrappers
(test/update/reset_pmd_bloom_filter()) that pass PMD-level mm_state->pmd_filters
as parameter. Then PUD-level filter pair can reuse the same hash,
double-buffering and reset. no behavior change yet.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 mm/vmscan.c | 41 ++++++++++++++++++++++++++++-------------
 1 file changed, 28 insertions(+), 13 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 912cbf86c7ff..062504f287da 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2842,14 +2842,13 @@ static void get_item_key(void *item, int *key)
 	key[1] = hash >> BLOOM_FILTER_SHIFT;
 }
 
-static bool test_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq,
-			      void *item)
+static bool __test_bloom_filter(unsigned long **filters, unsigned long seq, void *item)
 {
 	int key[2];
 	unsigned long *filter;
 	int gen = filter_gen_from_seq(seq);
 
-	filter = READ_ONCE(mm_state->pmd_filters[gen]);
+	filter = READ_ONCE(filters[gen]);
 	if (!filter)
 		return true;
 
@@ -2858,14 +2857,13 @@ static bool test_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long s
 	return test_bit(key[0], filter) && test_bit(key[1], filter);
 }
 
-static void update_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq,
-				void *item)
+static void __update_bloom_filter(unsigned long **filters, unsigned long seq, void *item)
 {
 	int key[2];
 	unsigned long *filter;
 	int gen = filter_gen_from_seq(seq);
 
-	filter = READ_ONCE(mm_state->pmd_filters[gen]);
+	filter = READ_ONCE(filters[gen]);
 	if (!filter)
 		return;
 
@@ -2877,12 +2875,12 @@ static void update_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long
 		set_bit(key[1], filter);
 }
 
-static void reset_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq)
+static void __reset_bloom_filter(unsigned long **filters, unsigned long seq)
 {
 	unsigned long *filter;
 	int gen = filter_gen_from_seq(seq);
 
-	filter = mm_state->pmd_filters[gen];
+	filter = filters[gen];
 	if (filter) {
 		bitmap_clear(filter, 0, BIT(BLOOM_FILTER_SHIFT));
 		return;
@@ -2890,7 +2888,24 @@ static void reset_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long
 
 	filter = bitmap_zalloc(BIT(BLOOM_FILTER_SHIFT),
 			       __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN);
-	WRITE_ONCE(mm_state->pmd_filters[gen], filter);
+	WRITE_ONCE(filters[gen], filter);
+}
+
+static bool test_pmd_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq,
+			      void *item)
+{
+	return __test_bloom_filter(mm_state->pmd_filters, seq, item);
+}
+
+static void update_pmd_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq,
+				void *item)
+{
+	__update_bloom_filter(mm_state->pmd_filters, seq, item);
+}
+
+static void reset_pmd_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq)
+{
+	__reset_bloom_filter(mm_state->pmd_filters, seq);
 }
 
 /******************************************************************************
@@ -3144,7 +3159,7 @@ static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite
 	spin_unlock(&mm_list->lock);
 
 	if (mm && first)
-		reset_bloom_filter(mm_state, walk->seq + 1);
+		reset_pmd_bloom_filter(mm_state, walk->seq + 1);
 
 	if (*iter)
 		mmdrop(*iter);
@@ -3782,7 +3797,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
 			walk_pmd_range_locked(pud, addr, vma, args, bitmap, &first);
 		}
 
-		if (!walk->force_scan && !test_bloom_filter(mm_state, walk->seq, pmd + i))
+		if (!walk->force_scan && !test_pmd_bloom_filter(mm_state, walk->seq, pmd + i))
 			continue;
 
 		walk->mm_stats[MM_NONLEAF_FOUND]++;
@@ -3793,7 +3808,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
 		walk->mm_stats[MM_NONLEAF_ADDED]++;
 
 		/* carry over to the next generation */
-		update_bloom_filter(mm_state, walk->seq + 1, pmd + i);
+		update_pmd_bloom_filter(mm_state, walk->seq + 1, pmd + i);
 	}
 
 	walk_pmd_range_locked(pud, -1, vma, args, bitmap, &first);
@@ -4422,7 +4437,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 
 	/* feedback from rmap walkers to page table walkers */
 	if (mm_state && suitable_to_scan(i, young))
-		update_bloom_filter(mm_state, max_seq, pvmw->pmd);
+		update_pmd_bloom_filter(mm_state, max_seq, pvmw->pmd);
 
 	mem_cgroup_put(memcg);
 
-- 
2.54.0

[PATCH 7/9] mm/mglru: skip PUD subtrees during aging

[Baoquan He]

The aging walk into present PUD and iterates all its 512 PMDs, testing the
PMD-level Bloom filter on each. Add a coarser PUD-level filter
(pud_filters) one level up:
 - walk_pmd_range() now reports whether it found any young leaf entries,
 - and walk_pud_range() records that in the PUD filter and,
 - on subsequent generations, skips the whole 1GB subtree when the filter says
   it had none last generation.

The double-buffered filter flips with each new iteration, and the existing
eviction feedback (lru_gen_look_around()) keeps hot regions marked, so newly
hot or migrated-in pages are re-checked promptly rather than suppressed
indefinitely. force_scan walks bypass the PUD test, so manual aging and newly
added mm's always rescan and re-populate the filter.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 mm/vmscan.c | 44 ++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 40 insertions(+), 4 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 062504f287da..7a6c15be1c2a 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2817,6 +2817,8 @@ static bool __maybe_unused seq_is_valid(struct lruvec *lruvec)
  * walk_pmd_range(); the eviction also report them when walking the rmap
  * in lru_gen_look_around().
  *
+ * A second, coarser pair of filters (pud_filters) sits one level up.
+ *
  * For future optimizations:
  * 1. It's not necessary to keep both filters all the time. The spare one can be
  *    freed after the RCU grace period and reallocated if needed again.
@@ -2908,6 +2910,23 @@ static void reset_pmd_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned l
 	__reset_bloom_filter(mm_state->pmd_filters, seq);
 }
 
+static bool test_pud_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq,
+				  void *item)
+{
+	return __test_bloom_filter(mm_state->pud_filters, seq, item);
+}
+
+static void update_pud_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq,
+				    void *item)
+{
+	__update_bloom_filter(mm_state->pud_filters, seq, item);
+}
+
+static void reset_pud_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq)
+{
+	__reset_bloom_filter(mm_state->pud_filters, seq);
+}
+
 /******************************************************************************
  *                          mm_struct list
  ******************************************************************************/
@@ -3158,8 +3177,10 @@ static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite
 
 	spin_unlock(&mm_list->lock);
 
-	if (mm && first)
+	if (mm && first) {
 		reset_pmd_bloom_filter(mm_state, walk->seq + 1);
+		reset_pud_bloom_filter(mm_state, walk->seq + 1);
+	}
 
 	if (*iter)
 		mmdrop(*iter);
@@ -3744,10 +3765,11 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
 	*first = -1;
 }
 
-static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
+static bool walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
 			   struct mm_walk *args)
 {
 	int i;
+	bool young = false;
 	pmd_t *pmd;
 	unsigned long next;
 	unsigned long addr;
@@ -3784,8 +3806,10 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
 
 			walk->mm_stats[MM_LEAF_TOTAL]++;
 
-			if (pfn != -1)
+			if (pfn != -1) {
 				walk_pmd_range_locked(pud, addr, vma, args, bitmap, &first);
+				young = true;
+			}
 			continue;
 		}
 
@@ -3795,6 +3819,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
 				continue;
 
 			walk_pmd_range_locked(pud, addr, vma, args, bitmap, &first);
+			young = true;
 		}
 
 		if (!walk->force_scan && !test_pmd_bloom_filter(mm_state, walk->seq, pmd + i))
@@ -3806,6 +3831,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
 			continue;
 
 		walk->mm_stats[MM_NONLEAF_ADDED]++;
+		young = true;
 
 		/* carry over to the next generation */
 		update_pmd_bloom_filter(mm_state, walk->seq + 1, pmd + i);
@@ -3815,6 +3841,8 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
 
 	if (i < PTRS_PER_PMD && get_next_vma(PUD_MASK, PMD_SIZE, args, &start, &end))
 		goto restart;
+
+	return young;
 }
 
 static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,
@@ -3825,6 +3853,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,
 	unsigned long addr;
 	unsigned long next;
 	struct lru_gen_mm_walk *walk = args->private;
+	struct lru_gen_mm_state *mm_state = get_mm_state(walk->lruvec);
 
 	VM_WARN_ON_ONCE(p4d_leaf(*p4d));
 
@@ -3838,7 +3867,12 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,
 		if (!pud_present(val) || WARN_ON_ONCE(pud_leaf(val)))
 			continue;
 
-		walk_pmd_range(&val, addr, next, args);
+		/* Skip a subtree whose 512 PMDs all failed the PMD-level filter last gen */
+		if (!walk->force_scan && !test_pud_bloom_filter(mm_state, walk->seq, pud + i))
+			continue;
+
+		if (walk_pmd_range(&val, addr, next, args))
+			update_pud_bloom_filter(mm_state, walk->seq + 1, pud + i);
 
 		if (need_resched() || walk->batched >= MAX_LRU_BATCH) {
 			end = (addr | ~PUD_MASK) + 1;
@@ -6027,6 +6061,8 @@ void lru_gen_exit_memcg(struct mem_cgroup *memcg)
 		for (i = 0; i < NR_BLOOM_FILTERS; i++) {
 			bitmap_free(mm_state->pmd_filters[i]);
 			mm_state->pmd_filters[i] = NULL;
+			bitmap_free(mm_state->pud_filters[i]);
+			mm_state->pud_filters[i] = NULL;
 		}
 	}
 }
-- 
2.54.0

[PATCH 8/9] mm/mglru: report hot PUDs from the rmap feedback path

[Baoquan He]

lru_gen_look_around() marks the PMD of a young PTE found during the eviction
rmap walk, feeding hot regions back to the aging walker. Mark the covering
PUD as well, so regions whose hotness is only observed by eviction will be
re-scanned.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 mm/vmscan.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 7a6c15be1c2a..cc41d3692bd5 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4470,8 +4470,14 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 	lazy_mmu_mode_disable();
 
 	/* feedback from rmap walkers to page table walkers */
-	if (mm_state && suitable_to_scan(i, young))
+	if (mm_state && suitable_to_scan(i, young)) {
+		/* the PUD entry covering the young PTEs scanned above */
+		pud_t *pud_p = pud_offset(p4d_offset(pgd_offset(vma->vm_mm, pvmw->address),
+						     pvmw->address), pvmw->address);
+
 		update_pmd_bloom_filter(mm_state, max_seq, pvmw->pmd);
+		update_pud_bloom_filter(mm_state, max_seq, pud_p);
+	}
 
 	mem_cgroup_put(memcg);
 
-- 
2.54.0

[PATCH 9/9] mm/mglru: count PUD subtrees skipped by the PUD-level filter

[Baoquan He]

Add MM_PUD_EMPTY_SKIPPED, incremented in walk_pud_range() whenever a PUD
subtree is skipped by the filter. Like other counters accumulated
per walk, shown in debugfs lru_gen output (the "S" column of TYFALWEES)
and via the mm_vmscan_lru_gen_walk tracepoint, so the avoided cost
by the filter can be measured directly.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 include/linux/mmzone.h        |  1 +
 include/trace/events/vmscan.h | 10 ++++++----
 mm/vmscan.c                   | 13 ++++++++-----
 3 files changed, 15 insertions(+), 9 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 64eb010624b2..a039df84f907 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -599,6 +599,7 @@ enum {
 	MM_WALK_TOTAL,		/* page-table walks completed */
 	MM_WALK_EMPTY,		/* walks that found no eligible folio */
 	MM_LEAF_TOTAL_EMPTY,	/* leaf entries scanned by empty walks */
+	MM_PUD_EMPTY_SKIPPED,	/* PUD subtrees skipped by the PUD-level filter */
 	NR_MM_STATS
 };
 
diff --git a/include/trace/events/vmscan.h b/include/trace/events/vmscan.h
index c7c2034715b6..923fbdb20de6 100644
--- a/include/trace/events/vmscan.h
+++ b/include/trace/events/vmscan.h
@@ -662,9 +662,9 @@ TRACE_EVENT(mm_vmscan_kswapd_clear_hopeless,
 TRACE_EVENT(mm_vmscan_lru_gen_walk,
 
 	TP_PROTO(int nid, unsigned long seq, int leaf_total,
-		 int leaf_eligible, bool empty),
+		 int leaf_eligible, bool empty, int pud_skipped),
 
-	TP_ARGS(nid, seq, leaf_total, leaf_eligible, empty),
+	TP_ARGS(nid, seq, leaf_total, leaf_eligible, empty, pud_skipped),
 
 	TP_STRUCT__entry(
 		__field(int, nid)
@@ -672,6 +672,7 @@ TRACE_EVENT(mm_vmscan_lru_gen_walk,
 		__field(int, leaf_total)
 		__field(int, leaf_eligible)
 		__field(bool, empty)
+		__field(int, pud_skipped)
 	),
 
 	TP_fast_assign(
@@ -680,11 +681,12 @@ TRACE_EVENT(mm_vmscan_lru_gen_walk,
 		__entry->leaf_total = leaf_total;
 		__entry->leaf_eligible = leaf_eligible;
 		__entry->empty = empty;
+		__entry->pud_skipped = pud_skipped;
 	),
 
-	TP_printk("nid=%d seq=%lu leaf_total=%d leaf_eligible=%d empty=%d",
+	TP_printk("nid=%d seq=%lu leaf_total=%d leaf_eligible=%d empty=%d pud_skipped=%d",
 		__entry->nid, __entry->seq, __entry->leaf_total,
-		__entry->leaf_eligible, __entry->empty)
+		__entry->leaf_eligible, __entry->empty, __entry->pud_skipped)
 );
 
 #endif /* _TRACE_VMSCAN_H */
diff --git a/mm/vmscan.c b/mm/vmscan.c
index cc41d3692bd5..fa437b187a05 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3868,8 +3868,10 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,
 			continue;
 
 		/* Skip a subtree whose 512 PMDs all failed the PMD-level filter last gen */
-		if (!walk->force_scan && !test_pud_bloom_filter(mm_state, walk->seq, pud + i))
+		if (!walk->force_scan && !test_pud_bloom_filter(mm_state, walk->seq, pud + i)) {
+			walk->mm_stats[MM_PUD_EMPTY_SKIPPED]++;
 			continue;
+		}
 
 		if (walk_pmd_range(&val, addr, next, args))
 			update_pud_bloom_filter(mm_state, walk->seq + 1, pud + i);
@@ -4213,7 +4215,8 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq,
 			trace_mm_vmscan_lru_gen_walk(
 					lruvec_pgdat(lruvec)->node_id, walk->seq,
 					walk->mm_stats[MM_LEAF_TOTAL],
-					walk->mm_stats[MM_LEAF_ELIGIBLE], empty);
+					walk->mm_stats[MM_LEAF_ELIGIBLE], empty,
+					walk->mm_stats[MM_PUD_EMPTY_SKIPPED]);
 		}
 	} while (mm);
 done:
@@ -5698,14 +5701,14 @@ static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruvec,
 
 	seq_puts(m, "                      ");
 	for (i = 0; i < NR_MM_STATS; i++) {
-		const char *s = "xxxxxxxx";
+		const char *s = "xxxxxxxxx";
 		unsigned long n = 0;
 
 		if (seq == max_seq && NR_HIST_GENS == 1) {
-			s = "TYFALWEE";
+			s = "TYFALWEES";
 			n = READ_ONCE(mm_state->stats[hist][i]);
 		} else if (seq != max_seq && NR_HIST_GENS > 1) {
-			s = "tyfalwee";
+			s = "tyfalwees";
 			n = READ_ONCE(mm_state->stats[hist][i]);
 		}
 
-- 
2.54.0

Re: [PATCH 0/9] mm/mglru: suppress empty page table walks during aging

[Baoquan He]

On 08/24/26 at 03:37pm, Baoquan He wrote:
> In the current MGLRU, lru_gen_use_mm() will mark an mm for all nodes at
> each context switch, so on each node aging will walk into each mm's page
> table independently. On a multi-NUMA node system, one process launched
> on one or a subset of nodes, its mm is walked by all other nodes's aging
> while finds on pages for their lruvec. This is pure waste (100% walks
              ^ s/on/no/, typo
> are empty on those other nodes)
> 
> This patch series suppresses these empty page table walks with two
> complementary mechanisms at different levels:
> 
>   - empty_map (cross-node specifc). At mm granularity, one process's mm
>     whose walk found no pages for one node's lruvec is skipped on that
>     node between re-scan passes.
> 
>   - PUD-level Bloom filter (general). One level up from the existing PMD
>     filters, it skips any 1GB PUD that had no young entries last
>     generation - namely whose 512 PMDs all failed PMD test. Cross-node
>     empty walks can prove its effect the best, but it also suppresses
>     purely code PUD inside local mms. So it reduces unnecessary walking
>     in any workload with cold areas.
> 
> The two complement each other: empty_map reduces the *number* of
> cross-node walks, the PUD-level filter reduces the *cost* of the walks
> that remain.
> 
> Measurement test (3-socket Xeon 6766E, 32 x 256MB workers pinned to node 0):
> 
>   node 1 (foreign mm walks):
>     empty_map off (PUD filter on): 3300 walks, all empty, ~1238 PTE scans/walk
>     both on:                       900 walks (-73%), ~0 PTE scans/walk
> 
>   - empty_map cuts node 1's walk count by ~73% (every 4th pass re-scans
>     to close migration/fault windows).
>   - the PUD-level filter makes the remaining walks nearly free: the 1GB
>     subtrees are skipped instead of iterating their 512 PMDs
>     (leaf_total drops from ~1238 to ~0 per walk).
>   - worker RSS is unchanged in both modes - no premature reclaim. A
>     force_scan pass confirms the local pages are found young and eligible
>     (leaf_eligible == young == 21320 on node 0) when the filters are
>     bypassed, i.e. the pages are hot and the aging walker can find them.
> 
> Performance regression test (make -j4 in a 3G memory cgroup, 4 vCPU / 8GB
> 2-NUMA VM, median of 3 runs):
> 
>                   baseline   patch    diff
>     build time     10m01s    9m41s    -3.4%
>     pgpgin        136868    138080   +0.9%
>     pgmajfault      1832      1862   +1.6%
> 
> Build time, page-in and major-fault counts are within run-to-run
> variance of a no-patch baseline - no measurable regression. The build
> time is if anything slightly lower, consistent with the suppression
> reducing reclaim overhead (fewer cross-node empty walks) during a real
> memory-pressure workload.
> 
> The test codes/scripts (numa_workload + run_test_v4.sh) are available at:
> https://github.com/baoquan-he/mglru-empty-walk-test
> 
> RFC-v1:
>   - Bloom filter helpers are named symmetrically -
>     test/update/reset_pmd_bloom_filter() alongside the PUD-level ones, and
>     the struct field filters -> pmd_filters (Barry).
> 
>   - An mm is marked empty only when its page tables were actually walked
>     (a failed mmap_read_trylock() or a stale seq is not empty), and the skip
>     is invalidated on the major page-fault and migration paths. (Sashiko)
> 
>   - Reworked the re-scan: the old shared counter across nodes got stuck at
>     the slowest node, so the skip never really engaged. Now each node re-scans
>     its empty mms every N passes on its own clock. N is the skip_empty knob
>     (default 4, proper read/write with input validation). (Sashiko)
> 
>   - Added measurement counters/tracepoint, and a kernel-build regression
>     test (no measurable impact).
> 
> 
> Baoquan He (9):
>   mm/mglru: add MM_WALK_EMPTY stats and tracepoint
>   mm/mglru: suppress cross-node empty page table walks
>   mm/mglru: add debugfs knob for the empty-walk skip threshold
>   mm/mglru: invalidate empty-walk skip on page fault and migration
>   mm/mglru: add PUD-level Bloom filter state
>   mm/mglru: refactor Bloom filter helpers for two filter levels
>   mm/mglru: skip PUD subtrees during aging
>   mm/mglru: report hot PUDs from the rmap feedback path
>   mm/mglru: count PUD subtrees skipped by the PUD-level filter
> 
>  include/linux/mm_types.h      |  22 ++++
>  include/linux/mmzone.h        |  13 ++-
>  include/trace/events/vmscan.h |  30 +++++
>  mm/huge_memory.c              |   3 +
>  mm/memory.c                   |  15 +++
>  mm/migrate.c                  |   4 +
>  mm/vmscan.c                   | 205 +++++++++++++++++++++++++++++-----
>  7 files changed, 264 insertions(+), 28 deletions(-)
> 
> 
> base-commit: efecab401cb15fd3bb9bc05990609acb6b267ff2
> -- 
> 2.54.0
> 
> 

Re: [PATCH 0/9] mm/mglru: suppress empty page table walks during aging

[Baoquan He]

On 08/24/26 at 03:37pm, Baoquan He wrote:
> In the current MGLRU, lru_gen_use_mm() will mark an mm for all nodes at
> each context switch, so on each node aging will walk into each mm's page
> table independently. On a multi-NUMA node system, one process launched
> on one or a subset of nodes, its mm is walked by all other nodes's aging
> while finds on pages for their lruvec. This is pure waste (100% walks
> are empty on those other nodes)
> 
> This patch series suppresses these empty page table walks with two
> complementary mechanisms at different levels:
> 
>   - empty_map (cross-node specifc). At mm granularity, one process's mm
>     whose walk found no pages for one node's lruvec is skipped on that
>     node between re-scan passes.
> 
>   - PUD-level Bloom filter (general). One level up from the existing PMD
>     filters, it skips any 1GB PUD that had no young entries last
>     generation - namely whose 512 PMDs all failed PMD test. Cross-node
>     empty walks can prove its effect the best, but it also suppresses
>     purely code PUD inside local mms. So it reduces unnecessary walking
>     in any workload with cold areas.
> 
> The two complement each other: empty_map reduces the *number* of
> cross-node walks, the PUD-level filter reduces the *cost* of the walks
> that remain.
> 
> Measurement test (3-socket Xeon 6766E, 32 x 256MB workers pinned to node 0):
> 
>   node 1 (foreign mm walks):
>     empty_map off (PUD filter on): 3300 walks, all empty, ~1238 PTE scans/walk
>     both on:                       900 walks (-73%), ~0 PTE scans/walk

node 1 (foreign mm walks):
  no (PUD filter|empty_map) (baseline): 3300 walks, all empty, ~1238 PTE scans/walk
  empty_map off (PUD filter on):        3300 walks, all empty, ~0 PTE scans/walk
  both on:                              900 walks (-73%), ~0 PTE scans/walk

Sorry, double check here and found I pasted the wrong content. The above
table is the right one. The 1st case means on baseline kernel, no pud
filter and no empty_map, on node 1 3300 walks are all empty walks, and
each walk scans ~1238 PTE.

> 
>   - empty_map cuts node 1's walk count by ~73% (every 4th pass re-scans
>     to close migration/fault windows).
>   - the PUD-level filter makes the remaining walks nearly free: the 1GB
>     subtrees are skipped instead of iterating their 512 PMDs
>     (leaf_total drops from ~1238 to ~0 per walk).
>   - worker RSS is unchanged in both modes - no premature reclaim. A
>     force_scan pass confirms the local pages are found young and eligible
>     (leaf_eligible == young == 21320 on node 0) when the filters are
>     bypassed, i.e. the pages are hot and the aging walker can find them.
> 
> Performance regression test (make -j4 in a 3G memory cgroup, 4 vCPU / 8GB
> 2-NUMA VM, median of 3 runs):
> 
>                   baseline   patch    diff
>     build time     10m01s    9m41s    -3.4%
>     pgpgin        136868    138080   +0.9%
>     pgmajfault      1832      1862   +1.6%
> 
> Build time, page-in and major-fault counts are within run-to-run
> variance of a no-patch baseline - no measurable regression. The build
> time is if anything slightly lower, consistent with the suppression
> reducing reclaim overhead (fewer cross-node empty walks) during a real
> memory-pressure workload.
> 
> The test codes/scripts (numa_workload + run_test_v4.sh) are available at:
> https://github.com/baoquan-he/mglru-empty-walk-test
> 
> RFC-v1:
>   - Bloom filter helpers are named symmetrically -
>     test/update/reset_pmd_bloom_filter() alongside the PUD-level ones, and
>     the struct field filters -> pmd_filters (Barry).
> 
>   - An mm is marked empty only when its page tables were actually walked
>     (a failed mmap_read_trylock() or a stale seq is not empty), and the skip
>     is invalidated on the major page-fault and migration paths. (Sashiko)
> 
>   - Reworked the re-scan: the old shared counter across nodes got stuck at
>     the slowest node, so the skip never really engaged. Now each node re-scans
>     its empty mms every N passes on its own clock. N is the skip_empty knob
>     (default 4, proper read/write with input validation). (Sashiko)
> 
>   - Added measurement counters/tracepoint, and a kernel-build regression
>     test (no measurable impact).
> 
> 
> Baoquan He (9):
>   mm/mglru: add MM_WALK_EMPTY stats and tracepoint
>   mm/mglru: suppress cross-node empty page table walks
>   mm/mglru: add debugfs knob for the empty-walk skip threshold
>   mm/mglru: invalidate empty-walk skip on page fault and migration
>   mm/mglru: add PUD-level Bloom filter state
>   mm/mglru: refactor Bloom filter helpers for two filter levels
>   mm/mglru: skip PUD subtrees during aging
>   mm/mglru: report hot PUDs from the rmap feedback path
>   mm/mglru: count PUD subtrees skipped by the PUD-level filter
> 
>  include/linux/mm_types.h      |  22 ++++
>  include/linux/mmzone.h        |  13 ++-
>  include/trace/events/vmscan.h |  30 +++++
>  mm/huge_memory.c              |   3 +
>  mm/memory.c                   |  15 +++
>  mm/migrate.c                  |   4 +
>  mm/vmscan.c                   | 205 +++++++++++++++++++++++++++++-----
>  7 files changed, 264 insertions(+), 28 deletions(-)
> 
> 
> base-commit: efecab401cb15fd3bb9bc05990609acb6b267ff2
> -- 
> 2.54.0
> 
> 

Re: [PATCH 1/9] mm/mglru: add MM_WALK_EMPTY stats and tracepoint

[Barry Song]

On Mon, Aug 24, 2026 at 3:38 PM Baoquan He <hebaoquan@kylinos.cn> wrote:
>
> Add per-walk counters to measure empty aging walks which traverse
> an mm's page tables but find no folio for the current lruvec (node+memcg).
> These are common on multi-NUMA node systems because lru_gen_use_mm() marks
> an mm for all nodes at every context switch.
>
> New counters (accumulated in mm_state->stats[]):
>
>   MM_LEAF_ELIGIBLE    - folios belonging to this lruvec
>   MM_WALK_TOTAL       - page-table walks completed
>   MM_WALK_EMPTY       - walks that found no eligible folio
>   MM_LEAF_TOTAL_EMPTY - leaf entries scanned by empty walks

Hi Baoquan,

I'm having a hard time understanding what MM_LEAF_ELIGIBLE means.
In particular, I'm not sure why "eligible" refers to folios here.

I'm also finding MM_LEAF_TOTAL_EMPTY quite difficult to understand
without looking at the implementation.

Could we come up with clearer and more descriptive names for these
counters?  might be?

MM_LEAF_ASSOCIATED,          /* folios associated with this lruvec */
MM_WALK_TOTAL,               /* completed page-table walks */
MM_WALK_WITHOUT_ASSOCIATED,  /* walks with no associated folio */
MM_LEAF_WITHOUT_ASSOCIATED,  /* leaf entries in such walks */

>
> A new tracepoint, mm_vmscan_lru_gen_walk(), fires after each walk, and the
> debugfs lru_gen output ("TYFALWEE") exposes the new counters.
>
> Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
> ---
>  include/linux/mmzone.h        |  4 ++++
>  include/trace/events/vmscan.h | 28 ++++++++++++++++++++++++++++
>  mm/vmscan.c                   | 30 ++++++++++++++++++++++++++----
>  3 files changed, 58 insertions(+), 4 deletions(-)
>
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index 94f9c3ff5416..229d27fbfb54 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -595,6 +595,10 @@ enum {
>         MM_LEAF_YOUNG,          /* young leaf entries */
>         MM_NONLEAF_FOUND,       /* non-leaf entries found in Bloom filters */
>         MM_NONLEAF_ADDED,       /* non-leaf entries added to Bloom filters */
> +       MM_LEAF_ELIGIBLE,       /* folios belonging to this lruvec (node+memcg) */
> +       MM_WALK_TOTAL,          /* page-table walks completed */
> +       MM_WALK_EMPTY,          /* walks that found no eligible folio */
> +       MM_LEAF_TOTAL_EMPTY,    /* leaf entries scanned by empty walks */
>         NR_MM_STATS
>  };
>

Best Regards
Barry

Re: [PATCH 0/9] mm/mglru: suppress empty page table walks during aging

[Barry Song]

On Mon, Aug 24, 2026 at 3:38 PM Baoquan He <hebaoquan@kylinos.cn> wrote:
>
> In the current MGLRU, lru_gen_use_mm() will mark an mm for all nodes at
> each context switch, so on each node aging will walk into each mm's page
> table independently. On a multi-NUMA node system, one process launched
> on one or a subset of nodes, its mm is walked by all other nodes's aging
> while finds on pages for their lruvec. This is pure waste (100% walks

I assume you mean “finds no pages” rather than “find on pages.”

> are empty on those other nodes)
>
> This patch series suppresses these empty page table walks with two
> complementary mechanisms at different levels:
>
>   - empty_map (cross-node specifc). At mm granularity, one process's mm
>     whose walk found no pages for one node's lruvec is skipped on that
>     node between re-scan passes.
>
>   - PUD-level Bloom filter (general). One level up from the existing PMD
>     filters, it skips any 1GB PUD that had no young entries last
>     generation - namely whose 512 PMDs all failed PMD test. Cross-node
>     empty walks can prove its effect the best, but it also suppresses
>     purely code PUD inside local mms. So it reduces unnecessary walking
>     in any workload with cold areas.

I assume you mean “purely cold PUD” rather than “purely code PUD”.

>
> The two complement each other: empty_map reduces the *number* of
> cross-node walks, the PUD-level filter reduces the *cost* of the walks
> that remain.

I wonder if the PUD-level filter alone would achieve your goal without
requiring the more complex `empty_map` code.

PUD is already a fairly large granularity, so even if we still do some
redundant work, I wonder if the overhead would be small enough to make
`empty_map` unnecessary?

Best Regards
Barry

Re: [PATCH 2/9] mm/mglru: suppress cross-node empty page table walks

[Barry Song]

On Mon, Aug 24, 2026 at 3:38 PM Baoquan He <hebaoquan@kylinos.cn> wrote:
>
> In the current MGLRU, lru_gen_use_mm() will mark one process's mm used
> on all nodes at each context switch. So each nodes's aging walks into
> each mm's page tables. For an mm with memory on one or only a subset of
> nodes, the other nodes' walks find no pages for one lruvec. While these
> empty walks are pure waste.
>
> Track per-mm, per-node empty-walk marks: bit N on mm->lru_gen.empty_map is
> set when node N's walk of the mm found no page for this lruvec, and
> get_next_mm() will skip the mm on node N between re-scan passes. The re-scan
> is driven by each node's own pass count (mm_state->seq), so every
> mglru_empty_skip_gens-th (default 4) pass re-walks all empty-marked mms to
> close migration/NUMA-balancing windows; keeping it on the node's own clock
> avoids a shared "oldest marking" sequence latching at the slowest node.
>
> A walk is "empty" when it traversed the page tables and found no folio for
> this lruvec.
>
> A page that appears on the node during the skip (fault or migration) is not
> aged until the re-scan; a later patch invalidates the skip on those
> paths. mm_struct grows by 8 bytes per process.

Hi Baoquan,

As mentioned in my reply to the cover letter, I wonder if this could
be achieved by the PUD filter instead.

For example, if we find no associated folios in a PUD, could we simply
filter out the entire PUD?

BTW, is this related to memory policies such as `MPOL_BIND`? If so,
could we inspect the mempolicy to avoid these empty walks in the first
place?

I'm not quite sure what the best solution is. My gut feeling is that
`empty_map` adds quite a bit of complexity, so I'd like to explore
whether there are alternative ways to avoid the extra code and the
additional space in `mm_struct` before going with this approach.

So far, I'm not really against `empty_map`; I'm just trying to get a
better understanding of it and explore whether there are simpler
alternatives.

I mean, I really like your PUD filter, but I'm not quite as fond of
the `empty_map` approach. :-)

>
> Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
> ---
>  include/linux/mm_types.h |  3 +++
>  include/linux/mmzone.h   |  2 ++
>  mm/vmscan.c              | 52 ++++++++++++++++++++++++++++++++++------
>  3 files changed, 50 insertions(+), 7 deletions(-)
>
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index 6d815f6440c9..3738e8877b73 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -1410,6 +1410,8 @@ struct mm_struct {
>                          * page table walkers cleared the corresponding bits.
>                          */
>                         unsigned long bitmap;
> +                       /* bit N: node N's last walk found no folio; skip until re-scan */
> +                       unsigned long empty_map;
>  #ifdef CONFIG_MEMCG
>                         /* points to the memcg of "owner" above */
>                         struct mem_cgroup *memcg;
> @@ -1503,6 +1505,7 @@ static inline void lru_gen_init_mm(struct mm_struct *mm)
>  {
>         INIT_LIST_HEAD(&mm->lru_gen.list);
>         mm->lru_gen.bitmap = 0;
> +       mm->lru_gen.empty_map = 0;
>  #ifdef CONFIG_MEMCG
>         mm->lru_gen.memcg = NULL;
>  #endif
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index 229d27fbfb54..30213a880db0 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -633,6 +633,8 @@ struct lru_gen_mm_walk {
>         int batched;
>         int swappiness;
>         bool force_scan;
> +       /* this aging pass is an empty-walk re-scan pass (every K-th) */
> +       bool rescan_pass;
>  };
>
>  /*
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 92cb83a78971..e8ba49683b28 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -2710,6 +2710,14 @@ static bool should_clear_pmd_young(void)
>         return arch_has_hw_nonleaf_pmd_young() && get_cap(LRU_GEN_NONLEAF_YOUNG);
>  }
>
> +/*
> + * Cross-node empty walk suppression. lru_gen_use_mm() marks an mm used on all
> + * nodes, so aging on a node where the mm has no memory wastes a full page table
> + * walk. Skip such an mm for up to MGLRU_EMPTY_SKIP_GENS generations after an
> + * empty walk, then force-rescan to close migration/mlock/NUMA-balancing windows.
> + */
> +#define MGLRU_EMPTY_SKIP_GENS 4

Is this related to `MAX_NR_GENS`? Does that mean that, over a full
aging cycle, we have a `1 / MAX_NR_GENS` chance of doing a rescan?

Best Regards
Barry

Re: [PATCH 2/9] mm/mglru: suppress cross-node empty page table walks

[Baoquan He]

On 08/28/26 at 02:35pm, Barry Song wrote:
> On Mon, Aug 24, 2026 at 3:38 PM Baoquan He <hebaoquan@kylinos.cn> wrote:
> >
> > In the current MGLRU, lru_gen_use_mm() will mark one process's mm used
> > on all nodes at each context switch. So each nodes's aging walks into
> > each mm's page tables. For an mm with memory on one or only a subset of
> > nodes, the other nodes' walks find no pages for one lruvec. While these
> > empty walks are pure waste.
> >
> > Track per-mm, per-node empty-walk marks: bit N on mm->lru_gen.empty_map is
> > set when node N's walk of the mm found no page for this lruvec, and
> > get_next_mm() will skip the mm on node N between re-scan passes. The re-scan
> > is driven by each node's own pass count (mm_state->seq), so every
> > mglru_empty_skip_gens-th (default 4) pass re-walks all empty-marked mms to
> > close migration/NUMA-balancing windows; keeping it on the node's own clock
> > avoids a shared "oldest marking" sequence latching at the slowest node.
> >
> > A walk is "empty" when it traversed the page tables and found no folio for
> > this lruvec.
> >
> > A page that appears on the node during the skip (fault or migration) is not
> > aged until the re-scan; a later patch invalidates the skip on those
> > paths. mm_struct grows by 8 bytes per process.
> 
> Hi Baoquan,
> 
> As mentioned in my reply to the cover letter, I wonder if this could
> be achieved by the PUD filter instead.
> 
> For example, if we find no associated folios in a PUD, could we simply
> filter out the entire PUD?
> 
> BTW, is this related to memory policies such as `MPOL_BIND`? If so,
> could we inspect the mempolicy to avoid these empty walks in the first
> place?
> 
> I'm not quite sure what the best solution is. My gut feeling is that
> `empty_map` adds quite a bit of complexity, so I'd like to explore
> whether there are alternative ways to avoid the extra code and the
> additional space in `mm_struct` before going with this approach.
> 
> So far, I'm not really against `empty_map`; I'm just trying to get a
> better understanding of it and explore whether there are simpler
> alternatives.
> 
> I mean, I really like your PUD filter, but I'm not quite as fond of
> the `empty_map` approach. :-)

Thanks a lot for your careful reviewing, Barry, really appreciated.

I totally understand your preference. When I found the defect of
mm->lru_gen.bitmap, I was also very hesitant about the final solution.
empty_map is the specific solution, PUD filter is a generic one
while benefit mm->lru_gen.bitmap too. I agree with you that PUD filter
is good enough to resolve the defect of mm->lru_gen.bitmap, and benefit
even non-NUMA systems. Maybe in future when huge system RAM, e.g several
TeraBytes of memory becomre normal, we can come back to consider adding
the empty_map solution.

That said, I will drop the empty_map related code changes and post v2.

> 
> >
> > Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
> > ---
> >  include/linux/mm_types.h |  3 +++
> >  include/linux/mmzone.h   |  2 ++
> >  mm/vmscan.c              | 52 ++++++++++++++++++++++++++++++++++------
> >  3 files changed, 50 insertions(+), 7 deletions(-)
> >
> > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> > index 6d815f6440c9..3738e8877b73 100644
> > --- a/include/linux/mm_types.h
> > +++ b/include/linux/mm_types.h
> > @@ -1410,6 +1410,8 @@ struct mm_struct {
> >                          * page table walkers cleared the corresponding bits.
> >                          */
> >                         unsigned long bitmap;
> > +                       /* bit N: node N's last walk found no folio; skip until re-scan */
> > +                       unsigned long empty_map;
> >  #ifdef CONFIG_MEMCG
> >                         /* points to the memcg of "owner" above */
> >                         struct mem_cgroup *memcg;
> > @@ -1503,6 +1505,7 @@ static inline void lru_gen_init_mm(struct mm_struct *mm)
> >  {
> >         INIT_LIST_HEAD(&mm->lru_gen.list);
> >         mm->lru_gen.bitmap = 0;
> > +       mm->lru_gen.empty_map = 0;
> >  #ifdef CONFIG_MEMCG
> >         mm->lru_gen.memcg = NULL;
> >  #endif
> > diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> > index 229d27fbfb54..30213a880db0 100644
> > --- a/include/linux/mmzone.h
> > +++ b/include/linux/mmzone.h
> > @@ -633,6 +633,8 @@ struct lru_gen_mm_walk {
> >         int batched;
> >         int swappiness;
> >         bool force_scan;
> > +       /* this aging pass is an empty-walk re-scan pass (every K-th) */
> > +       bool rescan_pass;
> >  };
> >
> >  /*
> > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > index 92cb83a78971..e8ba49683b28 100644
> > --- a/mm/vmscan.c
> > +++ b/mm/vmscan.c
> > @@ -2710,6 +2710,14 @@ static bool should_clear_pmd_young(void)
> >         return arch_has_hw_nonleaf_pmd_young() && get_cap(LRU_GEN_NONLEAF_YOUNG);
> >  }
> >
> > +/*
> > + * Cross-node empty walk suppression. lru_gen_use_mm() marks an mm used on all
> > + * nodes, so aging on a node where the mm has no memory wastes a full page table
> > + * walk. Skip such an mm for up to MGLRU_EMPTY_SKIP_GENS generations after an
> > + * empty walk, then force-rescan to close migration/mlock/NUMA-balancing windows.
> > + */
> > +#define MGLRU_EMPTY_SKIP_GENS 4
> 
> Is this related to `MAX_NR_GENS`? Does that mean that, over a full
> aging cycle, we have a `1 / MAX_NR_GENS` chance of doing a rescan?
> 
> Best Regards
> Barry
> 

Re: [PATCH 0/9] mm/mglru: suppress empty page table walks during aging

[Baoquan He]

On 08/28/26 at 02:11pm, Barry Song wrote:
> On Mon, Aug 24, 2026 at 3:38 PM Baoquan He <hebaoquan@kylinos.cn> wrote:
> >
> > In the current MGLRU, lru_gen_use_mm() will mark an mm for all nodes at
> > each context switch, so on each node aging will walk into each mm's page
> > table independently. On a multi-NUMA node system, one process launched
> > on one or a subset of nodes, its mm is walked by all other nodes's aging
> > while finds on pages for their lruvec. This is pure waste (100% walks
> 
> I assume you mean “finds no pages” rather than “find on pages.”

You are right, typo, thanks.

> 
> > are empty on those other nodes)
> >
> > This patch series suppresses these empty page table walks with two
> > complementary mechanisms at different levels:
> >
> >   - empty_map (cross-node specifc). At mm granularity, one process's mm
> >     whose walk found no pages for one node's lruvec is skipped on that
> >     node between re-scan passes.
> >
> >   - PUD-level Bloom filter (general). One level up from the existing PMD
> >     filters, it skips any 1GB PUD that had no young entries last
> >     generation - namely whose 512 PMDs all failed PMD test. Cross-node
> >     empty walks can prove its effect the best, but it also suppresses
> >     purely code PUD inside local mms. So it reduces unnecessary walking
> >     in any workload with cold areas.
> 
> I assume you mean “purely cold PUD” rather than “purely code PUD”.

Right, thanks.

> 
> >
> > The two complement each other: empty_map reduces the *number* of
> > cross-node walks, the PUD-level filter reduces the *cost* of the walks
> > that remain.
> 
> I wonder if the PUD-level filter alone would achieve your goal without
> requiring the more complex `empty_map` code.
> 
> PUD is already a fairly large granularity, so even if we still do some
> redundant work, I wonder if the overhead would be small enough to make
> `empty_map` unnecessary?

As replied to you in patch 2 thread, will take PUD filter alone in v2
as you suggested, thanks.

Re: [PATCH 1/9] mm/mglru: add MM_WALK_EMPTY stats and tracepoint

[Baoquan He]

On 08/28/26 at 01:37pm, Barry Song wrote:
> On Mon, Aug 24, 2026 at 3:38 PM Baoquan He <hebaoquan@kylinos.cn> wrote:
> >
> > Add per-walk counters to measure empty aging walks which traverse
> > an mm's page tables but find no folio for the current lruvec (node+memcg).
> > These are common on multi-NUMA node systems because lru_gen_use_mm() marks
> > an mm for all nodes at every context switch.
> >
> > New counters (accumulated in mm_state->stats[]):
> >
> >   MM_LEAF_ELIGIBLE    - folios belonging to this lruvec
> >   MM_WALK_TOTAL       - page-table walks completed
> >   MM_WALK_EMPTY       - walks that found no eligible folio
> >   MM_LEAF_TOTAL_EMPTY - leaf entries scanned by empty walks
> 
> Hi Baoquan,
> 
> I'm having a hard time understanding what MM_LEAF_ELIGIBLE means.
> In particular, I'm not sure why "eligible" refers to folios here.
> 
> I'm also finding MM_LEAF_TOTAL_EMPTY quite difficult to understand
> without looking at the implementation.
> 
> Could we come up with clearer and more descriptive names for these
> counters?  might be?
> 
> MM_LEAF_ASSOCIATED,          /* folios associated with this lruvec */
> MM_WALK_TOTAL,               /* completed page-table walks */
> MM_WALK_WITHOUT_ASSOCIATED,  /* walks with no associated folio */
> MM_LEAF_WITHOUT_ASSOCIATED,  /* leaf entries in such walks */

Sorry about the confusion, let me think about this, and your naming
suggestions sound good, I will update here if I have different
thinking. Thanks for careful reviewing.

> 
> >
> > A new tracepoint, mm_vmscan_lru_gen_walk(), fires after each walk, and the
> > debugfs lru_gen output ("TYFALWEE") exposes the new counters.
> >
> > Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
> > ---
> >  include/linux/mmzone.h        |  4 ++++
> >  include/trace/events/vmscan.h | 28 ++++++++++++++++++++++++++++
> >  mm/vmscan.c                   | 30 ++++++++++++++++++++++++++----
> >  3 files changed, 58 insertions(+), 4 deletions(-)
> >
> > diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> > index 94f9c3ff5416..229d27fbfb54 100644
> > --- a/include/linux/mmzone.h
> > +++ b/include/linux/mmzone.h
> > @@ -595,6 +595,10 @@ enum {
> >         MM_LEAF_YOUNG,          /* young leaf entries */
> >         MM_NONLEAF_FOUND,       /* non-leaf entries found in Bloom filters */
> >         MM_NONLEAF_ADDED,       /* non-leaf entries added to Bloom filters */
> > +       MM_LEAF_ELIGIBLE,       /* folios belonging to this lruvec (node+memcg) */
> > +       MM_WALK_TOTAL,          /* page-table walks completed */
> > +       MM_WALK_EMPTY,          /* walks that found no eligible folio */
> > +       MM_LEAF_TOTAL_EMPTY,    /* leaf entries scanned by empty walks */
> >         NR_MM_STATS
> >  };
> >
> 
> Best Regards
> Barry
>