[PATCH v2 0/4] skip empty PUD subtrees during aging with a PUD-level Bloom filter

[PATCH v2 0/4] skip empty PUD subtrees during aging with a PUD-level Bloom filter

[Baoquan He]

Problem
=======
 
MGLRU's aging walks every present PUD of every mm in the mm_list. When
an mm has no pages on a given NUMA node, all 512 PMDs in each PUD fail
the existing PMD-level Bloom filter test, yet the walker still walks
into every PUD and iterates every PMD. This is pure waste.
 
These cross-node empty walks are structural: lru_gen_use_mm() sets
mm->lru_gen.bitmap to -1 (all nodes) at each context switch, so the
aging walker on every node including kswap, direct reclaim or proactive
aging independently walks the same mm. On the test of 2-node KVM
guest, node 1's walks of the node-0-pinned workload scan ~1235 PTE
entries per walk while finding no folio for node 1's lruvec.

Approach
========
  
Add a PUD-level Bloom filter, one level above the existing PMD filter.
The walker now:
  
  1. Tests the PUD filter before descending into a PUD subtree.
  2. If the PUD had no young entries last generation, skips the entire
     1GB region -- avoiding 512 PMD lookups. 
  3. walk_pmd_range() reports whether it found any young leaf entries;
     walk_pud_range() records that in the double-buffered PUD filter.
  4. The rmap feedback path (lru_gen_look_around()) marks hot PUDs, so
     newly hot or migrated-in regions are re-examined promptly.
  5. force_scan walks bypass the PUD test, so manual aging and newly
     added mm's always re-populate the filter.
 
The double-buffered filter flips each aging generation and the eviction
feedback keeps hot regions marked, so correctness holds without tracking
per-page residency or touching the page fault / rmap hot paths.

Test
====
 
Empty-walk suppression (8GB / 2-NUMA KVM guest, 32 x 64MB workers pinned
to node 0 via numactl --membind=0, 100 aging passes, 3 runs):
 
  node 1 (foreign mm walks):
    baseline (no PUD filter):   ~1235-1254 PTE entries scanned per walk
    with PUD filter:              0.04 / 1.3 / 2.0 per walk
                                (>99.8% reduction; what remains is a few
                                 other processes' pages on node 1,the
                                 cross-node workload mms are fully suppressed)
    pud_skipped:                 ~3.1 1GB subtrees per walk
    worker RSS delta:             0% (no premature reclaim)
 
Build-time regression (make -j4 in a 3G memory cgroup, 4 vCPU / 8GB
2-NUMA KVM guest, 3 runs each; the host was kept idle during the runs):
 
              baseline   patched
  build time  11m48      11m26     (within run-to-run variance; one
                                    patched run of 10m37 was an outlier)
  pgpgin      138k       139k      (unchanged)
  pgmajfault  1950       1950      (unchanged)

The PUD-level filter reduces the leaf scanning of empty aging walks on
the foreign node by >99.8% with no premature reclaim. The build test
shows no measurable regression.

Change log:
==========

v1->v2:
 
- Drop the empty_map (mm-level skip) mechanism. Per Barry Song's
  review, the PUD-level filter alone achieves the goal; empty_map added
  complexity (an mm_struct field, a skip_empty knob, re-scan bookkeeping)
  without being necessary. It can be revisited if very large memory
  systems make the residual walk count meaningful again.
- Rename the counters: MM_LEAF_ELIGIBLE -> MM_LEAF_ASSOCIATED,
  MM_LEAF_TOTAL_EMPTY -> MM_LEAF_EMPTY_WALKS; MM_WALK_EMPTY kept with its
  comment clarified.
- Patch 1 adds the per-walk counters and the mm_vmscan_lru_gen_walk()
  tracepoint used for the measurements below; the series is now 4 patches.

RFC-v1:

- Bloom filter helpers are named symmetrically -
  test/update/reset_pmd_bloom_filter() alongside the PUD-level ones, and
  the struct field filters -> pmd_filters (Barry).
- An mm is marked empty only when its page tables were actually walked
  (a failed mmap_read_trylock() or a stale seq is not empty), and the skip
  is invalidated on the major page-fault and migration paths. (Sashiko)
- Reworked the re-scan: the old shared counter across nodes got stuck at
  the slowest node, so the skip never really engaged. Now each node re-scans
  its empty mms every N passes on its own clock. N is the skip_empty knob
  (default 4, proper read/write with input validation). (Sashiko)
- Added measurement counters/tracepoint, and a kernel-build regression
  test (no measurable impact).

Baoquan He (4):
  mm/mglru: add MM_WALK_EMPTY stats and tracepoint
  mm/mglru: add PUD-level Bloom filter state and generic helpers
  mm/mglru: skip cold PUD subtrees during aging
  mm/mglru: count PUD subtrees skipped by the PUD-level filter

 include/linux/mmzone.h        |  11 ++-
 include/trace/events/vmscan.h |  30 ++++++++
 mm/vmscan.c                   | 132 +++++++++++++++++++++++++++-------
 3 files changed, 147 insertions(+), 26 deletions(-)


base-commit: 42d64d4fef83a241c919c8693fdf0a21b2cb6061
-- 
2.54.0

[PATCH v2 1/4] mm/mglru: add MM_WALK_EMPTY stats and tracepoint

[Baoquan He]

Add per-walk counters to measure empty aging walks which traverse
an mm's page tables but find no folio for the current lruvec (node+memcg).
These are common on multi-NUMA node systems because lru_gen_use_mm() marks
an mm for all nodes at every context switch.

New counters (accumulated in mm_state->stats[]):

  MM_LEAF_ASSOCIATED   - leaf entries whose folio is in this lruvec
  MM_WALK_TOTAL        - page-table walks completed
  MM_WALK_EMPTY        - walks that found no folio in this lruvec
  MM_LEAF_EMPTY_WALKS  - leaf entries scanned during empty walks

A new tracepoint, mm_vmscan_lru_gen_walk(), fires after each walk, and the
debugfs lru_gen output ("TYFALWEE") exposes the new counters.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 include/linux/mmzone.h        |  4 ++++
 include/trace/events/vmscan.h | 28 ++++++++++++++++++++++++++++
 mm/vmscan.c                   | 32 ++++++++++++++++++++++++++++----
 3 files changed, 60 insertions(+), 4 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 94f9c3ff5416..f75daa6cd02d 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -595,6 +595,10 @@ enum {
 	MM_LEAF_YOUNG,		/* young leaf entries */
 	MM_NONLEAF_FOUND,	/* non-leaf entries found in Bloom filters */
 	MM_NONLEAF_ADDED,	/* non-leaf entries added to Bloom filters */
+	MM_LEAF_ASSOCIATED,	/* leaf entries whose folio is in this lruvec (node+memcg) */
+	MM_WALK_TOTAL,		/* page-table walks completed */
+	MM_WALK_EMPTY,		/* walks that found no folio in this lruvec */
+	MM_LEAF_EMPTY_WALKS,	/* leaf entries scanned during empty walks */
 	NR_MM_STATS
 };
 
diff --git a/include/trace/events/vmscan.h b/include/trace/events/vmscan.h
index b4bf7b8def1f..695d6fefc606 100644
--- a/include/trace/events/vmscan.h
+++ b/include/trace/events/vmscan.h
@@ -659,6 +659,34 @@ TRACE_EVENT(mm_vmscan_kswapd_clear_hopeless,
 		__entry->nid,
 		__print_symbolic(__entry->reason, kswapd_clear_hopeless_reason_ops))
 );
+TRACE_EVENT(mm_vmscan_lru_gen_walk,
+
+	TP_PROTO(int nid, unsigned long seq, int leaf_total,
+		 int leaf_associated, bool empty),
+
+	TP_ARGS(nid, seq, leaf_total, leaf_associated, empty),
+
+	TP_STRUCT__entry(
+		__field(int, nid)
+		__field(unsigned long, seq)
+		__field(int, leaf_total)
+		__field(int, leaf_associated)
+		__field(bool, empty)
+	),
+
+	TP_fast_assign(
+		__entry->nid = nid;
+		__entry->seq = seq;
+		__entry->leaf_total = leaf_total;
+		__entry->leaf_associated = leaf_associated;
+		__entry->empty = empty;
+	),
+
+	TP_printk("nid=%d seq=%lu leaf_total=%d leaf_associated=%d empty=%d",
+		__entry->nid, __entry->seq, __entry->leaf_total,
+		__entry->leaf_associated, __entry->empty)
+);
+
 #endif /* _TRACE_VMSCAN_H */
 
 /* This part must be outside protection */
diff --git a/mm/vmscan.c b/mm/vmscan.c
index f11491ee9ed5..55d43ab54459 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3588,6 +3588,8 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
 		if (!folio)
 			continue;
 
+		walk->mm_stats[MM_LEAF_ASSOCIATED]++;
+
 		if (folio_test_large(folio)) {
 			const unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
 
@@ -3688,6 +3690,8 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
 		if (!folio)
 			goto next;
 
+		walk->mm_stats[MM_LEAF_ASSOCIATED]++;
+
 		if (!pmdp_test_and_clear_young_notify(vma, addr, pmd + i))
 			goto next;
 
@@ -4109,8 +4113,28 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq,
 
 	do {
 		success = iterate_mm_list(walk, &mm);
-		if (mm)
+		if (mm) {
+			bool empty = false;
+
 			walk_mm(mm, walk);
+			/*
+			 * A walk that traversed page tables but found no folio
+			 * belonging to this lruvec (node+memcg) is pure waste.
+			 */
+			if (walk->mm_stats[MM_LEAF_TOTAL]) {
+				walk->mm_stats[MM_WALK_TOTAL]++;
+				if (walk->mm_stats[MM_LEAF_ASSOCIATED] == 0) {
+					walk->mm_stats[MM_WALK_EMPTY]++;
+					walk->mm_stats[MM_LEAF_EMPTY_WALKS] +=
+						walk->mm_stats[MM_LEAF_TOTAL];
+					empty = true;
+				}
+			}
+			trace_mm_vmscan_lru_gen_walk(
+					lruvec_pgdat(lruvec)->node_id, walk->seq,
+					walk->mm_stats[MM_LEAF_TOTAL],
+					walk->mm_stats[MM_LEAF_ASSOCIATED], empty);
+		}
 	} while (mm);
 done:
 	if (success) {
@@ -5585,14 +5609,14 @@ static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruvec,
 
 	seq_puts(m, "                      ");
 	for (i = 0; i < NR_MM_STATS; i++) {
-		const char *s = "xxxx";
+		const char *s = "xxxxxxxx";
 		unsigned long n = 0;
 
 		if (seq == max_seq && NR_HIST_GENS == 1) {
-			s = "TYFA";
+			s = "TYFALWEE";
 			n = READ_ONCE(mm_state->stats[hist][i]);
 		} else if (seq != max_seq && NR_HIST_GENS > 1) {
-			s = "tyfa";
+			s = "tyfalwee";
 			n = READ_ONCE(mm_state->stats[hist][i]);
 		}
 
-- 
2.54.0

[PATCH v2 2/4] mm/mglru: add PUD-level Bloom filter state and generic helpers

[Baoquan He]

Rename the existing PMD-level filter array (filters -> pmd_filters) and
add a second, coarser pair of PUD-level Bloom filters (pud_filters).
The PUL filsters operate at 1GB (PUD) granularity, whose 512 PMDs would
all fail the PMD-level filter - a general suppression of unnecessary page
table walks. Cross-node empty walks are the most visible case: a foreign
mm's PUDs have no young entries for this lruvec, so the entire PMD
iteration is pure waste.

Generalize the Bloom filter helpers (test/update/reset_bloom_filter ->
__test/update/reset_bloom_filter) to operate on a generic filters array,
and add thin wrappers (test/update/reset_pmd_bloom_filter) that pass the
PMD-level array, so the PUD-level pair can reuse the same hash,
double-buffering and reset.

No behavior change yet; the PUD filters are populated and used by later
patches.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 include/linux/mmzone.h |  6 ++++--
 mm/vmscan.c            | 45 ++++++++++++++++++++++++++++--------------
 2 files changed, 34 insertions(+), 17 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index f75daa6cd02d..faf28d35065d 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -612,8 +612,10 @@ struct lru_gen_mm_state {
 	struct list_head *head;
 	/* where the last iteration ended before */
 	struct list_head *tail;
-	/* Bloom filters flip after each iteration */
-	unsigned long *filters[NR_BLOOM_FILTERS];
+	/* PMD-level Bloom filters flip after each iteration */
+	unsigned long *pmd_filters[NR_BLOOM_FILTERS];
+	/* PUD-level Bloom filters flip after each iteration */
+	unsigned long *pud_filters[NR_BLOOM_FILTERS];
 	/* the mm stats for debugging */
 	unsigned long stats[NR_HIST_GENS][NR_MM_STATS];
 };
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 55d43ab54459..cd09c5b8af8c 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2842,14 +2842,13 @@ static void get_item_key(void *item, int *key)
 	key[1] = hash >> BLOOM_FILTER_SHIFT;
 }
 
-static bool test_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq,
-			      void *item)
+static bool __test_bloom_filter(unsigned long **filters, unsigned long seq, void *item)
 {
 	int key[2];
 	unsigned long *filter;
 	int gen = filter_gen_from_seq(seq);
 
-	filter = READ_ONCE(mm_state->filters[gen]);
+	filter = READ_ONCE(filters[gen]);
 	if (!filter)
 		return true;
 
@@ -2858,14 +2857,13 @@ static bool test_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long s
 	return test_bit(key[0], filter) && test_bit(key[1], filter);
 }
 
-static void update_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq,
-				void *item)
+static void __update_bloom_filter(unsigned long **filters, unsigned long seq, void *item)
 {
 	int key[2];
 	unsigned long *filter;
 	int gen = filter_gen_from_seq(seq);
 
-	filter = READ_ONCE(mm_state->filters[gen]);
+	filter = READ_ONCE(filters[gen]);
 	if (!filter)
 		return;
 
@@ -2877,12 +2875,12 @@ static void update_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long
 		set_bit(key[1], filter);
 }
 
-static void reset_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq)
+static void __reset_bloom_filter(unsigned long **filters, unsigned long seq)
 {
 	unsigned long *filter;
 	int gen = filter_gen_from_seq(seq);
 
-	filter = mm_state->filters[gen];
+	filter = filters[gen];
 	if (filter) {
 		bitmap_clear(filter, 0, BIT(BLOOM_FILTER_SHIFT));
 		return;
@@ -2890,7 +2888,24 @@ static void reset_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long
 
 	filter = bitmap_zalloc(BIT(BLOOM_FILTER_SHIFT),
 			       __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN);
-	WRITE_ONCE(mm_state->filters[gen], filter);
+	WRITE_ONCE(filters[gen], filter);
+}
+
+static bool test_pmd_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq,
+			      void *item)
+{
+	return __test_bloom_filter(mm_state->pmd_filters, seq, item);
+}
+
+static void update_pmd_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq,
+				void *item)
+{
+	__update_bloom_filter(mm_state->pmd_filters, seq, item);
+}
+
+static void reset_pmd_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq)
+{
+	__reset_bloom_filter(mm_state->pmd_filters, seq);
 }
 
 /******************************************************************************
@@ -3133,7 +3148,7 @@ static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite
 	spin_unlock(&mm_list->lock);
 
 	if (mm && first)
-		reset_bloom_filter(mm_state, walk->seq + 1);
+		reset_pmd_bloom_filter(mm_state, walk->seq + 1);
 
 	if (*iter)
 		mmdrop(*iter);
@@ -3771,7 +3786,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
 			walk_pmd_range_locked(pud, addr, vma, args, bitmap, &first);
 		}
 
-		if (!walk->force_scan && !test_bloom_filter(mm_state, walk->seq, pmd + i))
+		if (!walk->force_scan && !test_pmd_bloom_filter(mm_state, walk->seq, pmd + i))
 			continue;
 
 		walk->mm_stats[MM_NONLEAF_FOUND]++;
@@ -3782,7 +3797,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
 		walk->mm_stats[MM_NONLEAF_ADDED]++;
 
 		/* carry over to the next generation */
-		update_bloom_filter(mm_state, walk->seq + 1, pmd + i);
+		update_pmd_bloom_filter(mm_state, walk->seq + 1, pmd + i);
 	}
 
 	walk_pmd_range_locked(pud, -1, vma, args, bitmap, &first);
@@ -4383,7 +4398,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 
 	/* feedback from rmap walkers to page table walkers */
 	if (mm_state && suitable_to_scan(i, young))
-		update_bloom_filter(mm_state, max_seq, pvmw->pmd);
+		update_pmd_bloom_filter(mm_state, max_seq, pvmw->pmd);
 
 	mem_cgroup_put(memcg);
 
@@ -5946,8 +5961,8 @@ void lru_gen_exit_memcg(struct mem_cgroup *memcg)
 			continue;
 
 		for (i = 0; i < NR_BLOOM_FILTERS; i++) {
-			bitmap_free(mm_state->filters[i]);
-			mm_state->filters[i] = NULL;
+			bitmap_free(mm_state->pmd_filters[i]);
+			mm_state->pmd_filters[i] = NULL;
 		}
 	}
 }
-- 
2.54.0

[PATCH v2 3/4] mm/mglru: skip cold PUD subtrees during aging

[Baoquan He]

The aging walks into every PUD and runs the PMD-level Bloom filter
on each PMD. Add a coarser PUD-level filter (pud_filters) one level
up:
 - walk_pmd_range() now reports whether it found any young leaf entries,
 - walk_pud_range() records that in the PUD filter, and
 - on subsequent generations, skips the whole 1GB subtree when the filter
   says it had none last generation.

The double-buffered filter flips with each new iteration, so newly hot or
migrated-in pages are re-checked promptly rather than suppressed
indefinitely. force_scan walks bypass the PUD test, so manual aging and
newly added mm's always rescan and re-populate the filter.

To keep hot regions marked, also report the covering PUD from the rmap
feedback path (lru_gen_look_around()), so regions whose hotness is only
observed by eviction will be re-scanned.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 mm/vmscan.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 47 insertions(+), 5 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index cd09c5b8af8c..955585626387 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2817,6 +2817,8 @@ static bool __maybe_unused seq_is_valid(struct lruvec *lruvec)
  * walk_pmd_range(); the eviction also report them when walking the rmap
  * in lru_gen_look_around().
  *
+ * A second, coarser pair of filters (pud_filters) sits one level up.
+ *
  * For future optimizations:
  * 1. It's not necessary to keep both filters all the time. The spare one can be
  *    freed after the RCU grace period and reallocated if needed again.
@@ -2908,6 +2910,23 @@ static void reset_pmd_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned l
 	__reset_bloom_filter(mm_state->pmd_filters, seq);
 }
 
+static bool test_pud_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq,
+				  void *item)
+{
+	return __test_bloom_filter(mm_state->pud_filters, seq, item);
+}
+
+static void update_pud_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq,
+				    void *item)
+{
+	__update_bloom_filter(mm_state->pud_filters, seq, item);
+}
+
+static void reset_pud_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq)
+{
+	__reset_bloom_filter(mm_state->pud_filters, seq);
+}
+
 /******************************************************************************
  *                          mm_struct list
  ******************************************************************************/
@@ -3147,8 +3166,10 @@ static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite
 
 	spin_unlock(&mm_list->lock);
 
-	if (mm && first)
+	if (mm && first) {
 		reset_pmd_bloom_filter(mm_state, walk->seq + 1);
+		reset_pud_bloom_filter(mm_state, walk->seq + 1);
+	}
 
 	if (*iter)
 		mmdrop(*iter);
@@ -3733,10 +3754,11 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
 	*first = -1;
 }
 
-static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
+static bool walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
 			   struct mm_walk *args)
 {
 	int i;
+	bool young = false;
 	pmd_t *pmd;
 	unsigned long next;
 	unsigned long addr;
@@ -3773,8 +3795,10 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
 
 			walk->mm_stats[MM_LEAF_TOTAL]++;
 
-			if (pfn != -1)
+			if (pfn != -1) {
 				walk_pmd_range_locked(pud, addr, vma, args, bitmap, &first);
+				young = true;
+			}
 			continue;
 		}
 
@@ -3784,6 +3808,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
 				continue;
 
 			walk_pmd_range_locked(pud, addr, vma, args, bitmap, &first);
+			young = true;
 		}
 
 		if (!walk->force_scan && !test_pmd_bloom_filter(mm_state, walk->seq, pmd + i))
@@ -3795,6 +3820,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
 			continue;
 
 		walk->mm_stats[MM_NONLEAF_ADDED]++;
+		young = true;
 
 		/* carry over to the next generation */
 		update_pmd_bloom_filter(mm_state, walk->seq + 1, pmd + i);
@@ -3804,6 +3830,8 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
 
 	if (i < PTRS_PER_PMD && get_next_vma(PUD_MASK, PMD_SIZE, args, &start, &end))
 		goto restart;
+
+	return young;
 }
 
 static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,
@@ -3814,6 +3842,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,
 	unsigned long addr;
 	unsigned long next;
 	struct lru_gen_mm_walk *walk = args->private;
+	struct lru_gen_mm_state *mm_state = get_mm_state(walk->lruvec);
 
 	VM_WARN_ON_ONCE(p4d_leaf(*p4d));
 
@@ -3827,7 +3856,12 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,
 		if (!pud_present(val) || WARN_ON_ONCE(pud_leaf(val)))
 			continue;
 
-		walk_pmd_range(&val, addr, next, args);
+		/* Skip a subtree whose 512 PMDs all failed the PMD-level filter last gen */
+		if (!walk->force_scan && !test_pud_bloom_filter(mm_state, walk->seq, pud + i))
+			continue;
+
+		if (walk_pmd_range(&val, addr, next, args))
+			update_pud_bloom_filter(mm_state, walk->seq + 1, pud + i);
 
 		if (need_resched() || walk->batched >= MAX_LRU_BATCH) {
 			end = (addr | ~PUD_MASK) + 1;
@@ -4397,8 +4431,14 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 	lazy_mmu_mode_disable();
 
 	/* feedback from rmap walkers to page table walkers */
-	if (mm_state && suitable_to_scan(i, young))
+	if (mm_state && suitable_to_scan(i, young)) {
+		/* the PUD entry covering the young PTEs scanned above */
+		pud_t *pud_p = pud_offset(p4d_offset(pgd_offset(vma->vm_mm, pvmw->address),
+						     pvmw->address), pvmw->address);
+
 		update_pmd_bloom_filter(mm_state, max_seq, pvmw->pmd);
+		update_pud_bloom_filter(mm_state, max_seq, pud_p);
+	}
 
 	mem_cgroup_put(memcg);
 
@@ -5963,6 +6003,8 @@ void lru_gen_exit_memcg(struct mem_cgroup *memcg)
 		for (i = 0; i < NR_BLOOM_FILTERS; i++) {
 			bitmap_free(mm_state->pmd_filters[i]);
 			mm_state->pmd_filters[i] = NULL;
+			bitmap_free(mm_state->pud_filters[i]);
+			mm_state->pud_filters[i] = NULL;
 		}
 	}
 }
-- 
2.54.0

[PATCH v2 4/4] mm/mglru: count PUD subtrees skipped by the PUD-level filter

[Baoquan He]

Add MM_PUD_EMPTY_SKIPPED, incremented in walk_pud_range() whenever a PUD
subtree is skipped by the filter. Like other counters accumulated
per walk, shown in debugfs lru_gen output (the "S" column of TYFALWEES)
and via the mm_vmscan_lru_gen_walk tracepoint, so the avoided cost
by the filter can be measured directly.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 include/linux/mmzone.h        |  1 +
 include/trace/events/vmscan.h | 10 ++++++----
 mm/vmscan.c                   | 13 ++++++++-----
 3 files changed, 15 insertions(+), 9 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index faf28d35065d..276d0ca8d1c1 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -599,6 +599,7 @@ enum {
 	MM_WALK_TOTAL,		/* page-table walks completed */
 	MM_WALK_EMPTY,		/* walks that found no folio in this lruvec */
 	MM_LEAF_EMPTY_WALKS,	/* leaf entries scanned during empty walks */
+	MM_PUD_EMPTY_SKIPPED,	/* PUD subtrees skipped by the PUD-level filter */
 	NR_MM_STATS
 };
 
diff --git a/include/trace/events/vmscan.h b/include/trace/events/vmscan.h
index 695d6fefc606..1cf031d88290 100644
--- a/include/trace/events/vmscan.h
+++ b/include/trace/events/vmscan.h
@@ -662,9 +662,9 @@ TRACE_EVENT(mm_vmscan_kswapd_clear_hopeless,
 TRACE_EVENT(mm_vmscan_lru_gen_walk,
 
 	TP_PROTO(int nid, unsigned long seq, int leaf_total,
-		 int leaf_associated, bool empty),
+		 int leaf_associated, bool empty, int pud_skipped),
 
-	TP_ARGS(nid, seq, leaf_total, leaf_associated, empty),
+	TP_ARGS(nid, seq, leaf_total, leaf_associated, empty, pud_skipped),
 
 	TP_STRUCT__entry(
 		__field(int, nid)
@@ -672,6 +672,7 @@ TRACE_EVENT(mm_vmscan_lru_gen_walk,
 		__field(int, leaf_total)
 		__field(int, leaf_associated)
 		__field(bool, empty)
+		__field(int, pud_skipped)
 	),
 
 	TP_fast_assign(
@@ -680,11 +681,12 @@ TRACE_EVENT(mm_vmscan_lru_gen_walk,
 		__entry->leaf_total = leaf_total;
 		__entry->leaf_associated = leaf_associated;
 		__entry->empty = empty;
+		__entry->pud_skipped = pud_skipped;
 	),
 
-	TP_printk("nid=%d seq=%lu leaf_total=%d leaf_associated=%d empty=%d",
+	TP_printk("nid=%d seq=%lu leaf_total=%d leaf_associated=%d empty=%d pud_skipped=%d",
 		__entry->nid, __entry->seq, __entry->leaf_total,
-		__entry->leaf_associated, __entry->empty)
+		__entry->leaf_associated, __entry->empty, __entry->pud_skipped)
 );
 
 #endif /* _TRACE_VMSCAN_H */
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 955585626387..035bd9ea49aa 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3857,8 +3857,10 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,
 			continue;
 
 		/* Skip a subtree whose 512 PMDs all failed the PMD-level filter last gen */
-		if (!walk->force_scan && !test_pud_bloom_filter(mm_state, walk->seq, pud + i))
+		if (!walk->force_scan && !test_pud_bloom_filter(mm_state, walk->seq, pud + i)) {
+			walk->mm_stats[MM_PUD_EMPTY_SKIPPED]++;
 			continue;
+		}
 
 		if (walk_pmd_range(&val, addr, next, args))
 			update_pud_bloom_filter(mm_state, walk->seq + 1, pud + i);
@@ -4182,7 +4184,8 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq,
 			trace_mm_vmscan_lru_gen_walk(
 					lruvec_pgdat(lruvec)->node_id, walk->seq,
 					walk->mm_stats[MM_LEAF_TOTAL],
-					walk->mm_stats[MM_LEAF_ASSOCIATED], empty);
+					walk->mm_stats[MM_LEAF_ASSOCIATED], empty,
+					walk->mm_stats[MM_PUD_EMPTY_SKIPPED]);
 		}
 	} while (mm);
 done:
@@ -5664,14 +5667,14 @@ static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruvec,
 
 	seq_puts(m, "                      ");
 	for (i = 0; i < NR_MM_STATS; i++) {
-		const char *s = "xxxxxxxx";
+		const char *s = "xxxxxxxxx";
 		unsigned long n = 0;
 
 		if (seq == max_seq && NR_HIST_GENS == 1) {
-			s = "TYFALWEE";
+			s = "TYFALWEES";
 			n = READ_ONCE(mm_state->stats[hist][i]);
 		} else if (seq != max_seq && NR_HIST_GENS > 1) {
-			s = "tyfalwee";
+			s = "tyfalwees";
 			n = READ_ONCE(mm_state->stats[hist][i]);
 		}
 
-- 
2.54.0

Re: [PATCH v2 1/4] mm/mglru: add MM_WALK_EMPTY stats and tracepoint

[Barry Song]

On Tue, Sep 1, 2026 at 2:38 PM Baoquan He <hebaoquan@kylinos.cn> wrote:
>
> Add per-walk counters to measure empty aging walks which traverse
> an mm's page tables but find no folio for the current lruvec (node+memcg).
> These are common on multi-NUMA node systems because lru_gen_use_mm() marks
> an mm for all nodes at every context switch.
>
> New counters (accumulated in mm_state->stats[]):
>
>   MM_LEAF_ASSOCIATED   - leaf entries whose folio is in this lruvec
>   MM_WALK_TOTAL        - page-table walks completed
>   MM_WALK_EMPTY        - walks that found no folio in this lruvec
>   MM_LEAF_EMPTY_WALKS  - leaf entries scanned during empty walks
>
> A new tracepoint, mm_vmscan_lru_gen_walk(), fires after each walk, and the
> debugfs lru_gen output ("TYFALWEE") exposes the new counters.
>
> Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
> ---
[...]
>
> @@ -4109,8 +4113,28 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq,
>
>         do {
>                 success = iterate_mm_list(walk, &mm);
> -               if (mm)
> +               if (mm) {
> +                       bool empty = false;
> +
>                         walk_mm(mm, walk);
> +                       /*
> +                        * A walk that traversed page tables but found no folio
> +                        * belonging to this lruvec (node+memcg) is pure waste.
> +                        */
> +                       if (walk->mm_stats[MM_LEAF_TOTAL]) {
> +                               walk->mm_stats[MM_WALK_TOTAL]++;
> +                               if (walk->mm_stats[MM_LEAF_ASSOCIATED] == 0) {
> +                                       walk->mm_stats[MM_WALK_EMPTY]++;
> +                                       walk->mm_stats[MM_LEAF_EMPTY_WALKS] +=
> +                                               walk->mm_stats[MM_LEAF_TOTAL];
> +                                       empty = true;
> +                               }
> +                       }

Hi Baoquan,

Where are we clearing these counters between different
mms?

It seems the previous mm_stats will affect the next mm
if they aren't cleared.


> +                       trace_mm_vmscan_lru_gen_walk(
> +                                       lruvec_pgdat(lruvec)->node_id, walk->seq,
> +                                       walk->mm_stats[MM_LEAF_TOTAL],
> +                                       walk->mm_stats[MM_LEAF_ASSOCIATED], empty);
> +               }
>         } while (mm);
>  done:

Best Regards
Barry

Re: [PATCH v2 1/4] mm/mglru: add MM_WALK_EMPTY stats and tracepoint

[Baoquan He]

On 09/01/26 at 03:11pm, Barry Song wrote:
> On Tue, Sep 1, 2026 at 2:38 PM Baoquan He <hebaoquan@kylinos.cn> wrote:
> >
> > Add per-walk counters to measure empty aging walks which traverse
> > an mm's page tables but find no folio for the current lruvec (node+memcg).
> > These are common on multi-NUMA node systems because lru_gen_use_mm() marks
> > an mm for all nodes at every context switch.
> >
> > New counters (accumulated in mm_state->stats[]):
> >
> >   MM_LEAF_ASSOCIATED   - leaf entries whose folio is in this lruvec
> >   MM_WALK_TOTAL        - page-table walks completed
> >   MM_WALK_EMPTY        - walks that found no folio in this lruvec
> >   MM_LEAF_EMPTY_WALKS  - leaf entries scanned during empty walks
> >
> > A new tracepoint, mm_vmscan_lru_gen_walk(), fires after each walk, and the
> > debugfs lru_gen output ("TYFALWEE") exposes the new counters.
> >
> > Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
> > ---
> [...]
> >
> > @@ -4109,8 +4113,28 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq,
> >
> >         do {
> >                 success = iterate_mm_list(walk, &mm);
> > -               if (mm)
> > +               if (mm) {
> > +                       bool empty = false;
> > +
> >                         walk_mm(mm, walk);
> > +                       /*
> > +                        * A walk that traversed page tables but found no folio
> > +                        * belonging to this lruvec (node+memcg) is pure waste.
> > +                        */
> > +                       if (walk->mm_stats[MM_LEAF_TOTAL]) {
> > +                               walk->mm_stats[MM_WALK_TOTAL]++;
> > +                               if (walk->mm_stats[MM_LEAF_ASSOCIATED] == 0) {
> > +                                       walk->mm_stats[MM_WALK_EMPTY]++;
> > +                                       walk->mm_stats[MM_LEAF_EMPTY_WALKS] +=
> > +                                               walk->mm_stats[MM_LEAF_TOTAL];
> > +                                       empty = true;
> > +                               }
> > +                       }
> 
> Hi Baoquan,
> 
> Where are we clearing these counters between different
> mms?
> 
> It seems the previous mm_stats will affect the next mm
> if they aren't cleared.

No, they won't. The counters are cleared between mms. iterate_mm_list()
calls reset_mm_stats() from its "done:" path for every mm. reset_mm_stats()
accumulates walk->mm_stats into the per-hist mm_state->stats[hist] and
then zeroes walk->mm_stats[i].

> 
> 
> > +                       trace_mm_vmscan_lru_gen_walk(
> > +                                       lruvec_pgdat(lruvec)->node_id, walk->seq,
> > +                                       walk->mm_stats[MM_LEAF_TOTAL],
> > +                                       walk->mm_stats[MM_LEAF_ASSOCIATED], empty);
> > +               }
> >         } while (mm);
> >  done:
> 
> Best Regards
> Barry
> 

Re: [PATCH v2 1/4] mm/mglru: add MM_WALK_EMPTY stats and tracepoint

[Barry Song]

On Tue, Sep 1, 2026 at 4:18 PM Baoquan He <baoquan.he@linux.dev> wrote:
>
> On 09/01/26 at 03:11pm, Barry Song wrote:
> > On Tue, Sep 1, 2026 at 2:38 PM Baoquan He <hebaoquan@kylinos.cn> wrote:
> > >
> > > Add per-walk counters to measure empty aging walks which traverse
> > > an mm's page tables but find no folio for the current lruvec (node+memcg).
> > > These are common on multi-NUMA node systems because lru_gen_use_mm() marks
> > > an mm for all nodes at every context switch.
> > >
> > > New counters (accumulated in mm_state->stats[]):
> > >
> > >   MM_LEAF_ASSOCIATED   - leaf entries whose folio is in this lruvec
> > >   MM_WALK_TOTAL        - page-table walks completed
> > >   MM_WALK_EMPTY        - walks that found no folio in this lruvec
> > >   MM_LEAF_EMPTY_WALKS  - leaf entries scanned during empty walks
> > >
> > > A new tracepoint, mm_vmscan_lru_gen_walk(), fires after each walk, and the
> > > debugfs lru_gen output ("TYFALWEE") exposes the new counters.
> > >
> > > Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
> > > ---
> > [...]
> > >
> > > @@ -4109,8 +4113,28 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq,
> > >
> > >         do {
> > >                 success = iterate_mm_list(walk, &mm);
> > > -               if (mm)
> > > +               if (mm) {
> > > +                       bool empty = false;
> > > +
> > >                         walk_mm(mm, walk);
> > > +                       /*
> > > +                        * A walk that traversed page tables but found no folio
> > > +                        * belonging to this lruvec (node+memcg) is pure waste.
> > > +                        */
> > > +                       if (walk->mm_stats[MM_LEAF_TOTAL]) {
> > > +                               walk->mm_stats[MM_WALK_TOTAL]++;
> > > +                               if (walk->mm_stats[MM_LEAF_ASSOCIATED] == 0) {
> > > +                                       walk->mm_stats[MM_WALK_EMPTY]++;
> > > +                                       walk->mm_stats[MM_LEAF_EMPTY_WALKS] +=
> > > +                                               walk->mm_stats[MM_LEAF_TOTAL];
> > > +                                       empty = true;
> > > +                               }
> > > +                       }
> >
> > Hi Baoquan,
> >
> > Where are we clearing these counters between different
> > mms?
> >
> > It seems the previous mm_stats will affect the next mm
> > if they aren't cleared.
>
> No, they won't. The counters are cleared between mms. iterate_mm_list()
> calls reset_mm_stats() from its "done:" path for every mm. reset_mm_stats()
> accumulates walk->mm_stats into the per-hist mm_state->stats[hist] and
> then zeroes walk->mm_stats[i].

You’re right. Sorry for the noise—I missed that part.

Reviewed-by: Barry Song <baohua@kernel.org>

Re: [PATCH v2 3/4] mm/mglru: skip cold PUD subtrees during aging

[Barry Song]

On Tue, Sep 1, 2026 at 2:38 PM Baoquan He <hebaoquan@kylinos.cn> wrote:
>
> The aging walks into every PUD and runs the PMD-level Bloom filter
> on each PMD. Add a coarser PUD-level filter (pud_filters) one level
> up:
>  - walk_pmd_range() now reports whether it found any young leaf entries,
>  - walk_pud_range() records that in the PUD filter, and
>  - on subsequent generations, skips the whole 1GB subtree when the filter
>    says it had none last generation.
>
> The double-buffered filter flips with each new iteration, so newly hot or
> migrated-in pages are re-checked promptly rather than suppressed
> indefinitely. force_scan walks bypass the PUD test, so manual aging and
> newly added mm's always rescan and re-populate the filter.
>
> To keep hot regions marked, also report the covering PUD from the rmap
> feedback path (lru_gen_look_around()), so regions whose hotness is only
> observed by eviction will be re-scanned.
>
> Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
> ---
>  mm/vmscan.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++-----
>  1 file changed, 47 insertions(+), 5 deletions(-)
[...]
> @@ -4397,8 +4431,14 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
>         lazy_mmu_mode_disable();
>
>         /* feedback from rmap walkers to page table walkers */
> -       if (mm_state && suitable_to_scan(i, young))
> +       if (mm_state && suitable_to_scan(i, young)) {
> +               /* the PUD entry covering the young PTEs scanned above */
> +               pud_t *pud_p = pud_offset(p4d_offset(pgd_offset(vma->vm_mm, pvmw->address),
> +                                                    pvmw->address), pvmw->address);
> +
>                 update_pmd_bloom_filter(mm_state, max_seq, pvmw->pmd);
> +               update_pud_bloom_filter(mm_state, max_seq, pud_p);
> +       }


Hi Baoquan,

I am really not against this idea, and I believe it can benefit
NUMA cases.

That said, I might be being overly cautious, but I'm a bit concerned
that this re-walk of the page tables could slightly hurt machines that
don't benefit from the PUD filter at all. For example, Android devices
typically have relatively small amounts of memory, so a PUD covers a
really large range of their address space, which is unlikely to be
useful on an 8 GB Android device. Also, `lru_gen_look_around()` is a
really hot path. On Android, we've observed that it can consume a
significant amount of CPU.

Is there any possibility of implementing this in a low-cost way?

Best Regards
Barry

Re: [PATCH v2 3/4] mm/mglru: skip cold PUD subtrees during aging

[Baoquan He]

On 09/02/26 at 07:53am, Barry Song wrote:
> On Tue, Sep 1, 2026 at 2:38 PM Baoquan He <hebaoquan@kylinos.cn> wrote:
> >
> > The aging walks into every PUD and runs the PMD-level Bloom filter
> > on each PMD. Add a coarser PUD-level filter (pud_filters) one level
> > up:
> >  - walk_pmd_range() now reports whether it found any young leaf entries,
> >  - walk_pud_range() records that in the PUD filter, and
> >  - on subsequent generations, skips the whole 1GB subtree when the filter
> >    says it had none last generation.
> >
> > The double-buffered filter flips with each new iteration, so newly hot or
> > migrated-in pages are re-checked promptly rather than suppressed
> > indefinitely. force_scan walks bypass the PUD test, so manual aging and
> > newly added mm's always rescan and re-populate the filter.
> >
> > To keep hot regions marked, also report the covering PUD from the rmap
> > feedback path (lru_gen_look_around()), so regions whose hotness is only
> > observed by eviction will be re-scanned.
> >
> > Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
> > ---
> >  mm/vmscan.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++-----
> >  1 file changed, 47 insertions(+), 5 deletions(-)
> [...]
> > @@ -4397,8 +4431,14 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
> >         lazy_mmu_mode_disable();
> >
> >         /* feedback from rmap walkers to page table walkers */
> > -       if (mm_state && suitable_to_scan(i, young))
> > +       if (mm_state && suitable_to_scan(i, young)) {
> > +               /* the PUD entry covering the young PTEs scanned above */
> > +               pud_t *pud_p = pud_offset(p4d_offset(pgd_offset(vma->vm_mm, pvmw->address),
> > +                                                    pvmw->address), pvmw->address);
> > +
> >                 update_pmd_bloom_filter(mm_state, max_seq, pvmw->pmd);
> > +               update_pud_bloom_filter(mm_state, max_seq, pud_p);
> > +       }
> 
> 
> Hi Baoquan,
> 
> I am really not against this idea, and I believe it can benefit
> NUMA cases.
> 
> That said, I might be being overly cautious, but I'm a bit concerned
> that this re-walk of the page tables could slightly hurt machines that
> don't benefit from the PUD filter at all. For example, Android devices
> typically have relatively small amounts of memory, so a PUD covers a
> really large range of their address space, which is unlikely to be
> useful on an 8 GB Android device. Also, `lru_gen_look_around()` is a
> really hot path. On Android, we've observed that it can consume a
> significant amount of CPU.
> 
> Is there any possibility of implementing this in a low-cost way?

Hi Barry,

Thanks for the careful review, and both concerns are fair. Let me address
them separately.

(1) Aging walker overhead

The added cost in the aging walker is one PUD-level bloom test (2 bit
lookups) before walk_pmd_range(), plus a bool return from walk_pmd_range()
and a bloom update when young. This runs once per PUD per aging pass, not
per page fault, and is a tiny fraction of what the aging already does per
PUD. The test is also what lets the filter discover cold subtrees, so it
cannot simply be disabled.

(2) lru_gen_look_around() hot path

Agreed - this is the genuinely hot path (every fault), and on a
small-memory system where the PUD granularity is too coarse to be useful,
the pud_offset() + bloom update would be pure overhead. So I'll gate it
on a per-lruvec flag that tracks whether the aging walker is currently
skipping PUDs:

    /* iterate_mm_list(), at the start of a new generation's walk: */
    WRITE_ONCE(mm_state->pud_filter_used, false);

    /* walk_pud_range(), when a PUD is actually skipped: */
    WRITE_ONCE(mm_state->pud_filter_used, true);

    /* lru_gen_look_around(): */
    if (mm_state && suitable_to_scan(i, young) &&
        READ_ONCE(mm_state->pud_filter_used)) {
        pud_t *pud_p = pud_offset(...);
        update_pmd_bloom_filter(mm_state, max_seq, pvmw->pmd);
        update_pud_bloom_filter(mm_state, max_seq, pud_p);
    }

The flag is per-generation: cleared at the start of each aging generation
and set again when a PUD is skipped, so the feedback tracks whether the
filter is currently engaging. Before any PUD is skipped in a generation,
look_around() pays only a single flag loading.

I'll measure both on the 8GB test VM with a clean (fully unpatched)
baseline vs the patched kernel, and on a 3-node / 1.1TiB bare-metal
machine.

Thanks
Baoquan

Re: [PATCH v2 3/4] mm/mglru: skip cold PUD subtrees during aging

[Baoquan He]

On 09/02/26 at 07:53am, Barry Song wrote:
> On Tue, Sep 1, 2026 at 2:38 PM Baoquan He <hebaoquan@kylinos.cn> wrote:
> >
> > The aging walks into every PUD and runs the PMD-level Bloom filter
> > on each PMD. Add a coarser PUD-level filter (pud_filters) one level
> > up:
> >  - walk_pmd_range() now reports whether it found any young leaf entries,
> >  - walk_pud_range() records that in the PUD filter, and
> >  - on subsequent generations, skips the whole 1GB subtree when the filter
> >    says it had none last generation.
> >
> > The double-buffered filter flips with each new iteration, so newly hot or
> > migrated-in pages are re-checked promptly rather than suppressed
> > indefinitely. force_scan walks bypass the PUD test, so manual aging and
> > newly added mm's always rescan and re-populate the filter.
> >
> > To keep hot regions marked, also report the covering PUD from the rmap
> > feedback path (lru_gen_look_around()), so regions whose hotness is only
> > observed by eviction will be re-scanned.
> >
> > Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
> > ---
> >  mm/vmscan.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++-----
> >  1 file changed, 47 insertions(+), 5 deletions(-)
> [...]
> > @@ -4397,8 +4431,14 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
> >         lazy_mmu_mode_disable();
> >
> >         /* feedback from rmap walkers to page table walkers */
> > -       if (mm_state && suitable_to_scan(i, young))
> > +       if (mm_state && suitable_to_scan(i, young)) {
> > +               /* the PUD entry covering the young PTEs scanned above */
> > +               pud_t *pud_p = pud_offset(p4d_offset(pgd_offset(vma->vm_mm, pvmw->address),
> > +                                                    pvmw->address), pvmw->address);
> > +
> >                 update_pmd_bloom_filter(mm_state, max_seq, pvmw->pmd);
> > +               update_pud_bloom_filter(mm_state, max_seq, pud_p);
> > +       }
> 
> 
> Hi Baoquan,
> 
> I am really not against this idea, and I believe it can benefit
> NUMA cases.
> 
> That said, I might be being overly cautious, but I'm a bit concerned
> that this re-walk of the page tables could slightly hurt machines that
> don't benefit from the PUD filter at all. For example, Android devices
> typically have relatively small amounts of memory, so a PUD covers a
> really large range of their address space, which is unlikely to be
> useful on an 8 GB Android device. Also, `lru_gen_look_around()` is a
> really hot path. On Android, we've observed that it can consume a
> significant amount of CPU.
> 
> Is there any possibility of implementing this in a low-cost way?

I made a draft patch, could you help check and test if it's performing
better on Andriod device?

From 5b92a88925a7fe1733ca54595ece7be1bd122909 Mon Sep 17 00:00:00 2001
From: Baoquan He <hebaoquan@kylinos.cn>
Date: Wed, 2 Sep 2026 11:07:10 +0800
Subject: [PATCH] mm/mglru: gate the PUD rmap feedback on filter engagement
Content-type: text/plain

lru_gen_look_around() updates the PUD-level Bloom filter so that regions
whose hotness is only observed by eviction stay marked and are re-scanned
by the aging walker. But on systems where the PUD filter never skips
(e.g. small-memory devices where a PUD covers too much address space to
be useful), this is pure overhead on the page-fault hot path.

Gate the PUD feedback on a per-lruvec flag that is set when the aging
walker actually skips a PUD and cleared at the start of each generation.
look_around() then pays only a single flag load while no PUD is being
skipped, and the feedback engages exactly when the filter is active.
PUDs are only marked from the eviction side when the aging is really
skipping them. So hot regions stay protected where the filter is active,
and machines that never benefit get zero extra cost.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 include/linux/mmzone.h | 2 ++
 mm/vmscan.c            | 9 +++++++--
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 276d0ca8d1c1..803431dfac80 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -617,6 +617,8 @@ struct lru_gen_mm_state {
 	unsigned long *pmd_filters[NR_BLOOM_FILTERS];
 	/* PUD-level Bloom filters flip after each iteration */
 	unsigned long *pud_filters[NR_BLOOM_FILTERS];
+	/* whether the aging has skipped a PUD this generation */
+	bool pud_filter_used;
 	/* the mm stats for debugging */
 	unsigned long stats[NR_HIST_GENS][NR_MM_STATS];
 };
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 035bd9ea49aa..22851b02264f 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3143,8 +3143,11 @@ static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite
 	if (!mm_state->head)
 		mm_state->head = &mm_list->fifo;
 
-	if (mm_state->head == &mm_list->fifo)
+	if (mm_state->head == &mm_list->fifo) {
+		/* new generation: reset the PUD-filter engagement flag */
+		WRITE_ONCE(mm_state->pud_filter_used, false);
 		first = true;
+	}
 
 	do {
 		mm_state->head = mm_state->head->next;
@@ -3858,6 +3861,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,
 
 		/* Skip a subtree whose 512 PMDs all failed the PMD-level filter last gen */
 		if (!walk->force_scan && !test_pud_bloom_filter(mm_state, walk->seq, pud + i)) {
+			WRITE_ONCE(mm_state->pud_filter_used, true);
 			walk->mm_stats[MM_PUD_EMPTY_SKIPPED]++;
 			continue;
 		}
@@ -4434,7 +4438,8 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 	lazy_mmu_mode_disable();
 
 	/* feedback from rmap walkers to page table walkers */
-	if (mm_state && suitable_to_scan(i, young)) {
+	if (mm_state && suitable_to_scan(i, young) &&
+	    READ_ONCE(mm_state->pud_filter_used)) {
 		/* the PUD entry covering the young PTEs scanned above */
 		pud_t *pud_p = pud_offset(p4d_offset(pgd_offset(vma->vm_mm, pvmw->address),
 						     pvmw->address), pvmw->address);
-- 
2.54.0

Re: [PATCH v2 3/4] mm/mglru: skip cold PUD subtrees during aging

[Barry Song]

On Wed, Sep 2, 2026 at 11:22 AM Baoquan He <baoquan.he@linux.dev> wrote:
>
> On 09/02/26 at 07:53am, Barry Song wrote:
> > On Tue, Sep 1, 2026 at 2:38 PM Baoquan He <hebaoquan@kylinos.cn> wrote:
> > >
> > > The aging walks into every PUD and runs the PMD-level Bloom filter
> > > on each PMD. Add a coarser PUD-level filter (pud_filters) one level
> > > up:
> > >  - walk_pmd_range() now reports whether it found any young leaf entries,
> > >  - walk_pud_range() records that in the PUD filter, and
> > >  - on subsequent generations, skips the whole 1GB subtree when the filter
> > >    says it had none last generation.
> > >
> > > The double-buffered filter flips with each new iteration, so newly hot or
> > > migrated-in pages are re-checked promptly rather than suppressed
> > > indefinitely. force_scan walks bypass the PUD test, so manual aging and
> > > newly added mm's always rescan and re-populate the filter.
> > >
> > > To keep hot regions marked, also report the covering PUD from the rmap
> > > feedback path (lru_gen_look_around()), so regions whose hotness is only
> > > observed by eviction will be re-scanned.
> > >
> > > Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
> > > ---
> > >  mm/vmscan.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++-----
> > >  1 file changed, 47 insertions(+), 5 deletions(-)
> > [...]
> > > @@ -4397,8 +4431,14 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
> > >         lazy_mmu_mode_disable();
> > >
> > >         /* feedback from rmap walkers to page table walkers */
> > > -       if (mm_state && suitable_to_scan(i, young))
> > > +       if (mm_state && suitable_to_scan(i, young)) {
> > > +               /* the PUD entry covering the young PTEs scanned above */
> > > +               pud_t *pud_p = pud_offset(p4d_offset(pgd_offset(vma->vm_mm, pvmw->address),
> > > +                                                    pvmw->address), pvmw->address);
> > > +
> > >                 update_pmd_bloom_filter(mm_state, max_seq, pvmw->pmd);
> > > +               update_pud_bloom_filter(mm_state, max_seq, pud_p);
> > > +       }
> >
> >
> > Hi Baoquan,
> >
> > I am really not against this idea, and I believe it can benefit
> > NUMA cases.
> >
> > That said, I might be being overly cautious, but I'm a bit concerned
> > that this re-walk of the page tables could slightly hurt machines that
> > don't benefit from the PUD filter at all. For example, Android devices
> > typically have relatively small amounts of memory, so a PUD covers a
> > really large range of their address space, which is unlikely to be
> > useful on an 8 GB Android device. Also, `lru_gen_look_around()` is a
> > really hot path. On Android, we've observed that it can consume a
> > significant amount of CPU.
> >
> > Is there any possibility of implementing this in a low-cost way?
>
> I made a draft patch, could you help check and test if it's performing
> better on Andriod device?
>

Hi Baoquan,

Actually, it is quite difficult to run the latest kernel, or even a
patch against the latest kernel, on an Android device. However, I can
run it on my x86 PC, for example, by setting `mem=4096M` or `mem=8192M`
in the bootargs to simulate a low-end phone.

I may run kernel-build on it and collect the perf data. Once I have
the data, I'll get back to you.

On the other hand, you could also investigate this in parallel by
emulating a low-end device. :-)

Best Regards
Barry

Re: [PATCH v2 3/4] mm/mglru: skip cold PUD subtrees during aging

[Baoquan He]

On 09/04/26 at 11:41am, Barry Song wrote:
> On Wed, Sep 2, 2026 at 11:22 AM Baoquan He <baoquan.he@linux.dev> wrote:
> >
> > On 09/02/26 at 07:53am, Barry Song wrote:
> > > On Tue, Sep 1, 2026 at 2:38 PM Baoquan He <hebaoquan@kylinos.cn> wrote:
> > > >
> > > > The aging walks into every PUD and runs the PMD-level Bloom filter
> > > > on each PMD. Add a coarser PUD-level filter (pud_filters) one level
> > > > up:
> > > >  - walk_pmd_range() now reports whether it found any young leaf entries,
> > > >  - walk_pud_range() records that in the PUD filter, and
> > > >  - on subsequent generations, skips the whole 1GB subtree when the filter
> > > >    says it had none last generation.
> > > >
> > > > The double-buffered filter flips with each new iteration, so newly hot or
> > > > migrated-in pages are re-checked promptly rather than suppressed
> > > > indefinitely. force_scan walks bypass the PUD test, so manual aging and
> > > > newly added mm's always rescan and re-populate the filter.
> > > >
> > > > To keep hot regions marked, also report the covering PUD from the rmap
> > > > feedback path (lru_gen_look_around()), so regions whose hotness is only
> > > > observed by eviction will be re-scanned.
> > > >
> > > > Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
> > > > ---
> > > >  mm/vmscan.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++-----
> > > >  1 file changed, 47 insertions(+), 5 deletions(-)
> > > [...]
> > > > @@ -4397,8 +4431,14 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
> > > >         lazy_mmu_mode_disable();
> > > >
> > > >         /* feedback from rmap walkers to page table walkers */
> > > > -       if (mm_state && suitable_to_scan(i, young))
> > > > +       if (mm_state && suitable_to_scan(i, young)) {
> > > > +               /* the PUD entry covering the young PTEs scanned above */
> > > > +               pud_t *pud_p = pud_offset(p4d_offset(pgd_offset(vma->vm_mm, pvmw->address),
> > > > +                                                    pvmw->address), pvmw->address);
> > > > +
> > > >                 update_pmd_bloom_filter(mm_state, max_seq, pvmw->pmd);
> > > > +               update_pud_bloom_filter(mm_state, max_seq, pud_p);
> > > > +       }
> > >
> > >
> > > Hi Baoquan,
> > >
> > > I am really not against this idea, and I believe it can benefit
> > > NUMA cases.
> > >
> > > That said, I might be being overly cautious, but I'm a bit concerned
> > > that this re-walk of the page tables could slightly hurt machines that
> > > don't benefit from the PUD filter at all. For example, Android devices
> > > typically have relatively small amounts of memory, so a PUD covers a
> > > really large range of their address space, which is unlikely to be
> > > useful on an 8 GB Android device. Also, `lru_gen_look_around()` is a
> > > really hot path. On Android, we've observed that it can consume a
> > > significant amount of CPU.
> > >
> > > Is there any possibility of implementing this in a low-cost way?
> >
> > I made a draft patch, could you help check and test if it's performing
> > better on Andriod device?
> >
> 
> Hi Baoquan,
> 
> Actually, it is quite difficult to run the latest kernel, or even a
> patch against the latest kernel, on an Android device. However, I can
> run it on my x86 PC, for example, by setting `mem=4096M` or `mem=8192M`
> in the bootargs to simulate a low-end phone.
> 
> I may run kernel-build on it and collect the perf data. Once I have
> the data, I'll get back to you.
> 
> On the other hand, you could also investigate this in parallel by
> emulating a low-end device. :-)

Sure, I will test and investigate from my side. Thank you very much for
the effort on testing and investigation.