From: Baoquan He <baoquan.he@linux.dev>
To: linux-mm@kvack.org
Cc: akpm@linux-foundation.org, david@kernel.org, kasong@tencent.com,
baohua@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev,
axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com,
baoquan.he@linux.dev, baolin.wang@linux.alibaba.com
Subject: [PATCH 0/4] skip empty PUD subtrees during aging with a PUD-level Bloom filter
Date: Fri, 20 Nov 2026 16:58:02 +0800 [thread overview]
Message-ID: <20261120085806.781225-1-baoquan.he@linux.dev> (raw)
Problem
=======
MGLRU's aging walks every PUD of every mm in the mm_list. For a PUD whose
512 PMDs all fail the existing PMD-level Bloom filter, the walker still
descends into the PUD and iterates all 512 PMD entries. When an mm has no
pages on a given node this is structural waste: lru_gen_use_mm() sets
mm->lru_gen.bitmap to -1 (all nodes) on every context switch, so the aging
walker of *every* node, e.g kswapd, direct reclaim, proactive aging walks
the same mm.
On a 2-node KVM guest with the workload pinned to node 0, node 1's aging
walks of those mms scan 1,362 leaf entries per walk while finding no young
folio for node 1's lruvec, and 94% of those walks are entirely empty.
Approach
========
Add a PUD-level Bloom filter, one level above the existing PMD filter.
The walker now:
1. Tests the PUD filter before descending into a PUD subtree.
2. If the PUD had nothing young for this lruvec last generation, skips the
whole 1 GiB region -- avoiding 512 PMD lookups.
3. walk_pmd_range() reports whether it found anything young;
walk_pud_range() records that in the double-buffered PUD filter.
4. The rmap feedback path marks hot PUDs, so regions whose hotness is only
observed by eviction are re-examined promptly. This is done for both
mapping types: lru_gen_look_around() covers PTE-mapped folios, and
folio_referenced_one() covers PMD-mapped ones, which never reach
lru_gen_look_around().
5. force_scan walks bypass the PUD test, so manual aging and newly added
mms always rescan and re-populate the filter.
The filter key is (mm, PUD-aligned address), as suggested by Kairui. It is
a stable identity of the range on both sides of the interface: the aging
walk computes it from args->mm and addr, the rmap path from vma->vm_mm and
the address. So the feedback path needs no page-table walk at all.
The double-buffered filter flips each generation and force_scan bypasses
it, so a missed re-mark costs at most one generation, and reclaim is still
safe for a wrongly skipped subtree because folio_referenced() consults the
PTE young bits, not the filter.
Test
====
8 GiB KVM guest, 4 vCPU, two 4 GiB NUMA nodes, 7.4.0-rc3 with the series.
"base" below is the same kernel with the filter disabled (patch 1 only):
same .config, same workload, medians of 3 runs.
1) Cross-node empty walks. 16 workers x 128 MiB pinned to node 0
(numactl --membind=0), so node 1 owns none of these pages. 100 aging
passes, per pass, both nodes aged:
base filter change
node 1 leaf entries scanned 1,214 100 -92%
walks with leaves 17 1 -94%
PUDs skipped 0 52
node 0 leaf entries scanned 12,559 12,288 -2.2%
leaf entries assoc. 9,164 9,175 +0.1%
worker RSS delta 0% 0%
forced pass, node 1 leaf/walk
before the filtered passes 1,363 1,362
after them 1,363 1,369
A forced pass bypasses the filter, so it costs the same on both kernels
and the same before and after 100 filtered passes: the filter is a
cache, not a state. Node 0 keeps its young leaves (+0.1%), so the
coarser filter does not starve the node that owns them.
2) Correctness, 11 checks, all pass: a cold PUD is skipped; touching it
does not revive it; force_scan re-marks it and the next pass walks it
again; a munmap + mmap at the same address shows no stale suppression; a
1 GiB range migrated from node 0 to node 1 is re-scanned by node 1 after
one forced pass; no premature reclaim in any of them; dmesg clean.
3) A PUD that holds only PMD-mapped huge pages. 1 GiB is mapped with
THP=always (AnonHugePages = 1 GiB) and aged until its PUD is skipped
(S +4 over 3 passes). The memcg is then put under reclaim pressure
(memory.max below the current usage) and the range is touched again, so
folio_referenced() runs on PMD-mapped folios. The next normal aging
pass walks the range again:
leaf_total 1132, leaf_associated 513
This is what v4 adds. In v3 the range stayed skipped, because
folio_referenced_one() takes the pmdp_clear_flush_young_notify() branch
for huge pages and never reaches lru_gen_look_around().
4) Reclaim pressure. Four 512 MiB workers in a memory.max=2G/swap.max=1G
memcg, i.e. exactly at the limit, so reclaim is always active; each
touches one 256 MiB half and flips it every second, so a half is cold
for a few aging generations and then hot again. Aging runs on both
nodes every 0.5 s, for 600 s. The base ran twice, the filter three
times:
oom_kill 0 vs 0
hot half resident, min 262,051 vs 262,120 (of 262,144 pages)
pgsteal_anon 2.12M vs 2.24M
pswpout 2.12M vs 2.24M
workingset_refault_anon 2.12M vs 2.24M
pgscan_anon 14.2M vs 17.6M
node 1 walks with leaves 4,532 vs 4 -99.9%
No OOM and no residency loss. Evictions, swap-out and refaults overlap
between the two kernels (base 2.12-2.61M, filter 2.03-2.26M), so this is
not a measurable difference. pgscan_anon is 15% higher; note that it
counts folios examined by the LRU scanner on both nodes, not page-table
walks, so it is not a measure of the filter's own work. PUD subtrees
skipped on node 0 are 0.16-0.20% of its associated leaves.
5) Low memory. Aging cost on a single node, CPU time per pass taken from
the utime+stime of the task that writes to debugfs, 200 passes:
shape base filter
2 procs x 32 GiB VA, 0.2% touched 3,707 us 3,191 us -14%
20 procs x 3 GiB VA, 2% touched 11,008 us 10,573 us
20 procs x 64 MiB, fully touched 10,962 us 10,783 us
The leaves scanned, the leaves associated and the number of walks with
leaves are the same in all three rows. If a mm has only a few 1 GiB
subtrees the filter has nothing to save and it spends nothing. On the
wide shape it saves about 14%, i.e. what 512 PMD-level Bloom tests cost
per skipped subtree.
Change log
==========
v3->v4:
- Mark a PUD young only when a huge page is really young for this lruvec.
walk_pmd_range_locked() returns that now, and the accessed bit of a
non-leaf PMD no longer marks anything. (Sashiko)
- Mark the PUD from folio_referenced_one() too, so a PUD that holds only
PMD-mapped huge pages can be re-marked. (Sashiko)
- Statistics: count the batched entries of a large folio in
MM_LEAF_ASSOCIATED, move the tracepoint inside #ifdef CONFIG_LRU_GEN,
and word the comment so MM_LEAF_ASSOCIATED and MM_WALK_EMPTY clearly
mean "young". (Sashiko)
- Bloom filter helpers: fold the __ functions into
test/update/reset_bloom_filter(), and describe both levels in the
comment, including why they keep the same m and k and that two levels of
double buffering take 16 KiB per lruvec. (Kairui)
v2 -> v3:
- Use (mm, PUD-aligned address) as the PUD filter key instead of the PUD
page-table pointer. This removes the pud_offset(p4d_offset(pgd_offset()))
page-table walk that the v2 lru_gen_look_around() hunk had to do to
obtain the key with withc Barry is very worried about adding cost to a
hot paths. This is suggested by Kairui.
- Store both levels in one array, filters[BLOOM_FILTER_LEVELS]
[NR_BLOOM_FILTERS], instead of separate pmd_filters/pud_filters
(Kairui's idea).
- Count skipped subtrees (MM_PUD_EMPTY_SKIPPED, debugfs column "S" of
TYFALWEES, and a tracepoint field) so the avoided cost is measurable.
Baoquan He (4):
mm/mglru: add MM_WALK_EMPTY stats and tracepoint
mm/mglru: add PUD-level Bloom filter state and generic helpers
mm/mglru: skip cold PUD subtrees during aging
mm/mglru: count PUD subtrees skipped by the PUD-level filter
include/linux/mmzone.h | 21 ++++-
mm/rmap.c | 5 +-
mm/trace_vmscan.h | 30 ++++++++
mm/vmscan.c | 171 +++++++++++++++++++++++++++++++++--------
4 files changed, 192 insertions(+), 35 deletions(-)
base-commit: 9f004372ebeb18fdeb699f9880a0e472df90135b
--
2.52.0
next reply other threads:[~2026-11-20 8:58 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 8:58 Baoquan He [this message]
2026-11-20 8:58 ` [PATCH 1/4] mm/mglru: add MM_WALK_EMPTY stats and tracepoint Baoquan He
2026-11-20 8:58 ` [PATCH 2/4] mm/mglru: add PUD-level Bloom filter state and generic helpers Baoquan He
2026-11-20 8:58 ` [PATCH 3/4] mm/mglru: skip cold PUD subtrees during aging Baoquan He
2026-11-20 8:58 ` [PATCH 4/4] mm/mglru: count PUD subtrees skipped by the PUD-level filter Baoquan He
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120085806.781225-1-baoquan.he@linux.dev \
--to=baoquan.he@linux.dev \
--cc=akpm@linux-foundation.org \
--cc=axelrasmussen@google.com \
--cc=baohua@kernel.org \
--cc=baolin.wang@linux.alibaba.com \
--cc=david@kernel.org \
--cc=kasong@tencent.com \
--cc=linux-mm@kvack.org \
--cc=qi.zheng@linux.dev \
--cc=shakeel.butt@linux.dev \
--cc=weixugc@google.com \
--cc=yuanchu@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.