All of lore.kernel.org
 help / color / mirror / Atom feed
From: Baoquan He <baoquan.he@linux.dev>
To: linux-mm@kvack.org
Cc: akpm@linux-foundation.org, david@kernel.org, kasong@tencent.com,
	baohua@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev,
	axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com,
	baoquan.he@linux.dev, baolin.wang@linux.alibaba.com
Subject: [PATCH 0/4] skip empty PUD subtrees during aging with a PUD-level Bloom filter
Date: Fri, 20 Nov 2026 16:58:02 +0800	[thread overview]
Message-ID: <20261120085806.781225-1-baoquan.he@linux.dev> (raw)

Problem
=======

MGLRU's aging walks every PUD of every mm in the mm_list.  For a PUD whose
512 PMDs all fail the existing PMD-level Bloom filter, the walker still
descends into the PUD and iterates all 512 PMD entries.  When an mm has no
pages on a given node this is structural waste: lru_gen_use_mm() sets
mm->lru_gen.bitmap to -1 (all nodes) on every context switch, so the aging
walker of *every* node, e.g kswapd, direct reclaim, proactive aging walks
the same mm.

On a 2-node KVM guest with the workload pinned to node 0, node 1's aging
walks of those mms scan 1,362 leaf entries per walk while finding no young
folio for node 1's lruvec, and 94% of those walks are entirely empty.

Approach
========

Add a PUD-level Bloom filter, one level above the existing PMD filter.
The walker now:

  1. Tests the PUD filter before descending into a PUD subtree.
  2. If the PUD had nothing young for this lruvec last generation, skips the
     whole 1 GiB region -- avoiding 512 PMD lookups.
  3. walk_pmd_range() reports whether it found anything young;
     walk_pud_range() records that in the double-buffered PUD filter.
  4. The rmap feedback path marks hot PUDs, so regions whose hotness is only
     observed by eviction are re-examined promptly.  This is done for both
     mapping types: lru_gen_look_around() covers PTE-mapped folios, and
     folio_referenced_one() covers PMD-mapped ones, which never reach
     lru_gen_look_around().
  5. force_scan walks bypass the PUD test, so manual aging and newly added
     mms always rescan and re-populate the filter.

The filter key is (mm, PUD-aligned address), as suggested by Kairui.  It is
a stable identity of the range on both sides of the interface: the aging
walk computes it from args->mm and addr, the rmap path from vma->vm_mm and
the address. So the feedback path needs no page-table walk at all.

The double-buffered filter flips each generation and force_scan bypasses
it, so a missed re-mark costs at most one generation, and reclaim is still
safe for a wrongly skipped subtree because folio_referenced() consults the
PTE young bits, not the filter.

Test
====

8 GiB KVM guest, 4 vCPU, two 4 GiB NUMA nodes, 7.4.0-rc3 with the series.
"base" below is the same kernel with the filter disabled (patch 1 only):
same .config, same workload, medians of 3 runs.

1) Cross-node empty walks.  16 workers x 128 MiB pinned to node 0
   (numactl --membind=0), so node 1 owns none of these pages.  100 aging
   passes, per pass, both nodes aged:

                                       base   filter change
     node 1  leaf entries scanned     1,214      100   -92%
             walks with leaves           17        1   -94%
             PUDs skipped                 0       52
     node 0  leaf entries scanned    12,559   12,288  -2.2%
             leaf entries assoc.      9,164    9,175  +0.1%
             worker RSS delta            0%       0%

     forced pass, node 1 leaf/walk
       before the filtered passes   1,363  1,362
       after them                   1,363  1,369

   A forced pass bypasses the filter, so it costs the same on both kernels
   and the same before and after 100 filtered passes: the filter is a
   cache, not a state.  Node 0 keeps its young leaves (+0.1%), so the
   coarser filter does not starve the node that owns them.

2) Correctness, 11 checks, all pass: a cold PUD is skipped; touching it
   does not revive it; force_scan re-marks it and the next pass walks it
   again; a munmap + mmap at the same address shows no stale suppression; a
   1 GiB range migrated from node 0 to node 1 is re-scanned by node 1 after
   one forced pass; no premature reclaim in any of them; dmesg clean.

3) A PUD that holds only PMD-mapped huge pages. 1 GiB is mapped with
   THP=always (AnonHugePages = 1 GiB) and aged until its PUD is skipped
   (S +4 over 3 passes). The memcg is then put under reclaim pressure
   (memory.max below the current usage) and the range is touched again, so
   folio_referenced() runs on PMD-mapped folios. The next normal aging
   pass walks the range again:

     leaf_total 1132,  leaf_associated 513

   This is what v4 adds. In v3 the range stayed skipped, because
   folio_referenced_one() takes the pmdp_clear_flush_young_notify() branch
   for huge pages and never reaches lru_gen_look_around().

4) Reclaim pressure. Four 512 MiB workers in a memory.max=2G/swap.max=1G
   memcg, i.e. exactly at the limit, so reclaim is always active; each
   touches one 256 MiB half and flips it every second, so a half is cold
   for a few aging generations and then hot again. Aging runs on both
   nodes every 0.5 s, for 600 s. The base ran twice, the filter three
   times:

     oom_kill 0 vs 0
     hot half resident, min   262,051 vs 262,120  (of 262,144 pages)
     pgsteal_anon              2.12M  vs  2.24M
     pswpout                   2.12M  vs  2.24M
     workingset_refault_anon   2.12M  vs  2.24M
     pgscan_anon               14.2M  vs  17.6M
     node 1 walks with leaves  4,532  vs      4  -99.9%

   No OOM and no residency loss. Evictions, swap-out and refaults overlap
   between the two kernels (base 2.12-2.61M, filter 2.03-2.26M), so this is
   not a measurable difference. pgscan_anon is 15% higher; note that it
   counts folios examined by the LRU scanner on both nodes, not page-table
   walks, so it is not a measure of the filter's own work. PUD subtrees
   skipped on node 0 are 0.16-0.20% of its associated leaves.

5) Low memory. Aging cost on a single node, CPU time per pass taken from
   the utime+stime of the task that writes to debugfs, 200 passes:

     shape                              base      filter
     2 procs x 32 GiB VA, 0.2% touched   3,707 us   3,191 us  -14%
     20 procs x 3 GiB VA, 2% touched    11,008 us  10,573 us
     20 procs x 64 MiB, fully touched   10,962 us  10,783 us

   The leaves scanned, the leaves associated and the number of walks with
   leaves are the same in all three rows. If a mm has only a few 1 GiB
   subtrees the filter has nothing to save and it spends nothing. On the
   wide shape it saves about 14%, i.e. what 512 PMD-level Bloom tests cost
   per skipped subtree.

Change log
==========
v3->v4:
 - Mark a PUD young only when a huge page is really young for this lruvec.
   walk_pmd_range_locked() returns that now, and the accessed bit of a
   non-leaf PMD no longer marks anything.  (Sashiko)

 - Mark the PUD from folio_referenced_one() too, so a PUD that holds only
   PMD-mapped huge pages can be re-marked.  (Sashiko)

 - Statistics: count the batched entries of a large folio in
   MM_LEAF_ASSOCIATED, move the tracepoint inside #ifdef CONFIG_LRU_GEN,
   and word the comment so MM_LEAF_ASSOCIATED and MM_WALK_EMPTY clearly
   mean "young".  (Sashiko)

 - Bloom filter helpers: fold the __ functions into
   test/update/reset_bloom_filter(), and describe both levels in the
   comment, including why they keep the same m and k and that two levels of
   double buffering take 16 KiB per lruvec.  (Kairui)

v2 -> v3:
 - Use (mm, PUD-aligned address) as the PUD filter key instead of the PUD
   page-table pointer.  This removes the pud_offset(p4d_offset(pgd_offset()))
   page-table walk that the v2 lru_gen_look_around() hunk had to do to
   obtain the key with withc Barry is very worried about adding cost to a
   hot paths. This is suggested by Kairui.

 - Store both levels in one array, filters[BLOOM_FILTER_LEVELS]
   [NR_BLOOM_FILTERS], instead of separate pmd_filters/pud_filters
   (Kairui's idea).

 - Count skipped subtrees (MM_PUD_EMPTY_SKIPPED, debugfs column "S" of
   TYFALWEES, and a tracepoint field) so the avoided cost is measurable.

Baoquan He (4):
  mm/mglru: add MM_WALK_EMPTY stats and tracepoint
  mm/mglru: add PUD-level Bloom filter state and generic helpers
  mm/mglru: skip cold PUD subtrees during aging
  mm/mglru: count PUD subtrees skipped by the PUD-level filter

 include/linux/mmzone.h |  21 ++++-
 mm/rmap.c              |   5 +-
 mm/trace_vmscan.h      |  30 ++++++++
 mm/vmscan.c            | 171 +++++++++++++++++++++++++++++++++--------
 4 files changed, 192 insertions(+), 35 deletions(-)


base-commit: 9f004372ebeb18fdeb699f9880a0e472df90135b
-- 
2.52.0



             reply	other threads:[~2026-11-20  8:58 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20  8:58 Baoquan He [this message]
2026-11-20  8:58 ` [PATCH 1/4] mm/mglru: add MM_WALK_EMPTY stats and tracepoint Baoquan He
2026-11-20  8:58 ` [PATCH 2/4] mm/mglru: add PUD-level Bloom filter state and generic helpers Baoquan He
2026-11-20  8:58 ` [PATCH 3/4] mm/mglru: skip cold PUD subtrees during aging Baoquan He
2026-11-20  8:58 ` [PATCH 4/4] mm/mglru: count PUD subtrees skipped by the PUD-level filter Baoquan He

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120085806.781225-1-baoquan.he@linux.dev \
    --to=baoquan.he@linux.dev \
    --cc=akpm@linux-foundation.org \
    --cc=axelrasmussen@google.com \
    --cc=baohua@kernel.org \
    --cc=baolin.wang@linux.alibaba.com \
    --cc=david@kernel.org \
    --cc=kasong@tencent.com \
    --cc=linux-mm@kvack.org \
    --cc=qi.zheng@linux.dev \
    --cc=shakeel.butt@linux.dev \
    --cc=weixugc@google.com \
    --cc=yuanchu@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.