From: Kairui Song <ryncsn@gmail.com>
To: Baoquan He <baoquan.he@linux.dev>
Cc: linux-mm@kvack.org, akpm@linux-foundation.org, david@kernel.org,
kasong@tencent.com, baohua@kernel.org,
baolin.wang@linux.alibaba.com, qi.zheng@linux.dev,
shakeel.butt@linux.dev, axelrasmussen@google.com,
yuanchu@google.com, weixugc@google.com,
Baoquan He <hebaoquan@kylinos.cn>
Subject: Re: [PATCH v3 2/4] mm/mglru: add PUD-level Bloom filter state and generic helpers
Date: Tue, 17 Nov 2026 00:24:03 +0800 [thread overview]
Message-ID: <avq-hc6E4Urpf6x4@KASONG-MC4> (raw)
In-Reply-To: <20261113084953.412919-3-baoquan.he@linux.dev>
On Fri, Nov 13, 2026 at 04:49:51PM +0800, Baoquan He wrote:
> From: Baoquan He <hebaoquan@kylinos.cn>
>
> Generalize the Bloom filter state and helpers so that another filter
> level can reuse the same hash, double-buffering and reset logic.
>
> Replace the PMD-level filter array with a two-dimensional filters array
> indexed by level, and add BLOOM_FILTER_PUD and BLOOM_FILTER_PMD. The
> PUD-level filters operate at PUD granularity and generalize the
> suppression one level up: when all PMDs under a PUD fail the PMD-level
> filter, the whole PMD iteration can be skipped. Cross-node empty walks
> are the most visible case: a foreign mm's PUDs have no young entries for
> this lruvec, so the entire PMD iteration is pure waste.
>
> The existing PMD-level code now goes through the generic
> test/update/reset_bloom_filter() helpers.
>
> No behavior change yet; the PUD-level filters are populated and used by
> the next patch.
>
> Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
> Signed-off-by: Baoquan He <baoquan.he@linux.dev>
...
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 98037a562ab6..9dbb3f477d7d 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -2798,6 +2798,10 @@ static bool __maybe_unused seq_is_valid(struct lruvec *lruvec)
> * walk_pmd_range(); the eviction also report them when walking the rmap
> * in lru_gen_look_around().
> *
Not a problem for this patch, but the comment above may want an update
in the next commit. This describes things only at the PMD level, while
the PUD level, the ~1/5 and ~1/2 FP figures still hold for (same m, k
and n). But the PMD false positive only causes a wasted PTE walk if
the PUD gate also passes, so the effective FP rate is weighted by
the PUD part too. The PUD filter has its own FP, which is rarer but
costs a whole 1 GiB subtree, so maybe the 4 KiB bitmap for PUD
is not that obviously oversized (despite the PUD number are much
lower than PMD we are using same sized bitmaps).
> + * A second, coarser pair of filters sits one level up. The filters are
> + * indexed by level: BLOOM_FILTER_PMD for the PMD-level filters above, and
> + * BLOOM_FILTER_PUD for the PUD-level pair.
> + *
> * For future optimizations:
> * 1. It's not necessary to keep both filters all the time. The spare one can be
> * freed after the RCU grace period and reallocated if needed again.
Still not a problem but this optimization is more important now. We have a
extra 8KB peak overhead per lruvec if I did the math right, not too bad
as there won't be too many of them, even on large machines (memcg number
is currently hard limited to 64k and they are lazily allocated). But
worth mentioning and optimized later.
> @@ -2823,14 +2827,13 @@ static void get_item_key(void *item, int *key)
> key[1] = hash >> BLOOM_FILTER_SHIFT;
> }
>
> -static bool test_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq,
> - void *item)
> +static bool __test_bloom_filter(unsigned long **filters, unsigned long seq, void *item)
There is only one caller of this (test_bloom_filter), do we really
need the __ version?
> {
> int key[2];
> unsigned long *filter;
> int gen = filter_gen_from_seq(seq);
>
> - filter = READ_ONCE(mm_state->filters[gen]);
> + filter = READ_ONCE(filters[gen]);
> if (!filter)
> return true;
>
> @@ -2839,14 +2842,13 @@ static bool test_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long s
> return test_bit(key[0], filter) && test_bit(key[1], filter);
> }
>
> -static void update_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq,
> - void *item)
> +static void __update_bloom_filter(unsigned long **filters, unsigned long seq, void *item)
Also this one, only one caller, the __ intermediate layer seems unnecessary.
> {
> int key[2];
> unsigned long *filter;
> int gen = filter_gen_from_seq(seq);
>
> - filter = READ_ONCE(mm_state->filters[gen]);
> + filter = READ_ONCE(filters[gen]);
> if (!filter)
> return;
>
> @@ -2858,12 +2860,12 @@ static void update_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long
> set_bit(key[1], filter);
> }
>
> -static void reset_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq)
> +static void __reset_bloom_filter(unsigned long **filters, unsigned long seq)
> {
> unsigned long *filter;
> int gen = filter_gen_from_seq(seq);
>
> - filter = mm_state->filters[gen];
> + filter = filters[gen];
> if (filter) {
> bitmap_clear(filter, 0, BIT(BLOOM_FILTER_SHIFT));
> return;
> @@ -2871,7 +2873,25 @@ static void reset_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long
>
> filter = bitmap_zalloc(BIT(BLOOM_FILTER_SHIFT),
> __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN);
> - WRITE_ONCE(mm_state->filters[gen], filter);
> + WRITE_ONCE(filters[gen], filter);
> +}
> +
> +static bool test_bloom_filter(struct lru_gen_mm_state *mm_state, int level,
> + unsigned long seq, void *item)
> +{
> + return __test_bloom_filter(mm_state->filters[level], seq, item);
> +}
> +
> +static void update_bloom_filter(struct lru_gen_mm_state *mm_state, int level,
> + unsigned long seq, void *item)
> +{
> + __update_bloom_filter(mm_state->filters[level], seq, item);
> +}
> +
> +static void reset_bloom_filter(struct lru_gen_mm_state *mm_state, int level,
> + unsigned long seq)
> +{
> + __reset_bloom_filter(mm_state->filters[level], seq);
> }
>
> /******************************************************************************
> @@ -3114,7 +3134,7 @@ static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite
> spin_unlock(&mm_list->lock);
>
> if (mm && first)
> - reset_bloom_filter(mm_state, walk->seq + 1);
> + reset_bloom_filter(mm_state, BLOOM_FILTER_PMD, walk->seq + 1);
>
> if (*iter)
> mmdrop(*iter);
> @@ -3771,7 +3791,8 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
> walk_pmd_range_locked(pud, addr, vma, args, bitmap, &first);
> }
>
> - if (!walk->force_scan && !test_bloom_filter(mm_state, walk->seq, pmd + i))
> + if (!walk->force_scan &&
> + !test_bloom_filter(mm_state, BLOOM_FILTER_PMD, walk->seq, pmd + i))
This looks nice.
> continue;
>
> walk->mm_stats[MM_NONLEAF_FOUND]++;
> @@ -3782,7 +3803,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
> walk->mm_stats[MM_NONLEAF_ADDED]++;
>
> /* carry over to the next generation */
> - update_bloom_filter(mm_state, walk->seq + 1, pmd + i);
> + update_bloom_filter(mm_state, BLOOM_FILTER_PMD, walk->seq + 1, pmd + i);
> }
>
> walk_pmd_range_locked(pud, -1, vma, args, bitmap, &first);
> @@ -4417,8 +4438,8 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
>
> /* feedback from rmap walkers to page table walkers */
> if (mm_state && suitable_to_scan(i, young))
> - update_bloom_filter(mm_state, READ_ONCE(lruvec->lrugen.max_seq),
> - pvmw->pmd);
> + update_bloom_filter(mm_state, BLOOM_FILTER_PMD,
> + READ_ONCE(lruvec->lrugen.max_seq), pvmw->pmd);
>
> mem_cgroup_put(memcg);
>
> @@ -5981,6 +6002,7 @@ void lru_gen_init_memcg(struct mem_cgroup *memcg)
> void lru_gen_exit_memcg(struct mem_cgroup *memcg)
> {
> int i;
> + int level;
> int nid;
> struct lru_gen_mm_list *mm_list = get_mm_list(memcg);
>
> @@ -5998,9 +6020,11 @@ void lru_gen_exit_memcg(struct mem_cgroup *memcg)
> if (!mm_state)
> continue;
>
> - for (i = 0; i < NR_BLOOM_FILTERS; i++) {
> - bitmap_free(mm_state->filters[i]);
> - mm_state->filters[i] = NULL;
> + for (level = 0; level < BLOOM_FILTER_LEVELS; level++) {
> + for (i = 0; i < NR_BLOOM_FILTERS; i++) {
> + bitmap_free(mm_state->filters[level][i]);
> + mm_state->filters[level][i] = NULL;
> + }
So just a few nit style and comment issue from me, rest looks good, thanks!
Reviewed-by: Kairui Song <kasong@tencent.com>
next prev parent reply other threads:[~2026-11-16 16:24 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-13 8:49 [PATCH v3 0/4] skip empty PUD subtrees during aging with a PUD-level Bloom filter Baoquan He
2026-11-13 8:49 ` [PATCH v3 1/4] mm/mglru: add MM_WALK_EMPTY stats and tracepoint Baoquan He
2026-11-13 8:53 ` Baoquan He
2026-11-14 3:42 ` sashiko-bot
2026-11-16 7:24 ` Baoquan He
2026-11-13 8:49 ` [PATCH v3 2/4] mm/mglru: add PUD-level Bloom filter state and generic helpers Baoquan He
2026-11-16 16:24 ` Kairui Song [this message]
2026-11-13 8:49 ` [PATCH v3 3/4] mm/mglru: skip cold PUD subtrees during aging Baoquan He
2026-11-14 3:42 ` sashiko-bot
2026-11-16 9:13 ` Baoquan He
2026-11-13 8:49 ` [PATCH v3 4/4] mm/mglru: count PUD subtrees skipped by the PUD-level filter Baoquan He
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=avq-hc6E4Urpf6x4@KASONG-MC4 \
--to=ryncsn@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=axelrasmussen@google.com \
--cc=baohua@kernel.org \
--cc=baolin.wang@linux.alibaba.com \
--cc=baoquan.he@linux.dev \
--cc=david@kernel.org \
--cc=hebaoquan@kylinos.cn \
--cc=kasong@tencent.com \
--cc=linux-mm@kvack.org \
--cc=qi.zheng@linux.dev \
--cc=shakeel.butt@linux.dev \
--cc=weixugc@google.com \
--cc=yuanchu@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.