From: Max Kellermann <max.kellermann@ionos.com>
To: akpm@linux-foundation.org, kasong@tencent.com,
qi.zheng@linux.dev, shakeel.butt@linux.dev, baohua@kernel.org,
axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com,
baoquan.he@linux.dev, baolin.wang@linux.alibaba.com,
david@kernel.org, ljs@kernel.org, liam@infradead.org,
vbabka@kernel.org, rppt@kernel.org, surenb@google.com,
mhocko@suse.com, hannes@cmpxchg.org, linux-mm@kvack.org,
linux-kernel@vger.kernel.org, rcu@vger.kernel.org
Cc: Max Kellermann <max.kellermann@ionos.com>
Subject: [PATCH 2/5] mm/vmscan: move code to lru_gen_{add,del}_memcg()
Date: Mon, 12 Oct 2026 22:54:44 +0200 [thread overview]
Message-ID: <20261012205447.2717801-3-max.kellermann@ionos.com> (raw)
In-Reply-To: <20261012205447.2717801-1-max.kellermann@ionos.com>
Preparation for the following patch that adds tail caching.
For simplicity, hlist_nulls_del_init_rcu() is now always used, which
is also valid during rotation, because the entry is immediately
reinserted under `memcg_lru.lock`.
Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
---
mm/vmscan.c | 36 ++++++++++++++++++++++++++++--------
1 file changed, 28 insertions(+), 8 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index f11491ee9ed5..3d4bc12cf18e 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4379,6 +4379,29 @@ enum {
MEMCG_LRU_YOUNG,
};
+static void lru_gen_add_memcg(struct lru_gen_memcg *memcg_lru,
+ struct lru_gen_folio *lrugen,
+ int gen, int bin, bool at_head)
+{
+ struct hlist_nulls_head *head = &memcg_lru->fifo[gen][bin];
+
+ lockdep_assert_held(&memcg_lru->lock);
+
+ if (at_head)
+ hlist_nulls_add_head_rcu(&lrugen->list, head);
+ else
+ hlist_nulls_add_tail_rcu(&lrugen->list, head);
+}
+
+static void lru_gen_del_memcg(struct lru_gen_memcg *memcg_lru,
+ struct lru_gen_folio *lrugen,
+ int gen)
+{
+ lockdep_assert_held(&memcg_lru->lock);
+
+ hlist_nulls_del_init_rcu(&lrugen->list);
+}
+
static void lru_gen_rotate_memcg(struct lruvec *lruvec, int op)
{
int seg;
@@ -4409,12 +4432,9 @@ static void lru_gen_rotate_memcg(struct lruvec *lruvec, int op)
WRITE_ONCE(lruvec->lrugen.seg, seg);
WRITE_ONCE(lruvec->lrugen.gen, new);
- hlist_nulls_del_rcu(&lruvec->lrugen.list);
-
- if (op == MEMCG_LRU_HEAD || op == MEMCG_LRU_OLD)
- hlist_nulls_add_head_rcu(&lruvec->lrugen.list, &pgdat->memcg_lru.fifo[new][bin]);
- else
- hlist_nulls_add_tail_rcu(&lruvec->lrugen.list, &pgdat->memcg_lru.fifo[new][bin]);
+ lru_gen_del_memcg(&pgdat->memcg_lru, &lruvec->lrugen, old);
+ lru_gen_add_memcg(&pgdat->memcg_lru, &lruvec->lrugen, new, bin,
+ op == MEMCG_LRU_HEAD || op == MEMCG_LRU_OLD);
pgdat->memcg_lru.nr_memcgs[old]--;
pgdat->memcg_lru.nr_memcgs[new]++;
@@ -4445,7 +4465,7 @@ void lru_gen_online_memcg(struct mem_cgroup *memcg)
lruvec->lrugen.gen = gen;
- hlist_nulls_add_tail_rcu(&lruvec->lrugen.list, &pgdat->memcg_lru.fifo[gen][bin]);
+ lru_gen_add_memcg(&pgdat->memcg_lru, &lruvec->lrugen, gen, bin, false);
pgdat->memcg_lru.nr_memcgs[gen]++;
spin_unlock_irq(&pgdat->memcg_lru.lock);
@@ -4479,7 +4499,7 @@ void lru_gen_release_memcg(struct mem_cgroup *memcg)
gen = lruvec->lrugen.gen;
- hlist_nulls_del_init_rcu(&lruvec->lrugen.list);
+ lru_gen_del_memcg(&pgdat->memcg_lru, &lruvec->lrugen, gen);
pgdat->memcg_lru.nr_memcgs[gen]--;
if (!pgdat->memcg_lru.nr_memcgs[gen] && gen == get_memcg_gen(pgdat->memcg_lru.seq))
--
2.47.3
next prev parent reply other threads:[~2026-10-12 20:55 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-12 20:54 [PATCH 0/5] mm: multi-gen LRU: reduce memcg rotation overhead Max Kellermann
2026-10-12 20:54 ` [PATCH 1/5] include/linux/rculist_nulls: add hlist_nulls_add_behind_rcu() Max Kellermann
2026-10-12 20:54 ` Max Kellermann [this message]
2026-10-12 20:54 ` [PATCH 3/5] mm/mglru: cache the last memcg node, make tail insertion constant time Max Kellermann
2026-10-12 20:54 ` [PATCH 4/5] mm/mglru: avoid redundant rotations of young memcgs Max Kellermann
2026-10-12 20:54 ` [PATCH 5/5] mm/mglru: reschedule between memcg reclaim attempts Max Kellermann
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261012205447.2717801-3-max.kellermann@ionos.com \
--to=max.kellermann@ionos.com \
--cc=akpm@linux-foundation.org \
--cc=axelrasmussen@google.com \
--cc=baohua@kernel.org \
--cc=baolin.wang@linux.alibaba.com \
--cc=baoquan.he@linux.dev \
--cc=david@kernel.org \
--cc=hannes@cmpxchg.org \
--cc=kasong@tencent.com \
--cc=liam@infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ljs@kernel.org \
--cc=mhocko@suse.com \
--cc=qi.zheng@linux.dev \
--cc=rcu@vger.kernel.org \
--cc=rppt@kernel.org \
--cc=shakeel.butt@linux.dev \
--cc=surenb@google.com \
--cc=vbabka@kernel.org \
--cc=weixugc@google.com \
--cc=yuanchu@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.