[PATCH v2 0/5] mm: multi-gen LRU: reduce memcg rotation overhead

[PATCH v2 0/5] mm: multi-gen LRU: reduce memcg rotation overhead

[Max Kellermann]

This series addresses excessive time spent rotating memcgs during
MGLRU reclaim on systems with a large number of memcgs.

On some servers with tens of thousands of cgroups, we had repeated
soft lockups and RCU stalls:

 rcu: INFO: rcu_sched self-detected stall on CPU
 rcu:         60-...!: (1118 ticks this GP) idle=ca74/1/0x4000000000000000 softirq=140067260/140067260 fqs=10
 rcu:         (t=2101 jiffies g=180256409 q=94266 ncpus=96)
 rcu: rcu_sched kthread starved for 2073 jiffies! g180256409 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=53
 rcu:         Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
 rcu: RCU grace-period kthread stack dump:
 task:rcu_sched       state:R  running task     stack:0     pid:15    tgid:15    ppid:2      task_flags:0x208040 flags:0x00000010
 Call trace:
  __switch_to+0xc4/0x108 (T)
  __schedule+0x2b4/0x978
  schedule+0x2c/0xc0
  schedule_timeout+0x7c/0x110
  rcu_gp_fqs_loop+0xec/0x4e8
  rcu_gp_kthread+0x12c/0x160
  kthread+0x118/0x1e0
  ret_from_fork+0x10/0x20
 rcu: Stack dump where RCU GP kthread last ran:
 Sending NMI from CPU 60 to CPUs 53:
 NMI backtrace for cpu 53
 CPU: 53 UID: 3758108211 PID: 4052395 Comm: php-cgi8.3 Tainted: G        W    L      6.18.52-i1-ampere #1203 NONE
 Tainted: [W]=WARN, [L]=SOFTLOCKUP
 Hardware name: Supermicro ARS-110M-NR/R12SPD-A, BIOS 1.1b 10/17/2023
 pstate: 40400009 (nZcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
 pc : handle_softirqs+0xa4/0x2e8
 lr : __do_softirq+0x18/0x20
 [...]
 Call trace:
  handle_softirqs+0xa4/0x2e8 (P)
  __do_softirq+0x18/0x20
  ____do_softirq+0x14/0x28
  call_on_irq_stack+0x30/0x48
  do_softirq_own_stack+0x20/0x38
  __irq_exit_rcu+0xc0/0x100
  irq_exit_rcu+0x14/0x28
  el1_interrupt+0x3c/0x58
  el1h_64_irq_handler+0x14/0x20
  el1h_64_irq+0x6c/0x70
  lru_gen_rotate_memcg+0x210/0x280 (P)
  shrink_node+0xb14/0xcc8
  do_try_to_free_pages+0xd0/0x508
  try_to_free_pages+0x11c/0x230
  __alloc_frozen_pages_noprof+0x4f8/0xe00
  get_free_pages_noprof+0x20/0x60
  ceph_alloc_readdir_reply_buffer+0xa4/0x160
  ceph_readdir+0x248/0xeb0
  wrap_directory_iterator+0x54/0x88
  shared_ceph_readdir+0x1c/0x28

The problem was lru_gen_rotate_memcg().  Its memcg tail insertion
walked the destination bin while holding the per-node `memcg_lru.lock`
with interrupts disabled.  With so many cgroups, the lock duration
grew too long and thousands of concurrent reclaimers were contending
for the spinlock all the time.

Reclaim can also repeatedly promote memcgs that are already in the
young generation, and scans of empty or protected memcgs can bypass
the rescheduling points in page reclaim.

This patch set contains three parts:

The first 3 patches add a cache for each bin's tail to make tail
insertion constant time.

The fourth patch skips redundant YOUNG rotations when the memcg is
already in the default segment of the current young generation.

The last patch adds rescheduling points between memcg reclaim attempts
and at bin/retry boundaries to avoid potential remaining soft lockups
under extreme pressure.

I ran a micro benchmark on branch 6.18 on a 12-core Xeon 5118: 8
concurrent workers repeatedly write `1024M swappiness=0` to the root
cgroup's `memory.reclaim` for 60 seconds, against a previously
prepared set of 1500 memory cgroups.  Each worker issues its next
request when the previous one returns. The test measures request
throughput and latency under concurrent MGLRU reclaim.  All requests
returned EAGAIN, indicating that the requested reclaim target was not
met; the reported throughput counts completed reclaim attempts, not
reclaimed bytes.

Results:

                  Requests/sec  Mean latency  Maximum latency
 Unpatched               4.56     1739.85 ms       4083.55 ms
 Patches 1-3            43.42      183.53 ms        393.15 ms
 Patches 1-4            59.22      134.03 ms        393.44 ms
 Patches 1-5            57.80      137.63 ms        412.72 ms

Constant-time insertion increases request throughput by 9.5x.
Avoiding redundant YOUNG rotations adds another 36.4%.  The full patch
series improves the unpatched throughput by a factor of 12.7.

All requests returned EAGAIN.  These results measure reclaim attempts
that did not meet their requested target.

I did not measure scheduling-latency improvements of the last patch.
---
v1->v2: rebase after MEMCG_LRU_HEAD was removed by d5d1712c5928
  ("memcg: remove lru_gen_soft_reclaim()")

Max Kellermann (5):
  include/linux/rculist_nulls: add hlist_nulls_add_behind_rcu()
  mm/vmscan: move code to lru_gen_{add,del}_memcg()
  mm/mglru: cache the last memcg node, make tail insertion constant time
  mm/mglru: avoid redundant rotations of young memcgs
  mm/mglru: reschedule between memcg reclaim attempts

 include/linux/mmzone.h        |  4 ++
 include/linux/rculist_nulls.h | 36 ++++++++++++++---
 mm/vmscan.c                   | 76 ++++++++++++++++++++++++++++++-----
 3 files changed, 101 insertions(+), 15 deletions(-)

-- 
2.53.0

[PATCH v2 1/5] include/linux/rculist_nulls: add hlist_nulls_add_behind_rcu()

[Max Kellermann]

Move some code from hlist_nulls_add_tail_rcu() to the new function
hlist_nulls_add_behind_rcu() for callers that already know the last
node of an `hlist_nulls` list, to allow constant-time appends.

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
---
 include/linux/rculist_nulls.h | 36 +++++++++++++++++++++++++++++------
 1 file changed, 30 insertions(+), 6 deletions(-)

diff --git a/include/linux/rculist_nulls.h b/include/linux/rculist_nulls.h
index a97c3bcb1656..054e477dd093 100644
--- a/include/linux/rculist_nulls.h
+++ b/include/linux/rculist_nulls.h
@@ -115,6 +115,33 @@ static inline void hlist_nulls_add_head_rcu(struct hlist_nulls_node *n,
 		WRITE_ONCE(first->pprev, &n->next);
 }
 
+/**
+ * hlist_nulls_add_behind_rcu - append after the last list entry
+ * @n: the new element to add to the hash list.
+ * @last: the last element in the list.
+ *
+ * Adds the specified element after @last while permitting racing RCU
+ * traversals.  @last must be the last list entry, with its next
+ * pointer holding the terminating nulls marker.
+ *
+ * The caller must take whatever precautions are necessary
+ * (such as holding appropriate locks) to avoid racing
+ * with another list-mutation primitive, such as hlist_nulls_add_head_rcu()
+ * or hlist_nulls_del_rcu(), running on this same list.
+ * However, it is perfectly legal to run concurrently with
+ * the _rcu list-traversal primitives, such as
+ * hlist_nulls_for_each_entry_rcu(), used to prevent memory-consistency
+ * problems on Alpha CPUs.  Regardless of the type of CPU, the
+ * list-traversal primitive must be guarded by rcu_read_lock().
+ */
+static inline void hlist_nulls_add_behind_rcu(struct hlist_nulls_node *n,
+					      struct hlist_nulls_node *last)
+{
+	WRITE_ONCE(n->next, last->next);
+	WRITE_ONCE(n->pprev, &last->next);
+	rcu_assign_pointer(hlist_nulls_next_rcu(last), n);
+}
+
 /**
  * hlist_nulls_add_tail_rcu
  * @n: the element to add to the hash list.
@@ -143,13 +170,10 @@ static inline void hlist_nulls_add_tail_rcu(struct hlist_nulls_node *n,
 	for (i = h->first; !is_a_nulls(i); i = i->next)
 		last = i;
 
-	if (last) {
-		WRITE_ONCE(n->next, last->next);
-		WRITE_ONCE(n->pprev, &last->next);
-		rcu_assign_pointer(hlist_nulls_next_rcu(last), n);
-	} else {
+	if (last)
+		hlist_nulls_add_behind_rcu(n, last);
+	else
 		hlist_nulls_add_head_rcu(n, h);
-	}
 }
 
 /* after that hlist_nulls_del will work */
-- 
2.53.0

Re: [PATCH v2 1/5] include/linux/rculist_nulls: add hlist_nulls_add_behind_rcu()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] rculist_nulls: missing next->pprev update in hlist_nulls_add_behind_rcu()
--

commit 32c8749135647f4c1c26a14e364aff3736e91e2e
Author: Max Kellermann <max.kellermann@ionos.com>

include/linux/rculist_nulls: add hlist_nulls_add_behind_rcu()

This commit factors the non-empty tail insertion logic out of
hlist_nulls_add_tail_rcu() into a new hlist_nulls_add_behind_rcu() helper
so callers tracking the last node can append in constant time.

> diff --git a/include/linux/rculist_nulls.h b/include/linux/rculist_nulls.h
> index a97c3bcb1656e..054e477dd0937 100644
> --- a/include/linux/rculist_nulls.h
> +++ b/include/linux/rculist_nulls.h
> @@ -115,6 +115,33 @@ static inline void hlist_nulls_add_head_rcu(struct hlist_nulls_node *n,
>  		WRITE_ONCE(first->pprev, &n->next);
>  }
>  
> +/**
> + * hlist_nulls_add_behind_rcu - append after the last list entry
> + * @n: the new element to add to the hash list.
> + * @last: the last element in the list.
> + *
> + * Adds the specified element after @last while permitting racing RCU
> + * traversals.  @last must be the last list entry, with its next
> + * pointer holding the terminating nulls marker.
> + *
> + * The caller must take whatever precautions are necessary
> + * (such as holding appropriate locks) to avoid racing
> + * with another list-mutation primitive, such as hlist_nulls_add_head_rcu()
> + * or hlist_nulls_del_rcu(), running on this same list.
> + * However, it is perfectly legal to run concurrently with
> + * the _rcu list-traversal primitives, such as
> + * hlist_nulls_for_each_entry_rcu(), used to prevent memory-consistency
> + * problems on Alpha CPUs.  Regardless of the type of CPU, the
> + * list-traversal primitive must be guarded by rcu_read_lock().
> + */
> +static inline void hlist_nulls_add_behind_rcu(struct hlist_nulls_node *n,
> +					      struct hlist_nulls_node *last)
> +{
> +	WRITE_ONCE(n->next, last->next);
> +	WRITE_ONCE(n->pprev, &last->next);
> +	rcu_assign_pointer(hlist_nulls_next_rcu(last), n);
> +}

[Severity: Medium]
Should hlist_nulls_add_behind_rcu() in include/linux/rculist_nulls.h also
update n->next->pprev when !is_a_nulls(n->next), matching
hlist_add_behind_rcu() in include/linux/rculist.h?

In include/linux/rculist.h, hlist_add_behind_rcu() supports inserting behind
any existing node prev in the list by updating the successor's pprev pointer:

include/linux/rculist.h:hlist_add_behind_rcu() {
	n->next = prev->next;
	WRITE_ONCE(n->pprev, &prev->next);
	rcu_assign_pointer(hlist_next_rcu(prev), n);
	if (n->next)
		WRITE_ONCE(n->next->pprev, &n->next);
}

The other *_add_behind() primitives, hlist_add_behind() in
include/linux/list.h and hlist_bl_add_behind() in include/linux/list_bl.h,
similarly update n->next->pprev so they can be used on interior nodes as well
as the tail.

With hlist_nulls_add_behind_rcu() omitting that update, if a caller passes a
node whose next pointer is not a nulls marker (!is_a_nulls(last->next)),
n->next->pprev remains pointing to &last->next instead of &n->next. Removing
the successor via hlist_nulls_del_rcu() -> __hlist_nulls_del() or replacing
it via hlist_nulls_replace_rcu() would then overwrite last->next and silently
drop n from the list, or cause a use-after-free write if last was removed and
freed first.

Could hlist_nulls_add_behind_rcu() check if (!is_a_nulls(n->next)) and set
WRITE_ONCE(n->next->pprev, &n->next) just like hlist_nulls_add_head_rcu() and
hlist_nulls_replace_rcu() in the same file?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103113711.1832834-1-max.kellermann@ionos.com?part=1

[PATCH v2 2/5] mm/vmscan: move code to lru_gen_{add,del}_memcg()

[Max Kellermann]

Preparation for the following patch that adds tail caching.

For simplicity, hlist_nulls_del_init_rcu() is now always used, which
is also valid during rotation, because the entry is immediately
reinserted under `memcg_lru.lock`.

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
---
 mm/vmscan.c | 36 ++++++++++++++++++++++++++++--------
 1 file changed, 28 insertions(+), 8 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 9959ca158ce8..aed56d452881 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4556,6 +4556,29 @@ enum {
 	MEMCG_LRU_YOUNG,
 };
 
+static void lru_gen_add_memcg(struct lru_gen_memcg *memcg_lru,
+			      struct lru_gen_folio *lrugen,
+			      int gen, int bin, bool at_head)
+{
+	struct hlist_nulls_head *head = &memcg_lru->fifo[gen][bin];
+
+	lockdep_assert_held(&memcg_lru->lock);
+
+	if (at_head)
+		hlist_nulls_add_head_rcu(&lrugen->list, head);
+	else
+		hlist_nulls_add_tail_rcu(&lrugen->list, head);
+}
+
+static void lru_gen_del_memcg(struct lru_gen_memcg *memcg_lru,
+			      struct lru_gen_folio *lrugen,
+			      int gen)
+{
+	lockdep_assert_held(&memcg_lru->lock);
+
+	hlist_nulls_del_init_rcu(&lrugen->list);
+}
+
 static void lru_gen_rotate_memcg(struct lruvec *lruvec, int op)
 {
 	int seg;
@@ -4584,12 +4607,9 @@ static void lru_gen_rotate_memcg(struct lruvec *lruvec, int op)
 	WRITE_ONCE(lruvec->lrugen.seg, seg);
 	WRITE_ONCE(lruvec->lrugen.gen, new);
 
-	hlist_nulls_del_rcu(&lruvec->lrugen.list);
-
-	if (op == MEMCG_LRU_OLD)
-		hlist_nulls_add_head_rcu(&lruvec->lrugen.list, &pgdat->memcg_lru.fifo[new][bin]);
-	else
-		hlist_nulls_add_tail_rcu(&lruvec->lrugen.list, &pgdat->memcg_lru.fifo[new][bin]);
+	lru_gen_del_memcg(&pgdat->memcg_lru, &lruvec->lrugen, old);
+	lru_gen_add_memcg(&pgdat->memcg_lru, &lruvec->lrugen, new, bin,
+			  op == MEMCG_LRU_OLD);
 
 	pgdat->memcg_lru.nr_memcgs[old]--;
 	pgdat->memcg_lru.nr_memcgs[new]++;
@@ -4620,7 +4640,7 @@ void lru_gen_online_memcg(struct mem_cgroup *memcg)
 
 		lruvec->lrugen.gen = gen;
 
-		hlist_nulls_add_tail_rcu(&lruvec->lrugen.list, &pgdat->memcg_lru.fifo[gen][bin]);
+		lru_gen_add_memcg(&pgdat->memcg_lru, &lruvec->lrugen, gen, bin, false);
 		pgdat->memcg_lru.nr_memcgs[gen]++;
 
 		spin_unlock_irq(&pgdat->memcg_lru.lock);
@@ -4654,7 +4674,7 @@ void lru_gen_release_memcg(struct mem_cgroup *memcg)
 
 		gen = lruvec->lrugen.gen;
 
-		hlist_nulls_del_init_rcu(&lruvec->lrugen.list);
+		lru_gen_del_memcg(&pgdat->memcg_lru, &lruvec->lrugen, gen);
 		pgdat->memcg_lru.nr_memcgs[gen]--;
 
 		if (!pgdat->memcg_lru.nr_memcgs[gen] && gen == get_memcg_gen(pgdat->memcg_lru.seq))
-- 
2.53.0

Re: [PATCH v2 2/5] mm/vmscan: move code to lru_gen_{add,del}_memcg()

[Barry Song]

On Tue, Nov 3, 2026 at 7:37 PM Max Kellermann <max.kellermann@ionos.com> wrote:
>
> Preparation for the following patch that adds tail caching.
>
> For simplicity, hlist_nulls_del_init_rcu() is now always used, which
> is also valid during rotation, because the entry is immediately
> reinserted under `memcg_lru.lock`.
>
> Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
> ---
>  mm/vmscan.c | 36 ++++++++++++++++++++++++++++--------
>  1 file changed, 28 insertions(+), 8 deletions(-)
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 9959ca158ce8..aed56d452881 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -4556,6 +4556,29 @@ enum {
>         MEMCG_LRU_YOUNG,
>  };
>
> +static void lru_gen_add_memcg(struct lru_gen_memcg *memcg_lru,
> +                             struct lru_gen_folio *lrugen,
> +                             int gen, int bin, bool at_head)
> +{
> +       struct hlist_nulls_head *head = &memcg_lru->fifo[gen][bin];
> +
> +       lockdep_assert_held(&memcg_lru->lock);
> +
> +       if (at_head)
> +               hlist_nulls_add_head_rcu(&lrugen->list, head);
> +       else
> +               hlist_nulls_add_tail_rcu(&lrugen->list, head);

T.J. 's patch[1] has actually removed the needs for hlist_nulls_add_head_rcu
and it has been in mm-new branch.

> +}
> +
> +static void lru_gen_del_memcg(struct lru_gen_memcg *memcg_lru,
> +                             struct lru_gen_folio *lrugen,
> +                             int gen)
> +{
> +       lockdep_assert_held(&memcg_lru->lock);
> +
> +       hlist_nulls_del_init_rcu(&lrugen->list);
> +}
> +
>  static void lru_gen_rotate_memcg(struct lruvec *lruvec, int op)
>  {
>         int seg;
> @@ -4584,12 +4607,9 @@ static void lru_gen_rotate_memcg(struct lruvec *lruvec, int op)
>         WRITE_ONCE(lruvec->lrugen.seg, seg);
>         WRITE_ONCE(lruvec->lrugen.gen, new);
>
> -       hlist_nulls_del_rcu(&lruvec->lrugen.list);
> -
> -       if (op == MEMCG_LRU_OLD)
> -               hlist_nulls_add_head_rcu(&lruvec->lrugen.list, &pgdat->memcg_lru.fifo[new][bin]);
> -       else
> -               hlist_nulls_add_tail_rcu(&lruvec->lrugen.list, &pgdat->memcg_lru.fifo[new][bin]);
> +       lru_gen_del_memcg(&pgdat->memcg_lru, &lruvec->lrugen, old);
> +       lru_gen_add_memcg(&pgdat->memcg_lru, &lruvec->lrugen, new, bin,
> +                         op == MEMCG_LRU_OLD);

Also, MEMCG_LRU_OLD has been removed as I replied to you in v1[2]

[1]  https://lore.kernel.org/linux-mm/20261028010611.3929937-3-tjmercier@google.com/
[2]  https://lore.kernel.org/linux-mm/CAGsJ_4zQh9HrvgaqD32_A=bSdqmPqP4oV_SxMXMEDBrHe4jjhg@mail.gmail.com/

Best,
Barry

Re: [PATCH v2 2/5] mm/vmscan: move code to lru_gen_{add,del}_memcg()

[Max Kellermann]

On Wed, Nov 4, 2026 at 4:22 AM Barry Song <baohua@kernel.org> wrote:
> Also, MEMCG_LRU_OLD has been removed as I replied to you in v1[2]

Oh, I rebased on linus/master and not mm-text.  I saw MEMCG_LRU_HEAD
was removed there which caused a rebase conflict, and thought your
mention of removing MEMCG_LRU_OLD was a mixup on your side, but it was
really me.  I'll post v3 (as soon as I have figured out the rest of
your reviews).

-- 
Max Kellermann
Principal Architect
Hosting Technology

cm4all | Im Mediapark 6a | 50670 Köln | Germany
General information about the company can be found here:
https://www.cm4all.com/impressum
A member of the IONOS Group

[PATCH v2 3/5] mm/mglru: cache the last memcg node, make tail insertion constant time

[Max Kellermann]

Rotating or onlining a memcg currently uses hlist_nulls_add_tail_rcu()
which walks the destination bin to find its tail while holding
`memcg_lru.lock` with interrupts disabled.  With thousands of cgroups,
this takes a long time and causes severe lock contention.

Cache a pointer to each bin's last node, making tail insertion O(1).
This requires remembering the bin number (but the new struct field
occupies a hole, so it does not grow).

In a micro benchmark that creates 1500 cgroups and repeatedly writes
`1024M swappiness=0` to `memory.reclaim` in 8 threads, the request
rate increases by a factor of 9.5 and the average duration of each
write request is reduced by 89%.

All writes return EAGAIN in both runs; the benchmark measures the cost
of reclaim attempts that cannot meet their requested target, not
reclaimed throughput.

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
---
 include/linux/mmzone.h |  4 ++++
 mm/vmscan.c            | 24 +++++++++++++++++++++---
 2 files changed, 25 insertions(+), 3 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index ecd1ebd49756..08928179b8aa 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -570,6 +570,8 @@ struct lru_gen_folio {
 	u8 gen;
 	/* the list segment this lru_gen_folio belongs to */
 	u8 seg;
+	/* the memcg bin, protected by memcg_lru.lock */
+	u8 bin;
 	/* per-node lru_gen_folio list for global reclaim */
 	struct hlist_nulls_node list;
 };
@@ -664,6 +666,8 @@ struct lru_gen_memcg {
 	unsigned long nr_memcgs[MEMCG_NR_GENS];
 	/* per-node lru_gen_folio list for global reclaim */
 	struct hlist_nulls_head	fifo[MEMCG_NR_GENS][MEMCG_NR_BINS];
+	/* last entry in each bin, or NULL if empty */
+	struct hlist_nulls_node *tail[MEMCG_NR_GENS][MEMCG_NR_BINS];
 	/* protects the above */
 	spinlock_t lock;
 };
diff --git a/mm/vmscan.c b/mm/vmscan.c
index aed56d452881..3075949bf641 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4561,21 +4561,37 @@ static void lru_gen_add_memcg(struct lru_gen_memcg *memcg_lru,
 			      int gen, int bin, bool at_head)
 {
 	struct hlist_nulls_head *head = &memcg_lru->fifo[gen][bin];
+	struct hlist_nulls_node *last = memcg_lru->tail[gen][bin];
 
 	lockdep_assert_held(&memcg_lru->lock);
 
-	if (at_head)
+	lrugen->bin = bin;
+
+	if (at_head || !last)
 		hlist_nulls_add_head_rcu(&lrugen->list, head);
 	else
-		hlist_nulls_add_tail_rcu(&lrugen->list, head);
+		hlist_nulls_add_behind_rcu(&lrugen->list, last);
+
+	if (!at_head || !last)
+		memcg_lru->tail[gen][bin] = &lrugen->list;
 }
 
 static void lru_gen_del_memcg(struct lru_gen_memcg *memcg_lru,
 			      struct lru_gen_folio *lrugen,
 			      int gen)
 {
+	const int bin = lrugen->bin;
+	struct hlist_nulls_head *head = &memcg_lru->fifo[gen][bin];
+
 	lockdep_assert_held(&memcg_lru->lock);
 
+	/* pprev points to head->first for the only entry, otherwise prev->next. */
+	if (memcg_lru->tail[gen][bin] == &lrugen->list)
+		memcg_lru->tail[gen][bin] = lrugen->list.pprev == &head->first
+			? NULL
+			: container_of(lrugen->list.pprev, struct hlist_nulls_node, next);
+
+	/* Keep next intact for readers that still hold this entry. */
 	hlist_nulls_del_init_rcu(&lrugen->list);
 }
 
@@ -6101,8 +6117,10 @@ void lru_gen_init_pgdat(struct pglist_data *pgdat)
 	spin_lock_init(&pgdat->memcg_lru.lock);
 
 	for (i = 0; i < MEMCG_NR_GENS; i++) {
-		for (j = 0; j < MEMCG_NR_BINS; j++)
+		for (j = 0; j < MEMCG_NR_BINS; j++) {
 			INIT_HLIST_NULLS_HEAD(&pgdat->memcg_lru.fifo[i][j], i);
+			pgdat->memcg_lru.tail[i][j] = NULL;
+		}
 	}
 }
 
-- 
2.53.0

Re: [PATCH v2 3/5] mm/mglru: cache the last memcg node, make tail insertion constant time

[Barry Song]

On Tue, Nov 3, 2026 at 7:37 PM Max Kellermann <max.kellermann@ionos.com> wrote:
>
> Rotating or onlining a memcg currently uses hlist_nulls_add_tail_rcu()
> which walks the destination bin to find its tail while holding
> `memcg_lru.lock` with interrupts disabled.  With thousands of cgroups,
> this takes a long time and causes severe lock contention.

This is the main cause of the soft lockups and RCU stalls,
right? If so, we may want to add a Fixes tag.

The fix itself may not be directly applicable to stable
kernels, but based on the logic of this patch, we should
be able to work out a backport, right?

Another option is to leave the cleanups in patches 1 and 2
out for now, and make patch 3 self-contained as the first
patch in your series, if that makes backporting easier.

You can then move the cleanups from patches 1 and 2 to
follow the new patch 1.

>
> Cache a pointer to each bin's last node, making tail insertion O(1).
> This requires remembering the bin number (but the new struct field
> occupies a hole, so it does not grow).
>
> In a micro benchmark that creates 1500 cgroups and repeatedly writes
> `1024M swappiness=0` to `memory.reclaim` in 8 threads, the request
> rate increases by a factor of 9.5 and the average duration of each
> write request is reduced by 89%.
>
> All writes return EAGAIN in both runs; the benchmark measures the cost
> of reclaim attempts that cannot meet their requested target, not
> reclaimed throughput.

I'd appreciate it if you could also mention Ridong's earlier
work in the changelog:

https://lore.kernel.org/all/20251119083722.1365680-1-chenridong@huaweicloud.com/

>
> Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
> ---
[...]
> @@ -4561,21 +4561,37 @@ static void lru_gen_add_memcg(struct lru_gen_memcg *memcg_lru,
>                               int gen, int bin, bool at_head)
>  {
>         struct hlist_nulls_head *head = &memcg_lru->fifo[gen][bin];
> +       struct hlist_nulls_node *last = memcg_lru->tail[gen][bin];
>
>         lockdep_assert_held(&memcg_lru->lock);
>
> -       if (at_head)
> +       lrugen->bin = bin;
> +
> +       if (at_head || !last)
>                 hlist_nulls_add_head_rcu(&lrugen->list, head);

Do we still need at_head now that MEMCG_LRU_OLD has been removed?

>         else
> -               hlist_nulls_add_tail_rcu(&lrugen->list, head);
> +               hlist_nulls_add_behind_rcu(&lrugen->list, last);
> +
> +       if (!at_head || !last)
> +               memcg_lru->tail[gen][bin] = &lrugen->list;
>  }

Best,
Barry

Re: [PATCH v2 3/5] mm/mglru: cache the last memcg node, make tail insertion constant time

[Max Kellermann]

On Wed, Nov 4, 2026 at 4:40 AM Barry Song <baohua@kernel.org> wrote:
>
> On Tue, Nov 3, 2026 at 7:37 PM Max Kellermann <max.kellermann@ionos.com> wrote:
> >
> > Rotating or onlining a memcg currently uses hlist_nulls_add_tail_rcu()
> > which walks the destination bin to find its tail while holding
> > `memcg_lru.lock` with interrupts disabled.  With thousands of cgroups,
> > this takes a long time and causes severe lock contention.
>
> This is the main cause of the soft lockups and RCU stalls,
> right? If so, we may want to add a Fixes tag.

Yes. I added (for v3): Fixes: e4dde56cd208 ("mm: multi-gen LRU:
per-node lru_gen_folio lists")

This commit added lru_gen_rotate_memcg() which calls
hlist_nulls_add_tail_rcu() (full linked list walk under spinlock).

> The fix itself may not be directly applicable to stable
> kernels, but based on the logic of this patch, we should
> be able to work out a backport, right?

The primary copy of these patches lives in our kernel branch that is
based on 6.18 LTS.  So I already have a set of patches that can be
backported more easily.  Do you want me to submit these versions, too
- and if yes, how?

For me, it is not entirely clear if a stable backport is really
necessary (thus I did not Cc stable).  It's not a bug; just awful
performance when you have thousands of memcgs, but few people have
that.  But that's up for you to decide.

> Another option is to leave the cleanups in patches 1 and 2
> out for now, and make patch 3 self-contained as the first
> patch in your series, if that makes backporting easier.

I don't think that makes backporting any easier - they are not really
cleanups, but refactoring to make [3/5] as simple&small as possible.
I wouldn't squash them; it improves nothing but makes review harder.

> I'd appreciate it if you could also mention Ridong's earlier
> work in the changelog:
>
> https://lore.kernel.org/all/20251119083722.1365680-1-chenridong@huaweicloud.com/

Aye.

> > +       if (at_head || !last)
> >                 hlist_nulls_add_head_rcu(&lrugen->list, head);
>
> Do we still need at_head now that MEMCG_LRU_OLD has been removed?

Yes, for the "!last" case.  This is, unfortunately, a special case.
If you prefer, we could move this special-casing to the hlist_nulls
API so callers need not be aware of it.

-- 
Max Kellermann
Principal Architect
Hosting Technology

cm4all | Im Mediapark 6a | 50670 Köln | Germany
General information about the company can be found here:
https://www.cm4all.com/impressum
A member of the IONOS Group

[PATCH v2 4/5] mm/mglru: avoid redundant rotations of young memcgs

[Max Kellermann]

Concurrent reclaimers can request promotion after another reclaimer
has already moved a memcg into the young generation.  Requeueing it
into a random bin again adds list mutations without advancing the
old-generation scan.

Therefore, skip `MEMCG_LRU_YOUNG` when the memcg is already in the
default segment of the current young generation.

In my micro benchmark, this increases the request rate by 36% and
reduces the average duration of each write request by 27%.

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
---
 mm/vmscan.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 3075949bf641..899f756bd15b 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4620,6 +4620,14 @@ static void lru_gen_rotate_memcg(struct lruvec *lruvec, int op)
 	else
 		VM_WARN_ON_ONCE(true);
 
+	if (op == MEMCG_LRU_YOUNG && old == new && lruvec->lrugen.seg == seg)
+		/*
+		 * This memcg is already in the young generation and
+		 * reshuffling it again cannot help the scan of the
+		 * old generation.
+		 */
+		goto unlock;
+
 	WRITE_ONCE(lruvec->lrugen.seg, seg);
 	WRITE_ONCE(lruvec->lrugen.gen, new);
 
@@ -4633,6 +4641,7 @@ static void lru_gen_rotate_memcg(struct lruvec *lruvec, int op)
 	if (!pgdat->memcg_lru.nr_memcgs[old] && old == get_memcg_gen(pgdat->memcg_lru.seq))
 		WRITE_ONCE(pgdat->memcg_lru.seq, pgdat->memcg_lru.seq + 1);
 
+unlock:
 	spin_unlock_irqrestore(&pgdat->memcg_lru.lock, flags);
 }
 
-- 
2.53.0

Re: [PATCH v2 4/5] mm/mglru: avoid redundant rotations of young memcgs

[Barry Song]

On Tue, Nov 3, 2026 at 7:37 PM Max Kellermann <max.kellermann@ionos.com> wrote:
>
> Concurrent reclaimers can request promotion after another reclaimer
> has already moved a memcg into the young generation.  Requeueing it
> into a random bin again adds list mutations without advancing the
> old-generation scan.
>
> Therefore, skip `MEMCG_LRU_YOUNG` when the memcg is already in the
> default segment of the current young generation.
>
> In my micro benchmark, this increases the request rate by 36% and
> reduces the average duration of each write request by 27%.

Could you share your microbenchmark somewhere (or post it here
if it's not too lengthy), perhaps on GitHub, so others can
reproduce it? You could then refer to it in your changelog.

>
> Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
> ---
>  mm/vmscan.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 3075949bf641..899f756bd15b 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -4620,6 +4620,14 @@ static void lru_gen_rotate_memcg(struct lruvec *lruvec, int op)
>         else
>                 VM_WARN_ON_ONCE(true);
>
> +       if (op == MEMCG_LRU_YOUNG && old == new && lruvec->lrugen.seg == seg)
> +               /*
> +                * This memcg is already in the young generation and
> +                * reshuffling it again cannot help the scan of the
> +                * old generation.
> +                */
> +               goto unlock;

I think this is a correct change. I wonder if we can
reproduce the issue even without concurrent reclaimers.
In other words, can a single thread running in a
memory-limited memcg trigger it as well?

One thing I noticed is that you may not need to call
get_random_u32_below(MEMCG_NR_BINS) upfront. Could you
defer generating the random number until you actually need
to rotate the memcg? I wonder if this could further improve
your microbenchmark results.

Anyway, the compiler might be able to defer it for you.
I'm not sure, though. Could you take a look?

> +
>         WRITE_ONCE(lruvec->lrugen.seg, seg);
>         WRITE_ONCE(lruvec->lrugen.gen, new);
>
> @@ -4633,6 +4641,7 @@ static void lru_gen_rotate_memcg(struct lruvec *lruvec, int op)
>         if (!pgdat->memcg_lru.nr_memcgs[old] && old == get_memcg_gen(pgdat->memcg_lru.seq))
>                 WRITE_ONCE(pgdat->memcg_lru.seq, pgdat->memcg_lru.seq + 1);
>
> +unlock:
>         spin_unlock_irqrestore(&pgdat->memcg_lru.lock, flags);
>  }

Best,
Barry

Re: [PATCH v2 4/5] mm/mglru: avoid redundant rotations of young memcgs

[Max Kellermann]

On Wed, Nov 4, 2026 at 6:18 AM Barry Song <baohua@kernel.org> wrote:
> > In my micro benchmark, this increases the request rate by 36% and
> > reduces the average duration of each write request by 27%.
>
> Could you share your microbenchmark somewhere (or post it here
> if it's not too lengthy), perhaps on GitHub, so others can
> reproduce it? You could then refer to it in your changelog.

Will do.  A note of warning: it's ugly LLM slop, and if I had written
it manually, probably 99% of it could be removed because it's
pointless garbage.  The LLM wrote it for me before I had any clue
where the problem could be.  It helped me find the problem, but
uploading this garbage triggers a bit of shame in me ;-)

> I think this is a correct change. I wonder if we can
> reproduce the issue even without concurrent reclaimers.
> In other words, can a single thread running in a
> memory-limited memcg trigger it as well?

Your question is not really related to this [4/5] patch, is it?
Because this one is just an incremental optimization on top of the
real linked list scalability issue.

A single thread means there is no spinlock contention and the linked
list is only walked once.  I don't think the problem is visible then.

> One thing I noticed is that you may not need to call
> get_random_u32_below(MEMCG_NR_BINS) upfront. Could you
> defer generating the random number until you actually need
> to rotate the memcg? I wonder if this could further improve
> your microbenchmark results.

Good idea, optimization added for the v3 patch.  I don't think it
makes much of a difference, but it really makes sense.

(And I still hate that the Linux coding style requires having all
variable declarations at the top.  Code would be simpler by
declaring+initializing variables where you need them, and additionally
it opens more opportunities for compiler optimizations, but... it's
the way it is.)

> Anyway, the compiler might be able to defer it for you.
> I'm not sure, though. Could you take a look?

The compiler is not allowed to because it is not allowed to make
assumptions on what get_random_u8() does, therefore it must not
reorder the call.

(I wish C had a way to express a promise like "this function call
won't modify global state reachable by the caller's pointers", there's
just GCC's "pure" attribute which is too strong here.)

Minor note: deferring the PRNG call will move it inside spinlock
protection.  I think that's okay, but it can theoretically have a
performance impact, and I'm not expert enough on this part of the
kernel.

-- 
Max Kellermann
Principal Architect
Hosting Technology

cm4all | Im Mediapark 6a | 50670 Köln | Germany
General information about the company can be found here:
https://www.cm4all.com/impressum
A member of the IONOS Group

Re: [PATCH v2 4/5] mm/mglru: avoid redundant rotations of young memcgs

[Barry Song]

On Wed, Nov 4, 2026 at 4:27 PM Max Kellermann <max.kellermann@ionos.com> wrote:
>
> On Wed, Nov 4, 2026 at 6:18 AM Barry Song <baohua@kernel.org> wrote:
> > > In my micro benchmark, this increases the request rate by 36% and
> > > reduces the average duration of each write request by 27%.
> >
> > Could you share your microbenchmark somewhere (or post it here
> > if it's not too lengthy), perhaps on GitHub, so others can
> > reproduce it? You could then refer to it in your changelog.
>
> Will do.  A note of warning: it's ugly LLM slop, and if I had written
> it manually, probably 99% of it could be removed because it's
> pointless garbage.  The LLM wrote it for me before I had any clue
> where the problem could be.  It helped me find the problem, but
> uploading this garbage triggers a bit of shame in me ;-)

Thanks!

>
> > I think this is a correct change. I wonder if we can
> > reproduce the issue even without concurrent reclaimers.
> > In other words, can a single thread running in a
> > memory-limited memcg trigger it as well?
>
> Your question is not really related to this [4/5] patch, is it?
> Because this one is just an incremental optimization on top of the
> real linked list scalability issue.

I think it is related. Although lock contention isn't a concern
with a single thread, the early goto unlock could still save
some CPU cycles.

I can imagine a scenario on a system with a large amount
of memory, where a single memcg (e.g. one running a kernel
build) is limited to 1 GB. If this is the only memcg
constantly being aged and rotated, it should remain in
YOUNG most of the time.

We could still skip things like:

        hlist_nulls_del_rcu(&lruvec->lrugen.list);
        hlist_nulls_add_tail_rcu(&lruvec->lrugen.list,
&pgdat->memcg_lru.fifo[new][bin]);

>
> A single thread means there is no spinlock contention and the linked
> list is only walked once.  I don't think the problem is visible then.
>
> > One thing I noticed is that you may not need to call
> > get_random_u32_below(MEMCG_NR_BINS) upfront. Could you
> > defer generating the random number until you actually need
> > to rotate the memcg? I wonder if this could further improve
> > your microbenchmark results.
>
> Good idea, optimization added for the v3 patch.  I don't think it
> makes much of a difference, but it really makes sense.
>
> (And I still hate that the Linux coding style requires having all
> variable declarations at the top.  Code would be simpler by
> declaring+initializing variables where you need them, and additionally
> it opens more opportunities for compiler optimizations, but... it's
> the way it is.)
>
> > Anyway, the compiler might be able to defer it for you.
> > I'm not sure, though. Could you take a look?
>
> The compiler is not allowed to because it is not allowed to make
> assumptions on what get_random_u8() does, therefore it must not
> reorder the call.
>
> (I wish C had a way to express a promise like "this function call
> won't modify global state reachable by the caller's pointers", there's
> just GCC's "pure" attribute which is too strong here.)
>
> Minor note: deferring the PRNG call will move it inside spinlock
> protection.  I think that's okay, but it can theoretically have a
> performance impact, and I'm not expert enough on this part of the
> kernel.

Thanks for pointing this out. I missed it.

It might be a problem, as it could increase lock contention
by holding the spinlock for longer.
Only testing can tell for sure. If it does increase lock
contention, I think we can simply keep the redundant
random number generation.

Best,
Barry

[PATCH v2 5/5] mm/mglru: reschedule between memcg reclaim attempts

[Max Kellermann]

shrink_one() can return immediately for empty or protected memcgs, and
try_to_shrink_lruvec() can break out before reaching its
cond_resched().  A scan through many memcgs can repeatedly rotate
entries without giving other runnable tasks a scheduling opportunity.

Call cond_resched_tasks_rcu_qs() (as the existing shrink_lruvec() loop
does) after shrink_one(), while the memcg reference still pins the
cursor and before reacquiring the RCU read lock.  Also call it between
bins and nulls-marker retries, covering scans that skip every entry.

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
---
 mm/vmscan.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 899f756bd15b..5dc5e698e243 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -5395,6 +5395,9 @@ static void shrink_many(struct pglist_data *pgdat, struct scan_control *sc)
 	gen = get_memcg_gen(READ_ONCE(pgdat->memcg_lru.seq));
 	bin = first_bin = get_random_u32_below(MEMCG_NR_BINS);
 restart:
+	/* Yield between bins/retries that contain only skipped entries. */
+	cond_resched_tasks_rcu_qs();
+
 	op = 0;
 	memcg = NULL;
 
@@ -5425,6 +5428,14 @@ static void shrink_many(struct pglist_data *pgdat, struct scan_control *sc)
 
 		op = shrink_one(lruvec, sc);
 
+		/*
+		 * Empty or protected memcgs can bypass the
+		 * rescheduling points in page reclaim.  The memcg
+		 * reference pins our cursor while we yield outside
+		 * the RCU read-side critical section.
+		 */
+		cond_resched_tasks_rcu_qs();
+
 		rcu_read_lock();
 
 		if (should_abort_scan(lruvec, sc))
-- 
2.53.0