Linux-mm Archive on lore.kernel.org
 help / color / mirror / Atom feed
* [RFC patch] mm/mglru: advance memcg generation when reclaimers pile up
@ 2026-11-19  6:50 Ridong Chen
  0 siblings, 0 replies; only message in thread
From: Ridong Chen @ 2026-11-19  6:50 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner
  Cc: Kairui Song, Qi Zheng, Shakeel Butt, Barry Song, Axel Rasmussen,
	Yuanchu Xie, Wei Xu, Baoquan He, Baolin Wang, Liam R. Howlett,
	Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
	open list:MEMORY MANAGEMENT - MGLRU (MULTI-GEN LRU), linux-kernel,
	Ridong Chen, Ridong Chen

From: Ridong Chen <chenridong@xiaomi.com>

On Android, memcgs are used to group and account memory rather than to
limit it, so direct reclaim never hits a memcg limit and is always
global reclaim through shrink_many(). We observed long direct reclaim
latency on Android devices under memory pressure, when many tasks enter
direct reclaim at the same time and all of them walk the same per-node
memcg LRU. Some of the slow calls piled up on a few memcgs.

shrink_many() picks a random bin in the old memcg generation and moves
on to the next bin if it is empty. When only a few old bins still hold
memcgs, all reclaimers end up on those bins and pile up on the same
memcgs, which makes direct reclaim take much longer.

Count the reclaimers in shrink_many() per node. Before walking a bin,
advance the memcg generation when there are more reclaimers than
non-empty old bins and advancing would add bins: splice each old bin in
front of the young bin with the same index, so in each bin the
remaining old memcgs are still reclaimed first in the same order, and
bump the generation counter. The condition is checked again under
memcg_lru->lock before splicing. The per-memcg segment is kept, so a
memcg already moved to the tail segment (below memory.low or too small)
still gets its second visit before the young memcgs in its bin.

shrink_many() now reads the generation again at each restart instead of
once at entry, so reclaimers in the loop follow the new old generation
instead of walking drained bins. A walker can also go more than one
generation stale while it is in shrink_one(), and two advances would
wrap its stale generation around to the young one. Skip the rest of the
walk once the generation no longer matches the counter.

Advancing earlier means memcgs that were just rotated to the young
generation become eligible for reclaim sooner than before, when the old
generation had to drain first, and the order is only kept within each
bin. The splice walks the remaining old memcgs under memcg_lru->lock
with interrupts off; it only runs when some old bin is empty while the
young bin with the same index is not.

The pattern can be reproduced on x86. Test on an x86_64 machine
(i7-13700HX, 24 CPUs, zram swap): 64 memcgs each churning 24MiB of anon
memory, 16 threads writing 128K to the root memory.reclaim in a loop,
which goes through shrink_many(). Each run is 120s; the patch is turned
off (A) and on (B) at runtime with a debug knob that is not part of this
patch, alternating A B A B A B in the same boot.
Latency of each memory.reclaim write, in ms, averaged over the three
runs:

                    A        B      change
    p50          1.95     2.08       +6%
    p95          4.88     4.29      -12%
    p99          8.06     6.25      -22%
    p99.9       13.08    10.40      -20%
    avg          2.32     2.32        0%
    >= 10ms      3240      894      -72%  (calls per run)

The time with at most 4 non-empty old bins drops from 2.6% to 0.02%.
The cost is that each call visits about 7% more memcgs (shrink_one()
calls per call 118 -> 126), since reclaimers following the new old
generation may revisit memcgs they rotated earlier in the same call,
which shows up as the higher p50. Anon refaults, pswpin and pswpout
are within 2% between A and B.

Signed-off-by: Ridong Chen <chenridong@xiaomi.com>
---
 include/linux/mmzone.h |  9 ++++-
 mm/vmscan.c            | 86 ++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 90 insertions(+), 5 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 7d6e830c2420..bc9f9224bf79 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -624,7 +624,9 @@ struct lru_gen_mm_walk {
  * An onlining memcg is added to the tail of a random bin in the old generation.
  * The eviction starts at the head of a random bin in the old generation. The
  * per-node memcg generation counter, whose reminder (mod MEMCG_NR_GENS) indexes
- * the old generation, is incremented when all its bins become empty.
+ * the old generation, is incremented when all its bins become empty, or when
+ * there are more reclaimers than non-empty old bins and advancing adds bins
+ * (see try_to_inc_memcg_seq()).
  *
  * There are two operations:
  * 1. MEMCG_LRU_TAIL, which moves a memcg to the tail of a random bin in its
@@ -649,7 +651,8 @@ struct lru_gen_mm_walk {
  *    memcgs. For memcg reclaim, it still relies on mem_cgroup_iter().
  * 2. There are only two valid generations: old (seq) and young (seq+1).
  *    MEMCG_NR_GENS is set to three so that when reading the generation counter
- *    locklessly, a stale value (seq-1) does not wraparound to young.
+ *    locklessly, a stale value (seq-1) does not wraparound to young. Walkers
+ *    that can be more stale, e.g. across shrink_one(), recheck seq.
  */
 #define MEMCG_NR_GENS	3
 #define MEMCG_NR_BINS	8
@@ -663,6 +666,8 @@ struct lru_gen_memcg {
 	struct hlist_nulls_head	fifo[MEMCG_NR_GENS][MEMCG_NR_BINS];
 	/* protects the above */
 	spinlock_t lock;
+	/* the number of reclaimers in shrink_many() */
+	atomic_t nr_reclaimers;
 };
 
 void lru_gen_init_pgdat(struct pglist_data *pgdat);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 2d061439a863..cd8ceadbe74a 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -5337,6 +5337,76 @@ static int shrink_one(struct lruvec *lruvec, struct scan_control *sc)
 	       MEMCG_LRU_TAIL : MEMCG_LRU_YOUNG;
 }
 
+/*
+ * Advance the memcg generation when there are more reclaimers than non-empty
+ * old bins and advancing adds bins, so that they spread over more bins.
+ */
+static bool should_inc_memcg_seq(struct lru_gen_memcg *memcg_lru, int old)
+{
+	int bin;
+	int nr_old = 0, nr_new = 0;
+	int young = get_memcg_gen(old + 1);
+
+	/* non-empty old bins now and after the splice */
+	for (bin = 0; bin < MEMCG_NR_BINS; bin++) {
+		bool o = !hlist_nulls_empty(&memcg_lru->fifo[old][bin]);
+		bool y = !hlist_nulls_empty(&memcg_lru->fifo[young][bin]);
+
+		nr_old += o;
+		nr_new += o || y;
+	}
+
+	return atomic_read(&memcg_lru->nr_reclaimers) > nr_old && nr_new > nr_old;
+}
+
+static void try_to_inc_memcg_seq(struct pglist_data *pgdat)
+{
+	int old, young, bin;
+	unsigned long flags;
+	struct lru_gen_memcg *memcg_lru = &pgdat->memcg_lru;
+
+	old = get_memcg_gen(READ_ONCE(memcg_lru->seq));
+	if (!should_inc_memcg_seq(memcg_lru, old))
+		return;
+
+	young = get_memcg_gen(old + 1);
+
+	spin_lock_irqsave(&memcg_lru->lock, flags);
+
+	/* recheck under the lock, the bins may have changed */
+	if (old != get_memcg_gen(memcg_lru->seq) ||
+	    !should_inc_memcg_seq(memcg_lru, old))
+		goto unlock;
+
+	for (bin = 0; bin < MEMCG_NR_BINS; bin++) {
+		struct hlist_nulls_head *src = &memcg_lru->fifo[old][bin];
+		struct hlist_nulls_head *dst = &memcg_lru->fifo[young][bin];
+		struct hlist_nulls_node *node, *last = NULL;
+		struct lru_gen_folio *lrugen;
+
+		hlist_nulls_for_each_entry(lrugen, node, src, list) {
+			WRITE_ONCE(lrugen->gen, young);
+			last = node;
+		}
+		if (!last)
+			continue;
+
+		/* splice src in front of dst */
+		WRITE_ONCE(last->next, dst->first);
+		if (!is_a_nulls(dst->first))
+			WRITE_ONCE(dst->first->pprev, &last->next);
+		WRITE_ONCE(src->first->pprev, &dst->first);
+		rcu_assign_pointer(hlist_nulls_first_rcu(dst), src->first);
+		WRITE_ONCE(src->first, (struct hlist_nulls_node *)NULLS_MARKER(old));
+	}
+
+	memcg_lru->nr_memcgs[young] += memcg_lru->nr_memcgs[old];
+	memcg_lru->nr_memcgs[old] = 0;
+	WRITE_ONCE(memcg_lru->seq, memcg_lru->seq + 1);
+unlock:
+	spin_unlock_irqrestore(&memcg_lru->lock, flags);
+}
+
 static void shrink_many(struct pglist_data *pgdat, struct scan_control *sc)
 {
 	int op;
@@ -5348,9 +5418,14 @@ static void shrink_many(struct pglist_data *pgdat, struct scan_control *sc)
 	struct mem_cgroup *memcg;
 	struct hlist_nulls_node *pos;
 
-	gen = get_memcg_gen(READ_ONCE(pgdat->memcg_lru.seq));
+	atomic_inc(&pgdat->memcg_lru.nr_reclaimers);
+
 	bin = first_bin = get_random_u32_below(MEMCG_NR_BINS);
 restart:
+	/* advance the generation if needed and walk the current old one */
+	try_to_inc_memcg_seq(pgdat);
+	gen = get_memcg_gen(READ_ONCE(pgdat->memcg_lru.seq));
+
 	op = 0;
 	memcg = NULL;
 
@@ -5365,7 +5440,9 @@ static void shrink_many(struct pglist_data *pgdat, struct scan_control *sc)
 		mem_cgroup_put(memcg);
 		memcg = NULL;
 
-		if (gen != READ_ONCE(lrugen->gen))
+		/* skip the rest if gen has gone stale, e.g. across shrink_one() */
+		if (gen != READ_ONCE(lrugen->gen) ||
+		    gen != get_memcg_gen(READ_ONCE(pgdat->memcg_lru.seq)))
 			continue;
 
 		lruvec = container_of(lrugen, struct lruvec, lrugen);
@@ -5395,7 +5472,7 @@ static void shrink_many(struct pglist_data *pgdat, struct scan_control *sc)
 	mem_cgroup_put(memcg);
 
 	if (!is_a_nulls(pos))
-		return;
+		goto done;
 
 	/* restart if raced with lru_gen_rotate_memcg() */
 	if (gen != get_nulls_value(pos))
@@ -5405,6 +5482,8 @@ static void shrink_many(struct pglist_data *pgdat, struct scan_control *sc)
 	bin = get_memcg_bin(bin + 1);
 	if (bin != first_bin)
 		goto restart;
+done:
+	atomic_dec(&pgdat->memcg_lru.nr_reclaimers);
 }
 
 static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
@@ -6081,6 +6160,7 @@ void lru_gen_init_pgdat(struct pglist_data *pgdat)
 	int i, j;
 
 	spin_lock_init(&pgdat->memcg_lru.lock);
+	atomic_set(&pgdat->memcg_lru.nr_reclaimers, 0);
 
 	for (i = 0; i < MEMCG_NR_GENS; i++) {
 		for (j = 0; j < MEMCG_NR_BINS; j++)
-- 
2.34.1



^ permalink raw reply related	[flat|nested] only message in thread

only message in thread, other threads:[~2026-11-19  6:51 UTC | newest]

Thread overview: (only message) (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-19  6:50 [RFC patch] mm/mglru: advance memcg generation when reclaimers pile up Ridong Chen

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox