Re: [PATCH v1] memcg: fix cacheline contention between lruvec stat updates and LRU lists

[Shakeel Butt <shakeel.butt@linux.dev>]

On Tue, Oct 20, 2026 at 01:07:30PM -0700, Joanne Koong wrote:
> Every memcg lruvec (per-node) stat update reads the memcg and
> lruvec_stats_percpu pointers in the first 64 bytes of struct
> mem_cgroup_per_node, and the ones that go through mod_lruvec_state()
> (such as lruvec_stat_mod_folio()) also read lruvec->pgdat. These updates
> are extremely frequent (eg these updates happen several times for every
> page that's faulted in, freed, dirtied, or written back).
> 
> The memcg_pn_read_mostly cacheline group keeps those read-mostly fields
> on a different cacheline from the lruvec, but on CPUs that do spatial
> prefetching [1] (which is most Intel CPUs / modern CPUs), cachelines are
> fetched in 128-byte aligned pairs. Depending on where struct
> mem_cgroup_per_node lands in its slab, that cacheline can be paired with
> the cacheline holding the LRU list heads, and the lruvec->pgdat
> cacheline with the one holding lru_lock.
> 
> When this happens, every stat update also pulls in a copy of the LRU
> list heads and lru_lock cachelines, which are written to constantly by
> other CPUs under the lru_lock. The stat update has to fetch them from
> another CPU, which is unnecessary overhead since the stat update doesn't
> use the LRU list heads or lru_lock at all. But even worse, the lru_lock
> holder then has to invalidate those copies and wait for the
> acknowledgements to come back before it can do its next write, which
> makes the lru_lock critical section longer for everyone waiting on the
> lock.

Can you please reproduce on a non-intel machine?

Second, can you try the following patch instead of your on the intel
machine?

From a747ceeb4e3f10fb0c3e7daf5660076a49b77126 Mon Sep 17 00:00:00 2001
From: Shakeel Butt <shakeel.butt@linux.dev>
Date: Tue, 20 Oct 2026 15:33:26 -0700
Subject: [PATCH] memcg: rearrage mem_cgroup_per_node

Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
 include/linux/memcontrol.h | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 74110a324f9e..0e73f5c96e15 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -95,6 +95,17 @@ struct mem_cgroup_per_node {
 
 	__cacheline_group_end_aligned(memcg_pn_read_mostly);
 
+	/* Touched only when the memcg is reparented or freed. */
+	__cacheline_group_begin_aligned(memcg_pn_cold);
+	/*
+	 * orig_objcg preserves a pointer (and a reference) to the original
+	 * objcg until the end of life of memcg.
+	 */
+	struct obj_cgroup	*orig_objcg;
+	/* list of inherited objcgs, protected by objcg_lock */
+	struct list_head objcg_list;
+	__cacheline_group_end_aligned(memcg_pn_cold);
+
 	/*
 	 * Keep lruvec on its own lines. Sharing them with lru_zone_size[]
 	 * regressed, see commit f59adcf59332 ("mm: memcg: add cacheline
@@ -114,17 +125,6 @@ struct mem_cgroup_per_node {
 	atomic_t		slab_unreclaimable;
 #endif
 	__cacheline_group_end_aligned(memcg_pn_write_hot);
-
-	/* Touched only when the memcg is reparented or freed. */
-	__cacheline_group_begin_aligned(memcg_pn_cold);
-	/*
-	 * orig_objcg preserves a pointer (and a reference) to the original
-	 * objcg until the end of life of memcg.
-	 */
-	struct obj_cgroup	*orig_objcg;
-	/* list of inherited objcgs, protected by objcg_lock */
-	struct list_head objcg_list;
-	__cacheline_group_end_aligned(memcg_pn_cold);
 };
 
 struct mem_cgroup_threshold {
-- 
2.53.0-Meta