* [PATCH v3 1/4] mm: zswap: fix shrinker count for the root memcg
2026-11-14 0:28 [PATCH v3 0/4] mm: read root memcg stats from global counters instead of flushing Joanne Koong
@ 2026-11-14 0:28 ` Joanne Koong
2026-11-17 5:07 ` Chris Down
2026-11-14 0:28 ` [PATCH v3 2/4] mm: memcg: read root lruvec stats from the node counters Joanne Koong
` (2 subsequent siblings)
3 siblings, 1 reply; 9+ messages in thread
From: Joanne Koong @ 2026-11-14 0:28 UTC (permalink / raw)
To: akpm, david
Cc: yosry, mkoutny, shakeel.butt, hannes, mhocko, roman.gushchin,
muchun.song, nphamcs, chengming.zhou, sj, ljs, linux-mm, cgroups,
damon
zswap_shrinker_count() scales the number of freeable entries on a
memcg's zswap LRU by the memcg's compression ratio. That compression
ratio is computed from the memcg's MEMCG_ZSWAP_B and MEMCG_ZSWAPPED
stats.
However, for the root memcg, the zswap entries owned by the root memcg
itself are never accounted for in the stats (obj_cgroup_charge_zswap()
skips the root objcg, and before the root memcg had an objcg, its
entries weren't charged at all). The stats only reflect the entries
owned by the root memcg's descendants. When the root memcg's descendants
have no zswap entries (eg when the whole workload runs in the root
cgroup), the shrinker reports nothing to reclaim for the root memcg, and
its zswap entries are never written back by the shrinker. When its
descendants do have entries, the root memcg's entries are scaled by the
descendants' compression ratio instead of a ratio that includes them.
Fix this by using the zswap pool-wide counters for the root memcg
instead. The zswap pool-wide counters cover every zswap entry, including
those owned by the root memcg.
This also gets rid of having to flush the root memcg's stats, which
flushes every cgroup in the system.
In a VM running a reclaim-heavy workload in the root cgroup with the
zswap shrinker enabled, the shrinker wrote back no pages before this
change (zswpwb stayed at 0) and 1016 pages with it.
Fixes: b5ba474f3f51 ("zswap: shrink zswap pool based on memory pressure")
Acked-by: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: SJ Park <sj@kernel.org>
Acked-by: Nhat Pham <nphamcs@gmail.com>
Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
mm/zswap.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index a2e0314db1cf..6e1cd3e482f7 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1330,9 +1330,12 @@ static unsigned long zswap_shrinker_count(struct shrinker *shrinker,
* runtime-disabled: we can get sc->memcg == NULL, which is ok
* for the lruvec, but not for memcg_page_state().
*
- * Without memcg, use the zswap pool-wide metrics.
+ * Without memcg, or for the root memcg, use the zswap pool-wide
+ * metrics. The root memcg's stats don't count the entries it owns
+ * itself (obj_cgroup_charge_zswap() skips the root objcg), whereas the
+ * pool-wide metrics cover every entry and don't need a flush.
*/
- if (!mem_cgroup_disabled()) {
+ if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) {
mem_cgroup_flush_stats_ratelimited(memcg);
nr_backing = memcg_page_state(memcg, MEMCG_ZSWAP_B) >> PAGE_SHIFT;
nr_stored = memcg_page_state(memcg, MEMCG_ZSWAPPED);
--
2.52.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH v3 1/4] mm: zswap: fix shrinker count for the root memcg
2026-11-14 0:28 ` [PATCH v3 1/4] mm: zswap: fix shrinker count for the root memcg Joanne Koong
@ 2026-11-17 5:07 ` Chris Down
0 siblings, 0 replies; 9+ messages in thread
From: Chris Down @ 2026-11-17 5:07 UTC (permalink / raw)
To: Joanne Koong
Cc: akpm, david, yosry, mkoutny, shakeel.butt, hannes, mhocko,
roman.gushchin, muchun.song, nphamcs, chengming.zhou, sj, ljs,
linux-mm, cgroups, damon
Hi Joanne,
Thanks, I've reproduced this and the results are definitely much better now,
in that with this patch it now writes back pages for a workload in the root
cgroup under memory pressure.
Reviewed-by: Chris Down <chris@chrisdown.name>
Tested-by: Chris Down <chris@chrisdown.name>
Thanks!
Chris
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v3 2/4] mm: memcg: read root lruvec stats from the node counters
2026-11-14 0:28 [PATCH v3 0/4] mm: read root memcg stats from global counters instead of flushing Joanne Koong
2026-11-14 0:28 ` [PATCH v3 1/4] mm: zswap: fix shrinker count for the root memcg Joanne Koong
@ 2026-11-14 0:28 ` Joanne Koong
2026-11-17 5:07 ` Chris Down
2026-11-14 0:28 ` [PATCH v3 3/4] mm: workingset: skip the root memcg flush in workingset_test_recent() Joanne Koong
2026-11-14 0:28 ` [PATCH v3 4/4] mm/damon/core: use node counters instead of flushing the root memcg Joanne Koong
3 siblings, 1 reply; 9+ messages in thread
From: Joanne Koong @ 2026-11-14 0:28 UTC (permalink / raw)
To: akpm, david
Cc: yosry, mkoutny, shakeel.butt, hannes, mhocko, roman.gushchin,
muchun.song, nphamcs, chengming.zhou, sj, ljs, linux-mm, cgroups,
damon
For the root memcg, lruvec_page_state() returns the rstat value, which
needs a flush to be up to date. Flushing the root memcg is expensive, as
it flushes every memcg in the system.
Most memcg node stats only track memory charged to a memcg, so the root
memcg's hierarchical value equals the node counter, which is always up
to date and doesn't need a flush. For those stats, have
lruvec_page_state() and lruvec_page_state_monotonic() read the node
counter for the root memcg, as they already do when memcg is disabled.
Callers that only read the root memcg's own lruvec then no longer need
to flush it, which the following patches take advantage of. It also
makes the root memcg's stats more accurate, as callers that ratelimit
the flush can otherwise read values that are seconds old.
The exceptions are the kernel memory stats (slab, kernel stack, page
tables, secondary page tables and vmalloc), hugetlb, and
WORKINGSET_NODERECLAIM, which counts reclaimed xarray nodes (slab
objects). Their node counters also count memory that isn't charged to a
memcg, which falls into two categories:
a) the allocation isn't accounted to any memcg (eg kernel page tables,
or hugetlb when hugetlb accounting is off)
b) the allocation is accounted, but it comes from a task running in the
root cgroup, and the root memcg isn't charged for it
As such, these stats still need to read the rstat value, since the node
counter value can be higher than the rstat value.
One user-visible change is that the root memcg's memory.numa_stat (and
the hierarchical_* entries of the cgroup v1 memory.numa_stat) now
reports the node counters for the non-excluded stats, as it reads
lruvec_page_state(). This is more accurate, but as a result, it may not
exactly match the root memcg's memory.stat, which still reads the rstat
values.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
mm/memcontrol.c | 58 +++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 56 insertions(+), 2 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 091401cecf39..196c0382181b 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -504,13 +504,67 @@ struct lruvec_stats {
long state_pending[NR_MEMCG_NODE_STAT_ITEMS];
};
+/*
+ * Whether to read the node counter for @idx instead of @lruvec's rstat value.
+ * This is only done for the root memcg for stats where every update also
+ * updates a memcg's stats, so the root memcg's hierarchical value equals the
+ * node counter. The node counter is preferred since it is always up to date,
+ * and avoids an rstat flush which for the root memcg flushes every cgroup in
+ * the system.
+ */
+static bool lruvec_use_node_stat(const struct lruvec *lruvec,
+ enum node_stat_item idx)
+{
+ if (!mem_cgroup_is_root(lruvec_memcg(lruvec)))
+ return false;
+
+ switch (idx) {
+ case NR_INACTIVE_ANON:
+ case NR_ACTIVE_ANON:
+ case NR_INACTIVE_FILE:
+ case NR_ACTIVE_FILE:
+ case NR_UNEVICTABLE:
+ case WORKINGSET_REFAULT_ANON:
+ case WORKINGSET_REFAULT_FILE:
+ case WORKINGSET_ACTIVATE_ANON:
+ case WORKINGSET_ACTIVATE_FILE:
+ case WORKINGSET_RESTORE_ANON:
+ case WORKINGSET_RESTORE_FILE:
+#ifdef CONFIG_NUMA_BALANCING
+ case PGPROMOTE_SUCCESS:
+#endif
+ case PGDEMOTE_KSWAPD:
+ case PGDEMOTE_DIRECT:
+ case PGDEMOTE_KHUGEPAGED:
+ case PGDEMOTE_PROACTIVE:
+ case PGSTEAL_KSWAPD:
+ case PGSTEAL_DIRECT:
+ case PGSTEAL_KHUGEPAGED:
+ case PGSTEAL_PROACTIVE:
+ case PGSTEAL_ANON:
+ case PGSTEAL_FILE:
+ case PGSCAN_KSWAPD:
+ case PGSCAN_DIRECT:
+ case PGSCAN_KHUGEPAGED:
+ case PGSCAN_PROACTIVE:
+ case PGSCAN_ANON:
+ case PGSCAN_FILE:
+ case PGROTATE_ANON:
+ case PGROTATE_FILE:
+ case PGREFILL:
+ return true;
+ default:
+ return false;
+ }
+}
+
unsigned long lruvec_page_state(const struct lruvec *lruvec, enum node_stat_item idx)
{
const struct mem_cgroup_per_node *pn;
long x;
int i;
- if (mem_cgroup_disabled())
+ if (mem_cgroup_disabled() || lruvec_use_node_stat(lruvec, idx))
return node_page_state(lruvec_pgdat(lruvec), idx);
i = memcg_stats_index(idx);
@@ -551,7 +605,7 @@ unsigned long lruvec_page_state_monotonic(const struct lruvec *lruvec,
const struct mem_cgroup_per_node *pn;
int i;
- if (mem_cgroup_disabled())
+ if (mem_cgroup_disabled() || lruvec_use_node_stat(lruvec, idx))
return node_page_state_monotonic(lruvec_pgdat(lruvec), idx);
i = memcg_stats_index(idx);
--
2.52.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH v3 2/4] mm: memcg: read root lruvec stats from the node counters
2026-11-14 0:28 ` [PATCH v3 2/4] mm: memcg: read root lruvec stats from the node counters Joanne Koong
@ 2026-11-17 5:07 ` Chris Down
0 siblings, 0 replies; 9+ messages in thread
From: Chris Down @ 2026-11-17 5:07 UTC (permalink / raw)
To: Joanne Koong
Cc: akpm, david, yosry, mkoutny, shakeel.butt, hannes, mhocko,
roman.gushchin, muchun.song, nphamcs, chengming.zhou, sj, ljs,
linux-mm, cgroups, damon
Hi Joanne,
Joanne Koong writes:
>The exceptions are the kernel memory stats (slab, kernel stack, page
>tables, secondary page tables and vmalloc), hugetlb, and
>WORKINGSET_NODERECLAIM, which counts reclaimed xarray nodes (slab
>objects). Their node counters also count memory that isn't charged to a
>memcg, which falls into two categories:
This still describes the v2 denylist. With the allowlist, the root memcg
also keeps reading rstat for NR_ANON_MAPPED, NR_FILE_MAPPED, NR_SHMEM et
al, and some of those differ for reasons outside the two categories here,
like swapcache being counted differently in the node counter, for
example. Could the changelog say what is on the list instead? So that
would be the stats that only change through mod_lruvec_state(), with
everything else still reading rstat.
>+ case PGROTATE_ANON:
>+ case PGROTATE_FILE:
>+ case PGREFILL:
>+ return true;
NR_VMSCAN_WRITE isn't on the list, but prepare_scan_control() adds it to
WORKINGSET_RESTORE_ANON, so once the root flush in reclaim goes away,
root reclaim would mix a live node counter with a stale rstat value.
Unless I'm mistaken it should go here as well.
Once those are fixed, please feel free to add:
Reviewed-by: Chris Down <chris@chrisdown.name>
Thanks!
Chris
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v3 3/4] mm: workingset: skip the root memcg flush in workingset_test_recent()
2026-11-14 0:28 [PATCH v3 0/4] mm: read root memcg stats from global counters instead of flushing Joanne Koong
2026-11-14 0:28 ` [PATCH v3 1/4] mm: zswap: fix shrinker count for the root memcg Joanne Koong
2026-11-14 0:28 ` [PATCH v3 2/4] mm: memcg: read root lruvec stats from the node counters Joanne Koong
@ 2026-11-14 0:28 ` Joanne Koong
2026-11-17 5:07 ` Chris Down
2026-11-14 0:28 ` [PATCH v3 4/4] mm/damon/core: use node counters instead of flushing the root memcg Joanne Koong
3 siblings, 1 reply; 9+ messages in thread
From: Joanne Koong @ 2026-11-14 0:28 UTC (permalink / raw)
To: akpm, david
Cc: yosry, mkoutny, shakeel.butt, hannes, mhocko, roman.gushchin,
muchun.song, nphamcs, chengming.zhou, sj, ljs, linux-mm, cgroups,
damon
We can skip the rstat flush in workingset_test_recent() when the
eviction memcg is the root memcg. lruvec_page_state() now reads the LRU
sizes of the root memcg from the node counters, which are always up to
date, and workingset_test_recent() only reads the eviction memcg's own
lruvec. This avoids an expensive flush on the refault path, as flushing
the root memcg flushes every memcg in the system.
For reclaim however, we still need to keep the flush (for now) since
global and root reclaim also read the rstat stats of the child memcgs
they scan, which the flush is needed for in order to keep up to date.
On an 80-CPU machine, a root flush took 69us on average with little
cgroup activity and ~0.9ms with 128 memcgs actively allocating (wall
time, including waiting for the rstat lock). Reading a node counter
doesn't add any cost. It took the same amount of time as reading the
rstat counter (~150ns, which includes bpftrace probe overhead).
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
mm/workingset.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/mm/workingset.c b/mm/workingset.c
index 1504f91cdca5..383f951eaad4 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -483,10 +483,13 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset,
* section (for e.g, in cachestat) - these callers need to skip flushing
* stats (via the flush argument).
*
+ * For the root memcg, the LRU sizes are read directly from the node
+ * counters (see lruvec_page_state()).
+ *
* XXX: With per-memcg flushing and thresholding, is ratelimiting
* still needed here?
*/
- if (flush)
+ if (flush && !mem_cgroup_is_root(eviction_memcg))
mem_cgroup_flush_stats_ratelimited(eviction_memcg);
eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat);
--
2.52.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH v3 3/4] mm: workingset: skip the root memcg flush in workingset_test_recent()
2026-11-14 0:28 ` [PATCH v3 3/4] mm: workingset: skip the root memcg flush in workingset_test_recent() Joanne Koong
@ 2026-11-17 5:07 ` Chris Down
0 siblings, 0 replies; 9+ messages in thread
From: Chris Down @ 2026-11-17 5:07 UTC (permalink / raw)
To: Joanne Koong
Cc: akpm, david, yosry, mkoutny, shakeel.butt, hannes, mhocko,
roman.gushchin, muchun.song, nphamcs, chengming.zhou, sj, ljs,
linux-mm, cgroups, damon
Hi Joanne,
Joanne Koong writes:
>- if (flush)
>+ if (flush && !mem_cgroup_is_root(eviction_memcg))
> mem_cgroup_flush_stats_ratelimited(eviction_memcg);
Not a big deal, but in the v2 thread you mentioned you would try a
mem_cgroup_flush_lru_stats_ratelimited() helper for this, per Michal's
suggestion. That still seems like a good idea given this skip and the one in
DAMON rely on everything read after them staying in lruvec_use_node_stat(),
and at least speaking for myself, keeping the root check close to the list
in memcontrol.c makes that easier to see. It doesn't need
to hold this patch up though, and I've tested that refaults in the root
cgroup no longer flush, so either way:
Reviewed-by: Chris Down <chris@chrisdown.name>
Tested-by: Chris Down <chris@chrisdown.name>
Thanks!
Chris
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v3 4/4] mm/damon/core: use node counters instead of flushing the root memcg
2026-11-14 0:28 [PATCH v3 0/4] mm: read root memcg stats from global counters instead of flushing Joanne Koong
` (2 preceding siblings ...)
2026-11-14 0:28 ` [PATCH v3 3/4] mm: workingset: skip the root memcg flush in workingset_test_recent() Joanne Koong
@ 2026-11-14 0:28 ` Joanne Koong
2026-11-17 5:07 ` Chris Down
3 siblings, 1 reply; 9+ messages in thread
From: Joanne Koong @ 2026-11-14 0:28 UTC (permalink / raw)
To: akpm, david
Cc: yosry, mkoutny, shakeel.butt, hannes, mhocko, roman.gushchin,
muchun.song, nphamcs, chengming.zhou, sj, ljs, linux-mm, cgroups,
damon
damos_get_node_memcg_used_bp() flushes the memcg stats before reading
the LRU sizes of the goal's memcg on the goal's node. For the root
memcg, that flushes every cgroup in the system. The flush is not
ratelimited, which means it happens every time the goal is evaluated.
lruvec_page_state() now reads the LRU sizes of the root memcg from the
node counters, which are always up to date, so skip the flush for the
root memcg.
Reviewed-by: SJ Park <sj@kernel.org>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
mm/damon/core.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/mm/damon/core.c b/mm/damon/core.c
index 0a056949604f..14f2a36c1025 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -3057,7 +3057,12 @@ static unsigned long damos_get_node_memcg_used_bp(
return 10000;
}
- mem_cgroup_flush_stats(memcg);
+ /*
+ * The root memcg's LRU sizes are read directly from the node counters
+ * (see lruvec_page_state()).
+ */
+ if (!mem_cgroup_is_root(memcg))
+ mem_cgroup_flush_stats(memcg);
lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(goal->nid));
used_pages = lruvec_page_state(lruvec, NR_ACTIVE_ANON);
used_pages += lruvec_page_state(lruvec, NR_INACTIVE_ANON);
--
2.52.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH v3 4/4] mm/damon/core: use node counters instead of flushing the root memcg
2026-11-14 0:28 ` [PATCH v3 4/4] mm/damon/core: use node counters instead of flushing the root memcg Joanne Koong
@ 2026-11-17 5:07 ` Chris Down
0 siblings, 0 replies; 9+ messages in thread
From: Chris Down @ 2026-11-17 5:07 UTC (permalink / raw)
To: Joanne Koong
Cc: akpm, david, yosry, mkoutny, shakeel.butt, hannes, mhocko,
roman.gushchin, muchun.song, nphamcs, chengming.zhou, sj, ljs,
linux-mm, cgroups, damon
Hi Joanne,
I've tested that this works well.
Reviewed-by: Chris Down <chris@chrisdown.name>
Tested-by: Chris Down <chris@chrisdown.name>
Thanks!
Chris
^ permalink raw reply [flat|nested] 9+ messages in thread