[PATCH] mm: memcg: use ratelimited stats flush in obj_cgroup_may_zswap()

[PATCH] mm: memcg: use ratelimited stats flush in obj_cgroup_may_zswap()

[Song Hu]

obj_cgroup_may_zswap() runs on every folio swapped out through
zswap.  For each ancestor with a non-max zswap.max, it flushes the
cgroup rstat hierarchy synchronously with force=true, which skips
the ratelimit inside __mem_cgroup_flush_stats().  In a swap storm
with zswap.max configured, a container takes the global rstat lock
on every swapped-out folio.

zswap_shrinker_count() had the same pattern and switched to
mem_cgroup_flush_stats_ratelimited() in commit ea80da363a1f
("mm/zswap: use ratelimited stats flush in zswap_shrinker_count()"),
where the same flush on the shrinker side showed up at 2.88% of
kernel cycles under osq_lock on a 96-core machine.

Measured on a KVM guest with a swap storm under a cgroup with
zswap.max set: obj_cgroup_may_zswap() was entered 198,977 times
before the patch and 198,968 times after, while
__mem_cgroup_flush_stats() was entered 281,017 times before and
80,445 times after.  The removed 200,572 flushes match the store
attempt count almost exactly; the remainder comes from other stats
readers in the swap path.

The stats can now be up to one flusher cycle stale, so zswap.max
admission can overshoot for one cycle in a storm; the overshoot is
corrected as soon as the next flush lands and later stores see it,
the same tradeoff the shrinker side made.

Fixes: f4840ccfca25 ("zswap: memcg accounting")
Signed-off-by: Song Hu <husong@kylinos.cn>
---
 mm/memcontrol.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 17da1f43b7d3..7a8f689055c6 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -6000,8 +6000,7 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
 			break;
 		}
 
-		/* Force flush to get accurate stats for charging */
-		__mem_cgroup_flush_stats(memcg, true);
+		mem_cgroup_flush_stats_ratelimited(memcg);
 		pages = memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE;
 		if (pages < max)
 			continue;
-- 
2.43.0

Re: [PATCH] mm: memcg: use ratelimited stats flush in obj_cgroup_may_zswap()

[Shakeel Butt]

On Mon, Aug 17, 2026 at 09:18:43PM +0800, Song Hu wrote:
> obj_cgroup_may_zswap() runs on every folio swapped out through
> zswap.  For each ancestor with a non-max zswap.max, it flushes the
> cgroup rstat hierarchy synchronously with force=true, which skips
> the ratelimit inside __mem_cgroup_flush_stats().  In a swap storm
> with zswap.max configured, a container takes the global rstat lock
> on every swapped-out folio.

Any reason you are limiting zswap through zswap.max?

> 
> zswap_shrinker_count() had the same pattern and switched to
> mem_cgroup_flush_stats_ratelimited() in commit ea80da363a1f
> ("mm/zswap: use ratelimited stats flush in zswap_shrinker_count()"),
> where the same flush on the shrinker side showed up at 2.88% of
> kernel cycles under osq_lock on a 96-core machine.
> 
> Measured on a KVM guest with a swap storm under a cgroup with
> zswap.max set: obj_cgroup_may_zswap() was entered 198,977 times
> before the patch and 198,968 times after, while
> __mem_cgroup_flush_stats() was entered 281,017 times before and
> 80,445 times after.  The removed 200,572 flushes match the store
> attempt count almost exactly; the remainder comes from other stats
> readers in the swap path.

This is a known issue. Using ratelimited interface also comes with a drawback
that the kernel may react on stale information and the consequences might be
unneeded oom-kills.

There was orthogonal discussion on moving zswap limit enforcement away from
rstat. Yosry, any updates on that?

> 
> The stats can now be up to one flusher cycle stale, so zswap.max
> admission can overshoot for one cycle in a storm; the overshoot is
> corrected as soon as the next flush lands and later stores see it,
> the same tradeoff the shrinker side made.
> 
> Fixes: f4840ccfca25 ("zswap: memcg accounting")
> Signed-off-by: Song Hu <husong@kylinos.cn>
> ---
>  mm/memcontrol.c | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 17da1f43b7d3..7a8f689055c6 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -6000,8 +6000,7 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
>  			break;
>  		}
>  
> -		/* Force flush to get accurate stats for charging */
> -		__mem_cgroup_flush_stats(memcg, true);
> +		mem_cgroup_flush_stats_ratelimited(memcg);
>  		pages = memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE;
>  		if (pages < max)
>  			continue;
> -- 
> 2.43.0
>