memcg stats flushing (WAS "Re: [PATCH] fuse: disable default bdi strictlimiting")

memcg stats flushing (WAS "Re: [PATCH] fuse: disable default bdi strictlimiting")

[Joanne Koong]

Hi Yosry,

On Wed, Jul 22, 2026 at 1:38 PM Yosry Ahmed <yosry@kernel.org> wrote:
>
> > > > First let's see if the general solution is doable and simple otherwise we can
> > > > explore use-case specific solutions. In extreme case revert is an options as
> > > > well but I would prefer a more sophisticated ratelimiting (that Jan suggested)
> > > > than a revert.
> > > >
> > >
> > > Sounds good. I'll keep an eye out for the work Yosry's doing on this
> > > and look into hooking it up to writeback if/when his general solution
> > > becomes available.
> >
> > I am not sure if I had in mind would generalize well. For zswap, I was
> > just going to replace rstat with per-memcg atomic counters, and simply
> > walk the cgroup parents in the update path, as the zswap load/store path
> > shouldn't be too hot.
> >
> > However, looking at other in-kernel flushers (including
> > mem_cgroup_wb_stats()), I see they consume other stats like
> > NR_INACTIVE_FILE/NR_ACTIVE_FILE, which are updated in the page
> > allocation path. I am not sure if a cgroup parent walk with atomic
> > updates would fly there.
> >
> > A more problematic one is count_shadow_nodes(), which consumes slab
> > object stats. I think we definitely cannot do atomic updates in the slab
> > allocation path. Although it seems like count_shadow_nodes() is a rough
> > estimate and perhaps we can forgoe using the stats there.
> >
> > If we want a generic solution for in-kernel flushers to improve stats
> > accuracy without killing performance, I think we need a heavier lift to
> > rework rstat or move away from it completely.
>
> (Adding a few more people and mailing lists)
>
> So I experimented with a per-memcg atomics in zswap (see [1] for
> context), and it appears to be too expensive even for updating the
> zswap stats during reclaim / swap faults. So I think zswap will need a
> similar approach to writeback and other in-kernel users.

Thanks for running the experiment.

In a previous thread [1], you mentioned your team was investigating
some premature OOM kills that might be attributed to this same stale
stats issue in the vmscan path. Did that turn out to be the case?
Asking because if so, that might be something we'd hit at Meta too.

>
> One idea is to keep per-CPU per-cgroup counters, but on the update
> side we update all the parents' counters, instead of just updating the
> current cgroup's counters. Flushing the stats then only needs to
> iterate the per-CPU counters and doesn't need to walk the cgroup tree
> at all. There is added work on the update side, but hopefully it's
> cheap enough to update per-CPU counters, and cgroup hierarchies aren't
> usually too deep.

I'm not sure about the zswap case, but for the writeback case there
are significantly more updates than reads. There are four stat updates
for every folio that gets dirtied / written back, so four per page
with 4k folios, while the read happens around every few hundred pages
dirtied. In the scenario I was testing with memory.max set to 8 GB and
using the default 20% dirty ratio, during freerun the stat gets read
once every 256 to 512 pages (with flushes happening less frequently
than reads).

I'm not too familiar with memcg or vmstat so apologies if this is a
naive question, but does it make sense for memcg to just do a similar
approach to what vmstat does? As I understand it, mod_lruvec_state()
updates the node counter and the memcg counter (if memcg applies),
where for updating the node counter, mod_node_state() does some
batching where it updates the global counter for that stat only once
it crosses some threshold (pcp->stat_threshold). Could memcg keep its
per-CPU counter as the accumulator but once it reaches some threshold,
it then propagates it up the parent/ancestors into a per-memcg atomic
global counter? The atomic would then only have to be touched once per
batch than on every update, which might eliminate the overhead of the
per-memcg atomic you saw previously? It seems like this would get rid
of needing to do any flushing altogether, as we could just read that
global counter directly.

>
> A more radical approach is to completely drop the per-CPU rstat trees.
> So instead of walking the parents to add them to the per-CPU tree, we
> just walk the parents and directly update the stat. The flush side
> will then always iterate the per-CPU counters to accumulate them. With
> this approach, the update cost roughly remains the same, but the flush
> cost changes. Instead of walking the trees and flushing all cgroups,
> but only doing it on CPUs/cgroups that actually have updates, we
> always iterate and sum the per-CPU counters. The average flush cost
> may become more expensive, but it should be more consistent and won't
> scale with the cgroup tree size. I am not sure if this would be a net
> win or not.

I wonder if this scales for machines that have a significant number of
cores? Onsomething like the 96-core / 192-thread machine Yunzhao
mentioned in the zswap case, it seems like the flush now would have to
touch multiple additional cachelines per CPU when it seems like for
the majority of the stats, only a small fraction of those hardware
threads would have any updates on them?

>
> We can also choose to do the above for all the stats or only the ones
> used by in-kernel flushers, the latter would probably perform better
> but at the cost of more complexity.

I agree. For the writeback case, afaict I think we really only need to
do this for NR_FILE_DIRTY and NR_WRITEBACK, neither of which are
updated from the page allocation path.

>
> I honestly don't have any time to pursue any of these approaches, so
> this is mostly just a brain dump. But I do think we should try to more
> systematically solve the stats flushing problem as it has given us a
> lot of pain over the years.
>

Thanks for writing up your thoughts on this - I found it helpful!

Thanks,
Joanne

[1] https://lore.kernel.org/linux-mm/CAO9r8zOiwgG80EjqdZFgWQvODUe1RzecLemxcB7xXNm_f3XfbA@mail.gmail.com/

Re: memcg stats flushing (WAS "Re: [PATCH] fuse: disable default bdi strictlimiting")

[Joanne Koong]

On Thu, Aug 6, 2026 at 2:04 PM Joanne Koong <joannelkoong@gmail.com> wrote:
>
>

Sorry, I think I messed up how things got linked when trying to update
the thread title in gmail. This is a continuation of the previous
discussion thread ("PATCH] fuse: disable default bdi strictlimiting")
which is at https://lore.kernel.org/linux-fsdevel/s4kv7qep3jlncs67q72xfhserpd66hpbwzkoppiddc5raufyjd@ksuvlfqcrbk2/T/#t.

Thanks,
Joanne

Re: memcg stats flushing (WAS "Re: [PATCH] fuse: disable default bdi strictlimiting")

[Yosry Ahmed]

On Thu, Aug 6, 2026 at 2:04 PM Joanne Koong <joannelkoong@gmail.com> wrote:
>
> Hi Yosry,
>
> On Wed, Jul 22, 2026 at 1:38 PM Yosry Ahmed <yosry@kernel.org> wrote:
> >
> > > > > First let's see if the general solution is doable and simple otherwise we can
> > > > > explore use-case specific solutions. In extreme case revert is an options as
> > > > > well but I would prefer a more sophisticated ratelimiting (that Jan suggested)
> > > > > than a revert.
> > > > >
> > > >
> > > > Sounds good. I'll keep an eye out for the work Yosry's doing on this
> > > > and look into hooking it up to writeback if/when his general solution
> > > > becomes available.
> > >
> > > I am not sure if I had in mind would generalize well. For zswap, I was
> > > just going to replace rstat with per-memcg atomic counters, and simply
> > > walk the cgroup parents in the update path, as the zswap load/store path
> > > shouldn't be too hot.
> > >
> > > However, looking at other in-kernel flushers (including
> > > mem_cgroup_wb_stats()), I see they consume other stats like
> > > NR_INACTIVE_FILE/NR_ACTIVE_FILE, which are updated in the page
> > > allocation path. I am not sure if a cgroup parent walk with atomic
> > > updates would fly there.
> > >
> > > A more problematic one is count_shadow_nodes(), which consumes slab
> > > object stats. I think we definitely cannot do atomic updates in the slab
> > > allocation path. Although it seems like count_shadow_nodes() is a rough
> > > estimate and perhaps we can forgoe using the stats there.
> > >
> > > If we want a generic solution for in-kernel flushers to improve stats
> > > accuracy without killing performance, I think we need a heavier lift to
> > > rework rstat or move away from it completely.
> >
> > (Adding a few more people and mailing lists)
> >
> > So I experimented with a per-memcg atomics in zswap (see [1] for
> > context), and it appears to be too expensive even for updating the
> > zswap stats during reclaim / swap faults. So I think zswap will need a
> > similar approach to writeback and other in-kernel users.
>
> Thanks for running the experiment.
>
> In a previous thread [1], you mentioned your team was investigating
> some premature OOM kills that might be attributed to this same stale
> stats issue in the vmscan path. Did that turn out to be the case?
> Asking because if so, that might be something we'd hit at Meta too.

I don't think we had time to look into this closely. We have a test
that runs VMs with some memory pressure / reclaim, and we observed OOM
kills in that test after the ratelimited flush in the vmscan path.
Whether or not this translates to additional OOM kills in our prod is
currently unknown, but it's a signal that reclaim effectiveness may be
negatively affected in some cases.

> >
> > One idea is to keep per-CPU per-cgroup counters, but on the update
> > side we update all the parents' counters, instead of just updating the
> > current cgroup's counters. Flushing the stats then only needs to
> > iterate the per-CPU counters and doesn't need to walk the cgroup tree
> > at all. There is added work on the update side, but hopefully it's
> > cheap enough to update per-CPU counters, and cgroup hierarchies aren't
> > usually too deep.
>
> I'm not sure about the zswap case, but for the writeback case there
> are significantly more updates than reads. There are four stat updates
> for every folio that gets dirtied / written back, so four per page
> with 4k folios, while the read happens around every few hundred pages
> dirtied. In the scenario I was testing with memory.max set to 8 GB and
> using the default 20% dirty ratio, during freerun the stat gets read
> once every 256 to 512 pages (with flushes happening less frequently
> than reads).

Yeah I think we generally want to keep updates faster than reads,
because some of them are also in latency-sensitive paths like
networking. However, since reads are becoming too slow in some cases,
the question here is whether we can afford some extra cost on the
update side (to iterate parent per-CPU counters) to make the read path
more manageable.

> I'm not too familiar with memcg or vmstat so apologies if this is a
> naive question, but does it make sense for memcg to just do a similar
> approach to what vmstat does? As I understand it, mod_lruvec_state()
> updates the node counter and the memcg counter (if memcg applies),
> where for updating the node counter, mod_node_state() does some
> batching where it updates the global counter for that stat only once
> it crosses some threshold (pcp->stat_threshold). Could memcg keep its
> per-CPU counter as the accumulator but once it reaches some threshold,
> it then propagates it up the parent/ancestors into a per-memcg atomic
> global counter? The atomic would then only have to be touched once per
> batch than on every update, which might eliminate the overhead of the
> per-memcg atomic you saw previously? It seems like this would get rid
> of needing to do any flushing altogether, as we could just read that
> global counter directly.

The memcg stats perform similar per-CPU counting and thresholding. The
main difference from vmstat AFAICT is the cgroup hierarchy. In vmstat
we have per-CPU counters and a global atomic counter for each stat.
For memcg, we have the same for each cgroup, then we also have a
cgroup tree. When we read the stats of a cgroup, we usually want the
hierarchical stats including all of its children, so we can't just
read the global counter.

We use rstat, a cgroup framework that keeps track of which cgroups
have updates on which CPUs, and then memcg has thresholding logic on
top to only flush if the number of pending updates exceeds a
threshold. Essentially, if the magnitude updates on a CPU exceeds
MEMCG_CHARGE_BATCH, we add it to a global per-memcg counter. We only
flush if that global per-memcg counter has > NR_CPUS *
MEMCG_CHARGE_BATCH updates. So in theory I think we tolerate up to 2 *
NR_CPUS * MEMCG_CHARGE_BATCH of stale stats (combined for all the
stats).

See memcg_rstat_updated() and memcg_vmstats_needs_flush() for this logic.

Looking at this code again, it made me realize that we already iterate
the per-CPU stats_updates on every update to increment the number of
pending stats on this CPU. I wonder if we can replace this with
actually updating the stat on each parent, then the flush path only
needs to accumulate CPU counters. Or maybe as you mention, we update
the global per-memcg counter when updates on a CPU exceed a threshold,
and forgo the rstat flushing logic completely.

> > A more radical approach is to completely drop the per-CPU rstat trees.
> > So instead of walking the parents to add them to the per-CPU tree, we
> > just walk the parents and directly update the stat. The flush side
> > will then always iterate the per-CPU counters to accumulate them. With
> > this approach, the update cost roughly remains the same, but the flush
> > cost changes. Instead of walking the trees and flushing all cgroups,
> > but only doing it on CPUs/cgroups that actually have updates, we
> > always iterate and sum the per-CPU counters. The average flush cost
> > may become more expensive, but it should be more consistent and won't
> > scale with the cgroup tree size. I am not sure if this would be a net
> > win or not.
>
> I wonder if this scales for machines that have a significant number of
> cores? Onsomething like the 96-core / 192-thread machine Yunzhao
> mentioned in the zswap case, it seems like the flush now would have to
> touch multiple additional cachelines per CPU when it seems like for
> the majority of the stats, only a small fraction of those hardware
> threads would have any updates on them?

Probably not (hence I called it radical), but I think this can be
side-stepped by either keeping the tracking of CPUs that have updates
via rstat (or similar), or just propagating the per-CPU stat updates
when they reach a threshold like vmstat.

Re: memcg stats flushing (WAS "Re: [PATCH] fuse: disable default bdi strictlimiting")

[Joanne Koong]

On Thu, Aug 6, 2026 at 2:31 PM Yosry Ahmed <yosry@kernel.org> wrote:
>
> On Thu, Aug 6, 2026 at 2:04 PM Joanne Koong <joannelkoong@gmail.com> wrote:
> >
> > I'm not too familiar with memcg or vmstat so apologies if this is a
> > naive question, but does it make sense for memcg to just do a similar
> > approach to what vmstat does? As I understand it, mod_lruvec_state()
> > updates the node counter and the memcg counter (if memcg applies),
> > where for updating the node counter, mod_node_state() does some
> > batching where it updates the global counter for that stat only once
> > it crosses some threshold (pcp->stat_threshold). Could memcg keep its
> > per-CPU counter as the accumulator but once it reaches some threshold,
> > it then propagates it up the parent/ancestors into a per-memcg atomic
> > global counter? The atomic would then only have to be touched once per
> > batch than on every update, which might eliminate the overhead of the
> > per-memcg atomic you saw previously? It seems like this would get rid
> > of needing to do any flushing altogether, as we could just read that
> > global counter directly.
>
> The memcg stats perform similar per-CPU counting and thresholding. The
> main difference from vmstat AFAICT is the cgroup hierarchy. In vmstat
> we have per-CPU counters and a global atomic counter for each stat.
> For memcg, we have the same for each cgroup, then we also have a
> cgroup tree. When we read the stats of a cgroup, we usually want the
> hierarchical stats including all of its children, so we can't just
> read the global counter.
>
> We use rstat, a cgroup framework that keeps track of which cgroups
> have updates on which CPUs, and then memcg has thresholding logic on
> top to only flush if the number of pending updates exceeds a
> threshold. Essentially, if the magnitude updates on a CPU exceeds
> MEMCG_CHARGE_BATCH, we add it to a global per-memcg counter. We only
> flush if that global per-memcg counter has > NR_CPUS *
> MEMCG_CHARGE_BATCH updates. So in theory I think we tolerate up to 2 *
> NR_CPUS * MEMCG_CHARGE_BATCH of stale stats (combined for all the
> stats).
>
> See memcg_rstat_updated() and memcg_vmstats_needs_flush() for this logic.
>
> Looking at this code again, it made me realize that we already iterate
> the per-CPU stats_updates on every update to increment the number of
> pending stats on this CPU. I wonder if we can replace this with
> actually updating the stat on each parent, then the flush path only
> needs to accumulate CPU counters. Or maybe as you mention, we update
> the global per-memcg counter when updates on a CPU exceed a threshold,
> and forgo the rstat flushing logic completely.
>

I can run some experiments and see if updating the global per-memcg
counter per batch and forgoing flushing could be a viable solution. I
can use my setup to test it for writeback, but for the zswap use case,
what's the best way to test if the changes are too expensive? Is there
a benchmark program you're running?

For updating the stat on each parent and having the flush path only
accumulate CPU counters, I think this means we would have to
unconditionally walk the ancestors + update its counters on every
update, as we wouldn't be able to early break if the flushable
threshold was already exceeded. I don't think we could replace the
stats_update update, as it seems like readers still need some way of
knowing when to flush. It seems like this could be a nontrivial
performance hit, but maybe it's a non-factor in reality. If you think
this would be useful to get benchmark numbers on, I can run some
experiments on this too.

Thanks,
Joanne

Re: memcg stats flushing (WAS "Re: [PATCH] fuse: disable default bdi strictlimiting")

[Yosry Ahmed]

On Mon, Aug 10, 2026 at 09:49:31AM -0700, Joanne Koong wrote:
> On Thu, Aug 6, 2026 at 2:31 PM Yosry Ahmed <yosry@kernel.org> wrote:
> >
> > On Thu, Aug 6, 2026 at 2:04 PM Joanne Koong <joannelkoong@gmail.com> wrote:
> > >
> > > I'm not too familiar with memcg or vmstat so apologies if this is a
> > > naive question, but does it make sense for memcg to just do a similar
> > > approach to what vmstat does? As I understand it, mod_lruvec_state()
> > > updates the node counter and the memcg counter (if memcg applies),
> > > where for updating the node counter, mod_node_state() does some
> > > batching where it updates the global counter for that stat only once
> > > it crosses some threshold (pcp->stat_threshold). Could memcg keep its
> > > per-CPU counter as the accumulator but once it reaches some threshold,
> > > it then propagates it up the parent/ancestors into a per-memcg atomic
> > > global counter? The atomic would then only have to be touched once per
> > > batch than on every update, which might eliminate the overhead of the
> > > per-memcg atomic you saw previously? It seems like this would get rid
> > > of needing to do any flushing altogether, as we could just read that
> > > global counter directly.
> >
> > The memcg stats perform similar per-CPU counting and thresholding. The
> > main difference from vmstat AFAICT is the cgroup hierarchy. In vmstat
> > we have per-CPU counters and a global atomic counter for each stat.
> > For memcg, we have the same for each cgroup, then we also have a
> > cgroup tree. When we read the stats of a cgroup, we usually want the
> > hierarchical stats including all of its children, so we can't just
> > read the global counter.
> >
> > We use rstat, a cgroup framework that keeps track of which cgroups
> > have updates on which CPUs, and then memcg has thresholding logic on
> > top to only flush if the number of pending updates exceeds a
> > threshold. Essentially, if the magnitude updates on a CPU exceeds
> > MEMCG_CHARGE_BATCH, we add it to a global per-memcg counter. We only
> > flush if that global per-memcg counter has > NR_CPUS *
> > MEMCG_CHARGE_BATCH updates. So in theory I think we tolerate up to 2 *
> > NR_CPUS * MEMCG_CHARGE_BATCH of stale stats (combined for all the
> > stats).
> >
> > See memcg_rstat_updated() and memcg_vmstats_needs_flush() for this logic.
> >
> > Looking at this code again, it made me realize that we already iterate
> > the per-CPU stats_updates on every update to increment the number of
> > pending stats on this CPU. I wonder if we can replace this with
> > actually updating the stat on each parent, then the flush path only
> > needs to accumulate CPU counters. Or maybe as you mention, we update
> > the global per-memcg counter when updates on a CPU exceed a threshold,
> > and forgo the rstat flushing logic completely.
> >
> 
> I can run some experiments and see if updating the global per-memcg
> counter per batch and forgoing flushing could be a viable solution.
> I can use my setup to test it for writeback, but for the zswap use
> case, what's the best way to test if the changes are too expensive? Is
> there a benchmark program you're running?

I was running stress-ng and measuring the latency of zswap_load() and
zswap_store() with bpftrace. But I suspect whatever works for writeback
should work for zswap :)

> 
> For updating the stat on each parent and having the flush path only
> accumulate CPU counters, I think this means we would have to
> unconditionally walk the ancestors + update its counters on every
> update, as we wouldn't be able to early break if the flushable
> threshold was already exceeded. I don't think we could replace the
> stats_update update, as it seems like readers still need some way of
> knowing when to flush. It seems like this could be a nontrivial
> performance hit, but maybe it's a non-factor in reality. If you think
> this would be useful to get benchmark numbers on, I can run some
> experiments on this too.

I think we need to experiment with how much work we can shift around
between the update and reader sides.

Right now, on the update side, we call __css_rstat_updated(), which
iterates the parents and puts them on the per-CPU rstat tree, then we
iterate the parents again in memcg_rstat_updated() and:
- Check if flushing threshold is already met.
- Update per-CPU counter for stats_updates.
- If per-CPU update exceeds (another) threshold, we update an atomic for
  stats_updates.

The flush side checks the flushing threshold and does an rstat flush,
which will walk the update trees on all CPUs and iterates all child
cgroups on it. Pretty expensive when the threshold is met.

I think we discussed two different options:

(a) On stats updates, iterate the parents and update the per-CPU
counters of the stat directly.

In this case, we can't keep using rstat, as we would still need to add
the cgroup and all its ancestors to the update tree anyway and the flush
cost will remain similar. So on the flush path we'll need to iterate all
CPUs and add the counters. We can explore if use a cpumask can help
here, but perhaps not since we'll need to update it atomically on all
stat updates.

I imagine that if we do this we'll drop the stats_updates optimization
to avoid adding more cost to the update side. We'll be replacing parent
stats_updates modification with parent stat counter modification. But
maybe it's feasible to keep both to limit the cost of flusing.

This only works if the cost of iterate the CPUs on the flush side ends
up being cheaper than the current flush. I imagine the average cost will
be higher but the cost should be much more consistent vs the current
flush, so the tail should be better.

(b) On stats updates, iterate the parents and update the per-CPU
counters of the stat directly AND update a global atomic if the per-CPU
counter exceeds a threshold. I think updates may get too expensive as we
may need to update multiple atomics, especially when updating the root
counter as it will be contended by all cgroups.

The flush side becomes an atomic read, so should be very cheap, and we
can drop other heuristics and optimizations.

---

I think ultimately we need to experiment with these approachs (and
perhaps others) to check what actually works in practice. You'll
probably want some benchmarks or synthetic tests with many cgroups to
exercise the worst case scenarios.

For the zswap stats, I used to run stress-ng with a bunch of workers
(20, 50, 100, ..) in cgroups of different levels. Since all workers are
in the same cgroups, all the parents are common and all threads end up
competing on the same atomics, so worst case scenario. I was also
testing in a VM that has more CPUs than stress-ng workers, to exercise
the full extent of the concurrency and contention.

This could be a lot of work, so I am not really asking you to do it, but
just brainstorming and sharing ideas :)