Re: [PATCH] fuse: disable default bdi strictlimiting

[Shakeel Butt <shakeel.butt@linux.dev>]

Hi Jan & Joanne,

On Wed, Jul 15, 2026 at 06:28:16PM -0700, Joanne Koong wrote:
> On Wed, Jul 15, 2026 at 3:32 AM Jan Kara <jack@suse.cz> wrote:
> >
> > [resending once more, this time with correct Shakeel's address]
> >
> > On Tue 14-07-26 18:21:57, Joanne Koong wrote:
> > > On Tue, Jul 14, 2026 at 10:10 AM Jan Kara <jack@suse.cz> wrote:
> > > > On Mon 13-07-26 18:13:29, Joanne Koong wrote:
> > > > > On Sat, May 30, 2026 at 4:04 AM Jan Kara <jack@suse.cz> wrote:
> > > > > > On Thu 28-05-26 15:11:18, Joanne Koong wrote:
> > > > > > > On Thu, May 28, 2026 at 5:34 AM Jan Kara <jack@suse.cz> wrote:
> > > > > > > > > I think this is also going to be a problem for cgroups with large
> > > > > > > > > folios since they also, as I understand it, are constrained with a
> > > > > > > > > limited / tight dirty budget. I ran some initial benchmarks with
> > > > > > > > > cgroup memory constraints on NVMe and saw similar instability (a
> > > > > > > > > single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> > > > > > > > > 15 ms without the cgroup, with the balance_dirty_pages() throttling
> > > > > > > > > oscillating rather than settling near the set point).
> > > > > > > >
> > > > > > > > Ok, so this is more folios (819) than my 512 gut feeling estimate :) What
> > > > > > > > was the write throughput of the NVMe drive? The high drive throughput also
> > > > > > > > requires more dirty data to keep the drive saturated so that writeback
> > > > > > > > throughput doesn't oscilate too much.
> > > > > > >
> > > > > > > The write throughput of the NVMe drive I was using was around ~1.1
> > > > > > > Gb/s (measured by running direct I/O). I think with the 1.6GB dirty
> > > > > > > budget, the math for that ends up being that the device drains it in
> > > > > > > ~1.5 secs. The performance I was seeing with the initial cgroup
> > > > > > > benchmarks was with 4k pages (no large folios enabled) on btrfs.
> > > > > >
> > > > > > OK, you might want to experiment with some other filesystem (I suggest xfs
> > > > > > or ext4) as well. Btrfs writeback behavior is a bit special with its data
> > > > > > checksum computations etc. and thus latency of starting writeback. It could
> > > > > > contribute to the erratic behavior with the relatively low dirty limits.
> > > > >
> > > > > I reran the benchmarks on xfs and ext4 and saw similiar results. The
> > > > > max write latency under the memcg were
> > > > > xfs        5.1s
> > > > > ext4      5.2 s
> > > > > btrfs      6.2 s
> > > > >
> > > > > compared to the non-memcg case (0.4 to 2.0 ms). This didn't affect
> > > > > throughput though, just the tail latency. When I ran xfs with large
> > > > > folios disabled, I still saw ~4.1s.
> > > >
> > > > OK, interesting. Thanks for running these tests! Also it is good to know
> > > > this is not really related to large folios, that somewhat simplifies
> > > > matters.
> > > >
> > > > > From the balance_dirty_pages() ftrace tracepoints, the dirty_ratelimit
> > > > > value is consistently stable / accurate but it looks like what's
> > > > > happening is that during freerun, the writer essentially dirties at
> > > > > memcpy speed until the freerun ceiling, and then the soft/proportional
> > > > > throttling kicks in but doesn't kick in fast or hard enough, which
> > > > > allows the number of dirty pages to exceed the hard limit by ~2x, and
> > > > > then at that point the writer is forced into the loop where it sleeps
> > > > > max_pause (200ms) each iteration until writeback has drained the
> > > > > number of dirty pages back under the limit.
> > > >
> > > > OK, above you mentioned that the dirty limit for the memcgs is set at
> > > > 1.6GB. Does that mean that dirty throttling allows memcg to dirty up to
> > > > ~3.2GB of pages? I wouldn't have expected that...
> > >
> > > On a couple of the runs, I saw it get as high as up to 5GB.
> > >
> > > I think this is because the balance_dirty_pages() code uses the memcg
> > > stats (NR_FILE_DIRTY) but these stats are only periodically
> > > flushed/refreshed when reading it, so the balance dirty code is
> > > seeing/using lagging/non-uptodate values.
> > >
> > > I'm seeing this in
> > >   balance_dirty_pages()
> > >       balance_domain_limits(mdtc,...)
> > >           domain_dirty_avail()
> > >                 mem_cgroup_wb_stats()
> > >                        mem_cgroup_flush_stats_ratelimited()
> > >
> > > where mem_cgroup_flush_stats_ratelimted() has this logic:
> > >
> > > void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg)
> > > {
> > >         /* Only flush if the periodic flusher is one full cycle late */
> > >         if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME))
> > >                 mem_cgroup_flush_stats(memcg);
> > > }
> > >
> > > where FLUSH_TIME is defined as 2UL * HZ, which afaict means the
> > > flushing can be around 2 seconds stale (accounting for the periodic
> > > flusher in flush_memcg_stats_dwork() that flushes every FLUSH_TIME
> > > interval).
> > >
> > > From what I see, the freerun and hard limit checks use this stale
> > > under-reported value (in domain_dirty_freerun() and
> > > wb_position_ratio()), which means the writer gets to keep dirtying and
> > > blow past the limit for up to 2 extra seconds before the throttling
> > > limits it.
> >
> > I see. That would indeed explain why we can overrun the dirty limit so
> > much.
> >
> > > Maybe one idea is to in the memcg throttle path do the flush based on
> > > how many pages are getting dirtied instead of on how much time has
> > > elapsed? It looks like flushing grabs the rstat lock though, so maybe
> > > only doing it in the throttled case where it's past freerun would be
> > > best. It looks like the logic in the regular non-ratelimited
> > > mem_cgroup_flush_stats() call does something similar. I can run some
> > > experiments this week with this if you think it could be promising.
> >
> > Checking some git history, this actually seems to be a relatively recent
> > change from Shakeel (added to CC) - d9b3ce8769e3 ("mm: writeback: ratelimit
> > stat flush from mem_cgroup_wb_stats"). So you can check whether reverting
> > that change makes dirty throttling not overshoot so much.
> >
> 
> Reverting d9b3ce8769e3 fixes the issue. On xfs, I'm now seeing the
> number of dirty pages stays bounded by the limit instead of going 2x+
> over, and the max latency is now around ~0.33s instead of ~4s.
> 
> > Shakeel, your change to use mem_cgroup_flush_stats_ratelimited() in
> > mem_cgroup_wb_stats() results in applications overshooting memcg dirty
> > limits by several gigabytes which for small memcgs (with 8GB memory in this
> > case, which means a dirty limit of ~1.5GB) is a considerable issue. I guess
> > the ratelimiting should take into account the size of memcg (or ideally the
> > dirty limit in the case of mem_cgroup_wb_stats()) as well. Using
> > somewhat outdated numbers for dirty throttling is fine but when we can
> > overshoot the dirty limit more than three times, it leads to rather long
> > stalls in dirty throttling and similar issues...

Yeah I agree this is an issue.

> 
> It looks like commit d9b3ce8769e3 was landed in Feb 2024 when rstat
> was still using the single global lock. I see commit 748922dcfabd
> ("cgroup: use subsystem-specific rstat locks to avoid contention")
> later in the tree, which was landed around May 2025 and reduces that
> global lock contention. That might make the cost of the flush a lot
> lower than what it was when the ratelimit was needed in d9b3ce8769e3,
> so maybe reverting d9b3ce8769e3 wouldn't make much of a difference for
> write-heavy workloads and would be the simplest fix?

Unfortunetely global lock was just part of the cost and even per-subsystem
lock reduces contention across subsystems but flushing is still expensive as the
kernel may have to traverse the cgroup update tree for each cpu on the system.

CCing Yosry who is looking into similar issue but for zswap stats. Maybe a
general solution for such specific stats is to move them from rstat to a
separate mechanism where update can be a bit more expensive but the read side
can be cheap. 

> If it does still
> regress, I'd be happy to look into making the flush based on how many
> pages have been dirtied and scaled to the dirty limit, if that sounds
> like the best way forward. 

First let's see if the general solution is doable and simple otherwise we can
explore use-case specific solutions. In extreme case revert is an options as
well but I would prefer a more sophisticated ratelimiting (that Jan suggested)
than a revert.

> I tried running the sysbench
> oltp_write_only benchmark mentioned in d9b3ce8769e3 on my setup but
> couldn't get the database to dirty pages fast enough to trigger the
> balance dirty pages logic to see how it performs without the
> ratelimiting. If you still have the script for how you ran the test
> back in 2024, I'm happy to give it a go on my local machine.

Unfortunately I don't have those scripts or setup.