Re: [PATCH] fuse: disable default bdi strictlimiting

Re: [PATCH] fuse: disable default bdi strictlimiting

[Joanne Koong]

On Tue, May 12, 2026 at 1:56 PM Joanne Koong <joannelkoong@gmail.com> wrote:
>
> On Fri, May 8, 2026 at 2:42 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
> >
> > On Mon, 27 Oct 2025 at 23:39, Joanne Koong <joannelkoong@gmail.com> wrote:
> > > Miklos, could you share your thoughts on this? Are you in favor of
> > > disabling default strictlimiting? Or do you prefer to have it kept
> > > enabled by default, with some mount option or sysctl added for
> > > privileged servers to be able to disable strictlimiting + enable large
> > > folios if they use the writeback cache?
> >
> > So what I think we should do is implement some sort of slow writer
> > test, and see what happens with and without strictlimit.
> >
> > Tried to ask claude to do this for me, but not getting very far.
> >
> > So if I take this maintainership role seriously and not let myself
> > drown in the details, then the logical thing to do is to delegate ;)
> > Which is hard (for me at least) but I'll give it a try...
> >
> > Could you please check how things change if there's limited writeback
> > rate and we disable strictlimit?  And what happens if there are
> > several such instances running in parallel?
>
> I think for unprivileged fuse servers, strictlimting will always need
> to be enabled or else a malicious user can launch tons of unprivileged
> servers and eat up the global dirty page budget / starve writeback for
> the rest of the system. Similarly for privileged servers, it could be
> unintentionally slow or buggy and eat up the dirty page budget. I'll
> read through the writeback throttling code to verify this and run some
> local tests.

I read through the writeback throttling code and re-read Jan's very
helpful comments from this thread last year [1]. So for unprivileged
servers, I think we definitely cannot remove strictlimiting. If the
fuse server is slow or unresponsive with writing back the pages, it
will take up too much of the global dirty budget which will degrade
write throughput for other filesystems (their throttling will be
computed against the global dirty page count, eg the freerunning check
in balance_dirty_pages() and the pos_ratio calculation "pos_ratio =
pos_ratio_polynom(setpoint, dtc->dirty, limit)" (dtc->dirty is the
global dirty page count)) and any fuse stuck dirty pages are
essentially unreclaimable. Without strictlimiting, there will be no
hard cap on how many dirty pages a misbehaving server can accumulate.

With strictlimiting on and large folios enabled, the problem is that
the large folio size can potentially dwarf the server's dirty budget,
which can lead to excessive throttling. When I ran my benchmarks last
year, I and independently Jingbo saw severe performance regressions
for buffered writes with large folios (eg 2 GB/s BW w/o, and 200 MB/s
BW w/) [2] but I think that might have been because the machines had
limited RAM, resulting in a very small dirty budget. Fuse sets the max
ratio of the bdi to 1% of the global dirty threshold, so running
through some napkin math:

On a 64 GB machine:
  - DirtyThresh = 20% of 64 GB = 12.8 GB
  - BdiDirtyThresh = 12.8 GB / 100 = 128 MB
  - 128 MB / 2 MB folio = 64 dirty folios

On a 32 GB machine:
  - BdiDirtyThresh = 64 MB
  - 32 dirty folios

On an 8 GB machine:
  - BdiDirtyThresh = 16 MB
  - 8 dirty folios

On a 8GB machine (with assuming vm.dirty_ratio=20% and
vm.dirty_background_ratio=10%), we get 12 MB of freerun, 4 MB of
proportional throttling, and then full throtttling starts at 16 MB.
With 2MB folios, the 4MB zone between freerun and f ull throttling
doesn't leave that much room for the balance_dirty_pages() logic to
adjust the dirtier's speed, which I think causes the writes to
oscillate between freerunning and then being fully (overly) throttled.

I think this is also going to be a problem for cgroups with large
folios since they also, as I understand it, are constrained with a
limited / tight dirty budget. I ran some initial benchmarks with
cgroup memory constraints on NVMe and saw similar instability (a
single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
15 ms without the cgroup, with the balance_dirty_pages() throttling
oscillating rather than settling near the set point).

I think this problem gets untenable for random writes with large
folios, since dirtying just a few bytes will charge the whole folio
size to the dirty budget. I have a patchset from last year for adding
more granular dirty/writeback tracking [3], I'm going to pick this
series back up. I think it will be useful generically, not just for
fuse.

For getting this to work on fuse servers with strictlimiting, I think
the next steps are to
a) as Jan had suggested in [1], come up with some heuristic to
constrain the max order supported for large folios for these fuse
servers if they're running with the writeback cache enabled
b) benchmark ^ and if there are still regressions, then we should
probably just turn large folios off for these servers
c) add the granular writeback/dirty accounting for large folios
d) look into improving the balance_dirty_pages() throttling logic to
handle narrow gaps between the freerun and full throttling zones
better and reduce over-throttling

Does this sound like a reasonable way forward?

For privileged servers, I still think it makes sense to remove the
strictlimiting requirement or at the least, let admins opt out of that
if they are confident their server is well-behaved.


Thanks,
Joanne

[1] https://lore.kernel.org/linux-fsdevel/tglgxjxcs3wpm4msgxlvzk3hebzcguhuu752hs3eefku6wj4zv@2ixuho7rxbah/
[2] https://lore.kernel.org/linux-fsdevel/f9b63a41-ced7-4176-8f40-6cba8fce7a4c@linux.alibaba.com/
[3] https://lore.kernel.org/linux-fsdevel/20250829233942.3607248-1-joannelkoong@gmail.com/

>
> I think the question is whether we want to let admins opt out of
> strictlimit when they're confident their server is well-behaved eg
> through a sysctl an admin can set to disable strictlimiting for all
> servers. Otherwise, large folios will always have to be off for any
> server that runs with writeback caching.
>
> Thanks,
> Joanne
> >
> > Thanks,
> > Miklos

Re: Re: [PATCH] fuse: disable default bdi strictlimiting

[Horst Birthelmer]

On Tue, May 26, 2026 at 06:42:35PM -0700, Joanne Koong wrote:
> On Tue, May 12, 2026 at 1:56 PM Joanne Koong <joannelkoong@gmail.com> wrote:
> >
> > On Fri, May 8, 2026 at 2:42 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
> > >
> > > On Mon, 27 Oct 2025 at 23:39, Joanne Koong <joannelkoong@gmail.com> wrote:
> > > > Miklos, could you share your thoughts on this? Are you in favor of
> > > > disabling default strictlimiting? Or do you prefer to have it kept
> > > > enabled by default, with some mount option or sysctl added for
> > > > privileged servers to be able to disable strictlimiting + enable large
> > > > folios if they use the writeback cache?
> > >
> > > So what I think we should do is implement some sort of slow writer
> > > test, and see what happens with and without strictlimit.
> > >
> > > Tried to ask claude to do this for me, but not getting very far.
> > >
> > > So if I take this maintainership role seriously and not let myself
> > > drown in the details, then the logical thing to do is to delegate ;)
> > > Which is hard (for me at least) but I'll give it a try...
> > >
> > > Could you please check how things change if there's limited writeback
> > > rate and we disable strictlimit?  And what happens if there are
> > > several such instances running in parallel?
> >
> > I think for unprivileged fuse servers, strictlimting will always need
> > to be enabled or else a malicious user can launch tons of unprivileged
> > servers and eat up the global dirty page budget / starve writeback for
> > the rest of the system. Similarly for privileged servers, it could be
> > unintentionally slow or buggy and eat up the dirty page budget. I'll
> > read through the writeback throttling code to verify this and run some
> > local tests.
> 
> I read through the writeback throttling code and re-read Jan's very
> helpful comments from this thread last year [1]. So for unprivileged
> servers, I think we definitely cannot remove strictlimiting. If the
> fuse server is slow or unresponsive with writing back the pages, it
> will take up too much of the global dirty budget which will degrade
> write throughput for other filesystems (their throttling will be
> computed against the global dirty page count, eg the freerunning check
> in balance_dirty_pages() and the pos_ratio calculation "pos_ratio =
> pos_ratio_polynom(setpoint, dtc->dirty, limit)" (dtc->dirty is the
> global dirty page count)) and any fuse stuck dirty pages are
> essentially unreclaimable. Without strictlimiting, there will be no
> hard cap on how many dirty pages a misbehaving server can accumulate.
> 
> With strictlimiting on and large folios enabled, the problem is that
> the large folio size can potentially dwarf the server's dirty budget,
> which can lead to excessive throttling. When I ran my benchmarks last
> year, I and independently Jingbo saw severe performance regressions
> for buffered writes with large folios (eg 2 GB/s BW w/o, and 200 MB/s
> BW w/) [2] but I think that might have been because the machines had
> limited RAM, resulting in a very small dirty budget. Fuse sets the max
> ratio of the bdi to 1% of the global dirty threshold, so running
> through some napkin math:
> 
> On a 64 GB machine:
>   - DirtyThresh = 20% of 64 GB = 12.8 GB
>   - BdiDirtyThresh = 12.8 GB / 100 = 128 MB
>   - 128 MB / 2 MB folio = 64 dirty folios
> 
> On a 32 GB machine:
>   - BdiDirtyThresh = 64 MB
>   - 32 dirty folios
> 
> On an 8 GB machine:
>   - BdiDirtyThresh = 16 MB
>   - 8 dirty folios
> 
> On a 8GB machine (with assuming vm.dirty_ratio=20% and
> vm.dirty_background_ratio=10%), we get 12 MB of freerun, 4 MB of
> proportional throttling, and then full throtttling starts at 16 MB.
> With 2MB folios, the 4MB zone between freerun and f ull throttling
> doesn't leave that much room for the balance_dirty_pages() logic to
> adjust the dirtier's speed, which I think causes the writes to
> oscillate between freerunning and then being fully (overly) throttled.
> 
> I think this is also going to be a problem for cgroups with large
> folios since they also, as I understand it, are constrained with a
> limited / tight dirty budget. I ran some initial benchmarks with
> cgroup memory constraints on NVMe and saw similar instability (a
> single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> 15 ms without the cgroup, with the balance_dirty_pages() throttling
> oscillating rather than settling near the set point).
> 
> I think this problem gets untenable for random writes with large
> folios, since dirtying just a few bytes will charge the whole folio
> size to the dirty budget. I have a patchset from last year for adding
> more granular dirty/writeback tracking [3], I'm going to pick this
> series back up. I think it will be useful generically, not just for
> fuse.
> 
> For getting this to work on fuse servers with strictlimiting, I think
> the next steps are to
> a) as Jan had suggested in [1], come up with some heuristic to
> constrain the max order supported for large folios for these fuse
> servers if they're running with the writeback cache enabled
> b) benchmark ^ and if there are still regressions, then we should
> probably just turn large folios off for these servers
> c) add the granular writeback/dirty accounting for large folios
> d) look into improving the balance_dirty_pages() throttling logic to
> handle narrow gaps between the freerun and full throttling zones
> better and reduce over-throttling
> 
> Does this sound like a reasonable way forward?

Sounds good to me, since we have seen pretty much the same when we enabled
large folios for testing.

> 
> For privileged servers, I still think it makes sense to remove the
> strictlimiting requirement or at the least, let admins opt out of that
> if they are confident their server is well-behaved.
> 

Here I'm not really sure what the most logical and sane way would be.
I really don't like limits for no reason but I understand the necessity
to have limits enabled for unpriviledged servers.

Do you think a module parameter is the right way to go here?
The connection parameter might be a problem since an admin would have
to set it for a large number of mounts.

Horst

> 
> Thanks,
> Joanne
> 
> [1] https://lore.kernel.org/linux-fsdevel/tglgxjxcs3wpm4msgxlvzk3hebzcguhuu752hs3eefku6wj4zv@2ixuho7rxbah/
> [2] https://lore.kernel.org/linux-fsdevel/f9b63a41-ced7-4176-8f40-6cba8fce7a4c@linux.alibaba.com/
> [3] https://lore.kernel.org/linux-fsdevel/20250829233942.3607248-1-joannelkoong@gmail.com/
> 
> >
> > I think the question is whether we want to let admins opt out of
> > strictlimit when they're confident their server is well-behaved eg
> > through a sysctl an admin can set to disable strictlimiting for all
> > servers. Otherwise, large folios will always have to be off for any
> > server that runs with writeback caching.
> >
> > Thanks,
> > Joanne
> > >
> > > Thanks,
> > > Miklos
> 

Re: Re: [PATCH] fuse: disable default bdi strictlimiting

[Amir Goldstein]

On Wed, May 27, 2026 at 8:06 AM Horst Birthelmer <horst@birthelmer.de> wrote:
>
> On Tue, May 26, 2026 at 06:42:35PM -0700, Joanne Koong wrote:
> > On Tue, May 12, 2026 at 1:56 PM Joanne Koong <joannelkoong@gmail.com> wrote:
> > >
> > > On Fri, May 8, 2026 at 2:42 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
> > > >
> > > > On Mon, 27 Oct 2025 at 23:39, Joanne Koong <joannelkoong@gmail.com> wrote:
> > > > > Miklos, could you share your thoughts on this? Are you in favor of
> > > > > disabling default strictlimiting? Or do you prefer to have it kept
> > > > > enabled by default, with some mount option or sysctl added for
> > > > > privileged servers to be able to disable strictlimiting + enable large
> > > > > folios if they use the writeback cache?
> > > >
> > > > So what I think we should do is implement some sort of slow writer
> > > > test, and see what happens with and without strictlimit.
> > > >
> > > > Tried to ask claude to do this for me, but not getting very far.
> > > >
> > > > So if I take this maintainership role seriously and not let myself
> > > > drown in the details, then the logical thing to do is to delegate ;)
> > > > Which is hard (for me at least) but I'll give it a try...
> > > >
> > > > Could you please check how things change if there's limited writeback
> > > > rate and we disable strictlimit?  And what happens if there are
> > > > several such instances running in parallel?
> > >
> > > I think for unprivileged fuse servers, strictlimting will always need
> > > to be enabled or else a malicious user can launch tons of unprivileged
> > > servers and eat up the global dirty page budget / starve writeback for
> > > the rest of the system. Similarly for privileged servers, it could be
> > > unintentionally slow or buggy and eat up the dirty page budget. I'll
> > > read through the writeback throttling code to verify this and run some
> > > local tests.
> >
> > I read through the writeback throttling code and re-read Jan's very
> > helpful comments from this thread last year [1]. So for unprivileged
> > servers, I think we definitely cannot remove strictlimiting. If the
> > fuse server is slow or unresponsive with writing back the pages, it
> > will take up too much of the global dirty budget which will degrade
> > write throughput for other filesystems (their throttling will be
> > computed against the global dirty page count, eg the freerunning check
> > in balance_dirty_pages() and the pos_ratio calculation "pos_ratio =
> > pos_ratio_polynom(setpoint, dtc->dirty, limit)" (dtc->dirty is the
> > global dirty page count)) and any fuse stuck dirty pages are
> > essentially unreclaimable. Without strictlimiting, there will be no
> > hard cap on how many dirty pages a misbehaving server can accumulate.
> >
> > With strictlimiting on and large folios enabled, the problem is that
> > the large folio size can potentially dwarf the server's dirty budget,
> > which can lead to excessive throttling. When I ran my benchmarks last
> > year, I and independently Jingbo saw severe performance regressions
> > for buffered writes with large folios (eg 2 GB/s BW w/o, and 200 MB/s
> > BW w/) [2] but I think that might have been because the machines had
> > limited RAM, resulting in a very small dirty budget. Fuse sets the max
> > ratio of the bdi to 1% of the global dirty threshold, so running
> > through some napkin math:
> >
> > On a 64 GB machine:
> >   - DirtyThresh = 20% of 64 GB = 12.8 GB
> >   - BdiDirtyThresh = 12.8 GB / 100 = 128 MB
> >   - 128 MB / 2 MB folio = 64 dirty folios
> >
> > On a 32 GB machine:
> >   - BdiDirtyThresh = 64 MB
> >   - 32 dirty folios
> >
> > On an 8 GB machine:
> >   - BdiDirtyThresh = 16 MB
> >   - 8 dirty folios
> >
> > On a 8GB machine (with assuming vm.dirty_ratio=20% and
> > vm.dirty_background_ratio=10%), we get 12 MB of freerun, 4 MB of
> > proportional throttling, and then full throtttling starts at 16 MB.
> > With 2MB folios, the 4MB zone between freerun and f ull throttling
> > doesn't leave that much room for the balance_dirty_pages() logic to
> > adjust the dirtier's speed, which I think causes the writes to
> > oscillate between freerunning and then being fully (overly) throttled.
> >
> > I think this is also going to be a problem for cgroups with large
> > folios since they also, as I understand it, are constrained with a
> > limited / tight dirty budget. I ran some initial benchmarks with
> > cgroup memory constraints on NVMe and saw similar instability (a
> > single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> > 15 ms without the cgroup, with the balance_dirty_pages() throttling
> > oscillating rather than settling near the set point).
> >
> > I think this problem gets untenable for random writes with large
> > folios, since dirtying just a few bytes will charge the whole folio
> > size to the dirty budget. I have a patchset from last year for adding
> > more granular dirty/writeback tracking [3], I'm going to pick this
> > series back up. I think it will be useful generically, not just for
> > fuse.
> >
> > For getting this to work on fuse servers with strictlimiting, I think
> > the next steps are to
> > a) as Jan had suggested in [1], come up with some heuristic to
> > constrain the max order supported for large folios for these fuse
> > servers if they're running with the writeback cache enabled
> > b) benchmark ^ and if there are still regressions, then we should
> > probably just turn large folios off for these servers
> > c) add the granular writeback/dirty accounting for large folios
> > d) look into improving the balance_dirty_pages() throttling logic to
> > handle narrow gaps between the freerun and full throttling zones
> > better and reduce over-throttling
> >
> > Does this sound like a reasonable way forward?
>
> Sounds good to me, since we have seen pretty much the same when we enabled
> large folios for testing.
>
> >
> > For privileged servers, I still think it makes sense to remove the
> > strictlimiting requirement or at the least, let admins opt out of that
> > if they are confident their server is well-behaved.
> >
>
> Here I'm not really sure what the most logical and sane way would be.
> I really don't like limits for no reason but I understand the necessity
> to have limits enabled for unpriviledged servers.
>
> Do you think a module parameter is the right way to go here?
> The connection parameter might be a problem since an admin would have
> to set it for a large number of mounts.
>

Not saying no to a module option, but I am curious.
This admin opt-in per mount could easily be implemented in userspace
in either libfuse and/or mount helper.

For such a niche ("I know what I am doing") feature, why does it matter
to do this in the kernel module params?

Thanks,
Amir.

Re: Re: [PATCH] fuse: disable default bdi strictlimiting

[Joanne Koong]

On Wed, May 27, 2026 at 3:59 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Wed, May 27, 2026 at 8:06 AM Horst Birthelmer <horst@birthelmer.de> wrote:
> >
> > On Tue, May 26, 2026 at 06:42:35PM -0700, Joanne Koong wrote:
> > >
> > > For getting this to work on fuse servers with strictlimiting, I think
> > > the next steps are to
> > > a) as Jan had suggested in [1], come up with some heuristic to
> > > constrain the max order supported for large folios for these fuse
> > > servers if they're running with the writeback cache enabled
> > > b) benchmark ^ and if there are still regressions, then we should
> > > probably just turn large folios off for these servers
> > > c) add the granular writeback/dirty accounting for large folios
> > > d) look into improving the balance_dirty_pages() throttling logic to
> > > handle narrow gaps between the freerun and full throttling zones
> > > better and reduce over-throttling
> > >
> > > Does this sound like a reasonable way forward?
> >
> > Sounds good to me, since we have seen pretty much the same when we enabled
> > large folios for testing.

Awesome, glad to hear we've been seeing the same things.

> >
> > >
> > > For privileged servers, I still think it makes sense to remove the
> > > strictlimiting requirement or at the least, let admins opt out of that
> > > if they are confident their server is well-behaved.
> > >
> >
> > Here I'm not really sure what the most logical and sane way would be.
> > I really don't like limits for no reason but I understand the necessity
> > to have limits enabled for unpriviledged servers.
> >
> > Do you think a module parameter is the right way to go here?
> > The connection parameter might be a problem since an admin would have
> > to set it for a large number of mounts.
> >
>
> Not saying no to a module option, but I am curious.
> This admin opt-in per mount could easily be implemented in userspace
> in either libfuse and/or mount helper.
>
> For such a niche ("I know what I am doing") feature, why does it matter
> to do this in the kernel module params?
>

I'm hoping we're able to keep things as simple as possible by just
getting rid of strictlimiting by default for privileged servers on the
kernel side, without needing any module parameters or libfuse changes,
but I'm not sure if Miklos is comfortable with that. imo if the server
is privileged, it can be trusted to not misbehave and should be given
the same kind of capabilities as a local filesystem. But I guess there
might be the possibility of user regressions if it turns out people's
privileged fuse servers are accidentally buggy.

I think the module parameter makes sense if admins trust unprivileged
servers enough to turn off strictlimiting globally. Otherwise, I think
Miklos just mentioned there's already a /sys/class/bdi interface that
lets admins disable strictlimit on a per-mount basis, so I think it
would be cleaner (though more annoying on the userspace side :)) if
libfuse used this to turn it off by default. The max supported folio
order is set per-inode when the inode is first looked up, not at mount
time, so the admin can disable strictlimit via sysfs after mount
(which is when the bdi sysfs entry gets created) and new inodes will
pick up large folio support from there.

Thanks,
Joanne

> Thanks,
> Amir.

Re: [PATCH] fuse: disable default bdi strictlimiting

[Miklos Szeredi]

On Wed, 27 May 2026 at 03:42, Joanne Koong <joannelkoong@gmail.com> wrote:

> On a 8GB machine (with assuming vm.dirty_ratio=20% and
> vm.dirty_background_ratio=10%), we get 12 MB of freerun, 4 MB of
> proportional throttling, and then full throtttling starts at 16 MB.
> With 2MB folios, the 4MB zone between freerun and f ull throttling
> doesn't leave that much room for the balance_dirty_pages() logic to
> adjust the dirtier's speed, which I think causes the writes to
> oscillate between freerunning and then being fully (overly) throttled.

Some handwaving: would it be possible for the folio alloc logic to
modulate the folio size based on how close are we to the dirty
threshold?

E.g. if the number of folios to reach the dirty limit is less than 10,
then start allocating smaller folios.

> I think this problem gets untenable for random writes with large
> folios, since dirtying just a few bytes will charge the whole folio
> size to the dirty budget. I have a patchset from last year for adding
> more granular dirty/writeback tracking [3], I'm going to pick this
> series back up. I think it will be useful generically, not just for
> fuse.

Tracking partial dirty is not going to help free up those large
folios, unless the dirty part is migrated to a smaller folio.  Not
sure if it's worth the added complexity, maybe...

> For getting this to work on fuse servers with strictlimiting, I think
> the next steps are to
> a) as Jan had suggested in [1], come up with some heuristic to
> constrain the max order supported for large folios for these fuse
> servers if they're running with the writeback cache enabled
> b) benchmark ^ and if there are still regressions, then we should
> probably just turn large folios off for these servers

Definitely worth exploring the characteristics of the current system.

> c) add the granular writeback/dirty accounting for large folios
> d) look into improving the balance_dirty_pages() throttling logic to
> handle narrow gaps between the freerun and full throttling zones
> better and reduce over-throttling
>
> Does this sound like a reasonable way forward?

Yeah, makes sense.

> For privileged servers, I still think it makes sense to remove the
> strictlimiting requirement or at the least, let admins opt out of that
> if they are confident their server is well-behaved.

This is already possible via the /sys/class/bdi interface.

I'm fine with introducing a config option / module option to turn it
off by default.

Thanks,
Miklos

Re: [PATCH] fuse: disable default bdi strictlimiting

[Joanne Koong]

On Wed, May 27, 2026 at 5:25 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
>
> On Wed, 27 May 2026 at 03:42, Joanne Koong <joannelkoong@gmail.com> wrote:
>
> > On a 8GB machine (with assuming vm.dirty_ratio=20% and
> > vm.dirty_background_ratio=10%), we get 12 MB of freerun, 4 MB of
> > proportional throttling, and then full throtttling starts at 16 MB.
> > With 2MB folios, the 4MB zone between freerun and f ull throttling
> > doesn't leave that much room for the balance_dirty_pages() logic to
> > adjust the dirtier's speed, which I think causes the writes to
> > oscillate between freerunning and then being fully (overly) throttled.
>
> Some handwaving: would it be possible for the folio alloc logic to
> modulate the folio size based on how close are we to the dirty
> threshold?
>
> E.g. if the number of folios to reach the dirty limit is less than 10,
> then start allocating smaller folios.

I don't think this would help for the case where the writes are to
existing large folios already resident in the page cache, but I can do
some experimentation with this.

>
> > I think this problem gets untenable for random writes with large
> > folios, since dirtying just a few bytes will charge the whole folio
> > size to the dirty budget. I have a patchset from last year for adding
> > more granular dirty/writeback tracking [3], I'm going to pick this
> > series back up. I think it will be useful generically, not just for
> > fuse.
>
> Tracking partial dirty is not going to help free up those large
> folios, unless the dirty part is migrated to a smaller folio.  Not
> sure if it's worth the added complexity, maybe...

That's a good point... the whole point of throttling is to cap the
number of pages pinned in RAM...

>
> > For getting this to work on fuse servers with strictlimiting, I think
> > the next steps are to
> > a) as Jan had suggested in [1], come up with some heuristic to
> > constrain the max order supported for large folios for these fuse
> > servers if they're running with the writeback cache enabled
> > b) benchmark ^ and if there are still regressions, then we should
> > probably just turn large folios off for these servers
>
> Definitely worth exploring the characteristics of the current system.
>
> > c) add the granular writeback/dirty accounting for large folios
> > d) look into improving the balance_dirty_pages() throttling logic to
> > handle narrow gaps between the freerun and full throttling zones
> > better and reduce over-throttling
> >
> > Does this sound like a reasonable way forward?
>
> Yeah, makes sense.
>
> > For privileged servers, I still think it makes sense to remove the
> > strictlimiting requirement or at the least, let admins opt out of that
> > if they are confident their server is well-behaved.
>
> This is already possible via the /sys/class/bdi interface.

Nice, I think in that case we can just have libfuse disable this on
behalf of the privileged server.

Thanks,
Joanne

>
> I'm fine with introducing a config option / module option to turn it
> off by default.
>
> Thanks,
> Miklos

Re: [PATCH] fuse: disable default bdi strictlimiting

[Jan Kara]

On Tue 26-05-26 18:42:35, Joanne Koong wrote:
> On Tue, May 12, 2026 at 1:56 PM Joanne Koong <joannelkoong@gmail.com> wrote:
> >
> > On Fri, May 8, 2026 at 2:42 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
> > >
> > > On Mon, 27 Oct 2025 at 23:39, Joanne Koong <joannelkoong@gmail.com> wrote:
> > > > Miklos, could you share your thoughts on this? Are you in favor of
> > > > disabling default strictlimiting? Or do you prefer to have it kept
> > > > enabled by default, with some mount option or sysctl added for
> > > > privileged servers to be able to disable strictlimiting + enable large
> > > > folios if they use the writeback cache?
> > >
> > > So what I think we should do is implement some sort of slow writer
> > > test, and see what happens with and without strictlimit.
> > >
> > > Tried to ask claude to do this for me, but not getting very far.
> > >
> > > So if I take this maintainership role seriously and not let myself
> > > drown in the details, then the logical thing to do is to delegate ;)
> > > Which is hard (for me at least) but I'll give it a try...
> > >
> > > Could you please check how things change if there's limited writeback
> > > rate and we disable strictlimit?  And what happens if there are
> > > several such instances running in parallel?
> >
> > I think for unprivileged fuse servers, strictlimting will always need
> > to be enabled or else a malicious user can launch tons of unprivileged
> > servers and eat up the global dirty page budget / starve writeback for
> > the rest of the system. Similarly for privileged servers, it could be
> > unintentionally slow or buggy and eat up the dirty page budget. I'll
> > read through the writeback throttling code to verify this and run some
> > local tests.
> 
> I read through the writeback throttling code and re-read Jan's very
> helpful comments from this thread last year [1]. So for unprivileged
> servers, I think we definitely cannot remove strictlimiting. If the
> fuse server is slow or unresponsive with writing back the pages, it
> will take up too much of the global dirty budget which will degrade
> write throughput for other filesystems (their throttling will be
> computed against the global dirty page count, eg the freerunning check
> in balance_dirty_pages() and the pos_ratio calculation "pos_ratio =
> pos_ratio_polynom(setpoint, dtc->dirty, limit)" (dtc->dirty is the
> global dirty page count)) and any fuse stuck dirty pages are
> essentially unreclaimable. Without strictlimiting, there will be no
> hard cap on how many dirty pages a misbehaving server can accumulate.

Yes.

> With strictlimiting on and large folios enabled, the problem is that
> the large folio size can potentially dwarf the server's dirty budget,
> which can lead to excessive throttling. When I ran my benchmarks last
> year, I and independently Jingbo saw severe performance regressions
> for buffered writes with large folios (eg 2 GB/s BW w/o, and 200 MB/s
> BW w/) [2] but I think that might have been because the machines had
> limited RAM, resulting in a very small dirty budget. Fuse sets the max
> ratio of the bdi to 1% of the global dirty threshold, so running
> through some napkin math:
> 
> On a 64 GB machine:
>   - DirtyThresh = 20% of 64 GB = 12.8 GB
>   - BdiDirtyThresh = 12.8 GB / 100 = 128 MB
>   - 128 MB / 2 MB folio = 64 dirty folios
> 
> On a 32 GB machine:
>   - BdiDirtyThresh = 64 MB
>   - 32 dirty folios
> 
> On an 8 GB machine:
>   - BdiDirtyThresh = 16 MB
>   - 8 dirty folios
> 
> On a 8GB machine (with assuming vm.dirty_ratio=20% and
> vm.dirty_background_ratio=10%), we get 12 MB of freerun, 4 MB of
> proportional throttling, and then full throtttling starts at 16 MB.
> With 2MB folios, the 4MB zone between freerun and f ull throttling
> doesn't leave that much room for the balance_dirty_pages() logic to
> adjust the dirtier's speed, which I think causes the writes to
> oscillate between freerunning and then being fully (overly) throttled.

Thanks for spelling this out. Even with 64 dirty folios effective limit I
think the throttling code will be too unstable. In general I think we
should aim at having at least ~512 (gut feeling estimate ;) folios in a
dirty limit to have a decent wiggle room for the throttling algorithm.

> I think this is also going to be a problem for cgroups with large
> folios since they also, as I understand it, are constrained with a
> limited / tight dirty budget. I ran some initial benchmarks with
> cgroup memory constraints on NVMe and saw similar instability (a
> single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> 15 ms without the cgroup, with the balance_dirty_pages() throttling
> oscillating rather than settling near the set point).

Ok, so this is more folios (819) than my 512 gut feeling estimate :) What
was the write throughput of the NVMe drive? The high drive throughput also
requires more dirty data to keep the drive saturated so that writeback
throughput doesn't oscilate too much.

> I think this problem gets untenable for random writes with large
> folios, since dirtying just a few bytes will charge the whole folio
> size to the dirty budget. I have a patchset from last year for adding
> more granular dirty/writeback tracking [3], I'm going to pick this
> series back up. I think it will be useful generically, not just for
> fuse.
> 
> For getting this to work on fuse servers with strictlimiting, I think
> the next steps are to
> a) as Jan had suggested in [1], come up with some heuristic to
> constrain the max order supported for large folios for these fuse
> servers if they're running with the writeback cache enabled
> b) benchmark ^ and if there are still regressions, then we should
> probably just turn large folios off for these servers

Yes, as we briefly discussed at LSF, picking proper max folio order is
useful not only for dirty throttling but also for memory reclaim in
general in memory-constrained environments. Strictlimit is just one of the
cases where the limit is very small and so the problems with the memory
overhead of larger folios become very apparent.

So although we may decide for a FUSE specific heuristic setting the maximum
folio order, you properly point out memory limited memcgs will see similar
issues so I think a more general solution for selecting sensible max order
would be better.

> c) add the granular writeback/dirty accounting for large folios

As Miklos pointed out and I think we've discussed that also last year since
the full folio is pinned in memory by a single dirty fs block (or whatever
is going to be your unit), finer grained accounting alone doesn't help you
with the narrow limits. We'd also have to add logic to decide about
splitting large folios.

> d) look into improving the balance_dirty_pages() throttling logic to
> handle narrow gaps between the freerun and full throttling zones
> better and reduce over-throttling

Sure, this is always a possibility but this is easier said than done :) If
you get very coarse-grained signals, there's only so much you can do to
smooth out the behavior for the application. But the erratic behavior with
8GB memcg is certainly something I'd expect to work reasonably smoothly so
starting with investigating that case might be a good start.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [PATCH] fuse: disable default bdi strictlimiting

[Joanne Koong]

On Thu, May 28, 2026 at 5:34 AM Jan Kara <jack@suse.cz> wrote:
>
> On Tue 26-05-26 18:42:35, Joanne Koong wrote:
> > On Tue, May 12, 2026 at 1:56 PM Joanne Koong <joannelkoong@gmail.com> wrote:
> > >
> > > On Fri, May 8, 2026 at 2:42 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
> > > >
> > > > On Mon, 27 Oct 2025 at 23:39, Joanne Koong <joannelkoong@gmail.com> wrote:
> > > > > Miklos, could you share your thoughts on this? Are you in favor of
> > > > > disabling default strictlimiting? Or do you prefer to have it kept
> > > > > enabled by default, with some mount option or sysctl added for
> > > > > privileged servers to be able to disable strictlimiting + enable large
> > > > > folios if they use the writeback cache?
> > > >
> > > > So what I think we should do is implement some sort of slow writer
> > > > test, and see what happens with and without strictlimit.
> > > >
> > > > Tried to ask claude to do this for me, but not getting very far.
> > > >
> > > > So if I take this maintainership role seriously and not let myself
> > > > drown in the details, then the logical thing to do is to delegate ;)
> > > > Which is hard (for me at least) but I'll give it a try...
> > > >
> > > > Could you please check how things change if there's limited writeback
> > > > rate and we disable strictlimit?  And what happens if there are
> > > > several such instances running in parallel?
> > >
> > > I think for unprivileged fuse servers, strictlimting will always need
> > > to be enabled or else a malicious user can launch tons of unprivileged
> > > servers and eat up the global dirty page budget / starve writeback for
> > > the rest of the system. Similarly for privileged servers, it could be
> > > unintentionally slow or buggy and eat up the dirty page budget. I'll
> > > read through the writeback throttling code to verify this and run some
> > > local tests.
> >
> > I read through the writeback throttling code and re-read Jan's very
> > helpful comments from this thread last year [1]. So for unprivileged
> > servers, I think we definitely cannot remove strictlimiting. If the
> > fuse server is slow or unresponsive with writing back the pages, it
> > will take up too much of the global dirty budget which will degrade
> > write throughput for other filesystems (their throttling will be
> > computed against the global dirty page count, eg the freerunning check
> > in balance_dirty_pages() and the pos_ratio calculation "pos_ratio =
> > pos_ratio_polynom(setpoint, dtc->dirty, limit)" (dtc->dirty is the
> > global dirty page count)) and any fuse stuck dirty pages are
> > essentially unreclaimable. Without strictlimiting, there will be no
> > hard cap on how many dirty pages a misbehaving server can accumulate.
>
> Yes.
>
> > With strictlimiting on and large folios enabled, the problem is that
> > the large folio size can potentially dwarf the server's dirty budget,
> > which can lead to excessive throttling. When I ran my benchmarks last
> > year, I and independently Jingbo saw severe performance regressions
> > for buffered writes with large folios (eg 2 GB/s BW w/o, and 200 MB/s
> > BW w/) [2] but I think that might have been because the machines had
> > limited RAM, resulting in a very small dirty budget. Fuse sets the max
> > ratio of the bdi to 1% of the global dirty threshold, so running
> > through some napkin math:
> >
> > On a 64 GB machine:
> >   - DirtyThresh = 20% of 64 GB = 12.8 GB
> >   - BdiDirtyThresh = 12.8 GB / 100 = 128 MB
> >   - 128 MB / 2 MB folio = 64 dirty folios
> >
> > On a 32 GB machine:
> >   - BdiDirtyThresh = 64 MB
> >   - 32 dirty folios
> >
> > On an 8 GB machine:
> >   - BdiDirtyThresh = 16 MB
> >   - 8 dirty folios
> >
> > On a 8GB machine (with assuming vm.dirty_ratio=20% and
> > vm.dirty_background_ratio=10%), we get 12 MB of freerun, 4 MB of
> > proportional throttling, and then full throtttling starts at 16 MB.
> > With 2MB folios, the 4MB zone between freerun and f ull throttling
> > doesn't leave that much room for the balance_dirty_pages() logic to
> > adjust the dirtier's speed, which I think causes the writes to
> > oscillate between freerunning and then being fully (overly) throttled.
>
> Thanks for spelling this out. Even with 64 dirty folios effective limit I
> think the throttling code will be too unstable. In general I think we
> should aim at having at least ~512 (gut feeling estimate ;) folios in a
> dirty limit to have a decent wiggle room for the throttling algorithm.

Awesome, I'll do some benchmarking on this and gather some data.

>
> > I think this is also going to be a problem for cgroups with large
> > folios since they also, as I understand it, are constrained with a
> > limited / tight dirty budget. I ran some initial benchmarks with
> > cgroup memory constraints on NVMe and saw similar instability (a
> > single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> > 15 ms without the cgroup, with the balance_dirty_pages() throttling
> > oscillating rather than settling near the set point).
>
> Ok, so this is more folios (819) than my 512 gut feeling estimate :) What
> was the write throughput of the NVMe drive? The high drive throughput also
> requires more dirty data to keep the drive saturated so that writeback
> throughput doesn't oscilate too much.

The write throughput of the NVMe drive I was using was around ~1.1
Gb/s (measured by running direct I/O). I think with the 1.6GB dirty
budget, the math for that ends up being that the device drains it in
~1.5 secs. The performance I was seeing with the initial cgroup
benchmarks was with 4k pages (no large folios enabled) on btrfs.

>
> > I think this problem gets untenable for random writes with large
> > folios, since dirtying just a few bytes will charge the whole folio
> > size to the dirty budget. I have a patchset from last year for adding
> > more granular dirty/writeback tracking [3], I'm going to pick this
> > series back up. I think it will be useful generically, not just for
> > fuse.
> >
> > For getting this to work on fuse servers with strictlimiting, I think
> > the next steps are to
> > a) as Jan had suggested in [1], come up with some heuristic to
> > constrain the max order supported for large folios for these fuse
> > servers if they're running with the writeback cache enabled
> > b) benchmark ^ and if there are still regressions, then we should
> > probably just turn large folios off for these servers
>
> Yes, as we briefly discussed at LSF, picking proper max folio order is
> useful not only for dirty throttling but also for memory reclaim in
> general in memory-constrained environments. Strictlimit is just one of the
> cases where the limit is very small and so the problems with the memory
> overhead of larger folios become very apparent.
>
> So although we may decide for a FUSE specific heuristic setting the maximum
> folio order, you properly point out memory limited memcgs will see similar
> issues so I think a more general solution for selecting sensible max order
> would be better.

This sounds great. I'll run some more benchmarks to get more data and
then come up with a general solution and submit that as an rfc.

>
> > c) add the granular writeback/dirty accounting for large folios
>
> As Miklos pointed out and I think we've discussed that also last year since
> the full folio is pinned in memory by a single dirty fs block (or whatever
> is going to be your unit), finer grained accounting alone doesn't help you
> with the narrow limits. We'd also have to add logic to decide about
> splitting large folios.

If I'm inferring this right (and please let me know if I"m not), it
seems like it would be extremely useful to get the infrastructure
working better for splitting file-backed large folios. That seems like
the proper long-term solution if I'm understanding it correctly (eg
instead of capping the folio order statically across the board, larger
folios can be used until there's memory pressure that splits it or if
at write time we detect the dirty budget is tight, that will allow the
clean parts to be reclaimable and reduce memory pressure and
throttling)?

From what I see in __folio_split() in mm/huge_memory.c, it's not
currently possible to split dirty file-backed folios (eg
iomap_release_folio() will not release the folio if any part of it is
dirty), but I think we can get this working in the iomap layer for
supporting the per-folio dirty/uptodate state tracking for the split
folio instead of rejecting the release. Iomap already tracks which
subfolios are dirty vs not, so we have the information upfront as well
about how to split the dirty folio optimally. Do you think this would
be useful to get working?

Thanks,
Joanne

Re: [PATCH] fuse: disable default bdi strictlimiting

[Jan Kara]

On Thu 28-05-26 15:11:18, Joanne Koong wrote:
> On Thu, May 28, 2026 at 5:34 AM Jan Kara <jack@suse.cz> wrote:
> > > I think this is also going to be a problem for cgroups with large
> > > folios since they also, as I understand it, are constrained with a
> > > limited / tight dirty budget. I ran some initial benchmarks with
> > > cgroup memory constraints on NVMe and saw similar instability (a
> > > single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> > > 15 ms without the cgroup, with the balance_dirty_pages() throttling
> > > oscillating rather than settling near the set point).
> >
> > Ok, so this is more folios (819) than my 512 gut feeling estimate :) What
> > was the write throughput of the NVMe drive? The high drive throughput also
> > requires more dirty data to keep the drive saturated so that writeback
> > throughput doesn't oscilate too much.
> 
> The write throughput of the NVMe drive I was using was around ~1.1
> Gb/s (measured by running direct I/O). I think with the 1.6GB dirty
> budget, the math for that ends up being that the device drains it in
> ~1.5 secs. The performance I was seeing with the initial cgroup
> benchmarks was with 4k pages (no large folios enabled) on btrfs.

OK, you might want to experiment with some other filesystem (I suggest xfs
or ext4) as well. Btrfs writeback behavior is a bit special with its data
checksum computations etc. and thus latency of starting writeback. It could
contribute to the erratic behavior with the relatively low dirty limits.

> > > I think this problem gets untenable for random writes with large
> > > folios, since dirtying just a few bytes will charge the whole folio
> > > size to the dirty budget. I have a patchset from last year for adding
> > > more granular dirty/writeback tracking [3], I'm going to pick this
> > > series back up. I think it will be useful generically, not just for
> > > fuse.
> > >
> > > For getting this to work on fuse servers with strictlimiting, I think
> > > the next steps are to
> > > a) as Jan had suggested in [1], come up with some heuristic to
> > > constrain the max order supported for large folios for these fuse
> > > servers if they're running with the writeback cache enabled
> > > b) benchmark ^ and if there are still regressions, then we should
> > > probably just turn large folios off for these servers
> >
> > Yes, as we briefly discussed at LSF, picking proper max folio order is
> > useful not only for dirty throttling but also for memory reclaim in
> > general in memory-constrained environments. Strictlimit is just one of the
> > cases where the limit is very small and so the problems with the memory
> > overhead of larger folios become very apparent.
> >
> > So although we may decide for a FUSE specific heuristic setting the maximum
> > folio order, you properly point out memory limited memcgs will see similar
> > issues so I think a more general solution for selecting sensible max order
> > would be better.
> 
> This sounds great. I'll run some more benchmarks to get more data and
> then come up with a general solution and submit that as an rfc.
> 
> > > c) add the granular writeback/dirty accounting for large folios
> >
> > As Miklos pointed out and I think we've discussed that also last year since
> > the full folio is pinned in memory by a single dirty fs block (or whatever
> > is going to be your unit), finer grained accounting alone doesn't help you
> > with the narrow limits. We'd also have to add logic to decide about
> > splitting large folios.
> 
> If I'm inferring this right (and please let me know if I"m not), it
> seems like it would be extremely useful to get the infrastructure
> working better for splitting file-backed large folios. That seems like
> the proper long-term solution if I'm understanding it correctly (eg
> instead of capping the folio order statically across the board, larger
> folios can be used until there's memory pressure that splits it or if
> at write time we detect the dirty budget is tight, that will allow the
> clean parts to be reclaimable and reduce memory pressure and
> throttling)?
> 
> From what I see in __folio_split() in mm/huge_memory.c, it's not
> currently possible to split dirty file-backed folios (eg
> iomap_release_folio() will not release the folio if any part of it is
> dirty), but I think we can get this working in the iomap layer for
> supporting the per-folio dirty/uptodate state tracking for the split
> folio instead of rejecting the release. Iomap already tracks which
> subfolios are dirty vs not, so we have the information upfront as well
> about how to split the dirty folio optimally. Do you think this would
> be useful to get working?

I don't think people have reached concensus on what would be the best way
to address the issues with coarse granularity of large folios (even within
myself I haven't quite decided about the best solution :)). So we are just
experimenting and seeing what could work and what not. Limiting maximum
folio order is kind of a dumb solution but it is relatively easy to
implement and might be good enough.

Splitting of folios is definitely more complicated, requires support from
filesystems, can fail for various reasons (folio is pinned, you need to
allocate memory to split a folio, ...), and is generally more expensive.
So tuning that properly and making it work reliably under various
conditions is an order of magnitude harder task and I think at this point
you'd have a hard time justifying all the complexity without showing why
the dumb solution isn't good enough... So we might get to splitting folios
at some point in some shape or form I don't think we're there yet.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [PATCH] fuse: disable default bdi strictlimiting

[Joanne Koong]

On Sat, May 30, 2026 at 4:04 AM Jan Kara <jack@suse.cz> wrote:
>
> On Thu 28-05-26 15:11:18, Joanne Koong wrote:
> > On Thu, May 28, 2026 at 5:34 AM Jan Kara <jack@suse.cz> wrote:
> > > > I think this is also going to be a problem for cgroups with large
> > > > folios since they also, as I understand it, are constrained with a
> > > > limited / tight dirty budget. I ran some initial benchmarks with
> > > > cgroup memory constraints on NVMe and saw similar instability (a
> > > > single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> > > > 15 ms without the cgroup, with the balance_dirty_pages() throttling
> > > > oscillating rather than settling near the set point).
> > >
> > > Ok, so this is more folios (819) than my 512 gut feeling estimate :) What
> > > was the write throughput of the NVMe drive? The high drive throughput also
> > > requires more dirty data to keep the drive saturated so that writeback
> > > throughput doesn't oscilate too much.
> >
> > The write throughput of the NVMe drive I was using was around ~1.1
> > Gb/s (measured by running direct I/O). I think with the 1.6GB dirty
> > budget, the math for that ends up being that the device drains it in
> > ~1.5 secs. The performance I was seeing with the initial cgroup
> > benchmarks was with 4k pages (no large folios enabled) on btrfs.
>
> OK, you might want to experiment with some other filesystem (I suggest xfs
> or ext4) as well. Btrfs writeback behavior is a bit special with its data
> checksum computations etc. and thus latency of starting writeback. It could
> contribute to the erratic behavior with the relatively low dirty limits.

I reran the benchmarks on xfs and ext4 and saw similiar results. The
max write latency under the memcg were
xfs        5.1s
ext4      5.2 s
btrfs      6.2 s

compared to the non-memcg case (0.4 to 2.0 ms). This didn't affect
throughput though, just the tail latency. When I ran xfs with large
folios disabled, I still saw ~4.1s.

From the balance_dirty_pages() ftrace tracepoints, the dirty_ratelimit
value is consistently stable / accurate but it looks like what's
happening is that during freerun, the writer essentially dirties at
memcpy speed until the freerun ceiling, and then the soft/proportional
throttling kicks in but doesn't kick in fast or hard enough, which
allows the number of dirty pages to exceed the hard limit by ~2x, and
then at that point the writer is forced into the loop where it sleeps
max_pause (200ms) each iteration until writeback has drained the
number of dirty pages back under the limit.

If we wanted to fix this, it seems like we could for
memory-constrained memcgs either start soft/proportional throttling
earlier or make it throttle more aggressively as dirty approaches the
limit, or start writeback sooner so it finishes draining the dirty
pages faster. When I experimented with lowering
dirty_background_ratio, the worst-case latency dropped around 2-3x
(~6s to 2s) but there was still multi-second tail latencies. Maybe
this indicates more that the throttling needs to happen earlier/harder
than that writeback needs to happen earlier.

I'm not sure how much this matters in practice though, given that no
one has complained about it.

Ironically, for fuse there is not this problem because with
strictlimiting, it makes the dirty budget so small that even with
exceeding the hard limit by ~2x, there's not that many bytes to write
back so there's not that much latency waiting for it to finish, and
strictlimiting also inherently starts the soft/proportional throttling
more early/aggressively, which prevents too much overrun exceeding the
limit.

Thanks,
Joanne

Re: [PATCH] fuse: disable default bdi strictlimiting

[Jan Kara]

On Mon 13-07-26 18:13:29, Joanne Koong wrote:
> On Sat, May 30, 2026 at 4:04 AM Jan Kara <jack@suse.cz> wrote:
> >
> > On Thu 28-05-26 15:11:18, Joanne Koong wrote:
> > > On Thu, May 28, 2026 at 5:34 AM Jan Kara <jack@suse.cz> wrote:
> > > > > I think this is also going to be a problem for cgroups with large
> > > > > folios since they also, as I understand it, are constrained with a
> > > > > limited / tight dirty budget. I ran some initial benchmarks with
> > > > > cgroup memory constraints on NVMe and saw similar instability (a
> > > > > single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> > > > > 15 ms without the cgroup, with the balance_dirty_pages() throttling
> > > > > oscillating rather than settling near the set point).
> > > >
> > > > Ok, so this is more folios (819) than my 512 gut feeling estimate :) What
> > > > was the write throughput of the NVMe drive? The high drive throughput also
> > > > requires more dirty data to keep the drive saturated so that writeback
> > > > throughput doesn't oscilate too much.
> > >
> > > The write throughput of the NVMe drive I was using was around ~1.1
> > > Gb/s (measured by running direct I/O). I think with the 1.6GB dirty
> > > budget, the math for that ends up being that the device drains it in
> > > ~1.5 secs. The performance I was seeing with the initial cgroup
> > > benchmarks was with 4k pages (no large folios enabled) on btrfs.
> >
> > OK, you might want to experiment with some other filesystem (I suggest xfs
> > or ext4) as well. Btrfs writeback behavior is a bit special with its data
> > checksum computations etc. and thus latency of starting writeback. It could
> > contribute to the erratic behavior with the relatively low dirty limits.
> 
> I reran the benchmarks on xfs and ext4 and saw similiar results. The
> max write latency under the memcg were
> xfs        5.1s
> ext4      5.2 s
> btrfs      6.2 s
> 
> compared to the non-memcg case (0.4 to 2.0 ms). This didn't affect
> throughput though, just the tail latency. When I ran xfs with large
> folios disabled, I still saw ~4.1s.

OK, interesting. Thanks for running these tests! Also it is good to know
this is not really related to large folios, that somewhat simplifies
matters.

> From the balance_dirty_pages() ftrace tracepoints, the dirty_ratelimit
> value is consistently stable / accurate but it looks like what's
> happening is that during freerun, the writer essentially dirties at
> memcpy speed until the freerun ceiling, and then the soft/proportional
> throttling kicks in but doesn't kick in fast or hard enough, which
> allows the number of dirty pages to exceed the hard limit by ~2x, and
> then at that point the writer is forced into the loop where it sleeps
> max_pause (200ms) each iteration until writeback has drained the
> number of dirty pages back under the limit.

OK, above you mentioned that the dirty limit for the memcgs is set at
1.6GB. Does that mean that dirty throttling allows memcg to dirty up to
~3.2GB of pages? I wouldn't have expected that...

Also you mentioned that the disk you are using has writeback throughput of
~1.1GB/s. Even with 3.2GB of pages we should get below the dirty limit in
2s and not 5. So there's something going on I don't understand yet :)

BTW how much memory does the machine have in total?

> If we wanted to fix this, it seems like we could for
> memory-constrained memcgs either start soft/proportional throttling
> earlier or make it throttle more aggressively as dirty approaches the
> limit, or start writeback sooner so it finishes draining the dirty
> pages faster. When I experimented with lowering
> dirty_background_ratio, the worst-case latency dropped around 2-3x
> (~6s to 2s) but there was still multi-second tail latencies. Maybe
> this indicates more that the throttling needs to happen earlier/harder
> than that writeback needs to happen earlier.

Hum, can you share the script you are using for the benchmark? I can try
running in on my machine and see what happens. If I'm able to reproduce
these long waits, I want to do a bit of tracing to understand what's
happening with various writeback statistics. Because these large tail
latencies aren't really expected. The original design goal was to keep
maximum write latency (for fast enough disks) around 0.2s.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [PATCH] fuse: disable default bdi strictlimiting

[Joanne Koong]

On Tue, Jul 14, 2026 at 10:10 AM Jan Kara <jack@suse.cz> wrote:
>
> On Mon 13-07-26 18:13:29, Joanne Koong wrote:
> > On Sat, May 30, 2026 at 4:04 AM Jan Kara <jack@suse.cz> wrote:
> > >
> > > On Thu 28-05-26 15:11:18, Joanne Koong wrote:
> > > > On Thu, May 28, 2026 at 5:34 AM Jan Kara <jack@suse.cz> wrote:
> > > > > > I think this is also going to be a problem for cgroups with large
> > > > > > folios since they also, as I understand it, are constrained with a
> > > > > > limited / tight dirty budget. I ran some initial benchmarks with
> > > > > > cgroup memory constraints on NVMe and saw similar instability (a
> > > > > > single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> > > > > > 15 ms without the cgroup, with the balance_dirty_pages() throttling
> > > > > > oscillating rather than settling near the set point).
> > > > >
> > > > > Ok, so this is more folios (819) than my 512 gut feeling estimate :) What
> > > > > was the write throughput of the NVMe drive? The high drive throughput also
> > > > > requires more dirty data to keep the drive saturated so that writeback
> > > > > throughput doesn't oscilate too much.
> > > >
> > > > The write throughput of the NVMe drive I was using was around ~1.1
> > > > Gb/s (measured by running direct I/O). I think with the 1.6GB dirty
> > > > budget, the math for that ends up being that the device drains it in
> > > > ~1.5 secs. The performance I was seeing with the initial cgroup
> > > > benchmarks was with 4k pages (no large folios enabled) on btrfs.
> > >
> > > OK, you might want to experiment with some other filesystem (I suggest xfs
> > > or ext4) as well. Btrfs writeback behavior is a bit special with its data
> > > checksum computations etc. and thus latency of starting writeback. It could
> > > contribute to the erratic behavior with the relatively low dirty limits.
> >
> > I reran the benchmarks on xfs and ext4 and saw similiar results. The
> > max write latency under the memcg were
> > xfs        5.1s
> > ext4      5.2 s
> > btrfs      6.2 s
> >
> > compared to the non-memcg case (0.4 to 2.0 ms). This didn't affect
> > throughput though, just the tail latency. When I ran xfs with large
> > folios disabled, I still saw ~4.1s.
>
> OK, interesting. Thanks for running these tests! Also it is good to know
> this is not really related to large folios, that somewhat simplifies
> matters.
>
> > From the balance_dirty_pages() ftrace tracepoints, the dirty_ratelimit
> > value is consistently stable / accurate but it looks like what's
> > happening is that during freerun, the writer essentially dirties at
> > memcpy speed until the freerun ceiling, and then the soft/proportional
> > throttling kicks in but doesn't kick in fast or hard enough, which
> > allows the number of dirty pages to exceed the hard limit by ~2x, and
> > then at that point the writer is forced into the loop where it sleeps
> > max_pause (200ms) each iteration until writeback has drained the
> > number of dirty pages back under the limit.
>
> OK, above you mentioned that the dirty limit for the memcgs is set at
> 1.6GB. Does that mean that dirty throttling allows memcg to dirty up to
> ~3.2GB of pages? I wouldn't have expected that...

On a couple of the runs, I saw it get as high as up to 5GB.

I think this is because the balance_dirty_pages() code uses the memcg
stats (NR_FILE_DIRTY) but these stats are only periodically
flushed/refreshed when reading it, so the balance dirty code is
seeing/using lagging/non-uptodate values.

I'm seeing this in
  balance_dirty_pages()
      balance_domain_limits(mdtc,...)
          domain_dirty_avail()
                mem_cgroup_wb_stats()
                       mem_cgroup_flush_stats_ratelimited()

where mem_cgroup_flush_stats_ratelimted() has this logic:

void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg)
{
        /* Only flush if the periodic flusher is one full cycle late */
        if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME))
                mem_cgroup_flush_stats(memcg);
}

where FLUSH_TIME is defined as 2UL * HZ, which afaict means the
flushing can be around 2 seconds stale (accounting for the periodic
flusher in flush_memcg_stats_dwork() that flushes every FLUSH_TIME
interval).

From what I see, the freerun and hard limit checks use this stale
under-reported value (in domain_dirty_freerun() and
wb_position_ratio()), which means the writer gets to keep dirtying and
blow past the limit for up to 2 extra seconds before the throttling
limits it.

Maybe one idea is to in the memcg throttle path do the flush based on
how many pages are getting dirtied instead of on how much time has
elapsed? It looks like flushing grabs the rstat lock though, so maybe
only doing it in the throttled case where it's past freerun would be
best. It looks like the logic in the regular non-ratelimited
mem_cgroup_flush_stats() call does something similar. I can run some
experiments this week with this if you think it could be promising.

>
> Also you mentioned that the disk you are using has writeback throughput of
> ~1.1GB/s. Even with 3.2GB of pages we should get below the dirty limit in
> 2s and not 5. So there's something going on I don't understand yet :)

I'm not sure how the 5 got there either :) I reran it a couple of
times today and saw some variance, with the average of the runs
landing closer to around ~4s for xfs. From the traces, the writeback
throughput looks fine, so I don't think it's that the device is
writing back too slowly. I saw a trace where dirty went from ~4.7GB to
1.3 GB over 2.42 seconds which is about ~1.4 GB/s, which matches the
device's direct-io speed I saw. I think it's more that the number of
dirty pages is exceeding the limit by more than 2x and then it writes
back even below the hard limit (eg it keeps writing back for the
entire 200 ms hard throttle pause)

>
> BTW how much memory does the machine have in total?

The machine has 250 GB in total.

>
> > If we wanted to fix this, it seems like we could for
> > memory-constrained memcgs either start soft/proportional throttling
> > earlier or make it throttle more aggressively as dirty approaches the
> > limit, or start writeback sooner so it finishes draining the dirty
> > pages faster. When I experimented with lowering
> > dirty_background_ratio, the worst-case latency dropped around 2-3x
> > (~6s to 2s) but there was still multi-second tail latencies. Maybe
> > this indicates more that the throttling needs to happen earlier/harder
> > than that writeback needs to happen earlier.
>
> Hum, can you share the script you are using for the benchmark? I can try
> running in on my machine and see what happens. If I'm able to reproduce
> these long waits, I want to do a bit of tracing to understand what's
> happening with various writeback statistics. Because these large tail
> latencies aren't really expected. The original design goal was to keep
> maximum write latency (for fast enough disks) around 0.2s.

This is how I am running the benchmarks:

DEV=/dev/nvme5n1
# get device bandwidth (for reference)
fio --name=raw --filename=$DEV --rw=write --bs=1M --direct=1 --size=4G
--runtime=15 --time_based
mkfs.xfs -f $DEV && mount $DEV /mnt

mkdir -p /sys/fs/cgroup/balancedirtytest
echo 8G > /sys/fs/cgroup/balancedirtytest/memory.max

cd /sys/kernel/tracing
echo 131072 > buffer_size_kb
echo 1 > events/writeback/balance_dirty_pages/enable
echo 1 > events/writeback/global_dirty_state/enable
echo 1 > events/memcg/memcg_flush_stats/enable
echo > trace
echo 1 > tracing_on

echo 3 > /proc/sys/vm/drop_caches
( echo $BASHPID > /sys/fs/cgroup/balancedirtytest/cgroup.procs
fio --name=w --directory=/mnt --rw=write --bs=1M --ioengine=psync
--numjobs=1 --size=16G --time_based --runtime=60 --end_fsync=1 )

echo 0 > tracing_on
cp trace /tmp/xfs-data.trace

Thanks,
Joanne

Re: [PATCH] fuse: disable default bdi strictlimiting

[Jan Kara]

On Tue 14-07-26 18:21:57, Joanne Koong wrote:
> On Tue, Jul 14, 2026 at 10:10 AM Jan Kara <jack@suse.cz> wrote:
> > On Mon 13-07-26 18:13:29, Joanne Koong wrote:
> > > On Sat, May 30, 2026 at 4:04 AM Jan Kara <jack@suse.cz> wrote:
> > > > On Thu 28-05-26 15:11:18, Joanne Koong wrote:
> > > > > On Thu, May 28, 2026 at 5:34 AM Jan Kara <jack@suse.cz> wrote:
> > > > > > > I think this is also going to be a problem for cgroups with large
> > > > > > > folios since they also, as I understand it, are constrained with a
> > > > > > > limited / tight dirty budget. I ran some initial benchmarks with
> > > > > > > cgroup memory constraints on NVMe and saw similar instability (a
> > > > > > > single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> > > > > > > 15 ms without the cgroup, with the balance_dirty_pages() throttling
> > > > > > > oscillating rather than settling near the set point).
> > > > > >
> > > > > > Ok, so this is more folios (819) than my 512 gut feeling estimate :) What
> > > > > > was the write throughput of the NVMe drive? The high drive throughput also
> > > > > > requires more dirty data to keep the drive saturated so that writeback
> > > > > > throughput doesn't oscilate too much.
> > > > >
> > > > > The write throughput of the NVMe drive I was using was around ~1.1
> > > > > Gb/s (measured by running direct I/O). I think with the 1.6GB dirty
> > > > > budget, the math for that ends up being that the device drains it in
> > > > > ~1.5 secs. The performance I was seeing with the initial cgroup
> > > > > benchmarks was with 4k pages (no large folios enabled) on btrfs.
> > > >
> > > > OK, you might want to experiment with some other filesystem (I suggest xfs
> > > > or ext4) as well. Btrfs writeback behavior is a bit special with its data
> > > > checksum computations etc. and thus latency of starting writeback. It could
> > > > contribute to the erratic behavior with the relatively low dirty limits.
> > >
> > > I reran the benchmarks on xfs and ext4 and saw similiar results. The
> > > max write latency under the memcg were
> > > xfs        5.1s
> > > ext4      5.2 s
> > > btrfs      6.2 s
> > >
> > > compared to the non-memcg case (0.4 to 2.0 ms). This didn't affect
> > > throughput though, just the tail latency. When I ran xfs with large
> > > folios disabled, I still saw ~4.1s.
> >
> > OK, interesting. Thanks for running these tests! Also it is good to know
> > this is not really related to large folios, that somewhat simplifies
> > matters.
> >
> > > From the balance_dirty_pages() ftrace tracepoints, the dirty_ratelimit
> > > value is consistently stable / accurate but it looks like what's
> > > happening is that during freerun, the writer essentially dirties at
> > > memcpy speed until the freerun ceiling, and then the soft/proportional
> > > throttling kicks in but doesn't kick in fast or hard enough, which
> > > allows the number of dirty pages to exceed the hard limit by ~2x, and
> > > then at that point the writer is forced into the loop where it sleeps
> > > max_pause (200ms) each iteration until writeback has drained the
> > > number of dirty pages back under the limit.
> >
> > OK, above you mentioned that the dirty limit for the memcgs is set at
> > 1.6GB. Does that mean that dirty throttling allows memcg to dirty up to
> > ~3.2GB of pages? I wouldn't have expected that...
> 
> On a couple of the runs, I saw it get as high as up to 5GB.
> 
> I think this is because the balance_dirty_pages() code uses the memcg
> stats (NR_FILE_DIRTY) but these stats are only periodically
> flushed/refreshed when reading it, so the balance dirty code is
> seeing/using lagging/non-uptodate values.
> 
> I'm seeing this in
>   balance_dirty_pages()
>       balance_domain_limits(mdtc,...)
>           domain_dirty_avail()
>                 mem_cgroup_wb_stats()
>                        mem_cgroup_flush_stats_ratelimited()
> 
> where mem_cgroup_flush_stats_ratelimted() has this logic:
> 
> void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg)
> {
>         /* Only flush if the periodic flusher is one full cycle late */
>         if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME))
>                 mem_cgroup_flush_stats(memcg);
> }
> 
> where FLUSH_TIME is defined as 2UL * HZ, which afaict means the
> flushing can be around 2 seconds stale (accounting for the periodic
> flusher in flush_memcg_stats_dwork() that flushes every FLUSH_TIME
> interval).
> 
> From what I see, the freerun and hard limit checks use this stale
> under-reported value (in domain_dirty_freerun() and
> wb_position_ratio()), which means the writer gets to keep dirtying and
> blow past the limit for up to 2 extra seconds before the throttling
> limits it.

I see. That would indeed explain why we can overrun the dirty limit so
much.

> Maybe one idea is to in the memcg throttle path do the flush based on
> how many pages are getting dirtied instead of on how much time has
> elapsed? It looks like flushing grabs the rstat lock though, so maybe
> only doing it in the throttled case where it's past freerun would be
> best. It looks like the logic in the regular non-ratelimited
> mem_cgroup_flush_stats() call does something similar. I can run some
> experiments this week with this if you think it could be promising.

Checking some git history, this actually seems to be a relatively recent
change from Shakeel (added to CC) - d9b3ce8769e3 ("mm: writeback: ratelimit
stat flush from mem_cgroup_wb_stats"). So you can check whether reverting
that change makes dirty throttling not overshoot so much.

Shakeel, your change to use mem_cgroup_flush_stats_ratelimited() in
mem_cgroup_wb_stats() results in applications overshooting memcg dirty
limits by several gigabytes which for small memcgs (with 8GB memory in this
case, which means a dirty limit of ~1.5GB) is a considerable issue. I guess
the ratelimiting should take into account the size of memcg (or ideally the
dirty limit in the case of mem_cgroup_wb_stats()) as well. Using
somewhat outdated numbers for dirty throttling is fine but when we can
overshoot the dirty limit more than three times, it leads to rather long
stalls in dirty throttling and similar issues...

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [PATCH] fuse: disable default bdi strictlimiting

[Jan Kara]

[resending once more, this time with correct Shakeel's address]

On Tue 14-07-26 18:21:57, Joanne Koong wrote:
> On Tue, Jul 14, 2026 at 10:10 AM Jan Kara <jack@suse.cz> wrote:
> > On Mon 13-07-26 18:13:29, Joanne Koong wrote:
> > > On Sat, May 30, 2026 at 4:04 AM Jan Kara <jack@suse.cz> wrote:
> > > > On Thu 28-05-26 15:11:18, Joanne Koong wrote:
> > > > > On Thu, May 28, 2026 at 5:34 AM Jan Kara <jack@suse.cz> wrote:
> > > > > > > I think this is also going to be a problem for cgroups with large
> > > > > > > folios since they also, as I understand it, are constrained with a
> > > > > > > limited / tight dirty budget. I ran some initial benchmarks with
> > > > > > > cgroup memory constraints on NVMe and saw similar instability (a
> > > > > > > single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> > > > > > > 15 ms without the cgroup, with the balance_dirty_pages() throttling
> > > > > > > oscillating rather than settling near the set point).
> > > > > >
> > > > > > Ok, so this is more folios (819) than my 512 gut feeling estimate :) What
> > > > > > was the write throughput of the NVMe drive? The high drive throughput also
> > > > > > requires more dirty data to keep the drive saturated so that writeback
> > > > > > throughput doesn't oscilate too much.
> > > > >
> > > > > The write throughput of the NVMe drive I was using was around ~1.1
> > > > > Gb/s (measured by running direct I/O). I think with the 1.6GB dirty
> > > > > budget, the math for that ends up being that the device drains it in
> > > > > ~1.5 secs. The performance I was seeing with the initial cgroup
> > > > > benchmarks was with 4k pages (no large folios enabled) on btrfs.
> > > >
> > > > OK, you might want to experiment with some other filesystem (I suggest xfs
> > > > or ext4) as well. Btrfs writeback behavior is a bit special with its data
> > > > checksum computations etc. and thus latency of starting writeback. It could
> > > > contribute to the erratic behavior with the relatively low dirty limits.
> > >
> > > I reran the benchmarks on xfs and ext4 and saw similiar results. The
> > > max write latency under the memcg were
> > > xfs        5.1s
> > > ext4      5.2 s
> > > btrfs      6.2 s
> > >
> > > compared to the non-memcg case (0.4 to 2.0 ms). This didn't affect
> > > throughput though, just the tail latency. When I ran xfs with large
> > > folios disabled, I still saw ~4.1s.
> >
> > OK, interesting. Thanks for running these tests! Also it is good to know
> > this is not really related to large folios, that somewhat simplifies
> > matters.
> >
> > > From the balance_dirty_pages() ftrace tracepoints, the dirty_ratelimit
> > > value is consistently stable / accurate but it looks like what's
> > > happening is that during freerun, the writer essentially dirties at
> > > memcpy speed until the freerun ceiling, and then the soft/proportional
> > > throttling kicks in but doesn't kick in fast or hard enough, which
> > > allows the number of dirty pages to exceed the hard limit by ~2x, and
> > > then at that point the writer is forced into the loop where it sleeps
> > > max_pause (200ms) each iteration until writeback has drained the
> > > number of dirty pages back under the limit.
> >
> > OK, above you mentioned that the dirty limit for the memcgs is set at
> > 1.6GB. Does that mean that dirty throttling allows memcg to dirty up to
> > ~3.2GB of pages? I wouldn't have expected that...
> 
> On a couple of the runs, I saw it get as high as up to 5GB.
> 
> I think this is because the balance_dirty_pages() code uses the memcg
> stats (NR_FILE_DIRTY) but these stats are only periodically
> flushed/refreshed when reading it, so the balance dirty code is
> seeing/using lagging/non-uptodate values.
> 
> I'm seeing this in
>   balance_dirty_pages()
>       balance_domain_limits(mdtc,...)
>           domain_dirty_avail()
>                 mem_cgroup_wb_stats()
>                        mem_cgroup_flush_stats_ratelimited()
> 
> where mem_cgroup_flush_stats_ratelimted() has this logic:
> 
> void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg)
> {
>         /* Only flush if the periodic flusher is one full cycle late */
>         if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME))
>                 mem_cgroup_flush_stats(memcg);
> }
> 
> where FLUSH_TIME is defined as 2UL * HZ, which afaict means the
> flushing can be around 2 seconds stale (accounting for the periodic
> flusher in flush_memcg_stats_dwork() that flushes every FLUSH_TIME
> interval).
> 
> From what I see, the freerun and hard limit checks use this stale
> under-reported value (in domain_dirty_freerun() and
> wb_position_ratio()), which means the writer gets to keep dirtying and
> blow past the limit for up to 2 extra seconds before the throttling
> limits it.

I see. That would indeed explain why we can overrun the dirty limit so
much.

> Maybe one idea is to in the memcg throttle path do the flush based on
> how many pages are getting dirtied instead of on how much time has
> elapsed? It looks like flushing grabs the rstat lock though, so maybe
> only doing it in the throttled case where it's past freerun would be
> best. It looks like the logic in the regular non-ratelimited
> mem_cgroup_flush_stats() call does something similar. I can run some
> experiments this week with this if you think it could be promising.

Checking some git history, this actually seems to be a relatively recent
change from Shakeel (added to CC) - d9b3ce8769e3 ("mm: writeback: ratelimit
stat flush from mem_cgroup_wb_stats"). So you can check whether reverting
that change makes dirty throttling not overshoot so much.

Shakeel, your change to use mem_cgroup_flush_stats_ratelimited() in
mem_cgroup_wb_stats() results in applications overshooting memcg dirty
limits by several gigabytes which for small memcgs (with 8GB memory in this
case, which means a dirty limit of ~1.5GB) is a considerable issue. I guess
the ratelimiting should take into account the size of memcg (or ideally the
dirty limit in the case of mem_cgroup_wb_stats()) as well. Using
somewhat outdated numbers for dirty throttling is fine but when we can
overshoot the dirty limit more than three times, it leads to rather long
stalls in dirty throttling and similar issues...

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [PATCH] fuse: disable default bdi strictlimiting

[Joanne Koong]

On Wed, Jul 15, 2026 at 3:32 AM Jan Kara <jack@suse.cz> wrote:
>
> [resending once more, this time with correct Shakeel's address]
>
> On Tue 14-07-26 18:21:57, Joanne Koong wrote:
> > On Tue, Jul 14, 2026 at 10:10 AM Jan Kara <jack@suse.cz> wrote:
> > > On Mon 13-07-26 18:13:29, Joanne Koong wrote:
> > > > On Sat, May 30, 2026 at 4:04 AM Jan Kara <jack@suse.cz> wrote:
> > > > > On Thu 28-05-26 15:11:18, Joanne Koong wrote:
> > > > > > On Thu, May 28, 2026 at 5:34 AM Jan Kara <jack@suse.cz> wrote:
> > > > > > > > I think this is also going to be a problem for cgroups with large
> > > > > > > > folios since they also, as I understand it, are constrained with a
> > > > > > > > limited / tight dirty budget. I ran some initial benchmarks with
> > > > > > > > cgroup memory constraints on NVMe and saw similar instability (a
> > > > > > > > single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> > > > > > > > 15 ms without the cgroup, with the balance_dirty_pages() throttling
> > > > > > > > oscillating rather than settling near the set point).
> > > > > > >
> > > > > > > Ok, so this is more folios (819) than my 512 gut feeling estimate :) What
> > > > > > > was the write throughput of the NVMe drive? The high drive throughput also
> > > > > > > requires more dirty data to keep the drive saturated so that writeback
> > > > > > > throughput doesn't oscilate too much.
> > > > > >
> > > > > > The write throughput of the NVMe drive I was using was around ~1.1
> > > > > > Gb/s (measured by running direct I/O). I think with the 1.6GB dirty
> > > > > > budget, the math for that ends up being that the device drains it in
> > > > > > ~1.5 secs. The performance I was seeing with the initial cgroup
> > > > > > benchmarks was with 4k pages (no large folios enabled) on btrfs.
> > > > >
> > > > > OK, you might want to experiment with some other filesystem (I suggest xfs
> > > > > or ext4) as well. Btrfs writeback behavior is a bit special with its data
> > > > > checksum computations etc. and thus latency of starting writeback. It could
> > > > > contribute to the erratic behavior with the relatively low dirty limits.
> > > >
> > > > I reran the benchmarks on xfs and ext4 and saw similiar results. The
> > > > max write latency under the memcg were
> > > > xfs        5.1s
> > > > ext4      5.2 s
> > > > btrfs      6.2 s
> > > >
> > > > compared to the non-memcg case (0.4 to 2.0 ms). This didn't affect
> > > > throughput though, just the tail latency. When I ran xfs with large
> > > > folios disabled, I still saw ~4.1s.
> > >
> > > OK, interesting. Thanks for running these tests! Also it is good to know
> > > this is not really related to large folios, that somewhat simplifies
> > > matters.
> > >
> > > > From the balance_dirty_pages() ftrace tracepoints, the dirty_ratelimit
> > > > value is consistently stable / accurate but it looks like what's
> > > > happening is that during freerun, the writer essentially dirties at
> > > > memcpy speed until the freerun ceiling, and then the soft/proportional
> > > > throttling kicks in but doesn't kick in fast or hard enough, which
> > > > allows the number of dirty pages to exceed the hard limit by ~2x, and
> > > > then at that point the writer is forced into the loop where it sleeps
> > > > max_pause (200ms) each iteration until writeback has drained the
> > > > number of dirty pages back under the limit.
> > >
> > > OK, above you mentioned that the dirty limit for the memcgs is set at
> > > 1.6GB. Does that mean that dirty throttling allows memcg to dirty up to
> > > ~3.2GB of pages? I wouldn't have expected that...
> >
> > On a couple of the runs, I saw it get as high as up to 5GB.
> >
> > I think this is because the balance_dirty_pages() code uses the memcg
> > stats (NR_FILE_DIRTY) but these stats are only periodically
> > flushed/refreshed when reading it, so the balance dirty code is
> > seeing/using lagging/non-uptodate values.
> >
> > I'm seeing this in
> >   balance_dirty_pages()
> >       balance_domain_limits(mdtc,...)
> >           domain_dirty_avail()
> >                 mem_cgroup_wb_stats()
> >                        mem_cgroup_flush_stats_ratelimited()
> >
> > where mem_cgroup_flush_stats_ratelimted() has this logic:
> >
> > void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg)
> > {
> >         /* Only flush if the periodic flusher is one full cycle late */
> >         if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME))
> >                 mem_cgroup_flush_stats(memcg);
> > }
> >
> > where FLUSH_TIME is defined as 2UL * HZ, which afaict means the
> > flushing can be around 2 seconds stale (accounting for the periodic
> > flusher in flush_memcg_stats_dwork() that flushes every FLUSH_TIME
> > interval).
> >
> > From what I see, the freerun and hard limit checks use this stale
> > under-reported value (in domain_dirty_freerun() and
> > wb_position_ratio()), which means the writer gets to keep dirtying and
> > blow past the limit for up to 2 extra seconds before the throttling
> > limits it.
>
> I see. That would indeed explain why we can overrun the dirty limit so
> much.
>
> > Maybe one idea is to in the memcg throttle path do the flush based on
> > how many pages are getting dirtied instead of on how much time has
> > elapsed? It looks like flushing grabs the rstat lock though, so maybe
> > only doing it in the throttled case where it's past freerun would be
> > best. It looks like the logic in the regular non-ratelimited
> > mem_cgroup_flush_stats() call does something similar. I can run some
> > experiments this week with this if you think it could be promising.
>
> Checking some git history, this actually seems to be a relatively recent
> change from Shakeel (added to CC) - d9b3ce8769e3 ("mm: writeback: ratelimit
> stat flush from mem_cgroup_wb_stats"). So you can check whether reverting
> that change makes dirty throttling not overshoot so much.
>

Reverting d9b3ce8769e3 fixes the issue. On xfs, I'm now seeing the
number of dirty pages stays bounded by the limit instead of going 2x+
over, and the max latency is now around ~0.33s instead of ~4s.

> Shakeel, your change to use mem_cgroup_flush_stats_ratelimited() in
> mem_cgroup_wb_stats() results in applications overshooting memcg dirty
> limits by several gigabytes which for small memcgs (with 8GB memory in this
> case, which means a dirty limit of ~1.5GB) is a considerable issue. I guess
> the ratelimiting should take into account the size of memcg (or ideally the
> dirty limit in the case of mem_cgroup_wb_stats()) as well. Using
> somewhat outdated numbers for dirty throttling is fine but when we can
> overshoot the dirty limit more than three times, it leads to rather long
> stalls in dirty throttling and similar issues...

It looks like commit d9b3ce8769e3 was landed in Feb 2024 when rstat
was still using the single global lock. I see commit 748922dcfabd
("cgroup: use subsystem-specific rstat locks to avoid contention")
later in the tree, which was landed around May 2025 and reduces that
global lock contention. That might make the cost of the flush a lot
lower than what it was when the ratelimit was needed in d9b3ce8769e3,
so maybe reverting d9b3ce8769e3 wouldn't make much of a difference for
write-heavy workloads and would be the simplest fix? If it does still
regress, I'd be happy to look into making the flush based on how many
pages have been dirtied and scaled to the dirty limit, if that sounds
like the best way forward. I tried running the sysbench
oltp_write_only benchmark mentioned in d9b3ce8769e3 on my setup but
couldn't get the database to dirty pages fast enough to trigger the
balance dirty pages logic to see how it performs without the
ratelimiting. If you still have the script for how you ran the test
back in 2024, I'm happy to give it a go on my local machine.

Thanks,
Joanne

Re: [PATCH] fuse: disable default bdi strictlimiting

[Shakeel Butt]

Hi Jan & Joanne,

On Wed, Jul 15, 2026 at 06:28:16PM -0700, Joanne Koong wrote:
> On Wed, Jul 15, 2026 at 3:32 AM Jan Kara <jack@suse.cz> wrote:
> >
> > [resending once more, this time with correct Shakeel's address]
> >
> > On Tue 14-07-26 18:21:57, Joanne Koong wrote:
> > > On Tue, Jul 14, 2026 at 10:10 AM Jan Kara <jack@suse.cz> wrote:
> > > > On Mon 13-07-26 18:13:29, Joanne Koong wrote:
> > > > > On Sat, May 30, 2026 at 4:04 AM Jan Kara <jack@suse.cz> wrote:
> > > > > > On Thu 28-05-26 15:11:18, Joanne Koong wrote:
> > > > > > > On Thu, May 28, 2026 at 5:34 AM Jan Kara <jack@suse.cz> wrote:
> > > > > > > > > I think this is also going to be a problem for cgroups with large
> > > > > > > > > folios since they also, as I understand it, are constrained with a
> > > > > > > > > limited / tight dirty budget. I ran some initial benchmarks with
> > > > > > > > > cgroup memory constraints on NVMe and saw similar instability (a
> > > > > > > > > single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> > > > > > > > > 15 ms without the cgroup, with the balance_dirty_pages() throttling
> > > > > > > > > oscillating rather than settling near the set point).
> > > > > > > >
> > > > > > > > Ok, so this is more folios (819) than my 512 gut feeling estimate :) What
> > > > > > > > was the write throughput of the NVMe drive? The high drive throughput also
> > > > > > > > requires more dirty data to keep the drive saturated so that writeback
> > > > > > > > throughput doesn't oscilate too much.
> > > > > > >
> > > > > > > The write throughput of the NVMe drive I was using was around ~1.1
> > > > > > > Gb/s (measured by running direct I/O). I think with the 1.6GB dirty
> > > > > > > budget, the math for that ends up being that the device drains it in
> > > > > > > ~1.5 secs. The performance I was seeing with the initial cgroup
> > > > > > > benchmarks was with 4k pages (no large folios enabled) on btrfs.
> > > > > >
> > > > > > OK, you might want to experiment with some other filesystem (I suggest xfs
> > > > > > or ext4) as well. Btrfs writeback behavior is a bit special with its data
> > > > > > checksum computations etc. and thus latency of starting writeback. It could
> > > > > > contribute to the erratic behavior with the relatively low dirty limits.
> > > > >
> > > > > I reran the benchmarks on xfs and ext4 and saw similiar results. The
> > > > > max write latency under the memcg were
> > > > > xfs        5.1s
> > > > > ext4      5.2 s
> > > > > btrfs      6.2 s
> > > > >
> > > > > compared to the non-memcg case (0.4 to 2.0 ms). This didn't affect
> > > > > throughput though, just the tail latency. When I ran xfs with large
> > > > > folios disabled, I still saw ~4.1s.
> > > >
> > > > OK, interesting. Thanks for running these tests! Also it is good to know
> > > > this is not really related to large folios, that somewhat simplifies
> > > > matters.
> > > >
> > > > > From the balance_dirty_pages() ftrace tracepoints, the dirty_ratelimit
> > > > > value is consistently stable / accurate but it looks like what's
> > > > > happening is that during freerun, the writer essentially dirties at
> > > > > memcpy speed until the freerun ceiling, and then the soft/proportional
> > > > > throttling kicks in but doesn't kick in fast or hard enough, which
> > > > > allows the number of dirty pages to exceed the hard limit by ~2x, and
> > > > > then at that point the writer is forced into the loop where it sleeps
> > > > > max_pause (200ms) each iteration until writeback has drained the
> > > > > number of dirty pages back under the limit.
> > > >
> > > > OK, above you mentioned that the dirty limit for the memcgs is set at
> > > > 1.6GB. Does that mean that dirty throttling allows memcg to dirty up to
> > > > ~3.2GB of pages? I wouldn't have expected that...
> > >
> > > On a couple of the runs, I saw it get as high as up to 5GB.
> > >
> > > I think this is because the balance_dirty_pages() code uses the memcg
> > > stats (NR_FILE_DIRTY) but these stats are only periodically
> > > flushed/refreshed when reading it, so the balance dirty code is
> > > seeing/using lagging/non-uptodate values.
> > >
> > > I'm seeing this in
> > >   balance_dirty_pages()
> > >       balance_domain_limits(mdtc,...)
> > >           domain_dirty_avail()
> > >                 mem_cgroup_wb_stats()
> > >                        mem_cgroup_flush_stats_ratelimited()
> > >
> > > where mem_cgroup_flush_stats_ratelimted() has this logic:
> > >
> > > void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg)
> > > {
> > >         /* Only flush if the periodic flusher is one full cycle late */
> > >         if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME))
> > >                 mem_cgroup_flush_stats(memcg);
> > > }
> > >
> > > where FLUSH_TIME is defined as 2UL * HZ, which afaict means the
> > > flushing can be around 2 seconds stale (accounting for the periodic
> > > flusher in flush_memcg_stats_dwork() that flushes every FLUSH_TIME
> > > interval).
> > >
> > > From what I see, the freerun and hard limit checks use this stale
> > > under-reported value (in domain_dirty_freerun() and
> > > wb_position_ratio()), which means the writer gets to keep dirtying and
> > > blow past the limit for up to 2 extra seconds before the throttling
> > > limits it.
> >
> > I see. That would indeed explain why we can overrun the dirty limit so
> > much.
> >
> > > Maybe one idea is to in the memcg throttle path do the flush based on
> > > how many pages are getting dirtied instead of on how much time has
> > > elapsed? It looks like flushing grabs the rstat lock though, so maybe
> > > only doing it in the throttled case where it's past freerun would be
> > > best. It looks like the logic in the regular non-ratelimited
> > > mem_cgroup_flush_stats() call does something similar. I can run some
> > > experiments this week with this if you think it could be promising.
> >
> > Checking some git history, this actually seems to be a relatively recent
> > change from Shakeel (added to CC) - d9b3ce8769e3 ("mm: writeback: ratelimit
> > stat flush from mem_cgroup_wb_stats"). So you can check whether reverting
> > that change makes dirty throttling not overshoot so much.
> >
> 
> Reverting d9b3ce8769e3 fixes the issue. On xfs, I'm now seeing the
> number of dirty pages stays bounded by the limit instead of going 2x+
> over, and the max latency is now around ~0.33s instead of ~4s.
> 
> > Shakeel, your change to use mem_cgroup_flush_stats_ratelimited() in
> > mem_cgroup_wb_stats() results in applications overshooting memcg dirty
> > limits by several gigabytes which for small memcgs (with 8GB memory in this
> > case, which means a dirty limit of ~1.5GB) is a considerable issue. I guess
> > the ratelimiting should take into account the size of memcg (or ideally the
> > dirty limit in the case of mem_cgroup_wb_stats()) as well. Using
> > somewhat outdated numbers for dirty throttling is fine but when we can
> > overshoot the dirty limit more than three times, it leads to rather long
> > stalls in dirty throttling and similar issues...

Yeah I agree this is an issue.

> 
> It looks like commit d9b3ce8769e3 was landed in Feb 2024 when rstat
> was still using the single global lock. I see commit 748922dcfabd
> ("cgroup: use subsystem-specific rstat locks to avoid contention")
> later in the tree, which was landed around May 2025 and reduces that
> global lock contention. That might make the cost of the flush a lot
> lower than what it was when the ratelimit was needed in d9b3ce8769e3,
> so maybe reverting d9b3ce8769e3 wouldn't make much of a difference for
> write-heavy workloads and would be the simplest fix?

Unfortunetely global lock was just part of the cost and even per-subsystem
lock reduces contention across subsystems but flushing is still expensive as the
kernel may have to traverse the cgroup update tree for each cpu on the system.

CCing Yosry who is looking into similar issue but for zswap stats. Maybe a
general solution for such specific stats is to move them from rstat to a
separate mechanism where update can be a bit more expensive but the read side
can be cheap. 

> If it does still
> regress, I'd be happy to look into making the flush based on how many
> pages have been dirtied and scaled to the dirty limit, if that sounds
> like the best way forward. 

First let's see if the general solution is doable and simple otherwise we can
explore use-case specific solutions. In extreme case revert is an options as
well but I would prefer a more sophisticated ratelimiting (that Jan suggested)
than a revert.

> I tried running the sysbench
> oltp_write_only benchmark mentioned in d9b3ce8769e3 on my setup but
> couldn't get the database to dirty pages fast enough to trigger the
> balance dirty pages logic to see how it performs without the
> ratelimiting. If you still have the script for how you ran the test
> back in 2024, I'm happy to give it a go on my local machine.

Unfortunately I don't have those scripts or setup.

Re: [PATCH] fuse: disable default bdi strictlimiting

[Joanne Koong]

On Thu, Jul 16, 2026 at 8:12 AM Shakeel Butt <shakeel.butt@linux.dev> wrote:
>
> Hi Jan & Joanne,
>
> On Wed, Jul 15, 2026 at 06:28:16PM -0700, Joanne Koong wrote:
> > On Wed, Jul 15, 2026 at 3:32 AM Jan Kara <jack@suse.cz> wrote:
> > >
> > > [resending once more, this time with correct Shakeel's address]
> > >
> > > On Tue 14-07-26 18:21:57, Joanne Koong wrote:
> > > > On Tue, Jul 14, 2026 at 10:10 AM Jan Kara <jack@suse.cz> wrote:
> > > > > On Mon 13-07-26 18:13:29, Joanne Koong wrote:
> > > > > > On Sat, May 30, 2026 at 4:04 AM Jan Kara <jack@suse.cz> wrote:
> > > > > > > On Thu 28-05-26 15:11:18, Joanne Koong wrote:
> > > > > > > > On Thu, May 28, 2026 at 5:34 AM Jan Kara <jack@suse.cz> wrote:
> > > > > > > > > > I think this is also going to be a problem for cgroups with large
> > > > > > > > > > folios since they also, as I understand it, are constrained with a
> > > > > > > > > > limited / tight dirty budget. I ran some initial benchmarks with
> > > > > > > > > > cgroup memory constraints on NVMe and saw similar instability (a
> > > > > > > > > > single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> > > > > > > > > > 15 ms without the cgroup, with the balance_dirty_pages() throttling
> > > > > > > > > > oscillating rather than settling near the set point).
> > > > > > > > >
> > > > > > > > > Ok, so this is more folios (819) than my 512 gut feeling estimate :) What
> > > > > > > > > was the write throughput of the NVMe drive? The high drive throughput also
> > > > > > > > > requires more dirty data to keep the drive saturated so that writeback
> > > > > > > > > throughput doesn't oscilate too much.
> > > > > > > >
> > > > > > > > The write throughput of the NVMe drive I was using was around ~1.1
> > > > > > > > Gb/s (measured by running direct I/O). I think with the 1.6GB dirty
> > > > > > > > budget, the math for that ends up being that the device drains it in
> > > > > > > > ~1.5 secs. The performance I was seeing with the initial cgroup
> > > > > > > > benchmarks was with 4k pages (no large folios enabled) on btrfs.
> > > > > > >
> > > > > > > OK, you might want to experiment with some other filesystem (I suggest xfs
> > > > > > > or ext4) as well. Btrfs writeback behavior is a bit special with its data
> > > > > > > checksum computations etc. and thus latency of starting writeback. It could
> > > > > > > contribute to the erratic behavior with the relatively low dirty limits.
> > > > > >
> > > > > > I reran the benchmarks on xfs and ext4 and saw similiar results. The
> > > > > > max write latency under the memcg were
> > > > > > xfs        5.1s
> > > > > > ext4      5.2 s
> > > > > > btrfs      6.2 s
> > > > > >
> > > > > > compared to the non-memcg case (0.4 to 2.0 ms). This didn't affect
> > > > > > throughput though, just the tail latency. When I ran xfs with large
> > > > > > folios disabled, I still saw ~4.1s.
> > > > >
> > > > > OK, interesting. Thanks for running these tests! Also it is good to know
> > > > > this is not really related to large folios, that somewhat simplifies
> > > > > matters.
> > > > >
> > > > > > From the balance_dirty_pages() ftrace tracepoints, the dirty_ratelimit
> > > > > > value is consistently stable / accurate but it looks like what's
> > > > > > happening is that during freerun, the writer essentially dirties at
> > > > > > memcpy speed until the freerun ceiling, and then the soft/proportional
> > > > > > throttling kicks in but doesn't kick in fast or hard enough, which
> > > > > > allows the number of dirty pages to exceed the hard limit by ~2x, and
> > > > > > then at that point the writer is forced into the loop where it sleeps
> > > > > > max_pause (200ms) each iteration until writeback has drained the
> > > > > > number of dirty pages back under the limit.
> > > > >
> > > > > OK, above you mentioned that the dirty limit for the memcgs is set at
> > > > > 1.6GB. Does that mean that dirty throttling allows memcg to dirty up to
> > > > > ~3.2GB of pages? I wouldn't have expected that...
> > > >
> > > > On a couple of the runs, I saw it get as high as up to 5GB.
> > > >
> > > > I think this is because the balance_dirty_pages() code uses the memcg
> > > > stats (NR_FILE_DIRTY) but these stats are only periodically
> > > > flushed/refreshed when reading it, so the balance dirty code is
> > > > seeing/using lagging/non-uptodate values.
> > > >
> > > > I'm seeing this in
> > > >   balance_dirty_pages()
> > > >       balance_domain_limits(mdtc,...)
> > > >           domain_dirty_avail()
> > > >                 mem_cgroup_wb_stats()
> > > >                        mem_cgroup_flush_stats_ratelimited()
> > > >
> > > > where mem_cgroup_flush_stats_ratelimted() has this logic:
> > > >
> > > > void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg)
> > > > {
> > > >         /* Only flush if the periodic flusher is one full cycle late */
> > > >         if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME))
> > > >                 mem_cgroup_flush_stats(memcg);
> > > > }
> > > >
> > > > where FLUSH_TIME is defined as 2UL * HZ, which afaict means the
> > > > flushing can be around 2 seconds stale (accounting for the periodic
> > > > flusher in flush_memcg_stats_dwork() that flushes every FLUSH_TIME
> > > > interval).
> > > >
> > > > From what I see, the freerun and hard limit checks use this stale
> > > > under-reported value (in domain_dirty_freerun() and
> > > > wb_position_ratio()), which means the writer gets to keep dirtying and
> > > > blow past the limit for up to 2 extra seconds before the throttling
> > > > limits it.
> > >
> > > I see. That would indeed explain why we can overrun the dirty limit so
> > > much.
> > >
> > > > Maybe one idea is to in the memcg throttle path do the flush based on
> > > > how many pages are getting dirtied instead of on how much time has
> > > > elapsed? It looks like flushing grabs the rstat lock though, so maybe
> > > > only doing it in the throttled case where it's past freerun would be
> > > > best. It looks like the logic in the regular non-ratelimited
> > > > mem_cgroup_flush_stats() call does something similar. I can run some
> > > > experiments this week with this if you think it could be promising.
> > >
> > > Checking some git history, this actually seems to be a relatively recent
> > > change from Shakeel (added to CC) - d9b3ce8769e3 ("mm: writeback: ratelimit
> > > stat flush from mem_cgroup_wb_stats"). So you can check whether reverting
> > > that change makes dirty throttling not overshoot so much.
> > >
> >
> > Reverting d9b3ce8769e3 fixes the issue. On xfs, I'm now seeing the
> > number of dirty pages stays bounded by the limit instead of going 2x+
> > over, and the max latency is now around ~0.33s instead of ~4s.
> >
> > > Shakeel, your change to use mem_cgroup_flush_stats_ratelimited() in
> > > mem_cgroup_wb_stats() results in applications overshooting memcg dirty
> > > limits by several gigabytes which for small memcgs (with 8GB memory in this
> > > case, which means a dirty limit of ~1.5GB) is a considerable issue. I guess
> > > the ratelimiting should take into account the size of memcg (or ideally the
> > > dirty limit in the case of mem_cgroup_wb_stats()) as well. Using
> > > somewhat outdated numbers for dirty throttling is fine but when we can
> > > overshoot the dirty limit more than three times, it leads to rather long
> > > stalls in dirty throttling and similar issues...
>
> Yeah I agree this is an issue.
>
> >
> > It looks like commit d9b3ce8769e3 was landed in Feb 2024 when rstat
> > was still using the single global lock. I see commit 748922dcfabd
> > ("cgroup: use subsystem-specific rstat locks to avoid contention")
> > later in the tree, which was landed around May 2025 and reduces that
> > global lock contention. That might make the cost of the flush a lot
> > lower than what it was when the ratelimit was needed in d9b3ce8769e3,
> > so maybe reverting d9b3ce8769e3 wouldn't make much of a difference for
> > write-heavy workloads and would be the simplest fix?
>
> Unfortunetely global lock was just part of the cost and even per-subsystem
> lock reduces contention across subsystems but flushing is still expensive as the
> kernel may have to traverse the cgroup update tree for each cpu on the system.

Ah I see, thanks for the context.

>
> CCing Yosry who is looking into similar issue but for zswap stats. Maybe a
> general solution for such specific stats is to move them from rstat to a
> separate mechanism where update can be a bit more expensive but the read side
> can be cheap.
>
> > If it does still
> > regress, I'd be happy to look into making the flush based on how many
> > pages have been dirtied and scaled to the dirty limit, if that sounds
> > like the best way forward.
>
> First let's see if the general solution is doable and simple otherwise we can
> explore use-case specific solutions. In extreme case revert is an options as
> well but I would prefer a more sophisticated ratelimiting (that Jan suggested)
> than a revert.
>

Sounds good. I'll keep an eye out for the work Yosry's doing on this
and look into hooking it up to writeback if/when his general solution
becomes available.

Thanks,
Joanne

> > I tried running the sysbench
> > oltp_write_only benchmark mentioned in d9b3ce8769e3 on my setup but
> > couldn't get the database to dirty pages fast enough to trigger the
> > balance dirty pages logic to see how it performs without the
> > ratelimiting. If you still have the script for how you ran the test
> > back in 2024, I'm happy to give it a go on my local machine.
>
> Unfortunately I don't have those scripts or setup.

Re: [PATCH] fuse: disable default bdi strictlimiting

[Yosry Ahmed]

On Thu, Jul 16, 2026 at 03:31:00PM -0700, Joanne Koong wrote:
> On Thu, Jul 16, 2026 at 8:12 AM Shakeel Butt <shakeel.butt@linux.dev> wrote:
> >
> > Hi Jan & Joanne,
> >
> > On Wed, Jul 15, 2026 at 06:28:16PM -0700, Joanne Koong wrote:
> > > On Wed, Jul 15, 2026 at 3:32 AM Jan Kara <jack@suse.cz> wrote:
> > > >
> > > > [resending once more, this time with correct Shakeel's address]
> > > >
> > > > On Tue 14-07-26 18:21:57, Joanne Koong wrote:
> > > > > On Tue, Jul 14, 2026 at 10:10 AM Jan Kara <jack@suse.cz> wrote:
> > > > > > On Mon 13-07-26 18:13:29, Joanne Koong wrote:
> > > > > > > On Sat, May 30, 2026 at 4:04 AM Jan Kara <jack@suse.cz> wrote:
> > > > > > > > On Thu 28-05-26 15:11:18, Joanne Koong wrote:
> > > > > > > > > On Thu, May 28, 2026 at 5:34 AM Jan Kara <jack@suse.cz> wrote:
> > > > > > > > > > > I think this is also going to be a problem for cgroups with large
> > > > > > > > > > > folios since they also, as I understand it, are constrained with a
> > > > > > > > > > > limited / tight dirty budget. I ran some initial benchmarks with
> > > > > > > > > > > cgroup memory constraints on NVMe and saw similar instability (a
> > > > > > > > > > > single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> > > > > > > > > > > 15 ms without the cgroup, with the balance_dirty_pages() throttling
> > > > > > > > > > > oscillating rather than settling near the set point).
> > > > > > > > > >
> > > > > > > > > > Ok, so this is more folios (819) than my 512 gut feeling estimate :) What
> > > > > > > > > > was the write throughput of the NVMe drive? The high drive throughput also
> > > > > > > > > > requires more dirty data to keep the drive saturated so that writeback
> > > > > > > > > > throughput doesn't oscilate too much.
> > > > > > > > >
> > > > > > > > > The write throughput of the NVMe drive I was using was around ~1.1
> > > > > > > > > Gb/s (measured by running direct I/O). I think with the 1.6GB dirty
> > > > > > > > > budget, the math for that ends up being that the device drains it in
> > > > > > > > > ~1.5 secs. The performance I was seeing with the initial cgroup
> > > > > > > > > benchmarks was with 4k pages (no large folios enabled) on btrfs.
> > > > > > > >
> > > > > > > > OK, you might want to experiment with some other filesystem (I suggest xfs
> > > > > > > > or ext4) as well. Btrfs writeback behavior is a bit special with its data
> > > > > > > > checksum computations etc. and thus latency of starting writeback. It could
> > > > > > > > contribute to the erratic behavior with the relatively low dirty limits.
> > > > > > >
> > > > > > > I reran the benchmarks on xfs and ext4 and saw similiar results. The
> > > > > > > max write latency under the memcg were
> > > > > > > xfs        5.1s
> > > > > > > ext4      5.2 s
> > > > > > > btrfs      6.2 s
> > > > > > >
> > > > > > > compared to the non-memcg case (0.4 to 2.0 ms). This didn't affect
> > > > > > > throughput though, just the tail latency. When I ran xfs with large
> > > > > > > folios disabled, I still saw ~4.1s.
> > > > > >
> > > > > > OK, interesting. Thanks for running these tests! Also it is good to know
> > > > > > this is not really related to large folios, that somewhat simplifies
> > > > > > matters.
> > > > > >
> > > > > > > From the balance_dirty_pages() ftrace tracepoints, the dirty_ratelimit
> > > > > > > value is consistently stable / accurate but it looks like what's
> > > > > > > happening is that during freerun, the writer essentially dirties at
> > > > > > > memcpy speed until the freerun ceiling, and then the soft/proportional
> > > > > > > throttling kicks in but doesn't kick in fast or hard enough, which
> > > > > > > allows the number of dirty pages to exceed the hard limit by ~2x, and
> > > > > > > then at that point the writer is forced into the loop where it sleeps
> > > > > > > max_pause (200ms) each iteration until writeback has drained the
> > > > > > > number of dirty pages back under the limit.
> > > > > >
> > > > > > OK, above you mentioned that the dirty limit for the memcgs is set at
> > > > > > 1.6GB. Does that mean that dirty throttling allows memcg to dirty up to
> > > > > > ~3.2GB of pages? I wouldn't have expected that...
> > > > >
> > > > > On a couple of the runs, I saw it get as high as up to 5GB.
> > > > >
> > > > > I think this is because the balance_dirty_pages() code uses the memcg
> > > > > stats (NR_FILE_DIRTY) but these stats are only periodically
> > > > > flushed/refreshed when reading it, so the balance dirty code is
> > > > > seeing/using lagging/non-uptodate values.
> > > > >
> > > > > I'm seeing this in
> > > > >   balance_dirty_pages()
> > > > >       balance_domain_limits(mdtc,...)
> > > > >           domain_dirty_avail()
> > > > >                 mem_cgroup_wb_stats()
> > > > >                        mem_cgroup_flush_stats_ratelimited()
> > > > >
> > > > > where mem_cgroup_flush_stats_ratelimted() has this logic:
> > > > >
> > > > > void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg)
> > > > > {
> > > > >         /* Only flush if the periodic flusher is one full cycle late */
> > > > >         if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME))
> > > > >                 mem_cgroup_flush_stats(memcg);
> > > > > }
> > > > >
> > > > > where FLUSH_TIME is defined as 2UL * HZ, which afaict means the
> > > > > flushing can be around 2 seconds stale (accounting for the periodic
> > > > > flusher in flush_memcg_stats_dwork() that flushes every FLUSH_TIME
> > > > > interval).
> > > > >
> > > > > From what I see, the freerun and hard limit checks use this stale
> > > > > under-reported value (in domain_dirty_freerun() and
> > > > > wb_position_ratio()), which means the writer gets to keep dirtying and
> > > > > blow past the limit for up to 2 extra seconds before the throttling
> > > > > limits it.
> > > >
> > > > I see. That would indeed explain why we can overrun the dirty limit so
> > > > much.
> > > >
> > > > > Maybe one idea is to in the memcg throttle path do the flush based on
> > > > > how many pages are getting dirtied instead of on how much time has
> > > > > elapsed? It looks like flushing grabs the rstat lock though, so maybe
> > > > > only doing it in the throttled case where it's past freerun would be
> > > > > best. It looks like the logic in the regular non-ratelimited
> > > > > mem_cgroup_flush_stats() call does something similar. I can run some
> > > > > experiments this week with this if you think it could be promising.
> > > >
> > > > Checking some git history, this actually seems to be a relatively recent
> > > > change from Shakeel (added to CC) - d9b3ce8769e3 ("mm: writeback: ratelimit
> > > > stat flush from mem_cgroup_wb_stats"). So you can check whether reverting
> > > > that change makes dirty throttling not overshoot so much.
> > > >
> > >
> > > Reverting d9b3ce8769e3 fixes the issue. On xfs, I'm now seeing the
> > > number of dirty pages stays bounded by the limit instead of going 2x+
> > > over, and the max latency is now around ~0.33s instead of ~4s.
> > >
> > > > Shakeel, your change to use mem_cgroup_flush_stats_ratelimited() in
> > > > mem_cgroup_wb_stats() results in applications overshooting memcg dirty
> > > > limits by several gigabytes which for small memcgs (with 8GB memory in this
> > > > case, which means a dirty limit of ~1.5GB) is a considerable issue. I guess
> > > > the ratelimiting should take into account the size of memcg (or ideally the
> > > > dirty limit in the case of mem_cgroup_wb_stats()) as well. Using
> > > > somewhat outdated numbers for dirty throttling is fine but when we can
> > > > overshoot the dirty limit more than three times, it leads to rather long
> > > > stalls in dirty throttling and similar issues...
> >
> > Yeah I agree this is an issue.
> >
> > >
> > > It looks like commit d9b3ce8769e3 was landed in Feb 2024 when rstat
> > > was still using the single global lock. I see commit 748922dcfabd
> > > ("cgroup: use subsystem-specific rstat locks to avoid contention")
> > > later in the tree, which was landed around May 2025 and reduces that
> > > global lock contention. That might make the cost of the flush a lot
> > > lower than what it was when the ratelimit was needed in d9b3ce8769e3,
> > > so maybe reverting d9b3ce8769e3 wouldn't make much of a difference for
> > > write-heavy workloads and would be the simplest fix?
> >
> > Unfortunetely global lock was just part of the cost and even per-subsystem
> > lock reduces contention across subsystems but flushing is still expensive as the
> > kernel may have to traverse the cgroup update tree for each cpu on the system.
> 
> Ah I see, thanks for the context.
> 
> >
> > CCing Yosry who is looking into similar issue but for zswap stats. Maybe a
> > general solution for such specific stats is to move them from rstat to a
> > separate mechanism where update can be a bit more expensive but the read side
> > can be cheap.
> >
> > > If it does still
> > > regress, I'd be happy to look into making the flush based on how many
> > > pages have been dirtied and scaled to the dirty limit, if that sounds
> > > like the best way forward.
> >
> > First let's see if the general solution is doable and simple otherwise we can
> > explore use-case specific solutions. In extreme case revert is an options as
> > well but I would prefer a more sophisticated ratelimiting (that Jan suggested)
> > than a revert.
> >
> 
> Sounds good. I'll keep an eye out for the work Yosry's doing on this
> and look into hooking it up to writeback if/when his general solution
> becomes available.

I am not sure if I had in mind would generalize well. For zswap, I was
just going to replace rstat with per-memcg atomic counters, and simply
walk the cgroup parents in the update path, as the zswap load/store path
shouldn't be too hot.

However, looking at other in-kernel flushers (including
mem_cgroup_wb_stats()), I see they consume other stats like
NR_INACTIVE_FILE/NR_ACTIVE_FILE, which are updated in the page
allocation path. I am not sure if a cgroup parent walk with atomic
updates would fly there.

A more problematic one is count_shadow_nodes(), which consumes slab
object stats. I think we definitely cannot do atomic updates in the slab
allocation path. Although it seems like count_shadow_nodes() is a rough
estimate and perhaps we can forgoe using the stats there.

If we want a generic solution for in-kernel flushers to improve stats
accuracy without killing performance, I think we need a heavier lift to
rework rstat or move away from it completely.

Re: [PATCH] fuse: disable default bdi strictlimiting

[Joanne Koong]

On Tue, May 26, 2026 at 6:42 PM Joanne Koong <joannelkoong@gmail.com> wrote:
>
> I read through the writeback throttling code and re-read Jan's very
> helpful comments from this thread last year [1]. So for unprivileged
> servers, I think we definitely cannot remove strictlimiting. If the
> fuse server is slow or unresponsive with writing back the pages, it
> will take up too much of the global dirty budget which will degrade
> write throughput for other filesystems (their throttling will be
> computed against the global dirty page count, eg the freerunning check
> in balance_dirty_pages() and the pos_ratio calculation "pos_ratio =
> pos_ratio_polynom(setpoint, dtc->dirty, limit)" (dtc->dirty is the
> global dirty page count)) and any fuse stuck dirty pages are
> essentially unreclaimable. Without strictlimiting, there will be no
> hard cap on how many dirty pages a misbehaving server can accumulate.
>
> With strictlimiting on and large folios enabled, the problem is that
> the large folio size can potentially dwarf the server's dirty budget,
> which can lead to excessive throttling. When I ran my benchmarks last
> year, I and independently Jingbo saw severe performance regressions
> for buffered writes with large folios (eg 2 GB/s BW w/o, and 200 MB/s
> BW w/) [2] but I think that might have been because the machines had
> limited RAM, resulting in a very small dirty budget.
>

Update: this is no longer a problem. The large folio buffered write
performance regressions I saw last year was fixed by commit
494d2f508883 ('fuse: use default writeback accounting') [1], which
landed last August.

Prior to that commit, on a 2 GB VM, I was seeing for buffered writes:
No large folios:  840 MB/s
Large folios: 270 MB/s

After that commit, I now see around a 30% improvement compared to baseline:
Large folios: 1130 MB/s

Prior to commit 494d2f508883, fuse was doing its own custom writeback
accounting but the accounting wasn't updating wb->writeback_inodes
(which tracks when fuse inodes enter/finish writeback) or
wb->avg_write_bandwidth (which estimates the device write bandwidth).
As a result, this caused the balance_dirty_pages() throttling logic to
think the device is super slow since it's using stale underestimated
values for how fast the device can drain dirty pages. This caused
overthrottling based on the wrong estimated calcualtions.With large
folios this became a noticeably big issue because each folio charges
more dirty pages, where the balance_dirty_pages() logic thinks it will
take a super long time to drain those amount of dirty pages.

commit 494d2f508883 switched fuse to use the mm-internal writeback
accounting, which properly updated wb->writeback_inodes and
wb->avg_write_bandwidth, which fixed the incorrect bandwidth
estimations, which in turn fixed the overthrottling.

Overall, what I'm seeing is that enabling large folios improves
performance across the board for fuse, even with strictlimiting
enabled on memory-constrained systems. I'm going to do a closer audit
of the fuse code to see if we're missing any gaps that will be needed
for turning on large folios, and then will do some more testing. I'll
also rerun benchmarks on NVMe to make sure large folios don't lead to
any performance regressions on fast storage. After that, I think we
are good to go for turning on large folios.

I'm still planning to look into the cgroup dirty throttling
instability on NVMe I was seeing earlier and do some benchmarking /
experimentation with the folio order cap. I'll post updates on that
when I have them.

Thanks,
Joanne

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/fuse/file.c?id=494d2f508883a6e5c4530e5c6b3c8b2bbfb7318d