Re: [kdave-btrfs-devel:dev/guilherme/temp-fsid-v4] [btrfs] 6c9131ed0d: stress-ng.sync-file.ops_per_sec -44.2% regression

[Josef Bacik <josef@toxicpanda.com>]

On Tue, Sep 26, 2023 at 08:01:35PM +0100, Filipe Manana wrote:
> On Tue, Sep 26, 2023 at 03:34:59PM +0800, kernel test robot wrote:
> > 
> > 
> > Hello,
> > 
> > kernel test robot noticed a -44.2% regression of stress-ng.sync-file.ops_per_sec on:
> > 
> > 
> > commit: 6c9131ed0d644324adeeaccd2feeef8d04950b2d ("btrfs: always reserve space for delayed refs when starting transaction")
> > https://github.com/kdave/btrfs-devel.git dev/guilherme/temp-fsid-v4
> 
> David, can you remove this patch from misc-next/for-next in the meanwhile?
> 
> Starting to reserve space in advance for delayed refs is causing the slowdown,
> and I can reproduce it with the stress-ng test reported below.
> 
> By avoiding refilling the delayed block reserve I can recover about 60% of the
> lost performance, but that increases the chance in extreme scenarios of exhausting
> the global reserve and reaching a dead end -ENOSPC while committing transactions.
> It has happened rarely, both upstream and on SLE kernels.
> 
> At the moment I don't see how to keep both the upfront reservation of space for
> delayed refs and the refill of the delayed refs reserve without the performance
> impact on a test like that stress-ng test.

It's ok to eat a performance regression for correctness.  I think if you can
reduce the pain that's good, but going slower is better than not going at all.

Long term I have plans to make this better, stress-ng hilights the problem here
quite nicely.  The more pressure we have on the enospc machinery the more likely
we're going to do tickets and hit arbitrary waits.

What I want to do is take away the strict "we're over the limit, you now must
flush and wait" behavior and instead have something more akin to what MM does,
simply allow overcommit to occur, add the usage to a per-cpu counter everywhere
to avoid the spin lock contention, and then switch to the normal "ticket and
wait" system once we exceed certain thresholds.

Basically I want ENOSPC to not do anything until we're down to no chunk space
left and we're within 80% full in the available metadata block group.  This will
drastically improve performance, but is a larger project.

Until then trading performance for correctness is simply what we have to do.
Thanks,

Josef