Re: [PATCH] btrfs: introduce BTRFS_RESERVE_FLUSH_EMERGENCY

[David Sterba <dsterba@suse.cz>]

On Fri, Sep 09, 2022 at 09:35:01AM -0400, Josef Bacik wrote:
> Inside of FB, as well as some user reports, we've had a consistent
> problem of occasional ENOSPC transaction aborts.  Inside FB we were
> seeing ~100-200 ENOSPC aborts per day in the fleet, which is a really
> low occurrence rate given the size of our fleet, but it's not nothing.
> 
> There are two causes of this particular problem.
> 
> First is delayed allocation.  The reservation system for delalloc
> assumes that contiguous dirty ranges will result in 1 file extent item.
> However if there is memory pressure that results in fragmented writeout,
> or there is fragmentation in the block groups, this won't necessarily be
> true.  Consider the case where we do a single 256MiB write to a file and
> then close it.  We will have 1 reservation for the inode update, the
> reservations for the checksum updates, and 1 reservation for the file
> extent item.  At some point later we decide to write this entire range
> out, but we're so fragmented that we break this into 100 different file
> extents.  Since we've already closed the file and are no longer writing
> to it there's nothing to trigger a refill of the delalloc block rsv to
> satisfy the 99 new file extent reservations we need.  At this point we
> exhaust our delalloc reservation, and we begin to steal from the global
> reserve.  If you have enough of these cases going in parallel you can
> easily exhaust the global reserve, get an ENOSPC at
> btrfs_alloc_tree_block() time, and then abort the transaction.
> 
> The other case is the delayed refs reserve.  The delayed refs reserve
> updates its size based on outstanding delayed refs and dirty block
> groups.  However we only refill this block reserve when returning
> excess reservations and when we call btrfs_start_transaction(root, X).
> We will reserve 2*X credits at transaction start time, and fill in X
> into the delayed refs reserve to make sure it stays topped off.
> Generally this works well, but clearly has downsides.  If we do a
> particularly delayed ref heavy operation we may never catch up in our
> reservations.  Additionally running delayed refs generates more delayed
> refs, and at that point we may be committing the transaction and have no
> way to trigger a refill of our delayed refs rsv.  Then a similar thing
> occurs with the delalloc reserve.
> 
> Generally speaking we well over-reserve in all of our block rsvs.  If we
> reserve 1 credit we're usually reserving around 264k of space, but we'll
> often not use any of that reservation, or use a few blocks of that
> reservation.  We can be reasonably sure that as long as you were able to
> reserve space up front for your operation you'll be able to find space
> on disk for that reservation.
> 
> So introduce a new flushing state, BTRFS_RESERVE_FLUSH_EMERGENCY.  This
> gets used in the case that we've exhausted our reserve and the global
> reserve.  It simply forces a reservation if we have enough actual space
> on disk to make the reservation, which is almost always the case.  This
> keeps us from hitting ENOSPC aborts in these odd occurrences where we've
> not kept up with the delayed work.
> 
> Fixing this in a complete way is going to be relatively complicated and
> time consuming.  This patch is what I discussed with Filipe earlier this
> year, and what I put into our kernels inside FB.  With this patch we're
> down to 1-2 ENOSPC aborts per week, which is a significant reduction.
> This is a decent stop gap until we can work out a more wholistic
> solution to these two corner cases.
> 
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>

I'll add this to misc-next, we now have a full development cycle to find
problems and could remove it eventually.