Re: repeated enospc errors during balance on a filesystem with spare room - pls advise

[Zygo Blaxell <ce3g8jdj@umail.furryterror.org>]

[-- Attachment #1: Type: text/plain, Size: 4223 bytes --]

On Wed, Jan 01, 2020 at 11:05:00AM +0100, Swâmi Petaramesh wrote:
> Hi list,
> 
> Le 31/12/2019 à 23:58, Zygo Blaxell a écrit :
> > A full balance includes a metadata balance.  The primary effect
> > of metadata balance is to temporarily reduce space for metadata.
> > Reducing metadata space causes an assortment of problems for btrfs,
> > only one of which is hitting the 5.4 free space bug.  For all but a few
> > contrived test cases, btrfs manages metadata well without interference
> > from balance.  Too much metadata balancing (i.e. any at all) can make
> > a filesystem truly run out of metadata space on disk--a condition that
> > is sometimes difficult to reverse.
> 
> I however was hit by the "dummy zero free space condition" using 5.4 on
> a machine, and the resulting filesystem (on an external HD) then still
> showed 0% free using a 5.3 or a 4.15 kernel on other machines.

Interesting...so 5.4 can put the filesystem into a _persistent_ state
where the metadata allocator makes no further progress.  Two bugs in one!

I do note from the git history that metadata free space calculations
seem to have been dodgy for years.  The code that is touched from the
5.4 kernel bug fix patch dates back to v4.5, so probably all kernels
newer than 2015 have some metadata allocation bugs (kernels up to 4.4
would hit ENOSPC while reporting non-zero free space reported in df,
kernels 4.5 and later report zero free space in df first, then ENOSPC).
5.4 has other changes that make these problems more frequent, but these
are not new bugs.

The longer patch series Qu mentioned looks like it's trying to properly
solve the problem instead of using a quick guesstimate.

> It however passsed "btrfs check" without any error.

Not surprising.  "btrfs check" cannot detect kernel bugs.

"btrfs check" is concerned with on-disk data, while free space
calculations only exist in kernel memory structures.  btrfs check verifies
the metadata is correct, but check can't tell you if the kernel will read
the metadata and then just return the wrong free space number.

> The thing that fixed it and returned it to a « normal working state »
> has been running a "btrfs balance -m" (on the 5.4 machine) on it.
> 
> So I'm a bit puzzled reading that metadata balance is useless when it
> precisely fixed this issue on a FS here.

The kernel bug is triggered under very specific conditions.  Almost any
change to the filesystem could change those conditions to temporarily work
around the kernel bug.  Metadata balance is a change to the filesystem,
so sometimes it works--but it can also trigger the problem in a filesystem
that didn't have the problem before.

Other things that randomly allocate metadata, and might by chance
temporarily fix this problem without the bad side-effects of metadata
balances:

	- metadata defrag (btrfs fi defrag on a subvol without -r)

	- metadata modifications (deleting or touching files,
	creating and deleting empty directories)

	- running a data balance (allocates a lot of metadata)

	- using the metadata_ratio mount option

For 5 years our regression tests have been running on filesystems
that never run metadata balances.  We don't run metadata balances in
the test workloads or in production because 1) in production we never
have a requirement to balance metadata on any workload except for raid
conversions, and 2) balancing metadata is a pretty reliable reproducer
for the correct, non-bug metadata ENOSPC condition that a btrfs gets
into when it runs out of both metadata and unallocated space.  This is
especially bad on filesystems with less than ~250GB of free space, where
rounding to GB-sized block groups and free space fragmentation in data
block groups combine to make it very important to have a lot of slack
in metadata block groups.

We've probably been avoiding all the false-zero-free-space conditions
in kernels 4.5+ accidentally, simply as a side-effect of never running a
metadata balance.  To reproduce the 5.4 issue, I had to set up a custom
test workload which did metadata balances to force the bug to happen.

> Kind regards (and best wishes for a happy new year).
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]