Re: btrfs thinks fs is full, though 11GB should be still free

[Christoph Anton Mitterer <calestyo@scientia.org>]

Hey.

On Tue, 2023-12-12 at 13:00 +1030, Qu Wenruo wrote:
> IIRC compsize can do it.
> https://github.com/kilobyte/compsize

Okay... that seems promising:


/data/main/prometheus/metrics2# compsize 01H*
Processed 544 files, 399 regular extents (447 refs), 272 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL      100%       37G          37G          23G       
none       100%       37G          37G          23G       

01H* are the subdirs for the "semi-final" chunks.

So here's my stolen storage ;-)
I'm a bit puzzled how that can happen, I mean for the chunks I'd have
naively assumed that they just write them more or less at once and in
sequence.


Interestingly, the WAL seems good, though:
/data/main/prometheus/metrics2# compsize wal/
Processed 3723 files, 1617 regular extents (1617 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL      100%      1.9G         1.9G         1.9G       
none       100%      1.9G         1.9G         1.9G       



One thing I've noted by chance and don't understand:

I assumed "Referenced" was the number of unique bytes actually
references (by someone). So when I run compsize on a single file,
Reference should be the file size?


/data/main/prometheus/metrics2/wal# lll 00001030
251052 -rw-rw-r-- 1 106 106 ? 134217728 2023-12-10 04:51:58.665808973 +0100 00001030

/data/main/prometheus/metrics2/wal# compsize -b 00001030
Processed 1 file, 83 regular extents (83 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL      100%     134451200    134451200    134217728   
none       100%     134451200    134451200    134217728   

=> okay, here it is


/data/main/prometheus/metrics2/wal# lll 00001045
251947 -rw-rw-r-- 1 106 106 ? 33034564 2023-12-10 08:57:01.892017049 +0100 00001045

/data/main/prometheus/metrics2/wal# compsize -b 00001045
Processed 1 file, 316 regular extents (316 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL      100%     33116160     33116160     33038336    
none       100%     33116160     33116160     33038336    

=> here, Referenced is 3772 bytes larger than the actual file size?
   How can that happen?



On Tue, 2023-12-12 at 11:28 +1030, Qu Wenruo wrote:
> 
> 
> But WAL indeeds looks like a bad patter for btrfs.

> Thus we have "autodefrag" mount option for such use case.

Well the manpage warns from using on large DB workloads... I mean
Prometheus is not exactly like a DB, and I would have naively assumed
that at least the chunks were written not as many small random
writes... but apparently they are.

Also, this a VM, so the storage volume is actually something Ceph
backed, which the university's super computing centre provides us with.

I wonder if I do autodefrag on all that, if it doesn't just kill of our
performance even more?



Thanks :-)

Chris.