Re: btrfs thinks fs is full, though 11GB should be still free

Christoph Anton Mitterer
2023-12-12T03:27:36Z

Hey.

On Tue, 2023-12-12 at 13:00 +1030, Qu Wenruo wrote:
> IIRC compsize can do it.
> https://github.com/kilobyte/compsize

Okay... that seems promising:

/data/main/prometheus/metrics2# compsize 01H*
Processed 544 files, 399 regular extents (447 refs), 272 inline.
Type Perc Disk Usage Uncompressed Referenced
TOTAL 100% 37G 37G 23G
none 100% 37G 37G 23G

01H* are the subdirs for the "semi-final" chunks.

So here's my stolen storage ;-)
I'm a bit puzzled how that can happen, I mean for the chunks I'd have
naively assumed that they just write them more or less at once and in
sequence.

Interestingly, the WAL seems good, though:
/data/main/prometheus/metrics2# compsize wal/
Processed 3723 files, 1617 regular extents (1617 refs), 0 inline.
Type Perc Disk Usage Uncompressed Referenced
TOTAL 100% 1.9G 1.9G 1.9G
none 100% 1.9G 1.9G 1.9G

One thing I've noted by chance and don't understand:

I assumed "Referenced" was the number of unique bytes actually
references (by someone). So when I run compsize on a single file,
Reference should be the file size?

/data/main/prometheus/metrics2/wal# lll 00001030
251052 -rw-rw-r-- 1 106 106 ? 134217728 2023-12-10 04:51:58.665808973 +0100 00001030

/data/main/prometheus/metrics2/wal# compsize -b 00001030
Processed 1 file, 83 regular extents (83 refs), 0 inline.
Type Perc Disk Usage Uncompressed Referenced
TOTAL 100% 134451200 134451200 134217728
none 100% 134451200 134451200 134217728

=> okay, here it is

/data/main/prometheus/metrics2/wal# lll 00001045
251947 -rw-rw-r-- 1 106 106 ? 33034564 2023-12-10 08:57:01.892017049 +0100 00001045

/data/main/prometheus/metrics2/wal# compsize -b 00001045
Processed 1 file, 316 regular extents (316 refs), 0 inline.
Type Perc Disk Usage Uncompressed Referenced
TOTAL 100% 33116160 33116160 33038336
none 100% 33116160 33116160 33038336

=> here, Referenced is 3772 bytes larger than the actual file size?
How can that happen?

On Tue, 2023-12-12 at 11:28 +1030, Qu Wenruo wrote:
>
>
> But WAL indeeds looks like a bad patter for btrfs.

> Thus we have "autodefrag" mount option for such use case.

Well the manpage warns from using on large DB workloads... I mean
Prometheus is not exactly like a DB, and I would have naively assumed
that at least the chunks were written not as many small random
writes... but apparently they are.

Also, this a VM, so the storage volume is actually something Ceph
backed, which the university's super computing centre provides us with.

I wonder if I do autodefrag on all that, if it doesn't just kill of our
performance even more?

Thanks :-)

Chris.