Re: btrfs thinks fs is full, though 11GB should be still free

Christoph Anton Mitterer
2023-12-12T00:12:16Z

On Tue, 2023-12-12 at 10:24 +1030, Qu Wenruo wrote:
> Then the last thing is extent bookends.
>
> COW and small random writes can easily lead to extra space wasted by
> extent bookends.

Is there a way to check this? Would I just seem maaany extents when I
look at the files with filefrag?

I mean Prometheus, continuously collects metrics from a number of nodes
an (sooner or later) writes them to disk.
I don't really know their code, so I have no idea if they already write
every tiny metric, or only large bunches thereof.

Since they do maintain a WAL, I'd assume the former.

Every know and then, the WAL is written to chunk files which are rather
large, well ~160M or so in my case, but that depends on how many
metrics one collects. I think they always write data for a period of
2h.
Later on, they further compact that chunks (I think after 8 hours and
so on), in which case some larger rewritings would be done.
Though in my case this doesn't happen, as I run Thanos on top of
Prometheus, and for that one needs to disable Prometheus' own
compaction.

I've had already previously looked at the extents for these "compacted"
chunk files, but the worst file had only 32 extents (as reported by
filefrag).

Looking at the WAL files:
/data/main/prometheus/metrics2/wal# filefrag * | grep -v ' 0 extents
found'
00001030: 82 extents found
00001031: 81 extents found
00001032: 79 extents found
00001033: 82 extents found
00001034: 78 extents found
00001035: 78 extents found
00001036: 81 extents found
00001037: 79 extents found
00001038: 79 extents found
00001039: 89 extents found
00001040: 80 extents found
00001041: 74 extents found
00001042: 81 extents found
00001043: 97 extents found
00001044: 101 extents found
00001045: 316 extents found
checkpoint.00001029: FIBMAP/FIEMAP unsupported

(I did the grep -v, because there were a gazillion of empty wal files,
presumably created when the fs was already full).

The above numbers though still don't look to bad, do they?

And checking all:
# find /data/main/ -type f -execdir filefrag {} \; | cut -d : -f 2 |
sort | uniq -c | sort -V
3706 0 extents found
450 1 extent found
25 3 extents found
62 2 extents found
1 8 extents found
1 9 extents found
1 10 extents found
1 11 extents found
1 32 extents found
1 74 extents found
1 80 extents found
1 89 extents found
1 97 extents found
1 101 extents found
1 316 extents found
2 78 extents found
2 82 extents found
3 5 extents found
3 79 extents found
3 81 extents found
6 4 extents found

> E.g. You write a 16M data extents, then over-write the tailing 8M,
> now
> we have two data extents, the old 16M and the new 8M, wasting 8M
> space.
>
> In that case, you can try defrag, but you still need to delete some
> data
> first so that you can do defrag...

Well my main concern is rather how to prevent this from happening in
the first place... the data is already all backuped into Thanos, so I
could also just wipe the fs.
But this seems to occur repeatedly (well, okay only twice so far O:-)
).
So that would mean we have some IO pattern that "kills" btrfs.

Cheers,
Chris.