btrfs thinks fs is full, though 11GB should be still free

Christoph Anton Mitterer
2023-12-11T20:26:41Z

Hey.

I think the following might have already happened the 2nd time. I have
a Debian stable with kernel 6.1.55 running Prometheus.

There's one separate btrfs, just for Prometheus time series database.

# btrfs check /dev/vdb
Opening filesystem to check...
Checking filesystem on /dev/vdb
UUID: decdc81d-7cc4-431c-ab84-e03771f6de5d
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 42427637760 bytes used, no error found
total csum bytes: 27362284
total tree bytes: 32686080
total fs tree bytes: 1982464
total extent tree bytes: 360448
btree space waste bytes: 2839648
file data blocks allocated: 54877196288
referenced 28014796800
# mount /data/main/

# df | grep main
/dev/vdb btrfs 43G 43G 25k 100% /data/main

=> df thinks it's full

# btrfs filesystem usage /data/main/
Overall:
Device size: 40.00GiB
Device allocated: 40.00GiB
Device unallocated: 1.00MiB
Device missing: 0.00B
Device slack: 0.00B
Used: 39.54GiB
Free (estimated): 24.00KiB (min: 24.00KiB)
Free (statfs, df): 24.00KiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 29.22MiB (used: 0.00B)
Multiple profiles: no

Data,single: Size:39.48GiB, Used:39.48GiB (100.00%)
/dev/vdb 39.48GiB

Metadata,DUP: Size:256.00MiB, Used:31.16MiB (12.17%)
/dev/vdb 512.00MiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
/dev/vdb 16.00MiB

Unallocated:
/dev/vdb 1.00MiB

=> btrfs does so, too

# btrfs subvolume list -pagu /data/main/
ID 257 gen 2347947 parent 5 top level 5 uuid ae3fa7ff-f5a4-cf44-8555-ad579195036c path <FS_TREE>/data

=> no snapshots involved

# du --apparent-size --total -s --si /data/main/
29G /data/main/
29G total

=> but when actually counting the file sizes, there should be 11G left.

:/data/main/prometheus# dd if=/dev/zero of=foo bs=1M count=1
dd: error writing 'foo': No space left on device
1+0 records in
0+0 records out
0 bytes copied, 0,0876783 s, 0,0 kB/s

And it really is full.

Any ideas how this can happen?

Thanks,
Chris.