btrfs thinks fs is full, though 11GB should be still free

[Christoph Anton Mitterer <calestyo@scientia.org>]

Hey.

I think the following might have already happened the 2nd time. I have
a Debian stable with kernel 6.1.55 running Prometheus.

There's one separate btrfs, just for Prometheus time series database.


# btrfs check /dev/vdb 
Opening filesystem to check...
Checking filesystem on /dev/vdb
UUID: decdc81d-7cc4-431c-ab84-e03771f6de5d
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 42427637760 bytes used, no error found
total csum bytes: 27362284
total tree bytes: 32686080
total fs tree bytes: 1982464
total extent tree bytes: 360448
btree space waste bytes: 2839648
file data blocks allocated: 54877196288
 referenced 28014796800
# mount /data/main/


# df | grep main
/dev/vdb       btrfs      43G   43G   25k 100% /data/main

=> df thinks it's full


# btrfs filesystem usage /data/main/
Overall:
    Device size:		  40.00GiB
    Device allocated:		  40.00GiB
    Device unallocated:		   1.00MiB
    Device missing:		     0.00B
    Device slack:		     0.00B
    Used:			  39.54GiB
    Free (estimated):		  24.00KiB	(min: 24.00KiB)
    Free (statfs, df):		  24.00KiB
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		  29.22MiB	(used: 0.00B)
    Multiple profiles:		        no

Data,single: Size:39.48GiB, Used:39.48GiB (100.00%)
   /dev/vdb	  39.48GiB

Metadata,DUP: Size:256.00MiB, Used:31.16MiB (12.17%)
   /dev/vdb	 512.00MiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
   /dev/vdb	  16.00MiB

Unallocated:
   /dev/vdb	   1.00MiB

=> btrfs does so, too

# btrfs subvolume list -pagu /data/main/
ID 257 gen 2347947 parent 5 top level 5 uuid ae3fa7ff-f5a4-cf44-8555-ad579195036c path <FS_TREE>/data

=> no snapshots involved

# du --apparent-size --total -s --si /data/main/
29G	/data/main/
29G	total

=> but when actually counting the file sizes, there should be 11G left.


:/data/main/prometheus# dd if=/dev/zero of=foo bs=1M count=1
dd: error writing 'foo': No space left on device
1+0 records in
0+0 records out
0 bytes copied, 0,0876783 s, 0,0 kB/s


And it really is full.


Any ideas how this can happen?


Thanks,
Chris.