Re: btrfs thinks fs is full, though 11GB should be still free

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2023/12/12 06:56, Christoph Anton Mitterer wrote:
> Hey.
>
> I think the following might have already happened the 2nd time. I have
> a Debian stable with kernel 6.1.55 running Prometheus.
>
> There's one separate btrfs, just for Prometheus time series database.
>
>
> # btrfs check /dev/vdb
> Opening filesystem to check...
> Checking filesystem on /dev/vdb
> UUID: decdc81d-7cc4-431c-ab84-e03771f6de5d
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space tree
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 42427637760 bytes used, no error found
> total csum bytes: 27362284
> total tree bytes: 32686080
> total fs tree bytes: 1982464
> total extent tree bytes: 360448
> btree space waste bytes: 2839648
> file data blocks allocated: 54877196288
>   referenced 28014796800

That's pretty good.

> # mount /data/main/
>
>
> # df | grep main
> /dev/vdb       btrfs      43G   43G   25k 100% /data/main
>
> => df thinks it's full
>
>
> # btrfs filesystem usage /data/main/
> Overall:
>      Device size:		  40.00GiB
>      Device allocated:		  40.00GiB
>      Device unallocated:		   1.00MiB

Already full from the perspective of chunk space.

No new chunk can be allocated.

>      Device missing:		     0.00B
>      Device slack:		     0.00B
>      Used:			  39.54GiB
>      Free (estimated):		  24.00KiB	(min: 24.00KiB)
>      Free (statfs, df):		  24.00KiB
>      Data ratio:			      1.00
>      Metadata ratio:		      2.00
>      Global reserve:		  29.22MiB	(used: 0.00B)
>      Multiple profiles:		        no
>
> Data,single: Size:39.48GiB, Used:39.48GiB (100.00%)

Data chunks are already exhausted.

>     /dev/vdb	  39.48GiB
>
> Metadata,DUP: Size:256.00MiB, Used:31.16MiB (12.17%)

A single metadata chunk, which is not full.


>     /dev/vdb	 512.00MiB
>
> System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
>     /dev/vdb	  16.00MiB
>
> Unallocated:
>     /dev/vdb	   1.00MiB
>
> => btrfs does so, too
>
> # btrfs subvolume list -pagu /data/main/
> ID 257 gen 2347947 parent 5 top level 5 uuid ae3fa7ff-f5a4-cf44-8555-ad579195036c path <FS_TREE>/data

Is your current mounted subvolume the fs tree? Or already the data
subvolume?

If the latter case, there are some files you can not access from your
current mount point.

Thus it's recommended to use qgroup to show a correct full view of the
used space by each subvolume.

Thanks,
Qu

>
> => no snapshots involved
>
> # du --apparent-size --total -s --si /data/main/
> 29G	/data/main/
> 29G	total
>
> => but when actually counting the file sizes, there should be 11G left.
>
>
> :/data/main/prometheus# dd if=/dev/zero of=foo bs=1M count=1
> dd: error writing 'foo': No space left on device
> 1+0 records in
> 0+0 records out
> 0 bytes copied, 0,0876783 s, 0,0 kB/s
>
>
> And it really is full.
>
>
> Any ideas how this can happen?
>
>
> Thanks,
> Chris.
>