Re: Large BTRFS array suddenly says 53TiB Free, usage inconsistent

Max Spliethöver
2021-11-14T18:46:14Z

Hello everyone.
I observed the exact same behavior on my 2x4TB RAID1. After an update of my server that runs a btrfs RAID1 as data storage (root fs runs on different, non-btrfs disks) and running `sudo btrfs filesystem usage /tank`, I realized that the "Data ratio" and "Metadata ratio" had dropped from 2.00 (before upgrade) to 1.00 and that the Unallocated space on both drives jumped from ~550GB to 2.10TB. I sporadically checked the files and everything seems to be still there.

I would appreciate any help with explaining what happened and how to possibly fix this issue. Below I provided some information. If further outputs are required, please let me know.

```
$ sudo btrfs filesystem usage /tank
Overall:
Device size: 7.28TiB
Device allocated: 3.07TiB
Device unallocated: 4.21TiB
Device missing: 0.00B
Used: 3.07TiB
Free (estimated): 4.21TiB (min: 4.21TiB)
Free (statfs, df): 582.54GiB
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no

Data,RAID1: Size:3.07TiB, Used:3.06TiB (99.95%)
/dev/sdd1 1.53TiB
/dev/sde1 1.53TiB

Metadata,RAID1: Size:5.00GiB, Used:4.13GiB (82.59%)
/dev/sdd1 2.50GiB
/dev/sde1 2.50GiB

System,RAID1: Size:32.00MiB, Used:464.00KiB (1.42%)
/dev/sdd1 16.00MiB
/dev/sde1 16.00MiB

Unallocated:
/dev/sdd1 2.10TiB
/dev/sde1 2.10TiB
```

Also, `dmesg` and `btrfs check` do not show any errors.

```
$ sudo dmesg | grep BTRFS
[ 4.161867] BTRFS: device label tank devid 2 transid 204379 /dev/sde1 scanned by systemd-udevd (252)
[ 4.163715] BTRFS: device label tank devid 1 transid 204379 /dev/sdd1 scanned by systemd-udevd (234)
[ 300.416174] BTRFS info (device sdd1): flagging fs with big metadata feature
[ 300.416179] BTRFS info (device sdd1): disk space caching is enabled
[ 300.416181] BTRFS info (device sdd1): has skinny extents

$ sudo btrfs check -p /dev/sdd1
Opening filesystem to check...
Checking filesystem on /dev/sdd1
UUID: 37ce3698-b9d4-4475-8569-fc440c54ad82
[1/7] checking root items (0:00:11 elapsed, 698424 items checked)
[2/7] checking extents (0:00:42 elapsed, 270676 items checked)
[3/7] checking free space cache (0:00:02 elapsed, 3147 items checked)
[4/7] checking fs roots (0:00:11 elapsed, 22115 items checked)
[5/7] checking csums (without verifying data) (0:00:02 elapsed, 1439154 items checked)
[6/7] checking root refs (0:00:00 elapsed, 13 items checked)
[7/7] checking quota groups skipped (not enabled on this FS)
found 3374319136768 bytes used, no error found
total csum bytes: 3290097884
total tree bytes: 4434460672
total fs tree bytes: 363446272
total extent tree bytes: 62717952
btree space waste bytes: 721586744
file data blocks allocated: 4579386322944
referenced 4576433889280

$ sudo btrfs check -p /dev/sde1
Opening filesystem to check...
Checking filesystem on /dev/sde1
UUID: 37ce3698-b9d4-4475-8569-fc440c54ad82
[1/7] checking root items (0:00:11 elapsed, 698424 items checked)
[2/7] checking extents (0:00:43 elapsed, 270676 items checked)
[3/7] checking free space cache (0:00:02 elapsed, 3147 items checked)
[4/7] checking fs roots (0:00:11 elapsed, 22115 items checked)
[5/7] checking csums (without verifying data) (0:00:02 elapsed, 1439154 items checked)
[6/7] checking root refs (0:00:00 elapsed, 13 items checked)
[7/7] checking quota groups skipped (not enabled on this FS)
found 3374319136768 bytes used, no error found
total csum bytes: 3290097884
total tree bytes: 4434460672
total fs tree bytes: 363446272
total extent tree bytes: 62717952
btree space waste bytes: 721586744
file data blocks allocated: 4579386322944
referenced 4576433889280
```

Below, you can find some more useful outputs.

```
$ sudo btrfs fi show
Label: 'tank' uuid: 37ce3698-b9d4-4475-8569-fc440c54ad82
Total devices 2 FS bytes used 3.07TiB
devid 1 size 3.64TiB used 3.07TiB path /dev/sdd1
devid 2 size 3.64TiB used 3.07TiB path /dev/sde1

$ btrfs --version
btrfs-progs v5.15

$ uname -r
5.15.2-arch1-1
```

-Max

On 11/14/21 06:48, Joshua wrote:
> I have a large multi-device BTRFS array. (13 devices / 96TiB total usable space)
>
> As of yesterday, it had a little over 5 TiB reported as estimated free by 'btrfs fi usage'
>
> At exactly 7am this morning, my reporting tool reports that the "Free (estimated)" line of 'btrfs
> fi usage' jumped to 53TiB.
>
> Now I do use snapshots, managed by btrbk. I currently have 80 snapshots, and it is possible old
> snapshots were deleted at midnight, freeing up data. Perhaps the deletions didn't finish committing until 7am?
>
> However, the current state of the array is concerning to me:
>
> #> btrfs fi usage /mnt
> Overall:
> Device size: 96.42TiB
> Device allocated: 43.16TiB
> Device unallocated: 53.26TiB
> Device missing: 0.00B
> Used: 43.15TiB
> Free (estimated): 53.27TiB (min: 53.27TiB)
> Free (statfs, df): 4.71TiB
> Data ratio: 1.00
> Metadata ratio: 1.00
> Global reserve: 512.00MiB (used: 0.00B)
> Multiple profiles: no
>
> Data,RAID1: Size:43.10TiB, Used:43.09TiB (99.98%)
> {snip}
> Metadata,RAID1C3: Size:66.00GiB, Used:62.51GiB (94.71%)
> {snip}
> System,RAID1C3: Size:32.00MiB, Used:7.12MiB (22.27%)
> {snip}
> Unallocated:
> {snip}
>
> As you can see, it's showing all my data is Raid1 as it should be, and all my metadata is raid1c3
> as it should be.
> BUT it's showing data ratio: 1 and metadata ratio: 1
> Also, the allocated space is showing 43 TiB, which I know to be around the actual amount of used
> data by files. Since Raid1 is in use, Allocated data should be around 86....
>
> Any ideas as to what happened, why it's showing this erroneous data, or if I should be worried
> about my data in any way?
> As of right now, everything appears intact....
>
> --Joshua Villwock
>