Re: out of space (>865 GB free), filesystem remounts read-only - how to recover?

[Qu Wenruo <wqu@suse.com>]

在 2024/8/21 16:56, Tomasz Chmielewski 写道:
> On 2024-08-21 00:09, Qu Wenruo wrote:
>> 在 2024/8/20 23:07, Tomasz Chmielewski 写道:
>>> My 44 TB filesystem was getting full (~865 GB free out of 44 TB in
>>> total), so I thought removing some old data would be a good thing to do.
>>
>> btrfs fi usage output please.
>>
>> I bet it's some btrfs RAID setup, maybe RAID1/10 or RAID1C3/C4 for 
>> metadata.
>> And the disk usages got unbalanced, the metadata overcommitment did a
>> wrong estimation and cause the ENOSPC.
> 
> # btrfs filesystem usage /data1
> Overall:
>      Device size:                  43.75TiB
>      Device allocated:             43.66TiB
>      Device unallocated:          100.00GiB
>      Device missing:                  0.00B
>      Used:                         42.51TiB
>      Free (estimated):            951.71GiB      (min: 926.71GiB)
>      Free (statfs, df):           876.71GiB
>      Data ratio:                       1.33
>      Metadata ratio:                   2.00
>      Global reserve:              512.00MiB      (used: 64.00KiB)
>      Multiple profiles:                  no
> 
> Data,RAID5: Size:32.63TiB, Used:31.78TiB (97.38%)
>     /dev/sdc1      10.88TiB
>     /dev/sdd1      10.88TiB
>     /dev/sde1      10.88TiB
>     /dev/sdf1      10.88TiB
> 
> Metadata,RAID1: Size:73.00GiB, Used:71.89GiB (98.48%)
>     /dev/sdc1      37.00GiB
>     /dev/sdd1      37.00GiB
>     /dev/sde1      36.00GiB
>     /dev/sdf1      36.00GiB
> 
> System,RAID1: Size:8.00MiB, Used:2.30MiB (28.71%)
>     /dev/sde1       8.00MiB
>     /dev/sdf1       8.00MiB
> 
> Unallocated:
>     /dev/sdc1       1.00MiB
>     /dev/sdd1       1.00MiB
>     /dev/sde1       1.00MiB
>     /dev/sdf1       1.00MiB
>     /dev/loop0    100.00GiB
> # btrfs filesystem usage /data1
> Overall:
>      Device size:                  43.75TiB
>      Device allocated:             43.66TiB
>      Device unallocated:          100.00GiB
>      Device missing:                  0.00B
>      Used:                         42.51TiB
>      Free (estimated):            951.71GiB      (min: 926.71GiB)
>      Free (statfs, df):           876.71GiB
>      Data ratio:                       1.33
>      Metadata ratio:                   2.00
>      Global reserve:              512.00MiB      (used: 64.00KiB)
>      Multiple profiles:                  no
> 
> Data,RAID5: Size:32.63TiB, Used:31.78TiB (97.38%)
>     /dev/sdc1      10.88TiB
>     /dev/sdd1      10.88TiB
>     /dev/sde1      10.88TiB
>     /dev/sdf1      10.88TiB
> 
> Metadata,RAID1: Size:73.00GiB, Used:71.89GiB (98.48%)
>     /dev/sdc1      37.00GiB
>     /dev/sdd1      37.00GiB
>     /dev/sde1      36.00GiB
>     /dev/sdf1      36.00GiB

Your metadata is obviously near full.

But it's still a bug that btrfs free space reservation code didn't 
return ENOSPC early, other than hitting ENOSPC at transaction commit time.

> 
> System,RAID1: Size:8.00MiB, Used:2.30MiB (28.71%)
>     /dev/sde1       8.00MiB
>     /dev/sdf1       8.00MiB
> 
> Unallocated:
>     /dev/sdc1       1.00MiB
>     /dev/sdd1       1.00MiB
>     /dev/sde1       1.00MiB
>     /dev/sdf1       1.00MiB

At least one good news is, all your four devices are usage are very 
balanced.
>     /dev/loop0    100.00GiB

You just need to add another device to fulfill the RAID1 requirement.

Then you will still need to remove enough data to get at least 1GiB for 
two devices, then you can go balance data to free some space for your 
metadata.

Thanks,
Qu
> 
> 
> # btrfs filesystem df /data1
> Data, RAID5: total=32.63TiB, used=31.78TiB
> System, RAID1: total=8.00MiB, used=2.30MiB
> Metadata, RAID1: total=73.00GiB, used=71.89GiB
> GlobalReserve, single: total=512.00MiB, used=64.00KiB
> 
> 
> # btrfs filesystem show /data1
> Label: 'data'  uuid: a80ce575-8c39-4065-80ce-2ca015fa1d51
>          Total devices 5 FS bytes used 31.85TiB
>          devid    1 size 10.91TiB used 10.91TiB path /dev/sdc1
>          devid    2 size 10.91TiB used 10.91TiB path /dev/sdd1
>          devid    3 size 10.91TiB used 10.91TiB path /dev/sde1
>          devid    4 size 10.91TiB used 10.91TiB path /dev/sdf1
>          devid    5 size 100.00GiB used 0.00B path /dev/loop0
> 
> 
> Tomasz
>