Re: out of space (>865 GB free), filesystem remounts read-only - how to recover?

[Tomasz Chmielewski <mangoo@wpkg.org>]

On 2024-08-21 00:09, Qu Wenruo wrote:
> 在 2024/8/20 23:07, Tomasz Chmielewski 写道:
>> My 44 TB filesystem was getting full (~865 GB free out of 44 TB in
>> total), so I thought removing some old data would be a good thing to 
>> do.
> 
> btrfs fi usage output please.
> 
> I bet it's some btrfs RAID setup, maybe RAID1/10 or RAID1C3/C4 for 
> metadata.
> And the disk usages got unbalanced, the metadata overcommitment did a
> wrong estimation and cause the ENOSPC.

# btrfs filesystem usage /data1
Overall:
     Device size:                  43.75TiB
     Device allocated:             43.66TiB
     Device unallocated:          100.00GiB
     Device missing:                  0.00B
     Used:                         42.51TiB
     Free (estimated):            951.71GiB      (min: 926.71GiB)
     Free (statfs, df):           876.71GiB
     Data ratio:                       1.33
     Metadata ratio:                   2.00
     Global reserve:              512.00MiB      (used: 64.00KiB)
     Multiple profiles:                  no

Data,RAID5: Size:32.63TiB, Used:31.78TiB (97.38%)
    /dev/sdc1      10.88TiB
    /dev/sdd1      10.88TiB
    /dev/sde1      10.88TiB
    /dev/sdf1      10.88TiB

Metadata,RAID1: Size:73.00GiB, Used:71.89GiB (98.48%)
    /dev/sdc1      37.00GiB
    /dev/sdd1      37.00GiB
    /dev/sde1      36.00GiB
    /dev/sdf1      36.00GiB

System,RAID1: Size:8.00MiB, Used:2.30MiB (28.71%)
    /dev/sde1       8.00MiB
    /dev/sdf1       8.00MiB

Unallocated:
    /dev/sdc1       1.00MiB
    /dev/sdd1       1.00MiB
    /dev/sde1       1.00MiB
    /dev/sdf1       1.00MiB
    /dev/loop0    100.00GiB
# btrfs filesystem usage /data1
Overall:
     Device size:                  43.75TiB
     Device allocated:             43.66TiB
     Device unallocated:          100.00GiB
     Device missing:                  0.00B
     Used:                         42.51TiB
     Free (estimated):            951.71GiB      (min: 926.71GiB)
     Free (statfs, df):           876.71GiB
     Data ratio:                       1.33
     Metadata ratio:                   2.00
     Global reserve:              512.00MiB      (used: 64.00KiB)
     Multiple profiles:                  no

Data,RAID5: Size:32.63TiB, Used:31.78TiB (97.38%)
    /dev/sdc1      10.88TiB
    /dev/sdd1      10.88TiB
    /dev/sde1      10.88TiB
    /dev/sdf1      10.88TiB

Metadata,RAID1: Size:73.00GiB, Used:71.89GiB (98.48%)
    /dev/sdc1      37.00GiB
    /dev/sdd1      37.00GiB
    /dev/sde1      36.00GiB
    /dev/sdf1      36.00GiB

System,RAID1: Size:8.00MiB, Used:2.30MiB (28.71%)
    /dev/sde1       8.00MiB
    /dev/sdf1       8.00MiB

Unallocated:
    /dev/sdc1       1.00MiB
    /dev/sdd1       1.00MiB
    /dev/sde1       1.00MiB
    /dev/sdf1       1.00MiB
    /dev/loop0    100.00GiB


# btrfs filesystem df /data1
Data, RAID5: total=32.63TiB, used=31.78TiB
System, RAID1: total=8.00MiB, used=2.30MiB
Metadata, RAID1: total=73.00GiB, used=71.89GiB
GlobalReserve, single: total=512.00MiB, used=64.00KiB


# btrfs filesystem show /data1
Label: 'data'  uuid: a80ce575-8c39-4065-80ce-2ca015fa1d51
         Total devices 5 FS bytes used 31.85TiB
         devid    1 size 10.91TiB used 10.91TiB path /dev/sdc1
         devid    2 size 10.91TiB used 10.91TiB path /dev/sdd1
         devid    3 size 10.91TiB used 10.91TiB path /dev/sde1
         devid    4 size 10.91TiB used 10.91TiB path /dev/sdf1
         devid    5 size 100.00GiB used 0.00B path /dev/loop0


Tomasz