Re: 97% full system, dusage didn't help, musage strange

[Andrei Borzenkov <arvidjaar@gmail.com>]

14.12.2024 20:55, Leszek Dubiel wrote:
> 
> 
> My system is almost full:
> 
> 
> root@zefir:~# df -h
> 
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/sdb2       8.2T  7.9T  256G  97% /
> 
> 
> 
> 
> root@zefir:~# btrfs fi df /
> 
> Data, RAID1: total=8.08TiB, used=7.84TiB
> System, RAID1: total=32.00MiB, used=1.47MiB
> Metadata, RAID1: total=44.00GiB, used=36.26GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> 
> 
> 
> I have 256 GB free space, but almost no unallocated space:
> 
> 
> root@zefir:~# btrfs dev usa /
> /dev/sdb2, ID: 1
>      Device size:             5.43TiB
>      Device slack:              0.00B
>      Data,RAID1:              5.38TiB
>      Metadata,RAID1:         31.00GiB
>      System,RAID1:           32.00MiB
>      Unallocated:            11.00GiB
> 
> /dev/sdc2, ID: 2
>      Device size:             5.43TiB
>      Device slack:              0.00B
>      Data,RAID1:              5.39TiB
>      Metadata,RAID1:         22.00GiB
>      Unallocated:            10.03GiB
> 
> /dev/sda3, ID: 3
>      Device size:             5.43TiB
>      Device slack:              0.00B
>      Data,RAID1:              5.38TiB
>      Metadata,RAID1:         35.00GiB
>      System,RAID1:           32.00MiB
>      Unallocated:            11.00GiB
> 
> 
> 

Show

btrfs filesystem usage -T

> 
> I've been running whole day
> 
>             btrfs balance start -dusage=xxx,limit=8 /
> 
> with increasing numbers of xxx, until I reached dusage=90:
> 
> 
> root@zefir:~# btrfs bala start -dusage=20,limit=8 /
> Done, had to relocate 0 out of 8319 chunks
> 
> root@zefir:~# btrfs bala start -dusage=50,limit=8 /
> Done, had to relocate 0 out of 8319 chunks
> 
> root@zefir:~# btrfs bala start -dusage=80,limit=8 /
> Done, had to relocate 0 out of 8319 chunks
> 
> root@zefir:~# btrfs bala start -dusage=90,limit=8 /
> 
> 
> 
> 
> 
> I was running with -dusage=90 (90%) whole day, but
> unallocated space didn't increase.
> 
> On logs i can see:
> 
> 2024-12-09T08:46:13.001188+01:00 zefir kernel: [431476.446252] BTRFS
> info (device sda2): balance: start -dusage=90,limit=8
> 2024-12-09T08:46:13.013180+01:00 zefir kernel: [431476.458060] BTRFS
> info (device sda2): relocating block group 34750669520896 flags data|raid1
> 2024-12-09T08:46:40.389168+01:00 zefir kernel: [431503.832191] BTRFS
> info (device sda2): found 6 extents, stage: move data extents
> 2024-12-09T08:46:44.193216+01:00 zefir kernel: [431507.636729] BTRFS
> info (device sda2): found 6 extents, stage: update data pointers
> 2024-12-09T08:46:47.113166+01:00 zefir kernel: [431510.558009] BTRFS
> info (device sda2): relocating block group 34748522037248 flags data|raid1
> 2024-12-09T08:47:22.241196+01:00 zefir kernel: [431545.684216] BTRFS
> info (device sda2): found 11 extents, stage: move data extents
> 2024-12-09T08:47:23.933198+01:00 zefir kernel: [431547.378516] BTRFS
> info (device sda2): found 11 extents, stage: update data pointers
> 2024-12-09T08:47:25.137176+01:00 zefir kernel: [431548.582508] BTRFS
> info (device sda2): relocating block group 34731342168064 flags data|raid1
> 2024-12-09T08:48:01.897151+01:00 zefir kernel: [431585.342544] BTRFS
> info (device sda2): found 8 extents, stage: move data extents
> 2024-12-09T08:48:07.949185+01:00 zefir kernel: [431591.393774] BTRFS
> info (device sda2): found 8 extents, stage: update data pointers
> 2024-12-09T08:48:10.169177+01:00 zefir kernel: [431593.614676] BTRFS
> info (device sda2): relocating block group 34723825975296 flags data|raid1
> 2024-12-09T08:48:33.781190+01:00 zefir kernel: [431617.225031] BTRFS
> info (device sda2): found 10 extents, stage: move data extents
> 2024-12-09T08:48:44.353165+01:00 zefir kernel: [431627.799342] BTRFS
> info (device sda2): found 10 extents, stage: update data pointers
> 2024-12-09T08:48:47.453174+01:00 zefir kernel: [431630.899246] BTRFS
> info (device sda2): relocating block group 34721678491648 flags data|raid1
> 
> But unallocated space didn't increase.
> 
> 

Why did you expect it to increase? To free space balance need to pack 
more extents into less chunks. In your case chunks are near to full and 
extents are relatively large, so chunks simply may not have enough free 
space to accommodate more extents. You just move extents around.

> 
> 
> 
> So I started to play with metadata optimization, that is with musage.
> 
> 
> 
> When I put limit=0, no blocks are reallocated.
> When I put limit=1 or limit=2 always one block is reallocated.
> When I put limit greater then no blocks are reallocated.
> 
> 
> See the test:
> 
> root@zefir:~# for lim in 0 1 2 3 4 5 6; do echo "lim=$lim"; for f in
> $(seq 5); do btrfs bala start -musage=30,limit=$lim /; done; done
> lim=0
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> lim=1
> Done, had to relocate 1 out of 8318 chunks
> Done, had to relocate 1 out of 8318 chunks
> Done, had to relocate 1 out of 8318 chunks
> Done, had to relocate 1 out of 8318 chunks
> Done, had to relocate 1 out of 8318 chunks
> lim=2
> Done, had to relocate 1 out of 8318 chunks
> Done, had to relocate 1 out of 8318 chunks
> Done, had to relocate 1 out of 8318 chunks
> Done, had to relocate 1 out of 8318 chunks
> Done, had to relocate 1 out of 8318 chunks
> lim=3
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> lim=4
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> lim=5
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> lim=6
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> Done, had to relocate 0 out of 8318 chunks
> 
> 
> 
> 
> root@zefir:~# btrfs bala start -musage=30,limit=1 /
> Done, had to relocate 1 out of 8318 chunks
> 
> root@zefir:~# dmesg -T | tail
> [Sat Dec 14 18:50:00 2024] BTRFS info (device sdb2): balance: start
> -musage=30,limit=6 -susage=30,limit=6
> [Sat Dec 14 18:50:00 2024] BTRFS info (device sdb2): balance: ended with
> status: 0
> [Sat Dec 14 18:50:00 2024] BTRFS info (device sdb2): balance: start
> -musage=30,limit=6 -susage=30,limit=6
> [Sat Dec 14 18:50:00 2024] BTRFS info (device sdb2): balance: ended with
> status: 0
> [Sat Dec 14 18:50:00 2024] BTRFS info (device sdb2): balance: start
> -musage=30,limit=6 -susage=30,limit=6
> [Sat Dec 14 18:50:00 2024] BTRFS info (device sdb2): balance: ended with
> status: 0
> [Sat Dec 14 18:50:42 2024] BTRFS info (device sdb2): balance: start
> -musage=30,limit=1 -susage=30,limit=1
> [Sat Dec 14 18:50:42 2024] BTRFS info (device sdb2): relocating block
> group 38091650760704 flags system|raid1
> [Sat Dec 14 18:50:43 2024] BTRFS info (device sdb2): found 91 extents,
> stage: move data extents
> [Sat Dec 14 18:50:44 2024] BTRFS info (device sdb2): balance: ended with
> status: 0
> 
> 
> 
> 
> During those all operations level of Unallocated space is not increasing.

Relocating one chunk simply moves extents from this chunk to another 
location. It does not free any chunk. You can only get more unallocated 
space when you are able to pack extents from two (or more) chunks into 
one chunk. Which is only possible if chunks are filled to 50%.

> What should i do next?
> 
> 

It looks like your filesystem is simply full. Do you have reasons to 
believe that it is not true?