[bug] Balance fails due to ENOSPC

[bug] Balance fails due to ENOSPC

[Anand Jain]

Balance fails due to ENOSPC on 5.11.0-rc2+.


[ 2460.219094] BTRFS info (device sdb6): balance: start -f 
-dconvert=single -mconvert=single -sconvert=single
[ 2460.219224] BTRFS info (device sdb6): relocating block group 
194572713984 flags data
[ 2460.241299] ------------[ cut here ]------------
[ 2460.241302] BTRFS: Transaction aborted (error -28)
[ 2460.241328] WARNING: CPU: 3 PID: 4135 at fs/btrfs/volumes.c:3003 
btrfs_remove_chunk+0x60e/0x6c0 [btrfs]
::
[ 2460.241447] RIP: 0010:btrfs_remove_chunk+0x60e/0x6c0 [btrfs]
::
[ 2460.241497] Call Trace:
[ 2460.241500]  btrfs_relocate_chunk+0x9d/0xd0 [btrfs]
[ 2460.241533]  __btrfs_balance+0x413/0xa30 [btrfs]
[ 2460.241566]  btrfs_balance+0x436/0x4d0 [btrfs]
[ 2460.241597]  btrfs_ioctl_balance+0x2d5/0x380 [btrfs]
[ 2460.241647]  btrfs_ioctl+0x565/0x22b0 [btrfs]
[ 2460.241693]  ? selinux_file_ioctl+0x174/0x220
[ 2460.241698]  ? handle_mm_fault+0xd7/0x2b0
[ 2460.241702]  __x64_sys_ioctl+0x91/0xc0
[ 2460.241705]  ? __x64_sys_ioctl+0x91/0xc0
[ 2460.241707]  do_syscall_64+0x38/0x50
[ 2460.241711]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
::
[ 2460.241735] ---[ end trace 95b8b7b84503a477 ]---
[ 2460.241738] BTRFS: error (device sdb6) in btrfs_remove_chunk:3003: 
errno=-28 No space left
[ 2460.241743] BTRFS info (device sdb6): forced readonly
[ 2460.242646] BTRFS info (device sdb6): 1 enospc errors during balance
[ 2460.242649] BTRFS info (device sdb6): balance: ended with status: -30


btrfs fi us /Volumes/
Overall:
     Device size:		 154.97GiB
     Device allocated:		 154.97GiB
     Device unallocated:		   2.00MiB
     Device missing:		     0.00B
     Used:			 147.91GiB
     Free (estimated):		   2.75GiB	(min: 2.75GiB)
     Data ratio:			      1.99
     Metadata ratio:		      1.52
     Global reserve:		 183.72MiB	(used: 0.00B)

Data,single: Size:894.00MiB, Used:0.00B (0.00%)
    /dev/sdb6	 894.00MiB

Data,RAID1: Size:75.53GiB, Used:73.23GiB (96.96%)
    /dev/sdb6	  75.53GiB
    /dev/sdc2	  75.53GiB

Metadata,single: Size:964.00MiB, Used:0.00B (0.00%)
    /dev/sdb6	 964.00MiB

Metadata,RAID1: Size:1.00GiB, Used:737.25MiB (72.00%)
    /dev/sdb6	   1.00GiB
    /dev/sdc2	   1.00GiB

System,single: Size:32.00MiB, Used:0.00B (0.00%)
    /dev/sdb6	  32.00MiB

System,RAID1: Size:32.00MiB, Used:16.00KiB (0.05%)
    /dev/sdb6	  32.00MiB
    /dev/sdc2	  32.00MiB

Unallocated:
    /dev/sdb6	   1.00MiB
    /dev/sdc2	   1.00MiB

Re: [bug] Balance fails due to ENOSPC

[Qu Wenruo]

On 2021/1/12 下午6:35, Anand Jain wrote:
>
>
> Balance fails due to ENOSPC on 5.11.0-rc2+.
>
>
> [ 2460.219094] BTRFS info (device sdb6): balance: start -f
> -dconvert=single -mconvert=single -sconvert=single
> [ 2460.219224] BTRFS info (device sdb6): relocating block group
> 194572713984 flags data
> [ 2460.241299] ------------[ cut here ]------------
> [ 2460.241302] BTRFS: Transaction aborted (error -28)
> [ 2460.241328] WARNING: CPU: 3 PID: 4135 at fs/btrfs/volumes.c:3003
> btrfs_remove_chunk+0x60e/0x6c0 [btrfs]
> ::
> [ 2460.241447] RIP: 0010:btrfs_remove_chunk+0x60e/0x6c0 [btrfs]
> ::
> [ 2460.241497] Call Trace:
> [ 2460.241500]  btrfs_relocate_chunk+0x9d/0xd0 [btrfs]
> [ 2460.241533]  __btrfs_balance+0x413/0xa30 [btrfs]
> [ 2460.241566]  btrfs_balance+0x436/0x4d0 [btrfs]
> [ 2460.241597]  btrfs_ioctl_balance+0x2d5/0x380 [btrfs]
> [ 2460.241647]  btrfs_ioctl+0x565/0x22b0 [btrfs]
> [ 2460.241693]  ? selinux_file_ioctl+0x174/0x220
> [ 2460.241698]  ? handle_mm_fault+0xd7/0x2b0
> [ 2460.241702]  __x64_sys_ioctl+0x91/0xc0
> [ 2460.241705]  ? __x64_sys_ioctl+0x91/0xc0
> [ 2460.241707]  do_syscall_64+0x38/0x50
> [ 2460.241711]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> ::
> [ 2460.241735] ---[ end trace 95b8b7b84503a477 ]---
> [ 2460.241738] BTRFS: error (device sdb6) in btrfs_remove_chunk:3003:
> errno=-28 No space left
> [ 2460.241743] BTRFS info (device sdb6): forced readonly
> [ 2460.242646] BTRFS info (device sdb6): 1 enospc errors during balance
> [ 2460.242649] BTRFS info (device sdb6): balance: ended with status: -30
>
>
> btrfs fi us /Volumes/
> Overall:
>      Device size:         154.97GiB
>      Device allocated:         154.97GiB
>      Device unallocated:           2.00MiB
>      Device missing:             0.00B
>      Used:             147.91GiB
>      Free (estimated):           2.75GiB    (min: 2.75GiB)
>      Data ratio:                  1.99
>      Metadata ratio:              1.52
>      Global reserve:         183.72MiB    (used: 0.00B)
>
> Data,single: Size:894.00MiB, Used:0.00B (0.00%)
>     /dev/sdb6     894.00MiB
>
> Data,RAID1: Size:75.53GiB, Used:73.23GiB (96.96%)
>     /dev/sdb6      75.53GiB
>     /dev/sdc2      75.53GiB
>
> Metadata,single: Size:964.00MiB, Used:0.00B (0.00%)
>     /dev/sdb6     964.00MiB
>
> Metadata,RAID1: Size:1.00GiB, Used:737.25MiB (72.00%)
>     /dev/sdb6       1.00GiB
>     /dev/sdc2       1.00GiB

Metadata is only 1G, with 737M used, and 183 reserved as GlobalRSV.

Although the GlobalRSV is a little smaller than older kernel, but it
should be more or less enough.

In theory, we should be able to modify the fs stealing from GlobalRSV,
and since btrfs_remove_chunk:3003 is updating device time, either it
means we didn't really reserve enough metadata when we start the trans,
or we didn't do the steal when we really need.

Would you like to provide the dmesg if you can reproduce with "-o
enospc_debug" mount option?

Thanks,
Qu
>
> System,single: Size:32.00MiB, Used:0.00B (0.00%)
>     /dev/sdb6      32.00MiB
>
> System,RAID1: Size:32.00MiB, Used:16.00KiB (0.05%)
>     /dev/sdb6      32.00MiB
>     /dev/sdc2      32.00MiB
>
> Unallocated:
>     /dev/sdb6       1.00MiB
>     /dev/sdc2       1.00MiB