Re: unable to remove device due to enospc

[Ross Vandegrift <ross@kallisti.us>]

Hi Chris,

On Sat, Jan 15, 2022 at 10:49:01PM -0800, Ross Vandegrift wrote:
> On Sat, Jan 15, 2022 at 05:39:12PM -0700, Chris Murphy wrote:
> > On Tue, Jan 11, 2022 at 12:41 AM Ross Vandegrift <ross@kallisti.us> wrote:
> > >
> > > Unallocated:
> > >    /dev/mapper/backup      7.24TiB
> > >    /dev/mapper/backup_a    2.55TiB
> > >    /dev/mapper/backup_b    2.55TiB
> > 
> > You might be running into this bug I mentioned today in another
> > thread. It's an overcommit related bug where the initial overcommit is
> > based on a single device's unallocated space, in this case the 7T
> > device - but then later logic results in ENOSPC because there aren't
> > two devices that can handle that amount of overcommit. If you  can
> > remove /dev/mapper/backup, leaving just _a and _b with equal
> > unallocated, then try to reproduce. If you can't, you've likely hit
> > the bug I'm thinking of. If you still can, then you're hitting
> > something different.
> > 
> > However, since these unallocated values are orders of magnitude bigger
> > than discussed in the other thread, and orders of magnitude more space
> > than is needed to successfully create new chunks on multiple devices -
> > I'm not sure. Also I'm not sure without going through the change log
> > for 5.15 and 5.16 if this is fixed in one of those so the easiest
> > thing to do is try and reproduce it with 5.16. If ENOSPC happens, then
> > try to remove the device with 7T unallocated.
> 
> Same behavior with 5.16:

Any other ideas on this issue?  Would images of these devices be interesting or
useful?

Ross


> 
> $ uname -a
> Linux vanvanmojo 5.16.0 #3 SMP PREEMPT Sat Jan 15 21:39:38 PST 2022 x86_64 GNU/Linux
> $ sudo btrfs device remove /dev/mapper/backup /mnt/backup
> ERROR: error removing device '/dev/mapper/backup': No space left on device
> WARNING: Multiple block group profiles detected, see 'man btrfs(5)'.
> WARNING:   Data: single, raid1
> $ sudo btrfs fi usage /mnt/backup
> Overall:
>     Device size:	          18.80TiB
>     Device allocated:	          12.52TiB
>     Device unallocated:	           6.27TiB
>     Device missing:	             0.00B
>     Used:		          12.47TiB
>     Free (estimated):	           3.17TiB	(min: 3.16TiB)
>     Free (statfs, df):	           5.63TiB
>     Data ratio:	                      1.99
>     Metadata ratio:	              2.00
>     Global reserve:	         512.00MiB	(used: 0.00B)
>     Multiple profiles:	               yes	(data)
> 
> Data,single: Size:35.00GiB, Used:34.87GiB (99.63%)
>    /dev/mapper/backup	  35.00GiB
> 
> Data,RAID1: Size:6.23TiB, Used:6.20TiB (99.57%)
>    /dev/mapper/backup_b    6.23TiB
>    /dev/mapper/backup_a    6.23TiB
> 
> Metadata,RAID1: Size:16.00GiB, Used:14.85GiB (92.80%)
>    /dev/mapper/backup_b    16.00GiB
>    /dev/mapper/backup_a    16.00GiB
> 
> System,RAID1: Size:32.00MiB, Used:900.00KiB (2.75%)
>    /dev/mapper/backup_b    32.00MiB
>    /dev/mapper/backup_a    32.00MiB
> 
> Unallocated:
>    /dev/mapper/backup	 715.00GiB
>    /dev/mapper/backup_b    2.79TiB
>    /dev/mapper/backup_a    2.79TiB
> 
> 
> Here's the dmesg output from the removal attempt:
> [  581.830800] BTRFS info (device dm-16): relocating block group 749505347584 flags data
> [  648.464705] BTRFS info (device dm-16): relocating block group 419866607616 flags data
> [  724.635005] BTRFS info (device dm-16): relocating block group 411276673024 flags data
> [  752.047813] BTRFS info (device dm-16): relocating block group 403760480256 flags data
> [  756.749221] BTRFS info (device dm-16): relocating block group 395170545664 flags data
> [  760.778891] BTRFS info (device dm-16): relocating block group 387654352896 flags data
> [  787.116859] BTRFS info (device dm-16): relocating block group 379064418304 flags data
> [  812.351256] BTRFS info (device dm-16): relocating block group 749505347584 flags data
> [  838.589207] BTRFS info (device dm-16): relocating block group 419866607616 flags data
> [  885.964951] BTRFS info (device dm-16): relocating block group 411276673024 flags data
> [  901.889074] BTRFS info (device dm-16): relocating block group 403760480256 flags data
> [  904.306416] BTRFS info (device dm-16): relocating block group 395170545664 flags data
> [  906.514675] BTRFS info (device dm-16): relocating block group 387654352896 flags data
> [  923.680623] BTRFS info (device dm-16): relocating block group 379064418304 flags data
> 
> 
> Ross