Re: 'btrfs rescue' command (recommended by btrfs check) fails on old BTRFS RAID1 on (currently) openSUSE Leap 15.3

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2022/5/18 18:38, Johannes Kastl wrote:
> Hi Qu,
>
> TL;DR: took a while until I had all of the data backed up properly and
> had some time to test this. Unfortunately the filesystem is now no
> longer mountable...
>
> Any ideas?
>
> Johannes
>
> On 24.04.22 at 11:21 Qu Wenruo wrote:
>> On 2022/4/24 17:10, Johannes Kastl wrote:
>
>>> So would resizing the filesystem (to 8GiB) workaround this "limitation",
>>> so afterwards it could properly fix the device size?
>>
>> I'm not yet sure if it's a bug in progs causing false ENOSPC, or really
>> there isn't many space left.
>>
>> For the former case, no matter how much free space you have, it won't
>> help.
>>
>> For the latter case, it would definitely help.
>
> So, I deleted the partitions on both disks and re-created them with the
> new (bigger size), keeping the start sector and the btrfs signature intact.
>
> I could then resize both disks to the same value successfully. At least,
> the commands ran without errors.
>
> Fixing the device size fails nonetheless (see below). And I can no
> longer mount the filesystem, when I try I find this in the logs:
>
>> [87396.889043] BTRFS error (device sdb1): super_total_bytes
>> 15393162784768 mismatch with fs_devices total_rw_bytes 15393162788864
>> [87396.889974] BTRFS error (device sdb1): failed to read chunk tree: -22
>> [87396.892741] BTRFS error (device sdb1): open_ctree failed
>
> (Don't get confused by sdb1, this is from a rescue system with only some
> HDDs attached)
>
> Fixing the device-size on Leap 15.3:
>> # btrfs filesystem show /mnt/DUMBO_BACKUP_4TB/
>> Label: 'DUMBO_BACKUP_4TB'  uuid: 50651b41-bf33-47e7-8a08-afbc71ba0bf8
>>         Total devices 2 FS bytes used 3.17TiB
>>         devid    1 size 7.00TiB used 3.64TiB path /dev/sdd1
>>         devid    2 size 7.00TiB used 3.63TiB path /dev/sdc1

That's super weird, we have tons of unallocated space.

So definitely something wrong in btrfs-progs.
Normally `btrfs fi usage` would provide more info, but it needs the fs
to be mountable.

Can you prepare a building environment for btrfs-progs?

I can update the code to skip transaction commit so that we won't be
bother with -ENOSPC at all.

And since we're not really doing any metadata update, we don't really
need any new space.

And after your building environment prepared, you can fetch this branch
to compile the btrfs-progs and try to use the compiled `btrfs` command
to rescue the device again.

https://github.com/adam900710/btrfs-progs/tree/dirty_fix

I did some local tests, it shows no problem, but not sure if it would
work for you.

Thanks,
Qu

>>
>> # umount /mnt/DUMBO_BACKUP_4TB
>> # btrfs rescue fix-device-size /dev/sdd1
>> Unable to find block group for 0
>> Unable to find block group for 0
>> Unable to find block group for 0
>> transaction.c:189: btrfs_commit_transaction: BUG_ON `ret` triggered,
>> value -28
>> btrfs(+0x51f99)[0x55edf7a43f99]
>> btrfs(+0x525a9)[0x55edf7a445a9]
>> btrfs(btrfs_fix_super_size+0x98)[0x55edf7a2f438]
>> btrfs(btrfs_fix_device_and_super_size+0x84)[0x55edf7a2f584]
>> btrfs(+0x6ceee)[0x55edf7a5eeee]
>> btrfs(main+0x8e)[0x55edf7a1108e]
>> /lib64/libc.so.6(__libc_start_main+0xef)[0x7f672ad962bd]
>> btrfs(_start+0x2a)[0x55edf7a1128a]
>> Aborted (core dumped)
>> #
>
> I tested fixing the device-id by booting from a Tumbleweed rescue stick,
> running kernel 5.16 with btrfsprogs 5.16. This also fails, but spits out
> an error message that is a little different:
>
>  > [...]
>> Unable to find block group for 0
>> Error: failed to commit current transaction: -28 (No space left on
>> device)
>> No device size related problem found
>> ERROR: commit_root already set when starting transaction
>> extent buffer leak: start ... len 16384
>
> (I had to type this off of the screen)
>
> As the mounting failed with an error related to chunks, I tried the
> btrfs rescue chunk-recover command, but that also aborts and dumps a
> core, even on Tumbleweed with kernel 5.16...
>
> The error messages look something like this:
>  > Unable to find block group for 0
>  > Unable to find block group for 0
>  > Unable to find block group for 0
>
> followed by a "...BUG_ON `ret` triggered, value -28"
>
> So this could all be related to -28 (No space left on device)?
>