Re: Upgraded from Buster to Bullseye, unmountable Btrfs filesystem

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2021/11/10 10:17, S. wrote:
> Hi there, I run OpenMediaVault on an old LaCiE NAS with an armel
> processor. It has two HDDs, with the Debian root on an EXT4 partition
> and a simple BtrFS RAID-1 using `/dev/sda3` + `/dev/sdb2` for the
> OpenMediaVault data. Both drives have a few bad blocks, but I assumed
> that the filesystem was working around them, because it was running fine
> for almost 2 years on Buster. The SMART report for both drives says
> `SMART overall-health self-assessment test result: PASSED`. Today I
> upgraded to OpenMediaVault 6 and Debian Bullseye, and the BtrFS volume
> is no longer mountable. Here are my attempts thus far:

Newer kernel (mostly since v5.11?) has much strict sanity check inside
btrfs, thus it can detects things which doesn't show up in older kernels.

>
> https://paste.debian.net/1218866/
>
> For the search engines, these are the key errors:
>
> -----------------------------
>
>      [   86.110770] BTRFS critical (device sda3): corrupt leaf: root=9
> block=170459136 slot=0, invalid key objectid, have 1101835439474057344
> expect to be aligned to 4096

The root is UUID tree, thus the objectid is part of the UUID.

The problem is, the key type is corrupted, from your fsck result:

 >    Invalid key type(EXTENT_ITEM) found in root(UUID_TREE)
 >    ignoring invalid key

This bad key type gets caught by tree-checker, and the mount is rejected.

Further more, EXTENT_ITEM_KEY value is 168, while the correct key types
in UUID tree should be UUID_KEY_SUBVOL (251) or
UUID_KEY_RECEVIED_SUBVOL(252).
Which doesn't seem to be an simple bit flip.

But I still recommend to do a memtest to rule out memory problem.


For the repair, we can rebuilt UUID tree, but unfortunately btrfs-progs
doesn't have such ability yet.

Meanwhile, you can prepare a build environment to build btrfs-progs, I
can soon craft a branch for you to re-init UUID tree to solve the
problem first.

To be extra safe, please provide the dump of your UUID tree:

# btrfs ins dump-tree -t uuid /dev/sda3

Thanks,
Qu

>      [   86.125317] BTRFS error (device sda3): block=170459136 read time
> tree block corruption detected
>      [   86.149595] BTRFS critical (device sda3): corrupt leaf: root=9
> block=170459136 slot=0, invalid key objectid, have 1101835439474057344
> expect to be aligned to 4096
>      [   86.164280] BTRFS error (device sda3): block=170459136 read time
> tree block corruption detected
>      [   86.173099] BTRFS warning (device sda3): failed to read root
> (objectid=9): -5
>      [   86.268589] BTRFS critical (device sda3): corrupt leaf: root=9
> block=170459136 slot=0, invalid key objectid, have 1101835439474057344
> expect to be aligned to 4096
>      [   86.283207] BTRFS error (device sda3): block=170459136 read time
> tree block corruption detected
>      [   86.298934] BTRFS critical (device sda3): corrupt leaf: root=9
> block=170459136 slot=0, invalid key objectid, have 1101835439474057344
> expect to be aligned to 4096
>      [   86.313575] BTRFS error (device sda3): block=170459136 read time
> tree block corruption detected
>      [   86.322394] BTRFS warning (device sda3): failed to read root
> (objectid=9): -5
>      [   86.363745] BTRFS error (device sda3): parent transid verify
> failed on 78086144 wanted 8060 found 8063
>      [   86.390901] BTRFS error (device sda3): parent transid verify
> failed on 78086144 wanted 8060 found 8063
>      [   86.414379] BTRFS error (device sda3): parent transid verify
> failed on 79216640 wanted 8061 found 8063
>      [   86.434284] BTRFS error (device sda3): parent transid verify
> failed on 79216640 wanted 8061 found 8063
>      [   86.465467] BTRFS warning (device sda3): couldn't read tree root
>      [   86.500332] BTRFS error (device sda3): open_ctree failed
>
> -----------------------------
>
>      root@OpenMediaVault:~# btrfs check /dev/sda3
>      Opening filesystem to check...
>      Checking filesystem on /dev/sda3
>      UUID: 4a057760-998c-4c66-aa6a-2a08c51d5299
>      [1/7] checking root items
>      [2/7] checking extents
>      Invalid key type(EXTENT_ITEM) found in root(UUID_TREE)
>      ignoring invalid key
>      [3/7] checking free space cache
>      [4/7] checking fs roots
>      [5/7] checking only csums items (without verifying data)
>      [6/7] checking root refs
>      [7/7] checking quota groups skipped (not enabled on this FS)
>      found 704343715840 bytes used, no error found
>      total csum bytes: 686922316
>      total tree bytes: 757006336
>      total fs tree bytes: 14794752
>      total extent tree bytes: 13795328
>      btree space waste bytes: 32551835
>      file data blocks allocated: 707684626432
>       referenced 707430318080
>
> -----------------------------
>
> Any suggestions? Thanks a lot.