Re: mount: can't read superblock on - corrupt leaf - read time tree block corruption detected

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

在 2024/10/3 20:02, cwalou@gmail.com 写道:
> Le 03/10/2024 à 12:12, Qu Wenruo a écrit :
>>
>>
>> 在 2024/10/3 18:50, cwalou@gmail.com 写道:
>>> Le 03/10/2024 à 10:08, Qu Wenruo a écrit :
>>>>
>>>>
>>>> 在 2024/10/3 17:02, cwalou@gmail.com 写道:
>>>>> Hello.
>>>>>
>>>>> A 4TB drive taken out of a synology NAS. When I try to mount it, it
>>>>> won't. This is what I did :
>>>>
>>>> Synology has out-of-tree features that upstream kernel doesn't support.
>>>>
>>>> Please ask the vendor for their support.
>>>>
>>>> Thanks,
>>>> Qu
>>>
>>> Thank you for your answer.
>>>
>>> Just for my general knowledge, can you explain me what "out-of-tree
>>> features" means ?
>>
>> Out-of-tree means it's not merged into the upstream Linux kernel.
>>
>> Furthermore, they do not even bother to put a special compat-ro/
>> incompact flags into the super block.
>>
>> So upstream kernel will not even know the fs has unsupported features,
>> until the tree-checker checks the inode flags.
>>
>> Thanks,
>> Qu
>>
>
> Before I start a quest at Synology.
> Is there any way to get around the "can't read superblock on /dev/
> mapper/vg1000-lv." problem?
>
> I mean, is there an option I can pass to 'mount' to skip superblock
> check? Or is there a way to tell him a superblock is available at
> "65536" ("67108864" or "274877906944" which "btrfs rescue super-recover
> -v" tells me are good, see below) ?

The problem is not superblock, but the extra inode flags introduced:

Oct  2 17:31:01 user-NUC10i7FNH kernel: [ 1701.650935] BTRFS critical
(device dm-0: state C): corrupt leaf: root=257 block=2691220668416
slot=0 ino=6039235, unknown incompat flags detected: 0x40000000
Oct  2 17:31:01 user-NUC10i7FNH kernel: [ 1701.650969] BTRFS error
(device dm-0: state C): read time tree block corruption detected on
logical 2691220668416 mirror 1

So far there is no way to disable tree-checker (the sanity checks), as
it brings so many benefits that we can not ignore.

>
> My apologies if my questions seem naive. But file systems and storage
> media are not really my specialty. I'm just trying to help an
> acquaintance who has been diligently backing up his data but is now
> stuck restoring his data.
>
> I think if we solve this problem on this public list, it may help more
> people. That's why even I need to ask synology for help, I'll give
> feedback here.

Unfortunately, to the fs those unknown flags can also be an indicator of
bitflip, thus tree-checker rejects anything that it can not understand.

Considering how frequently those hardware memory bitflips are reported,
we do not provide a way to disable such comprehensive checks.

So this means any new feature introduced by non-upstream contributor
will be rejected, even if most of the fs can still be understood by
upstream kernel.

Thanks,
Qu
>
> Nevertheless, thank you for the answers you already gave me, for taking
> the time to answer me.
>
>>>
>>> I'll ask synology what's happening. Once I'll find a solution (if one
>>> day I find one) I'll let know here.
>>>
>>> Kind regards,
>>>
>>> Walou.
>>>
>>>
>>>>>
>>>>>
>>>>> root@user-NUC10i7FNH:~# fdisk -l /dev/sda
>>>>> Disk /dev/sda: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
>>>>> Disk model: 001-2MA101
>>>>> Units: sectors of 1 * 512 = 512 bytes
>>>>> Sector size (logical/physical): 512 bytes / 512 bytes
>>>>> I/O size (minimum/optimal): 512 bytes / 512 bytes
>>>>> Disklabel type: gpt
>>>>> Disk identifier: B7B80A4B-0294-44FD-A368-74B0455D6AF2
>>>>>
>>>>> Device          Start        End    Sectors   Size Type
>>>>> /dev/sda1        8192   16785407   16777216     8G Linux RAID
>>>>> /dev/sda2    16785408   20979711    4194304     2G Linux RAID
>>>>> /dev/sda5    21257952 1965122911 1943864960 926.9G Linux RAID
>>>>> /dev/sda6  1965139008 7813827135 5848688128   2.7T Linux RAID
>>>>>
>>>>>
>>>>> root@user-NUC10i7FNH:~# lsblk
>>>>> NAME            MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
>>>>> sda               8:0    0   3.6T  0 disk
>>>>> |-sda1            8:1    0     8G  0 part
>>>>> |-sda2            8:2    0     2G  0 part
>>>>> |-sda5            8:5    0 926.9G  0 part
>>>>> | `-md2           9:2    0 926.9G  0 raid1
>>>>> |   `-vg1000-lv 252:0    0   3.6T  0 lvm
>>>>> `-sda6            8:6    0   2.7T  0 part
>>>>>    `-md3           9:3    0   2.7T  0 raid1
>>>>>      `-vg1000-lv 252:0    0   3.6T  0 lvm
>>>>>
>>>>>
>>>>> root@user-NUC10i7FNH:~# cat /proc/mdstat
>>>>> Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
>>>>> md3 : active (auto-read-only) raid1 sda6[1]
>>>>>        2924343040 blocks super 1.2 [2/1] [_U]
>>>>>
>>>>> md2 : active raid1 sda5[3]
>>>>>        971931456 blocks super 1.2 [2/1] [U_]
>>>>>
>>>>> unused devices: <none>
>>>>>
>>>>>
>>>>> root@user-NUC10i7FNH:~# lvm pvscan
>>>>>    WARNING: PV /dev/md2 in VG vg1000 is using an old PV header, modify
>>>>> the VG to update.
>>>>>    WARNING: PV /dev/md3 in VG vg1000 is using an old PV header, modify
>>>>> the VG to update.
>>>>>    PV /dev/md2   VG vg1000          lvm2 [926.90 GiB / 0    free]
>>>>>    PV /dev/md3   VG vg1000          lvm2 [2.72 TiB / 0    free]
>>>>>    Total: 2 [<3.63 TiB] / in use: 2 [<3.63 TiB] / in no VG: 0 [0   ]
>>>>>
>>>>> root@user-NUC10i7FNH:~# lvm vgscan
>>>>>    WARNING: PV /dev/md2 in VG vg1000 is using an old PV header, modify
>>>>> the VG to update.
>>>>>    WARNING: PV /dev/md3 in VG vg1000 is using an old PV header, modify
>>>>> the VG to update.
>>>>>    Found volume group "vg1000" using metadata type lvm2
>>>>>
>>>>> root@user-NUC10i7FNH:~# lvm lvscan
>>>>>    WARNING: PV /dev/md2 in VG vg1000 is using an old PV header, modify
>>>>> the VG to update.
>>>>>    WARNING: PV /dev/md3 in VG vg1000 is using an old PV header, modify
>>>>> the VG to update.
>>>>>    ACTIVE            '/dev/vg1000/lv' [<3.63 TiB] inherit
>>>>>
>>>>>
>>>>> root@user-NUC10i7FNH:~# mount -t btrfs -o rescue=all,ro /dev/
>>>>> vg1000/lv /
>>>>> mnt/test/
>>>>> mount: /mnt/test: can't read superblock on /dev/mapper/vg1000-lv.
>>>>>
>>>>>
>>>>> root@user-NUC10i7FNH:~# ll /dev/vg1000/lv /dev/mapper/vg1000-lv
>>>>> lrwxrwxrwx 1 root root 7 oct.   2 17:34 /dev/mapper/vg1000-lv -
>>>>> > ../dm-0
>>>>> lrwxrwxrwx 1 root root 7 oct.   2 17:34 /dev/vg1000/lv -> ../dm-0
>>>>>
>>>>>
>>>>> root@user-NUC10i7FNH:~# tail log/kern.log
>>>>> Oct  2 17:30:57 user-NUC10i7FNH kernel: [ 1697.255079] BTRFS: device
>>>>> label 2017.12.01-16:57:32 v15217 devid 1 transid 15800483 /dev/mapper/
>>>>> vg1000-lv scanned by mount (2939)
>>>>> Oct  2 17:30:57 user-NUC10i7FNH kernel: [ 1697.257012] BTRFS info
>>>>> (device dm-0): first mount of filesystem
>>>>> 320f5288-777d-43eb-84e3-4ac70573ec6b
>>>>> Oct  2 17:30:57 user-NUC10i7FNH kernel: [ 1697.257061] BTRFS info
>>>>> (device dm-0): using crc32c (crc32c-intel) checksum algorithm
>>>>> Oct  2 17:30:57 user-NUC10i7FNH kernel: [ 1697.257079] BTRFS info
>>>>> (device dm-0): disk space caching is enabled
>>>>> Oct  2 17:31:01 user-NUC10i7FNH kernel: [ 1701.650935] BTRFS critical
>>>>> (device dm-0: state C): corrupt leaf: root=257 block=2691220668416
>>>>> slot=0 ino=6039235, unknown incompat flags detected: 0x40000000
>>>>> Oct  2 17:31:01 user-NUC10i7FNH kernel: [ 1701.650969] BTRFS error
>>>>> (device dm-0: state C): read time tree block corruption detected on
>>>>> logical 2691220668416 mirror 1
>>>>> Oct  2 17:31:01 user-NUC10i7FNH kernel: [ 1701.654160] BTRFS critical
>>>>> (device dm-0: state C): corrupt leaf: root=257 block=2691220668416
>>>>> slot=0 ino=6039235, unknown incompat flags detected: 0x40000000
>>>>> Oct  2 17:31:01 user-NUC10i7FNH kernel: [ 1701.654189] BTRFS error
>>>>> (device dm-0: state C): read time tree block corruption detected on
>>>>> logical 2691220668416 mirror 2
>>>>> Oct  2 17:31:01 user-NUC10i7FNH kernel: [ 1701.654337] BTRFS info
>>>>> (device dm-0: state C): last unmount of filesystem
>>>>> 320f5288-777d-43eb-84e3-4ac70573ec6b
>>>>>
>>>>>
>>>>> root@user-NUC10i7FNH:~# btrfs rescue super-recover -v /dev/vg1000/lv
>>>>> All Devices:
>>>>>          Device: id = 1, name = /dev/vg1000/lv
>>>>>
>>>>> Before Recovering:
>>>>>          [All good supers]:
>>>>>                  device name = /dev/vg1000/lv
>>>>>                  superblock bytenr = 65536
>>>>>
>>>>>                  device name = /dev/vg1000/lv
>>>>>                  superblock bytenr = 67108864
>>>>>
>>>>>                  device name = /dev/vg1000/lv
>>>>>                  superblock bytenr = 274877906944
>>>>>
>>>>>          [All bad supers]:
>>>>>
>>>>> All supers are valid, no need to recover
>>>>>
>>>>>
>>>>> root@user-NUC10i7FNH:~# btrfs rescue zero-log /dev/vg1000/lv
>>>>> Clearing log on /dev/vg1000/lv, previous log_root 0, level 0
>>>>>
>>>>>
>>>>> root@user-NUC10i7FNH:~# btrfs check /dev/vg1000/lv
>>>>> Opening filesystem to check...
>>>>> Checking filesystem on /dev/vg1000/lv
>>>>> UUID: 320f5288-777d-43eb-84e3-4ac70573ec6b
>>>>> [1/7] checking root items
>>>>> [2/7] checking extents
>>>>> Invalid key type(BLOCK_GROUP_ITEM) found in root(202)
>>>>> ignoring invalid key
>>>>> Invalid key type(BLOCK_GROUP_ITEM) found in root(202)
>>>>> [...line repeated many times
>>>>> Invalid key type(BLOCK_GROUP_ITEM) found in root(202)
>>>>> ignoring invalid key
>>>>> Invalid key type(BLOCK_GROUP_ITEM) found in root(202)
>>>>> ignoring invalid key
>>>>> [3/7] checking free space cache
>>>>> [4/7] checking fs roots
>>>>> [5/7] checking only csums items (without verifying data)
>>>>> [6/7] checking root refs
>>>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>>>> found 2726275964928 bytes used, no error found
>>>>> total csum bytes: 839025944
>>>>> total tree bytes: 3015049216
>>>>> total fs tree bytes: 1991966720
>>>>> total extent tree bytes: 95895552
>>>>> btree space waste bytes: 555710555
>>>>> file data blocks allocated: 3567579688960
>>>>>   referenced 2977409900544
>>>>>
>>>>>
>>>>> root@user-NUC10i7FNH:~# btrfs property get /dev/mapper/vg1000-lv
>>>>> label=2017.12.01-16:57:32 v15217
>>>>>
>>>>>
>>>>> root@user-NUC10i7FNH:~# btrfs version
>>>>> btrfs-progs v5.16.2
>>>>>
>>>>>
>>>>> The most surprising is that on a Windows 10, "DiskInternals Linux
>>>>> Reader" (a paid software) shows me the content of this disk (and
>>>>> asks me
>>>>> to pay for copying the data).
>>>>>
>>>>>
>>>>> Any idea ?
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>