Re: Root level mismatch after sudden shutdown

Chris Murphy
2022-02-09T21:04:14Z

On Wed, Feb 9, 2022 at 5:10 AM BERGUE Kevin
<kevin.bergue-externe@hemeria-group.com> wrote:
>
> Hello everyone.
>
>
> After a sudden shutdown my btrfs partition seems to be corrupted. After a few hours of reading documentation and trying various repair methods I found an error message I can't find anywhere so I'm sending it your way hoping you can at least explain what the issue is. The disk was running with linux 5.16.5 at the moment of the crash, my recovery environnement is a linux 5.15.16 machine with btrfs-progs v5.16.
>
>
> To retrace my steps a bit:
>
> - I tried to mount my partition normally:
> # mount /dev/mapper/SSD-Root /mnt/broken/
> mount: /mnt/broken: wrong fs type, bad option, bad superblock on /dev/mapper/SSD-Root, missing codepage or helper program, or other error.
>
> - I then looked at the relevant logs from dmesg:
> # dmesg
> [ 2118.381387] BTRFS info (device dm-1): flagging fs with big metadata feature
> [ 2118.381394] BTRFS info (device dm-1): disk space caching is enabled
> [ 2118.381395] BTRFS info (device dm-1): has skinny extents
> [ 2118.384626] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999
> [ 2118.384900] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999
> [ 2118.384905] BTRFS warning (device dm-1): failed to read root (objectid=4): -5
> [ 2118.385304] BTRFS error (device dm-1): open_ctree failed
>
> - After some reading about the "parent transid verify failed" errors I tried to mount the volume with the usebackuproot flag:
> # mount -t btrfs -o ro,rescue=usebackuproot /dev/mapper/SSD-Root /mnt/broken/
> mount: /mnt/broken: wrong fs type, bad option, bad superblock on /dev/mapper/SSD-Root, missing codepage or helper program, or other error.
> And the dmesg content:
> [ 2442.117867] BTRFS info (device dm-1): flagging fs with big metadata feature
> [ 2442.117871] BTRFS info (device dm-1): trying to use backup root at mount time
> [ 2442.117872] BTRFS info (device dm-1): disk space caching is enabled
> [ 2442.117873] BTRFS info (device dm-1): has skinny extents
> [ 2442.123056] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999
> [ 2442.123344] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999
> [ 2442.123348] BTRFS warning (device dm-1): failed to read root (objectid=4): -5
> [ 2442.124743] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999
> [ 2442.124939] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999
> [ 2442.124942] BTRFS warning (device dm-1): failed to read root (objectid=4): -5
> [ 2442.125196] BTRFS critical (device dm-1): corrupt leaf: block=1869863370752 slot=97 extent bytenr=920192651264 len=4096 invalid generation, have 527002 expect (0, 527001]
> [ 2442.125201] BTRFS error (device dm-1): block=1869863370752 read time tree block corruption detected
> [ 2442.125500] BTRFS critical (device dm-1): corrupt leaf: block=1869863370752 slot=97 extent bytenr=920192651264 len=4096 invalid generation, have 527002 expect (0, 527001]
> [ 2442.125502] BTRFS error (device dm-1): block=1869863370752 read time tree block corruption detected
> [ 2442.125508] BTRFS warning (device dm-1): couldn't read tree root
> [ 2442.125806] BTRFS critical (device dm-1): corrupt leaf: block=1869866401792 slot=117 extent bytenr=906206486528 len=4096 invalid generation, have 527003 expect (0, 527002]
> [ 2442.125808] BTRFS error (device dm-1): block=1869866401792 read time tree block corruption detected
> [ 2442.126174] BTRFS critical (device dm-1): corrupt leaf: block=1869866401792 slot=117 extent bytenr=906206486528 len=4096 invalid generation, have 527003 expect (0, 527002]
> [ 2442.126175] BTRFS error (device dm-1): block=1869866401792 read time tree block corruption detected
> [ 2442.126184] BTRFS warning (device dm-1): couldn't read tree root
> [ 2442.126599] BTRFS error (device dm-1): open_ctree failed

What do you get for uname -a? What are the mount options? In
particular I'm wondering if this drive is subject to discards, which
currently will render usebackuproots useless. And how old is the file
system?

If ro,rescue=usebackuproot doesn't work, it's not likely repairable,
but might still be something btrfs restore can scrape *if* you haven't
tried any repairs first. Repairs write changes to disk and can make
things worse and harder to recover. You pretty much have to go
straight to the scrape data off the drive step before the repair step
because it is lower risk and more reliable.

The central problem is some lower layer mishandled write order, most
likely by dropping flush/FUA, i.e. confirming to btrfs that flush/FUA
command succeeded when in fact critical metadata has not yet made it
to persistent media. And there is no way for Btrfs to know this until
it's too late.

>
> - I then tried a check:
> # btrfs check /dev/mapper/SSD-Root
> Opening filesystem to check...
> parent transid verify failed on 1869491683328 wanted 526959 found 526999
> parent transid verify failed on 1869491683328 wanted 526959 found 526999
> parent transid verify failed on 1869491683328 wanted 526959 found 526999
> Ignoring transid failure
> ERROR: root [4 0] level 0 does not match 1

What do you get for

btrfs rescue super -v $dev
btrfs insp dump-s -f $dev
btrfs insp dump-t -b 1869491683328 $dev
btrfs insp dump-t -b 1869863370752 $dev
btrfs insp dump-t -b 1869866401792 $dev

>
> Couldn't setup device tree
> ERROR: cannot open file system
>
>
> I think the "root [4 0] level 0 does not match 1" error is the real cuprit but I can't seem to find any info on this message anywhere. I tried a bunch of other commands including:
> - btrfs rescue zero-log
> - btrfs rescue chunk-recover
> - btrfs check --repair
> - btrfs rescue super-recover
> - btrfs check --repair with the tree root found by btrfs-find-root
> - btrfs check --repair --init-csum-tree --init-extent-tree
> - btrfs restore
>
> I'm aware I probably executed some commands that don't make a lot of sense in my context but all of them failed with either the "root [4 0] level 0 does not match 1" message or a more generic "could not open ctree" or equivalent.

--repair, but especially --init-extent-tree can make it next to
impossible for 'btrfs restore' command to work. It's not obvious that
trying to fix btrfs, can make it harder for the scrape tool to work.

--
Chris Murphy