btrfs repair and zero-log doesn't seem to do anything

btrfs repair and zero-log doesn't seem to do anything

[Calle Kabo]

[-- Attachment #1: Type: text/plain, Size: 3922 bytes --]

Hi,

I've got a Netgear NAS with an encrypted 4 disk RAID5 setup. One of the
hard drives was emailing me about errors so I swapped it out before it
died. While syncing another disk died. So I've cloned the disk I
replaced before it died and managed to get the array running again. You
can read more about that here:
https://community.netgear.com/t5/Using-your-ReadyNAS/Re-adding-disk-to-volume/m-p/1179982/highlight/false

But now BTRFS won't mount. I'd like to mount it read only so I can move
the stuff to another drive and then format the NAS into a new array. If
there's some data loss that's fine, the most important stuff is already
backed up, but there are a few things that aren't backed up that I find
it worth the effort of trying to restore if at all possible.

Mandatory information:
root@MyNAS:~# uname -a
Linux MyNAS 4.1.30.armada.1 #1 SMP Thu Sep 22 16:39:49 PDT 2016 armv7l
GNU/Linux
root@MyNAS:~# btrfs --version
btrfs-progs v4.7.3
root@MyNAS:~# btrfs fi show
Label: '0e35246e:data'  uuid: b6283e48-ebd3-4fdb-8976-b66b1f0868bc
        Total devices 1 FS bytes used 3.99TiB
        devid    1 size 8.17TiB used 4.22TiB path /dev/dm-0

Here's what I've done (/dev/mapper/data-0 is symlinked to /dev/dm-0 so
they're interchangeable):

root@MyNAS:~# mount -t btrfs -o ro,recovery,nospace_cache,clear_cache
/dev/dm-0 /data
mount: wrong fs type, bad option, bad superblock on /dev/mapper/data-0,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
root@MyNAS:~# dmesg | tail
[25168.491582] BTRFS info (device dm-0): enabling auto recovery
[25168.724395] BTRFS (device dm-0): parent transid verify failed on
2474687627264 wanted 558760 found 558769
[25168.732497] BTRFS (device dm-0): parent transid verify failed on
2474687627264 wanted 558760 found 558769
[25168.734444] BTRFS (device dm-0): parent transid verify failed on
2474687627264 wanted 558760 found 558769
[25168.736294] BTRFS (device dm-0): parent transid verify failed on
2474687627264 wanted 558760 found 558769
[25169.583328] BTRFS (device dm-0): bad tree block start
6746185026026631152 3067274493952
[25169.595978] BTRFS (device dm-0): bad tree block start
13892978140409233545 3067274493952
[25169.597777] BTRFS (device dm-0): bad tree block start
6746185026026631152 3067274493952
[25169.599472] BTRFS (device dm-0): bad tree block start
13892978140409233545 3067274493952
[25169.599516] BTRFS: Failed to read block groups: -5
[25169.640222] BTRFS: open_ctree failed

root@MyNAS:~# btrfs rescue zero-log /dev/mapper/data-0
parent transid verify failed on 2474648240128 wanted 558760 found 558765
parent transid verify failed on 2474648240128 wanted 558760 found 558765
parent transid verify failed on 2474648240128 wanted 558760 found 558765
parent transid verify failed on 2474648240128 wanted 558760 found 558765
Ignoring transid failure
parent transid verify failed on 2474688413696 wanted 558760 found 558769
parent transid verify failed on 2474688413696 wanted 558760 found 558769
parent transid verify failed on 2474688413696 wanted 558760 found 558769
parent transid verify failed on 2474688413696 wanted 558760 found 558769
Ignoring transid failure
parent transid verify failed on 2474691854336 wanted 558765 found 558758
parent transid verify failed on 2474691854336 wanted 558765 found 558758
parent transid verify failed on 2474691854336 wanted 558765 found 558758
parent transid verify failed on 2474691854336 wanted 558765 found 558758
Ignoring transid failure
leaf parent key incorrect 2474691854336
Clearing log on /dev/mapper/data-0, previous log_root 0, level 0
Unable to find block group for 0
extent-tree.c:289: find_search_start: Assertion `1` failed.

The output of btrfs check --repair /dev/mapper/data-0 is attached.

What should I do next? Run --init-csum-tree? btrfs restore to a USB
drive? Any help much appreciated.

/Calle

[-- Attachment #2: btrfsrepair.log --]
[-- Type: text/x-log, Size: 5633 bytes --]

parent transid verify failed on 2474648240128 wanted 558760 found 558765
parent transid verify failed on 2474648240128 wanted 558760 found 558765
parent transid verify failed on 2474648240128 wanted 558760 found 558765
parent transid verify failed on 2474648240128 wanted 558760 found 558765
Ignoring transid failure
parent transid verify failed on 2474688413696 wanted 558760 found 558769
parent transid verify failed on 2474688413696 wanted 558760 found 558769
parent transid verify failed on 2474688413696 wanted 558760 found 558769
parent transid verify failed on 2474688413696 wanted 558760 found 558769
Ignoring transid failure
parent transid verify failed on 2474691854336 wanted 558765 found 558758
parent transid verify failed on 2474691854336 wanted 558765 found 558758
parent transid verify failed on 2474691854336 wanted 558765 found 558758
parent transid verify failed on 2474691854336 wanted 558765 found 558758
Ignoring transid failure
leaf parent key incorrect 2474691854336
checking extents
parent transid verify failed on 2474682548224 wanted 558760 found 558769
parent transid verify failed on 2474682548224 wanted 558760 found 558769
parent transid verify failed on 2474682548224 wanted 558760 found 558769
parent transid verify failed on 2474682548224 wanted 558760 found 558769
Ignoring transid failure
parent transid verify failed on 2474721148928 wanted 558758 found 558769
parent transid verify failed on 2474721148928 wanted 558758 found 558769
parent transid verify failed on 2474721148928 wanted 558758 found 558769
parent transid verify failed on 2474721148928 wanted 558758 found 558769
Ignoring transid failure
checksum verify failed on 2474683826176 found A7762F46 wanted 1C12DF79
checksum verify failed on 2474683826176 found A7762F46 wanted 1C12DF79
checksum verify failed on 2474683826176 found 7B09FD0A wanted 6F493491
checksum verify failed on 2474683826176 found 7B09FD0A wanted 6F493491
bytenr mismatch, want=2474683826176, have=13830340915957161993
parent transid verify failed on 2474648338432 wanted 558765 found 558760
parent transid verify failed on 2474648338432 wanted 558765 found 558760
parent transid verify failed on 2474648338432 wanted 558765 found 558760
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
parent transid verify failed on 2474682548224 wanted 558760 found 558769
Ignoring transid failure
parent transid verify failed on 2474648338432 wanted 558765 found 558760
Ignoring transid failure
leaf parent key incorrect 2474682548224
bad block 2474682548224
Errors found in extent allocation tree or chunk allocation
parent transid verify failed on 2474691854336 wanted 558765 found 558758
Ignoring transid failure
leaf parent key incorrect 2474691854336
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
extent buffer leak: start 2474648240128 len 32768
enabling repair mode
Checking filesystem on /dev/mapper/data-0
UUID: b6283e48-ebd3-4fdb-8976-b66b1f0868bc

Re: btrfs repair and zero-log doesn't seem to do anything

[Qu Wenruo]

At 12/05/2016 10:28 AM, Calle Kabo wrote:
> Hi,
>
> I've got a Netgear NAS with an encrypted 4 disk RAID5 setup. One of the
> hard drives was emailing me about errors so I swapped it out before it
> died. While syncing another disk died. So I've cloned the disk I
> replaced before it died and managed to get the array running again. You
> can read more about that here:
> https://community.netgear.com/t5/Using-your-ReadyNAS/Re-adding-disk-to-volume/m-p/1179982/highlight/false
>
> But now BTRFS won't mount. I'd like to mount it read only so I can move
> the stuff to another drive and then format the NAS into a new array. If
> there's some data loss that's fine, the most important stuff is already
> backed up, but there are a few things that aren't backed up that I find
> it worth the effort of trying to restore if at all possible.
>
> Mandatory information:
> root@MyNAS:~# uname -a
> Linux MyNAS 4.1.30.armada.1 #1 SMP Thu Sep 22 16:39:49 PDT 2016 armv7l
> GNU/Linux
> root@MyNAS:~# btrfs --version
> btrfs-progs v4.7.3
> root@MyNAS:~# btrfs fi show
> Label: '0e35246e:data'  uuid: b6283e48-ebd3-4fdb-8976-b66b1f0868bc
>         Total devices 1 FS bytes used 3.99TiB
>         devid    1 size 8.17TiB used 4.22TiB path /dev/dm-0
>
> Here's what I've done (/dev/mapper/data-0 is symlinked to /dev/dm-0 so
> they're interchangeable):
>
> root@MyNAS:~# mount -t btrfs -o ro,recovery,nospace_cache,clear_cache
> /dev/dm-0 /data
> mount: wrong fs type, bad option, bad superblock on /dev/mapper/data-0,
>        missing codepage or helper program, or other error
>
>        In some cases useful info is found in syslog - try
>        dmesg | tail or so.
> root@MyNAS:~# dmesg | tail
> [25168.491582] BTRFS info (device dm-0): enabling auto recovery
> [25168.724395] BTRFS (device dm-0): parent transid verify failed on
> 2474687627264 wanted 558760 found 558769
> [25168.732497] BTRFS (device dm-0): parent transid verify failed on
> 2474687627264 wanted 558760 found 558769
> [25168.734444] BTRFS (device dm-0): parent transid verify failed on
> 2474687627264 wanted 558760 found 558769
> [25168.736294] BTRFS (device dm-0): parent transid verify failed on
> 2474687627264 wanted 558760 found 558769
> [25169.583328] BTRFS (device dm-0): bad tree block start
> 6746185026026631152 3067274493952
> [25169.595978] BTRFS (device dm-0): bad tree block start
> 13892978140409233545 3067274493952
> [25169.597777] BTRFS (device dm-0): bad tree block start
> 6746185026026631152 3067274493952
> [25169.599472] BTRFS (device dm-0): bad tree block start
> 13892978140409233545 3067274493952
> [25169.599516] BTRFS: Failed to read block groups: -5
> [25169.640222] BTRFS: open_ctree failed

According to the fsck report, extent tree seems to be corrupted heavily.
A lot of transid error and bad tree blocks.

No wonder kernel refuses to mount it.

>
> root@MyNAS:~# btrfs rescue zero-log /dev/mapper/data-0
> parent transid verify failed on 2474648240128 wanted 558760 found 558765
> parent transid verify failed on 2474648240128 wanted 558760 found 558765
> parent transid verify failed on 2474648240128 wanted 558760 found 558765
> parent transid verify failed on 2474648240128 wanted 558760 found 558765
> Ignoring transid failure
> parent transid verify failed on 2474688413696 wanted 558760 found 558769
> parent transid verify failed on 2474688413696 wanted 558760 found 558769
> parent transid verify failed on 2474688413696 wanted 558760 found 558769
> parent transid verify failed on 2474688413696 wanted 558760 found 558769
> Ignoring transid failure
> parent transid verify failed on 2474691854336 wanted 558765 found 558758
> parent transid verify failed on 2474691854336 wanted 558765 found 558758
> parent transid verify failed on 2474691854336 wanted 558765 found 558758
> parent transid verify failed on 2474691854336 wanted 558765 found 558758
> Ignoring transid failure
> leaf parent key incorrect 2474691854336
> Clearing log on /dev/mapper/data-0, previous log_root 0, level 0
> Unable to find block group for 0
> extent-tree.c:289: find_search_start: Assertion `1` failed.

Zero log won't help on such heavily damaged fs.

>
> The output of btrfs check --repair /dev/mapper/data-0 is attached.

Considering it's extent tree heavily damaged, normal repair won't help.

IMHO there will be 2 alternative method to recover:

1) Btrfs restore
    The safest method to recover files.
    Although it may need a lot of space to restore recovered data.

2) Btrfs check --init-extent-tree
    This will use fs tree to try to rebuild the extent tree.
    I don't believe it's only extent tree corrupted, so this may make
    things even *worse*, but at least it won't take several TBs to
    recover data which you may already have backup.

Thanks,
Qu
>
> What should I do next? Run --init-csum-tree? btrfs restore to a USB
> drive? Any help much appreciated.
>
> /Calle
>
>

Re: Re: btrfs repair and zero-log doesn't seem to do anything

[Calle Kabo]

> IMHO there will be 2 alternative method to recover:
>
> 1) Btrfs restore
>    The safest method to recover files.
>    Although it may need a lot of space to restore recovered data.
>
> 2) Btrfs check --init-extent-tree
>    This will use fs tree to try to rebuild the extent tree.
>    I don't believe it's only extent tree corrupted, so this may make
>    things even *worse*, but at least it won't take several TBs to
>    recover data which you may already have backup.

Thanks! I tested restoring a small directory and it worked beautifully.
Is there any way to list the files/directories? I don't think I can
remember all of the paths that were on there...

/Calle

Re: btrfs repair and zero-log doesn't seem to do anything

[Qu Wenruo]

At 12/05/2016 11:49 AM, Calle Kabo wrote:
>
>> IMHO there will be 2 alternative method to recover:
>>
>> 1) Btrfs restore
>>    The safest method to recover files.
>>    Although it may need a lot of space to restore recovered data.
>>
>> 2) Btrfs check --init-extent-tree
>>    This will use fs tree to try to rebuild the extent tree.
>>    I don't believe it's only extent tree corrupted, so this may make
>>    things even *worse*, but at least it won't take several TBs to
>>    recover data which you may already have backup.
>
> Thanks! I tested restoring a small directory and it worked beautifully.
> Is there any way to list the files/directories? I don't think I can
> remember all of the paths that were on there...
>
> /Calle

I'm not familiar with btrfs-restore tool.
But a quick glance at the man page shows there is some options like
-r <rooid>
--path-regex <path>
-d
--list-roots
-D

Those options would help.
Maybe -D would list all the file structures?


And if btrfs restore works perfectly fine without even a problem,
then it seems to indicate that only extent tree is corrupted.
In that case, --init-extent-tree may have a little chance to recover the fs.
(But under most case, maybe over 50%?, it may make things worse)

Thanks,
Qu

> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: btrfs repair and zero-log doesn't seem to do anything

[Calle Kabo]

> Maybe -D would list all the file structures?

I tried that first but didn't see anything, but when combined with -v it
shows the files that would have been restored.

e.g.
# btrfs restore -v -D --path-regex '^/(|Documents(|/.*))$'
/dev/mapper/data-0 /tmp


Awesome, big thanks for your help!

/Calle