Fi corruption on RAID1, generation doesn't match

Fi corruption on RAID1, generation doesn't match

[Andreas Hild]

Dear All,

The file system on a RAID1 Debian server seems corrupted in a major
way, with 99% of the files not found. This was the result of a
precarious shutdown after a crash that was preceded by an accidental
misconfiguration in /etc/fstab; it pointed "/" and "/tmp" to one and
the same UUID by omitting a subvol entry.

Is there any way to repair or recover a substantial part of this RAID?

The following output was obtained via a live disk boot.


Many thanks!

Best wishes,
Andreas


--- --- ---
uname -a
Linux debian 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3
(2016-01-17) x86_64 GNU/Linux
--

sudo btrfs fi show
Label: none  uuid: fc429e82-f46e-4018-a9fa-ded688cef161
        Total devices 2 FS bytes used 42.36MiB
        devid    1 size 455.76GiB used 169.03GiB path /dev/sdb4
        devid    2 size 455.76GiB used 169.03GiB path /dev/sda4

Btrfs v3.17

--
sudo btrfs fi df /mnt
Data, RAID1: total=168.00GiB, used=42.00MiB
System, RAID1: total=32.00MiB, used=48.00KiB
Metadata, RAID1: total=1.00GiB, used=320.00KiB
GlobalReserve, single: total=16.00MiB, used=0.00B
--

sudo btrfs subvol list /mnt/
ID 257 gen 192248 top level 5 path home
ID 258 gen 192228 top level 5 path usr
ID 259 gen 192226 top level 5 path tmp
ID 260 gen 192238 top level 5 path var
--

sudo btrfs-find-root /dev/sdb4
Super think's the tree root is at 647630274560, chunk root 647596376064
Well block 647629897728 seems great, but generation doesn't match,
have=192251, want=192253 level 1
Well block 647630012416 seems great, but generation doesn't match,
have=192251, want=192253 level 0
Well block 647630028800 seems great, but generation doesn't match,
have=192252, want=192253 level 1
Well block 647630110720 seems great, but generation doesn't match,
have=192249, want=192253 level 0
Well block 647630143488 seems great, but generation doesn't match,
have=192252, want=192253 level 0
Well block 647630159872 seems great, but generation doesn't match,
have=192252, want=192253 level 0
Well block 647630176256 seems great, but generation doesn't match,
have=192252, want=192253 level 0
Well block 647630192640 seems great, but generation doesn't match,
have=192252, want=192253 level 0
Well block 647630258176 seems great, but generation doesn't match,
have=192252, want=192253 level 0
Found tree root at 647630274560 gen 192253 level 1
--

sudo btrfs check /dev/sdb4
Checking filesystem on /dev/sdb4
UUID: fc429e82-f46e-4018-a9fa-ded688cef161
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
found 12173435 bytes used err is 0
total csum bytes: 0
total tree bytes: 376832
total fs tree bytes: 98304
total extent tree bytes: 49152
btree space waste bytes: 229409
file data blocks allocated: 44040192
 referenced 44040192
--

sudo btrfs-debug-tree -R /dev/sdb4
root tree: 647630274560 level 1
chunk tree: 647596376064 level 1
extent tree key (EXTENT_TREE ROOT_ITEM 0) 647630307328 level 1
device tree key (DEV_TREE ROOT_ITEM 0) 647630094336 level 1
fs tree key (FS_TREE ROOT_ITEM 0) 647730692096 level 0
checksum tree key (CSUM_TREE ROOT_ITEM 0) 647630422016 level 0
uuid tree key (UUID_TREE ROOT_ITEM 0) 647641219072 level 0
file tree key (257 ROOT_ITEM 0) 647629996032 level 0
file tree key (258 ROOT_ITEM 0) 647730413568 level 0
file tree key (259 ROOT_ITEM 0) 648013332480 level 0
file tree key (260 ROOT_ITEM 0) 647731609600 level 0
data reloc tree key (DATA_RELOC_TREE ROOT_ITEM 0) 648438398976 level 0
btrfs root backup slot 0
        tree root gen 192252 block 647630028800
                extent root gen 192252 block 647630045184
                chunk root gen 191843 block 647596376064
                device root gen 192252 block 647630094336
                csum root gen 192252 block 647630225408
                fs root gen 192233 block 647730692096
                44417024 used 978736644096 total 2 devices
btrfs root backup slot 1
        tree root gen 192253 block 647630274560
                extent root gen 192253 block 647630307328
                chunk root gen 191843 block 647596376064
                device root gen 192252 block 647630094336
                csum root gen 192253 block 647630422016
                fs root gen 192233 block 647730692096
                44417024 used 978736644096 total 2 devices
btrfs root backup slot 2
        tree root gen 192244 block 647731642368
                extent root gen 192244 block 647731691520
                chunk root gen 191843 block 647596376064
                device root gen 192244 block 647731724288
                csum root gen 192244 block 647731789824
                fs root gen 192233 block 647730692096
                44695552 used 978736644096 total 2 devices
btrfs root backup slot 3
        tree root gen 192245 block 647737933824
                extent root gen 192245 block 647737950208
                chunk root gen 191843 block 647596376064
                device root gen 192244 block 647731724288
                csum root gen 192245 block 647738081280
                fs root gen 192233 block 647730692096
                44695552 used 978736644096 total 2 devices
total bytes 978736644096
bytes used 44417024
uuid fc429e82-f46e-4018-a9fa-ded688cef161
--

Re: Fi corruption on RAID1, generation doesn't match

[Lionel Bouton]

Hi,

Le 07/02/2016 14:15, Andreas Hild a écrit :
> Dear All,
>
> The file system on a RAID1 Debian server seems corrupted in a major
> way, with 99% of the files not found. This was the result of a
> precarious shutdown after a crash that was preceded by an accidental
> misconfiguration in /etc/fstab; it pointed "/" and "/tmp" to one and
> the same UUID by omitting a subvol entry.
>
> Is there any way to repair or recover a substantial part of this RAID?

I don't think the RAID is damaged: most distributions including Debian
remove nearly all files from /tmp at boot.
If /tmp and / were as you described the same filesystem your server most
probably did what amounts to "rm -rf /". You would probably have got the
same result with any filesystem as mounting the same filesystem at
several points in the VFS is not BTRFS-specific.

Unless you can restore a snapshot or there is a way to debug the
filesystem to restore a previous state, I'm afraid there's nothing to be
done.

Best regards,

Lionel

Re: Fi corruption on RAID1, generation doesn't match

[Andreas Hild]

On 7 February 2016 at 20:27, Lionel Bouton
<lionel-subscription@bouton.name> wrote:
> Hi,
>
> Le 07/02/2016 14:15, Andreas Hild a écrit :
>> Dear All,
>>
>> The file system on a RAID1 Debian server seems corrupted in a major
>> way, with 99% of the files not found. This was the result of a
>> precarious shutdown after a crash that was preceded by an accidental
>> misconfiguration in /etc/fstab; it pointed "/" and "/tmp" to one and
>> the same UUID by omitting a subvol entry.
>>
>> Is there any way to repair or recover a substantial part of this RAID?
>
> I don't think the RAID is damaged: most distributions including Debian
> remove nearly all files from /tmp at boot.
> If /tmp and / were as you described the same filesystem your server most
> probably did what amounts to "rm -rf /". You would probably have got the
> same result with any filesystem as mounting the same filesystem at
> several points in the VFS is not BTRFS-specific.
>
> Unless you can restore a snapshot or there is a way to debug the
> filesystem to restore a previous state, I'm afraid there's nothing to be
> done.


Thank you for the fast response!

I see. Just to be certain here, I have noted a vast difference between
the output of the "regular" 'df' and 'btrfs fi df'. Regular df shows
about 85MB of data on this file system, whereas btrfs fi df shows the
expected 168.00GiB (the exact amount of data that was there).

What does this mean?


Best wishes,
Andreas

Re: Fi corruption on RAID1, generation doesn't match

[Qu Wenruo]

On 02/07/2016 09:15 PM, Andreas Hild wrote:
> Dear All,
>
> The file system on a RAID1 Debian server seems corrupted in a major
> way, with 99% of the files not found. This was the result of a
> precarious shutdown after a crash that was preceded by an accidental
> misconfiguration in /etc/fstab; it pointed "/" and "/tmp" to one and
> the same UUID by omitting a subvol entry.

So the data of the subvolume is just erased, in a proper manner.
>
> Is there any way to repair or recover a substantial part of this RAID?

As already mentioned by Lionel, the filesystem is not damaged, but just 
emptied.

There is still some chance to recovery to at most 4 transaction back, 
but I assume only the previous trans is possible.

First thing first, don't ever do *ANY* write to the filesystem.

Any write will just lead to at least new transaction, which will make 
the chance to recovery even smaller.

>
> The following output was obtained via a live disk boot.
>
>
> Many thanks!
>
> Best wishes,
> Andreas
>
>
> --- --- ---
> uname -a
> Linux debian 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3
> (2016-01-17) x86_64 GNU/Linux
> --
>
> sudo btrfs fi show
> Label: none  uuid: fc429e82-f46e-4018-a9fa-ded688cef161
>          Total devices 2 FS bytes used 42.36MiB
>          devid    1 size 455.76GiB used 169.03GiB path /dev/sdb4
>          devid    2 size 455.76GiB used 169.03GiB path /dev/sda4
>
> Btrfs v3.17
>
> --
> sudo btrfs fi df /mnt
> Data, RAID1: total=168.00GiB, used=42.00MiB

You are wondering why data is still 168G, but that's the allocated data 
chunk size.

It means 168G space is allocated to store data, but only 42M is used.
Matches with your vanilla df output.

So it doesn't mean you data are still here.

> System, RAID1: total=32.00MiB, used=48.00KiB
> Metadata, RAID1: total=1.00GiB, used=320.00KiB
> GlobalReserve, single: total=16.00MiB, used=0.00B
> --
>
> sudo btrfs subvol list /mnt/
> ID 257 gen 192248 top level 5 path home
> ID 258 gen 192228 top level 5 path usr
> ID 259 gen 192226 top level 5 path tmp
> ID 260 gen 192238 top level 5 path var
> --
>
> sudo btrfs-find-root /dev/sdb4
> Super think's the tree root is at 647630274560, chunk root 647596376064
> Well block 647629897728 seems great, but generation doesn't match,
> have=192251, want=192253 level 1
> Well block 647630012416 seems great, but generation doesn't match,
> have=192251, want=192253 level 0
> Well block 647630028800 seems great, but generation doesn't match,
> have=192252, want=192253 level 1
> Well block 647630110720 seems great, but generation doesn't match,
> have=192249, want=192253 level 0
> Well block 647630143488 seems great, but generation doesn't match,
> have=192252, want=192253 level 0
> Well block 647630159872 seems great, but generation doesn't match,
> have=192252, want=192253 level 0
> Well block 647630176256 seems great, but generation doesn't match,
> have=192252, want=192253 level 0
> Well block 647630192640 seems great, but generation doesn't match,
> have=192252, want=192253 level 0
> Well block 647630258176 seems great, but generation doesn't match,
> have=192252, want=192253 level 0
> Found tree root at 647630274560 gen 192253 level 1

Remember the generation 192253, which will be used later, if you didn't 
increased it by do any write into the filesystem.

> --
>
> sudo btrfs check /dev/sdb4
> Checking filesystem on /dev/sdb4
> UUID: fc429e82-f46e-4018-a9fa-ded688cef161
> checking extents
> checking free space cache
> checking fs roots
> checking csums
> checking root refs
> found 12173435 bytes used err is 0
> total csum bytes: 0
> total tree bytes: 376832
> total fs tree bytes: 98304
> total extent tree bytes: 49152
> btree space waste bytes: 229409
> file data blocks allocated: 44040192
>   referenced 44040192
> --
>
> sudo btrfs-debug-tree -R /dev/sdb4
> root tree: 647630274560 level 1
> chunk tree: 647596376064 level 1
> extent tree key (EXTENT_TREE ROOT_ITEM 0) 647630307328 level 1
> device tree key (DEV_TREE ROOT_ITEM 0) 647630094336 level 1
> fs tree key (FS_TREE ROOT_ITEM 0) 647730692096 level 0
> checksum tree key (CSUM_TREE ROOT_ITEM 0) 647630422016 level 0
> uuid tree key (UUID_TREE ROOT_ITEM 0) 647641219072 level 0
> file tree key (257 ROOT_ITEM 0) 647629996032 level 0
> file tree key (258 ROOT_ITEM 0) 647730413568 level 0
> file tree key (259 ROOT_ITEM 0) 648013332480 level 0
> file tree key (260 ROOT_ITEM 0) 647731609600 level 0
> data reloc tree key (DATA_RELOC_TREE ROOT_ITEM 0) 648438398976 level 0
> btrfs root backup slot 0
>          tree root gen 192252 block 647630028800
>                  extent root gen 192252 block 647630045184
>                  chunk root gen 191843 block 647596376064
>                  device root gen 192252 block 647630094336
>                  csum root gen 192252 block 647630225408
>                  fs root gen 192233 block 647730692096
>                  44417024 used 978736644096 total 2 devices

That's the previous transaction's tree root.
If you are in good luck, previous trans may has all your data.
But if you are in bad luck, it may already after the remove.

Normally, at least something should still be in previous trans.

Now use btrfsck to check if previous trans is in good shape.

# btrfsck --tree-root 647630028800

If btrfsck only reports minor problems, like space cache tree mismatch, 
then it means previous trans are almost OK.

Then let btrfsck to revert to previous trans:

# btrfsck --tree-root 647630028800 --repair

If you're really in good luck, then you can mount the fs and found 
something in it.

Thanks,
Qu

> btrfs root backup slot 1
>          tree root gen 192253 block 647630274560
>                  extent root gen 192253 block 647630307328
>                  chunk root gen 191843 block 647596376064
>                  device root gen 192252 block 647630094336
>                  csum root gen 192253 block 647630422016
>                  fs root gen 192233 block 647730692096
>                  44417024 used 978736644096 total 2 devices
> btrfs root backup slot 2
>          tree root gen 192244 block 647731642368
>                  extent root gen 192244 block 647731691520
>                  chunk root gen 191843 block 647596376064
>                  device root gen 192244 block 647731724288
>                  csum root gen 192244 block 647731789824
>                  fs root gen 192233 block 647730692096
>                  44695552 used 978736644096 total 2 devices
> btrfs root backup slot 3
>          tree root gen 192245 block 647737933824
>                  extent root gen 192245 block 647737950208
>                  chunk root gen 191843 block 647596376064
>                  device root gen 192244 block 647731724288
>                  csum root gen 192245 block 647738081280
>                  fs root gen 192233 block 647730692096
>                  44695552 used 978736644096 total 2 devices
> total bytes 978736644096
> bytes used 44417024
> uuid fc429e82-f46e-4018-a9fa-ded688cef161
> --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Fi corruption on RAID1, generation doesn't match

[Andreas Hild]

On 7 February 2016 at 20:56, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
> You are wondering why data is still 168G, but that's the allocated data
> chunk size.
>
> It means 168G space is allocated to store data, but only 42M is used.
> Matches with your vanilla df output.
>
> So it doesn't mean you data are still here.

I understand now. Thanks very much!


[...]
>
> That's the previous transaction's tree root.
> If you are in good luck, previous trans may has all your data.
> But if you are in bad luck, it may already after the remove.
>
> Normally, at least something should still be in previous trans.
>
> Now use btrfsck to check if previous trans is in good shape.
>
> # btrfsck --tree-root 647630028800
>
> If btrfsck only reports minor problems, like space cache tree mismatch, then
> it means previous trans are almost OK.
>
> Then let btrfsck to revert to previous trans:
>
> # btrfsck --tree-root 647630028800 --repair
>
> If you're really in good luck, then you can mount the fs and found something
> in it.


Great! I'll try this.

The 'tree-root' option must be a recent development. My 'stable'
Debian live disk does not recognize this option.

Could any one recommended a live disk that already includes this btrfsck option?

Many thanks!

Andreas

Re: Fi corruption on RAID1, generation doesn't match

[Qu Wenruo]

On 02/07/2016 10:23 PM, Andreas Hild wrote:
> On 7 February 2016 at 20:56, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>> You are wondering why data is still 168G, but that's the allocated data
>> chunk size.
>>
>> It means 168G space is allocated to store data, but only 42M is used.
>> Matches with your vanilla df output.
>>
>> So it doesn't mean you data are still here.
>
> I understand now. Thanks very much!
>
>
> [...]
>>
>> That's the previous transaction's tree root.
>> If you are in good luck, previous trans may has all your data.
>> But if you are in bad luck, it may already after the remove.
>>
>> Normally, at least something should still be in previous trans.
>>
>> Now use btrfsck to check if previous trans is in good shape.
>>
>> # btrfsck --tree-root 647630028800
>>
>> If btrfsck only reports minor problems, like space cache tree mismatch, then
>> it means previous trans are almost OK.
>>
>> Then let btrfsck to revert to previous trans:
>>
>> # btrfsck --tree-root 647630028800 --repair
>>
>> If you're really in good luck, then you can mount the fs and found something
>> in it.
>
>
> Great! I'll try this.
>
> The 'tree-root' option must be a recent development. My 'stable'
> Debian live disk does not recognize this option.
>
> Could any one recommended a live disk that already includes this btrfsck option?
>
> Many thanks!
>
> Andreas
>
Try latest installation ISO from Archlinux, which should be recent 
enough, if not already latest.

Another recommendation should be latest Fedora, which should be no later 
than Arch.
But with a better LiveCD environment, like with GUI, other than pure CLI 
from Archlinux ISO.

Thanks,
Qu

Re: Fi corruption on RAID1, generation doesn't match

[Andreas Hild]

On 7 February 2016 at 09:27, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
> On 02/07/2016 10:23 PM, Andreas Hild wrote:
>>
>> On 7 February 2016 at 20:56, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>> You are wondering why data is still 168G, but that's the allocated data
>>> chunk size.
>>>
>>> It means 168G space is allocated to store data, but only 42M is used.
>>> Matches with your vanilla df output.
>>>
>>> So it doesn't mean you data are still here.
>>
>>
>> I understand now. Thanks very much!
>>
>>
>> [...]
>>>
>>>
>>> That's the previous transaction's tree root.
>>> If you are in good luck, previous trans may has all your data.
>>> But if you are in bad luck, it may already after the remove.
>>>
>>> Normally, at least something should still be in previous trans.
>>>
>>> Now use btrfsck to check if previous trans is in good shape.
>>>
>>> # btrfsck --tree-root 647630028800
>>>
>>> If btrfsck only reports minor problems, like space cache tree mismatch,
>>> then
>>> it means previous trans are almost OK.
>>>
>>> Then let btrfsck to revert to previous trans:
>>>
>>> # btrfsck --tree-root 647630028800 --repair
>>>
>>> If you're really in good luck, then you can mount the fs and found
>>> something
>>> in it.
>>
>>
>>
>> Great! I'll try this.
>>
>> The 'tree-root' option must be a recent development. My 'stable'
>> Debian live disk does not recognize this option.
>>
>> Could any one recommended a live disk that already includes this btrfsck
>> option?
>>
>> Many thanks!
>>
>> Andreas
>>
> Try latest installation ISO from Archlinux, which should be recent enough,
> if not already latest.
>
> Another recommendation should be latest Fedora, which should be no later
> than Arch.
> But with a better LiveCD environment, like with GUI, other than pure CLI
> from Archlinux ISO.

An invaluable experience; though I wasn't able to recover data.

The list is confidence inspiring!

Many thanks!