Lost /home subvolume after btrfs crash

Lost /home subvolume after btrfs crash

[Martin Wilck]

Hello,

I have a broken btrfs file system on a laptop.

Debug material is available here:
  https://www.dropbox.com/sh/utv8b3qd0do6a04/zTwGQCrN9x

Most importantly, the /home subvolume is lost. All attempts to recover
data from it (btrfs-restore, mount -o recovery, btrfsck) have failed so
far (/home is simply empty in the btrfs-restore output), although I was
able to recover most of the other subvolumes and the main FS. The btrfs
tools would typically segfault with failed assertions when analyzing the
FS. Grepping through the entire volume shows that (at least parts of)
/home are still available, but they seem to be disconnected from the
main tree somehow, or the meta data is so corrupt that the tools bail
out trying to find files under it. Making matters worse, the important
data was encrypted using ecryptfs, so I need to recover the ciphertext
first and then find a way to recover the plaintext (I do have the
passphrase).

The crash happened with a rather old OpenSUSE 12.2 kernel (3.4.11-2.16).
The user says she was just surfing the web normally when the crash
occured (no screenshot of the original crash, unfortunately). On the
next boot,  the btrfs root file system couldn't be mounted any more.
After that I booted an OpenSUSE 12.3 rescue DVD and created the debug
material shown above.

Any hints how to retrieve files from the /home subvolume from this
corrupt file system would be highly appreciated.

Best regards
Martin

Re: Lost /home subvolume after btrfs crash

[Chris Murphy]

On Apr 16, 2014, at 7:53 AM, Martin Wilck <mwilck@arcor.de> wrote:
> 
> The crash happened with a rather old OpenSUSE 12.2 kernel (3.4.11-2.16).
> The user says she was just surfing the web normally when the crash
> occured (no screenshot of the original crash, unfortunately). On the
> next boot,  the btrfs root file system couldn't be mounted any more.
> After that I booted an OpenSUSE 12.3 rescue DVD and created the debug
> material shown above.

OpenSUSE 12.3 is using kernel 3.7 which is also old for this sort of recovery attempt. Even openSUSE 13.1 is at 3.11.6 which might work in a bind, but if it doesn't, inevitably someone will suggest you use something even newer. 

Current stable is 3.14.1, I suggest giving 3.13 or 3.14 a shot at this with -o ro,recovery as a first step and see if it at least mounts.

And an old kernel implies old btrfs-progs too, which is where the code for btrfsck and btrfs restore is contained. So that needs to be at least v 3.12. And hopefully you didn't use --repair with btrfsck yet.


Chris Murphy

Re: Lost /home subvolume after btrfs crash

[Martin Wilck]

Chris,

> OpenSUSE 12.3 is using kernel 3.7 which is also old for this sort of recovery attempt. Even openSUSE 13.1 is at 3.11.6 which might work in a bind, but if it doesn't, inevitably someone will suggest you use something even newer. 

Thanks for your reply, I appreciate it a lot.

> Current stable is 3.14.1, I suggest giving 3.13 or 3.14 a shot at this with -o ro,recovery as a first step and see if it at least mounts.

I will. Note that with 12.3, which was the most recent media I had at
hand at the time, the FS was actually mountable at first (-o
ro,recovery). But there was no /home and later attempts to actually
access data in other subvolumes failed with the messages in my debug
material.

> And an old kernel implies old btrfs-progs too, which is where the code for btrfsck and btrfs restore is contained. So that needs to be at least v 3.12. And hopefully you didn't use --repair with btrfsck yet.

I did, but I made a block-level copy of the device before.

Thanks again
Martin

Re: Lost /home subvolume after btrfs crash

[Chris Murphy]

On Apr 23, 2014, at 12:33 PM, Martin Wilck <mwilck@arcor.de> wrote:

> Chris,
> 
>> OpenSUSE 12.3 is using kernel 3.7 which is also old for this sort of recovery attempt. Even openSUSE 13.1 is at 3.11.6 which might work in a bind, but if it doesn't, inevitably someone will suggest you use something even newer. 
> 
> Thanks for your reply, I appreciate it a lot.
> 
>> Current stable is 3.14.1, I suggest giving 3.13 or 3.14 a shot at this with -o ro,recovery as a first step and see if it at least mounts.
> 
> I will. Note that with 12.3, which was the most recent media I had at
> hand at the time, the FS was actually mountable at first (-o
> ro,recovery). But there was no /home and later attempts to actually
> access data in other subvolumes failed with the messages in my debug
> material.

I'd skip 3.13 and go straight to 3.14.1 or 3.15rc2, and use mount option -o ro,recovery straight away and see if you can extract all of /home.

You could then try btrfs-progs v3.14's btrfs restore to extract /home. If that doesn't work then I'd give up on this instance of the file system as it sounds damaged by something possibly even the btrfsck --repair.

> 
>> And an old kernel implies old btrfs-progs too, which is where the code for btrfsck and btrfs restore is contained. So that needs to be at least v 3.12. And hopefully you didn't use --repair with btrfsck yet.
> 
> I did, but I made a block-level copy of the device before.

I wouldn't use --repair until a dev recommends it (even with btrfs-progs v3.14 let alone the I'm guessing v0.19 or v0.20 you were using); or once all other reasonable options are exhausted.


Chris Murphy

Re: Lost /home subvolume after btrfs crash

[Martin Wilck]

On 04/24/2014 05:15 AM, Chris Murphy wrote:

>>> Current stable is 3.14.1, I suggest giving 3.13 or 3.14 a shot at this with -o ro,recovery as a first step and see if it at least mounts.
>>
>> I will. Note that with 12.3, which was the most recent media I had at
>> hand at the time, the FS was actually mountable at first (-o
>> ro,recovery). But there was no /home and later attempts to actually
>> access data in other subvolumes failed with the messages in my debug
>> material.
> 
> I'd skip 3.13 and go straight to 3.14.1 or 3.15rc2, and use mount option -o ro,recovery straight away and see if you can extract all of /home.

I tried mount -o ro,recovery with 3.14.1 (3.14.1-vanilla from OpenSUSE
tumbleweed), but results weren't better than before. The kernel choked
on the FS. The attempt to mount the broken FS with -o ro,recovery ended
with a kernel Oops:

BTRFS error (device dm-18): unable to find ref byte nr 980791296 parent
0 root 2  owner 0 offset 0
BUG: unable to handle kernel paging request at ff000000
IP: [<c0324cab>] page_address+0xb/0xd0
[<f926deef>] btrfs_del_items+0xdf/0x390
[<f9275636>] __btrfs_free_extent+0x796/0xb30
[<f927a81c>] __btrfs_run_delayed_refs+0x8dc/0x10e0
[<f9276c21>] ? btrfs_delayed_refs_qgroup_accounting+0xa1/0xe0
[<f927ec96>] btrfs_run_delayed_refs.part.56+0x56/0x1f0
[<f92a963f>] ? btrfs_run_ordered_operations+0x19f/0x220
[<f927ee3f>] btrfs_run_delayed_refs+0xf/0x20
[<f928e0cc>] btrfs_commit_transaction+0x3c/0xbf0
...

After that, the system was basically dead, all IO would be blocked by
some deadlock in btrfs.

Full kernel log again under
https://www.dropbox.com/sh/utv8b3qd0do6a04/zTwGQCrN9x.

> You could then try btrfs-progs v3.14's btrfs restore to extract
> /home. 

btrfs restore from btrfs-progs 3.14.1 definitely looked more promising
than the version I tried previously. However, /home is still empty.

> If that doesn't work then I'd give up on this instance of the
> file system as it sounds damaged by something possibly even the
> btrfsck --repair.

I made the block-level copy before trying any write operation (IOW,
before copying, I tried only btrfs-restore and mount -ro,recovery).
So it should be a 1:1 copy of the broken FS.

I can't easily give up this data,

> I wouldn't use --repair until a dev recommends it (even with btrfs-progs v3.14 let alone the I'm guessing v0.19 or v0.20 you were using); or once all other reasonable options are exhausted.

Well it seems as if all "reasonable" options are indeed exhausted :-(
Still looking for ideas to recover the contents of /home.

Would it make sens to try to check out previous generations of the root
tree with btrfs recover?

Regards
Martin

> 
> 
> Chris Murphy