rootfs crash

rootfs crash

[Jogi Hofmüller]

[-- Attachment #1: Type: text/plain, Size: 1300 bytes --]

Dear all,

Yesterday the rootfs on my laptop crashed.  So far all attempts to fix
it have failed;  hence I am looking for some help (if possible).

I was creating a copy of a virtual machine (kvm) on my laptop and
started modifying the machine from the inside (installing new packages),
when the laptop froze.  Rebooting failed because /dev/sda3 (the rootfs)
could not be mounted anymore.

Some of the errors were

btrfs: failed to read log tree
btrfs: open_ctree failed

Recovery using a Debian/Testing boot stick (kernel 3.10.2, btrfs-tools
v0.20-rc1) were unsuccessful so far.  After btrfs-zero-log the device
was still not mountable.  Mount attempts fail with i/o erros now.

btrfs restore manages to restore some files from /boot/grub but then
fails with

failed to inflate: -6
and
failed to inflate: -8
and
*** Error in `btrfs': free(): invalid next size
(normal):0x0000000000c0e650 ***

The filesystem is setup as raid1 on devices sda3 and sdb3 using lzo
compression.  There are subvolumes for /home and /var.

Images created with btrfs-image are available from before and after
btrfs-zero-log was run.  I can make them available if needed.

Any hints appreciated!

Regards,
-- 
j.hofmüller                                http://users.mur.at/thesix/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 294 bytes --]

Re: rootfs crash

[Jogi Hofmüller]

Dear all,

After trying all the suggested options from

   http://permalink.gmane.org/gmane.comp.file-systems.btrfs/27999

I still get no mountable file system :(

Trying to mount I get

mount -oro,recovery,compress=lzo /dev/sda3 /rootfs

[  906.835314] btrfs: enabling auto recovery
[  906.836977] btrfs: use lzo compression
[  906.838611] btrfs: disk space caching is enabled
[  906.843788] btrfs: mismatching generation and generation_v2 found in 
root item. This root was probably mounted with an older kernel. 
Resetting all new fields.
[  906.898440] Btrfs detected SSD devices, enabling SSD mode
[  906.900352] parent transid verify failed on 71744598016 wanted 103531 
found 103528
[  906.902288] parent transid verify failed on 71744598016 wanted 103531 
found 103528
[  906.986653] btrfs: open_ctree failed
mount: mounting /dev/sda3 on /rootfs failed: Invalid argument

I am limited to working with the tools the Debian initramfs provides.  
This means kernel 3.10.2 (Debian 3.10.7-1) and btrfs-tools  
0.19+20130705-1.  The latter seems to be up to date with git although 
`btrfs version` says v0.20 rc1. All this is happening on an Asus Zen 
book  UX32V with two 128GB SSDs.

If anyone is interested in images produced by btrfs-image, they are 
available at

   http://plagi.at/images/

I am fresh out of ideas at the moment, so if anyone has a suggestion I 
am willing to try.

Regards!
-- 
j.hofmüller

Re: rootfs crash

[Sander]

Jogi Hofmüller wrote (ao):
> I am limited to working with the tools the Debian initramfs
> provides.  This means kernel 3.10.2 (Debian 3.10.7-1) and
> btrfs-tools  0.19+20130705-1.  The latter seems to be up to date
> with git although `btrfs version` says v0.20 rc1. All this is
> happening on an Asus Zen book  UX32V with two 128GB SSDs.
> 
> If anyone is interested in images produced by btrfs-image, they are
> available at
> 
>   http://plagi.at/images/
> 
> I am fresh out of ideas at the moment, so if anyone has a suggestion
> I am willing to try.

Did you try btrfs chunk-recover ?

	Sander

Re: rootfs crash

[Jogi Hofmüller]

Hi,

Am 2013-09-17 13:18, schrieb Sander:
> Jogi Hofmüller wrote (ao):

>> I am fresh out of ideas at the moment, so if anyone has a suggestion
>> I am willing to try.
>
> Did you try btrfs chunk-recover ?

Yes, did so.  It stopped with an error.  I will send the error message 
later since it resides on a different machine that I cannot access from 
where I am now.

Regards!
-- 
j.hofmüller

Re: rootfs crash

[Jogi Hofmüller]

[-- Attachment #1: Type: text/plain, Size: 976 bytes --]

Hi,

Am 2013-09-17 14:24, schrieb Jogi Hofmüller:

> Yes, did so.  It stopped with an error.  I will send the error
> message later since it resides on a different machine that I cannot
> access from where I am now.

This is the promised output of

btrfs chunk-recover -y -v /dev/sda3

[ 1090.896935] device fsid f75ca841-f09b-4341-8b59-8fcab921d8ce devid 2
transid 103536 /dev/sdb3
[ 1090.900656] device fsid f75ca841-f09b-4341-8b59-8fcab921d8ce devid 1
transid 103536 /dev/sda3
[ 1090.905844] device fsid f75ca841-f09b-4341-8b59-8fcab921d8ce devid 2
transid 103536 /dev/sdb3
[ 1090.909114] device fsid f75ca841-f09b-4341-8b59-8fcab921d8ce devid 1
transid 103536 /dev/sda3

All Devices:
    Device: id = 2, name = /dev/sdb3
    Device: id = 1, name = /dev/sda3

btrfs: cmds-chunk.c:125: process_extent_buffer: Assertion
`!(exists->nmirrors >=2)´ failed.

Regards!
-- 
j.hofmüller                                http://users.mur.at/thesix/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 294 bytes --]

Re: rootfs crash

[Clemens Eisserer]

Hi Jogi,

When I tried to recovered a broken btrfs-parition, I encountered the
same errors you did - btrfs restore seems to be a bit broken.

What helped was to:
a. comment out the free()-calls which led to crashes
b. re-run "btrfs restore" to account for the crashes which happen
elsewhere, hopefully it will progress each run a little further as it
did for me.

Regards, Clemens

Re: rootfs crash

[Jogi Hofmüller]

Hi Clemens, all,
Am 2013-09-17 20:53, schrieb Clemens Eisserer:

> a. comment out the free()-calls which led to crashes

Uh, well, I did not want to go that far ;)  I'm certainly not the
greatest programmer around, but not freeing allocated memory seems kind
of drastic.

Anyway, I checked out a copy of btrfs-progs from git and took a look at
the code.  Then I remembered valgrind and instead of running btrfs
restore on the broken machine I tried it on my images (saved with dd on
an external hard drive).  Running the self-compiled program using sudo
resulted in the same errors.

Now the (at least for me) fun part.  Running it under my UID I managed
to retrieve 13GB worth of data so far before hitting the same error:

  Error in `./btrfs': free(): invalid next size (normal)

I would be glad for someone's opinion on this.

Regards!
-- 
j.hofmüller

Re: rootfs crash

[Frank Holton]

Hello Jogi,

I'm not sure what's causing the free error, I'll keep looking.

Have you tried running restore on your home subvolume yet? By default
restore only works on the default subvolume which in your case is /

The following command will try to restore your /home subvolume
btrfs restore -i -r 258 [image] [destination]

Run that and see if you get your home folder restored. Hopefully the
errors are only on the root volume.

You can also try the /var subvolume which is 257 instead of 258.

Let me know if any of that helps,
Frank

On Wed, Sep 18, 2013 at 2:15 PM, Jogi Hofmüller <jogi@mur.at> wrote:
> Hi Clemens, all,
> Am 2013-09-17 20:53, schrieb Clemens Eisserer:
>
>> a. comment out the free()-calls which led to crashes
>
> Uh, well, I did not want to go that far ;)  I'm certainly not the
> greatest programmer around, but not freeing allocated memory seems kind
> of drastic.
>
> Anyway, I checked out a copy of btrfs-progs from git and took a look at
> the code.  Then I remembered valgrind and instead of running btrfs
> restore on the broken machine I tried it on my images (saved with dd on
> an external hard drive).  Running the self-compiled program using sudo
> resulted in the same errors.
>
> Now the (at least for me) fun part.  Running it under my UID I managed
> to retrieve 13GB worth of data so far before hitting the same error:
>
>   Error in `./btrfs': free(): invalid next size (normal)
>
> I would be glad for someone's opinion on this.
>
> Regards!
> --
> j.hofmüller
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rootfs crash

[Jogi Hofmüller]

[-- Attachment #1: Type: text/plain, Size: 1425 bytes --]

Hello Frank,

Am 2013-09-19 18:18, schrieb Frank Holton:

> I'm not sure what's causing the free error, I'll keep looking.

So am I :)  In fact I would really like to help fix the tool.  I hope I
can provide some suggestions for fixes, but I'm having my trouble with
the code ;)

All I can tell so far is that when you run `btrfs restore` via valgrind
you get much farther then without.  But even that eventually crashes :(

> Have you tried running restore on your home subvolume yet? By default
> restore only works on the default subvolume which in your case is /

I managed to retrieve almost everything from /home/.  I am even working
on the machine that had the crashed file system now :)

> The following command will try to restore your /home subvolume
> btrfs restore -i -r 258 [image] [destination]

Good point!  I could not find any references to restoring only parts of
the file system.  It's just a pain to see it restoring everything in
/var/cache/apt/archives ... and stopping before it get's to `valuable`
data ;)

> Run that and see if you get your home folder restored. Hopefully the
> errors are only on the root volume.

I will give it another try since I have backups of the corrupted devices
anyhow and will use them as test ground now.

Thanks for your reply!

Cheers!
-- 
j.hofmüller

Optimism doesn't alter the laws of physics.         - Subcommander T'Pol


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: rootfs crash

[Jérôme Poulin]

On Mon, Sep 16, 2013 at 6:48 AM, Jogi Hofmüller <jogi@mur.at> wrote:
>  After btrfs-zero-log the device
> was still not mountable.  Mount attempts fail with i/o erros now.
...
> All this is happening on an Asus Zen book  UX32V with two 128GB SSDs.


Did you make sure you aren't working with a dead SSD? That would make
everything much more complicated than it should be. I also had a
Zenbook from Asus with SSD with Btrfs, and it died 2 months ago. It
couldn't handle Btrfs for more than 6 months. Either those SSDs are
weak or Btrfs really is an SSD killer even though it is a COW FS.

Maybe using ddrescue first to get it off the defective media would
give you a chance to recover more data.

Re: rootfs crash

[Jogi Hofmüller]

[-- Attachment #1: Type: text/plain, Size: 838 bytes --]

Hi,

Am 2013-09-20 06:49, schrieb Jérôme Poulin:

> Did you make sure you aren't working with a dead SSD?

The SSDs are working quite fine.  I was worried about that too, but
since I managed to get raw disk images using `dd` I was pretty sure they
were alright.  In fact I reinstalled Debian on the Laptop and am using
it to write this email right now ;)

> That would make everything much more complicated than it should be. I
> also had a Zenbook from Asus with SSD with Btrfs, and it died 2
> months ago. It couldn't handle Btrfs for more than 6 months. Either
> those SSDs are weak or Btrfs really is an SSD killer even though it
> is a COW FS.

That sounds bad!  I will have a close look on the devices.

Regards!
-- 
j.hofmüller

Optimism doesn't alter the laws of physics.         - Subcommander T'Pol


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 230 bytes --]