btrfsck failes

btrfsck failes

[Holger Brandsmeier]

Dear BTRFS developers,

thanks for providing a filesystem with which I am happy so far!

Currently btrfsck failes to repair my partition, I get the output:

[root@ho-think bholger]# btrfsck --repair /dev/sda5
[...]
leaf parent key incorrect 600846336
parent transid verify failed on 600846336 wanted 23460 found 23463
Ignoring transid failure
leaf parent key incorrect 600846336
bad block 600395776
bad block 600518656
bad block 600547328
leaf parent key incorrect 600846336
bad block 600846336
bad block 601710592
bad block 603197440
Ignoring transid failure
parent transid verify failed on 601710592 wanted 23460 found 23463
Ignoring transid failure
parent transid verify failed on 602529792 wanted 23460 found 23463
Ignoring transid failure
btrfsck: disk-io.c:155: readahead_tree_block: Assertion `!(ret)' failed.
Aborted (core dumped)


>From dmesg I get the additional information:

[  629.447677] btrfs: device fsid f6907d81-f46f-4911-8600-858e8b6bd1a0
devid 1 transid 23462 /dev/sda5
[  629.671835] systemd-journald[182]: Vacuuming done, freed 0 bytes
[  629.843055] systemd-journald[182]: Failed to write entry (26 items,
416256620 bytes) despite vacuuming, ignoring: Argument list too long

I am running Arch linux with btrfs-progs-3.12-1 and
[root@ho-think bholger]# uname -r
3.12.7-2-ARCH

The filesystem got some problems today while being mounted, it got
mounted read-only due to an error, but I still had access to my data.
I had to unmount it to run btrfsck, but because it fails to repair my
disk, I cannot mount it any more.

Any advice?
Holger

Re: btrfsck failes

[Holger Brandsmeier]

Oh and if I do
mount -o recovery /dev/sda5 /some/path/

then I get the following in dmesg:

[ 1449.171259] btrfs: device fsid f6907d81-f46f-4911-8600-858e8b6bd1a0
devid 1 transid 23462 /dev/sda5
[ 1449.171635] btrfs: enabling auto recovery
[ 1449.171638] btrfs: disk space caching is enabled
[ 1451.848339] btrfs: space cache generation (23463) does not match
inode (23461)
[ 1451.848360] BTRFS error (device sda5): failed to load free space
cache for block group 1103101952
[ 1451.959865] parent transid verify failed on 601710592 wanted 23460
found 23463
[ 1451.969678] parent transid verify failed on 601710592 wanted 23460
found 23463
[ 1452.461075] btrfs: space cache generation (23463) does not match
inode (23461)
[ 1452.461094] BTRFS error (device sda5): failed to load free space
cache for block group 2176843776
[ 1456.915205] parent transid verify failed on 603377664 wanted 23460
found 23463
[ 1456.925074] parent transid verify failed on 603377664 wanted 23460
found 23463
[ 1456.925827] BTRFS error (device sda5) in open_ctree:2847: errno=-5
IO failure (Failed to recover log tree)
[ 1456.971476] btrfs: open_ctree failed


On Mon, Jan 13, 2014 at 11:58 PM, Holger Brandsmeier
<brandsmeier@gmail.com> wrote:
> Dear BTRFS developers,
>
> thanks for providing a filesystem with which I am happy so far!
>
> Currently btrfsck failes to repair my partition, I get the output:
>
> [root@ho-think bholger]# btrfsck --repair /dev/sda5
> [...]
> leaf parent key incorrect 600846336
> parent transid verify failed on 600846336 wanted 23460 found 23463
> Ignoring transid failure
> leaf parent key incorrect 600846336
> bad block 600395776
> bad block 600518656
> bad block 600547328
> leaf parent key incorrect 600846336
> bad block 600846336
> bad block 601710592
> bad block 603197440
> Ignoring transid failure
> parent transid verify failed on 601710592 wanted 23460 found 23463
> Ignoring transid failure
> parent transid verify failed on 602529792 wanted 23460 found 23463
> Ignoring transid failure
> btrfsck: disk-io.c:155: readahead_tree_block: Assertion `!(ret)' failed.
> Aborted (core dumped)
>
>
> From dmesg I get the additional information:
>
> [  629.447677] btrfs: device fsid f6907d81-f46f-4911-8600-858e8b6bd1a0
> devid 1 transid 23462 /dev/sda5
> [  629.671835] systemd-journald[182]: Vacuuming done, freed 0 bytes
> [  629.843055] systemd-journald[182]: Failed to write entry (26 items,
> 416256620 bytes) despite vacuuming, ignoring: Argument list too long
>
> I am running Arch linux with btrfs-progs-3.12-1 and
> [root@ho-think bholger]# uname -r
> 3.12.7-2-ARCH
>
> The filesystem got some problems today while being mounted, it got
> mounted read-only due to an error, but I still had access to my data.
> I had to unmount it to run btrfsck, but because it fails to repair my
> disk, I cannot mount it any more.
>
> Any advice?
> Holger

Re: btrfsck failes

[Holger Brandsmeier]

I modified the code (now I am using the git repository) of disk-io.c
to include the debug output

  if(ret) {
    printf("btrfs_map_block(bytenr=%u, length=%u) has code %d\n",
bytenr, length, ret);
  }

 before the assertion throws, and I get:

btrfs_map_block(bytenr=2684354560, length=4096) has code -2

i.e. the error is ENOENT.

-Holger

On Tue, Jan 14, 2014 at 12:15 AM, Holger Brandsmeier
<brandsmeier@gmail.com> wrote:
> Oh and if I do
> mount -o recovery /dev/sda5 /some/path/
>
> then I get the following in dmesg:
>
> [ 1449.171259] btrfs: device fsid f6907d81-f46f-4911-8600-858e8b6bd1a0
> devid 1 transid 23462 /dev/sda5
> [ 1449.171635] btrfs: enabling auto recovery
> [ 1449.171638] btrfs: disk space caching is enabled
> [ 1451.848339] btrfs: space cache generation (23463) does not match
> inode (23461)
> [ 1451.848360] BTRFS error (device sda5): failed to load free space
> cache for block group 1103101952
> [ 1451.959865] parent transid verify failed on 601710592 wanted 23460
> found 23463
> [ 1451.969678] parent transid verify failed on 601710592 wanted 23460
> found 23463
> [ 1452.461075] btrfs: space cache generation (23463) does not match
> inode (23461)
> [ 1452.461094] BTRFS error (device sda5): failed to load free space
> cache for block group 2176843776
> [ 1456.915205] parent transid verify failed on 603377664 wanted 23460
> found 23463
> [ 1456.925074] parent transid verify failed on 603377664 wanted 23460
> found 23463
> [ 1456.925827] BTRFS error (device sda5) in open_ctree:2847: errno=-5
> IO failure (Failed to recover log tree)
> [ 1456.971476] btrfs: open_ctree failed
>
>
> On Mon, Jan 13, 2014 at 11:58 PM, Holger Brandsmeier
> <brandsmeier@gmail.com> wrote:
>> Dear BTRFS developers,
>>
>> thanks for providing a filesystem with which I am happy so far!
>>
>> Currently btrfsck failes to repair my partition, I get the output:
>>
>> [root@ho-think bholger]# btrfsck --repair /dev/sda5
>> [...]
>> leaf parent key incorrect 600846336
>> parent transid verify failed on 600846336 wanted 23460 found 23463
>> Ignoring transid failure
>> leaf parent key incorrect 600846336
>> bad block 600395776
>> bad block 600518656
>> bad block 600547328
>> leaf parent key incorrect 600846336
>> bad block 600846336
>> bad block 601710592
>> bad block 603197440
>> Ignoring transid failure
>> parent transid verify failed on 601710592 wanted 23460 found 23463
>> Ignoring transid failure
>> parent transid verify failed on 602529792 wanted 23460 found 23463
>> Ignoring transid failure
>> btrfsck: disk-io.c:155: readahead_tree_block: Assertion `!(ret)' failed.
>> Aborted (core dumped)
>>
>>
>> From dmesg I get the additional information:
>>
>> [  629.447677] btrfs: device fsid f6907d81-f46f-4911-8600-858e8b6bd1a0
>> devid 1 transid 23462 /dev/sda5
>> [  629.671835] systemd-journald[182]: Vacuuming done, freed 0 bytes
>> [  629.843055] systemd-journald[182]: Failed to write entry (26 items,
>> 416256620 bytes) despite vacuuming, ignoring: Argument list too long
>>
>> I am running Arch linux with btrfs-progs-3.12-1 and
>> [root@ho-think bholger]# uname -r
>> 3.12.7-2-ARCH
>>
>> The filesystem got some problems today while being mounted, it got
>> mounted read-only due to an error, but I still had access to my data.
>> I had to unmount it to run btrfsck, but because it fails to repair my
>> disk, I cannot mount it any more.
>>
>> Any advice?
>> Holger

Re: btrfsck failes

[Chris Murphy]

On Jan 13, 2014, at 3:58 PM, Holger Brandsmeier <brandsmeier@gmail.com> wrote:
> 
> Currently btrfsck failes to repair my partition, I get the output:
> 
> [root@ho-think bholger]# btrfsck --repair /dev/sda5

This is almost the last resort and you probably should be posting to the list before using repair.


> [...]
> leaf parent key incorrect 600846336
> parent transid verify failed on 600846336 wanted 23460 found 23463
> Ignoring transid failure
> leaf parent key incorrect 600846336
> bad block 600395776
> bad block 600518656
> bad block 600547328

Please provide the following information for this volume:

btrfs fi show
btrfs fi df <mountpoint>
dmesg
smartctl -x <dev>


> I had to unmount it to run btrfsck, but because it fails to repair my
> disk, I cannot mount it any more.

Try to mount with -o clear_cache.

Chris Murphy

Re: btrfsck failes

[Chris Murphy]

On Jan 14, 2014, at 4:24 PM, Holger Brandsmeier <brandsmeier@gmail.com> wrote:

> Dear Chris,
> 
> as requested:
> 
> # btrfs fi show
> Label: none  uuid: 267f069c-11da-4d2a-88fa-00cab4e28149
>        Total devices 1 FS bytes used 12.68GiB
>        devid    1 size 29.82GiB used 24.27GiB path /dev/sdb1
> Label: none  uuid: f6907d81-f46f-4911-8600-858e8b6bd1a0
>        Total devices 1 FS bytes used 141.72GiB
>        devid    1 size 278.00GiB used 212.04GiB path /dev/sda5
> Btrfs v3.12
> 
> The problem makes `/dev/sda5`. I attached the output of `smartctl -x /dev/sda5`.

There are some reallocated sectors. dmesg reports 
[ 2023.770828] btrfs read error corrected: ino 1 off 605564928 (dev /dev/sda5 sector 1199128)

That might be metadata on a bad sector that was fixed from duplicate metadata. This is single device Btrfs with default DUP metadata?

> 
> # mount -o clear_cache /dev/sda5 data/
> mount: wrong fs type, bad option, bad superblock on /dev/sda5,
>       missing codepage or helper program, or other error
> 

Hmm. So maybe the 1st superblock is no good. But it might be worth trying btrfs-zero-log first. It's possible the repair made this worse.

https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg27613.html



Chris Murphy

Re: btrfsck failes

[Marc MERLIN]

On Mon, Jan 13, 2014 at 05:37:31PM -0700, Chris Murphy wrote:
> 
> On Jan 13, 2014, at 3:58 PM, Holger Brandsmeier <brandsmeier@gmail.com> wrote:
> > 
> > Currently btrfsck failes to repair my partition, I get the output:
> > 
> > [root@ho-think bholger]# btrfsck --repair /dev/sda5
> 
> This is almost the last resort and you probably should be posting to the list before using repair.

A reminder about my previous comment that brtfsck should really give
better usage info and --force or --really-try-to-fsck  ;)

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: btrfsck failes

[Mitch Harder]

On Mon, Jan 13, 2014 at 6:37 PM, Chris Murphy <lists@colorremedies.com> wrote:
>
> On Jan 13, 2014, at 3:58 PM, Holger Brandsmeier <brandsmeier@gmail.com> wrote:
>>
>> Currently btrfsck failes to repair my partition, I get the output:
>>
>> [root@ho-think bholger]# btrfsck --repair /dev/sda5
>
> This is almost the last resort and you probably should be posting to the list before using repair.
>
>

This is like saying:

"Yes, btrfs does now have a working btrfsck, but only for the select
few who manage to get through on the mailing list for support."

I'd like to think that's not the case.

Re: btrfsck failes

[Chris Murphy]

On Jan 15, 2014, at 8:15 AM, Marc MERLIN <marc@merlins.org> wrote:

> On Wed, Jan 15, 2014 at 08:53:55AM +0100, Holger Brandsmeier wrote:
>> # btrfs-zero-log /dev/sda5
>> parent transid verify failed on 602529792 wanted 23460 found 23463
>> parent transid verify failed on 602529792 wanted 23460 found 23463
>> parent transid verify failed on 602529792 wanted 23460 found 23463
>> parent transid verify failed on 602529792 wanted 23460 found 23463
>> Ignoring transid failure
>> btrfs-zero-log: disk-io.c:155: readahead_tree_block: Assertion `!(ret)' failed.
>> Aborted (core dumped)
> 
> I've been seeing this quite a bit too. Is this fixed already, or known
> broken?
> 
> I can run under gdb next time if needed.

I don't know. That alone might be worth a bz on kernel.org. If you do I'd also put the URL in this thread. And then also try building btrfs-progs from the integration branch, and seeing if it's still reproducible and include that information in the bug as well.


> # btrfs-image -c9 -t4 /dev/sda5 /some/path
> parent transid verify failed on 602529792 wanted 23460 found 23463
> parent transid verify failed on 602529792 wanted 23460 found 23463
> parent transid verify failed on 602529792 wanted 23460 found 23463
> parent transid verify failed on 602529792 wanted 23460 found 23463
> Ignoring transid failure
> Error going to next leaf -5
> create failed (Success)
> 
> means this failed or succeeded, it wrote a file which is 1.3mb large.

That last message is confusing. I'd say failed because your metadata allocation was much bigger than 1.3MB. If you build new progs from integration branch it might be worth filing this as a separate bug from recovery/repair attempt bug.


> The two files differ, but to me they both look equally bad. Should I
> try `btrfs-select-super` or `btrfsck --repair --init-extent-tree` /
> `btrfsck --repair --init-csum-tree`.

Archives contain some information on the consequences of those options. I'm pretty sure even if they work enough to let you mount the file system (to grab a more recent backup) that you will need to start over with a new file system. Before you do that I would file a bug for what you've done thus far, with the description being everything you've done, in order, up to this point. What kernel, what progs, that you started with btrfsck --repair, etc. The smartctl output as an attachment, and the dmesg you posted earlier showing the (likely) metadata repair after the read error.

Then upgrade to kernel 3.13rc8 because I can't tell you if any Btrfs changes since 3.12.7 will fix this problem or not and chances are it will be asked anyway. Retry mounting normally, then with -o recovery, -o ro,recovery and report on those results in the bug. Include user spaces messages and dmesg.

If that doesn't work, I would build btrfs-progs from integration branch and try the user space options in the order in Hugo's email. And report those results in the bug. If you get the btrfs-zero-log crash that's probably worth a separate gdb attachment before going on to anything btrfs repair (btrfsck) related.


At any time before btrfs repair you can bail out, include right now, with btrfs restore.
https://btrfs.wiki.kernel.org/index.php/Restore

Grab what you can, and then blow away the file system. But I think you might have an interesting case because something this broken is inherently valuable, especially if it was btrfs repair that made it worse.

Chris Murphy

Re: btrfsck failes

[Chris Murphy]

On Jan 15, 2014, at 9:15 AM, Mitch Harder <mitch.harder@sabayonlinux.org> wrote:

> On Mon, Jan 13, 2014 at 6:37 PM, Chris Murphy <lists@colorremedies.com> wrote:
>> 
>> On Jan 13, 2014, at 3:58 PM, Holger Brandsmeier <brandsmeier@gmail.com> wrote:
>>> 
>>> Currently btrfsck failes to repair my partition, I get the output:
>>> 
>>> [root@ho-think bholger]# btrfsck --repair /dev/sda5
>> 
>> This is almost the last resort and you probably should be posting to the list before using repair.
>> 
>> 
> 
> This is like saying:
> 
> "Yes, btrfs does now have a working btrfsck, but only for the select
> few who manage to get through on the mailing list for support."
> 
> I'd like to think that's not the case.

Yet that's exactly what the wiki suggests: "If you have a broken filesystem, it is probably better to use btrfsck with advice from one of the btrfs developers, just in case something goes wrong. (But even if it does go badly wrong, you've still got your backups, right?)"

I think it's understandably annoying that the repair tool could make things worse rather than fail gracefully, because restoring from backups is tedious. But the only way it gets better is if people break both the file system and the repair tools in ways the devs can't possibly predict. 

Chris Murphy

Re: btrfsck failes

[Marc MERLIN]

On Wed, Jan 15, 2014 at 10:16:13AM -0700, Chris Murphy wrote:
> 
> On Jan 15, 2014, at 9:15 AM, Mitch Harder <mitch.harder@sabayonlinux.org> wrote:
> 
> > On Mon, Jan 13, 2014 at 6:37 PM, Chris Murphy <lists@colorremedies.com> wrote:
> >> 
> >> On Jan 13, 2014, at 3:58 PM, Holger Brandsmeier <brandsmeier@gmail.com> wrote:
> >>> 
> >>> Currently btrfsck failes to repair my partition, I get the output:
> >>> 
> >>> [root@ho-think bholger]# btrfsck --repair /dev/sda5
> >> 
> >> This is almost the last resort and you probably should be posting to the list before using repair.
> >> 
> >> 
> > 
> > This is like saying:
> > 
> > "Yes, btrfs does now have a working btrfsck, but only for the select
> > few who manage to get through on the mailing list for support."
> > 
> > I'd like to think that's not the case.
> 
> Yet that's exactly what the wiki suggests: "If you have a broken filesystem, it is probably better to use btrfsck with advice from one of the btrfs developers, just in case something goes wrong. (But even if it does go badly wrong, you've still got your backups, right?)"
> 
> I think it's understandably annoying that the repair tool could make things worse rather than fail gracefully, because restoring from backups is tedious. But the only way it gets better is if people break both the file system and the repair tools in ways the devs can't possibly predict. 

I hate to be a broken record :) but telling people they should read/have
read a wiki when they are in the middle of fixing a filesystem is not
the right way to go.
On my laptop while travelling, it would even be my only way to boot and
maybe I can't get to the internet until my FS is fixed (maybe, maybe
not, but you get the idea).

My main point again (sorry) is still that the man page and usage info of btrfsck
should really warn users "this is likely NOT what you want to run,
please read the manpage, or HOWTO in /usr/share/doc/ with details about
mount recovery and other things to try first".

Think of it as a good thing, it means more btrfs users, and they are
used to working a certain way. It's for btrfs to adapt to how they're
used to working when possible.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: btrfsck failes

[Holger Brandsmeier]

[-- Attachment #1: Type: text/plain, Size: 3231 bytes --]

I agree to Marc, I created a small patch to `cmds-check.c` (see
attachments). It displays a warning message for 10 seconds if the
repair option is enabled. If you think this is a good idea, can
someone apply this patch?

I am going to roll in my backup soon. Is there anything more that I
should give you as a feedback? I did file one bug:

https://bugzilla.kernel.org/show_bug.cgi?id=68771

On Thu, Jan 16, 2014 at 1:19 PM, Marc MERLIN <marc@merlins.org> wrote:
> On Wed, Jan 15, 2014 at 10:16:13AM -0700, Chris Murphy wrote:
>>
>> On Jan 15, 2014, at 9:15 AM, Mitch Harder <mitch.harder@sabayonlinux.org> wrote:
>>
>> > On Mon, Jan 13, 2014 at 6:37 PM, Chris Murphy <lists@colorremedies.com> wrote:
>> >>
>> >> On Jan 13, 2014, at 3:58 PM, Holger Brandsmeier <brandsmeier@gmail.com> wrote:
>> >>>
>> >>> Currently btrfsck failes to repair my partition, I get the output:
>> >>>
>> >>> [root@ho-think bholger]# btrfsck --repair /dev/sda5
>> >>
>> >> This is almost the last resort and you probably should be posting to the list before using repair.
>> >>
>> >>
>> >
>> > This is like saying:
>> >
>> > "Yes, btrfs does now have a working btrfsck, but only for the select
>> > few who manage to get through on the mailing list for support."
>> >
>> > I'd like to think that's not the case.
>>
>> Yet that's exactly what the wiki suggests: "If you have a broken filesystem, it is probably better to use btrfsck with advice from one of the btrfs developers, just in case something goes wrong. (But even if it does go badly wrong, you've still got your backups, right?)"
>>
>> I think it's understandably annoying that the repair tool could make things worse rather than fail gracefully, because restoring from backups is tedious. But the only way it gets better is if people break both the file system and the repair tools in ways the devs can't possibly predict.
>
> I hate to be a broken record :) but telling people they should read/have
> read a wiki when they are in the middle of fixing a filesystem is not
> the right way to go.
> On my laptop while travelling, it would even be my only way to boot and
> maybe I can't get to the internet until my FS is fixed (maybe, maybe
> not, but you get the idea).
>
> My main point again (sorry) is still that the man page and usage info of btrfsck
> should really warn users "this is likely NOT what you want to run,
> please read the manpage, or HOWTO in /usr/share/doc/ with details about
> mount recovery and other things to try first".
>
> Think of it as a good thing, it means more btrfs users, and they are
> used to working a certain way. It's for btrfs to adapt to how they're
> used to working when possible.
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: cmds-check.patch --]
[-- Type: text/x-patch, Size: 732 bytes --]

--- cmds-check.c
+++ cmds-check.c
@@ -6427,6 +6427,15 @@ int cmd_check(int argc, char **argv)
        if (argc != 1)
                usage(cmd_check_usage);
 
+       if (repair != 0) {
+               printf("\nWARNING: you enabled repair mode, this option could make "
+                               "matters worse for your data. It is recommended to check with the "
+                               "linux-btrfs@vger.kernel.org mailing list before you proceed.\n");
+               printf("Sleeping for 10 seconds to give you time to consider..."
+                               " hit ctrl+c if you changed your mind.\n");
+               sleep(10);
+       }
+
        radix_tree_init();
        cache_tree_init(&root_cache);