Unable to open ctree

Unable to open ctree

[Leonidas Spyropoulos]

Hi all,

I'm a everyday user of btrfs. The system is a laptop with UEFI partition
and the following partition scheme:
# blkid 
/dev/sda1: UUID="." TYPE="vfat" PARTUUID="."
/dev/sda2: UUID="." TYPE="swap" PARTUUID="."
/dev/sda3: LABEL="root" UUID="." UUID_SUB="." TYPE="btrfs" PARTUUID="."
/dev/sda4: LABEL="home" UUID="." UUID_SUB="." TYPE="btrfs" PARTUUID="."

The partitions are surely not out of space:
# btrfs fi show
Label: 'root'  uuid: c7a773e0-e2be-4482-acdc-dbfa4c13c98c
    	Total devices 1 FS bytes used 9.98GiB
		devid    1 size 74.52GiB used 13.03GiB path /dev/sda3

Label: 'home'  uuid: 63f5219f-399a-4cce-be86-a8c7400853bd
		Total devices 1 FS bytes used 17.65GiB
		devid    1 size 188.87GiB used 25.03GiB path /dev/sda4

When the "root" partition is to be mounted during boot up it fails and I
am dropped on an emergency shell. The dmesg output is:

[  716.434137] BTRFS info (device sda3): disk space caching is enabled
[  716.434147] BTRFS info (device sda3): has skinny extents
[  716.436538] BTRFS info (device sda3): bdev /dev/sda3 errs: wr 0, rd
0, flush 0, corrupt 32, gen 0
[  716.439182] BTRFS warning (device sda3): sda3 checksum verify failed
on 11185160192 wanted 2670F0AB found FDB2F5BD level 0
[  716.439195] BTRFS error (device sda3): failed to read block groups:
-5
[  716.481402] BTRFS: open_ctree failed

The btrfs check /dev/sdc3 output is https://ptpb.pw/nc1d

dmesg output: https://ptpb.pw/G0Wv

The kernel I was running is right before this failure was 4.7.5 (with ck
patchset custom compiled) and there was not any kernel panics - just a
restart. Currently runnign through latest Archlinux liveCD

# uname -a
Linux archiso 4.7.5-1-ARCH #1 SMP PREEMPT Sat Sep 24 13:04:22 CEST 2016
x86_64 GNU/Linux

# btrfs --version
btrfs-progs v4.7.3

As this is the root partition I am not able to access it without a
liveCD. Can I perform any kind of steps to recaclulate checksums?

Thanks

-- 
Leonidas Spyropoulos

A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

Re: Unable to open ctree

[Chris Murphy]

On Sun, Oct 2, 2016 at 12:17 PM, Leonidas Spyropoulos
<artafinde@gmail.com> wrote:
> Hi all,
>
> I'm a everyday user of btrfs. The system is a laptop with UEFI partition
> and the following partition scheme:
> # blkid
> /dev/sda1: UUID="." TYPE="vfat" PARTUUID="."
> /dev/sda2: UUID="." TYPE="swap" PARTUUID="."
> /dev/sda3: LABEL="root" UUID="." UUID_SUB="." TYPE="btrfs" PARTUUID="."
> /dev/sda4: LABEL="home" UUID="." UUID_SUB="." TYPE="btrfs" PARTUUID="."
>
> The partitions are surely not out of space:
> # btrfs fi show
> Label: 'root'  uuid: c7a773e0-e2be-4482-acdc-dbfa4c13c98c
>         Total devices 1 FS bytes used 9.98GiB
>                 devid    1 size 74.52GiB used 13.03GiB path /dev/sda3
>
> Label: 'home'  uuid: 63f5219f-399a-4cce-be86-a8c7400853bd
>                 Total devices 1 FS bytes used 17.65GiB
>                 devid    1 size 188.87GiB used 25.03GiB path /dev/sda4
>
> When the "root" partition is to be mounted during boot up it fails and I
> am dropped on an emergency shell. The dmesg output is:
>
> [  716.434137] BTRFS info (device sda3): disk space caching is enabled
> [  716.434147] BTRFS info (device sda3): has skinny extents
> [  716.436538] BTRFS info (device sda3): bdev /dev/sda3 errs: wr 0, rd
> 0, flush 0, corrupt 32, gen 0
> [  716.439182] BTRFS warning (device sda3): sda3 checksum verify failed
> on 11185160192 wanted 2670F0AB found FDB2F5BD level 0
> [  716.439195] BTRFS error (device sda3): failed to read block groups:
> -5
> [  716.481402] BTRFS: open_ctree failed
>
> The btrfs check /dev/sdc3 output is https://ptpb.pw/nc1d
>
> dmesg output: https://ptpb.pw/G0Wv
>
> The kernel I was running is right before this failure was 4.7.5 (with ck
> patchset custom compiled) and there was not any kernel panics - just a
> restart. Currently runnign through latest Archlinux liveCD
>
> # uname -a
> Linux archiso 4.7.5-1-ARCH #1 SMP PREEMPT Sat Sep 24 13:04:22 CEST 2016
> x86_64 GNU/Linux
>
> # btrfs --version
> btrfs-progs v4.7.3
>
> As this is the root partition I am not able to access it without a
> liveCD. Can I perform any kind of steps to recaclulate checksums?

Well short of a bug, the problem aren't the checksums. The problem is
the metadata is wrong, so if you recalculate checksums you're likely
end up with an even more corrupted file system because it'll start out
trusting bad metadata.

If you're prepared to lose this filesystem, use --repair and see if it
can fix the metadata problems despite csum failures. If it were me,
I'd take a btrfs-image before --repair. In theory, if it makes things
worse you can restore the image, or donate the image to making the
btrfsck better.

If --repair doesn't work, try -b --repair.

If that doesn't work then I'd probably use --init-extent-tree which,
while it's a heavy hammer, at least still isn't going to pretend bad
metadata is good which is what --init-csum-tree will end up doing.

But before all of that I'm curious what you get for:

btrfs-debug-tree -b 11185160192 /dev/sda3
btrfs-find-root /dev/sda3





-- 
Chris Murphy

Re: Unable to open ctree

[Leonidas Spyropoulos]

Hi Chris,

Thanks for your suggestions

On 02/10/16, Chris Murphy wrote:
> Well short of a bug, the problem aren't the checksums. The problem is
> the metadata is wrong, so if you recalculate checksums you're likely
> end up with an even more corrupted file system because it'll start out
> trusting bad metadata.
> 
> If you're prepared to lose this filesystem, use --repair and see if it
> can fix the metadata problems despite csum failures. If it were me,
> I'd take a btrfs-image before --repair. In theory, if it makes things
> worse you can restore the image, or donate the image to making the
> btrfsck better.
> 
> If --repair doesn't work, try -b --repair.
> 
> If that doesn't work then I'd probably use --init-extent-tree which,
> while it's a heavy hammer, at least still isn't going to pretend bad
> metadata is good which is what --init-csum-tree will end up doing.
> 
> But before all of that I'm curious what you get for:
> 
> btrfs-debug-tree -b 11185160192 /dev/sda3
> btrfs-find-root /dev/sda3

The output from btrfs-debug-tree command is: https://ptpb.pw/0weU
The btrfs-find-root: https://ptpb.pw/PIZe
The btrfs-show-super -f : https://ptpb.pw/O4C9

I tried the btrfs-image but failed
# btrfs-image /dev/sda3 /root/sda3-btrfs-image.bin
with output https://ptpb.pw/jtDJ

Unless there's some other idea I will continue with the following:
# btrfs check --repair
# btrfs check -b --repair
# btrfs check --init-extent-tree --repair

and see if I can get something

Thanks,

-- 
Leonidas Spyropoulos

A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

Re: Unable to open ctree

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1039 bytes --]

On Sun, 2 Oct 2016 13:29:56 -0600
Chris Murphy <lists@colorremedies.com> wrote:

> Well short of a bug, the problem aren't the checksums. The problem is
> the metadata is wrong, so if you recalculate checksums you're likely
> end up with an even more corrupted file system because it'll start out
> trusting bad metadata.

It appears that metadata got created with profile "single", because the device
is SSD. If this was DUP metadata, this entire problem wouldn't happen.
It is a terrible idea to downgrade metadata to single on detecting SSDs at
mkfs. The original rationale was that "SSDs will deduplicate it anyways", but
there are many ways things can corrupt way before reaching the SSD (from the
point of view of which it will look like the computer sent two different
metadata blocks, if one got corrupted in flight), and secondly, the ability of
SSDs to perfectly deduplicate small 4K sized pieces of data at hundreds of
megabytes in read/write speeds is VASTLY overestimated here.


-- 
With respect,
Roman

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: Unable to open ctree

[Chris Murphy]

On Sun, Oct 2, 2016 at 2:22 PM, Roman Mamedov <rm@romanrm.net> wrote:
> On Sun, 2 Oct 2016 13:29:56 -0600
> Chris Murphy <lists@colorremedies.com> wrote:
>
>> Well short of a bug, the problem aren't the checksums. The problem is
>> the metadata is wrong, so if you recalculate checksums you're likely
>> end up with an even more corrupted file system because it'll start out
>> trusting bad metadata.
>
> It appears that metadata got created with profile "single", because the device
> is SSD. If this was DUP metadata, this entire problem wouldn't happen.
> It is a terrible idea to downgrade metadata to single on detecting SSDs at
> mkfs.

That's uncertain, we've seen file systems with dup also not mount with
csum errors in metadata.



-- 
Chris Murphy

Re: Unable to open ctree

[Leonidas Spyropoulos]

Hello,

After the command to rebuild the extent tree
# btrfs check --init-extent-tree --repair /dev/sda3

I was able to mount and run successfully a scrub (not sure if it would
detect something though).

Next was reboot and try to boot which was successful as well.

I'll keep an eye on the dmesh log next days but seems it's ok.

Thanks for suggestions all,

Regards,

-- 
Leonidas Spyropoulos

A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

Re: Unable to open ctree

[Leonidas Spyropoulos]

Hi Roman,


On 03/10/16, Roman Mamedov wrote:
> On Sun, 2 Oct 2016 13:29:56 -0600
> Chris Murphy <lists@colorremedies.com> wrote:
> 
> > Well short of a bug, the problem aren't the checksums. The problem is
> > the metadata is wrong, so if you recalculate checksums you're likely
> > end up with an even more corrupted file system because it'll start out
> > trusting bad metadata.
> 
> It appears that metadata got created with profile "single", because the device
> is SSD. If this was DUP metadata, this entire problem wouldn't happen.
> It is a terrible idea to downgrade metadata to single on detecting SSDs at
> mkfs. The original rationale was that "SSDs will deduplicate it anyways", but
> there are many ways things can corrupt way before reaching the SSD (from the
> point of view of which it will look like the computer sent two different
> metadata blocks, if one got corrupted in flight), and secondly, the ability of
> SSDs to perfectly deduplicate small 4K sized pieces of data at hundreds of
> megabytes in read/write speeds is VASTLY overestimated here.

I agree that the wear in SSDs due to metadata dublications is
overestimated and since my partition appears to be screwed I will make
sure next time to force it to have another copy there.

My progress from the repairs which both failed are: https://ptpb.pw/CtbC

Next step is initialize the extent tree
btrfs check --init-extent-tree --repair /dev/sda3

Thanks,

-- 
Leonidas Spyropoulos

A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?