Unable to mount, corrupt leaf

Unable to mount, corrupt leaf

[Cesar Strauss]

Hello,

After a BTRFS partition becoming read-only under use, it cannot be 
mounted anymore.

The output is:

# mount /dev/sdb5 /mnt/disk1
mount: /mnt/disk1: wrong fs type, bad option, bad superblock on 
/dev/sdb5, missing codepage or helper program, or other error.

Kernel output:
[ 2042.106654] BTRFS info (device sdb5): disk space caching is enabled
[ 2042.799537] BTRFS critical (device sdb5): corrupt leaf: root=2 
block=199940210688 slot=31, unexpected item end, have 268450090 expect 14634
[ 2042.807879] BTRFS critical (device sdb5): corrupt leaf: root=2 
block=199940210688 slot=31, unexpected item end, have 268450090 expect 14634
[ 2042.807947] BTRFS error (device sdb5): failed to read block groups: -5
[ 2042.832362] BTRFS error (device sdb5): open_ctree failed

# btrfs check /dev/sdb5
Opening filesystem to check...
incorrect offsets 14634 268450090
incorrect offsets 14634 268450090
incorrect offsets 14634 268450090
incorrect offsets 14634 268450090
ERROR: cannot open file system

Giving -s and -b options to "btrfs check" made no difference.

The usebackuproot mount option made no difference.

"btrfs restore" was successful in recovering most of the files, except 
for a couple instances of "Error copying data".

System information:

OS: Arch Linux

$ uname -a
Linux rescue 5.1.4-arch1-1-ARCH #1 SMP PREEMPT Wed May 22 08:06:56 UTC 
2019 x86_64 GNU/Linux

$ btrfs --version
btrfs-progs v5.1

I have since updated the kernel, with no difference:

$ uname -a
Linux rescue 5.1.5-arch1-2-ARCH #1 SMP PREEMPT Mon May 27 03:37:39 UTC 
2019 x86_64 GNU/Linux

Before making any recovery attempts, or even restoring from backup, I 
would like to ask for the best option to proceed.

Thanks,

Cesar

Re: Unable to mount, corrupt leaf

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2716 bytes --]

On Tue, May 28, 2019 at 03:39:36PM -0300, Cesar Strauss wrote:
> Hello,
> 
> After a BTRFS partition becoming read-only under use, it cannot be
> mounted anymore.
> 
> The output is:
> 
> # mount /dev/sdb5 /mnt/disk1
> mount: /mnt/disk1: wrong fs type, bad option, bad superblock on
> /dev/sdb5, missing codepage or helper program, or other error.
> 
> Kernel output:
> [ 2042.106654] BTRFS info (device sdb5): disk space caching is enabled
> [ 2042.799537] BTRFS critical (device sdb5): corrupt leaf: root=2
> block=199940210688 slot=31, unexpected item end, have 268450090
> expect 14634

   You have bad RAM.

The item end it's got on the disk:
>>> hex(268450090)
'0x1000392a'

The item end it should have (based on the other items and their
lengths and positions):
>>> hex(14634)
'0x392a'

   The good checksum on the block (it hasn't complained about the
csum, so it's good) indicates that the corruption happened in memory
at some point. The bit-flip in the data would strongly suggest that
it's caused by a stuck memory cell -- i.e. bad hardware.

   Run memtest86 for a minimum of 8 hours (preferably 24) and see what
shows up. Then fix the hardware.

   Hugo.

> [ 2042.807879] BTRFS critical (device sdb5): corrupt leaf: root=2
> block=199940210688 slot=31, unexpected item end, have 268450090
> expect 14634
> [ 2042.807947] BTRFS error (device sdb5): failed to read block groups: -5
> [ 2042.832362] BTRFS error (device sdb5): open_ctree failed
> 
> # btrfs check /dev/sdb5
> Opening filesystem to check...
> incorrect offsets 14634 268450090
> incorrect offsets 14634 268450090
> incorrect offsets 14634 268450090
> incorrect offsets 14634 268450090
> ERROR: cannot open file system
> 
> Giving -s and -b options to "btrfs check" made no difference.
> 
> The usebackuproot mount option made no difference.
> 
> "btrfs restore" was successful in recovering most of the files,
> except for a couple instances of "Error copying data".
> 
> System information:
> 
> OS: Arch Linux
> 
> $ uname -a
> Linux rescue 5.1.4-arch1-1-ARCH #1 SMP PREEMPT Wed May 22 08:06:56
> UTC 2019 x86_64 GNU/Linux
> 
> $ btrfs --version
> btrfs-progs v5.1
> 
> I have since updated the kernel, with no difference:
> 
> $ uname -a
> Linux rescue 5.1.5-arch1-2-ARCH #1 SMP PREEMPT Mon May 27 03:37:39
> UTC 2019 x86_64 GNU/Linux
> 
> Before making any recovery attempts, or even restoring from backup,
> I would like to ask for the best option to proceed.
> 
> Thanks,
> 
> Cesar

-- 
Hugo Mills             | You've read the project plan. Forget that. We're
hugo@... carfax.org.uk | going to Do Stuff and Have Fun doing it.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                                           Jeremy Frey

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Unable to mount, corrupt leaf

[Cesar Strauss]

On 05/28/2019 15:54, Hugo Mills wrote:
> 
>    You have bad RAM.
> 

On 05/28/2019 15:56, Hans van Kranenburg wrote:
> So, what you're observing here is a bitflip. One of the zeros flipped
> into a one, which caused the 14634 to suddenly become 268450090.

Indeed, I had RAM trouble some time ago.

Thank you both for the advice. I'll restore from a backup after I solve
the RAM problem.

Regards,

Cesar

Re: Unable to mount, corrupt leaf

[Hans van Kranenburg]

Hi,

On 5/28/19 8:39 PM, Cesar Strauss wrote:
> 
> After a BTRFS partition becoming read-only under use, it cannot be 
> mounted anymore.
> 
> The output is:
> 
> # mount /dev/sdb5 /mnt/disk1
> mount: /mnt/disk1: wrong fs type, bad option, bad superblock on 
> /dev/sdb5, missing codepage or helper program, or other error.
> 
> Kernel output:
> [ 2042.106654] BTRFS info (device sdb5): disk space caching is enabled
> [ 2042.799537] BTRFS critical (device sdb5): corrupt leaf: root=2 
> block=199940210688 slot=31, unexpected item end, have 268450090 expect 14634


bin(14634) ->
               '0b11100100101010'

bin(268450090) ->
'0b10000000000000011100100101010'

So, what you're observing here is a bitflip. One of the zeros flipped
into a one, which caused the 14634 to suddenly become 268450090.

Since this is detected as corruption, and not detected using the btrfs
checksums, it means that the bitflip did not happen when the value was
stored on disk, but it happened in your memory chips inside the computer
before the information was written to disk.

> [ 2042.807879] BTRFS critical (device sdb5): corrupt leaf: root=2 
> block=199940210688 slot=31, unexpected item end, have 268450090 expect 14634
> [ 2042.807947] BTRFS error (device sdb5): failed to read block groups: -5
> [ 2042.832362] BTRFS error (device sdb5): open_ctree failed
> 
> [...]
> 
> Before making any recovery attempts, or even restoring from backup, I 
> would like to ask for the best option to proceed.

The best thing to do is to not use this phsyical computer for anything
else right now than running a proper program to analyze and check the
memory chips, in order to find out which one is causing the problems.

Hans