Corrupted filesystem after power loss

Corrupted filesystem after power loss

[Benedikt Rips]

[-- Attachment #1: Type: text/plain, Size: 3355 bytes --]

Hi there,

two weeks ago, I forcibly shut down my system when it was frozen, by pressing
the power button for several seconds. At the next boot, I was not able to 
mount
the filesystem. I booted from a usb stick and at mounting my root filesystem
(which is btrfs), I got the following error messages:


# journalctl -qxeb | tail ... | head ...
Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): trying to use backup 
root at mount time
Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): use zstd 
compression, level 3
Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): disk space caching 
is enabled
Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): has skinny extents
Aug 28 16:43:21 archiso kernel: BTRFS warning (device dm-2): dm-2 checksum 
verify failed on 95634915328 wanted 59c7037e found 97021a59 level 0
Aug 28 16:43:21 archiso kernel: BTRFS warning (device dm-2): failed to read 
root (objectid=2): -5
Aug 28 16:43:21 archiso kernel: BTRFS critical (device dm-2): corrupt leaf: 
root=18446744073709551610 block=95602491392 slot=0 ino=10363009, invalid inode 
generation: has 1518459 expect [0, 1518458]
Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): block=95602491392 
read time tree block corruption detected
Aug 28 16:43:21 archiso kernel: BTRFS warning (device dm-2): failed to read 
tree root
Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): parent transid 
verify failed on 95599607808 wanted 1518455 found 1518457
Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): bdev /dev/mapper/
lvm-linux errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
Aug 28 16:43:21 archiso kernel: BTRFS critical (device dm-2): corrupt leaf: 
root=261 block=95596232704 slot=112 ino=11473161, invalid inode generation: 
has 1518457 expect (0, 1518456]
Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): block=95596232704 
read time tree block corruption detected
Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): failed to read 
block groups: -5
Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): open_ctree failed


The filesystem is on an SSD, my kernel version at the time of the failure was
v5.8.5 and my btrfs-progs version is v5.7.  The information regarding the SSD
are:


# smartctl --info /dev/nvme0
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.8.7-arch1-1] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       THNSN5256GPUK NVMe TOSHIBA 256GB
Serial Number:                      Y6EB70N0KMBU
Firmware Version:                   5KDA4103
PCI Vendor/Subsystem ID:            0x1179
IEEE OUI Identifier:                0x00080d
Controller ID:                      0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          256.060.514.304 [256 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            00080d 0300085baf
Local Time is:                      Wed Sep  9 00:18:09 2020 CEST


After reading through the btrfs wiki (btrfs-restore, btrfs-rescue, btrfs-
check),
I asked on the IRC channel for help.  With the advice of cmurf and darkling I
ran the following commands, trying to find the cause of this error and a
suitable backup root to restore data from: http://cwillu.com:
8080/37.201.170.65/1

Kind regards,
Benedikt

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Corrupted filesystem after power loss

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 4262 bytes --]



On 2020/9/9 下午8:13, Benedikt Rips wrote:
> Hi there,
> 
> two weeks ago, I forcibly shut down my system when it was frozen, by pressing
> the power button for several seconds. At the next boot, I was not able to 
> mount
> the filesystem. I booted from a usb stick and at mounting my root filesystem
> (which is btrfs), I got the following error messages:
> 
> 
> # journalctl -qxeb | tail ... | head ...
> Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): trying to use backup 
> root at mount time
> Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): use zstd 
> compression, level 3
> Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): disk space caching 
> is enabled
> Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): has skinny extents
> Aug 28 16:43:21 archiso kernel: BTRFS warning (device dm-2): dm-2 checksum 
> verify failed on 95634915328 wanted 59c7037e found 97021a59 level 0
> Aug 28 16:43:21 archiso kernel: BTRFS warning (device dm-2): failed to read 
> root (objectid=2): -5

This is the real problem, extent tree get csum corruption.

But since you're using backup roots, it should just rollback to next backup.


> Aug 28 16:43:21 archiso kernel: BTRFS critical (device dm-2): corrupt leaf: 
> root=18446744073709551610 block=95602491392 slot=0 ino=10363009, invalid inode 
> generation: has 1518459 expect [0, 1518458]

This is in fact not a bug, but indicates your fs finds a good backup,
but then log tree is triggering problems.

As log tree is newer than your backup root, tree-checker is rejecting it.

This is easy to handle, just zero the log by 'btrfs rescue zero-log'
should allow you mount the fs with "-o rescue=usebackuproot".

Furthermore, we should not allow log replay if we use "usebackuproot"
option at all.

I'll add extra check on that for the kernel.

Thanks,
Qu

> Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): block=95602491392 
> read time tree block corruption detected
> Aug 28 16:43:21 archiso kernel: BTRFS warning (device dm-2): failed to read 
> tree root
> Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): parent transid 
> verify failed on 95599607808 wanted 1518455 found 1518457
> Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): bdev /dev/mapper/
> lvm-linux errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
> Aug 28 16:43:21 archiso kernel: BTRFS critical (device dm-2): corrupt leaf: 
> root=261 block=95596232704 slot=112 ino=11473161, invalid inode generation: 
> has 1518457 expect (0, 1518456]
> Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): block=95596232704 
> read time tree block corruption detected
> Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): failed to read 
> block groups: -5
> Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): open_ctree failed
> 
> 
> The filesystem is on an SSD, my kernel version at the time of the failure was
> v5.8.5 and my btrfs-progs version is v5.7.  The information regarding the SSD
> are:
> 
> 
> # smartctl --info /dev/nvme0
> smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.8.7-arch1-1] (local build)
> Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
> 
> === START OF INFORMATION SECTION ===
> Model Number:                       THNSN5256GPUK NVMe TOSHIBA 256GB
> Serial Number:                      Y6EB70N0KMBU
> Firmware Version:                   5KDA4103
> PCI Vendor/Subsystem ID:            0x1179
> IEEE OUI Identifier:                0x00080d
> Controller ID:                      0
> Number of Namespaces:               1
> Namespace 1 Size/Capacity:          256.060.514.304 [256 GB]
> Namespace 1 Formatted LBA Size:     512
> Namespace 1 IEEE EUI-64:            00080d 0300085baf
> Local Time is:                      Wed Sep  9 00:18:09 2020 CEST
> 
> 
> After reading through the btrfs wiki (btrfs-restore, btrfs-rescue, btrfs-
> check),
> I asked on the IRC channel for help.  With the advice of cmurf and darkling I
> ran the following commands, trying to find the cause of this error and a
> suitable backup root to restore data from: http://cwillu.com:
> 8080/37.201.170.65/1
> 
> Kind regards,
> Benedikt
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]