Help requested: btrfs can't read superblock

Help requested: btrfs can't read superblock

[kind.moon1862]

Dear btrfs team,

I've been asked to help someone with a btrfs problem.  They use CentOS 7.  I've been told the host was shut down cleanly, but it won't mount a btrfs filesystem when powering on.

Below is what I've done so far.  Any guidance would be appreciated!

I booted using SystemRescue 11.0.  Some info about the rescue OS:

# uname -a
Linux sysrescue 6.6.14-1-lts #1 SMP PREEMPT_DYNAMIC Fri, 26 Jan 2024 11:54:58 +0000 x86_64 GNU/Linux

# btrfs --version
btrfs-progs v6.6.3

Here are the disks:

# btrfs fi show
Label: '<redacted>'  uuid: 834b4bfd-fbd4-40b9-8868-d1187c3c5a63
	Total devices 4 FS bytes used 17.44TiB
	devid    1 size 9.10TiB used 8.94TiB path /dev/sda
	devid    2 size 9.10TiB used 8.94TiB path /dev/sdb
	devid    3 size 9.10TiB used 8.94TiB path /dev/sdc
	devid    4 size 9.10TiB used 8.94TiB path /dev/sdd

I tried mounting:

# mount -t btrfs -o recovery,ro /dev/sda /mnt/data
mount: /mnt/data: can't read superblock on /dev/sda.
       dmesg(1) may have more information after failed mount system call.

dmesg shows this:

[ 1940.962632] BTRFS info (device sda): first mount of filesystem 834b4bfd-fbd4-40b9-8868-d1187c3c5a63
[ 1940.962665] BTRFS info (device sda): using crc32c (crc32c-intel) checksum algorithm
[ 1940.962686] BTRFS warning (device sda): 'recovery' is deprecated, use 'rescue=usebackuproot' instead
[ 1940.962692] BTRFS info (device sda): trying to use backup root at mount time
[ 1940.962698] BTRFS info (device sda): disk space caching is enabled
[ 1942.115796] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096
[ 1942.115819] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 2
[ 1942.116170] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096
[ 1942.116183] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 1
[ 1942.116212] BTRFS error (device sda): failed to read block groups: -5
[ 1942.140369] BTRFS error (device sda): open_ctree failed

I tried to recover the superblock:

# btrfs rescue super-recover -v /dev/sda

All Devices:
	Device: id = 3, name = /dev/sdc
	Device: id = 2, name = /dev/sdb
	Device: id = 4, name = /dev/sdd
	Device: id = 1, name = /dev/sda

Before Recovering:
	[All good supers]:
		device name = /dev/sdc
		superblock bytenr = 65536

		device name = /dev/sdc
		superblock bytenr = 67108864

		device name = /dev/sdc
		superblock bytenr = 274877906944

		device name = /dev/sdb
		superblock bytenr = 65536

		device name = /dev/sdb
		superblock bytenr = 67108864

		device name = /dev/sdb
		superblock bytenr = 274877906944

		device name = /dev/sdd
		superblock bytenr = 65536

		device name = /dev/sdd
		superblock bytenr = 67108864

		device name = /dev/sdd
		superblock bytenr = 274877906944

		device name = /dev/sda
		superblock bytenr = 65536

		device name = /dev/sda
		superblock bytenr = 67108864

		device name = /dev/sda
		superblock bytenr = 274877906944

	[All bad supers]:

All supers are valid, no need to recover

I checked the filesystem:

# btrfs check --readonly /dev/sda
Opening filesystem to check...
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
ERROR: failed to read block groups: Operation not permitted
ERROR: cannot open file system

Re: Help requested: btrfs can't read superblock

[Qu Wenruo]

在 2024/3/19 07:52, kind.moon1862@fastmail.com 写道:
>
> Dear btrfs team,
>
> I've been asked to help someone with a btrfs problem.  They use CentOS 7.  I've been told the host was shut down cleanly, but it won't mount a btrfs filesystem when powering on.
>
> Below is what I've done so far.  Any guidance would be appreciated!
>
> I booted using SystemRescue 11.0.  Some info about the rescue OS:
>
> # uname -a
> Linux sysrescue 6.6.14-1-lts #1 SMP PREEMPT_DYNAMIC Fri, 26 Jan 2024 11:54:58 +0000 x86_64 GNU/Linux
>
> # btrfs --version
> btrfs-progs v6.6.3
>
> Here are the disks:
>
> # btrfs fi show
> Label: '<redacted>'  uuid: 834b4bfd-fbd4-40b9-8868-d1187c3c5a63
> 	Total devices 4 FS bytes used 17.44TiB
> 	devid    1 size 9.10TiB used 8.94TiB path /dev/sda
> 	devid    2 size 9.10TiB used 8.94TiB path /dev/sdb
> 	devid    3 size 9.10TiB used 8.94TiB path /dev/sdc
> 	devid    4 size 9.10TiB used 8.94TiB path /dev/sdd
>
> I tried mounting:
>
> # mount -t btrfs -o recovery,ro /dev/sda /mnt/data
> mount: /mnt/data: can't read superblock on /dev/sda.
>         dmesg(1) may have more information after failed mount system call.

Use "mount -o rescue=all,ro" instead.

>
> dmesg shows this:
>
> [ 1940.962632] BTRFS info (device sda): first mount of filesystem 834b4bfd-fbd4-40b9-8868-d1187c3c5a63
> [ 1940.962665] BTRFS info (device sda): using crc32c (crc32c-intel) checksum algorithm
> [ 1940.962686] BTRFS warning (device sda): 'recovery' is deprecated, use 'rescue=usebackuproot' instead
> [ 1940.962692] BTRFS info (device sda): trying to use backup root at mount time
> [ 1940.962698] BTRFS info (device sda): disk space caching is enabled
> [ 1942.115796] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096

Bad extent item start bytenr, 5822947745796 = 0x54bc2bb6004, the last
0x004 looks like a bitflip.

Please run memtest to make sure your hardware memory is fine, or such
problem would just happen again and again.

Thankfully extent tree corruption is not a big deal for "rescue=all,ro"
mount.

Thanks,
Qu

> [ 1942.115819] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 2
> [ 1942.116170] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096
> [ 1942.116183] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 1
> [ 1942.116212] BTRFS error (device sda): failed to read block groups: -5
> [ 1942.140369] BTRFS error (device sda): open_ctree failed
>
> I tried to recover the superblock:
>
> # btrfs rescue super-recover -v /dev/sda
>
> All Devices:
> 	Device: id = 3, name = /dev/sdc
> 	Device: id = 2, name = /dev/sdb
> 	Device: id = 4, name = /dev/sdd
> 	Device: id = 1, name = /dev/sda
>
> Before Recovering:
> 	[All good supers]:
> 		device name = /dev/sdc
> 		superblock bytenr = 65536
>
> 		device name = /dev/sdc
> 		superblock bytenr = 67108864
>
> 		device name = /dev/sdc
> 		superblock bytenr = 274877906944
>
> 		device name = /dev/sdb
> 		superblock bytenr = 65536
>
> 		device name = /dev/sdb
> 		superblock bytenr = 67108864
>
> 		device name = /dev/sdb
> 		superblock bytenr = 274877906944
>
> 		device name = /dev/sdd
> 		superblock bytenr = 65536
>
> 		device name = /dev/sdd
> 		superblock bytenr = 67108864
>
> 		device name = /dev/sdd
> 		superblock bytenr = 274877906944
>
> 		device name = /dev/sda
> 		superblock bytenr = 65536
>
> 		device name = /dev/sda
> 		superblock bytenr = 67108864
>
> 		device name = /dev/sda
> 		superblock bytenr = 274877906944
>
> 	[All bad supers]:
>
> All supers are valid, no need to recover
>
> I checked the filesystem:
>
> # btrfs check --readonly /dev/sda
> Opening filesystem to check...
> corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
> corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
> corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
> corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
> ERROR: failed to read block groups: Operation not permitted
> ERROR: cannot open file system
>
>

Re: Help requested: btrfs can't read superblock

[kind.moon1862]

On Mon, Mar 18, 2024, at 17:06, Qu Wenruo wrote:

> Use "mount -o rescue=all,ro" instead.

Thank you for your suggestion.  Should I run it under CentOS 7 (the original OS) or under the newer kernel in the SystemRescue distro?

Re: Help requested: btrfs can't read superblock

[Qu Wenruo]

在 2024/3/19 11:03, kind.moon1862@fastmail.com 写道:
> On Mon, Mar 18, 2024, at 17:06, Qu Wenruo wrote:
>
>> Use "mount -o rescue=all,ro" instead.
>
> Thank you for your suggestion.  Should I run it under CentOS 7 (the original OS) or under the newer kernel in the SystemRescue distro?

Newer kernel please.

And still, please run memtest before doing any rescue.
As faulty hardware can always lead to weird problems no matter what.

And CentOS 7 is not recommended for btrfs usage (no proper btrfs related
backport and old kernels, IIRC it's already EOL)

Thanks,
Qu

Re: Help requested: btrfs can't read superblock

[kind.moon1862]

On Mon, Mar 18, 2024, at 17:53, Qu Wenruo wrote:

> And still, please run memtest before doing any rescue.
> As faulty hardware can always lead to weird problems no matter what.

memtest86+ found bad RAM.  I will replace that before proceeding.  Thanks again.

Re: Help requested: btrfs can't read superblock

[Qu Wenruo]

在 2024/3/20 07:14, kind.moon1862@fastmail.com 写道:
> On Mon, Mar 18, 2024, at 17:53, Qu Wenruo wrote:
>
>> And still, please run memtest before doing any rescue.
>> As faulty hardware can always lead to weird problems no matter what.
>
> memtest86+ found bad RAM.  I will replace that before proceeding.  Thanks again.

Btrfs tree-checker, the second best memory tester. :)