Help requested: btrfs can't read superblock

[kind.moon1862@fastmail.com]

Dear btrfs team,

I've been asked to help someone with a btrfs problem.  They use CentOS 7.  I've been told the host was shut down cleanly, but it won't mount a btrfs filesystem when powering on.

Below is what I've done so far.  Any guidance would be appreciated!

I booted using SystemRescue 11.0.  Some info about the rescue OS:

# uname -a
Linux sysrescue 6.6.14-1-lts #1 SMP PREEMPT_DYNAMIC Fri, 26 Jan 2024 11:54:58 +0000 x86_64 GNU/Linux

# btrfs --version
btrfs-progs v6.6.3

Here are the disks:

# btrfs fi show
Label: '<redacted>'  uuid: 834b4bfd-fbd4-40b9-8868-d1187c3c5a63
	Total devices 4 FS bytes used 17.44TiB
	devid    1 size 9.10TiB used 8.94TiB path /dev/sda
	devid    2 size 9.10TiB used 8.94TiB path /dev/sdb
	devid    3 size 9.10TiB used 8.94TiB path /dev/sdc
	devid    4 size 9.10TiB used 8.94TiB path /dev/sdd

I tried mounting:

# mount -t btrfs -o recovery,ro /dev/sda /mnt/data
mount: /mnt/data: can't read superblock on /dev/sda.
       dmesg(1) may have more information after failed mount system call.

dmesg shows this:

[ 1940.962632] BTRFS info (device sda): first mount of filesystem 834b4bfd-fbd4-40b9-8868-d1187c3c5a63
[ 1940.962665] BTRFS info (device sda): using crc32c (crc32c-intel) checksum algorithm
[ 1940.962686] BTRFS warning (device sda): 'recovery' is deprecated, use 'rescue=usebackuproot' instead
[ 1940.962692] BTRFS info (device sda): trying to use backup root at mount time
[ 1940.962698] BTRFS info (device sda): disk space caching is enabled
[ 1942.115796] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096
[ 1942.115819] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 2
[ 1942.116170] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096
[ 1942.116183] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 1
[ 1942.116212] BTRFS error (device sda): failed to read block groups: -5
[ 1942.140369] BTRFS error (device sda): open_ctree failed

I tried to recover the superblock:

# btrfs rescue super-recover -v /dev/sda

All Devices:
	Device: id = 3, name = /dev/sdc
	Device: id = 2, name = /dev/sdb
	Device: id = 4, name = /dev/sdd
	Device: id = 1, name = /dev/sda

Before Recovering:
	[All good supers]:
		device name = /dev/sdc
		superblock bytenr = 65536

		device name = /dev/sdc
		superblock bytenr = 67108864

		device name = /dev/sdc
		superblock bytenr = 274877906944

		device name = /dev/sdb
		superblock bytenr = 65536

		device name = /dev/sdb
		superblock bytenr = 67108864

		device name = /dev/sdb
		superblock bytenr = 274877906944

		device name = /dev/sdd
		superblock bytenr = 65536

		device name = /dev/sdd
		superblock bytenr = 67108864

		device name = /dev/sdd
		superblock bytenr = 274877906944

		device name = /dev/sda
		superblock bytenr = 65536

		device name = /dev/sda
		superblock bytenr = 67108864

		device name = /dev/sda
		superblock bytenr = 274877906944

	[All bad supers]:

All supers are valid, no need to recover

I checked the filesystem:

# btrfs check --readonly /dev/sda
Opening filesystem to check...
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
ERROR: failed to read block groups: Operation not permitted
ERROR: cannot open file system