* Help requested: btrfs can't read superblock
@ 2024-03-18 21:22 kind.moon1862
2024-03-19 0:06 ` Qu Wenruo
0 siblings, 1 reply; 6+ messages in thread
From: kind.moon1862 @ 2024-03-18 21:22 UTC (permalink / raw)
To: linux-btrfs
Dear btrfs team,
I've been asked to help someone with a btrfs problem. They use CentOS 7. I've been told the host was shut down cleanly, but it won't mount a btrfs filesystem when powering on.
Below is what I've done so far. Any guidance would be appreciated!
I booted using SystemRescue 11.0. Some info about the rescue OS:
# uname -a
Linux sysrescue 6.6.14-1-lts #1 SMP PREEMPT_DYNAMIC Fri, 26 Jan 2024 11:54:58 +0000 x86_64 GNU/Linux
# btrfs --version
btrfs-progs v6.6.3
Here are the disks:
# btrfs fi show
Label: '<redacted>' uuid: 834b4bfd-fbd4-40b9-8868-d1187c3c5a63
Total devices 4 FS bytes used 17.44TiB
devid 1 size 9.10TiB used 8.94TiB path /dev/sda
devid 2 size 9.10TiB used 8.94TiB path /dev/sdb
devid 3 size 9.10TiB used 8.94TiB path /dev/sdc
devid 4 size 9.10TiB used 8.94TiB path /dev/sdd
I tried mounting:
# mount -t btrfs -o recovery,ro /dev/sda /mnt/data
mount: /mnt/data: can't read superblock on /dev/sda.
dmesg(1) may have more information after failed mount system call.
dmesg shows this:
[ 1940.962632] BTRFS info (device sda): first mount of filesystem 834b4bfd-fbd4-40b9-8868-d1187c3c5a63
[ 1940.962665] BTRFS info (device sda): using crc32c (crc32c-intel) checksum algorithm
[ 1940.962686] BTRFS warning (device sda): 'recovery' is deprecated, use 'rescue=usebackuproot' instead
[ 1940.962692] BTRFS info (device sda): trying to use backup root at mount time
[ 1940.962698] BTRFS info (device sda): disk space caching is enabled
[ 1942.115796] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096
[ 1942.115819] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 2
[ 1942.116170] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096
[ 1942.116183] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 1
[ 1942.116212] BTRFS error (device sda): failed to read block groups: -5
[ 1942.140369] BTRFS error (device sda): open_ctree failed
I tried to recover the superblock:
# btrfs rescue super-recover -v /dev/sda
All Devices:
Device: id = 3, name = /dev/sdc
Device: id = 2, name = /dev/sdb
Device: id = 4, name = /dev/sdd
Device: id = 1, name = /dev/sda
Before Recovering:
[All good supers]:
device name = /dev/sdc
superblock bytenr = 65536
device name = /dev/sdc
superblock bytenr = 67108864
device name = /dev/sdc
superblock bytenr = 274877906944
device name = /dev/sdb
superblock bytenr = 65536
device name = /dev/sdb
superblock bytenr = 67108864
device name = /dev/sdb
superblock bytenr = 274877906944
device name = /dev/sdd
superblock bytenr = 65536
device name = /dev/sdd
superblock bytenr = 67108864
device name = /dev/sdd
superblock bytenr = 274877906944
device name = /dev/sda
superblock bytenr = 65536
device name = /dev/sda
superblock bytenr = 67108864
device name = /dev/sda
superblock bytenr = 274877906944
[All bad supers]:
All supers are valid, no need to recover
I checked the filesystem:
# btrfs check --readonly /dev/sda
Opening filesystem to check...
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
ERROR: failed to read block groups: Operation not permitted
ERROR: cannot open file system
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Help requested: btrfs can't read superblock
2024-03-18 21:22 Help requested: btrfs can't read superblock kind.moon1862
@ 2024-03-19 0:06 ` Qu Wenruo
2024-03-19 0:33 ` kind.moon1862
0 siblings, 1 reply; 6+ messages in thread
From: Qu Wenruo @ 2024-03-19 0:06 UTC (permalink / raw)
To: kind.moon1862, linux-btrfs
在 2024/3/19 07:52, kind.moon1862@fastmail.com 写道:
>
> Dear btrfs team,
>
> I've been asked to help someone with a btrfs problem. They use CentOS 7. I've been told the host was shut down cleanly, but it won't mount a btrfs filesystem when powering on.
>
> Below is what I've done so far. Any guidance would be appreciated!
>
> I booted using SystemRescue 11.0. Some info about the rescue OS:
>
> # uname -a
> Linux sysrescue 6.6.14-1-lts #1 SMP PREEMPT_DYNAMIC Fri, 26 Jan 2024 11:54:58 +0000 x86_64 GNU/Linux
>
> # btrfs --version
> btrfs-progs v6.6.3
>
> Here are the disks:
>
> # btrfs fi show
> Label: '<redacted>' uuid: 834b4bfd-fbd4-40b9-8868-d1187c3c5a63
> Total devices 4 FS bytes used 17.44TiB
> devid 1 size 9.10TiB used 8.94TiB path /dev/sda
> devid 2 size 9.10TiB used 8.94TiB path /dev/sdb
> devid 3 size 9.10TiB used 8.94TiB path /dev/sdc
> devid 4 size 9.10TiB used 8.94TiB path /dev/sdd
>
> I tried mounting:
>
> # mount -t btrfs -o recovery,ro /dev/sda /mnt/data
> mount: /mnt/data: can't read superblock on /dev/sda.
> dmesg(1) may have more information after failed mount system call.
Use "mount -o rescue=all,ro" instead.
>
> dmesg shows this:
>
> [ 1940.962632] BTRFS info (device sda): first mount of filesystem 834b4bfd-fbd4-40b9-8868-d1187c3c5a63
> [ 1940.962665] BTRFS info (device sda): using crc32c (crc32c-intel) checksum algorithm
> [ 1940.962686] BTRFS warning (device sda): 'recovery' is deprecated, use 'rescue=usebackuproot' instead
> [ 1940.962692] BTRFS info (device sda): trying to use backup root at mount time
> [ 1940.962698] BTRFS info (device sda): disk space caching is enabled
> [ 1942.115796] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096
Bad extent item start bytenr, 5822947745796 = 0x54bc2bb6004, the last
0x004 looks like a bitflip.
Please run memtest to make sure your hardware memory is fine, or such
problem would just happen again and again.
Thankfully extent tree corruption is not a big deal for "rescue=all,ro"
mount.
Thanks,
Qu
> [ 1942.115819] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 2
> [ 1942.116170] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096
> [ 1942.116183] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 1
> [ 1942.116212] BTRFS error (device sda): failed to read block groups: -5
> [ 1942.140369] BTRFS error (device sda): open_ctree failed
>
> I tried to recover the superblock:
>
> # btrfs rescue super-recover -v /dev/sda
>
> All Devices:
> Device: id = 3, name = /dev/sdc
> Device: id = 2, name = /dev/sdb
> Device: id = 4, name = /dev/sdd
> Device: id = 1, name = /dev/sda
>
> Before Recovering:
> [All good supers]:
> device name = /dev/sdc
> superblock bytenr = 65536
>
> device name = /dev/sdc
> superblock bytenr = 67108864
>
> device name = /dev/sdc
> superblock bytenr = 274877906944
>
> device name = /dev/sdb
> superblock bytenr = 65536
>
> device name = /dev/sdb
> superblock bytenr = 67108864
>
> device name = /dev/sdb
> superblock bytenr = 274877906944
>
> device name = /dev/sdd
> superblock bytenr = 65536
>
> device name = /dev/sdd
> superblock bytenr = 67108864
>
> device name = /dev/sdd
> superblock bytenr = 274877906944
>
> device name = /dev/sda
> superblock bytenr = 65536
>
> device name = /dev/sda
> superblock bytenr = 67108864
>
> device name = /dev/sda
> superblock bytenr = 274877906944
>
> [All bad supers]:
>
> All supers are valid, no need to recover
>
> I checked the filesystem:
>
> # btrfs check --readonly /dev/sda
> Opening filesystem to check...
> corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
> corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
> corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
> corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
> ERROR: failed to read block groups: Operation not permitted
> ERROR: cannot open file system
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Help requested: btrfs can't read superblock
2024-03-19 0:06 ` Qu Wenruo
@ 2024-03-19 0:33 ` kind.moon1862
2024-03-19 0:53 ` Qu Wenruo
0 siblings, 1 reply; 6+ messages in thread
From: kind.moon1862 @ 2024-03-19 0:33 UTC (permalink / raw)
To: Qu Wenruo, linux-btrfs
On Mon, Mar 18, 2024, at 17:06, Qu Wenruo wrote:
> Use "mount -o rescue=all,ro" instead.
Thank you for your suggestion. Should I run it under CentOS 7 (the original OS) or under the newer kernel in the SystemRescue distro?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Help requested: btrfs can't read superblock
2024-03-19 0:33 ` kind.moon1862
@ 2024-03-19 0:53 ` Qu Wenruo
2024-03-19 20:44 ` kind.moon1862
0 siblings, 1 reply; 6+ messages in thread
From: Qu Wenruo @ 2024-03-19 0:53 UTC (permalink / raw)
To: kind.moon1862, linux-btrfs
在 2024/3/19 11:03, kind.moon1862@fastmail.com 写道:
> On Mon, Mar 18, 2024, at 17:06, Qu Wenruo wrote:
>
>> Use "mount -o rescue=all,ro" instead.
>
> Thank you for your suggestion. Should I run it under CentOS 7 (the original OS) or under the newer kernel in the SystemRescue distro?
Newer kernel please.
And still, please run memtest before doing any rescue.
As faulty hardware can always lead to weird problems no matter what.
And CentOS 7 is not recommended for btrfs usage (no proper btrfs related
backport and old kernels, IIRC it's already EOL)
Thanks,
Qu
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Help requested: btrfs can't read superblock
2024-03-19 0:53 ` Qu Wenruo
@ 2024-03-19 20:44 ` kind.moon1862
2024-03-19 20:45 ` Qu Wenruo
0 siblings, 1 reply; 6+ messages in thread
From: kind.moon1862 @ 2024-03-19 20:44 UTC (permalink / raw)
To: Qu Wenruo, linux-btrfs
On Mon, Mar 18, 2024, at 17:53, Qu Wenruo wrote:
> And still, please run memtest before doing any rescue.
> As faulty hardware can always lead to weird problems no matter what.
memtest86+ found bad RAM. I will replace that before proceeding. Thanks again.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Help requested: btrfs can't read superblock
2024-03-19 20:44 ` kind.moon1862
@ 2024-03-19 20:45 ` Qu Wenruo
0 siblings, 0 replies; 6+ messages in thread
From: Qu Wenruo @ 2024-03-19 20:45 UTC (permalink / raw)
To: kind.moon1862, linux-btrfs
在 2024/3/20 07:14, kind.moon1862@fastmail.com 写道:
> On Mon, Mar 18, 2024, at 17:53, Qu Wenruo wrote:
>
>> And still, please run memtest before doing any rescue.
>> As faulty hardware can always lead to weird problems no matter what.
>
> memtest86+ found bad RAM. I will replace that before proceeding. Thanks again.
Btrfs tree-checker, the second best memory tester. :)
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-03-19 20:46 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-03-18 21:22 Help requested: btrfs can't read superblock kind.moon1862
2024-03-19 0:06 ` Qu Wenruo
2024-03-19 0:33 ` kind.moon1862
2024-03-19 0:53 ` Qu Wenruo
2024-03-19 20:44 ` kind.moon1862
2024-03-19 20:45 ` Qu Wenruo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).