* Unmountable 4-drive btrfs RAID after drive failure
@ 2023-12-01 20:36 gara
0 siblings, 0 replies; only message in thread
From: gara @ 2023-12-01 20:36 UTC (permalink / raw)
To: linux-btrfs
tl;dr: btrfs RAID doesn't mount after single drive failure, `btrfs
restore` also can't make progress - pls help save my data
I am currently running `btrfs rescue chunk-recover` and will for another
day or two, but before I ask it to rebuild anything (didnt get a prompt
to yet) I would be very thankful for any pointers on what else I could
do/check to get as much data as possible off of the FS.
Data loss would not be critical but it would hurt.
The situation / process so far:
I run a 4-drive NAS that with LUKS->BTRFS-RAID setup over all disks
(don't remember if it's a RAID5/6 or RAID10, is there a way to check?),
my rootfs is on a different drive so i can still boot into my box
> uname -a
Linux helios4 5.15.93-mvebu #23.02.2 SMP Fri Feb 17 23:47:24 UTC 2023 armv7l GNU/Linux
> btrfs --version
btrfs-progs v5.10.1
I noticed yesterday that my SSHFS mounts to this NAS are not writable.
After getting `Input/Output Error`s reading the mountpoint from bash
directly on the device, I rebooted.
After reboot /dev/sdd** is missing (it's clicking, so probably ded),
SMART info for the 3 remaining drives looks healthy.
Normal luksOpen works on all 3 drives, but mount fails and dmesg prints
[ 465.182744] BTRFS error (device dm-2): devid 4 uuid e79df2b2-5ad0-43b3-a280-3e240b9a2cff is missing
[ 465.182759] BTRFS error (device dm-2): failed to read the system array: -2
[ 465.183063] BTRFS error (device dm-2): open_ctree failed
> btrfs fi show
Label: none uuid: 90f61aa9-e02c-4c47-ac31-559f08083565
Total devices 1 FS bytes used 384.00KiB
devid 1 size 1.82TiB used 2.02GiB path /dev/sdd
warning, device 4 is missing
parent transid verify failed on 92405825536 wanted 40448 found 31913
parent transid verify failed on 92405825536 wanted 40448 found 31913
Ignoring transid failure
parent transid verify failed on 92405841920 wanted 33513 found 31913
parent transid verify failed on 92405841920 wanted 33513 found 31913
Ignoring transid failure
parent transid verify failed on 92405858304 wanted 35618 found 31923
parent transid verify failed on 92405858304 wanted 35618 found 31923
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=92405809152 item=6 parent level=1 child bytenr=92405858304 child level=1
Couldn't read chunk tree
Label: none uuid: 4a957a95-f784-41c3-a46c-13e975867a9f
Total devices 4 FS bytes used 2.50TiB
devid 1 size 5.46TiB used 1.25TiB path /dev/mapper/redA
devid 2 size 5.46TiB used 389.03GiB path /dev/mapper/blueB
devid 3 size 5.46TiB used 1.25TiB path /dev/mapper/redC
*** Some devices missing
mounting degraded fails too
> mount -t btrfs -o ro,degraded /dev/mapper/redA /mnt/nas/
mount: /mnt/nas: wrong fs type, bad option, bad superblock on /dev/mapper/redA, missing codepage or helper program, or other error.
and dmesg prints after mounting:
[56750.747574] BTRFS info (device dm-2): allowing degraded mounts
[56750.747590] BTRFS info (device dm-2): disk space caching is enabled
[56750.747594] BTRFS info (device dm-2): has skinny extents
[56750.932131] BTRFS error (device dm-2): parent transid verify failed on 92405825536 wanted 40448 found 31913
[56750.939473] BTRFS error (device dm-2): failed to read chunk tree: -5
[56750.994132] BTRFS error (device dm-2): open_ctree failed
I spent a good day googling and trying all nondestructive things i could
find to get the FS mounted or the data off, i.e.
> btrfs restore -si /dev/mapper/redA /mnt/rec/
warning, device 4 is missing
parent transid verify failed on 92405825536 wanted 40448 found 31913
parent transid verify failed on 92405825536 wanted 40448 found 31913
Ignoring transid failure
parent transid verify failed on 92405841920 wanted 33513 found 31913
parent transid verify failed on 92405841920 wanted 33513 found 31913
Ignoring transid failure
parent transid verify failed on 92405858304 wanted 35618 found 31923
parent transid verify failed on 92405858304 wanted 35618 found 31923
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=92405809152 item=6 parent level=1 child bytenr=92405858304 child level=1
Couldn't read chunk tree
Could not open root, trying backup super
warning, device 1 is missing
warning, device 2 is missing
warning, device 4 is missing
bad tree block 92405825536, bytenr mismatch, want=92405825536, have=0
Couldn't read chunk tree
Could not open root, trying backup super
warning, device 1 is missing
warning, device 2 is missing
warning, device 4 is missing
bad tree block 92405825536, bytenr mismatch, want=92405825536, have=0
Couldn't read chunk tree
Could not open root, trying backup super
> btrfs check --readonly --force /dev/mapper/redA
Opening filesystem to check...
warning, device 4 is missing
parent transid verify failed on 92405825536 wanted 40448 found 31913
parent transid verify failed on 92405825536 wanted 40448 found 31913
Ignoring transid failure
parent transid verify failed on 92405841920 wanted 33513 found 31913
parent transid verify failed on 92405841920 wanted 33513 found 31913
Ignoring transid failure
parent transid verify failed on 92405858304 wanted 35618 found 31923
parent transid verify failed on 92405858304 wanted 35618 found 31923
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=92405809152 item=6 parent level=1 child bytenr=92405858304 child level=1
Couldn't read chunk tree
ERROR: cannot open file system
> mount -t btrfs -o ro,degraded,rescue=all /dev/mapper/redA /mnt/nas/
mount: /mnt/nas: wrong fs type, bad option, bad superblock on /dev/mapper/redA, missing codepage or helper program, or other error.
> mount -t btrfs -o degraded,rescue=skip_bg /dev/mapper/redA /mnt/nas/
mount: /mnt/nas: wrong fs type, bad option, bad superblock on /dev/mapper/redA, missing codepage or helper program, or other error.
> mount -t btrfs -o degraded,ro,recovery /dev/mapper/redA /mnt/nas/
mount: /mnt/nas: wrong fs type, bad option, bad superblock on /dev/mapper/redA, missing codepage or helper program, or other error.
superblock dump in case it helps:
> btrfs inspect-internal dump-super /dev/mapper/redA
superblock: bytenr=65536, device=/dev/mapper/redA
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x7f35aa3d [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 4a957a95-f784-41c3-a46c-13e975867a9f
metadata_uuid 4a957a95-f784-41c3-a46c-13e975867a9f
label
generation 54015
root 3023220178944
sys_array_size 193
chunk_root_generation 40448
root_level 1
chunk_root 92405809152
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 24004633395200
bytes_used 2744322326528
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 4
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x161
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation 54015
uuid_tree_generation 54015
dev_item.uuid d64ab077-6466-4c54-8a89-7b60f86cacb9
dev_item.fsid 4a957a95-f784-41c3-a46c-13e975867a9f [match]
dev_item.type 0
dev_item.total_bytes 6001158348800
dev_item.bytes_used 1373349347328
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
^ permalink raw reply [flat|nested] only message in thread
only message in thread, other threads:[~2023-12-01 20:36 UTC | newest]
Thread overview: (only message) (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-12-01 20:36 Unmountable 4-drive btrfs RAID after drive failure gara
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox