From: gara <dw.gara@gmail.com>
To: linux-btrfs@vger.kernel.org
Subject: Unmountable 4-drive btrfs RAID after drive failure
Date: Fri, 1 Dec 2023 14:36:36 -0600 [thread overview]
Message-ID: <594d83ff-ebff-490e-aa8c-842c6a604a31@gmail.com> (raw)
tl;dr: btrfs RAID doesn't mount after single drive failure, `btrfs
restore` also can't make progress - pls help save my data
I am currently running `btrfs rescue chunk-recover` and will for another
day or two, but before I ask it to rebuild anything (didnt get a prompt
to yet) I would be very thankful for any pointers on what else I could
do/check to get as much data as possible off of the FS.
Data loss would not be critical but it would hurt.
The situation / process so far:
I run a 4-drive NAS that with LUKS->BTRFS-RAID setup over all disks
(don't remember if it's a RAID5/6 or RAID10, is there a way to check?),
my rootfs is on a different drive so i can still boot into my box
> uname -a
Linux helios4 5.15.93-mvebu #23.02.2 SMP Fri Feb 17 23:47:24 UTC 2023 armv7l GNU/Linux
> btrfs --version
btrfs-progs v5.10.1
I noticed yesterday that my SSHFS mounts to this NAS are not writable.
After getting `Input/Output Error`s reading the mountpoint from bash
directly on the device, I rebooted.
After reboot /dev/sdd** is missing (it's clicking, so probably ded),
SMART info for the 3 remaining drives looks healthy.
Normal luksOpen works on all 3 drives, but mount fails and dmesg prints
[ 465.182744] BTRFS error (device dm-2): devid 4 uuid e79df2b2-5ad0-43b3-a280-3e240b9a2cff is missing
[ 465.182759] BTRFS error (device dm-2): failed to read the system array: -2
[ 465.183063] BTRFS error (device dm-2): open_ctree failed
> btrfs fi show
Label: none uuid: 90f61aa9-e02c-4c47-ac31-559f08083565
Total devices 1 FS bytes used 384.00KiB
devid 1 size 1.82TiB used 2.02GiB path /dev/sdd
warning, device 4 is missing
parent transid verify failed on 92405825536 wanted 40448 found 31913
parent transid verify failed on 92405825536 wanted 40448 found 31913
Ignoring transid failure
parent transid verify failed on 92405841920 wanted 33513 found 31913
parent transid verify failed on 92405841920 wanted 33513 found 31913
Ignoring transid failure
parent transid verify failed on 92405858304 wanted 35618 found 31923
parent transid verify failed on 92405858304 wanted 35618 found 31923
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=92405809152 item=6 parent level=1 child bytenr=92405858304 child level=1
Couldn't read chunk tree
Label: none uuid: 4a957a95-f784-41c3-a46c-13e975867a9f
Total devices 4 FS bytes used 2.50TiB
devid 1 size 5.46TiB used 1.25TiB path /dev/mapper/redA
devid 2 size 5.46TiB used 389.03GiB path /dev/mapper/blueB
devid 3 size 5.46TiB used 1.25TiB path /dev/mapper/redC
*** Some devices missing
mounting degraded fails too
> mount -t btrfs -o ro,degraded /dev/mapper/redA /mnt/nas/
mount: /mnt/nas: wrong fs type, bad option, bad superblock on /dev/mapper/redA, missing codepage or helper program, or other error.
and dmesg prints after mounting:
[56750.747574] BTRFS info (device dm-2): allowing degraded mounts
[56750.747590] BTRFS info (device dm-2): disk space caching is enabled
[56750.747594] BTRFS info (device dm-2): has skinny extents
[56750.932131] BTRFS error (device dm-2): parent transid verify failed on 92405825536 wanted 40448 found 31913
[56750.939473] BTRFS error (device dm-2): failed to read chunk tree: -5
[56750.994132] BTRFS error (device dm-2): open_ctree failed
I spent a good day googling and trying all nondestructive things i could
find to get the FS mounted or the data off, i.e.
> btrfs restore -si /dev/mapper/redA /mnt/rec/
warning, device 4 is missing
parent transid verify failed on 92405825536 wanted 40448 found 31913
parent transid verify failed on 92405825536 wanted 40448 found 31913
Ignoring transid failure
parent transid verify failed on 92405841920 wanted 33513 found 31913
parent transid verify failed on 92405841920 wanted 33513 found 31913
Ignoring transid failure
parent transid verify failed on 92405858304 wanted 35618 found 31923
parent transid verify failed on 92405858304 wanted 35618 found 31923
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=92405809152 item=6 parent level=1 child bytenr=92405858304 child level=1
Couldn't read chunk tree
Could not open root, trying backup super
warning, device 1 is missing
warning, device 2 is missing
warning, device 4 is missing
bad tree block 92405825536, bytenr mismatch, want=92405825536, have=0
Couldn't read chunk tree
Could not open root, trying backup super
warning, device 1 is missing
warning, device 2 is missing
warning, device 4 is missing
bad tree block 92405825536, bytenr mismatch, want=92405825536, have=0
Couldn't read chunk tree
Could not open root, trying backup super
> btrfs check --readonly --force /dev/mapper/redA
Opening filesystem to check...
warning, device 4 is missing
parent transid verify failed on 92405825536 wanted 40448 found 31913
parent transid verify failed on 92405825536 wanted 40448 found 31913
Ignoring transid failure
parent transid verify failed on 92405841920 wanted 33513 found 31913
parent transid verify failed on 92405841920 wanted 33513 found 31913
Ignoring transid failure
parent transid verify failed on 92405858304 wanted 35618 found 31923
parent transid verify failed on 92405858304 wanted 35618 found 31923
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=92405809152 item=6 parent level=1 child bytenr=92405858304 child level=1
Couldn't read chunk tree
ERROR: cannot open file system
> mount -t btrfs -o ro,degraded,rescue=all /dev/mapper/redA /mnt/nas/
mount: /mnt/nas: wrong fs type, bad option, bad superblock on /dev/mapper/redA, missing codepage or helper program, or other error.
> mount -t btrfs -o degraded,rescue=skip_bg /dev/mapper/redA /mnt/nas/
mount: /mnt/nas: wrong fs type, bad option, bad superblock on /dev/mapper/redA, missing codepage or helper program, or other error.
> mount -t btrfs -o degraded,ro,recovery /dev/mapper/redA /mnt/nas/
mount: /mnt/nas: wrong fs type, bad option, bad superblock on /dev/mapper/redA, missing codepage or helper program, or other error.
superblock dump in case it helps:
> btrfs inspect-internal dump-super /dev/mapper/redA
superblock: bytenr=65536, device=/dev/mapper/redA
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x7f35aa3d [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 4a957a95-f784-41c3-a46c-13e975867a9f
metadata_uuid 4a957a95-f784-41c3-a46c-13e975867a9f
label
generation 54015
root 3023220178944
sys_array_size 193
chunk_root_generation 40448
root_level 1
chunk_root 92405809152
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 24004633395200
bytes_used 2744322326528
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 4
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x161
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation 54015
uuid_tree_generation 54015
dev_item.uuid d64ab077-6466-4c54-8a89-7b60f86cacb9
dev_item.fsid 4a957a95-f784-41c3-a46c-13e975867a9f [match]
dev_item.type 0
dev_item.total_bytes 6001158348800
dev_item.bytes_used 1373349347328
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
reply other threads:[~2023-12-01 20:36 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=594d83ff-ebff-490e-aa8c-842c6a604a31@gmail.com \
--to=dw.gara@gmail.com \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox