From: Abe <ml-btrfs@zeroloop.net>
To: linux-btrfs@vger.kernel.org
Subject: Broken RAID6, segfault on chunk-recover
Date: Mon, 4 Jan 2016 14:32:16 +0100 [thread overview]
Message-ID: <568A7460.6070607@zeroloop.net> (raw)
Hello,
Could you please help in recovering this multiple device filesystem?
I went up to a point where running chunk-recover looks to be promising,
but unfortunately the command will segfault consistently.
This was a 8 devices Btrfs RAID6 using LSI HBA. One disk died suddenly
and is 100% unavailable. At this point, I had evidences userland was
receiving corrupted data while reading files (few invalid bytes in
gigabytes files). System was rebooted. Since then I can't mount it as
degraded or use any recovery command.
Memtest is ok.
What would you advise ?
Is the segfault issue something you would like me to help debug before
going further ?
----------------------------------
# uname -a
Linux horo 4.3.0-1-amd64 #1 SMP Debian 4.3.3-2 (2015-12-17) x86_64 GNU/Linux
----------------------------------
# ./btrfs version
btrfs-progs v4.3.1
----------------------------------
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk
├─sda1 8:1 0 1M 0 part
└─sda2 8:2 0 118.3G 0 part /
sdb 8:16 0 477G 0 disk
└─sdb1 8:17 0 477G 0 part
sdc 8:32 0 477G 0 disk
└─sdc1 8:33 0 477G 0 part
sdd 8:48 0 477G 0 disk
└─sdd1 8:49 0 477G 0 part
sde 8:64 0 477G 0 disk
└─sde1 8:65 0 477G 0 part
sdf 8:80 0 477G 0 disk
└─sdf1 8:81 0 477G 0 part
sdg 8:96 0 477G 0 disk
└─sdg1 8:97 0 477G 0 part
sdh 8:112 0 477G 0 disk
└─sdh1 8:113 0 477G 0 part
----------------------------------
# ./btrfs fi show
warning, device 6 is missing
checksum verify failed on 4415566151680 found F8A6E83A wanted EB7CA66C
checksum verify failed on 4415566151680 found F8A6E83A wanted EB7CA66C
bytenr mismatch, want=4415566151680, have=526521381424393794
Couldn't read chunk root
Label: 'hive' uuid: bec7b9a0-c56c-494e-8631-072d3f89c0c9
Total devices 8 FS bytes used 2.22TiB
devid 1 size 476.94GiB used 325.73GiB path /dev/sdf1
devid 2 size 476.94GiB used 325.73GiB path /dev/sdc1
devid 3 size 476.94GiB used 325.73GiB path /dev/sdd1
devid 4 size 476.94GiB used 325.73GiB path /dev/sde1
devid 5 size 476.94GiB used 325.73GiB path /dev/sdb1
devid 7 size 476.94GiB used 325.73GiB path /dev/sdh1
devid 8 size 476.94GiB used 325.73GiB path /dev/sdg1
*** Some devices missing
----------------------------------
# mount -o ro,degraded,recovery /dev/sdb1 /mnt/temp
mount: wrong fs type, bad option, bad superblock on /dev/sdb1,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
[169387.880114] BTRFS info (device sdb1): allowing degraded mounts
[169387.880125] BTRFS info (device sdb1): enabling auto recovery
[169387.880130] BTRFS info (device sdb1): disk space caching is enabled
[169387.880132] BTRFS: has skinny extents
[169387.890940] BTRFS: bdev (null) errs: wr 27, rd 1535, flush 9,
corrupt 0, gen 0
[169388.014701] BTRFS (device sdb1): bad tree block start
8621721664010832405 6766851391488
[169388.015117] BTRFS (device sdb1): bad tree block start
8621721664010832405 6766851391488
[169388.015148] BTRFS: Failed to read block groups: -5
[169388.042731] BTRFS: open_ctree failed
----------------------------------
# ./btrfs check --readonly /dev/sdb1
warning, device 6 is missing
checksum verify failed on 4415566151680 found F8A6E83A wanted EB7CA66C
checksum verify failed on 4415566151680 found F8A6E83A wanted EB7CA66C
bytenr mismatch, want=4415566151680, have=526521381424393794
Couldn't read chunk root
Couldn't open file system
# ./btrfs check --readonly --tree-root 4415566151680 /dev/sdb1
warning, device 6 is missing
checksum verify failed on 4415566151680 found F8A6E83A wanted EB7CA66C
checksum verify failed on 4415566151680 found F8A6E83A wanted EB7CA66C
bytenr mismatch, want=4415566151680, have=526521381424393794
Couldn't read chunk root
Couldn't open file system
# ./btrfs check --readonly --tree-root 526521381424393794 /dev/sdb1
warning, device 6 is missing
checksum verify failed on 4415566151680 found F8A6E83A wanted EB7CA66C
checksum verify failed on 4415566151680 found F8A6E83A wanted EB7CA66C
bytenr mismatch, want=4415566151680, have=526521381424393794
Couldn't read chunk root
Couldn't open file system
# ./btrfs check --repair --init-csum-tree /dev/sdb1
enabling repair mode
Creating a new CRC tree
warning, device 6 is missing
checksum verify failed on 4415566151680 found F8A6E83A wanted EB7CA66C
checksum verify failed on 4415566151680 found F8A6E83A wanted EB7CA66C
bytenr mismatch, want=4415566151680, have=526521381424393794
Couldn't read chunk root
Couldn't open file system
----------------------------------
# ./btrfs rescue super-recover -v /dev/sd[bcdefgh]1
[...]
All supers are valid, no need to recover
----------------------------------
# ./btrfs rescue chunk-recover -v /dev/sdb1
All Devices:
Device: id = 1, name = /dev/sdf1
Device: id = 7, name = /dev/sdh1
Device: id = 8, name = /dev/sdg1
Device: id = 4, name = /dev/sde1
Device: id = 2, name = /dev/sdc1
Device: id = 3, name = /dev/sdd1
Device: id = 5, name = /dev/sdb1
Scanning: 0 in dev0, 0 in dev1, 0 in dev2, 0 in dev3, 0 in dev4, 0 in
dev5, 0 in dev6
Segmentation fault
# ./btrfs rescue chunk-recover -v /dev/sdc1
All Devices:
Device: id = 1, name = /dev/sdf1
Device: id = 7, name = /dev/sdh1
Device: id = 8, name = /dev/sdg1
Device: id = 4, name = /dev/sde1
Device: id = 3, name = /dev/sdd1
Device: id = 5, name = /dev/sdb1
Device: id = 2, name = /dev/sdc1
Scanning: 4096 in dev0, 2052096 in dev1, 2183168 in dev2, 1146880 in
dev3, 0 in dev4, 0 in dev5, 0 in dev6
Segmentation fault
# ./btrfs rescue chunk-recover -v /dev/sdd1
All Devices:
Device: id = 1, name = /dev/sdf1
Device: id = 7, name = /dev/sdh1
Device: id = 8, name = /dev/sdg1
Device: id = 4, name = /dev/sde1
Device: id = 2, name = /dev/sdc1
Device: id = 5, name = /dev/sdb1
Device: id = 3, name = /dev/sdd1
Scanning: 0 in dev0, 0 in dev1, 0 in dev2, 0 in dev3, 0 in dev4, 0 in
dev5, 0 in dev6
Segmentation fault
[...]
----------------------------------
Best regards
next reply other threads:[~2016-01-04 13:41 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-01-04 13:32 Abe [this message]
2016-01-06 22:51 ` Broken RAID6, segfault on chunk-recover Abe
2016-01-07 16:39 ` Abe
2016-01-08 1:42 ` Qu Wenruo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=568A7460.6070607@zeroloop.net \
--to=ml-btrfs@zeroloop.net \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).