From: Chris Kastorff <encryptio@gmail.com>
To: Duncan <1i5t5.duncan@cox.net>, linux-btrfs@vger.kernel.org
Subject: Re: Unmountable Array After Drive Failure During Device Deletion
Date: Thu, 19 Dec 2013 11:41:02 -0800 [thread overview]
Message-ID: <52B34BCE.2000409@gmail.com> (raw)
In-Reply-To: <52B3461A.4030300@gmail.com>
>>> I'm using btrfs in data and metadata RAID10 on drives (not on md or any
>>> other fanciness.)
>>>
>>> I was removing a drive (btrfs dev del) and during that operation, a
>>> different drive in the array failed. Having not had this happen before,
>>> I shut down the machine immediately due to the extremely loud piezo
>>> buzzer on the drive controller card. I attempted to do so cleanly, but
>>> the buzzer cut through my patience and after 4 minutes I cut the power.
>>>
>>> Afterwards, I located and removed the failed drive from the system, and
>>> then got back to linux. The array no longer mounts ("failed to read the
>>> system array on sdc"), with nearly identical messages when attempted
>>> with -o recovery and -o recovery,ro.
>>
>> This may be a stupid question, but you're missing a drive so the
>> filesystem will be degraded, but you didn't mention that in your mount
>> options, so...
>>
>> Did you try mounting with -o degraded (possibly with recovery, etc, also,
>> but just try -o degraded plus any normal options first)?
>>
>
> I did not try degraded because I didn't remember that there were two
> different options for handling broken btrfs volumes.
>
> mount -o degraded,ro yields:
>
> btrfs: device label lake devid 11 transid 4893967 /dev/sda
> btrfs: allowing degraded mounts
> btrfs: disk space caching is enabled
> parent transid verify failed on 87601116364800 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601116364800 (dev /dev/sdf
> sector 62986400)
> parent transid verify failed on 87601116381184 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601116381184 (dev /dev/sdf
> sector 62986432)
> parent transid verify failed on 87601115320320 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601115320320 (dev /dev/sdf
> sector 62985896)
> parent transid verify failed on 87601116368896 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601116368896 (dev /dev/sdf
> sector 62986408)
> parent transid verify failed on 87601116377088 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601116377088 (dev /dev/sdf
> sector 62986424)
> btrfs: bdev (null) errs: wr 344288, rd 230234, flush 0, corrupt 0, gen 0
> btrfs: bdev /dev/sdm1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
> btrfs: bdev /dev/sdg errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
> parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
> Failed to read block groups: -5
> btrfs: open_ctree failed
>
> mount -o degraded,recovery,ro yields:
>
> btrfs: device label lake devid 11 transid 4893967 /dev/sda
> btrfs: allowing degraded mounts
> btrfs: enabling auto recovery
> btrfs: disk space caching is enabled
> parent transid verify failed on 87601116798976 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601116798976 (dev /dev/sdg
> sector 113318256)
> parent transid verify failed on 87601119379456 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601119379456 (dev /dev/sdg
> sector 113319456)
> parent transid verify failed on 87601116774400 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601116774400 (dev /dev/sdg
> sector 113318208)
> parent transid verify failed on 87601119391744 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601119391744 (dev /dev/sdg
> sector 113319480)
> parent transid verify failed on 87601116778496 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601116778496 (dev /dev/sdg
> sector 113318216)
> parent transid verify failed on 87601116786688 wanted 4893969 found 4893849
> btrfs read error corrected: ino 1 off 87601116786688 (dev /dev/sdg
> sector 113318232)
> btrfs: bdev (null) errs: wr 344288, rd 230234, flush 0, corrupt 0, gen 0
> btrfs: bdev /dev/sdm1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
> btrfs: bdev /dev/sdg errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
> parent transid verify failed on 87601111515136 wanted 4893968 found 4893913
> btrfs read error corrected: ino 1 off 87601111515136 (dev /dev/sdg
> sector 113315616)
> parent transid verify failed on 87601111523328 wanted 4893968 found 4893913
> btrfs read error corrected: ino 1 off 87601111523328 (dev /dev/sdg
> sector 113315632)
> parent transid verify failed on 87601111535616 wanted 4893968 found 4893913
> btrfs read error corrected: ino 1 off 87601111535616 (dev /dev/sdg
> sector 113315656)
> parent transid verify failed on 87601111556096 wanted 4893968 found 4893913
> btrfs read error corrected: ino 1 off 87601111556096 (dev /dev/sdg
> sector 113315696)
> Failed to read block groups: -5
> btrfs: open_ctree failed
>
I should also mention that the "corrupt 4" errs on /dev/sdm1 and
/dev/sdg are there from an earlier btrfs extent corruption bug, and do
not exist on the filesystem anymore (a scrub hours before the device
deletion completed with 0 errors.)
next prev parent reply other threads:[~2013-12-19 19:41 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-12-19 9:26 Unmountable Array After Drive Failure During Device Deletion Chris Kastorff
2013-12-19 18:07 ` Duncan
2013-12-19 19:16 ` Chris Kastorff
2013-12-19 19:41 ` Chris Kastorff [this message]
2013-12-19 22:21 ` Chris Murphy
2013-12-20 0:06 ` Chris Kastorff
2013-12-20 3:47 ` Chris Murphy
2013-12-21 23:16 ` Chris Kastorff
2013-12-21 23:40 ` Chris Murphy
2013-12-22 1:15 ` ronnie sahlberg
2013-12-22 11:35 ` Duncan
2013-12-26 23:18 ` Chris Samuel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=52B34BCE.2000409@gmail.com \
--to=encryptio@gmail.com \
--cc=1i5t5.duncan@cox.net \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).