Re: Unmountable Array After Drive Failure During Device Deletion

[Chris Kastorff <encryptio@gmail.com>]

>> I'm using btrfs in data and metadata RAID10 on drives (not on md or any
>> other fanciness.)
>>
>> I was removing a drive (btrfs dev del) and during that operation, a
>> different drive in the array failed. Having not had this happen before,
>> I shut down the machine immediately due to the extremely loud piezo
>> buzzer on the drive controller card. I attempted to do so cleanly, but
>> the buzzer cut through my patience and after 4 minutes I cut the power.
>>
>> Afterwards, I located and removed the failed drive from the system, and
>> then got back to linux. The array no longer mounts ("failed to read the
>> system array on sdc"), with nearly identical messages when attempted
>> with -o recovery and -o recovery,ro.
> 
> This may be a stupid question, but you're missing a drive so the 
> filesystem will be degraded, but you didn't mention that in your mount 
> options, so...
> 
> Did you try mounting with -o degraded (possibly with recovery, etc, also, 
> but just try -o degraded plus any normal options first)?
> 

I did not try degraded because I didn't remember that there were two
different options for handling broken btrfs volumes.

mount -o degraded,ro yields:

btrfs: device label lake devid 11 transid 4893967 /dev/sda
btrfs: allowing degraded mounts
btrfs: disk space caching is enabled
parent transid verify failed on 87601116364800 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601116364800 (dev /dev/sdf
sector 62986400)
parent transid verify failed on 87601116381184 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601116381184 (dev /dev/sdf
sector 62986432)
parent transid verify failed on 87601115320320 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601115320320 (dev /dev/sdf
sector 62985896)
parent transid verify failed on 87601116368896 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601116368896 (dev /dev/sdf
sector 62986408)
parent transid verify failed on 87601116377088 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601116377088 (dev /dev/sdf
sector 62986424)
btrfs: bdev (null) errs: wr 344288, rd 230234, flush 0, corrupt 0, gen 0
btrfs: bdev /dev/sdm1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
btrfs: bdev /dev/sdg errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
Failed to read block groups: -5
btrfs: open_ctree failed

mount -o degraded,recovery,ro yields:

btrfs: device label lake devid 11 transid 4893967 /dev/sda
btrfs: allowing degraded mounts
btrfs: enabling auto recovery
btrfs: disk space caching is enabled
parent transid verify failed on 87601116798976 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601116798976 (dev /dev/sdg
sector 113318256)
parent transid verify failed on 87601119379456 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601119379456 (dev /dev/sdg
sector 113319456)
parent transid verify failed on 87601116774400 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601116774400 (dev /dev/sdg
sector 113318208)
parent transid verify failed on 87601119391744 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601119391744 (dev /dev/sdg
sector 113319480)
parent transid verify failed on 87601116778496 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601116778496 (dev /dev/sdg
sector 113318216)
parent transid verify failed on 87601116786688 wanted 4893969 found 4893849
btrfs read error corrected: ino 1 off 87601116786688 (dev /dev/sdg
sector 113318232)
btrfs: bdev (null) errs: wr 344288, rd 230234, flush 0, corrupt 0, gen 0
btrfs: bdev /dev/sdm1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
btrfs: bdev /dev/sdg errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
parent transid verify failed on 87601111515136 wanted 4893968 found 4893913
btrfs read error corrected: ino 1 off 87601111515136 (dev /dev/sdg
sector 113315616)
parent transid verify failed on 87601111523328 wanted 4893968 found 4893913
btrfs read error corrected: ino 1 off 87601111523328 (dev /dev/sdg
sector 113315632)
parent transid verify failed on 87601111535616 wanted 4893968 found 4893913
btrfs read error corrected: ino 1 off 87601111535616 (dev /dev/sdg
sector 113315656)
parent transid verify failed on 87601111556096 wanted 4893968 found 4893913
btrfs read error corrected: ino 1 off 87601111556096 (dev /dev/sdg
sector 113315696)
Failed to read block groups: -5
btrfs: open_ctree failed