Unmountable Array After Drive Failure During Device Deletion

Unmountable Array After Drive Failure During Device Deletion

[Chris Kastorff]

I'm using btrfs in data and metadata RAID10 on drives (not on md or any
other fanciness.)

I was removing a drive (btrfs dev del) and during that operation, a
different drive in the array failed. Having not had this happen before,
I shut down the machine immediately due to the extremely loud piezo
buzzer on the drive controller card. I attempted to do so cleanly, but
the buzzer cut through my patience and after 4 minutes I cut the power.

Afterwards, I located and removed the failed drive from the system, and
then got back to linux. The array no longer mounts ("failed to read the
system array on sdc"), with nearly identical messages when attempted
with -o recovery and -o recovery,ro.

btrfsck asserts and coredumps, as usual.

The drive that was being removed is devid 9 in the array, and is
/dev/sdm1 in the btrfs fi show seen below.

Kernel 3.12.4-1-ARCH, btrfs-progs v0.20-rc1-358-g194aa4a-dirty
(archlinux build.)

Can I recover the array?

== dmesg during failure ==

...
sd 0:2:3:0: [sdd] Unhandled error code
sd 0:2:3:0: [sdd]
Result: hostbyte=0x04 driverbyte=0x00
sd 0:2:3:0: [sdd] CDB:
cdb[0]=0x2a: 2a 00 26 89 5b 00 00 00 80 00
end_request: I/O error, dev sdd, sector 646535936
btrfs_dev_stat_print_on_error: 7791 callbacks suppressed
btrfs: bdev /dev/sdd errs: wr 315858, rd 230194, flush 0, corrupt 0, gen 0
sd 0:2:3:0: [sdd] Unhandled error code
sd 0:2:3:0: [sdd]
Result: hostbyte=0x04 driverbyte=0x00
sd 0:2:3:0: [sdd] CDB:
cdb[0]=0x2a: 2a 00 26 89 5b 80 00 00 80 00
end_request: I/O error, dev sdd, sector 646536064
...

== dmesg after new boot, mounting attempt ==

btrfs: device label lake devid 11 transid 4893967 /dev/sda
btrfs: disk space caching is enabled
btrfs: failed to read the system array on sdc
btrfs: open_ctree failed

== dmesg after new boot, mounting attempt with -o recovery,ro ==

btrfs: device label lake devid 11 transid 4893967 /dev/sda
btrfs: enabling auto recovery
btrfs: disk space caching is enabled
btrfs: failed to read the system array on sdc
btrfs: open_ctree failed

== btrfsck ==

deep# btrfsck /dev/sda
warning, device 14 is missing
warning devid 14 not found already
parent transid verify failed on 87601116364800 wanted 4893969 found 4893913
parent transid verify failed on 87601116364800 wanted 4893969 found 4893913
parent transid verify failed on 87601116381184 wanted 4893969 found 4893913
parent transid verify failed on 87601116381184 wanted 4893969 found 4893913
parent transid verify failed on 87601115320320 wanted 4893969 found 4893913
parent transid verify failed on 87601115320320 wanted 4893969 found 4893913
parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
Ignoring transid failure
Checking filesystem on /dev/sda
UUID: d5e17c49-d980-4bde-bd96-3c8bc95ea077
checking extents
parent transid verify failed on 87601117159424 wanted 4893969 found 4893913
parent transid verify failed on 87601117159424 wanted 4893969 found 4893913
parent transid verify failed on 87601116368896 wanted 4893969 found 4893913
parent transid verify failed on 87601116368896 wanted 4893969 found 4893913
parent transid verify failed on 87601117163520 wanted 4893969 found 4893913
parent transid verify failed on 87601117163520 wanted 4893969 found 4893913
parent transid verify failed on 87601117638656 wanted 4893969 found 4893913
parent transid verify failed on 87601117638656 wanted 4893969 found 4893913
Ignoring transid failure
parent transid verify failed on 87601117171712 wanted 4893969 found 4893913
parent transid verify failed on 87601117171712 wanted 4893969 found 4893913
parent transid verify failed on 87601117175808 wanted 4893969 found 4893913
parent transid verify failed on 87601117175808 wanted 4893969 found 4893913
parent transid verify failed on 87601117188096 wanted 4893969 found 4893913
parent transid verify failed on 87601117188096 wanted 4893969 found 4893913
parent transid verify failed on 87601116807168 wanted 4893969 found 4893913
parent transid verify failed on 87601116807168 wanted 4893969 found 4893913
Ignoring transid failure
parent transid verify failed on 87601117642752 wanted 4893969 found 4893913
parent transid verify failed on 87601117642752 wanted 4893969 found 4893913
Ignoring transid failure
parent transid verify failed on 87601117650944 wanted 4893969 found 4893913
parent transid verify failed on 87601117650944 wanted 4893969 found 4893913
Ignoring transid failure
Couldn't map the block 5764607523034234880
btrfsck: volumes.c:1019: btrfs_num_copies: Assertion `!(!ce)' failed.
zsh: abort (core dumped)  btrfsck /dev/sda

== btrfs fi show ==

Label: 'lake'  uuid: d5e17c49-d980-4bde-bd96-3c8bc95ea077
        Total devices 10 FS bytes used 7.43TB
        devid    9 size 1.82TB used 1.61TB path /dev/sdm1
        devid   12 size 1.82TB used 1.47TB path /dev/sdb
        devid   16 size 1.82TB used 1.47TB path /dev/sde
        devid   13 size 1.82TB used 1.47TB path /dev/sdc
        devid   11 size 1.82TB used 1.47TB path /dev/sda
        devid   19 size 1.82TB used 1.47TB path /dev/sdk
        devid   17 size 1.82TB used 1.47TB path /dev/sdf
        devid   18 size 1.82TB used 1.47TB path /dev/sdg
        devid   15 size 1.82TB used 1.47TB path /dev/sdd
        *** Some devices missing

Re: Unmountable Array After Drive Failure During Device Deletion

[Duncan]

Chris Kastorff posted on Thu, 19 Dec 2013 01:26:57 -0800 as excerpted:

> I'm using btrfs in data and metadata RAID10 on drives (not on md or any
> other fanciness.)
> 
> I was removing a drive (btrfs dev del) and during that operation, a
> different drive in the array failed. Having not had this happen before,
> I shut down the machine immediately due to the extremely loud piezo
> buzzer on the drive controller card. I attempted to do so cleanly, but
> the buzzer cut through my patience and after 4 minutes I cut the power.
> 
> Afterwards, I located and removed the failed drive from the system, and
> then got back to linux. The array no longer mounts ("failed to read the
> system array on sdc"), with nearly identical messages when attempted
> with -o recovery and -o recovery,ro.

This may be a stupid question, but you're missing a drive so the 
filesystem will be degraded, but you didn't mention that in your mount 
options, so...

Did you try mounting with -o degraded (possibly with recovery, etc, also, 
but just try -o degraded plus any normal options first)?

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Unmountable Array After Drive Failure During Device Deletion

[Chris Kastorff]

>> I'm using btrfs in data and metadata RAID10 on drives (not on md or any
>> other fanciness.)
>>
>> I was removing a drive (btrfs dev del) and during that operation, a
>> different drive in the array failed. Having not had this happen before,
>> I shut down the machine immediately due to the extremely loud piezo
>> buzzer on the drive controller card. I attempted to do so cleanly, but
>> the buzzer cut through my patience and after 4 minutes I cut the power.
>>
>> Afterwards, I located and removed the failed drive from the system, and
>> then got back to linux. The array no longer mounts ("failed to read the
>> system array on sdc"), with nearly identical messages when attempted
>> with -o recovery and -o recovery,ro.
> 
> This may be a stupid question, but you're missing a drive so the 
> filesystem will be degraded, but you didn't mention that in your mount 
> options, so...
> 
> Did you try mounting with -o degraded (possibly with recovery, etc, also, 
> but just try -o degraded plus any normal options first)?
> 

I did not try degraded because I didn't remember that there were two
different options for handling broken btrfs volumes.

mount -o degraded,ro yields:

btrfs: device label lake devid 11 transid 4893967 /dev/sda
btrfs: allowing degraded mounts
btrfs: disk space caching is enabled
parent transid verify failed on 87601116364800 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601116364800 (dev /dev/sdf
sector 62986400)
parent transid verify failed on 87601116381184 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601116381184 (dev /dev/sdf
sector 62986432)
parent transid verify failed on 87601115320320 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601115320320 (dev /dev/sdf
sector 62985896)
parent transid verify failed on 87601116368896 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601116368896 (dev /dev/sdf
sector 62986408)
parent transid verify failed on 87601116377088 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601116377088 (dev /dev/sdf
sector 62986424)
btrfs: bdev (null) errs: wr 344288, rd 230234, flush 0, corrupt 0, gen 0
btrfs: bdev /dev/sdm1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
btrfs: bdev /dev/sdg errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
Failed to read block groups: -5
btrfs: open_ctree failed

mount -o degraded,recovery,ro yields:

btrfs: device label lake devid 11 transid 4893967 /dev/sda
btrfs: allowing degraded mounts
btrfs: enabling auto recovery
btrfs: disk space caching is enabled
parent transid verify failed on 87601116798976 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601116798976 (dev /dev/sdg
sector 113318256)
parent transid verify failed on 87601119379456 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601119379456 (dev /dev/sdg
sector 113319456)
parent transid verify failed on 87601116774400 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601116774400 (dev /dev/sdg
sector 113318208)
parent transid verify failed on 87601119391744 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601119391744 (dev /dev/sdg
sector 113319480)
parent transid verify failed on 87601116778496 wanted 4893969 found 4893913
btrfs read error corrected: ino 1 off 87601116778496 (dev /dev/sdg
sector 113318216)
parent transid verify failed on 87601116786688 wanted 4893969 found 4893849
btrfs read error corrected: ino 1 off 87601116786688 (dev /dev/sdg
sector 113318232)
btrfs: bdev (null) errs: wr 344288, rd 230234, flush 0, corrupt 0, gen 0
btrfs: bdev /dev/sdm1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
btrfs: bdev /dev/sdg errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
parent transid verify failed on 87601111515136 wanted 4893968 found 4893913
btrfs read error corrected: ino 1 off 87601111515136 (dev /dev/sdg
sector 113315616)
parent transid verify failed on 87601111523328 wanted 4893968 found 4893913
btrfs read error corrected: ino 1 off 87601111523328 (dev /dev/sdg
sector 113315632)
parent transid verify failed on 87601111535616 wanted 4893968 found 4893913
btrfs read error corrected: ino 1 off 87601111535616 (dev /dev/sdg
sector 113315656)
parent transid verify failed on 87601111556096 wanted 4893968 found 4893913
btrfs read error corrected: ino 1 off 87601111556096 (dev /dev/sdg
sector 113315696)
Failed to read block groups: -5
btrfs: open_ctree failed

Re: Unmountable Array After Drive Failure During Device Deletion

[Chris Kastorff]

>>> I'm using btrfs in data and metadata RAID10 on drives (not on md or any
>>> other fanciness.)
>>>
>>> I was removing a drive (btrfs dev del) and during that operation, a
>>> different drive in the array failed. Having not had this happen before,
>>> I shut down the machine immediately due to the extremely loud piezo
>>> buzzer on the drive controller card. I attempted to do so cleanly, but
>>> the buzzer cut through my patience and after 4 minutes I cut the power.
>>>
>>> Afterwards, I located and removed the failed drive from the system, and
>>> then got back to linux. The array no longer mounts ("failed to read the
>>> system array on sdc"), with nearly identical messages when attempted
>>> with -o recovery and -o recovery,ro.
>>
>> This may be a stupid question, but you're missing a drive so the 
>> filesystem will be degraded, but you didn't mention that in your mount 
>> options, so...
>>
>> Did you try mounting with -o degraded (possibly with recovery, etc, also, 
>> but just try -o degraded plus any normal options first)?
>>
> 
> I did not try degraded because I didn't remember that there were two
> different options for handling broken btrfs volumes.
> 
> mount -o degraded,ro yields:
> 
> btrfs: device label lake devid 11 transid 4893967 /dev/sda
> btrfs: allowing degraded mounts
> btrfs: disk space caching is enabled
> parent transid verify failed on 87601116364800 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601116364800 (dev /dev/sdf
> sector 62986400)
> parent transid verify failed on 87601116381184 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601116381184 (dev /dev/sdf
> sector 62986432)
> parent transid verify failed on 87601115320320 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601115320320 (dev /dev/sdf
> sector 62985896)
> parent transid verify failed on 87601116368896 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601116368896 (dev /dev/sdf
> sector 62986408)
> parent transid verify failed on 87601116377088 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601116377088 (dev /dev/sdf
> sector 62986424)
> btrfs: bdev (null) errs: wr 344288, rd 230234, flush 0, corrupt 0, gen 0
> btrfs: bdev /dev/sdm1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
> btrfs: bdev /dev/sdg errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
> parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
> Failed to read block groups: -5
> btrfs: open_ctree failed
> 
> mount -o degraded,recovery,ro yields:
> 
> btrfs: device label lake devid 11 transid 4893967 /dev/sda
> btrfs: allowing degraded mounts
> btrfs: enabling auto recovery
> btrfs: disk space caching is enabled
> parent transid verify failed on 87601116798976 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601116798976 (dev /dev/sdg
> sector 113318256)
> parent transid verify failed on 87601119379456 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601119379456 (dev /dev/sdg
> sector 113319456)
> parent transid verify failed on 87601116774400 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601116774400 (dev /dev/sdg
> sector 113318208)
> parent transid verify failed on 87601119391744 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601119391744 (dev /dev/sdg
> sector 113319480)
> parent transid verify failed on 87601116778496 wanted 4893969 found 4893913
> btrfs read error corrected: ino 1 off 87601116778496 (dev /dev/sdg
> sector 113318216)
> parent transid verify failed on 87601116786688 wanted 4893969 found 4893849
> btrfs read error corrected: ino 1 off 87601116786688 (dev /dev/sdg
> sector 113318232)
> btrfs: bdev (null) errs: wr 344288, rd 230234, flush 0, corrupt 0, gen 0
> btrfs: bdev /dev/sdm1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
> btrfs: bdev /dev/sdg errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
> parent transid verify failed on 87601111515136 wanted 4893968 found 4893913
> btrfs read error corrected: ino 1 off 87601111515136 (dev /dev/sdg
> sector 113315616)
> parent transid verify failed on 87601111523328 wanted 4893968 found 4893913
> btrfs read error corrected: ino 1 off 87601111523328 (dev /dev/sdg
> sector 113315632)
> parent transid verify failed on 87601111535616 wanted 4893968 found 4893913
> btrfs read error corrected: ino 1 off 87601111535616 (dev /dev/sdg
> sector 113315656)
> parent transid verify failed on 87601111556096 wanted 4893968 found 4893913
> btrfs read error corrected: ino 1 off 87601111556096 (dev /dev/sdg
> sector 113315696)
> Failed to read block groups: -5
> btrfs: open_ctree failed
> 

I should also mention that the "corrupt 4" errs on /dev/sdm1 and
/dev/sdg are there from an earlier btrfs extent corruption bug, and do
not exist on the filesystem anymore (a scrub hours before the device
deletion completed with 0 errors.)

Re: Unmountable Array After Drive Failure During Device Deletion

[Chris Murphy]

On Dec 19, 2013, at 2:26 AM, Chris Kastorff <encryptio@gmail.com> wrote:

> btrfs-progs v0.20-rc1-358-g194aa4a-dirty

Most of what you're using is in the kernel so this is not urgent but if it gets to needing btrfs check/repair, I'd upgrade to v3.12 progs:
https://www.archlinux.org/packages/testing/x86_64/btrfs-progs/


> sd 0:2:3:0: [sdd] Unhandled error code
> sd 0:2:3:0: [sdd]
> Result: hostbyte=0x04 driverbyte=0x00
> sd 0:2:3:0: [sdd] CDB:
> cdb[0]=0x2a: 2a 00 26 89 5b 00 00 00 80 00
> end_request: I/O error, dev sdd, sector 646535936
> btrfs_dev_stat_print_on_error: 7791 callbacks suppressed
> btrfs: bdev /dev/sdd errs: wr 315858, rd 230194, flush 0, corrupt 0, gen 0
> sd 0:2:3:0: [sdd] Unhandled error code
> sd 0:2:3:0: [sdd]
> Result: hostbyte=0x04 driverbyte=0x00
> sd 0:2:3:0: [sdd] CDB:
> cdb[0]=0x2a: 2a 00 26 89 5b 80 00 00 80 00
> end_request: I/O error, dev sdd, sector 646536064

These are hardware errors. And you have missing devices, or at least a message of missing devices. So if a device went bad, and a new one added without deleting the missing one, then the new device only has new data. Data hasn't been recovered and replicated to the replacement. So it's possible with a missing device that's not removed, and a 2nd device failure, to lose some data.

> btrfs read error corrected: ino 1 off 87601116364800 (dev /dev/sdf
> sector 62986400)
> 
> btrfs read error corrected: ino 1 off 87601116798976 (dev /dev/sdg
> sector 113318256)

I'm not sure what constitutes a btrfs read error, maybe the device it originally requested data from didn't have it where it was expected but was able to find it on these devices. If the drive itself has a problem reading a sector and ECC can't correct it, it reports the read error to libata. So kernel messages report this with a line that starts with the word "exception" and then a line with "cmd" that shows what command and LBAs where issued to the drive, and then a "res" line that should contain an error mask with the actual error - bus error, media error. Very often you don't see these and instead see link reset messages, which means the drive is hanging doing something (probably attempting ECC) but then the linux SCSI layer hits its 30 second time out on the (hanged) queued command and resets the drive instead of waiting any longer. And that's a problem also because it prevents bad sectors from being fixed by Btrfs. So they just get worse to the point where then it can't do anything about the situation.

So I think you need to post a full dmesg somewhere rather than snippets. And I'd also like to see the result from smartctl -x for the above three drives, sdd, sdf, and sdg. And we need to know what this missing drive message is about, if you've done a drive replacement and exactly what commands you used to do that and how long ago.


Chris Murphy

Re: Unmountable Array After Drive Failure During Device Deletion

[Chris Kastorff]

On 12/19/2013 02:21 PM, Chris Murphy wrote:
>
> On Dec 19, 2013, at 2:26 AM, Chris Kastorff <encryptio@gmail.com> wrote:
>
>> btrfs-progs v0.20-rc1-358-g194aa4a-dirty
>
> Most of what you're using is in the kernel so this is not urgent but
if it gets to needing btrfs check/repair, I'd upgrade to v3.12 progs:
> https://www.archlinux.org/packages/testing/x86_64/btrfs-progs/

Adding the testing repository is a bad idea for this machine; turning
off the testing repository is extremely error prone.

Instead, I am now using the btrfs tools from
git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs.git's
master (specifically 8cae184), which reports itself as:

deep# ./btrfs version
Btrfs v3.12

>
>> sd 0:2:3:0: [sdd] Unhandled error code
>> sd 0:2:3:0: [sdd]
>> Result: hostbyte=0x04 driverbyte=0x00
>> sd 0:2:3:0: [sdd] CDB:
>> cdb[0]=0x2a: 2a 00 26 89 5b 00 00 00 80 00
>> end_request: I/O error, dev sdd, sector 646535936
>> btrfs_dev_stat_print_on_error: 7791 callbacks suppressed
>> btrfs: bdev /dev/sdd errs: wr 315858, rd 230194, flush 0, corrupt 0,
gen 0
>> sd 0:2:3:0: [sdd] Unhandled error code
>> sd 0:2:3:0: [sdd]
>> Result: hostbyte=0x04 driverbyte=0x00
>> sd 0:2:3:0: [sdd] CDB:
>> cdb[0]=0x2a: 2a 00 26 89 5b 80 00 00 80 00
>> end_request: I/O error, dev sdd, sector 646536064
>
> These are hardware errors. And you have missing devices, or at least
> a message of missing devices. So if a device went bad, and a new one
> added without deleting the missing one, then the new device only has
> new data. Data hasn't been recovered and replicated to the
> replacement. So it's possible with a missing device that's not
> removed, and a 2nd device failure, to lose some data.
>

This is not what happened, as I explained earlier; I shall explain
again, with more verbosity:

- Array is good. All drives are accounted for, btrfs scrub runs cleanly.
btrfs fi show shows no missing drives and reasonable allocations.
- I start btrfs dev del to remove devid 9. It chugs along with no
errors, until:
- Another drive in the array (NOT THE ONE I RAN DEV DEL ON) fails, and
all reads and writes to it fail, causing the SCSI errors above.
- I attempt clean shutdown. It takes too long for because my drive
controller card is buzzing loudly and the neighbors are sensitive to
noise, so:
- I power down the machine uncleanly.
- I remove the failed drive, NOT the one I ran dev del on.
- I reboot, attempt to mount with various options, all of which cause
the kernel to yell at me and the mount command returns failure.

>From what I understand, at all points there should be at least two
copies of every extent during a dev del when all chunks are allocated
RAID10 (and they are, according to btrfs fi df ran before on the mounted
fs).

Because of this, I expect to be able to use the chunks from the (not
successfully removed) devid=9, as I have done many many times before due
to other btrfs bugs that needed unclean shutdowns during dev del.

Under the assumption devid=9 is good, if a slightly out of date on
transid (which ALL data says is true), I should be able to completely
recover all data, because data that was not modified during the deletion
resides on devid=9, and data that was modified should be redundantly
(RAID10) stored on the remaining drives, and thus should work given this
case of a single drive failure.

Is this not the case? Does btrfs not maintain redundancy during device
removal?

>> btrfs read error corrected: ino 1 off 87601116364800 (dev /dev/sdf
>> sector 62986400)
>>
>> btrfs read error corrected: ino 1 off 87601116798976 (dev /dev/sdg
>> sector 113318256)
>
> I'm not sure what constitutes a btrfs read error, maybe the device it
> originally requested data from didn't have it where it was expected
> but was able to find it on these devices. If the drive itself has a
> problem reading a sector and ECC can't correct it, it reports the
> read error to libata. So kernel messages report this with a line that
> starts with the word "exception" and then a line with "cmd" that
> shows what command and LBAs where issued to the drive, and then a
> "res" line that should contain an error mask with the actual error -
> bus error, media error. Very often you don't see these and instead
> see link reset messages, which means the drive is hanging doing
> something (probably attempting ECC) but then the linux SCSI layer
> hits its 30 second time out on the (hanged) queued command and resets
> the drive instead of waiting any longer. And that's a problem also
> because it prevents bad sectors from being fixed by Btrfs. So they
> just get worse to the point where then it can't do anything about the
> situation.

There was a single drive immediately failing all its writes and reads
because that's how the controller card was configured. No ECC failures,
no timeouts. I have hit those issues on other arrays, but the drive
controller I'm using here correctly and immediately returned errors on
requests when the drive failed. I am no stranger to SCSI error messages
on both shitty drive interfaces (which behave as you described) and
reasonable ones (like the immediate failure I saw here.)

This single drive failure is the first case of errors after the most
recent (hours before) btrfs scrub which found no errors, and btrfs fi
show showed no warnings.

>
> So I think you need to post a full dmesg somewhere rather than
> snippets. And I'd also like to see the result from smartctl -x for
> the above three drives, sdd, sdf, and sdg. And we need to know what
> this missing drive message is about, if you've done a drive
> replacement and exactly what commands you used to do that and how
> long ago.

I cannot post the full dmesg during the drive failure; since it was
gigabytes of nothing but (completely expected) SCSI errors and btrfs log
lines saying write failed, as I snipped above, journald cut the top of
it because there was too much data.

Full dmesg after reboot annotated with mount and btrfsck commands as I
ran them is at https://encryptio.com/z/btrfs-failure-dmesg.txt ; My
annotations are lines starting with #, all other lines are complete and
unmodified.

Most of the drives are on an LSI MegaRAID 9260-8i card, but it is
configured write-through and with single drive RAID0 LUNs, which is as
close to passthru as you can get. btrfs handles redundancy, NOT the
drive controller;

This filesystem was originally spread across drives not on the
controller card; I used btrfs dev add /dev/<card drive> /mnt/lake for
each of the drives followed by btrfs dev del /dev/<old drive> /mnt/lake
followed by btrfs balance start /mnt/lake to move the filesystem to the
new drives.

The raid card software (an on-boot configuration tool) shows no SMART
errors or warnings on any of the remaining drives. Unfortunately I can't
get smartctl to actually grab any data through the controller:

deep# ./smartctl -x /dev/sdf
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.12.4-1-ARCH] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

Smartctl open device: /dev/sdf failed: DELL or MegaRaid controller,
please try adding '-d megaraid,N'

deep# ./smartctl -d megaraid,0 -x /dev/sdf
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.12.4-1-ARCH] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

Smartctl open device: /dev/sdf [megaraid_disk_00] failed: INQUIRY failed

Also, the drives renumber themselves often. If you want to ask me about
a specific drive, please use the btrfs devid, which I can look up in the
currently booted device ordering.

Re: Unmountable Array After Drive Failure During Device Deletion

[Chris Murphy]

On Dec 19, 2013, at 5:06 PM, Chris Kastorff <encryptio@gmail.com> wrote:

> On 12/19/2013 02:21 PM, Chris Murphy wrote:
>> 
>> On Dec 19, 2013, at 2:26 AM, Chris Kastorff <encryptio@gmail.com> wrote:
>> 
>>> btrfs-progs v0.20-rc1-358-g194aa4a-dirty
>> 
>> Most of what you're using is in the kernel so this is not urgent but
> if it gets to needing btrfs check/repair, I'd upgrade to v3.12 progs:
>> https://www.archlinux.org/packages/testing/x86_64/btrfs-progs/
> 
> Adding the testing repository is a bad idea for this machine; turning
> off the testing repository is extremely error prone.
> 
> Instead, I am now using the btrfs tools from
> git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs.git's
> master (specifically 8cae184), which reports itself as:
> 
> deep# ./btrfs version
> Btrfs v3.12

Good. As I thought about it again, you're using user space tools to add, remove, replace devices also, and that code has changed too, so better to use current.

> - Array is good. All drives are accounted for, btrfs scrub runs cleanly.
> btrfs fi show shows no missing drives and reasonable allocations.
> - I start btrfs dev del to remove devid 9. It chugs along with no
> errors, until:
> - Another drive in the array (NOT THE ONE I RAN DEV DEL ON) fails, and
> all reads and writes to it fail, causing the SCSI errors above.
> - I attempt clean shutdown. It takes too long for because my drive
> controller card is buzzing loudly and the neighbors are sensitive to
> noise, so:
> - I power down the machine uncleanly.
> - I remove the failed drive, NOT the one I ran dev del on.
> - I reboot, attempt to mount with various options, all of which cause
> the kernel to yell at me and the mount command returns failure.

devid 9 is "device delete" in-progress, and while that's occurring devid 15 fails completely. Is that correct?

Because previously you reported, in part this:
       devid   15 size 1.82TB used 1.47TB path /dev/sdd
       *** Some devices missing

And this:

sd 0:2:3:0: [sdd] Unhandled error code

That why I was confused. It looks like dead/missing device is one devid, and then devid 15 /dev/sdd is also having hardware problems - because all of this was posted at the same time. But I take it they're different boots and the /dev/sdd's are actually two different devids.

So devid 9 was "deleted" and then devid 14 failed. Right? Lovely when /dev/sdX changes between boots.


> From what I understand, at all points there should be at least two
> copies of every extent during a dev del when all chunks are allocated
> RAID10 (and they are, according to btrfs fi df ran before on the mounted
> fs).
> 
> Because of this, I expect to be able to use the chunks from the (not
> successfully removed) devid=9, as I have done many many times before due
> to other btrfs bugs that needed unclean shutdowns during dev del.

I haven't looked at the code or read anything this specific on the state of the file system during a device delete. But my expectation is that there are 1-2 chunks available for writes. And 2-3 chunks available for reads. Some writes must be only one copy because a chunk hasn't yet been replicated elsewhere, and presumably the device being "deleted" is not subject to writes as the transid also implies. Whereas devid 9 is one set of chunks for reading, those chunks have pre-existing copies elsewhere in the file system so that's two copies. And there's a replication in progress of the soon to be removed chunks. So that's up to three copies.

Problem is that for sure you've lost some chunks due to the failed/missing device. Normal raid10, it's unambiguous whether we've lost two mirrored sets. With Btrfs that's not clear as chunks are distributed. So it's possible that there are some chunks that don't exist at all for writes, and only 1 for reads. It may be no chunks are in common between devid 9 and the dead one. It may be only a couple of data or metadata chunks are in common.



> 
> Under the assumption devid=9 is good, if a slightly out of date on
> transid (which ALL data says is true), I should be able to completely
> recover all data, because data that was not modified during the deletion
> resides on devid=9, and data that was modified should be redundantly
> (RAID10) stored on the remaining drives, and thus should work given this
> case of a single drive failure.
> 
> Is this not the case? Does btrfs not maintain redundancy during device
> removal?

Good questions. I'm not certain. But the speculation seems reasonable, not accounting for the missing device. That's what makes this different.



>>> btrfs read error corrected: ino 1 off 87601116364800 (dev /dev/sdf
>>> sector 62986400)
>>> 
>>> btrfs read error corrected: ino 1 off 87601116798976 (dev /dev/sdg
>>> sector 113318256)
>> 
>> I'm not sure what constitutes a btrfs read error, maybe the device it
>> originally requested data from didn't have it where it was expected
>> but was able to find it on these devices. If the drive itself has a
>> problem reading a sector and ECC can't correct it, it reports the
>> read error to libata. So kernel messages report this with a line that
>> starts with the word "exception" and then a line with "cmd" that
>> shows what command and LBAs where issued to the drive, and then a
>> "res" line that should contain an error mask with the actual error -
>> bus error, media error. Very often you don't see these and instead
>> see link reset messages, which means the drive is hanging doing
>> something (probably attempting ECC) but then the linux SCSI layer
>> hits its 30 second time out on the (hanged) queued command and resets
>> the drive instead of waiting any longer. And that's a problem also
>> because it prevents bad sectors from being fixed by Btrfs. So they
>> just get worse to the point where then it can't do anything about the
>> situation.
> 
> There was a single drive immediately failing all its writes and reads
> because that's how the controller card was configured. 

OK so my next confusion stems from the error corrected messages above. I kinda expect error message to report the device with the error. But this seems to rather be an informational message that a read error was successfully correct, operations are normal. If so, that means /dev/sdf and /dev/sdg are the devices the recovered data came from?

> 
> Full dmesg after reboot annotated with mount and btrfsck commands as I
> ran them is at https://encryptio.com/z/btrfs-failure-dmesg.txt ;

Off topic:

This:
[    0.866365] ACPI Error: [DSSP] Namespace lookup failure, AE_NOT_FOUND (20130725/psargs-359)
[    0.866372] ACPI Error: Method parse/execution failed [\_SB_.PCI0.SAT0.SPT0._GTF] (Node ffff8807f3ca1460), AE_NOT_FOUND (20130725/psparse-536)
 
leads to these two:
https://bugzilla.redhat.com/show_bug.cgi?id=491588

http://markmail.org/message/eiodqm526mtejbpj#mid:srmaa4qaqkucz4gf+state:results

It suggests updating BIOS. Although nothing newer or in combination with 9260-8i…


Back to topic:

You have one missing/dead device. And devid 9 which was requested for 'device delete' but are uncertain about its state, whether the in-progress "delete" was halted upon another drive dying.

The chunks from the missing/dead drive are not automatically replicated. You need to use 'btrfs device delete missing' or you need to use 'btrfs replace' which is a newish command:

http://lwn.net/Articles/524589/

And then once that's complete it seems like you should be able to try deleting devid 9 again.

So where you're at now is you need a way to successfully mount the file system with -o degraded, not ro, so that you can successfully issue those commands, which I wouldn't expect work with an ro file system. But the problem is that you get a open_ctree failed message.

Next steps:

1. btrfs-image -c 9 -t <#cores> (see man page)
This is optional but one of the devs might want to see this because it should be a more rare case that either normal mount fix ups or additional recovery fix ups can't deal with this problem.

2. btrfs-zero-log <dev>
What's the full result from btrfs check <device>?

3. Try to mount again with -o degraded,recovery and report back.


> The raid card software (an on-boot configuration tool) shows no SMART
> errors or warnings on any of the remaining drives. Unfortunately I can't
> get smartctl to actually grab any data through the controller:

Hmm, you might check and see if LSI has some tools through the controller to access SAS SMART status of drives attached to it; or if it has a pass through function for smartmontools. This is a low priority at the moment. It's just to see if the drives have reported read, write, CRC, or bad sector errors.

I think where you're at with this is that the log is probably corrupt due to the unclean shutdown and that's why it won't mount. So zeroing the log should make it possible, while losing about 30 second worth of whatever was happening before the shutdown.

There's an ambiguity on the status of devid 9 being "deleted" when devid 14 subsequently failed. I think devid 14 needs to be successfully removed first because there's only one copy right now of its data, and possibly some of it is on devid 9. If that succeeds, next is to remove devid 9.

And I'd also check out the replace command to see if that's applicable for either of these rather than using delete. It's a new feature, I haven't used it yet.


Chris Murphy

Re: Unmountable Array After Drive Failure During Device Deletion

[Chris Kastorff]

>> - Array is good. All drives are accounted for, btrfs scrub runs cleanly.
>> btrfs fi show shows no missing drives and reasonable allocations.
>> - I start btrfs dev del to remove devid 9. It chugs along with no
>> errors, until:
>> - Another drive in the array (NOT THE ONE I RAN DEV DEL ON) fails, and
>> all reads and writes to it fail, causing the SCSI errors above.
>> - I attempt clean shutdown. It takes too long for because my drive
>> controller card is buzzing loudly and the neighbors are sensitive to
>> noise, so:
>> - I power down the machine uncleanly.
>> - I remove the failed drive, NOT the one I ran dev del on.
>> - I reboot, attempt to mount with various options, all of which cause
>> the kernel to yell at me and the mount command returns failure.
> 
> devid 9 is "device delete" in-progress, and while that's occurring devid 15 fails completely. Is that correct?

Either devid 14 or devid 10 (from memory) dropped out, devid 15 is still working.

> Because previously you reported, in part this:
>        devid   15 size 1.82TB used 1.47TB path /dev/sdd
>        *** Some devices missing
> 
> And this:
> 
> sd 0:2:3:0: [sdd] Unhandled error code

Yeah, those two are from different boots. sdd is the one that dropped out, and after a reboot another (working) drive was renumbered to sdd. Sorry for the confusion.

(Also note that if devid 15 was missing, it would not be reported in btrfs fi show.)

> That why I was confused. It looks like dead/missing device is one devid, and then devid 15 /dev/sdd is also having hardware problems - because all of this was posted at the same time. But I take it they're different boots and the /dev/sdd's are actually two different devids.
> 
> So devid 9 was "deleted" and then devid 14 failed. Right? Lovely when /dev/sdX changes between boots.

It never finished the deletion (was probably about halfway through,
based on previous dev dels), but otherwise yes.

>> From what I understand, at all points there should be at least two
>> copies of every extent during a dev del when all chunks are allocated
>> RAID10 (and they are, according to btrfs fi df ran before on the mounted
>> fs).
>>
>> Because of this, I expect to be able to use the chunks from the (not
>> successfully removed) devid=9, as I have done many many times before due
>> to other btrfs bugs that needed unclean shutdowns during dev del.
> 
> I haven't looked at the code or read anything this specific on the state of the file system during a device delete. But my expectation is that there are 1-2 chunks available for writes. And 2-3 chunks available for reads. Some writes must be only one copy because a chunk hasn't yet been replicated elsewhere, and presumably the device being "deleted" is not subject to writes as the transid also implies. Whereas devid 9 is one set of chunks for reading, those chunks have pre-existing copies elsewhere in the file system so that's two copies. And there's a replication in progress of the soon to be removed chunks. So that's up to three copies.
> 
> Problem is that for sure you've lost some chunks due to the failed/missing device. Normal raid10, it's unambiguous whether we've lost two mirrored sets. With Btrfs that's not clear as chunks are distributed. So it's possible that there are some chunks that don't exist at all for writes, and only 1 for reads. It may be no chunks are in common between devid 9 and the dead one. It may be only a couple of data or metadata chunks are in common.
> 
> 
> 
>>
>> Under the assumption devid=9 is good, if a slightly out of date on
>> transid (which ALL data says is true), I should be able to completely
>> recover all data, because data that was not modified during the deletion
>> resides on devid=9, and data that was modified should be redundantly
>> (RAID10) stored on the remaining drives, and thus should work given this
>> case of a single drive failure.
>>
>> Is this not the case? Does btrfs not maintain redundancy during device
>> removal?
> 
> Good questions. I'm not certain. But the speculation seems reasonable, not accounting for the missing device. That's what makes this different.
> 
> 
> 
>>>> btrfs read error corrected: ino 1 off 87601116364800 (dev /dev/sdf
>>>> sector 62986400)
>>>>
>>>> btrfs read error corrected: ino 1 off 87601116798976 (dev /dev/sdg
>>>> sector 113318256)
>>>
>>> I'm not sure what constitutes a btrfs read error, maybe the device it
>>> originally requested data from didn't have it where it was expected
>>> but was able to find it on these devices. If the drive itself has a
>>> problem reading a sector and ECC can't correct it, it reports the
>>> read error to libata. So kernel messages report this with a line that
>>> starts with the word "exception" and then a line with "cmd" that
>>> shows what command and LBAs where issued to the drive, and then a
>>> "res" line that should contain an error mask with the actual error -
>>> bus error, media error. Very often you don't see these and instead
>>> see link reset messages, which means the drive is hanging doing
>>> something (probably attempting ECC) but then the linux SCSI layer
>>> hits its 30 second time out on the (hanged) queued command and resets
>>> the drive instead of waiting any longer. And that's a problem also
>>> because it prevents bad sectors from being fixed by Btrfs. So they
>>> just get worse to the point where then it can't do anything about the
>>> situation.
>>
>> There was a single drive immediately failing all its writes and reads
>> because that's how the controller card was configured. 
> 
> OK so my next confusion stems from the error corrected messages above. I kinda expect error message to report the device with the error. But this seems to rather be an informational message that a read error was successfully correct, operations are normal. If so, that means /dev/sdf and /dev/sdg are the devices the recovered data came from?

That's what I guessed when I saw it too.

>> Full dmesg after reboot annotated with mount and btrfsck commands as I
>> ran them is at https://encryptio.com/z/btrfs-failure-dmesg.txt ;
> 
> Off topic:
> 
> This:
> [    0.866365] ACPI Error: [DSSP] Namespace lookup failure, AE_NOT_FOUND (20130725/psargs-359)
> [    0.866372] ACPI Error: Method parse/execution failed [\_SB_.PCI0.SAT0.SPT0._GTF] (Node ffff8807f3ca1460), AE_NOT_FOUND (20130725/psparse-536)
>  
> leads to these two:
> https://bugzilla.redhat.com/show_bug.cgi?id=491588
> 
> http://markmail.org/message/eiodqm526mtejbpj#mid:srmaa4qaqkucz4gf+state:results
> 
> It suggests updating BIOS. Although nothing newer or in combination with 9260-8i…
> 
> 
> Back to topic:
> 
> You have one missing/dead device. And devid 9 which was requested for 'device delete' but are uncertain about its state, whether the in-progress "delete" was halted upon another drive dying.
> 
> The chunks from the missing/dead drive are not automatically replicated. You need to use 'btrfs device delete missing' or you need to use 'btrfs replace' which is a newish command:
> 
> http://lwn.net/Articles/524589/
> 
> And then once that's complete it seems like you should be able to try deleting devid 9 again.
> 
> So where you're at now is you need a way to successfully mount the file system with -o degraded, not ro, so that you can successfully issue those commands, which I wouldn't expect work with an ro file system. But the problem is that you get a open_ctree failed message.
> 
> Next steps:
> 
> 1. btrfs-image -c 9 -t <#cores> (see man page)
> This is optional but one of the devs might want to see this because it should be a more rare case that either normal mount fix ups or additional recovery fix ups can't deal with this problem.

This fails:

deep# ./btrfs-image -c 9 -t 4 /dev/sda btrfsimg
warning, device 14 is missing
warning devid 14 not found already
parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
Ignoring transid failure
parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
Ignoring transid failure
Error going to next leaf -5
create failed (Bad file descriptor)

Adding -w causes it to hang forever (at least 16 hours) after an initial few
minutes of seemingly useful io:

deep# ./btrfs-image -c 9 -t 4 -w /dev/sda btrfsimg
warning, device 14 is missing
warning devid 14 not found already
parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
Ignoring transid failure
^C

> 2. btrfs-zero-log <dev>

This one gives the most useful (and scariest) error message I've seen so far:

deep# ./btrfs-zero-log /dev/sda
warning, device 14 is missing
warning devid 14 not found already
parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
Ignoring transid failure
parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
Ignoring transid failure
parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
Ignoring transid failure
Unable to find block group for 0
btrfs-zero-log: extent-tree.c:288: find_search_start: Assertion `!(1)'
failed.
zsh: abort (core dumped)  ./btrfs-zero-log /dev/sda

> What's the full result from btrfs check <device>?

deep# ./btrfs check /dev/sda
warning, device 14 is missing
warning devid 14 not found already
parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
Ignoring transid failure
Checking filesystem on /dev/sda
UUID: d5e17c49-d980-4bde-bd96-3c8bc95ea077
checking extents
parent transid verify failed on 87601117159424 wanted 4893969 found 4893913
parent transid verify failed on 87601117159424 wanted 4893969 found 4893913
parent transid verify failed on 87601117163520 wanted 4893969 found 4893913
parent transid verify failed on 87601117163520 wanted 4893969 found 4893913
parent transid verify failed on 87601117638656 wanted 4893969 found 4893913
parent transid verify failed on 87601117638656 wanted 4893969 found 4893913
Ignoring transid failure
parent transid verify failed on 87601117171712 wanted 4893969 found 4893913
parent transid verify failed on 87601117171712 wanted 4893969 found 4893913
parent transid verify failed on 87601117175808 wanted 4893969 found 4893913
parent transid verify failed on 87601117175808 wanted 4893969 found 4893913
parent transid verify failed on 87601117188096 wanted 4893969 found 4893913
parent transid verify failed on 87601117188096 wanted 4893969 found 4893913
parent transid verify failed on 87601116807168 wanted 4893969 found 4893913
parent transid verify failed on 87601116807168 wanted 4893969 found 4893913
Ignoring transid failure
parent transid verify failed on 87601117642752 wanted 4893969 found 4893913
parent transid verify failed on 87601117642752 wanted 4893969 found 4893913
Ignoring transid failure
parent transid verify failed on 87601117650944 wanted 4893969 found 4893913
parent transid verify failed on 87601117650944 wanted 4893969 found 4893913
Ignoring transid failure
Couldn't map the block 5764607523034234880
btrfs: volumes.c:1102: btrfs_num_copies: Assertion `!(!ce)' failed.
zsh: abort (core dumped)  ./btrfs check /dev/sda

> 3. Try to mount again with -o degraded,recovery and report back.

Since btrfs-zero-log (probably) didn't modify anything, the error
message is the same:

btrfs: allowing degraded mounts
btrfs: enabling auto recovery
btrfs: disk space caching is enabled
btrfs: bdev (null) errs: wr 344288, rd 230234, flush 0, corrupt 0, gen 0
btrfs: bdev /dev/sdm1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
btrfs: bdev /dev/sdg errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
Failed to read block groups: -5
btrfs: open_ctree failed

>> The raid card software (an on-boot configuration tool) shows no SMART
>> errors or warnings on any of the remaining drives. Unfortunately I can't
>> get smartctl to actually grab any data through the controller:
> 
> Hmm, you might check and see if LSI has some tools through the controller to access SAS SMART status of drives attached to it; or if it has a pass through function for smartmontools. This is a low priority at the moment. It's just to see if the drives have reported read, write, CRC, or bad sector errors.
> 
> I think where you're at with this is that the log is probably corrupt due to the unclean shutdown and that's why it won't mount. So zeroing the log should make it possible, while losing about 30 second worth of whatever was happening before the shutdown.
> 
> There's an ambiguity on the status of devid 9 being "deleted" when devid 14 subsequently failed. I think devid 14 needs to be successfully removed first because there's only one copy right now of its data, and possibly some of it is on devid 9. If that succeeds, next is to remove devid 9.
> 
> And I'd also check out the replace command to see if that's applicable for either of these rather than using delete. It's a new feature, I haven't used it yet.

If replace works with "missing", that seems like the right way to go.

btrfs-zero-log's "Unable to find block group for 0" combined with the
earlier kernel message on mount attempts "btrfs: failed to read the
system array on sdc" and btrfsck's "Couldn't map the block %ld" tells me
the (first) underlying problem is that the block group tree(?) in the
system allocated data is screwed up.

I have no idea where to go from here, aside from grabbing a compiler and
having at the disk structures myself.

Re: Unmountable Array After Drive Failure During Device Deletion

[Chris Murphy]

On Dec 21, 2013, at 4:16 PM, Chris Kastorff <encryptio@gmail.com> wrote:
>> 
>> 1. btrfs-image -c 9 -t <#cores> (see man page)
>> This is optional but one of the devs might want to see this because it should be a more rare case that either normal mount fix ups or additional recovery fix ups can't deal with this problem.
> 
> This fails:
> 
> deep# ./btrfs-image -c 9 -t 4 /dev/sda btrfsimg
> warning, device 14 is missing
> warning devid 14 not found already
> parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
> parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
> Ignoring transid failure
> parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
> Ignoring transid failure
> Error going to next leaf -5
> create failed (Bad file descriptor)

Well, that's unfortunate. Someone else is going to have to comment on the confusion of the tools trying to fix the file system while a device is missing, which cannot be removed due to the fact the file system can't be mounted, because it needs to be fixed first. Circular problem.

> 
>> 3. Try to mount again with -o degraded,recovery and report back.
> 
> Since btrfs-zero-log (probably) didn't modify anything, the error
> message is the same:
> 
> btrfs: allowing degraded mounts
> btrfs: enabling auto recovery
> btrfs: disk space caching is enabled
> btrfs: bdev (null) errs: wr 344288, rd 230234, flush 0, corrupt 0, gen 0
> btrfs: bdev /dev/sdm1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
> btrfs: bdev /dev/sdg errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
> parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
> Failed to read block groups: -5
> btrfs: open_ctree failed

How about:

-o skip_balance,degraded,recovery

If that fails, try:


-o skip_balance,degraded,recovery,ro

The ro file system probably doesn't let you delete missing, but it's worth a shot because the seems to be limiting repairs due to the missing device.

If you still have failure, it's worth repeating with the absolute latest kernel you can find or even build.

After that it gets really aggressive to dangerous and I'm not sure what to recommend next except avoid btrfs check --repair until dead last. I'd sooner go for the ro mount and use btrfs send/receive to get the current data you want off the file system, and create a new one from scratch. 

> 
> btrfs-zero-log's "Unable to find block group for 0" combined with the
> earlier kernel message on mount attempts "btrfs: failed to read the
> system array on sdc" and btrfsck's "Couldn't map the block %ld" tells me
> the (first) underlying problem is that the block group tree(?) in the
> system allocated data is screwed up.
> 
> I have no idea where to go from here, aside from grabbing a compiler and
> having at the disk structures myself.

There are some other options but they get progressively and quickly into possibly making things a lot worse. At a certain point it's an extraction operation rather than repair and continue.

Chris Murphy

Re: Unmountable Array After Drive Failure During Device Deletion

[ronnie sahlberg]

Similar things happened to me. (See my unanswered posts ~1Sep, this fs
is not really ready for production I think)

When you get wrong transid errors and reports that you have checksums
being repaired,
that is all bad news and no one can help you.

Unfortunately there are, I think, no real tools to fix basic fs erros.


I never managed to get the my in a state where it could be mounted at all
but did manage to recover most of my data using
btrfs restore from
https://github.com/FauxFaux/btrfs-progs

This is the argument from that command that I used to recover data :
I got most data back with ith but YMMV.

  commit 2a2a1fb21d375a46f9073e44a7b9d9bb7bfaa1e2
  Author: Peter Stuge <peter@stuge.se>
  Date:   Fri Nov 25 01:03:58 2011 +0100

    restore: Add regex matching of paths and files to be restored

    The option -m is used to specify the regex string. -c is used to
    specify case insensitive matching. -i was already taken.

    In order to restore only a single folder somewhere in the btrfs
    tree, it is unfortunately neccessary to construct a slightly
    nontrivial regex, e.g.:

    restore -m '^/(|home(|/username(|/Desktop(|/.*))))$' /dev/sdb2 /output

    This is needed in order to match each directory along the way to the
    Desktop directory, as well as all contents below the Desktop directory.

    Signed-off-by: Peter Stuge <peter@stuge.se>
    Signed-off-by: Josef Bacik <josef@redhat.com>


I wont give advice for your data.
For my data, I copied as much data as I could recover from the
filesystem over to a different filesystem
using the tools in the repo above.
After that destroy the damaged filesystem and rebuild from scratch.


Then  depending on how important your data is, you start making
backups regularely, or switch to a less fragile and unrepairable fs.


On Thu, Dec 19, 2013 at 1:26 AM, Chris Kastorff <encryptio@gmail.com> wrote:
> I'm using btrfs in data and metadata RAID10 on drives (not on md or any
> other fanciness.)
>
> I was removing a drive (btrfs dev del) and during that operation, a
> different drive in the array failed. Having not had this happen before,
> I shut down the machine immediately due to the extremely loud piezo
> buzzer on the drive controller card. I attempted to do so cleanly, but
> the buzzer cut through my patience and after 4 minutes I cut the power.
>
> Afterwards, I located and removed the failed drive from the system, and
> then got back to linux. The array no longer mounts ("failed to read the
> system array on sdc"), with nearly identical messages when attempted
> with -o recovery and -o recovery,ro.
>
> btrfsck asserts and coredumps, as usual.
>
> The drive that was being removed is devid 9 in the array, and is
> /dev/sdm1 in the btrfs fi show seen below.
>
> Kernel 3.12.4-1-ARCH, btrfs-progs v0.20-rc1-358-g194aa4a-dirty
> (archlinux build.)
>
> Can I recover the array?
>
> == dmesg during failure ==
>
> ...
> sd 0:2:3:0: [sdd] Unhandled error code
> sd 0:2:3:0: [sdd]
> Result: hostbyte=0x04 driverbyte=0x00
> sd 0:2:3:0: [sdd] CDB:
> cdb[0]=0x2a: 2a 00 26 89 5b 00 00 00 80 00
> end_request: I/O error, dev sdd, sector 646535936
> btrfs_dev_stat_print_on_error: 7791 callbacks suppressed
> btrfs: bdev /dev/sdd errs: wr 315858, rd 230194, flush 0, corrupt 0, gen 0
> sd 0:2:3:0: [sdd] Unhandled error code
> sd 0:2:3:0: [sdd]
> Result: hostbyte=0x04 driverbyte=0x00
> sd 0:2:3:0: [sdd] CDB:
> cdb[0]=0x2a: 2a 00 26 89 5b 80 00 00 80 00
> end_request: I/O error, dev sdd, sector 646536064
> ...
>
> == dmesg after new boot, mounting attempt ==
>
> btrfs: device label lake devid 11 transid 4893967 /dev/sda
> btrfs: disk space caching is enabled
> btrfs: failed to read the system array on sdc
> btrfs: open_ctree failed
>
> == dmesg after new boot, mounting attempt with -o recovery,ro ==
>
> btrfs: device label lake devid 11 transid 4893967 /dev/sda
> btrfs: enabling auto recovery
> btrfs: disk space caching is enabled
> btrfs: failed to read the system array on sdc
> btrfs: open_ctree failed
>
> == btrfsck ==
>
> deep# btrfsck /dev/sda
> warning, device 14 is missing
> warning devid 14 not found already
> parent transid verify failed on 87601116364800 wanted 4893969 found 4893913
> parent transid verify failed on 87601116364800 wanted 4893969 found 4893913
> parent transid verify failed on 87601116381184 wanted 4893969 found 4893913
> parent transid verify failed on 87601116381184 wanted 4893969 found 4893913
> parent transid verify failed on 87601115320320 wanted 4893969 found 4893913
> parent transid verify failed on 87601115320320 wanted 4893969 found 4893913
> parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
> parent transid verify failed on 87601117097984 wanted 4893969 found 4892460
> Ignoring transid failure
> Checking filesystem on /dev/sda
> UUID: d5e17c49-d980-4bde-bd96-3c8bc95ea077
> checking extents
> parent transid verify failed on 87601117159424 wanted 4893969 found 4893913
> parent transid verify failed on 87601117159424 wanted 4893969 found 4893913
> parent transid verify failed on 87601116368896 wanted 4893969 found 4893913
> parent transid verify failed on 87601116368896 wanted 4893969 found 4893913
> parent transid verify failed on 87601117163520 wanted 4893969 found 4893913
> parent transid verify failed on 87601117163520 wanted 4893969 found 4893913
> parent transid verify failed on 87601117638656 wanted 4893969 found 4893913
> parent transid verify failed on 87601117638656 wanted 4893969 found 4893913
> Ignoring transid failure
> parent transid verify failed on 87601117171712 wanted 4893969 found 4893913
> parent transid verify failed on 87601117171712 wanted 4893969 found 4893913
> parent transid verify failed on 87601117175808 wanted 4893969 found 4893913
> parent transid verify failed on 87601117175808 wanted 4893969 found 4893913
> parent transid verify failed on 87601117188096 wanted 4893969 found 4893913
> parent transid verify failed on 87601117188096 wanted 4893969 found 4893913
> parent transid verify failed on 87601116807168 wanted 4893969 found 4893913
> parent transid verify failed on 87601116807168 wanted 4893969 found 4893913
> Ignoring transid failure
> parent transid verify failed on 87601117642752 wanted 4893969 found 4893913
> parent transid verify failed on 87601117642752 wanted 4893969 found 4893913
> Ignoring transid failure
> parent transid verify failed on 87601117650944 wanted 4893969 found 4893913
> parent transid verify failed on 87601117650944 wanted 4893969 found 4893913
> Ignoring transid failure
> Couldn't map the block 5764607523034234880
> btrfsck: volumes.c:1019: btrfs_num_copies: Assertion `!(!ce)' failed.
> zsh: abort (core dumped)  btrfsck /dev/sda
>
> == btrfs fi show ==
>
> Label: 'lake'  uuid: d5e17c49-d980-4bde-bd96-3c8bc95ea077
>         Total devices 10 FS bytes used 7.43TB
>         devid    9 size 1.82TB used 1.61TB path /dev/sdm1
>         devid   12 size 1.82TB used 1.47TB path /dev/sdb
>         devid   16 size 1.82TB used 1.47TB path /dev/sde
>         devid   13 size 1.82TB used 1.47TB path /dev/sdc
>         devid   11 size 1.82TB used 1.47TB path /dev/sda
>         devid   19 size 1.82TB used 1.47TB path /dev/sdk
>         devid   17 size 1.82TB used 1.47TB path /dev/sdf
>         devid   18 size 1.82TB used 1.47TB path /dev/sdg
>         devid   15 size 1.82TB used 1.47TB path /dev/sdd
>         *** Some devices missing
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unmountable Array After Drive Failure During Device Deletion

[Duncan]

ronnie sahlberg posted on Sat, 21 Dec 2013 17:15:33 -0800 as excerpted:

> Similar things happened to me. (See my unanswered posts ~1Sep, this fs
> is not really ready for production I think)

No "I think" about that one.  Known fact.  Btrfs is not yet fully stable 
and is still under heavy development, and there are public warnings about 
having backups and being prepared to use them before testing it, too.

[snip]

> Then  depending on how important your data is, you start making backups
> regularely, or switch to a less fragile and unrepairable fs.

Of course, if the data's /that/ important, backups that are tested and 
ready to use have been done already, even if it's on a stable filesystem; 
certainly even more so if it's on a development filesystem.  Otherwise, 
the data simply isn't that important, no matter what people might claim, 
as their actions belie their words!

While the btrfs kernel config option no longer (as of 3.12 IIRC) directly 
calls the filesystem unstable, it /does/ say the disk format is "no 
longer expected to change", which doesn't exactly sound entirely stable, 
either (you don't see that sort of qualifier on ext4, for instance, let 
alone ext3, or reiserfs, the generation of filesystem that some of the 
folks who /really/ value their data are either still on or have just 
recently upgraded from).  Further, it points to 
http://btrfs.wiki.kernel.org , which on the main page under stability 
status has this to say:

>>>> The Btrfs code base is under heavy development.

... And under getting started, it says:

>>>> Note also that btrfs is still considered experimental. While many
>>>> people use it reliably, there are still problems being found.

... and (offset in red even)...

>>>> You should keep and test backups of your data, and be prepared to
>>>> use them.

So no doubt about it or secret there; people should have TESTED backups 
they are PREPARED TO USE before they stick anything on btrfs in the first 
place.

Failure to do that simply demonstrates in action if not in word that the 
data isn't considered valuable enough to bother reading and following the 
warnings about keeping tested backups they're prepared to use, while 
placing it on a filesystem known to be under heavy development and not 
fully stable yet, that being the reason for such warnings in the first 
place!

And the "but I didn't know" excuse isn't valid either, for the same 
reason.  If they care about their data, they have backups even on stable 
filesystems, and if they care /enough/, they've made it their business to 
know the stability of the entire system, including the filesystem, that 
data is on, too.

Anything else... simply demonstrates by their actions that they didn't 
care about their data as much as they might have said they did after 
all.  And when people play the odds on the safety of their data, instead 
of having redundant backups to a level of safety matching the level of 
value they claim to place on that data,  sometimes they lose.  It's as 
simple as that.

(Not that I'm always perfect about keeping current backups either, but if 
I lost the working copy and didn't have a current backup, I'd know 
exactly who to point the finger at... myself!  Meanwhile, while it's not 
always current, for the data I value I do tend to have multiple layers of 
backup, to the degree that if I lose them all, it'll mean something 
drastic enough has happened that I'll have more important things to worry 
about than recovering my data for awhile... like surviving whatever 
disaster took all those levels of backup away at once!  Other than that, 
if I lose the working copy and perhaps the first level of backup, yeah 
I'll be mad at myself for not having kept up with the backups, but oh, 
well, I'll pick up with the backups I have and life will go on.  I've 
done it before and if it comes to it I'll do it again!)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Unmountable Array After Drive Failure During Device Deletion

[Chris Samuel]

[-- Attachment #1: Type: text/plain, Size: 1159 bytes --]

On Sun, 22 Dec 2013 11:35:53 AM Duncan wrote:

> While the btrfs kernel config option no longer (as of 3.12 IIRC) directly 
> calls the filesystem unstable

It'll be in 3.13, the commit was:

$ git describe 4204617d142c0887e45fda2562cb5c58097b918e
v3.12-116-g4204617

The help text for btrfs now says:

          Btrfs is a general purpose copy-on-write filesystem with extents,
          writable snapshotting, support for multiple devices and many more
          features focused on fault tolerance, repair and easy administration.

          The filesystem disk format is no longer unstable, and it's not
          expected to change unless there are strong reasons to do so. If there
          is a format change, file systems with a unchanged format will
          continue to be mountable and usable by newer kernels.

          For more information, please see the web pages at
          http://btrfs.wiki.kernel.org.

cheers,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

This email may come with a PGP signature as a file. Do not panic.
For more info see: http://en.wikipedia.org/wiki/OpenPGP

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 482 bytes --]