Re: Expected behavior of bad sectors on one drive in a RAID1

[Austin S Hemmelgarn <ahferroin7@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3331 bytes --]

On 2015-10-20 00:45, Russell Coker wrote:
> On Tue, 20 Oct 2015 03:16:15 PM james harvey wrote:
>> sda appears to be going bad, with my low threshold of "going bad", and
>> will be replaced ASAP.  It just developed 16 reallocated sectors, and
>> has 40 current pending sectors.
>>
>> I'm currently running a "btrfs scrub start -B -d -r /terra", which
>> status on another term shows me has found 32 errors after running for
>> an hour.
>
> https://www.gnu.org/software/ddrescue/
>
> At this stage I would use ddrescue or something similar to copy data from the
> failing disk to a fresh disk, then do a BTRFS scrub to regenerate the missing
> data.
>
> I wouldn't remove the disk entirely because then you lose badly if you get
> another failure.  I wouldn't use a BTRFS replace because you already have the
> system apart and I expect ddrescue could copy the data faster.  Also as the
> drive has been causing system failures (I'm guessing a problem with the power
> connector) you REALLY don't want BTRFS to corrupt data on the other disks.  If
> you have a system with the failing disk and a new disk attached then there's
> no risk of further contamination.
BIG DISCLAIMER: For the filesystem to be safely mountable it is 
ABSOLUTELY NECESSARY to remove the old disk after doing a block level 
copy of it.  By all means, keep the disk around, but do not keep it 
visible to the kernel after doing a block level copy of it.  Also, you 
will probably have to run 'btrfs device scan' after copying the disk and 
removing it for the filesystem to work right.  This is an inherent 
result of how BTRFS's multi-device functionality works, and also applies 
to doing stuff like LVM snapshots of BTRFS filesystems.
>
>> Question 2 - Before having ran the scrub, booting off the raid with
>> bad sectors, would btrfs "on the fly" recognize it was getting bad
>> sector data with the checksum being off, and checking the other
>> drives?  Or, is it expected that I could get a bad sector read in a
>> critical piece of operating system and/or kernel, which could be
>> causing my lockup issues?
>
> Unless you have disabled CoW then BTRFS will not return bad data.
It is worth clarifying also that:
a. While BTRFS will not return bad data in this case, it also won't 
automatically repair the corruption.
b. In the unlikely event that both copies are bad, trying to read the 
data will return an IO error.
c. It is theoretically possible (although statistically impossible) that 
the block could become corrupted, but the checksum could still be 
correct (CRC32c is good at detecting small errors, but it's not hard to 
generate a hash collision for any arbitrary value, so if a large portion 
of the block goes bad, then it can theoretically still have a valid 
checksum).
>
>> Question 3 - Probably doesn't matter, but how can I see which files
>> (or metadata to files) the 40 current bad sectors are in?  (On extX,
>> I'd use tune2fs and debugfs to be able to see this information.)
>
> Read all the files in the system and syslog will report it.  But really don't
> do that until after you have copied the disk.
It may also be possible to use some of the debug tools from BTRFS to do 
this without hitting the disks so hard, but it will likely take a lot 
more effort.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]