Re: raid 5 mismatch_cnt errors

[Trey Scarborough <treys@locallinux.com>]

Neil Brown wrote:
> On Thu, 20 May 2010 12:02:23 -0500
> Trey Scarborough <treys@locallinux.com> wrote:
>
>   
>> I have a raid 5 array with 9 disks and I have a mismatch_cnt that keeps 
>> growing. This is causing file corruption on the underlaying file systems 
>> as well.  I can copy a group of 100 100mb files and then do a md5sum on 
>> them and 1-3 will be corrupt. If this is a drive that is bad is there 
>> anyway to run a report on the count per drive that these mismatches 
>> occur. I have run smarttools test and do not see one drive that stands 
>> out to be causing errors. Could something else be causing these errors?
>>     
>
>
> When RAID5 detects an inconsistency there is no way to know which device was
> wrong.
> SMART only detects some errors, not all.
> I have had hard drives before which appears to have a single-bit error in
> their internal buffer.  No error would be reported, but data you read would
> sometimes be wrong.
> RAID5 cannot help you with this sort of error.
>
> I would suggest backing up all your data (if it isn't already to late),
> breaking the array, and testing each device individually.
> e.g. create a filesystem on the device and try copying data on and reading it
> off.
>
> NeilBrown
>   
Thats what I was afraid of. The problem I have is if I back it up 
knowing what data is bad. Luckily it appears to be a write error because 
once written and correct I can do sums on all the files and I do not see 
anymore errors. I was thinking that there might be a way of do a resync 
and turning up the debug somehow so that it would log the mismatches 
with both the drives that it was reading from at the time. I could then 
take that information and considering there are 9 drives in the array 
the one that comes out having the most should be the culprit. I could 
then remove that drive from the array and test it leaving the rest in a 
state that could be rebuilt and the data being consistant because the 
drive with the bad write errors would be removed. Is this something that 
might be possible?

Thanks,
Trey