Re: md-raid paranoia mode?

[Brad Campbell <lists2009@fnarfbargle.com>]

On 11/06/14 14:48, Bart Kus wrote:
> Hello,
>
> As far as I understand, md-raid relies on the underlying devices to
> inform it of IO errors before it'll seek redundant/parity data to
> fulfill the read request.  I have, however, seen certain hard drives
> report successful reads while returning garbage data.

If you have drives that return garbage as valid data then you have far 
greater problems than what you are suggesting will fix. So much so I 
suggest you document these instances and start banging a drum announcing 
them in a name and shame campaign. That sort of behavior from storage 
devices is never ok, and the manufacturer needs to know that.

This comes up on the list at least once a year, and the upshot is that 
your storage platform needs to be reliable. Storage is *supposed* to be 
reliable. Even the cheapest solution is *supposed* to say "I'm sorry but 
that bit of data you asked for is toast". Even my 35c USB drives do that.

Whether you have a single drive or 10 mirrors, if you have a drive 
returning garbage you need to solve that problem first. Patching 
software that is based on the fundamental assumption that the storage 
stack knows when something is bad, to no longer trust that assumption 
makes all sorts of guarantees go out the window.

 From personal experience, I lost a 12TB RAID-6 and all the data on it 
due to a bad SATA controller. The controller would return corrupt reads 
under heavy load, and months of read/modify/write cycles combined with 
corrupt data spread the corruption all over the array. My immediate 
reaction was the same as yours. "RAID6 should be able to protect against 
this stuff", but after education from people that are more knowledgeable 
than I, it became apparent that bad hardware is JUST insidious and 
papering over one part of the stack would just lead to it biting me 
elsewhere anyway.

I learned 2 very valuable lessons.
- Don't deploy hardware unless you trust it. This may mean a month of 
burn-in testing in a spare machine, or delaying trusting it with 
valuable data. In my case it was a cheap 2 port PCIe SATA card procured 
to get me out of a tight spot, so I plugged it in and strapped drives to 
it blindly believing it would be ok.
- RAID is no substitute for backups.