Re: RAID 6 corruption : please help!

[Bill Davidsen <davidsen@tmr.com>]

Trevor Cordes wrote:

>>What was the bug, and is it maybe something that is reversible.. ?
>>    
>>
>
>That's what I had thought until I knew the details of the bug.
>
>Mr. Anvin says:
>
>"No, it's "random"."
>
>"The error was: when a write happened to a stripe that needs
>read-modify-write, it wouldn't properly schedule the reads, and would
>blindly write out whatever crap happened to be in the stripe cache."
>
>">Do you know where in the code the bug was?  If I can only discover
>  
>
>>exactly what it did I could write a program to try to clean it up?"
>>    
>>
>
>"No, it's timing-dependent and, in either case, involve writing non-data
>to the disks."
>
>On  6 Oct, Molle Bestefich wrote:
>  
>
>>What's stopping you from just pulling out the two new disks, mounting
>>the array using the old, almost OK disks, and fsck'ing your way out of
>>the couple of files that were corrupted when you were in rw mode?
>>    
>>
>
>That's kind of what I thought, but I had written to the disks and for
>each write lots of the entire stripe (in many cases) would get wiped out
>with random data.
>
>In the end, I ran fsck -y on it and crossed my fingers.  That recovered
>nearly 8/10ths of the data before it hit some fsck bug (dies on signal
>11).  The rest of the data I had 1 month old backups, so it actually
>turned out pretty good.  I'm certainly going to increase my backup
>frequency to weekly or twice weekly from now on -- even on a RAID6 setup
>that I was *really* trusting to protect my 2TB.
>

You want to google for "signal 11" it (usually?) means there are 
hardware problems with the system, frequently memory or bus occasional 
failures. Unfortunately you may not be out of the woods yet.

On the other hand, you did a lot of inadvisable things to get to this 
point...

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979