Data integrity and RAID

Data integrity and RAID

[peter.stevens]

I'm looking for a software solution to help replace my existing raid hardware
setup. The hardware mimics RAID-1 (in that it has 3 mirrors), except for the
fact that on read all 3 mirrors are compared and possibly error corrected
before data is returned. I don't neccessarily need a RAID-1 solution, it
just seems closer to what I already have and also that it also can recover
from 2 simulateous disk failures. 

So far I've played with a software RAID-1 array of USB flash drives. My
issue is that software RAID-1 does not check for or recover from data
corruption unless a read or write to disk actually fails. Integrity is a
major concern for me, I need to know that all data going to and from disk is
correct at all times. 

All advice and comments are welcomed.
-- 
View this message in context: http://www.nabble.com/Data-integrity-and-RAID-tp20915150p20915150.html
Sent from the linux-admin mailing list archive at Nabble.com.

Re: Data integrity and RAID

[Billy Crook]

At the moment, I would recommend using two drives in a raid1, and
rsync to the third.  Then use some scripts with sha1sum to compare
between the raid and the rsync'd copy, which would effectively be a
backup copy.  You could use a fuse filesystem called chironfs
http://www.furquim.org/chironfs/ to 'mirror' at the file level, and
sha1 or diff to compare all the disks at some regular interval.
chiron reads from the primary disk by default, and writes out to all
disks.  I don't know of anything that reads all, compares, tiebreaks,
and repairs on each file access.  I suspect that would perform quite
badly.

On Tue, Dec 9, 2008 at 07:55, peter.stevens <Peter.Stevens@gtech.com> wrote:
>
> I'm looking for a software solution to help replace my existing raid hardware
> setup. The hardware mimics RAID-1 (in that it has 3 mirrors), except for the
> fact that on read all 3 mirrors are compared and possibly error corrected
> before data is returned. I don't neccessarily need a RAID-1 solution, it
> just seems closer to what I already have and also that it also can recover
> from 2 simulateous disk failures.
>
> So far I've played with a software RAID-1 array of USB flash drives. My
> issue is that software RAID-1 does not check for or recover from data
> corruption unless a read or write to disk actually fails. Integrity is a
> major concern for me, I need to know that all data going to and from disk is
> correct at all times.
>
> All advice and comments are welcomed.

Data integrity and RAID

[peter.stevens]

I'm looking for a software solution to help replace my existing raid hardware
setup. The hardware mimics RAID-1 (in that it has 3 mirrors), except for the
fact that on read all 3 mirrors are compared and possibly error corrected
before data is returned. I don't neccessarily need a RAID-1 solution, it
just seems closer to what I already have and also that it also can recover
from 2 simulateous disk failures. 

So far I've played with a software RAID-1 array of USB flash drives. My
issue is that software RAID-1 does not check for or recover from data
corruption unless a read or write to disk actually fails. Integrity is a
major concern for me, I need to know that all data going to and from disk is
correct at all times.

All advice and comments are welcomed.
-- 
View this message in context: http://www.nabble.com/Data-integrity-and-RAID-tp20841008p20841008.html
Sent from the linux-raid mailing list archive at Nabble.com.

Re: Data integrity and RAID

[Justin Piszcz]

On Tue, 9 Dec 2008, peter.stevens wrote:

>
> I'm looking for a software solution to help replace my existing raid hardware
> setup. The hardware mimics RAID-1 (in that it has 3 mirrors), except for the
> fact that on read all 3 mirrors are compared and possibly error corrected
> before data is returned. I don't neccessarily need a RAID-1 solution, it
> just seems closer to what I already have and also that it also can recover
> from 2 simulateous disk failures.
>
> So far I've played with a software RAID-1 array of USB flash drives. My
> issue is that software RAID-1 does not check for or recover from data
> corruption unless a read or write to disk actually fails. Integrity is a
> major concern for me, I need to know that all data going to and from disk is
> correct at all times.
>
> All advice and comments are welcomed.
Well USB flash isn't a traditional disk that remaps bad sectors etc.. 
AFAIK md/raid will try and correct read-errors, but for write errors you 
have to trust the disks to do this for you, otherwise the bad disk will be 
kicked out of the array.  If you need the bad write to be re-mapped 
elsewhere in the array you need a hw raid controller, 3ware, etc..

Justin.

Re: Data integrity and RAID

[Justin Piszcz]

On Tue, 9 Dec 2008, Ryan Wagoner wrote:

> The issue I've had with RAID 1 using mdadm is the disk's aren't
> consistent. I've noticed that unlike RAID 5 when a write is aborted
> mdadm doesn't completely write the data.
>
> When I run a check on the RAID 1 I always have inconsistencies where I
> have never had one on a RAID 5. The issue here is you never know if
> the inconsistencies can be ignored or are signs of a failing disk,
> etc.
>
> I sent an email to the list previously on this with no response.
>
> http://marc.info/?l=linux-raid&m=122572326107399&w=2
I have asked about this before as well, do you have your swap on raid1? I 
recall someone replying that it was normal if that is the case.

Justin.

Re: Data integrity and RAID

[Justin Piszcz]

What kind of disks? Does it go away after a 'repair'?
Show smartctl -a of both disks in the raid?

On Tue, 9 Dec 2008, Ryan Wagoner wrote:

> My swap is on a different RAID 1, but my other RAID 1 with the root
> partition was showing inconsistencies as well. It does it on both my
> servers.
>
> Ryan
>
> On Tue, Dec 9, 2008 at 11:28 AM, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>>
>>
>> On Tue, 9 Dec 2008, Ryan Wagoner wrote:
>>
>>> The issue I've had with RAID 1 using mdadm is the disk's aren't
>>> consistent. I've noticed that unlike RAID 5 when a write is aborted
>>> mdadm doesn't completely write the data.
>>>
>>> When I run a check on the RAID 1 I always have inconsistencies where I
>>> have never had one on a RAID 5. The issue here is you never know if
>>> the inconsistencies can be ignored or are signs of a failing disk,
>>> etc.
>>>
>>> I sent an email to the list previously on this with no response.
>>>
>>> http://marc.info/?l=linux-raid&m=122572326107399&w=2
>>
>> I have asked about this before as well, do you have your swap on raid1? I
>> recall someone replying that it was normal if that is the case.
>>
>> Justin.
>>
>>
>

Re: Data integrity and RAID

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 2045 bytes --]

On Tue Dec 09, 2008 at 06:02:57AM -0800, peter.stevens wrote:

> 
> I'm looking for a software solution to help replace my existing raid hardware
> setup. The hardware mimics RAID-1 (in that it has 3 mirrors), except for the
> fact that on read all 3 mirrors are compared and possibly error corrected
> before data is returned. I don't neccessarily need a RAID-1 solution, it
> just seems closer to what I already have and also that it also can recover
> from 2 simulateous disk failures. 
> 
> So far I've played with a software RAID-1 array of USB flash drives. My
> issue is that software RAID-1 does not check for or recover from data
> corruption unless a read or write to disk actually fails. Integrity is a
> major concern for me, I need to know that all data going to and from disk is
> correct at all times.
> 
> All advice and comments are welcomed.
> 
I'm not aware of anything, no.  Software RAID-1 will do some of what
you're after but doesn't verify reads against all drives (this doesn't
guarantee integrity anyway, only reduce the chance of an error).  You
can run regular checks of the drives to detect (and repair) any
data mismatches though.

Mind you, to do this properly, you'd need to disable write caching on
all drives; do a read of all drives after every write (to verify that
the write was successful); and compare all drives on every read (to
verify that the read was successful).  You probably also want checksums
on the drives to help detect which drive(s) are incorrect on read
mismatches as well (I believe there's now a standard for doing this, but
I don't think it's supported end-to-end yet). Doing all this will really
limit performance though - generally you'll be better off building error
recovery into the file format instead.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Data integrity and RAID

[Justin Piszcz]

On Tue, 9 Dec 2008, Robin Hill wrote:

> On Tue Dec 09, 2008 at 06:02:57AM -0800, peter.stevens wrote:
>
>>
>> I'm looking for a software solution to help replace my existing raid hardware
>> setup. The hardware mimics RAID-1 (in that it has 3 mirrors), except for the
>> fact that on read all 3 mirrors are compared and possibly error corrected
>> before data is returned. I don't neccessarily need a RAID-1 solution, it
>> just seems closer to what I already have and also that it also can recover
>> from 2 simulateous disk failures.
>>
>> So far I've played with a software RAID-1 array of USB flash drives. My
>> issue is that software RAID-1 does not check for or recover from data
>> corruption unless a read or write to disk actually fails. Integrity is a
>> major concern for me, I need to know that all data going to and from disk is
>> correct at all times.
>>
>> All advice and comments are welcomed.
>>
> I'm not aware of anything, no.  Software RAID-1 will do some of what
> you're after but doesn't verify reads against all drives (this doesn't
> guarantee integrity anyway, only reduce the chance of an error).  You
> can run regular checks of the drives to detect (and repair) any
> data mismatches though.
>
> Mind you, to do this properly, you'd need to disable write caching on
> all drives; do a read of all drives after every write (to verify that
> the write was successful); and compare all drives on every read (to
> verify that the read was successful).  You probably also want checksums
> on the drives to help detect which drive(s) are incorrect on read
> mismatches as well (I believe there's now a standard for doing this, but
> I don't think it's supported end-to-end yet). Doing all this will really
> limit performance though - generally you'll be better off building error
> recovery into the file format instead.
Sounds like he should use ZFS.

RE: Data integrity and RAID

[David Lethe]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of peter.stevens
> Sent: Tuesday, December 09, 2008 8:03 AM
> To: linux-raid@vger.kernel.org
> Subject: Data integrity and RAID
> 
> 
> I'm looking for a software solution to help replace my existing raid
> hardware
> setup. The hardware mimics RAID-1 (in that it has 3 mirrors), except
> for the
> fact that on read all 3 mirrors are compared and possibly error
> corrected
> before data is returned. I don't neccessarily need a RAID-1 solution,
> it
> just seems closer to what I already have and also that it also can
> recover
> from 2 simulateous disk failures.
> 
> So far I've played with a software RAID-1 array of USB flash drives.
My
> issue is that software RAID-1 does not check for or recover from data
> corruption unless a read or write to disk actually fails. Integrity is
> a
> major concern for me, I need to know that all data going to and from
> disk is
> correct at all times.
> 
> All advice and comments are welcomed.
> --
> View this message in context:
http://www.nabble.com/Data-integrity-and-
> RAID-tp20841008p20841008.html
> Sent from the linux-raid mailing list archive at Nabble.com.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

A few things come to mind ...
1) Use Solaris w/ZFS instead of LINUX.   You can even put on target-mode
drivers and make this a SAN appliance or NAS box. Obviously this won't
work for you if you have application software you want to run on the
same machine, but just throwing it out.  ZFS supports n-Way mirroring.
2) Use disks that support BGMS.  This means late-generation Seagate SAS
disks (or SCSI or FC, just not SATA).  Then you can get software
(shameless plug here, but we have such a product for $90.) that runs
background media scans and repairs bad blocks 24x7 with near zero host
overhead.  Blocks that can't auto-repair will be reported, so you can
kick off a consistency repair once they are detected.  Many of the SATA
disks do have error detection/correction firmware, but their ability to
report specifics of problem and auto repair are extremely limited due to
fewer ECC bits and less sophisticated algorithms in the firmware).

Regards
David
david@santools.com