integrity verification on raid-5?

integrity verification on raid-5?

[Patrick H.]

Is there a way to do integrity verification on a raid-5 array? I'm 
working on building a storage system on SSDs under raid-5 and want to be 
able to perform periodic integrity checks. Basically just check the 
drives to make sure that they match what the parity drive has.
After a bit of googling I saw other people wanting the same thing but 
nobody with any result. I dont see why this cant be done, but is there 
any tool to do so?

Re: integrity verification on raid-5?

[Neil Brown]

On Fri, 17 Dec 2010 14:08:16 -0700 "Patrick H." <linux-raid@feystorm.net>
wrote:

> Is there a way to do integrity verification on a raid-5 array? I'm 
> working on building a storage system on SSDs under raid-5 and want to be 
> able to perform periodic integrity checks. Basically just check the 
> drives to make sure that they match what the parity drive has.
> After a bit of googling I saw other people wanting the same thing but 
> nobody with any result. I dont see why this cant be done, but is there 
> any tool to do so?
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

% man 4 md

search for 'check'.

NeilBrown

Re: integrity verification on raid-5?

[Patrick H.]

Perfect, thank you very much.

Hopefully google indexes the mailing list so this answer will be out 
there now :-)

-Patrick

Sent: Fri Dec 17 2010 14:17:45 GMT-0700 (Mountain Standard Time)
From: Neil Brown <neilb@suse.de>
To: Patrick H. <linux-raid@feystorm.net> linux-raid@vger.kernel.org
Subject: Re: integrity verification on raid-5?
> On Fri, 17 Dec 2010 14:08:16 -0700 "Patrick H." <linux-raid@feystorm.net>
> wrote:
>
>   
>> Is there a way to do integrity verification on a raid-5 array? I'm 
>> working on building a storage system on SSDs under raid-5 and want to be 
>> able to perform periodic integrity checks. Basically just check the 
>> drives to make sure that they match what the parity drive has.
>> After a bit of googling I saw other people wanting the same thing but 
>> nobody with any result. I dont see why this cant be done, but is there 
>> any tool to do so?
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>     
>
> % man 4 md
>
> search for 'check'.
>
> NeilBrown
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>   

Re: integrity verification on raid-5?

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]

On Fri Dec 17, 2010 at 02:08:16PM -0700, Patrick H. wrote:

> Is there a way to do integrity verification on a raid-5 array? I'm 
> working on building a storage system on SSDs under raid-5 and want to be 
> able to perform periodic integrity checks. Basically just check the 
> drives to make sure that they match what the parity drive has.
> After a bit of googling I saw other people wanting the same thing but 
> nobody with any result. I dont see why this cant be done, but is there 
> any tool to do so?

There's built-in functionality to do this.  To start the check, run:
    echo check > /sys/block/mdX/md/sync_action

You can check progress by catting /proc/mdstat, and the number of errors
is reported at the end in /sys/block/mdX/md/mismatch_cnt.  To rewrite
the parity data for any mismatches, use "repair" instead of "check" in
the first command.

Currently, there's no easy way to find out what file(s) are affected by
the mismatches though.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: integrity verification on raid-5?

[Patrick H.]

Sent: Fri Dec 17 2010 14:24:37 GMT-0700 (Mountain Standard Time)
From: Robin Hill <robin@robinhill.me.uk>
To: linux-raid@vger.kernel.org
Subject: Re: integrity verification on raid-5?
> On Fri Dec 17, 2010 at 02:08:16PM -0700, Patrick H. wrote:
>
>   
>> Is there a way to do integrity verification on a raid-5 array? I'm 
>> working on building a storage system on SSDs under raid-5 and want to be 
>> able to perform periodic integrity checks. Basically just check the 
>> drives to make sure that they match what the parity drive has.
>> After a bit of googling I saw other people wanting the same thing but 
>> nobody with any result. I dont see why this cant be done, but is there 
>> any tool to do so?
>>     
>
> There's built-in functionality to do this.  To start the check, run:
>     echo check > /sys/block/mdX/md/sync_action
>
> You can check progress by catting /proc/mdstat, and the number of errors
> is reported at the end in /sys/block/mdX/md/mismatch_cnt.  To rewrite
> the parity data for any mismatches, use "repair" instead of "check" in
> the first command.
>
> Currently, there's no easy way to find out what file(s) are affected by
> the mismatches though.
>
>   
The docs say that for both raid 5 & 6 it the repair function simply 
rewrites the parity drive(s). For raid-5 I can understand this as 
there's no way to tell if the data is incorrect, or if the parity is 
incorrect since there's only 1 parity. And while I dont know the details 
of the algorithms involved in raid-6, couldnt you do something like:
Calculate replacement data for both parity drives
If one of the 2 parity drives doesnt match its replacement data
    assume that drive is bad
Else if both parity drives dont match their replacement data
    one of the data drives must be bad
    calculate replacement data for each data drive and find the one that 
doesnt match
    If more than 1 data drive doesnt match its replacement data
       we have multiple-drive failure (could be any combination of 
parity & data drives) and cant determine which ones
Else
    the world is ok

Its probably a heck of a lot more computationally expensive, but it can 
isolate which drive is the bad one. But again, I'm not knowledgeable on 
the the internal details of raid-6 and might just be completely off my 
rocker.

-Patrick

> Cheers,
>     Robin
>   

Re: integrity verification on raid-5?

[Piergiorgio Sartor]

Hi,

> The docs say that for both raid 5 & 6 it the repair function simply
> rewrites the parity drive(s). For raid-5 I can understand this as
> there's no way to tell if the data is incorrect, or if the parity is
> incorrect since there's only 1 parity. And while I dont know the
> details of the algorithms involved in raid-6, couldnt you do
> something like:
> Calculate replacement data for both parity drives
> If one of the 2 parity drives doesnt match its replacement data
>    assume that drive is bad
> Else if both parity drives dont match their replacement data
>    one of the data drives must be bad
>    calculate replacement data for each data drive and find the one
> that doesnt match
>    If more than 1 data drive doesnt match its replacement data
>       we have multiple-drive failure (could be any combination of
> parity & data drives) and cant determine which ones
> Else
>    the world is ok
> 
> Its probably a heck of a lot more computationally expensive, but it
> can isolate which drive is the bad one. But again, I'm not
> knowledgeable on the the internal details of raid-6 and might just
> be completely off my rocker.
> 

Welcome to the club!

It seems this topic pops up more or less
each 6 months...

Unfortunately, it also seems that the
core developers do not have high priority
on this one.

BTW, did anybody else look into this?

Is there any possiblities to perform this
kind of check in user space?

bye,

-- 

piergiorgio

Re: integrity verification on raid-5?

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 448 bytes --]

On Fri, 17 Dec 2010 21:24:37 +0000
Robin Hill <robin@robinhill.me.uk> wrote:
 
> Currently, there's no easy way to find out what file(s) are affected by
> the mismatches though.

That's why it is also a good idea to keep around file-level checksums in each
directory (especially if you store a lot of files which shouldn't ever change,
like audio or video collections), using CFV - http://cfv.sourceforge.net/

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]