Raid-6 Rebuild question

Raid-6 Rebuild question

[Brad Campbell]

G'day all,

Here is an interesting question( well I think so in any case ). I just replaced a failed disk in my 
15 drive Raid-6.

Simply mdadm --add /dev/md0 /dev/sdl

Why, when there is no other activity on the array at all, is it writing to every disk during the 
recovery? I would have assumed it just read from the others and write to sdl.

This is an iostat -k 5 on that machine while rebuilding

avg-cpu:  %user   %nice    %sys %iowait   %idle
            0.00    0.00  100.00    0.00    0.00

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             121.08     14187.95       925.30      23552       1536
sdb             127.71     14187.95      1002.41      23552       1664
sdc             125.30     14187.95      1002.41      23552       1664
sdd             122.29     14187.95      1002.41      23552       1664
sde             125.30     14187.95      1002.41      23552       1664
sdf             127.71     14187.95      1002.41      23552       1664
sdg             125.90     14187.95       925.30      23552       1536
sdh             125.30     14187.95       925.30      23552       1536
sdi             134.34     14187.95       925.30      23552       1536
sdj             137.95     14187.95       925.30      23552       1536
sdk             140.36     14187.95      1850.60      23552       3072
sdl              79.52         0.00     14265.06          0      23680
sdm             133.13     14187.95       925.30      23552       1536
sdn             134.34     14187.95       925.30      23552       1536
sdo             133.73     14187.95       925.30      23552       1536
md0               0.00         0.00         0.00          0          0

storage1:/home/brad# cat /proc/mdstat
Personalities : [raid6]
md0 : active raid6 sdl[15] sdg[6] sda[0] sdo[14] sdn[13] sdm[12] sdk[10] sdj[9] sdi[8] sdh[7] sdf[5] 
sde[4] sdd[3] sdc[2] sdb[1]
       3186525056 blocks level 6, 128k chunk, algorithm 2 [15/14] [UUUUUUUUUUU_UUU]
       [>....................]  recovery =  1.8% (4518144/245117312) finish=838.3min speed=4782K/sec
unused devices: <none>

Regards,
Brad
-- 
"Human beings, who are almost unique in having the ability
to learn from the experience of others, are also remarkable
for their apparent disinclination to do so." -- Douglas Adams

Re: Raid-6 Rebuild question

[Brad Campbell]

Brad Campbell wrote:
> G'day all,
> 
> Here is an interesting question( well I think so in any case ). I just 
> replaced a failed disk in my 15 drive Raid-6.

Forgot the most important detail (as usual)

bklaptop:~>ssh storage1 uname -a
Linux storage1 2.6.11.7 #4 Fri Oct 7 20:00:25 GST 2005 i686 GNU/Linux

Regards,
Brad
-- 
"Human beings, who are almost unique in having the ability
to learn from the experience of others, are also remarkable
for their apparent disinclination to do so." -- Douglas Adams

Re: Raid-6 Rebuild question

[Neil Brown]

On Sunday November 13, brad@wasp.net.au wrote:
> G'day all,
> 
> Here is an interesting question( well I think so in any case ). I just replaced a failed disk in my 
> 15 drive Raid-6.
> 
> Simply mdadm --add /dev/md0 /dev/sdl
> 
> Why, when there is no other activity on the array at all, is it writing to every disk during the 
> recovery? I would have assumed it just read from the others and
> write to sdl.

The raid6 recovery code always writes out the P and Q blocks for every
stripe.  This is un-necessary and there is in fact a comment in the
code saying:
	/**** FIX: Should we really do both of these unconditionally? ****/

I recently reviewed and cleaned up this code, though I haven't tested
the new version yet.  I'll make sure the new code doesn't do
un-necessary writes (it may already not).  So there is a good chance
that 2.6.16 will do a better job here.  

Thanks for the report,
NeilBrown

RAID-6

[Bill Davidsen]

Based on some google searching on RAID-6, I find that it seems to be 
used to describe two different things. One is very similar to RAID-5, 
but with two redundancy blocks per stripe, one XOR and one CRC (or at 
any rate two methods are employed). The other sources define RAID-6 as 
RAID-5 with a distributed hot spare, AKA RAID-5E, which spreads head 
motion to all drives for performance.

Any clarification on this?

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: RAID-6

[Dan Stromberg]

My understanding is that RAID 5 -always- stripes parity.  If it didn't,
I believe it would be RAID 4.

You may find http://linux.cudeso.be/raid.php of interest.

I don't think RAID level 6 was in the original RAID paper, so vendors
may have decided on their own that it should mean what they're
selling.  :)

On Wed, 2005-11-16 at 12:54 -0500, Bill Davidsen wrote:
> Based on some google searching on RAID-6, I find that it seems to be 
> used to describe two different things. One is very similar to RAID-5, 
> but with two redundancy blocks per stripe, one XOR and one CRC (or at 
> any rate two methods are employed). The other sources define RAID-6 as 
> RAID-5 with a distributed hot spare, AKA RAID-5E, which spreads head 
> motion to all drives for performance.
> 
> Any clarification on this?
> 

Re: RAID-6

[H. Peter Anvin]

Followup to:  <1132173592.23464.459.camel@seki.nac.uci.edu>
By author:    Dan Stromberg <strombrg@dcs.nac.uci.edu>
In newsgroup: linux.dev.raid
>
> 
> My understanding is that RAID 5 -always- stripes parity.  If it didn't,
> I believe it would be RAID 4.
> 
> You may find http://linux.cudeso.be/raid.php of interest.
> 
> I don't think RAID level 6 was in the original RAID paper, so vendors
> may have decided on their own that it should mean what they're
> selling.  :)
> 

RAID-6 wasn't in the original RAID paper, but the term RAID-6 with the
P+Q parity defintion is by far the dominant use of the term, and I
believe it is/was recognized by the RAID Advisory Board, which is as
close as you can get to an official statement.  The RAB seems to have
gotten defunct, with a standard squatter page on their previous web
address.

	-hpa