RAID5E

RAID5E

[Bill Davidsen]

Where I was working most recently some systems were using RAID5E (RAID5 
with both the parity and hot spare distributed). This seems to be highly 
desirable for small arrays, where spreading head motion over one more 
drive will improve performance, and in all cases where a rebuild to the 
hot spare will avoid a bottleneck on a single drive.

Is there any plan to add this capability?

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: RAID5E

[Mattias Wadenstein]

On Wed, 31 May 2006, Bill Davidsen wrote:

> Where I was working most recently some systems were using RAID5E (RAID5 with 
> both the parity and hot spare distributed). This seems to be highly desirable 
> for small arrays, where spreading head motion over one more drive will 
> improve performance, and in all cases where a rebuild to the hot spare will 
> avoid a bottleneck on a single drive.
>
> Is there any plan to add this capability?

What advantage does that have over raid6? You use exactly as many drives 
(n+2), with the disadvantage of having to do a rebuild without parity when 
a drive fails and a raid failure at a double disk failure.

/Mattias Wadenstein

Re: RAID5E

[Erik Mouw]

On Wed, May 31, 2006 at 04:27:54PM +0200, Mattias Wadenstein wrote:
> On Wed, 31 May 2006, Bill Davidsen wrote:
> >Where I was working most recently some systems were using RAID5E (RAID5 
> >with both the parity and hot spare distributed). This seems to be highly 
> >desirable for small arrays, where spreading head motion over one more 
> >drive will improve performance, and in all cases where a rebuild to the 
> >hot spare will avoid a bottleneck on a single drive.
> >
> >Is there any plan to add this capability?
> 
> What advantage does that have over raid6? You use exactly as many drives 
> (n+2), with the disadvantage of having to do a rebuild without parity when 
> a drive fails and a raid failure at a double disk failure.

Advantage:
- Easier to calculate the checksum (RAID5 XOR instead of a generator
  polynome with RAID6)
- Higher throughput compared to standard RAID5
- Actually uses the hot spare

Disadvantage:
- Doesn't protect against double disk failures, but RAID5 also doesn't

Note that you could also do RAID6E.


Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands

Re: RAID5E

[Bill Davidsen]

Mattias Wadenstein wrote:

> On Wed, 31 May 2006, Bill Davidsen wrote:
>
>> Where I was working most recently some systems were using RAID5E 
>> (RAID5 with both the parity and hot spare distributed). This seems to 
>> be highly desirable for small arrays, where spreading head motion 
>> over one more drive will improve performance, and in all cases where 
>> a rebuild to the hot spare will avoid a bottleneck on a single drive.
>>
>> Is there any plan to add this capability?
>
>
> What advantage does that have over raid6? You use exactly as many 
> drives (n+2), with the disadvantage of having to do a rebuild without 
> parity when a drive fails and a raid failure at a double disk failure. 

The write overhead of RAID-6 vs. RAID-5 is much higher, both in terms of 
CPU and disk operations generated. See below re double drive failure and 
RAID-5 with spare.

As a starting point consider RAID-4, and four drives, one of which is a 
hot spare (I did clearly note small arrays). For reads you only have 
striping over two drives, for writes the parity drive has double the i/o 
of the data drives. The spare may spin down or not, but if it fails it 
will be just when you need it most.

With RAID-5 the load is evenly spread over the entire array of active 
drives. This improves both read and write performance, but the hot spare 
is still not used in normal operation. Note that in the two drive 
failure mode the only critical time is during the rebuild to hot spare, 
after that a second drive failure is tolerated without loss of data.

With RAID-5E the i/o is spread over all four drives, which gives a 
further improvement in the read and write performance, particularly if 
large reads and writes (relative to clunk size) are common. There is no 
hot spare which is mostly unused, and rebuild to the hot spare is 
distributed over all remaining drives.

Drawbacks: you can't share a single hot spare between multiple arrays. 
That's not really common, and I mention it for completeness. The other 
issue is that when a failed drive is replaced the rebuild appears to be 
somewhat complex because the new drive doesn't just become the new hot 
spare. The same issues apply regarding two drives failing, but the 
rebuild time is usually shorter, so the exposure is less.

And finally, with RAID-6 and the same number of drives, if you have a 
failure there's no hot spare, and the arrray runs in degraded mode until 
the failed drive is replaced. That is a different balance between 
reliability and performance after failure, and needs to be a 
per-instance choice.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: RAID5E

[Neil Brown]

On Wednesday May 31, davidsen@tmr.com wrote:
> Where I was working most recently some systems were using RAID5E (RAID5 
> with both the parity and hot spare distributed). This seems to be highly 
> desirable for small arrays, where spreading head motion over one more 
> drive will improve performance, and in all cases where a rebuild to the 
> hot spare will avoid a bottleneck on a single drive.
> 
> Is there any plan to add this capability?

I thought about it briefly....

As I understand it, the layout of raid5e when non-degraded is very
similar to raid6 - however the 'Q' block is simply not used.
This would be trivial to implement.

The interesting bit comes when a device fails and you want to rebuild
that distributed spare.
There are two possible ways that you could do this:

1/ Leave the spare where it is and write the correct data into each
 spare.  This would be fairly easy but would leave an array with an
 very ... interesting layout of data.
 When you add a replacement you just move everything back.

2/ reshape the array to be a regular raid5 layout.  This would be hard
 to do well without NVRAM as you are moving live data, but would result
 in a neat and tidy array.  Ofcourse adding a drive back in would be
 interesting again...

I had previously only thought of option '2', and so discarded the idea
as not worth the effort.  The more I think about it, the more possible
option 1 sounds.
I've put it back on my todo list, but I don't expect to get to it this
year.  Ofcourse if someone else wants to give it a try, I'm happy to
make suggestions and review code.

NeilBrown

Re: RAID5E

[Bill Davidsen]

Neil Brown wrote:

>On Wednesday May 31, davidsen@tmr.com wrote:
>  
>
>>Where I was working most recently some systems were using RAID5E (RAID5 
>>with both the parity and hot spare distributed). This seems to be highly 
>>desirable for small arrays, where spreading head motion over one more 
>>drive will improve performance, and in all cases where a rebuild to the 
>>hot spare will avoid a bottleneck on a single drive.
>>
>>Is there any plan to add this capability?
>>    
>>
>
>I thought about it briefly....
>
>As I understand it, the layout of raid5e when non-degraded is very
>similar to raid6 - however the 'Q' block is simply not used.
>This would be trivial to implement.
>
>The interesting bit comes when a device fails and you want to rebuild
>that distributed spare.
>There are two possible ways that you could do this:
>
>1/ Leave the spare where it is and write the correct data into each
> spare.  This would be fairly easy but would leave an array with an
> very ... interesting layout of data.
> When you add a replacement you just move everything back.
>
>2/ reshape the array to be a regular raid5 layout.  This would be hard
> to do well without NVRAM as you are moving live data, but would result
> in a neat and tidy array.  Ofcourse adding a drive back in would be
> interesting again...
>
>I had previously only thought of option '2', and so discarded the idea
>as not worth the effort.  The more I think about it, the more possible
>option 1 sounds.
>I've put it back on my todo list, but I don't expect to get to it this
>year.  Ofcourse if someone else wants to give it a try, I'm happy to
>make suggestions and review code.
>
I do appreciate being too busy, I'm just glad I have been able to 
clarify the tradeoffs of RAID5e, and get it on your list at all. I did 
look at the code a bit, and it would seem that if the "rebuild to hot 
spare" code is modified to handle a distributed spare, then it looks as 
if RAID6e might pretty much fall out. Feel free to tell me I'm dreaming.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979