Re: Raid selection questions (10 vs 6, n2 vs f2) on an 8 drive array

[Larry Schwerzler <larry@schwerzler.com>]

Joe, thanks for the info, response/questions inline.

On Fri, Feb 18, 2011 at 5:12 PM, Joe Landman <joe.landman@gmail.com> wrote:
> On 02/18/2011 03:55 PM, Larry Schwerzler wrote:
>
> [...]
>
>> Questions:
>>
>> 1. In my research of raid10 I very seldom hear of drive configurations
>> with more drives then 4, are there special considerations with having
>> an 8 drive raid10 array? I understand that I'll be loosing 2TB of
>> space from my current setup, but i'm not too worried about that.
>
> If you are going to set this up, I'd suggest a few things.
>
> 1st: try to use a PCI HBA with enough ports, not the motherboard ports.

I use the SANS DIGITAL HA-DAT-4ESPCIE PCI-Express x8 SATA II card with
the SANS DIGITAL TR8M-B 8 Bay SATA to eSATA (Port Multiplier) JBOD
Enclosure, so i'm most of the way there, just esata instead of sas. I
didn't realize that the esata connections had issues like this else I
would have avoided it, though at the time the extra cost of a sas card
that could expand to a total of 16 external hard drives would have
been prohibitive.

>
> 2nd: eSATA is probably not a good idea (see your issue below).
>
> 3rd: I'd suggest getting 10 drives and using 2 as hot spares.  Again, not
> using eSATA.  Use an internal PCIe card that provides a reasonable chip.  If
> you can't house the drives internal to your machine, get a x4 or x8
> JBOD/RAID cannister.  A single (or possibly 2) SAS cables.  But seriously,
> lose the eSATA setup.

I may see about getting an extra drive or two to act as hot spares.

>
>>
>> 2. One problem I'm having with my current setup is the esata cables
>> have been knocked loose which effectively drops 4 of my drives. I'd
>> really like to be able to survive this type of sudden drive loss. if
>> my drives are /dev/sd[abcdefgh] and abcd are on one esata channel
>> while efgh are on the other is there what drive order should I create
>> the array with? I'd guess /dev/sd[aebfcgdh] would that give me
>> survivability if one of my esata channels went dark?
>
> Usually the on-board eSATA chips are very low cost, low bandwidth units.
>  Spend another $150-200 on a dual external SAS HBA, and get the JBOD
> container.

I'd be interested in any specific recommendations anyone might have
for a $200 or so card and jbod enclosure that could house at least 8
drives. Off list is fine, so as to not spam the list.

I have zero experience with SAS, does it not experience the issues
that my esata setup runs into?

>
>>
>> 3. One of the concerns I have with raid10 is expandability, and I'm
>> glad to see reshaping raid10 as an item on the 2011 roadmap :) However
>> it will likely be a while before I'll see that ability in my distro
>> for a while. I did find a guide on expanding raid size when using lvm
>> by increasing the size of each drive and creating two partitions 1 the
>> size of the original drive, and one with the remainder of the new
>> space. Once you have done this for all drives you create a new raid10
>> array with the 2nd partitions on all the drives and add it to the lvm
>> storage group, effectively you have two raid10 arrays 1 on the first
>> half of the drives 1 on the 2nd half of the drives and the space
>> pooled together. I'm sure many of you are familiar with this scenario,
>> but I'm wondering if this scenario could be problematic, is having two
>> raid10 arrays on one drive an issue?
>
> We'd recommend against this.  Too much seeking.

So the raid10 expansion solution is again to wait for raid10 reshaping
in the mdraid tools, or start from scratch.

I thought that maybe with LVM since it wouldn't be striping the data
accross the arrays, it would mostly be accessing the info from one
array at a time. I don't know enough about the way that lvm stores the
data to know different though.
>
>>
>> 4. Part of the reason I'm wanting to switch is because of information
>> I read on the "BAARF" site pointing out some of the issues in the
>> parity raid's that can cause issues that people sometimes don't think
>> about. (site: http://www.miracleas.com/BAARF/BAARF2.html) A lot of the
>> information on the site is a few years old now and given how fast
>> things can change and the fact that I have not found many people
>> complaining about the parity raids I'm wondering if some/all of the
>> gotchas that they list are less of an issue now? Maybe my reasons for
>> moving to raid10 are no longer relevant?
>
> Things have gotten worse.  The BERs are improving a bit (most reasonable
> SATA drives report 1E-15 as their rate as compared with 1E-14 as previously.
>  Remember, 2TB = 1.6E13 bits.  So 10x 2TB drives together is 1.6E14 bits.  8
> scans or rebuilds will get you to a statistical near certainty of hitting an
> unrecoverable error.
>
> RAID6 buys you a little more time than RAID5, but you still have worries due
> to the time correlated second drive failure.  Google found a peak at 1000s
> after the first drive failure (which likely corresponds to an error on
> rebuild).  With RAID5, that second error is the end of your data.  With
> RAID6, you still have a fighting chance at recovery.
>

This i what really scares me, it seems like a false sense of security
as your drive size increases. Hoping for a better chance with raid10

>
>> Thank you in advance for any/all information given. And a big thank
>> you to Neil and the other developers of linux-raid for their efforts
>> on this great tool.
>
> Despite the occasional protestations to the contrary, MD raid is a robust
> and useful RAID layer, and not a "hobby" layer.  We use it extensively, as
> do many others.
>
>
> --
> Joe Landman
> landman@scalableinformatics.com
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html