Software RAID-0 striping over Hardware RAID-5

Software RAID-0 striping over Hardware RAID-5

[AndyLiebman]

Hi everyone, 

I have a Xeon machine (3.06 Ghz single processor, 2 GB RAM, SuperMicro 
motherboard) with two 3ware 9500S-8 cards. I'm getting good performance with 3ware's 
Hardware RAID-5 (I have reasons why I don't want to use Software RAID-5 on 
this machine). 

My question is, what kind of problems, if any, might I run into if I use 
Linux Software RAID to stripe the two RAID-5's together into a single RAID-0? The 
reason I'm thinking about doing this is to squeeze more performance out of the 
storage system for working with huge video files. 

(BTW, this is what Apple does with its XSeve RAID in order to work with high 
definition video. The XServe has two 7-drive RAID-5 arrays, each connected to 
a fiber channel port. The data from both arrays goes into a single dual 
channel fiber channel card on a G5 workstation, which then uses software RAID-0 to 
stripe the two RAIDS together. )

With the 2.6 kernel (versions 2.6.6 and above), should I expect any trouble 
if a drive goes down on ONE of the RAID-5s and I have to replace the disk and 
rebuild? Will the much lower performance on one of the RAID 5s during rebuild 
cause problems for the RAID-0? 

I guess it would be like making a RAID-0 with two drives of equal size, but 
very mismatched in terms of speed (i.e., a 4200 rpm drive and a 7200 rpm 
drive). 

Words of wisdom would be appreciated. 

Regards, 
Andy Liebman

Re: Software RAID-0 striping over Hardware RAID-5

[Joshua Baker-LePain]

On Fri, 17 Sep 2004 at 6:03pm, AndyLiebman@aol.com wrote

> I have a Xeon machine (3.06 Ghz single processor, 2 GB RAM, SuperMicro 
> motherboard) with two 3ware 9500S-8 cards. I'm getting good performance with 3ware's 
> Hardware RAID-5 (I have reasons why I don't want to use Software RAID-5 on 
> this machine). 
> 
> My question is, what kind of problems, if any, might I run into if I use 
> Linux Software RAID to stripe the two RAID-5's together into a single RAID-0? The 
> reason I'm thinking about doing this is to squeeze more performance out of the 
> storage system for working with huge video files. 

I have a couple of dual 3ware boxes I do this with, and have never had a 
problem.  I would make sure that you have a hot spare on your RAID5 
arrays, though.  Because if one of them dies, you'll obviously lose 
everything on both arrays.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: Software RAID-0 striping over Hardware RAID-5

[AndyLiebman]

jlb17@duke.edu writes:

> I have a couple of dual 3ware boxes I do this with, and have never had a 
> problem.  I would make sure that you have a hot spare on your RAID5 
> arrays, though.  Because if one of them dies, you'll obviously lose 
> everything on both arrays.

Thanks for the reply, Joshua. 

Have you ever tested your systems while one of your RAID5 arrays is 
rebuilding or verifying? (I'm about to do that right now). I'm wondering if Linux 
Software RAID-0 might falter if one of the two arrays is significantly slower than 
the other (while rebuilding)?

To test the integrity of the system, I'm going to remove a drive from one of 
the RAID5 arrays, add it back and rebuild the array, and while the rebuilding 
is taking place I'm going to run Bonnie++ several times (100GB of 
writing/reading per test). 

My inclination is to not use a hot spare. Instead, I rely on 3ware's email 
notification feature to alert me that a drive needs to be replaced. I always 
keep a spare handy that can be popped into place immediately. Also, I want to 
have control over when rebuilding takes place. It would be a bad thing to rebuild 
during certain critical high I/O moments of the day -- whereas rebuilding at 
night would be much less stressful on the system. Do you see anything wrong 
with this approach? 

Thanks again, 
Andy

RE: Software RAID-0 striping over Hardware RAID-5

[Guy]

Even in degraded mode or re-building, my array is faster much faster than my
network.  I have 100BaseTX Full duplex, so I can read and write a total of
20Meg per second.  My array is faster than that!

If your RAID5 arrays were software, the md driver will slow the re-build
when other disk access is needed.  These 2 files help control the speed:
/proc/sys/dev/raid/speed_limit_max
/proc/sys/dev/raid/speed_limit_min

Since your RAID5 arrays are hardware, you may have a re-build priority that
can be adjusted.  In your case you would want low priority.  I prefer to be
safe, I always give the array re-build the highest priority.

I would go with the hot spare.  Once the first disk fails, you are at risk
of losing everything if a second disk fails.  Even during the re-build, if a
second disk fails, game over man, game over!  But once the re-build is
finished, you are safe again.  Safe for another single disk failure that is.
But, if you have a daily backup, and you are willing to risk losing 1 days
work, that is your choice!

From what I have read, RAID0 will just go as fast as your slowest disk.
Your RAID5 re-building will not hurt the RAID0 array.  It will just slow it
down.

Guy

Sure you saved money, but at what cost? - "Guy Watkins"

-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of AndyLiebman@aol.com
Sent: Monday, September 20, 2004 9:51 AM
To: jlb17@duke.edu
Cc: linux-raid@vger.kernel.org
Subject: Re: Software RAID-0 striping over Hardware RAID-5

jlb17@duke.edu writes:

> I have a couple of dual 3ware boxes I do this with, and have never had a 
> problem.  I would make sure that you have a hot spare on your RAID5 
> arrays, though.  Because if one of them dies, you'll obviously lose 
> everything on both arrays.

Thanks for the reply, Joshua. 

Have you ever tested your systems while one of your RAID5 arrays is 
rebuilding or verifying? (I'm about to do that right now). I'm wondering if
Linux 
Software RAID-0 might falter if one of the two arrays is significantly
slower than 
the other (while rebuilding)?

To test the integrity of the system, I'm going to remove a drive from one of

the RAID5 arrays, add it back and rebuild the array, and while the
rebuilding 
is taking place I'm going to run Bonnie++ several times (100GB of 
writing/reading per test). 

My inclination is to not use a hot spare. Instead, I rely on 3ware's email 
notification feature to alert me that a drive needs to be replaced. I always

keep a spare handy that can be popped into place immediately. Also, I want
to 
have control over when rebuilding takes place. It would be a bad thing to
rebuild 
during certain critical high I/O moments of the day -- whereas rebuilding at

night would be much less stressful on the system. Do you see anything wrong 
with this approach? 

Thanks again, 
Andy
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Software RAID-0 striping over Hardware RAID-5

[Tim Moore]

>>From what I have read, RAID0 will just go as fast as your slowest disk.

Almost 2X faster.

[16:52] abit: > cat /proc/mdstat
Personalities : [raid0] [raid1] [raid5]
read_ahead 1024 sectors
md0 : active raid0 sdb5[1] sda5[0]
       4192768 blocks 32k chunks

md1 : active raid0 sdb6[1] sda6[0]
       2425600 blocks 32k chunks

md2 : active raid0 sdb7[1] sda7[0]
       12707200 blocks 16k chunks

md3 : active raid0 sdb8[1] sda8[0]
       20113152 blocks 64k chunks

unused devices: <none>

102400+0 records in
102400+0 records out
: 10.72 19% 0:10.72 s 2.12 u 0.02 mapf 112 mipf 12
: dd if=/dev/sda5 of=/dev/null bs=4k count=100k
102400+0 records in
102400+0 records out
: 11.33 19% 0:11.33 s 2.18 u 0.06 mapf 112 mipf 12
: dd if=/dev/sdb5 of=/dev/null bs=4k count=100k
102400+0 records in
102400+0 records out
: 5.81 32% 0:05.81 s 1.81 u 0.07 mapf 112 mipf 12
: dd if=/dev/md0 of=/dev/null bs=4k count=100k

[17:20] abit:~ > echo "scale=2;102400*4*1024;((10.72+11.33)/2)/5.81" | bc -q
419430400
1.89

Re: Software RAID-0 striping over Hardware RAID-5

[Scott T. Smith]

On Wed, 2004-09-22 at 17:23, Tim Moore wrote:
> >>From what I have read, RAID0 will just go as fast as your slowest disk.
> 
> Almost 2X faster.

writes will be as fast as the slowest disk; reads will be faster.

	Scott

Re: Software RAID-0 striping over Hardware RAID-5

[Doug Ledford]

On Wed, 2004-09-22 at 20:27, Scott T. Smith wrote:
> On Wed, 2004-09-22 at 17:23, Tim Moore wrote:
> > >>From what I have read, RAID0 will just go as fast as your slowest disk.
> > 
> > Almost 2X faster.
> 
> writes will be as fast as the slowest disk; reads will be faster.

Eh?  I think the more correct way to phrase RAID0 characteristics over
*any* selection of disks is that the minimum theoretical speed for the
overall array when it comes to sequential reads *and* writes is
n*speed_slowest_disk, where n is the number of drives in the array.

> 	Scott
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 
  Doug Ledford <dledford@redhat.com>     919-754-3700 x44233
         Red Hat, Inc.
         1801 Varsity Dr.
         Raleigh, NC 27606

Re: Software RAID-0 striping over Hardware RAID-5

[Joshua Baker-LePain]

On Mon, 20 Sep 2004 at 9:49am, AndyLiebman@aol.com wrote

> Have you ever tested your systems while one of your RAID5 arrays is 
> rebuilding or verifying? (I'm about to do that right now). I'm wondering if Linux 
> Software RAID-0 might falter if one of the two arrays is significantly slower than 
> the other (while rebuilding)?

I've had arrays under normal (sporadically heavy) usage during a rebuild, 
and there wasn't a problem.

> My inclination is to not use a hot spare. Instead, I rely on 3ware's email 
> notification feature to alert me that a drive needs to be replaced. I always 
> keep a spare handy that can be popped into place immediately. Also, I want to 
> have control over when rebuilding takes place. It would be a bad thing to rebuild 
> during certain critical high I/O moments of the day -- whereas rebuilding at 
> night would be much less stressful on the system. Do you see anything wrong 
> with this approach? 

Yes -- it requires you to be on call 24x7 for maximum protection.  And 
even then the level of protection is less than with a hot spare, given 
that it will take you time to get from whereever you are to whereever the 
server is.  Also, disks can sometimes have a tendency to go in quick 
succession.  I would *strongly* recommend a hot spare.

Remember, you're cutting your reliability in half by striping the two 
arrays.  Why not maximally protect those individual arrays?

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University