Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

[Rik Herrin]

Hi,
	I was interested in Linux's RAID capabilities and
read that mdadm was the tool of choice.  We are
currently comparing software RAID with hardware RAID
and to complete our comparison, we were wondering if
the following is supported by mdadm:

	1) OCE: Online Capacity Expansion:  From the latest
version of mdadm (v2.2), it ssems that there is
support for it with the -G option.  How well tested is
this?  Also, in the Readme / Man page, it mentions:
		This usage causes mdadm to attempt to reconfigure a
running array.  This is only possibly if the kernel
being used supports a particular reconfiguration.
	How can I know if the kernel I am using supports this
reconfiguration?  What if I'm compiling the kernel by
hand.  What options would I have to enable?
	2) RAID Level Migration:  Does mdadm currently
support this feature?
	3) Performance issues:  I'm currently thinking of
using either RAID 10 or LVM2 with RAID 5 to serve as a
RAID server.  The machine will be running either an
AMD 64 processor or a dual-core AMD 64 processor, so I
don't think the CPU will be a bottleneck.  In fact, it
should easily pass the speed of most "hardware" based
RAID systems.
	4) Would anyone recommend a certain hotswap
enclosure?
	5) Finally, I'm thinking of integrating this in a
custom SAN backend.  Anyone try this before?  Thanks
for your time.

PS.  Some of the terms used in the man page are a bit
ambiguous.  For example, what does --write-mostly
refer to?  Thanks for your time.


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

[Molle Bestefich]

Rik Herrin wrote:
> I was interested in Linux's RAID capabilities and
> read that mdadm was the tool of choice.  We are
> currently comparing software RAID with hardware RAID

MD is far superior to most of the hardware RAID solutions I've touched.
In short, it seems MD is developed with the goal of keeping your data
safe, not selling hardware.

I've had problems both with MD and with hardware RAID.  With hardware
RAID, once things go bad, they really go bad.  With MD, there's
usually a straight-forward way to rescue things.  And when there's
not, Neil's a real nice guy who always stands up to help and fix bugs.

I would trust my data with MD over any hardware RAID solution,
including professional server RAID solutions from eg. Compaq or IBM.

MD is a little more difficult to set up and also lacks in that it
doesn't integrate with BIOS level stuff and boot loaders (maybe
there's minimal MD RAID 1 support in Lilo, not sure).  Depending on
your choice of hardware, you might also get more features than MD can
currently offer.

> 	1) OCE: Online Capacity Expansion:  From the latest
> version of mdadm (v2.2), it ssems that there is
> support for it with the -G option.  How well tested is
> this?

New feature, so obviously not tested very well.

Neil said at one point that he was going to release this to the
general public when it's stable and when it can recover an interrupted
resize process.  Sounds like a very reasonable and sane goal to me, I
hope that this is still the case.

Otherwise, it's easy to work around - you can just create a new RAID
array on your new disks / extra disk space and then join it to the end
of the old array using MD's linear personality or DM.  Never tried it,
but should work just fine.

>  Also, in the Readme / Man page, it mentions:
> 		This usage causes mdadm to attempt to reconfigure a
> running array.  This is only possibly if the kernel
> being used supports a particular reconfiguration.
> 	How can I know if the kernel I am using supports this
> reconfiguration?  What if I'm compiling the kernel by
> hand.  What options would I have to enable?

Just the usual MD stuff I think.
You'll probably need a quite new kernel where Neil's bitmap patches
has been applied.
Hopefully MD will detect whether the kernel is new enough or not, but
I haven't tried myself ;-).

> 	2) RAID Level Migration:  Does mdadm currently
> support this feature?

I don't think so, but sounds like RAID5 --> RAID6 is planned.
Check back in a year or so ;-).

Or choose the RAID level you *really* want to begin with (duh).

Since you say "we", I assume you're part of a very large corporation
and thus intend to RAID a whole bunch of disks.  Go with RAID6 + a
couple of spares for that.  If you intend to use really many disks,
make multiple arrays.  (Not sure whether you can share spares across
arrays, but I think you can.)

> 	3) Performance issues:  I'm currently thinking of
> using either RAID 10 or LVM2 with RAID 5 to serve as a
> RAID server.  The machine will be running either an
> AMD 64 processor or a dual-core AMD 64 processor, so I
> don't think the CPU will be a bottleneck.  In fact, it
> should easily pass the speed of most "hardware" based
> RAID systems.

I think there's two issues to cover,
 * Throughput
 * Seek times

And of course they're not entirely separate issues - throughput will
be lower when you're doing random access (seeking) and seek times will
be higher when you're pulling lots of data out.

I've seen lots of MD tests, but none that covered profiling MD's
random access performance.  So I suppose that most hardware solutions
will do a lot better than MD here since they have been profiled with
this in mind.

Throughput-wise, I think MD is probably very good.
But I can't back that up with factual data, sorry.

> 	4) Would anyone recommend a certain hotswap
> enclosure?

I would, but can't remember their name, sorry :-)

Re: Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

[Ross Vandegrift]

On Thu, Dec 22, 2005 at 12:31:25PM +0000, Molle Bestefich wrote:
> Since you say "we", I assume you're part of a very large corporation
> and thus intend to RAID a whole bunch of disks.  Go with RAID6 + a
> couple of spares for that.  If you intend to use really many disks,
> make multiple arrays.  (Not sure whether you can share spares across
> arrays, but I think you can.)

A recent foray through mdadm's code verifies this.  If it noticies a
failure and there is a spare, it attempts to migrate the spare to the
array that needs it.  Very cool feature!

> I've seen lots of MD tests, but none that covered profiling MD's
> random access performance.  So I suppose that most hardware solutions
> will do a lot better than MD here since they have been profiled with
> this in mind.

Well, it depends on the RAID level, disk, configuration, and how
you're using it.  In general, RAID 0+1 has better seek properties
because reads can be done independantly from many disks.  RAID5 is
always going to be slow because n-1 disks need to all simultaneously
read their stripe, and this can cause spindle contention.

Of course, you lose more space to overhead as RAID 0+1 arrays grow...

-- 
Ross Vandegrift
ross@lug.udel.edu

"The good Christian should beware of mathematicians, and all those who
make empty prophecies. The danger already exists that the mathematicians
have made a covenant with the devil to darken the spirit and to confine
man in the bonds of Hell."
	--St. Augustine, De Genesi ad Litteram, Book II, xviii, 37

Re: Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

[Molle Bestefich]

Ross Vandegrift wrote:
> Molle Bestefich wrote:
> > Since you say "we", I assume you're part of a very large corporation
> > and thus intend to RAID a whole bunch of disks.  Go with RAID6 + a
> > couple of spares for that.  If you intend to use really many disks,
> > make multiple arrays.  (Not sure whether you can share spares across
> > arrays, but I think you can.)
>
> A recent foray through mdadm's code verifies this.  If it noticies a
> failure and there is a spare, it attempts to migrate the spare to the
> array that needs it.  Very cool feature!

Super =)

> > I've seen lots of MD tests, but none that covered profiling MD's
> > random access performance.  So I suppose that most hardware solutions
> > will do a lot better than MD here since they have been profiled with
> > this in mind.
>
> Well, it depends on the RAID level, disk, configuration, and how
> you're using it.

Still a shame for MD that noone has done these tests.

Also a shame that noone has setup up a test farm with:
 * A test box that regularly profiles MD, for seek time, throughput
and CPU usage with various personalities.
 * Another test box, exactly the same hardware as the first one,
suited for plugging in new hardware RAID adapters (while retaining
disks and other hardware) to test hardware RAID solutions with
consistent results even when new products come out.

> In general, RAID 0+1 has better seek properties
> because reads can be done independantly from many disks.  RAID5 is
> always going to be slow because n-1 disks need to all simultaneously
> read their stripe, and this can cause spindle contention.

Why would you need to send all of the disks in a RAID5 to the same
stripe if only fx. one block in the stripe was requested?  Doesn't
make much sense to me..

> Of course, you lose more space to overhead as RAID 0+1 arrays grow...

Of course.  It seems to me that RAID 1+0 has the exact same
performance properties as RAID 0+1 if things are done properly in
whatever RAID hardware/software used..

Re: Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

[Lajber Zoltan]

Hi,

On Thu, 22 Dec 2005, Rik Herrin wrote:

> 	1) OCE: Online Capacity Expansion:  From the latest
> version of mdadm (v2.2), it ssems that there is
> support for it with the -G option.  How well tested is

Well, we use LVM above raid, thins is not a big issue.

> 	3) Performance issues:  I'm currently thinking of

I have some simple test with bonnie++, the sw raid superior to hw raid,
except big-name storage systems.
http://zeus.gau.hu/~lajbi/diskbenchmarks.txt

> 	4) Would anyone recommend a certain hotswap
> enclosure?

We used ibm xseries and sun machines. The hotswap working well, but the
number of drives very limited.

Take a look at http://www.coraid.com : very well scalable, cheap storage.

Bye,
-=Lajbi=----------------------------------------------------------------
 LAJBER Zoltan               Szent Istvan Egyetem,  Informatika Hivatal
 Most of the time, if you think you are in trouble, crank that throttle!

Re: Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

[Molle Bestefich]

Lajber Zoltan wrote:
> I have some simple test with bonnie++, the sw raid superior to hw raid,
> except big-name storage systems.
> http://zeus.gau.hu/~lajbi/diskbenchmarks.txt

Cool.
But what does "gep, tip, diskvez, iras, olvasas" and "atlag" mean?

Re: Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

[Lajber Zoltan]

On Thu, 22 Dec 2005, Molle Bestefich wrote:

> Lajber Zoltan wrote:
> > I have some simple test with bonnie++, the sw raid superior to hw raid,
> > except big-name storage systems.
> > http://zeus.gau.hu/~lajbi/diskbenchmarks.txt
>
> Cool.
> But what does "gep, tip, diskvez, iras, olvasas" and "atlag" mean?

Oh, sorry, that is hungarian :)

gep= machine - a kind of id
tip= type
diskvez: disk controller
iras: write
olvasas: read
atlag: mean (of read and write)

Bye,
-=Lajbi=----------------------------------------------------------------
 LAJBER Zoltan               Szent Istvan Egyetem,  Informatika Hivatal
 Most of the time, if you think you are in trouble, crank that throttle!

Re: Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

[Andrew Burgess]

>	3) Performance issues:  I'm currently thinking of
>using either RAID 10 or LVM2 with RAID 5 to serve as a
>RAID server.

I think you always want LVM2 between raid and the filesystem. Not only can you
expand things but you can discard older raid arrays when you upgrade. At least
you should study the docs for LVM to see if it would be good for your
application.

Re: Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

[John Stoffel]

>>>>> "Andrew" == Andrew Burgess <aab@cichlid.com> writes:

>> 3) Performance issues:  I'm currently thinking of
>> using either RAID 10 or LVM2 with RAID 5 to serve as a
>> RAID server.

Andrew> I think you always want LVM2 between raid and the
Andrew> filesystem. Not only can you expand things but you can discard
Andrew> older raid arrays when you upgrade. At least you should study
Andrew> the docs for LVM to see if it would be good for your
Andrew> application.

I agree with this, having LVM between MD and the filesystem lets you
move the data from one MD to another without any major downtime or
even your users having to notice.  Which is cool.  

I really really like ext3 for it's reliability, but a filesystem which
you can grow/shrink while mounted would be nice.  I really like VxFS
from Veritas, but obviously you won't find it one Linux right now.  

I've been tempted by XFS at times, but I worry, esp since alot of
other people who are core developers don't care for XFS as much.  But
maybe I'll move that way for my next system.

John

Re: Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

[Robert Heinzmann]

Hi Rik,

Neil answered some of the questions earlier:


>	How can I know if the kernel I am using supports this
>reconfiguration?  What if I'm compiling the kernel by
>hand.  What options would I have to enable?
>  
>
>PS.  Some of the terms used in the man page are a bit
>ambiguous.  For example, what does --write-mostly
>refer to?  Thanks for your time.
>  
>
See Message : 
http://www.opensubscriber.com/message/linux-raid@vger.kernel.org/2852873.html


Reagards,
Robert Heinzmann

Re: Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

[Lajber Zoltan]

On Thu, 22 Dec 2005, John Stoffel wrote:

> I've been tempted by XFS at times, but I worry, esp since alot of
> other people who are core developers don't care for XFS as much.  But
> maybe I'll move that way for my next system.

We use xfs since 2.6.x in production. Typical config for us:
/ in raid1, ext3
swap in raid1
lvm2 for the remaining with XFS above it.

The lvm2 array raid1 or raid5. If raid1, the / and swap have hot spare
partitions.

The main reason for this is that you can't have boot block in XFS.

Bye,
-=Lajbi=----------------------------------------------------------------
 LAJBER Zoltan               Szent Istvan Egyetem,  Informatika Hivatal
 Most of the time, if you think you are in trouble, crank that throttle!

Re: Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

[jeane]

bonjour!
I was going thrugh the thread I am also in the processing of deciding
weither to procurz H/W raid, One significant matter would be
performances of the systems,
CPU utilization is an important issue as the workload involved lots of
CPU. but unfortunately for me there is no relative benchmarks to
convince peoples about the workload and performance comparison. If
somebody has done this  comparitive study it would greatly be of help
to me.
merci :)
                jeane

Re: Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

[Lajber Zoltan]

On Tue, 27 Dec 2005, jeane wrote:

> bonjour!
bonjur, ca va? :)

> I was going thrugh the thread I am also in the processing of deciding
> weither to procurz H/W raid, One significant matter would be
> performances of the systems,

On a server-class hardware, no significant cpu load can be observed.
We have serverworks/xeon or Adaptec, LSI and Fusion scsi ctrl, they are
good (in this order: Fusion MPT very impressiv :)
The sun v20z AMD-8131,  LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT
Dual Ultra320 SCSI is good to.

But, the cheap motherboards (sis chipset) and bad config can kill cpu.
Especialy the wrong PATA config, like raid1 on hda/hdb.

Bye,
-=Lajbi=----------------------------------------------------------------
 LAJBER Zoltan               Szent Istvan Egyetem,  Informatika Hivatal
 Most of the time, if you think you are in trouble, crank that throttle!

Re: Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

[Rik Herrin]

Sorry for taking so long to reply but it's been a
hectic week.  Thanks for your reply.  Out of
curiousity, what is MD's linear personality or DM? 
Would this affect performance if I expanded my RAID
using this?

--- Molle Bestefich <molle.bestefich@gmail.com> wrote:

> Rik Herrin wrote:
> > I was interested in Linux's RAID capabilities and
> > read that mdadm was the tool of choice.  We are
> > currently comparing software RAID with hardware
> RAID
> 
> MD is far superior to most of the hardware RAID
> solutions I've touched.
> In short, it seems MD is developed with the goal of
> keeping your data
> safe, not selling hardware.
> 
> I've had problems both with MD and with hardware
> RAID.  With hardware
> RAID, once things go bad, they really go bad.  With
> MD, there's
> usually a straight-forward way to rescue things. 
> And when there's
> not, Neil's a real nice guy who always stands up to
> help and fix bugs.
> 
> I would trust my data with MD over any hardware RAID
> solution,
> including professional server RAID solutions from
> eg. Compaq or IBM.
> 
> MD is a little more difficult to set up and also
> lacks in that it
> doesn't integrate with BIOS level stuff and boot
> loaders (maybe
> there's minimal MD RAID 1 support in Lilo, not
> sure).  Depending on
> your choice of hardware, you might also get more
> features than MD can
> currently offer.
> 
> > 	1) OCE: Online Capacity Expansion:  From the
> latest
> > version of mdadm (v2.2), it ssems that there is
> > support for it with the -G option.  How well
> tested is
> > this?
> 
> New feature, so obviously not tested very well.
> 
> Neil said at one point that he was going to release
> this to the
> general public when it's stable and when it can
> recover an interrupted
> resize process.  Sounds like a very reasonable and
> sane goal to me, I
> hope that this is still the case.
> 
> Otherwise, it's easy to work around - you can just
> create a new RAID
> array on your new disks / extra disk space and then
> join it to the end
> of the old array using MD's linear personality or
> DM.  Never tried it,
> but should work just fine.
> 
> >  Also, in the Readme / Man page, it mentions:
> > 		This usage causes mdadm to attempt to
> reconfigure a
> > running array.  This is only possibly if the
> kernel
> > being used supports a particular reconfiguration.
> > 	How can I know if the kernel I am using supports
> this
> > reconfiguration?  What if I'm compiling the kernel
> by
> > hand.  What options would I have to enable?
> 
> Just the usual MD stuff I think.
> You'll probably need a quite new kernel where Neil's
> bitmap patches
> has been applied.
> Hopefully MD will detect whether the kernel is new
> enough or not, but
> I haven't tried myself ;-).
> 
> > 	2) RAID Level Migration:  Does mdadm currently
> > support this feature?
> 
> I don't think so, but sounds like RAID5 --> RAID6 is
> planned.
> Check back in a year or so ;-).
> 
> Or choose the RAID level you *really* want to begin
> with (duh).
> 
> Since you say "we", I assume you're part of a very
> large corporation
> and thus intend to RAID a whole bunch of disks.  Go
> with RAID6 + a
> couple of spares for that.  If you intend to use
> really many disks,
> make multiple arrays.  (Not sure whether you can
> share spares across
> arrays, but I think you can.)
> 
> > 	3) Performance issues:  I'm currently thinking of
> > using either RAID 10 or LVM2 with RAID 5 to serve
> as a
> > RAID server.  The machine will be running either
> an
> > AMD 64 processor or a dual-core AMD 64 processor,
> so I
> > don't think the CPU will be a bottleneck.  In
> fact, it
> > should easily pass the speed of most "hardware"
> based
> > RAID systems.
> 
> I think there's two issues to cover,
>  * Throughput
>  * Seek times
> 
> And of course they're not entirely separate issues -
> throughput will
> be lower when you're doing random access (seeking)
> and seek times will
> be higher when you're pulling lots of data out.
> 
> I've seen lots of MD tests, but none that covered
> profiling MD's
> random access performance.  So I suppose that most
> hardware solutions
> will do a lot better than MD here since they have
> been profiled with
> this in mind.
> 
> Throughput-wise, I think MD is probably very good.
> But I can't back that up with factual data, sorry.
> 
> > 	4) Would anyone recommend a certain hotswap
> > enclosure?
> 
> I would, but can't remember their name, sorry :-)
> 



	
		
__________________________________ 
Yahoo! for Good - Make a difference this year. 
http://brand.yahoo.com/cybergivingweek2005/

Re: Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

[Rik Herrin]

Thanks Lajber.  Sorry for taking so long to reply but
it's been a busy week.  Your test benchmarks look
interesting.  Do they take into consideration the
random seeks that Molle mentioned in his email?  As
Molle mentioned: "I've seen lots of MD tests, but none
that covered profiling MD's random access
performance".  Does this cover MD's random access
performance?  

Also, have you thought of using iometer
(http://www.iometer.org) and comparing it's results
with bonnie++'s results?  Just a thought...

Thanks for your time...

--- Lajber Zoltan <lajbi@lajli.gau.hu> wrote:

> Hi,
> 
> On Thu, 22 Dec 2005, Rik Herrin wrote:
> 
> > 	1) OCE: Online Capacity Expansion:  From the
> latest
> > version of mdadm (v2.2), it ssems that there is
> > support for it with the -G option.  How well
> tested is
> 
> Well, we use LVM above raid, thins is not a big
> issue.
> 
> > 	3) Performance issues:  I'm currently thinking of
> 
> I have some simple test with bonnie++, the sw raid
> superior to hw raid,
> except big-name storage systems.
> http://zeus.gau.hu/~lajbi/diskbenchmarks.txt
> 
> > 	4) Would anyone recommend a certain hotswap
> > enclosure?
> 
> We used ibm xseries and sun machines. The hotswap
> working well, but the
> number of drives very limited.
> 
> Take a look at http://www.coraid.com : very well
> scalable, cheap storage.
> 
> Bye,
>
-=Lajbi=----------------------------------------------------------------
>  LAJBER Zoltan               Szent Istvan Egyetem, 
> Informatika Hivatal
>  Most of the time, if you think you are in trouble,
> crank that throttle!
> 



		
__________________________________________ 
Yahoo! DSL – Something to write home about. 
Just $16.99/mo. or less. 
dsl.yahoo.com 

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

[H. Peter Anvin]

Followup to:  <62b0912f0512220431l4d1ec130q137b737c1956ddd9@mail.gmail.com>
By author:    Molle Bestefich <molle.bestefich@gmail.com>
In newsgroup: linux.dev.raid
> 
> MD is a little more difficult to set up and also lacks in that it
> doesn't integrate with BIOS level stuff and boot loaders (maybe
> there's minimal MD RAID 1 support in Lilo, not sure).  Depending on
> your choice of hardware, you might also get more features than MD can
> currently offer.
> 

LILO and SYSLINUX/EXTLINUX can boot from RAID-1; some versions of GRUB can too.

	-hpa

Re: Linux RAID Enterprise-Level Capabilities and If It Supports Raid Level Migration and Online Capacity Expansion

[Dan Stromberg]

There are those who believe that if you have a production system with
xfs, that you should always have a UPS, because xfs journals aren't as
thorough as the journals of some other filesystems, including ext3.

On Fri, 2005-12-23 at 00:02 +0100, Lajber Zoltan wrote:
> On Thu, 22 Dec 2005, John Stoffel wrote:
> 
> > I've been tempted by XFS at times, but I worry, esp since alot of
> > other people who are core developers don't care for XFS as much.  But
> > maybe I'll move that way for my next system.
> 
> We use xfs since 2.6.x in production. Typical config for us:
> / in raid1, ext3
> swap in raid1
> lvm2 for the remaining with XFS above it.
> 
> The lvm2 array raid1 or raid5. If raid1, the / and swap have hot spare
> partitions.
> 
> The main reason for this is that you can't have boot block in XFS.
> 
> Bye,
> -=Lajbi=----------------------------------------------------------------
>  LAJBER Zoltan               Szent Istvan Egyetem,  Informatika Hivatal
>  Most of the time, if you think you are in trouble, crank that throttle!
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>