Performance question

Performance question

[Piergiorgio Sartor]

Hi all,

I'll have to setup some machines with two HDs (each)
in order to get some redundancy.

Reading the MD features I noticed there are several
possibilities to create a mirror.
I was wondering which one offer the best perfomances
and/or what are the compromises to accept between
the different solutions.

One possibility is a classic RAID-1 mirror.
Another is a RAID-10 far.
There would also be the RAID-10 near, but I guess
this is equivalent to RAID-1.

Any suggestion on which method offers higher "speed"?
Or there are other possibilities with 2 HDs (keeping
the redundancy, of course)?

Thanks a lot in advance,

bye,

-- 

piergiorgio

Re: Performance question

[David Lethe]

All we know is that you use 2 disks and md.  This is like posting to a TCP/IP architecture group and saying you have a network connection and want performance advice.   Read up, supply full config info, run benchmarks, then ask specific questions.  GI=GO.
-----Original Message-----

From:  "Piergiorgio Sartor" <piergiorgio.sartor@nexgo.de>
Subj:  Performance question
Date:  Sat Jan 17, 2009 11:18 am
Size:  874 bytes
To:  "linux-raid@vger.kernel.org" <linux-raid@vger.kernel.org>

Hi all, 
 
I'll have to setup some machines with two HDs (each) 
in order to get some redundancy. 
 
Reading the MD features I noticed there are several 
possibilities to create a mirror. 
I was wondering which one offer the best perfomances 
and/or what are the compromises to accept between 
the different solutions. 
 
One possibility is a classic RAID-1 mirror. 
Another is a RAID-10 far. 
There would also be the RAID-10 near, but I guess 
this is equivalent to RAID-1. 
 
Any suggestion on which method offers higher "speed"? 
Or there are other possibilities with 2 HDs (keeping 
the redundancy, of course)? 
 
Thanks a lot in advance, 
 
bye, 
 
--  
 
piergiorgio 
-- 
To unsubscribe from this list: send the line "unsubscribe linux-raid" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at  http://vger.kernel.org/majordomo-info.html 
 

Re: Performance question

[Piergiorgio Sartor]

Hi,

thanks for the answer.

Well what I would like to have is exactly a configuration
hint, eventually benchmarks and the like.

The requirements are: two disks, redundacy.
The question is: what configuration is reccommended
in view of performances (or "what can be achieved").

Is that specific enough?

Thanks again,

bye,

pg

On Sat, Jan 17, 2009 at 12:11:00PM -0600, David Lethe wrote:
> All we know is that you use 2 disks and md.  This is like posting to a TCP/IP architecture group and saying you have a network connection and want performance advice.   Read up, supply full config info, run benchmarks, then ask specific questions.  GI=GO.
> -----Original Message-----
> 
> From:  "Piergiorgio Sartor" <piergiorgio.sartor@nexgo.de>
> Subj:  Performance question
> Date:  Sat Jan 17, 2009 11:18 am
> Size:  874 bytes
> To:  "linux-raid@vger.kernel.org" <linux-raid@vger.kernel.org>
> 
> Hi all, 
>  
> I'll have to setup some machines with two HDs (each) 
> in order to get some redundancy. 
>  
> Reading the MD features I noticed there are several 
> possibilities to create a mirror. 
> I was wondering which one offer the best perfomances 
> and/or what are the compromises to accept between 
> the different solutions. 
>  
> One possibility is a classic RAID-1 mirror. 
> Another is a RAID-10 far. 
> There would also be the RAID-10 near, but I guess 
> this is equivalent to RAID-1. 
>  
> Any suggestion on which method offers higher "speed"? 
> Or there are other possibilities with 2 HDs (keeping 
> the redundancy, of course)? 
>  
> Thanks a lot in advance, 
>  
> bye, 
>  
> --  
>  
> piergiorgio 
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in 
> the body of a message to majordomo@vger.kernel.org 
> More majordomo info at  http://vger.kernel.org/majordomo-info.html 
>  
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 

piergiorgio

Re: Performance question

[Bill Davidsen]

Piergiorgio Sartor wrote:
> Hi all,
>
> I'll have to setup some machines with two HDs (each)
> in order to get some redundancy.
>
> Reading the MD features I noticed there are several
> possibilities to create a mirror.
> I was wondering which one offer the best perfomances
> and/or what are the compromises to accept between
> the different solutions.
>
> One possibility is a classic RAID-1 mirror.
> Another is a RAID-10 far.
> There would also be the RAID-10 near, but I guess
> this is equivalent to RAID-1.
>
> Any suggestion on which method offers higher "speed"?
> Or there are other possibilities with 2 HDs (keeping
> the redundancy, of course)?
>   

Mirrored array will offer slower write speed no matter how you do it, 
usually about the speed of a single drive. With raid10 far you should 
get about N times faster read than a single drive, where N is drives in 
the array. Clearly using three or more drives will help a LOT in typical 
performance.

-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

Re: Performance question

[Keld Jørn Simonsen]

On Sat, Jan 17, 2009 at 06:18:06PM +0100, Piergiorgio Sartor wrote:
> Hi all,
> 
> I'll have to setup some machines with two HDs (each)
> in order to get some redundancy.
> 
> Reading the MD features I noticed there are several
> possibilities to create a mirror.
> I was wondering which one offer the best perfomances
> and/or what are the compromises to accept between
> the different solutions.
> 
> One possibility is a classic RAID-1 mirror.
> Another is a RAID-10 far.
> There would also be the RAID-10 near, but I guess
> this is equivalent to RAID-1.

Yes, raid10,n2 is quite the same as raid1 for 2 drives,
That is the disk layout is the same. There may be some 
differences due to the use of different drivers, tho. It was reported at
some time that there were some errors that one of the drivers handled
better than the other. I am not sure which one was the better.
Also syncing and rebuilding etc. may have different performance.

> Any suggestion on which method offers higher "speed"?
> Or there are other possibilities with 2 HDs (keeping
> the redundancy, of course)?

raid10,f2 offers something like double the speed for sequential read,
while probably being a little faster on random read, and with a file
system about equal in performance on writes. Degraded performance (in
tha case that one disk is failing) could be worse for raid10,f2, but in
real life, with the fs elevator in operation, the penalty may be
minimal. IMHO you could normally replace raid1 and raid10,n2, and
raid1+0 with raid10,f2, except for boot devices.

Theoretically there is another possibility in raid5 with 2 drives,
but I am not sure it even works out in practice, and there is imho no
gain in it, except that you can expand the array with more disks.
Furthermore there is raid10,o2 which is viable, but does not
perform as well as raid10,f2.

For linux raid performance have a look at
http://linux-raid.osdl.org/index.php/Performance

For setting up a system with 2 disks so you can survive that one disk
fails, see
http://linux-raid.osdl.org/index.php/Preventing_against_a_failing_disk

I am the main author of both wiki pages, so I am interested in feedback.

Best regards
Keld

Re: Performance question

[Piergiorgio Sartor]

Hi,

thanks for the answer, that was exactly what I
was looking for.

Some feedback for you.
About the performance & benchmarking I've nothing
special to say.
About the setup of two disks, I've some questions,
in no particular order.

The creation of "mdadm.conf" is done by:

mdadm --detail --scan

Somewhere else I found:

mdadm --examine --scan

The two produce different results and the Fedora
installer seems to use the second one.

Which one is really correct? Can we use one or the
other interchangeably?

Second question.
The wiki page does not mention anything about
metadata types.
While it is clear that /boot must have the RAID
header at the end, it is not clear if the RAID-10,f2
could or should have the metadata at the beginning.
In this respect, it would be nice also to have some
clarification about the reccommended metadata version,
i.e. is it better 0.90 or 1.x? Why?

One note. Maybe it could be worth to mention that
further "partitioning" could be done with LVM on top
of the RAID, so only 3 md devices will be needed.

Hope this helps.

Thanks again,

bye,

pg

On Sat, Jan 17, 2009 at 11:08:49PM +0100, Keld Jørn Simonsen wrote:
> On Sat, Jan 17, 2009 at 06:18:06PM +0100, Piergiorgio Sartor wrote:
> > Hi all,
> > 
> > I'll have to setup some machines with two HDs (each)
> > in order to get some redundancy.
> > 
> > Reading the MD features I noticed there are several
> > possibilities to create a mirror.
> > I was wondering which one offer the best perfomances
> > and/or what are the compromises to accept between
> > the different solutions.
> > 
> > One possibility is a classic RAID-1 mirror.
> > Another is a RAID-10 far.
> > There would also be the RAID-10 near, but I guess
> > this is equivalent to RAID-1.
> 
> Yes, raid10,n2 is quite the same as raid1 for 2 drives,
> That is the disk layout is the same. There may be some 
> differences due to the use of different drivers, tho. It was reported at
> some time that there were some errors that one of the drivers handled
> better than the other. I am not sure which one was the better.
> Also syncing and rebuilding etc. may have different performance.
> 
> > Any suggestion on which method offers higher "speed"?
> > Or there are other possibilities with 2 HDs (keeping
> > the redundancy, of course)?
> 
> raid10,f2 offers something like double the speed for sequential read,
> while probably being a little faster on random read, and with a file
> system about equal in performance on writes. Degraded performance (in
> tha case that one disk is failing) could be worse for raid10,f2, but in
> real life, with the fs elevator in operation, the penalty may be
> minimal. IMHO you could normally replace raid1 and raid10,n2, and
> raid1+0 with raid10,f2, except for boot devices.
> 
> Theoretically there is another possibility in raid5 with 2 drives,
> but I am not sure it even works out in practice, and there is imho no
> gain in it, except that you can expand the array with more disks.
> Furthermore there is raid10,o2 which is viable, but does not
> perform as well as raid10,f2.
> 
> For linux raid performance have a look at
> http://linux-raid.osdl.org/index.php/Performance
> 
> For setting up a system with 2 disks so you can survive that one disk
> fails, see
> http://linux-raid.osdl.org/index.php/Preventing_against_a_failing_disk
> 
> I am the main author of both wiki pages, so I am interested in feedback.
> 
> Best regards
> Keld

-- 

piergiorgio
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Performance question

[Keld Jørn Simonsen]

On Mon, Jan 19, 2009 at 07:12:53PM +0100, Piergiorgio Sartor wrote:
> Hi,
> 
> thanks for the answer, that was exactly what I
> was looking for.

Good!

> Some feedback for you.
> About the performance & benchmarking I've nothing
> special to say.
> About the setup of two disks, I've some questions,
> in no particular order.
> 
> The creation of "mdadm.conf" is done by:
> 
> mdadm --detail --scan
> 
> Somewhere else I found:
> 
> mdadm --examine --scan
> 
> The two produce different results and the Fedora
> installer seems to use the second one.
> 
> Which one is really correct? Can we use one or the
> other interchangeably?

--detail looks at the running arrays, while --examine most
likely (depending on mdadm.conf) looks at all partitions
on the system. 

Given that the arrays are just created in the installation process, and
the active running arrays are most likely the ones you want your system
to know of, I think --detail is the better. --examine does on two of my
systems generate info that are in conflict and not suitable for a
mdadm.conf file, such as two /dev/md1 with different UUIDs.

> Second question.
> The wiki page does not mention anything about
> metadata types.
> While it is clear that /boot must have the RAID
> header at the end, it is not clear if the RAID-10,f2
> could or should have the metadata at the beginning.
> In this respect, it would be nice also to have some
> clarification about the reccommended metadata version,
> i.e. is it better 0.90 or 1.x? Why?

To me it does not matter that much, except for the booting device.
Each partition in the booting device must look like a normal (ext3)
partition, as grub and lilo does not know of raids, and just treats
a booting partition as a standalone partition. So here you should use
0.90 metadata, which is put at the end of the array.

For other arrays I think one important choice is if you have an array
greater than 2 TiB to not use 0.90 metadata, as this has a limit of 2
TiB.

> One note. Maybe it could be worth to mention that
> further "partitioning" could be done with LVM on top
> of the RAID, so only 3 md devices will be needed.

yes, I have been looking into that. Maybe I will add some words on this.

> Hope this helps.


yes, thanks for your feedback!

best regards
keld

Re: Performance question

[Richard Scobie]

Keld Jørn Simonsen wrote:

> For other arrays I think one important choice is if you have an array
> greater than 2 TiB to not use 0.90 metadata, as this has a limit of 2
> TiB.

This restriction only applies if the individual members of the array are 
larger than 2TB each.

Regards,

Richard
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Performance question

[Piergiorgio Sartor]

Hi again,

[--detail vs. --examine]
> --detail looks at the running arrays, while --examine most
> likely (depending on mdadm.conf) looks at all partitions
> on the system. 
> 
> Given that the arrays are just created in the installation process, and
> the active running arrays are most likely the ones you want your system
> to know of, I think --detail is the better. --examine does on two of my
> systems generate info that are in conflict and not suitable for a
> mdadm.conf file, such as two /dev/md1 with different UUIDs.

yes, but I noticed that with "--detail" and an
array (RAID-1) resyincing, it reports "spares=1"
too, while when the array is in sync, it prints
the correct geometry.
So, I was wondering, since I also noticed that
"--examine" produces the arrays with /dev/md/"name",
so if two arrays have same name, it ends up with
the same device.
Is this maybe a bug of mdadm?

[metadata position]
> To me it does not matter that much, except for the booting device.
> Each partition in the booting device must look like a normal (ext3)
> partition, as grub and lilo does not know of raids, and just treats
> a booting partition as a standalone partition. So here you should use
> 0.90 metadata, which is put at the end of the array.

Well, I was a bit mixing up things with this question.
In the back of my head the question was:

What about performances, RAID-10 f2, bitmap (important)
and metadata 1.0 vs. 1.1?

This could be a further test for performances. It would
be interesting to know if it is better to have the
metadata at the beginning or at the end of a RAID-10 f2,
with two HDs, having the bitmap enabled.
Or if it does not matter at all.

Reading around I found different "opinions" about bitmap
and performances, but I did not find a "convincing" test.

Thanks again.

Different item of the wiki, I run into it today.
Maybe the "initrd" description could be updated, since
it uses "mdassemble", while the "initrd" I have uses
directly "mdadm -As --auto=yes ..." (I do not remember
the full line).

Hope this helps,

bye,

-- 

piergiorgio

Re: Performance question

[Keld Jørn Simonsen]

On Wed, Jan 21, 2009 at 08:14:52PM +0100, Piergiorgio Sartor wrote:
> Hi again,
> 
> [--detail vs. --examine]
> > --detail looks at the running arrays, while --examine most
> > likely (depending on mdadm.conf) looks at all partitions
> > on the system. 
> > 
> > Given that the arrays are just created in the installation process, and
> > the active running arrays are most likely the ones you want your system
> > to know of, I think --detail is the better. --examine does on two of my
> > systems generate info that are in conflict and not suitable for a
> > mdadm.conf file, such as two /dev/md1 with different UUIDs.
> 
> yes, but I noticed that with "--detail" and an
> array (RAID-1) resyincing, it reports "spares=1"
> too, while when the array is in sync, it prints
> the correct geometry.
> So, I was wondering, since I also noticed that
> "--examine" produces the arrays with /dev/md/"name",
> so if two arrays have same name, it ends up with
> the same device.
> Is this maybe a bug of mdadm?

I leave this to others to answer this one.
I think it is strange for --detail to report "spares=1"
if it is syncing.

> [metadata position]
> > To me it does not matter that much, except for the booting device.
> > Each partition in the booting device must look like a normal (ext3)
> > partition, as grub and lilo does not know of raids, and just treats
> > a booting partition as a standalone partition. So here you should use
> > 0.90 metadata, which is put at the end of the array.
> 
> Well, I was a bit mixing up things with this question.
> In the back of my head the question was:
> 
> What about performances, RAID-10 f2, bitmap (important)
> and metadata 1.0 vs. 1.1?
> 
> This could be a further test for performances. It would
> be interesting to know if it is better to have the
> metadata at the beginning or at the end of a RAID-10 f2,
> with two HDs, having the bitmap enabled.
> Or if it does not matter at all.
> 
> Reading around I found different "opinions" about bitmap
> and performances, but I did not find a "convincing" test.

I have not tested it. So yes, I think this is something to do a performance test
on.  I think it should not matter much whether it is in the beginning or in
the end. However, if you make a test, then you most likely will do it on
a newly created raid, and then files would tend to be allocated in the
beginning of the file system, thus favouring a metadata block in the
beginning of the raid. In real operation this will tend to even out.
Another issue is that the sectors in the beginning of a disk are much 
faster, a factor of two perhaps, than the sectors in the end of the
drive.

> Thanks again.
> 
> Different item of the wiki, I run into it today.
> Maybe the "initrd" description could be updated, since
> it uses "mdassemble", while the "initrd" I have uses
> directly "mdadm -As --auto=yes ..." (I do not remember
> the full line).

mdasseble is specifically made for initrd, so why not use it here?

Best regards
keld

Re: Performance question

[Piergiorgio Sartor]

Hi,

thanks for the explanation about metadata.

> mdasseble is specifically made for initrd, so why not use it here?

I do not know, I just noticed that, on Fedora,
the initrd with RAID has /etc/mdadm.conf and it
calls "mdadm -As ...".
Which I found annoying, since I do not know
what will happen in case an array is changed
(UUID change, /etc/mdadm.conf not more consistent).

Anyway, if you say "mdassemble" is OK, no problem.

Thanks,

bye,

-- 

piergiorgio