raid10,f2 degraded read speed

raid10,f2 degraded read speed

[Keld Jørn Simonsen]

I found this old message, from before I joined the list, in the
archives.

Jon Nelson wrote  2007-12-23 14:23:40 GMT:
> On 12/23/07, maobo <maobo1983 <at> gmail.com> wrote:
> > Hi,all
> >
> > Yes, I agree some of you. But in my test both using real life trace
> > and
> > Iometer test I found that for absolutely read requests, RAID0 is
> > better than
> > RAID10 (with same data disks: 3 disks in RAID0, 6 disks in RAID10). I
> > don't
> > know why this happen.
> >
> > I read the code of RAID10 and RAID0 carefully and experiment with
> > printk to
> > track the process flow. The only conclusion I report is the complexity
> > of
> > RAID10 to process the read request. While for RAID0 it is so simple
> > that it
> > does the read more effectively.
> >
> > How do you think about this of absolutely read requests?
> > Thank you very much!
> 
> My own tests on identical hardware (same mobo, disks, partitions,
> everything) and same software, with the only difference being how
> mdadm is invoked (the only changes here being level and possibly
> layout) show that raid0 is about 15% faster on reads than the very
> fast raid10, f2 layout. raid10,f2 is approx. 50% of the write speed of
> raid0.
> 
> Does this make sense?

I am not sure. What are the real figures? 50 % of the *degraded* write
speed? or normal wrute speed? Measured as actual on the disk or
effective in the file system?

degraded raid10,f2 read speed and write speed should be the same, eg on
a 2-disk setup.  And effectiveliy the rate should be like random IO on a
single disk. There should be no substantial difference between sequential
and random read nor write on a degraded raid10,f2.

Or maybe the elevator is making tricks?

-----

I also find it interesting that raid10,f2 should be 15 % slower than 
raid0 (given the same disk size, or half the disk size? Would the
raid10,f2 file system be say 500 GB and the raid0 1 TB?) - could that
be due to CPU overhead? It seems like the overhead is substantial, maybe
10 % on raid10,f2 while it is 0 % on raid0.

I only found a 3 % reduction (150 MB/s vs 155 MB/s) in my reported
benchmark for raid10,f2 vs raid0.

Justin Piszcz reported with a 6 disk setup a bonnie++ test for seq read:
                   kB/s    cpu
  raid0          286240   21.33
  raid10,f2      335520	  26.33

http://home.comcast.net/~jpiszcz/raid/20080528/raid-levels.html

That is actually a 17 % improvement of raid10,f2 over raid0.

I repoted a teoretical overall improvement of 17 % of raid10,f2 compared to
raid0, due to disk geometry, and raid10,f2 only using half the space.

This bechmarks says 21 % cpu use for raid0, and 26 % cpu use for
raid10,f2. I wonder why all this pricessing is needed, and whether it
actually affects IO performance, or is actually carried out in parallel
with the IO.


----

I think I once found some benchmark (from Jon?) maybe on a Suse page
including various degraded raid10 arrays, even expressed in percentages
of normal one-disk performance. Can sombody provide a link?


Best regards
keld

Re: raid10,f2 degraded read speed

[Keld Jørn Simonsen]

On Tue, Jun 17, 2008 at 04:37:45PM +0200, Keld Jørn Simonsen wrote:
> 
> I think I once found some benchmark (from Jon?) maybe on a Suse page
> including various degraded raid10 arrays, even expressed in percentages
> of normal one-disk performance. Can sombody provide a link?

I found the link:
http://pycurious.blogspot.com/2007/12/some-raid10-performance-numbers.html

Is this Jon Nelson writing this?

I added the link + some benchmark from Bill Davidsen to the wiki pages
on performance.

Best regards
Keld
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raid10,f2 degraded read speed

[Jon Nelson]

Yep that's me.


On Wed, Jun 25, 2008 at 12:25 PM, Keld Jørn Simonsen <keld@dkuug.dk> wrote:
> On Tue, Jun 17, 2008 at 04:37:45PM +0200, Keld Jørn Simonsen wrote:
>>
>> I think I once found some benchmark (from Jon?) maybe on a Suse page
>> including various degraded raid10 arrays, even expressed in percentages
>> of normal one-disk performance. Can sombody provide a link?
>
> I found the link:
> http://pycurious.blogspot.com/2007/12/some-raid10-performance-numbers.html
>
> Is this Jon Nelson writing this?
>
> I added the link + some benchmark from Bill Davidsen to the wiki pages
> on performance.

Neat! I'll probably be going through a rather more extensive set of tests soon.
The problem for me is that I have to test on a deployed system.
The load is insanely low, but I can't exactly try "now do it without
the LVM layer.".

-- 
Jon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raid10,f2 degraded read speed

[Keld Jørn Simonsen]

On Wed, Jun 25, 2008 at 12:31:25PM -0500, Jon Nelson wrote:
> Yep that's me.

Good! I added your name in the wiki.

> > I found the link:
> > http://pycurious.blogspot.com/2007/12/some-raid10-performance-numbers.html
> >
> > Is this Jon Nelson writing this?
> >
> > I added the link + some benchmark from Bill Davidsen to the wiki pages
> > on performance.
> 
> Neat! I'll probably be going through a rather more extensive set of tests soon.
> The problem for me is that I have to test on a deployed system.
> The load is insanely low, but I can't exactly try "now do it without
> the LVM layer.".

I think we should actually have more figures from systems in production. 
The problem is of cause then what is measured. But is sometimes gives a
much better idea of what can be achieved. For example I have a Linux
mirror site with a raid10,f2 array of 4 1 TB disks, and I have maybe 100
concurrent proftpd processes. The fs produces something like 30 MB/s -
but that is not peak performance. I can then cat an ISO file with
about 60 - 100 MB/s - that is not bad. But the system then only delivers 
something like 90 - 130 MB/s where peak seq reading is 320 MB/s in idle
mode. Hmm, I think I normally get something like 200 MB/s for the
sequential read of one file, while there is some other activity, and I
am running a number of heavier administrative tasks right now...

But the point is: how do we measure production performance?
My best guess is that we can only give anedotical evidence, and report
of the admins feel of the system. 



Nice to have more benchmarks. Maybe you can write up something for the
wiki? 

For the wiki I tend to just make links to the articles with performance results,
instead of integrating them in the wiki. And that probably does not make 
due description as all the data on the test sometimes is not readily
referenced in the link.

Anyway what we should do for the wiki is kind of a canonical benchmark
results, so that it is easy to read and not repeated a lot of times, at
least on the main performance page. I think it is fine to have more
tests, maybe on subpages. And then I fear that some of the links will go
away, I would rather have the info incorporated into the wiki.

And to stand the teeth of time, I would like to have performance
expressed in terms of relative performance of raw read speed for one
disk, this figure is most likely very stable even over a 10 years'
timespan.

Best regards
keld

Re: raid10,f2 degraded read speed

[Jon Nelson]

On Thu, Jun 26, 2008 at 3:17 AM, Keld Jørn Simonsen <keld@dkuug.dk> wrote:
> I think we should actually have more figures from systems in production.
..
> Nice to have more benchmarks. Maybe you can write up something for the
> wiki?
..

Funny you should mention that - I was speaking with a colleague
recently and I told him that I'm going to try for a fairly
comprehensive test of raid5, raid10[n2,o2,f2] in both regular and
degraded mode, utilizing fstest to test ext3, jfs, xfs, and maybe
spadfs. I've got most of it scripted too so some refinements to those
scripts might be useful to somebody else. I plan on putting this on my
blog when I get the time.

However, I'm rather pressed for time right now so it will likely be a while.

> And to stand the teeth of time, I would like to have performance
> expressed in terms of relative performance of raw read speed for one
> disk, this figure is most likely very stable even over a 10 years'
> timespan.

I agree.

-- 
Jon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raid10,f2 degraded read speed

[Keld Jørn Simonsen]

On Thu, Jun 26, 2008 at 07:35:20AM -0500, Jon Nelson wrote:
> On Thu, Jun 26, 2008 at 3:17 AM, Keld Jørn Simonsen <keld@dkuug.dk> wrote:
> > I think we should actually have more figures from systems in production.
> ..
> > Nice to have more benchmarks. Maybe you can write up something for the
> > wiki?
> ..
> 
> Funny you should mention that - I was speaking with a colleague
> recently and I told him that I'm going to try for a fairly
> comprehensive test of raid5, raid10[n2,o2,f2] in both regular and
> degraded mode, utilizing fstest to test ext3, jfs, xfs, and maybe
> spadfs. I've got most of it scripted too so some refinements to those
> scripts might be useful to somebody else. I plan on putting this on my
> blog when I get the time.
> 
> However, I'm rather pressed for time right now so it will likely be a while.

Such is life. So then, when this is out, could I then use your findings
for the wiki?

I think the wiki is a good place to put up such tests, as more people
would have a chance finding the results there.

best regards
Keld
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raid10,f2 degraded read speed

[Jon Nelson]

On Thu, Jun 26, 2008 at 8:37 AM, Keld Jørn Simonsen <keld@dkuug.dk> wrote:
> Such is life. So then, when this is out, could I then use your findings
> for the wiki?

I don't see why not. My findings are just as useless as everybody else's.

-- 
Jon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raid10,f2 degraded read speed

[Keld Jørn Simonsen]

On Thu, Jun 26, 2008 at 08:38:47AM -0500, Jon Nelson wrote:
> On Thu, Jun 26, 2008 at 8:37 AM, Keld Jørn Simonsen <keld@dkuug.dk> wrote:
> > Such is life. So then, when this is out, could I then use your findings
> > for the wiki?
> 
> I don't see why not. My findings are just as useless as everybody else's.

Nah, I actually think that your findings were some of the more
pedagogical, and useful of the benchmarks I have seen on Linux RAID.
You did a comprehensive test of the interesting raid types, had it
converted to a factor of raw read speed, and even had graphics. 
Quite instructive.

So cheers up, grumpy middelaged man! I look forward to your new tests.
keld
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html