raid1 does not seem faster

raid1 does not seem faster

[Jan Engelhardt]

Hello list,


normally, I'd think that combining drives into a raid1 array would give 
me at least a little improvement in read speed. In my setup however, 
this does not seem to be the case.

14:16 opteron:/var/log # hdparm -t /dev/sda
 Timing buffered disk reads:  170 MB in  3.01 seconds =  56.52 MB/sec
14:17 opteron:/var/log # hdparm -t /dev/md3
 Timing buffered disk reads:  170 MB in  3.01 seconds =  56.45 MB/sec
(and dd_rescue shows the same numbers)

The raid array was created using
# mdadm -C /dev/md3 -b internal -e 1.0 -l 1 -n 2 /dev/sd[ab]3


Jan
-- 

Re: raid1 does not seem faster

[Henrik Holst]

On Sun, 2007-04-01 at 14:19 +0200, Jan Engelhardt wrote:
> Hello list,
> 
> 
> normally, I'd think that combining drives into a raid1 array would give 
> me at least a little improvement in read speed. In my setup however, 
> this does not seem to be the case.
> 
> 14:16 opteron:/var/log # hdparm -t /dev/sda
>  Timing buffered disk reads:  170 MB in  3.01 seconds =  56.52 MB/sec
> 14:17 opteron:/var/log # hdparm -t /dev/md3
>  Timing buffered disk reads:  170 MB in  3.01 seconds =  56.45 MB/sec
> (and dd_rescue shows the same numbers)
> 
> The raid array was created using
> # mdadm -C /dev/md3 -b internal -e 1.0 -l 1 -n 2 /dev/sd[ab]3
> 
> 
> Jan

From section 9.5 in [FAQ]

"To check out speed and performance of your RAID systems, do NOT use
hdparm. It won't do real benchmarking of the arrays." <snip>

I might recommend bonnie++. I think I've seen accepted benchmarks here
using bonnie++.

[FAQ] http://tldp.org/HOWTO/html_single/Software-RAID-HOWTO/#s9

-- 
Henrik Holst <holst@matmech.com>

Re: raid1 does not seem faster

[Al Boldi]

Jan Engelhardt wrote:
> normally, I'd think that combining drives into a raid1 array would give
> me at least a little improvement in read speed. In my setup however,
> this does not seem to be the case.
>
> 14:16 opteron:/var/log # hdparm -t /dev/sda
>  Timing buffered disk reads:  170 MB in  3.01 seconds =  56.52 MB/sec
> 14:17 opteron:/var/log # hdparm -t /dev/md3
>  Timing buffered disk reads:  170 MB in  3.01 seconds =  56.45 MB/sec
> (and dd_rescue shows the same numbers)

The problem is that raid1 one doesn't do striped reads, but rather uses 
read-balancing per proc.  Try your test with parallel reads; it should be 
faster.

You could use raid10, but then you loose single-disk-image compatibility.


Thanks!

--
Al

Re: raid1 does not seem faster

[Al Boldi]

Bill Davidsen wrote:
> Al Boldi wrote:
> > The problem is that raid1 one doesn't do striped reads, but rather uses
> > read-balancing per proc.  Try your test with parallel reads; it should
> > be faster.
:
:
> It would be nice if reads larger than some size were considered as
> candidates for multiple devices. By setting the readahead larger than
> that value speed increases would be noted for sequential access.

Actually, that's what I thought for a long time too, but as Neil once pointed 
out, for striped reads to be efficient, each chunk should be located 
sequentially, as to avoid any seeks.  This is only possible by introducing 
some offset layout, as in raid10, which infers a loss of raid1's 
single-disk-image compatibility.

What could be feasible, is some kind of an initial burst striped readahead, 
which could possibly improve small reads < (readahead * nr_of_disks).


Thanks!

--
Al

Re: raid1 does not seem faster

[Bill Davidsen]

Al Boldi wrote:
> Bill Davidsen wrote:
>   
>> Al Boldi wrote:
>>     
>>> The problem is that raid1 one doesn't do striped reads, but rather uses
>>> read-balancing per proc.  Try your test with parallel reads; it should
>>> be faster.
>>>       
> :
> :
>   
>> It would be nice if reads larger than some size were considered as
>> candidates for multiple devices. By setting the readahead larger than
>> that value speed increases would be noted for sequential access.
>>     
>
> Actually, that's what I thought for a long time too, but as Neil once pointed 
> out, for striped reads to be efficient, each chunk should be located 
> sequentially, as to avoid any seeks.  This is only possible by introducing 
> some offset layout, as in raid10, which infers a loss of raid1's 
> single-disk-image compatibility.
>   
I can't imaging that the offset needs to be physical, there's a 
translation done from the chunk address on the array to the physical 
address on the drive, and beyond that usually a translation from the 
logical position in a partition to a physical LBA location on the drive 
as a whole.
> What could be feasible, is some kind of an initial burst striped readahead, 
> which could possibly improve small reads < (readahead * nr_of_disks).
You are correct, but I think if an optimization were to be done, some balance between the read time, seek time, and read size could be done. Using more than one drive only makes sense when the read transfer time is significantly longer than the seek time. With an aggressive readahead set for the array that would happen regularly.

It's possible, it just takes the time to do it, like many other "nice" things.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: raid1 does not seem faster

[Iustin Pop]

On Wed, Apr 04, 2007 at 07:11:50PM -0400, Bill Davidsen wrote:
> You are correct, but I think if an optimization were to be done, some 
> balance between the read time, seek time, and read size could be done. 
> Using more than one drive only makes sense when the read transfer time is 
> significantly longer than the seek time. With an aggressive readahead set 
> for the array that would happen regularly.
> 
> It's possible, it just takes the time to do it, like many other "nice" 
> things.

Maybe yes, but why optimise the single-reader case? raid1 already can
read in parallel from the drives when multiple processes read from the
raid1. Optimising the single reader can help in hdparm or other
benchmark cases, but in real life I see very often the total throughput
of a (two drive) raid1 being around two times the throughput of a single
drive.

regards,
iustin

Re: raid1 does not seem faster

[Justin Piszcz]

On Thu, 5 Apr 2007, Iustin Pop wrote:

> On Wed, Apr 04, 2007 at 07:11:50PM -0400, Bill Davidsen wrote:
>> You are correct, but I think if an optimization were to be done, some
>> balance between the read time, seek time, and read size could be done.
>> Using more than one drive only makes sense when the read transfer time is
>> significantly longer than the seek time. With an aggressive readahead set
>> for the array that would happen regularly.
>>
>> It's possible, it just takes the time to do it, like many other "nice"
>> things.
>
> Maybe yes, but why optimise the single-reader case? raid1 already can
> read in parallel from the drives when multiple processes read from the
> raid1. Optimising the single reader can help in hdparm or other
> benchmark cases, but in real life I see very often the total throughput
> of a (two drive) raid1 being around two times the throughput of a single
> drive.
>
> regards,
> iustin
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Really? I have copied a file from a SW RAID1 (5GB) and I only saw 
60MB/s not the 120MB/s the (RAID1) is capable of to the 
destination (which can easily do > 160MB/s sustained read/write).

Justin.

Re: raid1 does not seem faster

[Bill Davidsen]

Iustin Pop wrote:
> On Wed, Apr 04, 2007 at 07:11:50PM -0400, Bill Davidsen wrote:
>   
>> You are correct, but I think if an optimization were to be done, some 
>> balance between the read time, seek time, and read size could be done. 
>> Using more than one drive only makes sense when the read transfer time is 
>> significantly longer than the seek time. With an aggressive readahead set 
>> for the array that would happen regularly.
>>
>> It's possible, it just takes the time to do it, like many other "nice" 
>> things.
>>     
>
> Maybe yes, but why optimise the single-reader case? raid1 already can
> read in parallel from the drives when multiple processes read from the
> raid1. Optimising the single reader can help in hdparm or other
> benchmark cases, but in real life I see very often the total throughput
> of a (two drive) raid1 being around two times the throughput of a single
> drive.

Why optimize the single thread case? Any process which has low CPU 
requirements (by current standards) becomes i/o bound. The obvious 
candidates are grep, sed, dd, or awk. And don't overlook the benefits of 
reliable swap.

Test script modifications taking place as I type, I post the script and 
some results later this week, barring urgent support issues.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: raid1 does not seem faster

[Iustin Pop]

On Thu, Apr 05, 2007 at 04:11:35AM -0400, Justin Piszcz wrote:
> 
> 
> On Thu, 5 Apr 2007, Iustin Pop wrote:
> 
> >On Wed, Apr 04, 2007 at 07:11:50PM -0400, Bill Davidsen wrote:
> >>You are correct, but I think if an optimization were to be done, some
> >>balance between the read time, seek time, and read size could be done.
> >>Using more than one drive only makes sense when the read transfer time is
> >>significantly longer than the seek time. With an aggressive readahead set
> >>for the array that would happen regularly.
> >>
> >>It's possible, it just takes the time to do it, like many other "nice"
> >>things.
> >
> >Maybe yes, but why optimise the single-reader case? raid1 already can
> >read in parallel from the drives when multiple processes read from the
> >raid1. Optimising the single reader can help in hdparm or other
> >benchmark cases, but in real life I see very often the total throughput
> >of a (two drive) raid1 being around two times the throughput of a single
> >drive.
> >
> >regards,
> >iustin
> >-
> >To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >the body of a message to majordomo@vger.kernel.org
> >More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 
> Really? I have copied a file from a SW RAID1 (5GB) and I only saw 
> 60MB/s not the 120MB/s the (RAID1) is capable of to the 
> destination (which can easily do > 160MB/s sustained read/write).

Did you copy it multi-threaded? I said "*multiple-readers* show improved
speed" and you said "I copied *one* file". Try copying two files in
parallel.

I'm doing in two xterms "cat file1 >/dev/null", "cat file2 >/dev/null"
and my raid1 shows ~110 MB/s, each drive doing about half.  On file only
does about 60 MB/s (this is over a PCI raid controller so the max 110
MB/s is a PCI bus limitation).

Iustin

Re: raid1 does not seem faster

[Iustin Pop]

On Thu, Apr 05, 2007 at 09:45:52AM -0400, Bill Davidsen wrote:
> Iustin Pop wrote:
> >On Wed, Apr 04, 2007 at 07:11:50PM -0400, Bill Davidsen wrote:
> >  
> >>You are correct, but I think if an optimization were to be done, some 
> >>balance between the read time, seek time, and read size could be done. 
> >>Using more than one drive only makes sense when the read transfer time is 
> >>significantly longer than the seek time. With an aggressive readahead set 
> >>for the array that would happen regularly.
> >>
> >>It's possible, it just takes the time to do it, like many other "nice" 
> >>things.
> >>    
> >
> >Maybe yes, but why optimise the single-reader case? raid1 already can
> >read in parallel from the drives when multiple processes read from the
> >raid1. Optimising the single reader can help in hdparm or other
> >benchmark cases, but in real life I see very often the total throughput
> >of a (two drive) raid1 being around two times the throughput of a single
> >drive.
> 
> Why optimize the single thread case? Any process which has low CPU 
> requirements (by current standards) becomes i/o bound. The obvious 
> candidates are grep, sed, dd, or awk. And don't overlook the benefits of 
> reliable swap.

"Premature optimisation is the root of all evil". Of course it's
possible to do it. But is it worth it? In terms of code complexity, of
possible bugs, etc. And what do you gain? raid0 performance and
mirroring? it's already available in raid10.

Reliable swap? Yes, I do that over raid1. But if raid raw I/O speed is
the bottleneck, the solution is add more ram instead of optimising
raid1.

And honestly, I like the raid1 code to be as simple as possible. If you
really find grep, sed, dd to be a bottleneck, you need to think if the
tools you are using are the right tools. As I said, raid1 shows good
read speed if using *more that one process doing I/O at a time*. Two
dd's in parallel are using two drives.

Iustin

Re: raid1 does not seem faster

[Justin Piszcz]

On Thu, 5 Apr 2007, Iustin Pop wrote:

> On Thu, Apr 05, 2007 at 04:11:35AM -0400, Justin Piszcz wrote:
>>
>>
>> On Thu, 5 Apr 2007, Iustin Pop wrote:
>>
>>> On Wed, Apr 04, 2007 at 07:11:50PM -0400, Bill Davidsen wrote:
>>>> You are correct, but I think if an optimization were to be done, some
>>>> balance between the read time, seek time, and read size could be done.
>>>> Using more than one drive only makes sense when the read transfer time is
>>>> significantly longer than the seek time. With an aggressive readahead set
>>>> for the array that would happen regularly.
>>>>
>>>> It's possible, it just takes the time to do it, like many other "nice"
>>>> things.
>>>
>>> Maybe yes, but why optimise the single-reader case? raid1 already can
>>> read in parallel from the drives when multiple processes read from the
>>> raid1. Optimising the single reader can help in hdparm or other
>>> benchmark cases, but in real life I see very often the total throughput
>>> of a (two drive) raid1 being around two times the throughput of a single
>>> drive.
>>>
>>> regards,
>>> iustin
>>> -
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>> Really? I have copied a file from a SW RAID1 (5GB) and I only saw
>> 60MB/s not the 120MB/s the (RAID1) is capable of to the
>> destination (which can easily do > 160MB/s sustained read/write).
>
> Did you copy it multi-threaded? I said "*multiple-readers* show improved
> speed" and you said "I copied *one* file". Try copying two files in
> parallel.
>
> I'm doing in two xterms "cat file1 >/dev/null", "cat file2 >/dev/null"
> and my raid1 shows ~110 MB/s, each drive doing about half.  On file only
> does about 60 MB/s (this is over a PCI raid controller so the max 110
> MB/s is a PCI bus limitation).
>
> Iustin
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Ah, nope, I only did a single copy, I did not parallelize the operations.

Justin.

Re: raid1 does not seem faster

[Bill Davidsen]

Justin Piszcz wrote:
>
>
> On Thu, 5 Apr 2007, Iustin Pop wrote:
>
>> On Thu, Apr 05, 2007 at 04:11:35AM -0400, Justin Piszcz wrote:
>>>
>>>
>>> On Thu, 5 Apr 2007, Iustin Pop wrote:
>>>
>>>> On Wed, Apr 04, 2007 at 07:11:50PM -0400, Bill Davidsen wrote:
>>>>> You are correct, but I think if an optimization were to be done, some
>>>>> balance between the read time, seek time, and read size could be 
>>>>> done.
>>>>> Using more than one drive only makes sense when the read transfer 
>>>>> time is
>>>>> significantly longer than the seek time. With an aggressive 
>>>>> readahead set
>>>>> for the array that would happen regularly.
>>>>>
>>>>> It's possible, it just takes the time to do it, like many other 
>>>>> "nice"
>>>>> things.
>>>>
>>>> Maybe yes, but why optimise the single-reader case? raid1 already can
>>>> read in parallel from the drives when multiple processes read from the
>>>> raid1. Optimising the single reader can help in hdparm or other
>>>> benchmark cases, but in real life I see very often the total 
>>>> throughput
>>>> of a (two drive) raid1 being around two times the throughput of a 
>>>> single
>>>> drive.
>>>>
>>>> regards,
>>>> iustin
>>>> -
>>>> To unsubscribe from this list: send the line "unsubscribe 
>>>> linux-raid" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>
>>> Really? I have copied a file from a SW RAID1 (5GB) and I only saw
>>> 60MB/s not the 120MB/s the (RAID1) is capable of to the
>>> destination (which can easily do > 160MB/s sustained read/write).
>>
>> Did you copy it multi-threaded? I said "*multiple-readers* show improved
>> speed" and you said "I copied *one* file". Try copying two files in
>> parallel.
>>
>> I'm doing in two xterms "cat file1 >/dev/null", "cat file2 >/dev/null"
>> and my raid1 shows ~110 MB/s, each drive doing about half.  On file only
>> does about 60 MB/s (this is over a PCI raid controller so the max 110
>> MB/s is a PCI bus limitation).
>>
>> Iustin
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> Ah, nope, I only did a single copy, I did not parallelize the operations.
And it would be nice not to have to jump through hoops to get better 
performance... You can implement raid[01] at the user level, too, why 
bother to have it in the kernel? ;-)

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: raid1 does not seem faster

[Justin Piszcz]

On Thu, 5 Apr 2007, Justin Piszcz wrote:

>
>
> On Thu, 5 Apr 2007, Iustin Pop wrote:
>
>> On Wed, Apr 04, 2007 at 07:11:50PM -0400, Bill Davidsen wrote:
>>> You are correct, but I think if an optimization were to be done, some
>>> balance between the read time, seek time, and read size could be done.
>>> Using more than one drive only makes sense when the read transfer time is
>>> significantly longer than the seek time. With an aggressive readahead set
>>> for the array that would happen regularly.
>>> 
>>> It's possible, it just takes the time to do it, like many other "nice"
>>> things.
>> 
>> Maybe yes, but why optimise the single-reader case? raid1 already can
>> read in parallel from the drives when multiple processes read from the
>> raid1. Optimising the single reader can help in hdparm or other
>> benchmark cases, but in real life I see very often the total throughput
>> of a (two drive) raid1 being around two times the throughput of a single
>> drive.
>> 
>> regards,
>> iustin
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 
>
> Really? I have copied a file from a SW RAID1 (5GB) and I only saw 60MB/s not 
> the 120MB/s the (RAID1) is capable of to the destination (which can easily do 
>> 160MB/s sustained read/write).
>
> Justin.
>

Using 2 threads made no difference either.

It was not until I did 3 simultaneous copies that I saw 110-130MB/s 
through vmstat 1, until then, it only used one drive, even with two cp's, 
how come it needs to be three or more?

Justin.

Re: raid1 does not seem faster

[Iustin Pop]

On Mon, Apr 09, 2007 at 06:53:26AM -0400, Justin Piszcz wrote:
> Using 2 threads made no difference either.
> 
> It was not until I did 3 simultaneous copies that I saw 110-130MB/s 
> through vmstat 1, until then, it only used one drive, even with two cp's, 
> how come it needs to be three or more?

Because, as I understand it, it's an optimisation, not a rule. Quoting
from the manpage (md):
"Once initialised, each device in a RAID1 array contains exactly the
same data.  Changes are written to all devices in parallel.  Data is
read from any one device.  The driver attempts to distribute read
requests across all devices to maximise performance."

The key word here is "attempts". I looked a while ago over the source
code and IIRC it says that it tries to direct a read request to the
drive whose head is closest to the requested sector, or if not possible
a random drive. To me, this seems a good strategy, which optimises
server-type workloads.

regards,
iustin