(was: Re: md: raid5 vs raid10 (f2,n2,o2) benchmarks [w/10 raptors])

(was: Re: md: raid5 vs raid10 (f2,n2,o2) benchmarks [w/10 raptors])

[Beolach]

On Sat, Mar 29, 2008 at 12:20 PM, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
<snip>
>
>  Results:
>  http://home.comcast.net/~jpiszcz/20080329-raid/
>
>  I have gone back to my RAID 5 optimized configuration, I am done testing
>  for now :)
>
>  __CREATION
>  1. Test RAID 10 with no optimizations to the disks or filesystems.
>     a. mdadm --create /dev/md3 --assume-clean --chunk=256 --level=raid10 --raid-devices=10 --spare-devices=0 --layout=f2 /dev/sd[c-l]1
>     b. mdadm --create /dev/md3 --assume-clean --chunk=256 --level=raid10 --raid-devices=10 --spare-devices=0 --layout=n2 /dev/sd[c-l]1
>     c. mdadm --create /dev/md3 --assume-clean --chunk=256 --level=raid10 --raid-devices=10 --spare-devices=0 --layout=o2 /dev/sd[c-l]1
>  2. Test RAID 5 with no optimizations as well using the default layout.
>     a. mdadm --create /dev/md3 --assume-clean --chunk=256 --level=raid5 --raid-devices=10 --spare-devices=0 /dev/sd[c-l]1
>  3. Test RAID 5 with optimizations.
>     a. mdadm --create /dev/md3 --assume-clean --chunk=1024 --level=raid5 --raid-devices=10 --spare-devices=0 /dev/sd[c-l]1
>
>  __SETUP
>  2. Format and set permissions and run benchmarks.
>     a. mkfs.xfs -f /dev/md3; mount /dev/md3 /r1; mkdir /r1/x
>        chown -R jpiszcz:users /r1
>
>  __TEST
>  3. Run the following bonnie++ benchmark 3 times and take the average.
>     a. /usr/bin/time /usr/sbin/bonnie++ -d /x/test -s 16384 -m p34 -n 16:100000:16:64
>     b. A script will be responsible for running it 3 times and the total time
>        of all runs will also be taken.
>
<snip>

Thanks for sharing your benchmarking methods.  I've been running
similar benchmarks, trying to pick the best chunksize for my hardware,
and I also found your previous thread "Fastest Chunk Size w/XFS For MD
Software RAID = 1024k" in the archives.  In my benchmarks[1], I see
256k & 128k giving the best results.  So now I'm wondering if that's
just because I have different harddrives (Seagate ST31000340NS 1000GB
32MB cache), or if the number of drives is also important - I only
have 4 drives right now, but I plan on getting more when I can afford
to.  If I had more drives, would the larger chunksizes (512k or 1024k)
be more likely to perform better than 256k?

Also, would you be willing to share your script for averaging 3
bonnie++ runs?  I'm too lazy to write my own, so I've just been doing
single runs.

[1] My bonnie++ results:
<http://www.xmission.com/~beolach/bonnie++_4disk-ls.html>


Thanks,
Conway S. Smith

Re: (was: Re: md: raid5 vs raid10 (f2,n2,o2) benchmarks [w/10 raptors])

[Justin Piszcz]

On Tue, 1 Apr 2008, Beolach wrote:

> On Sat, Mar 29, 2008 at 12:20 PM, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
> <snip>
>
> Thanks for sharing your benchmarking methods.  I've been running
> similar benchmarks, trying to pick the best chunksize for my hardware,
> and I also found your previous thread "Fastest Chunk Size w/XFS For MD
> Software RAID = 1024k" in the archives.  In my benchmarks[1], I see
> 256k & 128k giving the best results.  So now I'm wondering if that's
> just because I have different harddrives (Seagate ST31000340NS 1000GB
> 32MB cache), or if the number of drives is also important - I only
> have 4 drives right now, but I plan on getting more when I can afford
> to.  If I had more drives, would the larger chunksizes (512k or 1024k)
> be more likely to perform better than 256k?
>
> Also, would you be willing to share your script for averaging 3
> bonnie++ runs?  I'm too lazy to write my own, so I've just been doing
> single runs.
I do not have a single script to do it actually, it works like this:

# Run bonnie 3 times (script).
for i in 1 2 3
do
   /usr/bin/time /usr/sbin/bonnie++ -d /x/test -s 16384 -m p34 -n 16:100000:16:64 > $HOME/test"$i".txt 2>&1
done

# then get the results
$ cat test* | grep ,
p34,16G,80170,99,261521,43,109222,14,82516,99,527121,39,864.3,1,16:100000:16/6411428,83,+++++,+++,6603,30,7780,56,+++++,+++,8959,45
p34,16G,79428,99,266452,44,111190,14,82087,99,535667,39,884.3,1,16:100000:16/643388,26,+++++,+++,7185,34,6364,46,+++++,+++,4040,22
p34,16G,78346,99,255350,42,111591,14,82153,99,527210,38,850.4,1,16:100000:16/642916,21,+++++,+++,18495,81,5614,41,+++++,+++,15727,83

$ cat test* | grep , > results

$ avgbonnie results 
p34,16G,79314.7,99,261108,43,110668,14,82252,99,529999,38.6667,866.333,1,16:100000:16/64,5910.67,43.3333,0,0,10761,48.3333,6586,47.6667,0,0,9575.33,50

Nothing special for the average, just a long awk statement hardcoded for 3
runs:

grep ',' "$1" | awk -F',' '{print $1, $2, c += $3/3, d += $4/3, e += $5/3, f += $6/3, g += $7/3, h += $8/3, i += $9/3, j += $10/3, k += $11/3, l += $12/3, m += $13/3, n += $14/3, $15, p += $16/3, q += $17/3, r += $18/3, s += $19/3, t += $20/3, u += $21/3, v += $22/3, w += $23/3, x += $24/3, y += $25/3, z += $26/3, aa += $27/3}' | tail -n 1 | sed 's/\ /,/g'

$ grep ',' results | awk -F',' '{print $1, $2, c += $3/3, d += $4/3, e += $5/3, f += $6/3, g += $7/3, h += $8/3, i += $9/3, j += $10/3, k += $11/3, l += $12/3, m += $13/3, n += $14/3, $15, p += $16/3, q += $17/3, r += $18/3, s += $19/3, t += $20/3, u += $21/3, v += $22/3, w += $23/3, x += $24/3, y += $25/3, z += $26/3, aa += $27/3}' | tail -n 1 | sed 's/\ /,/g'
p34,16G,79314.7,99,261108,43,110668,14,82252,99,529999,38.6667,866.333,1,16:100000:16/64,5910.67,43.3333,0,0,10761,48.3333,6586,47.6667,0,0,9575.33,50

Hope this helps..

>
> [1] My bonnie++ results:
> <http://www.xmission.com/~beolach/bonnie++_4disk-ls.html>
Intriguing results you have there, nice sequential read speed.
What FS are you using?
Any special options?
What read-ahead are you using?
What is your stripe_cache_size?
These heavily affect performance.

Justin.

Re: (was: Re: md: raid5 vs raid10 (f2,n2,o2) benchmarks [w/10 raptors])

[Conway S. Smith]

On Wed, 2 Apr 2008 08:16:16 -0400 (EDT)
Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
> 
> 
> On Tue, 1 Apr 2008, Beolach wrote:
> 
<snip>
> >
> > Also, would you be willing to share your script for averaging 3
> > bonnie++ runs?  I'm too lazy to write my own, so I've just been
> > doing single runs.
> 
> I do not have a single script to do it actually, it works like this:
> 
> # Run bonnie 3 times (script).
> for i in 1 2 3
> do
>    /usr/bin/time /usr/sbin/bonnie++ -d /x/test -s 16384 -m p34 -n
> 16:100000:16:64 > $HOME/test"$i".txt 2>&1 done
> 
> # then get the results
> $ cat test* | grep ,
> p34,16G,80170,99,261521,43,109222,14,82516,99,527121,39,864.3,1,16:100000:16/6411428,83,+++++,+++,6603,30,7780,56,+++++,+++,8959,45
> p34,16G,79428,99,266452,44,111190,14,82087,99,535667,39,884.3,1,16:100000:16/643388,26,+++++,+++,7185,34,6364,46,+++++,+++,4040,22
> p34,16G,78346,99,255350,42,111591,14,82153,99,527210,38,850.4,1,16:100000:16/642916,21,+++++,+++,18495,81,5614,41,+++++,+++,15727,83
> 
> $ cat test* | grep , > results
> 
> $ avgbonnie results 
> p34,16G,79314.7,99,261108,43,110668,14,82252,99,529999,38.6667,866.333,1,16:100000:16/64,5910.67,43.3333,0,0,10761,48.3333,6586,47.6667,0,0,9575.33,50
> 
> Nothing special for the average, just a long awk statement
> hardcoded for 3 runs:
> 
> grep ',' "$1" | awk -F',' '{print $1, $2, c += $3/3, d += $4/3, e
> += $5/3, f += $6/3, g += $7/3, h += $8/3, i += $9/3, j += $10/3, k
> += $11/3, l += $12/3, m += $13/3, n += $14/3, $15, p += $16/3, q +=
> $17/3, r += $18/3, s += $19/3, t += $20/3, u += $21/3, v += $22/3,
> w += $23/3, x += $24/3, y += $25/3, z += $26/3, aa += $27/3}' |
> tail -n 1 | sed 's/\ /,/g'
> 
> $ grep ',' results | awk -F',' '{print $1, $2, c += $3/3, d +=
> $4/3, e += $5/3, f += $6/3, g += $7/3, h += $8/3, i += $9/3, j +=
> $10/3, k += $11/3, l += $12/3, m += $13/3, n += $14/3, $15, p +=
> $16/3, q += $17/3, r += $18/3, s += $19/3, t += $20/3, u += $21/3,
> v += $22/3, w += $23/3, x += $24/3, y += $25/3, z += $26/3, aa +=
> $27/3}' | tail -n 1 | sed 's/\ /,/g'
> p34,16G,79314.7,99,261108,43,110668,14,82252,99,529999,38.6667,866.333,1,16:100000:16/64,5910.67,43.3333,0,0,10761,48.3333,6586,47.6667,0,0,9575.33,50
> 
> Hope this helps..
> 

Thanks! Although now I'll have to get around to learning awk so I can
understand that. ;-)

> >
> > [1] My bonnie++ results:
> > <http://www.xmission.com/~beolach/bonnie++_4disk-ls.html>
> Intriguing results you have there, nice sequential read speed.
> What FS are you using?
> Any special options?

XFS, no special mkfs options, noatime,nodiratime mount options.

> What read-ahead are you using?
> What is your stripe_cache_size?
> These heavily affect performance.
> 

I haven't tried tweaking these yet.  Are they likely to change which
chunksize performs best?  I was thinking I'd figure out a chunksize,
and then look at other performance tweaks.  But I'm worried that I
might later find out a different chunksize would have been better,
and chunksize is much harder to change than read-ahead.

$ blockdev --getra /dev/md1
3072
$ cat /sys/block/md1/md/stripe_cache_size
256


Thanks,
Conway S. Smith

Re: (was: Re: md: raid5 vs raid10 (f2,n2,o2) benchmarks [w/10 raptors])

[Justin Piszcz]

On Wed, 2 Apr 2008, Conway S. Smith wrote:

> On Wed, 2 Apr 2008 08:16:16 -0400 (EDT)
> Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>>
>>
>> On Tue, 1 Apr 2008, Beolach wrote:
>>
> <snip>
>>>
>>> Also, would you be willing to share your script for averaging 3
>>> bonnie++ runs?  I'm too lazy to write my own, so I've just been
>>> doing single runs.
>>
>> I do not have a single script to do it actually, it works like this:
>>
>> # Run bonnie 3 times (script).
>> for i in 1 2 3
>> do
>>    /usr/bin/time /usr/sbin/bonnie++ -d /x/test -s 16384 -m p34 -n
>> 16:100000:16:64 > $HOME/test"$i".txt 2>&1 done
>>
>> # then get the results
>> $ cat test* | grep ,
>> p34,16G,80170,99,261521,43,109222,14,82516,99,527121,39,864.3,1,16:100000:16/6411428,83,+++++,+++,6603,30,7780,56,+++++,+++,8959,45
>> p34,16G,79428,99,266452,44,111190,14,82087,99,535667,39,884.3,1,16:100000:16/643388,26,+++++,+++,7185,34,6364,46,+++++,+++,4040,22
>> p34,16G,78346,99,255350,42,111591,14,82153,99,527210,38,850.4,1,16:100000:16/642916,21,+++++,+++,18495,81,5614,41,+++++,+++,15727,83
>>
>> $ cat test* | grep , > results
>>
>> $ avgbonnie results
>> p34,16G,79314.7,99,261108,43,110668,14,82252,99,529999,38.6667,866.333,1,16:100000:16/64,5910.67,43.3333,0,0,10761,48.3333,6586,47.6667,0,0,9575.33,50
>>
>> Nothing special for the average, just a long awk statement
>> hardcoded for 3 runs:
>>
>> grep ',' "$1" | awk -F',' '{print $1, $2, c += $3/3, d += $4/3, e
>> += $5/3, f += $6/3, g += $7/3, h += $8/3, i += $9/3, j += $10/3, k
>> += $11/3, l += $12/3, m += $13/3, n += $14/3, $15, p += $16/3, q +=
>> $17/3, r += $18/3, s += $19/3, t += $20/3, u += $21/3, v += $22/3,
>> w += $23/3, x += $24/3, y += $25/3, z += $26/3, aa += $27/3}' |
>> tail -n 1 | sed 's/\ /,/g'
>>
>> $ grep ',' results | awk -F',' '{print $1, $2, c += $3/3, d +=
>> $4/3, e += $5/3, f += $6/3, g += $7/3, h += $8/3, i += $9/3, j +=
>> $10/3, k += $11/3, l += $12/3, m += $13/3, n += $14/3, $15, p +=
>> $16/3, q += $17/3, r += $18/3, s += $19/3, t += $20/3, u += $21/3,
>> v += $22/3, w += $23/3, x += $24/3, y += $25/3, z += $26/3, aa +=
>> $27/3}' | tail -n 1 | sed 's/\ /,/g'
>> p34,16G,79314.7,99,261108,43,110668,14,82252,99,529999,38.6667,866.333,1,16:100000:16/64,5910.67,43.3333,0,0,10761,48.3333,6586,47.6667,0,0,9575.33,50
>>
>> Hope this helps..
>>
>
> Thanks! Although now I'll have to get around to learning awk so I can
> understand that. ;-)
>
>>>
>>> [1] My bonnie++ results:
>>> <http://www.xmission.com/~beolach/bonnie++_4disk-ls.html>
>> Intriguing results you have there, nice sequential read speed.
>> What FS are you using?
>> Any special options?
>
> XFS, no special mkfs options, noatime,nodiratime mount options.
>
>> What read-ahead are you using?
>> What is your stripe_cache_size?
>> These heavily affect performance.
>>
>
> I haven't tried tweaking these yet.  Are they likely to change which
> chunksize performs best?  I was thinking I'd figure out a chunksize,
> and then look at other performance tweaks.  But I'm worried that I
> might later find out a different chunksize would have been better,
> and chunksize is much harder to change than read-ahead.
>
> $ blockdev --getra /dev/md1
> 3072
> $ cat /sys/block/md1/md/stripe_cache_size
> 256
The stripe_cache_size will make a huge difference with RAID5, try up to 
32768.

256,512,1024,2048,4096,16384,32768

The sweet spot is 16384 for my config.

>
>
> Thanks,
> Conway S. Smith
>

Re: (was: Re: md: raid5 vs raid10 (f2,n2,o2) benchmarks [w/10 raptors])

[Justin Piszcz]

On Wed, 2 Apr 2008, Justin Piszcz wrote:

>
>
> On Wed, 2 Apr 2008, Conway S. Smith wrote:
>
>> On Wed, 2 Apr 2008 08:16:16 -0400 (EDT)
>> Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>>> 
>>> 
>>> On Tue, 1 Apr 2008, Beolach wrote:
>>> 
>> <snip>
>>>> 
>>>> Also, would you be willing to share your script for averaging 3
>>>> bonnie++ runs?  I'm too lazy to write my own, so I've just been
>>>> doing single runs.
>>> 
>>> I do not have a single script to do it actually, it works like this:
>>> 
>>> # Run bonnie 3 times (script).
>>> for i in 1 2 3
>>> do
>>>    /usr/bin/time /usr/sbin/bonnie++ -d /x/test -s 16384 -m p34 -n
>>> 16:100000:16:64 > $HOME/test"$i".txt 2>&1 done
>>> 
>>> # then get the results
>>> $ cat test* | grep ,
>>> p34,16G,80170,99,261521,43,109222,14,82516,99,527121,39,864.3,1,16:100000:16/6411428,83,+++++,+++,6603,30,7780,56,+++++,+++,8959,45
>>> p34,16G,79428,99,266452,44,111190,14,82087,99,535667,39,884.3,1,16:100000:16/643388,26,+++++,+++,7185,34,6364,46,+++++,+++,4040,22
>>> p34,16G,78346,99,255350,42,111591,14,82153,99,527210,38,850.4,1,16:100000:16/642916,21,+++++,+++,18495,81,5614,41,+++++,+++,15727,83
>>> 
>>> $ cat test* | grep , > results
>>> 
>>> $ avgbonnie results
>>> p34,16G,79314.7,99,261108,43,110668,14,82252,99,529999,38.6667,866.333,1,16:100000:16/64,5910.67,43.3333,0,0,10761,48.3333,6586,47.6667,0,0,9575.33,50
>>> 
>>> Nothing special for the average, just a long awk statement
>>> hardcoded for 3 runs:
>>> 
>>> grep ',' "$1" | awk -F',' '{print $1, $2, c += $3/3, d += $4/3, e
>>> += $5/3, f += $6/3, g += $7/3, h += $8/3, i += $9/3, j += $10/3, k
>>> += $11/3, l += $12/3, m += $13/3, n += $14/3, $15, p += $16/3, q +=
>>> $17/3, r += $18/3, s += $19/3, t += $20/3, u += $21/3, v += $22/3,
>>> w += $23/3, x += $24/3, y += $25/3, z += $26/3, aa += $27/3}' |
>>> tail -n 1 | sed 's/\ /,/g'
>>> 
>>> $ grep ',' results | awk -F',' '{print $1, $2, c += $3/3, d +=
>>> $4/3, e += $5/3, f += $6/3, g += $7/3, h += $8/3, i += $9/3, j +=
>>> $10/3, k += $11/3, l += $12/3, m += $13/3, n += $14/3, $15, p +=
>>> $16/3, q += $17/3, r += $18/3, s += $19/3, t += $20/3, u += $21/3,
>>> v += $22/3, w += $23/3, x += $24/3, y += $25/3, z += $26/3, aa +=
>>> $27/3}' | tail -n 1 | sed 's/\ /,/g'
>>> p34,16G,79314.7,99,261108,43,110668,14,82252,99,529999,38.6667,866.333,1,16:100000:16/64,5910.67,43.3333,0,0,10761,48.3333,6586,47.6667,0,0,9575.33,50
>>> 
>>> Hope this helps..
>>> 
>> 
>> Thanks! Although now I'll have to get around to learning awk so I can
>> understand that. ;-)
>> 
>>>> 
>>>> [1] My bonnie++ results:
>>>> <http://www.xmission.com/~beolach/bonnie++_4disk-ls.html>
>>> Intriguing results you have there, nice sequential read speed.
>>> What FS are you using?
>>> Any special options?
>> 
>> XFS, no special mkfs options, noatime,nodiratime mount options.
>> 
>>> What read-ahead are you using?
>>> What is your stripe_cache_size?
>>> These heavily affect performance.
>>> 
>> 
>> I haven't tried tweaking these yet.  Are they likely to change which
>> chunksize performs best?  I was thinking I'd figure out a chunksize,
>> and then look at other performance tweaks.  But I'm worried that I
>> might later find out a different chunksize would have been better,
>> and chunksize is much harder to change than read-ahead.
>> 
>> $ blockdev --getra /dev/md1
>> 3072
>> $ cat /sys/block/md1/md/stripe_cache_size
>> 256
> The stripe_cache_size will make a huge difference with RAID5, try up to 
> 32768.
>
> 256,512,1024,2048,4096,16384,32768
>
> The sweet spot is 16384 for my config.
>
>> 
>> 
>> Thanks,
>> Conway S. Smith
>> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Here is my config:

# Set read-ahead.
echo "Setting read-ahead to 64 MiB for /dev/md3"
blockdev --setra 65536 /dev/md3

# Set stripe-cache_size for RAID5.
echo "Setting stripe_cache_size to 16 MiB for /dev/md3"
echo 16384 > /sys/block/md3/md/stripe_cache_size

# Disable NCQ on all disks. (for raptors it increases the speed 30-40MiB/s)
echo "Disabling NCQ on all disks..."
for i in $DISKS
do
   echo "Disabling NCQ on $i"
   echo 1 > /sys/block/"$i"/device/queue_depth
done

Re: (was: Re: md: raid5 vs raid10 (f2,n2,o2) benchmarks [w/10 raptors])

[Keld Jørn Simonsen]

On Wed, Apr 02, 2008 at 01:49:44PM -0400, Justin Piszcz wrote:
> 
> 
> On Wed, 2 Apr 2008, Justin Piszcz wrote:
> 
> >
> >
> >On Wed, 2 Apr 2008, Conway S. Smith wrote:

I have referenced both of your benchmaks in the wiki on performance. So
now I just hope that your URLs will live forever. I also took down some
of your recomendations there. 

I note that raid10,f2 has a much higher cpu load than raid10,n2 or
raid10,o2. How come? it is 31-38 % for f2, where n2 and o2 is around 15 %.

Also I note that the sequential read rate for o2 is nowhere near that of
n2, it is more like half of the n2 speed. I have seen claims from
normally trustworthy sources that o2 would obtain striping speeds.
IMHO theory would also say that o2 could not be striping.

And the optimized version from Justin impedes performance for f2, by
around 90 MB/s. How come?

best regards
keld

Re: (was: Re: md: raid5 vs raid10 (f2,n2,o2) benchmarks [w/10 raptors])

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1219 bytes --]



On Thu, 3 Apr 2008, Keld Jørn Simonsen wrote:

> On Wed, Apr 02, 2008 at 01:49:44PM -0400, Justin Piszcz wrote:
>>
>>
>> On Wed, 2 Apr 2008, Justin Piszcz wrote:
>>
>>>
>>>
>>> On Wed, 2 Apr 2008, Conway S. Smith wrote:
>
> I have referenced both of your benchmaks in the wiki on performance. So
> now I just hope that your URLs will live forever. I also took down some
> of your recomendations there.
>
> I note that raid10,f2 has a much higher cpu load than raid10,n2 or
> raid10,o2. How come? it is 31-38 % for f2, where n2 and o2 is around 15 %
>
> Also I note that the sequential read rate for o2 is nowhere near that of
> n2, it is more like half of the n2 speed. I have seen claims from
> normally trustworthy sources that o2 would obtain striping speeds.
> IMHO theory would also say that o2 could not be striping.
>
> And the optimized version from Justin impedes performance for f2, by
> around 90 MB/s. How come?
This may be due to the optimizations I use (for RAID5) in general they 
apply to RAID10 somewhat but in some cases it impedes performance greately 
where the opposite is true of RAID5, if the optimizations are not in 
place, its nearly twice as slow.

Justin.

Re: (was: Re: md: raid5 vs raid10 (f2,n2,o2) benchmarks [w/10 raptors])

[Keld Jørn Simonsen]

On Thu, Apr 03, 2008 at 12:02:46PM +0200, Keld Jørn Simonsen wrote:
> On Wed, Apr 02, 2008 at 01:49:44PM -0400, Justin Piszcz wrote:
> > 
> > 
> > On Wed, 2 Apr 2008, Justin Piszcz wrote:
> > 
> > >
> > >
> > >On Wed, 2 Apr 2008, Conway S. Smith wrote:
> 
> I have referenced both of your benchmaks in the wiki on performance. So
> now I just hope that your URLs will live forever. I also took down some
> of your recomendations there. 
> 
> I note that raid10,f2 has a much higher cpu load than raid10,n2 or
> raid10,o2. How come? it is 31-38 % for f2, where n2 and o2 is around 15 %.

I found a reason for this, it seems that CPU usage and IO speed are very
related, so because the raid10,f2 has about double the IO performance
for sequential reading, it also has about double the cpu use.

Justin's benchmark is on http://home.comcast.net/~jpiszcz/20080329-raid/

Another of Justin's benchmarks also reveals the relation between
IO rate and CPU use:
http://home.comcast.net/~jpiszcz/raid/20080528/raid-levels.html

Why does IO use that much CPU? Is it mostly moving around the data from
the kernel to the user space? Does it matter here whether one is running
a 32 bit or a 64 bit system?

It seems like the RAM bus can be a bottleneck. I read that DDR-400
can have a peak performance of 1600 MB/s. If this is halved on 32 bit OS
then this is 800 MB/s. And you need to both read and write when you move
things around. So that is 400 MB/s... And you need to still be able to 
read in from the disk controller at 330 MB/s. For 64 bit systems this is
a max around 400 MB/s - given that there is a flow from the disk
controller to system disk buffers, then from kernel buffers to user
buffers and then from user buffers to some processing. Or am I wrong
here?

Best regards
Keld
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: (was: Re: md: raid5 vs raid10 (f2,n2,o2) benchmarks [w/10 raptors])

[David Lethe]

-----Original Message-----
From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Keld Jørn Simonsen
Sent: Wednesday, June 25, 2008 1:49 PM
To: Justin Piszcz
Cc: Conway S. Smith; linux-raid@vger.kernel.org
Subject: Re: (was: Re: md: raid5 vs raid10 (f2,n2,o2) benchmarks [w/10 raptors])

> On Thu, Apr 03, 2008 at 12:02:46PM +0200, Keld Jørn Simonsen wrote:
> > On Wed, Apr 02, 2008 at 01:49:44PM -0400, Justin Piszcz wrote:
> > > 
> > > 
> > > On Wed, 2 Apr 2008, Justin Piszcz wrote:
> > > 
> > > >
> > > >
> > > >On Wed, 2 Apr 2008, Conway S. Smith wrote:
> > 
> > I have referenced both of your benchmaks in the wiki on performance. So
> > now I just hope that your URLs will live forever. I also took down some
> > of your recomendations there. 
> > 
> > I note that raid10,f2 has a much higher cpu load than raid10,n2 or
> > raid10,o2. How come? it is 31-38 % for f2, where n2 and o2 is around 15 %.
>
> I found a reason for this, it seems that CPU usage and IO speed are very
> related, so because the raid10,f2 has about double the IO performance
> for sequential reading, it also has about double the cpu use.
>
> Justin's benchmark is on http://home.comcast.net/~jpiszcz/20080329-raid/
>
> Another of Justin's benchmarks also reveals the relation between
> IO rate and CPU use:
> http://home.comcast.net/~jpiszcz/raid/20080528/raid-levels.html
>
> Why does IO use that much CPU? Is it mostly moving around the data from
> the kernel to the user space? Does it matter here whether one is running
> a 32 bit or a 64 bit system?

> It seems like the RAM bus can be a bottleneck. I read that DDR-400
> can have a peak performance of 1600 MB/s. If this is halved on 32 bit OS
> then this is 800 MB/s. And you need to both read and write when you move
> things around. So that is 400 MB/s... And you need to still be able to 
> read in from the disk controller at 330 MB/s. For 64 bit systems this is
> a max around 400 MB/s - given that there is a flow from the disk
> controller to system disk buffers, then from kernel buffers to user
> buffers and then from user buffers to some processing. Or am I wrong
> here?
>
> Best regards
> Keld

These chunk sizes are profoundly meaningless if you plan on using them to estimate performance in the real world.   The relationship between IO rate, IO throughput, and CPU overhead will be dramatically different with default md settings.  Also consider that your kernel was recompiled with little or no cpu-specific optimization, so you are wasting cpu cycles .. and don't get me started on multicore vs. single core for such benchmarks.


- David


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: (was: Re: md: raid5 vs raid10 (f2,n2,o2) benchmarks [w/10 raptors])

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1058 bytes --]



On Wed, 25 Jun 2008, David Lethe wrote:

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Keld Jørn Simonsen
> Sent: Wednesday, June 25, 2008 1:49 PM
> To: Justin Piszcz
> Cc: Conway S. Smith; linux-raid@vger.kernel.org
> Subject: Re: (was: Re: md: raid5 vs raid10 (f2,n2,o2) benchmarks [w/10 raptors])
>
>
> These chunk sizes are profoundly meaningless if you plan on using them to estimate performance in the real world.   The relationship between IO rate, IO throughput, and CPU overhead will be dramatically different with default md settings.  Also consider that your kernel was recompiled with little or no cpu-specific optimization, so you are wasting cpu cycles .. and don't get me started on multicore vs. single core for such benchmarks.
>
>
> - David
>
>

1. In RAID5, these chunk sizes offered the best performance with bonnie++.
2. The CPU type in the config is optimized for the CPU in the host.
3. The host also runs IRQ balance.

Justin.

Re: (was: Re: md: raid5 vs raid10 (f2,n2,o2) benchmarks [w/10 raptors])

[Keld Jørn Simonsen]

On Wed, Jun 25, 2008 at 02:09:32PM -0500, David Lethe wrote:
> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Keld Jørn Simonsen
> Sent: Wednesday, June 25, 2008 1:49 PM
> To: Justin Piszcz
> Cc: Conway S. Smith; linux-raid@vger.kernel.org
> Subject: Re: (was: Re: md: raid5 vs raid10 (f2,n2,o2) benchmarks [w/10 raptors])
> 
> > On Thu, Apr 03, 2008 at 12:02:46PM +0200, Keld Jørn Simonsen wrote:
> > > On Wed, Apr 02, 2008 at 01:49:44PM -0400, Justin Piszcz wrote:
> > > > 
> > > > 
> > > > On Wed, 2 Apr 2008, Justin Piszcz wrote:
> > > > 
> > > > >
> > > > >
> > > > >On Wed, 2 Apr 2008, Conway S. Smith wrote:
> > > 
> > > I have referenced both of your benchmaks in the wiki on performance. So
> > > now I just hope that your URLs will live forever. I also took down some
> > > of your recomendations there. 
> > > 
> > > I note that raid10,f2 has a much higher cpu load than raid10,n2 or
> > > raid10,o2. How come? it is 31-38 % for f2, where n2 and o2 is around 15 %.
> >
> > I found a reason for this, it seems that CPU usage and IO speed are very
> > related, so because the raid10,f2 has about double the IO performance
> > for sequential reading, it also has about double the cpu use.
> >
> > Justin's benchmark is on http://home.comcast.net/~jpiszcz/20080329-raid/
> >
> > Another of Justin's benchmarks also reveals the relation between
> > IO rate and CPU use:
> > http://home.comcast.net/~jpiszcz/raid/20080528/raid-levels.html
> >
> > Why does IO use that much CPU? Is it mostly moving around the data from
> > the kernel to the user space? Does it matter here whether one is running
> > a 32 bit or a 64 bit system?
> 
> > It seems like the RAM bus can be a bottleneck. I read that DDR-400
> > can have a peak performance of 1600 MB/s. If this is halved on 32 bit OS
> > then this is 800 MB/s. And you need to both read and write when you move
> > things around. So that is 400 MB/s... And you need to still be able to 
> > read in from the disk controller at 330 MB/s. For 64 bit systems this is
> > a max around 400 MB/s - given that there is a flow from the disk
> > controller to system disk buffers, then from kernel buffers to user
> > buffers and then from user buffers to some processing. Or am I wrong
> > here?
> >
> > Best regards
> > Keld
> 
> These chunk sizes are profoundly meaningless if you plan on using them to estimate performance in the real world.   The relationship between IO rate, IO throughput, and CPU overhead will be dramatically different with default md settings.  Also consider that your kernel was recompiled with little or no cpu-specific optimization, so you are wasting cpu cycles .. and don't get me started on multicore vs. single core for such benchmarks.

I think something is wrong with the chunk sizes. 16 G and 7 G are most
likely erroneous. Justin?

David, if you have some benchmarks then please feel free to report them.

Best regards
keld
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: (was: Re: md: raid5 vs raid10 (f2,n2,o2) benchmarks [w/10 raptors])

[Dan Williams]

On Wed, Apr 2, 2008 at 10:44 AM, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>
>
>
>  On Wed, 2 Apr 2008, Conway S. Smith wrote:
>
>
> > On Wed, 2 Apr 2008 08:16:16 -0400 (EDT)
> > Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
> >
> > >
> > >
> > > On Tue, 1 Apr 2008, Beolach wrote:
> > >
> > >
> > <snip>
> >
> > >
> > > >
> > > > Also, would you be willing to share your script for averaging 3
> > > > bonnie++ runs?  I'm too lazy to write my own, so I've just been
> > > > doing single runs.
> > > >
> > >
> > > I do not have a single script to do it actually, it works like this:
> > >
> > > # Run bonnie 3 times (script).
> > > for i in 1 2 3
> > > do
> > >   /usr/bin/time /usr/sbin/bonnie++ -d /x/test -s 16384 -m p34 -n
> > > 16:100000:16:64 > $HOME/test"$i".txt 2>&1 done
> > >
> > > # then get the results
> > > $ cat test* | grep ,
> > >
> p34,16G,80170,99,261521,43,109222,14,82516,99,527121,39,864.3,1,16:100000:16/6411428,83,+++++,+++,6603,30,7780,56,+++++,+++,8959,45
> > >
> p34,16G,79428,99,266452,44,111190,14,82087,99,535667,39,884.3,1,16:100000:16/643388,26,+++++,+++,7185,34,6364,46,+++++,+++,4040,22
> > >
> p34,16G,78346,99,255350,42,111591,14,82153,99,527210,38,850.4,1,16:100000:16/642916,21,+++++,+++,18495,81,5614,41,+++++,+++,15727,83
> > >
> > > $ cat test* | grep , > results
> > >
> > > $ avgbonnie results
> > >
> p34,16G,79314.7,99,261108,43,110668,14,82252,99,529999,38.6667,866.333,1,16:100000:16/64,5910.67,43.3333,0,0,10761,48.3333,6586,47.6667,0,0,9575.33,50
> > >
> > > Nothing special for the average, just a long awk statement
> > > hardcoded for 3 runs:
> > >
> > > grep ',' "$1" | awk -F',' '{print $1, $2, c += $3/3, d += $4/3, e
> > > += $5/3, f += $6/3, g += $7/3, h += $8/3, i += $9/3, j += $10/3, k
> > > += $11/3, l += $12/3, m += $13/3, n += $14/3, $15, p += $16/3, q +=
> > > $17/3, r += $18/3, s += $19/3, t += $20/3, u += $21/3, v += $22/3,
> > > w += $23/3, x += $24/3, y += $25/3, z += $26/3, aa += $27/3}' |
> > > tail -n 1 | sed 's/\ /,/g'
> > >
> > > $ grep ',' results | awk -F',' '{print $1, $2, c += $3/3, d +=
> > > $4/3, e += $5/3, f += $6/3, g += $7/3, h += $8/3, i += $9/3, j +=
> > > $10/3, k += $11/3, l += $12/3, m += $13/3, n += $14/3, $15, p +=
> > > $16/3, q += $17/3, r += $18/3, s += $19/3, t += $20/3, u += $21/3,
> > > v += $22/3, w += $23/3, x += $24/3, y += $25/3, z += $26/3, aa +=
> > > $27/3}' | tail -n 1 | sed 's/\ /,/g'
> > >
> p34,16G,79314.7,99,261108,43,110668,14,82252,99,529999,38.6667,866.333,1,16:100000:16/64,5910.67,43.3333,0,0,10761,48.3333,6586,47.6667,0,0,9575.33,50
> > >
> > > Hope this helps..
> > >
> > >
> >
> > Thanks! Although now I'll have to get around to learning awk so I can
> > understand that. ;-)
> >
> >
> > >
> > > >
> > > > [1] My bonnie++ results:
> > > > <http://www.xmission.com/~beolach/bonnie++_4disk-ls.html>
> > > >
> > > Intriguing results you have there, nice sequential read speed.
> > > What FS are you using?
> > > Any special options?
> > >
> >
> > XFS, no special mkfs options, noatime,nodiratime mount options.
> >
> >
> > > What read-ahead are you using?
> > > What is your stripe_cache_size?
> > > These heavily affect performance.
> > >
> > >
> >
> > I haven't tried tweaking these yet.  Are they likely to change which
> > chunksize performs best?  I was thinking I'd figure out a chunksize,
> > and then look at other performance tweaks.  But I'm worried that I
> > might later find out a different chunksize would have been better,
> > and chunksize is much harder to change than read-ahead.
> >
> > $ blockdev --getra /dev/md1
> > 3072
> > $ cat /sys/block/md1/md/stripe_cache_size
> > 256
> >
>  The stripe_cache_size will make a huge difference with RAID5, try up to
> 32768.

That's 512MB for a 4-disk array.  Hopefully the get_priority_stripe
patch alleviates the pressure to push stripe_cache_size this high.

--
Dan