below 10MB/s write on raid5

below 10MB/s write on raid5

[Dexter Filmore]

I recently upgraded my file server, yet I'm still unsatisfied with the write 
speed.
Machine now is a Athlon64 3400+ (Socket 754) equipped with 1GB of RAM.
The four RAID disks are attached to the board's onbaord sATA controller 
(Sil3114 attached via PCI)
Kernel is 2.6.21.1, custom on Slackware 11.0.
RAID is on four Samsung SpinPoint disks, has LVM, 3 volumes atop of each XFS.

The machine does some other work, too, but still I would have suspected to get 
into the 20-30MB/s area. Too much asked for?

Dex

-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d--(+)@ s-:+ a- C++++ UL++ P+>++ L+++>++++ E-- W++ N o? K-
w--(---) !O M+ V- PS+ PE Y++ PGP t++(---)@ 5 X+(++) R+(++) tv--(+)@ 
b++(+++) DI+++ D- G++ e* h>++ r* y?
------END GEEK CODE BLOCK------

http://www.stop1984.com
http://www.againsttcpa.com

Re: below 10MB/s write on raid5

[Justin Piszcz]

On Mon, 11 Jun 2007, Dexter Filmore wrote:

> I recently upgraded my file server, yet I'm still unsatisfied with the write
> speed.
> Machine now is a Athlon64 3400+ (Socket 754) equipped with 1GB of RAM.
> The four RAID disks are attached to the board's onbaord sATA controller
> (Sil3114 attached via PCI)
> Kernel is 2.6.21.1, custom on Slackware 11.0.
> RAID is on four Samsung SpinPoint disks, has LVM, 3 volumes atop of each XFS.
>
> The machine does some other work, too, but still I would have suspected to get
> into the 20-30MB/s area. Too much asked for?
>
> Dex

What do you get without LVM?

Re: below 10MB/s write on raid5

[Dexter Filmore]

On Monday 11 June 2007 14:47:50 Justin Piszcz wrote:
> On Mon, 11 Jun 2007, Dexter Filmore wrote:
> > I recently upgraded my file server, yet I'm still unsatisfied with the
> > write speed.
> > Machine now is a Athlon64 3400+ (Socket 754) equipped with 1GB of RAM.
> > The four RAID disks are attached to the board's onbaord sATA controller
> > (Sil3114 attached via PCI)
> > Kernel is 2.6.21.1, custom on Slackware 11.0.
> > RAID is on four Samsung SpinPoint disks, has LVM, 3 volumes atop of each
> > XFS.
> >
> > The machine does some other work, too, but still I would have suspected
> > to get into the 20-30MB/s area. Too much asked for?
> >
> > Dex
>
> What do you get without LVM?

Hard to tell: the PV hogs all of the disk space, can't really do non-LVM 
tests.



-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d--(+)@ s-:+ a- C++++ UL++ P+>++ L+++>++++ E-- W++ N o? K-
w--(---) !O M+ V- PS+ PE Y++ PGP t++(---)@ 5 X+(++) R+(++) tv--(+)@ 
b++(+++) DI+++ D- G++ e* h>++ r* y?
------END GEEK CODE BLOCK------

http://www.stop1984.com
http://www.againsttcpa.com

Re: below 10MB/s write on raid5

[Justin Piszcz]

On Mon, 11 Jun 2007, Dexter Filmore wrote:

> On Monday 11 June 2007 14:47:50 Justin Piszcz wrote:
>> On Mon, 11 Jun 2007, Dexter Filmore wrote:
>>> I recently upgraded my file server, yet I'm still unsatisfied with the
>>> write speed.
>>> Machine now is a Athlon64 3400+ (Socket 754) equipped with 1GB of RAM.
>>> The four RAID disks are attached to the board's onbaord sATA controller
>>> (Sil3114 attached via PCI)
>>> Kernel is 2.6.21.1, custom on Slackware 11.0.
>>> RAID is on four Samsung SpinPoint disks, has LVM, 3 volumes atop of each
>>> XFS.
>>>
>>> The machine does some other work, too, but still I would have suspected
>>> to get into the 20-30MB/s area. Too much asked for?
>>>
>>> Dex
>>
>> What do you get without LVM?
>
> Hard to tell: the PV hogs all of the disk space, can't really do non-LVM
> tests.

You can do a read test.

10gb read test:

dd if=/dev/md0 bs=1M count=10240 of=/dev/null

What is the result?

I've read that LVM can incur a 30-50% slowdown.

Justin.

Re: below 10MB/s write on raid5

[Jon Nelson]

On Mon, 11 Jun 2007, Justin Piszcz wrote:

> 
> 
> On Mon, 11 Jun 2007, Dexter Filmore wrote:
> 
> > On Monday 11 June 2007 14:47:50 Justin Piszcz wrote:
> > > On Mon, 11 Jun 2007, Dexter Filmore wrote:
> > > > I recently upgraded my file server, yet I'm still unsatisfied with the
> > > > write speed.
> > > > Machine now is a Athlon64 3400+ (Socket 754) equipped with 1GB of RAM.
> > > > The four RAID disks are attached to the board's onbaord sATA controller
> > > > (Sil3114 attached via PCI)
> > > > Kernel is 2.6.21.1, custom on Slackware 11.0.
> > > > RAID is on four Samsung SpinPoint disks, has LVM, 3 volumes atop of each
> > > > XFS.
> > > >
> > > > The machine does some other work, too, but still I would have suspected
> > > > to get into the 20-30MB/s area. Too much asked for?
> > > >
> > > > Dex
> > >
> > > What do you get without LVM?
> >
> > Hard to tell: the PV hogs all of the disk space, can't really do non-LVM
> > tests.
> 
> You can do a read test.
> 
> 10gb read test:
> 
> dd if=/dev/md0 bs=1M count=10240 of=/dev/null

eek! Make sure to use iflag=direct
with that otherwise you'll get cached reads and that will throw
the numbers off considerably.


--
Jon Nelson <jnelson-linux-raid@jamponi.net>

Re: below 10MB/s write on raid5

[Justin Piszcz]

On Mon, 11 Jun 2007, Jon Nelson wrote:

> On Mon, 11 Jun 2007, Justin Piszcz wrote:
>
>>
>>
>> On Mon, 11 Jun 2007, Dexter Filmore wrote:
>>
>>> On Monday 11 June 2007 14:47:50 Justin Piszcz wrote:
>>>> On Mon, 11 Jun 2007, Dexter Filmore wrote:
>>>>> I recently upgraded my file server, yet I'm still unsatisfied with the
>>>>> write speed.
>>>>> Machine now is a Athlon64 3400+ (Socket 754) equipped with 1GB of RAM.
>>>>> The four RAID disks are attached to the board's onbaord sATA controller
>>>>> (Sil3114 attached via PCI)
>>>>> Kernel is 2.6.21.1, custom on Slackware 11.0.
>>>>> RAID is on four Samsung SpinPoint disks, has LVM, 3 volumes atop of each
>>>>> XFS.
>>>>>
>>>>> The machine does some other work, too, but still I would have suspected
>>>>> to get into the 20-30MB/s area. Too much asked for?
>>>>>
>>>>> Dex
>>>>
>>>> What do you get without LVM?
>>>
>>> Hard to tell: the PV hogs all of the disk space, can't really do non-LVM
>>> tests.
>>
>> You can do a read test.
>>
>> 10gb read test:
>>
>> dd if=/dev/md0 bs=1M count=10240 of=/dev/null
>
> eek! Make sure to use iflag=direct
> with that otherwise you'll get cached reads and that will throw
> the numbers off considerably.

Wow, makes a difference for faster devices.  Does bonnie++ use 
iflag=direct when benchmarking?

p34:~# dd if=/dev/md0 bs=1M count=1024 of=/dev/null iflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 15.0454 seconds, 71.4 MB/s
p34:~# dd if=/dev/md0 bs=1M count=1024 of=/dev/null
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 14.991 seconds, 71.6 MB/s
p34:~# dd if=/dev/md3 bs=1M count=1024 of=/dev/null iflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 3.08707 seconds, 348 MB/s
p34:~# dd if=/dev/md3 bs=1M count=1024 of=/dev/null
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 2.02948 seconds, 529 MB/s
p34:~#

p34:~# dd if=/dev/md3 bs=1M count=10024 of=/dev/null
10024+0 records in
10024+0 records out
10510925824 bytes (11 GB) copied, 17.7321 seconds, 593 MB/s
p34:~# sync
p34:~# dd if=/dev/md3 bs=1M count=10024 of=/dev/null iflag=direct
10024+0 records in
10024+0 records out
10510925824 bytes (11 GB) copied, 29.022 seconds, 362 MB/s
p34:~#

Re: below 10MB/s write on raid5

[Dexter Filmore]

> 10gb read test:
>
> dd if=/dev/md0 bs=1M count=10240 of=/dev/null
>
> What is the result?

71,7MB/s - but that's reading to null. *writing* real data however looks quite 
different.

>
> I've read that LVM can incur a 30-50% slowdown.
>
Even then the 8-10MB/s I get would be a little low. 



-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d--(+)@ s-:+ a- C++++ UL++ P+>++ L+++>++++ E-- W++ N o? K-
w--(---) !O M+ V- PS+ PE Y++ PGP t++(---)@ 5 X+(++) R+(++) tv--(+)@ 
b++(+++) DI+++ D- G++ e* h>++ r* y?
------END GEEK CODE BLOCK------

http://www.stop1984.com
http://www.againsttcpa.com

Re: below 10MB/s write on raid5

[Justin Piszcz]

On Mon, 11 Jun 2007, Dexter Filmore wrote:

>> 10gb read test:
>>
>> dd if=/dev/md0 bs=1M count=10240 of=/dev/null
>>
>> What is the result?
>
> 71,7MB/s - but that's reading to null. *writing* real data however looks quite
> different.
>
>>
>> I've read that LVM can incur a 30-50% slowdown.
>>
> Even then the 8-10MB/s I get would be a little low.
>
>
>
> -- 
> -----BEGIN GEEK CODE BLOCK-----
> Version: 3.12
> GCS d--(+)@ s-:+ a- C++++ UL++ P+>++ L+++>++++ E-- W++ N o? K-
> w--(---) !O M+ V- PS+ PE Y++ PGP t++(---)@ 5 X+(++) R+(++) tv--(+)@
> b++(+++) DI+++ D- G++ e* h>++ r* y?
> ------END GEEK CODE BLOCK------
>
> http://www.stop1984.com
> http://www.againsttcpa.com
>

Would it be possible to trash the LVM and write directly to the MD device? 
I'd be really curious to see what it gets with XFS ontop of MD.

Justin.

Re: below 10MB/s write on raid5

[Nix]

On 11 Jun 2007, Justin Piszcz told this:
> You can do a read test.
>
> 10gb read test:
>
> dd if=/dev/md0 bs=1M count=10240 of=/dev/null
>
> What is the result?
>
> I've read that LVM can incur a 30-50% slowdown.

FWIW I see a much smaller penalty than that.

loki:~# lvs -o +devices
  LV           VG    Attr   LSize   Origin Snap%  Move Log Copy%  Devices
[...]
  usr          raid  -wi-ao   6.00G                               /dev/md1(50)

loki:~# time dd if=/dev/md1 bs=1000 count=502400 of=/dev/null
502400+0 records in
502400+0 records out
502400000 bytes (502 MB) copied, 16.2995 s, 30.8 MB/s

real    0m16.360s
user    0m0.310s
sys     0m11.780s

loki:~# time dd if=/dev/raid/usr bs=1000 count=502400 of=/dev/null
502400+0 records in
502400+0 records out
502400000 bytes (502 MB) copied, 18.6172 s, 27.0 MB/s

real    0m18.790s
user    0m0.380s
sys     0m14.750s


So there's a penalty, sure, accounted for mostly in sys time, but it's
only about 10%: small enough that I at least can ignore it in exchange
for the administrative convenience of LVM.

-- 
`... in the sense that dragons logically follow evolution so they would
 be able to wield metal.' --- Kenneth Eng's colourless green ideas sleep
 furiously

Re: below 10MB/s write on raid5

[Jon Nelson]

On Mon, 11 Jun 2007, Nix wrote:

> On 11 Jun 2007, Justin Piszcz told this:
> > You can do a read test.
> >
> > 10gb read test:
> >
> > dd if=/dev/md0 bs=1M count=10240 of=/dev/null
> >
> > What is the result?
> >
> > I've read that LVM can incur a 30-50% slowdown.
> 
> FWIW I see a much smaller penalty than that.
> 
> loki:~# lvs -o +devices
>   LV           VG    Attr   LSize   Origin Snap%  Move Log Copy%  Devices
> [...]
>   usr          raid  -wi-ao   6.00G                               /dev/md1(50)
> 
> loki:~# time dd if=/dev/md1 bs=1000 count=502400 of=/dev/null
> 502400+0 records in
> 502400+0 records out
> 502400000 bytes (502 MB) copied, 16.2995 s, 30.8 MB/s
> 
> loki:~# time dd if=/dev/raid/usr bs=1000 count=502400 of=/dev/null
> 502400+0 records in
> 502400+0 records out
> 502400000 bytes (502 MB) copied, 18.6172 s, 27.0 MB/s

And what is it like with 'iflag=direct' which I really feel you have to 
use, otherwise you get caching.

--
Jon Nelson <jnelson-linux-raid@jamponi.net>

Re: below 10MB/s write on raid5

[Justin Piszcz]

On Tue, 12 Jun 2007, Bill Davidsen wrote:

> Dexter Filmore wrote:
>> I recently upgraded my file server, yet I'm still unsatisfied with the 
>> write speed.
>> Machine now is a Athlon64 3400+ (Socket 754) equipped with 1GB of RAM.
>> The four RAID disks are attached to the board's onbaord sATA controller 
>> (Sil3114 attached via PCI)
>> Kernel is 2.6.21.1, custom on Slackware 11.0.
>> RAID is on four Samsung SpinPoint disks, has LVM, 3 volumes atop of each 
>> XFS.
>> 
>> The machine does some other work, too, but still I would have suspected to 
>> get into the 20-30MB/s area. Too much asked for?
>> 
>
> Increase your stripe cache size in /sys/block/mdX/md/stripe_cache_size. If 
> you have a chunk size of 256, try setting the cache size to 8192 and see if 
> your write performance ends up ~100MB/s or so.
>
>  echo 8192 > /sys/block/mdX/md/stripe_cache_size
>
> Where "X" is your array name, of course.
>
> Note, larger values will help more, but it's definitely diminishing returns, 
> so don't get carried away. There was a report of problems with size > 32768, 
> I don't remember the details, so I would avoid that as well.
>
> -- 
> bill davidsen <davidsen@tmr.com>
> CTO TMR Associates, Inc
> Doing interesting things with small computers since 1979
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

I was the one who tried > 32768 it crashed the machine (on an older box).

Justin.

Re: below 10MB/s write on raid5

[Bill Davidsen]

Dexter Filmore wrote:
> I recently upgraded my file server, yet I'm still unsatisfied with the write 
> speed.
> Machine now is a Athlon64 3400+ (Socket 754) equipped with 1GB of RAM.
> The four RAID disks are attached to the board's onbaord sATA controller 
> (Sil3114 attached via PCI)
> Kernel is 2.6.21.1, custom on Slackware 11.0.
> RAID is on four Samsung SpinPoint disks, has LVM, 3 volumes atop of each XFS.
>
> The machine does some other work, too, but still I would have suspected to get 
> into the 20-30MB/s area. Too much asked for?
>   

Increase your stripe cache size in /sys/block/mdX/md/stripe_cache_size. 
If you have a chunk size of 256, try setting the cache size to 8192 and 
see if your write performance ends up ~100MB/s or so.

   echo 8192 > /sys/block/mdX/md/stripe_cache_size

Where "X" is your array name, of course.

Note, larger values will help more, but it's definitely diminishing 
returns, so don't get carried away. There was a report of problems with 
size > 32768, I don't remember the details, so I would avoid that as well.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: below 10MB/s write on raid5

[Nix]

On 12 Jun 2007, Jon Nelson told this:
> On Mon, 11 Jun 2007, Nix wrote:
>
>> On 11 Jun 2007, Justin Piszcz told this:
>> loki:~# time dd if=/dev/md1 bs=1000 count=502400 of=/dev/null
>> 502400+0 records in
>> 502400+0 records out
>> 502400000 bytes (502 MB) copied, 16.2995 s, 30.8 MB/s
>> 
>> loki:~# time dd if=/dev/raid/usr bs=1000 count=502400 of=/dev/null
>> 502400+0 records in
>> 502400+0 records out
>> 502400000 bytes (502 MB) copied, 18.6172 s, 27.0 MB/s
>
> And what is it like with 'iflag=direct' which I really feel you have to 
> use, otherwise you get caching.

I have little enough memory on this box that caching is really not
significant :)

With iflag=direct I get, um,

loki:/var/log# time dd if=/dev/md1 bs=1000 count=502400 of=/dev/null iflag=direct
dd: reading `/dev/md1': Invalid argument
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.0324791 s, 0.0 kB/s

real    0m0.085s
user    0m0.000s
sys     0m0.000s

so not exactly ideal.

-- 
`... in the sense that dragons logically follow evolution so they would
 be able to wield metal.' --- Kenneth Eng's colourless green ideas sleep
 furiously

Re: below 10MB/s write on raid5

[Nix]

On 12 Jun 2007, Jon Nelson told this:
> On Mon, 11 Jun 2007, Nix wrote:
>
>> On 11 Jun 2007, Justin Piszcz told this:
>> loki:~# time dd if=/dev/md1 bs=1000 count=502400 of=/dev/null
>> 502400+0 records in
>> 502400+0 records out
>> 502400000 bytes (502 MB) copied, 16.2995 s, 30.8 MB/s
>> 
>> loki:~# time dd if=/dev/raid/usr bs=1000 count=502400 of=/dev/null
>> 502400+0 records in
>> 502400+0 records out
>> 502400000 bytes (502 MB) copied, 18.6172 s, 27.0 MB/s
>
> And what is it like with 'iflag=direct' which I really feel you have to 
> use, otherwise you get caching.

I have little enough memory on this box that caching is really not
significant :)

With iflag=direct I get, um,

loki:/var/log# time dd if=/dev/md1 bs=1000 count=502400 of=/dev/null iflag=direct
dd: reading `/dev/md1': Invalid argument
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.0324791 s, 0.0 kB/s

real    0m0.085s
user    0m0.000s
sys     0m0.000s

so not exactly ideal.

-- 
`... in the sense that dragons logically follow evolution so they would
 be able to wield metal.' --- Kenneth Eng's colourless green ideas sleep
 furiously

Re: below 10MB/s write on raid5

[Michael Tokarev]

Nix wrote:
[]
> With iflag=direct I get, um,
> 
> loki:/var/log# time dd if=/dev/md1 bs=1000 count=502400 of=/dev/null iflag=direct
> dd: reading `/dev/md1': Invalid argument

For direct i/o, you have to use blocksize =512, 1024, ... - multiple of a sector size.

/mjt