3ware 7500-12, bad write speed

3ware 7500-12, bad write speed

[Mikael Abrahamsson]

I have a 6 drive (120 gig drives) RAID5 using a 3ware 7500-12. I have the
latest firmware (7.5.1), latest driver etc, everything looks up to spec. I
get great read speeds (60-80 megabyte/s), but I get very erratic write
speeds (average 15 megabyte/s or so).

Using "iostat -x 5" I see that writes are sometimes done at 25 meg/s to
the raid5, some intervals at 5 meg/s, etc, with an average of approx 12-15
megabyte/s. Writes seem very jerky.

Anyone seen this and solved it? I currently get better write speeds to a
single IBM 7200rpm 75 gig drive (~25-30 meg/s) than to the 6 drive RAID5
volume (I use the 3ware card to create the volume, no software raid).

3ware support seems at a loss as to what causes this. I see it both with 
my SMP system and another UP system (both using Redhat 7.3 supplied 
kernels and the latest firmware/drivers for each card). I use ext3 in all 
all these cases.

They suggested changing the bdflush values but I have already done that, 
and in my mind if it was a bdflush problem I would see the same problem to 
the single drive as well?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

RE: 3ware 7500-12, bad write speed

[Adam Radford]

This is because of the vmlayer and your bdflush settings and the fact that
you are
running a journalling filesystem.  Our raid 5 caching strategy gives the
best results when
given sequential writes.  Doing journaling fs and caching writes in the
vmlayer causes the firmware caching strategy to perform 'sub optimal' since
the lba's accessed aren't actually
sequential.  Thus your 15 MB/s.  We do do elevator seek in the FW, but this
can only help
you so much.

What you are benchmarking is VM layer performance+filesystem
performance+3ware device
performance.

Try some other settings or use a benchmark that issues raw asynchronous IO
(not filesystem benchmark) (Iometer for Linux?)

-Adam

-----Original Message-----
From: Mikael Abrahamsson [mailto:swmike@swm.pp.se]
Sent: Wednesday, September 25, 2002 1:43 PM
To: linux-raid@vger.kernel.org
Subject: 3ware 7500-12, bad write speed



I have a 6 drive (120 gig drives) RAID5 using a 3ware 7500-12. I have the
latest firmware (7.5.1), latest driver etc, everything looks up to spec. I
get great read speeds (60-80 megabyte/s), but I get very erratic write
speeds (average 15 megabyte/s or so).

Using "iostat -x 5" I see that writes are sometimes done at 25 meg/s to
the raid5, some intervals at 5 meg/s, etc, with an average of approx 12-15
megabyte/s. Writes seem very jerky.

Anyone seen this and solved it? I currently get better write speeds to a
single IBM 7200rpm 75 gig drive (~25-30 meg/s) than to the 6 drive RAID5
volume (I use the 3ware card to create the volume, no software raid).

3ware support seems at a loss as to what causes this. I see it both with 
my SMP system and another UP system (both using Redhat 7.3 supplied 
kernels and the latest firmware/drivers for each card). I use ext3 in all 
all these cases.

They suggested changing the bdflush values but I have already done that, 
and in my mind if it was a bdflush problem I would see the same problem to 
the single drive as well?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3ware 7500-12, bad write speed

[Maurice Hilarius]

With regards to your message at 02:43 PM 9/25/02, Mikael Abrahamsson. Where 
you stated:

>I have a 6 drive (120 gig drives) RAID5 using a 3ware 7500-12. I have the
>latest firmware (7.5.1), latest driver etc, everything looks up to spec. I
>get great read speeds (60-80 megabyte/s), but I get very erratic write
>speeds (average 15 megabyte/s or so).
Not so surprising.

>Anyone seen this and solved it? I currently get better write speeds to a
>single IBM 7200rpm 75 gig drive (~25-30 meg/s) than to the 6 drive RAID5
>volume (I use the 3ware card to create the volume, no software raid).
Yes, don't use the 3Ware BIOS RAID.
Run it in JBOD mode, and use Linux RAID.
Neil's md tools make this very nice, and if you go to the 3Ware site and 
look in the downloads they have CLI tools to support hotswap and other 
useful things.
On a set of 16 Maxtor 160GB disk, and a pair of 3Ware 8 port cards we 
regularly get RAID5 writes (using a 4GB block/file size measured by 
bonnie++) of over 75MB/sec and reads over 160MB/sec.

I do think you should be seeing better results with the card and it's 
native BIOS RAID, but I still think that software Linux RAID is a lot 
faster and more flexible.


With our best regards,

Maurice W. Hilarius       Telephone: 01-780-456-9771
Hard Data Ltd.               FAX:       01-780-456-9772
11060 - 166 Avenue        mailto:maurice@harddata.com
Edmonton, AB, Canada      http://www.harddata.com/
    T5X 1Y3

2.3TB RAID5 NAS server - dual AthlonMP CPU, Linux, $10,995 CAD / $6850 USD

RE: 3ware 7500-12, bad write speed

[Mikael Abrahamsson]

On Wed, 25 Sep 2002, Adam Radford wrote:

> This is because of the vmlayer and your bdflush settings and the fact
> that you are running a journalling filesystem.  Our raid 5 caching
> strategy gives the best results when given sequential writes.  Doing
> journaling fs and caching writes in the vmlayer causes the firmware
> caching strategy to perform 'sub optimal' since the lba's accessed
> aren't actually sequential.  Thus your 15 MB/s.  We do do elevator seek
> in the FW, but this can only help you so much.

Any specifics in the bdflush settings I should be aiming for? I have 
fiddeled some with the settings and I cannot seem to make much of a 
difference.

Would it help to mount that particular partition in with "sync" mount
option? I have already changed the mount option to data=writeback in order
to have less journaling data being thrown around.

What I also do not understand is that in some 5 second intervals I get 
iostat claiming there was only 500 kilobyte/s written when doing my test 
with dd if=/dev/zero of=testfile bs=1024000 count=2000. What in the world 
could cause the vm layer to have that big of an impact that it would only 
be able to write just 500 kilobytes in a second?

I have this bdflush setting at the moment:

echo 100 5000 640 2560 150 30000 5000 1884 2 > /proc/sys/vm/bdflush

I have made several radical changes to the bdflush settings and if any, it
made things worse.
 
-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

RE: 3ware 7500-12, bad write speed

[dean gaudet]

On Wed, 25 Sep 2002, Adam Radford wrote:

> What you are benchmarking is VM layer performance+filesystem
> performance+3ware device
> performance.
>
> Try some other settings or use a benchmark that issues raw asynchronous IO
> (not filesystem benchmark) (Iometer for Linux?)

unfortunately, raw io isn't a very realistic benchmark.  realistic
workloads have a filesystem :)

-dean

RE: 3ware 7500-12, bad write speed

[Jason]

> Yes, don't use the 3Ware BIOS RAID.
> Run it in JBOD mode, and use Linux RAID.
> Neil's md tools make this very nice, and if you go to the 3Ware site and
> look in the downloads they have CLI tools to support hotswap and other
> useful things.
> On a set of 16 Maxtor 160GB disk, and a pair of 3Ware 8 port cards we
> regularly get RAID5 writes (using a 4GB block/file size measured by
> bonnie++) of over 75MB/sec and reads over 160MB/sec.
>
> I do think you should be seeing better results with the card and it's
> native BIOS RAID, but I still think that software Linux RAID is a lot
> faster and more flexible.

I'm with Maurice on this one.  I changed over from using 3wares BIOS RAID to
using Linux software RAID and my write speeds went up to from 15mb/sec
(pathetic) to almost 120mb/sec which almost matches my read speeds (using
Bonnie++).  This was using 4x40GB Maxtor drives in a RAID 5 configuration.

One downside to using the drives in JBOD mode is with 3wares current drivers
if you use their software to remove the drive and then put it back in, it
then recognizes it as a CBOD drive of which then is unrecognizable by the
software RAID.  3ware had a few suggestions that simply were not working so
I placed the drive in a separate machine and ran fdisk to remove all
partitions.  Then upon putting it back in with a reboot rather than using
the software it was again seen as a JBOD drive.  This means no hot swapping
but a reboot would be required.  The good news is 3ware has told me they are
attempting to make a workaround in their drivers so you can add and remove a
JBOD drive without a shutdown.

Jay
>

RE: 3ware 7500-12, bad write speed

[Adam Radford]

Jason,

If you're using the 7.5.1 codeset, you can add a jbod with the following
command:

tw_cli maint add c0 p0 jbod

By default, you get a cbod, so you must specify jbod's and spare's.

-Adam

-----Original Message-----
From: Jason [mailto:bind@swift-web.com]
Sent: Wednesday, September 25, 2002 11:43 PM
To: Mikael Abrahamsson
Cc: linux-raid@vger.kernel.org
Subject: RE: 3ware 7500-12, bad write speed


> Yes, don't use the 3Ware BIOS RAID.
> Run it in JBOD mode, and use Linux RAID.
> Neil's md tools make this very nice, and if you go to the 3Ware site and
> look in the downloads they have CLI tools to support hotswap and other
> useful things.
> On a set of 16 Maxtor 160GB disk, and a pair of 3Ware 8 port cards we
> regularly get RAID5 writes (using a 4GB block/file size measured by
> bonnie++) of over 75MB/sec and reads over 160MB/sec.
>
> I do think you should be seeing better results with the card and it's
> native BIOS RAID, but I still think that software Linux RAID is a lot
> faster and more flexible.

I'm with Maurice on this one.  I changed over from using 3wares BIOS RAID to
using Linux software RAID and my write speeds went up to from 15mb/sec
(pathetic) to almost 120mb/sec which almost matches my read speeds (using
Bonnie++).  This was using 4x40GB Maxtor drives in a RAID 5 configuration.

One downside to using the drives in JBOD mode is with 3wares current drivers
if you use their software to remove the drive and then put it back in, it
then recognizes it as a CBOD drive of which then is unrecognizable by the
software RAID.  3ware had a few suggestions that simply were not working so
I placed the drive in a separate machine and ran fdisk to remove all
partitions.  Then upon putting it back in with a reboot rather than using
the software it was again seen as a JBOD drive.  This means no hot swapping
but a reboot would be required.  The good news is 3ware has told me they are
attempting to make a workaround in their drivers so you can add and remove a
JBOD drive without a shutdown.

Jay
>


-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3ware 7500-12, bad write speed

[raid]

Ok did a benchmark between software raid en hw raid
controller 3ware 7850 (7500-8 now?) with 5 * WDC 120GB 7200RPM (2mb cache)
On xeon dual 2GHZ (controller in 64bits pci)

results :

---raid5 5*120gb sw raid :

Version 1.02c       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
storagenew.a2000 1G 18850  99 78959  77 29982  29 24052  99 81547  22
510.5   2
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
/sec %CP
                 16  1933  97 +++++ +++ +++++ +++  1986  98 +++++ +++
5056  98
storagenew.a2000.nu,1G,18850,99,78959,77,29982,29,24052,99,81547,22,510.5,2,16,1933,97,+++++,+++,+++++,+++,1986,98,+++++,+++,5056,98


---raid5 5*120gb hw raid :

Version 1.02c       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
storagenew.a2000 1G 17100  85 32250  23 12053   5 22660  91 49603  10
466.9   1
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
/sec %CP
                 16  1872  93 +++++ +++ +++++ +++  1976  98 +++++ +++
5677 100
storagenew.a2000.nu,1G,17100,85,32250,23,12053,5,22660,91,49603,10,466.9,1,16,1872,93,+++++,+++,+++++,+++,1976,98,+++++,+++,5677,100

Re: 3ware 7500-12, bad write speed

[Mikael Abrahamsson]

On Thu, 10 Oct 2002 raid@ddx.a2000.nu wrote:

> Ok did a benchmark between software raid en hw raid
> controller 3ware 7850 (7500-8 now?) with 5 * WDC 120GB 7200RPM (2mb cache)
> On xeon dual 2GHZ (controller in 64bits pci)

Guess that is pretty much the same results I get, but it also shows that 
software raid5 also isn't very fast when it comes to writing, and cpu 
usage actually does rise a lot (your tests seem to indicate that there is 
a lot of cpu activity going on?).

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

RE: 3ware 7500-12, bad write speed

[raid]

Adam,

results after changing these settings
read is faster, but write is still half sw raid speed


Version 1.02c       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
storagenew.a2000 1G 18240  90 30134  22 12535   6 24634  98 82337  16
468.0   1
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
/sec %CP
                 16  1926  95 +++++ +++ +++++ +++  1960  97 +++++ +++
5627  99
storagenew.a2000.nu,1G,18240,90,30134,22,12535,6,24634,98,82337,16,468.0,1,16,1926,95,+++++,+++,+++++,+++,1960,97,+++++,+++,5627,99


On Thu, 10 Oct 2002, Adam Radford wrote:

> Can you run the following commands, then re-run your 3ware benchmark and see
> if the results change?
>
> echo "60" > /proc/sys/vm/bdflush
> echo "512" > /proc/sys/vm/max-readahead
> echo "256" > /proc/sys/vm/min-readahead
>
> -Adam
>
> -----Original Message-----
> From: raid@ddx.a2000.nu [mailto:raid@ddx.a2000.nu]
> Sent: Thursday, October 10, 2002 2:18 PM
> To: linux-raid@vger.kernel.org
> Subject: Re: 3ware 7500-12, bad write speed
>
>
> Ok did a benchmark between software raid en hw raid
> controller 3ware 7850 (7500-8 now?) with 5 * WDC 120GB 7200RPM (2mb cache)
> On xeon dual 2GHZ (controller in 64bits pci)
>
> results :
>
> ---raid5 5*120gb sw raid :
>
> Version 1.02c       ------Sequential Output------ --Sequential Input-
> --Random-
>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
> /sec %CP
> storagenew.a2000 1G 18850  99 78959  77 29982  29 24052  99 81547  22
> 510.5   2
>                     ------Sequential Create------ --------Random
> Create--------
>                     -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
> /sec %CP
>                  16  1933  97 +++++ +++ +++++ +++  1986  98 +++++ +++
> 5056  98
> storagenew.a2000.nu,1G,18850,99,78959,77,29982,29,24052,99,81547,22,510.5,2,
> 16,1933,97,+++++,+++,+++++,+++,1986,98,+++++,+++,5056,98
>
>
> ---raid5 5*120gb hw raid :
>
> Version 1.02c       ------Sequential Output------ --Sequential Input-
> --Random-
>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
> /sec %CP
> storagenew.a2000 1G 17100  85 32250  23 12053   5 22660  91 49603  10
> 466.9   1
>                     ------Sequential Create------ --------Random
> Create--------
>                     -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
> /sec %CP
>                  16  1872  93 +++++ +++ +++++ +++  1976  98 +++++ +++
> 5677 100
> storagenew.a2000.nu,1G,17100,85,32250,23,12053,5,22660,91,49603,10,466.9,1,1
> 6,1872,93,+++++,+++,+++++,+++,1976,98,+++++,+++,5677,100
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

RE: 3ware 7500-12, bad write speed

[raid]

On Thu, 10 Oct 2002, Adam Radford wrote:

> Can you run the following commands, then re-run your 3ware benchmark and see
> if the results change?
>
> echo "60" > /proc/sys/vm/bdflush
> echo "512" > /proc/sys/vm/max-readahead
> echo "256" > /proc/sys/vm/min-readahead

Do any of this settings (or any other settings in /proc/sys/vm) help sw
raid ?

also what are the best parameters for mke2fs (both sw and hw raid)
(i now run mke2fs -j -m 0 /dev/sdb1 (or /dev/md0)

RE: 3ware 7500-12, bad write speed

[Adam Radford]

You probably won't get any benefit of 3ware with software raid by
changing max/min readahead, since md.c sets its own readahead.

For the writes, try it and find out.

-Adam

-----Original Message-----
From: raid@ddx.a2000.nu [mailto:raid@ddx.a2000.nu]
Sent: Friday, October 11, 2002 3:05 PM
To: Adam Radford
Cc: linux-raid@vger.kernel.org
Subject: RE: 3ware 7500-12, bad write speed


On Thu, 10 Oct 2002, Adam Radford wrote:

> Can you run the following commands, then re-run your 3ware benchmark and
see
> if the results change?
>
> echo "60" > /proc/sys/vm/bdflush
> echo "512" > /proc/sys/vm/max-readahead
> echo "256" > /proc/sys/vm/min-readahead

Do any of this settings (or any other settings in /proc/sys/vm) help sw
raid ?

also what are the best parameters for mke2fs (both sw and hw raid)
(i now run mke2fs -j -m 0 /dev/sdb1 (or /dev/md0)



-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html