3ware bad write speed.

3ware bad write speed.

[Donghui Wen]

Hi,
 We used 3ware 7500-4 as ide raid controler on a redhat advanced server
machine.
(kernel 2.4.9-e12, lastest 3ware driver). But we got a pretty bad write
speed.


Sequential Writes
                              File  Blk   Num                   Avg
Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency
Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----
------  -------- -------- -----
2.4.9-e.12enterprise          1792  4096    1    8.99 6.920%     0.394
29064.03   0.00240  0.00218   130
2.4.9-e.12enterprise          1792  4096    2   10.55 17.61%     0.637
23474.59   0.00480  0.00218    60
2.4.9-e.12enterprise          1792  4096    4    1.94 11.22%     0.156
1861.13   0.00000  0.00000    17
2.4.9-e.12enterprise          1792  4096    8    0.83 7.171%     0.274
995.19   0.00000  0.00000    12

 I saw a thread (http://www.spinics.net/lists/raid/msg01762.html)
 discussing the simillar problem by your guys. The possible reason you
pointed out is:
      3ware card has its own caching mechanism to accelerate sequential
write, but combine with linux's
      own VM caching and journaling filesystem, the disk physical accesses
aren't actually sequential.

 I have several questions:

 (1) Does this problem only exists in 3ware ide raid adaptor? We test SCSI
too, it is much faster.
     If it is 3ware's driver problem, scsi raid should have the same
problem.
 (2) Has 3ware fixed this problem yet? They claim they support linux.
 (3) One solution is to setup 3ware in jbod mode and use linux
software-raid.
     what percentage of CPU used if using software-raid? It might cost too
much CPU power
     without using hardware raid.
 (4) Other than 3ware, is there any better raid card supported by linux?

 I appreciate your guys help


Donghui Wen

RE: 3ware bad write speed.

[Rechenberg, Andrew]

>  I have several questions:
> 
>  (1) Does this problem only exists in 3ware ide raid adaptor? 
> We test SCSI
> too, it is much faster.
>      If it is 3ware's driver problem, scsi raid should have the same
> problem.
>  (2) Has 3ware fixed this problem yet? They claim they support linux.


Not sure about these as I have no experience with the 3ware cards


>  (3) One solution is to setup 3ware in jbod mode and use linux
> software-raid.
>      what percentage of CPU used if using software-raid? It 
> might cost too
> much CPU power
>      without using hardware raid.

I just setup a 26 disk software RAID10 array on Linux and it has put a
minimal load on the CPU's (my box is a Dell PowerEdge 6600 with quad
Xeon 1.4GHz with HT enabled).  I will be moving this box to a 52 disk
software RAID10 setup as soon as I finish testing some kernel patches.

I can recommend software RAID on Linux without hesitation.  The kernel
developers have done a great job with MD.

>  (4) Other than 3ware, is there any better raid card 
> supported by linux?
> 

We moved from hardware to software RAID because we couldn't find a
hardware card that gave us the performance that we wanted.  I have the
52 disk RAID10 array described above setup on some test hardware at work
and tiobench, with 4 threads and 32KB blocks shows 515MB/s sequential
reads (ext3 default journaling mode).  And as I said earlier, the load
on the CPU's is minimal in my setup (OLTP and batch database
processing).


>  I appreciate your guys help
> 
> 
> Donghui Wen
> 
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: 3ware bad write speed.

[Donghui Wen]

Thanks, Andrew:
    What ATA RAID controllers are you using? Do you have any benchmark data
about sequential write?

Donghui
 
----- Original Message ----- 
From: "Rechenberg, Andrew" <ARechenberg@shermanfinancialgroup.com>
To: "Donghui Wen" <dhwen@protegonetworks.com>; <linux-raid@vger.kernel.org>
Sent: Saturday, March 22, 2003 1:07 PM
Subject: RE: 3ware bad write speed.


> >  I have several questions:
> > 
> >  (1) Does this problem only exists in 3ware ide raid adaptor? 
> > We test SCSI
> > too, it is much faster.
> >      If it is 3ware's driver problem, scsi raid should have the same
> > problem.
> >  (2) Has 3ware fixed this problem yet? They claim they support linux.
> 
> 
> Not sure about these as I have no experience with the 3ware cards
> 
> 
> >  (3) One solution is to setup 3ware in jbod mode and use linux
> > software-raid.
> >      what percentage of CPU used if using software-raid? It 
> > might cost too
> > much CPU power
> >      without using hardware raid.
> 
> I just setup a 26 disk software RAID10 array on Linux and it has put a
> minimal load on the CPU's (my box is a Dell PowerEdge 6600 with quad
> Xeon 1.4GHz with HT enabled).  I will be moving this box to a 52 disk
> software RAID10 setup as soon as I finish testing some kernel patches.
> 
> I can recommend software RAID on Linux without hesitation.  The kernel
> developers have done a great job with MD.
> 
> >  (4) Other than 3ware, is there any better raid card 
> > supported by linux?
> > 
> 
> We moved from hardware to software RAID because we couldn't find a
> hardware card that gave us the performance that we wanted.  I have the
> 52 disk RAID10 array described above setup on some test hardware at work
> and tiobench, with 4 threads and 32KB blocks shows 515MB/s sequential
> reads (ext3 default journaling mode).  And as I said earlier, the load
> on the CPU's is minimal in my setup (OLTP and batch database
> processing).
> 
> 
> >  I appreciate your guys help
> > 
> > 
> > Donghui Wen
> > 
> > -
> > To unsubscribe from this list: send the line "unsubscribe 
> > linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

RE: 3ware bad write speed.

[Rechenberg, Andrew]

>> -----Original Message-----
>> From: Donghui Wen [mailto:dhwen@protegonetworks.com] 
>> Sent: Saturday, March 22, 2003 4:37 PM
>> To: Rechenberg, Andrew; linux-raid@vger.kernel.org
>> Subject: Re: 3ware bad write speed.
>> 
>> 
>> Thanks, Andrew:
>>     What ATA RAID controllers are you using? Do you have any 
>> benchmark data
>> about sequential write?
>> 
>> Donghui 

We're actually using SCSI disks attached to Adaptec 39160 SCSI
controllers.  I then have a monster /etc/raidtab to setup the software
RAID arrays and I use mdadm to monitor the arrays for failed disks.  We
used to use the Dell PERC3/QC (OEM LSI MegaRAID Enterprise 1600) for
hardware RAID controllers but in basic tests we could get almost 50%
better performance from software RAID.

If you want (or have) to use ATA controllers for monetary reasons I'm
not really the person to ask :)  I have a Promise Ultra100TX2 in my home
workstation that works great for my home use, but I have not tested
these cards in a production environment.

I can't recall off-hand what the sequential writes were from tiobench,
but here are some bonnie++ numbers for that 52 SCSI disk array with an
8GB file:

Seq. Output (writing)                         
Per Char     Block        Rewrite
K/sec %CPU   K/sec %CPU   K/sec  %CPU
----------------------------------------
23719  99    129707 99    99141   54

Seq. Input  (reading)
Per Char     Block      
K/sec %CPU   K/sec %CPU 
-------------------------
27551  98    301288 63


So with one bonnie++ thread I was getting ~127MB/s sequential writes and
~294MB/s sequential reads.  I ran two other tests with the same
parameters and the averages over three tests were 128.8MB/s reads and
296.7MB/s reads.  The details for these numbers are below:

Red Hat Linux 7.3
Kernel 2.4.18-26.7.xbigmem with md-seq_file and LVM 1.0.7 patches
applied
Dell PowerEdge 4600 
2x2.4GHz Xeon with HT
4GB RAM
52 15K SCSI disks with equal 17GB partitions
1 RAID10 software RAID array (~442GB usable space)
1 Linux Logical Volume Manager (LVM) Volume Group (VG) on top of the MD
device
1 300GB Logical Volume (LV) carved out of the 442GB VG
ext3 filesystem on LV (mke2fs -j /dev/vg00/lv00) mounted data=ordered

The other portion of the LV was used to test LV snapshots.

Let me know if you have any other questions.

Thanks,
Andy.

Re: 3ware bad write speed.

[Joshua Baker-LePain]

On Sat, 22 Mar 2003 at 12:45pm, Donghui Wen wrote

> Hi,
>  We used 3ware 7500-4 as ide raid controler on a redhat advanced server
> machine.
> (kernel 2.4.9-e12, lastest 3ware driver). But we got a pretty bad write
> speed.
> 
> Sequential Writes
>                               File  Blk   Num                   Avg
> Maximum      Lat%     Lat%    CPU
> Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency
> Latency      >2s      >10s    Eff
> ---------------------------- ------ ----- ---  ------ ------ --------- -----
> ------  -------- -------- -----
> 2.4.9-e.12enterprise          1792  4096    1    8.99 6.920%     0.394
> 29064.03   0.00240  0.00218   130
> 2.4.9-e.12enterprise          1792  4096    2   10.55 17.61%     0.637
> 23474.59   0.00480  0.00218    60
> 2.4.9-e.12enterprise          1792  4096    4    1.94 11.22%     0.156
> 1861.13   0.00000  0.00000    17
> 2.4.9-e.12enterprise          1792  4096    8    0.83 7.171%     0.274
> 995.19   0.00000  0.00000    12

The card is capable of much better than that.  Here are my numbers from a 
7 disk RAID5 array (8th is a hot spare) on a 7500-8 board:

Sequential Writes
                              File  Blk   Num                   Avg      
Maximum
      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    
Latency
      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- 
---------
--  -------- -------- -----
2.4.18-17SGI_XFS_1.2pre3smp   4096  4096    1   49.50 20.52%     0.068     
2915.
71   0.00124  0.00000   241
2.4.18-17SGI_XFS_1.2pre3smp   4096  4096    2   45.17 23.21%     0.141     
7873.
45   0.00343  0.00000   195
2.4.18-17SGI_XFS_1.2pre3smp   4096  4096    4   39.98 23.29%     0.335    
13338.
23   0.00887  0.00010   172
2.4.18-17SGI_XFS_1.2pre3smp   4096  4096    8   39.23 24.30%     0.631     
7887.
96   0.01535  0.00000   161

Obviously I've got more spindles, but your speeds shouldn't be *that* bad.  
Some things to check:

1) You said latest drivers, but make sure that you upgraded everything -- 
a driver set from 3ware is driver+firmware+3dm.  Specifically, make sure 
your firmware is up to date with your driver.

2) Make sure that your kernel supports HIGHIO.  I don't know if that 
enterprise kernel does (do you need to run it?), but it makes a *big* 
difference.  The latest 3ware drivers support HIGHIO, but the kernel needs 
to as well.

3) Make sure your drives are listed as compatible.

4) Try a different FS.  The numbers above are with XFS -- with ext3 I was 
getting write speeds of only half that (at best).

Good luck.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: 3ware bad write speed.

[Donghui Wen]

Thanks! Joshua:
    We are runing oracle on rehat advanced server 2.1. Oracle only certified
this version of redhat with ext2. That' why we could not switch to another
linux
or file system. :-(

Donghui

> The card is capable of much better than that.  Here are my numbers from a
> 7 disk RAID5 array (8th is a hot spare) on a 7500-8 board:
>
> Sequential Writes
>                               File  Blk   Num                   Avg
> Maximum
>       Lat%     Lat%    CPU
> Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency
> Latency
>       >2s      >10s    Eff
> ---------------------------- ------ ----- ---  ------ ------ ---------
> ---------
> --  -------- -------- -----
> 2.4.18-17SGI_XFS_1.2pre3smp   4096  4096    1   49.50 20.52%     0.068
> 2915.
> 71   0.00124  0.00000   241
> 2.4.18-17SGI_XFS_1.2pre3smp   4096  4096    2   45.17 23.21%     0.141
> 7873.
> 45   0.00343  0.00000   195
> 2.4.18-17SGI_XFS_1.2pre3smp   4096  4096    4   39.98 23.29%     0.335
> 13338.
> 23   0.00887  0.00010   172
> 2.4.18-17SGI_XFS_1.2pre3smp   4096  4096    8   39.23 24.30%     0.631
> 7887.
> 96   0.01535  0.00000   161
>
> Obviously I've got more spindles, but your speeds shouldn't be *that* bad.
> Some things to check:
>
> 1) You said latest drivers, but make sure that you upgraded everything --
> a driver set from 3ware is driver+firmware+3dm.  Specifically, make sure
> your firmware is up to date with your driver.
>
> 2) Make sure that your kernel supports HIGHIO.  I don't know if that
> enterprise kernel does (do you need to run it?), but it makes a *big*
> difference.  The latest 3ware drivers support HIGHIO, but the kernel needs
> to as well.
>
> 3) Make sure your drives are listed as compatible.
>
> 4) Try a different FS.  The numbers above are with XFS -- with ext3 I was
> getting write speeds of only half that (at best).
>
> Good luck.
>
> --
> Joshua Baker-LePain
> Department of Biomedical Engineering
> Duke University
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: 3ware bad write speed.

[Stephan van Hienen]

On Mon, 24 Mar 2003, Joshua Baker-LePain wrote:

> 2) Make sure that your kernel supports HIGHIO.  I don't know if that
> enterprise kernel does (do you need to run it?), but it makes a *big*
> difference.  The latest 3ware drivers support HIGHIO, but the kernel needs
> to as well.

what is this HIGHIO (patch?) ?

Re: 3ware bad write speed.

[Joshua Baker-LePain]

On Wed, 26 Mar 2003 at 1:40am, Stephan van Hienen wrote

> On Mon, 24 Mar 2003, Joshua Baker-LePain wrote:
> 
> > 2) Make sure that your kernel supports HIGHIO.  I don't know if that
> > enterprise kernel does (do you need to run it?), but it makes a *big*
> > difference.  The latest 3ware drivers support HIGHIO, but the kernel needs
> > to as well.
> 
> what is this HIGHIO (patch?) ?

From Documentation/Configure.help:

HIGHMEM I/O support
CONFIG_HIGHIO
  If you want to be able to do I/O to high memory pages, say Y.
  Otherwise low memory pages are used as bounce buffers causing a
  degrade in performance.

It's in mainline as of 2.4.20, and I know that RH has included it for a 
while.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: 3ware bad write speed.

[Christian Diehl]

Hi * 

I am using several hardware raid systems based on 3Ware, Adaptec and 
AC&NC hardware. 

The Mainboard of the test machine is a Supermicro P4DMS-6GM with two
Xeon 2.4GHz Processors and 1 Gig of RAM. 

IDE (WD 120 Gig) disks are used in the AC&NC Jetstor III (14 disks) and
in the 3Ware (7 disks). The Adaptec Raids use SCSI (Seagate) Disks. 

All these systems show a remarkably bad performance (max. 80 MB/s write
and max. 70 MB/s read, exact figures (bonnie++,iostat) available). 

I am using kernel 2.4.20 without any patches (Highmem on / i2o off) 
and an ext3 filesystem. 
I have played around with different stride settings and block sizes as
well as chunk sizes on the raid and experienced minor differences within
the above limits. 

By setting 

echo 256 > /proc/sys/vm/max-readahead 
echo 128 > /proc/sys/vm/min-readahead 

I could convince the 3ware to read data with up to 114 MB/s. 

So basically I am experiencing the same bad speeds (especially write
speeds) as Donghui (and others). 

It seems to me like there might be a common reason behind the bad
performance. (the VM Layer settings?) 

While writing this email I just finished a bonnie++ run on a new
software raid using 8 SCSI Disks: 

softr.8disk.64chunk,2000M:64k,13630,99,41206,41,27280,27,15278,97,156480,62,329.8,12,16,1322,81,+++++,+++,29177,99,1510,99,+++++,+++,3981,98 

That's an amazing difference in reading speed to the hardware raids
(soft:156480;hard:70155). I will most probably not be able to convert
all my Raids to Software Raids, so I would like to reach a similar
performance with the Hardware Raid. 


Any ideas,comments,...? 


Christian