Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

["Giuseppe Ghibò" <ghibo@mandriva.com>]

Justin Piszcz ha scritto:

> I recently got a chance to test SW RAID5 using 750GB disks (10) in a 
> RAID5 on a 3ware card, model no: 9550SXU-12
> 
> The bottom line is the controller is doing some weird caching with 
> writes on SW RAID5 which makes it not worth using.
> 
> Recall, with SW RAID5 using regular SATA cards with (mind you) 10 raptors:
> write: 464MB/s
> read: 627MB/s
> 
> Yes, these drives are different, 7200RPM 750GB drives, but write should 
> not be 50-102MB/s as shown below.
> 
> First, lets test RAW performance of these 10 drives:
> 
> Create RAID 0 with 10 750GB Drives:
> # mdadm /dev/md0 --create --level=0 -n 10 /dev/sd[bcdefghjik]1
> mdadm: array /dev/md0 started.
> 
> --> XFS: (xfs default options, no optimizations)
> # dd if=/dev/zero of=10gb bs=1M count=10240
> 10737418240 bytes (11 GB) copied, 22.459 seconds, 478 MB/s
> # dd if=10gb of=/dev/zero bs=1M count=10240
> 10737418240 bytes (11 GB) copied, 28.7843 seconds, 373 MB/s
> 
> --> XFS: (xfs default options, enabled md-raid read optimizations)
> # dd if=/dev/zero of=10gb bs=1M count=10240
> 10737418240 bytes (11 GB) copied, 22.9623 seconds, 468 MB/s
> # dd if=10gb of=/dev/zero bs=1M count=10240
> 10737418240 bytes (11 GB) copied, 17.7328 seconds, 606 MB/s
> 
> Software RAID 5 on a real HW raid controller over 10 750GB disks JBOD:
> 
> UltraDense-AS-3ware-R5-9-disks,16G,50676,89,96019,34,46379,9,60267,99,501098,56,248.5,0,16:100000:16/64,240,3,21959,84,1109,10,286,4,22923,91,544,6 
> 
> UltraDense-AS-3ware-R5-9-disks,16G,49983,88,96902,37,47951,10,59002,99,529121,60,210.3,0,16:100000:16/64,250,3,25506,98,1163,10,268,3,18003,71,772,8 
> 
> UltraDense-AS-3ware-R5-9-disks,16G,49811,87,95759,35,48214,10,60153,99,538559,61,276.8,0,16:100000:16/64,233,3,25514,97,1100,9,279,3,21398,84,839,9 
> 
> 
> Write seems significantly impacted, where read is fine, the HW RAID 
> controller cache must be doing something strange:
> 
> --> XFS SW RAID 5: (xfs noatime only, enabled md-raid read optimizations)
> # dd if=/dev/zero of=10gb bs=1M count=10240
> 10737418240 bytes (11 GB) copied, 105.178 seconds, 102 MB/s
> # dd if=10gb of=/dev/zero bs=1M count=10240
> 10737418240 bytes (11 GB) copied, 17.4893 seconds, 614 MB/s
> 
> -----
> 
> I am sure one of your questions is, well, why use SW RAID5 on the 
> controller?  Because SW RAID5 is usually much faster than HW RAID5, at 
> least in my tests:
> 
> Ctl   Model        Ports   Drives   Units   NotOpt   RRate   VRate   BBU
> ------------------------------------------------------------------------
> c0    9550SXU-12   12      12       3       0        1       4       -
> 
> Unit  UnitType  Status         %Cmpl  Stripe  Size(GB)  Cache  AVerify  
> IgnECC
> ------------------------------------------------------------------------------ 
> 
> u0    RAID-1    OK             -      -       698.481   ON     ON       OFF
> u1    RAID-5    OK             -      64K     5587.85   ON     OFF      OFF
> u2    SPARE     OK             -      -       698.629   -      OFF      -
> 
> --> XFS:
> # dd if=/dev/zero of=10gb bs=1M count=10240
> 10737418240 bytes (11 GB) copied, 74.5648 seconds, 144 MB/s
> 
> --> JFS:
> # dd if=/dev/zero of=10gb bs=1M count=10240
> 10737418240 bytes (11 GB) copied, 108.631 seconds, 98.8 MB/s
> 
> The controller is set to performance, and this is nothing close to 
> performance.

How much is your RAM size? Is the size you tried (10G) at
least twice the size of the RAM seen by the OS? What
are the values returned by hdparm -t /dev/sda (it test only raw reading
speed)?

> 
> In RAID0, the controller is ok with the disks JBOD, but I cannot 
> recommend buying a controller (12,16,24 port) for Linux SW RAID 5.
> 
> Its too bad that there are no regular > 4 port SATA PCI-e controllers 
> out there.
> 
> Justin.
> 

Indeed not exists for PCI-e but Oden has spotted this PCI-X card
(which is around 97$), based on marvell chipset:

http://www.supermicro.com/products/accessories/addon/AoC-SAT2-MV8.cfm

which can be used on motherboard with PCI-X slot (the ASUS M2N32 WS Professional
AM2, or the ASUS P5W64-WS-PRO, both are for consumer desktop and have 2 PCI-X 
slots) though probably if you have either one of that mobo you already have at 
least 10 onboard SATA connectors.

Bye
Giuseppe.