Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Justin Piszcz <jpiszcz@lucidpixels.com>]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4688 bytes --]



On Wed, 18 Jul 2007, Giuseppe Ghibò wrote:

> Justin Piszcz ha scritto:
>
>> I recently got a chance to test SW RAID5 using 750GB disks (10) in a RAID5 
>> on a 3ware card, model no: 9550SXU-12
>> 
>> The bottom line is the controller is doing some weird caching with writes 
>> on SW RAID5 which makes it not worth using.
>> 
>> Recall, with SW RAID5 using regular SATA cards with (mind you) 10 raptors:
>> write: 464MB/s
>> read: 627MB/s
>> 
>> Yes, these drives are different, 7200RPM 750GB drives, but write should not 
>> be 50-102MB/s as shown below.
>> 
>> First, lets test RAW performance of these 10 drives:
>> 
>> Create RAID 0 with 10 750GB Drives:
>> # mdadm /dev/md0 --create --level=0 -n 10 /dev/sd[bcdefghjik]1
>> mdadm: array /dev/md0 started.
>> 
>> --> XFS: (xfs default options, no optimizations)
>> # dd if=/dev/zero of=10gb bs=1M count=10240
>> 10737418240 bytes (11 GB) copied, 22.459 seconds, 478 MB/s
>> # dd if=10gb of=/dev/zero bs=1M count=10240
>> 10737418240 bytes (11 GB) copied, 28.7843 seconds, 373 MB/s
>> 
>> --> XFS: (xfs default options, enabled md-raid read optimizations)
>> # dd if=/dev/zero of=10gb bs=1M count=10240
>> 10737418240 bytes (11 GB) copied, 22.9623 seconds, 468 MB/s
>> # dd if=10gb of=/dev/zero bs=1M count=10240
>> 10737418240 bytes (11 GB) copied, 17.7328 seconds, 606 MB/s
>> 
>> Software RAID 5 on a real HW raid controller over 10 750GB disks JBOD:
>> 
>> UltraDense-AS-3ware-R5-9-disks,16G,50676,89,96019,34,46379,9,60267,99,501098,56,248.5,0,16:100000:16/64,240,3,21959,84,1109,10,286,4,22923,91,544,6 
>> UltraDense-AS-3ware-R5-9-disks,16G,49983,88,96902,37,47951,10,59002,99,529121,60,210.3,0,16:100000:16/64,250,3,25506,98,1163,10,268,3,18003,71,772,8 
>> UltraDense-AS-3ware-R5-9-disks,16G,49811,87,95759,35,48214,10,60153,99,538559,61,276.8,0,16:100000:16/64,233,3,25514,97,1100,9,279,3,21398,84,839,9 
>> 
>> Write seems significantly impacted, where read is fine, the HW RAID 
>> controller cache must be doing something strange:
>> 
>> --> XFS SW RAID 5: (xfs noatime only, enabled md-raid read optimizations)
>> # dd if=/dev/zero of=10gb bs=1M count=10240
>> 10737418240 bytes (11 GB) copied, 105.178 seconds, 102 MB/s
>> # dd if=10gb of=/dev/zero bs=1M count=10240
>> 10737418240 bytes (11 GB) copied, 17.4893 seconds, 614 MB/s
>> 
>> -----
>> 
>> I am sure one of your questions is, well, why use SW RAID5 on the 
>> controller?  Because SW RAID5 is usually much faster than HW RAID5, at 
>> least in my tests:
>> 
>> Ctl   Model        Ports   Drives   Units   NotOpt   RRate   VRate   BBU
>> ------------------------------------------------------------------------
>> c0    9550SXU-12   12      12       3       0        1       4       -
>> 
>> Unit  UnitType  Status         %Cmpl  Stripe  Size(GB)  Cache  AVerify 
>> IgnECC
>> 
>> ------------------------------------------------------------------------------ 
>> u0    RAID-1    OK             -      -       698.481   ON     ON       OFF
>> u1    RAID-5    OK             -      64K     5587.85   ON     OFF      OFF
>> u2    SPARE     OK             -      -       698.629   -      OFF      -
>> 
>> --> XFS:
>> # dd if=/dev/zero of=10gb bs=1M count=10240
>> 10737418240 bytes (11 GB) copied, 74.5648 seconds, 144 MB/s
>> 
>> --> JFS:
>> # dd if=/dev/zero of=10gb bs=1M count=10240
>> 10737418240 bytes (11 GB) copied, 108.631 seconds, 98.8 MB/s
>> 
>> The controller is set to performance, and this is nothing close to 
>> performance.
>
> How much is your RAM size? Is the size you tried (10G) at
> least twice the size of the RAM seen by the OS? What
> are the values returned by hdparm -t /dev/sda (it test only raw reading
> speed)?
>
Total: 4GB of ram-- I am using the array for other things right now, did 
not get a chance to run that.

>> 
>> In RAID0, the controller is ok with the disks JBOD, but I cannot recommend 
>> buying a controller (12,16,24 port) for Linux SW RAID 5.
>> 
>> Its too bad that there are no regular > 4 port SATA PCI-e controllers out 
>> there.
>> 
>> Justin.
>> 
>
> Indeed not exists for PCI-e but Oden has spotted this PCI-X card
> (which is around 97$), based on marvell chipset:
>
> http://www.supermicro.com/products/accessories/addon/AoC-SAT2-MV8.cfm
>
> which can be used on motherboard with PCI-X slot (the ASUS M2N32 WS 
> Professional
> AM2, or the ASUS P5W64-WS-PRO, both are for consumer desktop and have 2 PCI-X 
> slots) though probably if you have either one of that mobo you already have 
> at least 10 onboard SATA connectors.
Indeed, wish there was a PCI-e version!

>
> Bye
> Giuseppe.
>