Software RAID5 Horrible Write Speed On 3ware Controller!!

Software RAID5 Horrible Write Speed On 3ware Controller!!

[Justin Piszcz]

I recently got a chance to test SW RAID5 using 750GB disks (10) in a RAID5 
on a 3ware card, model no: 9550SXU-12

The bottom line is the controller is doing some weird caching with writes 
on SW RAID5 which makes it not worth using.

Recall, with SW RAID5 using regular SATA cards with (mind you) 10 raptors:
write: 464MB/s
read: 627MB/s

Yes, these drives are different, 7200RPM 750GB drives, but write should 
not be 50-102MB/s as shown below.

First, lets test RAW performance of these 10 drives:

Create RAID 0 with 10 750GB Drives:
# mdadm /dev/md0 --create --level=0 -n 10 
/dev/sd[bcdefghjik]1
mdadm: array /dev/md0 started.

--> XFS: (xfs default options, no optimizations)
# dd if=/dev/zero of=10gb bs=1M count=10240
10737418240 bytes (11 GB) copied, 22.459 seconds, 478 MB/s
# dd if=10gb of=/dev/zero bs=1M count=10240
10737418240 bytes (11 GB) copied, 28.7843 seconds, 373 MB/s

--> XFS: (xfs default options, enabled md-raid read optimizations)
# dd if=/dev/zero of=10gb bs=1M count=10240
10737418240 bytes (11 GB) copied, 22.9623 seconds, 468 MB/s
# dd if=10gb of=/dev/zero bs=1M count=10240
10737418240 bytes (11 GB) copied, 17.7328 seconds, 606 MB/s

Software RAID 5 on a real HW raid controller over 10 750GB disks JBOD:

UltraDense-AS-3ware-R5-9-disks,16G,50676,89,96019,34,46379,9,60267,99,501098,56,248.5,0,16:100000:16/64,240,3,21959,84,1109,10,286,4,22923,91,544,6
UltraDense-AS-3ware-R5-9-disks,16G,49983,88,96902,37,47951,10,59002,99,529121,60,210.3,0,16:100000:16/64,250,3,25506,98,1163,10,268,3,18003,71,772,8
UltraDense-AS-3ware-R5-9-disks,16G,49811,87,95759,35,48214,10,60153,99,538559,61,276.8,0,16:100000:16/64,233,3,25514,97,1100,9,279,3,21398,84,839,9

Write seems significantly impacted, where read is fine, the HW RAID 
controller cache must be doing something strange:

--> XFS SW RAID 5: (xfs noatime only, enabled md-raid read optimizations)
# dd if=/dev/zero of=10gb bs=1M count=10240
10737418240 bytes (11 GB) copied, 105.178 seconds, 102 MB/s
# dd if=10gb of=/dev/zero bs=1M count=10240
10737418240 bytes (11 GB) copied, 17.4893 seconds, 614 MB/s

-----

I am sure one of your questions is, well, why use SW RAID5 on the 
controller?  Because SW RAID5 is usually much faster than HW RAID5, at 
least in my tests:

Ctl   Model        Ports   Drives   Units   NotOpt   RRate   VRate   BBU
------------------------------------------------------------------------
c0    9550SXU-12   12      12       3       0        1       4       -

Unit  UnitType  Status         %Cmpl  Stripe  Size(GB)  Cache  AVerify  IgnECC
------------------------------------------------------------------------------
u0    RAID-1    OK             -      -       698.481   ON     ON       OFF
u1    RAID-5    OK             -      64K     5587.85   ON     OFF      OFF
u2    SPARE     OK             -      -       698.629   -      OFF      -

--> XFS:
# dd if=/dev/zero of=10gb bs=1M count=10240
10737418240 bytes (11 GB) copied, 74.5648 seconds, 144 MB/s

--> JFS:
# dd if=/dev/zero of=10gb bs=1M count=10240
10737418240 bytes (11 GB) copied, 108.631 seconds, 98.8 MB/s

The controller is set to performance, and this is nothing close to 
performance.

In RAID0, the controller is ok with the disks JBOD, but I cannot recommend 
buying a controller (12,16,24 port) for Linux SW RAID 5.

Its too bad that there are no regular > 4 port SATA PCI-e controllers out 
there.

Justin.

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Hannes Dorbath]

On 18.07.2007 12:23, Justin Piszcz wrote:
> I am sure one of your questions is, well, why use SW RAID5 on the 
> controller?  Because SW RAID5 is usually much faster than HW RAID5, at 
> least in my tests:

Though that's no answer to your question, I really can't confirm that. 
I'm running a 3Ware 9650 with 8x 7200 SATAs in HW RAID 6. Writes are 
between 340-360MB/sec and reads peak at 430-450MB/sec. That seems to be 
the absolute possible maximum those drives can deliver. The card even 
seems to do some really smart stuff, as the read speed is in excess of 
the physical raw speed of 6 drives.

You have done the usual settings, 3Ware recommends?

echo 64 > /sys/block/sda/queue/max_sectors_kb
echo 512 > /sys/block/sda/queue/nr_requests
echo "deadline" > /sys/block/sda/queue/scheduler
blockdev --setra 16384 /dev/sdX

Have you set the correct su,sw options for XFS?


-- 
Regards,
Hannes Dorbath

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Al Boldi]

Justin Piszcz wrote:
> UltraDense-AS-3ware-R5-9-disks,16G,50676,89,96019,34,46379,9,60267,99,5010
>98,56,248.5,0,16:100000:16/64,240,3,21959,84,1109,10,286,4,22923,91,544,6
> UltraDense-AS-3ware-R5-9-disks,16G,49983,88,96902,37,47951,10,59002,99,529
>121,60,210.3,0,16:100000:16/64,250,3,25506,98,1163,10,268,3,18003,71,772,8
> UltraDense-AS-3ware-R5-9-disks,16G,49811,87,95759,35,48214,10,60153,99,538
>559,61,276.8,0,16:100000:16/64,233,3,25514,97,1100,9,279,3,21398,84,839,9

Is there any easy way to decipher these numbers?


Thanks!

--
Al

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Gabor Gombas]

On Wed, Jul 18, 2007 at 06:23:25AM -0400, Justin Piszcz wrote:

> I recently got a chance to test SW RAID5 using 750GB disks (10) in a RAID5 
> on a 3ware card, model no: 9550SXU-12
>
> The bottom line is the controller is doing some weird caching with writes 
> on SW RAID5 which makes it not worth using.

Did you use the settings documented in
http://www.3ware.com/KB/article.aspx?id=11050 ? Setting nr_requests and
the deadline scheduler doubled the seq write performance for me. Do you
have the latest firmware? Firmware updates can improve performance - at
least for RAID5/6; I somewhat doubt that they care about JBOD
performance that much...

Gabor

-- 
     ---------------------------------------------------------
     MTA SZTAKI Computer and Automation Research Institute
                Hungarian Academy of Sciences
     ---------------------------------------------------------

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Giuseppe Ghibò]

Justin Piszcz ha scritto:

> I recently got a chance to test SW RAID5 using 750GB disks (10) in a 
> RAID5 on a 3ware card, model no: 9550SXU-12
> 
> The bottom line is the controller is doing some weird caching with 
> writes on SW RAID5 which makes it not worth using.
> 
> Recall, with SW RAID5 using regular SATA cards with (mind you) 10 raptors:
> write: 464MB/s
> read: 627MB/s
> 
> Yes, these drives are different, 7200RPM 750GB drives, but write should 
> not be 50-102MB/s as shown below.
> 
> First, lets test RAW performance of these 10 drives:
> 
> Create RAID 0 with 10 750GB Drives:
> # mdadm /dev/md0 --create --level=0 -n 10 /dev/sd[bcdefghjik]1
> mdadm: array /dev/md0 started.
> 
> --> XFS: (xfs default options, no optimizations)
> # dd if=/dev/zero of=10gb bs=1M count=10240
> 10737418240 bytes (11 GB) copied, 22.459 seconds, 478 MB/s
> # dd if=10gb of=/dev/zero bs=1M count=10240
> 10737418240 bytes (11 GB) copied, 28.7843 seconds, 373 MB/s
> 
> --> XFS: (xfs default options, enabled md-raid read optimizations)
> # dd if=/dev/zero of=10gb bs=1M count=10240
> 10737418240 bytes (11 GB) copied, 22.9623 seconds, 468 MB/s
> # dd if=10gb of=/dev/zero bs=1M count=10240
> 10737418240 bytes (11 GB) copied, 17.7328 seconds, 606 MB/s
> 
> Software RAID 5 on a real HW raid controller over 10 750GB disks JBOD:
> 
> UltraDense-AS-3ware-R5-9-disks,16G,50676,89,96019,34,46379,9,60267,99,501098,56,248.5,0,16:100000:16/64,240,3,21959,84,1109,10,286,4,22923,91,544,6 
> 
> UltraDense-AS-3ware-R5-9-disks,16G,49983,88,96902,37,47951,10,59002,99,529121,60,210.3,0,16:100000:16/64,250,3,25506,98,1163,10,268,3,18003,71,772,8 
> 
> UltraDense-AS-3ware-R5-9-disks,16G,49811,87,95759,35,48214,10,60153,99,538559,61,276.8,0,16:100000:16/64,233,3,25514,97,1100,9,279,3,21398,84,839,9 
> 
> 
> Write seems significantly impacted, where read is fine, the HW RAID 
> controller cache must be doing something strange:
> 
> --> XFS SW RAID 5: (xfs noatime only, enabled md-raid read optimizations)
> # dd if=/dev/zero of=10gb bs=1M count=10240
> 10737418240 bytes (11 GB) copied, 105.178 seconds, 102 MB/s
> # dd if=10gb of=/dev/zero bs=1M count=10240
> 10737418240 bytes (11 GB) copied, 17.4893 seconds, 614 MB/s
> 
> -----
> 
> I am sure one of your questions is, well, why use SW RAID5 on the 
> controller?  Because SW RAID5 is usually much faster than HW RAID5, at 
> least in my tests:
> 
> Ctl   Model        Ports   Drives   Units   NotOpt   RRate   VRate   BBU
> ------------------------------------------------------------------------
> c0    9550SXU-12   12      12       3       0        1       4       -
> 
> Unit  UnitType  Status         %Cmpl  Stripe  Size(GB)  Cache  AVerify  
> IgnECC
> ------------------------------------------------------------------------------ 
> 
> u0    RAID-1    OK             -      -       698.481   ON     ON       OFF
> u1    RAID-5    OK             -      64K     5587.85   ON     OFF      OFF
> u2    SPARE     OK             -      -       698.629   -      OFF      -
> 
> --> XFS:
> # dd if=/dev/zero of=10gb bs=1M count=10240
> 10737418240 bytes (11 GB) copied, 74.5648 seconds, 144 MB/s
> 
> --> JFS:
> # dd if=/dev/zero of=10gb bs=1M count=10240
> 10737418240 bytes (11 GB) copied, 108.631 seconds, 98.8 MB/s
> 
> The controller is set to performance, and this is nothing close to 
> performance.

How much is your RAM size? Is the size you tried (10G) at
least twice the size of the RAM seen by the OS? What
are the values returned by hdparm -t /dev/sda (it test only raw reading
speed)?

> 
> In RAID0, the controller is ok with the disks JBOD, but I cannot 
> recommend buying a controller (12,16,24 port) for Linux SW RAID 5.
> 
> Its too bad that there are no regular > 4 port SATA PCI-e controllers 
> out there.
> 
> Justin.
> 

Indeed not exists for PCI-e but Oden has spotted this PCI-X card
(which is around 97$), based on marvell chipset:

http://www.supermicro.com/products/accessories/addon/AoC-SAT2-MV8.cfm

which can be used on motherboard with PCI-X slot (the ASUS M2N32 WS Professional
AM2, or the ASUS P5W64-WS-PRO, both are for consumer desktop and have 2 PCI-X 
slots) though probably if you have either one of that mobo you already have at 
least 10 onboard SATA connectors.

Bye
Giuseppe.

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Justin Piszcz]

On Wed, 18 Jul 2007, Gabor Gombas wrote:

> On Wed, Jul 18, 2007 at 06:23:25AM -0400, Justin Piszcz wrote:
>
>> I recently got a chance to test SW RAID5 using 750GB disks (10) in a RAID5
>> on a 3ware card, model no: 9550SXU-12
>>
>> The bottom line is the controller is doing some weird caching with writes
>> on SW RAID5 which makes it not worth using.
>
> Did you use the settings documented in
> http://www.3ware.com/KB/article.aspx?id=11050 ? Setting nr_requests and
> the deadline scheduler doubled the seq write performance for me. Do you
> have the latest firmware? Firmware updates can improve performance - at
> least for RAID5/6; I somewhat doubt that they care about JBOD
> performance that much...
>
> Gabor
>
> --
>     ---------------------------------------------------------
>     MTA SZTAKI Computer and Automation Research Institute
>                Hungarian Academy of Sciences
>     ---------------------------------------------------------
>

For the HW RAID tests (2) at the bottom of the e-mail, no, I did not set 
nr_requests or use the deadline scheduler.

For the SW RAID tests, I applied similar optimizations, I am probably not 
at the latest firmware.  The main thing I wanted to use though was SW RAID 
because the card itself offers so many ports, however, with write speed 
being that slow I am not sure its a good idea.

Justin.

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4688 bytes --]



On Wed, 18 Jul 2007, Giuseppe Ghibò wrote:

> Justin Piszcz ha scritto:
>
>> I recently got a chance to test SW RAID5 using 750GB disks (10) in a RAID5 
>> on a 3ware card, model no: 9550SXU-12
>> 
>> The bottom line is the controller is doing some weird caching with writes 
>> on SW RAID5 which makes it not worth using.
>> 
>> Recall, with SW RAID5 using regular SATA cards with (mind you) 10 raptors:
>> write: 464MB/s
>> read: 627MB/s
>> 
>> Yes, these drives are different, 7200RPM 750GB drives, but write should not 
>> be 50-102MB/s as shown below.
>> 
>> First, lets test RAW performance of these 10 drives:
>> 
>> Create RAID 0 with 10 750GB Drives:
>> # mdadm /dev/md0 --create --level=0 -n 10 /dev/sd[bcdefghjik]1
>> mdadm: array /dev/md0 started.
>> 
>> --> XFS: (xfs default options, no optimizations)
>> # dd if=/dev/zero of=10gb bs=1M count=10240
>> 10737418240 bytes (11 GB) copied, 22.459 seconds, 478 MB/s
>> # dd if=10gb of=/dev/zero bs=1M count=10240
>> 10737418240 bytes (11 GB) copied, 28.7843 seconds, 373 MB/s
>> 
>> --> XFS: (xfs default options, enabled md-raid read optimizations)
>> # dd if=/dev/zero of=10gb bs=1M count=10240
>> 10737418240 bytes (11 GB) copied, 22.9623 seconds, 468 MB/s
>> # dd if=10gb of=/dev/zero bs=1M count=10240
>> 10737418240 bytes (11 GB) copied, 17.7328 seconds, 606 MB/s
>> 
>> Software RAID 5 on a real HW raid controller over 10 750GB disks JBOD:
>> 
>> UltraDense-AS-3ware-R5-9-disks,16G,50676,89,96019,34,46379,9,60267,99,501098,56,248.5,0,16:100000:16/64,240,3,21959,84,1109,10,286,4,22923,91,544,6 
>> UltraDense-AS-3ware-R5-9-disks,16G,49983,88,96902,37,47951,10,59002,99,529121,60,210.3,0,16:100000:16/64,250,3,25506,98,1163,10,268,3,18003,71,772,8 
>> UltraDense-AS-3ware-R5-9-disks,16G,49811,87,95759,35,48214,10,60153,99,538559,61,276.8,0,16:100000:16/64,233,3,25514,97,1100,9,279,3,21398,84,839,9 
>> 
>> Write seems significantly impacted, where read is fine, the HW RAID 
>> controller cache must be doing something strange:
>> 
>> --> XFS SW RAID 5: (xfs noatime only, enabled md-raid read optimizations)
>> # dd if=/dev/zero of=10gb bs=1M count=10240
>> 10737418240 bytes (11 GB) copied, 105.178 seconds, 102 MB/s
>> # dd if=10gb of=/dev/zero bs=1M count=10240
>> 10737418240 bytes (11 GB) copied, 17.4893 seconds, 614 MB/s
>> 
>> -----
>> 
>> I am sure one of your questions is, well, why use SW RAID5 on the 
>> controller?  Because SW RAID5 is usually much faster than HW RAID5, at 
>> least in my tests:
>> 
>> Ctl   Model        Ports   Drives   Units   NotOpt   RRate   VRate   BBU
>> ------------------------------------------------------------------------
>> c0    9550SXU-12   12      12       3       0        1       4       -
>> 
>> Unit  UnitType  Status         %Cmpl  Stripe  Size(GB)  Cache  AVerify 
>> IgnECC
>> 
>> ------------------------------------------------------------------------------ 
>> u0    RAID-1    OK             -      -       698.481   ON     ON       OFF
>> u1    RAID-5    OK             -      64K     5587.85   ON     OFF      OFF
>> u2    SPARE     OK             -      -       698.629   -      OFF      -
>> 
>> --> XFS:
>> # dd if=/dev/zero of=10gb bs=1M count=10240
>> 10737418240 bytes (11 GB) copied, 74.5648 seconds, 144 MB/s
>> 
>> --> JFS:
>> # dd if=/dev/zero of=10gb bs=1M count=10240
>> 10737418240 bytes (11 GB) copied, 108.631 seconds, 98.8 MB/s
>> 
>> The controller is set to performance, and this is nothing close to 
>> performance.
>
> How much is your RAM size? Is the size you tried (10G) at
> least twice the size of the RAM seen by the OS? What
> are the values returned by hdparm -t /dev/sda (it test only raw reading
> speed)?
>
Total: 4GB of ram-- I am using the array for other things right now, did 
not get a chance to run that.

>> 
>> In RAID0, the controller is ok with the disks JBOD, but I cannot recommend 
>> buying a controller (12,16,24 port) for Linux SW RAID 5.
>> 
>> Its too bad that there are no regular > 4 port SATA PCI-e controllers out 
>> there.
>> 
>> Justin.
>> 
>
> Indeed not exists for PCI-e but Oden has spotted this PCI-X card
> (which is around 97$), based on marvell chipset:
>
> http://www.supermicro.com/products/accessories/addon/AoC-SAT2-MV8.cfm
>
> which can be used on motherboard with PCI-X slot (the ASUS M2N32 WS 
> Professional
> AM2, or the ASUS P5W64-WS-PRO, both are for consumer desktop and have 2 PCI-X 
> slots) though probably if you have either one of that mobo you already have 
> at least 10 onboard SATA connectors.
Indeed, wish there was a PCI-e version!

>
> Bye
> Giuseppe.
>

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Hannes Dorbath]

On 18.07.2007 13:19, Justin Piszcz wrote:
> For the HW RAID tests (2) at the bottom of the e-mail, no, I did not set 
> nr_requests or use the deadline scheduler.
> 
> For the SW RAID tests, I applied similar optimizations, I am probably 
> not at the latest firmware.  The main thing I wanted to use though was 
> SW RAID because the card itself offers so many ports, however, with 
> write speed being that slow I am not sure its a good idea.

I think what you might be experiencing is that XFS can read su,sw values 
from the MD device and will automatically optimize itself, while it 
can't do that for the HW RAID device. It is absolutely essential to 
align your file system, to prevent implicit reads, needed for parity 
calculations.

Set su to the stripe size you have configured in your controller (like 
128K) and sw to 9 (for a 10 disk RAID 5 array).


-- 
Regards,
Hannes Dorbath

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Sander]

Justin Piszcz wrote (ao):
> Its too bad that there are no regular > 4 port SATA PCI-e controllers
> out there.

Is there a disadvantage to using a SaS controller from for example
lsi.com ?

http://lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/index.html

I haven't tried them yet, but on paper it looks like an option, no?

	With kind regards, Sander

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Justin Piszcz]

On Wed, 18 Jul 2007, Hannes Dorbath wrote:

> On 18.07.2007 13:19, Justin Piszcz wrote:
>> For the HW RAID tests (2) at the bottom of the e-mail, no, I did not set 
>> nr_requests or use the deadline scheduler.
>> 
>> For the SW RAID tests, I applied similar optimizations, I am probably not 
>> at the latest firmware.  The main thing I wanted to use though was SW RAID 
>> because the card itself offers so many ports, however, with write speed 
>> being that slow I am not sure its a good idea.
>
> I think what you might be experiencing is that XFS can read su,sw values from 
> the MD device and will automatically optimize itself, while it can't do that 
> for the HW RAID device. It is absolutely essential to align your file system, 
> to prevent implicit reads, needed for parity calculations.
>
> Set su to the stripe size you have configured in your controller (like 128K) 
> and sw to 9 (for a 10 disk RAID 5 array).
>
>
> -- 
> Regards,
> Hannes Dorbath
>

Yes I agree here, but I guess my question is why is SW RAID5 so slow on 
the 3ware card?

Justin.

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Justin Piszcz]

On Wed, 18 Jul 2007, Sander wrote:

> Justin Piszcz wrote (ao):
>> Its too bad that there are no regular > 4 port SATA PCI-e controllers
>> out there.
>
> Is there a disadvantage to using a SaS controller from for example
> lsi.com ?
>
> http://lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/index.html
>
> I haven't tried them yet, but on paper it looks like an option, no?
>
> 	With kind regards, Sander
>
> -- 
> Humilis IT Services and Solutions
> http://www.humilis.net
>

Hmm that is also a good idea- but I am afraid there could be a similiar 
issue where it tries to do weird caching stuff, even when the drives are 
in JBOD.  Unless they offer a !raid SAS controller that supports SATA 
drives.

Justin.

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Hannes Dorbath]

On 18.07.2007 13:32, Justin Piszcz wrote:
> Yes I agree here, but I guess my question is why is SW RAID5 so slow on 
> the 3ware card?

I think the controller does disable the drives write cache when exported 
unconfigured. It does always disable, unless you put the drive in a unit 
  and explicit set the profile to `Performance'.


-- 
Regards,
Hannes Dorbath

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Justin Piszcz]

On Wed, 18 Jul 2007, Hannes Dorbath wrote:

> On 18.07.2007 13:32, Justin Piszcz wrote:
>> Yes I agree here, but I guess my question is why is SW RAID5 so slow on the 
>> 3ware card?
>
> I think the controller does disable the drives write cache when exported 
> unconfigured. It does always disable, unless you put the drive in a unit  and 
> explicit set the profile to `Performance'.
>
>
> -- 
> Regards,
> Hannes Dorbath
>

Ahh, that could explain it!

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Justin Piszcz]

On Wed, 18 Jul 2007, Al Boldi wrote:

> Justin Piszcz wrote:
>> UltraDense-AS-3ware-R5-9-disks,16G,50676,89,96019,34,46379,9,60267,99,5010
>> 98,56,248.5,0,16:100000:16/64,240,3,21959,84,1109,10,286,4,22923,91,544,6
>> UltraDense-AS-3ware-R5-9-disks,16G,49983,88,96902,37,47951,10,59002,99,529
>> 121,60,210.3,0,16:100000:16/64,250,3,25506,98,1163,10,268,3,18003,71,772,8
>> UltraDense-AS-3ware-R5-9-disks,16G,49811,87,95759,35,48214,10,60153,99,538
>> 559,61,276.8,0,16:100000:16/64,233,3,25514,97,1100,9,279,3,21398,84,839,9
>
> Is there any easy way to decipher these numbers?
>
>
> Thanks!
>
> --
> Al
>
>

Yeah, I put them in that format so people crawling the mailing list later 
could re-produce the HTML version:

cat that_file_containing_that_output | /usr/bin/bon_csv2html > out.html

Justin.

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Sander]

Justin Piszcz wrote (ao):
> On Wed, 18 Jul 2007, Sander wrote:
>> Justin Piszcz wrote (ao):
>>> Its too bad that there are no regular > 4 port SATA PCI-e controllers
>>> out there.
>>
>> Is there a disadvantage to using a SaS controller from for example
>> lsi.com ?
>>
>> http://lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/index.html
>>
>> I haven't tried them yet, but on paper it looks like an option, no?
>
> Hmm that is also a good idea- but I am afraid there could be a
> similiar issue where it tries to do weird caching stuff, even when the
> drives are in JBOD. Unless they offer a !raid SAS controller that
> supports SATA drives.

I guess they are cheap enough to try (I haven't yet):

PCIe:
http://lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/lsisas3080xr/index.html

PCI-X
http://lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/lsisas3081er/index.html

All SaS hardware (controllers, chassis, etc) support SATA disks AFAIK.

I have good experience with the Supermicro 8-Port SATA Card
http://www.supermicro.nl/products/accessories/addon/AoC-SAT2-MV8.cfm

But that is PCI-X while you requested PCIe.

	With kind regards, Sander

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Justin Piszcz]

On Wed, 18 Jul 2007, Sander wrote:

> Justin Piszcz wrote (ao):
>> On Wed, 18 Jul 2007, Sander wrote:
>>> Justin Piszcz wrote (ao):
>>>> Its too bad that there are no regular > 4 port SATA PCI-e controllers
>>>> out there.
>>>
>>> Is there a disadvantage to using a SaS controller from for example
>>> lsi.com ?
>>>
>>> http://lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/index.html
>>>
>>> I haven't tried them yet, but on paper it looks like an option, no?
>>
>> Hmm that is also a good idea- but I am afraid there could be a
>> similiar issue where it tries to do weird caching stuff, even when the
>> drives are in JBOD. Unless they offer a !raid SAS controller that
>> supports SATA drives.
>
> I guess they are cheap enough to try (I haven't yet):
>
> PCIe:
> http://lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/lsisas3080xr/index.html
>
> PCI-X
> http://lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/lsisas3081er/index.html
>
> All SaS hardware (controllers, chassis, etc) support SATA disks AFAIK.
>
> I have good experience with the Supermicro 8-Port SATA Card
> http://www.supermicro.nl/products/accessories/addon/AoC-SAT2-MV8.cfm
>
> But that is PCI-X while you requested PCIe.
>
> 	With kind regards, Sander
>
> -- 
> Humilis IT Services and Solutions
> http://www.humilis.net
>


Those links are reversed, but thanks, checking them out now!

Justin.

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 1379 bytes --]

On Wed Jul 18, 2007 at 01:26:11PM +0200, Hannes Dorbath wrote:

> I think what you might be experiencing is that XFS can read su,sw values 
> from the MD device and will automatically optimize itself, while it 
> can't do that for the HW RAID device. It is absolutely essential to 
> align your file system, to prevent implicit reads, needed for parity 
> calculations.
> 
> Set su to the stripe size you have configured in your controller (like 
> 128K) and sw to 9 (for a 10 disk RAID 5 array).
> 
Just to pick up on this one (as I'm about to reformat my array as XFS) -
does this actually work with a hardware controller?  Is there any
assurance that the XFS stripes align with the hardware RAID stripes?  Or
could you just end up offsetting everything so that every 128k chunk on
the XFS side of things fits half-and-half into two hardware raid chunks
(especially if the array has been partitioned)?  In which case would
it be better (performance-wise) to provide the su,sw values or not?

I'm planning on doing some benchmarking first but thought I'd check
whether there's any definitive answers first.

Cheers,
        Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Sander]

Justin Piszcz wrote (ao):
> On Wed, 18 Jul 2007, Sander wrote:
>> I guess they are cheap enough to try (I haven't yet):
>>
>> PCIe:
>> http://lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/lsisas3080xr/index.html
>>
>> PCI-X
>> http://lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/lsisas3081er/index.html
>>
>> All SaS hardware (controllers, chassis, etc) support SATA disks AFAIK.
>>
>> I have good experience with the Supermicro 8-Port SATA Card
>> http://www.supermicro.nl/products/accessories/addon/AoC-SAT2-MV8.cfm
>>
>> But that is PCI-X while you requested PCIe.

> Those links are reversed, but thanks, checking them out now!

Just to see if you are paying attention .. ;-)

If you will try them with Linux, please keep us posted if not too much
trouble.

	With kind regards, Sander

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Gabor Gombas]

On Wed, Jul 18, 2007 at 01:51:16PM +0100, Robin Hill wrote:

> Just to pick up on this one (as I'm about to reformat my array as XFS) -
> does this actually work with a hardware controller?  Is there any
> assurance that the XFS stripes align with the hardware RAID stripes?  Or
> could you just end up offsetting everything so that every 128k chunk on
> the XFS side of things fits half-and-half into two hardware raid chunks
> (especially if the array has been partitioned)?

If you partition the device and does not explicitely align the
partitions on a stripe boundary then you'll get that effect. Also if you
do not use partitions but use LVM instead, then the stripe size should
be a power of 2 meaning the number of data disks should also be a power
of 2 to get the best performance.

> In which case would
> it be better (performance-wise) to provide the su,sw values or not?

Only testing can tell... But if one logical file system block spans
multiple stripes then you will lose some performance; if that will be
noticable or not depends on your usage pattern.

Gabor

-- 
     ---------------------------------------------------------
     MTA SZTAKI Computer and Automation Research Institute
                Hungarian Academy of Sciences
     ---------------------------------------------------------

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Justin Piszcz]

On Wed, 18 Jul 2007, Hannes Dorbath wrote:

> On 18.07.2007 12:23, Justin Piszcz wrote:
>> I am sure one of your questions is, well, why use SW RAID5 on the 
>> controller?  Because SW RAID5 is usually much faster than HW RAID5, at 
>> least in my tests:
>
> Though that's no answer to your question, I really can't confirm that. I'm 
> running a 3Ware 9650 with 8x 7200 SATAs in HW RAID 6. Writes are between 
> 340-360MB/sec and reads peak at 430-450MB/sec. That seems to be the absolute 
> possible maximum those drives can deliver. The card even seems to do some 
> really smart stuff, as the read speed is in excess of the physical raw speed 
> of 6 drives.
>
> You have done the usual settings, 3Ware recommends?
>
> echo 64 > /sys/block/sda/queue/max_sectors_kb
> echo 512 > /sys/block/sda/queue/nr_requests
> echo "deadline" > /sys/block/sda/queue/scheduler
> blockdev --setra 16384 /dev/sdX
>
> Have you set the correct su,sw options for XFS?
>
>
> -- 
> Regards,
> Hannes Dorbath
>

I will try these options during future testing and re-visit the speeds of 
regular HW RAID5, thanks!  I know they are part of the 3ware doc and I 
need to re-bench with these.

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Bryan J. Smith]

On Wed, 2007-07-18 at 12:26 -0400, Justin Piszcz wrote:
> I will try these options during future testing and re-visit the speeds of 
> regular HW RAID5, thanks!  I know they are part of the 3ware doc and I 
> need to re-bench with these.

3Ware cards should be "tuned" for optimal performance, especially if you
have the battery backup unit (BBU) on the 9500 and later cards** , and
some of the settings can be made more "lax."  It's hard to gage hardware
RAID v. software RAID, especially since the latter can cache commits --
at least from the standpoint of various benchmark utilities (like
Bonnie).  There are various arguments that go back'n forth on this.

But if you're using 3Ware in JBOD with software RAID because you've been
fed the standard and quite _incorrect_ statement that you can do
hot-swap, understand that you can't get fully supported hot-swap without
using the 3Ware's hardware RAID volume management (with the segmentation
of its on-board microcontroller intelligence).  3Ware gets a bad rep
because too many MD advocates out there proliferate the 3Ware supports
hot-swap in JBOD -- you need to leverage kernel hot-plug and other
kernel facilities for hot-swap if you use MD software RAID.

You don't need to use the various kernel facilities if you let the 3Ware
intelligence management the volumes.  That's a major, major, missing
piece of information that I've seen far too many MD advocates omit.

-- Bryan

**NOTE:  Earlier cards (6000-9000, pre-9500) only use SRAM logic, no
DRAM cell, so they don't need a BBU.  Long story, SRAM combinational
circuits don't need to be "refreshed" and the board's capacitors can
store enough charge to keep values in SRAM circuits (at least long
enough to flush to disk or once the disks are powered again).  Of
course, on those earlier cards, the SRAM is small, too small to buffer
most sustained RAID-5 writes so I'd only use RAID-10 on them.

-- 
Bryan J. Smith         Professional, Technical Annoyance
mailto:b.j.smith@ieee.org   http://thebs413.blogspot.com
--------------------------------------------------------
        Fission Power:  An Inconvenient Solution

Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Bryan J. Smith]

On Wed, 2007-07-18 at 13:17 +0200, Giuseppe Ghibò wrote:
> Indeed not exists for PCI-e but Oden has spotted this PCI-X card
> (which is around 97$), based on marvell chipset:
> http://www.supermicro.com/products/accessories/addon/AoC-SAT2-MV8.cfm

Yes!  There are several Broadcom and Marvell ASICs that are eight (8)
SATA (and now even some SAS) channels in a single chip, of which, there
are several vendors selling them for $100 boards.  You do _not_ want an
on-board intelligence being an "intermediary" with software RAID, you
want _direct_ access to the SATA (or SAS) channels.  Besides, they are
cheaper.

Justin Piszcz ha scritto:  
> I am sure one of your questions is, well, why use SW RAID5 on the 
> controller?  Because SW RAID5 is usually much faster than HW RAID5,
> at least in my tests ...

Benchmarking direct disk access is a rather poor test of hardware RAID.
to truly evaluate if the off-load from the main system interconnect**
that a hardware RAID gives you, you need to benchmark your actual
_server_ application -- one with MD, the other with hardware RAID.
That's where you can tell if hardware RAID is going to buy you anything.

**NOTE:  It's the system interconnect bottleneck that is of concern, not
so much the CPU.  A modern, superscalar, multi-core x86-64 CPU can do
XORs in its sleep, with just MMX (won't even peg your CPU 10%).  It's
the LOAD/STO push just to get the XOR that ties up your system
interconnect (often with your CPU only being 10% busy if your
application is not processor-bound, but I/O ;-) that's the problem.

It's been my experience that for web services, hardware RAID buys you
_little_, because you're more processor bound than I/O bound, so you
have cycles you can use while your interconnect is doing I/O processing.
But more on the database and file server side, I pair a hardware RAID
(e.g., AMCC/3Ware PPC400-based or Areca IOP/X-Scale-based) with a RX TOE
(_Receive_ TCP Off-load Engine) HBA "NIC" (e.g., LeWiz 4-port GbE PCIe),
which keeps the system inteconnect free for pushing the application data
(instead of doing what is, essentially, "programmed I/O" for the XOR
software RAID operation).

Again benchmark your server _application_, not the disk I/O.


-- 
Bryan J. Smith         Professional, Technical Annoyance
mailto:b.j.smith@ieee.org   http://thebs413.blogspot.com
--------------------------------------------------------
        Fission Power:  An Inconvenient Solution

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html