Poor I/O Performance with MegaRaid SATA 150-4; bug or feature?

Poor I/O Performance with MegaRaid SATA 150-4; bug or feature?

[Matt M. Valites]

Hail List,

I've been banging my head against this for a few days, and I wanted to
see if anyone here could lend a hand.

I have the following configuration:
P4 3.x Ghz
2GB Ram;
2 x 36GB WD Raptors; in a RAID1 (sda)
2 x 74GB WD Raptor (those 10K RPM SATA drives) in a RAID1(sdb)
Two free PCI-X slots, one of which occupied by a LSI MegaRaid SATA 150-4.

The problem is I/O on either one of these RAID devices seems to
be less than half what I'm expecting.   The file system used in my testing is
XFS, and I'm running kernel 2.6.11.6.

The test I'm doing is a simple:
# time dd if=/dev/zero of=./crap.file bs=1024 count=209715
Which results in a runtime of about ~53s, in the best case, with all the
scary write cache enabled.    I've tried with deadline, and
anticipatory.  I've also tried several kernels, namely a recent 2.4, so
I could test megaraid and megaraid2, similar results.

On my desktop box, with one of these drives connected via SATA, i get
~25s, also XFS.  (2.6.11-gentoo-r6 x86_64).

Is this an expected result?  I'm seeing much higher numbers posted around the
'Net.  Most of those results are from Windows boxes.

I've uploaded my kernel config, lspci -v, and two opreports of a bonnie++ run
to: http://www.muixa.com/lkml/

Any thoughts would be appreciated.
Thanks,
--
Matt M. Valites

Re: Poor I/O Performance with MegaRaid SATA 150-4; bug or feature?

[Andre Tomt]

Matt M. Valites wrote:
> Hail List,
> 
> I've been banging my head against this for a few days, and I wanted to
> see if anyone here could lend a hand.
> 
> I have the following configuration:
> P4 3.x Ghz
> 2GB Ram;
> 2 x 36GB WD Raptors; in a RAID1 (sda)
> 2 x 74GB WD Raptor (those 10K RPM SATA drives) in a RAID1(sdb)
> Two free PCI-X slots, one of which occupied by a LSI MegaRaid SATA 150-4.
> 
> The problem is I/O on either one of these RAID devices seems to
> be less than half what I'm expecting.   The file system used in my testing is
> XFS, and I'm running kernel 2.6.11.6.
> 
> The test I'm doing is a simple:
> # time dd if=/dev/zero of=./crap.file bs=1024 count=209715
> Which results in a runtime of about ~53s, in the best case, with all the
> scary write cache enabled.    I've tried with deadline, and
> anticipatory.  I've also tried several kernels, namely a recent 2.4, so
> I could test megaraid and megaraid2, similar results.
> 
> On my desktop box, with one of these drives connected via SATA, i get
> ~25s, also XFS.  (2.6.11-gentoo-r6 x86_64).
> 
> Is this an expected result?  I'm seeing much higher numbers posted around the
> 'Net.  Most of those results are from Windows boxes.
> 
> I've uploaded my kernel config, lspci -v, and two opreports of a bonnie++ run
> to: http://www.muixa.com/lkml/

I also have one of those cards, at home. I've come to the conclusion 
that they're just too old. No NCQ and such other fancy features (for 
gods sake, the controllers on the card are sil 3112's!). It's probably 
not even PCI-X native.

The only thing that can bring its performance reseanably up to speed is 
using write-back instead of write-through on the array. Also try 
enabling the write-cache on the drives (all doable in the cards bios 
config, not sure if this is what you meant with "with all the scary 
write cache enabled"). Doing this is on the other hand not very good for 
your data integrity, not good at all.

If only NCQ/TCQ was in, it would have a chance of having decent 
performance using write-through. A cool experiment would be setting up 
the drives as invidual drives on the card, and use md software raid over it.

Next time I'll probably just use md software raid over a 3ware 9xxx 
(JBOD-mode) or AHCI controller. I'm feeling quite uneasy about vendor 
lock in nowadays. Groan.

-- 
Cheers,
André Tomt

Re: Poor I/O Performance with MegaRaid SATA 150-4; bug or feature?

[Matt M. Valites]

Andre Tomt wrote:

> Matt M. Valites wrote:
>
>> Hail List,
>>
>> I've been banging my head against this for a few days, and I wanted to
>> see if anyone here could lend a hand.
>>
>> I have the following configuration:
>> P4 3.x Ghz
>> 2GB Ram;
>> 2 x 36GB WD Raptors; in a RAID1 (sda)
>> 2 x 74GB WD Raptor (those 10K RPM SATA drives) in a RAID1(sdb)
>> Two free PCI-X slots, one of which occupied by a LSI MegaRaid SATA
>> 150-4.
>>
>> The problem is I/O on either one of these RAID devices seems to
>> be less than half what I'm expecting.   The file system used in my
>> testing is
>> XFS, and I'm running kernel 2.6.11.6.
>>
>> The test I'm doing is a simple:
>> # time dd if=/dev/zero of=./crap.file bs=1024 count=209715
>> Which results in a runtime of about ~53s, in the best case, with all the
>> scary write cache enabled.    I've tried with deadline, and
>> anticipatory.  I've also tried several kernels, namely a recent 2.4, so
>> I could test megaraid and megaraid2, similar results.
>>
>> On my desktop box, with one of these drives connected via SATA, i get
>> ~25s, also XFS.  (2.6.11-gentoo-r6 x86_64).
>>
>> Is this an expected result?  I'm seeing much higher numbers posted
>> around the
>> 'Net.  Most of those results are from Windows boxes.
>>
>> I've uploaded my kernel config, lspci -v, and two opreports of a
>> bonnie++ run
>> to: http://www.muixa.com/lkml/
>
>
> I also have one of those cards, at home. I've come to the conclusion
> that they're just too old. No NCQ and such other fancy features (for
> gods sake, the controllers on the card are sil 3112's!). It's probably
> not even PCI-X native.
>
> The only thing that can bring its performance reseanably up to speed
> is using write-back instead of write-through on the array. Also try
> enabling the write-cache on the drives (all doable in the cards bios
> config, not sure if this is what you meant with "with all the scary
> write cache enabled"). Doing this is on the other hand not very good
> for your data integrity, not good at all.
>
> If only NCQ/TCQ was in, it would have a chance of having decent
> performance using write-through. A cool experiment would be setting up
> the drives as invidual drives on the card, and use md software raid
> over it.
>
> Next time I'll probably just use md software raid over a 3ware 9xxx
> (JBOD-mode) or AHCI controller. I'm feeling quite uneasy about vendor
> lock in nowadays. Groan.
>
Andre,

Thanks for the reply.

I wouldn't have expected NCQ/TCQ to have such an effect on a direct I/O
test like dd.    Even more disturbing, changing to a raid0 gives me
close to the same results.
When I enable write-cache, and "cached I/O" in the cards BIOS, i get
results closer to a fast 7200RPM disk.    Still nothing like the
performance I know these raptors are capable of.  Write-cache isn't an
option for this machine, since it's slated to hold important
version-control data :)

I could punt on this HW RAID idea, and just go with AHCI, but i've only
got two ports free on the board.   So, i need at least two more ports to
pull it off, and I don't know of any SATA add-in card that has good
driver support...     Have you tested the RAID1 throughput of the 3ware
9x cards? 

(this is why I still like SCSI... )
--
Matt M. Valites