Promise SX8 driver performance

Promise SX8 driver performance

[Matt]

Hello all. I've got a Promise SX8 SATA controller and the performance
seems a little off. I'm using a debian-built 2.6.9 kernel and the
read/write I/O seems to level off around 80-90 MB/s. When I use the same
hardware in Windows, sequential I/O goes up to 218 MB/s. (That's about
the limit of my motherboard's buses.) If I move some disks to the
on-board SATA controller, I can get the I/O to go higher but never as
high as in Windows.

Is this a problem with my expectations, my testing, or the driver? 


Some Bonnie numbers for a raid 0 array of Seagate 7200.7's (2 GB file,
sequential, MB/s):

Disks/cntl        Write    Read
-----             -----    ----
1 (sx8)              57      63
2 (sx8)              69      75
2 (ESB)             111     121
3 (sx8)              75      79
3 (1 sx8, 2 ESB)    141     125
4 (sx8)              89      80
4 (2 sx8, 2 ESB)    135     150
5 (sx8)              88      79
6 (sx8)              91      75

Some numbers from Sandra (ugh) in W2K Pro with the latest Promise
drivers and using software RAID 0 (sequential, MB/s):

Disks/cntl      Write    Read
-----           -----    ----
1 (sx8)            56      62
2 (sx8)           106     122
3 (sx8)           151     176
4 (sx8)           178     209
5 (sx8)           180     218
6 (sx8)           179     218
6 (4 sx8, 2 ESB)  169     207


My full hardware:

Supermicro P4SCi motherboard (7210 chipset, 6300ESB SATA on-board)
3.0 GHz P4 Northwood w/ HT enabled
(2) 512MB DDR400 dual channel (non-ECC)
Promise SX8 SATA controller
(6) 200 GB Seagate 7200.7
(1) 40 GB Seagate Baraccuda IV (boot)
Adaptec 2940U

Promise SX8 performance issues and CARM_MAX_Q

[Bernard Blackham]

Hi,

Playing with a recently acquired Promise SX8 card, we've found
similar performance results to Matt's post to lkml a few months back
at http://marc.theaimsgroup.com/?l=linux-kernel&m=110175890323356&w=2

It appears that the driver is only submitting one command at a time
per port, which is at least one cause of the slowdowns. By raising
CARM_MAX_Q from 1 to 3 in drivers/block/sx8.c (it was 3 in an
earlier pre-merge incarnation of carmel.c), we're getting very
notable speed improvements, with no side effects just yet.

Knowing very little about what this change has actually done, I've a
few questions: 

 - Should this be considered dangerous?
 - Why was it taken from 3 to 1?
 - Is CARM_MAX_Q a number defined (or limited) by the hardware?

Thanks in advance,

Bernard.

-- 
 Bernard Blackham <bernard at blackham dot com dot au>

Re: Promise SX8 performance issues and CARM_MAX_Q

[Jeff Garzik]

Bernard Blackham wrote:
> Hi,
> 
> Playing with a recently acquired Promise SX8 card, we've found
> similar performance results to Matt's post to lkml a few months back
> at http://marc.theaimsgroup.com/?l=linux-kernel&m=110175890323356&w=2
> 
> It appears that the driver is only submitting one command at a time
> per port, which is at least one cause of the slowdowns. By raising
> CARM_MAX_Q from 1 to 3 in drivers/block/sx8.c (it was 3 in an
> earlier pre-merge incarnation of carmel.c), we're getting very
> notable speed improvements, with no side effects just yet.
> 
> Knowing very little about what this change has actually done, I've a
> few questions: 
> 
>  - Should this be considered dangerous?
>  - Why was it taken from 3 to 1?
>  - Is CARM_MAX_Q a number defined (or limited) by the hardware?

In multi-port stress tests, we couldn't get SX8 to function reliably 
without locking up or corrupting data, with more than one outstanding 
command.

Maybe a new firmware has solved this by now.

	Jeff

Re: Promise SX8 performance issues and CARM_MAX_Q

[Matthew Collins]

Jeff Garzik wrote:

> Bernard Blackham wrote:
>
>> Hi,
>>
>> Playing with a recently acquired Promise SX8 card, we've found
>> similar performance results to Matt's post to lkml a few months back
>> at http://marc.theaimsgroup.com/?l=linux-kernel&m=110175890323356&w=2
>>
>> It appears that the driver is only submitting one command at a time
>> per port, which is at least one cause of the slowdowns. By raising
>> CARM_MAX_Q from 1 to 3 in drivers/block/sx8.c (it was 3 in an
>> earlier pre-merge incarnation of carmel.c), we're getting very
>> notable speed improvements, with no side effects just yet.
>>
>> Knowing very little about what this change has actually done, I've a
>> few questions:
>>  - Should this be considered dangerous?
>>  - Why was it taken from 3 to 1?
>>  - Is CARM_MAX_Q a number defined (or limited) by the hardware?
>
>
> In multi-port stress tests, we couldn't get SX8 to function reliably 
> without locking up or corrupting data, with more than one outstanding 
> command.
>
> Maybe a new firmware has solved this by now.
>
>     Jeff
>
>
>
Indeed there does seem to be new firmware out as of 2/23/05. I ran my 
tests with the 9/10/04 firmware but I did not adjust the CARM_MAX_Q 
value. Do either of you happen to know what firmware revisions you 
tested under?

I've put the machine with the SX8 controller into production despite the 
performance issues so I'm not going to be of any use for testing 
revisions to the driver :(

Matt

Re: Promise SX8 performance issues and CARM_MAX_Q

[Jeff Garzik]

Matthew Collins wrote:
> Jeff Garzik wrote:
> 
>> Bernard Blackham wrote:
>>
>>> Hi,
>>>
>>> Playing with a recently acquired Promise SX8 card, we've found
>>> similar performance results to Matt's post to lkml a few months back
>>> at http://marc.theaimsgroup.com/?l=linux-kernel&m=110175890323356&w=2
>>>
>>> It appears that the driver is only submitting one command at a time
>>> per port, which is at least one cause of the slowdowns. By raising
>>> CARM_MAX_Q from 1 to 3 in drivers/block/sx8.c (it was 3 in an
>>> earlier pre-merge incarnation of carmel.c), we're getting very
>>> notable speed improvements, with no side effects just yet.
>>>
>>> Knowing very little about what this change has actually done, I've a
>>> few questions:
>>>  - Should this be considered dangerous?
>>>  - Why was it taken from 3 to 1?
>>>  - Is CARM_MAX_Q a number defined (or limited) by the hardware?
>>
>>
>>
>> In multi-port stress tests, we couldn't get SX8 to function reliably 
>> without locking up or corrupting data, with more than one outstanding 
>> command.
>>
>> Maybe a new firmware has solved this by now.
>>
>>     Jeff
>>
>>
>>
> Indeed there does seem to be new firmware out as of 2/23/05. I ran my 
> tests with the 9/10/04 firmware but I did not adjust the CARM_MAX_Q 
> value. Do either of you happen to know what firmware revisions you 
> tested under?
> 
> I've put the machine with the SX8 controller into production despite the 
> performance issues so I'm not going to be of any use for testing 
> revisions to the driver :(

The driver was developed on a pre-production board, so its entirely 
possible Promise fixed this issue.

The driver should be solid for _at least_ CARM_MAX_Q==31, presuming that 
the firmware doesn't choke.

Anybody willing to
(a) change CARM_MAX_Q
(b) use the latest firmware
(c) stress the card with fsx, badblocks, iozone, and other tools
     on -multiple ports- simultaneously.

?

	Jeff

Re: Promise SX8 performance issues and CARM_MAX_Q

[Bernard Blackham]

On Thu, Mar 24, 2005 at 11:09:17PM -0500, Jeff Garzik wrote:
> >Indeed there does seem to be new firmware out as of 2/23/05. I ran my 
> >tests with the 9/10/04 firmware but I did not adjust the CARM_MAX_Q 
> >value. Do either of you happen to know what firmware revisions you 
> >tested under?

BIOS boot messages didn't seem to indicate a firmware version, but
this card was only available in stock in early March, so its fairly
likely to have the latest firmware (2/23/05).

> The driver should be solid for _at least_ CARM_MAX_Q==31, presuming that 
> the firmware doesn't choke.

We've run it here with CARM_MAX_Q = 32 with a custom stress testing
script and haven't had any incidents so far. CARM_MAX_Q = 33 goes
bad instantly.

> (c) stress the card with fsx, badblocks, iozone, and other tools
>     on -multiple ports- simultaneously.

I'll give those a workout over the weekend and let you know the
results.

Thanks,

Bernard.

-- 
 Bernard Blackham <bernard at blackham dot com dot au>

Re: Promise SX8 performance issues and CARM_MAX_Q

[Bernard Blackham]

On Thu, Mar 24, 2005 at 11:09:17PM -0500, Jeff Garzik wrote:
> The driver was developed on a pre-production board, so its entirely 
> possible Promise fixed this issue.
> 
> The driver should be solid for _at least_ CARM_MAX_Q==31, presuming that 
> the firmware doesn't choke.

Stressing this SX8 card broke with CARM_MAX_Q anything higher than
16. Tests included simultaneous iozones, bonnie++'s, fsx, badblocks,
and mkfs'ing, on 4 ports. Setting CARM_MAX_Q to 17 or above
resulted in errors such as:

Buffer I/O error on device sx80_0p1, logical block 108828
lost page write due to I/O error on sx80_0p1
end_request: I/O error, dev sx80_2, sector 389167

It's been thrashing away with various tests most of today with
CARM_MAX_Q = 16, and it hasn't missed a beat.

Thanks must go to Matt Johnston who actually did all the work of
giving the card and drives a hard time.

Bernard.

-- 
 Bernard Blackham <bernard at blackham dot com dot au>