mpt performance differences

[David Zanetti <david.zanetti@catalyst.net.nz>]

[-- Attachment #1: Type: text/plain, Size: 4232 bytes --]

Hi there!

We have two machines, both with LSI MPT Fusion SCSI cards in them. They
have the same CPUs, memory, etc. Their disk performance is quite
different however.

The two machines are "crusaders" and "minsk". They're both dual Operton
servers, with six 10k 72G SCA disks in them, arranged as software
RAID-10. RAM in both is the same, 4G. Both are running Debian sarge
AMD64. 

Initally we dismissed the huge difference in hdparm -tT numbers as just
hdparm not really being a useful test, so we've run bonnie with default
settings, and there's still quite a difference between them:

Minsk
=====
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
minsk            8G 54409  98 147492  32 18438   5 25201  44 32090   6 601.4  1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  5569  99 +++++ +++ +++++ +++  5819  99 +++++ +++ 20239  99
/dev/md3:
 Timing cached reads:   3432 MB in  2.00 seconds = 1715.40 MB/sec
 Timing buffered disk reads:  224 MB in  3.02 seconds =  74.13 MB/sec

Crusaders
=========
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
crusaders        8G 52445  95 104509  19 21752   6 30754  54 86996  17 520.5  1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  5824  99 +++++ +++ +++++ +++  5917  99 +++++ +++ 19196  99
/dev/md1:
 Timing cached reads:   3836 MB in  2.00 seconds = 1916.37 MB/sec
 Timing buffered disk reads:  418 MB in  3.00 seconds = 139.17 MB/sec

The bit we're concerned about is the read speeds, block is way lower on
minsk than crusaders.

Kernel used is 2.6.12-rc3 with 3.03.00 patches from lsil.com applied.
I've tried various combinations of kernel, without any improvement.
We've also tried the kernel on crusaders, but performance was even worse
then (as in, around 3M/s, instead of 30 we get now..)

We suspect it's caused by a quirk of a newer version of the silicon on
the controller, they're slightly different in lspci:

Minsk
=====
0000:03:02.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030
PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 08)
        Subsystem: LSI Logic / Symbios Logic: Unknown device 1060
        Flags: bus master, 66MHz, medium devsel, latency 128, IRQ 30
        I/O ports at 3000 [size=256]
        Memory at fe120000 (64-bit, non-prefetchable) [size=128K]
        Memory at fe100000 (64-bit, non-prefetchable) [size=128K]
        Capabilities: [50] Power Management version 2
        Capabilities: [58] Message Signalled Interrupts: 64bit+
Queue=0/0 Enable-
        Capabilities: [68] PCI-X non-bridge device.

Crusaders
=========
0000:03:02.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030
PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
        Subsystem: LSI Logic / Symbios Logic: Unknown device 1060
        Flags: bus master, 66MHz, medium devsel, latency 128, IRQ 30
        I/O ports at 4000 [size=256]
        Memory at fe120000 (64-bit, non-prefetchable) [size=128K]
        Memory at fe100000 (64-bit, non-prefetchable) [size=128K]
        Capabilities: [50] Power Management version 2
        Capabilities: [58] Message Signalled Interrupts: 64bit+
Queue=0/0 Enable-
        Capabilities: [68] PCI-X non-bridge device.

Has anyone else seen this effect, or aware of a possible cause?

Thanks!

-- 
David Zanetti <david.zanetti@catalyst.net.nz>
Team Leader, Systems Administration
Catalyst IT Limited
+64-4-8032233 +64-21-402260

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]