* Possible performance regression?
@ 2006-01-02 15:58 Alan D. Brunelle
2006-01-02 16:19 ` James Bottomley
2006-01-03 10:35 ` Gerhard Schneider
0 siblings, 2 replies; 5+ messages in thread
From: Alan D. Brunelle @ 2006-01-02 15:58 UTC (permalink / raw)
To: linux-scsi
[-- Attachment #1: Type: text/plain, Size: 5555 bytes --]
As part of HP's Open Source and Linux Organization's Performance and
Scalability Group, I've noticed what looks to be a regression in U320
SCSI performance coming down the pike. Background: These measurements
were performed on Red Hat RHEL4 update 2 (2.6.9-based) and a "generic"
2.6.14-based kernel; I used an HP RX4640 with 4 dual-U320 LSI Logic /
Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 08)
adapters, each with two buses containing 5 72GB/15k U320 drives (for a
total of 40 drives). The runs were done in single-user mode (to minimize
effects from other tasks), and report results in megabytes per second. I
ran tests for various block sized IOs (1KB, 2KB, 4, ...256KB per IO).
While trying to gage other things, I ran across this set of strange
phenomena on single-disk IOs (meaning: tests run one at a time on an
individual disk):
Read averages:
BS(KB) RHEL4U2 2.6.14 %-diff
------ -------- -------- ------
1 23.8 12.3 -48.4%
2 46.6 21.3 -54.2%
4 87.5 44.0 -49.7%
8 89.9 51.8 -42.4%
16 89.9 74.2 -17.4%
32 89.9 89.9 0.0%
64 89.9 89.9 0.0%
128 89.9 89.9 0.0%
256 89.9 89.8 -0.1%
For reads, we see a huge average drop for block sizes up to 32KB.
Write averages:
BS(KB) RHEL4U2 2.6.14 %-diff
------ -------- -------- ------
1 3.6 3.6 -0.2%
2 8.2 8.1 -1.3%
4 19.7 19.0 -3.2%
8 52.6 49.7 -5.5%
16 84.8 78.3 -7.6%
32 84.7 78.8 -7.0%
64 84.8 79.9 -5.8%
128 84.6 81.8 -3.3%
256 84.7 82.0 -3.1%
For writes, an across the board smaller decrease in average performance.
The real kicker though came out when I show individual disk results:
Going back to the read cases: Here are a couple of (representative)
drives - all the data is added as an attachment - I see some drives with
monstrously large performance reductions like:
(0,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 87.8 89.6 89.5 89.6 89.5 89.5 89.5
2.6.14 3.9 7.8 15.6 31.2 62.5 89.6 89.5 89.5 89.6
Differ -83.6% -83.3% -82.2% -65.2% -30.2% 0.0% 0.0% 0.0% 0.0%
(0,1) means target 1 on bus 0, and then the columns represent different
transfer sizes (1K up to 256K). I also see some that show little if any
difference:
(6,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 88.5 90.0 89.9 89.9 89.9 90.0 89.9
2.6.14.2 24.1 47.0 88.5 90.0 89.9 90.0 89.9 89.9 89.9
Differ 0.8% 0.2% 0.0% 0.0% 0.0% 0.1% 0.0% -0.1% 0.0%
(Which tends to explain the average being "only" about 50% as listed in
the first table.) And there are other drives which show some unchanged,
and some changed results:
(3,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.1 86.6 91.2 91.2 91.2 91.2 91.2 91.2
2.6.14.2 23.9 46.3 66.9 31.2 91.2 91.2 91.2 91.2 91.2
Differ 0.4% 0.4% -22.7% -65.8% 0.0% 0.0% 0.0% 0.0% 0.0%
(3,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.0 86.1 89.8 89.7 89.8 89.8 89.8 89.8
2.6.14.2 23.8 46.2 15.6 89.8 62.5 89.8 89.8 89.8 89.3
Differ 0.8% 0.4% -81.9% 0.0% -30.3% 0.0% 0.0% 0.0% -0.6%
One thing I will point out: while I haven't looked to see if the same
drivers consistently show similar issues, I have done multiple runs with
the same characteristics noted in the averages -- meaning, it's not
single run peculiarities in the general sense. [Next I am going to see
if I can match drives from run-to-run to see if the same ones exhibit
"bad" read regressions, or whether that moves around... I am also going
to check out runs on straight 2.6.9 (instead of using Red Hat's kernel...)]
For the write case, the drives exhibited similar differences, but on a
much smaller scale - here are some examples:
(0,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 54.7 86.2 85.9 86.0 85.9 85.9
2.6.14.2 3.5 8.1 19.0 52.0 79.5 80.0 80.9 83.1 83.4
Differ -2.8% -1.2% -3.1% -4.9% -7.8% -6.9% -5.9% -3.3% -2.9%
(0,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.5 53.5 86.8 86.7 86.7 86.3 86.7
2.6.14.2 3.6 8.1 19.0 51.1 79.8 80.5 81.3 83.0 83.9
Differ 0.0% 0.0% -2.6% -4.5% -8.1% -7.2% -6.2% -3.8% -3.2%
(1,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 55.0 86.7 87.3 87.1 87.0 86.8
2.6.14.2 3.5 8.1 19.0 52.0 80.2 80.9 82.7 84.0 84.0
Differ -2.8% -1.2% -3.1% -5.5% -7.5% -7.3% -5.1% -3.4% -3.2%
(1,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.5 53.6 86.2 86.1 85.9 86.1 86.5
2.6.14.2 3.6 8.1 19.0 51.1 78.9 80.0 82.0 83.1 83.3
Differ 0.0% 0.0% -2.6% -4.7% -8.5% -7.1% -4.5% -3.5% -3.7%
So, the natural question - has anybody else noted such things? Any ideas
as to why?
Thanks,
Alan D. Brunelle
[-- Attachment #2: simple_change.txt --]
[-- Type: text/plain, Size: 24690 bytes --]
Read averages:
BS(KB) RHEL4U2 2.6.14.2 %-diff
------ -------- -------- ------
1 23.8 12.3 -48.4%
2 46.6 21.3 -54.2%
4 87.5 44.0 -49.7%
8 89.9 51.8 -42.4%
16 89.9 74.2 -17.4%
32 89.9 89.9 0.0%
64 89.9 89.9 0.0%
128 89.9 89.9 0.0%
256 89.9 89.8 -0.1%
Write averages:
BS(KB) RHEL4U2 2.6.14.2 %-diff
------ -------- -------- ------
1 3.6 3.6 -0.2%
2 8.2 8.1 -1.3%
4 19.7 19.0 -3.2%
8 52.6 49.7 -5.5%
16 84.8 78.3 -7.6%
32 84.7 78.8 -7.0%
64 84.8 79.9 -5.8%
128 84.6 81.8 -3.3%
256 84.7 82.0 -3.1%
-------------------------------
Read data:
(0,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 87.8 89.6 89.5 89.6 89.5 89.5 89.5
2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.6 89.5 89.5 89.6
Differ -83.6% -83.3% -82.2% -65.2% -30.2% 0.0% 0.0% 0.0% 0.1%
(0,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.9 87.6 89.7 89.7 89.7 89.7 89.7 89.7
2.6.14.2 3.9 7.8 15.6 31.3 62.5 89.7 89.7 89.7 89.7
Differ -83.6% -83.4% -82.2% -65.1% -30.3% 0.0% 0.0% 0.0% 0.0%
(0,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 88.5 89.6 89.7 89.6 89.6 89.7 89.6
2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.7 89.6 89.7 89.3
Differ -83.6% -83.3% -82.4% -65.2% -30.3% 0.1% 0.0% 0.0% -0.3%
(0,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 88.6 89.6 89.6 89.5 89.6 89.6 89.6
2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.6 89.6 89.6 89.6
Differ -83.6% -83.3% -82.4% -65.2% -30.2% 0.1% 0.0% 0.0% 0.0%
(0,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 88.3 89.7 89.6 89.7 89.6 89.6 89.6
2.6.14.2 3.9 7.8 15.6 31.2 61.7 89.7 89.6 89.6 89.5
Differ -83.6% -83.3% -82.3% -65.2% -31.1% 0.0% 0.0% 0.0% -0.1%
(1,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 88.3 89.6 89.6 89.7 89.6 89.6 89.6
2.6.14.2 3.9 7.8 16.5 31.2 66.8 89.7 89.6 89.6 89.7
Differ -83.6% -83.3% -81.3% -65.2% -25.4% 0.0% 0.0% 0.0% 0.1%
(1,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.8 88.3 90.0 89.9 90.0 89.9 90.0 89.9
2.6.14.2 24.0 7.8 15.6 31.2 62.5 89.6 90.0 90.0 89.9
Differ 0.4% -83.3% -82.3% -65.3% -30.5% -0.4% 0.1% 0.0% 0.0%
(1,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.8 87.9 90.4 90.4 90.3 90.3 90.3 90.4
2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.9 90.3 90.3 90.2
Differ -83.7% -83.3% -82.3% -65.5% -30.9% -0.4% 0.0% 0.0% -0.2%
(1,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 88.5 89.9 89.8 89.8 89.8 89.9 89.8
2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.9 89.8 89.8 89.7
Differ -83.6% -83.3% -82.4% -65.3% -30.4% 0.1% 0.0% -0.1% -0.1%
(1,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 87.7 89.9 89.8 89.8 89.9 89.8 89.8
2.6.14.2 3.9 7.8 15.6 31.3 61.4 89.8 89.8 89.8 89.8
Differ -83.6% -83.3% -82.2% -65.2% -31.6% 0.0% -0.1% 0.0% 0.0%
(2,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 88.6 89.7 89.7 89.8 89.8 89.8 89.8
2.6.14.2 3.9 16.7 88.8 31.2 89.7 89.7 89.7 89.7 89.8
Differ -83.6% -64.3% 0.2% -65.2% 0.0% -0.1% -0.1% -0.1% 0.0%
(2,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 88.4 90.0 89.9 89.9 89.9 89.9 89.9
2.6.14.2 3.9 10.6 88.5 89.9 89.9 90.0 90.0 90.0 90.0
Differ -83.7% -77.4% 0.1% -0.1% 0.0% 0.1% 0.1% 0.1% 0.1%
(2,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.9 88.1 90.2 90.3 90.2 90.2 90.2 90.2
2.6.14.2 3.9 29.0 88.1 90.3 90.3 90.3 90.2 90.2 90.2
Differ -83.6% -38.2% 0.0% 0.1% 0.0% 0.1% 0.0% 0.0% 0.0%
(2,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.8 87.8 89.9 89.8 89.9 89.8 89.8 89.8
2.6.14.2 3.9 47.0 88.0 89.9 89.8 89.9 89.9 89.8 89.7
Differ -83.7% 0.4% 0.2% 0.0% 0.0% 0.0% 0.1% 0.0% -0.1%
(2,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 87.9 89.9 90.0 90.0 89.9 90.0 90.0
2.6.14.2 24.0 11.5 87.9 90.0 89.9 90.0 89.9 89.9 89.9
Differ 0.8% -75.4% 0.0% 0.1% -0.1% 0.0% 0.0% -0.1% -0.1%
(3,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.1 86.6 91.2 91.2 91.2 91.2 91.2 91.2
2.6.14.2 23.9 46.3 66.9 31.2 91.2 91.2 91.2 91.2 91.2
Differ 0.4% 0.4% -22.7% -65.8% 0.0% 0.0% 0.0% 0.0% 0.0%
(3,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.0 86.1 89.8 89.7 89.8 89.8 89.8 89.8
2.6.14.2 23.8 46.2 15.6 89.8 62.5 89.8 89.8 89.8 89.3
Differ 0.8% 0.4% -81.9% 0.0% -30.3% 0.0% 0.0% 0.0% -0.6%
(3,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.0 86.0 89.7 89.7 89.7 89.7 89.7 89.7
2.6.14.2 23.7 46.2 86.2 31.2 89.7 89.7 89.7 89.7 89.3
Differ 0.4% 0.4% 0.2% -65.2% 0.0% 0.0% 0.0% 0.0% -0.4%
(3,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.0 86.5 89.9 89.9 89.9 90.0 90.0 89.9
2.6.14.2 23.8 46.2 15.6 90.0 62.5 90.0 89.9 89.9 90.0
Differ 0.8% 0.4% -82.0% 0.1% -30.5% 0.1% -0.1% -0.1% 0.1%
(3,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.0 85.9 89.9 89.9 89.9 89.9 89.9 89.9
2.6.14.2 23.7 46.1 86.2 89.9 89.9 89.6 90.0 89.9 89.8
Differ 0.4% 0.2% 0.3% 0.0% 0.0% -0.3% 0.1% 0.0% -0.1%
(4,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 87.7 89.6 89.5 89.6 89.6 89.5 89.6
2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.2 89.6 89.5 89.6
Differ -83.7% -83.4% -82.2% -65.2% -30.2% -0.4% 0.0% 0.0% 0.0%
(4,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 88.2 89.5 89.5 89.5 89.5 89.5 89.6
2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.6 89.5 89.6 89.5
Differ -83.7% -83.4% -82.3% -65.1% -30.2% 0.1% 0.0% 0.1% -0.1%
(4,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 88.8 90.9 90.8 90.9 90.8 90.9 90.9
2.6.14.2 3.9 7.8 15.6 31.2 62.5 90.9 90.9 90.9 90.6
Differ -83.7% -83.4% -82.4% -65.7% -31.2% 0.0% 0.1% 0.0% -0.3%
(4,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 88.6 89.7 89.6 89.7 89.6 89.6 89.7
2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.7 89.6 89.6 89.6
Differ -83.7% -83.4% -82.4% -65.2% -30.2% 0.0% 0.0% 0.0% -0.1%
(4,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 24.0 47.0 88.5 89.9 89.9 89.9 89.9 89.9 89.9
2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.9 89.9 89.9 89.9
Differ -83.8% -83.4% -82.4% -65.3% -30.5% 0.0% 0.0% 0.0% 0.0%
(5,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.2 86.0 89.7 89.8 89.7 89.7 89.7 89.7
2.6.14.2 3.9 7.8 15.6 31.3 62.5 89.7 89.7 89.7 89.6
Differ -83.6% -83.1% -81.9% -65.1% -30.4% 0.0% 0.0% 0.0% -0.1%
(5,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.2 86.1 89.9 89.9 89.9 89.9 89.9 89.9
2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.9 89.9 89.9 89.6
Differ -83.6% -83.1% -81.9% -65.3% -30.5% 0.0% 0.0% 0.0% -0.3%
(5,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.7 46.2 86.7 89.6 89.6 89.6 89.6 89.6 89.6
2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.6 89.6 89.6 89.5
Differ -83.5% -83.1% -82.0% -65.2% -30.2% 0.0% 0.0% 0.0% -0.1%
(5,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.7 46.1 86.5 89.7 89.6 89.7 89.6 89.7 89.6
2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.7 89.7 89.7 89.7
Differ -83.5% -83.1% -82.0% -65.2% -30.2% 0.0% 0.1% 0.0% 0.1%
(5,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.1 86.6 90.6 90.5 90.5 90.6 90.5 90.5
2.6.14.2 3.9 7.8 15.6 31.3 62.5 90.6 90.5 90.5 90.6
Differ -83.6% -83.1% -82.0% -65.5% -30.9% 0.1% -0.1% 0.0% 0.1%
(6,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 89.0 89.6 89.6 89.6 89.6 89.7 89.7
2.6.14.2 24.1 47.1 89.3 89.6 89.7 89.6 89.6 89.7 89.6
Differ 0.8% 0.4% 0.3% 0.0% 0.1% 0.0% 0.0% 0.0% -0.1%
(6,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 88.5 89.8 89.8 89.9 89.9 89.9 89.9
2.6.14.2 24.1 47.1 88.8 89.8 89.9 89.9 89.9 89.8 89.9
Differ 0.8% 0.4% 0.3% 0.0% 0.1% 0.0% 0.0% -0.1% 0.0%
(6,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.9 87.7 89.8 89.9 89.8 89.8 89.9 89.9
2.6.14.2 24.0 7.9 88.1 89.9 89.8 89.9 89.8 89.9 89.7
Differ 0.8% -83.2% 0.5% 0.1% -0.1% 0.1% 0.0% 0.0% -0.2%
(6,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 88.5 90.0 89.9 89.9 89.9 90.0 89.9
2.6.14.2 24.1 47.0 88.5 90.0 89.9 90.0 89.9 89.9 89.9
Differ 0.8% 0.2% 0.0% 0.0% 0.0% 0.1% 0.0% -0.1% 0.0%
(6,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 89.0 90.8 90.7 90.8 90.7 90.8 90.7
2.6.14.2 24.0 9.9 89.2 31.2 90.7 90.7 90.7 90.8 90.8
Differ 0.8% -78.8% 0.2% -65.6% 0.0% -0.1% 0.0% 0.0% 0.1%
(7,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.1 86.5 89.7 89.7 89.7 89.7 89.7 89.7
2.6.14.2 23.8 46.2 86.8 89.7 89.7 89.4 89.7 89.7 89.7
Differ 0.8% 0.2% 0.3% 0.0% 0.0% -0.3% 0.0% 0.0% 0.0%
(7,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.0 86.5 90.7 90.8 90.7 90.8 90.8 90.7
2.6.14.2 23.8 7.8 86.4 90.8 90.7 90.7 90.7 90.8 90.6
Differ 0.8% -83.0% -0.1% 0.1% -0.1% 0.0% -0.1% 0.0% -0.1%
(7,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.1 86.0 89.8 89.7 89.7 89.8 89.8 89.8
2.6.14.2 23.8 46.2 86.3 89.7 61.4 89.7 89.7 89.7 89.5
Differ 0.8% 0.2% 0.3% -0.1% -31.5% 0.0% -0.1% -0.1% -0.3%
(7,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.1 86.6 89.7 89.7 89.6 89.6 89.7 89.6
2.6.14.2 23.8 46.2 15.6 31.2 89.6 89.3 89.6 89.7 89.6
Differ 0.8% 0.2% -82.0% -65.2% -0.1% -0.3% 0.0% 0.0% 0.0%
(7,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.0 86.5 89.6 89.6 89.6 89.6 89.6 89.6
2.6.14.2 18.9 46.2 15.6 31.2 89.6 89.6 89.6 89.6 89.6
Differ -19.9% 0.4% -82.0% -65.2% 0.0% 0.0% 0.0% 0.0% 0.0%
-------------------------------
Write data:
(0,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.5 8.1 19.8 52.5 81.1 81.0 81.6 81.2 82.2
2.6.14.2 3.5 8.1 19.2 50.2 75.7 75.9 77.0 79.0 78.8
Differ 0.0% 0.0% -3.0% -4.4% -6.7% -6.3% -5.6% -2.7% -4.1%
(0,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 20.1 53.4 85.9 86.2 86.4 86.2 86.1
2.6.14.2 3.5 8.1 19.3 51.2 79.8 80.6 81.5 83.4 83.3
Differ -2.8% -1.2% -4.0% -4.1% -7.1% -6.5% -5.7% -3.2% -3.3%
(0,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.5 54.7 85.5 85.7 85.9 85.9 85.2
2.6.14.2 3.5 8.1 18.9 52.3 78.9 79.3 80.5 82.7 82.7
Differ -2.8% -1.2% -3.1% -4.4% -7.7% -7.5% -6.3% -3.7% -2.9%
(0,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 54.7 86.2 85.9 86.0 85.9 85.9
2.6.14.2 3.5 8.1 19.0 52.0 79.5 80.0 80.9 83.1 83.4
Differ -2.8% -1.2% -3.1% -4.9% -7.8% -6.9% -5.9% -3.3% -2.9%
(0,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.5 53.5 86.8 86.7 86.7 86.3 86.7
2.6.14.2 3.6 8.1 19.0 51.1 79.8 80.5 81.3 83.0 83.9
Differ 0.0% 0.0% -2.6% -4.5% -8.1% -7.2% -6.2% -3.8% -3.2%
(1,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 55.0 86.7 87.3 87.1 87.0 86.8
2.6.14.2 3.5 8.1 19.0 52.0 80.2 80.9 82.7 84.0 84.0
Differ -2.8% -1.2% -3.1% -5.5% -7.5% -7.3% -5.1% -3.4% -3.2%
(1,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.5 53.6 86.2 86.1 85.9 86.1 86.5
2.6.14.2 3.6 8.1 19.0 51.1 78.9 80.0 82.0 83.1 83.3
Differ 0.0% 0.0% -2.6% -4.7% -8.5% -7.1% -4.5% -3.5% -3.7%
(1,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 20.0 52.8 83.3 83.5 83.6 82.6 83.2
2.6.14.2 3.6 8.1 19.4 50.2 75.0 78.0 79.5 81.0 80.9
Differ 0.0% -1.2% -3.0% -4.9% -10.0% -6.6% -4.9% -1.9% -2.8%
(1,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 54.8 86.0 85.8 85.5 86.1 85.4
2.6.14.2 3.6 8.1 19.0 51.9 78.6 79.8 81.4 82.6 82.5
Differ 0.0% -1.2% -3.1% -5.3% -8.6% -7.0% -4.8% -4.1% -3.4%
(1,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 20.1 53.9 87.3 87.1 87.3 86.9 87.0
2.6.14.2 3.6 7.7 19.3 51.5 77.4 80.8 83.1 84.1 84.3
Differ 0.0% -6.1% -4.0% -4.5% -11.3% -7.2% -4.8% -3.2% -3.1%
(2,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 52.6 86.5 86.4 86.8 85.9 86.0
2.6.14.2 3.6 8.1 19.0 49.0 80.5 80.0 80.9 83.4 83.4
Differ 0.0% -1.2% -3.1% -6.8% -6.9% -7.4% -6.8% -2.9% -3.0%
(2,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.3 53.1 84.9 84.6 84.8 85.0 84.4
2.6.14.2 3.6 8.1 19.0 50.7 78.0 79.6 79.3 81.5 82.4
Differ 0.0% 0.0% -1.6% -4.5% -8.1% -5.9% -6.5% -4.1% -2.4%
(2,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.9 50.3 85.4 85.6 85.1 85.7 85.3
2.6.14.2 3.6 8.1 19.2 47.9 79.0 79.4 80.5 82.3 82.3
Differ 0.0% -1.2% -3.5% -4.8% -7.5% -7.2% -5.4% -4.0% -3.5%
(2,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.9 51.0 86.9 86.7 86.6 86.7 86.2
2.6.14.2 3.6 8.1 19.1 48.5 80.3 80.3 81.4 83.8 83.9
Differ 0.0% -1.2% -4.0% -4.9% -7.6% -7.4% -6.0% -3.3% -2.7%
(2,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.8 50.2 84.1 84.1 84.5 83.5 83.4
2.6.14.2 3.5 8.1 19.0 47.6 77.8 78.2 79.3 81.7 81.4
Differ -2.8% 0.0% -4.0% -5.2% -7.5% -7.0% -6.2% -2.2% -2.4%
(3,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.9 49.7 86.7 87.0 87.0 86.2 87.1
2.6.14.2 3.6 8.1 19.1 46.4 80.7 81.1 81.0 83.5 83.7
Differ 0.0% -1.2% -4.0% -6.6% -6.9% -6.8% -6.9% -3.1% -3.9%
(3,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.9 51.3 86.6 86.2 86.6 87.1 86.6
2.6.14.2 3.5 8.1 19.1 48.3 80.7 80.6 80.7 83.4 83.8
Differ -2.8% -1.2% -4.0% -5.8% -6.8% -6.5% -6.8% -4.2% -3.2%
(3,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.9 51.2 85.1 85.4 85.4 84.9 85.0
2.6.14.2 3.5 8.1 19.1 48.2 79.0 79.4 79.4 82.2 82.5
Differ -2.8% 0.0% -4.0% -5.9% -7.2% -7.0% -7.0% -3.2% -2.9%
(3,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 51.1 87.8 87.4 87.3 87.6 87.5
2.6.14.2 3.6 8.1 19.0 47.9 81.3 81.3 81.6 84.4 84.7
Differ 0.0% -1.2% -3.1% -6.3% -7.4% -7.0% -6.5% -3.7% -3.2%
(3,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.9 50.6 85.8 85.9 86.0 86.1 85.6
2.6.14.2 3.6 8.1 19.2 48.1 79.8 79.8 80.4 82.7 83.4
Differ 0.0% -1.2% -3.5% -4.9% -7.0% -7.1% -6.5% -3.9% -2.6%
(4,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.9 52.5 87.1 86.4 87.1 86.9 86.8
2.6.14.2 3.5 8.1 19.2 49.2 80.7 80.4 82.1 84.0 83.9
Differ -2.8% -1.2% -3.5% -6.3% -7.3% -6.9% -5.7% -3.3% -3.3%
(4,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.3 53.5 86.9 87.1 86.6 86.7 86.8
2.6.14.2 3.6 8.1 18.9 51.0 80.2 80.6 81.9 84.0 83.9
Differ 0.0% 0.0% -2.1% -4.7% -7.7% -7.5% -5.4% -3.1% -3.3%
(4,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.5 50.2 83.7 83.9 83.7 83.7 83.8
2.6.14.2 3.5 8.0 18.9 48.1 78.1 77.9 79.6 81.4 81.5
Differ -2.8% -1.2% -3.1% -4.2% -6.7% -7.2% -4.9% -2.7% -2.7%
(4,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.4 52.9 84.9 84.6 85.4 84.6 84.9
2.6.14.2 3.5 8.1 18.8 48.6 78.7 79.0 79.6 81.8 82.0
Differ -2.8% -1.2% -3.1% -8.1% -7.3% -6.6% -6.8% -3.3% -3.4%
(4,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.4 52.8 84.4 84.3 84.8 84.3 84.1
2.6.14.2 3.6 8.1 18.9 50.3 78.5 78.4 79.3 80.9 82.2
Differ 0.0% 0.0% -2.6% -4.7% -7.0% -7.0% -6.5% -4.0% -2.3%
(5,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.5 8.1 19.7 50.9 81.4 81.8 81.6 81.3 81.1
2.6.14.2 3.5 7.9 18.8 47.6 74.9 74.8 76.7 77.9 77.2
Differ 0.0% -2.5% -4.6% -6.5% -8.0% -8.6% -6.0% -4.2% -4.8%
(5,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.8 51.2 82.5 83.1 82.6 82.8 82.4
2.6.14.2 3.5 8.0 18.9 47.7 76.8 77.4 79.1 80.4 80.7
Differ -2.8% -1.2% -4.5% -6.8% -6.9% -6.9% -4.2% -2.9% -2.1%
(5,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.5 53.8 86.0 86.0 86.5 85.8 86.2
2.6.14.2 3.5 7.8 19.0 49.7 78.8 80.2 81.5 83.1 83.4
Differ -2.8% -4.9% -2.6% -7.6% -8.4% -6.7% -5.8% -3.1% -3.2%
(5,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.5 8.1 19.3 51.8 79.1 79.0 78.6 78.7 79.2
2.6.14.2 3.5 7.9 18.6 47.1 72.9 74.1 76.2 77.0 77.3
Differ 0.0% -2.5% -3.6% -9.1% -7.8% -6.2% -3.1% -2.2% -2.4%
(5,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 51.2 82.5 82.8 82.7 82.1 82.3
2.6.14.2 3.6 8.0 18.9 46.2 76.0 77.0 79.0 79.9 80.2
Differ 0.0% -2.4% -3.6% -9.8% -7.9% -7.0% -4.5% -2.7% -2.6%
(6,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.5 8.2 19.4 55.5 80.0 80.2 80.4 79.8 80.3
2.6.14.2 3.5 8.1 18.8 52.3 75.1 74.7 75.7 77.5 77.6
Differ 0.0% -1.2% -3.1% -5.8% -6.1% -6.9% -5.8% -2.9% -3.4%
(6,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.3 53.4 80.7 80.1 80.3 80.2 80.0
2.6.14.2 3.6 8.1 18.9 50.5 74.6 75.1 76.3 77.9 77.9
Differ 0.0% -1.2% -2.1% -5.4% -7.6% -6.2% -5.0% -2.9% -2.6%
(6,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 20.1 53.5 86.0 85.6 86.5 86.1 86.3
2.6.14.2 3.6 8.1 19.4 51.1 79.4 80.3 80.7 82.9 83.3
Differ 0.0% -1.2% -3.5% -4.5% -7.7% -6.2% -6.7% -3.7% -3.5%
(6,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.5 53.6 85.8 85.7 86.2 84.3 84.9
2.6.14.2 3.6 8.1 19.0 51.1 78.8 78.5 79.7 80.0 81.7
Differ 0.0% 0.0% -2.6% -4.7% -8.2% -8.4% -7.5% -5.1% -3.8%
(6,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 50.5 78.1 78.4 78.0 78.1 78.3
2.6.14.2 3.5 8.0 18.9 48.3 72.5 72.9 74.5 75.7 77.3
Differ -2.8% -2.4% -3.6% -4.4% -7.2% -7.0% -4.5% -3.1% -1.3%
(7,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.5 53.6 86.6 86.6 86.4 86.5 86.7
2.6.14.2 3.6 8.1 18.9 50.7 80.2 80.2 81.0 83.7 83.8
Differ 0.0% 0.0% -3.1% -5.4% -7.4% -7.4% -6.3% -3.2% -3.3%
(7,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 20.0 51.9 86.6 86.4 87.0 86.9 86.9
2.6.14.2 3.6 8.1 19.2 49.7 80.4 80.9 81.3 84.2 83.8
Differ 0.0% -1.2% -4.0% -4.2% -7.2% -6.4% -6.6% -3.1% -3.6%
(7,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.9 52.7 82.3 82.7 82.5 82.5 82.3
2.6.14.2 3.5 8.1 19.3 50.3 76.8 76.4 78.1 79.7 80.2
Differ -2.8% 0.0% -3.0% -4.6% -6.7% -7.6% -5.3% -3.4% -2.6%
(7,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.5 54.6 86.1 86.4 86.1 86.1 86.3
2.6.14.2 3.5 8.1 18.9 52.1 79.8 80.1 80.6 83.5 83.1
Differ -2.8% -1.2% -3.1% -4.6% -7.3% -7.3% -6.4% -3.0% -3.7%
(7,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.5 54.6 84.6 84.4 84.9 84.2 84.2
2.6.14.2 3.6 8.1 18.9 51.9 78.3 78.3 78.9 81.6 81.7
Differ 0.0% -1.2% -3.1% -4.9% -7.4% -7.2% -7.1% -3.1% -3.0%
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Possible performance regression?
2006-01-02 15:58 Possible performance regression? Alan D. Brunelle
@ 2006-01-02 16:19 ` James Bottomley
2006-01-03 10:35 ` Gerhard Schneider
1 sibling, 0 replies; 5+ messages in thread
From: James Bottomley @ 2006-01-02 16:19 UTC (permalink / raw)
To: Alan D. Brunelle; +Cc: linux-scsi
On Mon, 2006-01-02 at 10:58 -0500, Alan D. Brunelle wrote:
> As part of HP's Open Source and Linux Organization's Performance and
> Scalability Group, I've noticed what looks to be a regression in U320
> SCSI performance coming down the pike. Background: These measurements
> were performed on Red Hat RHEL4 update 2 (2.6.9-based) and a "generic"
> 2.6.14-based kernel; I used an HP RX4640 with 4 dual-U320 LSI Logic /
> Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 08)
> adapters, each with two buses containing 5 72GB/15k U320 drives (for a
> total of 40 drives). The runs were done in single-user mode (to minimize
> effects from other tasks), and report results in megabytes per second. I
> ran tests for various block sized IOs (1KB, 2KB, 4, ...256KB per IO).
This data doesn't seem to be mirrored by tiobench tests on other
drivers, so it's probably fusion specific.
When a similar problem was found in the adaptec driver a while back, it
was due to a bug which set the TCQ depth to 1, which looks very
consistent with your results, so you might try starting there.
James
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Possible performance regression?
2006-01-02 15:58 Possible performance regression? Alan D. Brunelle
2006-01-02 16:19 ` James Bottomley
@ 2006-01-03 10:35 ` Gerhard Schneider
2006-01-03 14:37 ` Alan D. Brunelle
1 sibling, 1 reply; 5+ messages in thread
From: Gerhard Schneider @ 2006-01-03 10:35 UTC (permalink / raw)
To: Alan D. Brunelle; +Cc: linux-scsi
[-- Attachment #1: Type: text/plain, Size: 1416 bytes --]
Alan D. Brunelle wrote:
>
>
> (0,1) 1 2 4 8 16 32 64 128 256
> RHEL4U2 23.8 46.8 87.8 89.6 89.5 89.6 89.5 89.5 89.5
> 2.6.14 3.9 7.8 15.6 31.2 62.5 89.6 89.5 89.5 89.6
> Differ -83.6% -83.3% -82.2% -65.2% -30.2% 0.0% 0.0% 0.0% 0.0%
>
> (0,1) means target 1 on bus 0, and then the columns represent different
> transfer sizes (1K up to 256K). I also see some that show little if any
> difference:
>
> (6,4) 1 2 4 8 16 32 64 128 256
> RHEL4U2 23.9 46.9 88.5 90.0 89.9 89.9 89.9 90.0 89.9
> 2.6.14.2 24.1 47.0 88.5 90.0 89.9 90.0 89.9 89.9 89.9
> Differ 0.8% 0.2% 0.0% 0.0% 0.0% 0.1% 0.0% -0.1% 0.0%
>
Did you apply [PATCH] mptfusion : dv performance fix by
Eric Dean Moore? (Dec. 1st, 2005) W/o that patch the first four disks of
an 1030 are performing 3.5 MB/s instead of 70 :-)
It was funny how slow a RAID 5-0 can be when four of fourteen disks
are slow...
GS
--
Gerhard Schneider
Institute of Lightweight Design and e-Mail:gs@ilsb.tuwien.ac.at
Structural Biomechanics (E317) Tel.: +43 1 58801 31716
Vienna University of Technology / Austria Fax: +43 1 58801 31799
A-1040 Wien, Gusshausstrasse 27-29 http://www.ilsb.tuwien.ac.at/~gs/
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Possible performance regression?
2006-01-03 10:35 ` Gerhard Schneider
@ 2006-01-03 14:37 ` Alan D. Brunelle
2006-01-03 18:37 ` Gerhard Schneider
0 siblings, 1 reply; 5+ messages in thread
From: Alan D. Brunelle @ 2006-01-03 14:37 UTC (permalink / raw)
To: Gerhard Schneider; +Cc: linux-scsi
Gerhard Schneider wrote:
>Alan D. Brunelle wrote:
>
>
>>(0,1) 1 2 4 8 16 32 64 128 256
>>RHEL4U2 23.8 46.8 87.8 89.6 89.5 89.6 89.5 89.5 89.5
>>2.6.14 3.9 7.8 15.6 31.2 62.5 89.6 89.5 89.5 89.6
>>Differ -83.6% -83.3% -82.2% -65.2% -30.2% 0.0% 0.0% 0.0% 0.0%
>>
>>(0,1) means target 1 on bus 0, and then the columns represent different
>>transfer sizes (1K up to 256K). I also see some that show little if any
>>difference:
>>
>>(6,4) 1 2 4 8 16 32 64 128 256
>>RHEL4U2 23.9 46.9 88.5 90.0 89.9 89.9 89.9 90.0 89.9
>>2.6.14.2 24.1 47.0 88.5 90.0 89.9 90.0 89.9 89.9 89.9
>>Differ 0.8% 0.2% 0.0% 0.0% 0.0% 0.1% 0.0% -0.1% 0.0%
>>
>>
>>
>
>Did you apply [PATCH] mptfusion : dv performance fix by
>Eric Dean Moore? (Dec. 1st, 2005) W/o that patch the first four disks of
>an 1030 are performing 3.5 MB/s instead of 70 :-)
>It was funny how slow a RAID 5-0 can be when four of fourteen disks
>are slow...
>
> GS
>
>
>
I do not know how to grab that specific patch, but I am just trying a
2.6.15-rc6 kernel, and things seem much better. I'm hoping that
2.6.15-rc6 contains the patch you mention?
Thanks,
Alan
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Possible performance regression?
2006-01-03 14:37 ` Alan D. Brunelle
@ 2006-01-03 18:37 ` Gerhard Schneider
0 siblings, 0 replies; 5+ messages in thread
From: Gerhard Schneider @ 2006-01-03 18:37 UTC (permalink / raw)
To: Alan D. Brunelle; +Cc: linux-scsi
[-- Attachment #1: Type: text/plain, Size: 743 bytes --]
Alan D. Brunelle schrieb:
> I do not know how to grab that specific patch, but I am just trying a
> 2.6.15-rc6 kernel, and things seem much better. I'm hoping that
> 2.6.15-rc6 contains the patch you mention?
>
I do not know -rc6, but if mptbase.c contains the functions
mpt_alt_ioc_wait and mpt_bringup_adapter the patch
has been applied. Worked for me :-)
GS
--
Gerhard Schneider
Institute of Lightweight Design and e-Mail: gs@ilsb.tuwien.ac.at
Structural Biomechanics (E317) Tel.: +43 1 58801 31716
Vienna University of Technology / Austria Fax: +43 1 58801 31799
A-1040 Wien, Gusshausstrasse 27-29 http://www.ilsb.tuwien.ac.at/~gs/
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2006-01-03 18:38 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2006-01-02 15:58 Possible performance regression? Alan D. Brunelle
2006-01-02 16:19 ` James Bottomley
2006-01-03 10:35 ` Gerhard Schneider
2006-01-03 14:37 ` Alan D. Brunelle
2006-01-03 18:37 ` Gerhard Schneider
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox