* Performance Degradation: 2.6.8-rc4-bk1 vs RHEL 2.4.21-15.0.3
@ 2004-08-12 12:45 Omar Kilani
[not found] ` <295911442.20040812150922@dns.toxicfilms.tv>
0 siblings, 1 reply; 4+ messages in thread
From: Omar Kilani @ 2004-08-12 12:45 UTC (permalink / raw)
To: linux-kernel
List,
I'm doing some "benchmarking" using ApacheBench against a bunch of
different operating systems and kernel versions, and noticed what
appears (by appears I mean "this may be intended behavior that I am
not aware of") a severe performance degradation between 2.4 and 2.6.
In this case, I'm testing the same version of Apache (2.0.50) on the
same hardware, using my own distributions (one is 2.4 based and uses
the RHEL 2.4.21-15.0.3 kernel, the other is 2.6 based and uses
2.6.8-rc3, although 2.6.8-rc4-bk1 exhibits the same issues.)
Hardware: Pentium 4 2.8GHz HT (running the SMP kernel) with 512MB of
RAM, and an 80Gb WD HDD.
Some numbers:
2.4.21-2ts:
[root@rio root]# hdparm -T -t /dev/hda
/dev/hda:
Timing buffer-cache reads: 2316 MB in 2.00 seconds = 1158.00 MB/sec
Timing buffered disk reads: 176 MB in 3.01 seconds = 58.47 MB/sec
[root@rio i686-pc-linux-gnu]# ./bw_tcp -s
[root@rio i686-pc-linux-gnu]# ./bw_tcp 127.0.0.1
0.065536 1284.40 MB/sec
[root@rio i686-pc-linux-gnu]# ./bw_tcp 127.0.0.1
0.065536 828.20 MB/sec
[root@rio i686-pc-linux-gnu]# ./bw_tcp 127.0.0.1
0.065536 1346.99 MB/sec
[root@rio i686-pc-linux-gnu]# ./bw_tcp 127.0.0.1
0.065536 1372.90 MB/sec
[root@rio i686-pc-linux-gnu]# ./bw_tcp 127.0.0.1
0.065536 871.46 MB/sec
[root@rio root]# ab -n 100000 -c 2 http://localhost/index.html
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.141 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Finished 100000 requests
Server Software: Apache/2.0.50
Server Hostname: localhost
Server Port: 80
Document Path: /index.html
Document Length: 51200 bytes
Concurrency Level: 2
Time taken for tests: 34.37600 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 852584179 bytes
HTML transferred: 825083904 bytes
Requests per second: 2937.93 [#/sec] (mean)
Time per request: 0.681 [ms] (mean)
Time per request: 0.340 [ms] (mean, across all concurrent requests)
Transfer rate: 24461.21 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 0 0 0.0 0 5
Waiting: 0 0 0.0 0 2
Total: 0 0 0.0 0 5
Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 0
95% 0
98% 0
99% 0
100% 5 (longest request)
Multiple runs of ab produce similar results.
2.6.8-rc4-bk1:
[root@odin vm]# hdparm -T -t /dev/hda
/dev/hda:
Timing buffer-cache reads: 2384 MB in 2.00 seconds = 1192.18 MB/sec
Timing buffered disk reads: 124 MB in 3.01 seconds = 41.20 MB/sec
[root@odin i686-pc-linux-gnu]# ./bw_tcp -s
[root@odin i686-pc-linux-gnu]# ./bw_tcp 127.0.0.1
0.065536 404.62 MB/sec
(Ran this 5 times with no real difference between runs.)
[root@odin ~]# ab -n 100000 -c 2 http://localhost/index.html
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.141 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Finished 100000 requests
Server Software: Apache/2.0.50
Server Hostname: localhost
Server Port: 80
Document Path: /index.html
Document Length: 51200 bytes
Concurrency Level: 2
Time taken for tests: 54.895184 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 852032704 bytes
HTML transferred: 825032704 bytes
Requests per second: 1821.65 [#/sec] (mean)
Time per request: 1.098 [ms] (mean)
Time per request: 0.549 [ms] (mean, across all concurrent requests)
Transfer rate: 15157.30 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.2 0 47
Processing: 0 0 0.6 0 55
Waiting: 0 0 0.2 0 54
Total: 0 0 0.6 0 55
Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 0
95% 1
98% 1
99% 1
100% 55 (longest request)
Multiple runs of ab produce the same results.
So it seems to me that IDE performance (chipset is ICH5) has
decreased, as well as net loopback performance, and this may be the
cause of the performance degradation.
Are there any known issues with either of these? What else could be
effecting these results?
Please Advise.
Regards,
Omar Kilani
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Performance Degradation: 2.6.8-rc4-bk1 vs RHEL 2.4.21-15.0.3
[not found] ` <295911442.20040812150922@dns.toxicfilms.tv>
@ 2004-08-12 13:35 ` Omar Kilani
2004-08-12 13:56 ` Omar Kilani
0 siblings, 1 reply; 4+ messages in thread
From: Omar Kilani @ 2004-08-12 13:35 UTC (permalink / raw)
To: Maciej Soltysiak; +Cc: linux-kernel
Maciej,
On Thu, 12 Aug 2004 15:09:22 +0200, Maciej Soltysiak
<solt@dns.toxicfilms.tv> wrote:
> Hi,
>
> Just a wild guess, please try the same tests with 2.6 kernels with
> echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
This halves all my performance values.
I've just tried 2.6.8-rc4-mm1 with the same results.
I also tested the 2.6.8-rc4-mm1 kernel on a dual P4 Xeon 3.2GHz with
2MB Cache (with HT, so 4 logical processors) using 15K RPM U320
Fujitsu SCSI drives.
[root@minbar root]# hdparm -T -t /dev/sda
/dev/sda:
Timing buffer-cache reads: 1852 MB in 2.00 seconds = 927.07 MB/sec
Timing buffered disk reads: 218 MB in 3.02 seconds = 72.29 MB/sec
[root@minbar root]# ./bw_tcp 127.0.0.1
0.065536 333.85 MB/sec
Ran this 5 times with no real difference in value.
[root@minbar root]# ab -n 100000 -c 2 http://localhost/index.html
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.141 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Finished 100000 requests
Server Software: Apache/2.0.50
Server Hostname: localhost
Server Port: 80
Document Path: /index.html
Document Length: 51200 bytes
Concurrency Level: 2
Time taken for tests: 43.787872 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 852032704 bytes
HTML transferred: 825032704 bytes
Requests per second: 2283.74 [#/sec] (mean)
Time per request: 0.876 [ms] (mean)
Time per request: 0.438 [ms] (mean, across all concurrent requests)
Transfer rate: 19002.13 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 0 0 0.1 0 12
Waiting: 0 0 0.0 0 1
Total: 0 0 0.1 0 12
Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 0
95% 0
98% 0
99% 0
100% 12 (longest request)
Ran this 5 times with no real difference in performance.
So a single 2.8GHz P4 HT with a 2.4 kernel has better performance than
a dual Xeon 3.2GHz HT with a 2.6 kernel. I think the SCSI drives
eliminate the IDE results as a factor, so this looks like it's a net
loopback performance issue. Maybe. :)
> I am curious if it might be related to my problems with networking.
> turning of TCP WS tcp_window_scaling helps my problems.
>
> Regards,
> Maciej
Regards,
Omar Kilani
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Performance Degradation: 2.6.8-rc4-bk1 vs RHEL 2.4.21-15.0.3
2004-08-12 13:35 ` Omar Kilani
@ 2004-08-12 13:56 ` Omar Kilani
2004-08-13 0:57 ` Nick Piggin
0 siblings, 1 reply; 4+ messages in thread
From: Omar Kilani @ 2004-08-12 13:56 UTC (permalink / raw)
To: linux-kernel
Replying to self,
Some additional stats. I ran 'vmstat 1' during the ab run. Results:
2.4.21-2ts (RHEL 2.4.21-15.0.3):
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 0 304852 18760 142880 0 0 0 0 104 22327 39 50 10 0
2 0 0 304256 18760 143196 0 0 0 0 103 22740 38 50 12 0
1 0 0 303176 18760 143504 0 0 0 0 106 22306 41 47 12 0
3 0 0 302808 18760 143824 0 0 0 0 102 22410 36 50 15 0
1 0 0 302548 18768 144140 0 0 0 1612 131 22384 39 50 11 0
3 0 0 302224 18768 144456 0 0 0 0 107 22707 34 57 9 0
3 0 0 301932 18768 144776 0 0 0 0 104 22399 42 44 14 0
3 0 0 301784 18768 145092 0 0 0 0 103 22635 36 52 11 0
2 0 0 301164 18768 145408 0 0 0 0 102 22613 32 57 11 0
2 0 0 301064 18776 145724 0 0 0 1608 134 22595 38 49 13 0
1 0 0 300672 18776 146040 0 0 0 0 102 22444 38 51 11 0
1 0 0 300088 18776 146356 0 0 0 0 103 21374 40 50 10 0
2 0 0 299628 18776 146668 0 0 0 0 109 20670 31 52 17 0
2 0 0 299440 18780 146988 0 0 0 0 103 21666 44 46 10 0
3 0 0 299100 18788 147304 0 0 0 1664 138 22305 36 51 12 0
2 0 0 298664 18788 147620 0 0 0 0 108 22328 36 49 15 0
2 0 0 298264 18788 147936 0 0 0 0 105 22404 34 50 17 0
1 0 0 297792 18788 148248 0 0 0 0 105 22415 37 50 13 0
1 0 0 297412 18788 148564 0 0 0 0 105 22480 37 50 13 0
2 0 0 296560 18804 148876 0 0 0 1640 138 22471 35 51 14 0
1 0 0 296040 18804 149188 0 0 0 0 104 22395 36 51 13 0
2 0 0 296048 18804 149504 0 0 0 0 104 22405 32 52 16 0
2 0 0 295440 18804 149820 0 0 0 0 109 22162 37 50 13 0
1 0 0 294676 18804 150136 0 0 0 0 106 22317 37 48 15 0
1 0 0 294384 18812 150452 0 0 0 1580 132 22068 41 46 12 1
1 0 0 294208 18812 150768 0 0 0 0 104 22168 42 49 9 0
3 0 0 293796 18816 151084 0 0 0 0 109 22345 40 49 12 0
2 0 0 293300 18816 151396 0 0 0 0 104 22793 37 49 14 0
1 1 0 292828 18816 151712 0 0 0 1568 106 22518 36 52 11 0
3 0 0 292680 18824 152028 0 0 0 40 139 22407 41 47 12 0
2 0 0 291952 18824 152344 0 0 0 0 105 21948 39 51 10 0
2 0 0 291092 18824 152668 0 0 0 0 104 20059 35 50 15 0
1 0 0 289944 18824 152908 0 0 0 0 114 13542 39 39 22 0
0 0 0 293840 18832 152908 0 0 0 1236 140 53 12 0 84 4
0 0 0 293840 18832 152908 0 0 0 0 103 11 0 0 100 0
2.6.8-rc4-mm1:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
4 0 0 396092 8300 55140 0 0 0 0 1014 13901 23 65 12 0
2 0 0 396796 8300 55140 0 0 0 0 1007 11587 30 66 4 0
1 0 0 395964 8300 55660 0 0 0 0 1005 14045 23 64 13 0
2 0 0 394492 8300 55660 0 0 0 0 1007 13333 24 68 8 0
2 0 0 393980 8308 55912 0 0 0 1044 1016 13234 26 68 7 0
2 0 0 393340 8308 56172 0 0 0 228 1012 13498 23 66 10 0
2 0 0 393588 8308 56172 0 0 0 0 1007 11800 29 67 4 0
2 0 0 393332 8308 56432 0 0 0 0 1009 13183 24 67 8 0
2 0 0 392820 8308 56692 0 0 0 0 1006 12992 27 68 6 0
2 0 0 392436 8316 56944 0 0 0 816 1016 13064 25 69 6 0
1 0 0 391988 8316 56944 0 0 0 0 1007 12894 27 69 5 0
2 0 0 391604 8316 57204 0 0 0 0 1009 13508 24 66 10 0
1 0 0 391412 8316 57464 0 0 0 0 1008 12780 27 69 4 0
1 0 0 391348 8316 57464 0 0 0 0 1007 13431 25 65 10 0
1 0 0 391028 8324 57716 0 0 0 992 1016 13699 25 63 12 0
3 0 0 390836 8324 57976 0 0 0 0 1006 12496 30 68 3 0
2 0 0 390708 8324 57976 0 0 0 0 1008 12498 27 71 2 0
2 0 0 390580 8324 58496 0 0 0 0 1009 13158 25 65 10 0
1 0 0 390452 8324 58496 0 0 0 0 1007 12676 26 66 8 0
2 0 0 390772 8332 58748 0 0 0 1024 1017 12255 27 62 10 0
2 0 0 389804 8336 58744 0 0 0 0 1005 13003 27 68 4 0
2 0 0 389612 8336 59264 0 0 0 0 1006 12848 26 70 5 0
2 0 0 389420 8336 59264 0 0 0 0 1008 12725 27 69 3 0
2 0 0 389036 8336 59524 0 0 0 0 1007 13007 26 69 6 0
2 0 0 388844 8344 59776 0 0 0 1000 1018 13096 26 68 7 0
2 0 0 388588 8344 59776 0 0 0 0 1007 13790 23 66 12 0
2 0 0 388396 8344 60036 0 0 0 0 1005 13315 24 67 9 0
2 0 0 388140 8344 60296 0 0 0 0 1007 12510 27 71 2 0
2 0 0 387820 8344 60296 0 0 0 0 1008 13011 25 69 6 0
2 0 0 387564 8352 60808 0 0 0 996 1018 13343 23 69 8 0
2 0 0 387436 8352 60808 0 0 0 4 1010 12532 29 70 2 0
1 0 0 386988 8352 60808 0 0 0 0 1010 13909 24 63 13 0
1 0 0 386988 8352 61328 0 0 0 0 1008 12854 26 67 7 0
2 0 0 385516 8352 61328 0 0 0 0 1013 12836 26 65 9 0
2 0 0 386284 8360 61580 0 0 0 1044 1017 12728 29 68 4 0
1 0 0 386924 8360 61840 0 0 0 68 1022 11707 29 65 6 0
1 0 0 386412 8360 61840 0 0 0 0 1009 14099 24 65 12 0
2 0 0 385836 8360 62100 0 0 0 0 1010 12837 27 69 4 0
2 0 0 384556 8360 62360 0 0 0 0 1008 12659 28 70 3 0
2 0 0 384172 8368 62612 0 0 0 1032 1022 13203 24 69 7 0
2 0 0 383916 8368 62612 0 0 0 0 1007 13499 24 67 9 0
2 0 0 383660 8372 63128 0 0 0 0 1010 12915 26 68 6 0
2 0 0 383404 8372 63128 0 0 0 0 1011 12856 26 70 5 0
2 0 0 383084 8372 63388 0 0 0 0 1009 13076 27 67 6 0
2 0 0 382828 8380 63640 0 0 0 1000 1022 13054 25 68 7 1
2 0 0 382572 8380 63640 0 0 0 0 1009 13559 24 66 10 0
2 0 0 382444 8380 63900 0 0 0 0 1010 12554 28 70 3 0
2 0 0 382196 8380 64160 0 0 0 0 1008 12754 27 69 4 0
2 0 0 381748 8380 64420 0 0 0 0 1008 13483 25 65 10 0
2 0 0 381492 8388 64412 0 0 0 996 1017 12901 28 67 6 0
0 0 0 385076 8388 64412 0 0 0 0 1024 6320 29 31 41 0
0 0 0 385076 8388 64412 0 0 0 0 1007 8 0 0 100 0
Anything else I should be monitoring?
Regards,
Omar Kilani
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Performance Degradation: 2.6.8-rc4-bk1 vs RHEL 2.4.21-15.0.3
2004-08-12 13:56 ` Omar Kilani
@ 2004-08-13 0:57 ` Nick Piggin
0 siblings, 0 replies; 4+ messages in thread
From: Nick Piggin @ 2004-08-13 0:57 UTC (permalink / raw)
To: Omar Kilani; +Cc: linux-kernel
Omar Kilani wrote:
>Anything else I should be monitoring?
>
>
Take a profile of each kernel. Boot with profile=1, then run:
echo 0 > /proc/profile ; ab --foo --bar ; readprofile | sort -nr |
head -n30 > profile.out
And send in the profiles for both kernels.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2004-08-13 0:57 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2004-08-12 12:45 Performance Degradation: 2.6.8-rc4-bk1 vs RHEL 2.4.21-15.0.3 Omar Kilani
[not found] ` <295911442.20040812150922@dns.toxicfilms.tv>
2004-08-12 13:35 ` Omar Kilani
2004-08-12 13:56 ` Omar Kilani
2004-08-13 0:57 ` Nick Piggin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox