From: Dave Sperry <dave_sperry@ieee.org>
To: linux-rt-users@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Subject: Poor UDP performance using 2.6.21-rc5-rt5
Date: Sun, 01 Apr 2007 15:15:26 -0400 [thread overview]
Message-ID: <461004CE.1030009@ieee.org> (raw)
Hi
I have a dual core Opteron machine that exhibits poor UDP performance
(RT consumes more than 2X cpu) with the 2.6.21-rc5-rt5 as compared to
2.6.21-rc5. Top shows the IRQ handler consuming a lot of CPU.
The mother board is a Supermicro H8DME-2 with one dual core Opteron
installed. The networking is provided by the on board nVidia MCP55Pro chip.
The RT test is done using netperf 2.4.3 with the server on an IBM LS20
blade running RHEL4U2 and the Supermicro running netperf under RHEL5
with 2.6.21-rc5-rt5.
The Non-RT test was done on the exact same setup except 2.6.21-rc5-rt5
was loaded on the SuperMicro board.
Cyclesoak was used to measure CPU utilization in all cases.
Here are the RT results
########################################################3
## 2.6.21-rc5-rt5
#######################################################
$ !netper
netperf -l 100 -H 192.168.70.11 -t UDP_STREAM -- -m 1025
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.70.11 (192.168.70.11) port 0 AF_INET
Socket Message Elapsed Messages
Size Size Time Okay Errors Throughput
bytes bytes secs # # 10^6bits/sec
126976 1025 100.00 8676376 0 711.46
135168 100.00 8676376 711.46
########## cyclesoak during test
$ ./cyclesoak
using 2 CPUs
System load: -0.1%
System load: 40.5%
System load: 51.6%
System load: 51.5%
System load: 50.9%
System load: 50.7%
System load: 50.8%
System load: 50.7%
System load: 50.6%
######## top during test
top - 13:26:48 up 8 min, 4 users, load average: 1.74, 0.46, 0.15
Tasks: 149 total, 4 running, 145 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 16.8%sy, 50.6%ni, 0.0%id, 0.0%wa, 25.6%hi, 6.3%si,
0.0%st
Mem: 2035444k total, 465888k used, 1569556k free, 28840k buffers
Swap: 3068372k total, 0k used, 3068372k free, 318668k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3865 eadi 39 19 6804 1164 108 R 100 0.1 0:38.25 cyclesoak
2715 root -51 -5 0 0 0 S 51 0.0 0:09.52 IRQ-8406
3867 eadi 25 0 6440 632 480 R 34 0.0 0:06.03 netperf
19 root -51 0 0 0 0 S 13 0.0 0:02.33 softirq-net-tx/
3866 eadi 39 19 6804 1164 108 R 1 0.1 0:20.47 cyclesoak
3167 root 25 0 29888 1180 888 S 0 0.1 0:00.93 automount
3861 eadi 15 0 12712 1076 788 R 0 0.1 0:00.19 top
1 root 18 0 10308 668 552 S 0 0.0 0:00.67 init
2 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
3 root RT 0 0 0 0 S 0 0.0 0:00.00 posix_cpu_timer
4 root -51 0 0 0 0 S 0 0.0 0:00.00 softirq-high/0
5 root -51 0 0 0 0 S 0 0.0 0:00.00 softirq-timer/0
6 root -51 0 0 0 0 S 0 0.0 0:00.00 softirq-net-tx/
7 root -51 0 0 0 0 S 0 0.0 0:00.00 softirq-net-rx/
8 root -51 0 0 0 0 S 0 0.0 0:00.00 softirq-block/0
########################
The baseline results:
RHEL5 with 2.6.21-rc5 kernel
##############################
$ netperf -l 100 -H 192.168.70.11 -t UDP_STREAM -- -m 1025
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.70.11 (192.168.70.11) port 0 AF_INET
Socket Message Elapsed Messages
Size Size Time Okay Errors Throughput
bytes bytes secs # # 10^6bits/sec
126976 1025 100.00 11405485 0 935.24
135168 100.00 11405485 935.24
#######################################
$ ./cyclesoak
using 2 CPUs
System load: 7.6%
System load: 29.6%
System load: 29.6%
System load: 28.9%
System load: 24.9%
System load: 25.0%
System load: 24.8%
System load: 24.9%
#######################################
top:top - 13:52:22 up 10 min, 6 users, load average: 1.46, 0.43, 0.17
Tasks: 118 total, 4 running, 114 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.5%us, 9.8%sy, 75.7%ni, 0.0%id, 0.0%wa, 5.8%hi, 8.1%si,
0.0%st
Mem: 2057200k total, 459128k used, 1598072k free, 29020k buffers
Swap: 3068372k total, 0k used, 3068372k free, 318968k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3882 eadi 39 19 6804 1164 108 R 100 0.1 0:52.11 cyclesoak
3881 eadi 39 19 6804 1164 108 R 65 0.1 0:38.47 cyclesoak
3883 eadi 15 0 6436 632 480 R 35 0.0 0:18.26 netperf
3879 eadi 15 0 12580 1052 788 R 0 0.1 0:00.15 top
1 root 18 0 10308 664 552 S 0 0.0 0:00.48 init
2 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.01 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0
5 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
6 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
Any thoughts on how to fix this?
Thanks,
-Dave
next reply other threads:[~2007-04-01 19:20 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-04-01 19:15 Dave Sperry [this message]
2007-04-01 20:07 ` Poor UDP performance using 2.6.21-rc5-rt5 Nivedita Singhvi
2007-04-01 22:00 ` Dave Sperry
2007-04-02 5:55 ` Ingo Molnar
2007-04-02 6:30 ` Ingo Molnar
2007-04-02 7:21 ` Ingo Molnar
2007-04-02 8:17 ` Dave Sperry
2007-04-02 9:37 ` Ingo Molnar
-- strict thread matches above, loose matches on Subject: below --
2007-04-02 14:09 dave_sperry@ieee.org
2007-04-02 14:23 ` Ingo Molnar
2007-04-02 17:17 dave_sperry@ieee.org
2007-04-02 17:50 dave_sperry@ieee.org
2007-04-02 19:04 ` Ingo Molnar
2007-04-03 0:09 ` David Sperry
2007-04-03 6:43 ` Ingo Molnar
2007-04-03 8:51 ` Ingo Molnar
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=461004CE.1030009@ieee.org \
--to=dave_sperry@ieee.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rt-users@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox