From: Stephen Hemminger <shemminger@linux-foundation.org>
To: Thomas Gleixner <tglx@linutronix.de>, Ulrich Drepper <drepper@gmail.com>
Cc: Maximilian Engelhardt <maxi@daemonizer.de>,
Michael Buesch <mb@bu3sch.de>,
linux-kernel <linux-kernel@vger.kernel.org>,
linux-wireless <linux-wireless@vger.kernel.org>,
Arnaldo Carvalho de Melo <acme@ghostprotocols.net>,
Jeff Garzik <jgarzik@pobox.com>,
Gary Zambrano <zambrano@broadcom.com>,
netdev@vger.kernel.org, Andrew Morton <akpm@linux-foundation.org>
Subject: iperf: performance regression (was b44 driver problem?)
Date: Mon, 4 Jun 2007 09:59:24 -0700 [thread overview]
Message-ID: <20070604095924.651d91c8@freepuppy> (raw)
In-Reply-To: <1180974958.4404.24.camel@chaos>
On Mon, 04 Jun 2007 18:35:58 +0200
Thomas Gleixner <tglx@linutronix.de> wrote:
> On Mon, 2007-06-04 at 09:09 -0700, Stephen Hemminger wrote:
> > > > I did the test with an 2.6.22-rc3-git4 kernel and the p54 driver built
> > > > external as module.
> > >
> > > Can you look at iperf to figure out, whether it does some weird timer
> > > stuff (high frequency interval timer or such) ? Either check the code or
> > > strace it.
> >
> > It is the receiver doing a tight loop doing gettimeofday/recv calls.
> >
> >
> > sendto(-1227715616, 0xc, 3085438964, 0, {...}, 3067249832) = 0
> > gettimeofday({1180973726, 981615}, NULL) = 0
> > gettimeofday({1180973726, 981751}, NULL) = 0
> > futex(0x8055c64, 0x5 /* FUTEX_??? */, 1) = 1
> > futex(0x8055c90, FUTEX_WAKE, 1) = 0
> > recv(4, "\0\0\0\0\0\0\0\1\0\0\23\211\0\0\0\0\0\0\0\0\377\377\364"..., 8192, 0) = 8192
> > gettimeofday({1180973726, 982754}, NULL) = 0
> > recv(4, "\0\0\0\0\0\0\0\1\0\0\23\211\0\0\0\0\0\0\0\0\377\377\364"..., 8192, 0) = 8192
> > gettimeofday({1180973726, 983790}, NULL) = 0
>
> Well, gettimeofday() is not affected by the highres code, but
>
> > nanosleep({0, 0}, NULL) = 0
> > nanosleep({0, 0}, NULL) = 0
>
> is. The nanosleep call with a relative timeout of 0 returns immediately
> with highres enabled, while it sleeps at least until the next tick
> arrives when highres is off. Are there more of those stupid sleeps in
> the code ?
GLIBC pthread_mutex does it, YES it is a problem!
Looks like the old behavior is required for ABI compatibility.
iperf server has several threads. One thread is using pthread_mutex_lock
to wait for the other thread. It looks like pthread_mutex_lock is using
nanosleep as yield().
Multi-thread strace shows how this could kill performance.
These are with old 2.6.20 kernel that doesn't have the problem:
sendto(-1210930208, 0xc, 3085438964, 0, {...}, 3084035240) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
bind(3, {sa_family=AF_INET, sin_port=htons(5001), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
listen(3, 5) = 0
futex(0x8055c64, 0x5 /* FUTEX_??? */, 1) = 1
accept(3, {sa_family=AF_INET, sin_port=htons(49973), sin_addr=inet_addr("192.168.0.14")}, [16]) = 4
getsockname(4, {sa_family=AF_INET, sin_port=htons(5001), sin_addr=inet_addr("192.168.0.12")}, [16]) = 0
recv(4, "\0\0\0\0\0\0\0\1\0\0\23\211\0\0\0\0\0\0\0\0\377\377\364"..., 24, 0) = 24
mmap2(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6528000
mprotect(0xb6528000, 4096, PROT_NONE) = 0
clone(child_stack=0xb6d284a4, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|CLONE_DETACHED, parent_tidptr=0xb6d28bd8, {entry_number:6, base_addr:0xb6d28b90, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}, child_tidptr=0xb6d28bd8) = 5622
accept(3, 0x8058b98, [128]) = ? ERESTARTSYS (To be restarted)
==============
sendto(-1219322912, 0xc, 3085438964, 0, {...}, 3075642536) = 0
futex(0x8055d04, 0x5 /* FUTEX_??? */, 1) = 1
futex(0x8055d30, FUTEX_WAKE, 1) = 1
futex(0x8055c64, FUTEX_WAIT, 1, NULL) = 0
futex(0x8055d04, 0x5 /* FUTEX_??? */, 1) = 1
futex(0x8055d30, FUTEX_WAKE, 1) = 1
futex(0x8055c90, FUTEX_WAKE, 1) = 0
getsockopt(3, SOL_SOCKET, SO_RCVBUF, [87380], [4]) = 0
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fb0000
write(1, "--------------------------------"..., 61) = 61
write(1, "Server listening on TCP port 500"..., 34) = 34
write(1, "TCP window size: 85.3 KByte (def"..., 38) = 38
write(1, "--------------------------------"..., 61) = 61
nanosleep({0, 0}, NULL) = 0
futex(0x8055d04, 0x5 /* FUTEX_??? */, 1) = 1
futex(0x8055d30, FUTEX_WAKE, 1) = 1
futex(0x8055c64, FUTEX_WAIT, 3, NULL) = 0
futex(0x8055c90, FUTEX_WAIT, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x8055d04, 0x5 /* FUTEX_??? */, 1) = 1
futex(0x8055d30, FUTEX_WAKE, 1) = 0
futex(0x8055c90, FUTEX_WAKE, 1) = 0
write(1, "[ 4] local 192.168.0.12 port 50"..., 74) = 74
nanosleep({0, 0}, NULL) = 0
nanosleep({0, 0}, NULL) = 0
nanosleep({0, 0}, NULL) = 0
nanosleep({0, 0}, NULL) = 0
nanosleep({0, 0}, NULL) = 0
nanosleep({0, 0}, NULL) = 0
nanosleep({0, 0}, NULL) = 0
nanosleep({0, 0}, NULL) = 0
...
nanosleep({0, 0}, NULL) = 0
nanosleep({0, 0}, NULL) = 0
nanosleep({0, 0}, NULL) = 0
nanosleep({0, 0}, NULL) = 0
nanosleep({0, 0}, NULL) = 0
nanosleep({0, 0}, NULL) = 0
write(1, "[ 4] 0.0-30.2 sec 336 MByte"..., 50) = 50
nanosleep({0, 0}, NULL) = 0
nanosleep({0, 0}, NULL) = 0
futex(0x8055d04, 0x5 /* FUTEX_??? */, 1) = 1
futex(0x8055d30, FUTEX_WAKE, 1) = 0
futex(0x8055c64, FUTEX_WAIT, 5, NULL) = -1 EINTR (Interrupted system call)
===============
sendto(-1227715616, 0xc, 3085438964, 0, {...}, 3067249832) = 0
gettimeofday({1180973726, 981615}, NULL) = 0
gettimeofday({1180973726, 981751}, NULL) = 0
futex(0x8055c64, 0x5 /* FUTEX_??? */, 1) = 1
futex(0x8055c90, FUTEX_WAKE, 1) = 0
recv(4, "\0\0\0\0\0\0\0\1\0\0\23\211\0\0\0\0\0\0\0\0\377\377\364"..., 8192, 0) = 8192
gettimeofday({1180973726, 982754}, NULL) = 0
recv(4, "\0\0\0\0\0\0\0\1\0\0\23\211\0\0\0\0\0\0\0\0\377\377\364"..., 8192, 0) = 8192
gettimeofday({1180973726, 983790}, NULL) = 0
recv(4, "\0\0\0\0\0\0\0\1\0\0\23\211\0\0\0\0\0\0\0\0\377\377\364"..., 8192, 0) = 8192
gettimeofday({1180973726, 984355}, NULL) = 0
recv(4, "\0\0\0\0\0\0\0\1\0\0\23\211\0\0\0\0\0\0\0\0\377\377\364"..., 8192, 0) = 8192
...
gettimeofday({1180973757, 172338}, NULL) = 0
recv(4, "45678901234567890123456789012345"..., 8192, 0) = 1448
gettimeofday({1180973757, 172437}, NULL) = 0
recv(4, "23456789012345678901234567890123"..., 8192, 0) = 1448
gettimeofday({1180973757, 172576}, NULL) = 0
recv(4, "01234567890123456789012345678901"..., 8192, 0) = 1632
gettimeofday({1180973757, 172752}, NULL) = 0
recv(4, "", 8192, 0) = 0
gettimeofday({1180973757, 172797}, NULL) = 0
gettimeofday({1180973757, 172817}, NULL) = 0
nanosleep({0, 0}, NULL) = 0
nanosleep({0, 0}, NULL) = 0
close(4) = 0
futex(0x8055d04, 0x5 /* FUTEX_??? */, 1) = 1
futex(0x8055d30, FUTEX_WAKE, 1) = 0
_exit(0) = ?
next prev parent reply other threads:[~2007-06-04 17:13 UTC|newest]
Thread overview: 46+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20070525172431.60affaca@freepuppy>
[not found] ` <200705261901.18110.mb@bu3sch.de>
2007-05-27 19:25 ` b44: regression in 2.6.22 (resend) Maximilian Engelhardt
2007-05-27 19:45 ` Michael Buesch
2007-05-27 20:36 ` Maximilian Engelhardt
2007-05-27 20:46 ` Michael Buesch
2007-05-27 21:46 ` Maximilian Engelhardt
2007-05-27 21:13 ` Michael Buesch
2007-05-27 21:16 ` Michael Buesch
2007-05-27 21:50 ` Maximilian Engelhardt
2007-05-27 22:15 ` Maximilian Engelhardt
2007-05-28 0:24 ` Michael Buesch
2007-05-28 0:40 ` Maximilian Engelhardt
2007-05-28 10:16 ` Michael Buesch
2007-05-28 14:09 ` Maximilian Engelhardt
2007-05-28 15:14 ` Michael Buesch
2007-05-28 15:32 ` Thomas Gleixner
2007-05-28 15:43 ` Michael Buesch
2007-05-28 17:44 ` Maximilian Engelhardt
2007-05-28 19:23 ` Thomas Gleixner
2007-05-28 20:55 ` Maximilian Engelhardt
2007-05-28 21:45 ` Thomas Gleixner
2007-05-29 18:28 ` Maximilian Engelhardt
2007-05-29 13:58 ` Gary Zambrano
2007-05-29 17:23 ` Maximilian Engelhardt
2007-06-03 16:26 ` Maximilian Engelhardt
2007-06-04 6:39 ` Thomas Gleixner
2007-06-04 16:09 ` Stephen Hemminger
2007-06-04 16:35 ` Thomas Gleixner
2007-06-04 16:59 ` Stephen Hemminger [this message]
2007-06-04 17:32 ` iperf: performance regression (was b44 driver problem?) Thomas Gleixner
2007-06-04 17:51 ` Stephen Hemminger
2007-06-04 19:00 ` Thomas Gleixner
2007-06-04 19:26 ` Thomas Gleixner
2007-06-04 19:32 ` Ingo Molnar
2007-06-04 19:47 ` Maximilian Engelhardt
2007-06-04 20:02 ` Stephen Hemminger
2007-06-04 20:52 ` Maximilian Engelhardt
2007-05-28 10:49 ` b44: regression in 2.6.22 (resend) Michael Buesch
2007-05-28 14:12 ` Maximilian Engelhardt
2007-05-28 14:55 ` Michael Buesch
2007-05-29 14:14 ` Gary Zambrano
2007-05-29 20:45 ` Michael Buesch
2007-05-29 21:01 ` Stephen Hemminger
2007-05-29 21:05 ` Gary Zambrano
2007-05-29 22:39 ` Jeff Garzik
2007-05-29 21:36 ` Gary Zambrano
2007-05-30 10:45 ` Michael Buesch
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20070604095924.651d91c8@freepuppy \
--to=shemminger@linux-foundation.org \
--cc=acme@ghostprotocols.net \
--cc=akpm@linux-foundation.org \
--cc=drepper@gmail.com \
--cc=jgarzik@pobox.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-wireless@vger.kernel.org \
--cc=maxi@daemonizer.de \
--cc=mb@bu3sch.de \
--cc=netdev@vger.kernel.org \
--cc=tglx@linutronix.de \
--cc=zambrano@broadcom.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).