From: Thomas Graf <tgraf@infradead.org>
To: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>,
David Miller <davem@davemloft.net>,
rick.jones2@hp.com, therbert@google.com, wsommerfeld@google.com,
daniel.baluta@gmail.com, netdev@vger.kernel.org
Subject: Re: SO_REUSEPORT - can it be done in kernel?
Date: Tue, 1 Mar 2011 06:07:08 -0500 [thread overview]
Message-ID: <20110301110708.GJ9763@canuck.infradead.org> (raw)
In-Reply-To: <1298975602.3284.13.camel@edumazet-laptop>
On Tue, Mar 01, 2011 at 11:33:22AM +0100, Eric Dumazet wrote:
> > I retested with net-2.6 w/o Herbert's patch:
> >
> > named -n 1: 36.9kqps
> > named: 16.2kqps
>
> Thats better ;)
>
> You could do "cat /proc/net/udp" to check if drops occur on port 53
> socket (last column)
>
> But maybe your queryperf is limited to few queries in flight (default is
> 20 per queryperf instance)
I tried -q 10, 20, 30, 50, 100. Starting with 20 I see drops, at q=50
queryperf reports 99% drops.
I also tested again on the Intel machine that did ~650kqps using SO_REUSEPORT.
net-2.6: 106.3kqps, 101.2kqps
net-2.6 lockless udp: 251.7kqps, 250.4kqps
I see drops in both test cases occur so I believe the rate supplied by the
clients is sufficient.
The difference is obvious when looking at top and mpstat:
UDP lockless (250kqps):
Cpu0 : 46.4%us, 28.8%sy, 0.0%ni, 24.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 2.0%us, 1.3%sy, 0.0%ni, 3.0%id, 0.0%wa, 0.0%hi, 93.6%si, 0.0%st
Cpu2 : 45.9%us, 28.2%sy, 0.0%ni, 25.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 50.0%us, 21.6%sy, 0.0%ni, 28.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 45.4%us, 27.8%sy, 0.0%ni, 26.5%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu5 : 50.7%us, 23.2%sy, 0.0%ni, 26.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 45.2%us, 28.9%sy, 0.0%ni, 25.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 50.5%us, 22.0%sy, 0.0%ni, 27.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu8 : 45.3%us, 29.3%sy, 0.0%ni, 25.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9 : 50.8%us, 20.8%sy, 0.0%ni, 28.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu10 : 46.1%us, 27.8%sy, 0.0%ni, 26.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu11 : 27.2%us, 11.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 58.1%si, 0.0%st
05:50:44 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
05:50:44 AM all 23.86 0.00 13.02 0.22 0.00 6.98 0.00 0.00 55.92
05:50:44 AM 0 26.16 0.00 17.20 0.73 0.00 0.30 0.00 0.00 55.61
05:50:44 AM 1 2.36 0.00 2.11 0.70 0.00 51.97 0.00 0.00 42.87
05:50:44 AM 2 25.90 0.00 16.38 0.32 0.00 0.03 0.00 0.00 57.36
05:50:44 AM 3 28.26 0.00 12.73 0.27 0.00 0.02 0.00 0.00 58.73
05:50:44 AM 4 25.63 0.00 16.04 0.13 0.00 0.03 0.00 0.00 58.17
05:50:44 AM 5 28.19 0.00 12.54 0.17 0.00 0.01 0.00 0.00 59.09
05:50:44 AM 6 25.28 0.00 15.21 0.02 0.00 1.95 0.00 0.00 57.54
05:50:44 AM 7 28.34 0.00 12.40 0.10 0.00 0.01 0.00 0.00 59.14
05:50:44 AM 8 25.70 0.00 15.91 0.01 0.00 0.02 0.00 0.00 58.37
05:50:44 AM 9 28.31 0.00 12.56 0.11 0.00 0.01 0.00 0.00 59.01
05:50:44 AM 10 25.85 0.00 15.65 0.01 0.00 0.02 0.00 0.00 58.47
05:50:44 AM 11 16.11 0.00 7.44 0.10 0.00 29.87 0.00 0.00 46.49
SO_REUSEPORT test (doing 640kqps):
Cpu0 : 57.3%us, 26.5%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 12.9%si, 0.0%st
Cpu1 : 25.7%us, 10.0%sy, 0.0%ni, 0.3%id, 0.0%wa, 0.0%hi, 64.0%si, 0.0%st
Cpu2 : 56.3%us, 28.8%sy, 0.0%ni, 3.0%id, 0.0%wa, 0.0%hi, 11.9%si, 0.0%st
Cpu3 : 29.1%us, 10.9%sy, 0.0%ni, 1.3%id, 0.0%wa, 0.0%hi, 58.6%si, 0.0%st
Cpu4 : 57.3%us, 28.5%sy, 0.0%ni, 2.3%id, 0.0%wa, 0.0%hi, 11.9%si, 0.0%st
Cpu5 : 64.8%us, 22.6%sy, 0.0%ni, 3.0%id, 0.0%wa, 0.0%hi, 9.6%si, 0.0%st
Cpu6 : 59.0%us, 26.7%sy, 0.0%ni, 2.7%id, 0.0%wa, 0.0%hi, 11.7%si, 0.0%st
Cpu7 : 64.1%us, 22.3%sy, 0.0%ni, 3.7%id, 0.0%wa, 0.0%hi, 10.0%si, 0.0%st
Cpu8 : 57.6%us, 27.5%sy, 0.0%ni, 3.0%id, 0.0%wa, 0.0%hi, 11.9%si, 0.0%st
Cpu9 : 65.2%us, 22.2%sy, 0.0%ni, 2.3%id, 0.0%wa, 0.0%hi, 10.3%si, 0.0%st
Cpu10 : 56.9%us, 28.3%sy, 0.0%ni, 3.0%id, 0.0%wa, 0.0%hi, 11.8%si, 0.0%st
Cpu11 : 40.2%us, 14.6%sy, 0.0%ni, 2.3%id, 0.0%wa, 0.0%hi, 42.9%si, 0.0%st
next prev parent reply other threads:[~2011-03-01 11:07 UTC|newest]
Thread overview: 91+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-01-27 10:07 SO_REUSEPORT - can it be done in kernel? Daniel Baluta
2011-01-27 15:55 ` Bill Sommerfeld
2011-01-27 21:32 ` Tom Herbert
2011-02-25 12:56 ` Thomas Graf
2011-02-25 19:18 ` Rick Jones
2011-02-25 19:20 ` David Miller
2011-02-26 0:57 ` Herbert Xu
2011-02-26 2:12 ` David Miller
2011-02-26 2:48 ` Herbert Xu
2011-02-26 3:07 ` David Miller
2011-02-26 3:11 ` Herbert Xu
2011-02-26 7:31 ` Eric Dumazet
2011-02-26 7:46 ` David Miller
2011-02-27 11:02 ` Thomas Graf
2011-02-27 11:06 ` Herbert Xu
2011-02-28 3:45 ` Tom Herbert
2011-02-28 4:26 ` Herbert Xu
2011-02-28 11:36 ` Herbert Xu
2011-02-28 13:32 ` Eric Dumazet
2011-02-28 14:13 ` Herbert Xu
2011-02-28 14:22 ` Eric Dumazet
2011-02-28 14:25 ` Herbert Xu
2011-02-28 14:53 ` Eric Dumazet
2011-02-28 15:01 ` Thomas Graf
2011-02-28 14:13 ` Thomas Graf
2011-02-28 16:22 ` Eric Dumazet
2011-02-28 16:37 ` Thomas Graf
2011-02-28 17:07 ` Eric Dumazet
2011-03-01 10:19 ` Thomas Graf
2011-03-01 10:33 ` Eric Dumazet
2011-03-01 11:07 ` Thomas Graf [this message]
2011-03-01 11:13 ` Eric Dumazet
2011-03-01 11:27 ` Thomas Graf
2011-03-01 11:45 ` Eric Dumazet
2011-03-01 11:53 ` Herbert Xu
2011-03-01 12:32 ` Herbert Xu
2011-03-01 13:04 ` Eric Dumazet
2011-03-01 13:11 ` Herbert Xu
2011-03-01 13:03 ` Eric Dumazet
2011-03-01 13:18 ` Herbert Xu
2011-03-01 13:52 ` Eric Dumazet
2011-03-01 13:58 ` Herbert Xu
2011-03-01 16:31 ` Eric Dumazet
2011-03-02 0:23 ` Herbert Xu
2011-03-02 2:00 ` Eric Dumazet
2011-03-02 2:39 ` Herbert Xu
2011-03-02 2:56 ` Eric Dumazet
2011-03-02 3:09 ` Herbert Xu
2011-03-02 3:44 ` Eric Dumazet
2011-03-02 7:12 ` Tom Herbert
2011-03-02 7:31 ` Herbert Xu
2011-03-02 8:04 ` Eric Dumazet
2011-03-02 8:07 ` Herbert Xu
2011-03-02 8:24 ` Eric Dumazet
2011-03-01 12:01 ` Thomas Graf
2011-03-01 12:15 ` Herbert Xu
2011-03-01 13:27 ` Herbert Xu
2011-03-01 12:18 ` Thomas Graf
2011-03-01 12:19 ` Herbert Xu
2011-03-01 13:50 ` Thomas Graf
2011-03-01 14:06 ` Eric Dumazet
2011-03-01 14:22 ` Thomas Graf
2011-03-01 14:30 ` Thomas Graf
2011-03-01 14:52 ` Eric Dumazet
2011-03-01 15:07 ` Thomas Graf
2011-03-01 5:33 ` Eric Dumazet
2011-03-01 12:35 ` Herbert Xu
2011-03-01 12:36 ` [PATCH 3/5] inet: Add ip_make_skb and ip_finish_skb Herbert Xu
2011-03-01 12:36 ` [PATCH 2/5] inet: Remove explicit write references to sk/inet in ip_append_data Herbert Xu
2011-03-02 6:15 ` inet: Replace left-over references to inet->cork Herbert Xu
2011-03-02 7:01 ` David Miller
2011-03-01 12:36 ` [PATCH 1/5] inet: Remove unused sk_sndmsg_* from UFO Herbert Xu
2011-03-01 12:36 ` [PATCH 5/5] udp: Add lockless transmit path Herbert Xu
2011-03-01 12:36 ` [PATCH 4/5] udp: Switch to ip_finish_skb Herbert Xu
2011-03-01 16:43 ` SO_REUSEPORT - can it be done in kernel? Eric Dumazet
2011-03-01 20:36 ` David Miller
2011-02-28 11:41 ` [PATCH 1/5] net: Remove unused sk_sndmsg_* from UFO Herbert Xu
2011-03-01 5:31 ` Eric Dumazet
2011-02-28 11:41 ` [PATCH 2/5] net: Remove explicit write references to sk/inet in ip_append_data Herbert Xu
2011-03-01 5:31 ` Eric Dumazet
2011-02-28 11:41 ` [PATCH 4/5] udp: Add lockless transmit path Herbert Xu
2011-02-28 11:41 ` Herbert Xu
2011-03-01 5:30 ` Eric Dumazet
2011-02-28 11:41 ` [PATCH 3/5] inet: Add ip_make_skb and ip_send_skb Herbert Xu
2011-03-01 5:31 ` Eric Dumazet
2011-02-25 19:21 ` SO_REUSEPORT - can it be done in kernel? Eric Dumazet
2011-02-25 22:48 ` Thomas Graf
2011-02-25 23:15 ` Rick Jones
2011-02-25 19:51 ` Tom Herbert
2011-02-25 22:58 ` Thomas Graf
2011-02-25 23:33 ` Bill Sommerfeld
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20110301110708.GJ9763@canuck.infradead.org \
--to=tgraf@infradead.org \
--cc=daniel.baluta@gmail.com \
--cc=davem@davemloft.net \
--cc=eric.dumazet@gmail.com \
--cc=herbert@gondor.apana.org.au \
--cc=netdev@vger.kernel.org \
--cc=rick.jones2@hp.com \
--cc=therbert@google.com \
--cc=wsommerfeld@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).