From: Thomas Graf <tgraf@infradead.org>
To: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Eric Dumazet <eric.dumazet@gmail.com>,
David Miller <davem@davemloft.net>,
rick.jones2@hp.com, therbert@google.com, wsommerfeld@google.com,
daniel.baluta@gmail.com, netdev@vger.kernel.org
Subject: Re: SO_REUSEPORT - can it be done in kernel?
Date: Tue, 1 Mar 2011 08:50:53 -0500 [thread overview]
Message-ID: <20110301135053.GA6684@canuck.infradead.org> (raw)
In-Reply-To: <20110301121951.GA7267@gondor.apana.org.au>
On Tue, Mar 01, 2011 at 08:19:51PM +0800, Herbert Xu wrote:
> On Tue, Mar 01, 2011 at 07:18:29AM -0500, Thomas Graf wrote:
> >
> >
> > ... makes it use CPU 5 for rxq2 and the qps goes up from 250kqps to 270kqps
>
> I think the increase here comes from the larger number of packets
> in flight more than anything.
>
> The bottleneck is still the TX queue (both software and hardware).
Disabled netfilter and reran test
Now does ~316kqps (rx was split over 2 queues)
----------------------------------------------------------------------------------------------------------------------
PerfTop: 30608 irqs/sec kernel:66.1% exact: 0.0% [1000Hz cpu-clock-msecs], (all, CPU: 1)
----------------------------------------------------------------------------------------------------------------------
samples pcnt function DSO
_______ _____ _____________________________ ___________________________________________________________
19237.00 5.6% _raw_spin_unlock_irqrestore /lib/modules/2.6.38-rc5+/build/vmlinux
17170.00 5.0% get_rx_page_info /lib/modules/2.6.38-rc5+/kernel/drivers/net/benet/be2net.ko
11411.00 3.3% be_poll_rx /lib/modules/2.6.38-rc5+/kernel/drivers/net/benet/be2net.ko
11320.00 3.3% isc_rwlock_lock /usr/lib64/libisc.so.62.0.1
10669.00 3.1% __do_softirq /lib/modules/2.6.38-rc5+/build/vmlinux
10655.00 3.1% get_page_from_freelist /lib/modules/2.6.38-rc5+/build/vmlinux
9523.00 2.8% intel_idle /lib/modules/2.6.38-rc5+/build/vmlinux
8677.00 2.5% __udp4_lib_lookup /lib/modules/2.6.38-rc5+/build/vmlinux
8379.00 2.4% sock_queue_rcv_skb /lib/modules/2.6.38-rc5+/build/vmlinux
8226.00 2.4% sk_run_filter /lib/modules/2.6.38-rc5+/build/vmlinux
6724.00 1.9% __netif_receive_skb /lib/modules/2.6.38-rc5+/build/vmlinux
6553.00 1.9% __alloc_skb /lib/modules/2.6.38-rc5+/build/vmlinux
6205.00 1.8% udp_queue_rcv_skb /lib/modules/2.6.38-rc5+/build/vmlinux
6038.00 1.7% _raw_spin_lock /lib/modules/2.6.38-rc5+/build/vmlinux
5868.00 1.7% isc_rwlock_unlock /usr/lib64/libisc.so.62.0.1
5696.00 1.6% dns_rbt_findnode /usr/lib64/libdns.so.69.0.1
5647.00 1.6% read_tsc /lib/modules/2.6.38-rc5+/build/vmlinux
5633.00 1.6% getnstimeofday /lib/modules/2.6.38-rc5+/build/vmlinux
5448.00 1.6% kmem_cache_alloc_node_trace /lib/modules/2.6.38-rc5+/build/vmlinux
5272.00 1.5% finish_task_switch /lib/modules/2.6.38-rc5+/build/vmlinux
4719.00 1.4% sock_def_readable /lib/modules/2.6.38-rc5+/build/vmlinux
4002.00 1.2% is_swiotlb_buffer /lib/modules/2.6.38-rc5+/build/vmlinux
3914.00 1.1% memcpy /lib/modules/2.6.38-rc5+/build/vmlinux
3717.00 1.1% isc_stats_increment /usr/lib64/libisc.so.62.0.1
3706.00 1.1% __udp4_lib_rcv /lib/modules/2.6.38-rc5+/build/vmlinux
3653.00 1.1% ip_rcv /lib/modules/2.6.38-rc5+/build/vmlinux
3598.00 1.0% kmem_cache_alloc_node /lib/modules/2.6.38-rc5+/build/vmlinux
3407.00 1.0% ip_route_input_common /lib/modules/2.6.38-rc5+/build/vmlinux
2683.00 0.8% be_post_rx_frags /lib/modules/2.6.38-rc5+/kernel/drivers/net/benet/be2net.ko
2666.00 0.8% __pthread_mutex_lock_internal /lib64/libpthread-2.12.so
2331.00 0.7% __phys_addr /lib/modules/2.6.38-rc5+/build/vmlinux
2230.00 0.6% __alloc_pages_nodemask /lib/modules/2.6.38-rc5+/build/vmlinux
2023.00 0.6% dns_name_fullcompare /usr/lib64/libdns.so.69.0.1
1972.00 0.6% packet_rcv /lib/modules/2.6.38-rc5+/build/vmlinux
1902.00 0.6% eth_type_trans /lib/modules/2.6.38-rc5+/build/vmlinux
1860.00 0.5% __pthread_mutex_unlock /lib64/libpthread-2.12.so
1804.00 0.5% fget_light /lib/modules/2.6.38-rc5+/build/vmlinux
1739.00 0.5% alloc_pages_current /lib/modules/2.6.38-rc5+/build/vmlinux
1736.00 0.5% dns_rbtnodechain_init /usr/lib64/libdns.so.69.0.1
----------------------------------------------------------------------------------------------------------------------
PerfTop: 29038 irqs/sec kernel:48.0% exact: 0.0% [1000Hz cpu-clock-msecs], (all, CPU: 11)
----------------------------------------------------------------------------------------------------------------------
samples pcnt function DSO
_______ _____ _____________________________ ___________________________________________________________
12833.00 7.5% intel_idle /lib/modules/2.6.38-rc5+/build/vmlinux
10771.00 6.3% isc_rwlock_lock /usr/lib64/libisc.so.62.0.1
8713.00 5.1% be_tx_compl_process /lib/modules/2.6.38-rc5+/kernel/drivers/net/benet/be2net.ko
6452.00 3.8% kfree /lib/modules/2.6.38-rc5+/build/vmlinux
5935.00 3.5% skb_release_data /lib/modules/2.6.38-rc5+/build/vmlinux
5552.00 3.2% kmem_cache_free /lib/modules/2.6.38-rc5+/build/vmlinux
5292.00 3.1% isc_rwlock_unlock /usr/lib64/libisc.so.62.0.1
4893.00 2.9% dns_rbt_findnode /usr/lib64/libdns.so.69.0.1
4413.00 2.6% kfree_skb /lib/modules/2.6.38-rc5+/build/vmlinux
3802.00 2.2% be_poll_tx_mcc /lib/modules/2.6.38-rc5+/kernel/drivers/net/benet/be2net.ko
3515.00 2.1% isc_stats_increment /usr/lib64/libisc.so.62.0.1
3016.00 1.8% _raw_spin_unlock_irqrestore /lib/modules/2.6.38-rc5+/build/vmlinux
2202.00 1.3% __do_softirq /lib/modules/2.6.38-rc5+/build/vmlinux
2027.00 1.2% _raw_spin_lock /lib/modules/2.6.38-rc5+/build/vmlinux
1935.00 1.1% finish_task_switch /lib/modules/2.6.38-rc5+/build/vmlinux
1906.00 1.1% __pthread_mutex_lock_internal /lib64/libpthread-2.12.so
1837.00 1.1% dns_name_fullcompare /usr/lib64/libdns.so.69.0.1
1702.00 1.0% dns_rbtnodechain_init /usr/lib64/libdns.so.69.0.1
1561.00 0.9% fget_light /lib/modules/2.6.38-rc5+/build/vmlinux
1559.00 0.9% dns_name_getlabelsequence /usr/lib64/libdns.so.69.0.1
1491.00 0.9% dns_name_equal /usr/lib64/libdns.so.69.0.1
1464.00 0.9% __pthread_mutex_unlock /lib64/libpthread-2.12.so
1454.00 0.9% dns_acl_match /usr/lib64/libdns.so.69.0.1
1293.00 0.8% dns_zone_attach /usr/lib64/libdns.so.69.0.1
1245.00 0.7% be_xmit /lib/modules/2.6.38-rc5+/kernel/drivers/net/benet/be2net.ko
1159.00 0.7% dns_message_rendersection /usr/lib64/libdns.so.69.0.1
1115.00 0.7% isc___mempool_get /usr/lib64/libisc.so.62.0.1
1100.00 0.6% copy_user_generic_string /lib/modules/2.6.38-rc5+/build/vmlinux
1030.00 0.6% dns_name_fromwire /usr/lib64/libdns.so.69.0.1
1015.00 0.6% dns_name_hash /usr/lib64/libdns.so.69.0.1
1013.00 0.6% isc_radix_search /usr/lib64/libisc.so.62.0.1
970.00 0.6% __ip_route_output_key /lib/modules/2.6.38-rc5+/build/vmlinux
917.00 0.5% fput /lib/modules/2.6.38-rc5+/build/vmlinux
817.00 0.5% dev_queue_xmit /lib/modules/2.6.38-rc5+/build/vmlinux
812.00 0.5% sk_run_filter /lib/modules/2.6.38-rc5+/build/vmlinux
806.00 0.5% avc_has_perm_noaudit /lib/modules/2.6.38-rc5+/build/vmlinux
802.00 0.5% sock_wfree /lib/modules/2.6.38-rc5+/build/vmlinux
793.00 0.5% dns_name_towire /usr/lib64/libdns.so.69.0.1
754.00 0.4% sock_alloc_send_pskb /lib/modules/2.6.38-rc5+/build/vmlinux
752.00 0.4% dns_message_parse /usr/lib64/libdns.so.69.0.1
749.00 0.4% dns_rdata_towire /usr/lib64/libdns.so.69.0.1
728.00 0.4% dns_rdataset_init /usr/lib64/libdns.so.69.0.1
709.00 0.4% isc___mempool_put /usr/lib64/libisc.so.62.0.1
699.00 0.4% skb_release_head_state /lib/modules/2.6.38-rc5+/build/vmlinux
685.00 0.4% _raw_spin_lock_bh /lib/modules/2.6.38-rc5+/build/vmlinux
683.00 0.4% dns_name_concatenate /usr/lib64/libdns.so.69.0.1
678.00 0.4% __ip_append_data /lib/modules/2.6.38-rc5+/build/vmlinux
673.00 0.4% tick_nohz_stop_sched_tick /lib/modules/2.6.38-rc5+/build/vmlinux
662.00 0.4% sys_sendmsg /lib/modules/2.6.38-rc5+/build/vmlinux
654.00 0.4% dns_compress_findglobal /usr/lib64/libdns.so.69.0.1
654.00 0.4% memcpy /lib64/libc-2.12.so
637.00 0.4% dns_compress_invalidate /usr/lib64/libdns.so.69.0.1
597.00 0.3% isc__buffer_init /usr/lib64/libisc.so.62.0.1
595.00 0.3% dns_zone_detach /usr/lib64/libdns.so.69.0.1
WARNING: failed to keep up with mmap data.
WARNING: failed to keep up with mmap data.
----------------------------------------------------------------------------------------------------------------------
PerfTop: 29539 irqs/sec kernel:47.0% exact: 0.0% [1000Hz cpu-clock-msecs], (all, CPU: 11)
----------------------------------------------------------------------------------------------------------------------
samples pcnt function DSO
_______ _____ _____________________________ ___________________________________________________________
14478.00 7.5% intel_idle /lib/modules/2.6.38-rc5+/build/vmlinux
12279.00 6.3% isc_rwlock_lock /usr/lib64/libisc.so.62.0.1
9844.00 5.1% be_tx_compl_process /lib/modules/2.6.38-rc5+/kernel/drivers/net/benet/be2net.ko
7368.00 3.8% kfree /lib/modules/2.6.38-rc5+/build/vmlinux
6696.00 3.5% skb_release_data /lib/modules/2.6.38-rc5+/build/vmlinux
6240.00 3.2% kmem_cache_free /lib/modules/2.6.38-rc5+/build/vmlinux
6034.00 3.1% isc_rwlock_unlock /usr/lib64/libisc.so.62.0.1
5547.00 2.9% dns_rbt_findnode /usr/lib64/libdns.so.69.0.1
5012.00 2.6% kfree_skb /lib/modules/2.6.38-rc5+/build/vmlinux
4290.00 2.2% be_poll_tx_mcc /lib/modules/2.6.38-rc5+/kernel/drivers/net/benet/be2net.ko
4024.00 2.1% isc_stats_increment /usr/lib64/libisc.so.62.0.1
3417.00 1.8% _raw_spin_unlock_irqrestore /lib/modules/2.6.38-rc5+/build/vmlinux
2470.00 1.3% __do_softirq /lib/modules/2.6.38-rc5+/build/vmlinux
2312.00 1.2% _raw_spin_lock /lib/modules/2.6.38-rc5+/build/vmlinux
2138.00 1.1% finish_task_switch /lib/modules/2.6.38-rc5+/build/vmlinux
2136.00 1.1% __pthread_mutex_lock_internal /lib64/libpthread-2.12.so
2061.00 1.1% dns_name_fullcompare /usr/lib64/libdns.so.69.0.1
1961.00 1.0% dns_rbtnodechain_init /usr/lib64/libdns.so.69.0.1
1797.00 0.9% dns_name_getlabelsequence /usr/lib64/libdns.so.69.0.1
1743.00 0.9% fget_light /lib/modules/2.6.38-rc5+/build/vmlinux
1723.00 0.9% dns_name_equal /usr/lib64/libdns.so.69.0.1
1673.00 0.9% __pthread_mutex_unlock /lib64/libpthread-2.12.so
1671.00 0.9% dns_acl_match /usr/lib64/libdns.so.69.0.1
1488.00 0.8% dns_zone_attach /usr/lib64/libdns.so.69.0.1
1428.00 0.7% be_xmit /lib/modules/2.6.38-rc5+/kernel/drivers/net/benet/be2net.ko
1369.00 0.7% dns_message_rendersection /usr/lib64/libdns.so.69.0.1
1278.00 0.7% isc___mempool_get /usr/lib64/libisc.so.62.0.1
1251.00 0.6% copy_user_generic_string /lib/modules/2.6.38-rc5+/build/vmlinux
1193.00 0.6% dns_name_fromwire /usr/lib64/libdns.so.69.0.1
1182.00 0.6% isc_radix_search /usr/lib64/libisc.so.62.0.1
next prev parent reply other threads:[~2011-03-01 13:51 UTC|newest]
Thread overview: 91+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-01-27 10:07 SO_REUSEPORT - can it be done in kernel? Daniel Baluta
2011-01-27 15:55 ` Bill Sommerfeld
2011-01-27 21:32 ` Tom Herbert
2011-02-25 12:56 ` Thomas Graf
2011-02-25 19:18 ` Rick Jones
2011-02-25 19:20 ` David Miller
2011-02-26 0:57 ` Herbert Xu
2011-02-26 2:12 ` David Miller
2011-02-26 2:48 ` Herbert Xu
2011-02-26 3:07 ` David Miller
2011-02-26 3:11 ` Herbert Xu
2011-02-26 7:31 ` Eric Dumazet
2011-02-26 7:46 ` David Miller
2011-02-27 11:02 ` Thomas Graf
2011-02-27 11:06 ` Herbert Xu
2011-02-28 3:45 ` Tom Herbert
2011-02-28 4:26 ` Herbert Xu
2011-02-28 11:36 ` Herbert Xu
2011-02-28 13:32 ` Eric Dumazet
2011-02-28 14:13 ` Herbert Xu
2011-02-28 14:22 ` Eric Dumazet
2011-02-28 14:25 ` Herbert Xu
2011-02-28 14:53 ` Eric Dumazet
2011-02-28 15:01 ` Thomas Graf
2011-02-28 14:13 ` Thomas Graf
2011-02-28 16:22 ` Eric Dumazet
2011-02-28 16:37 ` Thomas Graf
2011-02-28 17:07 ` Eric Dumazet
2011-03-01 10:19 ` Thomas Graf
2011-03-01 10:33 ` Eric Dumazet
2011-03-01 11:07 ` Thomas Graf
2011-03-01 11:13 ` Eric Dumazet
2011-03-01 11:27 ` Thomas Graf
2011-03-01 11:45 ` Eric Dumazet
2011-03-01 11:53 ` Herbert Xu
2011-03-01 12:32 ` Herbert Xu
2011-03-01 13:04 ` Eric Dumazet
2011-03-01 13:11 ` Herbert Xu
2011-03-01 13:03 ` Eric Dumazet
2011-03-01 13:18 ` Herbert Xu
2011-03-01 13:52 ` Eric Dumazet
2011-03-01 13:58 ` Herbert Xu
2011-03-01 16:31 ` Eric Dumazet
2011-03-02 0:23 ` Herbert Xu
2011-03-02 2:00 ` Eric Dumazet
2011-03-02 2:39 ` Herbert Xu
2011-03-02 2:56 ` Eric Dumazet
2011-03-02 3:09 ` Herbert Xu
2011-03-02 3:44 ` Eric Dumazet
2011-03-02 7:12 ` Tom Herbert
2011-03-02 7:31 ` Herbert Xu
2011-03-02 8:04 ` Eric Dumazet
2011-03-02 8:07 ` Herbert Xu
2011-03-02 8:24 ` Eric Dumazet
2011-03-01 12:01 ` Thomas Graf
2011-03-01 12:15 ` Herbert Xu
2011-03-01 13:27 ` Herbert Xu
2011-03-01 12:18 ` Thomas Graf
2011-03-01 12:19 ` Herbert Xu
2011-03-01 13:50 ` Thomas Graf [this message]
2011-03-01 14:06 ` Eric Dumazet
2011-03-01 14:22 ` Thomas Graf
2011-03-01 14:30 ` Thomas Graf
2011-03-01 14:52 ` Eric Dumazet
2011-03-01 15:07 ` Thomas Graf
2011-03-01 5:33 ` Eric Dumazet
2011-03-01 12:35 ` Herbert Xu
2011-03-01 12:36 ` [PATCH 2/5] inet: Remove explicit write references to sk/inet in ip_append_data Herbert Xu
2011-03-02 6:15 ` inet: Replace left-over references to inet->cork Herbert Xu
2011-03-02 7:01 ` David Miller
2011-03-01 12:36 ` [PATCH 1/5] inet: Remove unused sk_sndmsg_* from UFO Herbert Xu
2011-03-01 12:36 ` [PATCH 3/5] inet: Add ip_make_skb and ip_finish_skb Herbert Xu
2011-03-01 12:36 ` [PATCH 4/5] udp: Switch to ip_finish_skb Herbert Xu
2011-03-01 12:36 ` [PATCH 5/5] udp: Add lockless transmit path Herbert Xu
2011-03-01 16:43 ` SO_REUSEPORT - can it be done in kernel? Eric Dumazet
2011-03-01 20:36 ` David Miller
2011-02-28 11:41 ` [PATCH 1/5] net: Remove unused sk_sndmsg_* from UFO Herbert Xu
2011-03-01 5:31 ` Eric Dumazet
2011-02-28 11:41 ` [PATCH 2/5] net: Remove explicit write references to sk/inet in ip_append_data Herbert Xu
2011-03-01 5:31 ` Eric Dumazet
2011-02-28 11:41 ` [PATCH 3/5] inet: Add ip_make_skb and ip_send_skb Herbert Xu
2011-03-01 5:31 ` Eric Dumazet
2011-02-28 11:41 ` [PATCH 4/5] udp: Add lockless transmit path Herbert Xu
2011-02-28 11:41 ` Herbert Xu
2011-03-01 5:30 ` Eric Dumazet
2011-02-25 19:21 ` SO_REUSEPORT - can it be done in kernel? Eric Dumazet
2011-02-25 22:48 ` Thomas Graf
2011-02-25 23:15 ` Rick Jones
2011-02-25 19:51 ` Tom Herbert
2011-02-25 22:58 ` Thomas Graf
2011-02-25 23:33 ` Bill Sommerfeld
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20110301135053.GA6684@canuck.infradead.org \
--to=tgraf@infradead.org \
--cc=daniel.baluta@gmail.com \
--cc=davem@davemloft.net \
--cc=eric.dumazet@gmail.com \
--cc=herbert@gondor.apana.org.au \
--cc=netdev@vger.kernel.org \
--cc=rick.jones2@hp.com \
--cc=therbert@google.com \
--cc=wsommerfeld@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).