netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Thomas Graf <tgraf@infradead.org>
To: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>,
	David Miller <davem@davemloft.net>,
	rick.jones2@hp.com, therbert@google.com, wsommerfeld@google.com,
	daniel.baluta@gmail.com, netdev@vger.kernel.org
Subject: Re: SO_REUSEPORT - can it be done in kernel?
Date: Tue, 1 Mar 2011 06:27:59 -0500	[thread overview]
Message-ID: <20110301112759.GK9763@canuck.infradead.org> (raw)
In-Reply-To: <1298977984.3284.15.camel@edumazet-laptop>

On Tue, Mar 01, 2011 at 12:13:04PM +0100, Eric Dumazet wrote:
> Its a bit strange two cpus spend time in softirq, unless you have two
> queryperf sources, and a multiqueue NIC, or maybe you use two NICS ?

one NIC, 2 clients (12 instances per client)

[root@hp-bl460cg7-01 ~]# cat /sys/class/net/eth0/queues/rx-0/rps_cpus 
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000

[root@hp-bl460cg7-01 ~]# netstat -s | grep err
    1781377 packet receive errors

> Mind use "perf top -C 1" and "perf top -C 11" to check what these cpus
> do ?

--------------------------------------------------------------------------------------------------------------------
   PerfTop:   16198 irqs/sec  kernel:99.1%  exact:  0.0% [1000Hz cpu-clock-msecs],  (all, CPU: 1)
--------------------------------------------------------------------------------------------------------------------

             samples  pcnt function                    DSO
             _______ _____ ___________________________ ___________________________________________________________

            51675.00 33.2% _raw_spin_unlock_irqrestore [kernel.kallsyms]                                          
            12426.00  8.0% clflush_cache_range         [kernel.kallsyms]                                          
             5511.00  3.5% be_poll_rx                  /lib/modules/2.6.38-rc5+/kernel/drivers/net/benet/be2net.ko
             4567.00  2.9% __udp4_lib_lookup           [kernel.kallsyms]                                          
             3981.00  2.6% __kmalloc_node_track_caller [kernel.kallsyms]                                          
             3975.00  2.6% get_rx_page_info            /lib/modules/2.6.38-rc5+/kernel/drivers/net/benet/be2net.ko
             3725.00  2.4% sk_run_filter               [kernel.kallsyms]                                          
             3606.00  2.3% get_page_from_freelist      [kernel.kallsyms]                                          
             3178.00  2.0% __domain_mapping            [kernel.kallsyms]                                          
             3122.00  2.0% kmem_cache_alloc_node       [kernel.kallsyms]                                          
             2839.00  1.8% sock_queue_rcv_skb          [kernel.kallsyms]                                          
             2246.00  1.4% __netif_receive_skb         [kernel.kallsyms]                                          
             2245.00  1.4% nf_iterate                  [kernel.kallsyms]                                          
             2081.00  1.3% __udp4_lib_rcv              [kernel.kallsyms]                                          
             2042.00  1.3% ipt_do_table                [kernel.kallsyms]                                          
             1901.00  1.2% _raw_spin_lock              [kernel.kallsyms]                                          
             1856.00  1.2% __alloc_skb                 [kernel.kallsyms]                                          
             1645.00  1.1% read_tsc                    [kernel.kallsyms]                                          
             1562.00  1.0% nf_ct_tuple_equal           [kernel.kallsyms]                                          
             1562.00  1.0% ip_rcv                      [kernel.kallsyms]                                          
             1495.00  1.0% __nf_conntrack_find_get     [kernel.kallsyms]                                          
             1477.00  0.9% sock_def_readable           [kernel.kallsyms]                                          
             1363.00  0.9% find_first_bit              [kernel.kallsyms]                                          
             1360.00  0.9% domain_get_iommu            [kernel.kallsyms]                                          
             1255.00  0.8% udp_queue_rcv_skb           [kernel.kallsyms]                                          
             1174.00  0.8% xfrm4_policy_check.clone.0  [kernel.kallsyms]                                          
             1138.00  0.7% hash_conntrack_raw          [kernel.kallsyms]                                          
             1000.00  0.6% intel_unmap_page            [kernel.kallsyms]                                          
              959.00  0.6% load_pointer                [kernel.kallsyms]                                          
              957.00  0.6% sock_flag                   [kernel.kallsyms]                                          
              938.00  0.6% nf_conntrack_in             [kernel.kallsyms]                                          
              891.00  0.6% _local_bh_enable_ip         [kernel.kallsyms]                                          
              884.00  0.6% eth_type_trans              [kernel.kallsyms]                                          
              832.00  0.5% be_post_rx_frags            /lib/modules/2.6.38-rc5+/kernel/drivers/net/benet/be2net.ko
              829.00  0.5% __alloc_pages_nodemask      [kernel.kallsyms]                                          
              813.00  0.5% kmem_cache_alloc            [kernel.kallsyms]                                          
              802.00  0.5% netif_receive_skb           [kernel.kallsyms]                                          
              802.00  0.5% ip_route_input_common       [kernel.kallsyms]                                          
              723.00  0.5% nf_ct_get_tuple             [kernel.kallsyms]                                          
              720.00  0.5% __intel_map_single          [kernel.kallsyms]                                          
              720.00  0.5% udp_error                   [kernel.kallsyms]                                          

--------------------------------------------------------------------------------------------------------------------
   PerfTop:   16360 irqs/sec  kernel:72.6%  exact:  0.0% [1000Hz cpu-clock-msecs],  (all, CPU: 11)
--------------------------------------------------------------------------------------------------------------------

             samples  pcnt function                      DSO
             _______ _____ _____________________________ ___________________________________________________________

            16993.00 32.4% _raw_spin_unlock_irqrestore   [kernel.kallsyms]                                          
             5833.00 11.1% clflush_cache_range           [kernel.kallsyms]                                          
             3315.00  6.3% be_tx_compl_process           /lib/modules/2.6.38-rc5+/kernel/drivers/net/benet/be2net.ko
             1818.00  3.5% kmem_cache_free               [kernel.kallsyms]                                          
             1415.00  2.7% isc_rwlock_lock               /usr/lib64/libisc.so.62.0.1                                
             1090.00  2.1% be_poll_tx_mcc                /lib/modules/2.6.38-rc5+/kernel/drivers/net/benet/be2net.ko
              811.00  1.5% skb_release_head_state        [kernel.kallsyms]                                          
              772.00  1.5% skb_release_data              [kernel.kallsyms]                                          
              712.00  1.4% dns_rbt_findnode              /usr/lib64/libdns.so.69.0.1                                
              703.00  1.3% isc_rwlock_unlock             /usr/lib64/libisc.so.62.0.1                                
              695.00  1.3% dma_pte_clear_range           [kernel.kallsyms]                                          
              618.00  1.2% kfree_skb                     [kernel.kallsyms]                                          
              597.00  1.1% kfree                         [kernel.kallsyms]                                          
              553.00  1.1% intel_unmap_page              [kernel.kallsyms]                                          
              531.00  1.0% __do_softirq                  [kernel.kallsyms]                                          
              504.00  1.0% isc_stats_increment           /usr/lib64/libisc.so.62.0.1                                
              397.00  0.8% virt_to_head_page             [kernel.kallsyms]                                          
              306.00  0.6% _raw_spin_lock                [kernel.kallsyms]                                          
              270.00  0.5% domain_get_iommu              [kernel.kallsyms]                                          
              256.00  0.5% dns_name_fullcompare          /usr/lib64/libdns.so.69.0.1                                
              233.00  0.4% find_first_bit                [kernel.kallsyms]                                          
              222.00  0.4% dns_name_equal                /usr/lib64/libdns.so.69.0.1                                
              218.00  0.4% __pthread_mutex_lock_internal /lib64/libpthread-2.12.so                                  
              207.00  0.4% dns_rbtnodechain_init         /usr/lib64/libdns.so.69.0.1                                
              196.00  0.4% dns_acl_match                 /usr/lib64/libdns.so.69.0.1                                
              194.00  0.4% dma_pte_free_pagetable        [kernel.kallsyms]                                          
              192.00  0.4% dns_name_getlabelsequence     /usr/lib64/libdns.so.69.0.1                                


  reply	other threads:[~2011-03-01 11:28 UTC|newest]

Thread overview: 91+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2011-01-27 10:07 SO_REUSEPORT - can it be done in kernel? Daniel Baluta
2011-01-27 15:55 ` Bill Sommerfeld
2011-01-27 21:32 ` Tom Herbert
2011-02-25 12:56   ` Thomas Graf
2011-02-25 19:18     ` Rick Jones
2011-02-25 19:20       ` David Miller
2011-02-26  0:57         ` Herbert Xu
2011-02-26  2:12           ` David Miller
2011-02-26  2:48             ` Herbert Xu
2011-02-26  3:07               ` David Miller
2011-02-26  3:11                 ` Herbert Xu
2011-02-26  7:31                   ` Eric Dumazet
2011-02-26  7:46                     ` David Miller
2011-02-27 11:02           ` Thomas Graf
2011-02-27 11:06             ` Herbert Xu
2011-02-28  3:45               ` Tom Herbert
2011-02-28  4:26                 ` Herbert Xu
2011-02-28 11:36               ` Herbert Xu
2011-02-28 13:32                 ` Eric Dumazet
2011-02-28 14:13                   ` Herbert Xu
2011-02-28 14:22                     ` Eric Dumazet
2011-02-28 14:25                       ` Herbert Xu
2011-02-28 14:53                   ` Eric Dumazet
2011-02-28 15:01                     ` Thomas Graf
2011-02-28 14:13                 ` Thomas Graf
2011-02-28 16:22                   ` Eric Dumazet
2011-02-28 16:37                     ` Thomas Graf
2011-02-28 17:07                       ` Eric Dumazet
2011-03-01 10:19                         ` Thomas Graf
2011-03-01 10:33                           ` Eric Dumazet
2011-03-01 11:07                             ` Thomas Graf
2011-03-01 11:13                               ` Eric Dumazet
2011-03-01 11:27                                 ` Thomas Graf [this message]
2011-03-01 11:45                                   ` Eric Dumazet
2011-03-01 11:53                                     ` Herbert Xu
2011-03-01 12:32                                       ` Herbert Xu
2011-03-01 13:04                                         ` Eric Dumazet
2011-03-01 13:11                                           ` Herbert Xu
2011-03-01 13:03                                       ` Eric Dumazet
2011-03-01 13:18                                         ` Herbert Xu
2011-03-01 13:52                                           ` Eric Dumazet
2011-03-01 13:58                                             ` Herbert Xu
2011-03-01 16:31                                           ` Eric Dumazet
2011-03-02  0:23                                             ` Herbert Xu
2011-03-02  2:00                                               ` Eric Dumazet
2011-03-02  2:39                                                 ` Herbert Xu
2011-03-02  2:56                                                   ` Eric Dumazet
2011-03-02  3:09                                                     ` Herbert Xu
2011-03-02  3:44                                                       ` Eric Dumazet
2011-03-02  7:12                                                   ` Tom Herbert
2011-03-02  7:31                                                     ` Herbert Xu
2011-03-02  8:04                                                       ` Eric Dumazet
2011-03-02  8:07                                                         ` Herbert Xu
2011-03-02  8:24                                                           ` Eric Dumazet
2011-03-01 12:01                                     ` Thomas Graf
2011-03-01 12:15                                       ` Herbert Xu
2011-03-01 13:27                                       ` Herbert Xu
2011-03-01 12:18                                     ` Thomas Graf
2011-03-01 12:19                                       ` Herbert Xu
2011-03-01 13:50                                         ` Thomas Graf
2011-03-01 14:06                                           ` Eric Dumazet
2011-03-01 14:22                                             ` Thomas Graf
2011-03-01 14:30                                               ` Thomas Graf
2011-03-01 14:52                                                 ` Eric Dumazet
2011-03-01 15:07                                                   ` Thomas Graf
2011-03-01  5:33                 ` Eric Dumazet
2011-03-01 12:35                 ` Herbert Xu
2011-03-01 12:36                   ` [PATCH 2/5] inet: Remove explicit write references to sk/inet in ip_append_data Herbert Xu
2011-03-02  6:15                     ` inet: Replace left-over references to inet->cork Herbert Xu
2011-03-02  7:01                       ` David Miller
2011-03-01 12:36                   ` [PATCH 1/5] inet: Remove unused sk_sndmsg_* from UFO Herbert Xu
2011-03-01 12:36                   ` [PATCH 3/5] inet: Add ip_make_skb and ip_finish_skb Herbert Xu
2011-03-01 12:36                   ` [PATCH 4/5] udp: Switch to ip_finish_skb Herbert Xu
2011-03-01 12:36                   ` [PATCH 5/5] udp: Add lockless transmit path Herbert Xu
2011-03-01 16:43                   ` SO_REUSEPORT - can it be done in kernel? Eric Dumazet
2011-03-01 20:36                     ` David Miller
2011-02-28 11:41               ` [PATCH 2/5] net: Remove explicit write references to sk/inet in ip_append_data Herbert Xu
2011-03-01  5:31                 ` Eric Dumazet
2011-02-28 11:41               ` [PATCH 1/5] net: Remove unused sk_sndmsg_* from UFO Herbert Xu
2011-03-01  5:31                 ` Eric Dumazet
2011-02-28 11:41               ` [PATCH 4/5] udp: Add lockless transmit path Herbert Xu
2011-02-28 11:41                 ` Herbert Xu
2011-03-01  5:30                 ` Eric Dumazet
2011-02-28 11:41               ` [PATCH 3/5] inet: Add ip_make_skb and ip_send_skb Herbert Xu
2011-03-01  5:31                 ` Eric Dumazet
2011-02-25 19:21       ` SO_REUSEPORT - can it be done in kernel? Eric Dumazet
2011-02-25 22:48       ` Thomas Graf
2011-02-25 23:15         ` Rick Jones
2011-02-25 19:51     ` Tom Herbert
2011-02-25 22:58       ` Thomas Graf
2011-02-25 23:33       ` Bill Sommerfeld

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20110301112759.GK9763@canuck.infradead.org \
    --to=tgraf@infradead.org \
    --cc=daniel.baluta@gmail.com \
    --cc=davem@davemloft.net \
    --cc=eric.dumazet@gmail.com \
    --cc=herbert@gondor.apana.org.au \
    --cc=netdev@vger.kernel.org \
    --cc=rick.jones2@hp.com \
    --cc=therbert@google.com \
    --cc=wsommerfeld@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).