From: Eric Dumazet <eric.dumazet@gmail.com>
To: robert@herjulf.net
Cc: Jesper Dangaard Brouer <hawk@diku.dk>,
Peter P Waskiewicz Jr <peter.p.waskiewicz.jr@intel.com>,
Linux Netdev List <netdev@vger.kernel.org>
Subject: Re: ixgbe question
Date: Mon, 23 Nov 2009 17:59:25 +0100 [thread overview]
Message-ID: <4B0ABF6D.9000103@gmail.com> (raw)
In-Reply-To: <19210.54486.353397.804028@gargle.gargle.HOWL>
robert@herjulf.net a écrit :
> Eric Dumazet writes:
>
> > Jesper Dangaard Brouer a écrit :
> >
> > > How is your smp_affinity mask's set?
> > >
> > > grep . /proc/irq/*/fiber1-*/../smp_affinity
> >
>
> Weird... set clone_skb to 1 to be sure and vary dst or something so
> the HW classifier selects different queues and with proper RX affinty.
>
> You should see in /proc/net/softnet_stat something like:
>
> 012a7bb9 00000000 000000ae 00000000 00000000 00000000 00000000 00000000 00000000
> 01288d4c 00000000 00000049 00000000 00000000 00000000 00000000 00000000 00000000
> 0128fe28 00000000 00000043 00000000 00000000 00000000 00000000 00000000 00000000
> 01295387 00000000 00000047 00000000 00000000 00000000 00000000 00000000 00000000
> 0129a722 00000000 0000004a 00000000 00000000 00000000 00000000 00000000 00000000
> 0128c5e4 00000000 00000046 00000000 00000000 00000000 00000000 00000000 00000000
> 0128f718 00000000 00000043 00000000 00000000 00000000 00000000 00000000 00000000
> 012993e3 00000000 0000004a 00000000 00000000 00000000 00000000 00000000 00000000
>
slone_skb set to 1, this changes nothing but slows down pktgen (obviously)
Result: OK: 117614452(c117608705+d5746) nsec, 100000000 (60byte,0frags)
850235pps 408Mb/sec (408112800bps) errors: 0
All RX processing of 16 RX queues done by CPU 1 only.
# cat /proc/net/softnet_stat ; sleep 2 ; echo "--------------";cat /proc/net/softnet_stat
0039f331 00000000 00002e10 00000000 00000000 00000000 00000000 00000000 00000000
03f2ed19 00000000 00037ca2 00000000 00000000 00000000 00000000 00000000 00000000
00000024 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000041 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000028 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0000000b 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
000000c5 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0000010d 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000250 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000498 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000616 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0000012c 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
000000d2 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0000025d 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0000003c 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000127 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
--------------
0039f331 00000000 00002e10 00000000 00000000 00000000 00000000 00000000 00000000
03f66737 00000000 00038015 00000000 00000000 00000000 00000000 00000000 00000000
00000024 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000041 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000028 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0000000b 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
000000c5 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000110 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000250 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000499 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000616 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0000012c 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
000000d2 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000263 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0000003c 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000129 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ethtool -S fiber1 (to show how my trafic is equally distributed to 16 RX queues)
rx_queue_0_packets: 4867706
rx_queue_0_bytes: 292062360
rx_queue_1_packets: 4862472
rx_queue_1_bytes: 291748320
rx_queue_2_packets: 4867111
rx_queue_2_bytes: 292026660
rx_queue_3_packets: 4859897
rx_queue_3_bytes: 291593820
rx_queue_4_packets: 4862267
rx_queue_4_bytes: 291740814
rx_queue_5_packets: 4861517
rx_queue_5_bytes: 291691020
rx_queue_6_packets: 4862699
rx_queue_6_bytes: 291761940
rx_queue_7_packets: 4860523
rx_queue_7_bytes: 291631380
rx_queue_8_packets: 4856891
rx_queue_8_bytes: 291413460
rx_queue_9_packets: 4868794
rx_queue_9_bytes: 292127640
rx_queue_10_packets: 4859099
rx_queue_10_bytes: 291545940
rx_queue_11_packets: 4867599
rx_queue_11_bytes: 292055940
rx_queue_12_packets: 4861868
rx_queue_12_bytes: 291713374
rx_queue_13_packets: 4862655
rx_queue_13_bytes: 291759300
rx_queue_14_packets: 4860798
rx_queue_14_bytes: 291647880
rx_queue_15_packets: 4860951
rx_queue_15_bytes: 291657060
perf top -C 1 -E 25
------------------------------------------------------------------------------
PerfTop: 24419 irqs/sec kernel:100.0% [100000 cycles], (all, cpu: 1)
------------------------------------------------------------------------------
samples pcnt kernel function
_______ _____ _______________
46234.00 - 24.3% : ixgbe_clean_tx_irq [ixgbe]
21134.00 - 11.1% : __slab_free
17838.00 - 9.4% : _raw_spin_lock
17086.00 - 9.0% : skb_release_head_state
9410.00 - 5.0% : ixgbe_clean_rx_irq [ixgbe]
8639.00 - 4.5% : kmem_cache_free
6910.00 - 3.6% : kfree
5743.00 - 3.0% : __ip_route_output_key
5321.00 - 2.8% : ip_route_input
3138.00 - 1.7% : ip_rcv
2179.00 - 1.1% : kmem_cache_alloc_node
2002.00 - 1.1% : __kmalloc_node_track_caller
1907.00 - 1.0% : skb_put
1807.00 - 1.0% : __xfrm_lookup
1742.00 - 0.9% : get_partial_node
1727.00 - 0.9% : csum_partial_copy_generic
1541.00 - 0.8% : add_partial
1516.00 - 0.8% : __kfree_skb
1465.00 - 0.8% : __netdev_alloc_skb
1420.00 - 0.7% : icmp_send
1222.00 - 0.6% : dev_gro_receive
1159.00 - 0.6% : fib_table_lookup
1155.00 - 0.6% : __phys_addr
1050.00 - 0.6% : skb_release_data
982.00 - 0.5% : _raw_spin_unlock
next prev parent reply other threads:[~2009-11-23 16:59 UTC|newest]
Thread overview: 66+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-11-23 6:46 [PATCH] irq: Add node_affinity CPU masks for smarter irqbalance hints Peter P Waskiewicz Jr
2009-11-23 7:32 ` Yong Zhang
2009-11-23 9:36 ` Peter P Waskiewicz Jr
2009-11-23 10:21 ` ixgbe question Eric Dumazet
2009-11-23 10:30 ` Badalian Vyacheslav
2009-11-23 10:34 ` Waskiewicz Jr, Peter P
2009-11-23 10:37 ` Eric Dumazet
2009-11-23 14:05 ` Eric Dumazet
2009-11-23 21:26 ` David Miller
2009-11-23 14:10 ` Jesper Dangaard Brouer
2009-11-23 14:38 ` Eric Dumazet
2009-11-23 18:30 ` robert
2009-11-23 16:59 ` Eric Dumazet [this message]
2009-11-23 20:54 ` robert
2009-11-23 21:28 ` David Miller
2009-11-23 22:14 ` Robert Olsson
2009-11-23 23:28 ` Waskiewicz Jr, Peter P
2009-11-23 23:44 ` David Miller
2009-11-24 7:46 ` Eric Dumazet
2009-11-24 8:46 ` Badalian Vyacheslav
2009-11-24 9:07 ` Peter P Waskiewicz Jr
2009-11-24 9:55 ` Eric Dumazet
2009-11-24 10:06 ` Peter P Waskiewicz Jr
2009-11-24 11:37 ` [PATCH net-next-2.6] ixgbe: Fix TX stats accounting Eric Dumazet
2009-11-24 13:23 ` Eric Dumazet
2009-11-25 7:38 ` Jeff Kirsher
2009-11-25 9:31 ` Eric Dumazet
2009-11-25 9:38 ` Jeff Kirsher
2009-11-24 13:14 ` ixgbe question John Fastabend
2009-11-29 8:18 ` David Miller
2009-11-30 13:02 ` Eric Dumazet
2009-11-30 20:20 ` John Fastabend
2009-11-26 14:10 ` Badalian Vyacheslav
2009-11-23 17:05 ` [PATCH] irq: Add node_affinity CPU masks for smarter irqbalance hints Peter Zijlstra
2009-11-23 23:32 ` Waskiewicz Jr, Peter P
2009-11-24 8:38 ` Peter Zijlstra
2009-11-24 8:59 ` Peter P Waskiewicz Jr
2009-11-24 9:08 ` Peter Zijlstra
2009-11-24 9:15 ` Peter P Waskiewicz Jr
2009-11-24 14:43 ` Arjan van de Ven
2009-11-24 9:15 ` Peter Zijlstra
2009-11-24 10:07 ` Thomas Gleixner
2009-11-24 17:55 ` Peter P Waskiewicz Jr
2009-11-25 11:18 ` Peter Zijlstra
2009-11-24 6:07 ` Arjan van de Ven
2009-11-24 8:39 ` Peter Zijlstra
2009-11-24 14:42 ` Arjan van de Ven
2009-11-24 17:39 ` David Miller
2009-11-24 17:56 ` Peter P Waskiewicz Jr
2009-11-24 18:26 ` Eric Dumazet
2009-11-24 18:33 ` Peter P Waskiewicz Jr
2009-11-24 19:01 ` Eric Dumazet
2009-11-24 19:53 ` Peter P Waskiewicz Jr
2009-11-24 18:54 ` David Miller
2009-11-24 18:58 ` Eric Dumazet
2009-11-24 20:35 ` Andi Kleen
2009-11-24 20:46 ` Eric Dumazet
2009-11-25 10:30 ` Eric Dumazet
2009-11-25 10:37 ` Andi Kleen
2009-11-25 11:35 ` Eric Dumazet
2009-11-25 11:50 ` Andi Kleen
2009-11-26 11:43 ` Eric Dumazet
2009-11-24 5:17 ` Yong Zhang
2009-11-24 8:39 ` Peter P Waskiewicz Jr
-- strict thread matches above, loose matches on Subject: below --
2008-03-10 21:27 Ixgbe question Ben Greear
2008-03-11 1:01 ` Brandeburg, Jesse
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4B0ABF6D.9000103@gmail.com \
--to=eric.dumazet@gmail.com \
--cc=hawk@diku.dk \
--cc=netdev@vger.kernel.org \
--cc=peter.p.waskiewicz.jr@intel.com \
--cc=robert@herjulf.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).