From: Eric Dumazet <eric.dumazet@gmail.com>
To: "Waskiewicz Jr, Peter P" <peter.p.waskiewicz.jr@intel.com>
Cc: Linux Netdev List <netdev@vger.kernel.org>
Subject: Re: ixgbe question
Date: Mon, 23 Nov 2009 11:37:20 +0100 [thread overview]
Message-ID: <4B0A65E0.7060403@gmail.com> (raw)
In-Reply-To: <Pine.WNT.4.64.0911230225140.6352@ppwaskie-MOBL2.amr.corp.intel.com>
Waskiewicz Jr, Peter P a écrit :
> On Mon, 23 Nov 2009, Eric Dumazet wrote:
>
>> Hi Peter
>>
>> I tried a pktgen stress on 82599EB card and could not split RX load on multiple cpus.
>>
>> Setup is :
>>
>> One 82599 card with fiber0 looped to fiber1, 10Gb link mode.
>> machine is a HPDL380 G6 with dual quadcore E5530 @2.4GHz (16 logical cpus)
>
> Can you specify kernel version and driver version?
Well, I forgot to mention I am only working with net-next-2.6 tree.
Ubuntu 9.10 kernel (Fedora Core 12 installer was not able to recognize disks on this machine :( )
ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 2.0.44-k2
>
>> I use one pktgen thread sending to fiber0 one many dst IP, and checked that fiber1
>> was using many RX queues :
>>
>> grep fiber1 /proc/interrupts
>> 117: 1301 13060 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-0
>> 118: 601 1402 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-1
>> 119: 634 832 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-2
>> 120: 601 1303 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-3
>> 121: 620 1246 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-4
>> 122: 1287 13088 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-5
>> 123: 606 1354 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-6
>> 124: 653 827 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-7
>> 125: 639 825 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-8
>> 126: 596 1199 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-9
>> 127: 2013 24800 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-10
>> 128: 648 1353 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-11
>> 129: 601 1123 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-12
>> 130: 625 834 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-13
>> 131: 665 1409 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-14
>> 132: 2637 31699 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-15
>> 133: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1:lsc
>>
>>
>>
>> But only one CPU (CPU1) had a softirq running, 100%, and many frames were dropped
>>
>> root@demodl380g6:/usr/src# ifconfig fiber0
>> fiber0 Link encap:Ethernet HWaddr 00:1b:21:4a:fe:54
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> Packets reçus:4 erreurs:0 :0 overruns:0 frame:0
>> TX packets:309291576 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 lg file transmission:1000
>> Octets reçus:1368 (1.3 KB) Octets transmis:18557495682 (18.5 GB)
>>
>> root@demodl380g6:/usr/src# ifconfig fiber1
>> fiber1 Link encap:Ethernet HWaddr 00:1b:21:4a:fe:55
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> Packets reçus:55122164 erreurs:0 :254169411 overruns:0 frame:0
>> TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 lg file transmission:1000
>> Octets reçus:3307330968 (3.3 GB) Octets transmis:1368 (1.3 KB)
>
> I stay in the states too much. I love seeing net stats in French. :-)
Ok :)
>
>>
>> How and when multi queue rx can really start to use several cpus ?
>
> If you're sending one flow to many consumers, it's still one flow. Even
> using RSS won't help, since it requires differing flows to spread load
> (5-tuple matches for flow distribution).
Hm... I can try varying both src and dst on my pktgen test.
Thanks
next prev parent reply other threads:[~2009-11-23 10:37 UTC|newest]
Thread overview: 66+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-11-23 6:46 [PATCH] irq: Add node_affinity CPU masks for smarter irqbalance hints Peter P Waskiewicz Jr
2009-11-23 7:32 ` Yong Zhang
2009-11-23 9:36 ` Peter P Waskiewicz Jr
2009-11-23 10:21 ` ixgbe question Eric Dumazet
2009-11-23 10:30 ` Badalian Vyacheslav
2009-11-23 10:34 ` Waskiewicz Jr, Peter P
2009-11-23 10:37 ` Eric Dumazet [this message]
2009-11-23 14:05 ` Eric Dumazet
2009-11-23 21:26 ` David Miller
2009-11-23 14:10 ` Jesper Dangaard Brouer
2009-11-23 14:38 ` Eric Dumazet
2009-11-23 18:30 ` robert
2009-11-23 16:59 ` Eric Dumazet
2009-11-23 20:54 ` robert
2009-11-23 21:28 ` David Miller
2009-11-23 22:14 ` Robert Olsson
2009-11-23 23:28 ` Waskiewicz Jr, Peter P
2009-11-23 23:44 ` David Miller
2009-11-24 7:46 ` Eric Dumazet
2009-11-24 8:46 ` Badalian Vyacheslav
2009-11-24 9:07 ` Peter P Waskiewicz Jr
2009-11-24 9:55 ` Eric Dumazet
2009-11-24 10:06 ` Peter P Waskiewicz Jr
2009-11-24 11:37 ` [PATCH net-next-2.6] ixgbe: Fix TX stats accounting Eric Dumazet
2009-11-24 13:23 ` Eric Dumazet
2009-11-25 7:38 ` Jeff Kirsher
2009-11-25 9:31 ` Eric Dumazet
2009-11-25 9:38 ` Jeff Kirsher
2009-11-24 13:14 ` ixgbe question John Fastabend
2009-11-29 8:18 ` David Miller
2009-11-30 13:02 ` Eric Dumazet
2009-11-30 20:20 ` John Fastabend
2009-11-26 14:10 ` Badalian Vyacheslav
2009-11-23 17:05 ` [PATCH] irq: Add node_affinity CPU masks for smarter irqbalance hints Peter Zijlstra
2009-11-23 23:32 ` Waskiewicz Jr, Peter P
2009-11-24 8:38 ` Peter Zijlstra
2009-11-24 8:59 ` Peter P Waskiewicz Jr
2009-11-24 9:08 ` Peter Zijlstra
2009-11-24 9:15 ` Peter P Waskiewicz Jr
2009-11-24 14:43 ` Arjan van de Ven
2009-11-24 9:15 ` Peter Zijlstra
2009-11-24 10:07 ` Thomas Gleixner
2009-11-24 17:55 ` Peter P Waskiewicz Jr
2009-11-25 11:18 ` Peter Zijlstra
2009-11-24 6:07 ` Arjan van de Ven
2009-11-24 8:39 ` Peter Zijlstra
2009-11-24 14:42 ` Arjan van de Ven
2009-11-24 17:39 ` David Miller
2009-11-24 17:56 ` Peter P Waskiewicz Jr
2009-11-24 18:26 ` Eric Dumazet
2009-11-24 18:33 ` Peter P Waskiewicz Jr
2009-11-24 19:01 ` Eric Dumazet
2009-11-24 19:53 ` Peter P Waskiewicz Jr
2009-11-24 18:54 ` David Miller
2009-11-24 18:58 ` Eric Dumazet
2009-11-24 20:35 ` Andi Kleen
2009-11-24 20:46 ` Eric Dumazet
2009-11-25 10:30 ` Eric Dumazet
2009-11-25 10:37 ` Andi Kleen
2009-11-25 11:35 ` Eric Dumazet
2009-11-25 11:50 ` Andi Kleen
2009-11-26 11:43 ` Eric Dumazet
2009-11-24 5:17 ` Yong Zhang
2009-11-24 8:39 ` Peter P Waskiewicz Jr
-- strict thread matches above, loose matches on Subject: below --
2008-03-10 21:27 Ixgbe question Ben Greear
2008-03-11 1:01 ` Brandeburg, Jesse
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4B0A65E0.7060403@gmail.com \
--to=eric.dumazet@gmail.com \
--cc=netdev@vger.kernel.org \
--cc=peter.p.waskiewicz.jr@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).