From: Eric Dumazet <eric.dumazet@gmail.com>
To: "Waskiewicz Jr, Peter P" <peter.p.waskiewicz.jr@intel.com>
Cc: Linux Netdev List <netdev@vger.kernel.org>
Subject: Re: ixgbe question
Date: Mon, 23 Nov 2009 11:37:20 +0100 [thread overview]
Message-ID: <4B0A65E0.7060403@gmail.com> (raw)
In-Reply-To: <Pine.WNT.4.64.0911230225140.6352@ppwaskie-MOBL2.amr.corp.intel.com>
Waskiewicz Jr, Peter P a écrit :
> On Mon, 23 Nov 2009, Eric Dumazet wrote:
>
>> Hi Peter
>>
>> I tried a pktgen stress on 82599EB card and could not split RX load on multiple cpus.
>>
>> Setup is :
>>
>> One 82599 card with fiber0 looped to fiber1, 10Gb link mode.
>> machine is a HPDL380 G6 with dual quadcore E5530 @2.4GHz (16 logical cpus)
>
> Can you specify kernel version and driver version?
Well, I forgot to mention I am only working with net-next-2.6 tree.
Ubuntu 9.10 kernel (Fedora Core 12 installer was not able to recognize disks on this machine :( )
ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 2.0.44-k2
>
>> I use one pktgen thread sending to fiber0 one many dst IP, and checked that fiber1
>> was using many RX queues :
>>
>> grep fiber1 /proc/interrupts
>> 117: 1301 13060 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-0
>> 118: 601 1402 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-1
>> 119: 634 832 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-2
>> 120: 601 1303 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-3
>> 121: 620 1246 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-4
>> 122: 1287 13088 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-5
>> 123: 606 1354 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-6
>> 124: 653 827 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-7
>> 125: 639 825 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-8
>> 126: 596 1199 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-9
>> 127: 2013 24800 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-10
>> 128: 648 1353 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-11
>> 129: 601 1123 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-12
>> 130: 625 834 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-13
>> 131: 665 1409 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-14
>> 132: 2637 31699 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1-TxRx-15
>> 133: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge fiber1:lsc
>>
>>
>>
>> But only one CPU (CPU1) had a softirq running, 100%, and many frames were dropped
>>
>> root@demodl380g6:/usr/src# ifconfig fiber0
>> fiber0 Link encap:Ethernet HWaddr 00:1b:21:4a:fe:54
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> Packets reçus:4 erreurs:0 :0 overruns:0 frame:0
>> TX packets:309291576 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 lg file transmission:1000
>> Octets reçus:1368 (1.3 KB) Octets transmis:18557495682 (18.5 GB)
>>
>> root@demodl380g6:/usr/src# ifconfig fiber1
>> fiber1 Link encap:Ethernet HWaddr 00:1b:21:4a:fe:55
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> Packets reçus:55122164 erreurs:0 :254169411 overruns:0 frame:0
>> TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 lg file transmission:1000
>> Octets reçus:3307330968 (3.3 GB) Octets transmis:1368 (1.3 KB)
>
> I stay in the states too much. I love seeing net stats in French. :-)
Ok :)
>
>>
>> How and when multi queue rx can really start to use several cpus ?
>
> If you're sending one flow to many consumers, it's still one flow. Even
> using RSS won't help, since it requires differing flows to spread load
> (5-tuple matches for flow distribution).
Hm... I can try varying both src and dst on my pktgen test.
Thanks
next prev parent reply other threads:[~2009-11-23 10:37 UTC|newest]
Thread overview: 68+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-11-23 6:46 [PATCH] irq: Add node_affinity CPU masks for smarter irqbalance hints Peter P Waskiewicz Jr
2009-11-23 7:32 ` Yong Zhang
2009-11-23 7:32 ` Yong Zhang
2009-11-23 9:36 ` Peter P Waskiewicz Jr
2009-11-23 10:21 ` ixgbe question Eric Dumazet
2009-11-23 10:30 ` Badalian Vyacheslav
2009-11-23 10:34 ` Waskiewicz Jr, Peter P
2009-11-23 10:37 ` Eric Dumazet [this message]
2009-11-23 14:05 ` Eric Dumazet
2009-11-23 21:26 ` David Miller
2009-11-23 14:10 ` Jesper Dangaard Brouer
2009-11-23 14:38 ` Eric Dumazet
2009-11-23 18:30 ` robert
2009-11-23 16:59 ` Eric Dumazet
2009-11-23 20:54 ` robert
2009-11-23 21:28 ` David Miller
2009-11-23 22:14 ` Robert Olsson
2009-11-23 23:28 ` Waskiewicz Jr, Peter P
2009-11-23 23:44 ` David Miller
2009-11-24 7:46 ` Eric Dumazet
2009-11-24 8:46 ` Badalian Vyacheslav
2009-11-24 9:07 ` Peter P Waskiewicz Jr
2009-11-24 9:55 ` Eric Dumazet
2009-11-24 10:06 ` Peter P Waskiewicz Jr
2009-11-24 11:37 ` [PATCH net-next-2.6] ixgbe: Fix TX stats accounting Eric Dumazet
2009-11-24 13:23 ` Eric Dumazet
2009-11-25 7:38 ` Jeff Kirsher
2009-11-25 9:31 ` Eric Dumazet
2009-11-25 9:38 ` Jeff Kirsher
2009-11-24 13:14 ` ixgbe question John Fastabend
2009-11-29 8:18 ` David Miller
2009-11-30 13:02 ` Eric Dumazet
2009-11-30 20:20 ` John Fastabend
2009-11-26 14:10 ` Badalian Vyacheslav
2009-11-23 17:05 ` [PATCH] irq: Add node_affinity CPU masks for smarter irqbalance hints Peter Zijlstra
2009-11-23 23:32 ` Waskiewicz Jr, Peter P
2009-11-24 8:38 ` Peter Zijlstra
2009-11-24 8:59 ` Peter P Waskiewicz Jr
2009-11-24 9:08 ` Peter Zijlstra
2009-11-24 9:15 ` Peter P Waskiewicz Jr
2009-11-24 14:43 ` Arjan van de Ven
2009-11-24 9:15 ` Peter Zijlstra
2009-11-24 10:07 ` Thomas Gleixner
2009-11-24 17:55 ` Peter P Waskiewicz Jr
2009-11-25 11:18 ` Peter Zijlstra
2009-11-24 6:07 ` Arjan van de Ven
2009-11-24 8:39 ` Peter Zijlstra
2009-11-24 14:42 ` Arjan van de Ven
2009-11-24 17:39 ` David Miller
2009-11-24 17:56 ` Peter P Waskiewicz Jr
2009-11-24 18:26 ` Eric Dumazet
2009-11-24 18:33 ` Peter P Waskiewicz Jr
2009-11-24 19:01 ` Eric Dumazet
2009-11-24 19:53 ` Peter P Waskiewicz Jr
2009-11-24 18:54 ` David Miller
2009-11-24 18:58 ` Eric Dumazet
2009-11-24 20:35 ` Andi Kleen
2009-11-24 20:46 ` Eric Dumazet
2009-11-25 10:30 ` Eric Dumazet
2009-11-25 10:37 ` Andi Kleen
2009-11-25 11:35 ` Eric Dumazet
2009-11-25 11:50 ` Andi Kleen
2009-11-26 11:43 ` Eric Dumazet
2009-11-24 5:17 ` Yong Zhang
2009-11-24 5:17 ` Yong Zhang
2009-11-24 8:39 ` Peter P Waskiewicz Jr
-- strict thread matches above, loose matches on Subject: below --
2008-03-10 21:27 Ixgbe question Ben Greear
2008-03-11 1:01 ` Brandeburg, Jesse
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4B0A65E0.7060403@gmail.com \
--to=eric.dumazet@gmail.com \
--cc=netdev@vger.kernel.org \
--cc=peter.p.waskiewicz.jr@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.