From: Flavio Leitner <fbl@redhat.com>
To: Tom Herbert <therbert@google.com>
Cc: Linux Netdev List <netdev@vger.kernel.org>,
Jacob Keller <jacob.e.keller@intel.com>,
Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Subject: Re: RSS is not efficient when forwarding (ixgbe)
Date: Mon, 7 Jul 2014 07:55:23 -0300 [thread overview]
Message-ID: <20140707105523.GC2343@t520.home> (raw)
In-Reply-To: <CA+mtBx-dY2eoDzL-1K8F8Br224H+c4fzVtVPu9JmnjLdQKuAOg@mail.gmail.com>
On Sat, Jul 05, 2014 at 09:09:34AM -0700, Tom Herbert wrote:
> On Fri, Jul 4, 2014 at 12:36 PM, Flavio Leitner <fbl@redhat.com> wrote:
> > On Thu, Jul 03, 2014 at 07:44:08PM -0300, Flavio Leitner wrote:
> >>
> >> Hi,
> >>
> >> I have a simple router setup which forwards traffic from
> >> one ixgbe 82599ES to another ixgbe of the same model.
> >>
> >> kernel: 3.16.0-rc2-00262-ga921e2a
> >>
> >> p2p1: 192.168.155.1/24 is the gateway of the LAN
> >> p2p2: 192.168.156.1/24 is the gateway of the other LAN
> >>
> >> While the ARP is resolving, I can see the packets being spread
> >> among all the 8 queues (8 online CPUs) available and that is fine.
> >>
> >> However, as soon as the TCP traffic starts, all streams are
> >> merged to rx-queue-0 which overwhelms one single CPU, so the
> >> total throughput is about 4Gbits/sec.
> >>
> >> I can see the driver sending different skb->hash for each stream,
> >> so it can't be the NIC.
> >>
> >> Also, if I run a local http on the router, the skb->hash pattern
> >> doesn't change, but the workload is spread among all CPUs.
> >>
> >> debug output while reproducing all the streams on rx-queue-0:
> >> [...]
> >> [11685.885093] ixgbe_rx_skb:1713 skb(ffff880222a77200) hash: 0xC2AF4A27
> >> [11685.891454] ixgbe_rx_skb:1713 skb(ffff880222a77200) hash: 0x8C5B749D
> >> [11685.897820] ixgbe_rx_skb:1713 skb(ffff880222a77200) hash: 0xA33BA6D5
> >> [11690.845032] net_ratelimit: 3276406 callbacks suppressed
> >> [...]
> >
> > The command 'ehtool -S p2p1 | grep rx_queue' shows only rx queue #0
> > receiving packets. Also, /proc/interrupts shows only p2p1-TxRx-0
> > generating interrupts.
> >
> > Nothing changes when I start irqbalance in the middle of the test.
> > However, if I create new streams, they are correctly distributed
> > among the NIC queues/irqs/CPUs.
> >
> Could this be interaction with flow director? Is the application
> running on just one CPU?
It's a router forwarding traffic from one interface to another, so
I guess it's only the kernel. BTW, no firewall.
Flow Director needs to be enabled and I am using defaults.
> > I have tried the same setup and test with another card (bnx2, 1GbE)
> > and all queues got traffic by default though they were all assigned
> > to CPU#0 (no irqbalance), so this is expected and makes me think
> > this is specific to the ixgbe driver.
> >
> > Any ideas?
> >
> > Thanks,
> > fbl
> > --
> > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
next prev parent reply other threads:[~2014-07-07 10:55 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-07-03 22:44 RSS is not efficient when forwarding Flavio Leitner
2014-07-04 19:36 ` RSS is not efficient when forwarding (ixgbe) Flavio Leitner
2014-07-05 16:09 ` Tom Herbert
2014-07-07 10:55 ` Flavio Leitner [this message]
2014-07-07 16:33 ` Skidmore, Donald C
2014-07-08 0:28 ` Flavio Leitner
2014-07-08 17:09 ` Carlos Carvalho
2014-07-08 17:21 ` Flavio Leitner
2014-07-08 17:32 ` Carlos Carvalho
2014-07-09 5:22 ` Flavio Leitner
2014-07-10 0:08 ` Carlos Carvalho
2014-07-10 1:14 ` Flavio Leitner
2014-07-11 0:11 ` Carlos Carvalho
2014-07-11 14:11 ` Skidmore, Donald C
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140707105523.GC2343@t520.home \
--to=fbl@redhat.com \
--cc=jacob.e.keller@intel.com \
--cc=jeffrey.t.kirsher@intel.com \
--cc=netdev@vger.kernel.org \
--cc=therbert@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).