Multiqueue pktgen and ingress path (Was: [PATCH v5 2/2] pktgen: introduce xmit_mode '<start_xmit|netif_receive>')

[Jesper Dangaard Brouer <brouer@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 2772 bytes --]

On Fri, 8 May 2015 17:39:00 +0200
Jesper Dangaard Brouer <brouer@redhat.com> wrote:

> On Thu, 07 May 2015 19:11:58 +0200 Daniel Borkmann <daniel@iogearbox.net> wrote:
> 
> > On 05/07/2015 06:28 PM, Alexei Starovoitov wrote:  
> > > On 5/7/15 7:35 AM, Jesper Dangaard Brouer wrote:  
> > >> From: Alexei Starovoitov <ast@plumgrid.com>
> > >>  
> [...snip...]
> 
> > > btw, I've started to work on a patch on top of this one that allows
> > > multiple pktgen threads to submit into the same netdev.
> > > I've used it to stress test removal of spin_lock in ingress qdisc.
> > > The idea is to add another 'name' parameter to command:
> > > 'add_device name dev'
> > > 'name' will be used to identify this pktgen thread in /proc
> > > and 'dev' used as target net_device.
> > > I think it will be useful for start_xmit testing as well.
> > > I wonder why it wasn't done earlier? The queue configuration is
> > > already supported.  
> > 
> > You mean other than below commit (iow independant of queue mapping)?
> > 
> > commit e6fce5b916cd7f7f79b2b3e53ba74bbfc1d7cf8b
> > Author: Robert Olsson <robert.olsson@its.uu.se>
> > Date:   Thu Aug 7 02:23:01 2008 -0700
> > 
> >      pktgen: multiqueue etc.  
> 
> For completeness and others reading this threads...
> 
> Pktgen multiqueue is already supported via mentioned commit, which adds
> the device naming scheme: "add_device dev@number"
> 
> And yes, the documentation does not seem to mention this.  I've been
> using it for years now... My scripts[1] take param "-t" for "threads".
> [1] https://github.com/netoptimizer/network-testing/tree/master/pktgen
> 
> I've added a more plain version of a script, based on yours, below my
> signature.

Now attached.

> The funny thing now is that scaling does not "happen" as we stall on:
>    atomic_long_inc(&skb->dev->rx_dropped);

More interesting observations with the mentioned script (now attached).

On my system the scaling stopped a 24Mpps, when I increased the number
of threads the collective scaling was stuck at 24Mpps.

Then I simply removed/compiled-out the:
 atomic_long_inc(&skb->dev->rx_dropped);

And after that change, the scaling is basically infinite/perfect.

Single thread performance increased from 24.7Mpps to 31.1Mpps, which
corresponds perfectly with the cost of an atomic operation on this HW
(8.25ns).

Diff to before:
 * (1/24700988*10^9)-(1/31170819*10^9) = 8.40292328196 ns

When increasing the threads now, they all basically run at 31Mpps.
Tried it upto 12 threads.


I'm quite puzzled why a single atomic op could "freeze" my system from
scaling beyond 24Mpps.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

[-- Attachment #2: pktgen_multiqueue01_xmit_mode.sh --]
[-- Type: application/x-shellscript, Size: 1217 bytes --]