netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Jesper Dangaard Brouer <brouer@redhat.com>
To: Jesper Dangaard Brouer <brouer@redhat.com>, netdev@vger.kernel.org
Cc: Alexander Duyck <alexander.h.duyck@intel.com>,
	Jeff Kirsher <jeffrey.t.kirsher@intel.com>,
	Daniel Borkmann <dborkman@redhat.com>,
	Florian Westphal <fw@strlen.de>,
	"David S. Miller" <davem@davemloft.net>,
	Stephen Hemminger <shemminger@vyatta.com>,
	"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
	Robert Olsson <robert@herjulf.se>,
	Ben Greear <greearb@candelatech.com>,
	John Fastabend <john.r.fastabend@intel.com>,
	danieltt@kth.se, zhouzhouyi@gmail.com
Subject: [net-next PATCH 0/5] Optimizing "pktgen" for single CPU performance
Date: Wed, 14 May 2014 16:17:38 +0200	[thread overview]
Message-ID: <20140514141545.20309.28343.stgit@dragon> (raw)

I'm on a quest to push the packet per sec (pps) limits of our network
stack, with a special focus on single CPU performance.

My first action is to measure and identify bottlenecks in the transmit
path. For achieving this goal, I need a fast in-kernel packet
generator, like "pktgen". It turned out that "pktgen" were too slow.

Thus, this series focus on optimizing "pktgen" for single CPU performance.

Overview 1xCPU performance Packet Per Sec (pps) stats:
 * baseline: 3,930,068 pps
 * patch2:   5,362,722 pps -- TXSZ=1024
 * patch3:   5,608,781 pps --> 178.29ns per pkt
 * patch4:   5,857,065 pps --> 170.73ns ( -7.56ns)
 * patch5:   6,346,500 pps --> 157.56ns (-13.17ns)
 * No-lock:  6,642,948 pps --> 150.53ns ( -7.03ns)

The last result "No-lock" removes the HARD_TX_{UN}LOCK, and is not
applicable to upstream.  It removes two "LOCK" instructions (cost 8ns
each), thus I were expecting to see an improvement of 16ns, but we
only see 7ns.  This leads me to believe, that we have reached the
ixgbe driver limit, single queue.

Setup according to blogpost:
 http://netoptimizer.blogspot.dk/2014/04/basic-tuning-for-network-overload.html

Hardware:
 System: CPU E5-2630
 NIC: Intel ixgbe/82599 chip

Testing done with net-next git tree on top of
 commit 79e0f1c9f (ipv6: Need to sock_put on csum error).

Pktgen script exercising race condition:
 https://github.com/netoptimizer/network-testing/blob/master/pktgen/unit_test01_race_add_rem_device_loop.sh

---

Jesper Dangaard Brouer (5):
      pktgen: RCU'ify "if_list" to remove lock in next_to_run()
      pktgen: avoid expensive set_current_state() call in loop
      pktgen: avoid atomic_inc per packet in xmit loop
      ixgbe: increase default TX ring buffer to 1024
      ixgbe: trivial fixes while reading code


 drivers/net/ethernet/intel/ixgbe/ixgbe.h      |    2 
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c |    2 
 net/core/pktgen.c                             |  115 +++++++++++++------------
 3 files changed, 61 insertions(+), 58 deletions(-)

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

             reply	other threads:[~2014-05-14 14:18 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-05-14 14:17 Jesper Dangaard Brouer [this message]
2014-05-14 14:17 ` [net-next PATCH 1/5] ixgbe: trivial fixes while reading code Jesper Dangaard Brouer
2014-05-14 14:17 ` [net-next PATCH 2/5] ixgbe: increase default TX ring buffer to 1024 Jesper Dangaard Brouer
2014-05-14 14:28   ` David Laight
2014-05-14 19:25     ` Jesper Dangaard Brouer
2014-05-14 16:28   ` Alexander Duyck
2014-05-14 17:49     ` David Miller
2014-05-14 19:09       ` Jesper Dangaard Brouer
2014-05-14 19:54         ` David Miller
2014-05-15  9:16         ` David Laight
2014-05-29 15:29         ` Jesper Dangaard Brouer
2014-05-14 14:17 ` [net-next PATCH 3/5] pktgen: avoid atomic_inc per packet in xmit loop Jesper Dangaard Brouer
2014-05-14 14:35   ` Eric Dumazet
2014-05-14 15:13     ` Jesper Dangaard Brouer
2014-05-14 15:35       ` Eric Dumazet
2014-05-14 14:17 ` [net-next PATCH 4/5] pktgen: avoid expensive set_current_state() call in loop Jesper Dangaard Brouer
2014-05-14 14:18 ` [net-next PATCH 5/5] pktgen: RCU'ify "if_list" to remove lock in next_to_run() Jesper Dangaard Brouer
2014-06-26 11:16 ` [net-next PATCH V2 0/3] Optimizing pktgen for single CPU performance Jesper Dangaard Brouer
2014-06-26 11:16   ` [net-next PATCH V2 1/3] pktgen: document tuning for max NIC performance Jesper Dangaard Brouer
2014-06-26 11:16   ` [net-next PATCH V2 2/3] pktgen: avoid expensive set_current_state() call in loop Jesper Dangaard Brouer
2014-06-26 11:16   ` [net-next PATCH V2 3/3] pktgen: RCU-ify "if_list" to remove lock in next_to_run() Jesper Dangaard Brouer
2014-07-01 22:51   ` [net-next PATCH V2 0/3] Optimizing pktgen for single CPU performance David Miller

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20140514141545.20309.28343.stgit@dragon \
    --to=brouer@redhat.com \
    --cc=alexander.h.duyck@intel.com \
    --cc=danieltt@kth.se \
    --cc=davem@davemloft.net \
    --cc=dborkman@redhat.com \
    --cc=fw@strlen.de \
    --cc=greearb@candelatech.com \
    --cc=jeffrey.t.kirsher@intel.com \
    --cc=john.r.fastabend@intel.com \
    --cc=netdev@vger.kernel.org \
    --cc=paulmck@linux.vnet.ibm.com \
    --cc=robert@herjulf.se \
    --cc=shemminger@vyatta.com \
    --cc=zhouzhouyi@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).