netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
To: netdev@vger.kernel.org, jasowang@redhat.com
Cc: davem@davemloft.net, mst@redhat.com
Subject: Regression in throughput between kvm guests over virtual bridge
Date: Tue, 12 Sep 2017 13:56:15 -0400	[thread overview]
Message-ID: <4c7e2924-b10f-0e97-c388-c8809ecfdeeb@linux.vnet.ibm.com> (raw)

We are seeing a regression for a subset of workloads across KVM guests
over a virtual bridge between host kernel 4.12 and 4.13.  Bisecting
points to c67df11f "vhost_net: try batch dequing from skb array"

In the regressed environment, we are running 4 kvm guests, 2 running as
uperf servers and 2 running as uperf clients, all on a single host.
They are connected via a virtual bridge.  The uperf client profile looks
like:

<?xml version="1.0"?>
<profile name="TCP_STREAM">
  <group nprocs="1">
    <transaction iterations="1">
      <flowop type="connect" options="remotehost=192.168.122.103
protocol=tcp"/>
    </transaction>
    <transaction duration="300">
      <flowop type="write" options="count=16 size=30000"/>
    </transaction>
    <transaction iterations="1">
      <flowop type="disconnect"/>
    </transaction>
  </group>
</profile>

So, 1 tcp streaming instance per client.  When upgrading the host kernel
from 4.12->4.13, we see about a 30% drop in throughput for this
scenario.  After the bisect, I further verified that reverting c67df11f
on 4.13 "fixes" the throughput for this scenario.

On the other hand, if we increase the load by upping the number of
streaming instances to 50 (nprocs="50") or even 10, we see instead a
~10% increase in throughput when upgrading host from 4.12->4.13.

So it may be the issue is specific to "light load" scenarios.  I would
expect some overhead for the batching, but 30% seems significant...  Any
thoughts on what might be happening here?

             reply	other threads:[~2017-09-12 17:56 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-09-12 17:56 Matthew Rosato [this message]
2017-09-13  1:16 ` Regression in throughput between kvm guests over virtual bridge Jason Wang
2017-09-13  8:13   ` Jason Wang
2017-09-13 16:59     ` Matthew Rosato
2017-09-14  4:21       ` Jason Wang
2017-09-15  3:36         ` Matthew Rosato
2017-09-15  8:55           ` Jason Wang
2017-09-15 19:19             ` Matthew Rosato
2017-09-18  3:13               ` Jason Wang
2017-09-18  4:14                 ` [PATCH] vhost_net: conditionally enable tx polling kbuild test robot
2017-09-18  7:36                 ` Regression in throughput between kvm guests over virtual bridge Jason Wang
2017-09-18 18:11                   ` Matthew Rosato
2017-09-20  6:27                     ` Jason Wang
2017-09-20 19:38                       ` Matthew Rosato
2017-09-22  4:03                         ` Jason Wang
2017-09-25 20:18                           ` Matthew Rosato
2017-10-05 20:07                             ` Matthew Rosato
2017-10-11  2:41                               ` Jason Wang
2017-10-12 18:31                               ` Wei Xu
2017-10-18 20:17                                 ` Matthew Rosato
2017-10-23  2:06                                   ` Jason Wang
2017-10-23  2:13                                     ` Michael S. Tsirkin
2017-10-25 20:21                                     ` Matthew Rosato
2017-10-26  9:44                                       ` Wei Xu
2017-10-26 17:53                                         ` Matthew Rosato
2017-10-31  7:07                                           ` Wei Xu
2017-10-31  7:00                                             ` Jason Wang
2017-11-03  4:30                                             ` Matthew Rosato
2017-11-04 23:35                                               ` Wei Xu
2017-11-08  1:02                                                 ` Matthew Rosato
2017-11-11 20:59                                                   ` Matthew Rosato
2017-11-12 18:34                                                     ` Wei Xu
2017-11-14 20:11                                                       ` Matthew Rosato
2017-11-20 19:25                                                         ` Matthew Rosato
2017-11-27 16:21                                                           ` Wei Xu
2017-11-28  1:36                                                             ` Jason Wang
2017-11-28  2:44                                                               ` Matthew Rosato
2017-11-28 18:00                                                                 ` Wei Xu
2017-11-28  3:51                                                               ` Wei Xu
2017-11-12 15:40                                                   ` Wei Xu
2017-10-23 13:57                                   ` Wei Xu
2017-10-25 20:31                                     ` Matthew Rosato

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=4c7e2924-b10f-0e97-c388-c8809ecfdeeb@linux.vnet.ibm.com \
    --to=mjrosato@linux.vnet.ibm.com \
    --cc=davem@davemloft.net \
    --cc=jasowang@redhat.com \
    --cc=mst@redhat.com \
    --cc=netdev@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).