From: "Michael S. Tsirkin" <mst@redhat.com>
To: Steve Dobbelstein <steved@us.ibm.com>
Cc: kvm@vger.kernel.org
Subject: Re: Network performance with small packets
Date: Wed, 2 Feb 2011 20:38:47 +0200 [thread overview]
Message-ID: <20110202183847.GA14829@redhat.com> (raw)
In-Reply-To: <OFD293DCD2.7F0260F0-ON86257823.0061DC39-86257823.00743BB3@us.ibm.com>
On Tue, Jan 25, 2011 at 03:09:34PM -0600, Steve Dobbelstein wrote:
>
> I am working on a KVM network performance issue found in our lab running
> the DayTrader benchmark. The benchmark throughput takes a significant hit
> when running the application server in a KVM guest verses on bare metal.
> We have dug into the problem and found that DayTrader's use of small
> packets exposes KVM's overhead of handling network packets. I have been
> able to reproduce the performance hit with a simpler setup using the
> netperf benchmark with the TCP_RR test and the request and response sizes
> set to 256 bytes. I run the benchmark between two physical systems, each
> using a 1GB link. In order to get the maximum throughput for the system I
> have to run 100 instances of netperf. When I run the netserver processes
> in a guest, I see a maximum throughput that is 51% of what I get if I run
> the netserver processes directly on the host. The CPU utilization in the
> guest is only 85% at maximum throughput, whereas it is 100% on bare metal.
You are stressing the scheduler pretty hard with this test :)
Is your real benchmark also using a huge number of threads?
If it's not, you might be seeing a different issue.
IOW, the netperf degradation might not be network-related at all,
but have to do with speed of context switch in guest.
Thoughts?
> The KVM host has 16 CPUs. The KVM guest is configured with 2 VCPUs. When
> I run netperf on the host I boot the host with maxcpus=2 on the kernel
> command line. The host is running the current KVM upstream kernel along
> with the current upstream qemu. Here is the qemu command used to launch
> the guest:
> /build/qemu-kvm/x86_64-softmmu/qemu-system-x86_64 -name glasgow-RH60 -m 32768 -drive file=/build/guest-data/glasgow-RH60.img,if=virtio,index=0,boot=on
> -drive file=/dev/virt/WAS,if=virtio,index=1 -net nic,model=virtio,vlan=3,macaddr=00:1A:64:E5:00:63,netdev=nic0 -netdev tap,id=nic0,vhost=on -smp 2
> -vnc :1 -monitor telnet::4499,server,nowait -serial telnet::8899,server,nowait --mem-path /libhugetlbfs -daemonize
>
> We have tried various proposed fixes, each with varying amounts of success.
> One such fix was to add code to the vhost thread such that when it found
> the work queue empty it wouldn't just exit the thread but rather would
> delay for 50 microseconds and then recheck the queue. If there was work on
> the queue it would loop back and process it, else it would exit the thread.
> The change got us a 13% improvement in the DayTrader throughput.
>
> Running the same netperf configuration on the same hardware but using a
> different hypervisor gets us significantly better throughput numbers. The
> guest on that hypervisor runs at 100% CPU utilization. The various fixes
> we have tried have not gotten us close to the throughput seen on the other
> hypervisor. I'm looking for ideas/input from the KVM experts on how to
> make KVM perform better when handling small packets.
>
> Thanks,
> Steve
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2011-02-02 18:39 UTC|newest]
Thread overview: 88+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-01-25 21:09 Network performance with small packets Steve Dobbelstein
2011-01-26 15:17 ` Michael S. Tsirkin
2011-01-27 18:44 ` Shirley Ma
2011-01-27 19:00 ` Michael S. Tsirkin
2011-01-27 19:09 ` Shirley Ma
2011-01-27 19:31 ` Michael S. Tsirkin
2011-01-27 19:45 ` Shirley Ma
2011-01-27 20:05 ` Michael S. Tsirkin
2011-01-27 20:15 ` Shirley Ma
2011-01-28 18:29 ` Steve Dobbelstein
2011-01-28 22:51 ` Steve Dobbelstein
2011-02-01 15:52 ` [PATCHv2 dontapply] vhost-net tx tuning Michael S. Tsirkin
2011-02-01 23:07 ` Sridhar Samudrala
2011-02-01 23:27 ` Shirley Ma
2011-02-02 4:36 ` Michael S. Tsirkin
2011-01-27 21:02 ` Network performance with small packets David Miller
2011-01-27 21:30 ` Shirley Ma
2011-01-28 12:16 ` Michael S. Tsirkin
2011-02-01 0:24 ` Steve Dobbelstein
2011-02-01 1:30 ` Sridhar Samudrala
2011-02-01 5:56 ` Michael S. Tsirkin
2011-02-01 21:09 ` Shirley Ma
2011-02-01 21:24 ` Michael S. Tsirkin
2011-02-01 21:32 ` Shirley Ma
2011-02-01 21:42 ` Michael S. Tsirkin
2011-02-01 21:53 ` Shirley Ma
2011-02-01 21:56 ` Michael S. Tsirkin
2011-02-01 22:59 ` Shirley Ma
2011-02-02 4:40 ` Michael S. Tsirkin
2011-02-02 6:05 ` Shirley Ma
2011-02-02 6:19 ` Shirley Ma
2011-02-02 6:29 ` Michael S. Tsirkin
2011-02-02 7:14 ` Shirley Ma
2011-02-02 7:33 ` Shirley Ma
2011-02-02 10:49 ` Michael S. Tsirkin
2011-02-02 15:42 ` Shirley Ma
2011-02-02 15:48 ` Michael S. Tsirkin
2011-02-02 17:12 ` Shirley Ma
2011-02-02 18:20 ` Michael S. Tsirkin
2011-02-02 18:26 ` Shirley Ma
2011-02-02 10:48 ` Michael S. Tsirkin
2011-02-02 6:34 ` Krishna Kumar2
2011-02-02 7:03 ` Shirley Ma
2011-02-02 7:37 ` Krishna Kumar2
2011-02-02 10:48 ` Michael S. Tsirkin
2011-02-02 15:39 ` Shirley Ma
2011-02-02 15:47 ` Michael S. Tsirkin
2011-02-02 17:10 ` Shirley Ma
2011-02-02 17:32 ` Michael S. Tsirkin
2011-02-02 18:11 ` Shirley Ma
2011-02-02 18:27 ` Michael S. Tsirkin
2011-02-02 19:29 ` Shirley Ma
2011-02-02 20:17 ` Michael S. Tsirkin
2011-02-02 21:03 ` Shirley Ma
2011-02-02 21:20 ` Michael S. Tsirkin
2011-02-02 21:41 ` Shirley Ma
2011-02-03 5:59 ` Michael S. Tsirkin
2011-02-03 6:09 ` Shirley Ma
2011-02-03 6:16 ` Michael S. Tsirkin
2011-02-03 5:05 ` Shirley Ma
2011-02-03 6:13 ` Michael S. Tsirkin
2011-02-03 15:58 ` Shirley Ma
2011-02-03 16:20 ` Michael S. Tsirkin
2011-02-03 17:18 ` Shirley Ma
2011-02-01 5:54 ` Michael S. Tsirkin
2011-02-01 17:23 ` Michael S. Tsirkin
[not found] ` <1296590943.26937.797.camel@localhost.localdomain>
[not found] ` <20110201201715.GA30050@redhat.com>
2011-02-01 20:25 ` Shirley Ma
2011-02-01 21:21 ` Michael S. Tsirkin
2011-02-01 21:28 ` Shirley Ma
2011-02-01 21:41 ` Michael S. Tsirkin
2011-02-02 4:39 ` Krishna Kumar2
2011-02-02 4:42 ` Michael S. Tsirkin
2011-02-09 0:37 ` Rusty Russell
2011-02-09 0:53 ` Michael S. Tsirkin
2011-02-09 1:39 ` Rusty Russell
2011-02-09 1:55 ` Michael S. Tsirkin
2011-02-09 7:43 ` Stefan Hajnoczi
2011-03-08 21:57 ` Shirley Ma
2011-03-09 2:21 ` Andrew Theurer
2011-03-09 15:42 ` Shirley Ma
2011-03-10 1:49 ` Rusty Russell
2011-04-12 20:01 ` Michael S. Tsirkin
2011-04-14 11:28 ` Rusty Russell
2011-04-14 12:40 ` Michael S. Tsirkin
2011-04-14 16:03 ` Michael S. Tsirkin
2011-04-19 0:33 ` Rusty Russell
2011-02-02 18:38 ` Michael S. Tsirkin [this message]
2011-02-02 19:15 ` Steve Dobbelstein
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20110202183847.GA14829@redhat.com \
--to=mst@redhat.com \
--cc=kvm@vger.kernel.org \
--cc=steved@us.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).