Networking latency - what to expect?

[David Mohr <damailings@mcbf.net>]

Hi,

we were investigating some performance issue that an application on our 
kvm VMs had and noticed that network latency is much worse in a VM 
compared to actual hardware.

Initially we were using virtio network cards, but not vhost-net. After 
enabling vhost the performance improved quite a bit, but is still only 
at around 50% of what the actual hardware can do.

The question is, what can we expect? CPU utilization was never higher 
than 50% kvm and 20% vhost. Other details are below, please let me know 
if you can recommend other tests or important details that I could 
provide.

I ran the following tests using netperf -t UDP_RR (all results are 
transactions/sec):
* host->host            19k
* vm->its host          17k
* vm->vm (same host)    22k
* vm->vm (diff. hosts)   7k

Host:     Debian squeeze using a 3.5.2 kernel
KVM:      1.1.2 (bpo)
Host CPU: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
VM:       Debian squeeze using a 3.2.20 kernel
Network:  e1000 & gigabit switch
Command-line:
-cpu host -name testvm -m 12288 -smp 16 -pidfile 
/var/run/ganeti/kvm-hypervisor/pid/testvm -balloon virtio -daemonize 
-monitor 
unix:/var/run/ganeti/kvm-hypervisor/ctrl/testvm.monitor,server,nowait 
-serial 
unix:/var/run/ganeti/kvm-hypervisor/ctrl/testvm.serial,server,nowait 
-usbdevice tablet -vnc 127.0.0.1:5102 -netdev 
type=tap,id=netdev0,fd=8,vhost=on -device 
virtio-net-pci,mac=52:54:00:00:01:e6,netdev=netdev0 -netdev 
type=tap,id=netdev1,fd=9,vhost=on -device 
virtio-net-pci,mac=52:54:00:00:01:e7,netdev=netdev1 -qmp 
unix:/var/run/ganeti/kvm-hypervisor/ctrl/testvm.qmp,server,nowait -S

Thanks,
~David