High latency with VirtIO NIC

[Xupeng Yun <recordus@gmail.com>]

Hi,

I deployed KVM on my Gentoo servers for performance testing, but I am now having high 
latency problems with the VirtIO NIC:

diggle7 is the KVM host, diggle6 is another physical host in the same physical network, and
app211 is a KVM guest running on diggle7.

------------------------------- ping statistics -------------------------------
diggle7 ~ # ping -c 5 app211
PING app211 (192.168.1.211) 56(84) bytes of data.
64 bytes from app211 (192.168.1.211): icmp_seq=1 ttl=64 time=0.454 ms
64 bytes from app211 (192.168.1.211): icmp_seq=2 ttl=64 time=0.097 ms
64 bytes from app211 (192.168.1.211): icmp_seq=3 ttl=64 time=0.248 ms
64 bytes from app211 (192.168.1.211): icmp_seq=4 ttl=64 time=0.355 ms
64 bytes from app211 (192.168.1.211): icmp_seq=5 ttl=64 time=0.494 ms

--- app211 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3998ms
rtt min/avg/max/mdev = 0.097/0.329/0.494/0.145 ms

diggle7 ~ # ping -c 5 diggle6
PING diggle6 (192.168.1.44) 56(84) bytes of data.
64 bytes from diggle6 (192.168.1.44): icmp_seq=1 ttl=64 time=0.101 ms
64 bytes from diggle6 (192.168.1.44): icmp_seq=2 ttl=64 time=0.076 ms
64 bytes from diggle6 (192.168.1.44): icmp_seq=3 ttl=64 time=0.085 ms
64 bytes from diggle6 (192.168.1.44): icmp_seq=4 ttl=64 time=0.130 ms
64 bytes from diggle6 (192.168.1.44): icmp_seq=5 ttl=64 time=0.083 ms

--- diggle6 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3998ms
rtt min/avg/max/mdev = 0.076/0.095/0.130/0.019 ms
------------------------------- ping statistics -------------------------------

as above, the latency between Host and KVM guest(0.3+ms on average) is much higher
than the latency between the two physical hosts(~0.1ms on average), this might not be
acceptable for latency sensitive applications, for example memcached. 

is the latency normal? or is there something wrong with my configuration? here is detailed
information about my hardware and software environment:

Host hardware: DELL R710 + Intel Xeon E5620 @2.40GHz + 96GB memory + BCM5709 GB Ethernet
Host software: Gentoo amd64 + 2.6.36-gentoo-r5 + qemu-kvm-0.13.0

KVM related kernel configs for the host (diggle7):
diggle7 ~ # zgrep -E "(VIRTIO|KVM)" /proc/config.gz 
CONFIG_VIRTIO_BLK=m
CONFIG_VIRTIO_NET=m
# CONFIG_VIRTIO_CONSOLE is not set
CONFIG_HW_RANDOM_VIRTIO=m
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
# CONFIG_KVM_AMD is not set
CONFIG_VIRTIO=m
CONFIG_VIRTIO_RING=m
CONFIG_VIRTIO_PCI=m
CONFIG_VIRTIO_BALLOON=m

Kernel modules loaded on the host:
diggle7 ~ # lsmod
Module Size Used by
virtio_balloon 4098 0 
tun 12809 2 
bridge  63340 0 
stp  1871 1 bridge
llc  4595 2 bridge,stp
drbd 181114 7 
lru_cache  6273 1 drbd
virtio_net  10812 0 
virtio 3978 2 virtio_balloon,virtio_net
virtio_ring  4692 2 virtio_balloon,virtio_net
xt_state 1218 1 
iptable_filter 1626 1 
iptable_nat  3806 0 
nf_nat  15835 1 iptable_nat
nf_conntrack_ipv4 11106 4 iptable_nat,nf_nat
nf_defrag_ipv4 1361 1 nf_conntrack_ipv4
iptable_mangle 1591 0 
ipmi_devintf 7380 0 
ipmi_si 37990 0 
ipmi_msghandler 31256 2 ipmi_devintf,ipmi_si
kvm_intel 42059 18 
kvm  248729 1 kvm_intel
fuse  58921 2 

KVM related kernel configs for the guest (app211):
app211 ~ # zgrep -E "(VIRTIO|KVM)" /proc/config.gz
CONFIG_KVM_CLOCK=y
CONFIG_KVM_GUEST=y
CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_HAVE_KVM=y
# CONFIG_KVM is not set
CONFIG_VIRTIO=y
CONFIG_VIRTIO_RING=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BALLOON=y

Command line for starting the KVM guest:
diggle7 ~ # pgrep -fl qemu
18154 /usr/bin/qemu-system-x86_64 --enable-kvm -drive file=/vm/app211/app211.qcow2,if=virtio,boot=on,cache=writeback -cdrom /mfs/sysadmin/finnix.iso -boot c -vnc diggle7:211 -net nic,model=virtio,macaddr=52:54:C0:A8:01:D3 -net tap,ifname=app211,script=no -m 30720 -cpu host -smp 16 -daemonize

any help will be appreciated.

Thanks,
Xupeng