High latency with VirtIO NIC

High latency with VirtIO NIC

[Xupeng Yun]

Hi,

I deployed KVM on my Gentoo servers for performance testing, but I am now having high 
latency problems with the VirtIO NIC:

diggle7 is the KVM host, diggle6 is another physical host in the same physical network, and
app211 is a KVM guest running on diggle7.

------------------------------- ping statistics -------------------------------
diggle7 ~ # ping -c 5 app211
PING app211 (192.168.1.211) 56(84) bytes of data.
64 bytes from app211 (192.168.1.211): icmp_seq=1 ttl=64 time=0.454 ms
64 bytes from app211 (192.168.1.211): icmp_seq=2 ttl=64 time=0.097 ms
64 bytes from app211 (192.168.1.211): icmp_seq=3 ttl=64 time=0.248 ms
64 bytes from app211 (192.168.1.211): icmp_seq=4 ttl=64 time=0.355 ms
64 bytes from app211 (192.168.1.211): icmp_seq=5 ttl=64 time=0.494 ms

--- app211 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3998ms
rtt min/avg/max/mdev = 0.097/0.329/0.494/0.145 ms

diggle7 ~ # ping -c 5 diggle6
PING diggle6 (192.168.1.44) 56(84) bytes of data.
64 bytes from diggle6 (192.168.1.44): icmp_seq=1 ttl=64 time=0.101 ms
64 bytes from diggle6 (192.168.1.44): icmp_seq=2 ttl=64 time=0.076 ms
64 bytes from diggle6 (192.168.1.44): icmp_seq=3 ttl=64 time=0.085 ms
64 bytes from diggle6 (192.168.1.44): icmp_seq=4 ttl=64 time=0.130 ms
64 bytes from diggle6 (192.168.1.44): icmp_seq=5 ttl=64 time=0.083 ms

--- diggle6 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3998ms
rtt min/avg/max/mdev = 0.076/0.095/0.130/0.019 ms
------------------------------- ping statistics -------------------------------

as above, the latency between Host and KVM guest(0.3+ms on average) is much higher
than the latency between the two physical hosts(~0.1ms on average), this might not be
acceptable for latency sensitive applications, for example memcached. 

is the latency normal? or is there something wrong with my configuration? here is detailed
information about my hardware and software environment:

Host hardware: DELL R710 + Intel Xeon E5620 @2.40GHz + 96GB memory + BCM5709 GB Ethernet
Host software: Gentoo amd64 + 2.6.36-gentoo-r5 + qemu-kvm-0.13.0

KVM related kernel configs for the host (diggle7):
diggle7 ~ # zgrep -E "(VIRTIO|KVM)" /proc/config.gz 
CONFIG_VIRTIO_BLK=m
CONFIG_VIRTIO_NET=m
# CONFIG_VIRTIO_CONSOLE is not set
CONFIG_HW_RANDOM_VIRTIO=m
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
# CONFIG_KVM_AMD is not set
CONFIG_VIRTIO=m
CONFIG_VIRTIO_RING=m
CONFIG_VIRTIO_PCI=m
CONFIG_VIRTIO_BALLOON=m

Kernel modules loaded on the host:
diggle7 ~ # lsmod
Module Size Used by
virtio_balloon 4098 0 
tun 12809 2 
bridge  63340 0 
stp  1871 1 bridge
llc  4595 2 bridge,stp
drbd 181114 7 
lru_cache  6273 1 drbd
virtio_net  10812 0 
virtio 3978 2 virtio_balloon,virtio_net
virtio_ring  4692 2 virtio_balloon,virtio_net
xt_state 1218 1 
iptable_filter 1626 1 
iptable_nat  3806 0 
nf_nat  15835 1 iptable_nat
nf_conntrack_ipv4 11106 4 iptable_nat,nf_nat
nf_defrag_ipv4 1361 1 nf_conntrack_ipv4
iptable_mangle 1591 0 
ipmi_devintf 7380 0 
ipmi_si 37990 0 
ipmi_msghandler 31256 2 ipmi_devintf,ipmi_si
kvm_intel 42059 18 
kvm  248729 1 kvm_intel
fuse  58921 2 

KVM related kernel configs for the guest (app211):
app211 ~ # zgrep -E "(VIRTIO|KVM)" /proc/config.gz
CONFIG_KVM_CLOCK=y
CONFIG_KVM_GUEST=y
CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_HAVE_KVM=y
# CONFIG_KVM is not set
CONFIG_VIRTIO=y
CONFIG_VIRTIO_RING=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BALLOON=y

Command line for starting the KVM guest:
diggle7 ~ # pgrep -fl qemu
18154 /usr/bin/qemu-system-x86_64 --enable-kvm -drive file=/vm/app211/app211.qcow2,if=virtio,boot=on,cache=writeback -cdrom /mfs/sysadmin/finnix.iso -boot c -vnc diggle7:211 -net nic,model=virtio,macaddr=52:54:C0:A8:01:D3 -net tap,ifname=app211,script=no -m 30720 -cpu host -smp 16 -daemonize

any help will be appreciated.

Thanks,
Xupeng

Re: High latency with VirtIO NIC

[Avi Kivity]

On 03/24/2011 02:37 AM, Xupeng Yun wrote:
> Hi,
>
> I deployed KVM on my Gentoo servers for performance testing, but I am now having high
> latency problems with the VirtIO NIC:
>

Try vhost-net, that should have much better latencies.

-- 
error compiling committee.c: too many arguments to function

Re: High latency with VirtIO NIC

[Xupeng Yun]

On Thursday, March 24, 2011 at 6:22 PM, Avi Kivity wrote:
Try vhost-net, that should have much better latencies.
>  Indeed, the latency is much better after switching to vhost-net, it drops from 3+ms to ~1.5ms on average
when there is no network load on the KVM guest, but the latencies become bad again when there is 
heavy network load on the guest(~70Mbps RX and ~25Mbps TX).

--------------------------- ping KVM guest (app211) from the host ---------------------------
64 bytes from app211 (192.168.1.211): icmp_seq=3582 ttl=64 time=3.08 ms
64 bytes from app211 (192.168.1.211): icmp_seq=3583 ttl=64 time=1.10 ms
64 bytes from app211 (192.168.1.211): icmp_seq=3584 ttl=64 time=1.03 ms
64 bytes from app211 (192.168.1.211): icmp_seq=3585 ttl=64 time=3.55 ms
64 bytes from app211 (192.168.1.211): icmp_seq=3586 ttl=64 time=1.29 ms
64 bytes from app211 (192.168.1.211): icmp_seq=3587 ttl=64 time=0.710 ms
64 bytes from app211 (192.168.1.211): icmp_seq=3588 ttl=64 time=64.4 ms
64 bytes from app211 (192.168.1.211): icmp_seq=3589 ttl=64 time=0.271 ms
64 bytes from app211 (192.168.1.211): icmp_seq=3590 ttl=64 time=0.563 ms
64 bytes from app211 (192.168.1.211): icmp_seq=3591 ttl=64 time=1.66 ms
64 bytes from app211 (192.168.1.211): icmp_seq=3592 ttl=64 time=0.939 ms
64 bytes from app211 (192.168.1.211): icmp_seq=3593 ttl=64 time=0.819 ms
64 bytes from app211 (192.168.1.211): icmp_seq=3594 ttl=64 time=16.5 ms

--------------------------- ping KVM guest (app211) from the host ---------------------------

any idea?

Thanks,
Xupeng

Re: High latency with VirtIO NIC

[Xupeng Yun]

On Thu, Mar 24, 2011 at 21:45, Xupeng Yun <recordus@gmail.com> wrote:
>
> Indeed, the latency is much better after switching to vhost-net, it drops from 3+ms to ~1.5ms on average

Typo, should be "it drops from 0.3+ms to ~0.15ms on average"

--
I like Linux & Python
http://blog.xupeng.me

Re: High latency with VirtIO NIC

[Avi Kivity]

On 03/24/2011 03:45 PM, Xupeng Yun wrote:
> On Thursday, March 24, 2011 at 6:22 PM, Avi Kivity wrote:
> Try vhost-net, that should have much better latencies.
> >   Indeed, the latency is much better after switching to vhost-net, it drops from 3+ms to ~1.5ms on average
> when there is no network load on the KVM guest, but the latencies become bad again when there is
> heavy network load on the guest(~70Mbps RX and ~25Mbps TX).
>
> --------------------------- ping KVM guest (app211) from the host ---------------------------
> 64 bytes from app211 (192.168.1.211): icmp_seq=3582 ttl=64 time=3.08 ms
> 64 bytes from app211 (192.168.1.211): icmp_seq=3583 ttl=64 time=1.10 ms
> 64 bytes from app211 (192.168.1.211): icmp_seq=3584 ttl=64 time=1.03 ms
> 64 bytes from app211 (192.168.1.211): icmp_seq=3585 ttl=64 time=3.55 ms
> 64 bytes from app211 (192.168.1.211): icmp_seq=3586 ttl=64 time=1.29 ms
> 64 bytes from app211 (192.168.1.211): icmp_seq=3587 ttl=64 time=0.710 ms
> 64 bytes from app211 (192.168.1.211): icmp_seq=3588 ttl=64 time=64.4 ms
> 64 bytes from app211 (192.168.1.211): icmp_seq=3589 ttl=64 time=0.271 ms
> 64 bytes from app211 (192.168.1.211): icmp_seq=3590 ttl=64 time=0.563 ms
> 64 bytes from app211 (192.168.1.211): icmp_seq=3591 ttl=64 time=1.66 ms
> 64 bytes from app211 (192.168.1.211): icmp_seq=3592 ttl=64 time=0.939 ms
> 64 bytes from app211 (192.168.1.211): icmp_seq=3593 ttl=64 time=0.819 ms
> 64 bytes from app211 (192.168.1.211): icmp_seq=3594 ttl=64 time=16.5 ms
>
> --------------------------- ping KVM guest (app211) from the host ---------------------------
>
> any idea?
>

Isn't it normal?  With heavy load, packets start to queue.

-- 
error compiling committee.c: too many arguments to function

Re: High latency with VirtIO NIC

[Xupeng Yun]

On Thu, Mar 24, 2011 at 22:08, Avi Kivity <avi@redhat.com> wrote:
>> 64 bytes from app211 (192.168.1.211): icmp_seq=3582 ttl=64 time=3.08 ms
>> 64 bytes from app211 (192.168.1.211): icmp_seq=3583 ttl=64 time=1.10 ms
>> 64 bytes from app211 (192.168.1.211): icmp_seq=3584 ttl=64 time=1.03 ms
>> 64 bytes from app211 (192.168.1.211): icmp_seq=3585 ttl=64 time=3.55 ms
>> 64 bytes from app211 (192.168.1.211): icmp_seq=3586 ttl=64 time=1.29 ms
>> 64 bytes from app211 (192.168.1.211): icmp_seq=3587 ttl=64 time=0.710 ms
>> 64 bytes from app211 (192.168.1.211): icmp_seq=3588 ttl=64 time=64.4 ms
>> 64 bytes from app211 (192.168.1.211): icmp_seq=3589 ttl=64 time=0.271 ms
>> 64 bytes from app211 (192.168.1.211): icmp_seq=3590 ttl=64 time=0.563 ms
>> 64 bytes from app211 (192.168.1.211): icmp_seq=3591 ttl=64 time=1.66 ms
>> 64 bytes from app211 (192.168.1.211): icmp_seq=3592 ttl=64 time=0.939 ms
>> 64 bytes from app211 (192.168.1.211): icmp_seq=3593 ttl=64 time=0.819 ms
>> 64 bytes from app211 (192.168.1.211): icmp_seq=3594 ttl=64 time=16.5 ms
>>
>> --------------------------- ping KVM guest (app211) from the host
>> ---------------------------
>>
>> any idea?
>>
>
> Isn't it normal?  With heavy load, packets start to queue.


Thank you, if the performance drop (from 0.15ms to 1-64ms) at this
workload is normal,  I am okay with it.

-- 
I like Linux & Python
http://blog.xupeng.me

Re: High latency with VirtIO NIC

[Avi Kivity]

On 03/24/2011 04:24 PM, Xupeng Yun wrote:
> On Thu, Mar 24, 2011 at 22:08, Avi Kivity<avi@redhat.com>  wrote:
> >>  64 bytes from app211 (192.168.1.211): icmp_seq=3582 ttl=64 time=3.08 ms
> >>  64 bytes from app211 (192.168.1.211): icmp_seq=3583 ttl=64 time=1.10 ms
> >>  64 bytes from app211 (192.168.1.211): icmp_seq=3584 ttl=64 time=1.03 ms
> >>  64 bytes from app211 (192.168.1.211): icmp_seq=3585 ttl=64 time=3.55 ms
> >>  64 bytes from app211 (192.168.1.211): icmp_seq=3586 ttl=64 time=1.29 ms
> >>  64 bytes from app211 (192.168.1.211): icmp_seq=3587 ttl=64 time=0.710 ms
> >>  64 bytes from app211 (192.168.1.211): icmp_seq=3588 ttl=64 time=64.4 ms
> >>  64 bytes from app211 (192.168.1.211): icmp_seq=3589 ttl=64 time=0.271 ms
> >>  64 bytes from app211 (192.168.1.211): icmp_seq=3590 ttl=64 time=0.563 ms
> >>  64 bytes from app211 (192.168.1.211): icmp_seq=3591 ttl=64 time=1.66 ms
> >>  64 bytes from app211 (192.168.1.211): icmp_seq=3592 ttl=64 time=0.939 ms
> >>  64 bytes from app211 (192.168.1.211): icmp_seq=3593 ttl=64 time=0.819 ms
> >>  64 bytes from app211 (192.168.1.211): icmp_seq=3594 ttl=64 time=16.5 ms
> >>
> >>  --------------------------- ping KVM guest (app211) from the host
> >>  ---------------------------
> >>
> >>  any idea?
> >>
> >
> >  Isn't it normal?  With heavy load, packets start to queue.
>
>
> Thank you, if the performance drop (from 0.15ms to 1-64ms) at this
> workload is normal,  I am okay with it.

I don't know if this is exactly the point at which latency should rise; 
what I'm saying is that it can't remain low at all load levels.


-- 
error compiling committee.c: too many arguments to function

Re: High latency with VirtIO NIC

[Xupeng Yun]

On Thu, Mar 24, 2011 at 22:27, Avi Kivity <avi@redhat.com> wrote:
> I don't know if this is exactly the point at which latency should rise; what
> I'm saying is that it can't remain low at all load levels.

Agree with you, it's reasonable. what puzzled me is that I don't know
if there is something wrong with my configurations.