virtio-net has a sudden high latency when receiving

[yue <kvmluck@163.com>]

[-- Attachment #1: Type: text/plain, Size: 2654 bytes --]

Hi, all. virtio-net  has a sudden high latency when receiving.  i think it is irq related, but i have no idea to fix it.
centos 3.10.0-693.21.1.el7.x86_64 ; qemu-kvm-ev-2.6.0-28.el7_3.9.1.x86_64  ;   
physical machine has 40 cpus,   the two vms are on the same node,  cpu load is very low.
1) pic1:  inside vm1(192.168.91.4) , ping 192.168.91.5;  please notice seq  20



[1577252450.592441] 64 bytes from 192.168.91.5: icmp_seq=6 ttl=64 time=0.335 ms

[1577252451.592373] 64 bytes from 192.168.91.5: icmp_seq=7 ttl=64 time=0.308 ms

[1577252452.592449] 64 bytes from 192.168.91.5: icmp_seq=8 ttl=64 time=0.381 ms

[1577252453.597358] 64 bytes from 192.168.91.5: icmp_seq=9 ttl=64 time=2.21 ms

[1577252454.596415] 64 bytes from 192.168.91.5: icmp_seq=10 ttl=64 time=0.353 ms

[1577252455.596446] 64 bytes from 192.168.91.5: icmp_seq=11 ttl=64 time=0.369 ms

[1577252456.600426] 64 bytes from 192.168.91.5: icmp_seq=12 ttl=64 time=0.692 ms

[1577252457.600514] 64 bytes from 192.168.91.5: icmp_seq=13 ttl=64 time=0.462 ms

[1577252458.600590] 64 bytes from 192.168.91.5: icmp_seq=14 ttl=64 time=0.396 ms

[1577252459.600922] 64 bytes from 192.168.91.5: icmp_seq=15 ttl=64 time=0.888 ms

[1577252460.600445] 64 bytes from 192.168.91.5: icmp_seq=16 ttl=64 time=0.355 ms

[1577252461.601328] 64 bytes from 192.168.91.5: icmp_seq=17 ttl=64 time=0.311 ms

[1577252462.600449] 64 bytes from 192.168.91.5: icmp_seq=18 ttl=64 time=0.424 ms

[1577252463.601725] 64 bytes from 192.168.91.5: icmp_seq=19 ttl=64 time=0.322 ms

[1577252464.606752] 64 bytes from 192.168.91.5: icmp_seq=20 ttl=64 time=5.65 ms

[1577252465.603569] 64 bytes from 192.168.91.5: icmp_seq=21 ttl=64 time=0.438 ms

[1577252466.603679] 64 bytes from 192.168.91.5: icmp_seq=22 ttl=64 time=0.494 ms

[1577252467.603545] 64 bytes from 192.168.91.5: icmp_seq=23 ttl=64 time=0.431 ms



tcpdump inside vm1 ; 

13:41:04.601016 IP 192.168.91.4 > 192.168.91.5: ICMP echo request, id 18375, seq 20, length 64

13:41:04.606647 IP 192.168.91.5 > 192.168.91.4: ICMP echo reply, id 18375, seq 20, length 64

2) capture packet on its backend tap device on hypervisor ;    tcpdump -nn -i tap5803c575-4a icmp and host 192.168.91.5 and host 192.168.91.4

13:41:04.601166 IP 192.168.91.4 > 192.168.91.5: ICMP echo request, id 18375, seq 20, length 64

13:41:04.601362 IP 192.168.91.5 > 192.168.91.4: ICMP echo reply, id 18375, seq 20, length 64

it shows  tap takes no much time between send and receive.


606647 - 601362 = 5.3ms ,  this is consumed by virtio-net.


similarly, when 192.168.91.5 receive packets from its backend, it also has sudden high latency.


do you all have any ideas?


thanks







 

[-- Attachment #2: Type: text/html, Size: 10529 bytes --]