virtio-net has a sudden high latency

virtio-net has a sudden high latency

[yue]

Hi, all. virtio-net  has a sudden high latency when receiving.  i think it is irq related, but i have no idea to fix it.
centos 3.10.0-693.21.1.el7.x86_64 ; qemu-kvm-ev-2.6.0-28.el7_3.9.1.x86_64  ;   
physical machine has 40 cpus,   the two vms are on the same node,  cpu load is very low.


1) pic1:  inside vm1(192.168.91.4) , ping 192.168.91.5;  please notice icmp seq  20
[1577252456.600426] 64 bytes from 192.168.91.5: icmp_seq=12 ttl=64 time=0.692 ms
[1577252457.600514] 64 bytes from 192.168.91.5: icmp_seq=13 ttl=64 time=0.462 ms
[1577252458.600590] 64 bytes from 192.168.91.5: icmp_seq=14 ttl=64 time=0.396 ms
[1577252459.600922] 64 bytes from 192.168.91.5: icmp_seq=15 ttl=64 time=0.888 ms
[1577252460.600445] 64 bytes from 192.168.91.5: icmp_seq=16 ttl=64 time=0.355 ms
[1577252461.601328] 64 bytes from 192.168.91.5: icmp_seq=17 ttl=64 time=0.311 ms
[1577252462.600449] 64 bytes from 192.168.91.5: icmp_seq=18 ttl=64 time=0.424 ms
[1577252463.601725] 64 bytes from 192.168.91.5: icmp_seq=19 ttl=64 time=0.322 ms
[1577252464.606752] 64 bytes from 192.168.91.5: icmp_seq=20 ttl=64 time=5.65 ms
[1577252465.603569] 64 bytes from 192.168.91.5: icmp_seq=21 ttl=64 time=0.438 ms
[1577252466.603679] 64 bytes from 192.168.91.5: icmp_seq=22 ttl=64 time=0.494 ms
[1577252467.603545] 64 bytes from 192.168.91.5: icmp_seq=23 ttl=64 time=0.431 ms


tcpdump inside vm1 ; 

13:41:04.601016 IP 192.168.91.4 > 192.168.91.5: ICMP echo request, id 18375, seq 20, length 64
13:41:04.606647 IP 192.168.91.5 > 192.168.91.4: ICMP echo reply, id 18375, seq 20, length 64

2) capture packet on its backend tap device on hypervisor ;    tcpdump -nn -i tap5803c575-4a icmp and host 192.168.91.5 and host 192.168.91.4

13:41:04.601166 IP 192.168.91.4 > 192.168.91.5: ICMP echo request, id 18375, seq 20, length 64
13:41:04.601362 IP 192.168.91.5 > 192.168.91.4: ICMP echo reply, id 18375, seq 20, length 64
it shows  tap takes no much time between send and receive.

3)
606647 - 601362 = 5.3ms ,  this is consumed by virtio-net.
similarly, when 192.168.91.5 receive packets from its backend, it also has sudden high latency.


do you all have any ideas?


thanks