weird packet loss between two VMs

weird packet loss between two VMs

[Simon Chen]

Hello,

I have two KVM VMs running on two servers, which are connected to the
same physical switch.

Each server has a 10G port connected to the switch, which have two VLANs.

For the first vlan, I directly configured IPs on the vlan interfaces
on the two servers. With this vlan, the two servers can use iperf to
push 600Mbps UDP packets to each other with little loss:
[  3]  0.0- 2.0 sec   147 MBytes   617 Mbits/sec   0.017 ms  257/105263 (0.24%)
[  3]  2.0- 4.0 sec   147 MBytes   618 Mbits/sec   0.019 ms    0/105133 (0%)
[  3]  4.0- 6.0 sec   148 MBytes   619 Mbits/sec   0.019 ms    0/105244 (0%)
[  3]  6.0- 8.0 sec   148 MBytes   619 Mbits/sec   0.022 ms    0/105237 (0%)
[  3]  8.0-10.0 sec   148 MBytes   619 Mbits/sec   0.018 ms    0/105254 (0%)
...
[  3]  0.0-30.0 sec  2.16 GBytes   619 Mbits/sec   0.018 ms
256/1578752 (0.016%)

(Not sure what happend in the first 2 seconds...)

For the two VMs, one is using 100M VNIC, the other is using 1000M one.
The vnet interfaces for the two VMs are put on two bridges on the two
servers, both tap into the second vlan. I then run iperf to send UDP
packets from the 100M VM to the 1000M VM using the following
parameter:
iperf -c 10.6.6.17 -t 30 -i 2 -r -b 600M

And I see something like this:
[  4]  0.0- 1.0 sec  42.6 MBytes   358 Mbits/sec   0.046 ms   93/30508 (0.3%)
[  4]  1.0- 2.0 sec  42.9 MBytes   360 Mbits/sec   0.044 ms    0/30613 (0%)
[  4]  2.0- 3.0 sec  41.9 MBytes   352 Mbits/sec   0.041 ms  507/30420 (1.7%)
[  4]  3.0- 4.0 sec  41.9 MBytes   352 Mbits/sec   0.039 ms  687/30591 (2.2%)
[  4]  4.0- 5.0 sec  48.7 MBytes   408 Mbits/sec   0.037 ms    0/34704 (0%)
[  4]  5.0- 6.0 sec  50.5 MBytes   423 Mbits/sec   0.041 ms    0/36010 (0%)
[  4]  6.0- 7.0 sec  21.6 MBytes   181 Mbits/sec   0.064 ms 4023/19431 (21%)
[  4]  7.0- 8.0 sec  24.5 MBytes   206 Mbits/sec   0.225 ms 1501/19003 (7.9%)
[  4]  8.0- 9.0 sec  24.2 MBytes   203 Mbits/sec   0.157 ms 2541/19786 (13%)
[  4]  9.0-10.0 sec  24.1 MBytes   202 Mbits/sec   0.102 ms 2670/19859 (13%)
[  4] 10.0-11.0 sec  21.6 MBytes   181 Mbits/sec   0.273 ms 3672/19078 (19%)
[  4] 11.0-12.0 sec  20.7 MBytes   174 Mbits/sec   0.061 ms 5038/19811 (25%)
[  4] 12.0-13.0 sec  24.3 MBytes   204 Mbits/sec   0.137 ms 2096/19403 (11%)
[  4] 13.0-14.0 sec  21.2 MBytes   178 Mbits/sec   0.381 ms 4123/19252 (21%)
[  4] 14.0-15.0 sec  46.0 MBytes   386 Mbits/sec   0.047 ms  347/33174 (1%)
[  4] 15.0-16.0 sec  47.6 MBytes   399 Mbits/sec   0.039 ms    0/33936 (0%)
[  4] 16.0-17.0 sec  47.3 MBytes   397 Mbits/sec   0.040 ms    0/33750 (0%)
[  4] 17.0-18.0 sec  46.0 MBytes   386 Mbits/sec   0.049 ms  191/33005 (0.58%)
[  4] 18.0-19.0 sec  46.2 MBytes   387 Mbits/sec   0.051 ms    0/32923 (0%)
[  4] 19.0-20.0 sec  46.2 MBytes   387 Mbits/sec   0.049 ms    0/32942 (0%)
[  4] 20.0-21.0 sec  45.9 MBytes   385 Mbits/sec   0.032 ms    0/32711 (0%)
[  4] 21.0-22.0 sec  45.9 MBytes   385 Mbits/sec   0.037 ms   37/32769 (0.11%)
[  4] 22.0-23.0 sec  45.0 MBytes   378 Mbits/sec   0.052 ms   83/32205 (0.26%)
[  4] 23.0-24.0 sec  45.1 MBytes   378 Mbits/sec   0.039 ms    0/32180 (0%)
[  4] 24.0-25.0 sec  34.6 MBytes   290 Mbits/sec   0.051 ms    0/24689 (0%)
[  4] 25.0-26.0 sec  33.8 MBytes   284 Mbits/sec   0.047 ms    0/24132 (0%)
[  4] 26.0-27.0 sec  33.9 MBytes   285 Mbits/sec   0.050 ms    0/24208 (0%)
[  4] 27.0-28.0 sec  33.6 MBytes   282 Mbits/sec   0.057 ms    0/23994 (0%)
[  4] 28.0-29.0 sec  40.1 MBytes   337 Mbits/sec   0.040 ms    0/28616 (0%)
[  4]  0.0-30.0 sec  1.11 GBytes   318 Mbits/sec   0.038 ms 27608/838083 (3.3%)



So, my question is... Why do I see this kind of variable dropping
rate? Is it because of KVM doing some kind of rate throttling to
emulate the virtual nic speed? Is there a way to achieve a more
consistent dropping rate?

The biggest problem I am having is actually with TCP. For the first
several seconds, the drop rate is small enough for TCP to ramp up to a
good speed, but then the drop rate becomes larger and essentially
stalls my TCP. Here is what I am talking about:
[  4] 12.0-14.0 sec  26.0 MBytes   109 Mbits/sec
[  4] 14.0-16.0 sec  18.0 MBytes  75.5 Mbits/sec --> start to drop
[  4] 16.0-18.0 sec 0.00 Bytes  0.00 bits/sec --> stalled
[  4] 18.0-20.0 sec  47.9 MBytes   201 Mbits/sec --> back to speed
[  4] 20.0-22.0 sec  58.5 MBytes   245 Mbits/sec
[  4] 22.0-24.0 sec  44.6 MBytes   187 Mbits/sec

Thanks.
-Simon

Re: weird packet loss between two VMs

[Martin Mailand]

Hi Simon,
you are using a 100Mbits nic and you try to send with 600M, try a 
1000Mbits on the sending site as well.

-martin

Am 05.03.2012 00:57, schrieb Simon Chen:
> For the two VMs, one is using 100M VNIC, the other is using 1000M one.
> The vnet interfaces for the two VMs are put on two bridges on the two
> servers, both tap into the second vlan. I then run iperf to send UDP
> packets from the 100M VM to the 1000M VM using the following
> parameter:
> iperf -c 10.6.6.17 -t 30 -i 2 -r -b 600M