Network Overruns

Network Overruns

[Henri Cook]

Hi all,

I'm running ~13 virtual machines in production on KVM 62 (unfortunately
the best ubuntu-stable can provide for me :/) and i've noticed a lot of
TX overruns, is this something i should be worried about? Are my
machines losing data?

These machines are all bridged with options like this:

-net nic,macaddr=00:16:3e:00:00:05,vlan=0,model=e1000 -net
tap,fd=17,script=,vlan=0 (where fd differs for each machine)

ifconfig output enclosed below:

br0       Link encap:Ethernet  HWaddr 00:15:17:6b:2d:14 
          inet addr:my.ip.address  Bcast:my.ip.addr.255  Mask:255.255.255.0
          inet6 addr: my.ipv6 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:610572463 errors:0 dropped:0 overruns:0 frame:0
          TX packets:108729730 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:824841768 (786.6 MB)  TX bytes:3829964354 (3.5 GB)

eth0      Link encap:Ethernet  HWaddr 00:15:17:6b:2d:14 
          inet6 addr: my.ipv6 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1521528058 errors:0 dropped:108777021 overruns:0
frame:0
          TX packets:276414465 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:417285897 (397.9 MB)  TX bytes:3547313532 (3.3 GB)
          Base address:0xef00 Memory:fdde0000-fde00000

lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:64334 errors:0 dropped:0 overruns:0 frame:0
          TX packets:64334 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:7516953 (7.1 MB)  TX bytes:7516953 (7.1 MB)

vnet0     Link encap:Ethernet  HWaddr ca:c2:8b:75:ee:83 
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          inet6 addr: fe80::c8c2:8bff:fe75:ee83/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:468 (468.0 B)

vnet1     Link encap:Ethernet  HWaddr 00:ff:c1:ca:e0:18 
          inet6 addr: fe80::2ff:c1ff:feca:e018/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:306211 errors:0 dropped:0 overruns:0 frame:0
          TX packets:208417450 errors:0 dropped:0 overruns:292 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:84842203 (80.9 MB)  TX bytes:3341249731 (3.1 GB)

vnet2     Link encap:Ethernet  HWaddr 00:ff:55:94:72:9b 
          inet6 addr: fe80::2ff:55ff:fe94:729b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:243806 errors:0 dropped:0 overruns:0 frame:0
          TX packets:227992073 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:58875698 (56.1 MB)  TX bytes:1147433844 (1.0 GB)

vnet3     Link encap:Ethernet  HWaddr 00:ff:b1:b7:41:5f 
          inet6 addr: fe80::2ff:b1ff:feb7:415f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:813360 errors:0 dropped:0 overruns:0 frame:0
          TX packets:228960883 errors:0 dropped:0 overruns:16 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:64048107 (61.0 MB)  TX bytes:1217812685 (1.1 GB)

vnet4     Link encap:Ethernet  HWaddr 00:ff:6e:87:17:7b 
          inet6 addr: fe80::2ff:6eff:fe87:177b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:24463 errors:0 dropped:0 overruns:0 frame:0
          TX packets:227534964 errors:0 dropped:0 overruns:501 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:4312887 (4.1 MB)  TX bytes:1060554904 (1011.4 MB)

vnet5     Link encap:Ethernet  HWaddr 00:ff:50:6a:26:62 
          inet6 addr: fe80::2ff:50ff:fe6a:2662/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:57623 errors:0 dropped:0 overruns:0 frame:0
          TX packets:203904515 errors:0 dropped:0 overruns:491 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:9015385 (8.5 MB)  TX bytes:2856817773 (2.6 GB)

vnet6     Link encap:Ethernet  HWaddr 00:ff:3f:ff:32:86 
          inet6 addr: fe80::2ff:3fff:feff:3286/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:377799 errors:0 dropped:0 overruns:0 frame:0
          TX packets:233685442 errors:0 dropped:0 overruns:1 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:67794919 (64.6 MB)  TX bytes:2100174866 (1.9 GB)

vnet7     Link encap:Ethernet  HWaddr 00:ff:79:fd:64:94 
          inet6 addr: fe80::2ff:79ff:fefd:6494/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:306667 errors:0 dropped:0 overruns:0 frame:0
          TX packets:228114579 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:42935219 (40.9 MB)  TX bytes:1879470986 (1.7 GB)

vnet9     Link encap:Ethernet  HWaddr 00:ff:b4:9c:e6:31 
          inet6 addr: fe80::2ff:b4ff:fe9c:e631/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:227503853 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:0 (0.0 B)  TX bytes:1057149624 (1008.1 MB)

vnet10    Link encap:Ethernet  HWaddr 00:ff:6a:2f:08:5e 
          inet6 addr: fe80::2ff:6aff:fe2f:85e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:29123 errors:0 dropped:0 overruns:0 frame:0
          TX packets:227532228 errors:0 dropped:0 overruns:218 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:1591885 (1.5 MB)  TX bytes:1058977339 (1009.9 MB)

vnet11    Link encap:Ethernet  HWaddr 00:ff:d4:76:9d:de 
          inet6 addr: fe80::2ff:d4ff:fe76:9dde/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:218597 errors:0 dropped:0 overruns:0 frame:0
          TX packets:227728885 errors:0 dropped:0 overruns:566 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:31377899 (29.9 MB)  TX bytes:1084583551 (1.0 GB)

vnet12    Link encap:Ethernet  HWaddr d2:11:1d:f4:46:c0 
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          inet6 addr: fe80::d011:1dff:fef4:46c0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:468 (468.0 B)

vnet13    Link encap:Ethernet  HWaddr 00:ff:77:a1:0c:2a 
          inet6 addr: fe80::2ff:77ff:fea1:c2a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:30510393 errors:0 dropped:0 overruns:0 frame:0
          TX packets:249244012 errors:0 dropped:0 overruns:2036 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:3854788416 (3.5 GB)  TX bytes:2761388091 (2.5 GB)

Cheers,

Henri

Re: Network Overruns

[Avi Kivity]

Henri Cook wrote:
> Hi all,
>
> I'm running ~13 virtual machines in production on KVM 62 (unfortunately
> the best ubuntu-stable can provide for me :/) and i've noticed a lot of
> TX overruns, is this something i should be worried about? Are my
> machines losing data?
>   

There shouldn't be any data loss, since the guest expects the network to
be lossy.  But this can cause performance drops due to retries.

Most likely the guest's rx queue length is greater than the host's.  You
might try

   ifconfig vnet0 txqueuelen 1500

(and so on for every interface)


or perhaps reducing the guests' txqueuelen.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Network Overruns

[xming]

> Most likely the guest's rx queue length is greater than the host's.  You
> might try
>
>   ifconfig vnet0 txqueuelen 1500
>
> (and so on for every interface)
>
>
> or perhaps reducing the guests' txqueuelen.

I have the same issue (overruns) and I have stalled network (in my other
report I can only unstall the network by putting the NIC in promisc back
and forth).

The txqueuelen is indeed greater in the guest then the tap on the host,
and I also notice that it's almost ways the output NIc which get stuck.

So changing the tap (on the host) to 1500, I can't not reproduce the hang yet
and a ping -f -s 64000 (between guests) does not produce any packet loss.

27810 packets transmitted, 27809 received, 0% packet loss, time 259080ms
rtt min/avg/max/mdev = 5.013/7.639/33.999/1.536 ms, pipe 3, ipg/ewma
9.316/7.284 ms

I hope this is the solution for my network stalls and packet loss in games.

Re: Network Overruns

[Avi Kivity]

xming wrote:
>> Most likely the guest's rx queue length is greater than the host's.  You
>> might try
>>
>>   ifconfig vnet0 txqueuelen 1500
>>
>> (and so on for every interface)
>>
>>
>> or perhaps reducing the guests' txqueuelen.
>>     
>
> I have the same issue (overruns) and I have stalled network (in my other
> report I can only unstall the network by putting the NIC in promisc back
> and forth).
>
> The txqueuelen is indeed greater in the guest then the tap on the host,
> and I also notice that it's almost ways the output NIc which get stuck.
>
> So changing the tap (on the host) to 1500, I can't not reproduce the hang yet
> and a ping -f -s 64000 (between guests) does not produce any packet loss.
>
> 27810 packets transmitted, 27809 received, 0% packet loss, time 259080ms
> rtt min/avg/max/mdev = 5.013/7.639/33.999/1.536 ms, pipe 3, ipg/ewma
> 9.316/7.284 ms
>
> I hope this is the solution for my network stalls and packet loss in games.
>   

It's not a fix, rather a workaround.  Hopefully some networking guru
will come up with a real fix.

What NIC model are you using in the guest?

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Network Overruns

[Avi Kivity]

(re-adding list)

xming wrote:
>> What NIC model are you using in the guest?
>>     
>
> I have tried 8319, e1000 and virtio both had stalls. I am using virtio
> with txqueuelen 1500 on the tap.
> I didin't had this proble with kvm 69, 70.
>   

What is the oldest version that doesn't work?  (i.e. when was the 
regression introduced?)


-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: Network Overruns

[xming]

> What is the oldest version that doesn't work?  (i.e. when was the regression
> introduced?)

I can (almost) be sure the kvm-70 did not have stalls and that kvm 73 certainly
has this problem.

Never tried 71 and I am not sure about 72 (haven't ran it for long
time and upgraded to 73)

Re: Network Overruns

[xming]

On Sun, Sep 14, 2008 at 7:56 PM, xming <xmingske@gmail.com> wrote:
>> What is the oldest version that doesn't work?  (i.e. when was the regression
>> introduced?)
>
> I can (almost) be sure the kvm-70 did not have stalls and that kvm 73 certainly
> has this problem.
>
> Never tried 71 and I am not sure about 72 (haven't ran it for long
> time and upgraded to 73)
>

Ok I tried kvm 69 to 75, it does not seems to be kvm version related.
It seems to be host kernel version realted.

- with 2.6.26.x I have very bad network, stalled nfs, stalled https, packet loss
- with 2.6.25.x I don't have stalled (or at least much less frequent,
haven't seen one yet) and much less packet loss
- with 2.6.27-rcX network seems to be fine but can't test for long
because the virtio block hangs is back :(

Re: Network Overruns

[xming]

>>> What is the oldest version that doesn't work?  (i.e. when was the regression
>>> introduced?)
>>
>> I can (almost) be sure the kvm-70 did not have stalls and that kvm 73 certainly
>> has this problem.
>>
>> Never tried 71 and I am not sure about 72 (haven't ran it for long
>> time and upgraded to 73)
>>
>
> Ok I tried kvm 69 to 75, it does not seems to be kvm version related.
> It seems to be host kernel version realted.
>
> - with 2.6.26.x I have very bad network, stalled nfs, stalled https, packet loss
> - with 2.6.25.x I don't have stalled (or at least much less frequent,
> haven't seen one yet) and much less packet loss
> - with 2.6.27-rcX network seems to be fine but can't test for long
> because the virtio block hangs is back :(

An other user on the irc came to the same conclusion

< OpenTokix> I have just shy of 3000 new connections per second, and I
get strange timeouts on a server
                   running 2.6.26 (inside kvm) and not as many on
2.6.25 - both using virtio-interface