[ kvm-Bugs-2506814 ] TAP network lockup after some traffic

SourceForge.net
2009-08-05T11:02:59Z

Message generated for change (Comment added) made by mellen
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=893831&aid=2506814&group_id=180599

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Fabio Coatti (cova)
Assigned to: Nobody/Anonymous (nobody)
Summary: TAP network lockup after some traffic

Initial Comment:
Hi all,
we are experiencing severe network troubles using kvm+tap networking.
basically, after some network load (we have yet to identify the exact amount of traffic if one exist) network stops working.
During lockups, With tcpdump we can see arp requests on guest interface, then on tap, brX and physical interfaces on host system.
the arp answers can be seen, with tcpdump, only on physical host interface and bridge (brX), but not on tap device. Basically it seems that packets coming from external network get lost in tap device on the way to guest (kvm). Looking at tap data with ifconfig, the only weird thing is the TX packets overrun count that is > 0. every time the network stops working, overrun count increases.
This has been observed with several kvm releases (for sure, 76/77/78/79/80/81/82) and with different kernels (tried with some versions among 2.6.25.X, 26.X, 27.X, 28) both on guest and host side.
we tried several network drivers (virtio, e1000, rtl) and all shows the same problem. Only 100Mbit drivers seems to be unaffected so far. (only virtio has acceptable performance)
(btw: on host machine we have vlan on top of ethX devices)
cpu number on guest makes no difference.
we tried with vanilla kernel provided kvm modules and with kvm package provided modules.
guest: 32 bit
host: 64 bit
host machine:

2 x Quad-Core AMD Opteron(tm) Processor 2352 16GB, gentoo.

Of course I can provide more details or perform other tests and try patches, if someone can give me some hints and advices they will be most welcome.

Thanks.

----------------------------------------------------------------------

Comment By: Tais M. Hansen (mellen)
Date: 2009-08-05 13:02

Message:
All my tests have been made with the kernel modules.

----------------------------------------------------------------------

Comment By: Fabio Coatti (cova)
Date: 2009-08-05 12:56

Message:
mellen, did you tested with kernel modules or kvm-provided modules?

----------------------------------------------------------------------

Comment By: Tais M. Hansen (mellen)
Date: 2009-08-04 17:48

Message:
Re-tested with kvm-88 and host kernel 2.6.30.3 and guest kernel 2.6.30.4.
I've have not been able to reproduce the problem so far. The test has been
running for 8 hours and 15 minutes now.

----------------------------------------------------------------------

Comment By: Daniel Schwager (danny1)
Date: 2009-07-13 11:09

Message:
> I installed KVM-86 at the moment and the first test works fine - no
disconnect
> at the moment (..)

i got the disconnect with KVM-86 also..

c_jones, did you check it with the current KVM-88 release ?

----------------------------------------------------------------------

Comment By: Daniel Schwager (danny1)
Date: 2009-07-11 09:24

Message:
I can confirm exactly the same behavior with kvm-84 - all my machines uses
also a dhcp daemon for requesting there IP adr.

> My guest nodes are configured to get their IP addresses with DHCP. At
the
> DHCP server I see the DHCPDISCOVER requests from the guests, and I see
the
> DHCP server replying with DHCPOFFERs, but the guest never sees the
offers
> come in.

I installed KVM-86 at the moment and the first test works fine - no
disconnect
at the moment (..)

----------------------------------------------------------------------

Comment By: Chris Jones (c_jones)
Date: 2009-07-11 05:14

Message:
I'm experiencing exactly the same problem.

I've been running kvm-84 for quite a while and it works great there. But
I tried moving to kvm-87 and on kvm-87, I'm getting exactly the same
behavior you're reporting here -- except all the time (it doesn't ever
work).

My guest nodes are configured to get their IP addresses with DHCP. At the
DHCP server I see the DHCPDISCOVER requests from the guests, and I see the
DHCP server replying with DHCPOFFERs, but the guest never sees the offers
come in.

So, just like the other reports - packets outbound from the guest are
fine, but inbound they seem to get dropped.

Like the other reports, DHCP requests are UDP, so I guess this might lend
weight to the argument this could happen more often with UDP.

My environment (where it fails)
kvm-87
kernel 2.6.27.26
Using the kvm-87 user component and kernel component.

On kvm-84 all works fine (which seems to be in contrast to another
person's report).

Mine is easily reproducible - let me know if you want me to collect any
information.

----------------------------------------------------------------------

Comment By: Daniel Schwager (danny1)
Date: 2009-07-08 21:16

Message:
> Mellen, did you try KVM-85 ?
not -85, but did you try the current on: KVM-87 ?

----------------------------------------------------------------------

Comment By: Daniel Schwager (danny1)
Date: 2009-07-08 21:13

Message:
Hi,

i have the same (?) problem. We are running about 20 x WinXP's using RDP
on an 64bit FC9 with KVM-84, bridge-network, tap-device with e1000, smp-1.

Sometimes (most (!) 3-4 minutes after starting a winxp-vm), i loose
the RDP-connection. While this disconnection, a running ping inside the VM
tells's me about ping problems. After only 2-3 Seconds later, i can
reconnect the rdp and all works find ... until the next disconnect..

Mellen, did you try KVM-85 ?

regards
Danny

----------------------------------------------------------------------

Comment By: Tais M. Hansen (mellen)
Date: 2009-07-02 17:22

Message:
So far, I have been unable to reproduce this with a guest using only one
cpu (-smp 1). Previous tests have been made on guests with -smp 2 or 4.

----------------------------------------------------------------------

Comment By: Tais M. Hansen (mellen)
Date: 2009-06-29 18:43