possible bug: VM drops incoming network packets (tap)

possible bug: VM drops incoming network packets (tap)

[Joris]

Hi,


Please CC me, I'm not a subscriber to the list.

Also, please note this could very well be a guest-os problem, but I've
never experienced anything like this before.


I have a Debian unstable 64bit host on an intel S3000 quad core xeon
X3210. Kernel 2.6.22-2-amd64 (debian flavour) KVM46.
My guests are debian32 with 1 cpu and 1GB ram, connected via '-net
nic,macaddr=52:54:00:00:00:xx -net tap' and linux bridging. I'm
running 3 guests at the time.


Sometimes (see below) the machine appears to drop all incoming packets
from the network.
I can see the guest send arp requests, I can see the replies going
back in the bridge/tap interface on the host, but they don't appear on
the guest.

A fix is taking the guest's nic down and up again, using ifconfig.
Rebuilding the bridge on the host does not help.


This appears to happen on combined disk and network IO. I was
uploading data over a ssh connection (via FISH) to a disk on the
guest.
Doing a mild 100Mbit/s icmp flood to the guest does not trip this behavior.

I don't see any logic to the frequency. I can transfer multiple
gigabytes without a problem, or it can happen every few MB or seconds.


I don't think I've experienced this with KVM36 (my previous version),
but certainly with KVM46.


Any help or suggestions would be greatly appreciated.

Kind regards,
Joris

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: possible bug: VM drops incoming network packets (tap)

[Jim Paris]

Joris wrote:
> I have a Debian unstable 64bit host on an intel S3000 quad core xeon
> X3210. Kernel 2.6.22-2-amd64 (debian flavour) KVM46.
> My guests are debian32 with 1 cpu and 1GB ram, connected via '-net
> nic,macaddr=52:54:00:00:00:xx -net tap' and linux bridging. I'm
> running 3 guests at the time.
> 
> 
> Sometimes (see below) the machine appears to drop all incoming packets
> from the network.

Hi Joris,

I'm in the same boat -- upgraded from kvm-36 to kvm-46 and I'm seeing
identical network problems.

host: quad 64-bit 2.6.20.4
guests: 32-bit 2.6.18, 32-bit 2.6.21, 64-bit 2.6.18

No difference with -no-kvm-irqchip and with "taskset -c 1".  I haven't
had a chance to debug it much further yet, although trying a different
network card might be my next step.  I was also (reluctantly) going to
try rebooting the host in case unloading the kvm-36 modules left
something in a bad state.

An especially strange thing is that, before the network dies, sshing
from my desktop->host->guest has no problems (gigabit lan from
desktop->host).  But connecting directly desktop->guest leads to
frequent short delays and noticable lag when typing.  Since that just
seems weird I was going to try to figure that part out before I
reported this to the list :)

-jim

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: possible bug: VM drops incoming network packets (tap)

[Joris]

2007/10/20, Jim Paris <jim-XrPbb/hENzg@public.gmane.org>:


> I'm in the same boat -- upgraded from kvm-36 to kvm-46 and I'm seeing
> identical network problems.

Great I'm not the only one. Does it help to ifdown && ifup in your case?

> No difference with -no-kvm-irqchip and with "taskset -c 1".  I haven't
> had a chance to debug it much further yet, although trying a different
> network card might be my next step.  I was also (reluctantly) going to
> try rebooting the host in case unloading the kvm-36 modules left
> something in a bad state.

I have a host reboot scheduled for this weekend, I'll report on it.
A different nic is a good idea, I'll try that as well.


> An especially strange thing is that, before the network dies, sshing

I haven't really observed that, but I'll keep an eye out.


Kind regards,
Joris

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: possible bug: VM drops incoming network packets (tap)

[Joris]

Jim,


Both changing the nic (to rtl8139) and rebooting the host solved the
issue for me.
My VM's have now survived over 12 hours of intense activity without a glitch.


To be clear: soly changing the nic without rebooting the host worked;
after rebooting the host the default nic works again.



Kind regards,
Joris

2007/10/20, Joris <joris-3CvOTOCK0fQ@public.gmane.org>:
> 2007/10/20, Jim Paris <jim-XrPbb/hENzg@public.gmane.org>:
>
>
> > I'm in the same boat -- upgraded from kvm-36 to kvm-46 and I'm seeing
> > identical network problems.
>
> Great I'm not the only one. Does it help to ifdown && ifup in your case?
>
> > No difference with -no-kvm-irqchip and with "taskset -c 1".  I haven't
> > had a chance to debug it much further yet, although trying a different
> > network card might be my next step.  I was also (reluctantly) going to
> > try rebooting the host in case unloading the kvm-36 modules left
> > something in a bad state.
>
> I have a host reboot scheduled for this weekend, I'll report on it.
> A different nic is a good idea, I'll try that as well.
>
>
> > An especially strange thing is that, before the network dies, sshing
>
> I haven't really observed that, but I'll keep an eye out.
>
>
> Kind regards,
> Joris
>

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: possible bug: VM drops incoming network packets (tap)

[Jim Paris]

Joris wrote:
> Both changing the nic (to rtl8139) and rebooting the host solved the
> issue for me.
> My VM's have now survived over 12 hours of intense activity without a glitch.
> 
> To be clear: soly changing the nic without rebooting the host worked;
> after rebooting the host the default nic works again.

I rebooted the host (and upgraded to 2.6.23.1), but the default
ne2k-pci nic still had the same problems.  I switched my guests all to
rtl8139 and they seem to be working much better.

Seems that ne2k-pci has had some bugs introduced between kvm-36 and kvm-46.

-jim

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/