Re: network shutdown under heavy load

[Tom Lendacky <tahm@linux.vnet.ibm.com>]

On Tuesday 19 January 2010 05:57:53 pm Chris Wright wrote:
> * Tom Lendacky (tahm@linux.vnet.ibm.com) wrote:
> > On Wednesday 13 January 2010 03:52:28 pm Chris Wright wrote:
> > > (Mark cc'd, sound familiar?)
> > >
> > > * Tom Lendacky (tahm@linux.vnet.ibm.com) wrote:
> > > > On Sunday 10 January 2010 06:38:54 am Avi Kivity wrote:
> > > > > On 01/10/2010 02:35 PM, Herbert Xu wrote:
> > > > > > On Sun, Jan 10, 2010 at 02:30:12PM +0200, Avi Kivity wrote:
> > > > > >> This isn't in 2.6.27.y.  Herbert, can you send it there?
> > > > > >
> > > > > > It appears that now that TX is fixed we have a similar problem
> > > > > > with RX.  Once I figure that one out I'll send them together.
> > > >
> > > > I've been experiencing the network shutdown issue also.  I've been
> > > > running netperf tests across 10GbE adapters with Qemu 0.12.1.2,
> > > > RHEL5.4 guests and 2.6.32 kernel (from kvm.git) guests.  I
> > > > instrumented Qemu to print out some network statistics.  It appears
> > > > that at some point in the netperf test the receiving guest ends up
> > > > having the 10GbE device "receive_disabled" variable in its
> > > > VLANClientState structure stuck at 1. From looking at the code it
> > > > appears that the virtio-net driver in the guest should cause
> > > > qemu_flush_queued_packets in net.c to eventually run and clear the
> > > > "receive_disabled" variable but it's not happening.  I don't seem to
> > > > have these issues when I have a lot of debug settings active in the
> > > > guest kernel which results in very low/poor network performance -
> > > > maybe some kind of race condition?
> >
> > Ok, here's an update. After realizing that none of the ethtool offload
> > options were enabled in my guest, I found that I needed to be using the
> > -netdev option on the qemu command line.  Once I did that, some ethtool
> > offload options were enabled and the deadlock did not appear when I did
> > networking between guests on different machines.  However, the deadlock
> > did appear when I did networking between guests on the same machine.
> 
> What does your full command line look like?  And when the networking
> stops does your same receive_disabled hack make things work?

The command line when using the -net option for the tap device is:

/usr/local/bin/qemu-system-x86_64 -name cape-vm001 -m 1024 -drive 
file=/autobench/var/tmp/cape-vm001-
raw.img,if=virtio,index=0,media=disk,boot=on -net 
nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:51 -net 
tap,vlan=0,script=/autobench/var/tmp/ifup-kvm-
br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net 
nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:D1 -net 
tap,vlan=1,script=/autobench/var/tmp/ifup-kvm-
br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :1 -monitor 
telnet::5701,server,nowait -snapshot -daemonize

when using the -netdev option for the tap device:

/usr/local/bin/qemu-system-x86_64 -name cape-vm001 -m 1024 -drive 
file=/autobench/var/tmp/cape-vm001-
raw.img,if=virtio,index=0,media=disk,boot=on -net 
nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:51,netdev=cape-vm001-eth0 -
netdev tap,id=cape-vm001-eth0,script=/autobench/var/tmp/ifup-kvm-
br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net 
nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:D1,netdev=cape-vm001-eth1 -
netdev tap,id=cape-vm001-eth1,script=/autobench/var/tmp/ifup-kvm-
br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :1 -monitor 
telnet::5701,server,nowait -snapshot -daemonize


The first ethernet device is a 1GbE device for communicating with the 
automation infrastructure we have.  The second ethernet device is the 10GbE 
device that the netperf tests run on.

I can get the networking to work again by bringing down the interfaces and 
reloading the virtio_net module (modprobe -r virtio_net / modprobe 
virtio_net).

I haven't had a chance yet to run the tests against a modified version of qemu 
that does not set the receive_disabled variable.

Tom

> 
> thanks,
> -chris
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>