Re: network shutdown under heavy load

[Chris Wright <chrisw@sous-sol.org>]

* Tom Lendacky (tahm@linux.vnet.ibm.com) wrote:
> On Wednesday 13 January 2010 03:52:28 pm Chris Wright wrote:
> > (Mark cc'd, sound familiar?)
> > 
> > * Tom Lendacky (tahm@linux.vnet.ibm.com) wrote:
> > > On Sunday 10 January 2010 06:38:54 am Avi Kivity wrote:
> > > > On 01/10/2010 02:35 PM, Herbert Xu wrote:
> > > > > On Sun, Jan 10, 2010 at 02:30:12PM +0200, Avi Kivity wrote:
> > > > >> This isn't in 2.6.27.y.  Herbert, can you send it there?
> > > > >
> > > > > It appears that now that TX is fixed we have a similar problem
> > > > > with RX.  Once I figure that one out I'll send them together.
> > >
> > > I've been experiencing the network shutdown issue also.  I've been
> > > running netperf tests across 10GbE adapters with Qemu 0.12.1.2, RHEL5.4
> > > guests and 2.6.32 kernel (from kvm.git) guests.  I instrumented Qemu to
> > > print out some network statistics.  It appears that at some point in the
> > > netperf test the receiving guest ends up having the 10GbE device
> > > "receive_disabled" variable in its VLANClientState structure stuck at 1. 
> > > From looking at the code it appears that the virtio-net driver in the
> > > guest should cause qemu_flush_queued_packets in net.c to eventually run
> > > and clear the "receive_disabled" variable but it's not happening.  I
> > > don't seem to have these issues when I have a lot of debug settings
> > > active in the guest kernel which results in very low/poor network
> > > performance - maybe some kind of race condition?
> 
> Ok, here's an update. After realizing that none of the ethtool offload options
> were enabled in my guest, I found that I needed to be using the -netdev option
> on the qemu command line.  Once I did that, some ethtool offload options were 
> enabled and the deadlock did not appear when I did networking between guests 
> on different machines.  However, the deadlock did appear when I did networking
> between guests on the same machine.

What does your full command line look like?  And when the networking
stops does your same receive_disabled hack make things work?

thanks,
-chris