Re: virtio_net hang

[Mark McLoughlin <markmc@redhat.com>]

On Wed, 2008-11-19 at 13:13 +0000, Mark McLoughlin wrote:
> On Tue, 2008-11-18 at 19:37 +0100, Emmanuel Lacour wrote:
> > On Fri, Nov 14, 2008 at 06:26:44PM +0000, Mark McLoughlin wrote:
> > > 
> > > Right, the tap device tx queue is full because kvm-userspace isn't
> > > reading packets from it.
> > > 
> > > This could be because kvm-userspace has just stopped noticing that
> > > there's data available from the tapfd or because virtio_net in the guest
> > > has stopped noticing that packets are available in the ring.
> > > 
> > > One thing you could easily check is whether:
> > > 
> > >   ip link set eth0 down
> > >   ip link set eth0 up
> > > 
> > > in the host is sufficient to fix it? If it is, then it points to a guest
> > > driver issue.
> > > 
> > 
> > I made the test, putting link down then up fix it.
> 
> Thanks, that's very interesting.
> 
> Since bringing the interface up and down basically just causes the
> driver to re-schedule itself with NAPI, all I can see as a possibility
> is that we somehow (e.g. a race condition) had gotten ourselves into a
> state where we have rx ring interrupts disabled and we're not scheduled
> with NAPI.
> 
> We synchronise around the NAPI_STATE_SCHED bit with atomic operations
> and all the logic looks correct ... so I'm stumped, really.

I had a look at Emmanuel's strace log and it shows that qemu isn't
selecting on the tapfd, presumably because virtio_net_can_receive() sees
that we've exhausted all available receive buffers.

When qemu does poll the tapfd (after an ifdown/ifup in the guest), there
are a load of packets waiting in the queue and things proceed as normal.

That still jives with the theory that we're somehow getting into a state
where NAPI polling is de-scheduled while guest rx interrupts are also
disabled.

> Is it possible for you to try a newer guest kernel?

If you can try a newer kernel, or even try some debugging patches, that
would help a lot.

Cheers,
Mark.