From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mark McLoughlin Subject: Re: virtio_net hang Date: Wed, 19 Nov 2008 19:03:09 +0000 Message-ID: <1227121389.3698.136.camel@blaa> References: <20081113122709.GB14254@easter-eggs.com> <1226589153.19068.7.camel@blaa> <20081113152452.GI14254@easter-eggs.com> <20081114092339.GC11961@easter-eggs.com> <1226687204.9332.113.camel@blaa> <20081118183756.GP1897@easter-eggs.com> <1227100432.3698.47.camel@blaa> Reply-To: Mark McLoughlin Mime-Version: 1.0 Content-Type: text/plain Content-Transfer-Encoding: 7bit Cc: kvm@vger.kernel.org To: Emmanuel Lacour Return-path: Received: from mx2.redhat.com ([66.187.237.31]:43687 "EHLO mx2.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752377AbYKSTEW (ORCPT ); Wed, 19 Nov 2008 14:04:22 -0500 In-Reply-To: <1227100432.3698.47.camel@blaa> Sender: kvm-owner@vger.kernel.org List-ID: On Wed, 2008-11-19 at 13:13 +0000, Mark McLoughlin wrote: > On Tue, 2008-11-18 at 19:37 +0100, Emmanuel Lacour wrote: > > On Fri, Nov 14, 2008 at 06:26:44PM +0000, Mark McLoughlin wrote: > > > > > > Right, the tap device tx queue is full because kvm-userspace isn't > > > reading packets from it. > > > > > > This could be because kvm-userspace has just stopped noticing that > > > there's data available from the tapfd or because virtio_net in the guest > > > has stopped noticing that packets are available in the ring. > > > > > > One thing you could easily check is whether: > > > > > > ip link set eth0 down > > > ip link set eth0 up > > > > > > in the host is sufficient to fix it? If it is, then it points to a guest > > > driver issue. > > > > > > > I made the test, putting link down then up fix it. > > Thanks, that's very interesting. > > Since bringing the interface up and down basically just causes the > driver to re-schedule itself with NAPI, all I can see as a possibility > is that we somehow (e.g. a race condition) had gotten ourselves into a > state where we have rx ring interrupts disabled and we're not scheduled > with NAPI. > > We synchronise around the NAPI_STATE_SCHED bit with atomic operations > and all the logic looks correct ... so I'm stumped, really. I had a look at Emmanuel's strace log and it shows that qemu isn't selecting on the tapfd, presumably because virtio_net_can_receive() sees that we've exhausted all available receive buffers. When qemu does poll the tapfd (after an ifdown/ifup in the guest), there are a load of packets waiting in the queue and things proceed as normal. That still jives with the theory that we're somehow getting into a state where NAPI polling is de-scheduled while guest rx interrupts are also disabled. > Is it possible for you to try a newer guest kernel? If you can try a newer kernel, or even try some debugging patches, that would help a lot. Cheers, Mark.