From: "Michael S. Tsirkin" <mst@redhat.com>
To: Longjun Tang <lange_tang@163.com>
Cc: kuba@kernel.org, eperezma@redhat.com, jasowangio@gmail.com,
xuanzhuo@linux.alibaba.com, virtualization@lists.linux.dev,
netdev@vger.kernel.org, tanglongjun@kylinos.cn
Subject: Re: [PATCH v2 0/3] virtio_net: introduce an RX queue watchdog for stuck detection
Date: Fri, 20 Nov 2026 12:45:22 -0500 [thread overview]
Message-ID: <20261120124403-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <20261120071826.198604-1-lange_tang@163.com>
On Fri, Nov 20, 2026 at 03:18:23PM +0800, Longjun Tang wrote:
> Hi Michael, Kuba,
>
> I once ran into a problem where the virtio_net receive queue became
> permanently stuck.
>
> The virtio_net RX path relies on backend interrupt notifications.
> The backend(vhost_net/vhost-user) appends buffers to the used ring
> and then notifies the guest, relying on the hypervisor(KVM) to inject
> the interrupt. On receiving it, virtio_net schedules NAPI, drains the
> used ring and refills descriptors.
>
> If the hypervisor fails to inject the interrupt(e.g. a transient KVM
> failure), the guest never schedules NAPI, so it neither consumes buffers
> nor returns descriptors. The backend keeps appending until the ring is full,
> then stops and, with no free descriptors left, also stops sending
> notifications. Both sides now wait for the other, and the RX queue is
> permanently stuck.
>
> The root cause is the lost interrupt in the hypervisor, and the proper
> fix belongs there. Nevertheless, I believe virtio_net needs a way to
> observe and diagnose it. Today, once a queue hangs, there is no signal
> to the operator that anything is wrong: the RX path has no equivalent of
> the TX watchdog (ndo_tx_timeout). This series lets virtio_net detect and
> recover a stuck RX queue.
>
Why virtio net specifically?
> Implementation
> --------------
>
> Patch 1 exposes last_used_idx through a new helper, which is used by both
> the split and the packed ring and denotes the position the driver has
> last consumed. Together with the existing virtqueue_poll(), it lets a
> driver ask whether the device has produced buffers that the driver has
> not consumed yet ("backlog").
>
> Patch 2 adds a per-device RX watchdog that scans all RX queues once per
> second. A queue is considered stuck when, for 5 seconds, all of the
> following hold simultaneously:
> - the queue has a non-zero backlog (unconsumed buffers);
what does unconsumed mean?
> - the driver makes no consumption progress (last_used unchanged);
> - no new interrupt arrives (rq->calls unchanged).
> On detection it logs a warning, rate-limited to once per timeout, and
> reschedules NAPI, which drains the ring and refills the descriptors,
> so the queue recovers.
>
> Patch 3 adds a per-queue stalls statistic, exposed via ethtool -S, so
> the number of stuck events can be observed per queue.
> questions and comments are welcome.
>
> Thanks!
> Lange
>
> Changes in v2:
> - 1/3: fix the kdoc.
> - 2/3: drop the module parameter, use a fixed 5s timeout.
> - 3/3: rename rx_timeouts to stalls.
>
> Longjun Tang (3):
> virtio: add virtqueue_get_last_used_idx() helper
> virtio_net: introduce an RX queue watchdog for stuck detection
> virtio_net: add stalls per-queue statistic
>
> drivers/net/virtio_net.c | 105 +++++++++++++++++++++++++++++++++++
> drivers/virtio/virtio_ring.c | 18 +++++-
> include/linux/virtio.h | 2 +
> 3 files changed, 123 insertions(+), 2 deletions(-)
>
> --
> 2.25.1
prev parent reply other threads:[~2026-11-20 17:45 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 7:18 [PATCH v2 0/3] virtio_net: introduce an RX queue watchdog for stuck detection Longjun Tang
2026-11-20 7:18 ` [PATCH v2 1/3] virtio: add virtqueue_get_last_used_idx() helper Longjun Tang
2026-11-20 17:43 ` Michael S. Tsirkin
2026-11-20 7:18 ` [PATCH v2 2/3] virtio_net: introduce an RX queue watchdog for stuck detection Longjun Tang
2026-11-20 7:18 ` [PATCH v2 3/3] virtio_net: add stalls per-queue statistic Longjun Tang
2026-11-20 17:45 ` Michael S. Tsirkin [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120124403-mutt-send-email-mst@kernel.org \
--to=mst@redhat.com \
--cc=eperezma@redhat.com \
--cc=jasowangio@gmail.com \
--cc=kuba@kernel.org \
--cc=lange_tang@163.com \
--cc=netdev@vger.kernel.org \
--cc=tanglongjun@kylinos.cn \
--cc=virtualization@lists.linux.dev \
--cc=xuanzhuo@linux.alibaba.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.