From: "Michael S. Tsirkin" <mst@redhat.com>
To: Longjun Tang <lange_tang@163.com>
Cc: kuba@kernel.org, eperezma@redhat.com, jasowangio@gmail.com,
xuanzhuo@linux.alibaba.com, virtualization@lists.linux.dev,
netdev@vger.kernel.org, tanglongjun@kylinos.cn
Subject: Re: [PATCH v2 0/3] virtio_net: introduce an RX queue watchdog for stuck detection
Date: Fri, 20 Nov 2026 12:45:22 -0500 [thread overview]
Message-ID: <20261120124403-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <20261120071826.198604-1-lange_tang@163.com>
On Fri, Nov 20, 2026 at 03:18:23PM +0800, Longjun Tang wrote:
> Hi Michael, Kuba,
>
> I once ran into a problem where the virtio_net receive queue became
> permanently stuck.
>
> The virtio_net RX path relies on backend interrupt notifications.
> The backend(vhost_net/vhost-user) appends buffers to the used ring
> and then notifies the guest, relying on the hypervisor(KVM) to inject
> the interrupt. On receiving it, virtio_net schedules NAPI, drains the
> used ring and refills descriptors.
>
> If the hypervisor fails to inject the interrupt(e.g. a transient KVM
> failure), the guest never schedules NAPI, so it neither consumes buffers
> nor returns descriptors. The backend keeps appending until the ring is full,
> then stops and, with no free descriptors left, also stops sending
> notifications. Both sides now wait for the other, and the RX queue is
> permanently stuck.
>
> The root cause is the lost interrupt in the hypervisor, and the proper
> fix belongs there. Nevertheless, I believe virtio_net needs a way to
> observe and diagnose it. Today, once a queue hangs, there is no signal
> to the operator that anything is wrong: the RX path has no equivalent of
> the TX watchdog (ndo_tx_timeout). This series lets virtio_net detect and
> recover a stuck RX queue.
>
Why virtio net specifically?
> Implementation
> --------------
>
> Patch 1 exposes last_used_idx through a new helper, which is used by both
> the split and the packed ring and denotes the position the driver has
> last consumed. Together with the existing virtqueue_poll(), it lets a
> driver ask whether the device has produced buffers that the driver has
> not consumed yet ("backlog").
>
> Patch 2 adds a per-device RX watchdog that scans all RX queues once per
> second. A queue is considered stuck when, for 5 seconds, all of the
> following hold simultaneously:
> - the queue has a non-zero backlog (unconsumed buffers);
what does unconsumed mean?
> - the driver makes no consumption progress (last_used unchanged);
> - no new interrupt arrives (rq->calls unchanged).
> On detection it logs a warning, rate-limited to once per timeout, and
> reschedules NAPI, which drains the ring and refills the descriptors,
> so the queue recovers.
>
> Patch 3 adds a per-queue stalls statistic, exposed via ethtool -S, so
> the number of stuck events can be observed per queue.
> questions and comments are welcome.
>
> Thanks!
> Lange
>
> Changes in v2:
> - 1/3: fix the kdoc.
> - 2/3: drop the module parameter, use a fixed 5s timeout.
> - 3/3: rename rx_timeouts to stalls.
>
> Longjun Tang (3):
> virtio: add virtqueue_get_last_used_idx() helper
> virtio_net: introduce an RX queue watchdog for stuck detection
> virtio_net: add stalls per-queue statistic
>
> drivers/net/virtio_net.c | 105 +++++++++++++++++++++++++++++++++++
> drivers/virtio/virtio_ring.c | 18 +++++-
> include/linux/virtio.h | 2 +
> 3 files changed, 123 insertions(+), 2 deletions(-)
>
> --
> 2.25.1
prev parent reply other threads:[~2026-11-20 17:45 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 7:18 [PATCH v2 0/3] virtio_net: introduce an RX queue watchdog for stuck detection Longjun Tang
2026-11-20 7:18 ` [PATCH v2 1/3] virtio: add virtqueue_get_last_used_idx() helper Longjun Tang
2026-11-20 17:43 ` Michael S. Tsirkin
2026-11-20 7:18 ` [PATCH v2 2/3] virtio_net: introduce an RX queue watchdog for stuck detection Longjun Tang
2026-11-20 7:18 ` [PATCH v2 3/3] virtio_net: add stalls per-queue statistic Longjun Tang
2026-11-20 17:45 ` Michael S. Tsirkin [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120124403-mutt-send-email-mst@kernel.org \
--to=mst@redhat.com \
--cc=eperezma@redhat.com \
--cc=jasowangio@gmail.com \
--cc=kuba@kernel.org \
--cc=lange_tang@163.com \
--cc=netdev@vger.kernel.org \
--cc=tanglongjun@kylinos.cn \
--cc=virtualization@lists.linux.dev \
--cc=xuanzhuo@linux.alibaba.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox