Netdev List
 help / color / mirror / Atom feed
* [PATCH v2 0/3] virtio_net: introduce an RX queue watchdog for stuck detection
@ 2026-11-20  7:18 Longjun Tang
  2026-11-20  7:18 ` [PATCH v2 1/3] virtio: add virtqueue_get_last_used_idx() helper Longjun Tang
                   ` (3 more replies)
  0 siblings, 4 replies; 6+ messages in thread
From: Longjun Tang @ 2026-11-20  7:18 UTC (permalink / raw)
  To: mst, kuba, eperezma, jasowangio
  Cc: xuanzhuo, virtualization, netdev, tanglongjun, Longjun Tang

Hi Michael, Kuba,

I once ran into a problem where the virtio_net receive queue became
permanently stuck.

The virtio_net RX path relies on backend interrupt notifications.
The backend(vhost_net/vhost-user) appends buffers to the used ring
and then notifies the guest, relying on the hypervisor(KVM) to inject
the interrupt. On receiving it, virtio_net schedules NAPI, drains the
used ring and refills descriptors.

If the hypervisor fails to inject the interrupt(e.g. a transient KVM
failure), the guest never schedules NAPI, so it neither consumes buffers
nor returns descriptors. The backend keeps appending until the ring is full,
then stops and, with no free descriptors left, also stops sending
notifications. Both sides now wait for the other, and the RX queue is
permanently stuck.

The root cause is the lost interrupt in the hypervisor, and the proper
fix belongs there. Nevertheless, I believe virtio_net needs a way to
observe and diagnose it. Today, once a queue hangs, there is no signal
to the operator that anything is wrong: the RX path has no equivalent of
the TX watchdog (ndo_tx_timeout). This series lets virtio_net detect and
recover a stuck RX queue.


Implementation
--------------

Patch 1 exposes last_used_idx through a new helper, which is used by both
the split and the packed ring and denotes the position the driver has
last consumed. Together with the existing virtqueue_poll(), it lets a
driver ask whether the device has produced buffers that the driver has
not consumed yet ("backlog").

Patch 2 adds a per-device RX watchdog that scans all RX queues once per
second. A queue is considered stuck when, for 5 seconds, all of the
following hold simultaneously:
  - the queue has a non-zero backlog (unconsumed buffers);
  - the driver makes no consumption progress (last_used unchanged);
  - no new interrupt arrives (rq->calls unchanged).
On detection it logs a warning, rate-limited to once per timeout, and
reschedules NAPI, which drains the ring and refills the descriptors,
so the queue recovers.

Patch 3 adds a per-queue stalls statistic, exposed via ethtool -S, so
the number of stuck events can be observed per queue.

questions and comments are welcome.

Thanks!
Lange

Changes in v2:
- 1/3: fix the kdoc.
- 2/3: drop the module parameter, use a fixed 5s timeout.
- 3/3: rename rx_timeouts to stalls.

Longjun Tang (3):
  virtio: add virtqueue_get_last_used_idx() helper
  virtio_net: introduce an RX queue watchdog for stuck detection
  virtio_net: add stalls per-queue statistic

 drivers/net/virtio_net.c     | 105 +++++++++++++++++++++++++++++++++++
 drivers/virtio/virtio_ring.c |  18 +++++-
 include/linux/virtio.h       |   2 +
 3 files changed, 123 insertions(+), 2 deletions(-)

-- 
2.25.1


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2026-11-20 17:45 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20  7:18 [PATCH v2 0/3] virtio_net: introduce an RX queue watchdog for stuck detection Longjun Tang
2026-11-20  7:18 ` [PATCH v2 1/3] virtio: add virtqueue_get_last_used_idx() helper Longjun Tang
2026-11-20 17:43   ` Michael S. Tsirkin
2026-11-20  7:18 ` [PATCH v2 2/3] virtio_net: introduce an RX queue watchdog for stuck detection Longjun Tang
2026-11-20  7:18 ` [PATCH v2 3/3] virtio_net: add stalls per-queue statistic Longjun Tang
2026-11-20 17:45 ` [PATCH v2 0/3] virtio_net: introduce an RX queue watchdog for stuck detection Michael S. Tsirkin

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox