All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v2 0/3] virtio_net: introduce an RX queue watchdog for stuck detection
@ 2026-11-20  7:18 Longjun Tang
  2026-11-20  7:18 ` [PATCH v2 1/3] virtio: add virtqueue_get_last_used_idx() helper Longjun Tang
                   ` (3 more replies)
  0 siblings, 4 replies; 9+ messages in thread
From: Longjun Tang @ 2026-11-20  7:18 UTC (permalink / raw)
  To: mst, kuba, eperezma, jasowangio
  Cc: xuanzhuo, virtualization, netdev, tanglongjun, Longjun Tang

Hi Michael, Kuba,

I once ran into a problem where the virtio_net receive queue became
permanently stuck.

The virtio_net RX path relies on backend interrupt notifications.
The backend(vhost_net/vhost-user) appends buffers to the used ring
and then notifies the guest, relying on the hypervisor(KVM) to inject
the interrupt. On receiving it, virtio_net schedules NAPI, drains the
used ring and refills descriptors.

If the hypervisor fails to inject the interrupt(e.g. a transient KVM
failure), the guest never schedules NAPI, so it neither consumes buffers
nor returns descriptors. The backend keeps appending until the ring is full,
then stops and, with no free descriptors left, also stops sending
notifications. Both sides now wait for the other, and the RX queue is
permanently stuck.

The root cause is the lost interrupt in the hypervisor, and the proper
fix belongs there. Nevertheless, I believe virtio_net needs a way to
observe and diagnose it. Today, once a queue hangs, there is no signal
to the operator that anything is wrong: the RX path has no equivalent of
the TX watchdog (ndo_tx_timeout). This series lets virtio_net detect and
recover a stuck RX queue.


Implementation
--------------

Patch 1 exposes last_used_idx through a new helper, which is used by both
the split and the packed ring and denotes the position the driver has
last consumed. Together with the existing virtqueue_poll(), it lets a
driver ask whether the device has produced buffers that the driver has
not consumed yet ("backlog").

Patch 2 adds a per-device RX watchdog that scans all RX queues once per
second. A queue is considered stuck when, for 5 seconds, all of the
following hold simultaneously:
  - the queue has a non-zero backlog (unconsumed buffers);
  - the driver makes no consumption progress (last_used unchanged);
  - no new interrupt arrives (rq->calls unchanged).
On detection it logs a warning, rate-limited to once per timeout, and
reschedules NAPI, which drains the ring and refills the descriptors,
so the queue recovers.

Patch 3 adds a per-queue stalls statistic, exposed via ethtool -S, so
the number of stuck events can be observed per queue.

questions and comments are welcome.

Thanks!
Lange

Changes in v2:
- 1/3: fix the kdoc.
- 2/3: drop the module parameter, use a fixed 5s timeout.
- 3/3: rename rx_timeouts to stalls.

Longjun Tang (3):
  virtio: add virtqueue_get_last_used_idx() helper
  virtio_net: introduce an RX queue watchdog for stuck detection
  virtio_net: add stalls per-queue statistic

 drivers/net/virtio_net.c     | 105 +++++++++++++++++++++++++++++++++++
 drivers/virtio/virtio_ring.c |  18 +++++-
 include/linux/virtio.h       |   2 +
 3 files changed, 123 insertions(+), 2 deletions(-)

-- 
2.25.1


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 1/3] virtio: add virtqueue_get_last_used_idx() helper
  2026-11-20  7:18 [PATCH v2 0/3] virtio_net: introduce an RX queue watchdog for stuck detection Longjun Tang
@ 2026-11-20  7:18 ` Longjun Tang
  2026-11-20 17:43   ` Michael S. Tsirkin
  2026-11-21  7:18   ` sashiko-bot
  2026-11-20  7:18 ` [PATCH v2 2/3] virtio_net: introduce an RX queue watchdog for stuck detection Longjun Tang
                   ` (2 subsequent siblings)
  3 siblings, 2 replies; 9+ messages in thread
From: Longjun Tang @ 2026-11-20  7:18 UTC (permalink / raw)
  To: mst, kuba, eperezma, jasowangio
  Cc: xuanzhuo, virtualization, netdev, tanglongjun

From: Longjun Tang <tanglongjun@kylinos.cn>

Expose last_used_idx through a new helper, which is used by both the split
and the packed ring and denotes the position the driver has last consumed.
It is useful for drivers that need to detect whether the device has
produced buffers that have not been consumed yet, by passing the returned
value to virtqueue_poll().

The split ring now writes the field with WRITE_ONCE() (the packed ring
already did) to document that the new reader can run concurrently with
the poll path.

Signed-off-by: Longjun Tang <tanglongjun@kylinos.cn>
---
 drivers/virtio/virtio_ring.c | 18 ++++++++++++++++--
 include/linux/virtio.h       |  2 ++
 2 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index f683a2d6a200..e8af6d5c6432 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -1005,7 +1005,7 @@ static void *virtqueue_get_buf_ctx_split(struct vring_virtqueue *vq,
 	/* detach_buf_split clears data, so grab it now. */
 	ret = vq->split.desc_state[i].data;
 	detach_buf_split(vq, i, ctx);
-	vq->last_used_idx++;
+	WRITE_ONCE(vq->last_used_idx, vq->last_used_idx + 1);
 	/* If we expect an interrupt for the next entry, tell host
 	 * by writing event index and flush out the write before
 	 * the read in the next get_buf call. */
@@ -1072,7 +1072,7 @@ static void *virtqueue_get_buf_ctx_split_in_order(struct vring_virtqueue *vq,
 	ret = vq->split.desc_state[last_used].data;
 	detach_buf_split_in_order(vq, last_used, ctx);
 
-	vq->last_used_idx++;
+	WRITE_ONCE(vq->last_used_idx, vq->last_used_idx + 1);
 	vq->last_used += (vq->vq.num_free - num_free);
 	/* If we expect an interrupt for the next entry, tell host
 	 * by writing event index and flush out the write before
@@ -3236,6 +3236,20 @@ bool virtqueue_poll(struct virtqueue *_vq, unsigned int last_used_idx)
 }
 EXPORT_SYMBOL_GPL(virtqueue_poll);
 
+/**
+ * virtqueue_get_last_used_idx - get the ring's last consumed position
+ * @_vq: the struct virtqueue we're talking about.
+ *
+ * Returns: the vring_virtqueue's last_used_idx.
+ */
+unsigned int virtqueue_get_last_used_idx(const struct virtqueue *_vq)
+{
+	const struct vring_virtqueue *vq = to_vvq(_vq);
+
+	return READ_ONCE(vq->last_used_idx);
+}
+EXPORT_SYMBOL_GPL(virtqueue_get_last_used_idx);
+
 /**
  * virtqueue_enable_cb - restart callbacks after disable_cb.
  * @_vq: the struct virtqueue we're talking about.
diff --git a/include/linux/virtio.h b/include/linux/virtio.h
index f923e42cfd01..1aed4d9235c8 100644
--- a/include/linux/virtio.h
+++ b/include/linux/virtio.h
@@ -112,6 +112,8 @@ unsigned virtqueue_enable_cb_prepare(struct virtqueue *vq);
 
 bool virtqueue_poll(struct virtqueue *vq, unsigned);
 
+unsigned int virtqueue_get_last_used_idx(const struct virtqueue *vq);
+
 bool virtqueue_enable_cb_delayed(struct virtqueue *vq);
 
 void *virtqueue_detach_unused_buf(struct virtqueue *vq);
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH v2 2/3] virtio_net: introduce an RX queue watchdog for stuck detection
  2026-11-20  7:18 [PATCH v2 0/3] virtio_net: introduce an RX queue watchdog for stuck detection Longjun Tang
  2026-11-20  7:18 ` [PATCH v2 1/3] virtio: add virtqueue_get_last_used_idx() helper Longjun Tang
@ 2026-11-20  7:18 ` Longjun Tang
  2026-11-21  7:18   ` sashiko-bot
  2026-11-20  7:18 ` [PATCH v2 3/3] virtio_net: add stalls per-queue statistic Longjun Tang
  2026-11-20 17:45 ` [PATCH v2 0/3] virtio_net: introduce an RX queue watchdog for stuck detection Michael S. Tsirkin
  3 siblings, 1 reply; 9+ messages in thread
From: Longjun Tang @ 2026-11-20  7:18 UTC (permalink / raw)
  To: mst, kuba, eperezma, jasowangio
  Cc: xuanzhuo, virtualization, netdev, tanglongjun

From: Longjun Tang <tanglongjun@kylinos.cn>

The backend raises an interrupt after adding buffers to the used ring,
and the hypervisor injects it. If one is lost (e.g., a transient KVM
failure), NAPI will not be scheduled and the driver stops consuming
buffers. The ring eventually fills up, the backend stops notifying, and
the queue is left permanently stuck.

Add a per-device watchdog that scans the RX queues once per second and
detects a queue that has a non-zero backlog while making no consumption
progress and receiving no new interrupt for 5 seconds. On detection,
reschedule NAPI.

Signed-off-by: Longjun Tang <tanglongjun@kylinos.cn>
---
 drivers/net/virtio_net.c | 97 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 97 insertions(+)

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index ed20e76fbc4c..15191b841a60 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -36,6 +36,9 @@ module_param(csum, bool, 0444);
 module_param(gso, bool, 0444);
 module_param(napi_tx, bool, 0644);
 
+/* RX queue stall timeout in seconds. */
+#define VIRTNET_RX_WATCHDOG_TIMEO 5
+
 #define VIRTIO_OFFLOAD_MAP_MIN	46
 #define VIRTIO_OFFLOAD_MAP_MAX	47
 #define VIRTIO_FEATURES_MAP_MIN	65
@@ -318,6 +321,13 @@ struct send_queue {
 	dma_addr_t xsk_hdr_dma_addr;
 };
 
+struct virtnet_rq_watchdog {
+	u16 last_used;
+	u16 calls;
+	unsigned long jiffies;
+	bool stalled;
+};
+
 /* Internal representation of a receive virtqueue */
 struct receive_queue {
 	/* Virtqueue associated with this receive_queue */
@@ -332,6 +342,9 @@ struct receive_queue {
 	/* The number of rx notifications */
 	u16 calls;
 
+	/* RX watchdog state. */
+	struct virtnet_rq_watchdog watchdog;
+
 	/* Is dynamic interrupt moderation enabled? */
 	bool dim_enabled;
 
@@ -443,6 +456,9 @@ struct virtnet_info {
 	/* Work struct for setting rx mode */
 	struct work_struct rx_mode_work;
 
+	/* RX watchdog timer for stuck detection */
+	struct timer_list rx_watchdog;
+
 	/* Does the affinity hint is set for virtqueues? */
 	bool affinity_hint_set;
 
@@ -3067,6 +3083,74 @@ static int virtnet_poll(struct napi_struct *napi, int budget)
 	return received;
 }
 
+/*
+ * An RX queue is stalled if it has a backlog but makes no progress and
+ * gets no new interrupt for VIRTNET_RX_WATCHDOG_TIMEO seconds, usually
+ * because an interrupt was lost.  Mark it stalled and reschedule NAPI to
+ * drain the ring and refill the buffers.
+ */
+static void virtnet_rx_watchdog(struct timer_list *t)
+{
+	struct virtnet_info *vi = timer_container_of(vi, t, rx_watchdog);
+	unsigned long timeout = VIRTNET_RX_WATCHDOG_TIMEO * HZ;
+	int i;
+
+	for (i = 0; i < vi->curr_queue_pairs; i++) {
+		struct receive_queue *rq = &vi->rq[i];
+		u16 last_used = virtqueue_get_last_used_idx(rq->vq);
+		u16 calls = rq->calls;
+		bool backlog = virtqueue_poll(rq->vq, last_used);
+		bool progress = last_used != rq->watchdog.last_used;
+
+		if (!backlog || progress || calls != rq->watchdog.calls) {
+			rq->watchdog.last_used = last_used;
+			rq->watchdog.calls = calls;
+			rq->watchdog.jiffies = jiffies;
+			continue;
+		}
+
+		if (time_after(jiffies, rq->watchdog.jiffies + timeout)) {
+			unsigned int stall_ms =
+				jiffies_to_msecs(jiffies - rq->watchdog.jiffies);
+
+			netdev_warn(vi->dev,
+				    "RX queue %u stalled for %u ms, rescheduling NAPI\n",
+				    i, stall_ms);
+
+			rq->watchdog.stalled = true;
+			virtqueue_napi_schedule(&rq->napi, rq->vq);
+
+			/* Rate-limit to one event per timeout. */
+			rq->watchdog.jiffies = jiffies;
+		}
+	}
+
+	mod_timer(&vi->rx_watchdog, jiffies + HZ);
+}
+
+static void virtnet_rx_watchdog_init_queue(struct receive_queue *rq)
+{
+	rq->watchdog.last_used = virtqueue_get_last_used_idx(rq->vq);
+	rq->watchdog.calls = rq->calls;
+	rq->watchdog.jiffies = jiffies;
+	rq->watchdog.stalled = false;
+}
+
+static void virtnet_rx_watchdog_start(struct virtnet_info *vi)
+{
+	int i;
+
+	for (i = 0; i < vi->curr_queue_pairs; i++)
+		virtnet_rx_watchdog_init_queue(&vi->rq[i]);
+
+	mod_timer(&vi->rx_watchdog, jiffies + HZ);
+}
+
+static void virtnet_rx_watchdog_stop(struct virtnet_info *vi)
+{
+	timer_delete_sync(&vi->rx_watchdog);
+}
+
 static void virtnet_disable_queue_pair(struct virtnet_info *vi, int qp_index)
 {
 	virtnet_napi_tx_disable(&vi->sq[qp_index]);
@@ -3230,6 +3314,8 @@ static int virtnet_open(struct net_device *dev)
 		netif_carrier_on(dev);
 	}
 
+	virtnet_rx_watchdog_start(vi);
+
 	return 0;
 
 err_enable_qp:
@@ -3759,6 +3845,7 @@ static int virtnet_set_queues(struct virtnet_info *vi, u16 queue_pairs)
 	struct virtio_net_rss_config_trailer old_rss_trailer;
 	struct net_device *dev = vi->dev;
 	struct scatterlist sg;
+	u16 old_queue_pairs = vi->curr_queue_pairs;
 
 	if (!vi->has_cvq || !virtio_has_feature(vi->vdev, VIRTIO_NET_F_MQ))
 		return 0;
@@ -3816,7 +3903,14 @@ static int virtnet_set_queues(struct virtnet_info *vi, u16 queue_pairs)
 	if (vi->has_rss)
 		vi->rss_trailer.max_tx_vq = cpu_to_le16(queue_pairs);
 succ:
+	if (dev->flags & IFF_UP) {
+		/* Init new queues */
+		for (int i = old_queue_pairs; i < queue_pairs; i++)
+			virtnet_rx_watchdog_init_queue(&vi->rq[i]);
+	}
+
 	vi->curr_queue_pairs = queue_pairs;
+
 	if (dev->flags & IFF_UP) {
 		local_bh_disable();
 		for (int i = 0; i < vi->curr_queue_pairs; ++i)
@@ -3832,6 +3926,8 @@ static int virtnet_close(struct net_device *dev)
 	struct virtnet_info *vi = netdev_priv(dev);
 	int i;
 
+	virtnet_rx_watchdog_stop(vi);
+
 	/* Prevent the config change callback from changing carrier
 	 * after close
 	 */
@@ -6900,6 +6996,7 @@ static int virtnet_probe(struct virtio_device *vdev)
 	INIT_WORK(&vi->config_work, virtnet_config_changed_work);
 	INIT_WORK(&vi->rx_mode_work, virtnet_rx_mode_work);
 	disable_work(&vi->rx_mode_work);
+	timer_setup(&vi->rx_watchdog, virtnet_rx_watchdog, 0);
 
 	if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF)) {
 		vi->mergeable_rx_bufs = true;
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH v2 3/3] virtio_net: add stalls per-queue statistic
  2026-11-20  7:18 [PATCH v2 0/3] virtio_net: introduce an RX queue watchdog for stuck detection Longjun Tang
  2026-11-20  7:18 ` [PATCH v2 1/3] virtio: add virtqueue_get_last_used_idx() helper Longjun Tang
  2026-11-20  7:18 ` [PATCH v2 2/3] virtio_net: introduce an RX queue watchdog for stuck detection Longjun Tang
@ 2026-11-20  7:18 ` Longjun Tang
  2026-11-21  7:18   ` sashiko-bot
  2026-11-20 17:45 ` [PATCH v2 0/3] virtio_net: introduce an RX queue watchdog for stuck detection Michael S. Tsirkin
  3 siblings, 1 reply; 9+ messages in thread
From: Longjun Tang @ 2026-11-20  7:18 UTC (permalink / raw)
  To: mst, kuba, eperezma, jasowangio
  Cc: xuanzhuo, virtualization, netdev, tanglongjun

From: Longjun Tang <tanglongjun@kylinos.cn>

Add a stalls field to the per-queue RX statistics and expose it via
ethtool. Whenever an RX queue gets stuck, the NAPI poll increments
stalls, so it counts RX queue stall events.

Signed-off-by: Longjun Tang <tanglongjun@kylinos.cn>
---
 drivers/net/virtio_net.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 15191b841a60..61d51c5a50ea 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -134,6 +134,7 @@ struct virtnet_rq_stats {
 	u64_stats_t xdp_redirects;
 	u64_stats_t xdp_drops;
 	u64_stats_t kicks;
+	u64_stats_t stalls;
 };
 
 #define VIRTNET_SQ_STAT(name, m) {name, offsetof(struct virtnet_sq_stats, m), -1}
@@ -167,6 +168,7 @@ static const struct virtnet_stat_desc virtnet_rq_stats_desc[] = {
 	VIRTNET_RQ_STAT("xdp_redirects", xdp_redirects),
 	VIRTNET_RQ_STAT("xdp_drops",     xdp_drops),
 	VIRTNET_RQ_STAT("kicks",         kicks),
+	VIRTNET_RQ_STAT("stalls",   stalls),
 };
 
 static const struct virtnet_stat_desc virtnet_sq_stats_desc_qstat[] = {
@@ -2982,6 +2984,12 @@ static int virtnet_receive(struct receive_queue *rq, int budget,
 		u64_stats_add(item, u64_stats_read(src));
 	}
 
+	/* Count a stall recovered by this poll. */
+	if (rq->watchdog.stalled) {
+		u64_stats_inc(&rq->stats.stalls);
+		rq->watchdog.stalled = false;
+	}
+
 	u64_stats_add(&rq->stats.packets, u64_stats_read(&stats.packets));
 	u64_stats_add(&rq->stats.bytes, u64_stats_read(&stats.bytes));
 	u64_stats_add(&rq->stats.alloc_fail, u64_stats_read(&stats.alloc_fail));
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 9+ messages in thread

* Re: [PATCH v2 1/3] virtio: add virtqueue_get_last_used_idx() helper
  2026-11-20  7:18 ` [PATCH v2 1/3] virtio: add virtqueue_get_last_used_idx() helper Longjun Tang
@ 2026-11-20 17:43   ` Michael S. Tsirkin
  2026-11-21  7:18   ` sashiko-bot
  1 sibling, 0 replies; 9+ messages in thread
From: Michael S. Tsirkin @ 2026-11-20 17:43 UTC (permalink / raw)
  To: Longjun Tang
  Cc: kuba, eperezma, jasowangio, xuanzhuo, virtualization, netdev,
	tanglongjun

On Fri, Nov 20, 2026 at 03:18:24PM +0800, Longjun Tang wrote:
> From: Longjun Tang <tanglongjun@kylinos.cn>
> 
> Expose last_used_idx through a new helper, which is used by both the split
> and the packed ring and denotes the position the driver has last consumed.
> It is useful for drivers that need to detect whether the device has
> produced buffers that have not been consumed yet, by passing the returned
> value to virtqueue_poll().
> 
> The split ring now writes the field with WRITE_ONCE() (the packed ring
> already did) to document that the new reader can run concurrently with
> the poll path.
> 
> Signed-off-by: Longjun Tang <tanglongjun@kylinos.cn>



This kind of low level API is not reasonable in virtio.h



> ---
>  drivers/virtio/virtio_ring.c | 18 ++++++++++++++++--
>  include/linux/virtio.h       |  2 ++
>  2 files changed, 18 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
> index f683a2d6a200..e8af6d5c6432 100644
> --- a/drivers/virtio/virtio_ring.c
> +++ b/drivers/virtio/virtio_ring.c
> @@ -1005,7 +1005,7 @@ static void *virtqueue_get_buf_ctx_split(struct vring_virtqueue *vq,
>  	/* detach_buf_split clears data, so grab it now. */
>  	ret = vq->split.desc_state[i].data;
>  	detach_buf_split(vq, i, ctx);
> -	vq->last_used_idx++;
> +	WRITE_ONCE(vq->last_used_idx, vq->last_used_idx + 1);
>  	/* If we expect an interrupt for the next entry, tell host
>  	 * by writing event index and flush out the write before
>  	 * the read in the next get_buf call. */
> @@ -1072,7 +1072,7 @@ static void *virtqueue_get_buf_ctx_split_in_order(struct vring_virtqueue *vq,
>  	ret = vq->split.desc_state[last_used].data;
>  	detach_buf_split_in_order(vq, last_used, ctx);
>  
> -	vq->last_used_idx++;
> +	WRITE_ONCE(vq->last_used_idx, vq->last_used_idx + 1);
>  	vq->last_used += (vq->vq.num_free - num_free);
>  	/* If we expect an interrupt for the next entry, tell host
>  	 * by writing event index and flush out the write before
> @@ -3236,6 +3236,20 @@ bool virtqueue_poll(struct virtqueue *_vq, unsigned int last_used_idx)
>  }
>  EXPORT_SYMBOL_GPL(virtqueue_poll);
>  
> +/**
> + * virtqueue_get_last_used_idx - get the ring's last consumed position
> + * @_vq: the struct virtqueue we're talking about.
> + *
> + * Returns: the vring_virtqueue's last_used_idx.
> + */
> +unsigned int virtqueue_get_last_used_idx(const struct virtqueue *_vq)
> +{
> +	const struct vring_virtqueue *vq = to_vvq(_vq);
> +
> +	return READ_ONCE(vq->last_used_idx);
> +}
> +EXPORT_SYMBOL_GPL(virtqueue_get_last_used_idx);
> +
>  /**
>   * virtqueue_enable_cb - restart callbacks after disable_cb.
>   * @_vq: the struct virtqueue we're talking about.
> diff --git a/include/linux/virtio.h b/include/linux/virtio.h
> index f923e42cfd01..1aed4d9235c8 100644
> --- a/include/linux/virtio.h
> +++ b/include/linux/virtio.h
> @@ -112,6 +112,8 @@ unsigned virtqueue_enable_cb_prepare(struct virtqueue *vq);
>  
>  bool virtqueue_poll(struct virtqueue *vq, unsigned);
>  
> +unsigned int virtqueue_get_last_used_idx(const struct virtqueue *vq);
> +
>  bool virtqueue_enable_cb_delayed(struct virtqueue *vq);
>  
>  void *virtqueue_detach_unused_buf(struct virtqueue *vq);
> -- 
> 2.25.1


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2 0/3] virtio_net: introduce an RX queue watchdog for stuck detection
  2026-11-20  7:18 [PATCH v2 0/3] virtio_net: introduce an RX queue watchdog for stuck detection Longjun Tang
                   ` (2 preceding siblings ...)
  2026-11-20  7:18 ` [PATCH v2 3/3] virtio_net: add stalls per-queue statistic Longjun Tang
@ 2026-11-20 17:45 ` Michael S. Tsirkin
  3 siblings, 0 replies; 9+ messages in thread
From: Michael S. Tsirkin @ 2026-11-20 17:45 UTC (permalink / raw)
  To: Longjun Tang
  Cc: kuba, eperezma, jasowangio, xuanzhuo, virtualization, netdev,
	tanglongjun

On Fri, Nov 20, 2026 at 03:18:23PM +0800, Longjun Tang wrote:
> Hi Michael, Kuba,
> 
> I once ran into a problem where the virtio_net receive queue became
> permanently stuck.
> 
> The virtio_net RX path relies on backend interrupt notifications.
> The backend(vhost_net/vhost-user) appends buffers to the used ring
> and then notifies the guest, relying on the hypervisor(KVM) to inject
> the interrupt. On receiving it, virtio_net schedules NAPI, drains the
> used ring and refills descriptors.
> 
> If the hypervisor fails to inject the interrupt(e.g. a transient KVM
> failure), the guest never schedules NAPI, so it neither consumes buffers
> nor returns descriptors. The backend keeps appending until the ring is full,
> then stops and, with no free descriptors left, also stops sending
> notifications. Both sides now wait for the other, and the RX queue is
> permanently stuck.
> 
> The root cause is the lost interrupt in the hypervisor, and the proper
> fix belongs there. Nevertheless, I believe virtio_net needs a way to
> observe and diagnose it. Today, once a queue hangs, there is no signal
> to the operator that anything is wrong: the RX path has no equivalent of
> the TX watchdog (ndo_tx_timeout). This series lets virtio_net detect and
> recover a stuck RX queue.
> 

Why virtio net specifically?


> Implementation
> --------------
> 
> Patch 1 exposes last_used_idx through a new helper, which is used by both
> the split and the packed ring and denotes the position the driver has
> last consumed. Together with the existing virtqueue_poll(), it lets a
> driver ask whether the device has produced buffers that the driver has
> not consumed yet ("backlog").
> 
> Patch 2 adds a per-device RX watchdog that scans all RX queues once per
> second. A queue is considered stuck when, for 5 seconds, all of the
> following hold simultaneously:
>   - the queue has a non-zero backlog (unconsumed buffers);


what does unconsumed mean?

>   - the driver makes no consumption progress (last_used unchanged);
>   - no new interrupt arrives (rq->calls unchanged).
> On detection it logs a warning, rate-limited to once per timeout, and
> reschedules NAPI, which drains the ring and refills the descriptors,
> so the queue recovers.
> 
> Patch 3 adds a per-queue stalls statistic, exposed via ethtool -S, so
> the number of stuck events can be observed per queue.



> questions and comments are welcome.
> 
> Thanks!
> Lange
> 
> Changes in v2:
> - 1/3: fix the kdoc.
> - 2/3: drop the module parameter, use a fixed 5s timeout.
> - 3/3: rename rx_timeouts to stalls.
> 
> Longjun Tang (3):
>   virtio: add virtqueue_get_last_used_idx() helper
>   virtio_net: introduce an RX queue watchdog for stuck detection
>   virtio_net: add stalls per-queue statistic
> 
>  drivers/net/virtio_net.c     | 105 +++++++++++++++++++++++++++++++++++
>  drivers/virtio/virtio_ring.c |  18 +++++-
>  include/linux/virtio.h       |   2 +
>  3 files changed, 123 insertions(+), 2 deletions(-)
> 
> -- 
> 2.25.1


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2 1/3] virtio: add virtqueue_get_last_used_idx() helper
  2026-11-20  7:18 ` [PATCH v2 1/3] virtio: add virtqueue_get_last_used_idx() helper Longjun Tang
  2026-11-20 17:43   ` Michael S. Tsirkin
@ 2026-11-21  7:18   ` sashiko-bot
  1 sibling, 0 replies; 9+ messages in thread
From: sashiko-bot @ 2026-11-21  7:18 UTC (permalink / raw)
  To: Longjun Tang; +Cc: virtualization, Michael S. Tsirkin, Eugenio Perez

Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [Medium] virtio_ring: 4-byte WRITE_ONCE() on u16 last_used_idx in
  tools/virtio
- [Medium] virtio_ring: virtqueue_get_last_used_idx() misses in-order
  batch buffers
- [Medium] virtio: virtqueue_get_last_used_idx() ignores maintainer API
  objection
--

commit e3f548bc724c634761028f08e7565898a68423a0
Author: Longjun Tang <tanglongjun@kylinos.cn>

virtio: add virtqueue_get_last_used_idx() helper

This commit adds a virtqueue_get_last_used_idx() helper to expose
vring_virtqueue's last_used_idx and converts the split-ring updates of
last_used_idx to use WRITE_ONCE().

> diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
> index f683a2d6a2002..e8af6d5c6432b 100644
> --- a/drivers/virtio/virtio_ring.c
> +++ b/drivers/virtio/virtio_ring.c
> @@ -1005,7 +1005,7 @@ static void *virtqueue_get_buf_ctx_split(struct vring_virtqueue *vq,
>  	/* detach_buf_split clears data, so grab it now. */
>  	ret = vq->split.desc_state[i].data;
>  	detach_buf_split(vq, i, ctx);
> -	vq->last_used_idx++;
> +	WRITE_ONCE(vq->last_used_idx, vq->last_used_idx + 1);
>  	/* If we expect an interrupt for the next entry, tell host
>  	 * by writing event index and flush out the write before
>  	 * the read in the next get_buf call. */
> @@ -1072,7 +1072,7 @@ static void *virtqueue_get_buf_ctx_split_in_order(struct vring_virtqueue *vq,
>  	ret = vq->split.desc_state[last_used].data;
>  	detach_buf_split_in_order(vq, last_used, ctx);
>  
> -	vq->last_used_idx++;
> +	WRITE_ONCE(vq->last_used_idx, vq->last_used_idx + 1);
>  	vq->last_used += (vq->vq.num_free - num_free);

[Severity: Medium]
Does WRITE_ONCE(vq->last_used_idx, vq->last_used_idx + 1) perform a 4-byte
store into the 2-byte u16 last_used_idx field when compiled under
tools/virtio/?

Because vq->last_used_idx + 1 undergoes integer promotion to int, the
userspace WRITE_ONCE() stub in tools/virtio/linux/compiler.h casts the
destination pointer to typeof(val) * (4 bytes):

tools/virtio/linux/compiler.h:
#define WRITE_ONCE(var, val) \
	(*((volatile typeof(val) *)(&(var))) = (val))

In struct vring_virtqueue, u16 last_used_idx is immediately followed by
u16 last_used, so a 4-byte write clobbers vq->last_used right before
virtqueue_get_buf_ctx_split_in_order() increments it (and on big-endian
hosts writes zero into vq->last_used_idx in virtqueue_get_buf_ctx_split()).
Casting the value to u16 (or fixing the tools/virtio/ macro to use
typeof(var)) avoids the out-of-bounds write.

> @@ -3236,6 +3236,20 @@ bool virtqueue_poll(struct virtqueue *_vq, unsigned int last_used_idx)
>  }
>  EXPORT_SYMBOL_GPL(virtqueue_poll);
>  
> +/**
> + * virtqueue_get_last_used_idx - get the ring's last consumed position
> + * @_vq: the struct virtqueue we're talking about.
> + *
> + * Returns: the vring_virtqueue's last_used_idx.
> + */
> +unsigned int virtqueue_get_last_used_idx(const struct virtqueue *_vq)
> +{
> +	const struct vring_virtqueue *vq = to_vvq(_vq);
> +
> +	return READ_ONCE(vq->last_used_idx);
> +}
> +EXPORT_SYMBOL_GPL(virtqueue_get_last_used_idx);

[Severity: Medium]
Can passing virtqueue_get_last_used_idx() to virtqueue_poll() falsely
report no pending used buffers when VIRTIO_F_IN_ORDER is negotiated?

When a device completes a multi-buffer batch in order, it only writes the
batch's single used entry (or the first descriptor on a packed ring), and
virtqueue_get_buf_ctx_{split,packed}_in_order() caches the end of the batch
in vq->batch_last.id while advancing vq->last_used_idx on each detached
buffer.  If the driver stops consuming mid-batch (for example, when NAPI
exhausts its budget), vq->batch_last.id is not UINT_MAX, yet
virtqueue_poll() only inspects the ring at last_used_idx where the device
wrote no descriptor flag or used entry, returning false while unconsumed
buffers remain in the batch:

drivers/virtio/virtio_ring.c:more_used_packed_in_order() {
	if (vq->batch_last.id != UINT_MAX)
		return true;

	return virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx));
}

[Severity: Medium]
As reported by Michael S. Tsirkin on v1 [1], is exposing last_used_idx
through virtqueue_get_last_used_idx() an appropriate public virtio API given
that packed virtqueues do not have a used ring and encode both the
descriptor index and VRING_PACKED_EVENT_F_WRAP_CTR wrap counter in
vq->last_used_idx?

[1] https://lore.kernel.org/r/20261112174936-mutt-send-email-mst@kernel.org

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261120071826.198604-1-lange_tang@163.com?part=1

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2 2/3] virtio_net: introduce an RX queue watchdog for stuck detection
  2026-11-20  7:18 ` [PATCH v2 2/3] virtio_net: introduce an RX queue watchdog for stuck detection Longjun Tang
@ 2026-11-21  7:18   ` sashiko-bot
  0 siblings, 0 replies; 9+ messages in thread
From: sashiko-bot @ 2026-11-21  7:18 UTC (permalink / raw)
  To: Longjun Tang; +Cc: virtualization, Michael S. Tsirkin, Eugenio Perez

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] virtio_net: virtnet_rx_watchdog() races with
  virtqueue_resize() and reset
- [Low] virtio_net: unsynchronized watchdog init in virtnet_set_queues()
--

commit 093ca3617532e988a9fe20901b18a2e6a86aca17
Author: Longjun Tang <tanglongjun@kylinos.cn>

virtio_net: introduce an RX queue watchdog for stuck detection

This commit adds a per-device 1 Hz watchdog timer that checks active RX
queues for 5-second consumption stalls and reschedules NAPI when a queue
is stuck with a non-zero backlog.

> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 1cd64f9e12e74..7f21fe4f96a2b 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c

[ ... ]

> @@ -3090,6 +3106,74 @@ static int virtnet_poll(struct napi_struct *napi, int budget)
>  	return received;
>  }
>  
> +/*
> + * An RX queue is stalled if it has a backlog but makes no progress and
> + * gets no new interrupt for VIRTNET_RX_WATCHDOG_TIMEO seconds, usually
> + * because an interrupt was lost.  Mark it stalled and reschedule NAPI to
> + * drain the ring and refill the buffers.
> + */
> +static void virtnet_rx_watchdog(struct timer_list *t)
> +{
> +	struct virtnet_info *vi = timer_container_of(vi, t, rx_watchdog);
> +	unsigned long timeout = VIRTNET_RX_WATCHDOG_TIMEO * HZ;
> +	int i;
> +
> +	for (i = 0; i < vi->curr_queue_pairs; i++) {
> +		struct receive_queue *rq = &vi->rq[i];
> +		u16 last_used = virtqueue_get_last_used_idx(rq->vq);
> +		u16 calls = rq->calls;
> +		bool backlog = virtqueue_poll(rq->vq, last_used);

[Severity: Medium]
Can virtnet_rx_watchdog() race with virtqueue_resize() or
virtqueue_reset() and read freed vring memory?

When an RX ring is resized via virtnet_rx_resize() (or reset for AF_XDP in
virtnet_rq_bind_xsk_pool()) while the interface is up, virtnet_rx_pause()
disables NAPI and cancels DIM, but vi->rx_watchdog keeps firing:

drivers/net/virtio_net.c:virtnet_rx_pause() {
	bool running = netif_running(vi->dev);

	if (running) {
		virtnet_napi_disable(rq);
		virtnet_cancel_dim(vi, &rq->dim);
	}
}

Inside virtqueue_resize_split() and virtqueue_resize_packed(),
vring_free() frees the old vring before virtqueue_vring_attach_split() or
virtqueue_vring_attach_packed() installs the new ring pointers:

CPU0                                  CPU1 (<softirq>)
virtnet_rx_resize()
  virtnet_rx_pause(vi, rq)
  virtqueue_resize(rq->vq, ...)
    vring_free(&vq->vq)
                                      virtnet_rx_watchdog()
                                        last_used =
                                          virtqueue_get_last_used_idx(...)
                                        virtqueue_poll(rq->vq, last_used)
                                          <-- use-after-free
    virtqueue_init(vq, ...)
    virtqueue_vring_attach_split(...)

On packed virtqueues, if virtnet_rx_resize() shrinks the ring and
virtnet_rx_watchdog() reads last_used from the old ring right before
virtqueue_init() resets vq->last_used_idx and attaches the smaller
descriptor array, virtqueue_poll_packed() also indexes
vq->packed.vring.desc[used_idx] out of bounds.

In addition, both virtqueue_resize() and virtqueue_reset() reset
vq->last_used_idx in virtqueue_init(), but virtnet_rx_resume() does not
call virtnet_rx_watchdog_init_queue(rq) to resynchronize rq->watchdog.

[ ... ]

> @@ -3839,7 +3926,14 @@ static int virtnet_set_queues(struct virtnet_info *vi, u16 queue_pairs)
>  	if (vi->has_rss)
>  		vi->rss_trailer.max_tx_vq = cpu_to_le16(queue_pairs);
>  succ:
> +	if (dev->flags & IFF_UP) {
> +		/* Init new queues */
> +		for (int i = old_queue_pairs; i < queue_pairs; i++)
> +			virtnet_rx_watchdog_init_queue(&vi->rq[i]);
> +	}
> +
>  	vi->curr_queue_pairs = queue_pairs;

[Severity: Low]
Can virtnet_rx_watchdog() observe the updated vi->curr_queue_pairs before
the rq->watchdog fields initialized by virtnet_rx_watchdog_init_queue()?

virtnet_set_queues() runs while vi->rx_watchdog is active and uses plain
stores to initialize rq->watchdog and update vi->curr_queue_pairs.  By the
time execution reaches the succ label, the CVQ command has already allowed
the device to steer RX packets to the new queues:

CPU0                                  CPU1 (<softirq>)
virtnet_set_queues()
  virtnet_rx_watchdog_init_queue(rq)
    rq->watchdog.last_used = ...
    rq->watchdog.calls = ...
  vi->curr_queue_pairs = queue_pairs
                                      virtnet_rx_watchdog()
                                        i < vi->curr_queue_pairs
                                        time_after(jiffies,
                                          rq->watchdog.jiffies + timeout)
                                          <-- stale jiffies, false stall
    rq->watchdog.jiffies = jiffies

If a newly activated queue already has a buffer in its used ring and
virtnet_rx_watchdog() reads a stale rq->watchdog.jiffies (0 from initial
allocation or an old timestamp from before the queue count was lowered),
it logs a false RX stall warning and marks rq->watchdog.stalled.  If
vi->curr_queue_pairs is lowered and raised again while
virtnet_rx_watchdog() is mid-loop, both functions can also read and write
rq->watchdog concurrently.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261120071826.198604-1-lange_tang@163.com?part=2

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2 3/3] virtio_net: add stalls per-queue statistic
  2026-11-20  7:18 ` [PATCH v2 3/3] virtio_net: add stalls per-queue statistic Longjun Tang
@ 2026-11-21  7:18   ` sashiko-bot
  0 siblings, 0 replies; 9+ messages in thread
From: sashiko-bot @ 2026-11-21  7:18 UTC (permalink / raw)
  To: Longjun Tang; +Cc: virtualization, Michael S. Tsirkin, Eugenio Perez

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] virtio_net: race between virtnet_rx_watchdog() and RX queue
  pause/reset
--

commit fd9c9353e0dbe0759e73fd91a8963232b76ecc1c
Author: Longjun Tang <tanglongjun@kylinos.cn>

virtio_net: add stalls per-queue statistic

This commit adds a per-queue RX stalls statistic to virtio_net and exposes
it via ethtool when NAPI poll recovers a stalled RX queue.

> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 7f21fe4f96a2b..0b95fa10734d2 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c

[ ... ]

> @@ -3005,6 +3007,12 @@ static int virtnet_receive(struct receive_queue *rq, int budget,
>  		u64_stats_add(item, u64_stats_read(src));
>  	}
>  
> +	/* Count a stall recovered by this poll. */
> +	if (rq->watchdog.stalled) {
> +		u64_stats_inc(&rq->stats.stalls);
> +		rq->watchdog.stalled = false;
> +	}

[Severity: Medium]
Can virtnet_rx_watchdog() race with an RX queue resize or reset while the
interface is up, causing a use-after-free on the vring and false
rq->stats.stalls increments?

Commit "virtio_net: introduce an RX queue watchdog for stuck detection"
starts vi->rx_watchdog in virtnet_open() and only stops it in
virtnet_close().  When virtnet_rx_resize() or virtnet_rq_bind_xsk_pool()
pauses an active RX queue via virtnet_rx_pause(), it disables NAPI and DIM
but does not stop or synchronize with vi->rx_watchdog:

CPU0                                  CPU1 (<softirq>)
virtnet_rx_resize()
  virtnet_rx_pause(vi, rq)
  virtqueue_resize(rq->vq, ...)
                                      virtnet_rx_watchdog()
                                        last_used =
                                          virtqueue_get_last_used_idx(...)
    vring_free(&vq->vq)
                                        virtqueue_poll(rq->vq, last_used)
                                          <-- use-after-free

In addition, virtqueue_resize() and virtqueue_reset() reset
vq->last_used_idx, but neither path calls
virtnet_rx_watchdog_init_queue(rq).  Any stale rq->watchdog.last_used or
rq->watchdog.stalled state persists across the reset and causes
virtnet_receive() to falsely increment rq->stats.stalls after the queue is
resumed.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261120071826.198604-1-lange_tang@163.com?part=3

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2026-11-21  7:18 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20  7:18 [PATCH v2 0/3] virtio_net: introduce an RX queue watchdog for stuck detection Longjun Tang
2026-11-20  7:18 ` [PATCH v2 1/3] virtio: add virtqueue_get_last_used_idx() helper Longjun Tang
2026-11-20 17:43   ` Michael S. Tsirkin
2026-11-21  7:18   ` sashiko-bot
2026-11-20  7:18 ` [PATCH v2 2/3] virtio_net: introduce an RX queue watchdog for stuck detection Longjun Tang
2026-11-21  7:18   ` sashiko-bot
2026-11-20  7:18 ` [PATCH v2 3/3] virtio_net: add stalls per-queue statistic Longjun Tang
2026-11-21  7:18   ` sashiko-bot
2026-11-20 17:45 ` [PATCH v2 0/3] virtio_net: introduce an RX queue watchdog for stuck detection Michael S. Tsirkin

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.