From: "Michael S. Tsirkin" <mst@redhat.com>
To: Jinqian Yang <yangjinqian1@huawei.com>
Cc: jasowang@redhat.com, xuanzhuo@linux.alibaba.com,
eperezma@redhat.com, andrew+netdev@lunn.ch, davem@davemloft.net,
edumazet@google.com, kuba@kernel.org, pabeni@redhat.com,
netdev@vger.kernel.org, virtualization@lists.linux.dev,
linux-kernel@vger.kernel.org, liuyonglong@huawei.com,
wangzhou1@hisilicon.com, linuxarm@huawei.com
Subject: Re: [PATCH v2] virtio_net: fix infinite loop in virtnet_poll_cleantx when device is broken
Date: Thu, 16 Jul 2026 01:14:25 -0400 [thread overview]
Message-ID: <20260716011312-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <20260716035201.3736582-1-yangjinqian1@huawei.com>
On Thu, Jul 16, 2026 at 11:52:01AM +0800, Jinqian Yang wrote:
> virtnet_poll_cleantx() contains a do-while loop that cleans up
> transmitted TX buffers and calls virtqueue_enable_cb_delayed() to check
> whether more buffers need processing. When the virtio backend stops
> responding during guest reboot, used->idx is never updated, so
> virtqueue_enable_cb_delayed() always returns false and the loop never
> terminates. Then it will block reboot process, and the guest will hang.
>
> The problem occurs during guest reboot under network traffic:
>
> 1. kernel_restart() -> device_shutdown() traverses the device list
> 2. virtio_dev_shutdown() calls virtio_break_device() which sets
> vq->broken = true
> 3. virtio_dev_shutdown() then calls virtio_synchronize_cbs() to wait
> for in-flight callbacks to complete
> 4. A virtio interrupt fires, softirq is deferred to ksoftirqd which
> calls net_rx_action() -> virtnet_poll() -> virtnet_poll_cleantx()
> 5. virtnet_poll_cleantx() enters the do-while loop and never exits
> because the QEMU backend has stopped updating used->idx, despite
> vq->broken having been set to true in step 2.
>
> Since the loop runs inside ksoftirqd (a SCHED_OTHER kthread), it is
> visible to the scheduler and does not trigger a hard lockup. However,
> the kthread never leaves the loop, so RCU detects it as a CPU stall
> and reports it periodically. Meanwhile, the reboot process remains
> blocked in device_shutdown() because virtio_dev_shutdown() cannot
> complete its synchronization step, and the guest hangs permanently.
>
> This can be reproduced on a guest with a virtio-net device: run iperf3
> traffic in the guest, then trigger reboot. The reboot occasionally hangs
> permanently with RCU stall on ksoftirqd.
>
> Observed on ARM64 KVM guest:
>
> CPU#1 RCU stall (ksoftirqd/1), repeated periodically:
> virtqueue_enable_cb_delayed_split <- virtnet_poll <- __napi_poll <-
> net_rx_action <- handle_softirqs <- run_ksoftirqd <-
> smpboot_thread_fn <- kthread
>
> Fix by adding a virtqueue_is_broken() check to the loop condition, so
> that the loop exits immediately when the device is broken, allowing
> the device shutdown to proceed.
>
> Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com>
Good thanks! Just the subject needs change so it's clear
we are changing virtio core not virtio net.
> ---
> Changes in v2:
> - Moved vq->broken check to virtqueue_enable_cb_delayed().
>
> v1: https://lore.kernel.org/lkml/20260713132025.703147-1-yangjinqian1@huawei.com/
> ---
> drivers/virtio/virtio_ring.c | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
> index b438dc2ce1b8..5c169fbb418a 100644
> --- a/drivers/virtio/virtio_ring.c
> +++ b/drivers/virtio/virtio_ring.c
> @@ -3233,6 +3233,14 @@ bool virtqueue_enable_cb_delayed(struct virtqueue *_vq)
> {
> struct vring_virtqueue *vq = to_vvq(_vq);
>
> + /*
> + * When the device is broken there is no point in polling used->idx,
> + * the backend will never update it. Return true to let callers
> + * exit their cleanup loops instead of spinning forever.
> + */
> + if (unlikely(vq->broken))
> + return true;
> +
> if (vq->event_triggered)
> data_race(vq->event_triggered = false);
>
> --
> 2.33.0
next prev parent reply other threads:[~2026-07-16 5:14 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-07-16 3:52 [PATCH v2] virtio_net: fix infinite loop in virtnet_poll_cleantx when device is broken Jinqian Yang
2026-07-16 5:14 ` Michael S. Tsirkin [this message]
2026-07-16 6:05 ` Jinqian Yang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260716011312-mutt-send-email-mst@kernel.org \
--to=mst@redhat.com \
--cc=andrew+netdev@lunn.ch \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=eperezma@redhat.com \
--cc=jasowang@redhat.com \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linuxarm@huawei.com \
--cc=liuyonglong@huawei.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=virtualization@lists.linux.dev \
--cc=wangzhou1@hisilicon.com \
--cc=xuanzhuo@linux.alibaba.com \
--cc=yangjinqian1@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox