From: "Michael S. Tsirkin" <mst@redhat.com>
To: "Denis V. Lunev" <den@virtuozzo.com>
Cc: "David Hildenbrand (Arm)" <david@kernel.org>,
"Denis V. Lunev" <den@openvz.org>,
virtualization@lists.linux.dev, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 0/4] virtio_balloon: quiesce balloon work on device shutdown
Date: Mon, 13 Jul 2026 16:37:15 -0400 [thread overview]
Message-ID: <20260713163611-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <6697d41e-a2fd-4207-982d-999cafd3cd56@virtuozzo.com>
On Mon, Jul 13, 2026 at 10:32:29PM +0200, Denis V. Lunev wrote:
> On 7/3/26 00:33, Michael S. Tsirkin wrote:
> > On Thu, Jul 02, 2026 at 09:30:26PM +0200, David Hildenbrand (Arm) wrote:
> >> On 7/2/26 19:50, Denis V. Lunev wrote:
> >>> On 6/24/26 16:08, Denis V. Lunev wrote:
> >>>> This email originated from an IP that might not be authorized by the domain it was sent from.
> >>>> Do not click links or open attachments unless it is an email you expected to receive.
> >>>> Since commit 8bd2fa086a04 ("virtio: break and reset virtio devices on
> >>>> device_shutdown()") the virtio bus breaks and resets every virtio device
> >>>> during device_shutdown(), i.e. on reboot and kexec. virtio_balloon has no
> >>>> .shutdown of its own, so that generic path runs while the balloon's
> >>>> asynchronous work is still armed: the free page reporting worker, the
> >>>> inflate/deflate and stats workers, the OOM notifier and the free page
> >>>> shrinker.
> >>>>
> >>>> Once the device has been broken, virtqueue_add_inbuf() in
> >>>> virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE().
> >>>> On a kernel booted with panic_on_warn that turns an ordinary reboot into a
> >>>> fatal panic in the middle of device_shutdown(), so the machine never
> >>>> reaches the new kernel. The inflate/deflate and OOM paths do not warn but
> >>>> are no better off: they call wait_event(vb->acked, ...) and would block
> >>>> forever on a queue that can no longer complete.
> >>>>
> >>>> This was hit in the field as an intermittent failure of a virtualization
> >>>> cluster upgrade: guest storage nodes were rebooted via kexec into the new
> >>>> kernel, and the ones whose free page reporting happened to run during
> >>>> device_shutdown() panicked (the guests run with panic_on_warn) and never
> >>>> came back, stalling the rolling upgrade. The crash dump showed the WARN at
> >>>> virtio_balloon.c:216 in a page_reporting kworker, with all the balloon
> >>>> virtqueues already broken.
> >>>>
> >>>> Validated by churning balloon inflate/deflate from the host while
> >>>> kexec-rebooting the guest in a loop under panic_on_warn: the unpatched
> >>>> kernel reproduces the WARN within a couple of cycles, while the patched
> >>>> kernel survives many consecutive kexec cycles cleanly (12/12 in the final
> >>>> run, 0 WARNs). checkpatch is clean across the series.
> >>>>
> >>>> Changes in v2:
> >>>> - Add a virtio_device_shutdown() core helper and call it from the balloon
> >>>> .shutdown handler instead of open-coding break + synchronize_cbs + reset
> >>>> (David Hildenbrand).
> >>>> - New patch: make tell_host() warn and bail instead of hanging if a buffer
> >>>> add ever fails (David Hildenbrand); kept as a separate patch
> >>>> (Michael S. Tsirkin).
> >>>>
> >>>> v1: https://lore.kernel.org/all/20260622133715.3707707-1-den@openvz.org
> >>>>
> >>>> Denis V. Lunev (4):
> >>>> virtio: add virtio_device_shutdown() helper
> >>>> virtio_balloon: factor out virtballoon_quiesce()
> >>>> virtio_balloon: quiesce balloon work before device shutdown
> >>>> virtio_balloon: warn on failed buffer add in tell_host()
> >>>>
> >>>> drivers/virtio/virtio.c | 41 ++++++++++++++++++++++-----------
> >>>> drivers/virtio/virtio_balloon.c | 40 ++++++++++++++++++++++++--------
> >>>> include/linux/virtio.h | 1 +
> >>>> 3 files changed, 59 insertions(+), 23 deletions(-)
> >>>>
> >>> Hi, David!
> >> Hi! :)
> >>
> >>> Is this good to go in? I have not seen the confirmation that the
> >>> series is taken into your tree.
> >> I don't have a tree (yet), and once I have one it will likely be more mm focused :)
> >>
> >> @MST, I think this is good to go!
> >>
> >> --
> >> Cheers,
> >>
> >> David
> > Indeed, it's just a bugfix and I'm trying to get features into qemu
> > now before their freeze. I'll work on linux end of next week.
> >
> Hi, Michael!
>
> Have you had a chance to take series into your tree, i.e.
> should I continue to track this submission?
>
> Thank you in advance,
> Den
Hi!
It's on my queue and will be in the next Linux release, don't worry.
--
MST
prev parent reply other threads:[~2026-07-13 20:37 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-06-24 14:08 [PATCH v2 0/4] virtio_balloon: quiesce balloon work on device shutdown Denis V. Lunev
2026-06-24 14:08 ` [PATCH v2 1/4] virtio: add virtio_device_shutdown() helper Denis V. Lunev
2026-06-24 14:52 ` David Hildenbrand (Arm)
2026-06-24 14:08 ` [PATCH v2 2/4] virtio_balloon: factor out virtballoon_quiesce() Denis V. Lunev
2026-06-24 14:52 ` David Hildenbrand (Arm)
2026-06-24 14:08 ` [PATCH v2 3/4] virtio_balloon: quiesce balloon work before device shutdown Denis V. Lunev
2026-06-24 14:55 ` David Hildenbrand (Arm)
2026-06-24 15:00 ` Denis V. Lunev
2026-06-24 15:23 ` David Hildenbrand (Arm)
2026-06-24 14:08 ` [PATCH v2 4/4] virtio_balloon: warn on failed buffer add in tell_host() Denis V. Lunev
2026-06-24 14:57 ` David Hildenbrand (Arm)
2026-06-24 15:40 ` [PATCH v2 5/4] virtio_balloon: warn on failed buffer add in stats_handle_request() Denis V. Lunev
2026-06-24 16:56 ` David Hildenbrand (Arm)
2026-06-24 17:03 ` Denis V. Lunev
2026-07-02 17:50 ` [PATCH v2 0/4] virtio_balloon: quiesce balloon work on device shutdown Denis V. Lunev
2026-07-02 19:30 ` David Hildenbrand (Arm)
2026-07-02 22:33 ` Michael S. Tsirkin
2026-07-13 20:32 ` Denis V. Lunev
2026-07-13 20:37 ` Michael S. Tsirkin [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260713163611-mutt-send-email-mst@kernel.org \
--to=mst@redhat.com \
--cc=david@kernel.org \
--cc=den@openvz.org \
--cc=den@virtuozzo.com \
--cc=linux-kernel@vger.kernel.org \
--cc=virtualization@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.