From: "Michael S. Tsirkin" <mst@redhat.com>
To: "Denis V. Lunev" <den@virtuozzo.com>
Cc: "David Hildenbrand (Arm)" <david@kernel.org>,
"Denis V. Lunev" <den@openvz.org>,
virtualization@lists.linux.dev, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 0/4] virtio_balloon: quiesce balloon work on device shutdown
Date: Mon, 13 Jul 2026 16:37:15 -0400 [thread overview]
Message-ID: <20260713163611-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <6697d41e-a2fd-4207-982d-999cafd3cd56@virtuozzo.com>
On Mon, Jul 13, 2026 at 10:32:29PM +0200, Denis V. Lunev wrote:
> On 7/3/26 00:33, Michael S. Tsirkin wrote:
> > On Thu, Jul 02, 2026 at 09:30:26PM +0200, David Hildenbrand (Arm) wrote:
> >> On 7/2/26 19:50, Denis V. Lunev wrote:
> >>> On 6/24/26 16:08, Denis V. Lunev wrote:
> >>>> This email originated from an IP that might not be authorized by the domain it was sent from.
> >>>> Do not click links or open attachments unless it is an email you expected to receive.
> >>>> Since commit 8bd2fa086a04 ("virtio: break and reset virtio devices on
> >>>> device_shutdown()") the virtio bus breaks and resets every virtio device
> >>>> during device_shutdown(), i.e. on reboot and kexec. virtio_balloon has no
> >>>> .shutdown of its own, so that generic path runs while the balloon's
> >>>> asynchronous work is still armed: the free page reporting worker, the
> >>>> inflate/deflate and stats workers, the OOM notifier and the free page
> >>>> shrinker.
> >>>>
> >>>> Once the device has been broken, virtqueue_add_inbuf() in
> >>>> virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE().
> >>>> On a kernel booted with panic_on_warn that turns an ordinary reboot into a
> >>>> fatal panic in the middle of device_shutdown(), so the machine never
> >>>> reaches the new kernel. The inflate/deflate and OOM paths do not warn but
> >>>> are no better off: they call wait_event(vb->acked, ...) and would block
> >>>> forever on a queue that can no longer complete.
> >>>>
> >>>> This was hit in the field as an intermittent failure of a virtualization
> >>>> cluster upgrade: guest storage nodes were rebooted via kexec into the new
> >>>> kernel, and the ones whose free page reporting happened to run during
> >>>> device_shutdown() panicked (the guests run with panic_on_warn) and never
> >>>> came back, stalling the rolling upgrade. The crash dump showed the WARN at
> >>>> virtio_balloon.c:216 in a page_reporting kworker, with all the balloon
> >>>> virtqueues already broken.
> >>>>
> >>>> Validated by churning balloon inflate/deflate from the host while
> >>>> kexec-rebooting the guest in a loop under panic_on_warn: the unpatched
> >>>> kernel reproduces the WARN within a couple of cycles, while the patched
> >>>> kernel survives many consecutive kexec cycles cleanly (12/12 in the final
> >>>> run, 0 WARNs). checkpatch is clean across the series.
> >>>>
> >>>> Changes in v2:
> >>>> - Add a virtio_device_shutdown() core helper and call it from the balloon
> >>>> .shutdown handler instead of open-coding break + synchronize_cbs + reset
> >>>> (David Hildenbrand).
> >>>> - New patch: make tell_host() warn and bail instead of hanging if a buffer
> >>>> add ever fails (David Hildenbrand); kept as a separate patch
> >>>> (Michael S. Tsirkin).
> >>>>
> >>>> v1: https://lore.kernel.org/all/20260622133715.3707707-1-den@openvz.org
> >>>>
> >>>> Denis V. Lunev (4):
> >>>> virtio: add virtio_device_shutdown() helper
> >>>> virtio_balloon: factor out virtballoon_quiesce()
> >>>> virtio_balloon: quiesce balloon work before device shutdown
> >>>> virtio_balloon: warn on failed buffer add in tell_host()
> >>>>
> >>>> drivers/virtio/virtio.c | 41 ++++++++++++++++++++++-----------
> >>>> drivers/virtio/virtio_balloon.c | 40 ++++++++++++++++++++++++--------
> >>>> include/linux/virtio.h | 1 +
> >>>> 3 files changed, 59 insertions(+), 23 deletions(-)
> >>>>
> >>> Hi, David!
> >> Hi! :)
> >>
> >>> Is this good to go in? I have not seen the confirmation that the
> >>> series is taken into your tree.
> >> I don't have a tree (yet), and once I have one it will likely be more mm focused :)
> >>
> >> @MST, I think this is good to go!
> >>
> >> --
> >> Cheers,
> >>
> >> David
> > Indeed, it's just a bugfix and I'm trying to get features into qemu
> > now before their freeze. I'll work on linux end of next week.
> >
> Hi, Michael!
>
> Have you had a chance to take series into your tree, i.e.
> should I continue to track this submission?
>
> Thank you in advance,
> Den
Hi!
It's on my queue and will be in the next Linux release, don't worry.
--
MST
prev parent reply other threads:[~2026-07-13 20:37 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-06-24 14:08 [PATCH v2 0/4] virtio_balloon: quiesce balloon work on device shutdown Denis V. Lunev
2026-06-24 14:08 ` [PATCH v2 1/4] virtio: add virtio_device_shutdown() helper Denis V. Lunev
2026-06-24 14:52 ` David Hildenbrand (Arm)
2026-06-24 14:08 ` [PATCH v2 2/4] virtio_balloon: factor out virtballoon_quiesce() Denis V. Lunev
2026-06-24 14:52 ` David Hildenbrand (Arm)
2026-06-24 14:08 ` [PATCH v2 3/4] virtio_balloon: quiesce balloon work before device shutdown Denis V. Lunev
2026-06-24 14:55 ` David Hildenbrand (Arm)
2026-06-24 15:00 ` Denis V. Lunev
2026-06-24 15:23 ` David Hildenbrand (Arm)
2026-06-24 14:08 ` [PATCH v2 4/4] virtio_balloon: warn on failed buffer add in tell_host() Denis V. Lunev
2026-06-24 14:57 ` David Hildenbrand (Arm)
2026-06-24 15:40 ` [PATCH v2 5/4] virtio_balloon: warn on failed buffer add in stats_handle_request() Denis V. Lunev
2026-06-24 16:56 ` David Hildenbrand (Arm)
2026-06-24 17:03 ` Denis V. Lunev
2026-07-02 17:50 ` [PATCH v2 0/4] virtio_balloon: quiesce balloon work on device shutdown Denis V. Lunev
2026-07-02 19:30 ` David Hildenbrand (Arm)
2026-07-02 22:33 ` Michael S. Tsirkin
2026-07-13 20:32 ` Denis V. Lunev
2026-07-13 20:37 ` Michael S. Tsirkin [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260713163611-mutt-send-email-mst@kernel.org \
--to=mst@redhat.com \
--cc=david@kernel.org \
--cc=den@openvz.org \
--cc=den@virtuozzo.com \
--cc=linux-kernel@vger.kernel.org \
--cc=virtualization@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox