Linux virtualization list
 help / color / mirror / Atom feed
From: "David Hildenbrand (Arm)" <david@kernel.org>
To: "Denis V. Lunev" <den@virtuozzo.com>,
	"Denis V. Lunev" <den@openvz.org>,
	mst@redhat.com
Cc: virtualization@lists.linux.dev, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 0/4] virtio_balloon: quiesce balloon work on device shutdown
Date: Thu, 2 Jul 2026 21:30:26 +0200	[thread overview]
Message-ID: <00ebb86d-f7b8-4ddc-a17c-c4700cab73a9@kernel.org> (raw)
In-Reply-To: <c18ed056-72f3-4eab-a370-c02b735e536e@virtuozzo.com>

On 7/2/26 19:50, Denis V. Lunev wrote:
> On 6/24/26 16:08, Denis V. Lunev wrote:
>> This email originated from an IP that might not be authorized by the domain it was sent from.
>> Do not click links or open attachments unless it is an email you expected to receive.
>> Since commit 8bd2fa086a04 ("virtio: break and reset virtio devices on
>> device_shutdown()") the virtio bus breaks and resets every virtio device
>> during device_shutdown(), i.e. on reboot and kexec. virtio_balloon has no
>> .shutdown of its own, so that generic path runs while the balloon's
>> asynchronous work is still armed: the free page reporting worker, the
>> inflate/deflate and stats workers, the OOM notifier and the free page
>> shrinker.
>>
>> Once the device has been broken, virtqueue_add_inbuf() in
>> virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE().
>> On a kernel booted with panic_on_warn that turns an ordinary reboot into a
>> fatal panic in the middle of device_shutdown(), so the machine never
>> reaches the new kernel. The inflate/deflate and OOM paths do not warn but
>> are no better off: they call wait_event(vb->acked, ...) and would block
>> forever on a queue that can no longer complete.
>>
>> This was hit in the field as an intermittent failure of a virtualization
>> cluster upgrade: guest storage nodes were rebooted via kexec into the new
>> kernel, and the ones whose free page reporting happened to run during
>> device_shutdown() panicked (the guests run with panic_on_warn) and never
>> came back, stalling the rolling upgrade. The crash dump showed the WARN at
>> virtio_balloon.c:216 in a page_reporting kworker, with all the balloon
>> virtqueues already broken.
>>
>> Validated by churning balloon inflate/deflate from the host while
>> kexec-rebooting the guest in a loop under panic_on_warn: the unpatched
>> kernel reproduces the WARN within a couple of cycles, while the patched
>> kernel survives many consecutive kexec cycles cleanly (12/12 in the final
>> run, 0 WARNs). checkpatch is clean across the series.
>>
>> Changes in v2:
>> - Add a virtio_device_shutdown() core helper and call it from the balloon
>>   .shutdown handler instead of open-coding break + synchronize_cbs + reset
>>   (David Hildenbrand).
>> - New patch: make tell_host() warn and bail instead of hanging if a buffer
>>   add ever fails (David Hildenbrand); kept as a separate patch
>>   (Michael S. Tsirkin).
>>
>> v1: https://lore.kernel.org/all/20260622133715.3707707-1-den@openvz.org
>>
>> Denis V. Lunev (4):
>>   virtio: add virtio_device_shutdown() helper
>>   virtio_balloon: factor out virtballoon_quiesce()
>>   virtio_balloon: quiesce balloon work before device shutdown
>>   virtio_balloon: warn on failed buffer add in tell_host()
>>
>>  drivers/virtio/virtio.c         | 41 ++++++++++++++++++++++-----------
>>  drivers/virtio/virtio_balloon.c | 40 ++++++++++++++++++++++++--------
>>  include/linux/virtio.h          |  1 +
>>  3 files changed, 59 insertions(+), 23 deletions(-)
>>
> Hi, David!

Hi! :)

> 
> Is this good to go in? I have not seen the confirmation that the
> series is taken into your tree.

I don't have a tree (yet), and once I have one it will likely be more mm focused :)

@MST, I think this is good to go!

-- 
Cheers,

David

  reply	other threads:[~2026-07-02 19:30 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-06-24 14:08 [PATCH v2 0/4] virtio_balloon: quiesce balloon work on device shutdown Denis V. Lunev
2026-06-24 14:08 ` [PATCH v2 1/4] virtio: add virtio_device_shutdown() helper Denis V. Lunev
2026-06-24 14:52   ` David Hildenbrand (Arm)
2026-06-24 14:08 ` [PATCH v2 2/4] virtio_balloon: factor out virtballoon_quiesce() Denis V. Lunev
2026-06-24 14:52   ` David Hildenbrand (Arm)
2026-06-24 14:08 ` [PATCH v2 3/4] virtio_balloon: quiesce balloon work before device shutdown Denis V. Lunev
2026-06-24 14:55   ` David Hildenbrand (Arm)
2026-06-24 15:00     ` Denis V. Lunev
2026-06-24 15:23       ` David Hildenbrand (Arm)
2026-06-24 14:08 ` [PATCH v2 4/4] virtio_balloon: warn on failed buffer add in tell_host() Denis V. Lunev
2026-06-24 14:57   ` David Hildenbrand (Arm)
2026-06-24 15:40     ` [PATCH v2 5/4] virtio_balloon: warn on failed buffer add in stats_handle_request() Denis V. Lunev
2026-06-24 16:56       ` David Hildenbrand (Arm)
2026-06-24 17:03         ` Denis V. Lunev
2026-07-02 17:50 ` [PATCH v2 0/4] virtio_balloon: quiesce balloon work on device shutdown Denis V. Lunev
2026-07-02 19:30   ` David Hildenbrand (Arm) [this message]
2026-07-02 22:33     ` Michael S. Tsirkin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=00ebb86d-f7b8-4ddc-a17c-c4700cab73a9@kernel.org \
    --to=david@kernel.org \
    --cc=den@openvz.org \
    --cc=den@virtuozzo.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mst@redhat.com \
    --cc=virtualization@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox