From: "Daniel P. Berrangé" <berrange@redhat.com>
To: "Denis V. Lunev" <den@openvz.org>
Cc: qemu-devel@nongnu.org, "Paolo Bonzini" <pbonzini@redhat.com>,
"Richard Henderson" <richard.henderson@linaro.org>,
"Philippe Mathieu-Daudé" <philmd@oss.qualcomm.com>,
"Zhao Liu" <zhao1.liu@intel.com>
Subject: Re: [PATCH 0/10] system: add a watchdog for the Big QEMU Lock
Date: Fri, 20 Nov 2026 16:22:48 +0000 [thread overview]
Message-ID: <awBz2AP3W6-XNQ9U@redhat.com> (raw)
In-Reply-To: <20260915203455.580999-1-den@openvz.org>
On Tue, Sep 15, 2026 at 10:34:44PM +0200, Denis V. Lunev wrote:
> A vCPU thread reacquires the BQL on every exit that needs userspace. A
> holder that keeps it for tens of seconds stops every vCPU, the guest
> reports a soft lockup or panics, and qemu itself records nothing. The
> guest log is the only evidence, and it cannot say whether the hypervisor
> held the lock or the guest wedged on its own.
>
> This series adds a BQL watchdog. A sampler thread watches a sequence
> counter that bql_update_status() advances on every acquire and release.
> One odd value across the whole window means one owner held the lock for
> the whole window. On expiry the watchdog names the holder and prints the
> state of every other thread:
>
> qemu-system-x86_64: 2026-09-15T15:53:57Z BQL held for more than 100 ms by thread 168146
> 168146 qemu-system-x86 S
> pc 0x785a4dbcb884 liburing.so.2+0x2884
> wchan io_cqring_wait
> 168147 call_rcu S
> pc 0x785a4d534acb syscall+0x1b
> wchan futex_do_wait
> 168148 IO mon_iothread S
> pc 0x785a4d4ac802 libc.so.6+0xac802
> wchan poll_schedule_timeout.constprop.0
>
> With CAP_SYS_ADMIN the wchan line becomes the kernel stack. The pc line
> comes from signalling each thread and reading the program counter out of
> its signal frame, which is what tells a thread spinning in userspace
> from one asleep in a syscall.
>
> The watchdog is off by default and stays quiet under a debugger. Two
> machine properties drive it: bql-watchdog-ms sets the deadline, on the
> command line or over qom-set at any time, and bql-watchdog-abort turns
> the report into a core dump for hosts where the answer is worth more
> than the guest.
>
> Patches 1 to 6 are the pieces: a fix for bql_locked across the timed
> condvar wait, a debugger check, the per-thread state reader and its
> kernel stack and program counter extensions, and the signal reserved for
> the latter. Patches 7 to 9 are the watchdog itself. Patch 10 is a
> functional test that holds the BQL through an HMP qemu-io write to a
> null-co node with a latency longer than the deadline.
>
> The program counter lives at a host specific offset in the signal frame,
> so patch 6 reads it through a host/include header. The Linux headers for
> aarch64, ppc64, riscv64, s390x and loongarch64 repeat what linux-user's
> host-signal.h already knows about those frames, but only the x86_64 one
> has been compiled here, and the Windows half of patch 1 has not been
> compiled either.
>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Richard Henderson <richard.henderson@linaro.org>
> Cc: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
> Cc: Zhao Liu <zhao1.liu@intel.com>
> Signed-off-by: Denis V. Lunev <den@openvz.org>
>
> Denis V. Lunev (10):
> util/qemu-thread: track bql_locked across the timed condvar wait
> oslib: add qemu_debugger_attached()
> oslib: add qemu_thread_states()
> oslib: report kernel stacks in qemu_thread_states()
> util/qemu-thread: reserve a signal for thread introspection
> oslib: report each thread's program counter
> system/cpus: report a BQL held past a deadline
> system/cpus: hold off the BQL watchdog under a debugger
> system/cpus: add bql-watchdog-abort
> tests/functional: cover the BQL watchdog
>
> host/include/aarch64/host/signal-pc.h | 20 ++
> host/include/generic/host/signal-pc.h | 11 +
> host/include/loongarch64/host/signal-pc.h | 20 ++
> host/include/ppc64/host/signal-pc.h | 22 ++
> host/include/riscv64/host/signal-pc.h | 20 ++
> host/include/s390x/host/signal-pc.h | 20 ++
> host/include/x86_64/host/signal-pc.h | 20 ++
> hw/core/machine.c | 44 +++
> include/qemu/osdep.h | 29 ++
> include/system/cpus.h | 8 +
> meson.build | 2 +
> qemu-options.hx | 11 +
> system/cpus.c | 143 ++++++++
> tests/functional/generic/meson.build | 1 +
> tests/functional/generic/test_bql_watchdog.py | 60 ++++
> util/oslib-posix.c | 316 ++++++++++++++++++
As a general point that doesn't fit any single patch, IMHO
it is undesirable to be putting all this logic into the
oslib-posix.c file.
It likely justifies being put into a standalone file eg a
new 'debug.c'
With regards,
Daniel
--
|: https://berrange.com ~~ https://hachyderm.io/@berrange :|
|: https://libvirt.org ~~ https://entangle-photo.org :|
|: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
next prev parent reply other threads:[~2026-11-20 16:23 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-09-15 20:34 [PATCH 0/10] system: add a watchdog for the Big QEMU Lock Denis V. Lunev
2026-09-15 20:34 ` [PATCH 01/10] util/qemu-thread: track bql_locked across the timed condvar wait Denis V. Lunev
2026-10-14 22:36 ` Fabiano Rosas
2026-09-15 20:34 ` [PATCH 02/10] oslib: add qemu_debugger_attached() Denis V. Lunev
2026-11-17 13:45 ` Fabiano Rosas
2026-11-17 14:38 ` Peter Maydell
2026-11-20 16:19 ` Daniel P. Berrangé
2026-11-20 16:27 ` Daniel P. Berrangé
2026-09-15 20:34 ` [PATCH 03/10] oslib: add qemu_thread_states() Denis V. Lunev
2026-11-17 14:02 ` Fabiano Rosas
2026-09-15 20:34 ` [PATCH 04/10] oslib: report kernel stacks in qemu_thread_states() Denis V. Lunev
2026-11-17 18:13 ` Fabiano Rosas
2026-11-20 16:14 ` Denis V. Lunev
2026-11-20 15:28 ` Daniel P. Berrangé
2026-09-15 20:34 ` [PATCH 05/10] util/qemu-thread: reserve a signal for thread introspection Denis V. Lunev
2026-09-15 20:34 ` [PATCH 06/10] oslib: report each thread's program counter Denis V. Lunev
2026-11-17 18:19 ` Fabiano Rosas
2026-09-15 20:34 ` [PATCH 07/10] system/cpus: report a BQL held past a deadline Denis V. Lunev
2026-11-17 18:24 ` Fabiano Rosas
2026-09-15 20:34 ` [PATCH 08/10] system/cpus: hold off the BQL watchdog under a debugger Denis V. Lunev
2026-11-17 18:25 ` Fabiano Rosas
2026-09-15 20:34 ` [PATCH 09/10] system/cpus: add bql-watchdog-abort Denis V. Lunev
2026-10-14 23:30 ` Fabiano Rosas
2026-09-15 20:34 ` [PATCH 10/10] tests/functional: cover the BQL watchdog Denis V. Lunev
2026-09-16 4:57 ` Thomas Huth
2026-09-16 6:44 ` Marc-André Lureau
2026-09-16 7:56 ` Denis V. Lunev
2026-09-18 21:56 ` [PATCH 0/10] system: add a watchdog for the Big QEMU Lock Fabiano Rosas
2026-11-20 16:22 ` Daniel P. Berrangé [this message]
2026-11-20 16:25 ` Denis V. Lunev
2026-11-20 17:57 ` Paolo Bonzini
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=awBz2AP3W6-XNQ9U@redhat.com \
--to=berrange@redhat.com \
--cc=den@openvz.org \
--cc=pbonzini@redhat.com \
--cc=philmd@oss.qualcomm.com \
--cc=qemu-devel@nongnu.org \
--cc=richard.henderson@linaro.org \
--cc=zhao1.liu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.