Re: [PATCH 0/10] system: add a watchdog for the Big QEMU Lock

[Fabiano Rosas <farosas@suse.de>]

"Denis V. Lunev" <den@openvz.org> writes:

> A vCPU thread reacquires the BQL on every exit that needs userspace. A
> holder that keeps it for tens of seconds stops every vCPU, the guest
> reports a soft lockup or panics, and qemu itself records nothing. The
> guest log is the only evidence, and it cannot say whether the hypervisor
> held the lock or the guest wedged on its own.
>
> This series adds a BQL watchdog. A sampler thread watches a sequence
> counter that bql_update_status() advances on every acquire and release.
> One odd value across the whole window means one owner held the lock for
> the whole window. On expiry the watchdog names the holder and prints the
> state of every other thread:
>
>   qemu-system-x86_64: 2026-09-15T15:53:57Z BQL held for more than 100 ms by thread 168146
>     168146  qemu-system-x86  S
>         pc 0x785a4dbcb884 liburing.so.2+0x2884
>         wchan io_cqring_wait
>     168147  call_rcu         S
>         pc 0x785a4d534acb syscall+0x1b
>         wchan futex_do_wait
>     168148  IO mon_iothread  S
>         pc 0x785a4d4ac802 libc.so.6+0xac802
>         wchan poll_schedule_timeout.constprop.0
>
> With CAP_SYS_ADMIN the wchan line becomes the kernel stack. The pc line
> comes from signalling each thread and reading the program counter out of
> its signal frame, which is what tells a thread spinning in userspace
> from one asleep in a syscall.
>
> The watchdog is off by default and stays quiet under a debugger. Two
> machine properties drive it: bql-watchdog-ms sets the deadline, on the
> command line or over qom-set at any time, and bql-watchdog-abort turns
> the report into a core dump for hosts where the answer is worth more
> than the guest.
>
> Patches 1 to 6 are the pieces: a fix for bql_locked across the timed
> condvar wait, a debugger check, the per-thread state reader and its
> kernel stack and program counter extensions, and the signal reserved for
> the latter. Patches 7 to 9 are the watchdog itself. Patch 10 is a
> functional test that holds the BQL through an HMP qemu-io write to a
> null-co node with a latency longer than the deadline.
>
> The program counter lives at a host specific offset in the signal frame,
> so patch 6 reads it through a host/include header. The Linux headers for
> aarch64, ppc64, riscv64, s390x and loongarch64 repeat what linux-user's
> host-signal.h already knows about those frames, but only the x86_64 one
> has been compiled here, and the Windows half of patch 1 has not been
> compiled either.
>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Richard Henderson <richard.henderson@linaro.org>
> Cc: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
> Cc: Zhao Liu <zhao1.liu@intel.com>
> Signed-off-by: Denis V. Lunev <den@openvz.org>
>
> Denis V. Lunev (10):
>   util/qemu-thread: track bql_locked across the timed condvar wait
>   oslib: add qemu_debugger_attached()
>   oslib: add qemu_thread_states()
>   oslib: report kernel stacks in qemu_thread_states()
>   util/qemu-thread: reserve a signal for thread introspection
>   oslib: report each thread's program counter
>   system/cpus: report a BQL held past a deadline
>   system/cpus: hold off the BQL watchdog under a debugger
>   system/cpus: add bql-watchdog-abort
>   tests/functional: cover the BQL watchdog
>
>  host/include/aarch64/host/signal-pc.h         |  20 ++
>  host/include/generic/host/signal-pc.h         |  11 +
>  host/include/loongarch64/host/signal-pc.h     |  20 ++
>  host/include/ppc64/host/signal-pc.h           |  22 ++
>  host/include/riscv64/host/signal-pc.h         |  20 ++
>  host/include/s390x/host/signal-pc.h           |  20 ++
>  host/include/x86_64/host/signal-pc.h          |  20 ++
>  hw/core/machine.c                             |  44 +++
>  include/qemu/osdep.h                          |  29 ++
>  include/system/cpus.h                         |   8 +
>  meson.build                                   |   2 +
>  qemu-options.hx                               |  11 +
>  system/cpus.c                                 | 143 ++++++++
>  tests/functional/generic/meson.build          |   1 +
>  tests/functional/generic/test_bql_watchdog.py |  60 ++++
>  util/oslib-posix.c                            | 316 ++++++++++++++++++
>  util/oslib-win32.c                            |  10 +
>  util/qemu-thread-posix.c                      |   8 +-
>  util/qemu-thread-win32.c                      |   4 +-
>  19 files changed, 765 insertions(+), 4 deletions(-)
>  create mode 100644 host/include/aarch64/host/signal-pc.h
>  create mode 100644 host/include/generic/host/signal-pc.h
>  create mode 100644 host/include/loongarch64/host/signal-pc.h
>  create mode 100644 host/include/ppc64/host/signal-pc.h
>  create mode 100644 host/include/riscv64/host/signal-pc.h
>  create mode 100644 host/include/s390x/host/signal-pc.h
>  create mode 100644 host/include/x86_64/host/signal-pc.h
>  create mode 100755 tests/functional/generic/test_bql_watchdog.py

The problem you state is definitely a pervasive one and I've encountered
it many times. It's a pain do debug. I'm working (on and off, you know
how it goes) on a perf improvement for dirty logging disabling, which is
basically this issue after migrate_cancel. I've been writing custom code
to track the blocking windows, so this series looks like it could help a
lot.

I'm putting it on my queue to review, I'll make some time for it soon.

One other thing I have in flight is a change to qtests to allow testing
a linux guest while a worker thread does some BQL intensive work in
parallel such as serial IO. I'm currently checking the guest serial
console for panics and lockups, but the watchdog would report the issue
right at the source.