[PATCH 0/10] system: add a watchdog for the Big QEMU Lock

[PATCH 0/10] system: add a watchdog for the Big QEMU Lock

[Denis V. Lunev]

A vCPU thread reacquires the BQL on every exit that needs userspace. A
holder that keeps it for tens of seconds stops every vCPU, the guest
reports a soft lockup or panics, and qemu itself records nothing. The
guest log is the only evidence, and it cannot say whether the hypervisor
held the lock or the guest wedged on its own.

This series adds a BQL watchdog. A sampler thread watches a sequence
counter that bql_update_status() advances on every acquire and release.
One odd value across the whole window means one owner held the lock for
the whole window. On expiry the watchdog names the holder and prints the
state of every other thread:

  qemu-system-x86_64: 2026-09-15T15:53:57Z BQL held for more than 100 ms by thread 168146
    168146  qemu-system-x86  S
        pc 0x785a4dbcb884 liburing.so.2+0x2884
        wchan io_cqring_wait
    168147  call_rcu         S
        pc 0x785a4d534acb syscall+0x1b
        wchan futex_do_wait
    168148  IO mon_iothread  S
        pc 0x785a4d4ac802 libc.so.6+0xac802
        wchan poll_schedule_timeout.constprop.0

With CAP_SYS_ADMIN the wchan line becomes the kernel stack. The pc line
comes from signalling each thread and reading the program counter out of
its signal frame, which is what tells a thread spinning in userspace
from one asleep in a syscall.

The watchdog is off by default and stays quiet under a debugger. Two
machine properties drive it: bql-watchdog-ms sets the deadline, on the
command line or over qom-set at any time, and bql-watchdog-abort turns
the report into a core dump for hosts where the answer is worth more
than the guest.

Patches 1 to 6 are the pieces: a fix for bql_locked across the timed
condvar wait, a debugger check, the per-thread state reader and its
kernel stack and program counter extensions, and the signal reserved for
the latter. Patches 7 to 9 are the watchdog itself. Patch 10 is a
functional test that holds the BQL through an HMP qemu-io write to a
null-co node with a latency longer than the deadline.

The program counter lives at a host specific offset in the signal frame,
so patch 6 reads it through a host/include header. The Linux headers for
aarch64, ppc64, riscv64, s390x and loongarch64 repeat what linux-user's
host-signal.h already knows about those frames, but only the x86_64 one
has been compiled here, and the Windows half of patch 1 has not been
compiled either.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Cc: Zhao Liu <zhao1.liu@intel.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>

Denis V. Lunev (10):
  util/qemu-thread: track bql_locked across the timed condvar wait
  oslib: add qemu_debugger_attached()
  oslib: add qemu_thread_states()
  oslib: report kernel stacks in qemu_thread_states()
  util/qemu-thread: reserve a signal for thread introspection
  oslib: report each thread's program counter
  system/cpus: report a BQL held past a deadline
  system/cpus: hold off the BQL watchdog under a debugger
  system/cpus: add bql-watchdog-abort
  tests/functional: cover the BQL watchdog

 host/include/aarch64/host/signal-pc.h         |  20 ++
 host/include/generic/host/signal-pc.h         |  11 +
 host/include/loongarch64/host/signal-pc.h     |  20 ++
 host/include/ppc64/host/signal-pc.h           |  22 ++
 host/include/riscv64/host/signal-pc.h         |  20 ++
 host/include/s390x/host/signal-pc.h           |  20 ++
 host/include/x86_64/host/signal-pc.h          |  20 ++
 hw/core/machine.c                             |  44 +++
 include/qemu/osdep.h                          |  29 ++
 include/system/cpus.h                         |   8 +
 meson.build                                   |   2 +
 qemu-options.hx                               |  11 +
 system/cpus.c                                 | 143 ++++++++
 tests/functional/generic/meson.build          |   1 +
 tests/functional/generic/test_bql_watchdog.py |  60 ++++
 util/oslib-posix.c                            | 316 ++++++++++++++++++
 util/oslib-win32.c                            |  10 +
 util/qemu-thread-posix.c                      |   8 +-
 util/qemu-thread-win32.c                      |   4 +-
 19 files changed, 765 insertions(+), 4 deletions(-)
 create mode 100644 host/include/aarch64/host/signal-pc.h
 create mode 100644 host/include/generic/host/signal-pc.h
 create mode 100644 host/include/loongarch64/host/signal-pc.h
 create mode 100644 host/include/ppc64/host/signal-pc.h
 create mode 100644 host/include/riscv64/host/signal-pc.h
 create mode 100644 host/include/s390x/host/signal-pc.h
 create mode 100644 host/include/x86_64/host/signal-pc.h
 create mode 100755 tests/functional/generic/test_bql_watchdog.py

-- 
2.53.0

[PATCH 01/10] util/qemu-thread: track bql_locked across the timed condvar wait

[Denis V. Lunev]

From: Denis V. Lunev <den@openvz.org>

qemu_cond_timedwait_ts() releases and retakes the mutex around
pthread_cond_timedwait(), but calls the two trace points by hand
instead of going through the mutex hooks, so it never reaches
bql_update_status(). A thread waiting there still reports bql_locked
true while the mutex is released, which is the confusion the hooks were
given that job to end. migration/cpu-throttle.c reaches this through
qemu_cond_timedwait_bql(). The Windows qemu_cond_timedwait_impl() has
the same two trace calls, next to a qemu_cond_wait_impl() that already
uses the hooks.

Call the hooks, which emit the same two trace points.

Fixes: c89d1c879a ("bql: Fix bql_locked status with condvar APIs")
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Cc: Zhao Liu <zhao1.liu@intel.com>
Cc: Stefan Weil <sw@weilnetz.de>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 util/qemu-thread-posix.c | 4 ++--
 util/qemu-thread-win32.c | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/util/qemu-thread-posix.c b/util/qemu-thread-posix.c
index 089606c93f..580cdf8079 100644
--- a/util/qemu-thread-posix.c
+++ b/util/qemu-thread-posix.c
@@ -250,9 +250,9 @@ qemu_cond_timedwait_ts(QemuCond *cond, QemuMutex *mutex, struct timespec *ts,
     int err;
 
     assert(cond->initialized);
-    trace_qemu_mutex_unlock(mutex, file, line);
+    qemu_mutex_pre_unlock(mutex, file, line);
     err = pthread_cond_timedwait(&cond->cond, &mutex->lock, ts);
-    trace_qemu_mutex_locked(mutex, file, line);
+    qemu_mutex_post_lock(mutex, file, line);
     if (err && err != ETIMEDOUT) {
         error_exit(err, __func__);
     }
diff --git a/util/qemu-thread-win32.c b/util/qemu-thread-win32.c
index 3037732a6f..afdacdb5f2 100644
--- a/util/qemu-thread-win32.c
+++ b/util/qemu-thread-win32.c
@@ -177,11 +177,11 @@ bool qemu_cond_timedwait_impl(QemuCond *cond, QemuMutex *mutex, int ms,
     int rc = 0;
 
     assert(cond->initialized);
-    trace_qemu_mutex_unlock(mutex, file, line);
+    qemu_mutex_pre_unlock(mutex, file, line);
     if (!SleepConditionVariableSRW(&cond->var, &mutex->lock, ms, 0)) {
         rc = GetLastError();
     }
-    trace_qemu_mutex_locked(mutex, file, line);
+    qemu_mutex_post_lock(mutex, file, line);
     if (rc && rc != ERROR_TIMEOUT) {
         error_exit(rc, __func__);
     }
-- 
2.53.0

[PATCH 02/10] oslib: add qemu_debugger_attached()

[Denis V. Lunev]

From: Denis V. Lunev <den@openvz.org>

Code that watches for a hang has to tell a hang apart from a debugger
stop, because under a debugger every thread stops and whatever liveness
signal the watcher samples stops changing with them.

Linux reports the tracer in /proc/self/status and Windows has
IsDebuggerPresent(). Answer false elsewhere, which leaves the caller
behaving as it did before the check.

Match the Linux field at the start of a line. The Name: line holds
fifteen characters of comm, enough to spell the field name and appear
ahead of the real one.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Cc: Zhao Liu <zhao1.liu@intel.com>
Cc: Stefan Weil <sw@weilnetz.de>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 include/qemu/osdep.h |  9 +++++++++
 util/oslib-posix.c   | 22 ++++++++++++++++++++++
 util/oslib-win32.c   |  5 +++++
 3 files changed, 36 insertions(+)

diff --git a/include/qemu/osdep.h b/include/qemu/osdep.h
index 1ec5b42230..96afe09632 100644
--- a/include/qemu/osdep.h
+++ b/include/qemu/osdep.h
@@ -658,6 +658,15 @@ bool qemu_write_pidfile(const char *pidfile, Error **errp);
 
 int qemu_get_thread_id(void);
 
+/**
+ * qemu_debugger_attached:
+ *
+ * Returns true if a debugger is currently tracing this process. Meant for
+ * code that would otherwise mistake a debugger stop for a hang, and false
+ * on hosts where this cannot be determined.
+ */
+bool qemu_debugger_attached(void);
+
 /**
  * qemu_kill_thread:
  * @tid: thread id.
diff --git a/util/oslib-posix.c b/util/oslib-posix.c
index dc001da66d..4eb6a40628 100644
--- a/util/oslib-posix.c
+++ b/util/oslib-posix.c
@@ -112,6 +112,28 @@ int qemu_get_thread_id(void)
 #endif
 }
 
+bool qemu_debugger_attached(void)
+{
+#ifdef CONFIG_LINUX
+    static const char key[] = "TracerPid:";
+    g_autofree char *status = NULL;
+    const char *p = NULL;
+
+    if (!g_file_get_contents("/proc/self/status", &status, NULL, NULL)) {
+        return false;
+    }
+    while ((p = strstr(p ? p : status, key))) {
+        if (p == status || p[-1] == '\n') {
+            return atoi(p + sizeof(key) - 1) > 0;
+        }
+        p += sizeof(key) - 1;
+    }
+    return false;
+#else
+    return false;
+#endif
+}
+
 int qemu_kill_thread(int tid, int sig)
 {
 #if defined(__linux__)
diff --git a/util/oslib-win32.c b/util/oslib-win32.c
index 5f3e8f4d98..6663d83151 100644
--- a/util/oslib-win32.c
+++ b/util/oslib-win32.c
@@ -228,6 +228,11 @@ int qemu_get_thread_id(void)
     return GetCurrentThreadId();
 }
 
+bool qemu_debugger_attached(void)
+{
+    return IsDebuggerPresent();
+}
+
 char *
 qemu_get_local_state_dir(void)
 {
-- 
2.53.0

[PATCH 03/10] oslib: add qemu_thread_states()

[Denis V. Lunev]

From: Denis V. Lunev <den@openvz.org>

A watchdog that reports a hang wants to say what every thread was doing,
and that answer is host specific. Put it beside the other process
introspection helpers rather than in the code that happens to need it
first.

Describe each thread by id, name, scheduler state and the wait channel
it is parked on, all of which Linux exposes in /proc/self/task without
privilege. That is already enough to separate a thread blocked on a
userspace lock, which waits in a futex, from one blocked in the kernel,
which names the primitive it sits on. Skip the caller, which is
reporting rather than hanging.

Hosts with no way to answer return NULL, leaving the caller to print
its own message and nothing more.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Cc: Zhao Liu <zhao1.liu@intel.com>
Cc: Stefan Weil <sw@weilnetz.de>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 include/qemu/osdep.h | 10 +++++++
 util/oslib-posix.c   | 71 ++++++++++++++++++++++++++++++++++++++++++++
 util/oslib-win32.c   |  5 ++++
 3 files changed, 86 insertions(+)

diff --git a/include/qemu/osdep.h b/include/qemu/osdep.h
index 96afe09632..8039bf91b8 100644
--- a/include/qemu/osdep.h
+++ b/include/qemu/osdep.h
@@ -667,6 +667,16 @@ int qemu_get_thread_id(void);
  */
 bool qemu_debugger_attached(void);
 
+/**
+ * qemu_thread_states:
+ *
+ * Returns a newly allocated, newline terminated description of every thread
+ * in this process apart from the caller: its id, name, scheduler state and
+ * the wait channel it is parked on, or NULL on a host that cannot report any
+ * of that.
+ */
+char *qemu_thread_states(void);
+
 /**
  * qemu_kill_thread:
  * @tid: thread id.
diff --git a/util/oslib-posix.c b/util/oslib-posix.c
index 4eb6a40628..79887d370c 100644
--- a/util/oslib-posix.c
+++ b/util/oslib-posix.c
@@ -134,6 +134,77 @@ bool qemu_debugger_attached(void)
 #endif
 }
 
+#ifdef CONFIG_LINUX
+
+static char *read_task_file(const char *tid, const char *name)
+{
+    g_autofree char *path = g_strdup_printf("/proc/self/task/%s/%s", tid, name);
+    char *buf = NULL;
+
+    g_file_get_contents(path, &buf, NULL, NULL);
+    return buf;
+}
+
+static void append_thread_state(GString *out, const char *tid)
+{
+    g_autofree char *comm = read_task_file(tid, "comm");
+    g_autofree char *stat = NULL;
+    g_autofree char *wchan = NULL;
+    char state = '?';
+    char *rparen;
+
+    if (!comm) {
+        return;
+    }
+    g_strchomp(comm);
+
+    stat = read_task_file(tid, "stat");
+    if (stat) {
+        /* comm can hold anything, ')' included, so scan from the right */
+        rparen = strrchr(stat, ')');
+        if (rparen && rparen[1] == ' ') {
+            state = rparen[2];
+        }
+    }
+    g_string_append_printf(out, "  %-7s %-16s %c\n", tid, comm, state);
+
+    wchan = read_task_file(tid, "wchan");
+    if (wchan && wchan[0]) {
+        g_string_append_printf(out, "      wchan %s\n", wchan);
+    }
+}
+
+char *qemu_thread_states(void)
+{
+    g_autoptr(GDir) dir = g_dir_open("/proc/self/task", 0, NULL);
+    const char *tid;
+    GString *out;
+
+    if (!dir) {
+        return NULL;
+    }
+    out = g_string_new(NULL);
+    while ((tid = g_dir_read_name(dir))) {
+        if (atoi(tid) != qemu_get_thread_id()) {
+            append_thread_state(out, tid);
+        }
+    }
+    if (!out->len) {
+        g_string_free(out, TRUE);
+        return NULL;
+    }
+    return g_string_free(out, FALSE);
+}
+
+#else
+
+char *qemu_thread_states(void)
+{
+    return NULL;
+}
+
+#endif /* CONFIG_LINUX */
+
 int qemu_kill_thread(int tid, int sig)
 {
 #if defined(__linux__)
diff --git a/util/oslib-win32.c b/util/oslib-win32.c
index 6663d83151..d18886847c 100644
--- a/util/oslib-win32.c
+++ b/util/oslib-win32.c
@@ -233,6 +233,11 @@ bool qemu_debugger_attached(void)
     return IsDebuggerPresent();
 }
 
+char *qemu_thread_states(void)
+{
+    return NULL;
+}
+
 char *
 qemu_get_local_state_dir(void)
 {
-- 
2.53.0

[PATCH 04/10] oslib: report kernel stacks in qemu_thread_states()

[Denis V. Lunev]

From: Denis V. Lunev <den@openvz.org>

A wait channel names the primitive a thread is parked on but not the
path that got it there, and for a thread wedged in a syscall that path
is the answer.

Read /proc/tid/stack when the report is built, and keep the wait
channel for when it cannot be. The file is restricted to CAP_SYS_ADMIN
in the init namespace, so ask once rather than failing per thread, and
say why the report came back thinner. Skip a running thread, which has
no settled stack to unwind, and cap the frames, because the deep end of
a stack is rarely the part that explains the hang.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Cc: Zhao Liu <zhao1.liu@intel.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 include/qemu/osdep.h |  4 ++--
 util/oslib-posix.c   | 52 +++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 51 insertions(+), 5 deletions(-)

diff --git a/include/qemu/osdep.h b/include/qemu/osdep.h
index 8039bf91b8..c15ffd7880 100644
--- a/include/qemu/osdep.h
+++ b/include/qemu/osdep.h
@@ -672,8 +672,8 @@ bool qemu_debugger_attached(void);
  *
  * Returns a newly allocated, newline terminated description of every thread
  * in this process apart from the caller: its id, name, scheduler state and
- * the wait channel it is parked on, or NULL on a host that cannot report any
- * of that.
+ * kernel stack, or NULL on a host that cannot report any of that. Where the
+ * kernel stack is privileged the description falls back to a wait channel.
  */
 char *qemu_thread_states(void);
 
diff --git a/util/oslib-posix.c b/util/oslib-posix.c
index 79887d370c..6a683b849e 100644
--- a/util/oslib-posix.c
+++ b/util/oslib-posix.c
@@ -145,13 +145,39 @@ static char *read_task_file(const char *tid, const char *name)
     return buf;
 }
 
+#define THREAD_STACK_FRAMES 12
+
+/*
+ * /proc/tid/stack is restricted to CAP_SYS_ADMIN in the init namespace, so
+ * ask once instead of failing per thread, and say why the report is thinner.
+ */
+static bool thread_stacks_readable(void)
+{
+    static int readable = -1;
+    g_autofree char *path = NULL;
+    g_autofree char *buf = NULL;
+
+    if (readable >= 0) {
+        return readable;
+    }
+    path = g_strdup_printf("/proc/self/task/%d/stack", qemu_get_thread_id());
+    readable = g_file_get_contents(path, &buf, NULL, NULL);
+    if (!readable) {
+        warn_report("kernel stacks need CAP_SYS_ADMIN, falling back to wchan");
+    }
+    return readable;
+}
+
 static void append_thread_state(GString *out, const char *tid)
 {
     g_autofree char *comm = read_task_file(tid, "comm");
     g_autofree char *stat = NULL;
+    g_autofree char *stack = NULL;
     g_autofree char *wchan = NULL;
+    g_auto(GStrv) lines = NULL;
     char state = '?';
     char *rparen;
+    int i;
 
     if (!comm) {
         return;
@@ -168,9 +194,26 @@ static void append_thread_state(GString *out, const char *tid)
     }
     g_string_append_printf(out, "  %-7s %-16s %c\n", tid, comm, state);
 
-    wchan = read_task_file(tid, "wchan");
-    if (wchan && wchan[0]) {
-        g_string_append_printf(out, "      wchan %s\n", wchan);
+    /* A running thread has no settled kernel stack to unwind */
+    if (state != 'R' && thread_stacks_readable()) {
+        stack = read_task_file(tid, "stack");
+    }
+    if (!stack) {
+        wchan = read_task_file(tid, "wchan");
+        if (wchan && wchan[0]) {
+            g_string_append_printf(out, "      wchan %s\n", wchan);
+        }
+        return;
+    }
+
+    lines = g_strsplit(stack, "\n", 0);
+    for (i = 0; i < THREAD_STACK_FRAMES && lines[i]; i++) {
+        if (lines[i][0]) {
+            g_string_append_printf(out, "      %s\n", lines[i]);
+        }
+    }
+    if (lines[i]) {
+        g_string_append(out, "      ...\n");
     }
 }
 
@@ -183,6 +226,9 @@ char *qemu_thread_states(void)
     if (!dir) {
         return NULL;
     }
+    /* Emit the permission notice ahead of the listing, not inside it */
+    thread_stacks_readable();
+
     out = g_string_new(NULL);
     while ((tid = g_dir_read_name(dir))) {
         if (atoi(tid) != qemu_get_thread_id()) {
-- 
2.53.0

[PATCH 05/10] util/qemu-thread: reserve a signal for thread introspection

[Denis V. Lunev]

From: Denis V. Lunev <den@openvz.org>

qemu_thread_create() blocks every signal in the threads it starts so
that process directed signals reach the main loop and nowhere else.
That reasoning covers delivery the kernel is free to route: kill(2)
picks any thread not blocking the signal, so leaving SIGTERM or SIGCHLD
unblocked in a worker would let signalfd miss them.

A thread directed signal is not routed. tgkill(2) delivers to the named
thread whatever every other mask says, so a signal only ever sent that
way was never the main loop's to receive, and unblocking it takes
nothing from it.

Reserve SIGRTMIN for asking a thread what it is doing, and leave that
one unblocked. Nothing sends it yet, and until something installs a
handler the change is invisible: a real time signal terminates by
default, which is already what a stray one does through the main
thread, where it was never blocked either.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Cc: Zhao Liu <zhao1.liu@intel.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 include/qemu/osdep.h     | 9 +++++++++
 util/qemu-thread-posix.c | 4 ++++
 2 files changed, 13 insertions(+)

diff --git a/include/qemu/osdep.h b/include/qemu/osdep.h
index c15ffd7880..0bd08e0f7f 100644
--- a/include/qemu/osdep.h
+++ b/include/qemu/osdep.h
@@ -667,6 +667,15 @@ int qemu_get_thread_id(void);
  */
 bool qemu_debugger_attached(void);
 
+/*
+ * Signal reserved for asking a thread what it is doing. It is only ever
+ * sent thread directed, so it does not compete for the process directed
+ * signals that qemu routes to the main loop.
+ */
+#if defined(SIGRTMIN) && !defined(_WIN32)
+#define QEMU_SIG_INTROSPECT SIGRTMIN
+#endif
+
 /**
  * qemu_thread_states:
  *
diff --git a/util/qemu-thread-posix.c b/util/qemu-thread-posix.c
index 580cdf8079..66bde85608 100644
--- a/util/qemu-thread-posix.c
+++ b/util/qemu-thread-posix.c
@@ -444,6 +444,10 @@ void qemu_thread_create(QemuThread *thread, const char *name,
     sigdelset(&set, SIGFPE);
     sigdelset(&set, SIGILL);
     /* TODO avoid SIGBUS loss on macOS */
+#ifdef QEMU_SIG_INTROSPECT
+    /* Thread directed, so it was never the iothread's to receive */
+    sigdelset(&set, QEMU_SIG_INTROSPECT);
+#endif
     pthread_sigmask(SIG_SETMASK, &set, &oldset);
 
     qemu_thread_args = g_new0(QemuThreadArgs, 1);
-- 
2.53.0

[PATCH 06/10] oslib: report each thread's program counter

[Denis V. Lunev]

From: Denis V. Lunev <den@openvz.org>

A kernel stack describes a thread asleep in a syscall and says nothing
about one spinning in userspace, which is the other way a thread stops
making progress.

Ask each thread where it is by signalling it and reading the program
counter out of the signal frame. That is a register read and a store,
which a handler may do, unlike unwinding: backtrace() is not async
signal safe and takes the loader lock the first time it runs.

Ask all of them at once, once every passive read is taken. The signal
interrupts whatever a thread was blocked in, so probing earlier would
make the kernel stack and the wait channel describe the interruption
instead of the hang, and probing one at a time would spend the timeout
again for every thread that never answers. Insert the answers from the
back, so the offsets recorded during the reads still point where they
did.

A thread that never answers within the deadline can still run the
handler later, once whatever held it lets go, and by then the slots it
would write to are gone. Count handlers between entry and exit,
withdraw the slots when the deadline expires and wait for the last
handler out before returning to the caller that frees them. A handler
that arrives after the withdrawal finds nothing to write to.

Where the counter sits in the signal frame is a host property, so it is
answered from host/include like the other per-host questions, with a
generic fallback that reports no counter and leaves the host otherwise
untouched. The Linux hosts covered are the ones whose signal frames
linux-user already describes in host-signal.h.

Name the address through dladdr() where there is a symbol for it. There
often is not, because dladdr() sees exported symbols only and a thread
parked inside a library sits in its internals, so fall back to the
object and the offset within it. An offset resolves against the library
afterwards; the raw address does not survive the process.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Cc: Zhao Liu <zhao1.liu@intel.com>
Cc: Daniel P. Berrangé <berrange@redhat.com>
Cc: Marc-André Lureau <marcandre.lureau@redhat.com>
Cc: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 host/include/aarch64/host/signal-pc.h     |  20 +++
 host/include/generic/host/signal-pc.h     |  11 ++
 host/include/loongarch64/host/signal-pc.h |  20 +++
 host/include/ppc64/host/signal-pc.h       |  22 +++
 host/include/riscv64/host/signal-pc.h     |  20 +++
 host/include/s390x/host/signal-pc.h       |  20 +++
 host/include/x86_64/host/signal-pc.h      |  20 +++
 include/qemu/osdep.h                      |   7 +-
 meson.build                               |   2 +
 util/oslib-posix.c                        | 181 +++++++++++++++++++++-
 10 files changed, 318 insertions(+), 5 deletions(-)
 create mode 100644 host/include/aarch64/host/signal-pc.h
 create mode 100644 host/include/generic/host/signal-pc.h
 create mode 100644 host/include/loongarch64/host/signal-pc.h
 create mode 100644 host/include/ppc64/host/signal-pc.h
 create mode 100644 host/include/riscv64/host/signal-pc.h
 create mode 100644 host/include/s390x/host/signal-pc.h
 create mode 100644 host/include/x86_64/host/signal-pc.h

diff --git a/host/include/aarch64/host/signal-pc.h b/host/include/aarch64/host/signal-pc.h
new file mode 100644
index 0000000000..fe60ed1c96
--- /dev/null
+++ b/host/include/aarch64/host/signal-pc.h
@@ -0,0 +1,20 @@
+/*
+ * Program counter of an interrupted aarch64 thread, from its signal frame.
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#ifndef AARCH64_HOST_SIGNAL_PC_H
+#define AARCH64_HOST_SIGNAL_PC_H
+
+#ifdef CONFIG_LINUX
+#define HAVE_HOST_SIGNAL_PC 1
+
+static inline uintptr_t host_signal_pc(const ucontext_t *uc)
+{
+    return uc->uc_mcontext.pc;
+}
+#else
+#define HAVE_HOST_SIGNAL_PC 0
+#endif
+
+#endif
diff --git a/host/include/generic/host/signal-pc.h b/host/include/generic/host/signal-pc.h
new file mode 100644
index 0000000000..8ce3822069
--- /dev/null
+++ b/host/include/generic/host/signal-pc.h
@@ -0,0 +1,11 @@
+/*
+ * No host specific way to read the program counter from a signal frame.
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#ifndef HOST_SIGNAL_PC_H
+#define HOST_SIGNAL_PC_H
+
+#define HAVE_HOST_SIGNAL_PC 0
+
+#endif
diff --git a/host/include/loongarch64/host/signal-pc.h b/host/include/loongarch64/host/signal-pc.h
new file mode 100644
index 0000000000..ccb2ac1c56
--- /dev/null
+++ b/host/include/loongarch64/host/signal-pc.h
@@ -0,0 +1,20 @@
+/*
+ * Program counter of an interrupted loongarch64 thread, from its signal frame.
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#ifndef LOONGARCH64_HOST_SIGNAL_PC_H
+#define LOONGARCH64_HOST_SIGNAL_PC_H
+
+#ifdef CONFIG_LINUX
+#define HAVE_HOST_SIGNAL_PC 1
+
+static inline uintptr_t host_signal_pc(const ucontext_t *uc)
+{
+    return uc->uc_mcontext.__pc;
+}
+#else
+#define HAVE_HOST_SIGNAL_PC 0
+#endif
+
+#endif
diff --git a/host/include/ppc64/host/signal-pc.h b/host/include/ppc64/host/signal-pc.h
new file mode 100644
index 0000000000..1473570a39
--- /dev/null
+++ b/host/include/ppc64/host/signal-pc.h
@@ -0,0 +1,22 @@
+/*
+ * Program counter of an interrupted ppc64 thread, from its signal frame.
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#ifndef PPC64_HOST_SIGNAL_PC_H
+#define PPC64_HOST_SIGNAL_PC_H
+
+#ifdef CONFIG_LINUX
+#include <asm/ptrace.h>
+
+#define HAVE_HOST_SIGNAL_PC 1
+
+static inline uintptr_t host_signal_pc(const ucontext_t *uc)
+{
+    return uc->uc_mcontext.gp_regs[PT_NIP];
+}
+#else
+#define HAVE_HOST_SIGNAL_PC 0
+#endif
+
+#endif
diff --git a/host/include/riscv64/host/signal-pc.h b/host/include/riscv64/host/signal-pc.h
new file mode 100644
index 0000000000..362b446040
--- /dev/null
+++ b/host/include/riscv64/host/signal-pc.h
@@ -0,0 +1,20 @@
+/*
+ * Program counter of an interrupted riscv64 thread, from its signal frame.
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#ifndef RISCV64_HOST_SIGNAL_PC_H
+#define RISCV64_HOST_SIGNAL_PC_H
+
+#ifdef CONFIG_LINUX
+#define HAVE_HOST_SIGNAL_PC 1
+
+static inline uintptr_t host_signal_pc(const ucontext_t *uc)
+{
+    return uc->uc_mcontext.__gregs[REG_PC];
+}
+#else
+#define HAVE_HOST_SIGNAL_PC 0
+#endif
+
+#endif
diff --git a/host/include/s390x/host/signal-pc.h b/host/include/s390x/host/signal-pc.h
new file mode 100644
index 0000000000..6ad809c911
--- /dev/null
+++ b/host/include/s390x/host/signal-pc.h
@@ -0,0 +1,20 @@
+/*
+ * Program counter of an interrupted s390x thread, from its signal frame.
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#ifndef S390X_HOST_SIGNAL_PC_H
+#define S390X_HOST_SIGNAL_PC_H
+
+#ifdef CONFIG_LINUX
+#define HAVE_HOST_SIGNAL_PC 1
+
+static inline uintptr_t host_signal_pc(const ucontext_t *uc)
+{
+    return uc->uc_mcontext.psw.addr;
+}
+#else
+#define HAVE_HOST_SIGNAL_PC 0
+#endif
+
+#endif
diff --git a/host/include/x86_64/host/signal-pc.h b/host/include/x86_64/host/signal-pc.h
new file mode 100644
index 0000000000..c94ad2e976
--- /dev/null
+++ b/host/include/x86_64/host/signal-pc.h
@@ -0,0 +1,20 @@
+/*
+ * Program counter of an interrupted x86_64 thread, from its signal frame.
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#ifndef X86_64_HOST_SIGNAL_PC_H
+#define X86_64_HOST_SIGNAL_PC_H
+
+#ifdef CONFIG_LINUX
+#define HAVE_HOST_SIGNAL_PC 1
+
+static inline uintptr_t host_signal_pc(const ucontext_t *uc)
+{
+    return uc->uc_mcontext.gregs[REG_RIP];
+}
+#else
+#define HAVE_HOST_SIGNAL_PC 0
+#endif
+
+#endif
diff --git a/include/qemu/osdep.h b/include/qemu/osdep.h
index 0bd08e0f7f..00eeb5dcc9 100644
--- a/include/qemu/osdep.h
+++ b/include/qemu/osdep.h
@@ -680,9 +680,10 @@ bool qemu_debugger_attached(void);
  * qemu_thread_states:
  *
  * Returns a newly allocated, newline terminated description of every thread
- * in this process apart from the caller: its id, name, scheduler state and
- * kernel stack, or NULL on a host that cannot report any of that. Where the
- * kernel stack is privileged the description falls back to a wait channel.
+ * in this process apart from the caller: its id, name, scheduler state, the
+ * program counter it stopped at and its kernel stack, or NULL on a host that
+ * cannot report any of that. A privileged kernel stack falls back to a wait
+ * channel, and a host with no way to read a program counter omits it.
  */
 char *qemu_thread_states(void);
 
diff --git a/meson.build b/meson.build
index cfac634cf1..25e8842918 100644
--- a/meson.build
+++ b/meson.build
@@ -2646,6 +2646,8 @@ config_host_data.set('HAVE_GETIFADDRS', cc.has_function('getifaddrs'))
 config_host_data.set('HAVE_GLIB_WITH_SLICE_ALLOCATOR', glib_has_gslice)
 config_host_data.set('HAVE_OPENPTY', cc.has_function('openpty', dependencies: util))
 config_host_data.set('HAVE_STRCHRNUL', cc.has_function('strchrnul', prefix: osdep_prefix))
+config_host_data.set('CONFIG_DLADDR', cc.has_function('dladdr',
+                     prefix: osdep_prefix + '#include <dlfcn.h>'))
 config_host_data.set('HAVE_SYSTEM_FUNCTION', cc.has_function('system', prefix: '#include <stdlib.h>'))
 if rbd.found()
   config_host_data.set('HAVE_RBD_NAMESPACE_EXISTS',
diff --git a/util/oslib-posix.c b/util/oslib-posix.c
index 6a683b849e..ddc2b48df2 100644
--- a/util/oslib-posix.c
+++ b/util/oslib-posix.c
@@ -58,8 +58,13 @@
 #include <lwp.h>
 #endif
 
+#ifdef CONFIG_DLADDR
+#include <dlfcn.h>
+#endif
+
 #include "qemu/memalign.h"
 #include "qemu/mmap-alloc.h"
+#include "host/signal-pc.h"
 
 #define MAX_MEM_PREALLOC_THREAD_COUNT 32
 
@@ -168,8 +173,164 @@ static bool thread_stacks_readable(void)
     return readable;
 }
 
-static void append_thread_state(GString *out, const char *tid)
+/* Where a thread was, and where its line belongs in the report */
+typedef struct {
+    int tid;
+    size_t at;
+    void *pc;
+    int done;
+} ThreadPc;
+
+#if HAVE_HOST_SIGNAL_PC
+
+/* How long to wait for the answers, however many threads were asked */
+#define THREAD_PC_WAIT_MS 100
+
+/*
+ * Published to the handler, which runs in whichever thread was interrupted
+ * and so may only touch this with atomics: every qemu synchronisation
+ * primitive is built on a mutex, and none of them may be taken here.
+ * pc_handlers counts handlers between entry and exit, so the slots can be
+ * withdrawn and the last handler waited out before the caller frees them.
+ */
+static ThreadPc *pc_slots;
+static int pc_nslots;
+static int pc_handlers;
+
+static void thread_pc_handler(int sig, siginfo_t *si, void *ctx)
+{
+    ThreadPc *slots;
+    int me, i, n;
+
+    qatomic_inc(&pc_handlers);
+    smp_mb__after_rmw();
+    slots = qatomic_load_acquire(&pc_slots);
+    if (slots) {
+        n = qatomic_read(&pc_nslots);
+        me = qemu_get_thread_id();
+        for (i = 0; i < n; i++) {
+            if (qatomic_read(&slots[i].tid) == me) {
+                slots[i].pc = (void *)host_signal_pc(ctx);
+                qatomic_store_release(&slots[i].done, 1);
+                break;
+            }
+        }
+    }
+    qatomic_dec(&pc_handlers);
+}
+
+static bool thread_pc_armed(void)
+{
+    static int armed = -1;
+    struct sigaction sa;
+
+    if (armed >= 0) {
+        return armed;
+    }
+    memset(&sa, 0, sizeof(sa));
+    sa.sa_sigaction = thread_pc_handler;
+    sa.sa_flags = SA_SIGINFO | SA_RESTART;
+    sigemptyset(&sa.sa_mask);
+    armed = sigaction(QEMU_SIG_INTROSPECT, &sa, NULL) == 0;
+    return armed;
+}
+
+/*
+ * Ask every thread at once. Asking one at a time would spend the timeout
+ * again for each thread that never answers, and the threads that never
+ * answer are the ones a hung qemu has most of.
+ */
+static void thread_pc_collect(GArray *slots)
 {
+    int64_t deadline;
+    guint i;
+    int left;
+
+    if (!slots->len || !thread_pc_armed()) {
+        return;
+    }
+    qatomic_set(&pc_nslots, slots->len);
+    qatomic_store_release(&pc_slots, &g_array_index(slots, ThreadPc, 0));
+
+    for (i = 0; i < slots->len; i++) {
+        ThreadPc *slot = &g_array_index(slots, ThreadPc, i);
+
+        if (qemu_kill_thread(slot->tid, QEMU_SIG_INTROSPECT) < 0) {
+            qatomic_store_release(&slot->done, -1);
+        }
+    }
+
+    deadline = g_get_monotonic_time() + THREAD_PC_WAIT_MS * 1000;
+    do {
+        left = 0;
+        for (i = 0; i < slots->len; i++) {
+            if (!qatomic_load_acquire(&g_array_index(slots, ThreadPc,
+                                                     i).done)) {
+                left++;
+            }
+        }
+        if (!left) {
+            break;
+        }
+        g_usleep(1000);
+    } while (g_get_monotonic_time() < deadline);
+
+    qatomic_set(&pc_slots, NULL);
+    /* Pairs with the barrier in thread_pc_handler() */
+    smp_mb();
+    while (qatomic_read(&pc_handlers)) {
+        g_usleep(1000);
+    }
+}
+
+static void insert_thread_pc(GString *out, const ThreadPc *slot)
+{
+    g_autofree char *line = NULL;
+    void *pc = slot->pc;
+#ifdef CONFIG_DLADDR
+    Dl_info info = { };
+#endif
+
+    if (qatomic_read(&slot->done) != 1 || !pc) {
+        return;
+    }
+#ifdef CONFIG_DLADDR
+    if (dladdr(pc, &info) && info.dli_sname) {
+        line = g_strdup_printf("      pc %p %s+0x%tx\n", pc, info.dli_sname,
+                               (char *)pc - (char *)info.dli_saddr);
+    } else if (info.dli_fname && info.dli_fbase) {
+        /*
+         * dladdr() only knows exported symbols, so an address inside a
+         * library's own internals has none. The object and the offset
+         * within it still resolve later, the raw address does not.
+         */
+        g_autofree char *obj = g_path_get_basename(info.dli_fname);
+
+        line = g_strdup_printf("      pc %p %s+0x%tx\n", pc, obj,
+                               (char *)pc - (char *)info.dli_fbase);
+    }
+#endif
+    if (!line) {
+        line = g_strdup_printf("      pc %p\n", pc);
+    }
+    g_string_insert(out, slot->at, line);
+}
+
+#else
+
+static void thread_pc_collect(GArray *slots)
+{
+}
+
+static void insert_thread_pc(GString *out, const ThreadPc *slot)
+{
+}
+
+#endif /* HAVE_HOST_SIGNAL_PC */
+
+static void append_thread_state(GString *out, GArray *slots, const char *tid)
+{
+    ThreadPc slot = { };
     g_autofree char *comm = read_task_file(tid, "comm");
     g_autofree char *stat = NULL;
     g_autofree char *stack = NULL;
@@ -194,6 +355,10 @@ static void append_thread_state(GString *out, const char *tid)
     }
     g_string_append_printf(out, "  %-7s %-16s %c\n", tid, comm, state);
 
+    slot.tid = atoi(tid);
+    slot.at = out->len;
+    g_array_append_val(slots, slot);
+
     /* A running thread has no settled kernel stack to unwind */
     if (state != 'R' && thread_stacks_readable()) {
         stack = read_task_file(tid, "stack");
@@ -220,8 +385,10 @@ static void append_thread_state(GString *out, const char *tid)
 char *qemu_thread_states(void)
 {
     g_autoptr(GDir) dir = g_dir_open("/proc/self/task", 0, NULL);
+    g_autoptr(GArray) slots = g_array_new(FALSE, TRUE, sizeof(ThreadPc));
     const char *tid;
     GString *out;
+    guint i;
 
     if (!dir) {
         return NULL;
@@ -232,13 +399,23 @@ char *qemu_thread_states(void)
     out = g_string_new(NULL);
     while ((tid = g_dir_read_name(dir))) {
         if (atoi(tid) != qemu_get_thread_id()) {
-            append_thread_state(out, tid);
+            append_thread_state(out, slots, tid);
         }
     }
     if (!out->len) {
         g_string_free(out, TRUE);
         return NULL;
     }
+
+    /*
+     * Only now, with every passive read taken, disturb the threads. Insert
+     * from the back so the offsets recorded above stay where they were.
+     */
+    thread_pc_collect(slots);
+    for (i = slots->len; i > 0; i--) {
+        insert_thread_pc(out, &g_array_index(slots, ThreadPc, i - 1));
+    }
+
     return g_string_free(out, FALSE);
 }
 
-- 
2.53.0

[PATCH 07/10] system/cpus: report a BQL held past a deadline

[Denis V. Lunev]

From: Denis V. Lunev <den@openvz.org>

A vCPU thread reacquires the BQL on every exit that needs userspace, so
a lock held for tens of seconds stops those vCPUs dead and the guest
reports its own soft lockup. Nothing in qemu records that this
happened, which leaves a guest panic as the only evidence and no way to
tell a stuck holder from a guest that wedged on its own.

Add a BQL watchdog. A thread of its own samples a counter advanced on
every acquire and every release, so it is odd while the lock is held.
One odd value seen throughout the window means a single owner held the
lock for the whole window, because any handover would have advanced the
counter twice.

Advance it in bql_update_status(), which the mutex hooks already call
for exactly those two transitions, so a cond-wait that drops and
retakes the lock is covered without the wrappers knowing, and so is any
caller that reaches the BQL through the plain mutex API. Record the
holder's tid beside it, under the same lock, because the thread listing
alone cannot tell a thread parked on its own condvar from one blocked
on the BQL. Cache the tid per thread, since gettid is a syscall and
this is the lock path.

On expiry, name the holder and list every thread's state. The listing
separates a thread blocked on the BQL, which waits in a futex, from one
idle inside KVM, which waits in kvm_vcpu_block, and names the syscall a
wedged holder is stuck in. Report once per episode, or a lock held for
a minute buries the log in copies of one answer. Put the time on the
line unless -msg timestamp=on already does: a hold is only worth
reporting if it can be lined up against what the guest saw, and that
option has to be asked for in advance.

The deadline has to clear the longest legitimate hold, and migration
setup can hold the BQL for a while, so the watchdog stays off unless
asked for. The bql-watchdog-ms machine property is the whole interface:
it sets the deadline, on the command line or over qom-set at any time,
and 0 turns the watchdog off. Reject a deadline above a minute: a hold
that long has already shown up as a soft lockup inside the guest.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Cc: Zhao Liu <zhao1.liu@intel.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 hw/core/machine.c     |  27 ++++++++++
 include/system/cpus.h |   4 ++
 qemu-options.hx       |   6 +++
 system/cpus.c         | 120 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 157 insertions(+)

diff --git a/hw/core/machine.c b/hw/core/machine.c
index 8939ae1666..b55af89516 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -382,6 +382,27 @@ static void machine_set_dump_guest_core(Object *obj, bool value, Error **errp)
     ms->dump_guest_core = value;
 }
 
+static void machine_get_bql_watchdog_ms(Object *obj, Visitor *v,
+                                        const char *name, void *opaque,
+                                        Error **errp)
+{
+    uint64_t value = bql_watchdog_get_timeout_ms();
+
+    visit_type_uint64(v, name, &value, errp);
+}
+
+static void machine_set_bql_watchdog_ms(Object *obj, Visitor *v,
+                                        const char *name, void *opaque,
+                                        Error **errp)
+{
+    uint64_t value;
+
+    if (!visit_type_uint64(v, name, &value, errp)) {
+        return;
+    }
+    bql_watchdog_set_timeout_ms(value, errp);
+}
+
 static bool machine_get_new_accel_vmfd_on_reset(Object *obj, Error **errp)
 {
     MachineState *ms = MACHINE(obj);
@@ -1147,6 +1168,12 @@ static void machine_class_init(ObjectClass *oc, const void *data)
     object_class_property_set_description(oc, "dump-guest-core",
         "Include guest memory in a core dump");
 
+    object_class_property_add(oc, "bql-watchdog-ms", "uint64",
+        machine_get_bql_watchdog_ms, machine_set_bql_watchdog_ms, NULL, NULL);
+    object_class_property_set_description(oc, "bql-watchdog-ms",
+        "Watchdog deadline for a Big QEMU Lock hold, in ms. "
+        "Default: 0 (disabled)");
+
     object_class_property_add_bool(oc, "x-change-vmfd-on-reset",
         machine_get_new_accel_vmfd_on_reset,
         machine_set_new_accel_vmfd_on_reset);
diff --git a/include/system/cpus.h b/include/system/cpus.h
index bf36b82376..0e1e63bd85 100644
--- a/include/system/cpus.h
+++ b/include/system/cpus.h
@@ -40,4 +40,8 @@ void cpu_synchronize_all_post_reset(void);
 void cpu_synchronize_all_post_init(void);
 void cpu_synchronize_all_pre_loadvm(void);
 
+/* Watchdog deadline for a BQL hold. 0 disables the watchdog. */
+bool bql_watchdog_set_timeout_ms(uint64_t ms, Error **errp);
+uint64_t bql_watchdog_get_timeout_ms(void);
+
 #endif
diff --git a/qemu-options.hx b/qemu-options.hx
index 2f6863180b..9600620d34 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -32,6 +32,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \
     "                vmport=on|off|auto controls emulation of vmport (default: auto)\n"
     "                dump-guest-core=on|off include guest memory in a core dump (default=on)\n"
     "                mem-merge=on|off controls memory merge support (default: on)\n"
+    "                bql-watchdog-ms=ms report a Big QEMU Lock held longer than ms (default: 0, off)\n"
     "                aes-key-wrap=on|off controls support for AES key wrapping (default=on)\n"
     "                dea-key-wrap=on|off controls support for DEA key wrapping (default=on)\n"
     "                suppress-vmdesc=on|off disables self-describing migration (default=off)\n"
@@ -88,6 +89,11 @@ SRST
         supported by the host, de-duplicates identical memory pages
         among VMs instances (enabled by default).
 
+    ``bql-watchdog-ms=ms``
+        Report when the Big QEMU Lock has been held for more than ``ms``
+        milliseconds, up to a minute. The default is 0, which disables the
+        watchdog.
+
     ``aes-key-wrap=on|off``
         Enables or disables AES key wrapping support on s390-ccw hosts.
         This feature controls whether AES wrapping keys will be created
diff --git a/system/cpus.c b/system/cpus.c
index e11a5aab6a..9e03f303f2 100644
--- a/system/cpus.c
+++ b/system/cpus.c
@@ -31,6 +31,7 @@
 #include "exec/cpu-common.h"
 #include "qemu/thread.h"
 #include "qemu/main-loop.h"
+#include "qemu/error-report.h"
 #include "qemu/plugin.h"
 #include "system/cpus.h"
 #include "qemu/guest-random.h"
@@ -367,12 +368,129 @@ static QemuCond qemu_cpu_cond;
 /* system init */
 static QemuCond qemu_pause_cond;
 
+/*
+ * Advanced by bql_update_status() below, so odd while the BQL is held and
+ * even while it is free. A sampler that sees one odd value throughout its
+ * window knows a single owner held the lock for the whole window: any
+ * handover would have advanced it twice.
+ */
+static unsigned int bql_seq;
+
+/* Who advanced bql_seq to its current odd value, 0 while the BQL is free */
+static int bql_owner_tid;
+
+static int bql_this_tid(void)
+{
+    static __thread int tid;
+
+    if (tid) {
+        return tid;
+    }
+    tid = qemu_get_thread_id();
+    return tid;
+}
+
+/* Poll this many times per timeout, to bound how late the watchdog reacts */
+#define BQL_WATCHDOG_POLLS 4
+
+#define BQL_WATCHDOG_MAX_MS (60 * 1000)
+
+static QemuThread bql_watchdog_thread;
+static QemuSemaphore bql_watchdog_sem;
+static uint64_t bql_watchdog_timeout_ms;
+
+static void bql_report_thread_states(void)
+{
+    g_autofree char *states = qemu_thread_states();
+
+    if (!states) {
+        return;
+    }
+    qemu_flockfile(stderr);
+    fputs(states, stderr);
+    qemu_funlockfile(stderr);
+}
+
+static void *bql_watchdog_fn(void *opaque)
+{
+    unsigned int last = qatomic_read(&bql_seq);
+    unsigned int reported = 0;
+    int stuck_polls = 0;
+
+    for (;;) {
+        uint64_t timeout_ms = qatomic_read(&bql_watchdog_timeout_ms);
+        g_autofree char *when = NULL;
+        unsigned int seq;
+
+        if (!timeout_ms) {
+            /* Park rather than exit, so re-arming needs no new thread */
+            qemu_sem_wait(&bql_watchdog_sem);
+            last = qatomic_read(&bql_seq);
+            stuck_polls = 0;
+            continue;
+        }
+        qemu_sem_timedwait(&bql_watchdog_sem,
+                           MAX(timeout_ms / BQL_WATCHDOG_POLLS, 1));
+
+        seq = qatomic_read(&bql_seq);
+        if (seq != last || !(seq & 1)) {
+            last = seq;
+            stuck_polls = 0;
+            continue;
+        }
+        if (++stuck_polls < BQL_WATCHDOG_POLLS) {
+            continue;
+        }
+        if (seq == reported) {
+            continue;
+        }
+        reported = seq;
+        if (!message_with_timestamp) {
+            when = real_time_iso8601();
+        }
+        error_report("%s%sBQL held for more than %" PRIu64 " ms by thread %d",
+                     when ?: "", when ? " " : "", timeout_ms,
+                     qatomic_read(&bql_owner_tid));
+        bql_report_thread_states();
+    }
+}
+
+uint64_t bql_watchdog_get_timeout_ms(void)
+{
+    return qatomic_read(&bql_watchdog_timeout_ms);
+}
+
+bool bql_watchdog_set_timeout_ms(uint64_t ms, Error **errp)
+{
+    static bool started;
+
+    assert(bql_locked());
+    if (ms > BQL_WATCHDOG_MAX_MS) {
+        error_setg(errp, "BQL watchdog deadline must not exceed %d ms",
+                   BQL_WATCHDOG_MAX_MS);
+        return false;
+    }
+    qatomic_set(&bql_watchdog_timeout_ms, ms);
+    if (!ms) {
+        return true;
+    }
+    if (!started) {
+        started = true;
+        qemu_thread_create(&bql_watchdog_thread, "bql-watchdog",
+                           bql_watchdog_fn, NULL, QEMU_THREAD_DETACHED);
+    } else {
+        qemu_sem_post(&bql_watchdog_sem);
+    }
+    return true;
+}
+
 void qemu_init_cpu_loop(void)
 {
     qemu_init_sigbus();
     qemu_cond_init(&qemu_cpu_cond);
     qemu_cond_init(&qemu_pause_cond);
     qemu_mutex_init(&bql);
+    qemu_sem_init(&bql_watchdog_sem, 0);
 
     qemu_thread_get_self(&io_thread);
 }
@@ -477,6 +595,8 @@ void bql_update_status(bool locked)
     /* This function should only be used when an update happened.. */
     assert(bql_locked() != locked);
     set_bql_locked(locked);
+    qatomic_set(&bql_owner_tid, locked ? bql_this_tid() : 0);
+    qatomic_set(&bql_seq, qatomic_read(&bql_seq) + 1);
 }
 
 static uint32_t bql_unlock_blocked;
-- 
2.53.0

[PATCH 08/10] system/cpus: hold off the BQL watchdog under a debugger

[Denis V. Lunev]

From: Denis V. Lunev <den@openvz.org>

A debugger stops every thread, so the sequence counter freezes at an
odd value and the watchdog cannot tell that apart from an owner that
never lets go. Attaching gdb to a running guest would report a hang
that is not one.

Skip the report while a tracer is attached, and restart the count so
that a detach gets a full deadline before the watchdog can fire. Any
ptracer sets TracerPid, so strace holds the watchdog off as well.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Cc: Zhao Liu <zhao1.liu@intel.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 system/cpus.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/system/cpus.c b/system/cpus.c
index 9e03f303f2..03a07e4824 100644
--- a/system/cpus.c
+++ b/system/cpus.c
@@ -441,6 +441,10 @@ static void *bql_watchdog_fn(void *opaque)
         if (++stuck_polls < BQL_WATCHDOG_POLLS) {
             continue;
         }
+        if (qemu_debugger_attached()) {
+            stuck_polls = 0;
+            continue;
+        }
         if (seq == reported) {
             continue;
         }
-- 
2.53.0

[PATCH 09/10] system/cpus: add bql-watchdog-abort

[Denis V. Lunev]

From: Denis V. Lunev <den@openvz.org>

The report says where each thread was when the watchdog looked, not
how it got there. A core carries every thread's whole stack, at the
price of the guest. A qemu whose BQL has been held this long is not
running its vCPUs anyway, so on a host where the answer is worth more
than the guest, abort after the report.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Cc: Zhao Liu <zhao1.liu@intel.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 hw/core/machine.c     | 17 +++++++++++++++++
 include/system/cpus.h |  4 ++++
 qemu-options.hx       |  5 +++++
 system/cpus.c         | 19 +++++++++++++++++++
 4 files changed, 45 insertions(+)

diff --git a/hw/core/machine.c b/hw/core/machine.c
index b55af89516..2abb9e6295 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -403,6 +403,17 @@ static void machine_set_bql_watchdog_ms(Object *obj, Visitor *v,
     bql_watchdog_set_timeout_ms(value, errp);
 }
 
+static bool machine_get_bql_watchdog_abort(Object *obj, Error **errp)
+{
+    return bql_watchdog_get_abort();
+}
+
+static void machine_set_bql_watchdog_abort(Object *obj, bool value,
+                                           Error **errp)
+{
+    bql_watchdog_set_abort(value);
+}
+
 static bool machine_get_new_accel_vmfd_on_reset(Object *obj, Error **errp)
 {
     MachineState *ms = MACHINE(obj);
@@ -1174,6 +1185,12 @@ static void machine_class_init(ObjectClass *oc, const void *data)
         "Watchdog deadline for a Big QEMU Lock hold, in ms. "
         "Default: 0 (disabled)");
 
+    object_class_property_add_bool(oc, "bql-watchdog-abort",
+        machine_get_bql_watchdog_abort, machine_set_bql_watchdog_abort);
+    object_class_property_set_description(oc, "bql-watchdog-abort",
+        "Abort after the bql-watchdog-ms report, to leave a core dump. "
+        "Default: off");
+
     object_class_property_add_bool(oc, "x-change-vmfd-on-reset",
         machine_get_new_accel_vmfd_on_reset,
         machine_set_new_accel_vmfd_on_reset);
diff --git a/include/system/cpus.h b/include/system/cpus.h
index 0e1e63bd85..14dbc8ff0d 100644
--- a/include/system/cpus.h
+++ b/include/system/cpus.h
@@ -44,4 +44,8 @@ void cpu_synchronize_all_pre_loadvm(void);
 bool bql_watchdog_set_timeout_ms(uint64_t ms, Error **errp);
 uint64_t bql_watchdog_get_timeout_ms(void);
 
+/* Abort after reporting, to leave a core behind. */
+void bql_watchdog_set_abort(bool enable);
+bool bql_watchdog_get_abort(void);
+
 #endif
diff --git a/qemu-options.hx b/qemu-options.hx
index 9600620d34..f43d8ca480 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -33,6 +33,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \
     "                dump-guest-core=on|off include guest memory in a core dump (default=on)\n"
     "                mem-merge=on|off controls memory merge support (default: on)\n"
     "                bql-watchdog-ms=ms report a Big QEMU Lock held longer than ms (default: 0, off)\n"
+    "                bql-watchdog-abort=on|off abort after the bql-watchdog-ms report (default: off)\n"
     "                aes-key-wrap=on|off controls support for AES key wrapping (default=on)\n"
     "                dea-key-wrap=on|off controls support for DEA key wrapping (default=on)\n"
     "                suppress-vmdesc=on|off disables self-describing migration (default=off)\n"
@@ -94,6 +95,10 @@ SRST
         milliseconds, up to a minute. The default is 0, which disables the
         watchdog.
 
+    ``bql-watchdog-abort=on|off``
+        Abort after the ``bql-watchdog-ms`` report, to leave a core dump.
+        The default is off.
+
     ``aes-key-wrap=on|off``
         Enables or disables AES key wrapping support on s390-ccw hosts.
         This feature controls whether AES wrapping keys will be created
diff --git a/system/cpus.c b/system/cpus.c
index 03a07e4824..a200a07707 100644
--- a/system/cpus.c
+++ b/system/cpus.c
@@ -398,6 +398,7 @@ static int bql_this_tid(void)
 static QemuThread bql_watchdog_thread;
 static QemuSemaphore bql_watchdog_sem;
 static uint64_t bql_watchdog_timeout_ms;
+static bool bql_watchdog_abort;
 
 static void bql_report_thread_states(void)
 {
@@ -411,6 +412,16 @@ static void bql_report_thread_states(void)
     qemu_funlockfile(stderr);
 }
 
+bool bql_watchdog_get_abort(void)
+{
+    return qatomic_read(&bql_watchdog_abort);
+}
+
+void bql_watchdog_set_abort(bool enable)
+{
+    qatomic_set(&bql_watchdog_abort, enable);
+}
+
 static void *bql_watchdog_fn(void *opaque)
 {
     unsigned int last = qatomic_read(&bql_seq);
@@ -456,6 +467,14 @@ static void *bql_watchdog_fn(void *opaque)
                      when ?: "", when ? " " : "", timeout_ms,
                      qatomic_read(&bql_owner_tid));
         bql_report_thread_states();
+        if (!qatomic_read(&bql_watchdog_abort)) {
+            continue;
+        }
+        /*
+         * A core carries the whole of every stack, and a qemu whose BQL has
+         * been held this long has stopped running its vCPUs anyway.
+         */
+        abort();
     }
 }
 
-- 
2.53.0

[PATCH 10/10] tests/functional: cover the BQL watchdog

[Denis V. Lunev]

From: Denis V. Lunev <den@openvz.org>

Hold the BQL from the main thread and check that the watchdog says so.

human-monitor-command is not a coroutine command, so a write issued
through it runs to completion under AIO_WAIT_WHILE() in the main
thread, which keeps the BQL for as long as the backend takes to answer.
Point that at a null-co node with a latency ten times the deadline and
the hold is both long enough to report and short enough to end on its
own.

Arm the watchdog after launch rather than on the command line, because
machine creation takes the BQL in bursts of its own and a deadline this
short is meant for the hold, not for startup.

Check the quiet case too. A counter that stops distinguishing a held
lock from a free one would report a hang on a healthy qemu, and that
failure is invisible to a test that only looks for the report.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Cc: Zhao Liu <zhao1.liu@intel.com>
Cc: Thomas Huth <th.huth+qemu@posteo.eu>
Cc: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 tests/functional/generic/meson.build          |  1 +
 tests/functional/generic/test_bql_watchdog.py | 60 +++++++++++++++++++
 2 files changed, 61 insertions(+)
 create mode 100755 tests/functional/generic/test_bql_watchdog.py

diff --git a/tests/functional/generic/meson.build b/tests/functional/generic/meson.build
index c94105c62e..c20c2a91ea 100644
--- a/tests/functional/generic/meson.build
+++ b/tests/functional/generic/meson.build
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-or-later
 
 tests_generic_system = [
+  'bql_watchdog',
   'empty_cpu_model',
   'info_usernet',
   'linters',
diff --git a/tests/functional/generic/test_bql_watchdog.py b/tests/functional/generic/test_bql_watchdog.py
new file mode 100755
index 0000000000..193dad15e3
--- /dev/null
+++ b/tests/functional/generic/test_bql_watchdog.py
@@ -0,0 +1,60 @@
+#!/usr/bin/env python3
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Test the BQL watchdog against a main thread blocked on IO.
+#
+# Copyright (C) 2026 Virtuozzo International GmbH
+
+from qemu_test import QemuSystemTest
+
+deadline_ms = 100
+hold_ns = 1000 * 1000 * 1000
+
+
+class BqlWatchdog(QemuSystemTest):
+
+    def setUp(self):
+        super().setUp()
+        self.set_machine('none')
+        self.vm.add_args('-nodefaults')
+        self.vm.add_args('-blockdev',
+                         f'driver=null-co,node-name=null0,size=1048576,'
+                         f'latency-ns={hold_ns}')
+        self.vm.launch()
+
+    def arm(self, ms):
+        self.vm.cmd('qom-set', path='/machine',
+                    property='bql-watchdog-ms', value=ms)
+
+    def hold_the_bql(self):
+        """
+        human-monitor-command is not a coroutine command, so the write runs
+        to completion under AIO_WAIT_WHILE() in the main thread, which keeps
+        the BQL for as long as null-co takes to answer.
+        """
+        self.vm.cmd('human-monitor-command',
+                    command_line='qemu-io null0 "write 0 4k"')
+
+    def log_after_shutdown(self):
+        """The log only reaches the harness once the VM is reaped"""
+        self.vm.shutdown()
+        return self.vm.get_log() or ''
+
+    def test_reports_a_hold(self):
+        self.arm(deadline_ms)
+        self.hold_the_bql()
+        self.arm(0)
+
+        log = self.log_after_shutdown()
+        self.assertIn(f'BQL held for more than {deadline_ms} ms', log)
+
+    def test_silent_while_disarmed(self):
+        self.hold_the_bql()
+
+        log = self.log_after_shutdown()
+        self.assertNotIn('BQL held for more than', log)
+
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
-- 
2.53.0

Re: [PATCH 10/10] tests/functional: cover the BQL watchdog

[Thomas Huth]

On 15/09/2026 22.34, Denis V. Lunev wrote:
> From: Denis V. Lunev <den@openvz.org>
> 
> Hold the BQL from the main thread and check that the watchdog says so.
> 
> human-monitor-command is not a coroutine command, so a write issued
> through it runs to completion under AIO_WAIT_WHILE() in the main
> thread, which keeps the BQL for as long as the backend takes to answer.
> Point that at a null-co node with a latency ten times the deadline and
> the hold is both long enough to report and short enough to end on its
> own.
> 
> Arm the watchdog after launch rather than on the command line, because
> machine creation takes the BQL in bursts of its own and a deadline this
> short is meant for the hold, not for startup.
> 
> Check the quiet case too. A counter that stops distinguishing a held
> lock from a free one would report a hang on a healthy qemu, and that
> failure is invisible to a test that only looks for the report.
...
> diff --git a/tests/functional/generic/test_bql_watchdog.py b/tests/functional/generic/test_bql_watchdog.py
> new file mode 100755
> index 0000000000..193dad15e3
> --- /dev/null
> +++ b/tests/functional/generic/test_bql_watchdog.py
> @@ -0,0 +1,60 @@
> +#!/usr/bin/env python3
> +#
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Test the BQL watchdog against a main thread blocked on IO.
> +#
> +# Copyright (C) 2026 Virtuozzo International GmbH
> +
> +from qemu_test import QemuSystemTest
> +
> +deadline_ms = 100
> +hold_ns = 1000 * 1000 * 1000
> +
> +
> +class BqlWatchdog(QemuSystemTest):
> +
> +    def setUp(self):
> +        super().setUp()
> +        self.set_machine('none')
> +        self.vm.add_args('-nodefaults')
> +        self.vm.add_args('-blockdev',
> +                         f'driver=null-co,node-name=null0,size=1048576,'
> +                         f'latency-ns={hold_ns}')
> +        self.vm.launch()
> +
> +    def arm(self, ms):

Maybe it's just me (so feel free to ignore this comment), but when reading 
"arm" I'm thinking of qemu-system-arm ... maybe call the function 
"enable_wd" or so instead?

> +        self.vm.cmd('qom-set', path='/machine',
> +                    property='bql-watchdog-ms', value=ms)
> +
> +    def hold_the_bql(self):
> +        """
> +        human-monitor-command is not a coroutine command, so the write runs
> +        to completion under AIO_WAIT_WHILE() in the main thread, which keeps
> +        the BQL for as long as null-co takes to answer.
> +        """
> +        self.vm.cmd('human-monitor-command',
> +                    command_line='qemu-io null0 "write 0 4k"')

HMP has recently been made optional, you likely need a 
self.skipTestIfNoHMP() somewhere now to make sure that this test does not 
fail if HMP has not been compiled into the binary. See also:

  https://gitlab.com/qemu-project/qemu/-/commit/6a0195cd57b231c2047232c8

  Thomas

Re: [PATCH 10/10] tests/functional: cover the BQL watchdog

[Marc-André Lureau]

Hi

On Wed, Sep 16, 2026 at 8:58 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 15/09/2026 22.34, Denis V. Lunev wrote:
> > From: Denis V. Lunev <den@openvz.org>
> >
> > Hold the BQL from the main thread and check that the watchdog says so.
> >
> > human-monitor-command is not a coroutine command, so a write issued
> > through it runs to completion under AIO_WAIT_WHILE() in the main
> > thread, which keeps the BQL for as long as the backend takes to answer.
> > Point that at a null-co node with a latency ten times the deadline and
> > the hold is both long enough to report and short enough to end on its
> > own.
> >
> > Arm the watchdog after launch rather than on the command line, because
> > machine creation takes the BQL in bursts of its own and a deadline this
> > short is meant for the hold, not for startup.
> >
> > Check the quiet case too. A counter that stops distinguishing a held
> > lock from a free one would report a hang on a healthy qemu, and that
> > failure is invisible to a test that only looks for the report.
> ...
> > diff --git a/tests/functional/generic/test_bql_watchdog.py b/tests/functional/generic/test_bql_watchdog.py
> > new file mode 100755
> > index 0000000000..193dad15e3
> > --- /dev/null
> > +++ b/tests/functional/generic/test_bql_watchdog.py
> > @@ -0,0 +1,60 @@
> > +#!/usr/bin/env python3
> > +#
> > +# SPDX-License-Identifier: GPL-2.0-or-later
> > +#
> > +# Test the BQL watchdog against a main thread blocked on IO.
> > +#
> > +# Copyright (C) 2026 Virtuozzo International GmbH
> > +
> > +from qemu_test import QemuSystemTest
> > +
> > +deadline_ms = 100
> > +hold_ns = 1000 * 1000 * 1000
> > +
> > +
> > +class BqlWatchdog(QemuSystemTest):
> > +
> > +    def setUp(self):
> > +        super().setUp()
> > +        self.set_machine('none')
> > +        self.vm.add_args('-nodefaults')
> > +        self.vm.add_args('-blockdev',
> > +                         f'driver=null-co,node-name=null0,size=1048576,'
> > +                         f'latency-ns={hold_ns}')
> > +        self.vm.launch()
> > +
> > +    def arm(self, ms):
>
> Maybe it's just me (so feel free to ignore this comment), but when reading
> "arm" I'm thinking of qemu-system-arm ... maybe call the function
> "enable_wd" or so instead?
>
> > +        self.vm.cmd('qom-set', path='/machine',
> > +                    property='bql-watchdog-ms', value=ms)
> > +
> > +    def hold_the_bql(self):
> > +        """
> > +        human-monitor-command is not a coroutine command, so the write runs
> > +        to completion under AIO_WAIT_WHILE() in the main thread, which keeps
> > +        the BQL for as long as null-co takes to answer.
> > +        """
> > +        self.vm.cmd('human-monitor-command',
> > +                    command_line='qemu-io null0 "write 0 4k"')
>
> HMP has recently been made optional, you likely need a
> self.skipTestIfNoHMP() somewhere now to make sure that this test does not
> fail if HMP has not been compiled into the binary. See also:
>
>   https://gitlab.com/qemu-project/qemu/-/commit/6a0195cd57b231c2047232c8

Better yet, use QMP, amd x-qemu-io.
thanks

Re: [PATCH 10/10] tests/functional: cover the BQL watchdog

[Denis V. Lunev]

On 9/16/26 08:44, Marc-André Lureau wrote:
> Hi
>
> On Wed, Sep 16, 2026 at 8:58 AM Thomas Huth <thuth@redhat.com> wrote:
>> On 15/09/2026 22.34, Denis V. Lunev wrote:
>>> From: Denis V. Lunev <den@openvz.org>
>>>
>>> Hold the BQL from the main thread and check that the watchdog says so.
>>>
>>> human-monitor-command is not a coroutine command, so a write issued
>>> through it runs to completion under AIO_WAIT_WHILE() in the main
>>> thread, which keeps the BQL for as long as the backend takes to answer.
>>> Point that at a null-co node with a latency ten times the deadline and
>>> the hold is both long enough to report and short enough to end on its
>>> own.
>>>
>>> Arm the watchdog after launch rather than on the command line, because
>>> machine creation takes the BQL in bursts of its own and a deadline this
>>> short is meant for the hold, not for startup.
>>>
>>> Check the quiet case too. A counter that stops distinguishing a held
>>> lock from a free one would report a hang on a healthy qemu, and that
>>> failure is invisible to a test that only looks for the report.
>> ...
>>> diff --git a/tests/functional/generic/test_bql_watchdog.py b/tests/functional/generic/test_bql_watchdog.py
>>> new file mode 100755
>>> index 0000000000..193dad15e3
>>> --- /dev/null
>>> +++ b/tests/functional/generic/test_bql_watchdog.py
>>> @@ -0,0 +1,60 @@
>>> +#!/usr/bin/env python3
>>> +#
>>> +# SPDX-License-Identifier: GPL-2.0-or-later
>>> +#
>>> +# Test the BQL watchdog against a main thread blocked on IO.
>>> +#
>>> +# Copyright (C) 2026 Virtuozzo International GmbH
>>> +
>>> +from qemu_test import QemuSystemTest
>>> +
>>> +deadline_ms = 100
>>> +hold_ns = 1000 * 1000 * 1000
>>> +
>>> +
>>> +class BqlWatchdog(QemuSystemTest):
>>> +
>>> +    def setUp(self):
>>> +        super().setUp()
>>> +        self.set_machine('none')
>>> +        self.vm.add_args('-nodefaults')
>>> +        self.vm.add_args('-blockdev',
>>> +                         f'driver=null-co,node-name=null0,size=1048576,'
>>> +                         f'latency-ns={hold_ns}')
>>> +        self.vm.launch()
>>> +
>>> +    def arm(self, ms):
>> Maybe it's just me (so feel free to ignore this comment), but when reading
>> "arm" I'm thinking of qemu-system-arm ... maybe call the function
>> "enable_wd" or so instead?
>>
>>> +        self.vm.cmd('qom-set', path='/machine',
>>> +                    property='bql-watchdog-ms', value=ms)
>>> +
>>> +    def hold_the_bql(self):
>>> +        """th
>>> +        human-monitor-command is not a coroutine command, so the write runs
>>> +        to completion under AIO_WAIT_WHILE() in the main thread, which keeps
>>> +        the BQL for as long as null-co takes to answer.
>>> +        """
>>> +        self.vm.cmd('human-monitor-command',
>>> +                    command_line='qemu-io null0 "write 0 4k"')
>> HMP has recently been made optional, you likely need a
>> self.skipTestIfNoHMP() somewhere now to make sure that this test does not
>> fail if HMP has not been compiled into the binary. See also:
>>
>>   https://gitlab.com/qemu-project/qemu/-/commit/6a0195cd57b231c2047232c8
> Better yet, use QMP, amd x-qemu-io.
> thanks
yes, this is just another old series kept in depths
of my git.

The fix is minimal and I'll do it. The most important
question is the concept itself in the previous patches.

Thank you for your time,
    Den

Re: [PATCH 0/10] system: add a watchdog for the Big QEMU Lock

[Fabiano Rosas]

"Denis V. Lunev" <den@openvz.org> writes:

> A vCPU thread reacquires the BQL on every exit that needs userspace. A
> holder that keeps it for tens of seconds stops every vCPU, the guest
> reports a soft lockup or panics, and qemu itself records nothing. The
> guest log is the only evidence, and it cannot say whether the hypervisor
> held the lock or the guest wedged on its own.
>
> This series adds a BQL watchdog. A sampler thread watches a sequence
> counter that bql_update_status() advances on every acquire and release.
> One odd value across the whole window means one owner held the lock for
> the whole window. On expiry the watchdog names the holder and prints the
> state of every other thread:
>
>   qemu-system-x86_64: 2026-09-15T15:53:57Z BQL held for more than 100 ms by thread 168146
>     168146  qemu-system-x86  S
>         pc 0x785a4dbcb884 liburing.so.2+0x2884
>         wchan io_cqring_wait
>     168147  call_rcu         S
>         pc 0x785a4d534acb syscall+0x1b
>         wchan futex_do_wait
>     168148  IO mon_iothread  S
>         pc 0x785a4d4ac802 libc.so.6+0xac802
>         wchan poll_schedule_timeout.constprop.0
>
> With CAP_SYS_ADMIN the wchan line becomes the kernel stack. The pc line
> comes from signalling each thread and reading the program counter out of
> its signal frame, which is what tells a thread spinning in userspace
> from one asleep in a syscall.
>
> The watchdog is off by default and stays quiet under a debugger. Two
> machine properties drive it: bql-watchdog-ms sets the deadline, on the
> command line or over qom-set at any time, and bql-watchdog-abort turns
> the report into a core dump for hosts where the answer is worth more
> than the guest.
>
> Patches 1 to 6 are the pieces: a fix for bql_locked across the timed
> condvar wait, a debugger check, the per-thread state reader and its
> kernel stack and program counter extensions, and the signal reserved for
> the latter. Patches 7 to 9 are the watchdog itself. Patch 10 is a
> functional test that holds the BQL through an HMP qemu-io write to a
> null-co node with a latency longer than the deadline.
>
> The program counter lives at a host specific offset in the signal frame,
> so patch 6 reads it through a host/include header. The Linux headers for
> aarch64, ppc64, riscv64, s390x and loongarch64 repeat what linux-user's
> host-signal.h already knows about those frames, but only the x86_64 one
> has been compiled here, and the Windows half of patch 1 has not been
> compiled either.
>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Richard Henderson <richard.henderson@linaro.org>
> Cc: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
> Cc: Zhao Liu <zhao1.liu@intel.com>
> Signed-off-by: Denis V. Lunev <den@openvz.org>
>
> Denis V. Lunev (10):
>   util/qemu-thread: track bql_locked across the timed condvar wait
>   oslib: add qemu_debugger_attached()
>   oslib: add qemu_thread_states()
>   oslib: report kernel stacks in qemu_thread_states()
>   util/qemu-thread: reserve a signal for thread introspection
>   oslib: report each thread's program counter
>   system/cpus: report a BQL held past a deadline
>   system/cpus: hold off the BQL watchdog under a debugger
>   system/cpus: add bql-watchdog-abort
>   tests/functional: cover the BQL watchdog
>
>  host/include/aarch64/host/signal-pc.h         |  20 ++
>  host/include/generic/host/signal-pc.h         |  11 +
>  host/include/loongarch64/host/signal-pc.h     |  20 ++
>  host/include/ppc64/host/signal-pc.h           |  22 ++
>  host/include/riscv64/host/signal-pc.h         |  20 ++
>  host/include/s390x/host/signal-pc.h           |  20 ++
>  host/include/x86_64/host/signal-pc.h          |  20 ++
>  hw/core/machine.c                             |  44 +++
>  include/qemu/osdep.h                          |  29 ++
>  include/system/cpus.h                         |   8 +
>  meson.build                                   |   2 +
>  qemu-options.hx                               |  11 +
>  system/cpus.c                                 | 143 ++++++++
>  tests/functional/generic/meson.build          |   1 +
>  tests/functional/generic/test_bql_watchdog.py |  60 ++++
>  util/oslib-posix.c                            | 316 ++++++++++++++++++
>  util/oslib-win32.c                            |  10 +
>  util/qemu-thread-posix.c                      |   8 +-
>  util/qemu-thread-win32.c                      |   4 +-
>  19 files changed, 765 insertions(+), 4 deletions(-)
>  create mode 100644 host/include/aarch64/host/signal-pc.h
>  create mode 100644 host/include/generic/host/signal-pc.h
>  create mode 100644 host/include/loongarch64/host/signal-pc.h
>  create mode 100644 host/include/ppc64/host/signal-pc.h
>  create mode 100644 host/include/riscv64/host/signal-pc.h
>  create mode 100644 host/include/s390x/host/signal-pc.h
>  create mode 100644 host/include/x86_64/host/signal-pc.h
>  create mode 100755 tests/functional/generic/test_bql_watchdog.py

The problem you state is definitely a pervasive one and I've encountered
it many times. It's a pain do debug. I'm working (on and off, you know
how it goes) on a perf improvement for dirty logging disabling, which is
basically this issue after migrate_cancel. I've been writing custom code
to track the blocking windows, so this series looks like it could help a
lot.

I'm putting it on my queue to review, I'll make some time for it soon.

One other thing I have in flight is a change to qtests to allow testing
a linux guest while a worker thread does some BQL intensive work in
parallel such as serial IO. I'm currently checking the guest serial
console for panics and lockups, but the watchdog would report the issue
right at the source.