QEMU-Devel Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: Fabiano Rosas <farosas@suse.de>
To: "Denis V. Lunev" <den@openvz.org>, qemu-devel@nongnu.org
Cc: den@openvz.org, "Paolo Bonzini" <pbonzini@redhat.com>,
	"Richard Henderson" <richard.henderson@linaro.org>,
	"Philippe Mathieu-Daudé" <philmd@oss.qualcomm.com>,
	"Zhao Liu" <zhao1.liu@intel.com>
Subject: Re: [PATCH 04/10] oslib: report kernel stacks in qemu_thread_states()
Date: Tue, 17 Nov 2026 15:13:54 -0300	[thread overview]
Message-ID: <87h5hfs4v1.fsf@suse.de> (raw)
In-Reply-To: <20260915203455.580999-5-den@openvz.org>

"Denis V. Lunev" <den@openvz.org> writes:

> From: Denis V. Lunev <den@openvz.org>
>
> A wait channel names the primitive a thread is parked on but not the
> path that got it there, and for a thread wedged in a syscall that path
> is the answer.
>

Talking internally with other people that also deal with these kinds of
issues I got the impression (and that's my opinion as well) that having
the warning message and wchan is already very useful. I wonder if we
should separate the other "secondary" sources of data somehow so the
user can opt out of them.

For instance, the stack information will most likely never be available
in a production setup, so that could be behind a knob. Likewise, the PC
addition in patch 6 is nice, but slightly invasive.

One other passing thought is about large hosts with many CPUs. There's
probably a sweet spot of workload vs. amount of cpus vs. watchdog_ms
that would trigger a long sequence of processing and printing stacks
that the admin might not even spot in the logs if the VM operation
continues apparently unaffected.

I wonder whether the above might lead to the feature just being off all
the time.

Maybe putting a precedence to the information we print and providing a
bql-watchdog-level=[1-5] option so a more conservative environment could
just enable the warning message, then wchan, and so on. Just a thought,
I actually like the simplicity of having just _ms and _abort.

> Read /proc/tid/stack when the report is built, and keep the wait
> channel for when it cannot be. The file is restricted to CAP_SYS_ADMIN
> in the init namespace, so ask once rather than failing per thread, and
> say why the report came back thinner. Skip a running thread, which has
> no settled stack to unwind, and cap the frames, because the deep end of

What this "settled stack" statement means in practice? I think having
even a partial view of the stack up to a point would already be
elucidative, even if the last few frames are bogus.

For example, I see a few:

  1822761 CPU 0/KVM        R
      pc 0x7fb3bbcfae0f ioctl+0x41
      wchan 0

whereas something like this looks more interesting:

  1821448 CPU 0/KVM        R
      pc 0x7fbe91119e0f ioctl+0x41
      [<0>] preempt_schedule_thunk+0x16/0x30
      [<0>] srcu_gp_start_if_needed+0x38d/0x520
      [<0>] __synchronize_srcu.part.0+0x7d/0xb0
      [<0>] kvm_swap_active_memslots+0x133/0x180 [kvm]
      [<0>] kvm_set_memslot+0x47f/0x600 [kvm]
      [<0>] kvm_vm_ioctl+0x1113/0x16e0 [kvm]
      [<0>] __x64_sys_ioctl+0x97/0xc0
      [<0>] do_syscall_64+0x7d/0x160
      [<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e

> a stack is rarely the part that explains the hang.
>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Richard Henderson <richard.henderson@linaro.org>
> Cc: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
> Cc: Zhao Liu <zhao1.liu@intel.com>
> Signed-off-by: Denis V. Lunev <den@openvz.org>
> ---
>  include/qemu/osdep.h |  4 ++--
>  util/oslib-posix.c   | 52 +++++++++++++++++++++++++++++++++++++++++---
>  2 files changed, 51 insertions(+), 5 deletions(-)
>
> diff --git a/include/qemu/osdep.h b/include/qemu/osdep.h
> index 8039bf91b8..c15ffd7880 100644
> --- a/include/qemu/osdep.h
> +++ b/include/qemu/osdep.h
> @@ -672,8 +672,8 @@ bool qemu_debugger_attached(void);
>   *
>   * Returns a newly allocated, newline terminated description of every thread
>   * in this process apart from the caller: its id, name, scheduler state and
> - * the wait channel it is parked on, or NULL on a host that cannot report any
> - * of that.
> + * kernel stack, or NULL on a host that cannot report any of that. Where the
> + * kernel stack is privileged the description falls back to a wait channel.
>   */
>  char *qemu_thread_states(void);
>  
> diff --git a/util/oslib-posix.c b/util/oslib-posix.c
> index 79887d370c..6a683b849e 100644
> --- a/util/oslib-posix.c
> +++ b/util/oslib-posix.c
> @@ -145,13 +145,39 @@ static char *read_task_file(const char *tid, const char *name)
>      return buf;
>  }
>  
> +#define THREAD_STACK_FRAMES 12
> +
> +/*
> + * /proc/tid/stack is restricted to CAP_SYS_ADMIN in the init namespace, so
> + * ask once instead of failing per thread, and say why the report is thinner.
> + */
> +static bool thread_stacks_readable(void)
> +{
> +    static int readable = -1;
> +    g_autofree char *path = NULL;
> +    g_autofree char *buf = NULL;
> +
> +    if (readable >= 0) {
> +        return readable;
> +    }
> +    path = g_strdup_printf("/proc/self/task/%d/stack", qemu_get_thread_id());
> +    readable = g_file_get_contents(path, &buf, NULL, NULL);
> +    if (!readable) {
> +        warn_report("kernel stacks need CAP_SYS_ADMIN, falling back to wchan");
> +    }
> +    return readable;
> +}
> +
>  static void append_thread_state(GString *out, const char *tid)
>  {
>      g_autofree char *comm = read_task_file(tid, "comm");
>      g_autofree char *stat = NULL;
> +    g_autofree char *stack = NULL;
>      g_autofree char *wchan = NULL;
> +    g_auto(GStrv) lines = NULL;
>      char state = '?';
>      char *rparen;
> +    int i;
>  
>      if (!comm) {
>          return;
> @@ -168,9 +194,26 @@ static void append_thread_state(GString *out, const char *tid)
>      }
>      g_string_append_printf(out, "  %-7s %-16s %c\n", tid, comm, state);
>  
> -    wchan = read_task_file(tid, "wchan");
> -    if (wchan && wchan[0]) {
> -        g_string_append_printf(out, "      wchan %s\n", wchan);
> +    /* A running thread has no settled kernel stack to unwind */
> +    if (state != 'R' && thread_stacks_readable()) {
> +        stack = read_task_file(tid, "stack");
> +    }
> +    if (!stack) {
> +        wchan = read_task_file(tid, "wchan");
> +        if (wchan && wchan[0]) {
> +            g_string_append_printf(out, "      wchan %s\n", wchan);
> +        }
> +        return;
> +    }
> +
> +    lines = g_strsplit(stack, "\n", 0);
> +    for (i = 0; i < THREAD_STACK_FRAMES && lines[i]; i++) {
> +        if (lines[i][0]) {
> +            g_string_append_printf(out, "      %s\n", lines[i]);
> +        }
> +    }
> +    if (lines[i]) {
> +        g_string_append(out, "      ...\n");
>      }
>  }
>  
> @@ -183,6 +226,9 @@ char *qemu_thread_states(void)
>      if (!dir) {
>          return NULL;
>      }
> +    /* Emit the permission notice ahead of the listing, not inside it */
> +    thread_stacks_readable();
> +
>      out = g_string_new(NULL);
>      while ((tid = g_dir_read_name(dir))) {
>          if (atoi(tid) != qemu_get_thread_id()) {


  reply	other threads:[~2026-11-17 18:14 UTC|newest]

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-09-15 20:34 [PATCH 0/10] system: add a watchdog for the Big QEMU Lock Denis V. Lunev
2026-09-15 20:34 ` [PATCH 01/10] util/qemu-thread: track bql_locked across the timed condvar wait Denis V. Lunev
2026-10-14 22:36   ` Fabiano Rosas
2026-09-15 20:34 ` [PATCH 02/10] oslib: add qemu_debugger_attached() Denis V. Lunev
2026-11-17 13:45   ` Fabiano Rosas
2026-11-17 14:38   ` Peter Maydell
2026-09-15 20:34 ` [PATCH 03/10] oslib: add qemu_thread_states() Denis V. Lunev
2026-11-17 14:02   ` Fabiano Rosas
2026-09-15 20:34 ` [PATCH 04/10] oslib: report kernel stacks in qemu_thread_states() Denis V. Lunev
2026-11-17 18:13   ` Fabiano Rosas [this message]
2026-09-15 20:34 ` [PATCH 05/10] util/qemu-thread: reserve a signal for thread introspection Denis V. Lunev
2026-09-15 20:34 ` [PATCH 06/10] oslib: report each thread's program counter Denis V. Lunev
2026-11-17 18:19   ` Fabiano Rosas
2026-09-15 20:34 ` [PATCH 07/10] system/cpus: report a BQL held past a deadline Denis V. Lunev
2026-11-17 18:24   ` Fabiano Rosas
2026-09-15 20:34 ` [PATCH 08/10] system/cpus: hold off the BQL watchdog under a debugger Denis V. Lunev
2026-11-17 18:25   ` Fabiano Rosas
2026-09-15 20:34 ` [PATCH 09/10] system/cpus: add bql-watchdog-abort Denis V. Lunev
2026-10-14 23:30   ` Fabiano Rosas
2026-09-15 20:34 ` [PATCH 10/10] tests/functional: cover the BQL watchdog Denis V. Lunev
2026-09-16  4:57   ` Thomas Huth
2026-09-16  6:44     ` Marc-André Lureau
2026-09-16  7:56       ` Denis V. Lunev
2026-09-18 21:56 ` [PATCH 0/10] system: add a watchdog for the Big QEMU Lock Fabiano Rosas

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87h5hfs4v1.fsf@suse.de \
    --to=farosas@suse.de \
    --cc=den@openvz.org \
    --cc=pbonzini@redhat.com \
    --cc=philmd@oss.qualcomm.com \
    --cc=qemu-devel@nongnu.org \
    --cc=richard.henderson@linaro.org \
    --cc=zhao1.liu@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox