[PATCH 0/3] scripts/qemu-gdb: Make coroutine dumps to work with coredumps

[PATCH 0/3] scripts/qemu-gdb: Make coroutine dumps to work with coredumps

[Peter Xu]

Coroutines are used in many cases in block layers. It's also used in live
migration when on destination side, and it'll be handy to diagnose crashes
within a coroutine when we want to also know what other coroutines are
doing.

This series adds initial support for that, not pretty but it should start
working.  Since we can't use the trick to modify registers on the fly in
non-live gdb sessions, we do manual unwinds.

Thanks,

Peter Xu (3):
  scripts/qemu-gdb: Always do full stack dump for python errors
  scripts/qemu-gdb: Simplify fs_base fetching for coroutines
  scripts/qemu-gdb: Support coroutine dumps in coredumps

 scripts/qemu-gdb.py          |  2 +
 scripts/qemugdb/coroutine.py | 73 ++++++++++++++++++++++--------------
 2 files changed, 47 insertions(+), 28 deletions(-)

-- 
2.47.0

[PATCH 1/3] scripts/qemu-gdb: Always do full stack dump for python errors

[Peter Xu]

It's easier for either debugging plugin errors, or issue reports.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 scripts/qemu-gdb.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/qemu-gdb.py b/scripts/qemu-gdb.py
index 4d2a9f6c43..cfae94a2e9 100644
--- a/scripts/qemu-gdb.py
+++ b/scripts/qemu-gdb.py
@@ -45,3 +45,5 @@ def __init__(self):
 # Default to silently passing through SIGUSR1, because QEMU sends it
 # to itself a lot.
 gdb.execute('handle SIGUSR1 pass noprint nostop')
+# Always print full stack for python errors, easier to debug and report issues
+gdb.execute('set python print-stack full')
-- 
2.47.0

[PATCH 2/3] scripts/qemu-gdb: Simplify fs_base fetching for coroutines

[Peter Xu]

There're a bunch of code trying to fetch fs_base in different ways.  IIUC
the simplest way instead is "$fs_base".  It also has the benefit that it'll
work for both live gdb session or coredumps.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 scripts/qemugdb/coroutine.py | 23 ++---------------------
 1 file changed, 2 insertions(+), 21 deletions(-)

diff --git a/scripts/qemugdb/coroutine.py b/scripts/qemugdb/coroutine.py
index 7db46d4b68..20f76ed37b 100644
--- a/scripts/qemugdb/coroutine.py
+++ b/scripts/qemugdb/coroutine.py
@@ -13,28 +13,9 @@
 
 VOID_PTR = gdb.lookup_type('void').pointer()
 
-def get_fs_base():
-    '''Fetch %fs base value using arch_prctl(ARCH_GET_FS).  This is
-       pthread_self().'''
-    # %rsp - 120 is scratch space according to the SystemV ABI
-    old = gdb.parse_and_eval('*(uint64_t*)($rsp - 120)')
-    gdb.execute('call (int)arch_prctl(0x1003, $rsp - 120)', False, True)
-    fs_base = gdb.parse_and_eval('*(uint64_t*)($rsp - 120)')
-    gdb.execute('set *(uint64_t*)($rsp - 120) = %s' % old, False, True)
-    return fs_base
-
 def pthread_self():
-    '''Fetch pthread_self() from the glibc start_thread function.'''
-    f = gdb.newest_frame()
-    while f.name() != 'start_thread':
-        f = f.older()
-        if f is None:
-            return get_fs_base()
-
-    try:
-        return f.read_var("arg")
-    except ValueError:
-        return get_fs_base()
+    '''Fetch the base address of TLS.'''
+    return gdb.parse_and_eval("$fs_base")
 
 def get_glibc_pointer_guard():
     '''Fetch glibc pointer guard value'''
-- 
2.47.0

[PATCH 3/3] scripts/qemu-gdb: Support coroutine dumps in coredumps

[Peter Xu]

Dumping coroutines don't yet work with coredumps.  Let's make it work.

We still kept most of the old code because they can be either more
flexible, or prettier.  Only add the fallbacks when they stop working.

Currently the raw unwind is pretty ugly, but it works, like this:

(gdb) qemu bt
Coroutine at 0x7fc474728748:

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 scripts/qemugdb/coroutine.py | 50 +++++++++++++++++++++++++++++++-----
 1 file changed, 43 insertions(+), 7 deletions(-)

diff --git a/scripts/qemugdb/coroutine.py b/scripts/qemugdb/coroutine.py
index 20f76ed37b..b29ee16205 100644
--- a/scripts/qemugdb/coroutine.py
+++ b/scripts/qemugdb/coroutine.py
@@ -46,9 +46,30 @@ def get_jmpbuf_regs(jmpbuf):
         'r15': jmpbuf[JB_R15],
         'rip': glibc_ptr_demangle(jmpbuf[JB_PC], pointer_guard) }
 
-def bt_jmpbuf(jmpbuf):
-    '''Backtrace a jmpbuf'''
-    regs = get_jmpbuf_regs(jmpbuf)
+def symbol_lookup(addr):
+    # Example: "__clone3 + 44 in section .text of /lib64/libc.so.6"
+    result = gdb.execute(f"info symbol {hex(addr)}", to_string=True).strip()
+    return result.split(" in ")[0]
+
+def dump_backtrace(regs):
+    '''
+    Backtrace dump with raw registers, mimic GDB command 'bt'.
+    '''
+    # Here only rbp and rip that matter..
+    rbp = regs['rbp']
+    rip = regs['rip']
+    i = 0
+
+    while rbp:
+        print(f"#{i}\t{symbol_lookup(rip)}")
+        rip = gdb.parse_and_eval(f"*(uint64_t *)(uint64_t)({hex(rbp)} + 8)")
+        rbp = gdb.parse_and_eval(f"*(uint64_t *)(uint64_t)({hex(rbp)})")
+        i += 1
+
+def dump_backtrace_live(regs):
+    '''
+    Backtrace dump with gdb's 'bt' command, only usable in a live session.
+    '''
     old = dict()
 
     # remember current stack frame and select the topmost
@@ -69,6 +90,17 @@ def bt_jmpbuf(jmpbuf):
 
     selected_frame.select()
 
+def bt_jmpbuf(jmpbuf):
+    '''Backtrace a jmpbuf'''
+    regs = get_jmpbuf_regs(jmpbuf)
+    try:
+        # This reuses gdb's "bt" command, which can be slightly prettier
+        # but only works with live sessions.
+        dump_backtrace_live(regs)
+    except:
+        # If above doesn't work, fallback to poor man's unwind
+        dump_backtrace(regs)
+
 def co_cast(co):
     return co.cast(gdb.lookup_type('CoroutineUContext').pointer())
 
@@ -101,10 +133,14 @@ def invoke(self, arg, from_tty):
 
         gdb.execute("bt")
 
-        if gdb.parse_and_eval("qemu_in_coroutine()") == False:
-            return
-
-        co_ptr = gdb.parse_and_eval("qemu_coroutine_self()")
+        try:
+            # This only works with a live session
+            co_ptr = gdb.parse_and_eval("qemu_coroutine_self()")
+            if co_ptr == False:
+                return
+        except:
+            # Fallback to use hard-coded ucontext vars if it's coredump
+            co_ptr = gdb.parse_and_eval("co_tls_current")
 
         while True:
             co = co_cast(co_ptr)
-- 
2.47.0

Re: [PATCH 3/3] scripts/qemu-gdb: Support coroutine dumps in coredumps

[Peter Xu]

On Wed, Dec 11, 2024 at 03:17:39PM -0500, Peter Xu wrote:
> Dumping coroutines don't yet work with coredumps.  Let's make it work.
> 
> We still kept most of the old code because they can be either more
> flexible, or prettier.  Only add the fallbacks when they stop working.
> 
> Currently the raw unwind is pretty ugly, but it works, like this:
> 
> (gdb) qemu bt
> Coroutine at 0x7fc474728748:

It didn't get commited.. I forgot to indent.  It looks like this:

  (gdb) qemu bt
  #0  process_incoming_migration_co (opaque=0x0) at ../migration/migration.c:788
  #1  0x000055cf3894d4d9 in coroutine_trampoline (i0=1565638480, i1=21967) at ../util/coroutine-ucontext.c:175
  #2  0x00007fc481f72f40 in ??? () at /lib64/libc.so.6
  #3  0x00007ffc0c74a520 in ??? ()
  #4  0x0000000000000000 in ??? ()
  Coroutine at 0x7fc474728748:
  #0      qemu_coroutine_switch + 120
  #1      qemu_aio_coroutine_enter + 357
  #2      qemu_coroutine_enter + 35
  #3      migration_incoming_process + 44
  #4      migration_ioc_process_incoming + 491
  #5      migration_channel_process_incoming + 146
  #6      socket_accept_incoming_migration + 119
  #7      qio_net_listener_channel_func + 132
  #8      qio_channel_fd_source_dispatch + 79
  #9      g_main_context_dispatch_unlocked.lto_priv + 316
  #10     g_main_context_dispatch + 37
  #11     glib_pollfds_poll + 91
  #12     os_host_main_loop_wait + 129
  #13     main_loop_wait + 204
  #14     qemu_main_loop + 42
  #15     qemu_default_main + 20
  #16     main + 41
  #17     __libc_start_call_main + 120
  #18     __libc_start_main_impl + 139

> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  scripts/qemugdb/coroutine.py | 50 +++++++++++++++++++++++++++++++-----
>  1 file changed, 43 insertions(+), 7 deletions(-)
> 
> diff --git a/scripts/qemugdb/coroutine.py b/scripts/qemugdb/coroutine.py
> index 20f76ed37b..b29ee16205 100644
> --- a/scripts/qemugdb/coroutine.py
> +++ b/scripts/qemugdb/coroutine.py
> @@ -46,9 +46,30 @@ def get_jmpbuf_regs(jmpbuf):
>          'r15': jmpbuf[JB_R15],
>          'rip': glibc_ptr_demangle(jmpbuf[JB_PC], pointer_guard) }
>  
> -def bt_jmpbuf(jmpbuf):
> -    '''Backtrace a jmpbuf'''
> -    regs = get_jmpbuf_regs(jmpbuf)
> +def symbol_lookup(addr):
> +    # Example: "__clone3 + 44 in section .text of /lib64/libc.so.6"
> +    result = gdb.execute(f"info symbol {hex(addr)}", to_string=True).strip()
> +    return result.split(" in ")[0]
> +
> +def dump_backtrace(regs):
> +    '''
> +    Backtrace dump with raw registers, mimic GDB command 'bt'.
> +    '''
> +    # Here only rbp and rip that matter..
> +    rbp = regs['rbp']
> +    rip = regs['rip']
> +    i = 0
> +
> +    while rbp:
> +        print(f"#{i}\t{symbol_lookup(rip)}")
> +        rip = gdb.parse_and_eval(f"*(uint64_t *)(uint64_t)({hex(rbp)} + 8)")
> +        rbp = gdb.parse_and_eval(f"*(uint64_t *)(uint64_t)({hex(rbp)})")
> +        i += 1
> +
> +def dump_backtrace_live(regs):
> +    '''
> +    Backtrace dump with gdb's 'bt' command, only usable in a live session.
> +    '''
>      old = dict()
>  
>      # remember current stack frame and select the topmost
> @@ -69,6 +90,17 @@ def bt_jmpbuf(jmpbuf):
>  
>      selected_frame.select()
>  
> +def bt_jmpbuf(jmpbuf):
> +    '''Backtrace a jmpbuf'''
> +    regs = get_jmpbuf_regs(jmpbuf)
> +    try:
> +        # This reuses gdb's "bt" command, which can be slightly prettier
> +        # but only works with live sessions.
> +        dump_backtrace_live(regs)
> +    except:
> +        # If above doesn't work, fallback to poor man's unwind
> +        dump_backtrace(regs)
> +
>  def co_cast(co):
>      return co.cast(gdb.lookup_type('CoroutineUContext').pointer())
>  
> @@ -101,10 +133,14 @@ def invoke(self, arg, from_tty):
>  
>          gdb.execute("bt")
>  
> -        if gdb.parse_and_eval("qemu_in_coroutine()") == False:
> -            return
> -
> -        co_ptr = gdb.parse_and_eval("qemu_coroutine_self()")
> +        try:
> +            # This only works with a live session
> +            co_ptr = gdb.parse_and_eval("qemu_coroutine_self()")
> +            if co_ptr == False:
> +                return
> +        except:
> +            # Fallback to use hard-coded ucontext vars if it's coredump
> +            co_ptr = gdb.parse_and_eval("co_tls_current")
>  
>          while True:
>              co = co_cast(co_ptr)
> -- 
> 2.47.0
> 

-- 
Peter Xu

Re: [PATCH 0/3] scripts/qemu-gdb: Make coroutine dumps to work with coredumps

[Fabiano Rosas]

Peter Xu <peterx@redhat.com> writes:

> Coroutines are used in many cases in block layers. It's also used in live
> migration when on destination side, and it'll be handy to diagnose crashes
> within a coroutine when we want to also know what other coroutines are
> doing.

Not sure if you've seen this message on the list:

https://lore.kernel.org/r/f0ebccca-7a17-4da8-ac4a-71cf6d69abc3@mtasv.net

Re: [PATCH 0/3] scripts/qemu-gdb: Make coroutine dumps to work with coredumps

[Peter Xu]

On Wed, Dec 11, 2024 at 05:25:10PM -0300, Fabiano Rosas wrote:
> Peter Xu <peterx@redhat.com> writes:
> 
> > Coroutines are used in many cases in block layers. It's also used in live
> > migration when on destination side, and it'll be handy to diagnose crashes
> > within a coroutine when we want to also know what other coroutines are
> > doing.
> 
> Not sure if you've seen this message on the list:
> 
> https://lore.kernel.org/r/f0ebccca-7a17-4da8-ac4a-71cf6d69abc3@mtasv.net

No I didn't.  I only started looking at this because I got a bug a few days
ago that I need to look at the main coroutine where dest crashed, then
Stefan told me this script and also told me it only works with live session.
Ideally I'll need coredump debug-ability, then I figured it isn't too hard.

I saw that it didn't yet land gdb, and it's much more involved even if it
could be more generic.  Not sure how the block developers think, personally
I prefer this much smaller change because it works on old systems, where I
can easily install gdb with package managers.

-- 
Peter Xu

Re: [PATCH 0/3] scripts/qemu-gdb: Make coroutine dumps to work with coredumps

[Kevin Wolf]

Am 11.12.2024 um 22:39 hat Peter Xu geschrieben:
> On Wed, Dec 11, 2024 at 05:25:10PM -0300, Fabiano Rosas wrote:
> > Peter Xu <peterx@redhat.com> writes:
> > 
> > > Coroutines are used in many cases in block layers. It's also used in live
> > > migration when on destination side, and it'll be handy to diagnose crashes
> > > within a coroutine when we want to also know what other coroutines are
> > > doing.
> > 
> > Not sure if you've seen this message on the list:
> > 
> > https://lore.kernel.org/r/f0ebccca-7a17-4da8-ac4a-71cf6d69abc3@mtasv.net

Let me add Niall to this thread.

> No I didn't.  I only started looking at this because I got a bug a few days
> ago that I need to look at the main coroutine where dest crashed, then
> Stefan told me this script and also told me it only works with live session.
> Ideally I'll need coredump debug-ability, then I figured it isn't too hard.

I agree this is useful at least in the current state of things, and
possibly in the future for QEMUs built without support for new gdb
magic.

> I saw that it didn't yet land gdb, and it's much more involved even if it
> could be more generic.  Not sure how the block developers think, personally
> I prefer this much smaller change because it works on old systems, where I
> can easily install gdb with package managers.

I don't think this is a question of preferring one or the other! Your
simple change works today with existing QEMU builds and has fewer
requirements, but it's also more limited in functionality.

The gdb changes look really nice at the first sight, debugging
coroutines as if they were threads is exactly what you want in most
cases. So I'd love to see this. Of course, someone would have to first
understand what needs to be done in detail and then implement the QEMU
side of it once the gdb side is merged.

Kevin

Re: [PATCH 0/3] scripts/qemu-gdb: Make coroutine dumps to work with coredumps

[Peter Xu]

On Thu, Dec 12, 2024 at 11:28:52AM +0100, Kevin Wolf wrote:
> Am 11.12.2024 um 22:39 hat Peter Xu geschrieben:
> > On Wed, Dec 11, 2024 at 05:25:10PM -0300, Fabiano Rosas wrote:
> > > Peter Xu <peterx@redhat.com> writes:
> > > 
> > > > Coroutines are used in many cases in block layers. It's also used in live
> > > > migration when on destination side, and it'll be handy to diagnose crashes
> > > > within a coroutine when we want to also know what other coroutines are
> > > > doing.
> > > 
> > > Not sure if you've seen this message on the list:
> > > 
> > > https://lore.kernel.org/r/f0ebccca-7a17-4da8-ac4a-71cf6d69abc3@mtasv.net
> 
> Let me add Niall to this thread.
> 
> > No I didn't.  I only started looking at this because I got a bug a few days
> > ago that I need to look at the main coroutine where dest crashed, then
> > Stefan told me this script and also told me it only works with live session.
> > Ideally I'll need coredump debug-ability, then I figured it isn't too hard.
> 
> I agree this is useful at least in the current state of things, and
> possibly in the future for QEMUs built without support for new gdb
> magic.
> 
> > I saw that it didn't yet land gdb, and it's much more involved even if it
> > could be more generic.  Not sure how the block developers think, personally
> > I prefer this much smaller change because it works on old systems, where I
> > can easily install gdb with package managers.
> 
> I don't think this is a question of preferring one or the other! Your
> simple change works today with existing QEMU builds and has fewer
> requirements, but it's also more limited in functionality.

Ah, definitely..

I had no intention to mean that we should only merge one, but more of pure
thinking what I would like to use in the near future, even if the solution
landed at some point in both gdb and qemu.

I'll still need to look at the old systems that miss the generic solution,
so I'd still prefer using the scripts (which currently satisfies most of my
needs..).

> 
> The gdb changes look really nice at the first sight, debugging
> coroutines as if they were threads is exactly what you want in most
> cases. So I'd love to see this. Of course, someone would have to first
> understand what needs to be done in detail and then implement the QEMU
> side of it once the gdb side is merged.

Yes, I wish that could happen some day, that by default gdb on top of QEMU
can dump coroutines.

Thanks,

-- 
Peter Xu