[Qemu-devel] [PATCH 0/4] qemugdb: coroutine backtrace for coredump

[Qemu-devel] [PATCH 0/4] qemugdb: coroutine backtrace for coredump

[Vladimir Sementsov-Ogievskiy]

Hi all. Here are some qemugdb enhancements around qemu coroutine command.
The main feature is a backtrace for coroutine, when debugging with a
coredump file.

The problem is that we can't get coroutine backtrace through obvious way
 - set regs
 - bt
 - restore regs
when debugging a coredump.
So, let's go hard way: clone current coredump file, patch regs
in it and execute a subprocess gdb to get backtrace from this
patched coredump.

Vladimir Sementsov-Ogievskiy (4):
  scripts/qemugdb: get pthread_self from "info threads" command
  scripts/qemugdb: improve "qemu coroutine" command
  scripts/qemugdb: add coredump.py for coredump patching
  scripts/qemugdb: backtraces for coroutines in coredump

 scripts/qemugdb/coredump.py  | 51 ++++++++++++++++++++++++++++++++++++++++++
 scripts/qemugdb/coroutine.py | 53 +++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 99 insertions(+), 5 deletions(-)
 create mode 100644 scripts/qemugdb/coredump.py

-- 
2.11.1

[Qemu-devel] [PATCH 1/4] scripts/qemugdb: get pthread_self from "info threads" command

[Vladimir Sementsov-Ogievskiy]

When debugging a coredump, pthread_self can't be obtained from
function arch_prctl. Moreover if qemu crashed in coroutine, we
can't find 'start_thread' in current stack-trace. So, add a method,
actually proposed in 1138f24645e9e, which should work for gdb
version >= 7.3.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 scripts/qemugdb/coroutine.py | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/scripts/qemugdb/coroutine.py b/scripts/qemugdb/coroutine.py
index ab699794ab..ffaa45c464 100644
--- a/scripts/qemugdb/coroutine.py
+++ b/scripts/qemugdb/coroutine.py
@@ -14,6 +14,7 @@
 # GNU GPL, version 2 or (at your option) any later version.
 
 import gdb
+import re
 
 VOID_PTR = gdb.lookup_type('void').pointer()
 
@@ -28,7 +29,17 @@ def get_fs_base():
     return fs_base
 
 def pthread_self():
-    '''Fetch pthread_self() from the glibc start_thread function.'''
+    # Try read pthread_self from gdb command 'info threads'.
+    # Will fail for old gdb.
+    try:
+        threads = gdb.execute('info threads', False, True)
+        m = re.search('^\* 1    Thread (0x[0-9a-f]+)', threads, re.MULTILINE)
+        return int(m.group(1), 16)
+    except TypeError:
+        # gdb doesn't support third parameter for execute
+        pass
+
+    # Try fetch pthread_self() from the glibc start_thread function.
     f = gdb.newest_frame()
     while f.name() != 'start_thread':
         f = f.older()
-- 
2.11.1

Re: [Qemu-devel] [PATCH 1/4] scripts/qemugdb: get pthread_self from "info threads" command

[Paolo Bonzini]

On 28/03/2018 19:32, Vladimir Sementsov-Ogievskiy wrote:
> When debugging a coredump, pthread_self can't be obtained from
> function arch_prctl. Moreover if qemu crashed in coroutine, we
> can't find 'start_thread' in current stack-trace. So, add a method,
> actually proposed in 1138f24645e9e, which should work for gdb
> version >= 7.3.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

New-enough gdb can also use gdb.selected_thread()
> ---
>  scripts/qemugdb/coroutine.py | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/qemugdb/coroutine.py b/scripts/qemugdb/coroutine.py
> index ab699794ab..ffaa45c464 100644
> --- a/scripts/qemugdb/coroutine.py
> +++ b/scripts/qemugdb/coroutine.py
> @@ -14,6 +14,7 @@
>  # GNU GPL, version 2 or (at your option) any later version.
>  
>  import gdb
> +import re
>  
>  VOID_PTR = gdb.lookup_type('void').pointer()
>  
> @@ -28,7 +29,17 @@ def get_fs_base():
>      return fs_base
>  
>  def pthread_self():
> -    '''Fetch pthread_self() from the glibc start_thread function.'''
> +    # Try read pthread_self from gdb command 'info threads'.
> +    # Will fail for old gdb.
> +    try:
> +        threads = gdb.execute('info threads', False, True)
> +        m = re.search('^\* 1    Thread (0x[0-9a-f]+)', threads, re.MULTILINE)

I don't think hard-coding "1" works here, and the spacing of the table
might differ as well.  However, looking for the asterisk seems safe from
a quick look at gdb source, and "Thread" looks like it isn't localized.

Paolo

> +        return int(m.group(1), 16)
> +    except TypeError:
> +        # gdb doesn't support third parameter for execute
> +        pass
> +
> +    # Try fetch pthread_self() from the glibc start_thread function.
>      f = gdb.newest_frame()
>      while f.name() != 'start_thread':
>          f = f.older()
> 

Re: [Qemu-devel] [PATCH 1/4] scripts/qemugdb: get pthread_self from "info threads" command

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 609 bytes --]

On Wed, Mar 28, 2018 at 08:32:35PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> When debugging a coredump, pthread_self can't be obtained from
> function arch_prctl. Moreover if qemu crashed in coroutine, we
> can't find 'start_thread' in current stack-trace. So, add a method,
> actually proposed in 1138f24645e9e, which should work for gdb
> version >= 7.3.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
>  scripts/qemugdb/coroutine.py | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[Qemu-devel] [PATCH 2/4] scripts/qemugdb: improve "qemu coroutine" command

[Vladimir Sementsov-Ogievskiy]

 - print regs
 - catch exception for coredump debugging

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 scripts/qemugdb/coroutine.py | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)

diff --git a/scripts/qemugdb/coroutine.py b/scripts/qemugdb/coroutine.py
index ffaa45c464..7070a592f3 100644
--- a/scripts/qemugdb/coroutine.py
+++ b/scripts/qemugdb/coroutine.py
@@ -80,9 +80,8 @@ def get_jmpbuf_regs(jmpbuf):
         'r15': jmpbuf[JB_R15],
         'rip': glibc_ptr_demangle(jmpbuf[JB_PC], pointer_guard) }
 
-def bt_jmpbuf(jmpbuf):
-    '''Backtrace a jmpbuf'''
-    regs = get_jmpbuf_regs(jmpbuf)
+def bt_regs(regs):
+    '''Backtrace with specified regs'''
     old = dict()
 
     for i in regs:
@@ -113,7 +112,18 @@ class CoroutineCommand(gdb.Command):
             gdb.write('usage: qemu coroutine <coroutine-pointer>\n')
             return
 
-        bt_jmpbuf(coroutine_to_jmpbuf(gdb.parse_and_eval(argv[0])))
+        jmpbuf = coroutine_to_jmpbuf(gdb.parse_and_eval(argv[0]))
+        regs = get_jmpbuf_regs(jmpbuf)
+        for k, v in regs.iteritems():
+            gdb.write('%s: 0x%x\n' %(k,v))
+
+        gdb.write('\n')
+
+        try:
+            bt_regs(regs)
+        except gdb.error:
+            print "Coroutine backtrace can't be obtained without " \
+                  "a process to debug."
 
 class CoroutineSPFunction(gdb.Function):
     def __init__(self):
-- 
2.11.1

Re: [Qemu-devel] [PATCH 2/4] scripts/qemugdb: improve "qemu coroutine" command

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 390 bytes --]

On Wed, Mar 28, 2018 at 08:32:36PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>  - print regs
>  - catch exception for coredump debugging
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
>  scripts/qemugdb/coroutine.py | 18 ++++++++++++++----
>  1 file changed, 14 insertions(+), 4 deletions(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[Qemu-devel] [PATCH 3/4] scripts/qemugdb: add coredump.py for coredump patching

[Vladimir Sementsov-Ogievskiy]

The main function is write_regs_to_coredump, which opens coredump
file, searches for 'CORE' sign. The first one should correspond
to PRSTATUS note for first thread. Patch register values in
elf_prstatus structure, going after header with 'CORE' sign.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 scripts/qemugdb/coredump.py | 51 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)
 create mode 100644 scripts/qemugdb/coredump.py

diff --git a/scripts/qemugdb/coredump.py b/scripts/qemugdb/coredump.py
new file mode 100644
index 0000000000..8915461886
--- /dev/null
+++ b/scripts/qemugdb/coredump.py
@@ -0,0 +1,51 @@
+# Coredump patching
+#
+# Copyright (c) 2018 Virtuozzo International GmbH. All rights reserved.
+#
+# Authors:
+#  Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import struct
+import shutil
+
+def write_regs_to_coredump(fname, set_regs):
+    # asm/ptrace.h
+    pt_regs = ['r15', 'r14', 'r13', 'r12', 'rbp', 'rbx', 'r11', 'r10',
+               'r9', 'r8', 'rax', 'rcx', 'rdx', 'rsi', 'rdi', 'orig_rax',
+               'rip', 'cs', 'eflags', 'rsp', 'ss']
+
+    with open(fname, 'r+b') as f:
+        print 'patching core file "%s"' % fname
+
+        while f.read(4) != 'CORE':
+            pass
+
+        print 'found "CORE" at 0x%x' % f.tell()
+        f.seek(4, 1) # go to elf_prstatus
+        f.seek(112, 1) # offsetof(struct elf_prstatus, pr_reg)
+
+        print 'assume pt_regs at 0x%x' % f.tell()
+        for reg in pt_regs:
+            if reg in set_regs:
+                print 'write %s at 0x%x' % (reg, f.tell())
+                f.write(struct.pack('q', set_regs[reg]))
+            else:
+                f.seek(8, 1)
+
+def clone_coredump(source, target, set_regs):
+    shutil.copyfile(source, target)
+    write_regs_to_coredump(target, set_regs)
-- 
2.11.1

[Qemu-devel] [PATCH 4/4] scripts/qemugdb: backtraces for coroutines in coredump

[Vladimir Sementsov-Ogievskiy]

We can't get coroutine backtrace through obvious way
 - set regs
 - bt
 - restore regs
when debugging a coredump.
So, let's go hard way: clone current coredump file, patch regs
in it and execute a subprocess gdb to get backtrace from this
patched coredump.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 scripts/qemugdb/coroutine.py | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

diff --git a/scripts/qemugdb/coroutine.py b/scripts/qemugdb/coroutine.py
index 7070a592f3..2a05851e24 100644
--- a/scripts/qemugdb/coroutine.py
+++ b/scripts/qemugdb/coroutine.py
@@ -15,9 +15,32 @@
 
 import gdb
 import re
+import tempfile
+import subprocess
+import os
+import coredump
 
 VOID_PTR = gdb.lookup_type('void').pointer()
 
+def bt_regs_static(regs):
+    files = gdb.execute('info files', False, True).split('\n')
+    executable = re.match('^Symbols from "(.*)".$', files[0]).group(1)
+    dump = re.search("`(.*)'", files[2]).group(1)
+
+    with tempfile.NamedTemporaryFile(dir='/tmp', delete=False) as f:
+        temp = f.name
+
+    coredump.clone_coredump(dump, temp, regs)
+
+    cmd = ['gdb', '-batch', '-ex', "python print '----split----'",
+           '-ex', 'bt', executable, temp]
+    out = subprocess.check_output(cmd)
+    out = out.split('----split----')[1]
+
+    os.remove(temp)
+
+    print out
+
 def get_fs_base():
     '''Fetch %fs base value using arch_prctl(ARCH_GET_FS).  This is
        pthread_self().'''
@@ -122,8 +145,7 @@ class CoroutineCommand(gdb.Command):
         try:
             bt_regs(regs)
         except gdb.error:
-            print "Coroutine backtrace can't be obtained without " \
-                  "a process to debug."
+            bt_regs_static(regs)
 
 class CoroutineSPFunction(gdb.Function):
     def __init__(self):
-- 
2.11.1

Re: [Qemu-devel] [PATCH 4/4] scripts/qemugdb: backtraces for coroutines in coredump

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

On Wed, Mar 28, 2018 at 08:32:38PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> We can't get coroutine backtrace through obvious way
>  - set regs
>  - bt
>  - restore regs
> when debugging a coredump.
> So, let's go hard way: clone current coredump file, patch regs
> in it and execute a subprocess gdb to get backtrace from this
> patched coredump.

I have CCed you on a patch that uses 'select-frame' to temporarily
switch stacks.  This works in a coredump and does not require
cloning/patching the coredump file.

Please let me know if 'select-frame' works for you, it would be
preferrable to cloning/patching the coredump file.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]