[Qemu-devel] [PATCH] scripts/qemugdb: support coroutine backtrace in coredumps

[Qemu-devel] [PATCH] scripts/qemugdb: support coroutine backtrace in coredumps

[Stefan Hajnoczi]

Use the 'select-frame' GDB command to switch stacks instead of manually
setting the debugged thread's registers (this only works when debugging
a live process, not in a coredump).

Cc: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
Vladimir: Does this work for you?

 scripts/qemugdb/coroutine.py | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/scripts/qemugdb/coroutine.py b/scripts/qemugdb/coroutine.py
index ab699794ab..ed96434aee 100644
--- a/scripts/qemugdb/coroutine.py
+++ b/scripts/qemugdb/coroutine.py
@@ -77,13 +77,11 @@ def bt_jmpbuf(jmpbuf):
     for i in regs:
         old[i] = gdb.parse_and_eval('(uint64_t)$%s' % i)
 
-    for i in regs:
-        gdb.execute('set $%s = %s' % (i, regs[i]))
+    gdb.execute('select-frame %s %s' % (regs['rsp'], regs['rip']))
 
     gdb.execute('bt')
 
-    for i in regs:
-        gdb.execute('set $%s = %s' % (i, old[i]))
+    gdb.execute('select-frame %s %s' % (old['rsp'], old['rip']))
 
 def coroutine_to_jmpbuf(co):
     coroutine_pointer = co.cast(gdb.lookup_type('CoroutineUContext').pointer())
-- 
2.14.3

Re: [Qemu-devel] [PATCH] scripts/qemugdb: support coroutine backtrace in coredumps

[Vladimir Sementsov-Ogievskiy]

04.04.2018 13:34, Stefan Hajnoczi wrote:
> Use the 'select-frame' GDB command to switch stacks instead of manually
> setting the debugged thread's registers (this only works when debugging
> a live process, not in a coredump).
>
> Cc: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> Vladimir: Does this work for you?
>
>   scripts/qemugdb/coroutine.py | 6 ++----
>   1 file changed, 2 insertions(+), 4 deletions(-)
>
> diff --git a/scripts/qemugdb/coroutine.py b/scripts/qemugdb/coroutine.py
> index ab699794ab..ed96434aee 100644
> --- a/scripts/qemugdb/coroutine.py
> +++ b/scripts/qemugdb/coroutine.py
> @@ -77,13 +77,11 @@ def bt_jmpbuf(jmpbuf):
>       for i in regs:
>           old[i] = gdb.parse_and_eval('(uint64_t)$%s' % i)
>   
> -    for i in regs:
> -        gdb.execute('set $%s = %s' % (i, regs[i]))
> +    gdb.execute('select-frame %s %s' % (regs['rsp'], regs['rip']))
>   
>       gdb.execute('bt')
>   
> -    for i in regs:
> -        gdb.execute('set $%s = %s' % (i, old[i]))
> +    gdb.execute('select-frame %s %s' % (old['rsp'], old['rip']))
>   
>   def coroutine_to_jmpbuf(co):
>       coroutine_pointer = co.cast(gdb.lookup_type('CoroutineUContext').pointer())

strange, but it doesn't work. it prints the same backtrace, as if I just 
call bt.
(I applied it onto "[PATCH 2/4] scripts/qemugdb: improve "qemu 
coroutine" command")

also, I can just call select-frame with zeros or any garbage in gdb, 
with same effect:
(gdb) select-frame 0 0
(gdb) bt

and get same backtrace.

so, bt command not related to selected frame. also, up and down commands 
don't help too, they go to frames in current bt, instead of moving 
relatively to selected frame.

-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH] scripts/qemugdb: support coroutine backtrace in coredumps

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2474 bytes --]

On Mon, Apr 09, 2018 at 04:01:24PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> 04.04.2018 13:34, Stefan Hajnoczi wrote:
> > Use the 'select-frame' GDB command to switch stacks instead of manually
> > setting the debugged thread's registers (this only works when debugging
> > a live process, not in a coredump).
> > 
> > Cc: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> > ---
> > Vladimir: Does this work for you?
> > 
> >   scripts/qemugdb/coroutine.py | 6 ++----
> >   1 file changed, 2 insertions(+), 4 deletions(-)
> > 
> > diff --git a/scripts/qemugdb/coroutine.py b/scripts/qemugdb/coroutine.py
> > index ab699794ab..ed96434aee 100644
> > --- a/scripts/qemugdb/coroutine.py
> > +++ b/scripts/qemugdb/coroutine.py
> > @@ -77,13 +77,11 @@ def bt_jmpbuf(jmpbuf):
> >       for i in regs:
> >           old[i] = gdb.parse_and_eval('(uint64_t)$%s' % i)
> > -    for i in regs:
> > -        gdb.execute('set $%s = %s' % (i, regs[i]))
> > +    gdb.execute('select-frame %s %s' % (regs['rsp'], regs['rip']))
> >       gdb.execute('bt')
> > -    for i in regs:
> > -        gdb.execute('set $%s = %s' % (i, old[i]))
> > +    gdb.execute('select-frame %s %s' % (old['rsp'], old['rip']))
> >   def coroutine_to_jmpbuf(co):
> >       coroutine_pointer = co.cast(gdb.lookup_type('CoroutineUContext').pointer())
> 
> strange, but it doesn't work. it prints the same backtrace, as if I just
> call bt.
> (I applied it onto "[PATCH 2/4] scripts/qemugdb: improve "qemu coroutine"
> command")
> 
> also, I can just call select-frame with zeros or any garbage in gdb, with
> same effect:
> (gdb) select-frame 0 0
> (gdb) bt
> 
> and get same backtrace.
> 
> so, bt command not related to selected frame. also, up and down commands
> don't help too, they go to frames in current bt, instead of moving
> relatively to selected frame.

I wonder what the point of select-frame is then...

I have CCed the GDB mailing list.  Maybe someone can help us.  Context:

QEMU implements coroutines using jmpbuf.  We'd like to print coroutine
call stacks in GDB and have a script that works when a process is being
debugged (it sets the registers).

Now we'd like to extend the script to work on core dumps where it's not
possible to set registers (since there is no process being debugged).

Is there a way to backtrace an arbitrary call stack in a core dump?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [PATCH] scripts/qemugdb: support coroutine backtrace in coredumps

[Simon Marchi]

On 2018-04-09 10:08 PM, Stefan Hajnoczi wrote:
> I wonder what the point of select-frame is then...
> 
> I have CCed the GDB mailing list.  Maybe someone can help us.  Context:
> 
> QEMU implements coroutines using jmpbuf.  We'd like to print coroutine
> call stacks in GDB and have a script that works when a process is being
> debugged (it sets the registers).
> 
> Now we'd like to extend the script to work on core dumps where it's not
> possible to set registers (since there is no process being debugged).
> 
> Is there a way to backtrace an arbitrary call stack in a core dump?

Not that I know of.  The "frame <stack-addr> <pc-addr>" form of the frame
command sounds like it should be usable to achieve that, but it doesn't
seem to work in that way.  I really wonder if it's working as it was
intended initially.  I guess using that form of the frame command should
override/mask the real current values of $sp and $pc?

Simon

Re: [Qemu-devel] [PATCH] scripts/qemugdb: support coroutine backtrace in coredumps

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1407 bytes --]

On Sun, Apr 22, 2018 at 09:37:52PM -0400, Simon Marchi wrote:
> On 2018-04-09 10:08 PM, Stefan Hajnoczi wrote:
> > I wonder what the point of select-frame is then...
> > 
> > I have CCed the GDB mailing list.  Maybe someone can help us.  Context:
> > 
> > QEMU implements coroutines using jmpbuf.  We'd like to print coroutine
> > call stacks in GDB and have a script that works when a process is being
> > debugged (it sets the registers).
> > 
> > Now we'd like to extend the script to work on core dumps where it's not
> > possible to set registers (since there is no process being debugged).
> > 
> > Is there a way to backtrace an arbitrary call stack in a core dump?
> 
> Not that I know of.  The "frame <stack-addr> <pc-addr>" form of the frame
> command sounds like it should be usable to achieve that, but it doesn't
> seem to work in that way.  I really wonder if it's working as it was
> intended initially.  I guess using that form of the frame command should
> override/mask the real current values of $sp and $pc?

Yes, that is what I was expecting.

Vladimir has a script to copy the coredump file and manipulate the
registers on disk, but it would be nicer to do stack-switching inside
the GDB session without needing to create another coredump file.

Vladimir: Would you like to write a GDB patch?  If GDB is fixed then
workaround won't be necessary.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [PATCH] scripts/qemugdb: support coroutine backtrace in coredumps

[Vladimir Sementsov-Ogievskiy]

23.04.2018 12:33, Stefan Hajnoczi wrote:
> On Sun, Apr 22, 2018 at 09:37:52PM -0400, Simon Marchi wrote:
>> On 2018-04-09 10:08 PM, Stefan Hajnoczi wrote:
>>> I wonder what the point of select-frame is then...
>>>
>>> I have CCed the GDB mailing list.  Maybe someone can help us.  Context:
>>>
>>> QEMU implements coroutines using jmpbuf.  We'd like to print coroutine
>>> call stacks in GDB and have a script that works when a process is being
>>> debugged (it sets the registers).
>>>
>>> Now we'd like to extend the script to work on core dumps where it's not
>>> possible to set registers (since there is no process being debugged).
>>>
>>> Is there a way to backtrace an arbitrary call stack in a core dump?
>> Not that I know of.  The "frame <stack-addr> <pc-addr>" form of the frame
>> command sounds like it should be usable to achieve that, but it doesn't
>> seem to work in that way.  I really wonder if it's working as it was
>> intended initially.  I guess using that form of the frame command should
>> override/mask the real current values of $sp and $pc?
> Yes, that is what I was expecting.
>
> Vladimir has a script to copy the coredump file and manipulate the
> registers on disk, but it would be nicer to do stack-switching inside
> the GDB session without needing to create another coredump file.
>
> Vladimir: Would you like to write a GDB patch?  If GDB is fixed then
> workaround won't be necessary.

Hmm, unlikely. At least, not in the nearest future.

>
> Stefan


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH] scripts/qemugdb: support coroutine backtrace in coredumps

[Pedro Alves]

On 04/23/2018 02:37 AM, Simon Marchi wrote:
> On 2018-04-09 10:08 PM, Stefan Hajnoczi wrote:
>> I wonder what the point of select-frame is then...
>>
>> I have CCed the GDB mailing list.  Maybe someone can help us.  Context:
>>
>> QEMU implements coroutines using jmpbuf.  We'd like to print coroutine
>> call stacks in GDB and have a script that works when a process is being
>> debugged (it sets the registers).
>>
>> Now we'd like to extend the script to work on core dumps where it's not
>> possible to set registers (since there is no process being debugged).
>>
>> Is there a way to backtrace an arbitrary call stack in a core dump?
> 
> Not that I know of.  The "frame <stack-addr> <pc-addr>" form of the frame
> command sounds like it should be usable to achieve that, but it doesn't
> seem to work in that way.  I really wonder if it's working as it was
> intended initially.  I guess using that form of the frame command should
> override/mask the real current values of $sp and $pc?

Yeah, "frame <args>" has a lot of problems.

This series was working toward sorting out the "frame" command:

  https://sourceware.org/ml/gdb-patches/2015-09/msg00248.html

Follow the urls there for more background.

To me, the important questions to answer are here:
 https://sourceware.org/ml/gdb-patches/2015-09/msg00658.html

Unfortunately, I don't think the series moved past that point.

Thanks,
Pedro Alves

Re: [Qemu-devel] [PATCH] scripts/qemugdb: support coroutine backtrace in coredumps

[Vladimir Sementsov-Ogievskiy]

23.04.2018 16:28, Pedro Alves wrote:
> On 04/23/2018 02:37 AM, Simon Marchi wrote:
>> On 2018-04-09 10:08 PM, Stefan Hajnoczi wrote:
>>> I wonder what the point of select-frame is then...
>>>
>>> I have CCed the GDB mailing list.  Maybe someone can help us.  Context:
>>>
>>> QEMU implements coroutines using jmpbuf.  We'd like to print coroutine
>>> call stacks in GDB and have a script that works when a process is being
>>> debugged (it sets the registers).
>>>
>>> Now we'd like to extend the script to work on core dumps where it's not
>>> possible to set registers (since there is no process being debugged).
>>>
>>> Is there a way to backtrace an arbitrary call stack in a core dump?
>>
>> Not that I know of.  The "frame <stack-addr> <pc-addr>" form of the frame
>> command sounds like it should be usable to achieve that, but it doesn't
>> seem to work in that way.  I really wonder if it's working as it was
>> intended initially.  I guess using that form of the frame command should
>> override/mask the real current values of $sp and $pc?
> 
> Yeah, "frame <args>" has a lot of problems.
> 
> This series was working toward sorting out the "frame" command:
> 
>    https://sourceware.org/ml/gdb-patches/2015-09/msg00248.html
> 
> Follow the urls there for more background.
> 
> To me, the important questions to answer are here:
>   https://sourceware.org/ml/gdb-patches/2015-09/msg00658.html
> 
> Unfortunately, I don't think the series moved past that point.
> 
> Thanks,
> Pedro Alves
> 


Hi Pedro!

Hmm, returned to this topic. I've spent this day digging in gdb code, and found it much
more difficult than qemu)..

I've failed to find something like

create_frame_with_registers, or create_thread_with_registers.. Looks like registers comes
from some register caches, backed by different sources of registers or something like this.

So, I'd like to ask several questions:

1. Any news on the topic since April?

2. Can you propose a simple (maybe hacky) way (with or without patching gdb) to achieve the behavior like

set $rsp = ...
set $rbp = ...
set $rip = ...

bt #prints bt, starting from the frame corresponding to register values set
fr 5 #goes to frame in this bt, and allow to examine local variables

for debugging core-dumps?

***

May be, we can allow set registers while debugging core-dump? Why we can't set them? We have
same register caches, and anyway we can move between threads and frames and registers are changed...

May be, we can somehow add separate thread with given registers from user, like they are created from
coredump file..

Anything else?

***

For me, still, the simplest way is to add additional note segments to coredump file, to specify needed frames..
But it is not very comfortable, to recreate and reopen core file, when found new coroutine.

-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH] scripts/qemugdb: support coroutine backtrace in coredumps

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2204 bytes --]

On Thu, Dec 27, 2018 at 05:36:04PM +0000, Vladimir Sementsov-Ogievskiy wrote:
> 23.04.2018 16:28, Pedro Alves wrote:
> > On 04/23/2018 02:37 AM, Simon Marchi wrote:
> >> On 2018-04-09 10:08 PM, Stefan Hajnoczi wrote:
> >>> I wonder what the point of select-frame is then...
> >>>
> >>> I have CCed the GDB mailing list.  Maybe someone can help us.  Context:
> >>>
> >>> QEMU implements coroutines using jmpbuf.  We'd like to print coroutine
> >>> call stacks in GDB and have a script that works when a process is being
> >>> debugged (it sets the registers).
> >>>
> >>> Now we'd like to extend the script to work on core dumps where it's not
> >>> possible to set registers (since there is no process being debugged).
> >>>
> >>> Is there a way to backtrace an arbitrary call stack in a core dump?
> >>
> >> Not that I know of.  The "frame <stack-addr> <pc-addr>" form of the frame
> >> command sounds like it should be usable to achieve that, but it doesn't
> >> seem to work in that way.  I really wonder if it's working as it was
> >> intended initially.  I guess using that form of the frame command should
> >> override/mask the real current values of $sp and $pc?
> > 
> > Yeah, "frame <args>" has a lot of problems.
> > 
> > This series was working toward sorting out the "frame" command:
> > 
> >    https://sourceware.org/ml/gdb-patches/2015-09/msg00248.html
> > 
> > Follow the urls there for more background.
> > 
> > To me, the important questions to answer are here:
> >   https://sourceware.org/ml/gdb-patches/2015-09/msg00658.html
> > 
> > Unfortunately, I don't think the series moved past that point.
> > 
> > Thanks,
> > Pedro Alves
> > 
> 
> 
> Hi Pedro!
> 
> Hmm, returned to this topic. I've spent this day digging in gdb code, and found it much
> more difficult than qemu)..
> 
> I've failed to find something like
> 
> create_frame_with_registers, or create_thread_with_registers.. Looks like registers comes
> from some register caches, backed by different sources of registers or something like this.
> 
> So, I'd like to ask several questions:
> 
> 1. Any news on the topic since April?

Not on my side, sorry.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]