Debugging xm dump-core file

Debugging xm dump-core file

[Kieran Mansley]

I'm following the instructions in xen-
unstable.hg/tools/debugger/gdb/README on how to use gdb to debug a
running guest or core file.  The former is fine, and I can connect to
the server, halt it, get a backtrace etc.  I'm having problems using gdb
and gdbserver-xen with a core file generated by xm dump-core however.

I do the following:

xm dump-core 1

gdbserver-xen 127.0.0.1:9999 --file /var/xen/dump/<corefile>

gdb /home/kjm/xen-unstable.hg/build-linux-2.6.18-xenU_x86_32/vmlinux

(gdb) directory /home/kjm/xen-unstable.hg/build-linux-2.6.18-xenU_x86_32/
Source directories searched: /home/kjm/xen-unstable.hg/build-linux-2.6.18-xenU_x86_32:$cdir:$cwd
(gdb) target remote 127.0.0.1:9999
Remote debugging using 127.0.0.1:9999
[New Thread 0]
[Switching to Thread 0]
0xc02def0a in _spin_lock_irqsave (lock=0xcd0f6bd4)
    at include2/asm/mach-xen/asm/spinlock.h:73
73              asm(__raw_spin_lock_string_flags : "+m" (lock->slock) : "r" (flags) : "memory");
warning: shared library handler failed to enable breakpoint
(gdb) bt
#0  0xc02def0a in _spin_lock_irqsave (lock=0xcd0f6bd4)
    at include2/asm/mach-xen/asm/spinlock.h:73
Cannot access memory at address 0xc0361b40


The last error (Cannot access memory at...) is the problem.  It means
that I can't get a back trace, and although I can see that the problem
I'm trying to debug is due to a spin lock, I'd rather worked that out
for myself and was hoping that gdb would be able to give me the
backtrace that would throw some light on my spin lock bug.

Anyone have any idea what I might have done wrong?  Getting a back trace
is such a basic operation that I'm sure it should work.

Thanks 

Kieran

Re: Debugging xm dump-core file

[Akio Takebe]

Hi, Kieran

You can use crash-utils for debugging the core.
http://people.redhat.com/anderson/

# crash <vmlinux.debug> <corefile>

corefile is core dumped by "xm dump-core".
vmlinux.debug is vmlinux including debuginfo.
We can make the vmlinux.debug by compiling with CONFIG_DEBUG_INFO.

Best Regards,

Akio Takebe

Re: Debugging xm dump-core file

[Kieran Mansley]

On Fri, 2007-10-26 at 09:38 +0900, Akio Takebe wrote:
> Hi, Kieran
> 
> You can use crash-utils for debugging the core.
> http://people.redhat.com/anderson/
> 
> # crash <vmlinux.debug> <corefile>
> 
> corefile is core dumped by "xm dump-core".
> vmlinux.debug is vmlinux including debuginfo.
> We can make the vmlinux.debug by compiling with CONFIG_DEBUG_INFO.

I did try that but get the following error from crash:

crash: /var/xen/dump/2007-1025-1628.22-Wolf02Guest1.1.core: not a
supported file format

However, file seems to think it is the following:

file /var/xen/dump/2007-1025-1628.22-Wolf02Guest1.1.core
  /var/xen/dump/2007-1025-1628.22-Wolf02Guest1.1.core: 
  ELF 64-bit LSB core file Intel 80386, version 1

And the fact that gdb and gdb-server can get some information out of it
suggests that it's not a completely invalid dump.

Thanks

Kieran

Re: Debugging xm dump-core file

[Akio Takebe]

Hi,

>I did try that but get the following error from crash:
>
>crash: /var/xen/dump/2007-1025-1628.22-Wolf02Guest1.1.core: not a
>supported file format
What version of crash did you use?
If the crash is not the latest version, please use the latest one.

Best Regards,

Akio Takebe

Re: Debugging xm dump-core file

[Kieran Mansley]

On Fri, 2007-10-26 at 16:46 +0900, Akio Takebe wrote:
> Hi,
> 
> >I did try that but get the following error from crash:
> >
> >crash: /var/xen/dump/2007-1025-1628.22-Wolf02Guest1.1.core: not a
> >supported file format
> What version of crash did you use?
> If the crash is not the latest version, please use the latest one.

That helps enormously.  I was using the standard RHEL5 version
(4.0-3.14) but 4.0-4.7 seems to understand the format fine and gives the
back trace I was after.

Many thanks

Kieran