help on debugging kernel module in Xen

help on debugging kernel module in Xen

[Lily Huang]

[-- Attachment #1.1: Type: text/plain, Size: 228 bytes --]

Hi all,

I am developing a kernel module running in both Xen0 and XenU. I want to
debug it but don't know how. Does someone has experience on this?

I know printk is an option, but not very convenient.


Thanks in advance,
lily

[-- Attachment #1.2: Type: text/html, Size: 263 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: help on debugging kernel module in Xen

[Brendan Cully]

On Friday, 12 January 2007 at 13:41, Lily Huang wrote:
> Hi all,
> 
> I am developing a kernel module running in both Xen0 and XenU. I want to
> debug it but don't know how. Does someone has experience on this?
> 
> I know printk is an option, but not very convenient.

There's a nice howto for using gdb to debug guest kernels in
tools/debugger/gdb/README.

Re: help on debugging kernel module in Xen

[Lily Huang]

[-- Attachment #1.1: Type: text/plain, Size: 1680 bytes --]

Thanks, Brendan.

I followed the README instruction and did follows:

1. start a domU
2. in Dom0, run "gdbserver-xen 127.0.0.1:9999 --attach 5", where 5 is the
domain id of the domU
3. I got the response:
/*******************************/
Attached; pid = 5
Listening on port 9999
/*******************************/
And then, I lost connection with the domU, I tried both network and console,
neither works for me.

4. I run "gdb /boot/vmlinux-syms-2.6.16.33-xenU" in dom0, hoping to
reactivate domU with debug command.

the response is as follows:

/*******************************/
GNU gdb Red Hat Linux (6.3.0.0-1.84rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db
library "/lib/libthread_db.so.1".
/*******************************/

when I typed "cont" command in the gdb, I got "The program is not being
run."

Am I doing something wrong? Any further help will be highly appreciated.

lily



On 1/12/07, Brendan Cully <brendan@cs.ubc.ca> wrote:
>
> On Friday, 12 January 2007 at 13:41, Lily Huang wrote:
> > Hi all,
> >
> > I am developing a kernel module running in both Xen0 and XenU. I want to
> > debug it but don't know how. Does someone has experience on this?
> >
> > I know printk is an option, but not very convenient.
>
> There's a nice howto for using gdb to debug guest kernels in
> tools/debugger/gdb/README.
>

[-- Attachment #1.2: Type: text/html, Size: 2216 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: help on debugging kernel module in Xen

[Lily Huang]

[-- Attachment #1.1: Type: text/plain, Size: 2237 bytes --]

Sorry. I was wrong. I forgot to run the rest of gdb command.

Now I did the following commands in gdb:

1. "directory /boot"
response:  "Source directories searched: /boot:$cdir:$cwd"

2. br filp_open
response: "Breakpoint 1 at 0xc0160246: file fs/open.c, line 885."

3.  cont

4. But no matter what file I opened in domU, the breakpoint was never
encountered.

What's wrong?

Thanks,
lily


On 1/12/07, Lily Huang <ushuanglily@gmail.com> wrote:
>
> Thanks, Brendan.
>
> I followed the README instruction and did follows:
>
> 1. start a domU
> 2. in Dom0, run "gdbserver-xen 127.0.0.1:9999 --attach 5", where 5 is the
> domain id of the domU
> 3. I got the response:
> /*******************************/
> Attached; pid = 5
> Listening on port 9999
> /*******************************/
> And then, I lost connection with the domU, I tried both network and
> console, neither works for me.
>
> 4. I run "gdb /boot/vmlinux-syms-2.6.16.33-xenU" in dom0, hoping to
> reactivate domU with debug command.
>
> the response is as follows:
>
> /*******************************/
> GNU gdb Red Hat Linux (6.3.0.0-1.84rh)
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you
> are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for
> details.
> This GDB was configured as "i386-redhat-linux-gnu"...Using host
> libthread_db library "/lib/libthread_db.so.1".
> /*******************************/
>
> when I typed "cont" command in the gdb, I got "The program is not being
> run."
>
> Am I doing something wrong? Any further help will be highly appreciated.
>
> lily
>
>
>
> On 1/12/07, Brendan Cully <brendan@cs.ubc.ca> wrote:
> >
> > On Friday, 12 January 2007 at 13:41, Lily Huang wrote:
> > > Hi all,
> > >
> > > I am developing a kernel module running in both Xen0 and XenU. I want
> > to
> > > debug it but don't know how. Does someone has experience on this?
> > >
> > > I know printk is an option, but not very convenient.
> >
> > There's a nice howto for using gdb to debug guest kernels in
> > tools/debugger/gdb/README.
> >
>
>

[-- Attachment #1.2: Type: text/html, Size: 3272 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: help on debugging kernel module in Xen

[Brendan Cully]

On Friday, 12 January 2007 at 15:25, Lily Huang wrote:
> Thanks, Brendan.
> 
> I followed the README instruction and did follows:
> 
> 1. start a domU
> 2. in Dom0, run "gdbserver-xen 127.0.0.1:9999 --attach 5", where 5 is the
> domain id of the domU
> 3. I got the response:
> /*******************************/
> Attached; pid = 5
> Listening on port 9999
> /*******************************/
> And then, I lost connection with the domU, I tried both network and console,
> neither works for me.

when the debugger attaches, it pauses the guest.

> 4. I run "gdb /boot/vmlinux-syms-2.6.16.33-xenU" in dom0, hoping to
> reactivate domU with debug command.
> 
> the response is as follows:
> 
> /*******************************/
> GNU gdb Red Hat Linux (6.3.0.0-1.84rh)
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db
> library "/lib/libthread_db.so.1".
> /*******************************/
> 
> when I typed "cont" command in the gdb, I got "The program is not being
> run."
> 
> Am I doing something wrong? Any further help will be highly appreciated.

you need to attach to the domain from within gdb:

target remote localhost:9999

Re: help on debugging kernel module in Xen

[Lily Huang]

[-- Attachment #1.1: Type: text/plain, Size: 2383 bytes --]

Many thanks for your help!

I tried the command "target remote 127.0.0.1:9999", and got the response:

/******************************************************************************/
Remote debugging using 127.0.0.1:9999
[New Thread 0]
[Switching to Thread 0]
0xc01013a7 in hypercall_page ()
warning: shared library handler failed to enable breakpoint
<------------------ this could be a problem
/******************************************************************************/

then, I did "br filp_open", and then "cont".

But no matter what file I opened in domU, the breakpoint is never
encountered. It appears that the breakpoint is not activate.

I googled but cannot find answer so far.

Did you run into similar problems?

Thanks,
lily

On 1/12/07, Brendan Cully <brendan@cs.ubc.ca> wrote:
>
> On Friday, 12 January 2007 at 15:25, Lily Huang wrote:
> > Thanks, Brendan.
> >
> > I followed the README instruction and did follows:
> >
> > 1. start a domU
> > 2. in Dom0, run "gdbserver-xen 127.0.0.1:9999 --attach 5", where 5 is
> the
> > domain id of the domU
> > 3. I got the response:
> > /*******************************/
> > Attached; pid = 5
> > Listening on port 9999
> > /*******************************/
> > And then, I lost connection with the domU, I tried both network and
> console,
> > neither works for me.
>
> when the debugger attaches, it pauses the guest.
>
> > 4. I run "gdb /boot/vmlinux-syms-2.6.16.33-xenU" in dom0, hoping to
> > reactivate domU with debug command.
> >
> > the response is as follows:
> >
> > /*******************************/
> > GNU gdb Red Hat Linux (6.3.0.0-1.84rh)
> > Copyright 2004 Free Software Foundation, Inc.
> > GDB is free software, covered by the GNU General Public License, and you
> are
> > welcome to change it and/or distribute copies of it under certain
> > conditions.
> > Type "show copying" to see the conditions.
> > There is absolutely no warranty for GDB.  Type "show warranty" for
> details.
> > This GDB was configured as "i386-redhat-linux-gnu"...Using host
> libthread_db
> > library "/lib/libthread_db.so.1".
> > /*******************************/
> >
> > when I typed "cont" command in the gdb, I got "The program is not being
> > run."
> >
> > Am I doing something wrong? Any further help will be highly appreciated.
>
> you need to attach to the domain from within gdb:
>
> target remote localhost:9999
>

[-- Attachment #1.2: Type: text/html, Size: 3108 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: help on debugging kernel module in Xen

[Lily Huang]

[-- Attachment #1.1: Type: text/plain, Size: 2818 bytes --]

Oh. I found that gdbserver printed out some error messages:

unsupported xc_ptrace request PTRACE_SETFPXREGS
unsupported xc_ptrace request PTRACE_SETFPREGS

Are they the reason why the breakpoint does not work?

lily

On 1/12/07, Lily Huang <ushuanglily@gmail.com> wrote:
>
> Many thanks for your help!
>
> I tried the command "target remote 127.0.0.1:9999", and got the response:
>
> /******************************************************************************/
>
> Remote debugging using 127.0.0.1:9999
> [New Thread 0]
> [Switching to Thread 0]
> 0xc01013a7 in hypercall_page ()
> warning: shared library handler failed to enable breakpoint
> <------------------ this could be a problem
>
> /******************************************************************************/
>
> then, I did "br filp_open", and then "cont".
>
> But no matter what file I opened in domU, the breakpoint is never
> encountered. It appears that the breakpoint is not activate.
>
> I googled but cannot find answer so far.
>
> Did you run into similar problems?
>
> Thanks,
> lily
>
> On 1/12/07, Brendan Cully <brendan@cs.ubc.ca > wrote:
> >
> > On Friday, 12 January 2007 at 15:25, Lily Huang wrote:
> > > Thanks, Brendan.
> > >
> > > I followed the README instruction and did follows:
> > >
> > > 1. start a domU
> > > 2. in Dom0, run "gdbserver-xen 127.0.0.1:9999 --attach 5", where 5 is
> > the
> > > domain id of the domU
> > > 3. I got the response:
> > > /*******************************/
> > > Attached; pid = 5
> > > Listening on port 9999
> > > /*******************************/
> > > And then, I lost connection with the domU, I tried both network and
> > console,
> > > neither works for me.
> >
> > when the debugger attaches, it pauses the guest.
> >
> > > 4. I run "gdb /boot/vmlinux-syms-2.6.16.33-xenU" in dom0, hoping to
> > > reactivate domU with debug command.
> > >
> > > the response is as follows:
> > >
> > > /*******************************/
> > > GNU gdb Red Hat Linux (6.3.0.0-1.84rh)
> > > Copyright 2004 Free Software Foundation, Inc.
> > > GDB is free software, covered by the GNU General Public License, and
> > you are
> > > welcome to change it and/or distribute copies of it under certain
> > > conditions.
> > > Type "show copying" to see the conditions.
> > > There is absolutely no warranty for GDB.  Type "show warranty" for
> > details.
> > > This GDB was configured as "i386-redhat-linux-gnu"...Using host
> > libthread_db
> > > library "/lib/libthread_db.so.1".
> > > /*******************************/
> > >
> > > when I typed "cont" command in the gdb, I got "The program is not
> > being
> > > run."
> > >
> > > Am I doing something wrong? Any further help will be highly
> > appreciated.
> >
> > you need to attach to the domain from within gdb:
> >
> > target remote localhost:9999
> >
>
>

[-- Attachment #1.2: Type: text/html, Size: 4079 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: help on debugging kernel module in Xen

[Brendan Cully]

On Friday, 12 January 2007 at 16:10, Lily Huang wrote:
> Oh. I found that gdbserver printed out some error messages:
> 
> unsupported xc_ptrace request PTRACE_SETFPXREGS
> unsupported xc_ptrace request PTRACE_SETFPREGS
> 
> Are they the reason why the breakpoint does not work?

no, those are fairly harmless.

Re: help on debugging kernel module in Xen

[Lily Huang]

[-- Attachment #1.1: Type: text/plain, Size: 1685 bytes --]

Hi Brendan,

You are right. After I lift the breakpoint a little bit (to sys_open), the
breakpoint works.

But now the domU kept complaining after I did "target remote 127.0.0.1:9999"
and then "br sys_open":

/************************************************?
BUG: soft lockup detected on CPU#0!

Pid: 0, comm:              swapper
EIP: 0061:[<c01013a7>] CPU: 0
EIP is at 0xc01013a7
 EFLAGS: 00000346    Not tainted  (2.6.16.33-xenU #12)
EAX: 00000000 EBX: 00000001 ECX: 00000000 EDX: 00000000
ESI: 00000001 EDI: c0354000 EBP: c0355f84 DS: 007b ES: 007b
CR0: 8005003b CR2: b7f77000 CR3: 1d633000 CR4: 00000640
 [<c0105bb3>] show_trace+0x13/0x20
 [<c0103390>] show_regs+0x190/0x1e0
 [<c013cf48>] softlockup_tick+0x88/0xa0
 [<c0126d45>] do_timer+0x215/0x440
 [<c010858c>] timer_interrupt+0x1bc/0x6a0
 [<c013d09f>] handle_IRQ_event+0x3f/0xd0
 [<c013d1be>] __do_IRQ+0x8e/0xf0
 [<c0106efd>] do_IRQ+0x1d/0x30
 [<c025d2d1>] evtchn_do_upcall+0xa1/0xe0
 [<c0105309>] hypervisor_callback+0x3d/0x48
 [<c01039df>] xen_idle+0x2f/0x60
 [<c0103a82>] cpu_idle+0x72/0xc0
 [<c0102035>] rest_init+0x35/0x40
 [<c035656a>] start_kernel+0x2ea/0x3a0
 [<c010006f>] 0xc010006f

/********************************************************/

What's wrong? Looks like I am really doomed to run into so many weird
problems. :(

lily

On 1/12/07, Brendan Cully <brendan@cs.ubc.ca> wrote:
>
> On Friday, 12 January 2007 at 16:10, Lily Huang wrote:
> > Oh. I found that gdbserver printed out some error messages:
> >
> > unsupported xc_ptrace request PTRACE_SETFPXREGS
> > unsupported xc_ptrace request PTRACE_SETFPREGS
> >
> > Are they the reason why the breakpoint does not work?
>
> no, those are fairly harmless.
>

[-- Attachment #1.2: Type: text/html, Size: 2421 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: help on debugging kernel module in Xen

[Brendan Cully]

On Friday, 12 January 2007 at 16:32, Lily Huang wrote:
> Hi Brendan,
> 
> You are right. After I lift the breakpoint a little bit (to sys_open), the
> breakpoint works.
> 
> But now the domU kept complaining after I did "target remote 127.0.0.1:9999"
> and then "br sys_open":
> 
> /************************************************?
> BUG: soft lockup detected on CPU#0!
> 
> Pid: 0, comm:              swapper
> EIP: 0061:[<c01013a7>] CPU: 0
> EIP is at 0xc01013a7
> EFLAGS: 00000346    Not tainted  (2.6.16.33-xenU #12)
> EAX: 00000000 EBX: 00000001 ECX: 00000000 EDX: 00000000
> ESI: 00000001 EDI: c0354000 EBP: c0355f84 DS: 007b ES: 007b
> CR0: 8005003b CR2: b7f77000 CR3: 1d633000 CR4: 00000640
> [<c0105bb3>] show_trace+0x13/0x20
> [<c0103390>] show_regs+0x190/0x1e0
> [<c013cf48>] softlockup_tick+0x88/0xa0
> [<c0126d45>] do_timer+0x215/0x440
> [<c010858c>] timer_interrupt+0x1bc/0x6a0
> [<c013d09f>] handle_IRQ_event+0x3f/0xd0
> [<c013d1be>] __do_IRQ+0x8e/0xf0
> [<c0106efd>] do_IRQ+0x1d/0x30
> [<c025d2d1>] evtchn_do_upcall+0xa1/0xe0
> [<c0105309>] hypervisor_callback+0x3d/0x48
> [<c01039df>] xen_idle+0x2f/0x60
> [<c0103a82>] cpu_idle+0x72/0xc0
> [<c0102035>] rest_init+0x35/0x40
> [<c035656a>] start_kernel+0x2ea/0x3a0
> [<c010006f>] 0xc010006f
> 
> /********************************************************/
> 
> What's wrong? Looks like I am really doomed to run into so many weird
> problems. :(

soft lockup gets triggered when the kernel hasn't gotten a timer
interrupt in a while. This tends to happen frequently when the kernel
is being debugged. I think there's a kernel config option to turn this
off, or you can just live with the messages.

Re: help on debugging kernel module in Xen

[Lily Huang]

[-- Attachment #1.1: Type: text/plain, Size: 4094 bytes --]

That's a great point. I just disabled the "Detect Soft Lockup" option in the
"kernel hacking" group. Now the soft lockup disappeared and I can debug
kernel now.

Next, I tried to debug kernel module in domU. New problem occurs.

After I set "br xxx(function in a loaded kernel module" and then "cont", I
worked in the domu to activate the breakpoint.

Unfortunately, I found domU immediately print error messages:

/**********************************************************/
Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
normal kernel: int3: 0000 [#1]
<----------------------------------------------------- seems to be a problem

Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
normal kernel: SMP

Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
normal kernel: CPU:    0

Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
normal kernel: EIP is at nfs3_proc_readdir+0xd/0x100 [nfs]

Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
normal kernel: eax: daa0c740   ebx: 00001000   ecx: c1353aa0   edx: db25cd14

Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
normal kernel: esi: df02ced0   edi: daa0885c   ebp: da9c7d34   esp: da9c7c4c

Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
normal kernel: ds: 007b   es: 007b   ss: 0069

Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
normal kernel: Process ls (pid: 1670, threadinfo=da9c6000 task=c05d7070)
/**********************************************************/

Looks like "int 3" triggered kernel error. Why just for kernel module
debugging?


BTW: I built the kernel moduel with "-g" option and use gdb command to load
module symbols

/*************************************************/
add-symbol-file /path/to/module.ko 0xdf01f000 \
        -s .altinstr_replacement 0xdf036629 \
        -s .altinstructions 0xdf03add0 \
        -s .bss 0xdf042a60 \
        -s .data 0xdf041a20 \
        -s .exit.text 0xdf0365f0 \
        -s .gnu.linkonce.this_module 0xdf042840 \
        -s .init.text 0xdf045000 \
        -s .rodata 0xdf036680 \
        -s .rodata.str1.1 0xdf039ef4 \
        -s .rodata.str1.4 0xdf036cb8 \
        -s .strtab 0xdf03e450 \
        -s .symtab 0xdf03aec0
/*************************************************/

Thanks,
lily

On 1/12/07, Brendan Cully <brendan@cs.ubc.ca> wrote:
>
> On Friday, 12 January 2007 at 16:32, Lily Huang wrote:
> > Hi Brendan,
> >
> > You are right. After I lift the breakpoint a little bit (to sys_open),
> the
> > breakpoint works.
> >
> > But now the domU kept complaining after I did "target remote
> 127.0.0.1:9999"
> > and then "br sys_open":
> >
> > /************************************************?
> > BUG: soft lockup detected on CPU#0!
> >
> > Pid: 0, comm:              swapper
> > EIP: 0061:[<c01013a7>] CPU: 0
> > EIP is at 0xc01013a7
> > EFLAGS: 00000346    Not tainted  (2.6.16.33-xenU #12)
> > EAX: 00000000 EBX: 00000001 ECX: 00000000 EDX: 00000000
> > ESI: 00000001 EDI: c0354000 EBP: c0355f84 DS: 007b ES: 007b
> > CR0: 8005003b CR2: b7f77000 CR3: 1d633000 CR4: 00000640
> > [<c0105bb3>] show_trace+0x13/0x20
> > [<c0103390>] show_regs+0x190/0x1e0
> > [<c013cf48>] softlockup_tick+0x88/0xa0
> > [<c0126d45>] do_timer+0x215/0x440
> > [<c010858c>] timer_interrupt+0x1bc/0x6a0
> > [<c013d09f>] handle_IRQ_event+0x3f/0xd0
> > [<c013d1be>] __do_IRQ+0x8e/0xf0
> > [<c0106efd>] do_IRQ+0x1d/0x30
> > [<c025d2d1>] evtchn_do_upcall+0xa1/0xe0
> > [<c0105309>] hypervisor_callback+0x3d/0x48
> > [<c01039df>] xen_idle+0x2f/0x60
> > [<c0103a82>] cpu_idle+0x72/0xc0
> > [<c0102035>] rest_init+0x35/0x40
> > [<c035656a>] start_kernel+0x2ea/0x3a0
> > [<c010006f>] 0xc010006f
> >
> > /********************************************************/
> >
> > What's wrong? Looks like I am really doomed to run into so many weird
> > problems. :(
>
> soft lockup gets triggered when the kernel hasn't gotten a timer
> interrupt in a while. This tends to happen frequently when the kernel
> is being debugged. I think there's a kernel config option to turn this
> off, or you can just live with the messages.
>

[-- Attachment #1.2: Type: text/html, Size: 5568 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: help on debugging kernel module in Xen

[Lily Huang]

[-- Attachment #1.1: Type: text/plain, Size: 4645 bytes --]

Sorry, please skip my last question. I fixed the problem myself.

The problem occurred because I failed to run "add-symbol-file" command.
Somehow gdb failed to recognize the strtab and symtab sections.

After I removed them, it works now.

Again, thanks for your help!

lily

On 1/12/07, Lily Huang <ushuanglily@gmail.com> wrote:
>
> That's a great point. I just disabled the "Detect Soft Lockup" option in
> the "kernel hacking" group. Now the soft lockup disappeared and I can debug
> kernel now.
>
> Next, I tried to debug kernel module in domU. New problem occurs.
>
> After I set "br xxx(function in a loaded kernel module" and then "cont", I
> worked in the domu to activate the breakpoint.
>
> Unfortunately, I found domU immediately print error messages:
>
> /**********************************************************/
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: int3: 0000 [#1]
> <----------------------------------------------------- seems to be a problem
>
>
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: SMP
>
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: CPU:    0
>
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: EIP is at nfs3_proc_readdir+0xd/0x100 [nfs]
>
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: eax: daa0c740   ebx: 00001000   ecx: c1353aa0   edx:
> db25cd14
>
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: esi: df02ced0   edi: daa0885c   ebp: da9c7d34   esp:
> da9c7c4c
>
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: ds: 007b   es: 007b   ss: 0069
>
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: Process ls (pid: 1670, threadinfo=da9c6000 task=c05d7070)
> /**********************************************************/
>
> Looks like "int 3" triggered kernel error. Why just for kernel module
> debugging?
>
>
> BTW: I built the kernel moduel with "-g" option and use gdb command to
> load module symbols
>
> /*************************************************/
> add-symbol-file /path/to/module.ko 0xdf01f000 \
>         -s .altinstr_replacement 0xdf036629 \
>         -s .altinstructions 0xdf03add0 \
>         -s .bss 0xdf042a60 \
>         -s .data 0xdf041a20 \
>         -s .exit.text 0xdf0365f0 \
>         -s .gnu.linkonce.this_module 0xdf042840 \
>         -s .init.text 0xdf045000 \
>         -s .rodata 0xdf036680 \
>         -s .rodata.str1.1 0xdf039ef4 \
>         -s .rodata.str1.4 0xdf036cb8 \
>         -s .strtab 0xdf03e450 \
>         -s .symtab 0xdf03aec0
> /*************************************************/
>
> Thanks,
> lily
>
> On 1/12/07, Brendan Cully <brendan@cs.ubc.ca> wrote:
> >
> > On Friday, 12 January 2007 at 16:32, Lily Huang wrote:
> > > Hi Brendan,
> > >
> > > You are right. After I lift the breakpoint a little bit (to sys_open),
> > the
> > > breakpoint works.
> > >
> > > But now the domU kept complaining after I did "target remote
> > 127.0.0.1:9999"
> > > and then "br sys_open":
> > >
> > > /************************************************?
> > > BUG: soft lockup detected on CPU#0!
> > >
> > > Pid: 0, comm:              swapper
> > > EIP: 0061:[<c01013a7>] CPU: 0
> > > EIP is at 0xc01013a7
> > > EFLAGS: 00000346    Not tainted  (2.6.16.33-xenU #12)
> > > EAX: 00000000 EBX: 00000001 ECX: 00000000 EDX: 00000000
> > > ESI: 00000001 EDI: c0354000 EBP: c0355f84 DS: 007b ES: 007b
> > > CR0: 8005003b CR2: b7f77000 CR3: 1d633000 CR4: 00000640
> > > [<c0105bb3>] show_trace+0x13/0x20
> > > [<c0103390>] show_regs+0x190/0x1e0
> > > [<c013cf48>] softlockup_tick+0x88/0xa0
> > > [<c0126d45>] do_timer+0x215/0x440
> > > [<c010858c>] timer_interrupt+0x1bc/0x6a0
> > > [<c013d09f>] handle_IRQ_event+0x3f/0xd0
> > > [<c013d1be>] __do_IRQ+0x8e/0xf0
> > > [<c0106efd>] do_IRQ+0x1d/0x30
> > > [<c025d2d1>] evtchn_do_upcall+0xa1/0xe0
> > > [<c0105309>] hypervisor_callback+0x3d/0x48
> > > [<c01039df>] xen_idle+0x2f/0x60
> > > [<c0103a82>] cpu_idle+0x72/0xc0
> > > [<c0102035>] rest_init+0x35/0x40
> > > [<c035656a>] start_kernel+0x2ea/0x3a0
> > > [<c010006f>] 0xc010006f
> > >
> > > /********************************************************/
> > >
> > > What's wrong? Looks like I am really doomed to run into so many weird
> > > problems. :(
> >
> > soft lockup gets triggered when the kernel hasn't gotten a timer
> > interrupt in a while. This tends to happen frequently when the kernel
> > is being debugged. I think there's a kernel config option to turn this
> > off, or you can just live with the messages.
> >
>
>

[-- Attachment #1.2: Type: text/html, Size: 6468 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: help on debugging kernel module in Xen

[Brendan Cully]

On Friday, 12 January 2007 at 18:04, Lily Huang wrote:
> That's a great point. I just disabled the "Detect Soft Lockup" option in the
> "kernel hacking" group. Now the soft lockup disappeared and I can debug
> kernel now.
> 
> Next, I tried to debug kernel module in domU. New problem occurs.
> 
> After I set "br xxx(function in a loaded kernel module" and then "cont", I
> worked in the domu to activate the breakpoint.
> 
> Unfortunately, I found domU immediately print error messages:
> 
> /**********************************************************/
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: int3: 0000 [#1]
> <----------------------------------------------------- seems to be a problem
> 
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: SMP
> 
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: CPU:    0
> 
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: EIP is at nfs3_proc_readdir+0xd/0x100 [nfs]
> 
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: eax: daa0c740   ebx: 00001000   ecx: c1353aa0   edx: db25cd14
> 
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: esi: df02ced0   edi: daa0885c   ebp: da9c7d34   esp: da9c7c4c
> 
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: ds: 007b   es: 007b   ss: 0069
> 
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: Process ls (pid: 1670, threadinfo=da9c6000 task=c05d7070)
> /**********************************************************/
> 
> Looks like "int 3" triggered kernel error. Why just for kernel module
> debugging?

I'm afraid you're beyond my knowledge at this point.

Re: help on debugging kernel module in Xen

[Brendan Cully]

On Friday, 12 January 2007 at 16:07, Lily Huang wrote:
> Many thanks for your help!
> 
> I tried the command "target remote 127.0.0.1:9999", and got the response:
> 
> /******************************************************************************/
> Remote debugging using 127.0.0.1:9999
> [New Thread 0]
> [Switching to Thread 0]
> 0xc01013a7 in hypercall_page ()
> warning: shared library handler failed to enable breakpoint
> <------------------ this could be a problem
> /******************************************************************************/
> 
> then, I did "br filp_open", and then "cont".
> 
> But no matter what file I opened in domU, the breakpoint is never
> encountered. It appears that the breakpoint is not activate.
> 
> I googled but cannot find answer so far.
> 
> Did you run into similar problems?

I think I'm out of answers at this point. Perhaps it's possible that
the function has been inlined or something (though that sounds like a
function pointer). You might try breaking a little higher in the call
chain. And, until you figure out what's wrong with the debugger,
printk can help you make sure you're at least triggering the code you
expect to hit the breakpoint :)