Debugging Xen via serial console

Debugging Xen via serial console

[Emre Can Sezer]

I'm trying to debug Xen remotely using serial port.  I followed the 
instructions on the following post made in this xen-devel list:
http://lists.xensource.com/archives/html/xen-devel/2007-12/msg00678.html

I was able to connect gdb but I ran into a problem where even after I 
clear all the breakpoints, execution traps into gdb with SIGTRAP at 
restore_all_xen().  This only happens if I set a breakpoint and then try 
to continue after hitting the break point.  If I simply connect gdb but 
continue without setting any breakpoints, everything is OK.

The post mentions some stability issues and I haven't seen any follow 
ups.  Is this still a viable debugging method?  Or is there some other 
method I should try?

Thanks in advance,

John

Re: Debugging Xen via serial console

[Dan Doucette]

[-- Attachment #1.1: Type: text/plain, Size: 1808 bytes --]

Hi John,

The patches I submitted quickly fell out of sync with the 3.2 unstable
branch at the time.  I found a month or so later they stopped working, and I
didnt have the time to keep them up to date.
For Xen debugging, have you tried the kdb branch maintained my Mukesh Rathor
(Oracle)?  I used this a few months ago and it seemed to work.  The branch
is off the mainline, however.  Its usage is documented in the xen wiki
pages, I believe.

If you would prefer to use gdb, the functionality should be easy to
resurrect.  There really wasnt much to it.
I would recommend using my serial port splitter (ssplitd.c), which I found
easier to use and was complete in its implementation.  I submitted all the
changes in proper patch form after the posting you referenced above, if you
are interested.

Dan.




On Wed, Jan 28, 2009 at 10:53 AM, Emre Can Sezer <ecsezer@ncsu.edu> wrote:

> I'm trying to debug Xen remotely using serial port.  I followed the
> instructions on the following post made in this xen-devel list:
> http://lists.xensource.com/archives/html/xen-devel/2007-12/msg00678.html
>
> I was able to connect gdb but I ran into a problem where even after I clear
> all the breakpoints, execution traps into gdb with SIGTRAP at
> restore_all_xen().  This only happens if I set a breakpoint and then try to
> continue after hitting the break point.  If I simply connect gdb but
> continue without setting any breakpoints, everything is OK.
>
> The post mentions some stability issues and I haven't seen any follow ups.
>  Is this still a viable debugging method?  Or is there some other method I
> should try?
>
> Thanks in advance,
>
> John
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

[-- Attachment #1.2: Type: text/html, Size: 2538 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Debugging Xen via serial console

[Emre Can Sezer]

Hi Dan,

I'm currently using your version of ssplitd as it is.  I haven't tried
kdb.  For some reason I thought that was for debugging guest domains.
I'll look into it.  As for resurrecting the gdbstub, I'm *NOT* familiar 
enough
with it to try it myself. I do have a few people here who might be though.
I'll post something if we get it working.

Thanks,

John

Dan Doucette wrote:
> Hi John,
>  
> The patches I submitted quickly fell out of sync with the 3.2 unstable 
> branch at the time.  I found a month or so later they stopped working, 
> and I didnt have the time to keep them up to date.
> For Xen debugging, have you tried the kdb branch maintained my Mukesh 
> Rathor (Oracle)?  I used this a few months ago and it seemed to work.  
> The branch is off the mainline, however.  Its usage is documented in 
> the xen wiki pages, I believe.
>  
> If you would prefer to use gdb, the functionality should be easy to 
> resurrect.  There really wasnt much to it.
> I would recommend using my serial port splitter (ssplitd.c), which I 
> found easier to use and was complete in its implementation.  I 
> submitted all the changes in proper patch form after the posting you 
> referenced above, if you are interested.
>  
> Dan.
>  
>
>
>  
> On Wed, Jan 28, 2009 at 10:53 AM, Emre Can Sezer <ecsezer@ncsu.edu 
> <mailto:ecsezer@ncsu.edu>> wrote:
>
>     I'm trying to debug Xen remotely using serial port.  I followed
>     the instructions on the following post made in this xen-devel list:
>     http://lists.xensource.com/archives/html/xen-devel/2007-12/msg00678.html
>
>     I was able to connect gdb but I ran into a problem where even
>     after I clear all the breakpoints, execution traps into gdb with
>     SIGTRAP at restore_all_xen().  This only happens if I set a
>     breakpoint and then try to continue after hitting the break point.
>      If I simply connect gdb but continue without setting any
>     breakpoints, everything is OK.
>
>     The post mentions some stability issues and I haven't seen any
>     follow ups.  Is this still a viable debugging method?  Or is there
>     some other method I should try?
>
>     Thanks in advance,
>
>     John
>
>
>
>
>     _______________________________________________
>     Xen-devel mailing list
>     Xen-devel@lists.xensource.com <mailto:Xen-devel@lists.xensource.com>
>     http://lists.xensource.com/xen-devel
>
>

Re: Debugging Xen via serial console

[Mukesh Rathor]

Hi,

  kdb: to debug xen hypervisor, could also debug guests
  gdbsx: to debug PV/HVM linux guests

The tree is : http://xenbits.xensource.com/ext/debuggers.hg

See README-dbg. You'll need to setup serial access for kdb.

Thanks,
Mukesh

 >
 > Hi Dan,
 >
 > I'm currently using your version of ssplitd as it is.  I haven't tried
 > kdb.  For some reason I thought that was for debugging guest domains.
 > I'll look into it.  As for resurrecting the gdbstub, I'm *NOT* familiar
 > enough
 > with it to try it myself. I do have a few people here who might be though.
 > I'll post something if we get it working.
 >
 > Thanks,
 >
 > John
 >
 > Dan Doucette wrote:
 >> Hi John,
 >>
 >> The patches I submitted quickly fell out of sync with the 3.2 unstable
 >> branch at the time.  I found a month or so later they stopped working,
 >> and I didnt have the time to keep them up to date.
 >> For Xen debugging, have you tried the kdb branch maintained my Mukesh
 >> Rathor (Oracle)?  I used this a few months ago and it seemed to work.
 >> The branch is off the mainline, however.  Its usage is documented in
 >> the xen wiki pages, I believe.
 >>
 >> If you would prefer to use gdb, the functionality should be easy to
 >> resurrect.  There really wasnt much to it.
 >> I would recommend using my serial port splitter (ssplitd.c), which I
 >> found easier to use and was complete in its implementation.  I
 >> submitted all the changes in proper patch form after the posting you
 >> referenced above, if you are interested.
 >>
 >> Dan.
 >>
 >>
 >>
 >>
 >> On Wed, Jan 28, 2009 at 10:53 AM, Emre Can Sezer <ecsezer@ncsu.edu
 >> <mailto:ecsezer@ncsu.edu>> wrote:
 >>
 >>     I'm trying to debug Xen remotely using serial port.  I followed
 >>     the instructions on the following post made in this xen-devel list:
 >>     http://lists.xensource.com/archives/html/xen-devel/2007-12/msg00678.html
 >>
 >>     I was able to connect gdb but I ran into a problem where even
 >>     after I clear all the breakpoints, execution traps into gdb with
 >>     SIGTRAP at restore_all_xen().  This only happens if I set a
 >>     breakpoint and then try to continue after hitting the break point.
 >>      If I simply connect gdb but continue without setting any
 >>     breakpoints, everything is OK.
 >>
 >>     The post mentions some stability issues and I haven't seen any
 >>     follow ups.  Is this still a viable debugging method?  Or is there
 >>     some other method I should try?
 >>
 >>     Thanks in advance,
 >>
 >>     John
 >>
 >>
 >>
 >>
 >>     _______________________________________________
 >>     Xen-devel mailing list
 >>     Xen-devel@lists.xensource.com <mailto:Xen-devel@lists.xensource.com>
 >>     http://lists.xensource.com/xen-devel

Compile error in KDB branch when debug=y.

[Emre Can Sezer]

Hi Mukesh,

Got a compile time type mismatch error from debug.c:70 when compiling 
with debug=y.  Below is the single line patch to fix it.

--- xen-unstable/xen/arch/x86/debug.c    2009-02-19 14:27:02.000000000 -0500
+++ xen-debug/xen/arch/x86/debug.c    2009-02-18 17:26:09.000000000 -0500
@@ -67,7 +67,7 @@
     }
 
 #if XEN_SUBVERSION >= 2             /* xen >= 3.2.x */
-        mfn = gfn_to_mfn(dp, gfn, &gfntype);
+        mfn = mfn_x(gfn_to_mfn(dp, gfn, &gfntype));
     if (p2m_is_readonly(gfntype) && toaddr) {
         DBGP2("kdb:p2m_is_readonly: gfntype:%x\n", gfntype);
         return INVALID_MFN;


Cheers,

John



Mukesh Rathor wrote:
>
> Hi,
>
>  kdb: to debug xen hypervisor, could also debug guests
>  gdbsx: to debug PV/HVM linux guests
>
> The tree is : http://xenbits.xensource.com/ext/debuggers.hg
>
> See README-dbg. You'll need to setup serial access for kdb.
>
> Thanks,
> Mukesh
>
> >
> > Hi Dan,
> >
> > I'm currently using your version of ssplitd as it is.  I haven't tried
> > kdb.  For some reason I thought that was for debugging guest domains.
> > I'll look into it.  As for resurrecting the gdbstub, I'm *NOT* familiar
> > enough
> > with it to try it myself. I do have a few people here who might be 
> though.
> > I'll post something if we get it working.
> >
> > Thanks,
> >
> > John
> >
> > Dan Doucette wrote:
> >> Hi John,
> >>
> >> The patches I submitted quickly fell out of sync with the 3.2 unstable
> >> branch at the time.  I found a month or so later they stopped working,
> >> and I didnt have the time to keep them up to date.
> >> For Xen debugging, have you tried the kdb branch maintained my Mukesh
> >> Rathor (Oracle)?  I used this a few months ago and it seemed to work.
> >> The branch is off the mainline, however.  Its usage is documented in
> >> the xen wiki pages, I believe.
> >>
> >> If you would prefer to use gdb, the functionality should be easy to
> >> resurrect.  There really wasnt much to it.
> >> I would recommend using my serial port splitter (ssplitd.c), which I
> >> found easier to use and was complete in its implementation.  I
> >> submitted all the changes in proper patch form after the posting you
> >> referenced above, if you are interested.
> >>
> >> Dan.
> >>
> >>
> >>
> >>
> >> On Wed, Jan 28, 2009 at 10:53 AM, Emre Can Sezer <ecsezer@ncsu.edu
> >> <mailto:ecsezer@ncsu.edu>> wrote:
> >>
> >>     I'm trying to debug Xen remotely using serial port.  I followed
> >>     the instructions on the following post made in this xen-devel 
> list:
> >>     
> http://lists.xensource.com/archives/html/xen-devel/2007-12/msg00678.html
> >>
> >>     I was able to connect gdb but I ran into a problem where even
> >>     after I clear all the breakpoints, execution traps into gdb with
> >>     SIGTRAP at restore_all_xen().  This only happens if I set a
> >>     breakpoint and then try to continue after hitting the break point.
> >>      If I simply connect gdb but continue without setting any
> >>     breakpoints, everything is OK.
> >>
> >>     The post mentions some stability issues and I haven't seen any
> >>     follow ups.  Is this still a viable debugging method?  Or is there
> >>     some other method I should try?
> >>
> >>     Thanks in advance,
> >>
> >>     John
> >>
> >>
> >>
> >>
> >>     _______________________________________________
> >>     Xen-devel mailing list
> >>     Xen-devel@lists.xensource.com 
> <mailto:Xen-devel@lists.xensource.com>
> >>     http://lists.xensource.com/xen-devel

Re: Compile error in KDB branch when debug=y.

[Mukesh Rathor]

Hi John,

Thanks for the patch.
Actually, I noticed it last week when I refreshed the tree to latest and
debug was turned on by default. There are problems running kdb with
debug=y, like:

(XEN) Xen BUG at spinlock.c:23
*** kdb (Fatal Error on cpu:1 vec:6 Invalid Opcode):
ffff828c8011b865: check_lock+3d                  ud2
[1]xkdb>
[1]xkdb> f
(XEN)    [<ffff828c8011b865>] check_lock+0x3d/0x4e
(XEN)    [<ffff828c8011b9eb>] _spin_lock+0x11/0x24
(XEN)    [<ffff828c8015bd5c>] do_settime+0x26/0x87
(XEN)    [<ffff828c8015c1f5>] kdb_time_resume+0x90/0x121
(XEN)    [<ffff828c801d94d4>] kdb_end_session+0x8f/0x298
(XEN)    [<ffff828c801d9e3b>] kdbmain+0x75e/0x867
(XEN)    [<ffff828c801d9fa2>] kdb_keyboard+0x11/0x13
(XEN)    [<ffff828c80128e99>] serial_rx+0x2f/0xd8
(XEN)    [<ffff828c8012a2e7>] serial_rx_interrupt+0xa9/0xbf


So I'd recommend turning it off, even tho I've fixed the build break.  Since
I always have debug=n (with frame_pointer=yes) when I debug,  I've not
looked at any of these issues.

BTW, I've refreshed the tree to latest, so please check it out:
http://xenbits.xensource.com/ext/debuggers.hg

Thanks,
Mukesh


Emre Can Sezer wrote:
 > Hi Mukesh,
 >
 > Got a compile time type mismatch error from debug.c:70 when compiling
 > with debug=y.  Below is the single line patch to fix it.
 >
 > --- xen-unstable/xen/arch/x86/debug.c    2009-02-19 14:27:02.000000000
 > -0500
 > +++ xen-debug/xen/arch/x86/debug.c    2009-02-18 17:26:09.000000000 -0500
 > @@ -67,7 +67,7 @@
 >     }
 >
 > #if XEN_SUBVERSION >= 2             /* xen >= 3.2.x */
 > -        mfn = gfn_to_mfn(dp, gfn, &gfntype);
 > +        mfn = mfn_x(gfn_to_mfn(dp, gfn, &gfntype));
 >     if (p2m_is_readonly(gfntype) && toaddr) {
 >         DBGP2("kdb:p2m_is_readonly: gfntype:%x\n", gfntype);
 >         return INVALID_MFN;
 >
 >
 > Cheers,
 >
 > John
 >
 >
 >
 > Mukesh Rathor wrote:
 >>
 >> Hi,
 >>
 >>  kdb: to debug xen hypervisor, could also debug guests
 >>  gdbsx: to debug PV/HVM linux guests
 >>
 >> The tree is : http://xenbits.xensource.com/ext/debuggers.hg
 >>
 >> See README-dbg. You'll need to setup serial access for kdb.
 >>
 >> Thanks,
 >> Mukesh
 >>
 >> >
 >> > Hi Dan,
 >> >
 >> > I'm currently using your version of ssplitd as it is.  I haven't tried
 >> > kdb.  For some reason I thought that was for debugging guest domains.
 >> > I'll look into it.  As for resurrecting the gdbstub, I'm *NOT* familiar
 >> > enough
 >> > with it to try it myself. I do have a few people here who might be
 >> though.
 >> > I'll post something if we get it working.
 >> >
 >> > Thanks,
 >> >
 >> > John
 >> >
 >> > Dan Doucette wrote:
 >> >> Hi John,
 >> >>
 >> >> The patches I submitted quickly fell out of sync with the 3.2 unstable
 >> >> branch at the time.  I found a month or so later they stopped working,
 >> >> and I didnt have the time to keep them up to date.
 >> >> For Xen debugging, have you tried the kdb branch maintained my Mukesh
 >> >> Rathor (Oracle)?  I used this a few months ago and it seemed to work.
 >> >> The branch is off the mainline, however.  Its usage is documented in
 >> >> the xen wiki pages, I believe.
 >> >>
 >> >> If you would prefer to use gdb, the functionality should be easy to
 >> >> resurrect.  There really wasnt much to it.
 >> >> I would recommend using my serial port splitter (ssplitd.c), which I
 >> >> found easier to use and was complete in its implementation.  I
 >> >> submitted all the changes in proper patch form after the posting you
 >> >> referenced above, if you are interested.
 >> >>
 >> >> Dan.
 >> >>
 >> >>
 >> >>
 >> >>
 >> >> On Wed, Jan 28, 2009 at 10:53 AM, Emre Can Sezer <ecsezer@ncsu.edu
 >> >> <mailto:ecsezer@ncsu.edu>> wrote:
 >> >>
 >> >>     I'm trying to debug Xen remotely using serial port.  I followed
 >> >>     the instructions on the following post made in this xen-devel
 >> list:
 >> >>
 >> http://lists.xensource.com/archives/html/xen-devel/2007-12/msg00678.html
 >> >>
 >> >>     I was able to connect gdb but I ran into a problem where even
 >> >>     after I clear all the breakpoints, execution traps into gdb with
 >> >>     SIGTRAP at restore_all_xen().  This only happens if I set a
 >> >>     breakpoint and then try to continue after hitting the break point.
 >> >>      If I simply connect gdb but continue without setting any
 >> >>     breakpoints, everything is OK.
 >> >>
 >> >>     The post mentions some stability issues and I haven't seen any
 >> >>     follow ups.  Is this still a viable debugging method?  Or is there
 >> >>     some other method I should try?
 >> >>
 >> >>     Thanks in advance,
 >> >>
 >> >>     John
 >> >>
 >> >>
 >> >>
 >> >>
 >> >>     _______________________________________________
 >> >>     Xen-devel mailing list
 >> >>     Xen-devel@lists.xensource.com
 >> <mailto:Xen-devel@lists.xensource.com>
 >> >>     http://lists.xensource.com/xen-devel