Debugging the MIPS processor using GDB

Debugging the MIPS processor using GDB

[TriKri]

The questions are many, and I don't really know where to begin, but I shall
make a try.

I am developing software to debug a MIPS processor in an embedded system
using an EJTAG probe. Currently I have software to tell the processor to
perform certain instructions, and read/write from/to certain register
addresses. Now, I don't really know how to continue from here; I'm trying to
find out how to make GDB benefit from these functions in order to be able to
debug the processor.

I know that GDB needs a gdb-stub, but I don't really know what it is and
what it does, or why GDB needs it. The GDB manual suggests taking a look at
sparc-stub.c coming along with GDB, since it's the best organized. Not
finding any main function in it, I'm having a hard time figuring out to what
it does. How should this stub be implemented and how do I know what to write
in the stub file I'm going to write?

There is also something called gdb-hook, which I don't really know what it
is either. Do I need it?

Finally, there's a program called gdbserver, which comes with GDB. If I
write a remote stub, do I need this program? Where should it be run? Where
should my program be run? Since the stub is a c file, but lacks of a main
function, how do I compile it?

There are just so many unanswered questions, I don't know where to start
working or where to start looking for information. I haven't used GDB before
either, possibly contributing to some of the confusion; I hope I'll get this
straight soon, though.

/Kristofer
-- 
View this message in context: http://www.nabble.com/Debugging-the-MIPS-processor-using-GDB-tp18944199p18944199.html
Sent from the linux-mips main mailing list archive at Nabble.com.

Re: Debugging the MIPS processor using GDB

[Martin Gebert]

> Finally, there's a program called gdbserver, which comes with GDB. If I
> write a remote stub, do I need this program? Where should it be run? Where
> should my program be run? Since the stub is a c file, but lacks of a main
> function, how do I compile it?
>   

At least this I can answer. In short, you need to call gdbserver on your 
target machine in order to do remote debugging from your workstation. 
The first Google match for "using gdbserver" reveals this:

http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/gdb/server.html

HTH

Martin

Re: Debugging the MIPS processor using GDB

[Brian Foster]

On Tuesday 12 August 2008 16:14:22 Martin Gebert wrote:
> > Finally, there's a program called gdbserver, which comes with GDB. If I
> > write a remote stub, do I need this program? Where should it be run? Where
> > should my program be run? Since the stub is a c file, but lacks of a main
> > function, how do I compile it?
> 
> At least this I can answer. In short, you need to call gdbserver on your 
> target machine in order to do remote debugging from your workstation. 
> The first Google match for "using gdbserver" reveals this:
> 
> http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/gdb/server.html

  To the best of my knowledge, ‘gdbserver’ itself is not normally
 used with (E)JTAG probes; it's mostly for debugging userland
 code on a remote target machine whilst running ‘gdb’ itself on
 a local host workstation.

  I'm using the commercial FS² (First Silicon Systems, now owned
 by MIPS) EJTAG probe.  The local ‘gdb’ on the workstation talks
 to the local FS² software on the workstation, which talks to the
 probe (in my case, over USB, but there is also an Ethernet model).
 There is no ‘gdbserver’ in this setup per se, albeit I suppose
 the protocol between ‘gdb’ and the FS² software (which is called
 something like ‘jnetserver’?) might be similar/identical (I have
 no idea!).

cheers!
	-blf-

-- 
“How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools.” |      http://www.stopesso.com

Re: Debugging the MIPS processor using GDB

[Maciej W. Rozycki]

On Tue, 12 Aug 2008, Brian Foster wrote:

>   I'm using the commercial FS² (First Silicon Systems, now owned
>  by MIPS) EJTAG probe.  The local ‘gdb’ on the workstation talks
>  to the local FS² software on the workstation, which talks to the
>  probe (in my case, over USB, but there is also an Ethernet model).
>  There is no ‘gdbserver’ in this setup per se, albeit I suppose
>  the protocol between ‘gdb’ and the FS² software (which is called
>  something like ‘jnetserver’?) might be similar/identical (I have
>  no idea!).

 Not really -- it uses a C API called MDI -- the spec is available from
MIPS Technologies.  I am happy to read somebody finds it useful. :)  
Debugging the Linux kernel with GDB and this piece of hardware is
certainly a lot of fun.

  Maciej

Re: Debugging the MIPS processor using GDB

[Brian Foster]

On Tuesday 12 August 2008 18:27:42 Maciej W. Rozycki wrote:
> On Tue, 12 Aug 2008, Brian Foster wrote:
> >   I'm using the commercial FS² (First Silicon Systems, now owned
> >  by MIPS) EJTAG probe.  [ ... ]  There is no ‘gdbserver’ in this
> >  setup per se, albeit I suppose the protocol between ‘gdb’ and
> >  the FS² software [ ... ] might be similar/identical[?]
> 
>  Not really -- it uses a C API called MDI -- the spec is available from
> MIPS Technologies.  I am happy to read somebody finds it useful. :) 
> Debugging the Linux kernel with GDB and this piece of hardware is
> certainly a lot of fun.

Maciej,

  Thanks for the clarification.  I didn't know if MDI
 was related to the remote-‘gdbserver’ stuff or not.

  Re the FS²:  When it works, my (somewhat limited)
 experience to-date is it works Ok.  And the use of
 TCL on the Host workstation side allows some neat
 tricks.  However, at least one thing doesn't work
 reliably for me, albeit I've never investigated:
 Breakpoints in the Linux kernel.  They do detonate.
 Then, sometimes, I can ‘c’(ontinue) or ‘s’(tep) Ok.
 But other times, when I ‘c’ or ‘s’, the breakpoint
 detonates again and I'm stuck.  I cannot proceed.
 (The same breakpoint might even work once or twice
 and then fail.)   Any ideas?   AFAICR, this can also
 happen if I try to use the ‘sysnav’ console instead
 of ‘gdb’.

  I understand my predecessor in my job I gave up on
 the FS² (very possibly because of this breakpoint
 issue?) and used a competing (E?)JTAG probe.

  Weirdly, I've only seen this effect with the Linux
 kernel — other kernel-mode software (e.g., the trivial
 custom bootloader) — doesn't seem to suffer from these
 “flakey FS² breakpoints”?

cheers!
	-blf-

-- 
“How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools.” |      http://www.stopesso.com

Re: Debugging the MIPS processor using GDB

[Maciej W. Rozycki]

On Wed, 13 Aug 2008, Brian Foster wrote:

>   Re the FS²:  When it works, my (somewhat limited)
>  experience to-date is it works Ok.  And the use of
>  TCL on the Host workstation side allows some neat
>  tricks.  However, at least one thing doesn't work
>  reliably for me, albeit I've never investigated:
>  Breakpoints in the Linux kernel.  They do detonate.
>  Then, sometimes, I can ‘c’(ontinue) or ‘s’(tep) Ok.
>  But other times, when I ‘c’ or ‘s’, the breakpoint
>  detonates again and I'm stuck.  I cannot proceed.
>  (The same breakpoint might even work once or twice
>  and then fail.)   Any ideas?   AFAICR, this can also
>  happen if I try to use the ‘sysnav’ console instead
>  of ‘gdb’.

 Hmm, odd.  It looks like a cache coherence issue.  It could be a bug in
your version of FS2 software -- did you raise the issue with them?  
Anyway, as a workaround try setting "coherent=on" (quoting from memory) in
fs2.ini (just an idea -- it may not work and you will lose some
performance though) or use hardware breakpoints.

  Maciej

Re: Debugging the MIPS processor using GDB

[Jon Fraser]

FYI,

      I use the FS2 probe with sde-gdb on a nearly daily basis.
Are you compiling your kernel with -g -O1 ?

You can also try 'hbreak' instead of 'break' in sde-gdb.

Jon


On Wed, 2008-08-13 at 09:05 +0200, Brian Foster wrote:
> On Tuesday 12 August 2008 18:27:42 Maciej W. Rozycki wrote:
> > On Tue, 12 Aug 2008, Brian Foster wrote:
> > >   I'm using the commercial FS² (First Silicon Systems, now owned
> > >  by MIPS) EJTAG probe.  [ ... ]  There is no ‘gdbserver’ in this
> > >  setup per se, albeit I suppose the protocol between ‘gdb’ and
> > >  the FS² software [ ... ] might be similar/identical[?]
> > 
> >  Not really -- it uses a C API called MDI -- the spec is available from
> > MIPS Technologies.  I am happy to read somebody finds it useful. :) 
> > Debugging the Linux kernel with GDB and this piece of hardware is
> > certainly a lot of fun.
> 
> Maciej,
> 
>   Thanks for the clarification.  I didn't know if MDI
>  was related to the remote-‘gdbserver’ stuff or not.
> 
>   Re the FS²:  When it works, my (somewhat limited)
>  experience to-date is it works Ok.  And the use of
>  TCL on the Host workstation side allows some neat
>  tricks.  However, at least one thing doesn't work
>  reliably for me, albeit I've never investigated:
>  Breakpoints in the Linux kernel.  They do detonate.
>  Then, sometimes, I can ‘c’(ontinue) or ‘s’(tep) Ok.
>  But other times, when I ‘c’ or ‘s’, the breakpoint
>  detonates again and I'm stuck.  I cannot proceed.
>  (The same breakpoint might even work once or twice
>  and then fail.)   Any ideas?   AFAICR, this can also
>  happen if I try to use the ‘sysnav’ console instead
>  of ‘gdb’.
> 
>   I understand my predecessor in my job I gave up on
>  the FS² (very possibly because of this breakpoint
>  issue?) and used a competing (E?)JTAG probe.
> 
>   Weirdly, I've only seen this effect with the Linux
>  kernel — other kernel-mode software (e.g., the trivial
>  custom bootloader) — doesn't seem to suffer from these
>  “flakey FS² breakpoints”?
> 
> cheers!
> 	-blf-
> 

Re: Debugging the MIPS processor using GDB

[Brian Foster]

On Wednesday 13 August 2008 15:49:39 Maciej W. Rozycki wrote:
> On Wed, 13 Aug 2008, Brian Foster wrote:
> 
> >   Re the FS²:  [ ... ]  at least one thing doesn't work
> >  reliably for me [ ... ]:  Breakpoints in the Linux kernel.
> >  They do detonate.  Then, sometimes, I can ‘c’(ontinue) or
> >  ‘s’(tep) Ok.  But other times, when I ‘c’ or ‘s’, the
> >  breakpoint detonates again and I'm stuck.  [ ... ]
> 
>  Hmm, odd.  It looks like a cache coherence issue.

Maciej,

  That would be my guess also.

>  It could be a bug in your version of FS2 software -- did you raise the issue with them?  

  I've been trying to.  (I cannot say more on this ATM, sorry!)

  I'm using the most recent FS² (2.4.4) with the most recent
 SDE-Lite from MIPS (V6.06.01).  Older versions of FS²/SDE
 had the same(?) issue.  (This is with a 4KSd core, running
 Little Endian.)

  What _might_ be an issue/cause is we're using our own
 home-grown ‘gdb’ scripts (to init the memory, load the
 kernel, etc.).  I didn't write them, but I have looked
 them over, and they _look_ Ok to me.


> Anyway, as a workaround try setting "coherent=on" (quoting from memory)
> in fs2.ini (just an idea -- it may not work and you will lose some
> performance though) or use hardware breakpoints.

  As it turns out, I _have_ been running Coherent On!
 So I tried turning if Off, just to see what would happen.
 No difference.

  The behavior I saw was:

    1. gdb     b xxx_open
    2. target  cat /dev/xxx
    3. (breakpoint detonates)
    4. gdb     x/i $pc     (all is Ok)
    5. gdb     c           (Ok)
    6. target  cat /dev/xxx
    7. (breakpoint detonates)
    8. gdb     x/i $pc     (wrong! instruction is ‘sdbbp’.)

  I'm now stuck.  Any attempt to ‘c’ or ‘s’ just hits the
 breakpoint again.

  I tried some “mdi cacheflush” at some plausible-seeming
 points, all to no effect.  I also tried deleting the
 breakpoint (after step 8), which was a disaster:  (From
 memory) when I then ‘c’(ontinued), ‘gdb’ hung, and the
 ‘sysnav’ went into an infinite loop of reporting a
 breakpoint.  ;-(

 ( I seem to recall also having an issue with hardware
  breakpoints, but cannot recall for certain ATM; tests
  will have to wait until later ....  ;-\  )

  All ideas and suggestions are very welcome!

cheers!
	-blf-

-- 
“How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools.” |      http://www.stopesso.com

Re: Debugging the MIPS processor using GDB

[Brian Foster]

On Wednesday 13 August 2008 16:41:34 Jon Fraser wrote:
> I use the FS2 probe with sde-gdb on a nearly daily basis.
> Are you compiling your kernel with -g -O1 ?

Jon,

  ‘-g’, yes.  ‘-O1’?  Not sure, I will have to check when
 I get a chance.  (But I don't see how either would have
 the effect on breakpoints I'm suffering from?)

> You can also try 'hbreak' instead of 'break' in sde-gdb.

  I'll have to (re-)try that when I get a chance.  I did
 try it once, and have (very! vague!!) memories of some
 issue, but now cannot recall anything useful.  ;-\ 

cheers!
	-blf-

-- 
“How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools.” |      http://www.stopesso.com

Re: Debugging the MIPS processor using GDB

[Maciej W. Rozycki]

On Wed, 13 Aug 2008, Brian Foster wrote:

>   What _might_ be an issue/cause is we're using our own
>  home-grown ‘gdb’ scripts (to init the memory, load the
>  kernel, etc.).  I didn't write them, but I have looked
>  them over, and they _look_ Ok to me.

 That should not matter.  For example you could boot your system in the
usual way provided by the firmware and attach to an already running
kernel.  The probe does not know or care about that.

>   I tried some “mdi cacheflush” at some plausible-seeming
>  points, all to no effect.  I also tried deleting the
>  breakpoint (after step 8), which was a disaster:  (From
>  memory) when I then ‘c’(ontinued), ‘gdb’ hung, and the
>  ‘sysnav’ went into an infinite loop of reporting a
>  breakpoint.  ;-(
> 
>  ( I seem to recall also having an issue with hardware
>   breakpoints, but cannot recall for certain ATM; tests
>   will have to wait until later ....  ;-\  )
> 
>   All ideas and suggestions are very welcome!

 Your situation looks pretty miserable -- you should definitely pester 
FS2.

  Maciej

Re: Debugging the MIPS processor using GDB (and FS2 EJTAG probe breakpoint issues)

[Brian Foster]

On Wednesday 13 August 2008 17:07:30 Brian Foster wrote:
>[ ... ]
>   On Wed, 13 Aug 2008, Brian Foster wrote:
>   >   Re the FS²:  [ ... ]  at least one thing doesn't work
>   >  reliably for me [ ... ]:  Breakpoints in the Linux kernel.
>   >  They do detonate.  Then, sometimes, I can ‘c’(ontinue) or
>   >  ‘s’(tep) Ok.  But other times, when I ‘c’ or ‘s’, the
>   >  breakpoint detonates again and I'm stuck.  [ ... ]
>[ ... ]
>   I'm using the most recent FS² (2.4.4) with the most recent
>  SDE-Lite from MIPS (V6.06.01).  Older versions of FS²/SDE
>  had the same(?) issue.  (This is with a 4KSd core, running
>  Little Endian.)
>[ ... ]
>   The behavior [ is ]:
> 
>    s1. gdb     b xxx_open
>    s2. target  cat /dev/xxx
>    s3. (breakpoint detonates)
>    s4. gdb     x/i $pc     (all is Ok)
>    s5. gdb     c           (Ok)
>    s6. target  cat /dev/xxx
>    s7. (breakpoint detonates)
>    s8. gdb     x/i $pc     (wrong! instruction is ‘sdbbp’.)
> 
>   I'm now stuck.  Any attempt to ‘c’ or ‘s’ just hits the
>  breakpoint again.
>[ ... ]
>  ( I seem to recall also having an issue with hardware
>   breakpoints, but cannot recall for certain ATM; tests
>   will have to wait until later ....  ;-\  )

Hello!

 Just in case someone recognizes these symptoms (or has
 any ideas) ....

  I tried hardware breakpoints from ‘gdb’ and got:

     h1. gdb     hb xxx_open
     h2. gdb says:
           warning: Cannot query hardware breakpoint/watchpoint resources.
           warning: Available functionality may be limited.
     h3. target  cat /dev/xxx
     h4. Nothing happens!  (The breakpoint does not detonate.)

 The warnings from ‘sde-gdb’ (h2) have me puzzled.  As I read
 the VHDL synthesis definitions for our implementation of the
 4KSd core — an advantage of working for a HW company! — we
 should have 4 I and 2 D hardware breakpoints in Debug (EJTAG)
 Mode.   Related, and maybe significantly, ‘info reg’ from
 ‘sde-gdb’ reports “xxxxxxxx” for the ‘debug’ register (WTF?).

  I also tried the ‘sysnav’ command ‘bkpt’ (via the ‘monitor’
 command in ‘gdb’).   It works!   ‘set’, ‘setsw’, and ‘sethw’
 all work as expected in my simple test (below, 0xVIRTUAL is
 the virtual address of xxx_open()):

     m1. gdb     monitor bkpt set 0xVIRTUAL
     m2. target  cat /dev/xxx
     m3. (breakpoint detonates)
     m4. gdb     x/i $pc     (all is Ok)
     m5. gdb     c           (Ok)
     m6. target  cat /dev/xxx
     m7. (breakpoint detonates)
     m8. gdb     x/i $pc     (all is Ok)
     m9. gdb     c           (Ok)
     (... and repeat ad infinitum (‘s’ also works) ...)

 I presume I can cook up some ‘gdb’ macros to can (simplify)
 those horrible ‘monitor bkpt ...’ commands, but I shouldn't
 have to do that, should I?  ;-\ 

  Any ideas?  (And yes, I'm attempting to pursue this with
 FS²/MIPS, but .... .)

cheers!
	-blf-

-- 
“How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools.” |      http://www.stopesso.com

Re: Debugging the MIPS processor using GDB (and FS2 EJTAG probe breakpoint issues)

[Maciej W. Rozycki]

On Thu, 14 Aug 2008, Brian Foster wrote:

>  Just in case someone recognizes these symptoms (or has
>  any ideas) ....
> 
>   I tried hardware breakpoints from ‘gdb’ and got:
> 
>      h1. gdb     hb xxx_open
>      h2. gdb says:
>            warning: Cannot query hardware breakpoint/watchpoint resources.
>            warning: Available functionality may be limited.

 This warning is issued because FS2's MDIlib does not implement the
MDIHwBpQuery() call, which tells the MDI client what hardware breakpoint
capabilities the given implementation of MDIlib provides.  They generally
correspond to what the debug port provides, but there is room for some
abstraction to be provided by MDIlib, for example a given library
implementation may emulate ways to specify address ranges different to
what hardware implements.

 Therefore GDB cannot determine whether the intended breakpoint type will
or will not fail at the time MDISetBp() is called to actually insert the
breakpoint into the target, which only happens when execution is resumed.  
There is a heuristic implemented in GDB that tries to figure out what the
MDIlib supports by trial and error, but in the end the target may not 
really support what GDB requests and you'll get a failure when you try to 
start execution.  The message therefore serves as a precautionary warning.

 They really ought to implement this call -- I suppose if customers like
you keep pushing, they may eventually decide it is worth the effort.

 Anyway, the warning is normal and happened for all releases of FS2
software I have ever used.

>      h3. target  cat /dev/xxx
>      h4. Nothing happens!  (The breakpoint does not detonate.)

 You should be able to see a hardware breakpoint set by GDB from their
System Navigator console as soon as execution is resumed.  I don't
remember the exact command to use -- it's in their reference manual.

 I suggest you verify this way the breakpoint is indeed set in the target
and then whether it is really at an address that guarantees it to trigger.  
A good way of determining whether hardware breakpoints work is to set one
at the reset vector i.e.  0xffffffffbfc00000 (remove "f" digits as
appropriate for your platform), resume execution and hit the NMI or soft
reset button.

>  I presume I can cook up some ‘gdb’ macros to can (simplify)
>  those horrible ‘monitor bkpt ...’ commands, but I shouldn't
>  have to do that, should I?  ;-\ 

 You won't get proper handling from GDB side (condition checking, 
breakpoint commands, etc.) if you set breakpoints behind GDB's back 
anyway -- all you will get are unrecognised traps.  And resumption under 
these circumstances is rather cumbersome as you have to perform all the 
actions GDB normally does itself manually.

>   Any ideas?  (And yes, I'm attempting to pursue this with
>  FS²/MIPS, but .... .)

 Well, it's their product and they should be able to sort out your 
problem.  It's not a problem with Linux nor GDB even, so it's quite 
unlikely anybody on this list is able to provide you with any advice 
beyond what has been written already, so I suggest we stop bothering 
people with the noise.

  Maciej