Linux Kernel Debuggers, KDB or KGDB?

Linux Kernel Debuggers, KDB or KGDB?

[Paul J Albrecht]

Where can I find an analysis of the relative strengths and weaknesses of KDB
and KGDB for kernel debug? Has the linux community come to any consensus
regarding the utility one or the other?

-- 
Paul J Albrecht
pjalbrecht@home.com

Re: Linux Kernel Debuggers, KDB or KGDB?

[Jeff Dike]

pjalbrecht@home.com said:
> Where can I find an analysis of the relative strengths and weaknesses
> of KDB and KGDB for kernel debug? Has the linux community come to any
> consensus regarding the utility one or the other? 

You ought to add UML to the list, since it is useful for debugging any part of 
the kernel that's not arch code or a hardware device driver (except that 
there's now USB support for UML).

				Jeff

Re: Linux Kernel Debuggers, KDB or KGDB?

[Keith Owens]

On Mon, 30 Apr 2001 16:17:22 -0500, 
Paul J Albrecht <pjalbrecht@home.com> wrote:
>Where can I find an analysis of the relative strengths and weaknesses of KDB
>and KGDB for kernel debug? Has the linux community come to any consensus
>regarding the utility one or the other?

kdb is a really low level debugger which understands the kernel
structures.  It does its utmost to stop all kernel activity while it is
running and to use as few kernel services as possible so it can run
even when the kernel is dead.  It (currently) has no source level
debugging.

kgdb relies on gdb so you loose the knowledge of kernel internals (no,
I am *not* going to teach gdb about kernel stacks, out of line lock
code etc.).  kgdb has more of a dependency on a working kernel.  It
provides source level debugging, although stack backtrace tends not to
work unless you compile the kernel with frame pointers.

UML is great for debugging generic kernel code such as filesystems, but
cannot be used for most arch code or hardware drivers.

My ideal debugger is one that combines the internal knowledge of kdb
with the source level debugging of gdb.  I know how to do this over a
serial line, finding time to write the code is the problem.

Re: Linux Kernel Debuggers, KDB or KGDB?

[Aaron Passey]

On Tue, 01 May 2001 11:16:50 +1000, Keith Owens <kaos@ocs.com.au> wrote:
>kgdb relies on gdb so you loose the knowledge of kernel internals (no,
>I am *not* going to teach gdb about kernel stacks, out of line lock
>code etc.).  kgdb has more of a dependency on a working kernel.  It
>provides source level debugging, although stack backtrace tends not to
>work unless you compile the kernel with frame pointers.
>
>UML is great for debugging generic kernel code such as filesystems, but
>cannot be used for most arch code or hardware drivers.
>
>My ideal debugger is one that combines the internal knowledge of kdb
>with the source level debugging of gdb.  I know how to do this over a
>serial line, finding time to write the code is the problem.

I've been thinking about this a little bit and I suspect the right thing
may be to combine a kgdb style debuging stub with the Mission Critical
Linux crash code (http://oss.missioncriticallinux.com/projects/crash/).
Crash is based around gdb and adds the ability to easily examine the
process table, memory maps, kernel logs, wait queues, timers, etc.  Crash
already is able to examine a live system by reading /dev/mem.  The only
thing you'd need to add is the ability to attach to a live system over a
serial port (probably not too hard since gdb already knows how to do that).

Aaron

Re: Linux Kernel Debuggers, KDB or KGDB?

[Ingo Oeser]

On Mon, Apr 30, 2001 at 07:11:35PM -0500, Jeff Dike wrote:
> pjalbrecht@home.com said:
> > Where can I find an analysis of the relative strengths and weaknesses
> > of KDB and KGDB for kernel debug? Has the linux community come to any
> > consensus regarding the utility one or the other? 
> 
> You ought to add UML to the list, since it is useful for
> debugging any part of the kernel that's not arch code or a
> hardware device driver (except that there's now USB support for
> UML).

Basically you could add support for ALL generic subsystems, that
support dummy hardware, like SCSI and ISDN for example.

Is that planned or do I suggest sth. stupid here? ;-)

Regards

Ingo Oeser
-- 
10.+11.03.2001 - 3. Chemnitzer LinuxTag <http://www.tu-chemnitz.de/linux/tag>
         <<<<<<<<<<<<     been there and had much fun   >>>>>>>>>>>>

Re: Linux Kernel Debuggers, KDB or KGDB?

[Amit S. Kale]

Hi,

I mostly agree with Keith.

kdb and kgdb have similar capabilities. kgdb can be used
to do kdb like assembly level debugging too, though it
needs some knowledge of gdb and kgdb internals.

You can analyze any problems with either of the debuggers.
You'll need to decide which debugger to go for depending
on your requirements.

I've got many queries asking kgdb capabilities.
I guess it's time for a webpage comparing the two
debuggers. People who are new to linux kernel and
have not used any of the debuggers can hardly arrive
at a decision as to which debugger is better suited for
their work.

I always suggest for kgdb over kdb because, IMO source
level debugging makes a programmers life much easier.

UML too is a good kernel debugging tool. It offers
the advantage of source level debugging on a single
machine. IMHO it's more useful for beginners.

Keith Owens wrote:
> 
> On Mon, 30 Apr 2001 16:17:22 -0500,
> Paul J Albrecht <pjalbrecht@home.com> wrote:
> >Where can I find an analysis of the relative strengths and weaknesses of KDB
> >and KGDB for kernel debug? Has the linux community come to any consensus
> >regarding the utility one or the other?
> 
> kdb is a really low level debugger which understands the kernel
> structures.  It does its utmost to stop all kernel activity while it is
> running and to use as few kernel services as possible so it can run
> even when the kernel is dead.  It (currently) has no source level
> debugging.
> 
> kgdb relies on gdb so you loose the knowledge of kernel internals (no,
> I am *not* going to teach gdb about kernel stacks, out of line lock
> code etc.).  kgdb has more of a dependency on a working kernel.  It
> provides source level debugging, although stack backtrace tends not to
> work unless you compile the kernel with frame pointers.
> 
> UML is great for debugging generic kernel code such as filesystems, but
> cannot be used for most arch code or hardware drivers.
> 
> My ideal debugger is one that combines the internal knowledge of kdb
> with the source level debugging of gdb.  I know how to do this over a
> serial line, finding time to write the code is the problem.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Amit Kale
Veritas Software ( http://www.veritas.com )

Re: Linux Kernel Debuggers, KDB or KGDB?

[Jeff Dike]

ingo.oeser@informatik.tu-chemnitz.de said:
> Basically you could add support for ALL generic subsystems, that
> support dummy hardware, like SCSI and ISDN for example.

> Is that planned or do I suggest sth. stupid here? ;-) 

Neither.  I know squat about hardware, so I had no idea that SCSI and ISDN 
would be easy to do from UML.

If the SCSI and ISDN people want to produce appropriate UML drivers, I take 
patches :-)

				Jeff

Re: Linux Kernel Debuggers, KDB or KGDB?

[Ingo Oeser]

On Tue, May 01, 2001 at 10:22:14AM -0500, Jeff Dike wrote:
> ingo.oeser@informatik.tu-chemnitz.de said:
> > Basically you could add support for ALL generic subsystems, that
> > support dummy hardware, like SCSI and ISDN for example.
> > Is that planned or do I suggest sth. stupid here? ;-) 
> 
> Neither.  I know squat about hardware, so I had no idea that SCSI and ISDN 
> would be easy to do from UML.
> 
> If the SCSI and ISDN people want to produce appropriate UML drivers, I take 
> patches :-)

Everything is there. SCSI and ISDN have the equivalent devices of the
"lo" driver for the networking layer. Or the equivalent of
tun/tap devices for the ethernet layer.

It just have to be an config.in option in UML and every other
adapters switched off.

The problem is: I still do not really get how UML really works.
Many of the mapping rules (Kernel machanism on normal arch ->
UML) are not quite clear to me.

Is there a paper or sth. like that describing the design a bit
more in detail? I only found usage papers on the user-mode-linux
home page.

Regards

Ingo Oeser
-- 
10.+11.03.2001 - 3. Chemnitzer LinuxTag <http://www.tu-chemnitz.de/linux/tag>
         <<<<<<<<<<<<     been there and had much fun   >>>>>>>>>>>>

Re: Linux Kernel Debuggers, KDB or KGDB?

[Andi Kleen]

On Tue, May 01, 2001 at 11:16:50AM +1000, Keith Owens wrote:
> My ideal debugger is one that combines the internal knowledge of kdb
> with the source level debugging of gdb.  I know how to do this over a
> serial line, finding time to write the code is the problem.

http://pice.sourceforge.net is one approach to it; but it's currently
a bit limited: e.g. ATM UP-i386-2.2 only and somewhat windowish; also the
approach only really works for modules not complete kernel sources unless
you have a *lot* of memory. Still in its limits it works.

-Andi

Re: Linux Kernel Debuggers, KDB or KGDB?

[Alan Cox]

> > Everything is there. SCSI and ISDN have the equivalent devices of the
> > "lo" driver for the networking layer. Or the equivalent of tun/tap
> > devices for the ethernet layer.
> 
> Is this sufficient to do driver development?  TUN/TAP doesn't let me write 
> ethernet drivers inside UML.

For ISDN not really. For SCSI yes - scsi generic would let you write a virtual
scsi adapter 'owning' some physical devices

Re: Linux Kernel Debuggers, KDB or KGDB?

[Jeff Dike]

ingo.oeser@informatik.tu-chemnitz.de said:
> Everything is there. SCSI and ISDN have the equivalent devices of the
> "lo" driver for the networking layer. Or the equivalent of tun/tap
> devices for the ethernet layer.

Is this sufficient to do driver development?  TUN/TAP doesn't let me write 
ethernet drivers inside UML.

> It just have to be an config.in option in UML and every other adapters
> switched off.

This is a matter of fiddling the config process.  It might not be easy, but 
there isn't much doubt that it's possible :-)

> The problem is: I still do not really get how UML really works. Many
> of the mapping rules (Kernel machanism on normal arch -> UML) are not
> quite clear to me.
> Is there a paper or sth. like that describing the design a bit more in
> detail? I only found usage papers on the user-mode-linux home page. 

First, have a look at the USB patch at http://user-mode-linux.sourceforge.net/p
atches/uml-hcd-2.4.3.patch

My ALS paper has a description of how UML basically works inside and it's not 
too much out of date - http://user-mode-linux.sourceforge.net/als2000/index.htm
l

Also, my ALS and LCA slides have this sort of information -

http://user-mode-linux.sourceforge.net/slides/als2000/slides.html
http://user-mode-linux.sourceforge.net/slides/lca2001/lca.html

The LCA slides are probably better.  They are from a few months later, and I 
had a longer slot, so I needed more slides.

If you have questions which aren't answered on my site, send me mail.

				Jeff

Re: Linux Kernel Debuggers, KDB or KGDB?

[Bill Nottingham]

Jeff Dike (jdike@karaya.com) said: 
> > > Is this sufficient to do driver development?  TUN/TAP doesn't let me
> > > write 
> > > ethernet drivers inside UML.
> > For ISDN not really. For SCSI yes - scsi generic would let you write a
> > virtual scsi adapter 'owning' some physical devices 
> 
> Fine, so go ahead and write a UML SCSI adapter...  
> 
> I would love to see this happen.  If you need UML help that's not on the site, 
> let me know, and I'll be happy to do what I can.

There is the simulator SCSI, net, and serial drivers in the ia64
tree. Dunno how similar that would be to what you need.

Bill

Re: Linux Kernel Debuggers, KDB or KGDB?

[Jeff Dike]

alan@lxorguk.ukuu.org.uk said:
> > Is this sufficient to do driver development?  TUN/TAP doesn't let me
> > write 
> > ethernet drivers inside UML.
> For ISDN not really. For SCSI yes - scsi generic would let you write a
> virtual scsi adapter 'owning' some physical devices 

Fine, so go ahead and write a UML SCSI adapter...  

I would love to see this happen.  If you need UML help that's not on the site, 
let me know, and I'll be happy to do what I can.

				Jeff

Re: Linux Kernel Debuggers, KDB or KGDB?

[Paul J Albrecht]

On Mon, 30 Apr 2001, Keith Owens wrote:
> On Mon, 30 Apr 2001 16:17:22 -0500, 
> Paul J Albrecht <pjalbrecht@home.com> wrote:
> >Where can I find an analysis of the relative strengths and weaknesses of KDB
> >and KGDB for kernel debug? Has the linux community come to any consensus
> >regarding the utility one or the other?
> 
> kdb is a really low level debugger which understands the kernel
> structures.  It does its utmost to stop all kernel activity while it is
> running and to use as few kernel services as possible so it can run
> even when the kernel is dead.  It (currently) has no source level
> debugging.
> 

I'd like to know more about your plans to enhance KDB with source level debug
capability. What sort of compiler debug output would be supported? STABs or
DWARF?  Both? What hardware platforms would be supported? What about gui
support?

KDB is console debugger, i.e., it doesn't support a remote serial protocol so
I don't understand how source level debug would work. Would you have to boot
an unstripped kernel executable whenever you wanted to debug?

KGDB is better because it's a source level debugger and because you get a
graphical interface, Data Display Debugger (DDD), for free when you use gdb.

KGDB is better because it's more platform agnostic than KDB. The only
architecture dependent part in the kernel is the gdb stub which is small and
well defined.

> kgdb relies on gdb so you loose the knowledge of kernel
internals (no, > I am *not* going to teach gdb about kernel stacks, out of line
lock > code etc.).  kgdb has more of a dependency on a working kernel.  It
> provides source level debugging, although stack backtrace tends not to
> work unless you compile the kernel with frame pointers.
> 

I don't know what you mean when you say "loose knowledge of kernel internals"
with KGDB ... I'd rather develop a set of standard gdb user defined commands to
expose kernel internals than hard code them in the linux kernel.

As for stack frame analysis, both debuggers have problems with backtrace if the
kernel is compiled without CONFIG_FRAME_POINTER option. Whether the problem is
fixed in gdb for KGDB users or the kernel for KDB users is immaterial in
choosing one debugger over the other.

-- 
Paul J Albrecht
pjalbrecht@home.com

Re: Linux Kernel Debuggers, KDB or KGDB?

[Keith Owens]

On Wed, 2 May 2001 16:06:15 -0500, 
Paul J Albrecht <pjalbrecht@home.com> wrote:
>I'd like to know more about your plans to enhance KDB with source level debug
>capability.

Use a combination of gdb and kdb.  kdb to support kernel internals, gdb
to take the kdb output and add source level data.  It needs two
machines, one that is running to support gdb, the second machine is
being debugged, with a serial console between them.

The problem will be stopping gdb from making assumptions about the
machine being debugged.  Instead of changing gdb code, use a gdb
wrapper program to intercept user commands and gdb serial protocol and
convert them to kdb commands.

>Would you have to boot an unstripped kernel executable whenever you
>wanted to debug?

Boot, no.  But the machine running gdb will need an copy of the
unstripped vmlinux and module objects to get the debug information.

Re: Linux Kernel Debuggers, KDB or KGDB?

[Amit S. Kale]

Keith Owens wrote:
> 
> On Wed, 2 May 2001 16:06:15 -0500,
> Paul J Albrecht <pjalbrecht@home.com> wrote:
> >I'd like to know more about your plans to enhance KDB with source level debug
> >capability.
> 
> Use a combination of gdb and kdb.  kdb to support kernel internals, gdb
> to take the kdb output and add source level data.  It needs two
> machines, one that is running to support gdb, the second machine is
> being debugged, with a serial console between them.

This is how solaris ksld (kernel source level debugger
works). solaris dbx connects to a kernel stub which
serves as kadb (solaris assembly level debugger) as
well as a debugger stub.

> The problem will be stopping gdb from making assumptions about the
> machine being debugged.  Instead of changing gdb code, use a gdb
> wrapper program to intercept user commands and gdb serial protocol and
> convert them to kdb commands.

I am not sure if kdb provides access to kernel threads
in a form that can be conveniently by a wrapper
program.

Interested people can check whether all features of kgdb
(http://kgdb.sourceforge.net/) are available in convenient
form in kdb.

The logic of holding slave cpus (the cpu in the debugger
is the master while others are slaves) is different in kdb
and kgdb. Handling of nmi-watchdog too is different.

> 
> >Would you have to boot an unstripped kernel executable whenever you
> >wanted to debug?
> 
> Boot, no.  But the machine running gdb will need an copy of the
> unstripped vmlinux and module objects to get the debug information.

Plus all the sources.

All this is required by gdb. In theory gdb could do well
with just source code and symbol information only, though
for lack of a symbol information requesting packet in
the gdb remote protocol, gdb can't get symbol information
directly from the stub.

There is some effort in gdb world to allow a stub to query
gdb for symbol information. Probably the reverse can also
be added. Need someone good understanding of gdb to do
that.

It's because of these reasons that kgdb module debugging setup
is combersome.

> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Amit Kale
Veritas Software ( http://www.veritas.com )