[PATCH v4 00/13] Add gdb python scripts as kernel debugging helpers

[PATCH v4 00/13] Add gdb python scripts as kernel debugging helpers

[Jan Kiszka]

Version 4 of this series is a rebase over latest 3.8-rc4+. Moreover, I
updated the mechanism that implements automatic symbol loading for new
modules. It was affected by the refactorings around finit_module.

While waiting for feedback who could imagine picking this up for merge,
I wrote a tiny tutorial, see below.


Here is the original series intro again:

This adds the infrastructure and first tools that make kernel debugging
through gdb more comfortable. Since 7.0, gdb supports python scripting.
And this opens the doors to automate steps like the tedious loading of
module symbols at the right address, resolving per-cpu variables or even
retrieving the current kernel log without resuming an stopped target.

Many of the helpers naturally depend on the layout of structures or
internal mechanics of the kernel. So the best place to maintain such
things, keeping them consistent with the corresponding kernel is, well,
the kernel itself.

While these scripts have been originally developed for debugging via
QEMU/KVM, I've now also added the required bits for KGDB. Works fine,
but as QEMU/KVM tends to outperform KGDB it remains the recommendation
- when available.

There are two architecture dependencies so far, one regarding per-cpu,
the other regarding thread_info calculation. None of them I was able to
test on a target, so I'm counting on review/testing by the corresponding
communities.

This series should be considered the foundation of much more kernel
state exploration helpers, e.g. around tasks, timers, locks, sockets -
I guess people will have even more ideas.


And this is a tutorial for the gdb extension using QEMU/KVM as target
platform:

 o Set up a virtual Linux machine for KVM (see www.linux-kvm.org and
   www.qemu.org for more details)

 o Build the kernel with this series applied, enabling CONFIG_DEBUG_INFO
   (but leave CONFIG_DEBUG_INFO_REDUCED off)

 o Install that kernel on the guest

 o Enable the gdb stub of QEMU/KVM, either
    - at VM startup time by appending "-s" to the QEMU command line
   or
    - during runtime by issuing "gdbserver" from the QEMU monitor
      console

 o cd /path/to/linux-build

 o Start gdb: gdb vmlinux

 o Attach to the booted guest:
    (gdb) target remote :1234

 o Load module (and main kernel) symbols:
    (gdb) lx-symbols
    loading vmlinux
    scanning for modules in /home/user/linux/build
    loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
    loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
    loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
    loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
    loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
    ...
    loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko

 o Set a breakpoint on some not yet loaded module function, e.g.:
    (gdb) b btrfs_init_sysfs
    Function "btrfs_init_sysfs" not defined.
    Make breakpoint pending on future shared library load? (y or [n]) y
    Breakpoint 1 (btrfs_init_sysfs) pending.

 o Continue the target

 o Load the module on the target and watch what happens:
    loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
    loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
    loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
    loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko

    Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
    36              btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);

 o Let's examine the current task a bit:
    (gdb) p ().pid
     = 4998
    (gdb) p ().comm
     = "modprobe\000\000\000\000\000\000\000"

 o Dump the log buffer of target kernel:
    (gdb) lx-dmesg
    [     0.000000] Initializing cgroup subsys cpuset
    [     0.000000] Initializing cgroup subsys cpu
    [     0.000000] Linux version 3.8.0-rc4-dbg+ (...
    [     0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
    [     0.000000] e820: BIOS-provided physical RAM map:
    [     0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
    [     0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
    ....

 o Make use of the per-cpu helper for the current or a specified CPU:
    (gdb) p ("runqueues").nr_running
     = 1
    (gdb) p ("runqueues", 2).nr_running
     = 0

 o And now we are digging deep into hrtimers using the container_of
   helper:
    (gdb) set  = ("hrtimer_bases").clock_base[0].active.next
    (gdb) p *(, "struct hrtimer", "node")
     = {
      node = {
        node = {
          __rb_parent_color = 18446612133355256072,
          rb_right = 0x0 <irq_stack_union>,
          rb_left = 0x0 <irq_stack_union>
        },
        expires = {
          tv64 = 1835268000000
        }
      },
      _softexpires = {
        tv64 = 1835268000000
      },
      function = 0xffffffff81078232 <tick_sched_timer>,
      base = 0xffff88003fd0d6f0,
      state = 1,
      start_pid = 0,
      start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
      start_comm = "swapper/2\000\000\000\000\000\000"
    }

Hope this provided some ideas and inspirations on how the commands and
helper functions can support kernel development.

Enjoy,
Jan

PS: Also available via git://git.kiszka.org/linux.git queues/gdb-scripts

CC: "David S. Miller" <davem@davemloft.net>
CC: Fenghua Yu <fenghua.yu@intel.com>
CC: Kay Sievers <kay@vrfy.org>
CC: linux-ia64@vger.kernel.org
CC: linux-kbuild@vger.kernel.org
CC: Michal Marek <mmarek@suse.cz>
CC: sparclinux@vger.kernel.org
CC: Tony Luck <tony.luck@intel.com>

Jan Kiszka (13):
  scripts/gdb: Add infrastructure
  scripts/gdb: Add container_of helper and convenience function
  scripts/gdb: Add lx-symbols command
  scripts/gdb: Add get_target_endianness helper
  scripts/gdb: Add read_u16/32/64 helpers
  scripts/gdb: Add lx-dmesg command
  scripts/gdb: Add task iteration helper
  scripts/gdb: Add helper and convenience function to look up tasks
  scripts/gdb: Add is_target_arch helper
  scripts/gdb: Add internal helper and convenience function to retrieve
    thread_info
  scripts/gdb: Add get_gdbserver_type helper
  scripts/gdb: Add internal helper and convenience function for per-cpu
    lookup
  scripts/gdb: Add lx_current convenience function

 Makefile                   |    5 +-
 scripts/Makefile           |    3 +-
 scripts/gdb/Makefile       |    9 +++
 scripts/gdb/dmesg.py       |   63 ++++++++++++++++++
 scripts/gdb/percpu.py      |   76 ++++++++++++++++++++++
 scripts/gdb/symbols.py     |  153 ++++++++++++++++++++++++++++++++++++++++++++
 scripts/gdb/task.py        |  108 +++++++++++++++++++++++++++++++
 scripts/gdb/utils.py       |  137 +++++++++++++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |   28 ++++++++
 9 files changed, 580 insertions(+), 2 deletions(-)
 create mode 100644 scripts/gdb/Makefile
 create mode 100644 scripts/gdb/dmesg.py
 create mode 100644 scripts/gdb/percpu.py
 create mode 100644 scripts/gdb/symbols.py
 create mode 100644 scripts/gdb/task.py
 create mode 100644 scripts/gdb/utils.py
 create mode 100644 scripts/gdb/vmlinux-gdb.py

-- 
1.7.3.4

[PATCH v4 10/13] scripts/gdb: Add internal helper and convenience function to retrieve thread_info

[Jan Kiszka]

Add the internal helper get_thread_info that calculated the thread_info
from a given task variable. Also export this service as a convenience
function.

Note: ia64 version is untested.

CC: Tony Luck <tony.luck@intel.com>
CC: Fenghua Yu <fenghua.yu@intel.com>
CC: linux-ia64@vger.kernel.org
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/task.py |   39 +++++++++++++++++++++++++++++++++++++++
 1 files changed, 39 insertions(+), 0 deletions(-)

diff --git a/scripts/gdb/task.py b/scripts/gdb/task.py
index 93e6c0c..0a45b3c 100644
--- a/scripts/gdb/task.py
+++ b/scripts/gdb/task.py
@@ -67,3 +67,42 @@ class LxTaskByPidFunc(gdb.Function):
 			raise gdb.GdbError("No task of PID " + str(pid))
 
 LxTaskByPidFunc()
+
+
+thread_info_ptr_type = None
+
+def get_thread_info_ptr_type():
+	global thread_info_ptr_type
+	if thread_info_ptr_type = None:
+		thread_info_ptr_type = get_type('struct thread_info').pointer()
+	return thread_info_ptr_type
+
+ia64_task_size = None
+
+def get_thread_info(task):
+	if is_target_arch("ia64"):
+		global ia64_task_size
+		if ia64_task_size = None:
+			ia64_task_size = gdb.parse_and_eval(
+						"sizeof(struct task_struct)")
+		thread_info_addr = task.address + ia64_task_size
+		thread_info = thread_info_addr.cast(get_thread_info_ptr_type())
+	else:
+		thread_info = task['stack'].cast(get_thread_info_ptr_type())
+	return thread_info.dereference()
+
+
+class LxThreadInfoFunc (gdb.Function):
+	# Calculate Linux thread_info from task variable.
+	__doc__ = "Calculate Linux thread_info from task variable.\n" \
+		  "\n" \
+		  "$lx_thread_info(TASK): Given TASK, return the corresponding thread_info\n" \
+		  "variable.\n"
+
+	def __init__(self):
+		super(LxThreadInfoFunc, self).__init__("lx_thread_info")
+
+	def invoke(self, task):
+		return get_thread_info(task)
+
+LxThreadInfoFunc()
-- 
1.7.3.4

Re: [PATCH v4 00/13] Add gdb python scripts as kernel debugging helpers

[Andi Kleen]

> 
> And this is a tutorial for the gdb extension using QEMU/KVM as target
> platform:

Can you add the tutorial as a file in Documentation? 
Other than that everything looks good to me.

-Andi

Re: [PATCH v4 00/13] Add gdb python scripts as kernel debugging helpers

[Ben Widawsky]

On Mon, 21 Jan 2013 18:06:07 +0100
Jan Kiszka <jan.kiszka@siemens.com> wrote:

> Version 4 of this series is a rebase over latest 3.8-rc4+. Moreover, I
> updated the mechanism that implements automatic symbol loading for new
> modules. It was affected by the refactorings around finit_module.
> 
> While waiting for feedback who could imagine picking this up for merge,
> I wrote a tiny tutorial, see below.
> 
> 
> Here is the original series intro again:
> 
> This adds the infrastructure and first tools that make kernel debugging
> through gdb more comfortable. Since 7.0, gdb supports python scripting.
> And this opens the doors to automate steps like the tedious loading of
> module symbols at the right address, resolving per-cpu variables or even
> retrieving the current kernel log without resuming an stopped target.
> 
> Many of the helpers naturally depend on the layout of structures or
> internal mechanics of the kernel. So the best place to maintain such
> things, keeping them consistent with the corresponding kernel is, well,
> the kernel itself.
> 
> While these scripts have been originally developed for debugging via
> QEMU/KVM, I've now also added the required bits for KGDB. Works fine,
> but as QEMU/KVM tends to outperform KGDB it remains the recommendation
> - when available.
> 
> There are two architecture dependencies so far, one regarding per-cpu,
> the other regarding thread_info calculation. None of them I was able to
> test on a target, so I'm counting on review/testing by the corresponding
> communities.
> 
> This series should be considered the foundation of much more kernel
> state exploration helpers, e.g. around tasks, timers, locks, sockets -
> I guess people will have even more ideas.
> 
> 
> And this is a tutorial for the gdb extension using QEMU/KVM as target
> platform:
> 
>  o Set up a virtual Linux machine for KVM (see www.linux-kvm.org and
>    www.qemu.org for more details)
> 
>  o Build the kernel with this series applied, enabling CONFIG_DEBUG_INFO
>    (but leave CONFIG_DEBUG_INFO_REDUCED off)
> 
>  o Install that kernel on the guest
> 
>  o Enable the gdb stub of QEMU/KVM, either
>     - at VM startup time by appending "-s" to the QEMU command line
>    or
>     - during runtime by issuing "gdbserver" from the QEMU monitor
>       console
> 
>  o cd /path/to/linux-build
> 
>  o Start gdb: gdb vmlinux
> 
>  o Attach to the booted guest:
>     (gdb) target remote :1234
> 
>  o Load module (and main kernel) symbols:
>     (gdb) lx-symbols
>     loading vmlinux
>     scanning for modules in /home/user/linux/build
>     loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
>     loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
>     loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
>     loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
>     loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
>     ...
>     loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
> 
>  o Set a breakpoint on some not yet loaded module function, e.g.:
>     (gdb) b btrfs_init_sysfs
>     Function "btrfs_init_sysfs" not defined.
>     Make breakpoint pending on future shared library load? (y or [n]) y
>     Breakpoint 1 (btrfs_init_sysfs) pending.
> 
>  o Continue the target
> 
>  o Load the module on the target and watch what happens:
>     loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
>     loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
>     loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
>     loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
> 
>     Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
>     36              btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
> 
>  o Let's examine the current task a bit:
>     (gdb) p ().pid
>      = 4998
>     (gdb) p ().comm
>      = "modprobe\000\000\000\000\000\000\000"
> 
>  o Dump the log buffer of target kernel:
>     (gdb) lx-dmesg
>     [     0.000000] Initializing cgroup subsys cpuset
>     [     0.000000] Initializing cgroup subsys cpu
>     [     0.000000] Linux version 3.8.0-rc4-dbg+ (...
>     [     0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
>     [     0.000000] e820: BIOS-provided physical RAM map:
>     [     0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
>     [     0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
>     ....
> 
>  o Make use of the per-cpu helper for the current or a specified CPU:
>     (gdb) p ("runqueues").nr_running
>      = 1
>     (gdb) p ("runqueues", 2).nr_running
>      = 0
> 
>  o And now we are digging deep into hrtimers using the container_of
>    helper:
>     (gdb) set  = ("hrtimer_bases").clock_base[0].active.next
>     (gdb) p *(, "struct hrtimer", "node")
>      = {
>       node = {
>         node = {
>           __rb_parent_color = 18446612133355256072,
>           rb_right = 0x0 <irq_stack_union>,
>           rb_left = 0x0 <irq_stack_union>
>         },
>         expires = {
>           tv64 = 1835268000000
>         }
>       },
>       _softexpires = {
>         tv64 = 1835268000000
>       },
>       function = 0xffffffff81078232 <tick_sched_timer>,
>       base = 0xffff88003fd0d6f0,
>       state = 1,
>       start_pid = 0,
>       start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
>       start_comm = "swapper/2\000\000\000\000\000\000"
>     }
> 
> Hope this provided some ideas and inspirations on how the commands and
> helper functions can support kernel development.
> 
> Enjoy,
> Jan
> 
> PS: Also available via git://git.kiszka.org/linux.git queues/gdb-scripts
> 
> CC: "David S. Miller" <davem@davemloft.net>
> CC: Fenghua Yu <fenghua.yu@intel.com>
> CC: Kay Sievers <kay@vrfy.org>
> CC: linux-ia64@vger.kernel.org
> CC: linux-kbuild@vger.kernel.org
> CC: Michal Marek <mmarek@suse.cz>
> CC: sparclinux@vger.kernel.org
> CC: Tony Luck <tony.luck@intel.com>

My less than useful from v3 still applies:
Acked-by: Ben Widawsky <ben@bwidawsk.net>

I'm not really an appropriate person to review, but I've mde heavy use
of lx-symbols and lx-dmesg and am very happy.

-- 
Ben Widawsky, Intel Open Source Technology Center

Re: [PATCH v4 00/13] Add gdb python scripts as kernel debugging helpers

[Andi Kleen]

> 
>  o Install that kernel on the guest

If you use a static kernel you can also do 

- copy the initrd out of the guest once

qemu...  -bzImage kernel -initrd initrd

This saves the step of getting the kernel into the kernel.
Doesn't work with modules unfortunately.

-Andi

Re: [PATCH v4 00/13] Add gdb python scripts as kernel debugging helpers

[Jan Kiszka]

On 2013-01-21 22:21, Andi Kleen wrote:
>>
>> And this is a tutorial for the gdb extension using QEMU/KVM as target
>> platform:
> 
> Can you add the tutorial as a file in Documentation? 

Sure, will do.

> Other than that everything looks good to me.

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v4 00/13] Add gdb python scripts as kernel debugging helpers

[Jan Kiszka]

On 2013-01-21 23:15, Andi Kleen wrote:
>>
>>  o Install that kernel on the guest
> 
> If you use a static kernel you can also do 
> 
> - copy the initrd out of the guest once
> 
> qemu...  -bzImage kernel -initrd initrd

-kernel/append/initrd, of course.

> 
> This saves the step of getting the kernel into the kernel.
> Doesn't work with modules unfortunately.

True. An alternative can be nfsroot. Or root on virtfs.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v4 00/13] Add gdb python scripts as kernel debugging helpers

[Borislav Petkov]

Ok, first of all, I gotta say, this is very cool stuff, thanks for doing
this. I'm playing with your tutorial and I'm having some trouble, see
below:

On Mon, Jan 21, 2013 at 06:06:07PM +0100, Jan Kiszka wrote:
>  o Let's examine the current task a bit:
>     (gdb) p ().pid

when I do that, it says:

(gdb) p ().pid
A syntax error in expression, near `).pid'

Here's what I do:

$ gdb --data-directory $(pwd)/scripts/gdb/ ./vmlinux
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /w/kernel/linux-2.6/vmlinux...done.
(gdb) target remote :1234
Remote debugging using :1234
[New Thread 2]
[Switching to Thread 2]
default_idle () at arch/x86/kernel/process.c:391
391             current_thread_info()->status |= TS_POLLING;
(gdb) p ().pid
A syntax error in expression, near `).pid'.
(gdb) show data-directory
GDB's data directory is "/w/kernel/linux-2.6/scripts/gdb/".
(gdb) lx-symbols
loading vmlinux
no modules found
(gdb) p ().pid
A syntax error in expression, near `).pid'.
(gdb)

So, I thought I should point it explicitly to the python scripts with
--data-directory but it still doesn't fly. And gdb is 7.4.1 so it should
be recent enough.

lx-dmesg works, for example:

(gdb) lx-dmesg
[     0.000000] Linux version 3.8.0-rc4+ (boris@pd) (gcc version 4.7.2 (Debian 4.7.2-5) ) #2 SMP PREEMPT Wed Jan 23 11:59:58 CET 2013
[     0.000000] Command line: vga=0 root=/dev/sda1 debug ignore_loglevel console=ttyS0,115200 console=tty0
[     0.000000] e820: BIOS-provided physical RAM map:
[     0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009f3ff] usable
[     0.000000] BIOS-e820: [mem 0x000000000009f400-0x000000000009ffff] reserved
[     0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[     0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffdfff] usable
...

So what am I missing?

Thanks.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH v4 00/13] Add gdb python scripts as kernel debugging helpers

[Jan Kiszka]

On 2013-01-23 12:32, Borislav Petkov wrote:
> Ok, first of all, I gotta say, this is very cool stuff, thanks for doing
> this. I'm playing with your tutorial and I'm having some trouble, see
> below:
> 
> On Mon, Jan 21, 2013 at 06:06:07PM +0100, Jan Kiszka wrote:
>>  o Let's examine the current task a bit:
>>     (gdb) p ().pid
> 
> when I do that, it says:
> 
> (gdb) p ().pid
> A syntax error in expression, near `).pid'

Ouch, my git-send script ate all the "$foo" in the cover letter :(. It
has to be $lx_current, $lx_per_cpu and $container_of. Will write an
add-on patch that adds the tutorial as a text file later on.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux