[PATCH 0/1] crosstap: a systemtap usability script

[PATCH 0/1] crosstap: a systemtap usability script

[tom.zanussi]

From: Tom Zanussi <tom.zanussi@intel.com>

This patch implements a script called 'crosstap', which essentially
encapsulates and automates the tedious steps until now required to
use systemtap on a remote embedded target, outlined here:

https://wiki.yoctoproject.org/wiki/Tracing_and_Profiling

There's actually a Yocto bug tracking this work, see for a little
more background:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=1551

I've tested the below scripts on a couple qemu targets (qemuarm and
qemux86) as well as on real hardware (emenlow).  I'll be doing a lot
more testing of this script in the coming weeks as I address another
enhancement (tracing and profiling HOWTOs):

https://bugzilla.yoctoproject.org/show_bug.cgi?id=1640

The examples tested were:

1) The simplest possible probe, which just prints 'hello world':

probe begin
{
	print ("hello world\n")
	exit ()
}

[trz@empanada tutorial]$ crosstap root@192.168.7.10 helloworld.stp
~/tracing/systemtap/scripts/tutorial ~/tracing/systemtap/scripts/tutorial
~/tracing/systemtap/scripts/tutorial
root@192.168.7.10's password: 
hello world

2) The next simplest possible probe, which prints 'hello world', but also
demonstrates that any number of arguements can be passed to the systemtap
script:

probe begin
{
	printf("hello world with cmdline args, numeric arg: %d, string arg: %s\n", $1, @2)
	exit ()
}

[trz@empanada tutorial]$ crosstap root@192.168.7.10 helloworld_args.stp 99 ninetynine -v
~/tracing/systemtap/scripts/tutorial ~/tracing/systemtap/scripts/tutorial
~/tracing/systemtap/scripts/tutorial
root@192.168.7.10's password: 
Pass 1: parsed user script and 83 library script(s) using 150500virt/22236res/2148shr/20612data kb, in 80usr/0sys/81real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 150896virt/22760res/2224shr/21008data kb, in 0usr/0sys/2real ms.
Pass 3: translated to C into "/tmp/stapJsYotR/helloworld_args_src.c" using 151020virt/23208res/2616shr/21132data kb, in 0usr/0sys/0real ms.
Pass 4: compiled C into "helloworld_args.ko" in 1030usr/450sys/1568real ms.
Pass 5: starting run.
hello world with cmdline args, numeric arg: 99, string arg: ninetynine
Pass 5: run completed in 10usr/0sys/426real ms.

3) A more interesting, marginally useful test, instrumenting and printing
out information about every 'open' syscall that happens on the system:

probe syscall.open
{
	printf ("%s(%d) open (%s)\n", execname(), pid(), argstr)
}

probe timer.ms(9000) # after 9 seconds
{
	exit ()
}

On the target system:
# cat /etc/*

[trz@empanada tutorial]$ crosstap root@192.168.7.10 trace_open.stp
~/tracing/systemtap/scripts/tutorial ~/tracing/systemtap/scripts/tutorial
~/tracing/systemtap/scripts/tutorial
root@192.168.7.10's password: 
syslogd(811) open ("/var/log/messages", O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE|O_NOCTTY|O_NONBLOCK, 0666)
cat(963) open ("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC|O_CLOEXEC)
cat(963) open ("/lib/libc.so.6", O_RDONLY|O_CLOEXEC|O_CLOEXEC)
cat(963) open ("X11", O_RDONLY|O_LARGEFILE)
cat(963) open ("apm", O_RDONLY|O_LARGEFILE)
cat(963) open ("asound.conf", O_RDONLY|O_LARGEFILE)
.
.
.
cat(963) open ("init.d", O_RDONLY|O_LARGEFILE)
cat(963) open ("inittab", O_RDONLY|O_LARGEFILE)
cat(963) open ("inputrc", O_RDONLY|O_LARGEFILE)
cat(963) open ("issue", O_RDONLY|O_LARGEFILE)
cat(963) open ("issue.net", O_RDONLY|O_LARGEFILE)
cat(963) open ("ld.so.cache", O_RDONLY|O_LARGEFILE)
matchbox-termin(891) open ("/tmp/vteNQTCKW", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600)
matchbox-termin(891) open ("/tmp/vteQOVCKW", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600)
matchbox-termin(891) open ("/tmp/vte0QKDKW", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600)
matchbox-termin(891) open ("/tmp/vteUBAFKW", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600)
cat(963) open ("ld.so.conf", O_RDONLY|O_LARGEFILE)
cat(963) open ("libnl", O_RDONLY|O_LARGEFILE)
cat(963) open ("limits", O_RDONLY|O_LARGEFILE)
cat(963) open ("login.access", O_RDONLY|O_LARGEFILE)
cat(963) open ("login.defs", O_RDONLY|O_LARGEFILE)
cat(963) open ("logrotate.d", O_RDONLY|O_LARGEFILE)
cat(963) open ("matchbox", O_RDONLY|O_LARGEFILE)
.
.
.


The following changes since commit 9ba1e33e2d14362971d6441ee6142bcb0857df1a:

  sstate: Ensure master.list exists if it doesn't already (2012-08-30 22:45:56 -0700)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib.git tzanussi/crosstap.v1
  http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=tzanussi/crosstap.v1

Tom Zanussi (1):
  crosstap: new script

 scripts/crosstap | 148 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 148 insertions(+)
 create mode 100755 scripts/crosstap

-- 
1.7.11.4

[PATCH 1/1] crosstap: new script

[tom.zanussi]

From: Tom Zanussi <tom.zanussi@intel.com>

This script makes it easy to run a systemtap script on a remote target
from a build host.

The script itself contains documentation on basic setup and usage -
see that for details.  In a nutshell, if you have a systemtap script
on the build host, this script allows that systemtap script to be
compiled and run on the target host and for the output appear on the
host terminal.

The crosstap script requires to an sdk build of the target (or a build
with 'tools-profile' added to EXTRA_IMAGE_FEATURES) because it needs
to be able to run the native systemtap built by the systemtap recipe,
which in turn needs access to the kernel built for the target in order
to build the kernel modules that implement the systemtap probe
specified by the systemtap script and which ultimately get shipped to
the target and insmod'ed there.

The crosstap script also needs to be able to ssh to the target host in
order insert the modules on the target, so the target system needs to
support incoming ssh connections.

Signed-off-by: Tom Zanussi <tom.zanussi@intel.com>
---
 scripts/crosstap | 148 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 148 insertions(+)
 create mode 100755 scripts/crosstap

diff --git a/scripts/crosstap b/scripts/crosstap
new file mode 100755
index 0000000..783bf2f
--- /dev/null
+++ b/scripts/crosstap
@@ -0,0 +1,148 @@
+#!/bin/bash
+#
+# Run a systemtap script on remote target
+#
+# Examples (run on build host, target is 192.168.1.xxx):
+#   $ source oe-init-build-env"
+#   $ cd ~/my/systemtap/scripts"
+#
+#   $ crosstap root@192.168.1.xxx myscript.stp"
+#   $ crosstap root@192.168.1.xxx myscript-with-args.stp 99 ninetynine"
+#
+# Copyright (c) 2012, Intel Corporation.
+# All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+function usage() {
+    echo "Usage: $0 <user@hostname> <sytemtap-script> [additional systemtap-script args]"
+}
+
+function setup_usage() {
+    echo ""
+    echo "'crosstap' requires a local sdk build of the target system"
+    echo "(or a build that includes 'tools-profile') in order to build"
+    echo "kernel modules that can probe the target system."
+    echo ""
+    echo "Practically speaking, that means you need to do the following:"
+    echo "  - If you're running a pre-built image, download the release"
+    echo "    and/or BSP tarballs used to build the image."
+    echo "  - If you're working from git sources, just clone the metadata"
+    echo "    and BSP layers needed to build the image you'll be booting."
+    echo "  - Make sure you're properly set up to build a new image (see"
+    echo "    the BSP README and/or the widely available basic documentation"
+    echo "    that discusses how to build images)."
+    echo "  - Build an -sdk version of the image e.g.:"
+    echo "      $ bitbake core-image-sato-sdk"
+    echo "  OR"
+    echo "  - Build a non-sdk image but include the profiling tools:"
+    echo "      [ edit local.conf and add 'tools-profile' to the end of"
+    echo "        the EXTRA_IMAGE_FEATURES variable ]"
+    echo "      $ bitbake core-image-sato"
+    echo ""
+    echo "  [ NOTE that 'crosstap' needs to be able to ssh into the target"
+    echo "    system, which isn't enabled by default in -minimal images. ]"
+    echo ""
+    echo "Once you've build the image on the host system, you're ready to"
+    echo "boot it (or the equivalent pre-built image) and use 'crosstap'"
+    echo "to probe it (you need to source the environment as usual first):"
+    echo ""
+    echo "    $ source oe-init-build-env"
+    echo "    $ cd ~/my/systemtap/scripts"
+    echo "    $ crosstap root@192.168.1.xxx myscript.stp"
+    echo ""
+}
+
+function systemtap_target_arch() {
+    SYSTEMTAP_TARGET_ARCH=$1
+    case $SYSTEMTAP_TARGET_ARCH in
+        i?86)
+            SYSTEMTAP_TARGET_ARCH="i386"
+            ;;
+        x86_64*)
+            SYSTEMTAP_TARGET_ARCH="x86_64"
+            ;;
+        arm*)
+            SYSTEMTAP_TARGET_ARCH="arm"
+            ;;
+        powerpc*)
+            SYSTEMTAP_TARGET_ARCH="powerpc"
+            ;;
+        *)
+            ;;
+    esac
+}
+
+if [ $# -lt 2 ]; then
+	usage
+	exit 1
+fi
+
+if [ -z "$BUILDDIR" ]; then
+    echo "Error: Unable to find the BUILDDIR environment variable."
+    echo "Did you forget to source your build system environment setup script?"
+    exit 1
+fi
+
+pushd $PWD
+cd $BUILDDIR
+BITBAKE_VARS=`bitbake -e virtual/kernel`
+popd
+
+STAGING_BINDIR_TOOLCHAIN=$(echo "$BITBAKE_VARS" | grep ^STAGING_BINDIR_TOOLCHAIN \
+  | cut -d '=' -f2 | cut -d '"' -f2)
+STAGING_BINDIR_TOOLPREFIX=$(echo "$BITBAKE_VARS" | grep ^TARGET_PREFIX \
+  | cut -d '=' -f2 | cut -d '"' -f2)
+SYSTEMTAP_HOST_INSTALLDIR=$(echo "$BITBAKE_VARS" | grep ^STAGING_DIR_NATIVE \
+  | cut -d '=' -f2 | cut -d '"' -f2)
+TARGET_ARCH=$(echo "$BITBAKE_VARS" | grep ^TRANSLATED_TARGET_ARCH \
+  | cut -d '=' -f2 | cut -d '"' -f2)
+TARGET_KERNEL_BUILDDIR=$(echo "$BITBAKE_VARS" | grep ^B= \
+  | cut -d '=' -f2 | cut -d '"' -f2)
+
+systemtap_target_arch "$TARGET_ARCH"
+
+if [ ! -d $TARGET_KERNEL_BUILDDIR ] ||
+   [ ! -f $TARGET_KERNEL_BUILDDIR/vmlinux ]; then
+    echo -e "\nError: No target kernel build found."
+    echo -e "Did you forget to create a local build of your image?"
+    setup_usage
+    exit 1
+fi
+
+if [ ! -f $SYSTEMTAP_HOST_INSTALLDIR/usr/bin/stap ]; then
+    echo -e "\nError: Native (host) systemtap not found."
+    echo -e "Did you accidentally build a local non-sdk image? (or forget to"
+    echo -e "add 'tools-profile' to EXTRA_IMAGE_FEATURES in your local.conf)?"
+    setup_usage
+    exit 1
+fi
+
+target_user_hostname="$1"
+full_script_name="$2"
+script_name=$(basename "$2")
+script_base=${script_name%.*}
+shift 2
+
+${SYSTEMTAP_HOST_INSTALLDIR}/usr/bin/stap \
+  -a ${SYSTEMTAP_TARGET_ARCH} \
+  -B CROSS_COMPILE="${STAGING_BINDIR_TOOLCHAIN}/${STAGING_BINDIR_TOOLPREFIX}" \
+  -r ${TARGET_KERNEL_BUILDDIR} \
+  -I ${SYSTEMTAP_HOST_INSTALLDIR}/usr/share/systemtap/tapset \
+  -R ${SYSTEMTAP_HOST_INSTALLDIR}/usr/share/systemtap/runtime \
+  --remote=$target_user_hostname \
+  -m $script_base \
+   $full_script_name "$@"
+
+exit 0
-- 
1.7.11.4