ubuntu 11.10, systemtap and building my own kernel

ubuntu 11.10, systemtap and building my own kernel

[Robert P. J. Day]

  (sort of a combo of ubuntu, systemtap and kernel question, hoping
it's not too far off-topic.)

  for the purpose of an upcoming tutorial, i decided to install
systemtap on my fully-updated 64-bit ubuntu 11.10 laptop, then build a
new kernel and show how simple it is to use systemtap.  it didn't turn
out well.

  since i would be using a self-built kernel, i started here:

  http://sourceware.org/systemtap/wiki/SystemTapWithSelfBuiltKernel

but i'm not convinced that page is entirely correct.

  while i made sure all of the listed kernel options were selected, my
first concern is that i'm not sure why i should have to do "make
headers_install".  all that does is populate the in-tree usr/include/
directory with the exported header files.  i'm not sure why that's
necessary.  is it really required?  does systemtap actually go looking
there for header files?

  also, the rest of that page seems to explain how to make sure
systemtap can find your kernel build tree.  but if you build and
install, then /lib/modules/<release>/build is a symlink to the build
tree, so i'm not sure anything else has to be done.  the rest of that
page also seems unnecessary.  thoughts?

  in any event, i configured, built and installed my new kernel (based
on latest "git pull" from development kernel source), and rebooted to
get:

$ uname -r
3.4.0-rc2+
$

which is correct so, theoretically, i should be read to try a simple
systemtap example.  reading here:

  http://www.domaigne.com/blog/random/running-systemtap-on-ubuntu/

i tried:

$ stap -e 'probe kernel.function("sys_open") {log("hello world") exit()}'

and got:

===== start =====

In file included from /usr/share/systemtap/runtime/transport/transport.c:53:0,
                 from /usr/share/systemtap/runtime/print.c:18,
                 from /usr/share/systemtap/runtime/runtime.h:128,
                 from /tmp/stapadEYdA/stap_59bddd394f89d5f186da7e250db36434_876.c:55:
/usr/share/systemtap/runtime/transport/relay_v2.c:242:2: error: initialization from incompatible pointer type [-Werror]
/usr/share/systemtap/runtime/transport/relay_v2.c:242:2: error: (near initialization for ?__stp_relay_callbacks.create_buf_file?) [-Werror]
In file included from /usr/share/systemtap/runtime/print.c:18:0,
                 from /usr/share/systemtap/runtime/runtime.h:128,
                 from /tmp/stapadEYdA/stap_59bddd394f89d5f186da7e250db36434_876.c:55:
/usr/share/systemtap/runtime/transport/transport.c: In function ?_stp_get_root_dir?:
/usr/share/systemtap/runtime/transport/transport.c:380:8: error: ?struct hlist_head? has no member named ?next?
In file included from /tmp/stapadEYdA/stap_59bddd394f89d5f186da7e250db36434_876.c:57:0:
/usr/share/systemtap/runtime/stat.c: In function ?_stp_stat_get?:
/usr/share/systemtap/runtime/stat.c:213:2: error: ?cpu_possible_map? undeclared (first use in this function)
/usr/share/systemtap/runtime/stat.c:213:2: note: each undeclared identifier is reported only once for each function it appears in
/usr/share/systemtap/runtime/stat.c: In function ?_stp_stat_clear?:
/usr/share/systemtap/runtime/stat.c:248:2: error: ?cpu_possible_map? undeclared (first use in this function)
cc1: all warnings being treated as errors

make[1]: *** [/tmp/stapadEYdA/stap_59bddd394f89d5f186da7e250db36434_876.o] Error 1
make: *** [_module_/tmp/stapadEYdA] Error 2
Pass 4: compilation failed.  Try again with another '--vp 0001' option.

===== end =====

  that doesn't look good.  in particular, this error just looks
strange:

/usr/share/systemtap/runtime/transport/transport.c:380:8: error: ?struct hlist_head? has no member named ?next?

  to make a long story short, that diagnostic is entirely correct --
"struct hlist_head" as it's defined in <types.h> is:

struct hlist_head {
        struct hlist_node *first;
};

but that systemtap source file is trying to refer to a member called
"next", which clearly does not exist.

  am i doing something silly?  can someone else with an ubuntu system
try this and let me know what happens?  thanks.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

ubuntu 11.10, systemtap and building my own kernel

[Robert P. J. Day]

On Tue, 10 Apr 2012, Frank Ch. Eigler wrote:

>
> rpjday wrote:
>
> > [...]
> > $ stap -e 'probe kernel.function("sys_open") {log("hello world") exit()}'
> > and got:
> > ===== start =====
> > [...]
> > cc1: all warnings being treated as errors
> > make[1]: *** [/tmp/stapadEYdA/stap_59bddd394f89d5f186da7e250db36434_876.o] Error 1
> > make: *** [_module_/tmp/stapadEYdA] Error 2
> > Pass 4: compilation failed.  Try again with another '--vp 0001' option.
>
> Please try systemtap 1.7, or build one out of
> <git://sourceware.org/git/systemtap.git>.  Newer kernels sometimes
> break systemtap assumptions; these are fixed pretty quickly in the
> source tree, and get rolled into later releases.
>
> Please report any further problems at <systemtap@sourceware.org>,
> including /usr/bin/stap-report output, if able.

  ok, but i'm curious ... i took a quick look at earlier kernels and i
don't see where "struct hlist_head" *ever* had a "next" member field
that is required by this version of systemtap (1.4).

  can you clarify which kernel version(s) would have worked with this
version of systemtap?  thanks.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

ubuntu 11.10, systemtap and building my own kernel

[Frank Ch. Eigler]

Hi -

On Tue, Apr 10, 2012 at 02:23:43PM -0400, Robert P. J. Day wrote:
> [...]
> ok, but i'm curious ... i took a quick look at earlier kernels and i
> don't see where "struct hlist_head" *ever* had a "next" member field
> that is required by this version of systemtap (1.4).

I believe this was related to a viro's linux commit 9591718a5a.

> can you clarify which kernel version(s) would have worked with this
> version of systemtap?  thanks.

According to http://sourceware.org/ml/systemtap/2011-q1/msg00036.html,
it was run with 2.6.9 ... 2.6.37.

- FChE

ubuntu 11.10, systemtap and building my own kernel

[Robert P. J. Day]

On Tue, 10 Apr 2012, Frank Ch. Eigler wrote:

>
> rpjday wrote:
>
> > [...]
> > $ stap -e 'probe kernel.function("sys_open") {log("hello world") exit()}'
> > and got:
> > ===== start =====
> > [...]
> > cc1: all warnings being treated as errors
> > make[1]: *** [/tmp/stapadEYdA/stap_59bddd394f89d5f186da7e250db36434_876.o] Error 1
> > make: *** [_module_/tmp/stapadEYdA] Error 2
> > Pass 4: compilation failed.  Try again with another '--vp 0001' option.
>
> Please try systemtap 1.7, or build one out of
> <git://sourceware.org/git/systemtap.git>.  Newer kernels sometimes
> break systemtap assumptions; these are fixed pretty quickly in the
> source tree, and get rolled into later releases.
>
> Please report any further problems at <systemtap@sourceware.org>,
> including /usr/bin/stap-report output, if able.

  ok, that solved that problem, but i'm still getting:

/home/rpjday/systemtap/share/systemtap/runtime/stat.c:214:2: error:
?cpu_possible_map? undeclared (first use in this function)

if this is a simple fix, could you enlighten me?  or if it isn't, i
can move this to the systemtap mailing list.

rday


-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

ubuntu 11.10, systemtap and building my own kernel

[Robert P. J. Day]

On Tue, 10 Apr 2012, Frank Ch. Eigler wrote:

>
> rpjday wrote:
>
> > [...]
> > $ stap -e 'probe kernel.function("sys_open") {log("hello world") exit()}'
> > and got:
> > ===== start =====
> > [...]
> > cc1: all warnings being treated as errors
> > make[1]: *** [/tmp/stapadEYdA/stap_59bddd394f89d5f186da7e250db36434_876.o] Error 1
> > make: *** [_module_/tmp/stapadEYdA] Error 2
> > Pass 4: compilation failed.  Try again with another '--vp 0001' option.
>
> Please try systemtap 1.7, or build one out of
> <git://sourceware.org/git/systemtap.git>.  Newer kernels sometimes
> break systemtap assumptions; these are fixed pretty quickly in the
> source tree, and get rolled into later releases.
>
> Please report any further problems at <systemtap@sourceware.org>,
> including /usr/bin/stap-report output, if able.

  just FYI, this is the current problem:

===== begin =====

$ git show 615399c84d1b8d8d8752629e5e5ab4e5044d6918
commit 615399c84d1b8d8d8752629e5e5ab4e5044d6918
Author: Rusty Russell <rusty@rustcorp.com.au>
Date:   Thu Mar 29 15:38:31 2012 +1030

    cpumask: remove old cpu_*_map.

    These are obsolete: cpu_*_mask provides (const) pointers.

    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

diff --git a/include/linux/cpumask.h b/include/linux/cpumask.h
index 1ffdb98..a2c819d 100644
--- a/include/linux/cpumask.h
+++ b/include/linux/cpumask.h
@@ -764,12 +764,6 @@ static inline const struct cpumask *get_cpu_mask(unsigned int cpu)
  *
  */
 #ifndef CONFIG_DISABLE_OBSOLETE_CPUMASK_FUNCTIONS
-/* These strip const, as traditionally they weren't const. */
-#define cpu_possible_map       (*(cpumask_t *)cpu_possible_mask)
-#define cpu_online_map         (*(cpumask_t *)cpu_online_mask)
-#define cpu_present_map                (*(cpumask_t *)cpu_present_mask)
-#define cpu_active_map         (*(cpumask_t *)cpu_active_mask)
-
 #define cpumask_of_cpu(cpu) (*get_cpu_mask(cpu))

 #define CPU_MASK_LAST_WORD BITMAP_LAST_WORD_MASK(NR_CPUS)

===== end =====

  in current systemtap, runtime.h:

#define stp_for_each_cpu(cpu)  for_each_cpu_mask((cpu), cpu_possible_map)

so that's definitely going to fail.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

ubuntu 11.10, systemtap and building my own kernel

[Frank Ch. Eigler]

Hi -

On Tue, Apr 10, 2012 at 04:17:25PM -0400, Robert P. J. Day wrote:
> [...]
>   just FYI, this is the current problem:
> [...]

Yup, found it.  For now, you could adjust your copy of the runtime.h
header file s/map/mask/ and have at it.  We'll plop presently a
polished patch into proper place.

- FChE

ubuntu 11.10, systemtap and building my own kernel

[Robert P. J. Day]

On Tue, 10 Apr 2012, Frank Ch. Eigler wrote:

> Hi -
>
> On Tue, Apr 10, 2012 at 04:17:25PM -0400, Robert P. J. Day wrote:
> > [...]
> >   just FYI, this is the current problem:
> > [...]
>
> Yup, found it.  For now, you could adjust your copy of the runtime.h
> header file s/map/mask/ and have at it.  We'll plop presently a
> polished patch into proper place.

  actually, would it be possible to simplify the code?  i see that
systemtap's stat.c defines its own CPU iteration macro:

#define stp_for_each_cpu(cpu)  for_each_cpu_mask((cpu), cpu_possible_map)

is there no way to directly use the kernel's <linux/cpumask.h> file,
and something like "for_each_possible_cpu"?

  in any event, i "git clone"d systemtap so i can just pull the fix
when it's committed.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

ubuntu 11.10, systemtap and building my own kernel

[Frank Ch. Eigler]

Hi -

On Tue, Apr 10, 2012 at 05:08:04PM -0400, Robert P. J. Day wrote:
> actually, would it be possible to simplify the code?  [...]

Yes, will do.  (Some of this complexity came about from the desire to
support very old kernels.)

- FChE