[RFC] ktap: Another dynamic tracing tool for Linux

[RFC] ktap: Another dynamic tracing tool for Linux

[Jovi Zhang]

Hi,

This mail is RFC for discuss on a new dynamic tracing tool, I name it
ktap. (only experimental project now)

I'm working at Linux embedded world long time,
when I noticed systemtap is not suit for my dynamic tracing
requirement(systemtap need to run gcc),
so I decide to write this experimental project ktap, for safe,
lightweight, extendable, easy to use, and funny.
ktap use script language to let user trace Linux kernel dynamically,
ktap is like Systemtap, and Dtrace in Solaris.

what ktap differentiates with Systemtap is:
1). ktap don't depend on gcc, you don't need to compile kernel module,
and even without root privileged in future.
      this resolve some limitation of Systemtap.
      In many usecase, it's not possible to compile source file to ko
easily, depend on product environment (like embedded).

2). ktap have good portability, because it compile source file to
bytecode, like python and Java.

3). ktap use it own simple dynamic typed script language, this
language include built-in hash table support.

4). ktap is safe, whatever you doing in script file, you will never
crash your kernel.

5). ktap will be open source completely, with GPL license, it might be
merge into mainline in someday, that's very convince for tracing user.


what ktap differentiates with Dtrace in Solaris is:
6). it's not Dtrace porting for Linux, it a new tool build from scratch.

7). The language is not D, it's a dynamic typed language which like lua
      currently ktap use lua syntax and bytecode, but it may change it
when need in future.

8). ktap is GPL licensed, so there don't have any license issue when
upstream(if possible in future)

ktap have a userspace compiler tool "ktapc", ktapc compile ktap source
file to .ktap bytecode chunk file,
then ktap binary load and run the chunk bytecode file, there have a
kernel module ktapvm executing .ktap bytecode.

ktap use lua language syntax and bytecode as initial implementation,
why lua? the reason is:
lua is small, syntax is clean, no complex language feature, few
bytecode, expendable, the virtual machine is
smaller than other script language like python and perl, it have
built-in hashtable data structure support, which proving
helpful for tracing purpose.
(ktap just borrow syntax and bytecode from lua initially,  there
should not have any relation with lua after ktap release,
ktap will have its own road)


ktap implementation is not completely finish yet, there already have a
workable ktap version,
it support basic syscall and tracepoint tracing currently, in future,
it could support kprobe, uprobe, userspace probe, etc.

that's all about ktap until now.
I wish you can give me some technical architecture pre-review for
ktap, before ktap release 1.0.
Any comments is welcome, thanks very much.

.jovi

Re: [RFC] ktap: Another dynamic tracing tool for Linux

[Frank Ch. Eigler]

bookjovi wrote:


> [...]  This mail is RFC for discuss on a new dynamic tracing tool, I
> name it ktap. (only experimental project now)

Welcome to the problem domain!


> [...]
> what ktap differentiates with Systemtap is:
> [...]
> 2). ktap have good portability, because it compile source file to
> bytecode, like python and Java.

(From this PoV, systemtap is just as portable as the kernel, as it
generates the same sort of C code the kernel is built from.)

> [...]
> 5). ktap will be open source completely, with GPL license, it might be
> merge into mainline in someday, that's very convince for tracing user.

(systemtap has always been GPLv2, ever since its beginning in 2005.)


> [...]
> ktap use lua language syntax and bytecode as initial implementation,

Interesting approach.  I recall we considered it way back when, but
rejected it for a couple of reasons, including the at-the-time
perceived unwelcomeness of a serious bytecode interpreter within the
kernel.


> it could support kprobe, uprobe, userspace probe, etc.

Great.

> I wish you can give me some technical architecture pre-review for
> ktap, before ktap release 1.0.
> Any comments is welcome, thanks very much.

Have you made any source code available yet?


- FChE

Re: [RFC] ktap: Another dynamic tracing tool for Linux

[Jovi Zhang]

On Tue, Jan 1, 2013 at 2:58 AM, Frank Ch. Eigler <fche@redhat.com> wrote:
>
> bookjovi wrote:
>
>
>> [...]  This mail is RFC for discuss on a new dynamic tracing tool, I
>> name it ktap. (only experimental project now)
>
> Welcome to the problem domain!
Thanks very much, frank, I'm so luck to discussing this topic with you. :)
>
>
>> [...]
>> what ktap differentiates with Systemtap is:
>> [...]
>> 2). ktap have good portability, because it compile source file to
>> bytecode, like python and Java.
>
> (From this PoV, systemtap is just as portable as the kernel, as it
> generates the same sort of C code the kernel is built from.)
>
the portability for ktap what I mean here is binary portability, you
will not need to
compile arm and x86 binary in ktap, it's all bytecode.
but as you pointed, systemtap also have good portability in source
level, for more clear,
I will change the readme text a little bit about this portability.  thanks.

>> [...]
>> 5). ktap will be open source completely, with GPL license, it might be
>> merge into mainline in someday, that's very convince for tracing user.
>
> (systemtap has always been GPLv2, ever since its beginning in 2005.)

Yes, you are right, both is GPL licensed.
>
>
>> [...]
>> ktap use lua language syntax and bytecode as initial implementation,
>
> Interesting approach.  I recall we considered it way back when, but
> rejected it for a couple of reasons, including the at-the-time
> perceived unwelcomeness of a serious bytecode interpreter within the
> kernel.
>
ktap is more like Dtrace, without gcc compiling.
Obviously, the bytecode design is the biggest differences with systemtap.
systemtap really is a valueable tracing tool, but in many case we cannot use it.
many Linux box doesn't deverily with gcc, this is very common in
embedded device,
even there installed gcc, but it's hard(impossible) to get matched
kernel source.
embeded world have many hardware architectures, arm/x86/mips/ppc, when
those different
hardware board is combined into a cluster, you need to prepare several
systemtap kernel module
in development machine, then upload all kernel module to cluster. but
if you have a bytecode
mechanism, that's totaly not neccessary, just a source script file or
bytecode chunk file.
This is just one advantage of bytecode mechanism.

Of course we must need to afford some penalty on performance compare
with raw C design(systemtap),
but gaining better convenient, one directing of ktap is reducing
bytecode executing overhead to minimal.

On clear that, ktap is not a replacement to systemtap, it's supplementation.
>
>> it could support kprobe, uprobe, userspace probe, etc.
>
> Great.
>
>> I wish you can give me some technical architecture pre-review for
>> ktap, before ktap release 1.0.
>> Any comments is welcome, thanks very much.
>
> Have you made any source code available yet?
>
I will try to make ktap develop progress more faster, and make ktap
source code public available soon.

.jovi

Re: [RFC] ktap: Another dynamic tracing tool for Linux

[Frank Ch. Eigler]

Hi -

bookjovi wrote:

> >> [...]
> >> ktap use lua language syntax and bytecode as initial implementation,
> >
> > Interesting approach.  I recall we considered it way back when, but
> > rejected it for a couple of reasons, including the at-the-time
> > perceived unwelcomeness of a serious bytecode interpreter within the
> > kernel.

> [...]  Obviously, the bytecode design is the biggest differences
> with systemtap.  [...]  many Linux box doesn't deliver with gcc,
> this is very common in embedded device, even there installed gcc,
> but it's hard(impossible) to get matched kernel source.  [...]

Right, that is a strong attraction.


> [...]  On clear that, ktap is not a replacement to systemtap, it's
> supplementation.

FWIW, it would be reasonable to have systemtap emit bytecodes as an
alternative backend.  All that has been lacking is a powerful and
pervasive enough interpreter.  The script language is not tied to its
implementation via C code generation.


That reminds me of your item #4 on your ktap-systemtap comparison:

> 4). ktap is safe, whatever you doing in script file, you will never
> crash your kernel.

This is roughly as true for systemtap as for anything else.  The
scripts are constrainted to be safe.  Kernel crashes that occur are
due to occasional bugs in the systemtap runtime library, or down in
the kernel facilities being used.  You would likely encounter both
classes of problems in a kernel-side bytecode interpreter, regardless
of the theoretical safety of the scripting language.  (This is one of
the reasons we've been building out the pure-userspace dyninst-based
systemtap backend.)


> I will try to make ktap develop progress more faster, and make ktap
> source code public available soon.

Yes, please.  (RFC/experimental code need not be complete before
posting; why not develop straight on github?)


- FChE

Re: [RFC] ktap: Another dynamic tracing tool for Linux

[Jovi Zhang]

On Fri, Jan 11, 2013 at 10:19 PM, Michel Dagenais
<michel.dagenais@polymtl.ca> wrote:
> You may be interested in KGTP which implements a simple bytecode interpreter
> in the kernel to accept GDB tracepoints http://code.google.com/p/kgtp/
>
> The bytecode is quite limited but would be easy to extend.
KGTP is still not meet my requirement on Linux tracing.
ktap don't have gcc or gdb dependence, it's build from scratch, with a
clean design, this is very important.

You will find more differences if you watch ktap and kgtp deeply. :)

>
> Eventually we should be able to connect LTTng http://lttng.org/ and KGTP in
> order to benefit from the efficiency of LTTng for activating probes and
> retrieving data.
You are right, LTTng should be possible, and I already planed it, also
on some functionality of ftrace and systemtap.

Let's wait public ktap for a little time, you then can get a basic
overview on ktap.

Re: [RFC] ktap: Another dynamic tracing tool for Linux

[Michel Dagenais]

----- "Jovi Zhang" <bookjovi@gmail.com> wrote:
> On Fri, Jan 11, 2013 at 10:19 PM, Michel Dagenais
> <michel.dagenais@polymtl.ca> wrote:
> > You may be interested in KGTP which implements a simple bytecode interpreter
> > in the kernel to accept GDB tracepoints
> http://code.google.com/p/kgtp/
> >
> > The bytecode is quite limited but would be easy to extend.
> KGTP is still not meet my requirement on Linux tracing.
> ktap don't have gcc or gdb dependence, it's build from scratch, with a
> clean design, this is very important.

KGTP uses the GDB remote protocol but does not use/require GDB. If you have the needed information about symbols, offsets... you can generate and send the bytecode yourself. 

> > Eventually we should be able to connect LTTng http://lttng.org/ and KGTP in
> > order to benefit from the efficiency of LTTng for activating probes and
> > retrieving data.
> You are right, LTTng should be possible, and I already planed it, also
> on some functionality of ftrace and systemtap.

Such dynamic capabilities are definitely of interest both in kernel and userspace tracing.

Re: [RFC] ktap: Another dynamic tracing tool for Linux

[Jovi Zhang]

On Fri, Jan 4, 2013 at 11:19 PM, Frank Ch. Eigler <fche@redhat.com> wrote:
> Hi -
>
> bookjovi wrote:
>
>> >> [...]
>> >> ktap use lua language syntax and bytecode as initial implementation,
>> >
>> > Interesting approach.  I recall we considered it way back when, but
>> > rejected it for a couple of reasons, including the at-the-time
>> > perceived unwelcomeness of a serious bytecode interpreter within the
>> > kernel.
>
>> [...]  Obviously, the bytecode design is the biggest differences
>> with systemtap.  [...]  many Linux box doesn't deliver with gcc,
>> this is very common in embedded device, even there installed gcc,
>> but it's hard(impossible) to get matched kernel source.  [...]
>
> Right, that is a strong attraction.
>
>
>> [...]  On clear that, ktap is not a replacement to systemtap, it's
>> supplementation.
>
> FWIW, it would be reasonable to have systemtap emit bytecodes as an
> alternative backend.  All that has been lacking is a powerful and
> pervasive enough interpreter.  The script language is not tied to its
> implementation via C code generation.
>
>
> That reminds me of your item #4 on your ktap-systemtap comparison:
>
>> 4). ktap is safe, whatever you doing in script file, you will never
>> crash your kernel.
>
> This is roughly as true for systemtap as for anything else.  The
> scripts are constrainted to be safe.  Kernel crashes that occur are
> due to occasional bugs in the systemtap runtime library, or down in
> the kernel facilities being used.  You would likely encounter both
> classes of problems in a kernel-side bytecode interpreter, regardless
> of the theoretical safety of the scripting language.  (This is one of
> the reasons we've been building out the pure-userspace dyninst-based
> systemtap backend.)
>
>
>> I will try to make ktap develop progress more faster, and make ktap
>> source code public available soon.
>
> Yes, please.  (RFC/experimental code need not be complete before
> posting; why not develop straight on github?)
>
>
> - FChE

Let us continue this ktap topic in this thread :).

ktap code is public available at github, please clone from:

https://github.com/ktap/ktap.git

Brief introduction for initial building ktap:
-----------------------------------------------------------------------------------------------
ktap is a new dynamic tracing tool for Linux, it is a script environment
running in Linux kernel(similar with systemtap and Dtrace).

ktap is GPL licensed, the compiler and virtual machine is borrowed
from lua language as initial implementation.

compiler and loader is included in userspace tool ktap

compiling:
    make ktap    --- generate userspace ktap tool
    make         --- generate ktapvm kernel module

try to running ktap:
    ./ktap scripts/tracepoint.ktap

This commit is just a initial pulbic code release, it have basic
tracepoint and syscall support, please waiting ktap release 1.0. :)

I suggest you put this ktap directory into linux/kernel/trace/.

(this initial code have one issue: it need to patch ftrace code in Linux kernel,
 see ftrace.patch, I will try to get rid it sooner.)

For any question, please contact the author, Jovi Zhang.

ktap is a open source project, welcome hacking and contributing.

.jovi

Re: [RFC] ktap: Another dynamic tracing tool for Linux

[Frank Ch. Eigler]

Hi -

On Fri, Jan 18, 2013 at 09:24:55AM +0800, Jovi Zhang wrote:
> Let us continue this ktap topic in this thread :).
> ktap code is public available at github, please clone from:
> https://github.com/ktap/ktap.git
> [...]

Neat stuff.  I have one set of initial observations: even with a nice
small bytecode language, the VM has to be skeptical and careful.  For
example, some dangers:

- any dynamic memory allocation from within the context of an
  arbitrary tracepoints (table_* functions, stack frames)
- unchecked arithmetic (division-by-zero - OP_DIV, overflows,
  function arity limits)
- time-quantity of computation (limit loop iterations / #-bytecodes?,
  DO_JMP infinite loops)
- stack-frame usage of interpreter (if internally recursive, or if 
  too much state kept on stack)
- trusting validity of bytecode (imagine an attacker sending random
  or harmful junk, jumping out of bytecode ranges)
- calls out from interpreter into native code - to ensure that *all*
  those functions (transitively) are valid to call from general
  e.g. tracepoint context (e.g. sleeps often aren't)
- (and these problems get even worse with evaluation from the
  context of kprobes)


- FChE

Re: [RFC] ktap: Another dynamic tracing tool for Linux

[Jovi Zhang]

On Fri, Jan 18, 2013 at 11:35 AM, Frank Ch. Eigler <fche@redhat.com> wrote:
> Hi -
>
> On Fri, Jan 18, 2013 at 09:24:55AM +0800, Jovi Zhang wrote:
>> Let us continue this ktap topic in this thread :).
>> ktap code is public available at github, please clone from:
>> https://github.com/ktap/ktap.git
>> [...]
>
> Neat stuff.  I have one set of initial observations: even with a nice
> small bytecode language, the VM has to be skeptical and careful.  For
> example, some dangers:
>
> - any dynamic memory allocation from within the context of an
>   arbitrary tracepoints (table_* functions, stack frames)
> - unchecked arithmetic (division-by-zero - OP_DIV, overflows,
>   function arity limits)
> - time-quantity of computation (limit loop iterations / #-bytecodes?,
>   DO_JMP infinite loops)
> - stack-frame usage of interpreter (if internally recursive, or if
>   too much state kept on stack)
> - trusting validity of bytecode (imagine an attacker sending random
>   or harmful junk, jumping out of bytecode ranges)
> - calls out from interpreter into native code - to ensure that *all*
>   those functions (transitively) are valid to call from general
>   e.g. tracepoint context (e.g. sleeps often aren't)
> - (and these problems get even worse with evaluation from the
>   context of kprobes)
>
>
> - FChE

Yeh, that's why this code is called initial implementation.
most of dangers you pointed I have thought about, most of them should not
have too much hard to solve it, it just need some time, some of them already
listed in Todo-list.txt.

Thanks your quick observations, the comments is valueable, and need to
be address step by step
in future
development (I also think most issues is very common for script
dynamic tracing tool, not specific for ktap,
but ktap have to go through those barrier one by one)

Let's waiting for ktap official release 0.1, the code quality would be
more better than now :)

.jovi