Re: [RFD] Kprobes/Kretprobes perf support

[Masami Hiramatsu <mhiramat@redhat.com>]

Ingo Molnar wrote:
> 
> * Masami Hiramatsu <mhiramat@redhat.com> wrote:
> 
>> Frederic Weisbecker wrote:
>>> Ftrace events are supported by perfcounter currently but Kprobes
>>> dynamic ftrace events are of a different nature: we must create them
>>> before any toggling.
>>>
>>> So a large part is already done through the ftrace events and the fact
>>> that you create one dynamically for each kprobes (we'll just need
>>> a little callback for perf sample submission but that's a small
>>> point).
>>>
>>> The largest work that remains is to port the current powerful interface
>>> to create these k{ret}probes (with requested  arguments, etc...) through
>>> ftrace but using perf open syscall.
>>>
>>> And I imagine it won't be trivial.
>>>
>>> Ingo, Peter do you have an idea on how we could do that?
>>> We should be able to choose between a kprobe and kretprobe (these can
>>> be two separate counters). And also one must be able to request the dump
>>> of random desired parameters (or return values in case of kretprobe)
>>> or registers...
>>>
>>> May be we should use the perf attr by passing a __user address to a buffer
>>> that contains all these options?
>>> Once we get that to the kernel, that can be passed to ftrace-kprobe that
>>> can parse it, create the desired trace event and rely on perf to create
>>> a counter for it.
>>>
>>> I guess that won't imply so much adds to Masami's patchset. Most of
>>> the work is on the perf tools (parsing the user request).
>>>
>>> ./perf kprobes -e (func|addr):(c|r):(a1,a2,a3,... | rax,rbx,rcx,...)
>>>                                ^  ^
>>>                             c = call = kprobe
>>>                             r = return = kretprobe

It is better to support C/source syntax too.

./perf kprobes [-m kmod] [-k vmlinux] -e event-definition [-a arg-definition]
 or
./perf kprobes [-m kmod] [-k vmlinux] -f definition-file
 or
./perf kprobes [-m kmod] [-k vmlinux] -

event-definition:
 (p|r):[event-name]:probepoint

p = kprobe
r = kretprobe

probepoint (with debuginfo):
 function[+offs][@file]
 or
 @file:line

probepoint (without debuginfo):
 function[+offs]
 or
 address

arg-definition:
 a1,a2,a3,... | %ax,%bx,%cx,...| $var1,$var2,...

$var1,... are converted to register or memory address
by using debuginfo.

Thus, you can use perf like this.

./perf kprobes -e p::@mm/filemap.c:339 -a $inode,$pos


>>
>> If it is possible that libdwarf can be linked to the perf tool, I 
>> think it might be better to support 'C source line/local variable' 
>> style too, because basic dwarf decoding logic has already been 
>> done in c2kpe which I posted yesterday :-).
> 
> Sure - we can link it - and C/source syntax beats everything else, 
> hands down. We can also do the kind of automatic 'conditional 
> linking' we do for C++ symbol demangling - i.e. if libdwarf is not 
> installed we just emit a warning and dont build that functionality 
> but otherwise perf will still be built fine.
> 
> Thus there will be no dependency on libdwarf.

That's fine to me. If there is no libdwarf or CONFIG_DEBUG_INFO=n (or
just couldn't find vmlinux), 'perf kprobes' just doesn't accept
C/source syntax (and fail back to symbol/address syntax).

> 
> One thing that occured to me is that it would be nice to have sanity 
> checks of all sorts. For example we could expose the md5sum of the 
> kernel image and perf would double check against that when looking 
> around in the debuginfo - or something like that.

Systemtap has been done it with build-id.
http://sources.redhat.com/bugzilla/show_bug.cgi?id=4886

Perhaps, we can also use build-id to check the vmlinux version.

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com