Re: Re: [BUG] perf probe can't insert return kprobe

[Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>]

(2015/01/10 18:51), Namhyung Kim wrote:
> Hi Jiri,
> 
> On Fri, Jan 09, 2015 at 04:44:21PM +0100, Jiri Olsa wrote:
>> On Fri, Jan 09, 2015 at 04:30:56PM +0100, Jiri Olsa wrote:
>>> On Sat, Jan 10, 2015 at 12:21:13AM +0900, Namhyung Kim wrote:
>>>> On Fri, Jan 09, 2015 at 03:55:39PM +0100, Jiri Olsa wrote:
>>>>> hi,
>>>>> I couldn't use following perf command to insert return probe:
>>>>>
>>>>>   # perf probe -a fork_exit=do_fork%return
>>>>>   Added new event:
>>>>>   Failed to write event: Invalid argument
>>>>>     Error: Failed to add events.
>>>>>
>>>>>
>>>>> I'm pretty sure I used this command before, so seems like
>>>>> it's broken. I can still use debugfs tracing interface to
>>>>> do that:
>>>>>   # echo 'r:do_fork_entry do_fork' > kprobe_events
>>>>>
>>>>> I used Arnaldo's latest perf/core and FC20 kernel:
>>>>>
>>>>>   # uname -a
>>>>>   Linux krava 3.17.7-200.fc20.x86_64 #1 SMP Wed Dec 17 03:35:33 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>>>>>   # ./perf version
>>>>>   perf version 3.18.g6a7d78
>>>>>
>>>>
>>>> Is it just return probe?  Did it work for normal kprobes?
>>>
>>> yep, works for normal probes
>>>
>>>> Maybe it's related to the below:
>>>>
>>>> https://lkml.org/lkml/2014/12/31/15
>>>>
>>>> Have you check the acme/perf/urgent too?
>>>
>>> hum.. can't access lkml, I'll check, also with perf/urgent
>>
>> neither helped..
> 
> I think I've found the reason.
> 
> The commit dfef99cd0b2c ("perf probe: Use ref_reloc_sym based address
> instead of the symbol name") converts kprobes to use ref_reloc_sym
> (i.e. _stext) and offset instead of using symbol's name directly.  So
> on my system, adding do_fork ends up with like below:
> 
>   $ sudo perf probe -v --add do_fork%return
>   probe-definition(0): do_fork%return
>   symbol:do_fork file:(null) line:0 offset:0 return:1 lazy:(null)
>   0 arguments
>   Looking at the vmlinux_path (7 entries long)
>   Using /lib/modules/3.17.6-1-ARCH/build/vmlinux for symbols
>   Could not open debuginfo. Try to use symbols.
>   Opening /sys/kernel/debug/tracing/kprobe_events write=1
>   Added new event:
>   Writing event: r:probe/do_fork _stext+456136
>   Failed to write event: Invalid argument
>     Error: Failed to add events. Reason: Operation not permitted (Code: -1)
>   
> 
> As you can see, the do_fork was translated to _stext+456136.  This was
> because to support (local) symbols that have same name.  But the
> problem is that kretprobe requires to be inserted at function start
> point so it simply checks whether it's called with offset 0.  And if
> not, it'll return with -EINVAL.  You can see it with dmesg.
> 
>   $ dmesg | tail -1
>   [125621.764103] Return probe must be used without offset.
> 
> So we need to use the symbol name instead of ref_reloc_sym in case of
> return probes.  During the tracking down, I found a couple of problems
> in the code.  I'll send fixes soon.

Oops, thank you for analyzing the problem:) I've completely forgot about
return probes with KASLR. This also should be fixed in kprobe-tracer,
which should convert given _stext+offset to actual symbol in kernel.

Thanks,

> 
> Thanks,
> Namhyung
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com