failed to add c++ probe

failed to add c++ probe

[Jiri Olsa]

hi,
Thomas reported failure to add an uprobe on libstdc++.so.6 like:

[jolsa@krava perf]$ sudo ./perf probe -x /usr/lib64/libstdc++.so.6 'std::ostream::flush'
Semantic error :There is non-digit char in line number.
  Error: Command Parse Error.

it does not work also if I use the mangled name like:

[jolsa@krava perf]$ sudo ./perf probe -x /usr/lib64/libstdc++.so.6 '_ZSt5flushIwSt11char_traitsIwEERSt13basic_ostreamIT_T0_ES6_'
Probe point '_ZSt5flushIwSt11char_traitsIwEERSt13basic_ostreamIT_T0_ES6_' not found.
  Error: Failed to add events.

thanks,
jirka

Re: failed to add c++ probe

[Masami Hiramatsu]

On Mon, 19 Sep 2016 11:54:02 +0200
Jiri Olsa <jolsa@redhat.com> wrote:

> hi,
> Thomas reported failure to add an uprobe on libstdc++.so.6 like:
> 
> [jolsa@krava perf]$ sudo ./perf probe -x /usr/lib64/libstdc++.so.6 'std::ostream::flush'
> Semantic error :There is non-digit char in line number.
>   Error: Command Parse Error.

Ah, I should start adding mangling support on perf probe...


> it does not work also if I use the mangled name like:
> 
> [jolsa@krava perf]$ sudo ./perf probe -x /usr/lib64/libstdc++.so.6 '_ZSt5flushIwSt11char_traitsIwEERSt13basic_ostreamIT_T0_ES6_'
> Probe point '_ZSt5flushIwSt11char_traitsIwEERSt13basic_ostreamIT_T0_ES6_' not found.
>   Error: Failed to add events.

Hmm, could you try to find mangled symbol from the library as below?

./perf probe -x /usr/lib64/libstdc++.so.6 -F \*

And also try to do adding probe with -v option again?

In my case, I could find the mangled symbol, but failed to setup the probe...

$ sudo ./perf probe -x /usr/lib/x86_64-linux-gnu/libstdc++.so.6 -v _ZSt5flushIwSt11char_traitsIwEERSt13basic_ostreamIT_T0_ES6_
probe-definition(0): _ZSt5flushIwSt11char_traitsIwEERSt13basic_ostreamIT_T0_ES6_ 
symbol:_ZSt5flushIwSt11char_traitsIwEERSt13basic_ostreamIT_T0_ES6_ file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
symbol:catch file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:throw file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:rethrow file:(null) line:0 offset:0 return:0 lazy:(null)
Could not open debuginfo. Try to use symbols.
Opening /sys/kernel/debug/tracing//uprobe_events write=1
Writing event: p:probe_libstdc++/_ZSt5flushIwSt11char_traitsIwEERSt13basic_ostreamIT_T0_ES6_ /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21:0x1114c0
Failed to write event: Invalid argument
  Error: Failed to add events. Reason: Invalid argument (Code: -22)

dmsg -k said;

[143725.339609] Failed to allocate trace_uprobe.(-22)
[143725.339612] Failed to parse address or file.

It seems we need to fix trace_uprobe.c at least.

Thank you!

-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: failed to add c++ probe

[Thomas Gleixner]

Masami,

On Tue, 20 Sep 2016, Masami Hiramatsu wrote:
> Hmm, could you try to find mangled symbol from the library as below?
> 
> ./perf probe -x /usr/lib64/libstdc++.so.6 -F \*
> 
> And also try to do adding probe with -v option again?
> 
> In my case, I could find the mangled symbol, but failed to setup the probe...

./perf probe -x /usr/lib/x86_64-linux-gnu/libstdc++.so.6 -v _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv
probe-definition(0): _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv 
symbol:_X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
symbol:catch file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:throw file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:rethrow file:(null) line:0 offset:0 return:0 lazy:(null)
Open Debuginfo file: /usr/lib/debug/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
Try to find probe point from debuginfo.
Symbol _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv address found : 8c620
Matched function: is_open
found inline addr: 0x8cad0
Probe point found: is_open+0
found inline addr: 0x8cac0
Probe point found: is_open+0
found inline addr: 0x8cab0
Probe point found: is_open+0
Matched function: is_open
An error occurred in debuginfo analysis (-2).
Trying to use symbols.
Opening /sys/kernel/tracing//uprobe_events write=1
Writing event: p:probe_libstdc++/_X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22:0x8c620
Failed to write event: Invalid argument
  Error: Failed to add events. Reason: Invalid argument (Code: -22)

[16568.176464] Failed to allocate trace_uprobe.(-22)
[16568.176468] Failed to parse address or file.

Aside of that mangled symbols with a @plt suffix do not work at all.

Thanks,

	tglx

Re: failed to add c++ probe

[Masami Hiramatsu]

On Tue, 20 Sep 2016 01:32:04 +0200 (CEST)
Thomas Gleixner <tglx@linutronix.de> wrote:

> Masami,
> 
> On Tue, 20 Sep 2016, Masami Hiramatsu wrote:
> > Hmm, could you try to find mangled symbol from the library as below?
> > 
> > ./perf probe -x /usr/lib64/libstdc++.so.6 -F \*
> > 
> > And also try to do adding probe with -v option again?
> > 
> > In my case, I could find the mangled symbol, but failed to setup the probe...
> 
> ./perf probe -x /usr/lib/x86_64-linux-gnu/libstdc++.so.6 -v _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv
> probe-definition(0): _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv 
> symbol:_X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv file:(null) line:0 offset:0 return:0 lazy:(null)
> 0 arguments
> symbol:catch file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:throw file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:rethrow file:(null) line:0 offset:0 return:0 lazy:(null)
> Open Debuginfo file: /usr/lib/debug/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
> Try to find probe point from debuginfo.
> Symbol _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv address found : 8c620
> Matched function: is_open
> found inline addr: 0x8cad0
> Probe point found: is_open+0
> found inline addr: 0x8cac0
> Probe point found: is_open+0
> found inline addr: 0x8cab0
> Probe point found: is_open+0
> Matched function: is_open
> An error occurred in debuginfo analysis (-2).
> Trying to use symbols.
> Opening /sys/kernel/tracing//uprobe_events write=1
> Writing event: p:probe_libstdc++/_X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22:0x8c620
> Failed to write event: Invalid argument
>   Error: Failed to add events. Reason: Invalid argument (Code: -22)
> 
> [16568.176464] Failed to allocate trace_uprobe.(-22)
> [16568.176468] Failed to parse address or file.
> 
> Aside of that mangled symbols with a @plt suffix do not work at all.

Oh, OK. I could reproduce it. I'll investigate it. :)

Thank you,


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: failed to add c++ probe

[Masami Hiramatsu]

On Wed, 21 Sep 2016 12:37:59 +0900
Masami Hiramatsu <mhiramat@kernel.org> wrote:

> On Tue, 20 Sep 2016 01:32:04 +0200 (CEST)
> Thomas Gleixner <tglx@linutronix.de> wrote:
> 
> > Masami,
> > 
> > On Tue, 20 Sep 2016, Masami Hiramatsu wrote:
> > > Hmm, could you try to find mangled symbol from the library as below?
> > > 
> > > ./perf probe -x /usr/lib64/libstdc++.so.6 -F \*
> > > 
> > > And also try to do adding probe with -v option again?
> > > 
> > > In my case, I could find the mangled symbol, but failed to setup the probe...
> > 
> > ./perf probe -x /usr/lib/x86_64-linux-gnu/libstdc++.so.6 -v _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv
> > probe-definition(0): _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv 
> > symbol:_X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv file:(null) line:0 offset:0 return:0 lazy:(null)
> > 0 arguments
> > symbol:catch file:(null) line:0 offset:0 return:0 lazy:(null)
> > symbol:throw file:(null) line:0 offset:0 return:0 lazy:(null)
> > symbol:rethrow file:(null) line:0 offset:0 return:0 lazy:(null)
> > Open Debuginfo file: /usr/lib/debug/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
> > Try to find probe point from debuginfo.
> > Symbol _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv address found : 8c620
> > Matched function: is_open
> > found inline addr: 0x8cad0
> > Probe point found: is_open+0
> > found inline addr: 0x8cac0
> > Probe point found: is_open+0
> > found inline addr: 0x8cab0
> > Probe point found: is_open+0
> > Matched function: is_open
> > An error occurred in debuginfo analysis (-2).
> > Trying to use symbols.
> > Opening /sys/kernel/tracing//uprobe_events write=1
> > Writing event: p:probe_libstdc++/_X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22:0x8c620
> > Failed to write event: Invalid argument
> >   Error: Failed to add events. Reason: Invalid argument (Code: -22)
> > 
> > [16568.176464] Failed to allocate trace_uprobe.(-22)
> > [16568.176468] Failed to parse address or file.
> > 
> > Aside of that mangled symbols with a @plt suffix do not work at all.
> 
> Oh, OK. I could reproduce it. I'll investigate it. :)
> 

Hmm, I found there are 3 issues in perf-probe to define event on C++ libs.

1) No mangle/demangle symbol support. This needs a) fix option parser to
   accept C++ method name correctly, b) mangle that method, and c) demangle
   it when we show the probe point.

2) @plt symbol should be skipped since the real function will be called(jumped)
   from plt, when searching probe point.

3) Event group name is including some characters which can not be used (e.g. "++").
   Thus we have to remove it or replace it with '_' or other alphabet.("PP"?)

Thank you,

-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: failed to add c++ probe

[Peter Zijlstra]

On Wed, Sep 21, 2016 at 01:43:24PM +0900, Masami Hiramatsu wrote:
> On Wed, 21 Sep 2016 12:37:59 +0900
> Masami Hiramatsu <mhiramat@kernel.org> wrote:

> Hmm, I found there are 3 issues in perf-probe to define event on C++ libs.
> 
> 1) No mangle/demangle symbol support. This needs a) fix option parser to
>    accept C++ method name correctly, b) mangle that method, and c) demangle
>    it when we show the probe point.

I think we should also allow using the mangled name, supporting (de)mangling
is of course nice, but not required per-se. Esp. since there's more than
just C++ mangling. Having the mangling stuff as optional makes it a
convenience but still allows using this on other languages for which we
do not support the mangling.

> 2) @plt symbol should be skipped since the real function will be called(jumped)
>    from plt, when searching probe point.
> 
> 3) Event group name is including some characters which can not be used (e.g. "++").
>    Thus we have to remove it or replace it with '_' or other alphabet.("PP"?)

What is the problem with the '+' character? Why cannot this be used? 

Re: failed to add c++ probe

[Masami Hiramatsu]

On Thu, 22 Sep 2016 10:54:06 +0200
Peter Zijlstra <peterz@infradead.org> wrote:

> On Wed, Sep 21, 2016 at 01:43:24PM +0900, Masami Hiramatsu wrote:
> > On Wed, 21 Sep 2016 12:37:59 +0900
> > Masami Hiramatsu <mhiramat@kernel.org> wrote:
> 
> > Hmm, I found there are 3 issues in perf-probe to define event on C++ libs.
> > 
> > 1) No mangle/demangle symbol support. This needs a) fix option parser to
> >    accept C++ method name correctly, b) mangle that method, and c) demangle
> >    it when we show the probe point.
> 
> I think we should also allow using the mangled name, supporting (de)mangling
> is of course nice, but not required per-se. Esp. since there's more than
> just C++ mangling. Having the mangling stuff as optional makes it a
> convenience but still allows using this on other languages for which we
> do not support the mangling.

Agreed. I also learned that mangling depends on the compiler and no
good way to get demangled name to mangled name. Moreover, the demangled
name will include arguments, templates etc. because of supporting polymorphism.
At this phase, I would like to support only "method name" or "mangled name".

> > 2) @plt symbol should be skipped since the real function will be called(jumped)
> >    from plt, when searching probe point.
> > 
> > 3) Event group name is including some characters which can not be used (e.g. "++").
> >    Thus we have to remove it or replace it with '_' or other alphabet.("PP"?)
> 
> What is the problem with the '+' character? Why cannot this be used? 

In the ftrace, it is not considered to use such characters for event/group
name, because the name is used for generating symbols for tracepoints.
IOW, other tracepoint events have the name same as C naming conventions.
(Maybe that is also good for out-of-tree tools and scripts to parse it.)

Thank you,

-- 
Masami Hiramatsu <mhiramat@kernel.org>