perf probe an addr without debuginfo

perf probe an addr without debuginfo

[Brendan Gregg]

G'Day,

Is there a trick to getting perf to probe a user-level address without
debuginfo? Eg (on Linux 4.0):

# perf probe -x /root/tick '@+0x583'
Debuginfo-analysis is not supported.
  Error: Failed to add events.

I can do this using ftrace ok, eg, "p:tick_0x583 /root/tick:0x583"
works. Thanks,

Brendan

Re: perf probe an addr without debuginfo

[Benjamin King]

Hi Brendan

>Is there a trick to getting perf to probe a user-level address without
>debuginfo? Eg (on Linux 4.0):
>[...]
>I can do this using ftrace ok, eg, "p:tick_0x583 /root/tick:0x583"
>works. Thanks,

Not quite what you have asked for, but you can add the probe via ftrace and
then use it from perf. Probes from /sys/kernel/debug/tracing/uprobe_events
will show up in 'perf list' as well.

Cheers,
  Benjamin

Re: perf probe an addr without debuginfo

[Arnaldo Carvalho de Melo]

Em Sun, Jun 28, 2015 at 05:46:51PM +0200, Benjamin King escreveu:
> Hi Brendan
> 
> >Is there a trick to getting perf to probe a user-level address without
> >debuginfo? Eg (on Linux 4.0):
> >[...]
> >I can do this using ftrace ok, eg, "p:tick_0x583 /root/tick:0x583"
> >works. Thanks,
> 
> Not quite what you have asked for, but you can add the probe via ftrace and
> then use it from perf. Probes from /sys/kernel/debug/tracing/uprobe_events
> will show up in 'perf list' as well.

Masami,

	Is this already possible?

- Arnaldo

P.S.: If you're not subscribed to this list, please consider it :-)

Re: perf probe an addr without debuginfo

[Benjamin King]

[-- Attachment #1: Type: text/plain, Size: 1842 bytes --]

On Mon, Jun 29, 2015 at 12:00:38PM -0300, Arnaldo Carvalho de Melo wrote:
>Em Sun, Jun 28, 2015 at 05:46:51PM +0200, Benjamin King escreveu:
>> >Is there a trick to getting perf to probe a user-level address without
>> >debuginfo? Eg (on Linux 4.0):
>> >[...]
>> >I can do this using ftrace ok, eg, "p:tick_0x583 /root/tick:0x583"
>> >works. Thanks,
>>
>> Not quite what you have asked for, but you can add the probe via ftrace and
>> then use it from perf. Probes from /sys/kernel/debug/tracing/uprobe_events
>> will show up in 'perf list' as well.
>
>Masami,
>
>	Is this already possible?

Hi Arnaldo,

for me this works in 3.19.0. I'm using the attached script to help me add
probes to /sys/kernel/debug/tracing/uprobe_events which I can then see with
'sudo perf list' and use from the other perf subcommands. Tracing arguments is
also working fine:

--- snip ---
$ uprobe /lib/x86_64-linux-gnu/libc.so.6 malloc %di
$ sudo perf stat  -e uprobes:* ls
...
 Performance counter stats for 'ls':
...
               218      uprobes:malloc_1                                            
$ sudo perf record -euprobes:malloc_1 ls >/dev/null 2>&1
$ sudo perf script
              ls 21084 [003] 10825.526815: uprobes:malloc_1: (7f772bfc2700) arg1=0x238
              ls 21084 [003] 10825.526917: uprobes:malloc_1: (7f772bfc2700) arg1=0x238
              ls 21084 [003] 10825.526968: uprobes:malloc_1: (7f772bfc2700) arg1=0x78
              ls 21084 [003] 10825.527089: uprobes:malloc_1: (7f772bfc2700) arg1=0x5
              ls 21084 [003] 10825.527132: uprobes:malloc_1: (7f772bfc2700) arg1=0x78
...
--- snip ---

This is superuseful to me, so thank you all for working on perf!

I failed to manage placing proper probes in C++-Code with perf probe, so I
hacked the script together after reading Brendans ftrace article on LWN.

Cheers,
  Benjamin

[-- Attachment #2: uprobe --]
[-- Type: text/plain, Size: 2965 bytes --]

#!/bin/bash

usage() {
  cat <<EOF
Add and remove uprobes

USAGE
  $0 [options] <library or executable> <regexp for function/address> [<additional arguments to uprobe>]
  
  This will try to find a matching symbol in the executable and add a probe at its entry point with the given arguments.

OPTIONS
  -C  Clear uprobe_events, then exit
  -r  Probe on return rather than function entry
  -n  dry-run, only show what would be done, rather than doing it. Implies -v
  -v  verbose, show commands
  -D  demangle symbol names before checking the regexp

SEE ALSO
  https://www.kernel.org/doc/Documentation/trace/uprobetracer.txt
EOF
  exit 1
}

fail() {
  echo "$@"
  exit 1
}

log() {
  VERBOSITY=$1
  shift
  (( $VERBOSITY <= $VERBOSE )) && echo "$@"
}

CLEAR=0
DEMANGLER=cat
PROBETYPE=p:
DRYRUN=0
VERBOSE=0
[[ $# == 0 ]] && usage
while getopts "CDrnvh" flag
do
  case $flag in
    C ) CLEAR=1;;
    D ) DEMANGLER=c++filt;;
    r ) PROBETYPE=r:;;
    n ) DRYRUN=1;;
    v ) VERBOSE=$((VERBOSE + 1));;
    * ) usage;;
  esac
done
shift $(($OPTIND - 1 ))

if (( $CLEAR ))
then
  echo "" | sudo tee /sys/kernel/debug/tracing/uprobe_events
  exit 0
fi

(( $# < 2 )) && usage

DSO=$1
REGEXP=$2
shift 2
ADDITIONAL_ARGUMENTS="$@"

#echo $DEMANGLER
#echo $PROBETYPE
#echo $DRYRUN
#echo $VERBOSE
#echo $DSO
#echo $REGEXP
#echo $ADDITIONAL_ARGUMENTS

[[ -f "$DSO" ]] || fail "Not found: '$DSO'"

# Build string to pipe into uprobe_events
TODO=$( ( eu-readelf -S "$DSO" | sed 's/^\[ */[/'; eu-readelf -s "$DSO" | $DEMANGLER ) |
          grep -- "^\[\|$REGEXP" |
          grep -v UNDEF |
          awk -vADDITIONAL_ARGUMENTS="$ADDITIONAL_ARGUMENTS" \
              -vPROBETYPE=$PROBETYPE \
              -vDSO="$DSO" \
              -vPROBENAME=$(echo "$REGEXP" | tr -cd "0-9a-zA-Z") \
              '/^\[/ {
                 # First eu-readelf checks segments. We need the offset that
                 # maps adresses to file offsets
                 seg=$1
                 fileoffset[seg]=strtonum("0x"$4)-strtonum("0x"$5)
                 #print seg, fileoffset[seg]
               }
               $4 == "FUNC" {
                 addr = $2
                 seg = $7
                 $1 = ""; $2 = ""; $3 = ""; $4 = ""; $5 = ""; $6 = ""; $7 = ""
                 symbol[addr]=$0
                 segment[addr]=seg
               }
               END {
                 for ( addr in symbol )
                 {
                   #print segment[addr], fileoffset["["segment[addr]"]"], addr, symbol[addr]
                   printf "%s%s %s:0x%x %s\n", PROBETYPE, PROBENAME"_"(++num), DSO, strtonum("0x"addr)-fileoffset["["segment[addr]"]"], ADDITIONAL_ARGUMENTS
                 }
               }' )

log 1 "Would do this:
echo
$TODO | sudo tee -a /sys/kernel/debug/tracing/uprobe_events"
(( $DRYRUN )) || echo "$TODO" | sudo tee -a /sys/kernel/debug/tracing/uprobe_events
log 1 "
You now have this in uprobe_events:
$( sudo cat /sys/kernel/debug/tracing/uprobe_events )"

Re: Re: perf probe an addr without debuginfo

[Masami Hiramatsu]

On 2015/06/30 15:44, Benjamin King wrote:
> On Mon, Jun 29, 2015 at 12:00:38PM -0300, Arnaldo Carvalho de Melo wrote:
>> Em Sun, Jun 28, 2015 at 05:46:51PM +0200, Benjamin King escreveu:
>>>> Is there a trick to getting perf to probe a user-level address without
>>>> debuginfo? Eg (on Linux 4.0):
>>>> [...]
>>>> I can do this using ftrace ok, eg, "p:tick_0x583 /root/tick:0x583"
>>>> works. Thanks,
>>>
>>> Not quite what you have asked for, but you can add the probe via ftrace and
>>> then use it from perf. Probes from /sys/kernel/debug/tracing/uprobe_events
>>> will show up in 'perf list' as well.
>>
>> Masami,
>>
>> 	Is this already possible?
> 
> Hi Arnaldo,
> 
> for me this works in 3.19.0. I'm using the attached script to help me add
> probes to /sys/kernel/debug/tracing/uprobe_events which I can then see with
> 'sudo perf list' and use from the other perf subcommands. Tracing arguments is
> also working fine:

Yeah, perf probe should be able to do the same thing without debuginfo.
If not, there must be a bug. :(

> 
> --- snip ---
> $ uprobe /lib/x86_64-linux-gnu/libc.so.6 malloc %di
> $ sudo perf stat  -e uprobes:* ls
> ...
>  Performance counter stats for 'ls':
> ...
>                218      uprobes:malloc_1                                            
> $ sudo perf record -euprobes:malloc_1 ls >/dev/null 2>&1
> $ sudo perf script
>               ls 21084 [003] 10825.526815: uprobes:malloc_1: (7f772bfc2700) arg1=0x238
>               ls 21084 [003] 10825.526917: uprobes:malloc_1: (7f772bfc2700) arg1=0x238
>               ls 21084 [003] 10825.526968: uprobes:malloc_1: (7f772bfc2700) arg1=0x78
>               ls 21084 [003] 10825.527089: uprobes:malloc_1: (7f772bfc2700) arg1=0x5
>               ls 21084 [003] 10825.527132: uprobes:malloc_1: (7f772bfc2700) arg1=0x78
> ...
> --- snip ---
> 
> This is superuseful to me, so thank you all for working on perf!
> 
> I failed to manage placing proper probes in C++-Code with perf probe, so I
> hacked the script together after reading Brendans ftrace article on LWN.

Hmm, that must be a bug, so it should be fixed.

Thank you!



-- 
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: masami.hiramatsu.pt@hitachi.com

Re: perf probe an addr without debuginfo

[Brendan Gregg]

On Mon, Jun 29, 2015 at 11:44 PM, Benjamin King <benjaminking@web.de> wrote:
>
> On Mon, Jun 29, 2015 at 12:00:38PM -0300, Arnaldo Carvalho de Melo wrote:
>>
>> Em Sun, Jun 28, 2015 at 05:46:51PM +0200, Benjamin King escreveu:
>>>
>>> >Is there a trick to getting perf to probe a user-level address without
>>> >debuginfo? Eg (on Linux 4.0):
>>> >[...]
>>> >I can do this using ftrace ok, eg, "p:tick_0x583 /root/tick:0x583"
>>> >works. Thanks,
>>>
>>> Not quite what you have asked for, but you can add the probe via ftrace and
>>> then use it from perf. Probes from /sys/kernel/debug/tracing/uprobe_events
>>> will show up in 'perf list' as well.
>>
>>
>> Masami,
>>
>>         Is this already possible?
>
>
> Hi Arnaldo,
>
> for me this works in 3.19.0. I'm using the attached script to help me add
> probes to /sys/kernel/debug/tracing/uprobe_events which I can then see with
> 'sudo perf list' and use from the other perf subcommands. Tracing arguments is
> also working fine:
>
> --- snip ---
> $ uprobe /lib/x86_64-linux-gnu/libc.so.6 malloc %di

Nice - and I'd been thinking along the same lines, that it wouldn't be
hard to write a helper, and published something to do it all from
ftrace: http://www.brendangregg.com/blog/2015-06-28/linux-ftrace-uprobe.html

But I was thinking about doing this from perf as well, or instead.

[...]
>
> I failed to manage placing proper probes in C++-Code with perf probe, so I
> hacked the script together after reading Brendans ftrace article on LWN.
>
> Cheers,
>  Benjamin

Re: perf probe an addr without debuginfo

[Benjamin King]

Hi Brendan,

>Nice - and I'd been thinking along the same lines, that it wouldn't be
>hard to write a helper, and published something to do it all from
>ftrace: http://www.brendangregg.com/blog/2015-06-28/linux-ftrace-uprobe.html

Ah, missed that one, much cleaner than my attempt, thanks.

The only thing I would miss is the "match symbol by regular expression"
feature from my script. Needing to work with C++, where each Foo has new and
exciting ideas on how to implement Foo::bar(), tracing the bar's Pokemon
style (probe em all!) often helps me maintain sanity :-)

Cheers,
  Benjamin

Re: perf probe an addr without debuginfo

[Masami Hiramatsu]

On 2015/06/30 0:00, Arnaldo Carvalho de Melo wrote:
> Em Sun, Jun 28, 2015 at 05:46:51PM +0200, Benjamin King escreveu:
>> Hi Brendan
>>
>>> Is there a trick to getting perf to probe a user-level address without
>>> debuginfo? Eg (on Linux 4.0):
>>> [...]
>>> I can do this using ftrace ok, eg, "p:tick_0x583 /root/tick:0x583"
>>> works. Thanks,
>>
>> Not quite what you have asked for, but you can add the probe via ftrace and
>> then use it from perf. Probes from /sys/kernel/debug/tracing/uprobe_events
>> will show up in 'perf list' as well.
> 
> Masami,
> 
> 	Is this already possible?

Yes, it should be possible. However, I don't recommend you to
probe the address inside a function without debuginfo. You must
check the instruction boundary in that case.

> 
> - Arnaldo
> 
> P.S.: If you're not subscribed to this list, please consider it :-)

Oh, I didn't know the list! I will.
Thanks!

> 


-- 
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: masami.hiramatsu.pt@hitachi.com

Re: perf probe an addr without debuginfo

[Brendan Gregg]

[-- Attachment #1: Type: text/plain, Size: 2676 bytes --]

G'Day Masami,

On Tue, Jun 30, 2015 at 12:37 AM, Masami Hiramatsu
<masami.hiramatsu.pt@hitachi.com> wrote:
> On 2015/06/30 0:00, Arnaldo Carvalho de Melo wrote:
>> Em Sun, Jun 28, 2015 at 05:46:51PM +0200, Benjamin King escreveu:
>>> Hi Brendan
>>>
>>>> Is there a trick to getting perf to probe a user-level address without
>>>> debuginfo? Eg (on Linux 4.0):
>>>> [...]
>>>> I can do this using ftrace ok, eg, "p:tick_0x583 /root/tick:0x583"
>>>> works. Thanks,
>>>
>>> Not quite what you have asked for, but you can add the probe via ftrace and
>>> then use it from perf. Probes from /sys/kernel/debug/tracing/uprobe_events
>>> will show up in 'perf list' as well.
>>
>> Masami,
>>
>>       Is this already possible?
>
> Yes, it should be possible. However, I don't recommend you to
> probe the address inside a function without debuginfo. You must
> check the instruction boundary in that case.

Ah, because otherwise I could accidentally trace at a non-zero offsite
inside an instruction? That probably explains the only failure I saw
on uprobes on linux 4.0.

Here's the original use case:

# readelf -n /root/node/node
[...]
  stapsdt              0x00000082 NT_STAPSDT (SystemTap probe descriptors)
    Provider: node
    Name: http__client__request
    Location: 0x0000000000bf48ff, Base: 0x0000000000f22464, Semaphore:
0x0000000001243024
    Arguments: 8@%rax 8@%rdx 8@-136(%rbp) -4@-140(%rbp) 8@-72(%rbp)
8@-80(%rbp) -4@-144(%rbp)
[...]

These user SDT probes (or "USDT probes") can't currently be traced
using perf (although Hemant Kumar was working on a patch).

I can trace them via ftrace, and have already written a helper script.
But that's another story. I'm interested in doing this from perf as
well! (Without needing to create the probes via ftrace.)

I have their instruction offset above from readelf (and need to
subtract the base load address). Here's what doesn't work:

A) Tracing via the absolute address. Eg, "perf probe -x node '@+0x7f48ff'

B) Tracing via the offset from main. Eg, "perf probe -x node
'main+0x...", since perf complains I'm tracing at an offset bigger
than main.

C) Tracing via the offset from the function that contains the probe.
In this case, because the function name is too big! (Due to C++) Eg:

# ./perf probe -x /root/node/node --filter="*"
'_ZN4node26DTRACE_HTTP_CLIENT_REQUESTERKN2v820FunctionCallbackInfoINS0_5ValueEEE+0x42f'
Added new event:
  Error: Failed to add events.

I've attached a trivial patch for (C), it just increases the buffer
size in __add_probe_trace_events() from 64 to 128. It's still a bit of
a nuisance to resolve the containing function (addr2line) when I have
the absolute address...

Brendan

[-- Attachment #2: perf_probename.diff --]
[-- Type: text/plain, Size: 712 bytes --]

# diff -u linux-4.0.0+/tools/perf/util/probe-event.c linux-4.0.0+brendan/tools/perf/util/probe-event.c
--- linux-4.0.0+/tools/perf/util/probe-event.c	2015-03-31 18:13:30.001009754 +0000
+++ linux-4.0.0+brendan/tools/perf/util/probe-event.c	2015-06-30 18:20:57.914308026 +0000
@@ -2383,7 +2383,7 @@
 {
 	int i, fd, ret;
 	struct probe_trace_event *tev = NULL;
-	char buf[64];
+	char buf[128];
 	const char *event, *group;
 	struct strlist *namelist;
 	LIST_HEAD(blacklist);
@@ -2437,7 +2437,7 @@
 			group = PERFPROBE_GROUP;
 
 		/* Get an unused new event name */
-		ret = get_new_event_name(buf, 64, event,
+		ret = get_new_event_name(buf, 128, event,
 					 namelist, allow_suffix);
 		if (ret < 0)
 			break;

Re: Re: perf probe an addr without debuginfo

[Masami Hiramatsu]

On 2015/07/01 3:40, Brendan Gregg wrote:
> G'Day Masami,
> 
> On Tue, Jun 30, 2015 at 12:37 AM, Masami Hiramatsu
> <masami.hiramatsu.pt@hitachi.com> wrote:
>> On 2015/06/30 0:00, Arnaldo Carvalho de Melo wrote:
>>> Em Sun, Jun 28, 2015 at 05:46:51PM +0200, Benjamin King escreveu:
>>>> Hi Brendan
>>>>
>>>>> Is there a trick to getting perf to probe a user-level address without
>>>>> debuginfo? Eg (on Linux 4.0):
>>>>> [...]
>>>>> I can do this using ftrace ok, eg, "p:tick_0x583 /root/tick:0x583"
>>>>> works. Thanks,
>>>>
>>>> Not quite what you have asked for, but you can add the probe via ftrace and
>>>> then use it from perf. Probes from /sys/kernel/debug/tracing/uprobe_events
>>>> will show up in 'perf list' as well.
>>>
>>> Masami,
>>>
>>>       Is this already possible?
>>
>> Yes, it should be possible. However, I don't recommend you to
>> probe the address inside a function without debuginfo. You must
>> check the instruction boundary in that case.
> 
> Ah, because otherwise I could accidentally trace at a non-zero offsite
> inside an instruction? That probably explains the only failure I saw
> on uprobes on linux 4.0.
> 
> Here's the original use case:
> 
> # readelf -n /root/node/node
> [...]
>   stapsdt              0x00000082 NT_STAPSDT (SystemTap probe descriptors)
>     Provider: node
>     Name: http__client__request
>     Location: 0x0000000000bf48ff, Base: 0x0000000000f22464, Semaphore:
> 0x0000000001243024
>     Arguments: 8@%rax 8@%rdx 8@-136(%rbp) -4@-140(%rbp) 8@-72(%rbp)
> 8@-80(%rbp) -4@-144(%rbp)
> [...]
> 
> These user SDT probes (or "USDT probes") can't currently be traced
> using perf (although Hemant Kumar was working on a patch).

Yes, and I'm working on it.

> I can trace them via ftrace, and have already written a helper script.
> But that's another story. I'm interested in doing this from perf as
> well! (Without needing to create the probes via ftrace.)
> 
> I have their instruction offset above from readelf (and need to
> subtract the base load address). Here's what doesn't work:
> 
> A) Tracing via the absolute address. Eg, "perf probe -x node '@+0x7f48ff'
> 
> B) Tracing via the offset from main. Eg, "perf probe -x node
> 'main+0x...", since perf complains I'm tracing at an offset bigger
> than main.
> 
> C) Tracing via the offset from the function that contains the probe.
> In this case, because the function name is too big! (Due to C++) Eg:
> 
> # ./perf probe -x /root/node/node --filter="*"
> '_ZN4node26DTRACE_HTTP_CLIENT_REQUESTERKN2v820FunctionCallbackInfoINS0_5ValueEEE+0x42f'
> Added new event:
>   Error: Failed to add events.
> 
> I've attached a trivial patch for (C), it just increases the buffer
> size in __add_probe_trace_events() from 64 to 128. It's still a bit of
> a nuisance to resolve the containing function (addr2line) when I have
> the absolute address...

The buffer size will be fixed with strbuf in my series.

https://lkml.org/lkml/2015/5/31/84

Anyway, thank you for reporting it. I'll check why it doesn't work.

Thanks!

> 
> Brendan
> 


-- 
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Re: perf probe an addr without debuginfo

[Brendan Gregg]

G'Day Masami-san,

On Wed, Jul 1, 2015 at 4:05 AM, Masami Hiramatsu
<masami.hiramatsu.pt@hitachi.com> wrote:
> On 2015/07/01 3:40, Brendan Gregg wrote:
>> G'Day Masami,
>>
>> On Tue, Jun 30, 2015 at 12:37 AM, Masami Hiramatsu
>> <masami.hiramatsu.pt@hitachi.com> wrote:
>>> On 2015/06/30 0:00, Arnaldo Carvalho de Melo wrote:
>>>> Em Sun, Jun 28, 2015 at 05:46:51PM +0200, Benjamin King escreveu:
>>>>> Hi Brendan
>>>>>
>>>>>> Is there a trick to getting perf to probe a user-level address without
>>>>>> debuginfo? Eg (on Linux 4.0):
>>>>>> [...]
>>>>>> I can do this using ftrace ok, eg, "p:tick_0x583 /root/tick:0x583"
>>>>>> works. Thanks,
>>>>>
>>>>> Not quite what you have asked for, but you can add the probe via ftrace and
>>>>> then use it from perf. Probes from /sys/kernel/debug/tracing/uprobe_events
>>>>> will show up in 'perf list' as well.
>>>>
>>>> Masami,
>>>>
>>>>       Is this already possible?
>>>
>>> Yes, it should be possible. However, I don't recommend you to
>>> probe the address inside a function without debuginfo. You must
>>> check the instruction boundary in that case.
>>
>> Ah, because otherwise I could accidentally trace at a non-zero offsite
>> inside an instruction? That probably explains the only failure I saw
>> on uprobes on linux 4.0.
>>
>> Here's the original use case:
>>
>> # readelf -n /root/node/node
>> [...]
>>   stapsdt              0x00000082 NT_STAPSDT (SystemTap probe descriptors)
>>     Provider: node
>>     Name: http__client__request
>>     Location: 0x0000000000bf48ff, Base: 0x0000000000f22464, Semaphore:
>> 0x0000000001243024
>>     Arguments: 8@%rax 8@%rdx 8@-136(%rbp) -4@-140(%rbp) 8@-72(%rbp)
>> 8@-80(%rbp) -4@-144(%rbp)
>> [...]
>>
>> These user SDT probes (or "USDT probes") can't currently be traced
>> using perf (although Hemant Kumar was working on a patch).
>
> Yes, and I'm working on it.

Oh, can't wait! :)

If it's helpful, I did some USDT hacking with ftrace:
http://www.brendangregg.com/blog/2015-07-03/hacking-linux-usdt-ftrace.html
, although this is really a place-holder until we have a proper
implementation.

Brendan