Trying to use 'perf probe' to debug perf itself

Trying to use 'perf probe' to debug perf itself

[Arnaldo Carvalho de Melo]

Hi Masami,

	I'm adding thread refcounting in perf, and in this process I am
investigating a problem, so thought about using perf itself to match
thread__get with thread__put:

[root@ssdandy ~]# perf probe -x ~/bin/perf -V thread__put 
Available variables at thread__put
        @<thread__put+0>
                struct thread*  thread
[root@ssdandy ~]# perf probe -x ~/bin/perf -V thread__get
Available variables at thread__get
        @<thread__get+0>
                struct thread*  thread
[root@ssdandy ~]#

cool, so I thought it would be just a matter of asking to put the probes
and get the value of 'thread', then match things, but:


[root@ssdandy ~]# perf probe -v ~/bin/perf thread__put thread
probe-definition(0): thread__put thread 
symbol:thread__put file:(null) line:0 offset:0 return:0 lazy:(null)
parsing arg: thread into thread
1 arguments
Open Debuginfo file: /root/bin/perf
Try to find probe point from debuginfo.
Probe point found: thread__put+0
Searching 'thread' variable in context.
Converting variable thread into trace event.
thread type is (null).
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/uprobe_events write=1
Added new event:
Writing event: p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64
Failed to write event: Invalid argument
  Error: Failed to add events. Reason: Invalid argument (Code: -22)
[root@ssdandy ~]#

Not possible :-\ 

please let me know if you need some file, here is the readelf -wi for
those two routines:

# readelf -wi ~/bin/perf
<SNIP>
 <1><e25f1>: Abbrev Number: 35 (DW_TAG_subprogram)
    <e25f2>   DW_AT_external    : 1     
    <e25f2>   DW_AT_name        : (indirect string, offset: 0x4c5c3): thread__get       
    <e25f6>   DW_AT_decl_file   : 6     
    <e25f7>   DW_AT_decl_line   : 84    
    <e25f8>   DW_AT_prototyped  : 1     
    <e25f8>   DW_AT_type        : <0xe15cc>     
    <e25fc>   DW_AT_low_pc      : 0x4d038c      
    <e2604>   DW_AT_high_pc     : 0x46  
    <e260c>   DW_AT_frame_base  : 1 byte block: 9c      (DW_OP_call_frame_cfa)
    <e260e>   DW_AT_GNU_all_tail_call_sites: 1  
    <e260e>   DW_AT_sibling     : <0xe2621>     
 <2><e2612>: Abbrev Number: 32 (DW_TAG_formal_parameter)
    <e2613>   DW_AT_name        : (indirect string, offset: 0x43dc7): thread    
    <e2617>   DW_AT_decl_file   : 6     
    <e2618>   DW_AT_decl_line   : 84    
    <e2619>   DW_AT_type        : <0xe15cc>     
    <e261d>   DW_AT_location    : 2 byte block: 91 58   (DW_OP_fbreg: -40)
 <2><e2620>: Abbrev Number: 0
 <1><e2621>: Abbrev Number: 39 (DW_TAG_subprogram)
    <e2622>   DW_AT_external    : 1     
    <e2622>   DW_AT_name        : (indirect string, offset: 0x4c699): thread__put       
    <e2626>   DW_AT_decl_file   : 6     
    <e2627>   DW_AT_decl_line   : 90    
    <e2628>   DW_AT_prototyped  : 1     
    <e2628>   DW_AT_low_pc      : 0x4d03d2      
    <e2630>   DW_AT_high_pc     : 0x6c  
    <e2638>   DW_AT_frame_base  : 1 byte block: 9c      (DW_OP_call_frame_cfa)
    <e263a>   DW_AT_GNU_all_tail_call_sites: 1  
    <e263a>   DW_AT_sibling     : <0xe264d>     
 <2><e263e>: Abbrev Number: 32 (DW_TAG_formal_parameter)
    <e263f>   DW_AT_name        : (indirect string, offset: 0x43dc7): thread    
    <e2643>   DW_AT_decl_file   : 6     
    <e2644>   DW_AT_decl_line   : 90    
    <e2645>   DW_AT_type        : <0xe15cc>     
<SNIP>

Will see what I can do with just:

[root@ssdandy ~]# perf probe ~/bin/perf thread__get
Added new event:
  probe_perf:thread__get (on thread__get in /root/bin/perf)

You can now use it in all perf tools, such as:

	perf record -e probe_perf:thread__get -aR sleep 1

[root@ssdandy ~]# perf probe ~/bin/perf thread__put
Added new event:
  probe_perf:thread__put (on thread__put in /root/bin/perf)

You can now use it in all perf tools, such as:

	perf record -e probe_perf:thread__put -aR sleep 1

[root@ssdandy ~]# perf list probe*:*

List of pre-defined events (to be used in -e):

  probe_perf:thread__put                             [Tracepoint event]
  probe_perf:thread__get                             [Tracepoint event]
[root@ssdandy ~]# perf probe -l
  probe_perf:thread__get (on thread__get@util/thread.c in /root/bin/perf)
  probe_perf:thread__put (on thread__put@util/thread.c in /root/bin/perf)
[root@ssdandy ~]# 

- Arnaldo

Re: Trying to use 'perf probe' to debug perf itself

[Masami Hiramatsu]

Hi Arnaldo,

(2015/02/25 3:49), Arnaldo Carvalho de Melo wrote:
> Hi Masami,
> 
> 	I'm adding thread refcounting in perf, and in this process I am
> investigating a problem, so thought about using perf itself to match
> thread__get with thread__put:
> 
> [root@ssdandy ~]# perf probe -x ~/bin/perf -V thread__put 
> Available variables at thread__put
>         @<thread__put+0>
>                 struct thread*  thread
> [root@ssdandy ~]# perf probe -x ~/bin/perf -V thread__get
> Available variables at thread__get
>         @<thread__get+0>
>                 struct thread*  thread
> [root@ssdandy ~]#
> 
> cool, so I thought it would be just a matter of asking to put the probes
> and get the value of 'thread', then match things, but:
> 
> 
> [root@ssdandy ~]# perf probe -v ~/bin/perf thread__put thread
> probe-definition(0): thread__put thread 
> symbol:thread__put file:(null) line:0 offset:0 return:0 lazy:(null)
> parsing arg: thread into thread
> 1 arguments
> Open Debuginfo file: /root/bin/perf
> Try to find probe point from debuginfo.
> Probe point found: thread__put+0
> Searching 'thread' variable in context.
> Converting variable thread into trace event.
> thread type is (null).
> Found 1 probe_trace_events.
> Opening /sys/kernel/debug/tracing/uprobe_events write=1
> Added new event:
> Writing event: p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64
> Failed to write event: Invalid argument
>   Error: Failed to add events. Reason: Invalid argument (Code: -22)
> [root@ssdandy ~]#
> 
> Not possible :-\ 

Hmm, strange. Could you tell me the version of your kernel?
It seems that the kernel newer than 3.14 supports uprobes with
memory dereference (e.g. -32(%sp) )feature.
(it was actually introduced by 5baaa59e, and git-describe told
 it was v3.13-rc4-22-g5baaa59)

And also, could you try to write the result command to uprobe_event as
below?

# echo "p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64" >> \
 /sys/kernel/debug/tracing/uprobe_events


> 
> please let me know if you need some file, here is the readelf -wi for
> those two routines:

This should not be the problem of dwarf-analysis. It seems kernel-side
(uprobe) problem.

Thank you,

> 
> # readelf -wi ~/bin/perf
> <SNIP>
>  <1><e25f1>: Abbrev Number: 35 (DW_TAG_subprogram)
>     <e25f2>   DW_AT_external    : 1     
>     <e25f2>   DW_AT_name        : (indirect string, offset: 0x4c5c3): thread__get       
>     <e25f6>   DW_AT_decl_file   : 6     
>     <e25f7>   DW_AT_decl_line   : 84    
>     <e25f8>   DW_AT_prototyped  : 1     
>     <e25f8>   DW_AT_type        : <0xe15cc>     
>     <e25fc>   DW_AT_low_pc      : 0x4d038c      
>     <e2604>   DW_AT_high_pc     : 0x46  
>     <e260c>   DW_AT_frame_base  : 1 byte block: 9c      (DW_OP_call_frame_cfa)
>     <e260e>   DW_AT_GNU_all_tail_call_sites: 1  
>     <e260e>   DW_AT_sibling     : <0xe2621>     
>  <2><e2612>: Abbrev Number: 32 (DW_TAG_formal_parameter)
>     <e2613>   DW_AT_name        : (indirect string, offset: 0x43dc7): thread    
>     <e2617>   DW_AT_decl_file   : 6     
>     <e2618>   DW_AT_decl_line   : 84    
>     <e2619>   DW_AT_type        : <0xe15cc>     
>     <e261d>   DW_AT_location    : 2 byte block: 91 58   (DW_OP_fbreg: -40)
>  <2><e2620>: Abbrev Number: 0
>  <1><e2621>: Abbrev Number: 39 (DW_TAG_subprogram)
>     <e2622>   DW_AT_external    : 1     
>     <e2622>   DW_AT_name        : (indirect string, offset: 0x4c699): thread__put       
>     <e2626>   DW_AT_decl_file   : 6     
>     <e2627>   DW_AT_decl_line   : 90    
>     <e2628>   DW_AT_prototyped  : 1     
>     <e2628>   DW_AT_low_pc      : 0x4d03d2      
>     <e2630>   DW_AT_high_pc     : 0x6c  
>     <e2638>   DW_AT_frame_base  : 1 byte block: 9c      (DW_OP_call_frame_cfa)
>     <e263a>   DW_AT_GNU_all_tail_call_sites: 1  
>     <e263a>   DW_AT_sibling     : <0xe264d>     
>  <2><e263e>: Abbrev Number: 32 (DW_TAG_formal_parameter)
>     <e263f>   DW_AT_name        : (indirect string, offset: 0x43dc7): thread    
>     <e2643>   DW_AT_decl_file   : 6     
>     <e2644>   DW_AT_decl_line   : 90    
>     <e2645>   DW_AT_type        : <0xe15cc>     
> <SNIP>
> 
> Will see what I can do with just:
> 
> [root@ssdandy ~]# perf probe ~/bin/perf thread__get
> Added new event:
>   probe_perf:thread__get (on thread__get in /root/bin/perf)
> 
> You can now use it in all perf tools, such as:
> 
> 	perf record -e probe_perf:thread__get -aR sleep 1
> 
> [root@ssdandy ~]# perf probe ~/bin/perf thread__put
> Added new event:
>   probe_perf:thread__put (on thread__put in /root/bin/perf)
> 
> You can now use it in all perf tools, such as:
> 
> 	perf record -e probe_perf:thread__put -aR sleep 1
> 
> [root@ssdandy ~]# perf list probe*:*
> 
> List of pre-defined events (to be used in -e):
> 
>   probe_perf:thread__put                             [Tracepoint event]
>   probe_perf:thread__get                             [Tracepoint event]
> [root@ssdandy ~]# perf probe -l
>   probe_perf:thread__get (on thread__get@util/thread.c in /root/bin/perf)
>   probe_perf:thread__put (on thread__put@util/thread.c in /root/bin/perf)
> [root@ssdandy ~]# 
> 
> - Arnaldo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Trying to use 'perf probe' to debug perf itself

[Arnaldo Carvalho de Melo]

Em Wed, Feb 25, 2015 at 11:53:16AM +0900, Masami Hiramatsu escreveu:
> (2015/02/25 3:49), Arnaldo Carvalho de Melo wrote:
> > Available variables at thread__get
> >         @<thread__get+0>
> >                 struct thread*  thread
> > [root@ssdandy ~]#

> > cool, so I thought it would be just a matter of asking to put the probes
> > and get the value of 'thread', then match things, but:

> > [root@ssdandy ~]# perf probe -v ~/bin/perf thread__put thread
<SNIP>
> > Writing event: p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64
> > Failed to write event: Invalid argument
> >   Error: Failed to add events. Reason: Invalid argument (Code: -22)
> > [root@ssdandy ~]#

> > Not possible :-\ 
 
> Hmm, strange. Could you tell me the version of your kernel?

[root@ssdandy ~]# uname -r
3.10.0-210.el7.x86_64

> It seems that the kernel newer than 3.14 supports uprobes with
> memory dereference (e.g. -32(%sp) )feature.

Right, that must be the case, will test, but then, would it be possible
for the kernel, in such cases, return something line EOPNOTSUP?

I will try to figure out a better error message on the tooling side,
something like:

. Realize we're asking for memory dereference in uprobes
. If it fails with EINVAL, check the kernel version and say something
  like:

Please upgrade your kernel to at least x.y.z to have access to feature
FOO_BAR.

- Arnaldo

> (it was actually introduced by 5baaa59e, and git-describe told
>  it was v3.13-rc4-22-g5baaa59)
> 
> And also, could you try to write the result command to uprobe_event as
> below?
> 
> # echo "p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64" >> \
>  /sys/kernel/debug/tracing/uprobe_events

Well, I'll try that if it fails after I upgrade to 3.14.

> > please let me know if you need some file, here is the readelf -wi for
> > those two routines:
> 
> This should not be the problem of dwarf-analysis. It seems kernel-side
> (uprobe) problem.

Thanks a lot!

- Arnaldo

Re: Trying to use 'perf probe' to debug perf itself

[Masami Hiramatsu]

(2015/02/25 22:25), Arnaldo Carvalho de Melo wrote:
> Em Wed, Feb 25, 2015 at 11:53:16AM +0900, Masami Hiramatsu escreveu:
>> (2015/02/25 3:49), Arnaldo Carvalho de Melo wrote:
>>> Available variables at thread__get
>>>         @<thread__get+0>
>>>                 struct thread*  thread
>>> [root@ssdandy ~]#
> 
>>> cool, so I thought it would be just a matter of asking to put the probes
>>> and get the value of 'thread', then match things, but:
> 
>>> [root@ssdandy ~]# perf probe -v ~/bin/perf thread__put thread
> <SNIP>
>>> Writing event: p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64
>>> Failed to write event: Invalid argument
>>>   Error: Failed to add events. Reason: Invalid argument (Code: -22)
>>> [root@ssdandy ~]#
> 
>>> Not possible :-\ 
>  
>> Hmm, strange. Could you tell me the version of your kernel?
> 
> [root@ssdandy ~]# uname -r
> 3.10.0-210.el7.x86_64
> 
>> It seems that the kernel newer than 3.14 supports uprobes with
>> memory dereference (e.g. -32(%sp) )feature.
> 
> Right, that must be the case, will test, but then, would it be possible
> for the kernel, in such cases, return something line EOPNOTSUP?

Yeah, but for now, it is already supported in kernel.
Of cause we can try to test feature with adding temporary event
from perftools too.

> I will try to figure out a better error message on the tooling side,
> something like:
> 
> . Realize we're asking for memory dereference in uprobes
> . If it fails with EINVAL, check the kernel version and say something
>   like:
> 
> Please upgrade your kernel to at least x.y.z to have access to feature
> FOO_BAR.

OK, it may be worth for users (I'm not sure RHEL can update their kernel
to include that enhancement)

Thank you,

> 
> - Arnaldo
> 
>> (it was actually introduced by 5baaa59e, and git-describe told
>>  it was v3.13-rc4-22-g5baaa59)
>>
>> And also, could you try to write the result command to uprobe_event as
>> below?
>>
>> # echo "p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64" >> \
>>  /sys/kernel/debug/tracing/uprobe_events
> 
> Well, I'll try that if it fails after I upgrade to 3.14.
> 
>>> please let me know if you need some file, here is the readelf -wi for
>>> those two routines:
>>
>> This should not be the problem of dwarf-analysis. It seems kernel-side
>> (uprobe) problem.
> 
> Thanks a lot!
> 
> - Arnaldo
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Trying to use 'perf probe' to debug perf itself

[Arnaldo Carvalho de Melo]

Em Thu, Feb 26, 2015 at 01:57:42PM +0900, Masami Hiramatsu escreveu:
> (2015/02/25 22:25), Arnaldo Carvalho de Melo wrote:
> > Em Wed, Feb 25, 2015 at 11:53:16AM +0900, Masami Hiramatsu escreveu:
> >> (2015/02/25 3:49), Arnaldo Carvalho de Melo wrote:
> >> Hmm, strange. Could you tell me the version of your kernel?

> > [root@ssdandy ~]# uname -r
> > 3.10.0-210.el7.x86_64

> >> It seems that the kernel newer than 3.14 supports uprobes with
> >> memory dereference (e.g. -32(%sp) )feature.

> > Right, that must be the case, will test, but then, would it be possible
> > for the kernel, in such cases, return something line EOPNOTSUP?
 
> Yeah, but for now, it is already supported in kernel.
> Of cause we can try to test feature with adding temporary event
> from perftools too.

Yeah, capability querying is done in several places in tools/perf/
already, like when checking if we have perf_event_attr enabled features,
such as sample_id_all, etc.

Sometimes we can fallback to alternative mechanisms, sometimes we just
need to provide a less cryptic message.
 
> > I will try to figure out a better error message on the tooling side,
> > something like:

> > . Realize we're asking for memory dereference in uprobes
> > . If it fails with EINVAL, check the kernel version and say something
> >   like:
> > 
> > Please upgrade your kernel to at least x.y.z to have access to feature
> > FOO_BAR.
 
> OK, it may be worth for users (I'm not sure RHEL can update their kernel
> to include that enhancement)

That may be possible, will check with Jiri if we can include that in a
future backport.

Talking about these tools in conferences will make more people try the
things mentioned, and as we talk about the latest and greatest, we need
to try harder to handle error paths in a better way, providing less
cryptic messages.

Rebuilding a newer kernel now to retry all this, thanks!

- Arnaldo

[PATCH perf/core ] perf-probe: Warn if given uprobe event accesses memory on older kernel

[Masami Hiramatsu]

Warn if given uprobe event accesses memory on older kernel.
Until 3.14, uprobe event only supports accessing registers
so this warns to upgrade kernel if uprobe-event returns
-EINVAL and an argument of the event accesses memory ($stack,
@+offset, and +|-offs() symtax).

With this patch (on 3.10.0-123.13.2.el7.x86_64);
  -----
  # ./perf probe -x ./perf warn_uprobe_event_compat stack=-0\(%sp\)
  Added new event:
  Failed to write event: Invalid argument
  Please upgrade your kernel to at least 3.14 to have access to feature -0(%sp)
    Error: Failed to add events.
  -----

Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
 tools/perf/util/probe-event.c |   23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 7c0e765..1c570c2fa7 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -2199,6 +2199,27 @@ static int get_new_event_name(char *buf, size_t len, const char *base,
 	return ret;
 }
 
+/* Warn if the current kernel's uprobe implementation is old */
+static void warn_uprobe_event_compat(struct probe_trace_event *tev)
+{
+	int i;
+	char *buf = synthesize_probe_trace_command(tev);
+
+	/* Old uprobe event doesn't support memory dereference */
+	if (!tev->uprobes || tev->nargs == 0 || !buf)
+		goto out;
+
+	for (i = 0; i < tev->nargs; i++)
+		if (strglobmatch(tev->args[i].value, "[$@+-]*")) {
+			pr_warning("Please upgrade your kernel to at least "
+				   "3.14 to have access to feature %s\n",
+				   tev->args[i].value);
+			break;
+		}
+out:
+	free(buf);
+}
+
 static int __add_probe_trace_events(struct perf_probe_event *pev,
 				     struct probe_trace_event *tevs,
 				     int ntevs, bool allow_suffix)
@@ -2295,6 +2316,8 @@ static int __add_probe_trace_events(struct perf_probe_event *pev,
 		 */
 		allow_suffix = true;
 	}
+	if (ret == -EINVAL && pev->uprobes)
+		warn_uprobe_event_compat(tev);
 
 	/* Note that it is possible to skip all events because of blacklist */
 	if (ret >= 0 && tev->event) {

Re: [PATCH perf/core ] perf-probe: Warn if given uprobe event accesses memory on older kernel

[Masami Hiramatsu]

(2015/02/28 11:53), Masami Hiramatsu wrote:
> Warn if given uprobe event accesses memory on older kernel.
> Until 3.14, uprobe event only supports accessing registers
> so this warns to upgrade kernel if uprobe-event returns
> -EINVAL and an argument of the event accesses memory ($stack,
> @+offset, and +|-offs() symtax).
> 
> With this patch (on 3.10.0-123.13.2.el7.x86_64);
>   -----
>   # ./perf probe -x ./perf warn_uprobe_event_compat stack=-0\(%sp\)
>   Added new event:
>   Failed to write event: Invalid argument
>   Please upgrade your kernel to at least 3.14 to have access to feature -0(%sp)
>     Error: Failed to add events.
>   -----
> 

Oops,

Suggested-by: Arnaldo Carvalho de Melo <acme@kernel.org>

Thanks,

> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
> ---
>  tools/perf/util/probe-event.c |   23 +++++++++++++++++++++++
>  1 file changed, 23 insertions(+)
> 
> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> index 7c0e765..1c570c2fa7 100644
> --- a/tools/perf/util/probe-event.c
> +++ b/tools/perf/util/probe-event.c
> @@ -2199,6 +2199,27 @@ static int get_new_event_name(char *buf, size_t len, const char *base,
>  	return ret;
>  }
>  
> +/* Warn if the current kernel's uprobe implementation is old */
> +static void warn_uprobe_event_compat(struct probe_trace_event *tev)
> +{
> +	int i;
> +	char *buf = synthesize_probe_trace_command(tev);
> +
> +	/* Old uprobe event doesn't support memory dereference */
> +	if (!tev->uprobes || tev->nargs == 0 || !buf)
> +		goto out;
> +
> +	for (i = 0; i < tev->nargs; i++)
> +		if (strglobmatch(tev->args[i].value, "[$@+-]*")) {
> +			pr_warning("Please upgrade your kernel to at least "
> +				   "3.14 to have access to feature %s\n",
> +				   tev->args[i].value);
> +			break;
> +		}
> +out:
> +	free(buf);
> +}
> +
>  static int __add_probe_trace_events(struct perf_probe_event *pev,
>  				     struct probe_trace_event *tevs,
>  				     int ntevs, bool allow_suffix)
> @@ -2295,6 +2316,8 @@ static int __add_probe_trace_events(struct perf_probe_event *pev,
>  		 */
>  		allow_suffix = true;
>  	}
> +	if (ret == -EINVAL && pev->uprobes)
> +		warn_uprobe_event_compat(tev);
>  
>  	/* Note that it is possible to skip all events because of blacklist */
>  	if (ret >= 0 && tev->event) {
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 
> 
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [PATCH perf/core ] perf-probe: Warn if given uprobe event accesses memory on older kernel

[Arnaldo Carvalho de Melo]

Em Sat, Feb 28, 2015 at 12:21:05PM +0900, Masami Hiramatsu escreveu:
> (2015/02/28 11:53), Masami Hiramatsu wrote:
> > Warn if given uprobe event accesses memory on older kernel.
> > Until 3.14, uprobe event only supports accessing registers
> > so this warns to upgrade kernel if uprobe-event returns
> > -EINVAL and an argument of the event accesses memory ($stack,
> > @+offset, and +|-offs() symtax).
> > 
> > With this patch (on 3.10.0-123.13.2.el7.x86_64);
> >   -----
> >   # ./perf probe -x ./perf warn_uprobe_event_compat stack=-0\(%sp\)
> >   Added new event:
> >   Failed to write event: Invalid argument
> >   Please upgrade your kernel to at least 3.14 to have access to feature -0(%sp)
> >     Error: Failed to add events.
> >   -----
> > 
> 
> Oops,
> 
> Suggested-by: Arnaldo Carvalho de Melo <acme@kernel.org>

:-)

Thanks for considering my suggestion and coming up with a patch so
quickly!

I'll test it and merge,

- Arnaldo
 
> Thanks,
> 
> > Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
> > ---
> >  tools/perf/util/probe-event.c |   23 +++++++++++++++++++++++
> >  1 file changed, 23 insertions(+)
> > 
> > diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> > index 7c0e765..1c570c2fa7 100644
> > --- a/tools/perf/util/probe-event.c
> > +++ b/tools/perf/util/probe-event.c
> > @@ -2199,6 +2199,27 @@ static int get_new_event_name(char *buf, size_t len, const char *base,
> >  	return ret;
> >  }
> >  
> > +/* Warn if the current kernel's uprobe implementation is old */
> > +static void warn_uprobe_event_compat(struct probe_trace_event *tev)
> > +{
> > +	int i;
> > +	char *buf = synthesize_probe_trace_command(tev);
> > +
> > +	/* Old uprobe event doesn't support memory dereference */
> > +	if (!tev->uprobes || tev->nargs == 0 || !buf)
> > +		goto out;
> > +
> > +	for (i = 0; i < tev->nargs; i++)
> > +		if (strglobmatch(tev->args[i].value, "[$@+-]*")) {
> > +			pr_warning("Please upgrade your kernel to at least "
> > +				   "3.14 to have access to feature %s\n",
> > +				   tev->args[i].value);
> > +			break;
> > +		}
> > +out:
> > +	free(buf);
> > +}
> > +
> >  static int __add_probe_trace_events(struct perf_probe_event *pev,
> >  				     struct probe_trace_event *tevs,
> >  				     int ntevs, bool allow_suffix)
> > @@ -2295,6 +2316,8 @@ static int __add_probe_trace_events(struct perf_probe_event *pev,
> >  		 */
> >  		allow_suffix = true;
> >  	}
> > +	if (ret == -EINVAL && pev->uprobes)
> > +		warn_uprobe_event_compat(tev);
> >  
> >  	/* Note that it is possible to skip all events because of blacklist */
> >  	if (ret >= 0 && tev->event) {
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> > 
> > 
> > 
> 
> 
> -- 
> Masami HIRAMATSU
> Software Platform Research Dept. Linux Technology Research Center
> Hitachi, Ltd., Yokohama Research Laboratory
> E-mail: masami.hiramatsu.pt@hitachi.com
> 

[tip:perf/core] perf probe: Warn if given uprobe event accesses memory on older kernel

[tip-bot for Masami Hiramatsu]

Commit-ID:  79702f614187f652a814061e8f5875ddcc9e732d
Gitweb:     http://git.kernel.org/tip/79702f614187f652a814061e8f5875ddcc9e732d
Author:     Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
AuthorDate: Sat, 28 Feb 2015 11:53:29 +0900
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Mon, 2 Mar 2015 12:27:43 -0300

perf probe: Warn if given uprobe event accesses memory on older kernel

Warn if given uprobe event accesses memory on older kernel.

Until 3.14, uprobe event only supports accessing registers so this warns
to upgrade kernel if uprobe-event returns -EINVAL and an argument of the
event accesses memory ($stack, @+offset, and +|-offs() symtax).

With this patch (on 3.10.0-123.13.2.el7.x86_64);
  -----
  # ./perf probe -x ./perf warn_uprobe_event_compat stack=-0\(%sp\)
  Added new event:
  Failed to write event: Invalid argument
  Please upgrade your kernel to at least 3.14 to have access to feature -0(%sp)
    Error: Failed to add events.
  -----

Suggested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/20150228025329.32106.70581.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/probe-event.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 7c0e765..1c570c2 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -2199,6 +2199,27 @@ static int get_new_event_name(char *buf, size_t len, const char *base,
 	return ret;
 }
 
+/* Warn if the current kernel's uprobe implementation is old */
+static void warn_uprobe_event_compat(struct probe_trace_event *tev)
+{
+	int i;
+	char *buf = synthesize_probe_trace_command(tev);
+
+	/* Old uprobe event doesn't support memory dereference */
+	if (!tev->uprobes || tev->nargs == 0 || !buf)
+		goto out;
+
+	for (i = 0; i < tev->nargs; i++)
+		if (strglobmatch(tev->args[i].value, "[$@+-]*")) {
+			pr_warning("Please upgrade your kernel to at least "
+				   "3.14 to have access to feature %s\n",
+				   tev->args[i].value);
+			break;
+		}
+out:
+	free(buf);
+}
+
 static int __add_probe_trace_events(struct perf_probe_event *pev,
 				     struct probe_trace_event *tevs,
 				     int ntevs, bool allow_suffix)
@@ -2295,6 +2316,8 @@ static int __add_probe_trace_events(struct perf_probe_event *pev,
 		 */
 		allow_suffix = true;
 	}
+	if (ret == -EINVAL && pev->uprobes)
+		warn_uprobe_event_compat(tev);
 
 	/* Note that it is possible to skip all events because of blacklist */
 	if (ret >= 0 && tev->event) {