Re: Trying to use 'perf probe' to debug perf itself

[Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>]

Hi Arnaldo,

(2015/02/25 3:49), Arnaldo Carvalho de Melo wrote:
> Hi Masami,
> 
> 	I'm adding thread refcounting in perf, and in this process I am
> investigating a problem, so thought about using perf itself to match
> thread__get with thread__put:
> 
> [root@ssdandy ~]# perf probe -x ~/bin/perf -V thread__put 
> Available variables at thread__put
>         @<thread__put+0>
>                 struct thread*  thread
> [root@ssdandy ~]# perf probe -x ~/bin/perf -V thread__get
> Available variables at thread__get
>         @<thread__get+0>
>                 struct thread*  thread
> [root@ssdandy ~]#
> 
> cool, so I thought it would be just a matter of asking to put the probes
> and get the value of 'thread', then match things, but:
> 
> 
> [root@ssdandy ~]# perf probe -v ~/bin/perf thread__put thread
> probe-definition(0): thread__put thread 
> symbol:thread__put file:(null) line:0 offset:0 return:0 lazy:(null)
> parsing arg: thread into thread
> 1 arguments
> Open Debuginfo file: /root/bin/perf
> Try to find probe point from debuginfo.
> Probe point found: thread__put+0
> Searching 'thread' variable in context.
> Converting variable thread into trace event.
> thread type is (null).
> Found 1 probe_trace_events.
> Opening /sys/kernel/debug/tracing/uprobe_events write=1
> Added new event:
> Writing event: p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64
> Failed to write event: Invalid argument
>   Error: Failed to add events. Reason: Invalid argument (Code: -22)
> [root@ssdandy ~]#
> 
> Not possible :-\ 

Hmm, strange. Could you tell me the version of your kernel?
It seems that the kernel newer than 3.14 supports uprobes with
memory dereference (e.g. -32(%sp) )feature.
(it was actually introduced by 5baaa59e, and git-describe told
 it was v3.13-rc4-22-g5baaa59)

And also, could you try to write the result command to uprobe_event as
below?

# echo "p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64" >> \
 /sys/kernel/debug/tracing/uprobe_events


> 
> please let me know if you need some file, here is the readelf -wi for
> those two routines:

This should not be the problem of dwarf-analysis. It seems kernel-side
(uprobe) problem.

Thank you,

> 
> # readelf -wi ~/bin/perf
> <SNIP>
>  <1><e25f1>: Abbrev Number: 35 (DW_TAG_subprogram)
>     <e25f2>   DW_AT_external    : 1     
>     <e25f2>   DW_AT_name        : (indirect string, offset: 0x4c5c3): thread__get       
>     <e25f6>   DW_AT_decl_file   : 6     
>     <e25f7>   DW_AT_decl_line   : 84    
>     <e25f8>   DW_AT_prototyped  : 1     
>     <e25f8>   DW_AT_type        : <0xe15cc>     
>     <e25fc>   DW_AT_low_pc      : 0x4d038c      
>     <e2604>   DW_AT_high_pc     : 0x46  
>     <e260c>   DW_AT_frame_base  : 1 byte block: 9c      (DW_OP_call_frame_cfa)
>     <e260e>   DW_AT_GNU_all_tail_call_sites: 1  
>     <e260e>   DW_AT_sibling     : <0xe2621>     
>  <2><e2612>: Abbrev Number: 32 (DW_TAG_formal_parameter)
>     <e2613>   DW_AT_name        : (indirect string, offset: 0x43dc7): thread    
>     <e2617>   DW_AT_decl_file   : 6     
>     <e2618>   DW_AT_decl_line   : 84    
>     <e2619>   DW_AT_type        : <0xe15cc>     
>     <e261d>   DW_AT_location    : 2 byte block: 91 58   (DW_OP_fbreg: -40)
>  <2><e2620>: Abbrev Number: 0
>  <1><e2621>: Abbrev Number: 39 (DW_TAG_subprogram)
>     <e2622>   DW_AT_external    : 1     
>     <e2622>   DW_AT_name        : (indirect string, offset: 0x4c699): thread__put       
>     <e2626>   DW_AT_decl_file   : 6     
>     <e2627>   DW_AT_decl_line   : 90    
>     <e2628>   DW_AT_prototyped  : 1     
>     <e2628>   DW_AT_low_pc      : 0x4d03d2      
>     <e2630>   DW_AT_high_pc     : 0x6c  
>     <e2638>   DW_AT_frame_base  : 1 byte block: 9c      (DW_OP_call_frame_cfa)
>     <e263a>   DW_AT_GNU_all_tail_call_sites: 1  
>     <e263a>   DW_AT_sibling     : <0xe264d>     
>  <2><e263e>: Abbrev Number: 32 (DW_TAG_formal_parameter)
>     <e263f>   DW_AT_name        : (indirect string, offset: 0x43dc7): thread    
>     <e2643>   DW_AT_decl_file   : 6     
>     <e2644>   DW_AT_decl_line   : 90    
>     <e2645>   DW_AT_type        : <0xe15cc>     
> <SNIP>
> 
> Will see what I can do with just:
> 
> [root@ssdandy ~]# perf probe ~/bin/perf thread__get
> Added new event:
>   probe_perf:thread__get (on thread__get in /root/bin/perf)
> 
> You can now use it in all perf tools, such as:
> 
> 	perf record -e probe_perf:thread__get -aR sleep 1
> 
> [root@ssdandy ~]# perf probe ~/bin/perf thread__put
> Added new event:
>   probe_perf:thread__put (on thread__put in /root/bin/perf)
> 
> You can now use it in all perf tools, such as:
> 
> 	perf record -e probe_perf:thread__put -aR sleep 1
> 
> [root@ssdandy ~]# perf list probe*:*
> 
> List of pre-defined events (to be used in -e):
> 
>   probe_perf:thread__put                             [Tracepoint event]
>   probe_perf:thread__get                             [Tracepoint event]
> [root@ssdandy ~]# perf probe -l
>   probe_perf:thread__get (on thread__get@util/thread.c in /root/bin/perf)
>   probe_perf:thread__put (on thread__put@util/thread.c in /root/bin/perf)
> [root@ssdandy ~]# 
> 
> - Arnaldo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com