[RFE] perf probe: Support globbing/regex in -a

[RFE] perf probe: Support globbing/regex in -a

[Arnaldo Carvalho de Melo]

Hi Masami,

	In tools/perf constructors are named <CLASS>__new, and right now I want
to know hoe many of each <CLASS> objects are being allocated, so I expected to
be able to do:

[root@zoo ~]# perf probe -x ~/bin/perf -a "*__new"
Probe point '*__new' not found.
  Error: Failed to add events.
[root@zoo ~]#

And get the same result that I get from:

[root@zoo ~]# perf probe -x ~/bin/perf `perf probe -x ~/bin/perf -F | grep __new | sed 's/^/-a /g' | sort -u`
Added new event:
  probe_perf:call_path__new (on call_path__new in /home/acme/bin/perf)

You can now use it in all perf tools, such as:

	perf record -e probe_perf:call_path__new -aR sleep 1

Added new event:
  probe_perf:call_path_root__new (on call_path_root__new in /home/acme/bin/perf)

You can now use it in all perf tools, such as:

	perf record -e probe_perf:call_path_root__new -aR sleep 1

Added new event:
  probe_perf:call_return_processor__new (on call_return_processor__new in /home/acme/bin/perf)


<SNIP>

I.e. that I end up with:

[root@zoo ~]# perf probe -l  2>&1 | grep probe_perf:.*__new
  probe_perf:call_path__new (on call_path__new@util/thread-stack.c in /home/acme/bin/perf)
  probe_perf:call_path_root__new (on call_path_root__new@util/thread-stack.c in /home/acme/bin/perf)
  probe_perf:call_return_processor__new (on call_return_processor__new@util/thread-stack.c in /home/acme/bin/perf)
  probe_perf:comm__new (on comm__new@util/comm.c in /home/acme/bin/perf)
  probe_perf:cpu_map__new (on cpu_map__new@util/cpumap.c in /home/acme/bin/perf)
  probe_perf:debuginfo__new (on debuginfo__new@util/probe-finder.c in /home/acme/bin/perf)
  probe_perf:disasm_line__new (on disasm_line__new@util/annotate.c in /home/acme/bin/perf)
  probe_perf:dso__new  (on dso__new@util/dso.c in /home/acme/bin/perf)
  probe_perf:dso__new_map (on dso__new_map@util/dso.c in /home/acme/bin/perf)
  probe_perf:fdarray__new (on fdarray__new@fd/array.c in /home/acme/bin/perf)
  probe_perf:hist_browser__new (on hist_browser__new@ui/browsers/hists.c in /home/acme/bin/perf)
<SNIP>
  probe_perf:thread_map__new_by_uid (on thread_map__new_by_uid@util/thread_map.c in /home/acme/bin/perf)
  probe_perf:thread_map__new_dummy (on thread_map__new_dummy@util/thread_map.c in /home/acme/bin/perf)
  probe_perf:thread_map__new_str (on thread_map__new_str@util/thread_map.c in /home/acme/bin/perf)
[root@zoo ~]#

Some more goodies to have, please consider better compressing the output of successfully installed probes,
Something like:

[root@zoo ~]# perf probe -x ~/bin/perf `perf probe -x ~/bin/perf -F  | grep __new | sed 's/^/-a /g' | sort -u`
Added 46 new events:
  probe_perf:call_path__new (on call_path__new in /home/acme/bin/perf)
  probe_perf:call_path_root__new (on call_path_root__new in /home/acme/bin/perf)
  probe_perf:call_return_processor__new (on call_return_processor__new in /home/acme/bin/perf)
<SNIP>

You can now use them in all perf tools, e.g.:

	perf record -e probe_perf:call_path_root__new -aR sleep 1

------

Also that -l should output to stdout, so that we can grep it, also it would be nice if it -l worked
like in perf list:

[root@zoo ~]# perf probe -l probe_perf:*
Semantic error :There is non-digit char in line number.
  Error: Command Parse Error.
[root@zoo ~]# 

Fails, but:

[root@zoo ~]# perf list *:*switch*

List of pre-defined events (to be used in -e):

  cfg80211:cfg80211_ch_switch_notify                 [Tracepoint event]
  cfg80211:rdev_channel_switch                       [Tracepoint event]
  mac80211:api_chswitch_done                         [Tracepoint event]
  mac80211:drv_channel_switch                        [Tracepoint event]
  mac80211:drv_channel_switch_beacon                 [Tracepoint event]
  mac80211:drv_switch_vif_chanctx                    [Tracepoint event]
  sched:sched_switch                                 [Tracepoint event]

[root@zoo ~]# 

Works :-)

- Arnaldo

Re: [RFE] perf probe: Support globbing/regex in -a

[Masami Hiramatsu]

Hi Arnaldo,

(2015/04/09 4:48), Arnaldo Carvalho de Melo wrote:
> Hi Masami,
> 
> 	In tools/perf constructors are named <CLASS>__new, and right now I want
> to know hoe many of each <CLASS> objects are being allocated, so I expected to
> be able to do:
> 
> [root@zoo ~]# perf probe -x ~/bin/perf -a "*__new"
> Probe point '*__new' not found.
>   Error: Failed to add events.
> [root@zoo ~]#

Actually, I had sent it　:)

https://lkml.org/lkml/2014/10/31/207
https://lkml.org/lkml/2014/10/31/213

I'll update and resend the wildcard support.

> 
> And get the same result that I get from:
> 
> [root@zoo ~]# perf probe -x ~/bin/perf `perf probe -x ~/bin/perf -F | grep __new | sed 's/^/-a /g' | sort -u`
> Added new event:
>   probe_perf:call_path__new (on call_path__new in /home/acme/bin/perf)
> 
> You can now use it in all perf tools, such as:
> 
> 	perf record -e probe_perf:call_path__new -aR sleep 1
> 
> Added new event:
>   probe_perf:call_path_root__new (on call_path_root__new in /home/acme/bin/perf)
> 
> You can now use it in all perf tools, such as:
> 
> 	perf record -e probe_perf:call_path_root__new -aR sleep 1
> 
> Added new event:
>   probe_perf:call_return_processor__new (on call_return_processor__new in /home/acme/bin/perf)
> 
> 
> <SNIP>
> 
> I.e. that I end up with:
> 
> [root@zoo ~]# perf probe -l  2>&1 | grep probe_perf:.*__new
>   probe_perf:call_path__new (on call_path__new@util/thread-stack.c in /home/acme/bin/perf)
>   probe_perf:call_path_root__new (on call_path_root__new@util/thread-stack.c in /home/acme/bin/perf)
>   probe_perf:call_return_processor__new (on call_return_processor__new@util/thread-stack.c in /home/acme/bin/perf)
>   probe_perf:comm__new (on comm__new@util/comm.c in /home/acme/bin/perf)
>   probe_perf:cpu_map__new (on cpu_map__new@util/cpumap.c in /home/acme/bin/perf)
>   probe_perf:debuginfo__new (on debuginfo__new@util/probe-finder.c in /home/acme/bin/perf)
>   probe_perf:disasm_line__new (on disasm_line__new@util/annotate.c in /home/acme/bin/perf)
>   probe_perf:dso__new  (on dso__new@util/dso.c in /home/acme/bin/perf)
>   probe_perf:dso__new_map (on dso__new_map@util/dso.c in /home/acme/bin/perf)
>   probe_perf:fdarray__new (on fdarray__new@fd/array.c in /home/acme/bin/perf)
>   probe_perf:hist_browser__new (on hist_browser__new@ui/browsers/hists.c in /home/acme/bin/perf)
> <SNIP>
>   probe_perf:thread_map__new_by_uid (on thread_map__new_by_uid@util/thread_map.c in /home/acme/bin/perf)
>   probe_perf:thread_map__new_dummy (on thread_map__new_dummy@util/thread_map.c in /home/acme/bin/perf)
>   probe_perf:thread_map__new_str (on thread_map__new_str@util/thread_map.c in /home/acme/bin/perf)
> [root@zoo ~]#
> 
> Some more goodies to have, please consider better compressing the output of successfully installed probes,
> Something like:
> 
> [root@zoo ~]# perf probe -x ~/bin/perf `perf probe -x ~/bin/perf -F  | grep __new | sed 's/^/-a /g' | sort -u`
> Added 46 new events:
>   probe_perf:call_path__new (on call_path__new in /home/acme/bin/perf)
>   probe_perf:call_path_root__new (on call_path_root__new in /home/acme/bin/perf)
>   probe_perf:call_return_processor__new (on call_return_processor__new in /home/acme/bin/perf)
> <SNIP>
> 
> You can now use them in all perf tools, e.g.:
> 
> 	perf record -e probe_perf:call_path_root__new -aR sleep 1
> 
> ------
> 
> Also that -l should output to stdout, so that we can grep it, also it would be nice if it -l worked
> like in perf list:
> 
> [root@zoo ~]# perf probe -l probe_perf:*
> Semantic error :There is non-digit char in line number.
>   Error: Command Parse Error.


Hmm, right. This should be fixed.

Thank you!


> [root@zoo ~]# 
> 
> Fails, but:
> 
> [root@zoo ~]# perf list *:*switch*
> 
> List of pre-defined events (to be used in -e):
> 
>   cfg80211:cfg80211_ch_switch_notify                 [Tracepoint event]
>   cfg80211:rdev_channel_switch                       [Tracepoint event]
>   mac80211:api_chswitch_done                         [Tracepoint event]
>   mac80211:drv_channel_switch                        [Tracepoint event]
>   mac80211:drv_channel_switch_beacon                 [Tracepoint event]
>   mac80211:drv_switch_vif_chanctx                    [Tracepoint event]
>   sched:sched_switch                                 [Tracepoint event]
> 
> [root@zoo ~]# 
> 
> Works :-)
> 
> - Arnaldo
> 


-- 
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [RFE] perf probe: Support globbing/regex in -a

[Arnaldo Carvalho de Melo]

Em Thu, Apr 09, 2015 at 02:27:52PM +0900, Masami Hiramatsu escreveu:
> Hi Arnaldo,
> 
> (2015/04/09 4:48), Arnaldo Carvalho de Melo wrote:
> > Hi Masami,
> > 
> > 	In tools/perf constructors are named <CLASS>__new, and right now I want
> > to know hoe many of each <CLASS> objects are being allocated, so I expected to
> > be able to do:
> > 
> > [root@zoo ~]# perf probe -x ~/bin/perf -a "*__new"
> > Probe point '*__new' not found.
> >   Error: Failed to add events.
> > [root@zoo ~]#
> 
> Actually, I had sent it　:)
> 
> https://lkml.org/lkml/2014/10/31/207
> https://lkml.org/lkml/2014/10/31/213
> 
> I'll update and resend the wildcard support.

Oops, but it was together with that cache thing, could it be sent
separately?

<SNIP>
 
> > Also that -l should output to stdout, so that we can grep it, also it would be nice if it -l worked
> > like in perf list:
> > 
> > [root@zoo ~]# perf probe -l probe_perf:*
> > Semantic error :There is non-digit char in line number.
> >   Error: Command Parse Error.
> 
> Hmm, right. This should be fixed.

Thanks!

- Arnaldo

Re: Re: [RFE] perf probe: Support globbing/regex in -a

[Masami Hiramatsu]

(2015/04/09 19:36), Arnaldo Carvalho de Melo wrote:
> Em Thu, Apr 09, 2015 at 02:27:52PM +0900, Masami Hiramatsu escreveu:
>> Hi Arnaldo,
>>
>> (2015/04/09 4:48), Arnaldo Carvalho de Melo wrote:
>>> Hi Masami,
>>>
>>> 	In tools/perf constructors are named <CLASS>__new, and right now I want
>>> to know hoe many of each <CLASS> objects are being allocated, so I expected to
>>> be able to do:
>>>
>>> [root@zoo ~]# perf probe -x ~/bin/perf -a "*__new"
>>> Probe point '*__new' not found.
>>>   Error: Failed to add events.
>>> [root@zoo ~]#
>>
>> Actually, I had sent it　:)
>>
>> https://lkml.org/lkml/2014/10/31/207
>> https://lkml.org/lkml/2014/10/31/213
>>
>> I'll update and resend the wildcard support.
> 
> Oops, but it was together with that cache thing, could it be sent
> separately?

Yes, it can be applied separately. However, it may open the Pandora's box,
which we have to face to someday anyway :)

 - "perf probe -a \*" can try to put probes on all functions in the kernel
   including the inline functions. The total number of probes will be
   more than one hundred-thousand (100,000)...
 - Currently ftrace-kprobe doesn't limit the number of dynamic events, so
   it can cause out of memory on some machine.
 - Also, enabling such events at once, could cause performance problem,
   which I had tackled last year. See below slide.
http://events.linuxfoundation.org/sites/events/files/slides/Handling%20the%20Massive%20Multiple%20Kprobes%20v2_1.pdf
   (note that this series is not applied yet)

So, currently perf probe limits the maximum number of definable events at once
you can expand it with --max-probes anyway. And also, I've introduced --no-inlines
option to perf probe which reduces the number of matched functions when user
puts the probes with wild cards. (I'm considering to apply this option automatically
if the function name includes wild cards)

Thank you,

> 
> <SNIP>
>  
>>> Also that -l should output to stdout, so that we can grep it, also it would be nice if it -l worked
>>> like in perf list:
>>>
>>> [root@zoo ~]# perf probe -l probe_perf:*
>>> Semantic error :There is non-digit char in line number.
>>>   Error: Command Parse Error.
>>
>> Hmm, right. This should be fixed.
> 
> Thanks!
> 
> - Arnaldo
> 


-- 
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Re: [RFE] perf probe: Support globbing/regex in -a

[Arnaldo Carvalho de Melo]

Em Sat, Apr 11, 2015 at 09:25:54AM +0900, Masami Hiramatsu escreveu:
> (2015/04/09 19:36), Arnaldo Carvalho de Melo wrote:
> > Em Thu, Apr 09, 2015 at 02:27:52PM +0900, Masami Hiramatsu escreveu:
> >> (2015/04/09 4:48), Arnaldo Carvalho de Melo wrote:
> >>> 	In tools/perf constructors are named <CLASS>__new, and right now I want
> >>> to know hoe many of each <CLASS> objects are being allocated, so I expected to
> >>> be able to do:

> >>> [root@zoo ~]# perf probe -x ~/bin/perf -a "*__new"
> >>> Probe point '*__new' not found.
> >>>   Error: Failed to add events.
> >>> [root@zoo ~]#

> >> Actually, I had sent it　:)

> >> https://lkml.org/lkml/2014/10/31/207
> >> https://lkml.org/lkml/2014/10/31/213

> >> I'll update and resend the wildcard support.

> > Oops, but it was together with that cache thing, could it be sent
> > separately?
 
> Yes, it can be applied separately. However, it may open the Pandora's box,
> which we have to face to someday anyway :)

yeah!
 
>  - "perf probe -a \*" can try to put probes on all functions in the kernel
>    including the inline functions. The total number of probes will be
>    more than one hundred-thousand (100,000)...
>  - Currently ftrace-kprobe doesn't limit the number of dynamic events, so
>    it can cause out of memory on some machine.
>  - Also, enabling such events at once, could cause performance problem,

So don't do it?

>    which I had tackled last year. See below slide.
> http://events.linuxfoundation.org/sites/events/files/slides/Handling%20the%20Massive%20Multiple%20Kprobes%20v2_1.pdf
>    (note that this series is not applied yet)
 
> So, currently perf probe limits the maximum number of definable events at once
> you can expand it with --max-probes anyway. And also, I've introduced --no-inlines
> option to perf probe which reduces the number of matched functions when user
> puts the probes with wild cards. (I'm considering to apply this option automatically
> if the function name includes wild cards)

I.e. we have some mechanisms in place to limit the number of probes,
user gets warned when that limit is hit, no problem, right?

And on top of that, you already have a patchkit that tries to reduce the
overhead, good deal! :-)

I'll look again at those patches and see if they apply.

Thanks for all the info about this topic, appreciated!

- Arnaldo
 
> Thank you,
> 
> > 
> > <SNIP>
> >  
> >>> Also that -l should output to stdout, so that we can grep it, also it would be nice if it -l worked
> >>> like in perf list:
> >>>
> >>> [root@zoo ~]# perf probe -l probe_perf:*
> >>> Semantic error :There is non-digit char in line number.
> >>>   Error: Command Parse Error.
> >>
> >> Hmm, right. This should be fixed.
> > 
> > Thanks!
> > 
> > - Arnaldo
> > 
> 
> 
> -- 
> Masami HIRAMATSU
> Linux Technology Research Center, System Productivity Research Dept.
> Center for Technology Innovation - Systems Engineering
> Hitachi, Ltd., Research & Development Group
> E-mail: masami.hiramatsu.pt@hitachi.com
>