Re: [RFC PATCH perf/core v2 00/16] perf-probe --cache and SDT support

[Hemant Kumar <hemant@linux.vnet.ibm.com>]

Hi Masami,

Apologies for the delayed response.

On 07/17/2015 08:51 AM, Masami Hiramatsu wrote:
> Hi Hemant,
>
> On 2015/07/16 12:13, Hemant Kumar wrote:
>> Hi Masami,
>>
>> On 07/15/2015 02:43 PM, Masami Hiramatsu wrote:
>>> Hi,
>>>
>>> Here is the 2nd version of the patchset for probe-cache and
>>> initial SDT support which are going to be perf-cache finally.
>> Thanks for adding the SDT support.
>>
>>> The perf-probe is useful for debugging, but it strongly depends
>>> on the debuginfo. Without debuginfo, it is just a frontend of
>>> ftrace's dynamic events. This can usually happen in server
>>> farms or on cloud system, since no one wants to distribute
>>> big debuginfo packages.
>>>
>>> To solve this issue, I had tried to make a pre-analyzed probes
>>> ( https://lkml.org/lkml/2014/10/31/207 ) but it has a problm
>>> that we can't ensure the probed binary is same as what we analyzed.
>>> Arnaldo gave me an idea to reuse build-id cache for that perpose
>>> and this series is the first prototype of that.
>>>
>>> At the same time, Hemant has started to support SDT probes which
>>> also use the cache file of SDT info. So I decided to merge this
>>> into the same build-id cache.
>>> In this version, SDT support is still very limited, it works
>>> as a part of probe-cache.
>>>
>>> In this version, perf probe supports --cache option which means
>>> that perf probe manipulate probe caches, for example,
>>>
>>>     # perf probe --cache --add "probe-desc"
>>>
>>> does not only add probe events but also add "probe-desc" and
>>> it's result on the cache. (Note that the cached entry is always
>>> referred even without --cache)
>>> The --list and --del commands also support --cache. Note that
>>> both are only manipulate caches, not real events.
>>>
>>> To use SDT, we have to scan the target binary at first by using
>>> perf-buildid-cache, e.g.
>>>
>>>     # perf buildid-cache --add /lib/libc-2.17.so
>>>
>>> And perf probe --cache --list shows what SDTs are scanned.
>>>
>>>     # perf probe --cache --list
>>>     /usr/lib/libc-2.17.so (a6fb821bdf53660eb2c29f778757aef294d3d392):
>>>     libc:setjmp=setjmp
>>>     libc:longjmp=longjmp
>>>     libc:longjmp_target=longjmp_target
>>>     libc:memory_heap_new=memory_heap_new
>>>     libc:memory_sbrk_less=memory_sbrk_less
>>>     libc:memory_arena_reuse_free_list=memory_arena_reuse_free_list
>>>     libc:memory_arena_reuse=memory_arena_reuse
>>>     ...
>>>
>>> To use the SDT events, perf probe -x BIN %SDTEVENT allows you to
>>> add a probe on SDTEVENT@BIN.
>>>
>>>     # perf probe -x /lib/libc-2.17.so %memory_heap_new
>>>
>>> If you define a cached probe with event name, you can also reuse
>>> it as same as SDT events.
>>>
>>>     # perf probe -x ./perf --cache -n 'myevent=dso__load $params'
>>>
>>> (Note that "-n" option only updates caches)
>>> To use the above "myevent", you just have to add "%myevent".
>>>
>>>     # perf probe -x ./perf %myevent
>>>
>>>
>>> TODOs:
>>>    - Show available cached/SDT events by perf-list
>>>    - Allow perf-record to use cached/SDT events directly
>> As I was already working on SDT events' recording
>> https://lkml.org/lkml/2014/11/2/73,
>> I can re-spin the patches on top of your patchset and make the
>> required changes to implement the above TODOs.
> Sounds great! :)
> Note that you'll need to re-implement almost from scratch, since
> now the SDT is implemented on buildid-cache. Maybe I have to work
> on the buildid-cache one more to filter out binaries which are gone
> or different version from current running one (e.g. old vmlinux).
> It could help you to get available SDTs when showing it via perf-list.

Sure. That would be great.

>> What would you suggest?
> Now I'm thinking that we should avoid using %event syntax for perf-list
> and perf-record to avoid confusion. For example, suppose that we have
> "libfoo:bar" SDT event, when we just scanned the libfoo binary and
> use it via perf-record, we'll run perf record -e "%libfoo:bar".
> However, after we set the probe via perf-probe, we have to run
> perf record -e "libfoo:bar". That difference looks no good.
> So, I think in both case it should accept -e "libfoo:bar" syntax.

Although I agree to have "perf record" as a higher level tool and not bother
this tool to distinguish between its events, but that way we end up looking
into kprobe_events, uprobe_events, kernel tracepoints and then the entire
cache for any event (which may or may not be an SDT event or even a valid
event) lookup. Right?

The idea behind '%' was to identify the SDT events and take a different path
to lookup through the cache, put a probe, record and then delete the probe.
Or, do you want "perf record" to record any event this way (not just an sdt
event).

Please correct me if I missed something.

> In this series I've introduced %event syntax only to recall cached event
> setting explicitly, because perf-probe is a lower layer tool to set up
> new event. IMO, perf-list and perf-record should be higher tools which
> handle abstract events.
>
> Thanks!
>
>

-- 
Thanks,
Hemant Kumar