perf tools: Call graph from Intel BTS

perf tools: Call graph from Intel BTS

[Adrian Hunter]

Hi

Alexander Shishkin is working on the Intel PT driver for perf
and has included a driver for Intel BTS.  I have taken that and
extended perf tools to enable the creation of context-sensitive
call graphs.  A screen shot from an example call graph from
tracing "ls" is shown here:

	http://git.infradead.org/~ahunter/call-graph-bts-ls.png

The patches are here:

	http://git.infradead.org/users/ahunter/linux-perf.git

There are many perf tools patches and it would be helpful to start
considering how to get them into mainline.  Many need to wait for
the driver, but others could be taken sooner.

The perf tools python API has been extended to facilitate exporting
to databases, a script is provided to export to PostgreSQL, and
another script reads the database and displays the call graph.

The Intel BTS data provides branches (address from / to) but to
create the call graph the type of instruction (call, return etc)
is needed.  That is found by reading the object code, but for the
kernel that requires an accurate image.  A copy of /proc/kcore
is needed but it must be made under the same conditions as the
trace.  That has proven sufficiently awkward to warrant creating
a script to do it.

Here is how the "ls" call graph was created:


$ sudo bash -c 'echo 0 > /proc/sys/kernel/kptr_restrict'
$ sudo bash -c 'echo -1 > /proc/sys/kernel/perf_event_paranoid'
$ sudo bash -c 'chmod go+rx /sys/kernel/debug'
$ ~/libexec/perf-core/perf-with-kcore record bts_ls  --per-thread -e intel_bts// -- ls
Recording
Using /home/ahunter/bin/perf
perf version 3.16.rc2.linux.local.gaabf
/home/ahunter/bin/perf record -o bts_ls/perf.data --per-thread -e intel_bts//  -- ls
CREDITS    drivers               ipc            MAINTAINERS   modules.order   REPORTING-BUGS  signing_key.priv  tools             vmlinux.o
arch       crypto                firmware       Kconfig       Makefile        Module.symvers  samples           signing_key.x509  usr             x509.genkey
block      fs                    include        kernel        mm              net             scripts           sound             virt
bts_ls     COPYING               Documentation  init          Kbuild          lib             modules.builtin   README            security        System.map        vmlinux
[ perf record: Woken up 100 times to write data ]
[ perf record: Captured and wrote 6.945 MB bts_ls/perf.data ]
Copying kcore
Done
$ ~/libexec/perf-core/perf-with-kcore script bts_ls -s ~/libexec/perf-core/scripts/python/export-to-postgresql.py bts_ls branches calls
Using /home/ahunter/bin/perf
perf version 3.16.rc2.linux.local.gaabf
/home/ahunter/bin/perf script -i bts_ls/perf.data --kallsyms=bts_ls/kcore_dir/kallsyms -s ~/libexec/perf-core/scripts/python/export-to-postgresql.py bts_ls branches calls
2014-07-11 15:13:38.313702 Creating database...
2014-07-11 15:13:39.150277 Writing to intermediate files...
2014-07-11 15:13:40.182816 Copying to database...
2014-07-11 15:13:41.776815 Removing intermediate files...
2014-07-11 15:13:41.783443 Adding primary keys
2014-07-11 15:13:44.290786 Adding foreign keys
2014-07-11 15:13:44.974865 Done
$ ~/libexec/perf-core/scripts/python/call-graph-from-postgresql.py bts_ls
$


Regards
Adrian

Re: perf tools: Call graph from Intel BTS

[Arnaldo Carvalho de Melo]

Em Fri, Jul 11, 2014 at 05:36:41PM +0300, Adrian Hunter escreveu:
> There are many perf tools patches and it would be helpful to start
> considering how to get them into mainline.  Many need to wait for
> the driver, but others could be taken sooner.

We can go on looking at each of the patches to see which ones can be
cherry picked, i.e. the ones that are fixes and not related to the work
you're doing, like:

commit 244c87b15b124914827f3ce28d8e70c8d147c9d0
Author: Adrian Hunter <adrian.hunter@intel.com>
Date:   Wed Jun 11 09:33:17 2014 +0300

    perf tools: Fix the value used for unknown pids
    
    The value used for unknown pids cannot be zero
    because that is used by the "idle" task.
    Use -1 instead.  Also handle the unknown pid
    case when creating map groups.
    
    Note that, threads with an unknown pid should not
    occur because fork (or synthesized) events precede
    the thread's existence.
    
    Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>

But then one by one they need to be reviewed to check if the changes were made
to the whole tools/perf/ tree and if perhaps something new came along since you
changed some assumption, like 0 meaning unknown thread, in the above patch:

[acme@ssdandy linux]$ find tools -name "*.[ch]" | xargs grep machine__findnew_thread | grep 0
tools/perf/util/session.c:	thread = machine__findnew_thread(&session->machines.host, 0, 0);
tools/perf/tests/thread-mg-share.c:	leader = machine__findnew_thread(machine, 0, 0);
tools/perf/tests/thread-mg-share.c:	t1     = machine__findnew_thread(machine, 0, 1);
tools/perf/tests/thread-mg-share.c:	t2     = machine__findnew_thread(machine, 0, 2);
tools/perf/tests/thread-mg-share.c:	t3     = machine__findnew_thread(machine, 0, 3);
[acme@ssdandy linux]$

So I think that one way to reduce the size of that branch is to do just that:
start fresh from tip/perf/core, and go cherry picking those patches, making sure that they
take into account the whole current tools/perf/ tree, then ask for this patch to be pulled.

You could then rebase the old branch on top of the resulting branch once it is
merged upstream, rinse repeat.

- Arnaldo

Re: perf tools: Call graph from Intel BTS

[Peter Zijlstra]

On Fri, Jul 11, 2014 at 05:36:41PM +0300, Adrian Hunter wrote:
> Hi
> 
> Alexander Shishkin is working on the Intel PT driver for perf
> and has included a driver for Intel BTS.  I have taken that and

There is already a BTS driver, although I've not used it ever, since
there's no useful tool for it. One way would be to use that, and migrate
to PT data later.

Re: perf tools: Call graph from Intel BTS

[Adrian Hunter]

On 11/07/2014 6:18 p.m., Arnaldo Carvalho de Melo wrote:
> Em Fri, Jul 11, 2014 at 05:36:41PM +0300, Adrian Hunter escreveu:
>> There are many perf tools patches and it would be helpful to start
>> considering how to get them into mainline.  Many need to wait for
>> the driver, but others could be taken sooner.
>
> We can go on looking at each of the patches to see which ones can be
> cherry picked, i.e. the ones that are fixes and not related to the work
> you're doing, like:
>
> commit 244c87b15b124914827f3ce28d8e70c8d147c9d0
> Author: Adrian Hunter <adrian.hunter@intel.com>
> Date:   Wed Jun 11 09:33:17 2014 +0300
>
>      perf tools: Fix the value used for unknown pids
>
>      The value used for unknown pids cannot be zero
>      because that is used by the "idle" task.
>      Use -1 instead.  Also handle the unknown pid
>      case when creating map groups.
>
>      Note that, threads with an unknown pid should not
>      occur because fork (or synthesized) events precede
>      the thread's existence.
>
>      Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>
> But then one by one they need to be reviewed to check if the changes were made
> to the whole tools/perf/ tree and if perhaps something new came along since you
> changed some assumption, like 0 meaning unknown thread, in the above patch:
>
> [acme@ssdandy linux]$ find tools -name "*.[ch]" | xargs grep machine__findnew_thread | grep 0
> tools/perf/util/session.c:	thread = machine__findnew_thread(&session->machines.host, 0, 0);

That's the idle thread.

> tools/perf/tests/thread-mg-share.c:	leader = machine__findnew_thread(machine, 0, 0);
> tools/perf/tests/thread-mg-share.c:	t1     = machine__findnew_thread(machine, 0, 1);
> tools/perf/tests/thread-mg-share.c:	t2     = machine__findnew_thread(machine, 0, 2);
> tools/perf/tests/thread-mg-share.c:	t3     = machine__findnew_thread(machine, 0, 3);

Those are valid pids for that test.

> [acme@ssdandy linux]$
>
> So I think that one way to reduce the size of that branch is to do just that:
> start fresh from tip/perf/core, and go cherry picking those patches, making sure that they
> take into account the whole current tools/perf/ tree, then ask for this patch to be pulled.
>
> You could then rebase the old branch on top of the resulting branch once it is
> merged upstream, rinse repeat.

Sounds good, thanks!  It is currently based on tip/perf/core from a few days ago, so the
current patches should be mostly ok.  I will make a selection and check them again.

Re: perf tools: Call graph from Intel BTS

[Adrian Hunter]

On 11/07/2014 6:36 p.m., Peter Zijlstra wrote:
> On Fri, Jul 11, 2014 at 05:36:41PM +0300, Adrian Hunter wrote:
>> Alexander Shishkin is working on the Intel PT driver for perf
>> and has included a driver for Intel BTS.  I have taken that and
>
> There is already a BTS driver, although I've not used it ever, since
> there's no useful tool for it. One way would be to use that, and migrate
> to PT data later.

Currently that driver does not allow kernel tracing.  Last time I tried it
with that restriction removed, it would lose data on larger workloads.
Generally, any loss of data ruins the call graph.

For me it does not make sense to try to support it, since I am primarily
trying to pave the way for the Intel PT driver.