off-box analysis of perf.data file

off-box analysis of perf.data file

[David S. Ahern]

I'm trying to understand how the latest perf tools are designed with
respect to off-line analysis.

Specifically, I have a system running binaries and libraries that are
stripped and no vmlinux is installed. For analysis, I have a local tree
based on software versions with full debug symbols. Events are collected
into a perf.data file and the file is pushed off-box for analysis.

Without assuming write access to local tree or even ownership of the
perf.data I imagine running a command something like:

perf report -fi /path/to/perf.data \
    --rootfs /path/to/symboled-tree \
    --kallsyms /path/to/kallsyms
    ...

where /proc/kallsyms is dumped for the running system and retrieved
along with the perf.data file.

I've been reading the source and can't say I follow all of the build-id
stuff. Before I spend time working on code modifications I wanted to
better understand the expectations and options for the latest code.

Thanks,

David

Re: off-box analysis of perf.data file

[Arnaldo Carvalho de Melo]

Em Sun, Nov 14, 2010 at 09:03:20PM -0700, David S. Ahern escreveu:
> 
> I'm trying to understand how the latest perf tools are designed with
> respect to off-line analysis.
> 
> Specifically, I have a system running binaries and libraries that are
> stripped and no vmlinux is installed. For analysis, I have a local tree
> based on software versions with full debug symbols. Events are collected
> into a perf.data file and the file is pushed off-box for analysis.
> 
> Without assuming write access to local tree or even ownership of the
> perf.data I imagine running a command something like:
> 
> perf report -fi /path/to/perf.data \
>     --rootfs /path/to/symboled-tree \
>     --kallsyms /path/to/kallsyms
>     ...
> 
> where /proc/kallsyms is dumped for the running system and retrieved
> along with the perf.data file.
> 
> I've been reading the source and can't say I follow all of the build-id
> stuff. Before I spend time working on code modifications I wanted to
> better understand the expectations and options for the latest code.

The usual workflow is the following, after it I'll talk at how to change it to
fit your described setup:

On stripped machine:

$ perf record
$ perf archive
$ scp perf.data.* devel-machine

on devel machine:

$ tar xvf perf.data.tar.bz2 -C ~/.debug
$ perf report

This all assumes that build-ids are enabled, to see if they are, try
this:

[root@felicio ~]# perf record -a
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.311 MB perf.data (~13580 samples) ]

[root@felicio ~]# perf buildid-list
c6127e36741c6c9a2bdce6b15c311e0d484c6f5c [kernel.kallsyms]
c9bc1e043301a5683a70b9103fb92428120fa6b8 /lib64/ld-2.12.90.so
67775f20bdc4ed7c2c0153d9a9e3aaf264299a53 /lib64/libc-2.12.90.so
0bf7d0a3161677af75e1000b1b3230747f1bb723 /lib64/libpthread-2.12.90.so
03516b51e7b3d7ac03875b0bf1cdae075027f7ae /lib64/libdbus-1.so.3.5.2
b6689690a70811d6d0695721c5b70d7e8df87ee1 /lib64/libglib-2.0.so.0.2600.0
0293adb02b5eb6b105943c2023c7ed251f186068 /lib64/libgthread-2.0.so.0.2600.0
326e1b23b38fc735bfc8b80040f72e07366f57d4 /usr/lib64/libxcb.so.1.1.0
6a839d574266e21e5c7419897f32763c819a57c4 /usr/lib64/libX11.so.6.3.0
6dcb8890286a9a4f07bc02c72c040efcd9779bb1 /usr/lib64/libpixman-1.so.0.18.4
cb73d470ae58707322d98487eccf4c2c6dbe86bd /usr/lib64/libcairo.so.2.11000.0
eeee4ac30808d39985ffc1d09e9bdb77a6c9f790 /lib64/libnspr4.so
10452af31d660b66c59271054b88d3b0fa86837c /usr/libexec/hald-addon-input
fb88c2545ac77dbbca04b762de09fa1a2be0298d /usr/lib64/libstdc++.so.6.0.14
29662806dc8d2d288f5550169735c8f55f7e150c /usr/bin/Xorg
30e3c8c0e52233f880adbb4bbcb4ff054f2b659f /usr/lib64/firefox-3.6/firefox
c23d65d019845220376b2a16f6775722a424d0f9 /usr/lib64/xulrunner-1.9.2/libmozjs.so
6f5ec07169f334d1095e04bb181873ad8d45b3b8 /usr/lib64/xulrunner-1.9.2/libxul.so
501727fac876dbf2e3c636e0896f275b7d9042dd /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/server/libjvm.so
9ed0ef8bc9b9b3d98b21e4a8e0b248b6c983f54c /home/acme/bin/perf
[root@felicio ~]#

See the long hexadecimal numbers? Those are the build-ids, now look at:

[root@felicio ~]# ls -la ~/.debug/lib64/libnspr4.so/
total 244
drwxr-xr-x.  2 root root   4096 Oct  2 16:36 .
drwxr-xr-x. 22 root root   4096 Nov 16 20:04 ..
-rwxr-xr-x.  1 root root 239056 Aug 25 17:50 eeee4ac30808d39985ffc1d09e9bdb77a6c9f790
[root@felicio ~]# ls -la ~/.debug/lib64/libnspr4.so/eeee4ac30808d39985ffc1d09e9bdb77a6c9f790 
-rwxr-xr-x. 1 root root 239056 Aug 25 17:50 /root/.debug/lib64/libnspr4.so/eeee4ac30808d39985ffc1d09e9bdb77a6c9f790
[root@felicio ~]# set -o vi
[root@felicio ~]# ls -la ~/.debug/.build-id/ee/ee4ac30808d39985ffc1d09e9bdb77a6c9f790 
lrwxrwxrwx. 1 root root 64 Oct  2 16:36 /root/.debug/.build-id/ee/ee4ac30808d39985ffc1d09e9bdb77a6c9f790 -> ../../lib64/libnspr4.so/eeee4ac30808d39985ffc1d09e9bdb77a6c9f790
[root@felicio ~]#

Each DSO has its build-id in a special ELF section:

[root@felicio ~]# readelf -SW /lib64/libnspr4.so | grep build
  [ 1] .note.gnu.build-id NOTE            0000003683600190 000190 000024 00   A  0   0  4
[root@felicio ~]#

To fit your setup you need:

. Populate your devel machine ~/.debug build-id database with the unstripped
  binaries, do that using another perf command, buildid-cache, example:

[root@felicio ~]# perf buildid-cache -v -a /lib64/libc-2.12.90.so 
Adding 67775f20bdc4ed7c2c0153d9a9e3aaf264299a53 /lib64/libc-2.12.90.so: FAIL
/lib64/libc-2.12.90.so already in the cache
[root@felicio ~]# perf buildid-cache -v -a /lib64/libcrypt-2.12.90.so 
Adding 41189b36a158b2727e0e0b54b2dc869bce4e038d /lib64/libcrypt-2.12.90.so: Ok
[root@felicio ~]#

I.e. everytime you build something you intend to ship to your devices, add it
to your build-id database in your devel (where perf report runs).

Its like git, content based, not filename based. You may have lots of versions
of binaries in multiple supported "firmware" versions for your target devices
and the perf tools will find the right binaries/symtabs when decoding.

Please let me know if you need more details on how this is implemented.

Since you mentioned kallsyms, if vmlinux is not found, that is what is used:

[acme@doppio ~]$ l ~/.debug/\[kernel.kallsyms\]/
total 27504
drwxr-xr-x 2 acme acme    4096 2010-09-01 18:57 ./
drwxr-xr-x 9 acme acme    4096 2010-01-24 13:46 ../
-rw-rw-r-- 1 acme acme 2908658 2010-04-29 21:45 0c0a71b29fa20b8f7606180757d01406c8712110
-rw-rw-r-- 1 acme acme 2742642 2010-03-14 13:21 2622047749fd7128d77c015d148aed6d9b25f793
-rw-rw-r-- 1 acme acme 2842888 2010-05-20 22:57 349199a1e1b7cc25ac4f7004deed260f0146665e
-rw-r--r-- 1 acme acme 3205476 2010-08-18 11:09 364300f12afee3dd5b1a87269d60de586f6b61ce
-rw-rw-r-- 1 acme acme 2650594 2010-01-24 13:46 5a49c70ff35d0ba5959ba53bb05096ef46c88978
-rw-rw-r-- 1 acme acme 2797835 2010-06-24 11:12 6723f2dbda348d3fbc8aa5e88d4cf7d5fbbb7106
-rw-rw-r-- 1 acme acme 2734508 2010-09-01 18:57 6d2698625887fdfdb2a52611e8fca78f7cd1d238
-rw-rw-r-- 1 acme acme 2795481 2010-04-05 13:19 712f38010a4e3c01292ed3a3d4576692cba4c233
-rw-rw-r-- 1 acme acme 2648017 2010-01-14 23:06 74f9930ee94475b6b3238caf3725a50d59cb994b
-rw-rw-r-- 1 acme acme 2761144 2010-07-19 10:39 9921d66cf0e9ebf4f1e29d9b11d223d7805931f7
[acme@doppio ~]$

[acme@doppio ~]$ tail -5 /home/acme/.debug/\[kernel.kallsyms\]/9921d66cf0e9ebf4f1e29d9b11d223d7805931f7 
ffffffffa0000410 d video_output_attributes	[output]
ffffffffa0000460 d __this_module	[output]
ffffffffa00001d4 t cleanup_module	[output]
ffffffffa00000e7 T video_output_register	[output]
ffffffffa00000ce T video_output_unregister	[output]
[acme@doppio ~]$ 

10 different kernels :-)

The running kernel build-id is at:

[root@felicio ~]# ls -la /sys/kernel/notes 
-r--r--r-- 1 root root 380 Nov 16 20:03 /sys/kernel/notes
[root@felicio ~]# 

See the symbols.c in the tools/perf/util/ kernel sources directory to see how it is
parsed.

Regards,

- Arnaldo

Re: off-box analysis of perf.data file

[Arnaldo Carvalho de Melo]

Em Tue, Nov 16, 2010 at 09:40:42PM -0200, Arnaldo Carvalho de Melo escreveu:
> [root@felicio ~]#
> 
> To fit your setup you need:
> 
> . Populate your devel machine ~/.debug build-id database with the unstripped
>   binaries, do that using another perf command, buildid-cache, example:

Something I forgot: you can disable the cache populating in the stripped
machine since you have them anyway on the devel machine, that can be achieved
by what is described in this changeset:

commit a1ac1d3c085420ea8c809ebbee3bb212ed3616bd
Author: Stephane Eranian <eranian@google.com>
Date:   Thu Jun 17 11:39:01 2010 +0200

    perf record: Add option to avoid updating buildid cache
    
    There are situations where there is enough information in the perf.data
    to process the samples. Updating the buildid cache may add unecessary
    overhead in terms of disk space and time (copying large elf images).
    
    A persistent option to do this already exists via the perfconfig file,
    simply do:
    
    [buildid]
    dir = /dev/null
    
    This patch provides a way to suppress builid cache updates on a per-run
    basis.  It addds a new option, -N, to perf record. Buildids are still
    generated in the perf.data file.
    
    Cc: David S. Miller <davem@davemloft.net>
    Cc: Frédéric Weisbecker <fweisbec@gmail.com>
    Cc: Ingo Molnar <mingo@elte.hu>
    Cc: Paul Mackerras <paulus@samba.org>
    Cc: Peter Zijlstra <peterz@infradead.org>
    LKML-Reference: <4c19ef89.93ecd80a.40dc.fffff8e9@mx.google.com>
    Signed-off-by: Stephane Eranian <eranian@google.com>
    Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>

So just use -N and that step will be skipped.


- Arnaldo

Re: off-box analysis of perf.data file

[David S. Ahern]

On 11/16/10 16:40, Arnaldo Carvalho de Melo wrote:
> Em Sun, Nov 14, 2010 at 09:03:20PM -0700, David S. Ahern escreveu:
>>
>> I'm trying to understand how the latest perf tools are designed with
>> respect to off-line analysis.
>>
>> Specifically, I have a system running binaries and libraries that are
>> stripped and no vmlinux is installed. For analysis, I have a local tree
>> based on software versions with full debug symbols. Events are collected
>> into a perf.data file and the file is pushed off-box for analysis.
>>
>> Without assuming write access to local tree or even ownership of the
>> perf.data I imagine running a command something like:
>>
>> perf report -fi /path/to/perf.data \
>>     --rootfs /path/to/symboled-tree \
>>     --kallsyms /path/to/kallsyms
>>     ...
>>
>> where /proc/kallsyms is dumped for the running system and retrieved
>> along with the perf.data file.
>>
>> I've been reading the source and can't say I follow all of the build-id
>> stuff. Before I spend time working on code modifications I wanted to
>> better understand the expectations and options for the latest code.
> 
> The usual workflow is the following, after it I'll talk at how to change it to
> fit your described setup:
> 
> On stripped machine:
> 
> $ perf record
> $ perf archive
> $ scp perf.data.* devel-machine
> 
> on devel machine:
> 
> $ tar xvf perf.data.tar.bz2 -C ~/.debug
> $ perf report
> 
> This all assumes that build-ids are enabled, to see if they are, try
> this:

Thank you for the detailed response. Very helpful. I also read the
Fedora project page for a bit of background on build-ids:
http://fedoraproject.org/wiki/Releases/FeatureBuildId

I spent a bit of time looking at the perf-archive and seeing how it
works. I get the general idea now. The build-id option is supported by
our linker, so something we could enable. However, ...

> . Populate your devel machine ~/.debug build-id database with the unstripped
>   binaries, do that using another perf command, buildid-cache, example:

The need for populating ~/.debug does not seem practical for a
"large-scale" development organization - at least not for our setup.

In our case the product has "official" builds at regular intervals
(e.g., daily) with image trees in some automount area (e.g.,
/auto/${majorver}/${release}). Then developers create builds in personal
workspace areas, e.g. /auto/${user/my-build-${ver}-${release}. The
builds are done using cross-compile environments, so there is no single
"development machine".

When you consider 100's of engineers working on 100's of versions of
software images with a mix-and-match of who generates a perf.data file
and who is analyzing the data (and where) it does not seem practical to
require pulling in the debug data into a ~/.debug tree. e.g., QA or
system test can record and pull a perf.data file for analysis by one or
more DEs.

We already generate trees with links to files with full debugging
symbols for analyzing core files - leveraging the outputs of the builds.
It would be ideal to reuse that tree for perf too.

--

Kernel versions and kallsyms: We have a single code base that generates
images for N different platforms. Multiple platforms use the same kernel
version but requires different kernel modules, and each image can
contain slightly different builds of the same kernel module name. Would
this cause a conflict in a cached area from a build-id perspective? ie.,
is the build-id based on the kernel version and not a function of loaded
modules?

--

Development systems: We use a cross-compile environment for builds and
lab servers which would be used for analyzing a perf.data file can be
running a variety of linux versions and flavors - from RHEL4, RHEL5,
Fedora 10, Fedora 14, etc. In this case we would not want perf to look
at the vmlinux, kallsyms, libs and binaries on the system where the
analysis is done - something that perf does today as part of of its
default paths in dso__load.

--

The above factors are why I was expecting arguments such as --rootfs and
--kallsyms - to point an analysis invocation to a specific tree to use
and a specific kallsyms file.

David

Re: off-box analysis of perf.data file

[Arnaldo Carvalho de Melo]

Em Wed, Nov 17, 2010 at 04:31:53PM -0700, David S. Ahern escreveu:
> On 11/16/10 16:40, Arnaldo Carvalho de Melo wrote:
> > Em Sun, Nov 14, 2010 at 09:03:20PM -0700, David S. Ahern escreveu:
> >> I've been reading the source and can't say I follow all of the build-id
> >> stuff. Before I spend time working on code modifications I wanted to
> >> better understand the expectations and options for the latest code.
> > 
> > The usual workflow is the following, after it I'll talk at how to change it to
> > fit your described setup:
> 
> Thank you for the detailed response. Very helpful. I also read the
> Fedora project page for a bit of background on build-ids:
> http://fedoraproject.org/wiki/Releases/FeatureBuildId
> 
> I spent a bit of time looking at the perf-archive and seeing how it
> works. I get the general idea now. The build-id option is supported by
> our linker, so something we could enable. However, ...
> 
> > . Populate your devel machine ~/.debug build-id database with the unstripped
> >   binaries, do that using another perf command, buildid-cache, example:
> 
> The need for populating ~/.debug does not seem practical for a
> "large-scale" development organization - at least not for our setup.

You don't need to have one single ~/.debug database, its up to you to
determine what is the best setup for your organization, I believe that
it is possible to have multiple databases and for you to specify which
one to use.

> In our case the product has "official" builds at regular intervals
> (e.g., daily) with image trees in some automount area (e.g.,
> /auto/${majorver}/${release}). Then developers create builds in personal
> workspace areas, e.g. /auto/${user/my-build-${ver}-${release}. The
> builds are done using cross-compile environments, so there is no single
> "development machine".

But it is possible to have a single development server or to have
multiple databases according to your needs.
 
> When you consider 100's of engineers working on 100's of versions of
> software images with a mix-and-match of who generates a perf.data file
> and who is analyzing the data (and where) it does not seem practical to
> require pulling in the debug data into a ~/.debug tree. e.g., QA or
> system test can record and pull a perf.data file for analysis by one or
> more DEs.
> 
> We already generate trees with links to files with full debugging
> symbols for analyzing core files - leveraging the outputs of the builds.
> It would be ideal to reuse that tree for perf too.

Well, you can, just create the ~/.debug/.build-id/ links pointing to
whatever places where you already keep your binaries.
 
> Kernel versions and kallsyms: We have a single code base that generates
> images for N different platforms. Multiple platforms use the same kernel
> version but requires different kernel modules, and each image can
> contain slightly different builds of the same kernel module name. Would
> this cause a conflict in a cached area from a build-id perspective? ie.,
> is the build-id based on the kernel version and not a function of loaded
> modules?

Nope, the build-id is not based on versions, it is based on contents:

$ pinfo ld
<SNIP>
`--build-id'
`--build-id=STYLE'
     Request creation of `.note.gnu.build-id' ELF note section.  The
     contents of the note are unique bits identifying this linked file.
     STYLE can be `uuid' to use 128 random bits, `sha1' to use a
     160-bit SHA1 hash on the normative parts of the output contents,
     `md5' to use a 128-bit MD5 hash on the normative parts of the
     output contents, or `0xHEXSTRING' to use a chosen bit string
     specified as an even number of hexadecimal digits (`-' and `:'
     characters between digit pairs are ignored).  If STYLE is omitted,
     `sha1' is used.

     The `md5' and `sha1' styles produces an identifier that is always
     the same in an identical output file, but will be unique among all
     nonidentical output files.  It is not intended to be compared as a
     checksum for the file's contents.  A linked file may be changed
     later by other tools, but the build ID bit string identifying the
     original linked file does not change.

     Passing `none' for STYLE disables the setting from any
     `--build-id' options earlier on the command line.
<SNIP>

If you look at the kernel Makefile:

# Use --build-id when available.
LDFLAGS_BUILD_ID = $(patsubst -Wl$(comma)%,%,\
                              $(call cc-ldoption, -Wl$(comma)--build-id,))


no style passed? sha1 used, hence, git like content based unique ID.

> Development systems: We use a cross-compile environment for builds and
> lab servers which would be used for analyzing a perf.data file can be
> running a variety of linux versions and flavors - from RHEL4, RHEL5,
> Fedora 10, Fedora 14, etc. In this case we would not want perf to look
> at the vmlinux, kallsyms, libs and binaries on the system where the
> analysis is done - something that perf does today as part of of its
> default paths in dso__load.

"perf" is too vague, covers many subcommands, what specific subsystem, perhaps
the above explanations made this question moot?
 
- Arnaldo

Re: off-box analysis of perf.data file

[David S. Ahern]

On 11/18/10 13:19, Arnaldo Carvalho de Melo wrote:
>> In our case the product has "official" builds at regular intervals
>> (e.g., daily) with image trees in some automount area (e.g.,
>> /auto/${majorver}/${release}). Then developers create builds in personal
>> workspace areas, e.g. /auto/${user/my-build-${ver}-${release}. The
>> builds are done using cross-compile environments, so there is no single
>> "development machine".
> 
> But it is possible to have a single development server or to have
> multiple databases according to your needs.

Hmmm.... From a user perspective maintaining build-id databases for
short-use builds is more complex than using a --rootfs argument. i.e.,
add rootfs to symbol_conf with a default value of "", add a --rootfs
argument to analysis commands to allow a user to specify a path, and
within dso__load prepend the paths within rootfs. This appears to be
similar to what has been done for builtin-kvm and analyzing guest data.
Furthermore, this allows a wide range of options - loop mounting KVM
disk images, NFS trees, bootable USB keys, initrds, etc. Anything with
an OS tree can be analyzed from anywhere without the need to populate a
local data store.

Similar argument for kallsyms point to a file rather than pointing to
/proc/kallsyms -- which again parallels what has been done for builtin-kvm.

I guess my point is that perf contains a lot of hardcoded paths which is
great for standard use cases - default settings for users. But allow a
nerd knob that changes the path or offsets the path -- either for each
origin or at a minimum for the ORIG_DSO.


>> Development systems: We use a cross-compile environment for builds and
>> lab servers which would be used for analyzing a perf.data file can be
>> running a variety of linux versions and flavors - from RHEL4, RHEL5,
>> Fedora 10, Fedora 14, etc. In this case we would not want perf to look
>> at the vmlinux, kallsyms, libs and binaries on the system where the
>> analysis is done - something that perf does today as part of of its
>> default paths in dso__load.
> 
> "perf" is too vague, covers many subcommands, what specific subsystem, perhaps
> the above explanations made this question moot?

"perf" in the above meant all the commands that eventually call
dso__load. In my F14 - F13 test case (data collected on a Fedora 14
system and analyzed on a Fedora 13 system), 'strace -e trace=open perf
report ...' shows perf opening a lot of files on the F13 system - even
though the ~/.debug tree was populated using perf-archive.

David

>  
> - Arnaldo

Re: off-box analysis of perf.data file

[Arnaldo Carvalho de Melo]

Em Thu, Nov 18, 2010 at 02:54:27PM -0700, David S. Ahern escreveu:
> Hmmm.... From a user perspective maintaining build-id databases for
> short-use builds is more complex than using a --rootfs argument. i.e.,
> add rootfs to symbol_conf with a default value of "", add a --rootfs
> argument to analysis commands to allow a user to specify a path, and
> within dso__load prepend the paths within rootfs. This appears to be
> similar to what has been done for builtin-kvm and analyzing guest
> data.  Furthermore, this allows a wide range of options - loop
> mounting KVM disk images, NFS trees, bootable USB keys, initrds, etc.
> Anything with an OS tree can be analyzed from anywhere without the
> need to populate a local data store.
 
> Similar argument for kallsyms point to a file rather than pointing to
> /proc/kallsyms -- which again parallels what has been done for builtin-kvm.

Yeah, I think it is worth having it for maximum flexibility, can you try
to implement it and post the patch for discussion?

> "perf" in the above meant all the commands that eventually call
> dso__load. In my F14 - F13 test case (data collected on a Fedora 14
> system and analyzed on a Fedora 13 system), 'strace -e trace=open perf
> report ...' shows perf opening a lot of files on the F13 system - even
> though the ~/.debug tree was populated using perf-archive.

What 'perf buildid-list' shows on this specific perf.data file? Perhaps
it is not finding the build-ids somehow and so fallbacks to looking at
the system binaries, checking if they have the same build-id, if
build-ids are present on the perf.data file header.

There is a bit of historical baggage here, in the pre-build-id days the
tools would look just at the pathnames, assuming the binaries hadn't
changed, etc.

- Arnaldo

Re: off-box analysis of perf.data file

[David S. Ahern]

On 11/19/10 05:50, Arnaldo Carvalho de Melo wrote:
> Yeah, I think it is worth having it for maximum flexibility, can you try
> to implement it and post the patch for discussion?

What git tree should be used for posting patches?

David

Re: off-box analysis of perf.data file

[Arnaldo Carvalho de Melo]

Em Sat, Nov 20, 2010 at 08:44:06AM -0700, David S. Ahern escreveu:
> 
> 
> On 11/19/10 05:50, Arnaldo Carvalho de Melo wrote:
> > Yeah, I think it is worth having it for maximum flexibility, can you try
> > to implement it and post the patch for discussion?
> 
> What git tree should be used for posting patches?

Use tip perf/core

http://people.redhat.com/mingo/tip.git/README

- Arnaldo