From: Milian Wolff <milian.wolff@kdab.com>
To: acme@kernel.org
Cc: linux-perf-users@vger.kernel.org, dsahern@gmail.com, hekuang@huawei.com
Subject: Re: Cross platform perf reporting
Date: Mon, 15 Aug 2016 14:20:15 +0200 [thread overview]
Message-ID: <18625452.F1CIFXVKL2@milian-kdab2> (raw)
In-Reply-To: <20160812153125.GK27651@kernel.org>
[-- Attachment #1: Type: text/plain, Size: 13417 bytes --]
On Friday, August 12, 2016 12:31:25 PM CEST Arnaldo Carvalho de Melo wrote:
> Em Fri, Aug 12, 2016 at 05:10:18PM +0200, Milian Wolff escreveu:
> > On Friday, August 12, 2016 2:22:29 PM CEST Milian Wolff wrote:
> > > I'm trying to analyze perf.data files that I recorded on one machine
> > > (usually arm or arm64) on my development machine (x86_64). I have
> > > compiled
> > > a new version of perf (and the corresponding dependencies, where needed)
> > > to
> > > do this.
>
> Adding He Kuang, that did work on this area recently.
>
> > > Now I do:
> > >
> > > ~~~~~~~~~~~~~~~~~~~~~~~~
> > > aarch64$ perf record -g ...
> > > aarch64$ perf archive
> > >
> > > x86_64$ scp aarch64:perf.data* .
> > > x86_64$ tar xvf perf.data.tar.bz2 -C ~/.debug
> > > x86_64$ perf report |& less
> > > -> lots of errors, none of which are displayed in the TUI, such as:
> > >
> > > unwind: target platform=arm64 is not supported
> > > (often repeated)
> > > Failed to open /lib/ld-2.21.so, continuing without symbols
> > > Failed to open /lib/libc-2.21.so, continuing without symbols
> > > Failed to open /lib/libpthread-2.21.so, continuing without symbols
> > > Failed to open /lib/libdl-2.21.so, continuing without symbols
> > > ...
> > > ~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > What I notice, is that `perf buildid-list` does not contain any of these
>
> That is a limitation that needs fixing, what gets into the build-id are
> entries that had hits, DSOs, but those are sample hits, not callchain
> hits.
>
> We would have to traverse all callchains and add build-id entries for
> the DSOs in it.
>
> I need to do this :-\
>
> But if you have a copy of your target in a file and use --symfs with it it
> should work, because then, because it doesn't have the build-id for the DSOs
> in the callchains that didn't had hits, it will resort to using just the
> pathname when looking for a symtab or the whole ELF for annotation, etc.
>
> > files. I also don't have the corresponding files in my target's ~/.debug
> > folder, where everything seems to work as expected. Is that, because there
> > the symbols are read directly from the file and nothing is cached in
> > ~/.debug?
>
> That is a possibility, yes. But it should be adding it to ~/.debug, that
> is the default, unless you have something disabling that in your
> ~/.perfconfig or some sort of problem is happening that is preventing it
> to do the hardlink/copy.
I now realized that the culprit here is actually a broken platform, i.e. the
libs above simply don't have a build id:
eu-readelf -n libc.so.6
Note section [ 1] '.note.ABI-tag' of 32 bytes at offset 0x270:
Owner Data size Type
GNU 16 VERSION
OS: Linux, ABI: 3.18.0
So no wonder they are not included in perf archive. I think a warning would be
helpful here to tell users that analyzing the data on the remote machine will
require --symfs (which I wasn't aware of, thanks!).
> > Can I set an environment variable to let perf know to look in secondary
> > paths for the libraries? I.e. I hope to make it look into the cross
> > compile sysroot for these libraries, instead of literally interpreting
> > the path as a valid path on my development machine.
>
> That is the --symfs David talked about, no?
>
> [acme@jouet linux]$ perf report -h symfs
>
> Usage: perf report [<options>]
>
> --symfs <directory>
> Look for files with symbols relative to this
> directory
Yes, symfs sounds exactly like what I was looking for. I have now tested this,
and seem to hit some issues with it:
I unpacked a perf archive from a platform which includes a libc with buildit:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
milian@milian-kdab2:/tmp$ tar -tvf perf.data.tar.bz2 | grep libc-2.19
lrwxrwxrwx root/root 0 2000-01-01 18:36 .build-id/
45/7c5918872338e1b82bf8bf41ae9f55b34604ea -> ../../lib/arm-linux-gnueabihf/
libc-2.19.so/457c5918872338e1b82bf8bf41ae9f55b34604ea
drwxr-xr-x root/root 0 2000-01-01 18:36 lib/arm-linux-gnueabihf/
libc-2.19.so/457c5918872338e1b82bf8bf41ae9f55b34604ea/
-rwxr-xr-x root/root 911068 2016-03-01 01:55 lib/arm-linux-gnueabihf/
libc-2.19.so/457c5918872338e1b82bf8bf41ae9f55b34604ea/elf
-rw-r--r-- root/root 0 2000-01-01 18:36 lib/arm-linux-gnueabihf/
libc-2.19.so/457c5918872338e1b82bf8bf41ae9f55b34604ea/probes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Analyzing it for some reason isn't using the build id:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
milian@milian-kdab2:/tmp$ strace -e file -f perf report --stdio -s sym,srcline
|& grep libc-2.19
stat("/lib/arm-linux-gnueabihf/libc-2.19.so", 0x7ffde9316200) = -1 ENOENT (No
such file or directory)
stat("/lib/arm-linux-gnueabihf/libc-2.19.so", 0x7ffde9316180) = -1 ENOENT (No
such file or directory)
stat("/usr/lib/debug/lib/arm-linux-gnueabihf/libc-2.19.so.debug",
0x7ffde9316200) = -1 ENOENT (No such file or directory)
stat("/usr/lib/debug/lib/arm-linux-gnueabihf/libc-2.19.so", 0x7ffde9316200) =
-1 ENOENT (No such file or directory)
stat("/lib/arm-linux-gnueabihf/libc-2.19.so", 0x7ffde9316200) = -1 ENOENT (No
such file or directory)
stat("/lib/arm-linux-gnueabihf/.debug/libc-2.19.so", 0x7ffde9316200) = -1
ENOENT (No such file or directory)
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
0.25% 0.25% [.] 0x000000000005336c libc-2.19.so[5336c]
0.25% 0.00% [.] 0xffffffff4936136c
libc-2.19.so[ffffffff4936136
0.24% 0.24% [.] 0x0000000000053422 libc-2.19.so[53422]
0.24% 0.00% [.] 0xffffffff49361422
libc-2.19.so[ffffffff4936142
0.21% 0.21% [.] 0x0000000000053664 libc-2.19.so[53664]
0.21% 0.00% [.] 0xffffffff49361664
libc-2.19.so[ffffffff4936166
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Setting symfs:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
milian@milian-kdab2:/tmp$ locate libc-2.19.so
/home/milian/.debug/lib/arm-linux-gnueabihf/libc-2.19.so
/home/milian/.debug/lib/arm-linux-gnueabihf/libc-2.19.so/
457c5918872338e1b82bf8bf41ae9f55b34604ea
/home/milian/.debug/lib/arm-linux-gnueabihf/libc-2.19.so/
457c5918872338e1b82bf8bf41ae9f55b34604ea/elf
/home/milian/.debug/lib/arm-linux-gnueabihf/libc-2.19.so/
457c5918872338e1b82bf8bf41ae9f55b34604ea/probes
/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/lib/arm-
linux-gnueabihf/libc-2.19.so
milian@milian-kdab2:/tmp$ strace -e file -f perf report --symfs=/ssd/milian/
projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/ --stdio -s
sym,srcline |& grep libc-2.19
open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = 8
stat("/lib/arm-linux-gnueabihf/libc-2.19.so", 0x7ffc4da5e740) = -1 ENOENT (No
such file or directory)
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
lib/arm-linux-gnueabihf/libc-2.19.so", {st_mode=S_IFREG|0755, st_size=911068,
...}) = 0
open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = 9
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
lib/arm-linux-gnueabihf/libc-2.19.so", {st_mode=S_IFREG|0755, st_size=911068,
...}) = 0
open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = 9
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
usr/lib/debug/lib/arm-linux-gnueabihf/libc-2.19.so.debug", 0x7ffc4da5e740) =
-1 ENOENT (No such file or directory)
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
usr/lib/debug/lib/arm-linux-gnueabihf/libc-2.19.so", 0x7ffc4da5e740) = -1
ENOENT (No such file or directory)
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
lib/arm-linux-gnueabihf/libc-2.19.so", {st_mode=S_IFREG|0755, st_size=911068,
...}) = 0
open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = 10
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
lib/arm-linux-gnueabihf/.debug/libc-2.19.so", 0x7ffc4da5e740) = -1 ENOENT (No
such file or directory)
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
0.25% 0.25% [.] 0x000000000005336c libc-2.19.so[5336c]
0.25% 0.00% [.] 0xffffffff4936136c libc-2.19.so[ffffffff4936136
0.24% 0.24% [.] 0x0000000000053422 libc-2.19.so[53422]
0.24% 0.00% [.] 0xffffffff49361422 libc-2.19.so[ffffffff4936142
0.21% 0.21% [.] 0x0000000000053664 libc-2.19.so[53664]
0.21% 0.00% [.] 0xffffffff49361664 libc-2.19.so[ffffffff4936166
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Note how it does look into symfs sometimes, but not everywhere...
To make things worse, it seems to break the buildid lookup, i.e. when you get
samples in a file that you deployed manually to a target, which is not
included in your sysroot that you pass to symfs:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ perf report --stdio -s sym,srcline |& grep particles
34.79% 34.79% [.] main particles.cpp:34
32.62% 32.62% [.] main particles.cpp:33
9.85% 9.85% [.] main particles.cpp:32
$ perf report --symfs=/ssd/milian/projects/pandaboard/debian-8.5-minimal-
armhf-2016-06-06/ --stdio -s sym,srcline |& grep particles
/root/particles with build id 1c10a17afa25337ab368f8d9e690184e536c6563 not
found, continuing without symbols
addr2line_init failed for /root/particles
# above line is repeated a lot
26.99% 26.99% [.] 0x00000000000005a4 particles[5a4]
26.99% 0.00% [.] 0xffffffffffff05a4 particles[ffffffffffff05a4]
21.83% 21.83% [.] 0x00000000000005a6 particles[5a6]
21.83% 0.00% [.] 0xffffffffffff05a6 particles[ffffffffffff05a6]
14.55% 0.00% [.] 0xffffffffffff0596 particles[ffffffffffff0596]
14.54% 14.54% [.] 0x0000000000000596 particles[596]
...
$ strace -e file -f perf report --symfs=/ssd/milian/projects/pandaboard/
debian-8.5-minimal-armhf-2016-06-06/ --stdio -s sym,srcline |& grep particles
open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
root/particles", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/root/particles", 0x7ffe7535bf40) = -1 EACCES (Permission denied)
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
root/particles", 0x7ffe7535bec0) = -1 ENOENT (No such file or directory)
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
usr/lib/debug/root/particles.debug", 0x7ffe7535bf40) = -1 ENOENT (No such file
or directory)
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
usr/lib/debug/root/particles", 0x7ffe7535bf40) = -1 ENOENT (No such file or
directory)
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
root/particles", 0x7ffe7535bf40) = -1 ENOENT (No such file or directory)
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
root/.debug/particles", 0x7ffe7535bf40) = -1 ENOENT (No such file or
directory)
/root/particles with build id 1c10a17afa25337ab368f8d9e690184e536c6563 not
found, continuing without symbols
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Note how the above does not look for the buildid in my ~/.debug anymore...
Cheers
--
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]
next prev parent reply other threads:[~2016-08-15 12:20 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-08-12 12:22 Cross platform perf reporting Milian Wolff
2016-08-12 14:38 ` David Ahern
2016-08-12 15:10 ` Milian Wolff
2016-08-12 15:31 ` Arnaldo Carvalho de Melo
2016-08-15 12:20 ` Milian Wolff [this message]
2016-08-15 14:42 ` David Ahern
2016-08-15 15:22 ` Milian Wolff
2016-08-15 16:08 ` David Ahern
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=18625452.F1CIFXVKL2@milian-kdab2 \
--to=milian.wolff@kdab.com \
--cc=acme@kernel.org \
--cc=dsahern@gmail.com \
--cc=hekuang@huawei.com \
--cc=linux-perf-users@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).