* Missing stack symbols with perf_event's perf report, despite -fno-omit-frame-pointer compilation
@ 2016-04-19 0:07 Mark Davis
2016-04-19 0:18 ` David Ahern
2016-04-19 2:26 ` Taeung Song
0 siblings, 2 replies; 5+ messages in thread
From: Mark Davis @ 2016-04-19 0:07 UTC (permalink / raw)
To: linux-perf-users
I'm struggling getting perf_events to give me stack traces with symbols,
despite reading many tutorials on the subject and doing (I think) all the
necessary things. It's possible that my local install of perf (details on that
below) is somehow botched? Anyway, here's what I did:
main.cpp is a simple C++ program that calls a few functions defined in the same
file, allocates some memory and frees it, and prints a few things out.
compilation command:
gcc -std=c++11 -lstdc++ main.cpp -Og -fno-omit-frame-pointer -fno-inline -o
arr_test
profile command:
perf record -a -g -- ./arr_test && perf report --stdio
I do get the following warnings about kernel symbols, but I don't think this
should matter given that I only care about symbols in my application for now:
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.052 MB perf.data (~2285 samples) ]
[kernel.kallsyms] with build id e22966849c48748782a1be4fe0ce94db6838b806
not found, continuing without symbols
[kernel.kallsyms] with build id e22966849c48748782a1be4fe0ce94db6838b806
not found, continuing without symbols
Warning:
Kernel address maps (/proc/{kallsyms,modules}) were restricted.
Check /proc/sys/kernel/kptr_restrict before running 'perf record'.
As no suitable kallsyms nor vmlinux was found, kernel samples
can't be resolved.
Samples in kernel modules can't be resolved as well.
Here's a snippet of the output:
# Overhead Command Shared Object
# ........ ........ .................
#
83.27% arr_test arr_test
|
|--34.12%-- 0x400908
| 0x7fe72b381ec5
|
|--10.48%-- 0x400903
| 0x7fe72b381ec5
|
|--10.08%-- 0x4008b8
| 0x7fe72b381ec5
|
|--9.22%-- 0x4008e5
| 0x7fe72b381ec5
|
|--9.05%-- 0x4008da
| 0x7fe72b381ec5
|
|--8.49%-- 0x4008f0
| 0x7fe72b381ec5
|
|--6.87%-- 0x4008d5
| 0x7fe72b381ec5
|
|--6.23%-- 0x4008c2
| 0x7fe72b381ec5
|
|--4.76%-- 0x4008fd
| 0x7fe72b381ec5
--0.70%-- [...]
8.02% arr_test [kernel.kallsyms]
|
|--4.87%-- 0xffffffff81140b64
| 0xffffffff81146646
| 0xffffffff81182751
| 0xffffffff811829eb
| 0xffffffff8173317d
| 0x7fe72bab86a7
| 0x7fe72baa7e00
file info (shows "not stripped"):
$ file arr_test
arr_test: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically
linked (uses shared libs), for GNU/Linux 2.6.24, not stripped
Details on my perf install (do any of these warnings prevent me from seeing
symbols in stacks?)
Auto-detecting system features:
... backtrace: [ on ]
... dwarf: [ OFF ]
... fortify-source: [ on ]
... glibc: [ on ]
... gtk2: [ on ]
... gtk2-infobar: [ on ]
... libaudit: [ OFF ]
... libbfd: [ OFF ]
... libelf: [ OFF ]
... libelf-getphdrnum: [ OFF ]
... libelf-mmap: [ OFF ]
... libnuma: [ on ]
... libperl: [ on ]
... libpython: [ on ]
... libpython-version: [ on ]
... libslang: [ on ]
... libunwind: [ OFF ]
... on-exit: [ on ]
... stackprotector: [ on ]
... stackprotector-all: [ on ]
... timerfd: [ on ]
config/Makefile:264: No libelf found, disables 'probe' tool, please install
elfutils-libelf-devel/libelf-dev
config/Makefile:329: No libunwind found, disabling post unwind support.
Please install libunwind-dev[el] >= 1.1
config/Makefile:354: No libaudit.h found, disables 'trace' tool, please
install audit-libs-devel or libaudit-dev
How can I find my symbols in perf?
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Missing stack symbols with perf_event's perf report, despite -fno-omit-frame-pointer compilation
2016-04-19 0:07 Missing stack symbols with perf_event's perf report, despite -fno-omit-frame-pointer compilation Mark Davis
@ 2016-04-19 0:18 ` David Ahern
2016-04-25 9:01 ` Milian Wolff
2016-04-19 2:26 ` Taeung Song
1 sibling, 1 reply; 5+ messages in thread
From: David Ahern @ 2016-04-19 0:18 UTC (permalink / raw)
To: Mark Davis, linux-perf-users
On 4/18/16 6:07 PM, Mark Davis wrote:
> Auto-detecting system features:
> ... backtrace: [ on ]
> ... dwarf: [ OFF ]
> ... fortify-source: [ on ]
> ... glibc: [ on ]
> ... gtk2: [ on ]
> ... gtk2-infobar: [ on ]
> ... libaudit: [ OFF ]
> ... libbfd: [ OFF ]
> ... libelf: [ OFF ]
Install those 2 development packages.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Missing stack symbols with perf_event's perf report, despite -fno-omit-frame-pointer compilation
2016-04-19 0:07 Missing stack symbols with perf_event's perf report, despite -fno-omit-frame-pointer compilation Mark Davis
2016-04-19 0:18 ` David Ahern
@ 2016-04-19 2:26 ` Taeung Song
1 sibling, 0 replies; 5+ messages in thread
From: Taeung Song @ 2016-04-19 2:26 UTC (permalink / raw)
To: Mark Davis, linux-perf-users
Hi,
On 04/19/2016 09:07 AM, Mark Davis wrote:
> I'm struggling getting perf_events to give me stack traces with symbols,
> despite reading many tutorials on the subject and doing (I think) all the
> necessary things. It's possible that my local install of perf (details on that
> below) is somehow botched? Anyway, here's what I did:
>
> main.cpp is a simple C++ program that calls a few functions defined in the same
> file, allocates some memory and frees it, and prints a few things out.
>
> compilation command:
>
> gcc -std=c++11 -lstdc++ main.cpp -Og -fno-omit-frame-pointer -fno-inline -o
> arr_test
>
> profile command:
>
> perf record -a -g -- ./arr_test && perf report --stdio
>
>
> I do get the following warnings about kernel symbols, but I don't think this
> should matter given that I only care about symbols in my application for now:
>
As a guide..
If you add ":u" to a event name, you can measure only at the user level e.g.
perf record -ag -e cycles:u ./arr_test
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Missing stack symbols with perf_event's perf report, despite -fno-omit-frame-pointer compilation
2016-04-19 0:18 ` David Ahern
@ 2016-04-25 9:01 ` Milian Wolff
2016-04-26 1:03 ` Arnaldo Carvalho de Melo
0 siblings, 1 reply; 5+ messages in thread
From: Milian Wolff @ 2016-04-25 9:01 UTC (permalink / raw)
To: David Ahern; +Cc: Mark Davis, linux-perf-users
[-- Attachment #1: Type: text/plain, Size: 1339 bytes --]
On Monday, April 18, 2016 6:18:45 PM CEST David Ahern wrote:
> On 4/18/16 6:07 PM, Mark Davis wrote:
> > Auto-detecting system features:
> > ... backtrace: [ on ]
> > ... dwarf: [ OFF ]
> > ... fortify-source: [ on ]
> > ... glibc: [ on ]
> > ... gtk2: [ on ]
> > ... gtk2-infobar: [ on ]
> > ... libaudit: [ OFF ]
> > ... libbfd: [ OFF ]
> > ... libelf: [ OFF ]
>
> Install those 2 development packages.
And if that is still not helping, you may run into the case where the samples
are recorded in a library (like libstdc++, libc,...) which was provided by
your distribution without frame pointers. In such a case, the backtrace will
still be broken.
If you want to use frame pointers, and operate on user space code, my advise
is to recompile all dependencies with frame pointers. On Yocto/Gentoo that is
easily doable, elsewhere you'll have a hard time and waste a ton of time.
I suggest you simply use Dwarf unwinding.
Bye
--
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Missing stack symbols with perf_event's perf report, despite -fno-omit-frame-pointer compilation
2016-04-25 9:01 ` Milian Wolff
@ 2016-04-26 1:03 ` Arnaldo Carvalho de Melo
0 siblings, 0 replies; 5+ messages in thread
From: Arnaldo Carvalho de Melo @ 2016-04-26 1:03 UTC (permalink / raw)
To: Milian Wolff; +Cc: David Ahern, Mark Davis, linux-perf-users
Em Mon, Apr 25, 2016 at 11:01:54AM +0200, Milian Wolff escreveu:
> On Monday, April 18, 2016 6:18:45 PM CEST David Ahern wrote:
> > On 4/18/16 6:07 PM, Mark Davis wrote:
> > > Auto-detecting system features:
> > > ... backtrace: [ on ]
> > > ... dwarf: [ OFF ]
> > > ... fortify-source: [ on ]
> > > ... glibc: [ on ]
> > > ... gtk2: [ on ]
> > > ... gtk2-infobar: [ on ]
> > > ... libaudit: [ OFF ]
> > > ... libbfd: [ OFF ]
> > > ... libelf: [ OFF ]
> > Install those 2 development packages.
> And if that is still not helping, you may run into the case where the samples
> are recorded in a library (like libstdc++, libc,...) which was provided by
> your distribution without frame pointers. In such a case, the backtrace will
> still be broken.
> If you want to use frame pointers, and operate on user space code, my advise
> is to recompile all dependencies with frame pointers. On Yocto/Gentoo that is
> easily doable, elsewhere you'll have a hard time and waste a ton of time.
> I suggest you simply use Dwarf unwinding.
Please try '--call-graph lbr' to check if your hardware has LBR, will be
much cheaper than callchains.
For instance:
[acme@jouet linux]$ perf record --call-graph lbr usleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.018 MB perf.data (9 samples) ]
[acme@jouet linux]$ perf evlist -v
cycles:ppp: size: 112, { sample_period, sample_freq }: 4000,
sample_type: IP|TID|TIME|CALLCHAIN|PERIOD|BRANCH_STACK, disabled: 1,
inherit: 1, mmap: 1, comm: 1, freq: 1, enable_on_exec: 1, task: 1,
precise_ip: 3, sample_id_all: 1, exclude_guest: 1, mmap2: 1, comm_exec:
1, branch_sample_type: USER|CALL_STACK|NO_FLAGS|NO_CYCLES
[acme@jouet linux]$
- 59.69% usleep [kernel] [k] vma_interval_tree_insert
→vma_interval_tree_insert [kernel]
vma_adjust [kernel]
__split_vma.isra.31 [kernel]
split_vma [kernel]
mprotect_fixup [kernel]
sys_mprotect [kernel]
entry_SYSCALL_64_fastpath [kernel]
mprotect ld-2.22.so
_dl_relocate_object ld-2.22.so
memcpy@GLIBC_2.2.5 libc-2.22.so
_dl_relocate_object ld-2.22.so
__gettimeofday libc-2.22.so
_dl_vdso_vsym libc-2.22.so
_dl_lookup_symbol_x ld-2.22.so
This was done on a Broadwell system (ThinkPad t450s).
I now need to continue investigation why this doesn't seem to work from
tracepoints...
- Arnaldo
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2016-04-26 1:04 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-04-19 0:07 Missing stack symbols with perf_event's perf report, despite -fno-omit-frame-pointer compilation Mark Davis
2016-04-19 0:18 ` David Ahern
2016-04-25 9:01 ` Milian Wolff
2016-04-26 1:03 ` Arnaldo Carvalho de Melo
2016-04-19 2:26 ` Taeung Song
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).