* Missing stack symbols with perf_event's perf report, despite -fno-omit-frame-pointer compilation @ 2016-04-19 0:07 Mark Davis 2016-04-19 0:18 ` David Ahern 2016-04-19 2:26 ` Taeung Song 0 siblings, 2 replies; 5+ messages in thread From: Mark Davis @ 2016-04-19 0:07 UTC (permalink / raw) To: linux-perf-users I'm struggling getting perf_events to give me stack traces with symbols, despite reading many tutorials on the subject and doing (I think) all the necessary things. It's possible that my local install of perf (details on that below) is somehow botched? Anyway, here's what I did: main.cpp is a simple C++ program that calls a few functions defined in the same file, allocates some memory and frees it, and prints a few things out. compilation command: gcc -std=c++11 -lstdc++ main.cpp -Og -fno-omit-frame-pointer -fno-inline -o arr_test profile command: perf record -a -g -- ./arr_test && perf report --stdio I do get the following warnings about kernel symbols, but I don't think this should matter given that I only care about symbols in my application for now: [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.052 MB perf.data (~2285 samples) ] [kernel.kallsyms] with build id e22966849c48748782a1be4fe0ce94db6838b806 not found, continuing without symbols [kernel.kallsyms] with build id e22966849c48748782a1be4fe0ce94db6838b806 not found, continuing without symbols Warning: Kernel address maps (/proc/{kallsyms,modules}) were restricted. Check /proc/sys/kernel/kptr_restrict before running 'perf record'. As no suitable kallsyms nor vmlinux was found, kernel samples can't be resolved. Samples in kernel modules can't be resolved as well. Here's a snippet of the output: # Overhead Command Shared Object # ........ ........ ................. # 83.27% arr_test arr_test | |--34.12%-- 0x400908 | 0x7fe72b381ec5 | |--10.48%-- 0x400903 | 0x7fe72b381ec5 | |--10.08%-- 0x4008b8 | 0x7fe72b381ec5 | |--9.22%-- 0x4008e5 | 0x7fe72b381ec5 | |--9.05%-- 0x4008da | 0x7fe72b381ec5 | |--8.49%-- 0x4008f0 | 0x7fe72b381ec5 | |--6.87%-- 0x4008d5 | 0x7fe72b381ec5 | |--6.23%-- 0x4008c2 | 0x7fe72b381ec5 | |--4.76%-- 0x4008fd | 0x7fe72b381ec5 --0.70%-- [...] 8.02% arr_test [kernel.kallsyms] | |--4.87%-- 0xffffffff81140b64 | 0xffffffff81146646 | 0xffffffff81182751 | 0xffffffff811829eb | 0xffffffff8173317d | 0x7fe72bab86a7 | 0x7fe72baa7e00 file info (shows "not stripped"): $ file arr_test arr_test: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, not stripped Details on my perf install (do any of these warnings prevent me from seeing symbols in stacks?) Auto-detecting system features: ... backtrace: [ on ] ... dwarf: [ OFF ] ... fortify-source: [ on ] ... glibc: [ on ] ... gtk2: [ on ] ... gtk2-infobar: [ on ] ... libaudit: [ OFF ] ... libbfd: [ OFF ] ... libelf: [ OFF ] ... libelf-getphdrnum: [ OFF ] ... libelf-mmap: [ OFF ] ... libnuma: [ on ] ... libperl: [ on ] ... libpython: [ on ] ... libpython-version: [ on ] ... libslang: [ on ] ... libunwind: [ OFF ] ... on-exit: [ on ] ... stackprotector: [ on ] ... stackprotector-all: [ on ] ... timerfd: [ on ] config/Makefile:264: No libelf found, disables 'probe' tool, please install elfutils-libelf-devel/libelf-dev config/Makefile:329: No libunwind found, disabling post unwind support. Please install libunwind-dev[el] >= 1.1 config/Makefile:354: No libaudit.h found, disables 'trace' tool, please install audit-libs-devel or libaudit-dev How can I find my symbols in perf? ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Missing stack symbols with perf_event's perf report, despite -fno-omit-frame-pointer compilation 2016-04-19 0:07 Missing stack symbols with perf_event's perf report, despite -fno-omit-frame-pointer compilation Mark Davis @ 2016-04-19 0:18 ` David Ahern 2016-04-25 9:01 ` Milian Wolff 2016-04-19 2:26 ` Taeung Song 1 sibling, 1 reply; 5+ messages in thread From: David Ahern @ 2016-04-19 0:18 UTC (permalink / raw) To: Mark Davis, linux-perf-users On 4/18/16 6:07 PM, Mark Davis wrote: > Auto-detecting system features: > ... backtrace: [ on ] > ... dwarf: [ OFF ] > ... fortify-source: [ on ] > ... glibc: [ on ] > ... gtk2: [ on ] > ... gtk2-infobar: [ on ] > ... libaudit: [ OFF ] > ... libbfd: [ OFF ] > ... libelf: [ OFF ] Install those 2 development packages. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Missing stack symbols with perf_event's perf report, despite -fno-omit-frame-pointer compilation 2016-04-19 0:18 ` David Ahern @ 2016-04-25 9:01 ` Milian Wolff 2016-04-26 1:03 ` Arnaldo Carvalho de Melo 0 siblings, 1 reply; 5+ messages in thread From: Milian Wolff @ 2016-04-25 9:01 UTC (permalink / raw) To: David Ahern; +Cc: Mark Davis, linux-perf-users [-- Attachment #1: Type: text/plain, Size: 1339 bytes --] On Monday, April 18, 2016 6:18:45 PM CEST David Ahern wrote: > On 4/18/16 6:07 PM, Mark Davis wrote: > > Auto-detecting system features: > > ... backtrace: [ on ] > > ... dwarf: [ OFF ] > > ... fortify-source: [ on ] > > ... glibc: [ on ] > > ... gtk2: [ on ] > > ... gtk2-infobar: [ on ] > > ... libaudit: [ OFF ] > > ... libbfd: [ OFF ] > > ... libelf: [ OFF ] > > Install those 2 development packages. And if that is still not helping, you may run into the case where the samples are recorded in a library (like libstdc++, libc,...) which was provided by your distribution without frame pointers. In such a case, the backtrace will still be broken. If you want to use frame pointers, and operate on user space code, my advise is to recompile all dependencies with frame pointers. On Yocto/Gentoo that is easily doable, elsewhere you'll have a hard time and waste a ton of time. I suggest you simply use Dwarf unwinding. Bye -- Milian Wolff | milian.wolff@kdab.com | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt Experts [-- Attachment #2: smime.p7s --] [-- Type: application/pkcs7-signature, Size: 5903 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Missing stack symbols with perf_event's perf report, despite -fno-omit-frame-pointer compilation 2016-04-25 9:01 ` Milian Wolff @ 2016-04-26 1:03 ` Arnaldo Carvalho de Melo 0 siblings, 0 replies; 5+ messages in thread From: Arnaldo Carvalho de Melo @ 2016-04-26 1:03 UTC (permalink / raw) To: Milian Wolff; +Cc: David Ahern, Mark Davis, linux-perf-users Em Mon, Apr 25, 2016 at 11:01:54AM +0200, Milian Wolff escreveu: > On Monday, April 18, 2016 6:18:45 PM CEST David Ahern wrote: > > On 4/18/16 6:07 PM, Mark Davis wrote: > > > Auto-detecting system features: > > > ... backtrace: [ on ] > > > ... dwarf: [ OFF ] > > > ... fortify-source: [ on ] > > > ... glibc: [ on ] > > > ... gtk2: [ on ] > > > ... gtk2-infobar: [ on ] > > > ... libaudit: [ OFF ] > > > ... libbfd: [ OFF ] > > > ... libelf: [ OFF ] > > Install those 2 development packages. > And if that is still not helping, you may run into the case where the samples > are recorded in a library (like libstdc++, libc,...) which was provided by > your distribution without frame pointers. In such a case, the backtrace will > still be broken. > If you want to use frame pointers, and operate on user space code, my advise > is to recompile all dependencies with frame pointers. On Yocto/Gentoo that is > easily doable, elsewhere you'll have a hard time and waste a ton of time. > I suggest you simply use Dwarf unwinding. Please try '--call-graph lbr' to check if your hardware has LBR, will be much cheaper than callchains. For instance: [acme@jouet linux]$ perf record --call-graph lbr usleep 1 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.018 MB perf.data (9 samples) ] [acme@jouet linux]$ perf evlist -v cycles:ppp: size: 112, { sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|CALLCHAIN|PERIOD|BRANCH_STACK, disabled: 1, inherit: 1, mmap: 1, comm: 1, freq: 1, enable_on_exec: 1, task: 1, precise_ip: 3, sample_id_all: 1, exclude_guest: 1, mmap2: 1, comm_exec: 1, branch_sample_type: USER|CALL_STACK|NO_FLAGS|NO_CYCLES [acme@jouet linux]$ - 59.69% usleep [kernel] [k] vma_interval_tree_insert →vma_interval_tree_insert [kernel] vma_adjust [kernel] __split_vma.isra.31 [kernel] split_vma [kernel] mprotect_fixup [kernel] sys_mprotect [kernel] entry_SYSCALL_64_fastpath [kernel] mprotect ld-2.22.so _dl_relocate_object ld-2.22.so memcpy@GLIBC_2.2.5 libc-2.22.so _dl_relocate_object ld-2.22.so __gettimeofday libc-2.22.so _dl_vdso_vsym libc-2.22.so _dl_lookup_symbol_x ld-2.22.so This was done on a Broadwell system (ThinkPad t450s). I now need to continue investigation why this doesn't seem to work from tracepoints... - Arnaldo ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Missing stack symbols with perf_event's perf report, despite -fno-omit-frame-pointer compilation 2016-04-19 0:07 Missing stack symbols with perf_event's perf report, despite -fno-omit-frame-pointer compilation Mark Davis 2016-04-19 0:18 ` David Ahern @ 2016-04-19 2:26 ` Taeung Song 1 sibling, 0 replies; 5+ messages in thread From: Taeung Song @ 2016-04-19 2:26 UTC (permalink / raw) To: Mark Davis, linux-perf-users Hi, On 04/19/2016 09:07 AM, Mark Davis wrote: > I'm struggling getting perf_events to give me stack traces with symbols, > despite reading many tutorials on the subject and doing (I think) all the > necessary things. It's possible that my local install of perf (details on that > below) is somehow botched? Anyway, here's what I did: > > main.cpp is a simple C++ program that calls a few functions defined in the same > file, allocates some memory and frees it, and prints a few things out. > > compilation command: > > gcc -std=c++11 -lstdc++ main.cpp -Og -fno-omit-frame-pointer -fno-inline -o > arr_test > > profile command: > > perf record -a -g -- ./arr_test && perf report --stdio > > > I do get the following warnings about kernel symbols, but I don't think this > should matter given that I only care about symbols in my application for now: > As a guide.. If you add ":u" to a event name, you can measure only at the user level e.g. perf record -ag -e cycles:u ./arr_test ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2016-04-26 1:04 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-04-19 0:07 Missing stack symbols with perf_event's perf report, despite -fno-omit-frame-pointer compilation Mark Davis 2016-04-19 0:18 ` David Ahern 2016-04-25 9:01 ` Milian Wolff 2016-04-26 1:03 ` Arnaldo Carvalho de Melo 2016-04-19 2:26 ` Taeung Song
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).