* Re: Two questions on perf dwarf callchains
@ 2014-01-17 23:42 Mauro Andreolini
0 siblings, 0 replies; 18+ messages in thread
From: Mauro Andreolini @ 2014-01-17 23:42 UTC (permalink / raw)
To: linux-perf-users@vger.kernel.org
[Sorry for the repost, I can't see my reply in the gmane interface]
Il giorno ven, 17/01/2014 alle 12.11 -0800, Andi Kleen ha scritto:
> Mauro Andreolini <mauro.andreolini@gmail.com> writes:
> > file to get perf report to print the function names.
> > Is this expected behaviour? How do I get perf to automatically
extract
> > symbols from compressed files?
>
> Someone would need to write the code to do that I guess.
OK.
> >
> > To which function would 0x7facc1df12a0 match? It does not even seem
to
> > be in the mapped address space of "ls". I am clueless.
>
> The unwinder got confused. A common cause is incomplete or partially
> broken dwarf unwind information. May be a compiler problem.
Thanks for your time. Right now I am using gcc 4.8.2 20131219
(prerelease) and libunwind 1.1. I noticed the following
compile flags: "-O2 -g -fvar-tracking-assignments". I thought that
maybe optimizations like function inlining and the like would not play
together with debugging info, so I changed them to "-Og -g3
-fvar-tracking-assignments" and recompiled anything, albeit with no
luck. I must be missing something big here. How can I be sure to give
correct dwarf unwind information?
Mauro
^ permalink raw reply [flat|nested] 18+ messages in thread
* Two questions on perf dwarf callchains
@ 2014-01-17 10:21 Mauro Andreolini
2014-01-17 20:11 ` Andi Kleen
` (2 more replies)
0 siblings, 3 replies; 18+ messages in thread
From: Mauro Andreolini @ 2014-01-17 10:21 UTC (permalink / raw)
To: linux-perf-users; +Cc: Mauro Andreolini
Hi all,
I have installed a textual Arch Linux guest (the hypervisor
is KVM) and I am playing with perf and dwarf callchains.
In particular, I am trying to get meaningful stacktraces
for a simple command such as "ls -lR /".
Here is what I did.
* recompile the kernel with
CONFIG_DEBUG_INFO=y
CONFIG_FRAME_POINTER=Y
* recompile the C library with symbols (-g)
* recompile all other dependencies of "ls" (ldd /usr/bin/ls)
with symbols (on my system, that would be libcap, libacl,
libattr)
* recompile the coreutils package with symbols
After that, I executed the following commands:
perf record --call-graph dwarf -g -- ls -lR /
<wait 10s; hit CTRL-C>
perf report --call-graph --stdio
I hit two problems.
Problem n.1: perf report warns me about missing kernel symbols.
No kallsyms or vmlinux with build-id 4c6ab941f31dde04c2a481430218994849927c8d was found
[ext4] with build id 4c6ab941f31dde04c2a481430218994849927c8d not found, continuing without symbols
No kallsyms or vmlinux with build-id 21f163b0a8a8b924efdd173b8f83ad723d378958 was found
[drm] with build id 21f163b0a8a8b924efdd173b8f83ad723d378958 not found, continuing without symbols
No kallsyms or vmlinux with build-id d0bd7dd69886c2dccde18dee760b6d39cc0b5a55 was found[qxl] with build id d0bd7dd69886c2dccde18dee760b6d39cc0b5a55 not found, continuing without symbols
No kallsyms or vmlinux with build-id 8d13e6f0c4d4f8b4628c8c4bd6c660b0a00576df was found[ttm] with build id 8d13e6f0c4d4f8b4628c8c4bd6c660b0a00576df not found, continuing without symbols
...
#
# Samples: 4K of event 'cpu-clock'
# Event count (approx.): 1201250000
#
# Overhead Command Shared Object
Symbol
# ........ ....... ................. .................................
...
7.12% ls [qxl] [k] 0x0000000000009b20 <--- HERE
|
|--4.09%-- 0xffffffffa048c2c2
| |
| |--42.86%-- 0xffffffffa048ee8b
| | 0xffffffffa048ba8f
| | bit_bmove
| | fbcon_redraw_blit.isra.22
| | fbcon_scroll
| | scrup
| | lf
| | do_con_trol
| | do_con_write.part.18
| | con_write
| | do_output_char
| | n_tty_write
| | tty_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
For example, the 0x0000000000009b20 address should refer to some
function in the qxl module for which no symbol was generated. As far as
I could understand, perf does not seem to be able to extract symbol
information from compressed modules. I had to decompress the qxl.ko.gz
file to get perf report to print the function names.
Is this expected behaviour? How do I get perf to automatically extract
symbols from compressed files?
Problem n.2: sometimes perf prints bogus callchains (in particular
when the various printf() variants are involved).
|--1.46%-- 0xffffffffa048a275
| 0xffffffffa048c110
| 0xffffffffa04909e7
| |
| |--80.00%-- 0xffffffffa0490afa
| | 0xffffffffa048ee8b
| | 0xffffffffa048ba8f
| | bit_bmove
| | fbcon_redraw_blit.isra.22
| | fbcon_scroll
| | scrup
| | lf
| | do_con_trol
| | do_con_write.part.18
| | con_write
| | do_output_char
| | n_tty_write
| | tty_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| | |
| | |--50.00%-- vfprintf
| | | 0x0 <---HERE
| | |
| | |--25.00%-- main
| | | __libc_start_main
| | | _start
| | | 0x0 <---HERE
| | |
| | --25.00%-- 0x7facc1df12a0 HERE
To which function would 0x7facc1df12a0 match? It does not even seem to
be in the mapped address space of "ls". I am clueless.
Thanks for your time.
Bye
Mauro
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: Two questions on perf dwarf callchains
2014-01-17 10:21 Mauro Andreolini
@ 2014-01-17 20:11 ` Andi Kleen
2014-01-17 23:37 ` Mauro Andreolini
2014-01-29 11:18 ` Jiri Olsa
2014-01-29 13:02 ` Mauro Andreolini
2 siblings, 1 reply; 18+ messages in thread
From: Andi Kleen @ 2014-01-17 20:11 UTC (permalink / raw)
To: mauro.andreolini; +Cc: linux-perf-users
Mauro Andreolini <mauro.andreolini@gmail.com> writes:
> file to get perf report to print the function names.
> Is this expected behaviour? How do I get perf to automatically extract
> symbols from compressed files?
Someone would need to write the code to do that I guess.
>
> To which function would 0x7facc1df12a0 match? It does not even seem to
> be in the mapped address space of "ls". I am clueless.
The unwinder got confused. A common cause is incomplete or partially
broken dwarf unwind information. May be a compiler problem.
-Andi
--
ak@linux.intel.com -- Speaking for myself only
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Two questions on perf dwarf callchains
2014-01-17 20:11 ` Andi Kleen
@ 2014-01-17 23:37 ` Mauro Andreolini
2014-01-18 4:42 ` Andi Kleen
0 siblings, 1 reply; 18+ messages in thread
From: Mauro Andreolini @ 2014-01-17 23:37 UTC (permalink / raw)
To: Andi Kleen; +Cc: linux-perf-users
Il giorno ven, 17/01/2014 alle 12.11 -0800, Andi Kleen ha scritto:
> Mauro Andreolini <mauro.andreolini@gmail.com> writes:
> > file to get perf report to print the function names.
> > Is this expected behaviour? How do I get perf to automatically extract
> > symbols from compressed files?
>
> Someone would need to write the code to do that I guess.
OK.
> >
> > To which function would 0x7facc1df12a0 match? It does not even seem to
> > be in the mapped address space of "ls". I am clueless.
>
> The unwinder got confused. A common cause is incomplete or partially
> broken dwarf unwind information. May be a compiler problem.
Thanks for your time. Right now I am using gcc 4.8.2 20131219
(prerelease) and libunwind 1.1. I noticed the following
compile flags: "-O2 -g -fvar-tracking-assignments". I thought that
maybe optimizations like function inlining and the like would not play
together with debugging info, so I changed them to "-Og -g3
-fvar-tracking-assignments" and recompiled anything, albeit with no
luck. I must be missing something big here. How can I be sure to give
correct dwarf unwind information?
Mauro
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Two questions on perf dwarf callchains
2014-01-17 23:37 ` Mauro Andreolini
@ 2014-01-18 4:42 ` Andi Kleen
2014-01-22 14:40 ` Mauro Andreolini
0 siblings, 1 reply; 18+ messages in thread
From: Andi Kleen @ 2014-01-18 4:42 UTC (permalink / raw)
To: Mauro Andreolini; +Cc: Andi Kleen, linux-perf-users
> Thanks for your time. Right now I am using gcc 4.8.2 20131219
> (prerelease) and libunwind 1.1. I noticed the following
> compile flags: "-O2 -g -fvar-tracking-assignments". I thought that
> maybe optimizations like function inlining and the like would not play
> together with debugging info, so I changed them to "-Og -g3
> -fvar-tracking-assignments" and recompiled anything, albeit with no
> luck. I must be missing something big here. How can I be sure to give
> correct dwarf unwind information?
The dwarf information should be present unless explicitely disabled.
It's either a libunwind or compiler problem. You could file a gcc bug
I suppose. Also check if gdb can backtrace through it. If that works
it's likely unwind.
unwind problems are unfortunately somewhat common (but most seem to
be in handwritten assembler)
You'll also notice that dwarf unwind profiling is very expensive in overhead.
I usually avoid it because of that.
If you have a Haswell system one alternative is the LBR callstack
patchkit on Linux kernel (unfortunately not merged so far)
-Andi
--
ak@linux.intel.com -- Speaking for myself only.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Two questions on perf dwarf callchains
2014-01-18 4:42 ` Andi Kleen
@ 2014-01-22 14:40 ` Mauro Andreolini
2014-01-22 17:55 ` Andi Kleen
0 siblings, 1 reply; 18+ messages in thread
From: Mauro Andreolini @ 2014-01-22 14:40 UTC (permalink / raw)
To: Andi Kleen; +Cc: linux-perf-users
Il giorno sab, 18/01/2014 alle 05.42 +0100, Andi Kleen ha scritto:
> The dwarf information should be present unless explicitely disabled.
> It's either a libunwind or compiler problem. You could file a gcc bug
> I suppose. Also check if gdb can backtrace through it. If that works
> it's likely unwind.
gdb can backtrace perfectly through ls -lR /. Also, recompiling the
whole system with -fno-omit-frame-pointer makes perf record & report
work like a charm. So I guess it must be the unwinder.
> unwind problems are unfortunately somewhat common (but most seem to
> be in handwritten assembler)
>
> You'll also notice that dwarf unwind profiling is very expensive in overhead.
> I usually avoid it because of that.
Yes, I did, unfortunately.
I'll look into the other issue regarding the compressed kernel modules.
Again, thanks for your time.
Cheers
Mauro
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Two questions on perf dwarf callchains
2014-01-22 14:40 ` Mauro Andreolini
@ 2014-01-22 17:55 ` Andi Kleen
2014-01-22 18:02 ` Mauro Andreolini
0 siblings, 1 reply; 18+ messages in thread
From: Andi Kleen @ 2014-01-22 17:55 UTC (permalink / raw)
To: Mauro Andreolini; +Cc: Andi Kleen, linux-perf-users
On Wed, Jan 22, 2014 at 03:40:26PM +0100, Mauro Andreolini wrote:
> Il giorno sab, 18/01/2014 alle 05.42 +0100, Andi Kleen ha scritto:
> > The dwarf information should be present unless explicitely disabled.
> > It's either a libunwind or compiler problem. You could file a gcc bug
> > I suppose. Also check if gdb can backtrace through it. If that works
> > it's likely unwind.
> gdb can backtrace perfectly through ls -lR /. Also, recompiling the
You tested it on the same IP as perf failed on, correct?
It will depend on the exact location.
> whole system with -fno-omit-frame-pointer makes perf record & report
> work like a charm. So I guess it must be the unwinder.
Please report it to the libunwind developers.
-Andi
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Two questions on perf dwarf callchains
2014-01-22 17:55 ` Andi Kleen
@ 2014-01-22 18:02 ` Mauro Andreolini
0 siblings, 0 replies; 18+ messages in thread
From: Mauro Andreolini @ 2014-01-22 18:02 UTC (permalink / raw)
To: Andi Kleen; +Cc: linux-perf-users
Il giorno mer, 22/01/2014 alle 18.55 +0100, Andi Kleen ha scritto:
> On Wed, Jan 22, 2014 at 03:40:26PM +0100, Mauro Andreolini wrote:
> > Il giorno sab, 18/01/2014 alle 05.42 +0100, Andi Kleen ha scritto:
> > > The dwarf information should be present unless explicitely disabled.
> > > It's either a libunwind or compiler problem. You could file a gcc bug
> > > I suppose. Also check if gdb can backtrace through it. If that works
> > > it's likely unwind.
> > gdb can backtrace perfectly through ls -lR /. Also, recompiling the
>
> You tested it on the same IP as perf failed on, correct?
> It will depend on the exact location.
Yes, I did.
>
> > whole system with -fno-omit-frame-pointer makes perf record & report
> > work like a charm. So I guess it must be the unwinder.
>
> Please report it to the libunwind developers.
OK. Will do.
Mauro
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Two questions on perf dwarf callchains
2014-01-17 10:21 Mauro Andreolini
2014-01-17 20:11 ` Andi Kleen
@ 2014-01-29 11:18 ` Jiri Olsa
2014-01-29 15:09 ` Mauro Andreolini
2014-01-29 13:02 ` Mauro Andreolini
2 siblings, 1 reply; 18+ messages in thread
From: Jiri Olsa @ 2014-01-29 11:18 UTC (permalink / raw)
To: Mauro Andreolini; +Cc: linux-perf-users
On Fri, Jan 17, 2014 at 11:21:23AM +0100, Mauro Andreolini wrote:
> Hi all,
hi
>
> No kallsyms or vmlinux with build-id 4c6ab941f31dde04c2a481430218994849927c8d was found
> [ext4] with build id 4c6ab941f31dde04c2a481430218994849927c8d not found, continuing without symbols
> No kallsyms or vmlinux with build-id 21f163b0a8a8b924efdd173b8f83ad723d378958 was found
> [drm] with build id 21f163b0a8a8b924efdd173b8f83ad723d378958 not found, continuing without symbols
> No kallsyms or vmlinux with build-id d0bd7dd69886c2dccde18dee760b6d39cc0b5a55 was found[qxl] with build id d0bd7dd69886c2dccde18dee760b6d39cc0b5a55 not found, continuing without symbols
> No kallsyms or vmlinux with build-id 8d13e6f0c4d4f8b4628c8c4bd6c660b0a00576df was found[ttm] with build id 8d13e6f0c4d4f8b4628c8c4bd6c660b0a00576df not found, continuing without symbols
hum, do you have kernel-debuginfo* packages installed?
SNIP
>
> Problem n.2: sometimes perf prints bogus callchains (in particular
> when the various printf() variants are involved).
>
> |--1.46%-- 0xffffffffa048a275
> | 0xffffffffa048c110
> | 0xffffffffa04909e7
> | |
> | |--80.00%-- 0xffffffffa0490afa
> | | 0xffffffffa048ee8b
> | | 0xffffffffa048ba8f
> | | bit_bmove
> | | fbcon_redraw_blit.isra.22
> | | fbcon_scroll
> | | scrup
> | | lf
> | | do_con_trol
> | | do_con_write.part.18
> | | con_write
> | | do_output_char
> | | n_tty_write
> | | tty_write
> | | vfs_write
> | | sys_write
> | | system_call_fastpath
> | | __GI___libc_write
> | | |
> | | |--50.00%-- vfprintf
> | | | 0x0 <---HERE
> | | |
> | | |--25.00%-- main
> | | | __libc_start_main
> | | | _start
> | | | 0x0 <---HERE
> | | |
> | | --25.00%-- 0x7facc1df12a0 HERE
this one could be caused by:
http://marc.info/?l=linux-kernel&m=134934717011451&w=2
I never managed to push this change through.. need to revisit it ;-)
if you want to try, I rebased the patch and put it in here:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
branch perf/rsp_fix1
commit:
https://git.kernel.org/cgit/linux/kernel/git/jolsa/perf.git/commit/?h=perf/rsp_fix1&id=5eed39431f9b1e65e740a2e3e197c79f0a1bcf6b
it still fixies some of the backtraces for me
jirka
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Two questions on perf dwarf callchains
2014-01-29 11:18 ` Jiri Olsa
@ 2014-01-29 15:09 ` Mauro Andreolini
2014-01-30 11:47 ` Jiri Olsa
0 siblings, 1 reply; 18+ messages in thread
From: Mauro Andreolini @ 2014-01-29 15:09 UTC (permalink / raw)
To: Jiri Olsa; +Cc: linux-perf-users
Il giorno mer, 29/01/2014 alle 12.18 +0100, Jiri Olsa ha scritto:
> this one could be caused by:
> http://marc.info/?l=linux-kernel&m=134934717011451&w=2
> I never managed to push this change through.. need to revisit it ;-)
>
> if you want to try, I rebased the patch and put it in here:
> git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
> branch perf/rsp_fix1
>
> commit:
> https://git.kernel.org/cgit/linux/kernel/git/jolsa/perf.git/commit/?h=perf/rsp_fix1&id=5eed39431f9b1e65e740a2e3e197c79f0a1bcf6b
>
> it still fixies some of the backtraces for me
Hi, the patch improves things considerably. However, I still get several
"0x0" on top of the _start() function, e.g.:
15.79% ls [kernel.kallsyms] [k]
_raw_spin_unlock_irqrestore
|
--- _raw_spin_unlock_irqrestore
|
|--33.33%-- console_unlock
| con_write
| n_tty_write
| tty_write
| vfs_write
| sys_write
| system_call
| __GI___libc_write
| _IO_file_write@@GLIBC_2.2.5
| new_do_write
| _IO_do_write@@GLIBC_2.2.5
| _IO_file_overflow@@GLIBC_2.2.5
| print_many_per_line
| print_current_files
| print_dir
| main
| __libc_start_main
| _start
| 0x0 <---- HERE
I also get an occasional screw in the dynamic linker code path:
5.26% ls [kernel.kallsyms] [k]
lock_release
|
--- lock_release
__mem_cgroup_try_charge
mem_cgroup_charge_common
mem_cgroup_newpage_charge
do_wp_page
handle_mm_fault
__do_page_fault
do_page_fault
do_async_page_fault
async_page_fault
__init_cpu_features
0x7fd2357470de <---- HERE
_dl_relocate_object
dl_main
_dl_sysdep_start
_dl_start
_dl_start_user
and in the exit() path:
5.26% ls [kernel.kallsyms] [k]
do_raw_spin_trylock
|
--- do_raw_spin_trylock
unmap_single_vma
unmap_vmas
exit_mmap
mmput
do_exit
do_group_exit
0xffffffff81068184
system_call
Other than that, it looks perfect. Thanks a lot!
Mauro
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: Two questions on perf dwarf callchains
2014-01-29 15:09 ` Mauro Andreolini
@ 2014-01-30 11:47 ` Jiri Olsa
2014-01-30 14:43 ` Mauro Andreolini
0 siblings, 1 reply; 18+ messages in thread
From: Jiri Olsa @ 2014-01-30 11:47 UTC (permalink / raw)
To: Mauro Andreolini; +Cc: linux-perf-users
On Wed, Jan 29, 2014 at 04:09:06PM +0100, Mauro Andreolini wrote:
> Il giorno mer, 29/01/2014 alle 12.18 +0100, Jiri Olsa ha scritto:
SNIP
> | new_do_write
> | _IO_do_write@@GLIBC_2.2.5
> | _IO_file_overflow@@GLIBC_2.2.5
> | print_many_per_line
> | print_current_files
> | print_dir
> | main
> | __libc_start_main
> | _start
> | 0x0 <---- HERE
zero's are taken care by:
http://marc.info/?t=138909914500003&r=1&w=2
within following patchset:
http://marc.info/?l=linux-kernel&m=138909914404574&w=2
which implements libdw dwarf unwind.. you might
actualy want to try it, it seems to be little faster
than libunwind unwinder
>
> I also get an occasional screw in the dynamic linker code path:
>
> 5.26% ls [kernel.kallsyms] [k]
> lock_release
> |
> --- lock_release
> __mem_cgroup_try_charge
> mem_cgroup_charge_common
> mem_cgroup_newpage_charge
> do_wp_page
> handle_mm_fault
> __do_page_fault
> do_page_fault
> do_async_page_fault
> async_page_fault
> __init_cpu_features
> 0x7fd2357470de <---- HERE
not sure if that's just untranslatable part (.plt .got?)
or an error.. need to be checked some day ;-)
jirka
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Two questions on perf dwarf callchains
2014-01-30 11:47 ` Jiri Olsa
@ 2014-01-30 14:43 ` Mauro Andreolini
2014-01-30 15:40 ` Jiri Olsa
0 siblings, 1 reply; 18+ messages in thread
From: Mauro Andreolini @ 2014-01-30 14:43 UTC (permalink / raw)
To: Jiri Olsa; +Cc: linux-perf-users
> zero's are taken care by:
> http://marc.info/?t=138909914500003&r=1&w=2
> within following patchset:
> http://marc.info/?l=linux-kernel&m=138909914404574&w=2
> which implements libdw dwarf unwind.. you might
> actualy want to try it, it seems to be little faster
> than libunwind unwinder
Since I do not yet have elfutils 0.158 installed, I tried
to apply the single patch both to linux-3.12 and to the kernel you
pointed me at, and used perf with the libunwind unwinder. With the 3.12
kernel I get segfaults on perf report; with the 3.13 branch zeroes are
still present. Maybe I messed up.
I think I'll wait for the newest elfutils and give a shot ad the
libdw unwinder.
> >
> > I also get an occasional screw in the dynamic linker code path:
> >
> > 5.26% ls [kernel.kallsyms] [k]
> > lock_release
> > |
> > --- lock_release
> > __mem_cgroup_try_charge
> > mem_cgroup_charge_common
> > mem_cgroup_newpage_charge
> > do_wp_page
> > handle_mm_fault
> > __do_page_fault
> > do_page_fault
> > do_async_page_fault
> > async_page_fault
> > __init_cpu_features
> > 0x7fd2357470de <---- HERE
>
> not sure if that's just untranslatable part (.plt .got?)
> or an error.. need to be checked some day ;-)
OK. Thanks for your time.
Mauro
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Two questions on perf dwarf callchains
2014-01-30 14:43 ` Mauro Andreolini
@ 2014-01-30 15:40 ` Jiri Olsa
2014-02-03 14:30 ` Mauro Andreolini
0 siblings, 1 reply; 18+ messages in thread
From: Jiri Olsa @ 2014-01-30 15:40 UTC (permalink / raw)
To: Mauro Andreolini; +Cc: linux-perf-users
On Thu, Jan 30, 2014 at 03:43:19PM +0100, Mauro Andreolini wrote:
> > zero's are taken care by:
> > http://marc.info/?t=138909914500003&r=1&w=2
> > within following patchset:
> > http://marc.info/?l=linux-kernel&m=138909914404574&w=2
>
> > which implements libdw dwarf unwind.. you might
> > actualy want to try it, it seems to be little faster
> > than libunwind unwinder
> Since I do not yet have elfutils 0.158 installed, I tried
> to apply the single patch both to linux-3.12 and to the kernel you
> pointed me at, and used perf with the libunwind unwinder. With the 3.12
> kernel I get segfaults on perf report; with the 3.13 branch zeroes are
> still present. Maybe I messed up.
> I think I'll wait for the newest elfutils and give a shot ad the
> libdw unwinder.
I use elfutils git tree ;-)
git://git.fedorahosted.org/git/elfutils.git
and the latest patchset (recently rebased) is in here:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/core_libdw_unwind
jirka
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Two questions on perf dwarf callchains
2014-01-30 15:40 ` Jiri Olsa
@ 2014-02-03 14:30 ` Mauro Andreolini
2014-02-04 15:28 ` Mauro Andreolini
0 siblings, 1 reply; 18+ messages in thread
From: Mauro Andreolini @ 2014-02-03 14:30 UTC (permalink / raw)
To: Jiri Olsa; +Cc: linux-perf-users
> I use elfutils git tree ;-)
> git://git.fedorahosted.org/git/elfutils.git
Installed fine.
> and the latest patchset (recently rebased) is in here:
> git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
> perf/core_libdw_unwind
Pulled the repo, switched branch to core_libdw_unwind, recompiled
the kernel and rebooted fine. When I try to compile perf
(make tools/perf NO_LIBUNWIND=1 PYTHON=/usr/bin/python2) I get this
error message, though:
CHK -fstack-protector-all
CHK -Wstack-protector
CHK -Wvolatile-register-var
CHK bionic
CHK libelf
CHK libdw
CHK -DLIBELF_MMAP
CHK libaudit
CHK libnewt
CHK gtk2
CHK -DHAVE_GTK_INFO_BAR
CHK perl
CHK python
CHK python version
CHK libbfd
CHK liberty
CHK libz
CHK demangle
CHK -DHAVE_STRLCPY
CHK -DHAVE_ON_EXIT
CHK -DBACKTRACE_SUPPORT
CHK libnuma
* new build flags or prefix
PERF_VERSION = 3.9.rc2.g7c6baa
...
[SNIP make output]
...
In file included from util/scripting-engines/trace-event-perl.c:30:0:
/usr/lib/perl5/core_perl/CORE/perl.h:548:5: error:
"SILENT_NO_TAINT_SUPPORT" is not defined [-Werror=undef]
#if SILENT_NO_TAINT_SUPPORT && !defined(NO_TAINT_SUPPORT)
^
/usr/lib/perl5/core_perl/CORE/perl.h:556:5: error: "NO_TAINT_SUPPORT" is
not defined [-Werror=undef]
#if NO_TAINT_SUPPORT
^
In file included from /usr/lib/perl5/core_perl/CORE/perl.h:3471:0,
from util/scripting-engines/trace-event-perl.c:30:
/usr/lib/perl5/core_perl/CORE/sv.h:1455:5: error: "NO_TAINT_SUPPORT" is
not defined [-Werror=undef]
#if NO_TAINT_SUPPORT
^
In file included from /usr/lib/perl5/core_perl/CORE/perl.h:3472:0,
from util/scripting-engines/trace-event-perl.c:30:
/usr/lib/perl5/core_perl/CORE/regexp.h:436:5: error: "NO_TAINT_SUPPORT"
is not defined [-Werror=undef]
#if NO_TAINT_SUPPORT
^
In file included from /usr/lib/perl5/core_perl/CORE/hv.h:593:0,
from /usr/lib/perl5/core_perl/CORE/perl.h:3480,
from util/scripting-engines/trace-event-perl.c:30:
/usr/lib/perl5/core_perl/CORE/hv_func.h: In function
‘S_perl_hash_siphash_2_4’:
/usr/lib/perl5/core_perl/CORE/hv_func.h:222:3: error: switch missing
default case [-Werror=switch-default]
switch( left )
^
/usr/lib/perl5/core_perl/CORE/hv_func.h: In function
‘S_perl_hash_superfast’:
/usr/lib/perl5/core_perl/CORE/hv_func.h:274:5: error: switch missing
default case [-Werror=switch-default]
switch (rem) { \
^
/usr/lib/perl5/core_perl/CORE/hv_func.h: In function
‘S_perl_hash_murmur3’:
/usr/lib/perl5/core_perl/CORE/hv_func.h:398:5: error: switch missing
default case [-Werror=switch-default]
switch(bytes_in_carry) { /* how many bytes in carry */
^
cc1: all warnings being treated as errors
make: *** [util/scripting-engines/trace-event-perl.o] Errore 1
Mauro
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: Two questions on perf dwarf callchains
2014-02-03 14:30 ` Mauro Andreolini
@ 2014-02-04 15:28 ` Mauro Andreolini
2014-02-04 15:35 ` Jiri Olsa
0 siblings, 1 reply; 18+ messages in thread
From: Mauro Andreolini @ 2014-02-04 15:28 UTC (permalink / raw)
To: Jiri Olsa; +Cc: linux-perf-users
> /usr/lib/perl5/core_perl/CORE/hv_func.h:398:5: error: switch missing
> default case [-Werror=switch-default]
> switch(bytes_in_carry) { /* how many bytes in carry */
> ^
> cc1: all warnings being treated as errors
> make: *** [util/scripting-engines/trace-event-perl.o] Errore 1
I solved this one by applying this:
https://lkml.org/lkml/2013/7/23/60
Wow! Unwinding is noticeably faster than libunwind! The trace is almost
perfect. I think I found only one unresolved symbol (it's always the
same one, maybe I screwed up the compilation):
0.01% ls [kernel.kallsyms] [k] do_raw_read_unlock
|
--- do_raw_read_unlock
_raw_read_unlock
0xffffffffa0154479 <--- HERE
jbd2__journal_start
__ext4_journal_start_sb
ext4_dirty_inode
__mark_inode_dirty
update_time
touch_atime
sys_readlinkat
sys_readlink
system_call
__GI___readlink
get_link_name
gobble_file
print_dir
main
__libc_start_main
_start
Is this going into mainline soon?
Mauro
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: Two questions on perf dwarf callchains
2014-02-04 15:28 ` Mauro Andreolini
@ 2014-02-04 15:35 ` Jiri Olsa
2014-02-04 16:01 ` Mauro Andreolini
0 siblings, 1 reply; 18+ messages in thread
From: Jiri Olsa @ 2014-02-04 15:35 UTC (permalink / raw)
To: Mauro Andreolini; +Cc: linux-perf-users, Jean Pihet
On Tue, Feb 04, 2014 at 04:28:59PM +0100, Mauro Andreolini wrote:
> > /usr/lib/perl5/core_perl/CORE/hv_func.h:398:5: error: switch missing
> > default case [-Werror=switch-default]
> > switch(bytes_in_carry) { /* how many bytes in carry */
> > ^
> > cc1: all warnings being treated as errors
> > make: *** [util/scripting-engines/trace-event-perl.o] Errore 1
> I solved this one by applying this:
> https://lkml.org/lkml/2013/7/23/60
>
> Wow! Unwinding is noticeably faster than libunwind! The trace is almost
> perfect. I think I found only one unresolved symbol (it's always the
> same one, maybe I screwed up the compilation):
>
> 0.01% ls [kernel.kallsyms] [k] do_raw_read_unlock
> |
> --- do_raw_read_unlock
> _raw_read_unlock
> 0xffffffffa0154479 <--- HERE
hum, is there any symbol around that address?
you can check /proc/kallsyms or vmlinux file from debug package
> jbd2__journal_start
> __ext4_journal_start_sb
> ext4_dirty_inode
> __mark_inode_dirty
> update_time
> touch_atime
> sys_readlinkat
> sys_readlink
> system_call
> __GI___readlink
> get_link_name
> gobble_file
> print_dir
> main
> __libc_start_main
> _start
>
> Is this going into mainline soon?
hopefully ;-) it's on review now..
thanks a lot for testing! please let us know any issues you spot
jirka
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: Two questions on perf dwarf callchains
2014-02-04 15:35 ` Jiri Olsa
@ 2014-02-04 16:01 ` Mauro Andreolini
0 siblings, 0 replies; 18+ messages in thread
From: Mauro Andreolini @ 2014-02-04 16:01 UTC (permalink / raw)
To: Jiri Olsa; +Cc: linux-perf-users, Jean Pihet
> I think I found only one unresolved symbol (it's always the
> > same one, maybe I screwed up the compilation):
> >
> > 0.01% ls [kernel.kallsyms] [k] do_raw_read_unlock
> > |
> > --- do_raw_read_unlock
> > _raw_read_unlock
> > 0xffffffffa0154479 <--- HERE
>
> hum, is there any symbol around that address?
> you can check /proc/kallsyms or vmlinux file from debug package
[Damn sorry for using my other e-mail account, I made a mistake].
/proc/kallsyms shows this:
ffffffffa01780c0 R crc16_table [crc16]
ffffffffa0178000 T crc16 [crc16]
ffffffffa0154000 start_this_handle [jbd2]
ffffffffa01669e0 b __key.27934 [jbd2]
I guess it's in start_this_handle()?
> > Is this going into mainline soon?
>
> hopefully ;-) it's on review now..
>
> thanks a lot for testing! please let us know any issues you spot
You welcome!
Mauro
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Two questions on perf dwarf callchains
2014-01-17 10:21 Mauro Andreolini
2014-01-17 20:11 ` Andi Kleen
2014-01-29 11:18 ` Jiri Olsa
@ 2014-01-29 13:02 ` Mauro Andreolini
2 siblings, 0 replies; 18+ messages in thread
From: Mauro Andreolini @ 2014-01-29 13:02 UTC (permalink / raw)
To: linux-perf-users
Il giorno mer, 29/01/2014 alle 12.18 +0100, Jiri Olsa ha scritto:
> On Fri, Jan 17, 2014 at 11:21:23AM +0100, Mauro Andreolini wrote:
> > Hi all,
>
> hi
>
> >
> > No kallsyms or vmlinux with build-id
4c6ab941f31dde04c2a481430218994849927c8d was found
> > [ext4] with build id 4c6ab941f31dde04c2a481430218994849927c8d not
found, continuing without symbols
> > No kallsyms or vmlinux with build-id
21f163b0a8a8b924efdd173b8f83ad723d378958 was found
> > [drm] with build id 21f163b0a8a8b924efdd173b8f83ad723d378958 not
found, continuing without symbols
> > No kallsyms or vmlinux with build-id
d0bd7dd69886c2dccde18dee760b6d39cc0b5a55 was found[qxl] with build id
d0bd7dd69886c2dccde18dee760b6d39cc0b5a55 not found, continuing without
symbols
> > No kallsyms or vmlinux with build-id
8d13e6f0c4d4f8b4628c8c4bd6c660b0a00576df was found[ttm] with build id
8d13e6f0c4d4f8b4628c8c4bd6c660b0a00576df not found, continuing without
symbols
>
> hum, do you have kernel-debuginfo* packages installed?
Hi, thanks for your time. I am using Arch Linux, which does not provide
separate -debuginfo packages as Fedora. However, I have compiled the
kernel with debug-info and with frame pointers. Now I am not seeing
those messages anymore. Sorry for that noise.
>
>
> SNIP
>
> >
> > Problem n.2: sometimes perf prints bogus callchains (in particular
> > when the various printf() variants are involved).
> >
> > |--1.46%-- 0xffffffffa048a275
> > | 0xffffffffa048c110
> > | 0xffffffffa04909e7
> > | |
> > | |--80.00%-- 0xffffffffa0490afa
> > | | 0xffffffffa048ee8b
> > | | 0xffffffffa048ba8f
> > | | bit_bmove
> > | | fbcon_redraw_blit.isra.22
> > | | fbcon_scroll
> > | | scrup
> > | | lf
> > | | do_con_trol
> > | | do_con_write.part.18
> > | | con_write
> > | | do_output_char
> > | | n_tty_write
> > | | tty_write
> > | | vfs_write
> > | | sys_write
> > | | system_call_fastpath
> > | | __GI___libc_write
> > | | |
> > | | |--50.00%-- vfprintf
> > | | | 0x0
<---HERE
> > | | |
> > | | |--25.00%-- main
> > | | | __libc_start_main
> > | | | _start
> > | | | 0x0
<---HERE
> > | | |
> > | | --25.00%-- 0x7facc1df12a0
HERE
>
> this one could be caused by:
> http://marc.info/?l=linux-kernel&m=134934717011451&w=2
> I never managed to push this change through.. need to revisit it ;-)
>
> if you want to try, I rebased the patch and put it in here:
> git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
> branch perf/rsp_fix1
>
> commit:
>
https://git.kernel.org/cgit/linux/kernel/git/jolsa/perf.git/commit/?h=perf/rsp_fix1&id=5eed39431f9b1e65e740a2e3e197c79f0a1bcf6b
>
> it still fixies some of the backtraces for me
Oh, thanks a lot. I will try this ASAP. Meanwhile, using the frame
pointer
for kernel, libraries and executables works like a charm.
Thanks for your time.
Mauro
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2014-02-04 16:01 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-01-17 23:42 Two questions on perf dwarf callchains Mauro Andreolini
-- strict thread matches above, loose matches on Subject: below --
2014-01-17 10:21 Mauro Andreolini
2014-01-17 20:11 ` Andi Kleen
2014-01-17 23:37 ` Mauro Andreolini
2014-01-18 4:42 ` Andi Kleen
2014-01-22 14:40 ` Mauro Andreolini
2014-01-22 17:55 ` Andi Kleen
2014-01-22 18:02 ` Mauro Andreolini
2014-01-29 11:18 ` Jiri Olsa
2014-01-29 15:09 ` Mauro Andreolini
2014-01-30 11:47 ` Jiri Olsa
2014-01-30 14:43 ` Mauro Andreolini
2014-01-30 15:40 ` Jiri Olsa
2014-02-03 14:30 ` Mauro Andreolini
2014-02-04 15:28 ` Mauro Andreolini
2014-02-04 15:35 ` Jiri Olsa
2014-02-04 16:01 ` Mauro Andreolini
2014-01-29 13:02 ` Mauro Andreolini
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).