[PATCH] perf symbols: don't use modules or try vmlinux unless needed

[PATCH] perf symbols: don't use modules or try vmlinux unless needed

[Jamie Iles]

Commit 75be6cf (perf symbols: Make symbol_conf global) does what it says
on the tin, but also initialises the member fields use_modules and
try_vmlinux_path to true rather than the 'false' value they would have
had when symbol_conf was static.

When there is no vmlinux on the system and modules loaded, perf top will
not show any kernel symbols and perf report will only list raw IP's
rather than symbol names. Unloading the modules will allow perf to give
normal output.

Restore previous behaviour by initialising these fields to false.

Signed-off-by: Jamie Iles <jamie@jamieiles.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Paul Mackerras <paulus@samba.org>
---
 tools/perf/util/symbol.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index 79ca6a0..9467c29 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -40,8 +40,8 @@ static char **vmlinux_path;
 
 struct symbol_conf symbol_conf = {
 	.exclude_other	  = true,
-	.use_modules	  = true,
-	.try_vmlinux_path = true,
+	.use_modules	  = false,
+	.try_vmlinux_path = false,
 };
 
 bool dso__loaded(const struct dso *self, enum map_type type)
-- 
1.6.5.4

Re: [PATCH] perf symbols: don't use modules or try vmlinux unless needed

[Arnaldo Carvalho de Melo]

Em Mon, Jan 04, 2010 at 10:23:54AM +0000, Jamie Iles escreveu:
> Commit 75be6cf (perf symbols: Make symbol_conf global) does what it says
> on the tin, but also initialises the member fields use_modules and
> try_vmlinux_path to true rather than the 'false' value they would have
> had when symbol_conf was static.
> 
> When there is no vmlinux on the system and modules loaded, perf top will
> not show any kernel symbols and perf report will only list raw IP's
> rather than symbol names. Unloading the modules will allow perf to give
> normal output.
> 
> Restore previous behaviour by initialising these fields to false.

James,

	I just tried using perf top here in such conditions:

[root@ana ~]# strace -e open -o /tmp/perf.open.strace perf top
------------------------------------------------------------------------
   PerfTop:      65 irqs/sec kernel:64.6% [1000Hz cycles], (all, 2 CPUs)
------------------------------------------------------------------------

             samples  pcnt function           DSO
             _______ _____ __________________ _________________

               12.00  7.0% format_decode      [kernel.kallsyms]
               10.00  5.8% __strstr_ia32      /lib/libc-2.11.so
                9.00  5.3% read_hpet          [kernel.kallsyms]
                7.00  4.1% module_get_kallsym [kernel.kallsyms]
                5.00  2.9% vsnprintf          [kernel.kallsyms]
                5.00  2.9% __memchr           /lib/libc-2.11.so
/usr/lib/libgtk-x11-2.0.so.0.1800.5.#prelink#.aAFN89 was updated,
restart the long running apps that use it!

[root@ana ~]# grep vmlinux /tmp/perf.open.strace 
open("vmlinux", O_RDONLY|O_LARGEFILE)   = -1 ENOENT (No such file or directory)
open("/boot/vmlinux", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/boot/vmlinux-2.6.31.9-174.fc12.i686", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/lib/modules/2.6.31.9-174.fc12.i686/build/vmlinux", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/debug/lib/modules/2.6.31.9-174.fc12.i686/vmlinux", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)

And it is resolving the kernel symbols.

Maybe some changes I made that aren't yet in Ingo's tree or upstream
fixed it, can you please try after those patches are merged?

- Arnaldo

Re: [PATCH] perf symbols: don't use modules or try vmlinux unless needed

[Jamie Iles]

On Fri, Jan 08, 2010 at 09:36:57AM -0200, Arnaldo Carvalho de Melo wrote:
> Ecanfcan Mon, Jan 04, 2010 at 10:23:54AM +0000, Jamie Iles escreveu:
> > Commit 75be6cf (perf symbols: Make symbol_conf global) does what it says
> > on the tin, but also initialises the member fields use_modules and
> > try_vmlinux_path to true rather than the 'false' value they would have
> > had when symbol_conf was static.
> > 
> > When there is no vmlinux on the system and modules loaded, perf top will
> > not show any kernel symbols and perf report will only list raw IP's
> > rather than symbol names. Unloading the modules will allow perf to give
> > normal output.
> > 
> > Restore previous behaviour by initialising these fields to false.
> 
> James,
> 
> 	I just tried using perf top here in such conditions:
> 
[snip]
> 
> And it is resolving the kernel symbols.
> 
> Maybe some changes I made that aren't yet in Ingo's tree or upstream
> fixed it, can you please try after those patches are merged?
Ok, could you let me know when the relevant ones are in there? From todays
next tree, without the patch I still only get userspace symbols (system loaded
with 'dd if=/dev/zero of=/dev/null' to make sure it's busy):

               49.00 17.6% T.276                   /root/tools/perf    
               25.00  9.0% __GI_strcmp             /lib/libc-2.8.so    
               21.00  7.6% symbol_filter           /root/tools/perf    
               17.00  6.1% strncmp                 /root/tools/perf    
	       ...

But with the patch applied output looks good:

              597.00 17.0% _raw_spin_unlock_irq        [kernel.kallsyms]
              589.00 16.8% lock_acquire                [kernel.kallsyms]
              367.00 10.4% _raw_spin_unlock_irqrestore [kernel.kallsyms]
              230.00  6.5% vector_swi                  [kernel.kallsyms]
              122.00  3.5% __memzero                   [kernel.kallsyms]
               71.00  2.0% lock_release                [kernel.kallsyms]
	       ...

Jamie

Analysing an ARM perf.data file on a x86-64 workstation

[Arnaldo Carvalho de Melo]

CCing linux-perf-users, this may be interesting when investigating
similar problems for other arches.

Thanks James, got the vmlinux file and and already found and fixed two
bugs, patches sent CCed to you.

Ok, now how does it look like?

[acme@doppio linux-2.6-tip]$ perf report -i ~/iles/perf.data --vmlinux ~/iles/vmlinux 
Failed to open /lib/ld-2.8.so, continuing without symbols
Failed to open /root/perf, continuing without symbols
Failed to open /lib/libc-2.8.so, continuing without symbols
# Samples: 40653353521
#
# Overhead          Command      Shared Object  Symbol
# ........  ...............  .................  ......
#
    99.94%          swapper           c0020ac4  [k] 0x000000c0020ac4
     0.04%             perf           c0068f48  [k] 0x000000c0068f48
     0.01%             perf  libc-2.8.so        [.] 0x00000000067198
     0.01%            sleep           c00acb44  [k] 0x000000c00acb44
     0.00%             perf  perf               [.] 0x0000000000265c
     0.00%             perf  ld-2.8.so          [.] 0x00000000015fa8
     0.00%            sleep  ld-2.8.so          [.] 0x00000000008ff0
     0.00%             perf  [kernel.kallsyms]  [k] smp_call_function_single
#
# (For a higher level overview, try: perf report --sort comm,dso)
#
[acme@doppio linux-2.6-tip]$

So we manage to decode at least one kernel entry, yay!

To make sure we really did it right, we can use -v to show the address
too:

[acme@doppio linux-2.6-tip]$ perf report -v -i ~/iles/perf.data --vmlinux ~/iles/vmlinux 
Failed to open /lib/ld-2.8.so, continuing without symbols
Failed to open /root/perf, continuing without symbols
Failed to open /lib/libc-2.8.so, continuing without symbols
# Samples: 40653353521
#
# Overhead          Command      Shared Object  Symbol
# ........  ...............  .................  ......
#
    99.94%          swapper           c0020ac4  0x00000000c0020ac4 ! [k] 0x000000c0020ac4
     0.04%             perf           c0068f48  0x00000000c0068f48 ! [k] 0x000000c0068f48
     0.01%             perf  /lib/libc-2.8.so   0x0000000000067198 K [.] 0x00000000067198
     0.01%            sleep           c00acb44  0x00000000c00acb44 ! [k] 0x000000c00acb44
     0.00%             perf  /root/perf         0x000000000000265c K [.] 0x0000000000265c
     0.00%             perf  /lib/ld-2.8.so     0x0000000000015fa8 K [.] 0x00000000015fa8
     0.00%            sleep  /lib/ld-2.8.so     0x0000000000008ff0 K [.] 0x00000000008ff0
     0.00%             perf  [kernel.kallsyms]  0x00000000c0068f48 ! [k] smp_call_function_single
#
# (For a higher level overview, try: perf report --sort comm,dso)
#
[acme@doppio linux-2.6-tip]$

Ok, so smp_call_function_single is 0xc0068f48, lets see if that matches
what is in the vmlinux symtab you provided me:

[acme@doppio linux-2.6-tip]$ file ~/iles/vmlinux 
/home/acme/iles/vmlinux: ELF 32-bit LSB executable, ARM, version 1 (SYSV), statically linked, not stripped
[acme@doppio linux-2.6-tip]$ readelf -s ~/iles/vmlinux | grep smp_call_function_single
 57421: c0068f18    60 FUNC    GLOBAL DEFAULT    3 smp_call_function_single
[acme@doppio linux-2.6-tip]$

Ok, it matches, as 0xc0068f18 <= 0xc0068f48 < (0xc0068f18 + 0x60).

But what about the other addresses for [k]ernel space perf hits such as
0xc0020ac4, 0xc0068f48, etc?

0xc0020ac4 is around here:

   385: 00000000     0 FILE    LOCAL  DEFAULT  ABS process.c
<SNIP>
   411: c0020aa0    40 FUNC    LOCAL  DEFAULT    3 default_idle
   412: c0020ac8     0 NOTYPE  LOCAL  DEFAULT    3 $a

so its well possible that this is really a very idle machine, right?

0xc0068f48 is around here:

  7997: 00000000     0 FILE    LOCAL  DEFAULT  ABS up.c
  7998: c0068f18     0 NOTYPE  LOCAL  DEFAULT    3 $a
  7999: c0068f50     0 NOTYPE  LOCAL  DEFAULT    3 $d

0xc00acb44 around here:

 13645: 00000000     0 FILE    LOCAL  DEFAULT  ABS mmap.c
<SNIP>
 13679: c00aca98   216 FUNC    LOCAL  DEFAULT    3 __vma_link_file
 13680: c00acb70     0 NOTYPE  LOCAL  DEFAULT    3 $a
 13681: c00acb70   428 FUNC    LOCAL  DEFAULT    3 unmap_region

We now need to try to investigate why these things are being
misannotated in the symtab.

- Arnaldo

Re: Analysing an ARM perf.data file on a x86-64 workstation

[Jamie Iles]

Hi Arnaldo,

On Tue, Jan 19, 2010 at 11:09:58AM -0200, Arnaldo Carvalho de Melo wrote:
> CCing linux-perf-users, this may be interesting when investigating
> similar problems for other arches.
> 
> Thanks James, got the vmlinux file and and already found and fixed two
> bugs, patches sent CCed to you.
> 
> Ok, now how does it look like?
Well, with both patches applied, I still don't see any kernel symbols in perf
top (even the raw addresses) or resolved names in report if I specify a
vmlinux or if one gets autodetected.

For example, when keeping the CPU busy with 'dd if=/dev/zero of=/dev/null', If
I record with 'perf record -af -- sleep 1' then:

root@picopc7302:~# ./perf record -af -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.071 MB perf.data (~3092 samples) ]
root@picopc7302:~# ./perf report --vmlinux=/tmp/vmlinux -v | head -10
no symbols found in /bin/busybox, maybe install a debug package?
# Samples: 435596970
#
# Overhead          Command           Shared Object  Symbol
# ........  ...............  ......................  ......
#   
    81.14%               dd                c0020ac8  0x00000000c0020ac8 ! [k] 0x000000c0020ac8
     9.26%             perf                c0071bfc  0x00000000c0071bfc ! [k] 0x000000c0071bfc
     3.53%               dd  /bin/busybox            0x000000000001d6f4 K [.] 0x0000000001d6f4
     1.55%            sleep                c006c3a0  0x00000000c006c3a0 ! [k] 0x000000c006c3a0
     1.38%               dd  /lib/libc-2.8.so        0x00000000000b4a2c d [.] __GI___libc_read
root@picopc7302:~# mv /boot/vmlinux /boot/vmlinux
root@picopc7302:~# mv /boot/vmlinux /boot/v
root@picopc7302:~# ./perf report -v | head -10
Looking at the vmlinux_path (5 entries long)
no symbols found in /bin/busybox, maybe install a debug package?
# Samples: 435596970
#
# Overhead          Command           Shared Object  Symbol
# ........  ...............  ......................  ......
#   
    32.57%               dd  [kernel.kallsyms]       0x00000000c02aaef0 k [k] _raw_spin_unlock_irqrestore
    12.30%               dd  [kernel.kallsyms]       0x00000000c006b5c4 k [k] lock_acquire
     6.23%               dd  [kernel.kallsyms]       0x00000000c006c220 k [k] lock_release
     5.38%               dd  [kernel.kallsyms]       0x00000000c006b9f0 k [k] lock_acquired
     5.04%               dd  [kernel.kallsyms]       0x00000000c0020ac8 k [k] vector_swi
> 
[snip]
> 
> But what about the other addresses for [k]ernel space perf hits such as
> 0xc0020ac4, 0xc0068f48, etc?
> 
> 0xc0020ac4 is around here:
> 
>    385: 00000000     0 FILE    LOCAL  DEFAULT  ABS process.c
> <SNIP>
>    411: c0020aa0    40 FUNC    LOCAL  DEFAULT    3 default_idle
>    412: c0020ac8     0 NOTYPE  LOCAL  DEFAULT    3 $a
> 
> so its well possible that this is really a very idle machine, right?
Yes, the run I sent you was pretty much just idling away.

Jamie

Re: Analysing an ARM perf.data file on a x86-64 workstation

[Arnaldo Carvalho de Melo]

Em Tue, Jan 19, 2010 at 01:51:45PM +0000, Jamie Iles escreveu:
> Hi Arnaldo,
> 
> On Tue, Jan 19, 2010 at 11:09:58AM -0200, Arnaldo Carvalho de Melo wrote:
> > CCing linux-perf-users, this may be interesting when investigating
> > similar problems for other arches.
> > 
> > Thanks James, got the vmlinux file and and already found and fixed two
> > bugs, patches sent CCed to you.
> > 
> > Ok, now how does it look like?
> Well, with both patches applied, I still don't see any kernel symbols in perf
> top (even the raw addresses) or resolved names in report if I specify a
> vmlinux or if one gets autodetected.

What if you try to do what I did, i.e. to try to decode the perf.data
file on your x86 workstation? Does specifying --vmlinux
/path/to/arm-vmlinux gets the symbols decoded?
 
- Arnaldo