how to use perf annotate on the kernel

how to use perf annotate on the kernel

[Stephane Eranian]

Arnaldo,

I am trying to understand how one is supposed
to use perf annotate to get sample correlations
with kernel symbols.

I do:

# perf record -a -e cycles:k dd if=/dev/zero of=/dev/null count=1000000
# perf report

# Events: 14K cycles
#
# Overhead          Command        Shared Object  Symbol
# ........  ...............  ...................  ......
#
     8.76%               dd  [kernel.kallsyms]    [k] __lock_acquire
     7.49%               dd  [kernel.kallsyms]    [k] sched_clock
     6.41%               dd  [kernel.kallsyms]    [k] __clear_user
     5.50%               dd  [kernel.kallsyms]    [k] sched_clock_local
     3.86%               dd  [kernel.kallsyms]    [k] cpu_clock
     3.82%               dd  [kernel.kallsyms]    [k] trace_hardirqs_off_caller
     3.50%               dd  [kernel.kallsyms]    [k] lock_release
     3.27%               dd  [kernel.kallsyms]    [k] system_call_after_swapgs
     3.15%               dd  [kernel.kallsyms]    [k] system_call

# perf annotate (or annotate -d '[kernel.kallsyms]'?)
Can't annotate __lock_acquire: No vmlinux file was found in the path

# perf annotate -k vmlinux
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file

Am I missing something here?
Thanks.

Re: how to use perf annotate on the kernel

[Arnaldo Carvalho de Melo]

Em Thu, Jun 03, 2010 at 07:23:02PM +0200, Stephane Eranian escreveu:
> Arnaldo,
> 
> I am trying to understand how one is supposed
> to use perf annotate to get sample correlations
> with kernel symbols.
> 
> I do:
> 
> # perf record -a -e cycles:k dd if=/dev/zero of=/dev/null count=1000000
> # perf report
> 
> # Events: 14K cycles
> #
> # Overhead          Command        Shared Object  Symbol
> # ........  ...............  ...................  ......
> #
>      8.76%               dd  [kernel.kallsyms]    [k] __lock_acquire
>      7.49%               dd  [kernel.kallsyms]    [k] sched_clock
> 
> # perf annotate (or annotate -d '[kernel.kallsyms]'?)
> Can't annotate __lock_acquire: No vmlinux file was found in the path
> 
> # perf annotate -k vmlinux
> objdump: '[kernel.kallsyms]': No such file

This looks like a bug in how it reports this problem, probably vmlinux
is not in the current directory, or it is but has a build-id that
doesn't matches the one in the perf.data file.

But the message is wrong, should be like it is in the tui, for this same
situation:

	ui_helpline__puts("No vmlinux file found, can't "
			  "annotate with just a kallsyms file");

> Am I missing something here?

If you don't specify it with --vmlinux/-k it will try to find it in one
of these places:

        vmlinux_path[vmlinux_path__nr_entries] = strdup("vmlinux");
        if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
                goto out_fail;
        ++vmlinux_path__nr_entries;
        vmlinux_path[vmlinux_path__nr_entries] = strdup("/boot/vmlinux");
        if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
                goto out_fail;
        ++vmlinux_path__nr_entries;
        snprintf(bf, sizeof(bf), "/boot/vmlinux-%s", uts.release);
        vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
        if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
                goto out_fail;
        ++vmlinux_path__nr_entries;
        snprintf(bf, sizeof(bf), "/lib/modules/%s/build/vmlinux", uts.release);
        vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
        if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
                goto out_fail;
        ++vmlinux_path__nr_entries;
        snprintf(bf, sizeof(bf), "/usr/lib/debug/lib/modules/%s/vmlinux",
                 uts.release);
        vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);

It also now looks at /sys/kernel/notes, gets the build-id, and looks up
in the build-id cache.

- Arnaldo

Re: how to use perf annotate on the kernel

[Arnaldo Carvalho de Melo]

Em Thu, Jun 03, 2010 at 02:54:45PM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Thu, Jun 03, 2010 at 07:23:02PM +0200, Stephane Eranian escreveu:
> > # perf annotate (or annotate -d '[kernel.kallsyms]'?)
> > Can't annotate __lock_acquire: No vmlinux file was found in the path
> > 
> > # perf annotate -k vmlinux
> > objdump: '[kernel.kallsyms]': No such file
> 
> This looks like a bug in how it reports this problem, probably vmlinux
> is not in the current directory, or it is but has a build-id that
> doesn't matches the one in the perf.data file.
> 
> But the message is wrong, should be like it is in the tui, for this same
> situation:
> 
> 	ui_helpline__puts("No vmlinux file found, can't "
> 			  "annotate with just a kallsyms file");
> 
> > Am I missing something here?
> 
> If you don't specify it with --vmlinux/-k it will try to find it in one
> of these places:
> 
>         vmlinux_path[vmlinux_path__nr_entries] = strdup("vmlinux");
>         vmlinux_path[vmlinux_path__nr_entries] = strdup("/boot/vmlinux");
>         snprintf(bf, sizeof(bf), "/boot/vmlinux-%s", uts.release);
>         vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
>         snprintf(bf, sizeof(bf), "/lib/modules/%s/build/vmlinux", uts.release);
>         vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
>         snprintf(bf, sizeof(bf), "/usr/lib/debug/lib/modules/%s/vmlinux",
>                  uts.release);
>         vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
> 
> It also now looks at /sys/kernel/notes, gets the build-id, and looks up
> in the build-id cache.

For instance, here using -vvv I get:

[root@emilia ~]# perf annotate -vvv memcpy > /dev/null
build id event received for [kernel.kallsyms]: 310920b54cbbf242dbfa09052f652806d2c73821
build id event received for /lib64/libtinfo.so.5.7: cd9da7b5b1fcb25ee6b3762bb9fe62f782590fd5
Looking at the vmlinux_path (6 entries long)
No build_id in /root/.debug/.build-id/31/0920b54cbbf242dbfa09052f652806d2c73821, ignoring it
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.35-rc1, ignoring it
Using /lib/modules/2.6.35-rc1/build/vmlinux for symbols
hist_entry__inc_addr_samples: ip=0xffffffff811ea69e
0xffffffff811ea690 memcpy: period++ [ip: 0xffffffff811ea69e, 0xe] => 1
hist_entry__annotate: filename=/lib/modules/2.6.35-rc1/build/vmlinux, sym=memcpy, start=0xffffffff811ea690, end=0xffffffff811ea730
annotating [0x187e770] /lib/modules/2.6.35-rc1/build/vmlinux : [0x19b1f70]                         memcpy
Executing: objdump --start-address=0xffffffff811ea690 --stop-address=0xffffffff811ea730 -dS /lib/modules/2.6.35-rc1/build/vmlinux|grep -v /lib/modules/2.6.35-rc1/build/vmlinux|expand
[root@emilia ~]# 

I'm removing all vmlinux from the path so that I can reproduce what you see.

- Arnaldo

Re: how to use perf annotate on the kernel

[Arnaldo Carvalho de Melo]

Em Thu, Jun 03, 2010 at 03:06:13PM -0300, Arnaldo Carvalho de Melo escreveu:
> For instance, here using -vvv I get:
> 
> [root@emilia ~]# perf annotate -vvv memcpy > /dev/null
> build id event received for [kernel.kallsyms]: 310920b54cbbf242dbfa09052f652806d2c73821
> build id event received for /lib64/libtinfo.so.5.7: cd9da7b5b1fcb25ee6b3762bb9fe62f782590fd5
> Looking at the vmlinux_path (6 entries long)
> No build_id in /root/.debug/.build-id/31/0920b54cbbf242dbfa09052f652806d2c73821, ignoring it
> No build_id in vmlinux, ignoring it
> No build_id in /boot/vmlinux, ignoring it
> No build_id in /boot/vmlinux-2.6.35-rc1, ignoring it
> Using /lib/modules/2.6.35-rc1/build/vmlinux for symbols
> hist_entry__inc_addr_samples: ip=0xffffffff811ea69e
> 0xffffffff811ea690 memcpy: period++ [ip: 0xffffffff811ea69e, 0xe] => 1
> hist_entry__annotate: filename=/lib/modules/2.6.35-rc1/build/vmlinux, sym=memcpy, start=0xffffffff811ea690, end=0xffffffff811ea730
> annotating [0x187e770] /lib/modules/2.6.35-rc1/build/vmlinux : [0x19b1f70]                         memcpy
> Executing: objdump --start-address=0xffffffff811ea690 --stop-address=0xffffffff811ea730 -dS /lib/modules/2.6.35-rc1/build/vmlinux|grep -v /lib/modules/2.6.35-rc1/build/vmlinux|expand
> [root@emilia ~]# 
> 
> I'm removing all vmlinux from the path so that I can reproduce what you see.

[root@emilia ~]# mv /lib/modules/2.6.35-rc1/build/vmlinux /lib/modules/2.6.35-rc1/build/vmlinux.OFF
[root@emilia ~]# perf annotate -vvv memcpy > /dev/null
build id event received for [kernel.kallsyms]: 310920b54cbbf242dbfa09052f652806d2c73821
build id event received for /lib64/libtinfo.so.5.7: cd9da7b5b1fcb25ee6b3762bb9fe62f782590fd5
Looking at the vmlinux_path (6 entries long)
No build_id in /root/.debug/.build-id/31/0920b54cbbf242dbfa09052f652806d2c73821, ignoring it
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.35-rc1, ignoring it
No build_id in /lib/modules/2.6.35-rc1/build/vmlinux, ignoring it
No build_id in /usr/lib/debug/lib/modules/2.6.35-rc1/vmlinux, ignoring it
Using /proc/kallsyms for symbols
[root@emilia ~]# 
[root@emilia ~]# vi ~/.perfconfig
[root@emilia ~]# cat ~/.perfconfig
[tui]

	report = on
	annotate = off
[root@emilia ~]# perf annotate memcpy
[root@emilia ~]# perf annotate memcpy
[root@emilia ~]# perf annotate -k vmlinux memcpy
[kernel.kallsyms] with build id 310920b54cbbf242dbfa09052f652806d2c73821 not found, continuing without symbols

	Yeah, these error messages suckz, will work on having proper messages
for these cases :-\

- Arnaldo

Re: how to use perf annotate on the kernel

[Stephane Eranian]

perf annotate __lock_acquire -k /home/eranian/perfmon/official/tip/build/vmlinux
build id event received for [kernel.kallsyms]:
5c110a704293a259fbe11158dc1b25107ad7a23e
build id event received for
/lib/modules/2.6.35-rc1-tip/kernel/arch/x86/kernel/cpu/cpufreq/mperf.ko:
cb5
build id event received for
/lib/modules/2.6.35-rc1-tip/kernel/drivers/cpufreq/cpufreq_ondemand.ko:
cdb5
Using /home/eranian/perfmon/official/tip/build/vmlinux for symbols
hist_entry__inc_addr_samples: ip=0xffffffff81073b36
0xffffffff81073b30 __lock_acquire: period++ [ip: 0xffffffff81073b36, 0x6] => 1
hist_entry__inc_addr_samples: ip=0xffffffff81073c94

But in the end I get:
objdump: '[kernel.kallsyms]': No such file


------------------------------------------------
 Percent |      Source code & Disassembly of [kernel.kallsyms]
------------------------------------------------


On Thu, Jun 3, 2010 at 8:11 PM, Arnaldo Carvalho de Melo
<acme@infradead.org> wrote:
> Em Thu, Jun 03, 2010 at 03:06:13PM -0300, Arnaldo Carvalho de Melo escreveu:
>> For instance, here using -vvv I get:
>>
>> [root@emilia ~]# perf annotate -vvv memcpy > /dev/null
>> build id event received for [kernel.kallsyms]: 310920b54cbbf242dbfa09052f652806d2c73821
>> build id event received for /lib64/libtinfo.so.5.7: cd9da7b5b1fcb25ee6b3762bb9fe62f782590fd5
>> Looking at the vmlinux_path (6 entries long)
>> No build_id in /root/.debug/.build-id/31/0920b54cbbf242dbfa09052f652806d2c73821, ignoring it
>> No build_id in vmlinux, ignoring it
>> No build_id in /boot/vmlinux, ignoring it
>> No build_id in /boot/vmlinux-2.6.35-rc1, ignoring it
>> Using /lib/modules/2.6.35-rc1/build/vmlinux for symbols
>> hist_entry__inc_addr_samples: ip=0xffffffff811ea69e
>> 0xffffffff811ea690 memcpy: period++ [ip: 0xffffffff811ea69e, 0xe] => 1
>> hist_entry__annotate: filename=/lib/modules/2.6.35-rc1/build/vmlinux, sym=memcpy, start=0xffffffff811ea690, end=0xffffffff811ea730
>> annotating [0x187e770] /lib/modules/2.6.35-rc1/build/vmlinux : [0x19b1f70]                         memcpy
>> Executing: objdump --start-address=0xffffffff811ea690 --stop-address=0xffffffff811ea730 -dS /lib/modules/2.6.35-rc1/build/vmlinux|grep -v /lib/modules/2.6.35-rc1/build/vmlinux|expand
>> [root@emilia ~]#
>>
>> I'm removing all vmlinux from the path so that I can reproduce what you see.
>
> [root@emilia ~]# mv /lib/modules/2.6.35-rc1/build/vmlinux /lib/modules/2.6.35-rc1/build/vmlinux.OFF
> [root@emilia ~]# perf annotate -vvv memcpy > /dev/null
> build id event received for [kernel.kallsyms]: 310920b54cbbf242dbfa09052f652806d2c73821
> build id event received for /lib64/libtinfo.so.5.7: cd9da7b5b1fcb25ee6b3762bb9fe62f782590fd5
> Looking at the vmlinux_path (6 entries long)
> No build_id in /root/.debug/.build-id/31/0920b54cbbf242dbfa09052f652806d2c73821, ignoring it
> No build_id in vmlinux, ignoring it
> No build_id in /boot/vmlinux, ignoring it
> No build_id in /boot/vmlinux-2.6.35-rc1, ignoring it
> No build_id in /lib/modules/2.6.35-rc1/build/vmlinux, ignoring it
> No build_id in /usr/lib/debug/lib/modules/2.6.35-rc1/vmlinux, ignoring it
> Using /proc/kallsyms for symbols
> [root@emilia ~]#
> [root@emilia ~]# vi ~/.perfconfig
> [root@emilia ~]# cat ~/.perfconfig
> [tui]
>
>        report = on
>        annotate = off
> [root@emilia ~]# perf annotate memcpy
> [root@emilia ~]# perf annotate memcpy
> [root@emilia ~]# perf annotate -k vmlinux memcpy
> [kernel.kallsyms] with build id 310920b54cbbf242dbfa09052f652806d2c73821 not found, continuing without symbols
>
>        Yeah, these error messages suckz, will work on having proper messages
> for these cases :-\
>
> - Arnaldo
>

Re: how to use perf annotate on the kernel

[Arnaldo Carvalho de Melo]

Em Thu, Jun 03, 2010 at 08:18:04PM +0200, Stephane Eranian escreveu:
> perf annotate __lock_acquire -k /home/eranian/perfmon/official/tip/build/vmlinux
> build id event received for [kernel.kallsyms]:
> 5c110a704293a259fbe11158dc1b25107ad7a23e
> build id event received for
> /lib/modules/2.6.35-rc1-tip/kernel/arch/x86/kernel/cpu/cpufreq/mperf.ko:
> cb5
> build id event received for
> /lib/modules/2.6.35-rc1-tip/kernel/drivers/cpufreq/cpufreq_ondemand.ko:
> cdb5
> Using /home/eranian/perfmon/official/tip/build/vmlinux for symbols
> hist_entry__inc_addr_samples: ip=0xffffffff81073b36
> 0xffffffff81073b30 __lock_acquire: period++ [ip: 0xffffffff81073b36, 0x6] => 1
> hist_entry__inc_addr_samples: ip=0xffffffff81073c94
> 
> But in the end I get:
> objdump: '[kernel.kallsyms]': No such file

Reproduced, working on it now. I've been testing only with vmlinux in
the path it searches for, not by directly specifying it, will get that
fixed, thanks,

- Arnaldo

Re: how to use perf annotate on the kernel

[Arnaldo Carvalho de Melo]

Em Thu, Jun 03, 2010 at 09:20:39PM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Thu, Jun 03, 2010 at 08:18:04PM +0200, Stephane Eranian escreveu:
> > perf annotate __lock_acquire -k /home/eranian/perfmon/official/tip/build/vmlinux
> > build id event received for [kernel.kallsyms]:
> > 5c110a704293a259fbe11158dc1b25107ad7a23e
> > build id event received for
> > /lib/modules/2.6.35-rc1-tip/kernel/arch/x86/kernel/cpu/cpufreq/mperf.ko:
> > cb5
> > build id event received for
> > /lib/modules/2.6.35-rc1-tip/kernel/drivers/cpufreq/cpufreq_ondemand.ko:
> > cdb5
> > Using /home/eranian/perfmon/official/tip/build/vmlinux for symbols
> > hist_entry__inc_addr_samples: ip=0xffffffff81073b36
> > 0xffffffff81073b30 __lock_acquire: period++ [ip: 0xffffffff81073b36, 0x6] => 1
> > hist_entry__inc_addr_samples: ip=0xffffffff81073c94
> > 
> > But in the end I get:
> > objdump: '[kernel.kallsyms]': No such file
> 
> Reproduced, working on it now. I've been testing only with vmlinux in
> the path it searches for, not by directly specifying it, will get that
> fixed, thanks,

Should be fixed by this patch:

[root@emilia ~]# perf annotate -k /lib/modules/2.6.35-rc1/build/vmlinux.OFF n_tty_write | head -50


------------------------------------------------
 Percent |	Source code & Disassembly of vmlinux.OFF
------------------------------------------------
         :
         :
         :
         :	Disassembly of section .text:
         :
         :	ffffffff81288820 <n_tty_write>:
         :	 *                lock themselves)
         :	 */
         :
         :	static ssize_t n_tty_write(struct tty_struct *tty, struct file *file,
         :	                           const unsigned char *buf, size_t nr)
         :	{
    0.00 :	ffffffff81288820:       41 57                   push   %r15
         :
         :	DECLARE_PER_CPU(struct task_struct *, current_task);
         :
         :	static __always_inline struct task_struct *get_current(void)
         :	{
         :	        return percpu_read_stable(current_task);
    0.00 :	ffffffff81288822:       65 48 8b 04 25 40 cc    mov    %gs:0xcc40,%rax
    0.00 :	ffffffff81288829:       00 00 
    0.00 :	ffffffff8128882b:       41 56                   push   %r14
    0.00 :	ffffffff8128882d:       41 55                   push   %r13
    0.00 :	ffffffff8128882f:       49 89 cd                mov    %rcx,%r13
    0.00 :	ffffffff81288832:       41 54                   push   %r12
    0.00 :	ffffffff81288834:       49 89 d4                mov    %rdx,%r12
    0.00 :	ffffffff81288837:       55                      push   %rbp
    0.00 :	ffffffff81288838:       48 89 f5                mov    %rsi,%rbp
    0.00 :	ffffffff8128883b:       53                      push   %rbx
    0.00 :	ffffffff8128883c:       48 89 fb                mov    %rdi,%rbx
    0.00 :	ffffffff8128883f:       48 83 ec 58             sub    $0x58,%rsp
         :	        const unsigned char *b = buf;
         :	        DECLARE_WAITQUEUE(wait, current);
    0.00 :	ffffffff81288843:       48 c7 44 24 20 00 00    movq   $0x0,0x20(%rsp)
    0.00 :	ffffffff8128884a:       00 00 
    7.14 :	ffffffff8128884c:       48 c7 44 24 38 00 00    movq   $0x0,0x38(%rsp)
    0.00 :	ffffffff81288853:       00 00 
    0.00 :	ffffffff81288855:       48 c7 44 24 40 00 00    movq   $0x0,0x40(%rsp)
    0.00 :	ffffffff8128885c:       00 00 
    7.14 :	ffffffff8128885e:       48 89 44 24 28          mov    %rax,0x28(%rsp)
    0.00 :	ffffffff81288863:       48 c7 44 24 30 30 49    movq   $0xffffffff81044930,0x30(%rsp)
    0.00 :	ffffffff8128886a:       04 81 
         :	        int c;
         :	        ssize_t retval = 0;
         :
[root@emilia ~]#

>From 842b66de6807e72bddfd24a63742bacd0f853872 Mon Sep 17 00:00:00 2001
From: Arnaldo Carvalho de Melo <acme@redhat.com>
Date: Thu, 3 Jun 2010 18:35:55 -0300
Subject: [PATCH 1/1] perf symbols: Set the DSO long name when using symbol_conf.vmlinux_name
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

We need to set the long name to the name specified via, for instance,
'perf annotate --vmlinux /path/to/vmlinux', if not it will remain as
'[kernel.kallsyms]' and that will make annotate fail when passing this
as the vmlinux name in the call to objdump.

The way this is setup grew unwieldly and dso__load_vmlinux is the
function that should allocate space for the long name, with callers not
assuming that filenames should be allocated somehow by then (strdup,
dso__build_id_filename, etc).

For now this is the minimalistic patch, a proper fix for .36 will be
made.

Reported-by: Stephane Eranian <eranian@google.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Cc: Tom Zanussi <tzanussi@gmail.com>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/symbol.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index 7fd6b15..b63e571 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -1745,7 +1745,12 @@ static int dso__load_kernel_sym(struct dso *self, struct map *map,
 	if (symbol_conf.vmlinux_name != NULL) {
 		err = dso__load_vmlinux(self, map,
 					symbol_conf.vmlinux_name, filter);
-		goto out_try_fixup;
+		if (err > 0) {
+			dso__set_long_name(self,
+					   strdup(symbol_conf.vmlinux_name));
+			goto out_fixup;
+		}
+		return err;
 	}
 
 	if (vmlinux_path != NULL) {
@@ -1806,7 +1811,6 @@ do_kallsyms:
 		pr_debug("Using %s for symbols\n", kallsyms_filename);
 	free(kallsyms_allocated_filename);
 
-out_try_fixup:
 	if (err > 0) {
 out_fixup:
 		if (kallsyms_filename != NULL)
-- 
1.6.5.2

Re: how to use perf annotate on the kernel

[Stephane Eranian]

Hi,

This patch seems to work now.
Thanks.

On Fri, Jun 4, 2010 at 2:39 AM, Arnaldo Carvalho de Melo
<acme@infradead.org> wrote:
> Em Thu, Jun 03, 2010 at 09:20:39PM -0300, Arnaldo Carvalho de Melo escreveu:
>> Em Thu, Jun 03, 2010 at 08:18:04PM +0200, Stephane Eranian escreveu:
>> > perf annotate __lock_acquire -k /home/eranian/perfmon/official/tip/build/vmlinux
>> > build id event received for [kernel.kallsyms]:
>> > 5c110a704293a259fbe11158dc1b25107ad7a23e
>> > build id event received for
>> > /lib/modules/2.6.35-rc1-tip/kernel/arch/x86/kernel/cpu/cpufreq/mperf.ko:
>> > cb5
>> > build id event received for
>> > /lib/modules/2.6.35-rc1-tip/kernel/drivers/cpufreq/cpufreq_ondemand.ko:
>> > cdb5
>> > Using /home/eranian/perfmon/official/tip/build/vmlinux for symbols
>> > hist_entry__inc_addr_samples: ip=0xffffffff81073b36
>> > 0xffffffff81073b30 __lock_acquire: period++ [ip: 0xffffffff81073b36, 0x6] => 1
>> > hist_entry__inc_addr_samples: ip=0xffffffff81073c94
>> >
>> > But in the end I get:
>> > objdump: '[kernel.kallsyms]': No such file
>>
>> Reproduced, working on it now. I've been testing only with vmlinux in
>> the path it searches for, not by directly specifying it, will get that
>> fixed, thanks,
>
> Should be fixed by this patch:
>
> [root@emilia ~]# perf annotate -k /lib/modules/2.6.35-rc1/build/vmlinux.OFF n_tty_write | head -50
>
>
> ------------------------------------------------
>  Percent |      Source code & Disassembly of vmlinux.OFF
> ------------------------------------------------
>         :
>         :
>         :
>         :      Disassembly of section .text:
>         :
>         :      ffffffff81288820 <n_tty_write>:
>         :       *                lock themselves)
>         :       */
>         :
>         :      static ssize_t n_tty_write(struct tty_struct *tty, struct file *file,
>         :                                 const unsigned char *buf, size_t nr)
>         :      {
>    0.00 :      ffffffff81288820:       41 57                   push   %r15
>         :
>         :      DECLARE_PER_CPU(struct task_struct *, current_task);
>         :
>         :      static __always_inline struct task_struct *get_current(void)
>         :      {
>         :              return percpu_read_stable(current_task);
>    0.00 :      ffffffff81288822:       65 48 8b 04 25 40 cc    mov    %gs:0xcc40,%rax
>    0.00 :      ffffffff81288829:       00 00
>    0.00 :      ffffffff8128882b:       41 56                   push   %r14
>    0.00 :      ffffffff8128882d:       41 55                   push   %r13
>    0.00 :      ffffffff8128882f:       49 89 cd                mov    %rcx,%r13
>    0.00 :      ffffffff81288832:       41 54                   push   %r12
>    0.00 :      ffffffff81288834:       49 89 d4                mov    %rdx,%r12
>    0.00 :      ffffffff81288837:       55                      push   %rbp
>    0.00 :      ffffffff81288838:       48 89 f5                mov    %rsi,%rbp
>    0.00 :      ffffffff8128883b:       53                      push   %rbx
>    0.00 :      ffffffff8128883c:       48 89 fb                mov    %rdi,%rbx
>    0.00 :      ffffffff8128883f:       48 83 ec 58             sub    $0x58,%rsp
>         :              const unsigned char *b = buf;
>         :              DECLARE_WAITQUEUE(wait, current);
>    0.00 :      ffffffff81288843:       48 c7 44 24 20 00 00    movq   $0x0,0x20(%rsp)
>    0.00 :      ffffffff8128884a:       00 00
>    7.14 :      ffffffff8128884c:       48 c7 44 24 38 00 00    movq   $0x0,0x38(%rsp)
>    0.00 :      ffffffff81288853:       00 00
>    0.00 :      ffffffff81288855:       48 c7 44 24 40 00 00    movq   $0x0,0x40(%rsp)
>    0.00 :      ffffffff8128885c:       00 00
>    7.14 :      ffffffff8128885e:       48 89 44 24 28          mov    %rax,0x28(%rsp)
>    0.00 :      ffffffff81288863:       48 c7 44 24 30 30 49    movq   $0xffffffff81044930,0x30(%rsp)
>    0.00 :      ffffffff8128886a:       04 81
>         :              int c;
>         :              ssize_t retval = 0;
>         :
> [root@emilia ~]#
>
> From 842b66de6807e72bddfd24a63742bacd0f853872 Mon Sep 17 00:00:00 2001
> From: Arnaldo Carvalho de Melo <acme@redhat.com>
> Date: Thu, 3 Jun 2010 18:35:55 -0300
> Subject: [PATCH 1/1] perf symbols: Set the DSO long name when using symbol_conf.vmlinux_name
> MIME-Version: 1.0
> Content-Type: text/plain; charset=UTF-8
> Content-Transfer-Encoding: 8bit
>
> We need to set the long name to the name specified via, for instance,
> 'perf annotate --vmlinux /path/to/vmlinux', if not it will remain as
> '[kernel.kallsyms]' and that will make annotate fail when passing this
> as the vmlinux name in the call to objdump.
>
> The way this is setup grew unwieldly and dso__load_vmlinux is the
> function that should allocate space for the long name, with callers not
> assuming that filenames should be allocated somehow by then (strdup,
> dso__build_id_filename, etc).
>
> For now this is the minimalistic patch, a proper fix for .36 will be
> made.
>
> Reported-by: Stephane Eranian <eranian@google.com>
> Cc: David S. Miller <davem@davemloft.net>
> Cc: Frédéric Weisbecker <fweisbec@gmail.com>
> Cc: Ingo Molnar <mingo@elte.hu>
> Cc: Mike Galbraith <efault@gmx.de>
> Cc: Paul Mackerras <paulus@samba.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Stephane Eranian <eranian@google.com>
> Cc: Tom Zanussi <tzanussi@gmail.com>
> LKML-Reference: <new-submission>
> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
> ---
>  tools/perf/util/symbol.c |    8 ++++++--
>  1 files changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
> index 7fd6b15..b63e571 100644
> --- a/tools/perf/util/symbol.c
> +++ b/tools/perf/util/symbol.c
> @@ -1745,7 +1745,12 @@ static int dso__load_kernel_sym(struct dso *self, struct map *map,
>        if (symbol_conf.vmlinux_name != NULL) {
>                err = dso__load_vmlinux(self, map,
>                                        symbol_conf.vmlinux_name, filter);
> -               goto out_try_fixup;
> +               if (err > 0) {
> +                       dso__set_long_name(self,
> +                                          strdup(symbol_conf.vmlinux_name));
> +                       goto out_fixup;
> +               }
> +               return err;
>        }
>
>        if (vmlinux_path != NULL) {
> @@ -1806,7 +1811,6 @@ do_kallsyms:
>                pr_debug("Using %s for symbols\n", kallsyms_filename);
>        free(kallsyms_allocated_filename);
>
> -out_try_fixup:
>        if (err > 0) {
>  out_fixup:
>                if (kallsyms_filename != NULL)
> --
> 1.6.5.2
>
>

Re: how to use perf annotate on the kernel

[Stephane Eranian]

Hi,

My kernel is indeed not in the current subdir, I shortened
the cmdline in my post.

This is the live kernel but I don't have it in /lib/.

With verbose, I get output such as:

objdump --start-address=0xffffffff81246ad0
--stop-address=0xffffffff81246b44 -dS -C [kernel.kallsyms]|grep -v
[kernel.kallsyms]|expand


On Thu, Jun 3, 2010 at 8:06 PM, Arnaldo Carvalho de Melo
<acme@infradead.org> wrote:
> Em Thu, Jun 03, 2010 at 02:54:45PM -0300, Arnaldo Carvalho de Melo escreveu:
>> Em Thu, Jun 03, 2010 at 07:23:02PM +0200, Stephane Eranian escreveu:
>> > # perf annotate (or annotate -d '[kernel.kallsyms]'?)
>> > Can't annotate __lock_acquire: No vmlinux file was found in the path
>> >
>> > # perf annotate -k vmlinux
>> > objdump: '[kernel.kallsyms]': No such file
>>
>> This looks like a bug in how it reports this problem, probably vmlinux
>> is not in the current directory, or it is but has a build-id that
>> doesn't matches the one in the perf.data file.
>>
>> But the message is wrong, should be like it is in the tui, for this same
>> situation:
>>
>>       ui_helpline__puts("No vmlinux file found, can't "
>>                         "annotate with just a kallsyms file");
>>
>> > Am I missing something here?
>>
>> If you don't specify it with --vmlinux/-k it will try to find it in one
>> of these places:
>>
>>         vmlinux_path[vmlinux_path__nr_entries] = strdup("vmlinux");
>>         vmlinux_path[vmlinux_path__nr_entries] = strdup("/boot/vmlinux");
>>         snprintf(bf, sizeof(bf), "/boot/vmlinux-%s", uts.release);
>>         vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
>>         snprintf(bf, sizeof(bf), "/lib/modules/%s/build/vmlinux", uts.release);
>>         vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
>>         snprintf(bf, sizeof(bf), "/usr/lib/debug/lib/modules/%s/vmlinux",
>>                  uts.release);
>>         vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
>>
>> It also now looks at /sys/kernel/notes, gets the build-id, and looks up
>> in the build-id cache.
>
> For instance, here using -vvv I get:
>
> [root@emilia ~]# perf annotate -vvv memcpy > /dev/null
> build id event received for [kernel.kallsyms]: 310920b54cbbf242dbfa09052f652806d2c73821
> build id event received for /lib64/libtinfo.so.5.7: cd9da7b5b1fcb25ee6b3762bb9fe62f782590fd5
> Looking at the vmlinux_path (6 entries long)
> No build_id in /root/.debug/.build-id/31/0920b54cbbf242dbfa09052f652806d2c73821, ignoring it
> No build_id in vmlinux, ignoring it
> No build_id in /boot/vmlinux, ignoring it
> No build_id in /boot/vmlinux-2.6.35-rc1, ignoring it
> Using /lib/modules/2.6.35-rc1/build/vmlinux for symbols
> hist_entry__inc_addr_samples: ip=0xffffffff811ea69e
> 0xffffffff811ea690 memcpy: period++ [ip: 0xffffffff811ea69e, 0xe] => 1
> hist_entry__annotate: filename=/lib/modules/2.6.35-rc1/build/vmlinux, sym=memcpy, start=0xffffffff811ea690, end=0xffffffff811ea730
> annotating [0x187e770] /lib/modules/2.6.35-rc1/build/vmlinux : [0x19b1f70]                         memcpy
> Executing: objdump --start-address=0xffffffff811ea690 --stop-address=0xffffffff811ea730 -dS /lib/modules/2.6.35-rc1/build/vmlinux|grep -v /lib/modules/2.6.35-rc1/build/vmlinux|expand
> [root@emilia ~]#
>
> I'm removing all vmlinux from the path so that I can reproduce what you see.
>
> - Arnaldo
>