perf-probe: issue with latest fedora kernel

perf-probe: issue with latest fedora kernel

[Francis Moreau]

Hello,

I'm trying to use perf-probe(1) with the latest F14 kernel
(2.6.35.9-64.fc14.x86_64).

I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which
contains the running vmlinux image AFAICT.

I added it to the buildid cache:

$ perf buildid-cache -v -a /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
Adding 4d89e23415d8ab491cfc8ef8aa67764b91cc6787 /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux: OK

But when running perf-probe(1) like the following:

  $ perf probe schedule cpu
  Failed to find path of kernel moduleFailed to open debuginfo file.
  Error: Failed to add events. (-2)

it fails.

One strange thing is that when perf read the running kernel build id, it
get: 882b1b53eb1d65320e8fa710273aa8493896c4.

Could anybody tell me what I'm doing wrong ?

Thanks
-- 
Francis

Re: perf-probe: issue with latest fedora kernel

[Arnaldo Carvalho de Melo]

Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
> I'm trying to use perf-probe(1) with the latest F14 kernel
> (2.6.35.9-64.fc14.x86_64).
> 
> I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which
> contains the running vmlinux image AFAICT.
> 
> I added it to the buildid cache:
> 
> $ perf buildid-cache -v -a /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
> Adding 4d89e23415d8ab491cfc8ef8aa67764b91cc6787 /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux: OK
> 
> But when running perf-probe(1) like the following:
> 
>   $ perf probe schedule cpu
>   Failed to find path of kernel moduleFailed to open debuginfo file.
>   Error: Failed to add events. (-2)
> 
> it fails.
> 
> One strange thing is that when perf read the running kernel build id, it
> get: 882b1b53eb1d65320e8fa710273aa8493896c4.
> 
> Could anybody tell me what I'm doing wrong ?

Can you try using:

$ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>

So that it tries it directly instead of looking into the buildid cache?

I'll check if perf probe actually looks in the buildid cache, till then,
please try specifying it directly,

Regards,

- Arnaldo

Re: perf-probe: issue with latest fedora kernel

[Francis Moreau]

Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:

> Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
>> I'm trying to use perf-probe(1) with the latest F14 kernel
>> (2.6.35.9-64.fc14.x86_64).
>> 
>
>> I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which
>> contains the running vmlinux image AFAICT.
>> 
>> I added it to the buildid cache:
>> 
>> $ perf buildid-cache -v -a /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
>> Adding 4d89e23415d8ab491cfc8ef8aa67764b91cc6787 /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux: OK
>> 
>> But when running perf-probe(1) like the following:
>> 
>>   $ perf probe schedule cpu
>>   Failed to find path of kernel moduleFailed to open debuginfo file.
>>   Error: Failed to add events. (-2)
>> 
>> it fails.
>> 
>> One strange thing is that when perf read the running kernel build id, it
>> get: 882b1b53eb1d65320e8fa710273aa8493896c4.
>> 
>> Could anybody tell me what I'm doing wrong ?
>
> Can you try using:
>
> $ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
>
> So that it tries it directly instead of looking into the buildid cache?
>

Well that still fail:

$ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux schedule cpu
Failed to find path of kernel moduleFailed to open debuginfo file.
  Error: Failed to add events. (-2)

but that's not really suprising if perf checks that the running kernel
build-id is the same as the one in:
/usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux

It really looks like something is broken in the F14 kernel-debuginfo
package.

thanks
-- 
Francis

Re: perf-probe: issue with latest fedora kernel

[Masami Hiramatsu]

(2010/12/10 1:34), Francis Moreau wrote:
> Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
> 
>> Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
>>> I'm trying to use perf-probe(1) with the latest F14 kernel
>>> (2.6.35.9-64.fc14.x86_64).
>>>
>>
>>> I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which
>>> contains the running vmlinux image AFAICT.
>>>
>>> I added it to the buildid cache:
>>>
>>> $ perf buildid-cache -v -a /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
>>> Adding 4d89e23415d8ab491cfc8ef8aa67764b91cc6787 /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux: OK
>>>
>>> But when running perf-probe(1) like the following:
>>>
>>>   $ perf probe schedule cpu
>>>   Failed to find path of kernel moduleFailed to open debuginfo file.
>>>   Error: Failed to add events. (-2)
>>>
>>> it fails.
>>>
>>> One strange thing is that when perf read the running kernel build id, it
>>> get: 882b1b53eb1d65320e8fa710273aa8493896c4.
>>>
>>> Could anybody tell me what I'm doing wrong ?
>>
>> Can you try using:
>>
>> $ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
>>
>> So that it tries it directly instead of looking into the buildid cache?
>>
> 
> Well that still fail:
> 
> $ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux schedule cpu
> Failed to find path of kernel moduleFailed to open debuginfo file.
>   Error: Failed to add events. (-2)
> 
> but that's not really suprising if perf checks that the running kernel
> build-id is the same as the one in:
> /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
> 
> It really looks like something is broken in the F14 kernel-debuginfo
> package.

Hmm, curious. I'll check it after setting up Fedora 14.
Thank you for reporting :-)


-- 
Masami HIRAMATSU
2nd Dept. Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: perf-probe: issue with latest fedora kernel

[Franck Bui-Huu]

Hi,

Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> writes:

> (2010/12/10 1:34), Francis Moreau wrote:

[...]

>> 
>> Well that still fail:
>> 
>> $ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux schedule cpu
>> Failed to find path of kernel moduleFailed to open debuginfo file.
>>   Error: Failed to add events. (-2)
>> 
>> but that's not really suprising if perf checks that the running kernel
>> build-id is the same as the one in:
>> /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
>> 
>> It really looks like something is broken in the F14 kernel-debuginfo
>> package.
>
> Hmm, curious. I'll check it after setting up Fedora 14.
> Thank you for reporting :-)

BTW, it seems to me that perf-probe(1) is ignoring the kernel path given
by '-k' switch.

Looking at kernel_get_module_path(), it only considers the standard
paths given by vmlinux_path[].

I would have seen this, no ?

diff --git a/tools/perf/builtin-probe.c b/tools/perf/builtin-probe.c
index 2e000c0..add163c 100644
--- a/tools/perf/builtin-probe.c
+++ b/tools/perf/builtin-probe.c
@@ -249,6 +249,11 @@ int cmd_probe(int argc, const char **argv, const char *prefix __used)
 	     !params.show_lines))
 		usage_with_options(probe_usage, options);
 
+	/*
+	 * Only consider the user's kernel image path if given.
+	 */
+	symbol_conf.try_vmlinux_path = (symbol_conf.vmlinux_name == NULL);
+
 	if (params.list_events) {
 		if (params.mod_events) {
 			pr_err("  Error: Don't use --list with --add/--del.\n");
diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 694f04b..558545e 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -114,6 +114,8 @@ static struct symbol *__find_kernel_function_by_name(const char *name,
 const char *kernel_get_module_path(const char *module)
 {
 	struct dso *dso;
+	struct map *map;
+	const char *vmlinux_name;
 
 	if (module) {
 		list_for_each_entry(dso, &machine.kernel_dsos, node) {
@@ -123,14 +125,23 @@ const char *kernel_get_module_path(const char *module)
 		}
 		pr_debug("Failed to find module %s.\n", module);
 		return NULL;
-	} else {
-		dso = machine.vmlinux_maps[MAP__FUNCTION]->dso;
-		if (dso__load_vmlinux_path(dso,
-			 machine.vmlinux_maps[MAP__FUNCTION], NULL) < 0) {
-			pr_debug("Failed to load kernel map.\n");
-			return NULL;
-		}
 	}
+
+	map = machine.vmlinux_maps[MAP__FUNCTION];
+	dso = map->dso;
+
+	vmlinux_name = symbol_conf.vmlinux_name;
+	if (vmlinux_name) {
+		if (dso__load_vmlinux(dso, map, vmlinux_name, NULL) == 0)
+			goto found;
+		return NULL;
+	}
+
+	if (dso__load_vmlinux_path(dso, map, NULL) < 0) {
+		pr_debug("Failed to load kernel map.\n");
+		return NULL;
+	}
+
 found:
 	return dso->long_name;
 }
diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index a3a0c43..4e42e0d 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -1780,8 +1780,8 @@ out_failure:
 	return -1;
 }
 
-static int dso__load_vmlinux(struct dso *self, struct map *map,
-			     const char *vmlinux, symbol_filter_t filter)
+int dso__load_vmlinux(struct dso *self, struct map *map,
+		      const char *vmlinux, symbol_filter_t filter)
 {
 	int err = -1, fd;
 
diff --git a/tools/perf/util/symbol.h b/tools/perf/util/symbol.h
index 038f220..6c6eafd 100644
--- a/tools/perf/util/symbol.h
+++ b/tools/perf/util/symbol.h
@@ -166,6 +166,8 @@ void dso__sort_by_name(struct dso *self, enum map_type type);
 struct dso *__dsos__findnew(struct list_head *head, const char *name);
 
 int dso__load(struct dso *self, struct map *map, symbol_filter_t filter);
+int dso__load_vmlinux(struct dso *self, struct map *map,
+		      const char *vmlinux, symbol_filter_t filter);
 int dso__load_vmlinux_path(struct dso *self, struct map *map,
 			   symbol_filter_t filter);
 int dso__load_kallsyms(struct dso *self, const char *filename, struct map *map,


-- 
		Franck

Re: perf-probe: issue with latest fedora kernel

[Masami Hiramatsu]

(2010/12/10 16:53), Franck Bui-Huu wrote:
> 
> BTW, it seems to me that perf-probe(1) is ignoring the kernel path given
> by '-k' switch.
> 
> Looking at kernel_get_module_path(), it only considers the standard
> paths given by vmlinux_path[].
> 
> I would have seen this, no ?

Right! Thank you for finding the bug!


> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> index 694f04b..558545e 100644
> --- a/tools/perf/util/probe-event.c
> +++ b/tools/perf/util/probe-event.c
> @@ -114,6 +114,8 @@ static struct symbol *__find_kernel_function_by_name(const char *name,
>  const char *kernel_get_module_path(const char *module)
>  {
>  	struct dso *dso;
> +	struct map *map;
> +	const char *vmlinux_name;
>  
>  	if (module) {
>  		list_for_each_entry(dso, &machine.kernel_dsos, node) {
> @@ -123,14 +125,23 @@ const char *kernel_get_module_path(const char *module)
>  		}
>  		pr_debug("Failed to find module %s.\n", module);
>  		return NULL;
> -	} else {
> -		dso = machine.vmlinux_maps[MAP__FUNCTION]->dso;
> -		if (dso__load_vmlinux_path(dso,
> -			 machine.vmlinux_maps[MAP__FUNCTION], NULL) < 0) {
> -			pr_debug("Failed to load kernel map.\n");
> -			return NULL;
> -		}
>  	}
> +
> +	map = machine.vmlinux_maps[MAP__FUNCTION];
> +	dso = map->dso;
> +
> +	vmlinux_name = symbol_conf.vmlinux_name;
> +	if (vmlinux_name) {
> +		if (dso__load_vmlinux(dso, map, vmlinux_name, NULL) == 0)
> +			goto found;
> +		return NULL;
> +	}
> +
> +	if (dso__load_vmlinux_path(dso, map, NULL) < 0) {
> +		pr_debug("Failed to load kernel map.\n");
> +		return NULL;
> +	}
> +

Hmm, could you put these changes into an else block?
Because this part is only for the kernel vmlinux, and
not for modules.

Thanks,


-- 
Masami HIRAMATSU
2nd Dept. Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: perf-probe: issue with latest fedora kernel

[Franck Bui-Huu]

Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> writes:

> (2010/12/10 16:53), Franck Bui-Huu wrote:

[...]

>> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
>> index 694f04b..558545e 100644
>> --- a/tools/perf/util/probe-event.c
>> +++ b/tools/perf/util/probe-event.c
>> @@ -114,6 +114,8 @@ static struct symbol *__find_kernel_function_by_name(const char *name,
>>  const char *kernel_get_module_path(const char *module)
>>  {
>>  	struct dso *dso;
>> +	struct map *map;
>> +	const char *vmlinux_name;
>>  
>>  	if (module) {
>>  		list_for_each_entry(dso, &machine.kernel_dsos, node) {
>> @@ -123,14 +125,23 @@ const char *kernel_get_module_path(const char *module)
>>  		}
>>  		pr_debug("Failed to find module %s.\n", module);
>>  		return NULL;
>> -	} else {
>> -		dso = machine.vmlinux_maps[MAP__FUNCTION]->dso;
>> -		if (dso__load_vmlinux_path(dso,
>> -			 machine.vmlinux_maps[MAP__FUNCTION], NULL) < 0) {
>> -			pr_debug("Failed to load kernel map.\n");
>> -			return NULL;
>> -		}
>>  	}
>> +
>> +	map = machine.vmlinux_maps[MAP__FUNCTION];
>> +	dso = map->dso;
>> +
>> +	vmlinux_name = symbol_conf.vmlinux_name;
>> +	if (vmlinux_name) {
>> +		if (dso__load_vmlinux(dso, map, vmlinux_name, NULL) == 0)

BTW, the condition seems wrong, it should have been '<= 0', that is if
an error occured or no symbol was loaded.

>> +			goto found;
>> +		return NULL;
>> +	}
>> +
>> +	if (dso__load_vmlinux_path(dso, map, NULL) < 0) {

This one should be '<= 0' too, since if no symbol is loaded, the image
is probably wrong.

>> +		pr_debug("Failed to load kernel map.\n");
>> +		return NULL;
>> +	}
>> +
>
> Hmm, could you put these changes into an else block?
> Because this part is only for the kernel vmlinux, and
> not for modules.

Ok, I'll do that.

Thanks.
-- 
		Franck

[PATCH 1/2] perf-probe: do use the kernel image path given by 'k' option

[Franck Bui-Huu]

From: Franck Bui-Huu <fbuihuu@gmail.com>

Signed-off-by: Franck Bui-Huu <fbuihuu@gmail.com>
---
 tools/perf/builtin-probe.c    |    5 +++++
 tools/perf/util/probe-event.c |   15 ++++++++++++---
 tools/perf/util/symbol.c      |    4 ++--
 tools/perf/util/symbol.h      |    2 ++
 4 files changed, 21 insertions(+), 5 deletions(-)

diff --git a/tools/perf/builtin-probe.c b/tools/perf/builtin-probe.c
index 2e000c0..add163c 100644
--- a/tools/perf/builtin-probe.c
+++ b/tools/perf/builtin-probe.c
@@ -249,6 +249,11 @@ int cmd_probe(int argc, const char **argv, const char *prefix __used)
 	     !params.show_lines))
 		usage_with_options(probe_usage, options);
 
+	/*
+	 * Only consider the user's kernel image path if given.
+	 */
+	symbol_conf.try_vmlinux_path = (symbol_conf.vmlinux_name == NULL);
+
 	if (params.list_events) {
 		if (params.mod_events) {
 			pr_err("  Error: Don't use --list with --add/--del.\n");
diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 694f04b..99f9933 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -114,6 +114,8 @@ static struct symbol *__find_kernel_function_by_name(const char *name,
 const char *kernel_get_module_path(const char *module)
 {
 	struct dso *dso;
+	struct map *map;
+	const char *vmlinux_name;
 
 	if (module) {
 		list_for_each_entry(dso, &machine.kernel_dsos, node) {
@@ -123,10 +125,17 @@ const char *kernel_get_module_path(const char *module)
 		}
 		pr_debug("Failed to find module %s.\n", module);
 		return NULL;
+	}
+
+	map = machine.vmlinux_maps[MAP__FUNCTION];
+	dso = map->dso;
+
+	vmlinux_name = symbol_conf.vmlinux_name;
+	if (vmlinux_name) {
+		if (dso__load_vmlinux(dso, map, vmlinux_name, NULL) <= 0)
+			return NULL;
 	} else {
-		dso = machine.vmlinux_maps[MAP__FUNCTION]->dso;
-		if (dso__load_vmlinux_path(dso,
-			 machine.vmlinux_maps[MAP__FUNCTION], NULL) < 0) {
+		if (dso__load_vmlinux_path(dso, map, NULL) < 0) {
 			pr_debug("Failed to load kernel map.\n");
 			return NULL;
 		}
diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index a3a0c43..4e42e0d 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -1780,8 +1780,8 @@ out_failure:
 	return -1;
 }
 
-static int dso__load_vmlinux(struct dso *self, struct map *map,
-			     const char *vmlinux, symbol_filter_t filter)
+int dso__load_vmlinux(struct dso *self, struct map *map,
+		      const char *vmlinux, symbol_filter_t filter)
 {
 	int err = -1, fd;
 
diff --git a/tools/perf/util/symbol.h b/tools/perf/util/symbol.h
index 038f220..6c6eafd 100644
--- a/tools/perf/util/symbol.h
+++ b/tools/perf/util/symbol.h
@@ -166,6 +166,8 @@ void dso__sort_by_name(struct dso *self, enum map_type type);
 struct dso *__dsos__findnew(struct list_head *head, const char *name);
 
 int dso__load(struct dso *self, struct map *map, symbol_filter_t filter);
+int dso__load_vmlinux(struct dso *self, struct map *map,
+		      const char *vmlinux, symbol_filter_t filter);
 int dso__load_vmlinux_path(struct dso *self, struct map *map,
 			   symbol_filter_t filter);
 int dso__load_kallsyms(struct dso *self, const char *filename, struct map *map,
-- 
1.7.3.2

[PATCH 2/2] perf-probe: fail if the kernel image contains no symbol

[Franck Bui-Huu]

From: Franck Bui-Huu <fbuihuu@gmail.com>

This fix is for kernel images found in the standard paths.

Signed-off-by: Franck Bui-Huu <fbuihuu@gmail.com>
---
 tools/perf/util/probe-event.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 99f9933..fc2b0b4 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -135,7 +135,7 @@ const char *kernel_get_module_path(const char *module)
 		if (dso__load_vmlinux(dso, map, vmlinux_name, NULL) <= 0)
 			return NULL;
 	} else {
-		if (dso__load_vmlinux_path(dso, map, NULL) < 0) {
+		if (dso__load_vmlinux_path(dso, map, NULL) <= 0) {
 			pr_debug("Failed to load kernel map.\n");
 			return NULL;
 		}
-- 
1.7.3.2


-- 
		Franck

Re: perf-probe: issue with latest fedora kernel

[Franck Bui-Huu]

Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:

> Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
>> I'm trying to use perf-probe(1) with the latest F14 kernel
>> (2.6.35.9-64.fc14.x86_64).
>> 
>
>> I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which
>> contains the running vmlinux image AFAICT.
>> 
>> I added it to the buildid cache:
>> 
>> $ perf buildid-cache -v -a /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
>> Adding 4d89e23415d8ab491cfc8ef8aa67764b91cc6787 /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux: OK
>> 
>> But when running perf-probe(1) like the following:
>> 
>>   $ perf probe schedule cpu
>>   Failed to find path of kernel moduleFailed to open debuginfo file.
>>   Error: Failed to add events. (-2)
>> 
>> it fails.
>> 
>> One strange thing is that when perf read the running kernel build id, it
>> get: 882b1b53eb1d65320e8fa710273aa8493896c4.
>> 
>> Could anybody tell me what I'm doing wrong ?
>
> Can you try using:
>
> $ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
>
> So that it tries it directly instead of looking into the buildid cache?
>
> I'll check if perf probe actually looks in the buildid cache, till then,
> please try specifying it directly,

Looks like a standard path is missing...

What about adding the one above since it looks like Fedora is installing
their debug kernel images at this place ?

diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index d628c8d..0f914c7 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -2260,7 +2262,7 @@ static int vmlinux_path__init(void)
 	if (uname(&uts) < 0)
 		return -1;
 
-	vmlinux_path = malloc(sizeof(char *) * 5);
+	vmlinux_path = malloc(sizeof(char *) * 6);
 	if (vmlinux_path == NULL)
 		return -1;
 
@@ -2288,6 +2290,12 @@ static int vmlinux_path__init(void)
 	if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
 		goto out_fail;
 	++vmlinux_path__nr_entries;
+	snprintf(bf, sizeof(bf), "/usr/lib/debug/lib/modules/%s.debug/vmlinux",
+		 uts.release);
+	vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
+	if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
+		goto out_fail;
+	++vmlinux_path__nr_entries;
 
 	return 0;
 
-- 
		Franck

Re: perf-probe: issue with latest fedora kernel

[Arnaldo Carvalho de Melo]

Em Fri, Dec 10, 2010 at 08:30:19AM +0100, Franck Bui-Huu escreveu:
> Looks like a standard path is missing...
> 
> What about adding the one above since it looks like Fedora is installing
> their debug kernel images at this place ?

Can I get your signed-off-by on this one?

- Arnaldo
 
> diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
> index d628c8d..0f914c7 100644
> --- a/tools/perf/util/symbol.c
> +++ b/tools/perf/util/symbol.c
> @@ -2260,7 +2262,7 @@ static int vmlinux_path__init(void)
>  	if (uname(&uts) < 0)
>  		return -1;
>  
> -	vmlinux_path = malloc(sizeof(char *) * 5);
> +	vmlinux_path = malloc(sizeof(char *) * 6);
>  	if (vmlinux_path == NULL)
>  		return -1;
>  
> @@ -2288,6 +2290,12 @@ static int vmlinux_path__init(void)
>  	if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
>  		goto out_fail;
>  	++vmlinux_path__nr_entries;
> +	snprintf(bf, sizeof(bf), "/usr/lib/debug/lib/modules/%s.debug/vmlinux",
> +		 uts.release);
> +	vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
> +	if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
> +		goto out_fail;
> +	++vmlinux_path__nr_entries;
>  
>  	return 0;
>  
> -- 
> 		Franck

Re: perf-probe: issue with latest fedora kernel

[Franck Bui-Huu]

Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:

> Em Fri, Dec 10, 2010 at 08:30:19AM +0100, Franck Bui-Huu escreveu:
>> Looks like a standard path is missing...
>> 
>> What about adding the one above since it looks like Fedora is installing
>> their debug kernel images at this place ?
>
> Can I get your signed-off-by on this one?
>

Of course.

[PATCH] perf-tools: add one more default search path for kernel image

From: Franck Bui-Huu <fbuihuu@gmail.com>

It looks like Fedora is using this one.

Reported-by: Francis Moreau <francis.moro@gmail.com>
Signed-off-by: Franck Bui-Huu <fbuihuu@gmail.com>
---
 tools/perf/util/symbol.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index d628c8d..a3a0c43 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -2260,7 +2260,7 @@ static int vmlinux_path__init(void)
 	if (uname(&uts) < 0)
 		return -1;
 
-	vmlinux_path = malloc(sizeof(char *) * 5);
+	vmlinux_path = malloc(sizeof(char *) * 6);
 	if (vmlinux_path == NULL)
 		return -1;
 
@@ -2288,6 +2288,12 @@ static int vmlinux_path__init(void)
 	if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
 		goto out_fail;
 	++vmlinux_path__nr_entries;
+	snprintf(bf, sizeof(bf), "/usr/lib/debug/lib/modules/%s.debug/vmlinux",
+		 uts.release);
+	vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
+	if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
+		goto out_fail;
+	++vmlinux_path__nr_entries;
 
 	return 0;
 
-- 
1.7.3.2

Re: perf-probe: issue with latest fedora kernel

[Arnaldo Carvalho de Melo]

Em Sun, Dec 12, 2010 at 11:15:26PM +0100, Franck Bui-Huu escreveu:
> Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
> 
> > Em Fri, Dec 10, 2010 at 08:30:19AM +0100, Franck Bui-Huu escreveu:
> >> Looks like a standard path is missing...
> >> 
> >> What about adding the one above since it looks like Fedora is installing
> >> their debug kernel images at this place ?
> >
> > Can I get your signed-off-by on this one?
> >
> 
> Of course.
> 
> [PATCH] perf-tools: add one more default search path for kernel image
> 
> From: Franck Bui-Huu <fbuihuu@gmail.com>
> 
> It looks like Fedora is using this one.

Nope, this is not needed, see the rest of the discussion please.

- Arnaldo

Re: perf-probe: issue with latest fedora kernel

[Francis Moreau]

Hi,

Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:

> Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
>> I'm trying to use perf-probe(1) with the latest F14 kernel
>> (2.6.35.9-64.fc14.x86_64).
>> 
>
>> I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which
>> contains the running vmlinux image AFAICT.
>> 
>> I added it to the buildid cache:
>> 
>> $ perf buildid-cache -v -a /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
>> Adding 4d89e23415d8ab491cfc8ef8aa67764b91cc6787 /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux: OK
>> 
>> But when running perf-probe(1) like the following:
>> 
>>   $ perf probe schedule cpu
>>   Failed to find path of kernel moduleFailed to open debuginfo file.
>>   Error: Failed to add events. (-2)
>> 
>> it fails.
>> 
>> One strange thing is that when perf read the running kernel build id, it
>> get: 882b1b53eb1d65320e8fa710273aa8493896c4.
>> 
>> Could anybody tell me what I'm doing wrong ?
>
> Can you try using:
>
> $ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
>
> So that it tries it directly instead of looking into the buildid cache?
>
> I'll check if perf probe actually looks in the buildid cache, till then,
> please try specifying it directly,

Does anybody know where I can report this issue to Fedora community ?

-- 
Francis

Re: perf-probe: issue with latest fedora kernel

[Arnaldo Carvalho de Melo]

Em Mon, Dec 13, 2010 at 11:02:52AM +0100, Francis Moreau escreveu:
> Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
> > Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
> >> I'm trying to use perf-probe(1) with the latest F14 kernel
> >> (2.6.35.9-64.fc14.x86_64).

> >> I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which
> >> contains the running vmlinux image AFAICT.

> > Can you try using:

> > $ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>

> > So that it tries it directly instead of looking into the buildid cache?

> Does anybody know where I can report this issue to Fedora community ?

Please create a ticket at:

http://bugzilla.redhat.com

More specifically:

https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora

Mention this changeset:

http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=commit;h=34d2ea084a3bce03d73ffe64350db2813f07260c

It will make the -k argument work.

There is another that will look automatically at this location, but I
haven't merged yet, will post here when done.

- Arnaldo

Re: perf-probe: issue with latest fedora kernel

[Francis Moreau]

Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:

> Em Mon, Dec 13, 2010 at 11:02:52AM +0100, Francis Moreau escreveu:
>> Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
>> > Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
>> >> I'm trying to use perf-probe(1) with the latest F14 kernel
>> >> (2.6.35.9-64.fc14.x86_64).
>
>> >> I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which
>> >> contains the running vmlinux image AFAICT.
>
>> > Can you try using:
>
>> > $ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
>
>> > So that it tries it directly instead of looking into the buildid cache?
>
>> Does anybody know where I can report this issue to Fedora community ?
>
> Please create a ticket at:
>
> http://bugzilla.redhat.com
>
> More specifically:
>
> https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora
>
> Mention this changeset:
>
> http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=commit;h=34d2ea084a3bce03d73ffe64350db2813f07260c
>
> It will make the -k argument work.
>
> There is another that will look automatically at this location, but I
> haven't merged yet, will post here when done.

Hmm, I don't think that any patches recently posted in response of this
thread will solve the problem.

The primarily concern here, is that the running kernel has a different
build-id than the debug image installed by the corresponding debug
package.

I think it's a distrib issue, hence my question about Fedora report.

Thanks
-- 
Francis

Re: perf-probe: issue with latest fedora kernel

[Masami Hiramatsu]

(2010/12/13 22:08), Francis Moreau wrote:
> Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
> 
>> Em Mon, Dec 13, 2010 at 11:02:52AM +0100, Francis Moreau escreveu:
>>> Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
>>>> Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
>>>>> I'm trying to use perf-probe(1) with the latest F14 kernel
>>>>> (2.6.35.9-64.fc14.x86_64).
>>
>>>>> I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which
>>>>> contains the running vmlinux image AFAICT.
>>
>>>> Can you try using:
>>
>>>> $ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
>>
>>>> So that it tries it directly instead of looking into the buildid cache?
>>
>>> Does anybody know where I can report this issue to Fedora community ?
>>
>> Please create a ticket at:
>>
>> http://bugzilla.redhat.com
>>
>> More specifically:
>>
>> https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora
>>
>> Mention this changeset:
>>
>> http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=commit;h=34d2ea084a3bce03d73ffe64350db2813f07260c
>>
>> It will make the -k argument work.
>>
>> There is another that will look automatically at this location, but I
>> haven't merged yet, will post here when done.
> 
> Hmm, I don't think that any patches recently posted in response of this
> thread will solve the problem.

I agree with you.

> The primarily concern here, is that the running kernel has a different
> build-id than the debug image installed by the corresponding debug
> package.
> 
> I think it's a distrib issue, hence my question about Fedora report.

Yes, that is just for fedora people who can easily check
how to reproduce and make sure the problem is here.
Then they can try to solve the problem until the issue
is resolved.

Thank you,



-- 
Masami HIRAMATSU
2nd Dept. Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: perf-probe: issue with latest fedora kernel

[Francis Moreau]

Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> writes:

> (2010/12/13 22:08), Francis Moreau wrote:
>> Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
>> 
>>> Em Mon, Dec 13, 2010 at 11:02:52AM +0100, Francis Moreau escreveu:
>
>>>> Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
>>>>> Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
>>>>>> I'm trying to use perf-probe(1) with the latest F14 kernel
>>>>>> (2.6.35.9-64.fc14.x86_64).
>>>
>>>>>> I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which
>>>>>> contains the running vmlinux image AFAICT.
>>>
>>>>> Can you try using:
>>>
>>>>> $ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
>>>
>>>>> So that it tries it directly instead of looking into the buildid cache?
>>>
>>>> Does anybody know where I can report this issue to Fedora community ?
>>>
>>> Please create a ticket at:
>>>
>>> http://bugzilla.redhat.com
>>>
>>> More specifically:
>>>
>>> https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora
>>>
>>> Mention this changeset:
>>>
>>> http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=commit;h=34d2ea084a3bce03d73ffe64350db2813f07260c
>>>
>>> It will make the -k argument work.
>>>
>>> There is another that will look automatically at this location, but I
>>> haven't merged yet, will post here when done.
>> 
>> Hmm, I don't think that any patches recently posted in response of this
>> thread will solve the problem.
>
> I agree with you.
>
>> The primarily concern here, is that the running kernel has a different
>> build-id than the debug image installed by the corresponding debug
>> package.
>> 
>> I think it's a distrib issue, hence my question about Fedora report.
>
> Yes, that is just for fedora people who can easily check
> how to reproduce and make sure the problem is here.
> Then they can try to solve the problem until the issue
> is resolved.

I created this report: 

https://bugzilla.redhat.com/show_bug.cgi?id=662653

Thanks
-- 
Francis

Re: perf-probe: issue with latest fedora kernel

[Francis Moreau]

Francis Moreau <francis.moro@gmail.com> writes:

> Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> writes:
>
>> (2010/12/13 22:08), Francis Moreau wrote:
>>> Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
>>> 
>>>> Em Mon, Dec 13, 2010 at 11:02:52AM +0100, Francis Moreau escreveu:
>>
>>>>> Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
>>>>>> Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
>>>>>>> I'm trying to use perf-probe(1) with the latest F14 kernel
>>>>>>> (2.6.35.9-64.fc14.x86_64).
>>>>
>>>>>>> I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which
>>>>>>> contains the running vmlinux image AFAICT.
>>>>
>>>>>> Can you try using:
>>>>
>>>>>> $ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
>>>>
>>>>>> So that it tries it directly instead of looking into the buildid cache?
>>>>
>>>>> Does anybody know where I can report this issue to Fedora community ?
>>>>
>>>> Please create a ticket at:
>>>>
>>>> http://bugzilla.redhat.com
>>>>
>>>> More specifically:
>>>>
>>>> https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora
>>>>
>>>> Mention this changeset:
>>>>
>>>> http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=commit;h=34d2ea084a3bce03d73ffe64350db2813f07260c
>>>>
>>>> It will make the -k argument work.
>>>>
>>>> There is another that will look automatically at this location, but I
>>>> haven't merged yet, will post here when done.
>>> 
>>> Hmm, I don't think that any patches recently posted in response of this
>>> thread will solve the problem.
>>
>> I agree with you.
>>
>>> The primarily concern here, is that the running kernel has a different
>>> build-id than the debug image installed by the corresponding debug
>>> package.
>>> 
>>> I think it's a distrib issue, hence my question about Fedora report.
>>
>> Yes, that is just for fedora people who can easily check
>> how to reproduce and make sure the problem is here.
>> Then they can try to solve the problem until the issue
>> is resolved.
>
> I created this report: 
>
> https://bugzilla.redhat.com/show_bug.cgi?id=662653
>

Now closed.

-- 
Francis

Re: perf-probe: issue with latest fedora kernel

[Arnaldo Carvalho de Melo]

Em Mon, Dec 13, 2010 at 11:04:36PM +0900, Masami Hiramatsu escreveu:
> (2010/12/13 22:08), Francis Moreau wrote:
> > Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
> > 
> >> Em Mon, Dec 13, 2010 at 11:02:52AM +0100, Francis Moreau escreveu:
> >>> Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
> >>>> Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
> >>>>> I'm trying to use perf-probe(1) with the latest F14 kernel
> >>>>> (2.6.35.9-64.fc14.x86_64).
> >>
> >>>>> I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which
> >>>>> contains the running vmlinux image AFAICT.

> > The primarily concern here, is that the running kernel has a different
> > build-id than the debug image installed by the corresponding debug
> > package.
> > 
> > I think it's a distrib issue, hence my question about Fedora report.
> 
> Yes, that is just for fedora people who can easily check
> how to reproduce and make sure the problem is here.
> Then they can try to solve the problem until the issue
> is resolved.

I just tried it, works for me.

I added the sequence of commands, perhaps he can find where the problem
lies following them.

- Arnaldo

Re: perf-probe: issue with latest fedora kernel

[Arnaldo Carvalho de Melo]

Em Mon, Dec 13, 2010 at 02:08:56PM +0100, Francis Moreau escreveu:
> Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
> > Em Mon, Dec 13, 2010 at 11:02:52AM +0100, Francis Moreau escreveu:
> >> Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
> >> > Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
> >> >> I'm trying to use perf-probe(1) with the latest F14 kernel
> >> >> (2.6.35.9-64.fc14.x86_64).
> >> >> I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which
> >> >> contains the running vmlinux image AFAICT.
> >> > Can you try using:

> >> > $ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>

> >> > So that it tries it directly instead of looking into the buildid cache?

> >> Does anybody know where I can report this issue to Fedora community ?

> > Please create a ticket at:

> > http://bugzilla.redhat.com

> Hmm, I don't think that any patches recently posted in response of this
> thread will solve the problem.
> 
> The primarily concern here, is that the running kernel has a different
> build-id than the debug image installed by the corresponding debug
> package.

So? The problem for me is that:

1. looks at /sys/kernel/notes and gets the running kernel build-id
2. fails to locate a vmlinux with a matching build-id
3. ignores the -k specified path
4. The mentioned patch makes it consider the -k specified path

So it should look at the -k specified path and make sure that it has a
matching build-id, if it has, problem solved, no?
 
> I think it's a distrib issue, hence my question about Fedora report.

Well, here I have fedora 14 and:

[root@felicio /]# uname -r
2.6.35.9-64.fc14.x86_64
[root@felicio /]# rpm -qa | grep kernel-debuginfo
kernel-debuginfo-2.6.35.9-64.fc14.x86_64
kernel-debuginfo-common-x86_64-2.6.35.9-64.fc14.x86_64
[root@felicio /]# perf probe -L icmp_rcv
Failed to open net/ipv4/icmp.c: No such file or directory
  Error: Failed to show lines. (-2)
<icmp_rcv:0>
[root@felicio /]# rpm -ql kernel-debuginfo-common-x86_64 | grep net/ipv4/icmp.c
/usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64/net/ipv4/icmp.c
[root@felicio /]# cd /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64/
[root@felicio linux-2.6.35.x86_64]# perf probe -L icmp_rcv | head -30
Source file is shorter than expected.
<icmp_rcv:0>
      0  int icmp_rcv(struct sk_buff *skb)
      1  {
         	struct icmphdr *icmph;
         	struct rtable *rt = skb_rtable(skb);
         	struct net *net = dev_net(rt->u.dst.dev);
         
      6  	if (!xfrm4_policy_check(NULL, XFRM_POLICY_IN, skb)) {
         		struct sec_path *sp = skb_sec_path(skb);
         		int nh;
         
     10  		if (!(sp && sp->xvec[sp->len - 1]->props.flags &
         				 XFRM_STATE_ICMP))
         			goto drop;
         
     14  		if (!pskb_may_pull(skb, sizeof(*icmph) + sizeof(struct iphdr)))
         			goto drop;
         
         		nh = skb_network_offset(skb);
         		skb_set_network_header(skb, sizeof(*icmph));
         
     20  		if (!xfrm4_policy_check_reverse(NULL, XFRM_POLICY_IN, skb))
         			goto drop;
         
         		skb_set_network_header(skb, nh);
         	}
         
     26  	ICMP_INC_STATS_BH(net, ICMP_MIB_INMSGS);

[root@felicio linux-2.6.35.x86_64]# cat /proc/net/snmp | grep Icmp:
Icmp: InMsgs InErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps
Icmp: 4 1 2 0 0 0 0 1 1 0 0 0 0 119 0 116 0 0 0 0 3 0 0 0 0 0
[root@felicio linux-2.6.35.x86_64]# perf probe icmp_rcv:26
Add new event:
  probe:icmp_rcv       (on icmp_rcv:26)

You can now use it on all perf tools, such as:

	perf record -e probe:icmp_rcv -aR sleep 1

[root@felicio linux-2.6.35.x86_64]# perf record -e probe:icmp_rcv -ag
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.535 MB perf.data (~23357 samples) ]

[root@felicio linux-2.6.35.x86_64]# perf report
# Events: 3  cycles
#
# Overhead          Command      Shared Object  Symbol
# ........  ...............  .................  ......
#
    66.67%          swapper  [kernel.kallsyms]  [k] icmp_rcv
                    |
                    --- icmp_rcv
                        ip_local_deliver_finish
                        NF_HOOK.clone.8
                        ip_local_deliver
                        ip_rcv_finish
                        NF_HOOK.clone.8
                        ip_rcv
                        __netif_receive_skb
                        netif_receive_skb
                        napi_skb_finish
                        napi_gro_receive
                        igb_poll
                        net_rx_action
                        __do_softirq
                        call_softirq
                        do_softirq
                        irq_exit
                        do_IRQ
                        common_interrupt
                        default_idle
                        cpu_idle
                        start_secondary

    33.33%              ssh  [kernel.kallsyms]  [k] icmp_rcv
                        |
                        --- icmp_rcv
                            ip_local_deliver_finish
                            NF_HOOK.clone.8
                            ip_local_deliver
                            ip_rcv_finish
                            NF_HOOK.clone.8
                            ip_rcv
                            __netif_receive_skb
                            netif_receive_skb
                            napi_skb_finish
                            napi_gro_receive
                            igb_poll
                            net_rx_action
                            __do_softirq
                            call_softirq
[root@felicio linux-2.6.35.x86_64]#
[root@felicio linux-2.6.35.x86_64]# cat /proc/net/snmp | grep Icmp:
Icmp: InMsgs InErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps
Icmp: 7 1 2 0 0 0 0 4 1 0 0 0 0 123 0 117 0 0 0 0 3 3 0 0 0 0
[root@felicio linux-2.6.35.x86_64]#



All seems to work, right?

There are two problems here:

on the 'report' header, as we didn't collect "cycles", but
"probe:icmp_rcv" events, that is another fix that needs to be backported to
fedora, but a minor one.

Also 'perf probe -L' should look use
/usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 as a prefix to look for
sources.

That can be found using:

[root@felicio linux-2.6.35.x86_64]# readelf -wi /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux | grep DW_AT_comp_dir | head -5
    <2e>   DW_AT_comp_dir    : /usr/src/debug////////kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <75e8>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <900e>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
    <16670>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
[root@felicio linux-2.6.35.x86_64]# 

 <0><8b>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <8c>   DW_AT_producer    : (indirect string, offset: 0x22f4): GNU C 4.5.1 20100924 (Red Hat 4.5.1-4)        
    <90>   DW_AT_language    : 1        (ANSI C)
    <91>   DW_AT_name        : (indirect string, offset: 0x37a): arch/x86/kernel/head64.c       
    <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 
    <99>   DW_AT_low_pc      : 0x0      
    <a1>   DW_AT_entry_pc    : 0x0      
    <a9>   DW_AT_ranges      : 0x340    
    <ad>   DW_AT_stmt_list   : 0x114

I.e. using the DW_AT_comp_dir attribute in the DW_TAG_compile_unit DWARF tags,
Masami?

All this with:

[root@felicio linux-2.6.35.x86_64]# type perf
perf is hashed (/usr/bin/perf)
[root@felicio linux-2.6.35.x86_64]# rpm -qf /usr/bin/perf
perf-2.6.35.9-64.fc14.x86_64
[root@felicio linux-2.6.35.x86_64]# 

I.e. the perf binary shipping in fedora, without the recent patches.

To finish the session:

[root@felicio linux-2.6.35.x86_64]# perf probe --list
  probe:icmp_rcv       (on icmp_rcv:26@ipv4/icmp.c)
[root@felicio linux-2.6.35.x86_64]# perf probe --del icmp_rcv
Remove event: probe:icmp_rcv
[root@felicio linux-2.6.35.x86_64]# perf probe --list
[root@felicio linux-2.6.35.x86_64]# 

And if you run it in verbose mode we can see what was the vmlinux file used:

[root@felicio linux-2.6.35.x86_64]# perf probe -v icmp_rcv:26
probe-definition(0): icmp_rcv:26 
symbol:icmp_rcv file:(null) line:26 offset:0 return:0 lazy:(null)
0 arguments
Looking at the vmlinux_path (6 entries long)
Using /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux for symbols
Try to open /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux
Probe line found: line[571]:999 addr:0xffffffff81416655
Probe point found: icmp_rcv+208
find 1 kprobe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1
Add new event:
Writing event: p:probe/icmp_rcv icmp_rcv+208
  probe:icmp_rcv       (on icmp_rcv:26)

You can now use it on all perf tools, such as:

	perf record -e probe:icmp_rcv -aR sleep 1

[root@felicio linux-2.6.35.x86_64]#

It was /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux, that is present on:

[root@felicio linux-2.6.35.x86_64]# rpm -qf /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux
kernel-debuginfo-2.6.35.9-64.fc14.x86_64

And...

[root@felicio linux-2.6.35.x86_64]# perf buildid-list
882b1b53eb1d653200e8fa7100273aa8493896c4 [kernel.kallsyms]

This is the build-id of the running kernel, and:

[root@felicio linux-2.6.35.x86_64]# perf buildid-cache -v --add /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux
Adding 882b1b53eb1d653200e8fa7100273aa8493896c4 /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux: FAIL
/usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux already in the cache
[root@felicio linux-2.6.35.x86_64]# 

The running kernel build-id matches the build-id in the vmlinux file found in
the kernel-debuginfo package.

So can you please give us the versions for the kernel, kernel-debuginfo package?
Perhaps you have a new debuginfo package and is running an older kernel?

Use the perf buildid-list trick to get the build-id in readable format.

Thanks,

- Arnaldo

Re: perf-probe: issue with latest fedora kernel

[Francis Moreau]

Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:

> Em Mon, Dec 13, 2010 at 02:08:56PM +0100, Francis Moreau escreveu:
>> Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
>> > Em Mon, Dec 13, 2010 at 11:02:52AM +0100, Francis Moreau escreveu:
>> >> Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
>
>> >> > Em Thu, Dec 09, 2010 at 04:00:08PM +0100, Francis Moreau escreveu:
>> >> >> I'm trying to use perf-probe(1) with the latest F14 kernel
>> >> >> (2.6.35.9-64.fc14.x86_64).
>> >> >> I also installed kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 which
>> >> >> contains the running vmlinux image AFAICT.
>> >> > Can you try using:
>
>> >> > $ perf probe -k /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux <rest of the probe definition>
>
>> >> > So that it tries it directly instead of looking into the buildid cache?
>
>> >> Does anybody know where I can report this issue to Fedora community ?
>
>> > Please create a ticket at:
>
>> > http://bugzilla.redhat.com
>
>> Hmm, I don't think that any patches recently posted in response of this
>> thread will solve the problem.
>> 
>> The primarily concern here, is that the running kernel has a different
>> build-id than the debug image installed by the corresponding debug
>> package.
>
> So? The problem for me is that:
>
> 1. looks at /sys/kernel/notes and gets the running kernel build-id
> 2. fails to locate a vmlinux with a matching build-id
> 3. ignores the -k specified path
> 4. The mentioned patch makes it consider the -k specified path
>
> So it should look at the -k specified path and make sure that it has a
> matching build-id, if it has, problem solved, no?

But as I said, there's no matching build-id. I understand that -k is
currently broken but suppose that it works, which vmlinux should I use
then ?

$ rpm -qa | grep kernel
kernel-debuginfo-common-x86_64-2.6.35.9-64.fc14.x86_64
kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64
kernel-headers-2.6.35.6-48.fc14.x86_64
kernel-devel-2.6.35.9-64.fc14.x86_64
kernel-2.6.35.9-64.fc14.x86_64

$ rpm -ql kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 | grep vmlinux
/usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux

So the debug package installed the kernel image with debug info into

   /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug

directory. Please note that this path can't be found by perf without the
recent patches.

To check the build id of this image, I do:

$ perf buildid-cache -v -a /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
Adding 4d89e23415d8ab491cfc8ef8aa67764b91cc6787 ...

This is the only way I know to dump the debug id of an object.

The buildid of the running kernel:

$ perf buildid-list
882b1b53eb1d653200e8fa7100273aa8493896c4 [kernel.kallsyms]

So they both don't match.

>
>  
>> I think it's a distrib issue, hence my question about Fedora report.
>
> Well, here I have fedora 14 and:

[...]

>
> All this with:
>
> [root@felicio linux-2.6.35.x86_64]# type perf
> perf is hashed (/usr/bin/perf)
> [root@felicio linux-2.6.35.x86_64]# rpm -qf /usr/bin/perf
> perf-2.6.35.9-64.fc14.x86_64
> [root@felicio linux-2.6.35.x86_64]# 
>
> I.e. the perf binary shipping in fedora, without the recent patches.

In that case how perf can find the right vmlinux in your system since it
needs to be patched to find the vmlinux image in the
/usr/lib/debug/lib/modules/%s.debug directory ?

>
>
> To finish the session:
>

[...]

>
> And...
>
> [root@felicio linux-2.6.35.x86_64]# perf buildid-list
> 882b1b53eb1d653200e8fa7100273aa8493896c4 [kernel.kallsyms]
>
> This is the build-id of the running kernel, and:
>
> [root@felicio linux-2.6.35.x86_64]# perf buildid-cache -v --add /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux
> Adding 882b1b53eb1d653200e8fa7100273aa8493896c4 /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux: FAIL
> /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux already in the cache

Ah, that's interesting...

In the path you're using, you loose the '.debug', therefore this image
doesn't seem to belong to kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64.

And that explains how perf can find the debug kernel image.

-- 
Francis

Re: perf-probe: issue with latest fedora kernel

[Arnaldo Carvalho de Melo]

Em Mon, Dec 13, 2010 at 04:11:16PM +0100, Francis Moreau escreveu:
> But as I said, there's no matching build-id. I understand that -k is
> currently broken but suppose that it works, which vmlinux should I use
> then ?
> 
> $ rpm -qa | grep kernel
> kernel-debuginfo-common-x86_64-2.6.35.9-64.fc14.x86_64
> kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64
> kernel-headers-2.6.35.6-48.fc14.x86_64
> kernel-devel-2.6.35.9-64.fc14.x86_64
> kernel-2.6.35.9-64.fc14.x86_64

Bzzt, here is the problem: you're not using the right package,
kernel-debug-debuginfo is for the kernel-debug package, a kernel built
with lots of CONFIG_DEBUG_ options turned on, please install the right
package, which is kernel-debuginfo-2.6.35.9-64.fc14.x86_64
 
> $ rpm -ql kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64 | grep vmlinux
> /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug/vmlinux
> 
> So the debug package installed the kernel image with debug info into
> 
>    /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64.debug

Right, this matches what is in the kernel in the kernel-debug package.

Please install the right debuginfo package and try again

- Arnaldo

Re: perf-probe: issue with latest fedora kernel

[Francis Moreau]

Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:

> Em Mon, Dec 13, 2010 at 04:11:16PM +0100, Francis Moreau escreveu:
>> But as I said, there's no matching build-id. I understand that -k is
>> currently broken but suppose that it works, which vmlinux should I use
>> then ?
>
>> 
>> $ rpm -qa | grep kernel
>> kernel-debuginfo-common-x86_64-2.6.35.9-64.fc14.x86_64
>> kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64
>> kernel-headers-2.6.35.6-48.fc14.x86_64
>> kernel-devel-2.6.35.9-64.fc14.x86_64
>> kernel-2.6.35.9-64.fc14.x86_64
>
> Bzzt, here is the problem: you're not using the right package,
> kernel-debug-debuginfo is for the kernel-debug package, a kernel built
> with lots of CONFIG_DEBUG_ options turned on, please install the right
> package, which is kernel-debuginfo-2.6.35.9-64.fc14.x86_64

Ah, that's it.

Thanks !
-- 
Francis

Re: perf-probe: issue with latest fedora kernel

[Arnaldo Carvalho de Melo]

Em Mon, Dec 13, 2010 at 08:08:08PM +0100, Francis Moreau escreveu:
> Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
> 
> > Em Mon, Dec 13, 2010 at 04:11:16PM +0100, Francis Moreau escreveu:
> >> But as I said, there's no matching build-id. I understand that -k is
> >> currently broken but suppose that it works, which vmlinux should I use
> >> then ?

> >> $ rpm -qa | grep kernel
> >> kernel-debuginfo-common-x86_64-2.6.35.9-64.fc14.x86_64
> >> kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64
> >> kernel-headers-2.6.35.6-48.fc14.x86_64
> >> kernel-devel-2.6.35.9-64.fc14.x86_64
> >> kernel-2.6.35.9-64.fc14.x86_64

> > Bzzt, here is the problem: you're not using the right package,
> > kernel-debug-debuginfo is for the kernel-debug package, a kernel built
> > with lots of CONFIG_DEBUG_ options turned on, please install the right
> > package, which is kernel-debuginfo-2.6.35.9-64.fc14.x86_64
> 
> Ah, that's it.

Yeah, those package names are a bit confusing :-)

Please let us know about any other problem you have using these tools,
any feedback is _greatly_ appreciated.

- Arnaldo

Re: perf-probe: issue with latest fedora kernel

[Francis Moreau]

Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:

> Em Mon, Dec 13, 2010 at 08:08:08PM +0100, Francis Moreau escreveu:
>> Arnaldo Carvalho de Melo <acme@ghostprotocols.net> writes:
>> 
>> > Em Mon, Dec 13, 2010 at 04:11:16PM +0100, Francis Moreau escreveu:
>
>> >> But as I said, there's no matching build-id. I understand that -k is
>> >> currently broken but suppose that it works, which vmlinux should I use
>> >> then ?
>
>> >> $ rpm -qa | grep kernel
>> >> kernel-debuginfo-common-x86_64-2.6.35.9-64.fc14.x86_64
>> >> kernel-debug-debuginfo-2.6.35.9-64.fc14.x86_64
>> >> kernel-headers-2.6.35.6-48.fc14.x86_64
>> >> kernel-devel-2.6.35.9-64.fc14.x86_64
>> >> kernel-2.6.35.9-64.fc14.x86_64
>
>> > Bzzt, here is the problem: you're not using the right package,
>> > kernel-debug-debuginfo is for the kernel-debug package, a kernel built
>> > with lots of CONFIG_DEBUG_ options turned on, please install the right
>> > package, which is kernel-debuginfo-2.6.35.9-64.fc14.x86_64
>> 
>> Ah, that's it.
>
> Yeah, those package names are a bit confusing :-)
>
> Please let us know about any other problem you have using these tools,
> any feedback is _greatly_ appreciated.

Don't worry, I won't miss an opportunity.

Thanks
-- 
Francis

Re: perf-probe: issue with latest fedora kernel

[Masami Hiramatsu]

(2010/12/13 23:21), Arnaldo Carvalho de Melo wrote:

> There are two problems here:
> 
> on the 'report' header, as we didn't collect "cycles", but
> "probe:icmp_rcv" events, that is another fix that needs to be backported to
> fedora, but a minor one.
> 
> Also 'perf probe -L' should look use
> /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 as a prefix to look for
> sources.
> 
> That can be found using:
> 
> [root@felicio linux-2.6.35.x86_64]# readelf -wi /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux | grep DW_AT_comp_dir | head -5
>     <2e>   DW_AT_comp_dir    : /usr/src/debug////////kernel-2.6.35.fc14/linux-2.6.35.x86_64	
>     <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
>     <75e8>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
>     <900e>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
>     <16670>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
> [root@felicio linux-2.6.35.x86_64]# 
> 
>  <0><8b>: Abbrev Number: 1 (DW_TAG_compile_unit)
>     <8c>   DW_AT_producer    : (indirect string, offset: 0x22f4): GNU C 4.5.1 20100924 (Red Hat 4.5.1-4)        
>     <90>   DW_AT_language    : 1        (ANSI C)
>     <91>   DW_AT_name        : (indirect string, offset: 0x37a): arch/x86/kernel/head64.c       
>     <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 
>     <99>   DW_AT_low_pc      : 0x0      
>     <a1>   DW_AT_entry_pc    : 0x0      
>     <a9>   DW_AT_ranges      : 0x340    
>     <ad>   DW_AT_stmt_list   : 0x114
> 
> I.e. using the DW_AT_comp_dir attribute in the DW_TAG_compile_unit DWARF tags,
> Masami?

It already supports DW_AT_comp_dir on upstream kernel,
see find_line_range() in utils/probe-finder.c ;-)

Thank you,


-- 
Masami HIRAMATSU
2nd Dept. Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: perf-probe: issue with latest fedora kernel

[Arnaldo Carvalho de Melo]

Em Wed, Dec 15, 2010 at 05:57:11PM +0900, Masami Hiramatsu escreveu:
> (2010/12/13 23:21), Arnaldo Carvalho de Melo wrote:
> > Also 'perf probe -L' should look use
> > /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 as a prefix to look for
> > sources.

> > That can be found using:

> >  <0><8b>: Abbrev Number: 1 (DW_TAG_compile_unit)
> >     <8c>   DW_AT_producer    : (indirect string, offset: 0x22f4): GNU C 4.5.1 20100924 (Red Hat 4.5.1-4)        
> >     <90>   DW_AT_language    : 1        (ANSI C)
> >     <91>   DW_AT_name        : (indirect string, offset: 0x37a): arch/x86/kernel/head64.c       
> >     <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 

> > I.e. using the DW_AT_comp_dir attribute in the DW_TAG_compile_unit DWARF tags,
> > Masami?

> It already supports DW_AT_comp_dir on upstream kernel,
> see find_line_range() in utils/probe-finder.c ;-)

I happily stand corrected, was using the f14 one :-)

Great work!

- Arnaldo

Re: perf-probe: issue with latest fedora kernel

[Franck Bui-Huu]

Hello Masami,

Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> writes:

> (2010/12/13 23:21), Arnaldo Carvalho de Melo wrote:
>
>> There are two problems here:
>> 
>> on the 'report' header, as we didn't collect "cycles", but
>> "probe:icmp_rcv" events, that is another fix that needs to be backported to
>> fedora, but a minor one.
>> 
>> Also 'perf probe -L' should look use
>> /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 as a prefix to look for
>> sources.
>> 
>> That can be found using:
>> 
>> [root@felicio linux-2.6.35.x86_64]# readelf -wi /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux | grep DW_AT_comp_dir | head -5
>>     <2e>   DW_AT_comp_dir    : /usr/src/debug////////kernel-2.6.35.fc14/linux-2.6.35.x86_64	
>>     <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
>>     <75e8>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
>>     <900e>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
>>     <16670>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
>> [root@felicio linux-2.6.35.x86_64]# 
>> 
>>  <0><8b>: Abbrev Number: 1 (DW_TAG_compile_unit)
>>     <8c>   DW_AT_producer    : (indirect string, offset: 0x22f4): GNU C 4.5.1 20100924 (Red Hat 4.5.1-4)        
>>     <90>   DW_AT_language    : 1        (ANSI C)
>>     <91>   DW_AT_name        : (indirect string, offset: 0x37a): arch/x86/kernel/head64.c       
>>     <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 
>>     <99>   DW_AT_low_pc      : 0x0      
>>     <a1>   DW_AT_entry_pc    : 0x0      
>>     <a9>   DW_AT_ranges      : 0x340    
>>     <ad>   DW_AT_stmt_list   : 0x114
>> 
>> I.e. using the DW_AT_comp_dir attribute in the DW_TAG_compile_unit DWARF tags,
>> Masami?
>
> It already supports DW_AT_comp_dir on upstream kernel,
> see find_line_range() in utils/probe-finder.c ;-)
>

It looks like adding probe with lazy matching pattern does not:

   $ cd /dev
   $ perf probe -n -a "icmp.c;sk=*"
   Failed to open net/ipv4/icmp.c: Operation not permitted
     Error: Failed to add events. (-2)
   $ cd /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64/
   $ perf probe -n -a "icmp.c;sk=*"
   Add new events:
     probe:icmp_push_reply (on @cmp.c)
     probe:icmp_reply     (on @cmp.c)
     probe:icmp_reply_1   (on @cmp.c)
     probe:icmp_send      (on @cmp.c)
     probe:icmp_send_1    (on @cmp.c)
     probe:icmp_error     (on @cmp.c)
     probe:icmp_error_1   (on @cmp.c)
     probe:icmp_error_2   (on @cmp.c)
     probe:icmp_error_3   (on @cmp.c)
   
   You can now use it on all perf tools, such as:
   
   	perf record -e probe:icmp_error_3 -aR sleep 1

Below is a quick and dirty fix (sorry, I don't have any time to make it
better for now), also note that the note of the file (icmp.c) reported
is screwed...

diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 10ad1ad..53c2f97 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -241,7 +241,7 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
  * a newly allocated path on success.
  * Return 0 if file was found and readable, -errno otherwise.
  */
-static int get_real_path(const char *raw_path, const char *comp_dir,
+int get_real_path(const char *raw_path, const char *comp_dir,
 			 char **new_path)
 {
 	const char *prefix = symbol_conf.source_prefix;
diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
index 3991d73..88829ef 100644
--- a/tools/perf/util/probe-finder.c
+++ b/tools/perf/util/probe-finder.c
@@ -1126,9 +1126,17 @@ static int find_probe_point_lazy(Dwarf_Die *sp_die, struct probe_finder *pf)
 	int ret = 0;
 
 	if (list_empty(&pf->lcache)) {
+		const char *comp_dir = cu_get_comp_dir(&pf->cu_die);
+		char *fullpath;
+
+		ret = get_real_path(pf->fname, comp_dir, &fullpath);
+		if (ret < 0)
+			return ret;
+
 		/* Matching lazy line pattern */
-		ret = find_lazy_match_lines(&pf->lcache, pf->fname,
+		ret = find_lazy_match_lines(&pf->lcache, fullpath,
 					    pf->pev->point.lazy_line);
+		free(fullpath);
 		if (ret == 0) {
 			pr_debug("No matched lines found in %s.\n", pf->fname);
 			return 0;
diff --git a/tools/perf/util/probe-finder.h b/tools/perf/util/probe-finder.h
index bba69d4..665938a 100644
--- a/tools/perf/util/probe-finder.h
+++ b/tools/perf/util/probe-finder.h
@@ -15,6 +15,9 @@ static inline int is_c_varname(const char *name)
 	return isalpha(name[0]) || name[0] == '_';
 }
 
+extern int get_real_path(const char *raw_path, const char *comp_dir,
+		  char **new_path);
+
 #ifdef DWARF_SUPPORT
 /* Find probe_trace_events specified by perf_probe_event from debuginfo */
 extern int find_probe_trace_events(int fd, struct perf_probe_event *pev,


-- 
		Franck

Re: perf-probe: issue with latest fedora kernel

[Franck Bui-Huu]

Masami,

It seems you forget this one ....

Thanks

Franck Bui-Huu <vagabon.xyz@gmail.com> writes:

> Hello Masami,
>
> Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> writes:
>
>> (2010/12/13 23:21), Arnaldo Carvalho de Melo wrote:
>>
>>> There are two problems here:
>>> 
>>> on the 'report' header, as we didn't collect "cycles", but
>>> "probe:icmp_rcv" events, that is another fix that needs to be backported to
>>> fedora, but a minor one.
>>> 
>>> Also 'perf probe -L' should look use
>>> /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 as a prefix to look for
>>> sources.
>>> 
>>> That can be found using:
>>> 
>>> [root@felicio linux-2.6.35.x86_64]# readelf -wi /usr/lib/debug/lib/modules/2.6.35.9-64.fc14.x86_64/vmlinux | grep DW_AT_comp_dir | head -5
>>>     <2e>   DW_AT_comp_dir    : /usr/src/debug////////kernel-2.6.35.fc14/linux-2.6.35.x86_64	
>>>     <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
>>>     <75e8>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
>>>     <900e>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
>>>     <16670>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64	
>>> [root@felicio linux-2.6.35.x86_64]# 
>>> 
>>>  <0><8b>: Abbrev Number: 1 (DW_TAG_compile_unit)
>>>     <8c>   DW_AT_producer    : (indirect string, offset: 0x22f4): GNU C 4.5.1 20100924 (Red Hat 4.5.1-4)        
>>>     <90>   DW_AT_language    : 1        (ANSI C)
>>>     <91>   DW_AT_name        : (indirect string, offset: 0x37a): arch/x86/kernel/head64.c       
>>>     <95>   DW_AT_comp_dir    : (indirect string, offset: 0x24d0): /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64 
>>>     <99>   DW_AT_low_pc      : 0x0      
>>>     <a1>   DW_AT_entry_pc    : 0x0      
>>>     <a9>   DW_AT_ranges      : 0x340    
>>>     <ad>   DW_AT_stmt_list   : 0x114
>>> 
>>> I.e. using the DW_AT_comp_dir attribute in the DW_TAG_compile_unit DWARF tags,
>>> Masami?
>>
>> It already supports DW_AT_comp_dir on upstream kernel,
>> see find_line_range() in utils/probe-finder.c ;-)
>>
>
> It looks like adding probe with lazy matching pattern does not:
>
>    $ cd /dev
>    $ perf probe -n -a "icmp.c;sk=*"
>    Failed to open net/ipv4/icmp.c: Operation not permitted
>      Error: Failed to add events. (-2)
>    $ cd /usr/src/debug/kernel-2.6.35.fc14/linux-2.6.35.x86_64/
>    $ perf probe -n -a "icmp.c;sk=*"
>    Add new events:
>      probe:icmp_push_reply (on @cmp.c)
>      probe:icmp_reply     (on @cmp.c)
>      probe:icmp_reply_1   (on @cmp.c)
>      probe:icmp_send      (on @cmp.c)
>      probe:icmp_send_1    (on @cmp.c)
>      probe:icmp_error     (on @cmp.c)
>      probe:icmp_error_1   (on @cmp.c)
>      probe:icmp_error_2   (on @cmp.c)
>      probe:icmp_error_3   (on @cmp.c)
>    
>    You can now use it on all perf tools, such as:
>    
>    	perf record -e probe:icmp_error_3 -aR sleep 1
>
> Below is a quick and dirty fix (sorry, I don't have any time to make it
> better for now), also note that the note of the file (icmp.c) reported
> is screwed...
>
> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> index 10ad1ad..53c2f97 100644
> --- a/tools/perf/util/probe-event.c
> +++ b/tools/perf/util/probe-event.c
> @@ -241,7 +241,7 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
>   * a newly allocated path on success.
>   * Return 0 if file was found and readable, -errno otherwise.
>   */
> -static int get_real_path(const char *raw_path, const char *comp_dir,
> +int get_real_path(const char *raw_path, const char *comp_dir,
>  			 char **new_path)
>  {
>  	const char *prefix = symbol_conf.source_prefix;
> diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
> index 3991d73..88829ef 100644
> --- a/tools/perf/util/probe-finder.c
> +++ b/tools/perf/util/probe-finder.c
> @@ -1126,9 +1126,17 @@ static int find_probe_point_lazy(Dwarf_Die *sp_die, struct probe_finder *pf)
>  	int ret = 0;
>  
>  	if (list_empty(&pf->lcache)) {
> +		const char *comp_dir = cu_get_comp_dir(&pf->cu_die);
> +		char *fullpath;
> +
> +		ret = get_real_path(pf->fname, comp_dir, &fullpath);
> +		if (ret < 0)
> +			return ret;
> +
>  		/* Matching lazy line pattern */
> -		ret = find_lazy_match_lines(&pf->lcache, pf->fname,
> +		ret = find_lazy_match_lines(&pf->lcache, fullpath,
>  					    pf->pev->point.lazy_line);
> +		free(fullpath);
>  		if (ret == 0) {
>  			pr_debug("No matched lines found in %s.\n", pf->fname);
>  			return 0;
> diff --git a/tools/perf/util/probe-finder.h b/tools/perf/util/probe-finder.h
> index bba69d4..665938a 100644
> --- a/tools/perf/util/probe-finder.h
> +++ b/tools/perf/util/probe-finder.h
> @@ -15,6 +15,9 @@ static inline int is_c_varname(const char *name)
>  	return isalpha(name[0]) || name[0] == '_';
>  }
>  
> +extern int get_real_path(const char *raw_path, const char *comp_dir,
> +		  char **new_path);
> +
>  #ifdef DWARF_SUPPORT
>  /* Find probe_trace_events specified by perf_probe_event from debuginfo */
>  extern int find_probe_trace_events(int fd, struct perf_probe_event *pev,

-- 
		Franck