* problems with perf probe and system libraries
@ 2014-12-30 4:21 David Ahern
2014-12-30 22:18 ` Masami Hiramatsu
2014-12-31 6:27 ` [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols Masami Hiramatsu
0 siblings, 2 replies; 6+ messages in thread
From: David Ahern @ 2014-12-30 4:21 UTC (permalink / raw)
To: Masami Hiramatsu; +Cc: Arnaldo Carvalho de Melo, LKML
Hi Masami:
I have been looking at perf-probe again and having a number of problems
with top of tree.
Here's the first one I have isolated:
$ perf probe -x /lib64/libc-2.18.so -a 'malloc size=%di'
Probe point 'malloc' not found.
Error: Failed to add events.
$ perf probe -x /lib64/libc-2.18.so -F | grep malloc
malloc
malloc@plt
malloc_atfork
malloc_check
malloc_consolidate
malloc_hook_ini
malloc_info
malloc_printerr
mallochook
ptmalloc_init
ptmalloc_init.part.8
ptmalloc_lock_all
ptmalloc_unlock_all
ptmalloc_unlock_all2
tr_mallochook
$ perf probe -x /lib64/libc-2.18.so -a 'malloc=malloc size=%di'
Probe point 'malloc' not found.
Error: Failed to add events.
A year ago (v3.12) this worked fine so I did a git bisect which points to:
commit fb7345bbf7fad9bf72ef63a19c707970b9685812
Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Date: Thu Dec 26 05:41:53 2013 +0000
perf probe: Support basic dwarf-based operations on uprobe events
I have tried top of tree on Fedora 16, 18 and 20 with a variety of
kernels - and a variety of results. Reverting to
8a613d40e389b723fd5889ac8d4033ed4030be31 which is the commit before this
one and it works again.
David
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: problems with perf probe and system libraries 2014-12-30 4:21 problems with perf probe and system libraries David Ahern @ 2014-12-30 22:18 ` Masami Hiramatsu 2014-12-31 6:27 ` [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols Masami Hiramatsu 1 sibling, 0 replies; 6+ messages in thread From: Masami Hiramatsu @ 2014-12-30 22:18 UTC (permalink / raw) To: David Ahern; +Cc: Arnaldo Carvalho de Melo, LKML Hi David, (2014/12/30 13:21), David Ahern wrote: > Hi Masami: > > I have been looking at perf-probe again and having a number of problems > with top of tree. > > Here's the first one I have isolated: > > $ perf probe -x /lib64/libc-2.18.so -a 'malloc size=%di' > Probe point 'malloc' not found. > Error: Failed to add events. It seems to fail when failed to open debuginfo. I confirmed it on ubuntu 14.04 too. I'll try to fix that. Thank you for reporting! > > $ perf probe -x /lib64/libc-2.18.so -F | grep malloc > malloc > malloc@plt > malloc_atfork > malloc_check > malloc_consolidate > malloc_hook_ini > malloc_info > malloc_printerr > mallochook > ptmalloc_init > ptmalloc_init.part.8 > ptmalloc_lock_all > ptmalloc_unlock_all > ptmalloc_unlock_all2 > tr_mallochook > > $ perf probe -x /lib64/libc-2.18.so -a 'malloc=malloc size=%di' > Probe point 'malloc' not found. > Error: Failed to add events. > > A year ago (v3.12) this worked fine so I did a git bisect which points to: > > commit fb7345bbf7fad9bf72ef63a19c707970b9685812 > Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> > Date: Thu Dec 26 05:41:53 2013 +0000 > > perf probe: Support basic dwarf-based operations on uprobe events > > I have tried top of tree on Fedora 16, 18 and 20 with a variety of > kernels - and a variety of results. Reverting to > 8a613d40e389b723fd5889ac8d4033ed4030be31 which is the commit before this > one and it works again. > > David > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > -- Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu.pt@hitachi.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols 2014-12-30 4:21 problems with perf probe and system libraries David Ahern 2014-12-30 22:18 ` Masami Hiramatsu @ 2014-12-31 6:27 ` Masami Hiramatsu 2014-12-31 8:02 ` Namhyung Kim 2015-01-08 9:52 ` [tip:perf/urgent] perf probe: " tip-bot for Masami Hiramatsu 1 sibling, 2 replies; 6+ messages in thread From: Masami Hiramatsu @ 2014-12-31 6:27 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, David Ahern Cc: namhyung, yrl.pp-manager.tt, Ingo Molnar, Linux Kernel Mailing List Fix to fall back to find a probe point in symbols if perf fails to find it in debuginfo. This can happen when the target function is an alias of another function. Such alias doesn't have an entry in debuginfo but in symbols. David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355 I ensured the problem and deeper investigation discovers it. ----- eu-readelf --debug-dump=info /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep \"malloc\" -A6 name (strp) "malloc" decl_file (data1) 25 decl_line (data2) 466 prototyped (flag_present) type (ref4) [ 81b5] declaration (flag_present) [ 8f58] formal_parameter -- name (strp) "malloc" decl_file (data1) 23 decl_line (data2) 466 prototyped (flag_present) type (ref4) [ 9f4a] declaration (flag_present) sibling (ref4) [ bb29] ... ----- All these entires have no instances (all of them are declarations) This is why the perf probe failed to find it in debuginfo. However, there are some malloc instances in symbols. ----- eu-readelf --symbols /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep malloc$ 1181: 0000000000080700 5332 FUNC LOCAL DEFAULT 12 _int_malloc 4537: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __GI___libc_malloc 5545: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __malloc 6063: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 malloc 7302: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 __libc_malloc ----- As you an see, malloc and __libc_malloc have same address, and actually __libc_malloc has an entry in debuginfo. So you can set up a probe on __libc_malloc. To fix this problem shortly, perf probe simply falls back to find probe point(malloc) in symbols if it is not found in debuginfo. Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Reported-by: David Ahern <dsahern@gmail.com> --- tools/perf/util/probe-event.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c index 28eb141..7f9b863 100644 --- a/tools/perf/util/probe-event.c +++ b/tools/perf/util/probe-event.c @@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev, } if (ntevs == 0) { /* No error but failed to find probe point. */ - pr_warning("Probe point '%s' not found.\n", + pr_warning("Probe point '%s' not found in debuginfo.\n", synthesize_perf_probe_point(&pev->point)); - return -ENOENT; + if (need_dwarf) + return -ENOENT; + return 0; } /* Error path : ntevs < 0 */ pr_debug("An error occurred in debuginfo analysis (%d).\n", ntevs); ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols 2014-12-31 6:27 ` [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols Masami Hiramatsu @ 2014-12-31 8:02 ` Namhyung Kim 2015-01-04 4:41 ` Masami Hiramatsu 2015-01-08 9:52 ` [tip:perf/urgent] perf probe: " tip-bot for Masami Hiramatsu 1 sibling, 1 reply; 6+ messages in thread From: Namhyung Kim @ 2014-12-31 8:02 UTC (permalink / raw) To: Masami Hiramatsu Cc: Arnaldo Carvalho de Melo, David Ahern, yrl.pp-manager.tt, Ingo Molnar, Linux Kernel Mailing List Hi Masami, On Wed, Dec 31, 2014 at 03:27:47PM +0900, Masami Hiramatsu wrote: > Fix to fall back to find a probe point in symbols if perf fails > to find it in debuginfo. > > This can happen when the target function is an alias of another > function. Such alias doesn't have an entry in debuginfo but in > symbols. > > David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355 > > I ensured the problem and deeper investigation discovers it. > ----- > eu-readelf --debug-dump=info /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep \"malloc\" -A6 > name (strp) "malloc" > decl_file (data1) 25 > decl_line (data2) 466 > prototyped (flag_present) > type (ref4) [ 81b5] > declaration (flag_present) > [ 8f58] formal_parameter > -- > name (strp) "malloc" > decl_file (data1) 23 > decl_line (data2) 466 > prototyped (flag_present) > type (ref4) [ 9f4a] > declaration (flag_present) > sibling (ref4) [ bb29] > ... > ----- > All these entires have no instances (all of them are declarations) > This is why the perf probe failed to find it in debuginfo. > > However, there are some malloc instances in symbols. > ----- > eu-readelf --symbols /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep malloc$ > 1181: 0000000000080700 5332 FUNC LOCAL DEFAULT 12 _int_malloc > 4537: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __GI___libc_malloc > 5545: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __malloc > 6063: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 malloc > 7302: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 __libc_malloc > ----- > As you an see, malloc and __libc_malloc have same address, and actually > __libc_malloc has an entry in debuginfo. So you can set up a probe on > __libc_malloc. > > To fix this problem shortly, perf probe simply falls back to find probe > point(malloc) in symbols if it is not found in debuginfo. Idealy, it'd be great if it can go to the definition. I think perf only keeps a single symbol at a same address so if there're multiple aliases at a specific address it might be missed even with this patch. But that will be a rare case, so Acked-by: Namhyung Kim <namhyung@kernel.org> Thanks, Namhyung > > Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> > Reported-by: David Ahern <dsahern@gmail.com> > --- > tools/perf/util/probe-event.c | 6 ++++-- > 1 file changed, 4 insertions(+), 2 deletions(-) > > diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c > index 28eb141..7f9b863 100644 > --- a/tools/perf/util/probe-event.c > +++ b/tools/perf/util/probe-event.c > @@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev, > } > > if (ntevs == 0) { /* No error but failed to find probe point. */ > - pr_warning("Probe point '%s' not found.\n", > + pr_warning("Probe point '%s' not found in debuginfo.\n", > synthesize_perf_probe_point(&pev->point)); > - return -ENOENT; > + if (need_dwarf) > + return -ENOENT; > + return 0; > } > /* Error path : ntevs < 0 */ > pr_debug("An error occurred in debuginfo analysis (%d).\n", ntevs); > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Re: [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols 2014-12-31 8:02 ` Namhyung Kim @ 2015-01-04 4:41 ` Masami Hiramatsu 0 siblings, 0 replies; 6+ messages in thread From: Masami Hiramatsu @ 2015-01-04 4:41 UTC (permalink / raw) To: Namhyung Kim Cc: Arnaldo Carvalho de Melo, David Ahern, yrl.pp-manager.tt, Ingo Molnar, Linux Kernel Mailing List (2014/12/31 17:02), Namhyung Kim wrote: > Hi Masami, > > On Wed, Dec 31, 2014 at 03:27:47PM +0900, Masami Hiramatsu wrote: >> Fix to fall back to find a probe point in symbols if perf fails >> to find it in debuginfo. >> >> This can happen when the target function is an alias of another >> function. Such alias doesn't have an entry in debuginfo but in >> symbols. >> >> David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355 >> >> I ensured the problem and deeper investigation discovers it. >> ----- >> eu-readelf --debug-dump=info /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep \"malloc\" -A6 >> name (strp) "malloc" >> decl_file (data1) 25 >> decl_line (data2) 466 >> prototyped (flag_present) >> type (ref4) [ 81b5] >> declaration (flag_present) >> [ 8f58] formal_parameter >> -- >> name (strp) "malloc" >> decl_file (data1) 23 >> decl_line (data2) 466 >> prototyped (flag_present) >> type (ref4) [ 9f4a] >> declaration (flag_present) >> sibling (ref4) [ bb29] >> ... >> ----- >> All these entires have no instances (all of them are declarations) >> This is why the perf probe failed to find it in debuginfo. >> >> However, there are some malloc instances in symbols. >> ----- >> eu-readelf --symbols /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep malloc$ >> 1181: 0000000000080700 5332 FUNC LOCAL DEFAULT 12 _int_malloc >> 4537: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __GI___libc_malloc >> 5545: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __malloc >> 6063: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 malloc >> 7302: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 __libc_malloc >> ----- >> As you an see, malloc and __libc_malloc have same address, and actually >> __libc_malloc has an entry in debuginfo. So you can set up a probe on >> __libc_malloc. >> >> To fix this problem shortly, perf probe simply falls back to find probe >> point(malloc) in symbols if it is not found in debuginfo. > > Idealy, it'd be great if it can go to the definition. I think perf > only keeps a single symbol at a same address so if there're multiple > aliases at a specific address it might be missed even with this patch. Hmm, anyway, that should be fixed by improving perf's map, instead of introducing new symbol finder. > But that will be a rare case, so > > Acked-by: Namhyung Kim <namhyung@kernel.org> Thank you! > > Thanks, > Namhyung > > >> >> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> >> Reported-by: David Ahern <dsahern@gmail.com> >> --- >> tools/perf/util/probe-event.c | 6 ++++-- >> 1 file changed, 4 insertions(+), 2 deletions(-) >> >> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c >> index 28eb141..7f9b863 100644 >> --- a/tools/perf/util/probe-event.c >> +++ b/tools/perf/util/probe-event.c >> @@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev, >> } >> >> if (ntevs == 0) { /* No error but failed to find probe point. */ >> - pr_warning("Probe point '%s' not found.\n", >> + pr_warning("Probe point '%s' not found in debuginfo.\n", >> synthesize_perf_probe_point(&pev->point)); >> - return -ENOENT; >> + if (need_dwarf) >> + return -ENOENT; >> + return 0; >> } >> /* Error path : ntevs < 0 */ >> pr_debug("An error occurred in debuginfo analysis (%d).\n", ntevs); >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> Please read the FAQ at http://www.tux.org/lkml/ > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > -- Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu.pt@hitachi.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* [tip:perf/urgent] perf probe: Fix to fall back to find probe point in symbols 2014-12-31 6:27 ` [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols Masami Hiramatsu 2014-12-31 8:02 ` Namhyung Kim @ 2015-01-08 9:52 ` tip-bot for Masami Hiramatsu 1 sibling, 0 replies; 6+ messages in thread From: tip-bot for Masami Hiramatsu @ 2015-01-08 9:52 UTC (permalink / raw) To: linux-tip-commits Cc: masami.hiramatsu.pt, hpa, acme, mingo, namhyung, dsahern, tglx, linux-kernel Commit-ID: 906451b98b6774042b707a5dcebb6a93dbd14c85 Gitweb: http://git.kernel.org/tip/906451b98b6774042b707a5dcebb6a93dbd14c85 Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> AuthorDate: Wed, 31 Dec 2014 15:27:47 +0900 Committer: Arnaldo Carvalho de Melo <acme@redhat.com> CommitDate: Fri, 2 Jan 2015 12:42:17 -0300 perf probe: Fix to fall back to find probe point in symbols Fix to fall back to find a probe point in symbols if perf fails to find it in debuginfo. This can happen when the target function is an alias of another function. Such alias doesn't have an entry in debuginfo but in symbols. David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355 I ensured the problem and deeper investigation discovers it. ----- eu-readelf --debug-dump=info /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep \"malloc\" -A6 name (strp) "malloc" decl_file (data1) 25 decl_line (data2) 466 prototyped (flag_present) type (ref4) [ 81b5] declaration (flag_present) [ 8f58] formal_parameter -- name (strp) "malloc" decl_file (data1) 23 decl_line (data2) 466 prototyped (flag_present) type (ref4) [ 9f4a] declaration (flag_present) sibling (ref4) [ bb29] ... ----- All these entires have no instances (all of them are declarations) This is why the perf probe failed to find it in debuginfo. However, there are some malloc instances in symbols. ----- eu-readelf --symbols /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep malloc$ 1181: 0000000000080700 5332 FUNC LOCAL DEFAULT 12 _int_malloc 4537: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __GI___libc_malloc 5545: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __malloc 6063: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 malloc 7302: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 __libc_malloc ----- As you an see, malloc and __libc_malloc have same address, and actually __libc_malloc has an entry in debuginfo. So you can set up a probe on __libc_malloc. To fix this problem shortly, perf probe simply falls back to find probe point(malloc) in symbols if it is not found in debuginfo. Reported-by: David Ahern <dsahern@gmail.com> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Acked-by: Namhyung Kim <namhyung@kernel.org> Cc: David Ahern <dsahern@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Cc: yrl.pp-manager.tt@hitachi.com Link: http://lkml.kernel.org/r/20141231062747.2087.80961.stgit@localhost.localdomain Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> --- tools/perf/util/probe-event.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c index 28eb141..7f9b863 100644 --- a/tools/perf/util/probe-event.c +++ b/tools/perf/util/probe-event.c @@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev, } if (ntevs == 0) { /* No error but failed to find probe point. */ - pr_warning("Probe point '%s' not found.\n", + pr_warning("Probe point '%s' not found in debuginfo.\n", synthesize_perf_probe_point(&pev->point)); - return -ENOENT; + if (need_dwarf) + return -ENOENT; + return 0; } /* Error path : ntevs < 0 */ pr_debug("An error occurred in debuginfo analysis (%d).\n", ntevs); ^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2015-01-08 9:52 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-12-30 4:21 problems with perf probe and system libraries David Ahern 2014-12-30 22:18 ` Masami Hiramatsu 2014-12-31 6:27 ` [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols Masami Hiramatsu 2014-12-31 8:02 ` Namhyung Kim 2015-01-04 4:41 ` Masami Hiramatsu 2015-01-08 9:52 ` [tip:perf/urgent] perf probe: " tip-bot for Masami Hiramatsu
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox