* uprobes enable condition (user space) - sysfs interface
@ 2017-11-29 18:34 bhargavb
2017-12-01 16:40 ` Arnaldo Carvalho de Melo
0 siblings, 1 reply; 11+ messages in thread
From: bhargavb @ 2017-11-29 18:34 UTC (permalink / raw)
To: linux-rt-users, linux-perf-users
Hello,
I am trying to understand user space profiling using linux inbuilt
mechanism. (Trying with perf also but could not get complete clear
documentation and could not proceed, so retained linux-perf-users in
to list as the underlying interface used is same for both).
I observed below behaviour when I tried as per directions using URL:
https://opensource.com/article/17/7/dynamic-tracing-linux-user-and-kernel-space#comment-136366
:
echo 'p:<func_entry> ./test:0x420' >
/sys/kernel/debug/tracing/uprobe_events (for creating
uprobe)
echo 1 > /sys/kernel/debug/tracing/events/enable
(enabling uproble)
There is another enable parameter:
/sys/kernel/debug/tracing/events/uprobes/enable
I understand unless enabled both events/enable and
events/uprobes/enable, the /sys/kernel/debug/tracing/trace is not
complete. I do not understand the relevance of
.....events/uprobes/enable as I could not get any trace data in trace
log for 'func_entry' the user space function being added for tracing.
Irrespective of the probe function added, the trace log shows complete
system trace log during the period of execution of the user
application being traced.
Can this be clearly mentioned as to when to use one of them, role of
both and and also, I could not see uprobe function in particular at
all but generic trace in /sys/kernel/debug/tracing/trace. Kindly guide
me in understanding uprobe enable logic in user space in detail.
Regards,
Bhargav
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: uprobes enable condition (user space) - sysfs interface 2017-11-29 18:34 uprobes enable condition (user space) - sysfs interface bhargavb @ 2017-12-01 16:40 ` Arnaldo Carvalho de Melo 2017-12-03 17:14 ` bhargavb 2017-12-04 12:36 ` Masami Hiramatsu 0 siblings, 2 replies; 11+ messages in thread From: Arnaldo Carvalho de Melo @ 2017-12-01 16:40 UTC (permalink / raw) To: bhargavb; +Cc: linux-rt-users, linux-perf-users, Masami Hiramatsu Adding Masami to the CC list, to see if he can come up with a fix for the 'perf probe' failure below, improving the message or ignoring versioned symbols (with a warning perhaps). Em Thu, Nov 30, 2017 at 12:04:23AM +0530, bhargavb escreveu: > Hello, > > I am trying to understand user space profiling using linux inbuilt > mechanism. (Trying with perf also but could not get complete clear > documentation and could not proceed, so retained linux-perf-users in > to list as the underlying interface used is same for both). So yo uwant to trace function entry/exit in some userspace library? You can try using the tracefs as below or leave it to perf to do that, first you set up the probes, say for some glibc functions: [root@jouet ~]# perf probe -F -x /lib64/libc-2.25.so | grep ^malloc malloc malloc_check malloc_consolidate.part.1 malloc_get_state@GLIBC_2.2.5 malloc_hook_ini malloc_info malloc_printerr malloc_set_state@GLIBC_2.2.5 malloc_stats malloc_trim malloc_usable_size mallochook [root@jouet ~]# [root@jouet ~]# perf probe -x /lib64/libc-2.25.so malloc* Failed to write event: Invalid argument Error: Failed to add events. [root@jouet ~]# For now ignore that error, its a bug I just found, what matters is that it _has_ put in place probes for most of those functions, as you can see by looking at: [root@jouet ~]# perf probe -l probe_libc:malloc (on __malloc in /usr/lib64/libc-2.25.so) probe_libc:malloc_check (on malloc_check in /usr/lib64/libc-2.25.so) probe_libc:malloc_consolidate (on malloc_consolidate.part.1 in /usr/lib64/libc-2.25.so) probe_libc:malloc_hook_ini (on malloc_hook_ini in /usr/lib64/libc-2.25.so) probe_libc:malloc_info (on __malloc_info in /usr/lib64/libc-2.25.so) probe_libc:malloc_printerr (on malloc_printerr in /usr/lib64/libc-2.25.so) probe_libc:malloc_stats (on __malloc_stats in /usr/lib64/libc-2.25.so) probe_libc:malloc_trim (on __malloc_trim in /usr/lib64/libc-2.25.so) probe_libc:malloc_usable_size (on __malloc_usable_size in /usr/lib64/libc-2.25.so) probe_libc:mallochook (on mallochook in /usr/lib64/libc-2.25.so) [root@jouet ~]# Masami, this one also doesn't work: [root@jouet ~]# perf probe -x /lib64/libc-2.25.so 'malloc*%return' Error: event "malloc_printerr" already exists. <SNIP> doing it a bit differently: [root@jouet ~]# perf probe -F -x /lib64/libc-2.25.so | egrep ^malloc[^@]+$ | while read function ; do perf probe -x /lib64/libc-2.25.so "${function}_return=$function%return" ; done Added new event: probe_libc:malloc_check_return (on malloc_check%return in /usr/lib64/libc-2.25.so) You can now use it in all perf tools, such as: perf record -e probe_libc:malloc_check_return -aR sleep 1 <SNIP> ----------- This now makes us have: [root@jouet ~]# perf probe -l probe:check_stack_object (on check_stack_object%return@acme/git/linux/mm/usercopy.c with ret) probe_libc:malloc (on __malloc in /usr/lib64/libc-2.25.so) probe_libc:malloc_check (on malloc_check in /usr/lib64/libc-2.25.so) probe_libc:malloc_consolidate (on malloc_consolidate.part.1 in /usr/lib64/libc-2.25.so) probe_libc:malloc_hook_ini (on malloc_hook_ini in /usr/lib64/libc-2.25.so) probe_libc:malloc_info (on __malloc_info in /usr/lib64/libc-2.25.so) probe_libc:malloc_printerr (on malloc_printerr in /usr/lib64/libc-2.25.so) probe_libc:malloc_stats (on __malloc_stats in /usr/lib64/libc-2.25.so) probe_libc:malloc_trim (on __malloc_trim in /usr/lib64/libc-2.25.so) probe_libc:malloc_usable_size (on __malloc_usable_size in /usr/lib64/libc-2.25.so) probe_libc:mallochook (on mallochook in /usr/lib64/libc-2.25.so) probe_libc:malloc_check_return (on malloc_check%return in /usr/lib64/libc-2.25.so) probe_libc:malloc_hook_ini_return (on malloc_hook_ini%return in /usr/lib64/libc-2.25.so) probe_libc:malloc_info_return (on __malloc_info%return in /usr/lib64/libc-2.25.so) probe_libc:malloc_printerr_return (on malloc_printerr%return in /usr/lib64/libc-2.25.so) probe_libc:malloc_stats_return (on __malloc_stats%return in /usr/lib64/libc-2.25.so) probe_libc:malloc_trim_return (on __malloc_trim%return in /usr/lib64/libc-2.25.so) probe_libc:malloc_usable_size_return (on __malloc_usable_size%return in /usr/lib64/libc-2.25.so) probe_libc:mallochook_return (on mallochook%return in /usr/lib64/libc-2.25.so) probe_libc:malloc_return (on __malloc%return in /usr/lib64/libc-2.25.so) [root@jouet ~]# There were some usability snags that prevented this from being put in place, so do it manually: [root@jouet ~]# perf probe -x /lib64/libc-2.25.so 'malloc_consolidate_return=malloc_consolidate.part.1%return' Added new event: probe_libc:malloc_consolidate_return (on malloc_consolidate.part.1%return in /usr/lib64/libc-2.25.so) You can now use it in all perf tools, such as: perf record -e probe_libc:malloc_consolidate_return -aR sleep 1 [root@jouet ~]# Ok, now we can trace this syswide using: [root@jouet ~]# perf trace --no-syscalls -e probe_libc:* <BIG SNIP> 339.557 probe_libc:malloc:(7fda55c22970)) 339.562 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) 339.567 probe_libc:malloc_consolidate:(7fda55c1b7b0)) 339.570 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) 339.596 probe_libc:malloc:(7fda55c22970)) 339.600 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) 339.622 probe_libc:malloc:(7fda55c22970)) 339.626 probe_libc:malloc_return:(7fda55c22970 <- 7fda56ba9a39)) 339.631 probe_libc:malloc:(7fda55c22970)) 339.634 probe_libc:malloc_return:(7fda55c22970 <- 7fda578f1713)) 339.637 probe_libc:malloc:(7fda55c22970)) 339.641 probe_libc:malloc_return:(7fda55c22970 <- 7fda578f14ae)) 339.657 probe_libc:malloc_consolidate:(7fda55c1b7b0)) 339.661 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1dbd6)) 339.664 probe_libc:malloc:(7fda55c22970)) 339.667 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) 339.670 probe_libc:malloc:(7fda55c22970)) 339.673 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) 339.678 probe_libc:malloc_consolidate:(7fda55c1b7b0)) 339.681 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) 339.685 probe_libc:malloc_consolidate:(7fda55c1b7b0)) 339.688 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) 339.693 probe_libc:malloc_consolidate:(7fda55c1b7b0)) 339.696 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) 339.778 probe_libc:malloc:(7fda55c22970)) 339.783 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) 339.788 probe_libc:malloc_consolidate:(7fda55c1b7b0)) ^C(7fda55c22970 <- 7fda578f14ae)) [root@jouet ~]# It works as well with 'perf record' + 'perf script'. Other features from perf can be added to the mix, like callchains, etc. But see below for doing it via tracefs > I observed below behaviour when I tried as per directions using URL: > https://opensource.com/article/17/7/dynamic-tracing-linux-user-and-kernel-space#comment-136366 > : > > > echo 'p:<func_entry> ./test:0x420' > > /sys/kernel/debug/tracing/uprobe_events (for creating > uprobe) > echo 1 > /sys/kernel/debug/tracing/events/enable > (enabling uproble) > There is another enable parameter: > /sys/kernel/debug/tracing/events/uprobes/enable > > I understand unless enabled both events/enable and > events/uprobes/enable, the /sys/kernel/debug/tracing/trace is not > complete. I do not understand the relevance of > .....events/uprobes/enable as I could not get any trace data in trace > log for 'func_entry' the user space function being added for tracing. > Irrespective of the probe function added, the trace log shows complete > system trace log during the period of execution of the user > application being traced. Is this what you want: [root@jouet ~]# cd /sys/kernel/debug/tracing/ [root@jouet tracing]# cat available_tracers hwlat blk mmiotrace function_graph wakeup_dl wakeup_rt wakeup function nop [root@jouet tracing]# echo nop > current_tracer [root@jouet tracing]# echo 1 > events/probe_libc/malloc/enable [root@jouet tracing]# echo 1 > tracing_on [root@jouet tracing]# tail trace tail-1581 [002] d... 118484.507167: malloc: (0x7fdb32d67970) tail-1581 [002] d... 118484.507172: malloc: (0x7fdb32d67970) tail-1581 [002] d... 118484.507189: malloc: (0x7fdb32d67970) tail-1581 [002] d... 118484.507193: malloc: (0x7fdb32d67970) tail-1581 [002] d... 118484.507198: malloc: (0x7fdb32d67970) tail-1581 [002] d... 118484.507203: malloc: (0x7fdb32d67970) tail-1581 [002] d... 118484.507208: malloc: (0x7fdb32d67970) tail-1581 [002] d... 118484.507214: malloc: (0x7fdb32d67970) tail-1581 [002] d... 118484.507224: malloc: (0x7fdb32d67970) tail-1581 [002] d... 118484.507237: malloc: (0x7fdb32d67970) [root@jouet tracing]# ? > Can this be clearly mentioned as to when to use one of them, role of > both and and also, I could not see uprobe function in particular at > all but generic trace in /sys/kernel/debug/tracing/trace. Kindly guide > me in understanding uprobe enable logic in user space in detail. > > Regards, > Bhargav > -- > To unsubscribe from this list: send the line "unsubscribe linux-perf-users" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: uprobes enable condition (user space) - sysfs interface 2017-12-01 16:40 ` Arnaldo Carvalho de Melo @ 2017-12-03 17:14 ` bhargavb 2017-12-04 6:03 ` Namhyung Kim 2017-12-04 12:36 ` Masami Hiramatsu 1 sibling, 1 reply; 11+ messages in thread From: bhargavb @ 2017-12-03 17:14 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: linux-rt-users, linux-perf-users, Masami Hiramatsu Dear Arnaldo, Exactly, the last one was what I was looking for. Thank you very much... That apart I am also exploring various profiling options that can give me a full control of monitoring running processes (user space profiling ). sysfs seems powerful and very useful, though I got to know lately (and not much mentioned, I think as tools like perf etc). I was confused when u said tracefs in your response, I understand it is same as sysfs. Please correct me if I am wrong. Thanks and regards, bhargav On Fri, Dec 1, 2017 at 10:10 PM, Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote: > Adding Masami to the CC list, to see if he can come up with a fix for > the 'perf probe' failure below, improving the message or ignoring > versioned symbols (with a warning perhaps). > > Em Thu, Nov 30, 2017 at 12:04:23AM +0530, bhargavb escreveu: >> Hello, >> >> I am trying to understand user space profiling using linux inbuilt >> mechanism. (Trying with perf also but could not get complete clear >> documentation and could not proceed, so retained linux-perf-users in >> to list as the underlying interface used is same for both). > > So yo uwant to trace function entry/exit in some userspace library? You > can try using the tracefs as below or leave it to perf to do that, first > you set up the probes, say for some glibc functions: > > [root@jouet ~]# perf probe -F -x /lib64/libc-2.25.so | grep ^malloc > malloc > malloc_check > malloc_consolidate.part.1 > malloc_get_state@GLIBC_2.2.5 > malloc_hook_ini > malloc_info > malloc_printerr > malloc_set_state@GLIBC_2.2.5 > malloc_stats > malloc_trim > malloc_usable_size > mallochook > [root@jouet ~]# > > [root@jouet ~]# perf probe -x /lib64/libc-2.25.so malloc* > Failed to write event: Invalid argument > Error: Failed to add events. > [root@jouet ~]# > > For now ignore that error, its a bug I just found, what matters is that > it _has_ put in place probes for most of those functions, as you can see > by looking at: > > [root@jouet ~]# perf probe -l > probe_libc:malloc (on __malloc in /usr/lib64/libc-2.25.so) > probe_libc:malloc_check (on malloc_check in /usr/lib64/libc-2.25.so) > probe_libc:malloc_consolidate (on malloc_consolidate.part.1 in /usr/lib64/libc-2.25.so) > probe_libc:malloc_hook_ini (on malloc_hook_ini in /usr/lib64/libc-2.25.so) > probe_libc:malloc_info (on __malloc_info in /usr/lib64/libc-2.25.so) > probe_libc:malloc_printerr (on malloc_printerr in /usr/lib64/libc-2.25.so) > probe_libc:malloc_stats (on __malloc_stats in /usr/lib64/libc-2.25.so) > probe_libc:malloc_trim (on __malloc_trim in /usr/lib64/libc-2.25.so) > probe_libc:malloc_usable_size (on __malloc_usable_size in /usr/lib64/libc-2.25.so) > probe_libc:mallochook (on mallochook in /usr/lib64/libc-2.25.so) > [root@jouet ~]# > > Masami, this one also doesn't work: > > [root@jouet ~]# perf probe -x /lib64/libc-2.25.so 'malloc*%return' > Error: event "malloc_printerr" already exists. > <SNIP> > > doing it a bit differently: > > [root@jouet ~]# perf probe -F -x /lib64/libc-2.25.so | egrep ^malloc[^@]+$ | while read function ; do perf probe -x /lib64/libc-2.25.so "${function}_return=$function%return" ; done > Added new event: > probe_libc:malloc_check_return (on malloc_check%return in /usr/lib64/libc-2.25.so) > > You can now use it in all perf tools, such as: > > perf record -e probe_libc:malloc_check_return -aR sleep 1 > > <SNIP> > ----------- > > This now makes us have: > > [root@jouet ~]# perf probe -l > probe:check_stack_object (on check_stack_object%return@acme/git/linux/mm/usercopy.c with ret) > probe_libc:malloc (on __malloc in /usr/lib64/libc-2.25.so) > probe_libc:malloc_check (on malloc_check in /usr/lib64/libc-2.25.so) > probe_libc:malloc_consolidate (on malloc_consolidate.part.1 in /usr/lib64/libc-2.25.so) > probe_libc:malloc_hook_ini (on malloc_hook_ini in /usr/lib64/libc-2.25.so) > probe_libc:malloc_info (on __malloc_info in /usr/lib64/libc-2.25.so) > probe_libc:malloc_printerr (on malloc_printerr in /usr/lib64/libc-2.25.so) > probe_libc:malloc_stats (on __malloc_stats in /usr/lib64/libc-2.25.so) > probe_libc:malloc_trim (on __malloc_trim in /usr/lib64/libc-2.25.so) > probe_libc:malloc_usable_size (on __malloc_usable_size in /usr/lib64/libc-2.25.so) > probe_libc:mallochook (on mallochook in /usr/lib64/libc-2.25.so) > probe_libc:malloc_check_return (on malloc_check%return in /usr/lib64/libc-2.25.so) > probe_libc:malloc_hook_ini_return (on malloc_hook_ini%return in /usr/lib64/libc-2.25.so) > probe_libc:malloc_info_return (on __malloc_info%return in /usr/lib64/libc-2.25.so) > probe_libc:malloc_printerr_return (on malloc_printerr%return in /usr/lib64/libc-2.25.so) > probe_libc:malloc_stats_return (on __malloc_stats%return in /usr/lib64/libc-2.25.so) > probe_libc:malloc_trim_return (on __malloc_trim%return in /usr/lib64/libc-2.25.so) > probe_libc:malloc_usable_size_return (on __malloc_usable_size%return in /usr/lib64/libc-2.25.so) > probe_libc:mallochook_return (on mallochook%return in /usr/lib64/libc-2.25.so) > probe_libc:malloc_return (on __malloc%return in /usr/lib64/libc-2.25.so) > [root@jouet ~]# > > There were some usability snags that prevented this from being put in > place, so do it manually: > > [root@jouet ~]# perf probe -x /lib64/libc-2.25.so 'malloc_consolidate_return=malloc_consolidate.part.1%return' > Added new event: > probe_libc:malloc_consolidate_return (on malloc_consolidate.part.1%return in /usr/lib64/libc-2.25.so) > > You can now use it in all perf tools, such as: > > perf record -e probe_libc:malloc_consolidate_return -aR sleep 1 > > [root@jouet ~]# > > Ok, now we can trace this syswide using: > > [root@jouet ~]# perf trace --no-syscalls -e probe_libc:* > <BIG SNIP> > 339.557 probe_libc:malloc:(7fda55c22970)) > 339.562 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > 339.567 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > 339.570 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > 339.596 probe_libc:malloc:(7fda55c22970)) > 339.600 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > 339.622 probe_libc:malloc:(7fda55c22970)) > 339.626 probe_libc:malloc_return:(7fda55c22970 <- 7fda56ba9a39)) > 339.631 probe_libc:malloc:(7fda55c22970)) > 339.634 probe_libc:malloc_return:(7fda55c22970 <- 7fda578f1713)) > 339.637 probe_libc:malloc:(7fda55c22970)) > 339.641 probe_libc:malloc_return:(7fda55c22970 <- 7fda578f14ae)) > 339.657 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > 339.661 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1dbd6)) > 339.664 probe_libc:malloc:(7fda55c22970)) > 339.667 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > 339.670 probe_libc:malloc:(7fda55c22970)) > 339.673 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > 339.678 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > 339.681 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > 339.685 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > 339.688 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > 339.693 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > 339.696 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > 339.778 probe_libc:malloc:(7fda55c22970)) > 339.783 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > 339.788 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > ^C(7fda55c22970 <- 7fda578f14ae)) > [root@jouet ~]# > > It works as well with 'perf record' + 'perf script'. > > Other features from perf can be added to the mix, like callchains, etc. > > But see below for doing it via tracefs > >> I observed below behaviour when I tried as per directions using URL: >> https://opensource.com/article/17/7/dynamic-tracing-linux-user-and-kernel-space#comment-136366 >> : >> >> >> echo 'p:<func_entry> ./test:0x420' > >> /sys/kernel/debug/tracing/uprobe_events (for creating >> uprobe) >> echo 1 > /sys/kernel/debug/tracing/events/enable >> (enabling uproble) >> There is another enable parameter: >> /sys/kernel/debug/tracing/events/uprobes/enable >> >> I understand unless enabled both events/enable and >> events/uprobes/enable, the /sys/kernel/debug/tracing/trace is not >> complete. I do not understand the relevance of >> .....events/uprobes/enable as I could not get any trace data in trace >> log for 'func_entry' the user space function being added for tracing. >> Irrespective of the probe function added, the trace log shows complete >> system trace log during the period of execution of the user >> application being traced. > > Is this what you want: > > [root@jouet ~]# cd /sys/kernel/debug/tracing/ > [root@jouet tracing]# cat available_tracers > hwlat blk mmiotrace function_graph wakeup_dl wakeup_rt wakeup function nop > [root@jouet tracing]# echo nop > current_tracer > [root@jouet tracing]# echo 1 > events/probe_libc/malloc/enable > [root@jouet tracing]# echo 1 > tracing_on > [root@jouet tracing]# tail trace > tail-1581 [002] d... 118484.507167: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507172: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507189: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507193: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507198: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507203: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507208: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507214: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507224: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507237: malloc: (0x7fdb32d67970) > [root@jouet tracing]# > > ? > >> Can this be clearly mentioned as to when to use one of them, role of >> both and and also, I could not see uprobe function in particular at >> all but generic trace in /sys/kernel/debug/tracing/trace. Kindly guide >> me in understanding uprobe enable logic in user space in detail. >> >> Regards, >> Bhargav >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-perf-users" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- Bhargav ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: uprobes enable condition (user space) - sysfs interface 2017-12-03 17:14 ` bhargavb @ 2017-12-04 6:03 ` Namhyung Kim 0 siblings, 0 replies; 11+ messages in thread From: Namhyung Kim @ 2017-12-04 6:03 UTC (permalink / raw) To: bhargavb Cc: Arnaldo Carvalho de Melo, linux-rt-users, linux-perf-users, Masami Hiramatsu, kernel-team Hi, On Sun, Dec 03, 2017 at 10:44:22PM +0530, bhargavb wrote: > Dear Arnaldo, > Exactly, the last one was what I was looking for. Thank you very much... > That apart I am also exploring various profiling options that can give > me a full control of monitoring running processes (user space > profiling ). > sysfs seems powerful and very useful, though I got to know lately (and > not much mentioned, I think as tools like perf etc). I was confused > when u said tracefs in your response, I understand it is same as > sysfs. Please correct me if I am wrong. It's a different filesystem for controlling tracing features but just happens to be mounted on top of a sysfs (or debugfs) directory. $ mount | grep tracefs tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime) Also, writing to the 'enable' file changes state of every events in that directory and its subdirectory. So $TRACEFS/events/enable changes all events. Thanks, Namhyung > > > On Fri, Dec 1, 2017 at 10:10 PM, Arnaldo Carvalho de Melo > <arnaldo.melo@gmail.com> wrote: > > Adding Masami to the CC list, to see if he can come up with a fix for > > the 'perf probe' failure below, improving the message or ignoring > > versioned symbols (with a warning perhaps). > > > > Em Thu, Nov 30, 2017 at 12:04:23AM +0530, bhargavb escreveu: > >> Hello, > >> > >> I am trying to understand user space profiling using linux inbuilt > >> mechanism. (Trying with perf also but could not get complete clear > >> documentation and could not proceed, so retained linux-perf-users in > >> to list as the underlying interface used is same for both). > > > > So yo uwant to trace function entry/exit in some userspace library? You > > can try using the tracefs as below or leave it to perf to do that, first > > you set up the probes, say for some glibc functions: > > > > [root@jouet ~]# perf probe -F -x /lib64/libc-2.25.so | grep ^malloc > > malloc > > malloc_check > > malloc_consolidate.part.1 > > malloc_get_state@GLIBC_2.2.5 > > malloc_hook_ini > > malloc_info > > malloc_printerr > > malloc_set_state@GLIBC_2.2.5 > > malloc_stats > > malloc_trim > > malloc_usable_size > > mallochook > > [root@jouet ~]# > > > > [root@jouet ~]# perf probe -x /lib64/libc-2.25.so malloc* > > Failed to write event: Invalid argument > > Error: Failed to add events. > > [root@jouet ~]# > > > > For now ignore that error, its a bug I just found, what matters is that > > it _has_ put in place probes for most of those functions, as you can see > > by looking at: > > > > [root@jouet ~]# perf probe -l > > probe_libc:malloc (on __malloc in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_check (on malloc_check in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_consolidate (on malloc_consolidate.part.1 in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_hook_ini (on malloc_hook_ini in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_info (on __malloc_info in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_printerr (on malloc_printerr in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_stats (on __malloc_stats in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_trim (on __malloc_trim in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_usable_size (on __malloc_usable_size in /usr/lib64/libc-2.25.so) > > probe_libc:mallochook (on mallochook in /usr/lib64/libc-2.25.so) > > [root@jouet ~]# > > > > Masami, this one also doesn't work: > > > > [root@jouet ~]# perf probe -x /lib64/libc-2.25.so 'malloc*%return' > > Error: event "malloc_printerr" already exists. > > <SNIP> > > > > doing it a bit differently: > > > > [root@jouet ~]# perf probe -F -x /lib64/libc-2.25.so | egrep ^malloc[^@]+$ | while read function ; do perf probe -x /lib64/libc-2.25.so "${function}_return=$function%return" ; done > > Added new event: > > probe_libc:malloc_check_return (on malloc_check%return in /usr/lib64/libc-2.25.so) > > > > You can now use it in all perf tools, such as: > > > > perf record -e probe_libc:malloc_check_return -aR sleep 1 > > > > <SNIP> > > ----------- > > > > This now makes us have: > > > > [root@jouet ~]# perf probe -l > > probe:check_stack_object (on check_stack_object%return@acme/git/linux/mm/usercopy.c with ret) > > probe_libc:malloc (on __malloc in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_check (on malloc_check in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_consolidate (on malloc_consolidate.part.1 in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_hook_ini (on malloc_hook_ini in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_info (on __malloc_info in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_printerr (on malloc_printerr in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_stats (on __malloc_stats in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_trim (on __malloc_trim in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_usable_size (on __malloc_usable_size in /usr/lib64/libc-2.25.so) > > probe_libc:mallochook (on mallochook in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_check_return (on malloc_check%return in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_hook_ini_return (on malloc_hook_ini%return in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_info_return (on __malloc_info%return in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_printerr_return (on malloc_printerr%return in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_stats_return (on __malloc_stats%return in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_trim_return (on __malloc_trim%return in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_usable_size_return (on __malloc_usable_size%return in /usr/lib64/libc-2.25.so) > > probe_libc:mallochook_return (on mallochook%return in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_return (on __malloc%return in /usr/lib64/libc-2.25.so) > > [root@jouet ~]# > > > > There were some usability snags that prevented this from being put in > > place, so do it manually: > > > > [root@jouet ~]# perf probe -x /lib64/libc-2.25.so 'malloc_consolidate_return=malloc_consolidate.part.1%return' > > Added new event: > > probe_libc:malloc_consolidate_return (on malloc_consolidate.part.1%return in /usr/lib64/libc-2.25.so) > > > > You can now use it in all perf tools, such as: > > > > perf record -e probe_libc:malloc_consolidate_return -aR sleep 1 > > > > [root@jouet ~]# > > > > Ok, now we can trace this syswide using: > > > > [root@jouet ~]# perf trace --no-syscalls -e probe_libc:* > > <BIG SNIP> > > 339.557 probe_libc:malloc:(7fda55c22970)) > > 339.562 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > > 339.567 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > 339.570 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > > 339.596 probe_libc:malloc:(7fda55c22970)) > > 339.600 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > > 339.622 probe_libc:malloc:(7fda55c22970)) > > 339.626 probe_libc:malloc_return:(7fda55c22970 <- 7fda56ba9a39)) > > 339.631 probe_libc:malloc:(7fda55c22970)) > > 339.634 probe_libc:malloc_return:(7fda55c22970 <- 7fda578f1713)) > > 339.637 probe_libc:malloc:(7fda55c22970)) > > 339.641 probe_libc:malloc_return:(7fda55c22970 <- 7fda578f14ae)) > > 339.657 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > 339.661 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1dbd6)) > > 339.664 probe_libc:malloc:(7fda55c22970)) > > 339.667 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > > 339.670 probe_libc:malloc:(7fda55c22970)) > > 339.673 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > > 339.678 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > 339.681 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > > 339.685 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > 339.688 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > > 339.693 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > 339.696 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > > 339.778 probe_libc:malloc:(7fda55c22970)) > > 339.783 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > > 339.788 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > ^C(7fda55c22970 <- 7fda578f14ae)) > > [root@jouet ~]# > > > > It works as well with 'perf record' + 'perf script'. > > > > Other features from perf can be added to the mix, like callchains, etc. > > > > But see below for doing it via tracefs > > > >> I observed below behaviour when I tried as per directions using URL: > >> https://opensource.com/article/17/7/dynamic-tracing-linux-user-and-kernel-space#comment-136366 > >> : > >> > >> > >> echo 'p:<func_entry> ./test:0x420' > > >> /sys/kernel/debug/tracing/uprobe_events (for creating > >> uprobe) > >> echo 1 > /sys/kernel/debug/tracing/events/enable > >> (enabling uproble) > >> There is another enable parameter: > >> /sys/kernel/debug/tracing/events/uprobes/enable > >> > >> I understand unless enabled both events/enable and > >> events/uprobes/enable, the /sys/kernel/debug/tracing/trace is not > >> complete. I do not understand the relevance of > >> .....events/uprobes/enable as I could not get any trace data in trace > >> log for 'func_entry' the user space function being added for tracing. > >> Irrespective of the probe function added, the trace log shows complete > >> system trace log during the period of execution of the user > >> application being traced. > > > > Is this what you want: > > > > [root@jouet ~]# cd /sys/kernel/debug/tracing/ > > [root@jouet tracing]# cat available_tracers > > hwlat blk mmiotrace function_graph wakeup_dl wakeup_rt wakeup function nop > > [root@jouet tracing]# echo nop > current_tracer > > [root@jouet tracing]# echo 1 > events/probe_libc/malloc/enable > > [root@jouet tracing]# echo 1 > tracing_on > > [root@jouet tracing]# tail trace > > tail-1581 [002] d... 118484.507167: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507172: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507189: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507193: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507198: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507203: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507208: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507214: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507224: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507237: malloc: (0x7fdb32d67970) > > [root@jouet tracing]# > > > > ? > > > >> Can this be clearly mentioned as to when to use one of them, role of > >> both and and also, I could not see uprobe function in particular at > >> all but generic trace in /sys/kernel/debug/tracing/trace. Kindly guide > >> me in understanding uprobe enable logic in user space in detail. > >> > >> Regards, > >> Bhargav > >> -- > >> To unsubscribe from this list: send the line "unsubscribe linux-perf-users" in > >> the body of a message to majordomo@vger.kernel.org > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > -- > Bhargav > -- > To unsubscribe from this list: send the line "unsubscribe linux-perf-users" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: uprobes enable condition (user space) - sysfs interface 2017-12-01 16:40 ` Arnaldo Carvalho de Melo 2017-12-03 17:14 ` bhargavb @ 2017-12-04 12:36 ` Masami Hiramatsu 2017-12-04 14:55 ` Arnaldo Carvalho de Melo 1 sibling, 1 reply; 11+ messages in thread From: Masami Hiramatsu @ 2017-12-04 12:36 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: bhargavb, linux-rt-users, linux-perf-users, Masami Hiramatsu On Fri, 1 Dec 2017 13:40:39 -0300 Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > Adding Masami to the CC list, to see if he can come up with a fix for > the 'perf probe' failure below, improving the message or ignoring > versioned symbols (with a warning perhaps). Thanks. Should we ignore the versioned symbols? or can we put a probe on that? If we have symbol address, we can put a probe on it, even if it is versioned. > > Em Thu, Nov 30, 2017 at 12:04:23AM +0530, bhargavb escreveu: > > Hello, > > > > I am trying to understand user space profiling using linux inbuilt > > mechanism. (Trying with perf also but could not get complete clear > > documentation and could not proceed, so retained linux-perf-users in > > to list as the underlying interface used is same for both). > > So yo uwant to trace function entry/exit in some userspace library? You > can try using the tracefs as below or leave it to perf to do that, first > you set up the probes, say for some glibc functions: > > [root@jouet ~]# perf probe -F -x /lib64/libc-2.25.so | grep ^malloc > malloc > malloc_check > malloc_consolidate.part.1 > malloc_get_state@GLIBC_2.2.5 > malloc_hook_ini > malloc_info > malloc_printerr > malloc_set_state@GLIBC_2.2.5 > malloc_stats > malloc_trim > malloc_usable_size > mallochook > [root@jouet ~]# > > [root@jouet ~]# perf probe -x /lib64/libc-2.25.so malloc* > Failed to write event: Invalid argument > Error: Failed to add events. > [root@jouet ~]# OK, this is not good, at least it should analyse the reason(versioned) and warn that (Or, I think we can just remove the suffix after "@"). > > For now ignore that error, its a bug I just found, what matters is that > it _has_ put in place probes for most of those functions, as you can see > by looking at: > > [root@jouet ~]# perf probe -l > probe_libc:malloc (on __malloc in /usr/lib64/libc-2.25.so) > probe_libc:malloc_check (on malloc_check in /usr/lib64/libc-2.25.so) > probe_libc:malloc_consolidate (on malloc_consolidate.part.1 in /usr/lib64/libc-2.25.so) > probe_libc:malloc_hook_ini (on malloc_hook_ini in /usr/lib64/libc-2.25.so) > probe_libc:malloc_info (on __malloc_info in /usr/lib64/libc-2.25.so) > probe_libc:malloc_printerr (on malloc_printerr in /usr/lib64/libc-2.25.so) > probe_libc:malloc_stats (on __malloc_stats in /usr/lib64/libc-2.25.so) > probe_libc:malloc_trim (on __malloc_trim in /usr/lib64/libc-2.25.so) > probe_libc:malloc_usable_size (on __malloc_usable_size in /usr/lib64/libc-2.25.so) > probe_libc:mallochook (on mallochook in /usr/lib64/libc-2.25.so) > [root@jouet ~]# > > Masami, this one also doesn't work: > > [root@jouet ~]# perf probe -x /lib64/libc-2.25.so 'malloc*%return' > Error: event "malloc_printerr" already exists. > <SNIP> Hm, this looks like a collision happens on that event name. "-f"(--force) option allows you to add events with number suffix (e.g. _1, _2, ... ) > > doing it a bit differently: > > [root@jouet ~]# perf probe -F -x /lib64/libc-2.25.so | egrep ^malloc[^@]+$ | while read function ; do perf probe -x /lib64/libc-2.25.so "${function}_return=$function%return" ; done > Added new event: > probe_libc:malloc_check_return (on malloc_check%return in /usr/lib64/libc-2.25.so) > > You can now use it in all perf tools, such as: > > perf record -e probe_libc:malloc_check_return -aR sleep 1 Hmm, would you think we should add __return suffix by default? Thanks, > > <SNIP> > ----------- > > This now makes us have: > > [root@jouet ~]# perf probe -l > probe:check_stack_object (on check_stack_object%return@acme/git/linux/mm/usercopy.c with ret) > probe_libc:malloc (on __malloc in /usr/lib64/libc-2.25.so) > probe_libc:malloc_check (on malloc_check in /usr/lib64/libc-2.25.so) > probe_libc:malloc_consolidate (on malloc_consolidate.part.1 in /usr/lib64/libc-2.25.so) > probe_libc:malloc_hook_ini (on malloc_hook_ini in /usr/lib64/libc-2.25.so) > probe_libc:malloc_info (on __malloc_info in /usr/lib64/libc-2.25.so) > probe_libc:malloc_printerr (on malloc_printerr in /usr/lib64/libc-2.25.so) > probe_libc:malloc_stats (on __malloc_stats in /usr/lib64/libc-2.25.so) > probe_libc:malloc_trim (on __malloc_trim in /usr/lib64/libc-2.25.so) > probe_libc:malloc_usable_size (on __malloc_usable_size in /usr/lib64/libc-2.25.so) > probe_libc:mallochook (on mallochook in /usr/lib64/libc-2.25.so) > probe_libc:malloc_check_return (on malloc_check%return in /usr/lib64/libc-2.25.so) > probe_libc:malloc_hook_ini_return (on malloc_hook_ini%return in /usr/lib64/libc-2.25.so) > probe_libc:malloc_info_return (on __malloc_info%return in /usr/lib64/libc-2.25.so) > probe_libc:malloc_printerr_return (on malloc_printerr%return in /usr/lib64/libc-2.25.so) > probe_libc:malloc_stats_return (on __malloc_stats%return in /usr/lib64/libc-2.25.so) > probe_libc:malloc_trim_return (on __malloc_trim%return in /usr/lib64/libc-2.25.so) > probe_libc:malloc_usable_size_return (on __malloc_usable_size%return in /usr/lib64/libc-2.25.so) > probe_libc:mallochook_return (on mallochook%return in /usr/lib64/libc-2.25.so) > probe_libc:malloc_return (on __malloc%return in /usr/lib64/libc-2.25.so) > [root@jouet ~]# > > There were some usability snags that prevented this from being put in > place, so do it manually: > > [root@jouet ~]# perf probe -x /lib64/libc-2.25.so 'malloc_consolidate_return=malloc_consolidate.part.1%return' > Added new event: > probe_libc:malloc_consolidate_return (on malloc_consolidate.part.1%return in /usr/lib64/libc-2.25.so) > > You can now use it in all perf tools, such as: > > perf record -e probe_libc:malloc_consolidate_return -aR sleep 1 > > [root@jouet ~]# > > Ok, now we can trace this syswide using: > > [root@jouet ~]# perf trace --no-syscalls -e probe_libc:* > <BIG SNIP> > 339.557 probe_libc:malloc:(7fda55c22970)) > 339.562 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > 339.567 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > 339.570 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > 339.596 probe_libc:malloc:(7fda55c22970)) > 339.600 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > 339.622 probe_libc:malloc:(7fda55c22970)) > 339.626 probe_libc:malloc_return:(7fda55c22970 <- 7fda56ba9a39)) > 339.631 probe_libc:malloc:(7fda55c22970)) > 339.634 probe_libc:malloc_return:(7fda55c22970 <- 7fda578f1713)) > 339.637 probe_libc:malloc:(7fda55c22970)) > 339.641 probe_libc:malloc_return:(7fda55c22970 <- 7fda578f14ae)) > 339.657 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > 339.661 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1dbd6)) > 339.664 probe_libc:malloc:(7fda55c22970)) > 339.667 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > 339.670 probe_libc:malloc:(7fda55c22970)) > 339.673 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > 339.678 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > 339.681 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > 339.685 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > 339.688 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > 339.693 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > 339.696 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > 339.778 probe_libc:malloc:(7fda55c22970)) > 339.783 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > 339.788 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > ^C(7fda55c22970 <- 7fda578f14ae)) > [root@jouet ~]# > > It works as well with 'perf record' + 'perf script'. > > Other features from perf can be added to the mix, like callchains, etc. > > But see below for doing it via tracefs > > > I observed below behaviour when I tried as per directions using URL: > > https://opensource.com/article/17/7/dynamic-tracing-linux-user-and-kernel-space#comment-136366 > > : > > > > > > echo 'p:<func_entry> ./test:0x420' > > > /sys/kernel/debug/tracing/uprobe_events (for creating > > uprobe) > > echo 1 > /sys/kernel/debug/tracing/events/enable > > (enabling uproble) > > There is another enable parameter: > > /sys/kernel/debug/tracing/events/uprobes/enable > > > > I understand unless enabled both events/enable and > > events/uprobes/enable, the /sys/kernel/debug/tracing/trace is not > > complete. I do not understand the relevance of > > .....events/uprobes/enable as I could not get any trace data in trace > > log for 'func_entry' the user space function being added for tracing. > > Irrespective of the probe function added, the trace log shows complete > > system trace log during the period of execution of the user > > application being traced. > > Is this what you want: > > [root@jouet ~]# cd /sys/kernel/debug/tracing/ > [root@jouet tracing]# cat available_tracers > hwlat blk mmiotrace function_graph wakeup_dl wakeup_rt wakeup function nop > [root@jouet tracing]# echo nop > current_tracer > [root@jouet tracing]# echo 1 > events/probe_libc/malloc/enable > [root@jouet tracing]# echo 1 > tracing_on > [root@jouet tracing]# tail trace > tail-1581 [002] d... 118484.507167: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507172: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507189: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507193: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507198: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507203: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507208: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507214: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507224: malloc: (0x7fdb32d67970) > tail-1581 [002] d... 118484.507237: malloc: (0x7fdb32d67970) > [root@jouet tracing]# > > ? > > > Can this be clearly mentioned as to when to use one of them, role of > > both and and also, I could not see uprobe function in particular at > > all but generic trace in /sys/kernel/debug/tracing/trace. Kindly guide > > me in understanding uprobe enable logic in user space in detail. > > > > Regards, > > Bhargav > > -- -- Masami Hiramatsu <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: uprobes enable condition (user space) - sysfs interface 2017-12-04 12:36 ` Masami Hiramatsu @ 2017-12-04 14:55 ` Arnaldo Carvalho de Melo 2017-12-04 15:23 ` Paul Clarke 2017-12-05 6:51 ` Masami Hiramatsu 0 siblings, 2 replies; 11+ messages in thread From: Arnaldo Carvalho de Melo @ 2017-12-04 14:55 UTC (permalink / raw) To: Masami Hiramatsu; +Cc: bhargavb, linux-rt-users, linux-perf-users Em Mon, Dec 04, 2017 at 09:36:19PM +0900, Masami Hiramatsu escreveu: > On Fri, 1 Dec 2017 13:40:39 -0300 > Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > > > Adding Masami to the CC list, to see if he can come up with a fix for > > the 'perf probe' failure below, improving the message or ignoring > > versioned symbols (with a warning perhaps). > > Thanks. Should we ignore the versioned symbols? or can we put a probe > on that? If we have symbol address, we can put a probe on it, even > if it is versioned. Lemme try... [root@jouet ~]# perf probe -x /usr/lib64/libc-2.25.so malloc_get_state_225=malloc_get_state@GLIBC_2.2.5 The /usr/lib64/libc-2.25.so file has no debug information. Rebuild with -g, or install an appropriate debuginfo package. Error: Failed to add events. [root@jouet ~]# perf probe -x /usr/lib64/libc-2.25.so malloc_get_state@GLIBC_2.2.5 The /usr/lib64/libc-2.25.so file has no debug information. Rebuild with -g, or install an appropriate debuginfo package. Error: Failed to add events. [root@jouet ~]# perf probe -x /usr/lib64/libc-2.25.so malloc Added new event: probe_libc:malloc (on malloc in /usr/lib64/libc-2.25.so) You can now use it in all perf tools, such as: perf record -e probe_libc:malloc -aR sleep 1 [root@jouet ~]# So here the message is misleading, as we did manage to put a probe on 'malloc', but not on the versioned symbol, both when trying to give it a different name (malloc_get_state_225) and when doing it directly (malloc_get_state@GLIBC_2.2.5). > > > > Em Thu, Nov 30, 2017 at 12:04:23AM +0530, bhargavb escreveu: > > > Hello, > > > > > > I am trying to understand user space profiling using linux inbuilt > > > mechanism. (Trying with perf also but could not get complete clear > > > documentation and could not proceed, so retained linux-perf-users in > > > to list as the underlying interface used is same for both). > > > > So yo uwant to trace function entry/exit in some userspace library? You > > can try using the tracefs as below or leave it to perf to do that, first > > you set up the probes, say for some glibc functions: > > > > [root@jouet ~]# perf probe -F -x /lib64/libc-2.25.so | grep ^malloc > > malloc > > malloc_check > > malloc_consolidate.part.1 > > malloc_get_state@GLIBC_2.2.5 > > malloc_hook_ini > > malloc_info > > malloc_printerr > > malloc_set_state@GLIBC_2.2.5 > > malloc_stats > > malloc_trim > > malloc_usable_size > > mallochook > > [root@jouet ~]# > > > > [root@jouet ~]# perf probe -x /lib64/libc-2.25.so malloc* > > Failed to write event: Invalid argument > > Error: Failed to add events. > > [root@jouet ~]# > > OK, this is not good, at least it should analyse the reason(versioned) > and warn that (Or, I think we can just remove the suffix after "@"). I think it is a matter of replacing special characters with some other. I think we should support versioned symbols, i.e. one may well want to intercept calls to a specific version of a symbol. > > > > For now ignore that error, its a bug I just found, what matters is that > > it _has_ put in place probes for most of those functions, as you can see > > by looking at: > > > > [root@jouet ~]# perf probe -l > > probe_libc:malloc (on __malloc in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_check (on malloc_check in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_consolidate (on malloc_consolidate.part.1 in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_hook_ini (on malloc_hook_ini in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_info (on __malloc_info in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_printerr (on malloc_printerr in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_stats (on __malloc_stats in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_trim (on __malloc_trim in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_usable_size (on __malloc_usable_size in /usr/lib64/libc-2.25.so) > > probe_libc:mallochook (on mallochook in /usr/lib64/libc-2.25.so) > > [root@jouet ~]# > > > > Masami, this one also doesn't work: > > > > [root@jouet ~]# perf probe -x /lib64/libc-2.25.so 'malloc*%return' > > Error: event "malloc_printerr" already exists. > > <SNIP> > > Hm, this looks like a collision happens on that event name. "-f"(--force) > option allows you to add events with number suffix (e.g. _1, _2, ... ) Probably this was because the same name was used for both entry and exit, see below on automatically adding __return to %return probes. > > > > doing it a bit differently: > > > > [root@jouet ~]# perf probe -F -x /lib64/libc-2.25.so | egrep ^malloc[^@]+$ | while read function ; do perf probe -x /lib64/libc-2.25.so "${function}_return=$function%return" ; done > > Added new event: > > probe_libc:malloc_check_return (on malloc_check%return in /usr/lib64/libc-2.25.so) > > > > You can now use it in all perf tools, such as: > > > > perf record -e probe_libc:malloc_check_return -aR sleep 1 > > Hmm, would you think we should add __return suffix by default? I think so, as it is a common operation to ask for entry+exit. - Arnaldo > Thanks, > > > > > <SNIP> > > ----------- > > > > This now makes us have: > > > > [root@jouet ~]# perf probe -l > > probe:check_stack_object (on check_stack_object%return@acme/git/linux/mm/usercopy.c with ret) > > probe_libc:malloc (on __malloc in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_check (on malloc_check in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_consolidate (on malloc_consolidate.part.1 in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_hook_ini (on malloc_hook_ini in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_info (on __malloc_info in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_printerr (on malloc_printerr in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_stats (on __malloc_stats in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_trim (on __malloc_trim in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_usable_size (on __malloc_usable_size in /usr/lib64/libc-2.25.so) > > probe_libc:mallochook (on mallochook in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_check_return (on malloc_check%return in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_hook_ini_return (on malloc_hook_ini%return in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_info_return (on __malloc_info%return in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_printerr_return (on malloc_printerr%return in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_stats_return (on __malloc_stats%return in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_trim_return (on __malloc_trim%return in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_usable_size_return (on __malloc_usable_size%return in /usr/lib64/libc-2.25.so) > > probe_libc:mallochook_return (on mallochook%return in /usr/lib64/libc-2.25.so) > > probe_libc:malloc_return (on __malloc%return in /usr/lib64/libc-2.25.so) > > [root@jouet ~]# > > > > There were some usability snags that prevented this from being put in > > place, so do it manually: > > > > [root@jouet ~]# perf probe -x /lib64/libc-2.25.so 'malloc_consolidate_return=malloc_consolidate.part.1%return' > > Added new event: > > probe_libc:malloc_consolidate_return (on malloc_consolidate.part.1%return in /usr/lib64/libc-2.25.so) > > > > You can now use it in all perf tools, such as: > > > > perf record -e probe_libc:malloc_consolidate_return -aR sleep 1 > > > > [root@jouet ~]# > > > > Ok, now we can trace this syswide using: > > > > [root@jouet ~]# perf trace --no-syscalls -e probe_libc:* > > <BIG SNIP> > > 339.557 probe_libc:malloc:(7fda55c22970)) > > 339.562 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > > 339.567 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > 339.570 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > > 339.596 probe_libc:malloc:(7fda55c22970)) > > 339.600 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > > 339.622 probe_libc:malloc:(7fda55c22970)) > > 339.626 probe_libc:malloc_return:(7fda55c22970 <- 7fda56ba9a39)) > > 339.631 probe_libc:malloc:(7fda55c22970)) > > 339.634 probe_libc:malloc_return:(7fda55c22970 <- 7fda578f1713)) > > 339.637 probe_libc:malloc:(7fda55c22970)) > > 339.641 probe_libc:malloc_return:(7fda55c22970 <- 7fda578f14ae)) > > 339.657 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > 339.661 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1dbd6)) > > 339.664 probe_libc:malloc:(7fda55c22970)) > > 339.667 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > > 339.670 probe_libc:malloc:(7fda55c22970)) > > 339.673 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > > 339.678 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > 339.681 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > > 339.685 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > 339.688 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > > 339.693 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > 339.696 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > > 339.778 probe_libc:malloc:(7fda55c22970)) > > 339.783 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > > 339.788 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > ^C(7fda55c22970 <- 7fda578f14ae)) > > [root@jouet ~]# > > > > It works as well with 'perf record' + 'perf script'. > > > > Other features from perf can be added to the mix, like callchains, etc. > > > > But see below for doing it via tracefs > > > > > I observed below behaviour when I tried as per directions using URL: > > > https://opensource.com/article/17/7/dynamic-tracing-linux-user-and-kernel-space#comment-136366 > > > : > > > > > > > > > echo 'p:<func_entry> ./test:0x420' > > > > /sys/kernel/debug/tracing/uprobe_events (for creating > > > uprobe) > > > echo 1 > /sys/kernel/debug/tracing/events/enable > > > (enabling uproble) > > > There is another enable parameter: > > > /sys/kernel/debug/tracing/events/uprobes/enable > > > > > > I understand unless enabled both events/enable and > > > events/uprobes/enable, the /sys/kernel/debug/tracing/trace is not > > > complete. I do not understand the relevance of > > > .....events/uprobes/enable as I could not get any trace data in trace > > > log for 'func_entry' the user space function being added for tracing. > > > Irrespective of the probe function added, the trace log shows complete > > > system trace log during the period of execution of the user > > > application being traced. > > > > Is this what you want: > > > > [root@jouet ~]# cd /sys/kernel/debug/tracing/ > > [root@jouet tracing]# cat available_tracers > > hwlat blk mmiotrace function_graph wakeup_dl wakeup_rt wakeup function nop > > [root@jouet tracing]# echo nop > current_tracer > > [root@jouet tracing]# echo 1 > events/probe_libc/malloc/enable > > [root@jouet tracing]# echo 1 > tracing_on > > [root@jouet tracing]# tail trace > > tail-1581 [002] d... 118484.507167: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507172: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507189: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507193: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507198: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507203: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507208: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507214: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507224: malloc: (0x7fdb32d67970) > > tail-1581 [002] d... 118484.507237: malloc: (0x7fdb32d67970) > > [root@jouet tracing]# > > > > ? > > > > > Can this be clearly mentioned as to when to use one of them, role of > > > both and and also, I could not see uprobe function in particular at > > > all but generic trace in /sys/kernel/debug/tracing/trace. Kindly guide > > > me in understanding uprobe enable logic in user space in detail. > > > > > > Regards, > > > Bhargav > > > -- > > -- > Masami Hiramatsu <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: uprobes enable condition (user space) - sysfs interface 2017-12-04 14:55 ` Arnaldo Carvalho de Melo @ 2017-12-04 15:23 ` Paul Clarke 2017-12-05 6:53 ` Masami Hiramatsu 2017-12-05 7:45 ` Masami Hiramatsu 2017-12-05 6:51 ` Masami Hiramatsu 1 sibling, 2 replies; 11+ messages in thread From: Paul Clarke @ 2017-12-04 15:23 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, Masami Hiramatsu Cc: bhargavb, linux-rt-users, linux-perf-users On 12/04/2017 08:55 AM, Arnaldo Carvalho de Melo wrote: > Em Mon, Dec 04, 2017 at 09:36:19PM +0900, Masami Hiramatsu escreveu: >> On Fri, 1 Dec 2017 13:40:39 -0300 >> Arnaldo Carvalho de Melo <acme@kernel.org> wrote: >> >>> Adding Masami to the CC list, to see if he can come up with a fix for >>> the 'perf probe' failure below, improving the message or ignoring >>> versioned symbols (with a warning perhaps). >> >> Thanks. Should we ignore the versioned symbols? or can we put a probe >> on that? If we have symbol address, we can put a probe on it, even >> if it is versioned. > > Lemme try... > > [root@jouet ~]# perf probe -x /usr/lib64/libc-2.25.so malloc_get_state_225=malloc_get_state@GLIBC_2.2.5 > The /usr/lib64/libc-2.25.so file has no debug information. > Rebuild with -g, or install an appropriate debuginfo package. > Error: Failed to add events. > [root@jouet ~]# perf probe -x /usr/lib64/libc-2.25.so malloc_get_state@GLIBC_2.2.5 > The /usr/lib64/libc-2.25.so file has no debug information. > Rebuild with -g, or install an appropriate debuginfo package. > Error: Failed to add events. commit d80406453ad4a69932dc17a617d5b7bc7ae80b8f should have allowed adding probes on versioned symbols if you omit the versioning part of the symbol. So, in the above example, this should work: # perf probe -x /usr/lib64/libc-2.25.so malloc_get_state PC ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: uprobes enable condition (user space) - sysfs interface 2017-12-04 15:23 ` Paul Clarke @ 2017-12-05 6:53 ` Masami Hiramatsu 2017-12-05 7:45 ` Masami Hiramatsu 1 sibling, 0 replies; 11+ messages in thread From: Masami Hiramatsu @ 2017-12-05 6:53 UTC (permalink / raw) To: Paul Clarke Cc: Arnaldo Carvalho de Melo, bhargavb, linux-rt-users, linux-perf-users On Mon, 4 Dec 2017 09:23:16 -0600 Paul Clarke <pc@us.ibm.com> wrote: > On 12/04/2017 08:55 AM, Arnaldo Carvalho de Melo wrote: > > Em Mon, Dec 04, 2017 at 09:36:19PM +0900, Masami Hiramatsu escreveu: > >> On Fri, 1 Dec 2017 13:40:39 -0300 > >> Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > >> > >>> Adding Masami to the CC list, to see if he can come up with a fix for > >>> the 'perf probe' failure below, improving the message or ignoring > >>> versioned symbols (with a warning perhaps). > >> > >> Thanks. Should we ignore the versioned symbols? or can we put a probe > >> on that? If we have symbol address, we can put a probe on it, even > >> if it is versioned. > > > > Lemme try... > > > > [root@jouet ~]# perf probe -x /usr/lib64/libc-2.25.so malloc_get_state_225=malloc_get_state@GLIBC_2.2.5 > > The /usr/lib64/libc-2.25.so file has no debug information. > > Rebuild with -g, or install an appropriate debuginfo package. > > Error: Failed to add events. > > [root@jouet ~]# perf probe -x /usr/lib64/libc-2.25.so malloc_get_state@GLIBC_2.2.5 > > The /usr/lib64/libc-2.25.so file has no debug information. > > Rebuild with -g, or install an appropriate debuginfo package. > > Error: Failed to add events. > > commit d80406453ad4a69932dc17a617d5b7bc7ae80b8f should have allowed adding probes on versioned symbols if you omit the versioning part of the symbol. So, in the above example, this should work: > > # perf probe -x /usr/lib64/libc-2.25.so malloc_get_state Would there be no chance that single binary has different versioned symbols? Thank you, > > PC > -- Masami Hiramatsu <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: uprobes enable condition (user space) - sysfs interface 2017-12-04 15:23 ` Paul Clarke 2017-12-05 6:53 ` Masami Hiramatsu @ 2017-12-05 7:45 ` Masami Hiramatsu 2017-12-05 13:58 ` Paul Clarke 1 sibling, 1 reply; 11+ messages in thread From: Masami Hiramatsu @ 2017-12-05 7:45 UTC (permalink / raw) To: Paul Clarke Cc: Arnaldo Carvalho de Melo, bhargavb, linux-rt-users, linux-perf-users On Mon, 4 Dec 2017 09:23:16 -0600 Paul Clarke <pc@us.ibm.com> wrote: > On 12/04/2017 08:55 AM, Arnaldo Carvalho de Melo wrote: > > Em Mon, Dec 04, 2017 at 09:36:19PM +0900, Masami Hiramatsu escreveu: > >> On Fri, 1 Dec 2017 13:40:39 -0300 > >> Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > >> > >>> Adding Masami to the CC list, to see if he can come up with a fix for > >>> the 'perf probe' failure below, improving the message or ignoring > >>> versioned symbols (with a warning perhaps). > >> > >> Thanks. Should we ignore the versioned symbols? or can we put a probe > >> on that? If we have symbol address, we can put a probe on it, even > >> if it is versioned. > > > > Lemme try... > > > > [root@jouet ~]# perf probe -x /usr/lib64/libc-2.25.so malloc_get_state_225=malloc_get_state@GLIBC_2.2.5 > > The /usr/lib64/libc-2.25.so file has no debug information. > > Rebuild with -g, or install an appropriate debuginfo package. > > Error: Failed to add events. > > [root@jouet ~]# perf probe -x /usr/lib64/libc-2.25.so malloc_get_state@GLIBC_2.2.5 > > The /usr/lib64/libc-2.25.so file has no debug information. > > Rebuild with -g, or install an appropriate debuginfo package. > > Error: Failed to add events. > > commit d80406453ad4a69932dc17a617d5b7bc7ae80b8f should have allowed adding probes on versioned symbols if you omit the versioning part of the symbol. So, in the above example, this should work: > > # perf probe -x /usr/lib64/libc-2.25.so malloc_get_state No, it doesn't, the above commit improves *find_symbol_by_name(), but perf probe uses its own lookup function, because it has to support glob pattern matching. Let me see, how I can improve that... Thanks, > > PC > -- Masami Hiramatsu <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: uprobes enable condition (user space) - sysfs interface 2017-12-05 7:45 ` Masami Hiramatsu @ 2017-12-05 13:58 ` Paul Clarke 0 siblings, 0 replies; 11+ messages in thread From: Paul Clarke @ 2017-12-05 13:58 UTC (permalink / raw) To: Masami Hiramatsu Cc: Arnaldo Carvalho de Melo, bhargavb, linux-rt-users, linux-perf-users On 12/05/2017 01:45 AM, Masami Hiramatsu wrote: > On Mon, 4 Dec 2017 09:23:16 -0600 > Paul Clarke <pc@us.ibm.com> wrote: >> On 12/04/2017 08:55 AM, Arnaldo Carvalho de Melo wrote: >>> Em Mon, Dec 04, 2017 at 09:36:19PM +0900, Masami Hiramatsu escreveu: >>>> On Fri, 1 Dec 2017 13:40:39 -0300 >>>> Arnaldo Carvalho de Melo <acme@kernel.org> wrote: >>>> >>>>> Adding Masami to the CC list, to see if he can come up with a fix for >>>>> the 'perf probe' failure below, improving the message or ignoring >>>>> versioned symbols (with a warning perhaps). >>>> >>>> Thanks. Should we ignore the versioned symbols? or can we put a probe >>>> on that? If we have symbol address, we can put a probe on it, even >>>> if it is versioned. >>> >>> Lemme try... >>> >>> [root@jouet ~]# perf probe -x /usr/lib64/libc-2.25.so malloc_get_state_225=malloc_get_state@GLIBC_2.2.5 >>> The /usr/lib64/libc-2.25.so file has no debug information. >>> Rebuild with -g, or install an appropriate debuginfo package. >>> Error: Failed to add events. >>> [root@jouet ~]# perf probe -x /usr/lib64/libc-2.25.so malloc_get_state@GLIBC_2.2.5 >>> The /usr/lib64/libc-2.25.so file has no debug information. >>> Rebuild with -g, or install an appropriate debuginfo package. >>> Error: Failed to add events. >> >> commit d80406453ad4a69932dc17a617d5b7bc7ae80b8f should have allowed adding probes on versioned symbols if you omit the versioning part of the symbol. So, in the above example, this should work: >> >> # perf probe -x /usr/lib64/libc-2.25.so malloc_get_state > > No, it doesn't, the above commit improves *find_symbol_by_name(), > but perf probe uses its own lookup function, because it has to > support glob pattern matching. That commit was explicitly to allow perf probe on versioned symbols. However, it was limited to creating probes on _default_ versioned symbols (marked by '@@"). So, you're correct that it would not work on this symbol in particular. The value of the commit was to prevent something like the following from failing, even though the symbol existed, but was versioned (presuming one of the versions of the symbol was marked as default ('@@')): # perf probe perf probe -x /lib/powerpc64le-linux-gnu/libpthread.so.0 pthread_create PC ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: uprobes enable condition (user space) - sysfs interface 2017-12-04 14:55 ` Arnaldo Carvalho de Melo 2017-12-04 15:23 ` Paul Clarke @ 2017-12-05 6:51 ` Masami Hiramatsu 1 sibling, 0 replies; 11+ messages in thread From: Masami Hiramatsu @ 2017-12-05 6:51 UTC (permalink / raw) To: Arnaldo Carvalho de Melo; +Cc: bhargavb, linux-rt-users, linux-perf-users On Mon, 4 Dec 2017 11:55:58 -0300 Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > Em Mon, Dec 04, 2017 at 09:36:19PM +0900, Masami Hiramatsu escreveu: > > On Fri, 1 Dec 2017 13:40:39 -0300 > > Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > > > > > Adding Masami to the CC list, to see if he can come up with a fix for > > > the 'perf probe' failure below, improving the message or ignoring > > > versioned symbols (with a warning perhaps). > > > > Thanks. Should we ignore the versioned symbols? or can we put a probe > > on that? If we have symbol address, we can put a probe on it, even > > if it is versioned. > > Lemme try... > > [root@jouet ~]# perf probe -x /usr/lib64/libc-2.25.so malloc_get_state_225=malloc_get_state@GLIBC_2.2.5 > The /usr/lib64/libc-2.25.so file has no debug information. > Rebuild with -g, or install an appropriate debuginfo package. > Error: Failed to add events. > [root@jouet ~]# perf probe -x /usr/lib64/libc-2.25.so malloc_get_state@GLIBC_2.2.5 > The /usr/lib64/libc-2.25.so file has no debug information. > Rebuild with -g, or install an appropriate debuginfo package. > Error: Failed to add events. Ah, those cases, perf-probe parsed A@B as function@file... > [root@jouet ~]# perf probe -x /usr/lib64/libc-2.25.so malloc > Added new event: > probe_libc:malloc (on malloc in /usr/lib64/libc-2.25.so) > > You can now use it in all perf tools, such as: > > perf record -e probe_libc:malloc -aR sleep 1 > > [root@jouet ~]# > > So here the message is misleading, as we did manage to put a probe on > 'malloc', but not on the versioned symbol, both when trying to give it a > different name (malloc_get_state_225) and when doing it directly > (malloc_get_state@GLIBC_2.2.5). So, yes, the message is misleading. That should say something like "malloc_get_state@GLIBC_2.2.5 requires source file(GLIBC_2.2.5) information, but /usr/lib64/libc-2.25.so file has no debug information." [..] > > > [root@jouet ~]# perf probe -x /lib64/libc-2.25.so malloc* > > > Failed to write event: Invalid argument > > > Error: Failed to add events. > > > [root@jouet ~]# > > > > OK, this is not good, at least it should analyse the reason(versioned) > > and warn that (Or, I think we can just remove the suffix after "@"). > > I think it is a matter of replacing special characters with some other. > I think we should support versioned symbols, i.e. one may well want to > intercept calls to a specific version of a symbol. Then, there are 2 issues. Parser and event-name generator. Parser has to understand that the versioned event to find it from symbol-map (I have to check but versioned-suffix is not recorded in debuginfo). Also, event-name generator should handle that. (I think this part is easy to cut off the suffix) Anyway, I think that (tracing specific versioned symbol) is rare case, so for the parser we just need to introduce an "escape" backslash, like perf probe -x /usr/lib64/libc-2.25.so malloc_get_state\\@GLIBC_2.2.5 (note that bash/sh will consume the first '\', so we need '\\'.) This will allow user to specify version. [..] > > > > > > Masami, this one also doesn't work: > > > > > > [root@jouet ~]# perf probe -x /lib64/libc-2.25.so 'malloc*%return' > > > Error: event "malloc_printerr" already exists. > > > <SNIP> > > > > Hm, this looks like a collision happens on that event name. "-f"(--force) > > option allows you to add events with number suffix (e.g. _1, _2, ... ) > > Probably this was because the same name was used for both entry and > exit, see below on automatically adding __return to %return probes. > > > > > > > doing it a bit differently: > > > > > > [root@jouet ~]# perf probe -F -x /lib64/libc-2.25.so | egrep ^malloc[^@]+$ | while read function ; do perf probe -x /lib64/libc-2.25.so "${function}_return=$function%return" ; done > > > Added new event: > > > probe_libc:malloc_check_return (on malloc_check%return in /usr/lib64/libc-2.25.so) > > > > > > You can now use it in all perf tools, such as: > > > > > > perf record -e probe_libc:malloc_check_return -aR sleep 1 > > > > Hmm, would you think we should add __return suffix by default? > > I think so, as it is a common operation to ask for entry+exit. OK, I'll do that. Thank you, > > - Arnaldo > > > Thanks, > > > > > > > > <SNIP> > > > ----------- > > > > > > This now makes us have: > > > > > > [root@jouet ~]# perf probe -l > > > probe:check_stack_object (on check_stack_object%return@acme/git/linux/mm/usercopy.c with ret) > > > probe_libc:malloc (on __malloc in /usr/lib64/libc-2.25.so) > > > probe_libc:malloc_check (on malloc_check in /usr/lib64/libc-2.25.so) > > > probe_libc:malloc_consolidate (on malloc_consolidate.part.1 in /usr/lib64/libc-2.25.so) > > > probe_libc:malloc_hook_ini (on malloc_hook_ini in /usr/lib64/libc-2.25.so) > > > probe_libc:malloc_info (on __malloc_info in /usr/lib64/libc-2.25.so) > > > probe_libc:malloc_printerr (on malloc_printerr in /usr/lib64/libc-2.25.so) > > > probe_libc:malloc_stats (on __malloc_stats in /usr/lib64/libc-2.25.so) > > > probe_libc:malloc_trim (on __malloc_trim in /usr/lib64/libc-2.25.so) > > > probe_libc:malloc_usable_size (on __malloc_usable_size in /usr/lib64/libc-2.25.so) > > > probe_libc:mallochook (on mallochook in /usr/lib64/libc-2.25.so) > > > probe_libc:malloc_check_return (on malloc_check%return in /usr/lib64/libc-2.25.so) > > > probe_libc:malloc_hook_ini_return (on malloc_hook_ini%return in /usr/lib64/libc-2.25.so) > > > probe_libc:malloc_info_return (on __malloc_info%return in /usr/lib64/libc-2.25.so) > > > probe_libc:malloc_printerr_return (on malloc_printerr%return in /usr/lib64/libc-2.25.so) > > > probe_libc:malloc_stats_return (on __malloc_stats%return in /usr/lib64/libc-2.25.so) > > > probe_libc:malloc_trim_return (on __malloc_trim%return in /usr/lib64/libc-2.25.so) > > > probe_libc:malloc_usable_size_return (on __malloc_usable_size%return in /usr/lib64/libc-2.25.so) > > > probe_libc:mallochook_return (on mallochook%return in /usr/lib64/libc-2.25.so) > > > probe_libc:malloc_return (on __malloc%return in /usr/lib64/libc-2.25.so) > > > [root@jouet ~]# > > > > > > There were some usability snags that prevented this from being put in > > > place, so do it manually: > > > > > > [root@jouet ~]# perf probe -x /lib64/libc-2.25.so 'malloc_consolidate_return=malloc_consolidate.part.1%return' > > > Added new event: > > > probe_libc:malloc_consolidate_return (on malloc_consolidate.part.1%return in /usr/lib64/libc-2.25.so) > > > > > > You can now use it in all perf tools, such as: > > > > > > perf record -e probe_libc:malloc_consolidate_return -aR sleep 1 > > > > > > [root@jouet ~]# > > > > > > Ok, now we can trace this syswide using: > > > > > > [root@jouet ~]# perf trace --no-syscalls -e probe_libc:* > > > <BIG SNIP> > > > 339.557 probe_libc:malloc:(7fda55c22970)) > > > 339.562 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > > > 339.567 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > > 339.570 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > > > 339.596 probe_libc:malloc:(7fda55c22970)) > > > 339.600 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > > > 339.622 probe_libc:malloc:(7fda55c22970)) > > > 339.626 probe_libc:malloc_return:(7fda55c22970 <- 7fda56ba9a39)) > > > 339.631 probe_libc:malloc:(7fda55c22970)) > > > 339.634 probe_libc:malloc_return:(7fda55c22970 <- 7fda578f1713)) > > > 339.637 probe_libc:malloc:(7fda55c22970)) > > > 339.641 probe_libc:malloc_return:(7fda55c22970 <- 7fda578f14ae)) > > > 339.657 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > > 339.661 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1dbd6)) > > > 339.664 probe_libc:malloc:(7fda55c22970)) > > > 339.667 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > > > 339.670 probe_libc:malloc:(7fda55c22970)) > > > 339.673 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > > > 339.678 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > > 339.681 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > > > 339.685 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > > 339.688 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > > > 339.693 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > > 339.696 probe_libc:malloc_consolidate_return:(7fda55c1b7b0 <- 7fda55c1f886)) > > > 339.778 probe_libc:malloc:(7fda55c22970)) > > > 339.783 probe_libc:malloc_return:(7fda55c22970 <- 7fda55c23918)) > > > 339.788 probe_libc:malloc_consolidate:(7fda55c1b7b0)) > > > ^C(7fda55c22970 <- 7fda578f14ae)) > > > [root@jouet ~]# > > > > > > It works as well with 'perf record' + 'perf script'. > > > > > > Other features from perf can be added to the mix, like callchains, etc. > > > > > > But see below for doing it via tracefs > > > > > > > I observed below behaviour when I tried as per directions using URL: > > > > https://opensource.com/article/17/7/dynamic-tracing-linux-user-and-kernel-space#comment-136366 > > > > : > > > > > > > > > > > > echo 'p:<func_entry> ./test:0x420' > > > > > /sys/kernel/debug/tracing/uprobe_events (for creating > > > > uprobe) > > > > echo 1 > /sys/kernel/debug/tracing/events/enable > > > > (enabling uproble) > > > > There is another enable parameter: > > > > /sys/kernel/debug/tracing/events/uprobes/enable > > > > > > > > I understand unless enabled both events/enable and > > > > events/uprobes/enable, the /sys/kernel/debug/tracing/trace is not > > > > complete. I do not understand the relevance of > > > > .....events/uprobes/enable as I could not get any trace data in trace > > > > log for 'func_entry' the user space function being added for tracing. > > > > Irrespective of the probe function added, the trace log shows complete > > > > system trace log during the period of execution of the user > > > > application being traced. > > > > > > Is this what you want: > > > > > > [root@jouet ~]# cd /sys/kernel/debug/tracing/ > > > [root@jouet tracing]# cat available_tracers > > > hwlat blk mmiotrace function_graph wakeup_dl wakeup_rt wakeup function nop > > > [root@jouet tracing]# echo nop > current_tracer > > > [root@jouet tracing]# echo 1 > events/probe_libc/malloc/enable > > > [root@jouet tracing]# echo 1 > tracing_on > > > [root@jouet tracing]# tail trace > > > tail-1581 [002] d... 118484.507167: malloc: (0x7fdb32d67970) > > > tail-1581 [002] d... 118484.507172: malloc: (0x7fdb32d67970) > > > tail-1581 [002] d... 118484.507189: malloc: (0x7fdb32d67970) > > > tail-1581 [002] d... 118484.507193: malloc: (0x7fdb32d67970) > > > tail-1581 [002] d... 118484.507198: malloc: (0x7fdb32d67970) > > > tail-1581 [002] d... 118484.507203: malloc: (0x7fdb32d67970) > > > tail-1581 [002] d... 118484.507208: malloc: (0x7fdb32d67970) > > > tail-1581 [002] d... 118484.507214: malloc: (0x7fdb32d67970) > > > tail-1581 [002] d... 118484.507224: malloc: (0x7fdb32d67970) > > > tail-1581 [002] d... 118484.507237: malloc: (0x7fdb32d67970) > > > [root@jouet tracing]# > > > > > > ? > > > > > > > Can this be clearly mentioned as to when to use one of them, role of > > > > both and and also, I could not see uprobe function in particular at > > > > all but generic trace in /sys/kernel/debug/tracing/trace. Kindly guide > > > > me in understanding uprobe enable logic in user space in detail. > > > > > > > > Regards, > > > > Bhargav > > > > -- > > > > -- > > Masami Hiramatsu <mhiramat@kernel.org> -- Masami Hiramatsu <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2017-12-05 13:58 UTC | newest] Thread overview: 11+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-11-29 18:34 uprobes enable condition (user space) - sysfs interface bhargavb 2017-12-01 16:40 ` Arnaldo Carvalho de Melo 2017-12-03 17:14 ` bhargavb 2017-12-04 6:03 ` Namhyung Kim 2017-12-04 12:36 ` Masami Hiramatsu 2017-12-04 14:55 ` Arnaldo Carvalho de Melo 2017-12-04 15:23 ` Paul Clarke 2017-12-05 6:53 ` Masami Hiramatsu 2017-12-05 7:45 ` Masami Hiramatsu 2017-12-05 13:58 ` Paul Clarke 2017-12-05 6:51 ` Masami Hiramatsu
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).