Re: tprobe event tracing error

[Masami Hiramatsu (Google) <mhiramat@kernel.org>]

Hi,
(Cc: linux-kernel-trace ML for sharing this knowledge)

On Mon, 26 Feb 2024 16:36:29 +0300
Максим Морсков <xxxa0c@mail.ru> wrote:

> 
> Hello, dear Masami.
> I am researching Linux event tracing subsystem in part of tprobes,
> and found interesting behavior in kernel version 6.6:
>  
> echo  't:my_fchmodat sys_enter_fchmodat' | sudo tee ‘/sys/kernel/tracing/dynamic_events’
> bash: line 1: echo: write error: Invalid argument

Yeah, I understand that you are confused by this behavior, but it is
actually expected behavior. syscalls:* events looks like trace events
based on tracepoint, but those are software generated trace event.

You can find raw_syscalls:* trace events, that is based on the tracepoint,
and other syscalls:* are based on that raw_syscalls:* trace points.
(IOW, those are a kind of pre-compiled dynamic events)

e.g.

/sys/kernel/tracing # echo "t sys_enter \$arg*" >> dynamic_events 
/sys/kernel/tracing # cat dynamic_events 
t:tracepoints/sys_enter sys_enter regs=regs id=id

/sys/kernel/tracing # echo "t sys_enter_open \$arg*" >> dynamic_events 
sh: write error: Invalid argument
/sys/kernel/tracing # cat error_log 
[  227.981347] trace_fprobe: error: Tracepoint is not found
  Command: t sys_enter_open $arg*
             ^

So, tprobe can not find the hard-coded tracepoints for those dynamically
generated syscall trace events. But raw_syscall sys_enter/sys_exit are OK.

Thank you,

>  
> sudo cat /sys/kernel/tracing/error_log
> trace_fprobe: error: Tracepoint is not found 
>  Command: t:my_fchmodat sys_enter_fchmodat
>                                                     ^
>  
> But such tracepoint ( sys_enter_fchmodat ) exists in system:
> sudo perf list | grep chmod   
>  syscalls:sys_enter_ chmod                           [Tracepoint event] 
>  syscalls:sys_enter_f chmod                          [Tracepoint event] 
>  syscalls:sys_enter_f chmod at                        [Tracepoint event] 
>  syscalls:sys_enter_f chmod at2                       [Tracepoint event] 
>  syscalls:sys_exit_ chmod                            [Tracepoint event] 
>  syscalls:sys_exit_f chmod                           [Tracepoint event] 
>  syscalls:sys_exit_f chmod at                         [Tracepoint event] 
>  syscalls:sys_exit_f chmod at2                        [Tracepoint event]
>  
> sudo ls -lha /sys/kernel/tracing/events/syscalls/ | grep chmod
> drwxr-xr-x 1 root root 0 фев 26 16:07 sys_enter_ chmod
> drwxr-xr-x 1 root root 0 фев 26 16:07 sys_enter_f chmod
> drwxr-xr-x 1 root root 0 фев 26 16:07 sys_enter_f chmod at 
> drwxr-xr-x 1 root root 0 фев 26 16:07 sys_enter_f chmod at2 
> drwxr-xr-x 1 root root 0 фев 26 16:07 sys_exit_ chmod
> drwxr-xr-x 1 root root 0 фев 26 16:07 sys_exit_f chmod
> drwxr-xr-x 1 root root 0 фев 26 16:07 sys_exit_f chmod at 
> drwxr-xr-x 1 root root 0 фев 26 16:07 sys_exit_f chmod at2
>  
> My kernel config in the attachment.
> I noticed that you are developing event tracing tprobe subsystem.
> May be you could explain such behavior?
> Thank you in advance!
>  
> --
> Best regards
> Maksim Morskov

-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>