Re: kprobe not always triggering

[Masami Hiramatsu (Google) <mhiramat@kernel.org>]

On Thu, 16 Jan 2025 13:04:18 +0100
Daniel Wagner <dwagner@suse.de> wrote:

> On Thu, Jan 16, 2025 at 08:48:18AM +0900, Masami Hiramatsu wrote:
> > Have you tried to use `perf probe`? Even if it is inlined, you can find them
> > by
> > 
> >  perf probe -a nvme_change_ctrl_state new_state
> 
> I had to build my own perf, as the version from the distro didn't like
> my kernel and just crashed. Thus I added a bunch libraries to my build
> system. The resulting perf installed the probe and the missing events
> show up:
> 
>           <idle>-0       [000] ..s1.   245.406139: nvme_complete_rq: nvme1: qid=0, cmdid=36865, res=0x0, retries=0, flags=0x0, status=0x371
>           <idle>-0       [000] .Ns1.   245.407192: nvme_change_ctrl_state: (nvme_change_ctrl_state+0x0/0x180 [nvme_core]) new_state=0x2
>    kworker/u33:0-65      [000] .N...   245.407258: nvme_change_ctrl_state: (nvme_change_ctrl_state+0x0/0x180 [nvme_core]) new_state=0x2
>    kworker/u33:1-78      [000] .....   245.421290: nvme_change_ctrl_state: (nvme_change_ctrl_state+0x0/0x180 [nvme_core]) new_state=0x3
> 

Nice.

> I double checked my previous setup with kprobes and now the missing
> events show up as well. It looks like my setup changes makes clang to
> not inline this function anymore, or there is no additional debug info
> available?

Hmm, I don't think (don't hope) the clang change the optimization
silently if debuginfo is enabled. This is a kind of heisenbug :(

> 
> > But perhaps, some of them are inlined with constant numbers (maybe you can
> > find them as nvme_change_ctrl_state.constprop.XXX in kallsyms)
> 
> # cat /proc/kallsyms | grep nvme_change_ctrl_state
> ffffffffc03dc04d r __kstrtab_nvme_change_ctrl_state     [nvme_core]
> ffffffffc03dc064 r __kstrtabns_nvme_change_ctrl_state   [nvme_core]
> ffffffffc03ca130 r __ksymtab_nvme_change_ctrl_state     [nvme_core]
> ffffffffc0405de0 t __pfx_nvme_change_ctrl_state [nvme_core]
> ffffffffc0405df0 t nvme_change_ctrl_state       [nvme_core]
> 
> This is with the new kernel, so after installing the libraries for the
> perf build.

IIRC, constprop.XXX is gcc's suffix, clang has different one.

> 
> > If my guess is correct, it is not inlined but the constant propagation
> > optimization[1] has been done by the compiler.
> > 
> > [1] https://en.wikipedia.org/wiki/Constant_folding#Constant_propagation
> > 
> > IIRC, commit 66f69b219716 ("perf probe: Support DW_AT_const_value constant value")
> > should support this case.
> 
> Okay, this commit is already in (v5.4), so it was always in.
> 
> Anyway, I can't reproduce anymore.

Feel free to notice if you see similar issue.

Thank you,

> 
> Daniel


-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>