kprobe on local function

kprobe on local function

[Nam Cao]

Hi,

I installed a kretprobe on the function "rt_mutex_slowunlock". Although the
function is called, the probe is never hit.

The reason is because there are 3 copies of rt_mutex_slowunlock() in
vmlinux (at least for x86 defconfig + CONFIG_PREEMPT_RT). But probe point
is only installed in 1 copy.

Is this considered a bug? Or is kprobe not intended to be used for local
function? Should we modify kprobe to support such cases?

Best regards,
Nam

Re: kprobe on local function

[Steven Rostedt]

On Wed, 22 Jan 2025 08:59:39 +0100
Nam Cao <namcao@linutronix.de> wrote:

> Hi,
> 
> I installed a kretprobe on the function "rt_mutex_slowunlock". Although the
> function is called, the probe is never hit.
> 

How did you install the kretprobe?

> The reason is because there are 3 copies of rt_mutex_slowunlock() in
> vmlinux (at least for x86 defconfig + CONFIG_PREEMPT_RT). But probe point
> is only installed in 1 copy.
> 
> Is this considered a bug? Or is kprobe not intended to be used for local
> function? Should we modify kprobe to support such cases?

It's not a bug, but just the way the compiler works.

Have you tried "perf probe" to install the probes? That uses dwarf info to
find all the functions.

-- Steve

Re: kprobe on local function

[Masami Hiramatsu]

On Wed, 22 Jan 2025 17:29:36 -0500
Steven Rostedt <rostedt@goodmis.org> wrote:

> On Wed, 22 Jan 2025 08:59:39 +0100
> Nam Cao <namcao@linutronix.de> wrote:
> 
> > Hi,
> > 
> > I installed a kretprobe on the function "rt_mutex_slowunlock". Although the
> > function is called, the probe is never hit.
> > 
> 
> How did you install the kretprobe?
> 

If you specify one original 'symbol' name to tracefs/kprobe_events, it
only sets the kretprobe event on that symbol.


> > The reason is because there are 3 copies of rt_mutex_slowunlock() in
> > vmlinux (at least for x86 defconfig + CONFIG_PREEMPT_RT). But probe point
> > is only installed in 1 copy.

You can add those copies manually on the same event but,...

> > 
> > Is this considered a bug? Or is kprobe not intended to be used for local
> > function? Should we modify kprobe to support such cases?
> 
> It's not a bug, but just the way the compiler works.
> 
> Have you tried "perf probe" to install the probes? That uses dwarf info to
> find all the functions.

Yes, I recommend you to use `perf probe` (under tools/perf) to put kretprobes
on those copies. See tools/perf/Documentation/perf-probe.txt for details.


Thank you,

> 
> -- Steve


-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: kprobe on local function

[Nam Cao]

On Thu, Jan 23, 2025 at 10:48:42AM +0900, Masami Hiramatsu wrote:
> On Wed, 22 Jan 2025 17:29:36 -0500
> Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > On Wed, 22 Jan 2025 08:59:39 +0100
> > Nam Cao <namcao@linutronix.de> wrote:
> > > I installed a kretprobe on the function "rt_mutex_slowunlock". Although the
> > > function is called, the probe is never hit.
> > > 
> > 
> > How did you install the kretprobe?

Sorry for being unclear. I install it from within the kernel:

	struct kretprobe mutex_event = {
		.kp.symbol_name = "rt_mutex_slowunlock",
		.entry_handler = entry_handler,
		.handler = return_handler,
	};

	register_kretprobe(&mutex_event);

...
> > Have you tried "perf probe" to install the probes? That uses dwarf info to
> > find all the functions.
> 
> Yes, I recommend you to use `perf probe` (under tools/perf) to put kretprobes
> on those copies. See tools/perf/Documentation/perf-probe.txt for details.

So a similar solution for in-kernel code is kallsyms_on_each_match_symbol().
But I expect register_kretprobe() to do that by itself; or at least report
a failure if caller specifies the symbol name, but multiple symbols of the
same name are found. Because at least to me, my code "looks correct and
should work" but it doesn't.

Thank you both for your replies,
Nam

Re: kprobe on local function

[Steven Rostedt]

On Thu, 23 Jan 2025 03:42:24 +0100
Nam Cao <namcao@linutronix.de> wrote:

> So a similar solution for in-kernel code is kallsyms_on_each_match_symbol().
> But I expect register_kretprobe() to do that by itself; or at least report
> a failure if caller specifies the symbol name, but multiple symbols of the
> same name are found. Because at least to me, my code "looks correct and
> should work" but it doesn't.

I guess that could be an enhancement to the code.

Masami?

-- Steve

Re: kprobe on local function

[Masami Hiramatsu]

On Thu, 23 Jan 2025 03:42:24 +0100
Nam Cao <namcao@linutronix.de> wrote:

> On Thu, Jan 23, 2025 at 10:48:42AM +0900, Masami Hiramatsu wrote:
> > On Wed, 22 Jan 2025 17:29:36 -0500
> > Steven Rostedt <rostedt@goodmis.org> wrote:
> > 
> > > On Wed, 22 Jan 2025 08:59:39 +0100
> > > Nam Cao <namcao@linutronix.de> wrote:
> > > > I installed a kretprobe on the function "rt_mutex_slowunlock". Although the
> > > > function is called, the probe is never hit.
> > > > 
> > > 
> > > How did you install the kretprobe?
> 
> Sorry for being unclear. I install it from within the kernel:
> 
> 	struct kretprobe mutex_event = {
> 		.kp.symbol_name = "rt_mutex_slowunlock",
> 		.entry_handler = entry_handler,
> 		.handler = return_handler,
> 	};
> 
> 	register_kretprobe(&mutex_event);

This is using symbol name to install kretprobe. Thus if there
are rt_mutex_slowunlock.XXX symbols are generated by the compiler,
it can not install kretprobe to those symbols.

> 
> ...
> > > Have you tried "perf probe" to install the probes? That uses dwarf info to
> > > find all the functions.
> > 
> > Yes, I recommend you to use `perf probe` (under tools/perf) to put kretprobes
> > on those copies. See tools/perf/Documentation/perf-probe.txt for details.
> 
> So a similar solution for in-kernel code is kallsyms_on_each_match_symbol().
> But I expect register_kretprobe() to do that by itself; or at least report
> a failure if caller specifies the symbol name, but multiple symbols of the
> same name are found. Because at least to me, my code "looks correct and
> should work" but it doesn't.

Now what you could know is that the kernel does not call the
"rt_mutex_slowunlock" symbol and it may call another copied symbol. That's
a good to know right?

Note that this in-kernel kretprobe API is a low-layer API, so that user is
responsible for controlling it carefully. This kretprobe API itself does NOT
know which copy of the optimized symbols are used.
(there could be inlined instance too)

If you need to know how many copies are generated or inlined, you need to
use the debuginfo as "perf probe" tool does, which can not handled by the
kernel itself because it is too big to be loaded. That is why I made the
"perf probe" as an external tool.

Thank you,

> 
> Thank you both for your replies,
> Nam


-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: kprobe on local function

[Nam Cao]

On Sat, Jan 25, 2025 at 01:16:55AM +0900, Masami Hiramatsu wrote:
> Now what you could know is that the kernel does not call the
> "rt_mutex_slowunlock" symbol and it may call another copied symbol. That's
> a good to know right?
> 
> Note that this in-kernel kretprobe API is a low-layer API, so that user is
> responsible for controlling it carefully. This kretprobe API itself does NOT
> know which copy of the optimized symbols are used.
> (there could be inlined instance too)
> 
> If you need to know how many copies are generated or inlined, you need to
> use the debuginfo as "perf probe" tool does, which can not handled by the
> kernel itself because it is too big to be loaded. That is why I made the
> "perf probe" as an external tool.

Got it, thanks so much for the information!

Best regards,
Nam