[PATCH 0/2] tracing: Add a way to filter function addresses to function names

[PATCH 0/2] tracing: Add a way to filter function addresses to function names

[Steven Rostedt]

There's been several times where an event records a function address in
its field and I needed to filter on that address for a specific function
name. It required looking up the function in kallsyms, finding its size,
and doing a compare of "field >= function_start && field < function_end".

But this would change from boot to boot and is unreliable in scripts.
Also, it is useful to have this at boot up, where the addresses will not
be known. For example, on the boot command line:

  trace_trigger="initcall_finish.traceoff if func.function == acpi_init"

To implement this, add a ".function" prefix, that will check that the
field is of size long, and the only operations allowed (so far) are "=="
and "!=".

Changes since v1: https://lore.kernel.org/linux-trace-kernel/20221214125209.09d736dd@gandalf.local.home/

 - Fix commit log "initcall_finish.function" to "func.function" (Ross Zwisler)

 - Fixed processing of address (Ross Zwisler)

 - Added selftest (Masami Hiramatsu)

 - Just use "trigger" instead of "strsep(&trigger, "")" (Zheng Yejian)

Steven Rostedt (Google) (2):
      tracing: Add a way to filter function addresses to function names
      tracing/selftests: Add test for event filtering on function name

----
 Documentation/trace/events.rst                     | 12 +++
 kernel/trace/trace_events.c                        |  2 +-
 kernel/trace/trace_events_filter.c                 | 93 +++++++++++++++++++++-
 .../ftrace/test.d/filter/event-filter-function.tc  | 58 ++++++++++++++
 4 files changed, 163 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/ftrace/test.d/filter/event-filter-function.tc

[PATCH 1/2] tracing: Add a way to filter function addresses to function names

[Steven Rostedt]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

There's been several times where an event records a function address in
its field and I needed to filter on that address for a specific function
name. It required looking up the function in kallsyms, finding its size,
and doing a compare of "field >= function_start && field < function_end".

But this would change from boot to boot and is unreliable in scripts.
Also, it is useful to have this at boot up, where the addresses will not
be known. For example, on the boot command line:

  trace_trigger="initcall_finish.traceoff if func.function == acpi_init"

To implement this, add a ".function" prefix, that will check that the
field is of size long, and the only operations allowed (so far) are "=="
and "!=".

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 Documentation/trace/events.rst     | 12 ++++
 kernel/trace/trace_events.c        |  2 +-
 kernel/trace/trace_events_filter.c | 93 +++++++++++++++++++++++++++++-
 3 files changed, 105 insertions(+), 2 deletions(-)

diff --git a/Documentation/trace/events.rst b/Documentation/trace/events.rst
index c47f381d0c00..d0fd5c7220b7 100644
--- a/Documentation/trace/events.rst
+++ b/Documentation/trace/events.rst
@@ -207,6 +207,18 @@ field name::
 As the kernel will have to know how to retrieve the memory that the pointer
 is at from user space.
 
+You can convert any long type to a function address and search by function name::
+
+  call_site.function == security_prepare_creds
+
+The above will filter when the field "call_site" falls on the address within
+"security_prepare_creds". That is, it will compare the value of "call_site" and
+the filter will return true if it is greater than or equal to the start of
+the function "security_prepare_creds" and less than the end of that function.
+
+The ".function" postfix can only be attached to values of size long, and can only
+be compared with "==" or "!=".
+
 5.2 Setting filters
 -------------------
 
diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c
index 33e0b4f8ebe6..b07931ad97de 100644
--- a/kernel/trace/trace_events.c
+++ b/kernel/trace/trace_events.c
@@ -2822,7 +2822,7 @@ static __init int setup_trace_triggers(char *str)
 		if (!trigger)
 			break;
 		bootup_triggers[i].event = strsep(&trigger, ".");
-		bootup_triggers[i].trigger = strsep(&trigger, ".");
+		bootup_triggers[i].trigger = trigger;
 		if (!bootup_triggers[i].trigger)
 			break;
 	}
diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c
index 96acc2b71ac7..996b3c458c1a 100644
--- a/kernel/trace/trace_events_filter.c
+++ b/kernel/trace/trace_events_filter.c
@@ -64,6 +64,7 @@ enum filter_pred_fn {
 	FILTER_PRED_FN_PCHAR_USER,
 	FILTER_PRED_FN_PCHAR,
 	FILTER_PRED_FN_CPU,
+	FILTER_PRED_FN_FUNCTION,
 	FILTER_PRED_FN_,
 	FILTER_PRED_TEST_VISITED,
 };
@@ -71,6 +72,7 @@ enum filter_pred_fn {
 struct filter_pred {
 	enum filter_pred_fn 	fn_num;
 	u64 			val;
+	u64 			val2;
 	struct regex		regex;
 	unsigned short		*ops;
 	struct ftrace_event_field *field;
@@ -103,6 +105,7 @@ struct filter_pred {
 	C(INVALID_FILTER,	"Meaningless filter expression"),	\
 	C(IP_FIELD_ONLY,	"Only 'ip' field is supported for function trace"), \
 	C(INVALID_VALUE,	"Invalid value (did you forget quotes)?"), \
+	C(NO_FUNCTION,		"Function not found"),			\
 	C(ERRNO,		"Error"),				\
 	C(NO_FILTER,		"No filter found")
 
@@ -876,6 +879,17 @@ static int filter_pred_comm(struct filter_pred *pred, void *event)
 	return cmp ^ pred->not;
 }
 
+/* Filter predicate for functions. */
+static int filter_pred_function(struct filter_pred *pred, void *event)
+{
+	unsigned long *addr = (unsigned long *)(event + pred->offset);
+	unsigned long start = (unsigned long)pred->val;
+	unsigned long end = (unsigned long)pred->val2;
+	int ret = *addr >= start && *addr < end;
+
+	return pred->op == OP_EQ ? ret : !ret;
+}
+
 /*
  * regex_match_foo - Basic regex callbacks
  *
@@ -1335,6 +1349,8 @@ static int filter_pred_fn_call(struct filter_pred *pred, void *event)
 		return filter_pred_pchar(pred, event);
 	case FILTER_PRED_FN_CPU:
 		return filter_pred_cpu(pred, event);
+	case FILTER_PRED_FN_FUNCTION:
+		return filter_pred_function(pred, event);
 	case FILTER_PRED_TEST_VISITED:
 		return test_pred_visited_fn(pred, event);
 	default:
@@ -1350,8 +1366,13 @@ static int parse_pred(const char *str, void *data,
 	struct trace_event_call *call = data;
 	struct ftrace_event_field *field;
 	struct filter_pred *pred = NULL;
+	unsigned long offset;
+	unsigned long size;
+	unsigned long ip;
 	char num_buf[24];	/* Big enough to hold an address */
 	char *field_name;
+	char *name;
+	bool function = false;
 	bool ustring = false;
 	char q;
 	u64 val;
@@ -1393,6 +1414,12 @@ static int parse_pred(const char *str, void *data,
 		i += len;
 	}
 
+	/* See if the field is a kernel function name */
+	if ((len = str_has_prefix(str + i, ".function"))) {
+		function = true;
+		i += len;
+	}
+
 	while (isspace(str[i]))
 		i++;
 
@@ -1423,7 +1450,71 @@ static int parse_pred(const char *str, void *data,
 	pred->offset = field->offset;
 	pred->op = op;
 
-	if (ftrace_event_is_function(call)) {
+	if (function) {
+		/* The field must be the same size as long */
+		if (field->size != sizeof(long)) {
+			parse_error(pe, FILT_ERR_ILLEGAL_FIELD_OP, pos + i);
+			goto err_free;
+		}
+
+		/* Function only works with '==' or '!=' and an unquoted string */
+		switch (op) {
+		case OP_NE:
+		case OP_EQ:
+			break;
+		default:
+			parse_error(pe, FILT_ERR_INVALID_OP, pos + i);
+			goto err_free;
+		}
+
+		if (isdigit(str[i])) {
+			/* We allow 0xDEADBEEF */
+			while (isalnum(str[i]))
+				i++;
+
+			len = i - s;
+			/* 0xfeedfacedeadbeef is 18 chars max */
+			if (len >= sizeof(num_buf)) {
+				parse_error(pe, FILT_ERR_OPERAND_TOO_LONG, pos + i);
+				goto err_free;
+			}
+
+			strncpy(num_buf, str + s, len);
+			num_buf[len] = 0;
+
+			ret = kstrtoul(num_buf, 0, &ip);
+			if (ret) {
+				parse_error(pe, FILT_ERR_INVALID_VALUE, pos + i);
+				goto err_free;
+			}
+		} else {
+			s = i;
+			for (; str[i] && !isspace(str[i]); i++)
+				;
+
+			len = i - s;
+			name = kmemdup_nul(str + s, len, GFP_KERNEL);
+			if (!name)
+				goto err_mem;
+			ip = kallsyms_lookup_name(name);
+			kfree(name);
+			if (!ip) {
+				parse_error(pe, FILT_ERR_NO_FUNCTION, pos + i);
+				goto err_free;
+			}
+		}
+
+		/* Now find the function start and end address */
+		if (!kallsyms_lookup_size_offset(ip, &size, &offset)) {
+			parse_error(pe, FILT_ERR_NO_FUNCTION, pos + i);
+			goto err_free;
+		}
+
+		pred->fn_num = FILTER_PRED_FN_FUNCTION;
+		pred->val = ip - offset;
+		pred->val2 = pred->val + size;
+
+	} else if (ftrace_event_is_function(call)) {
 		/*
 		 * Perf does things different with function events.
 		 * It only allows an "ip" field, and expects a string.
-- 
2.35.1

Re: [PATCH 1/2] tracing: Add a way to filter function addresses to function names

[Ross Zwisler]

On Mon, Dec 19, 2022 at 01:31:07PM -0500, Steven Rostedt wrote:
> From: "Steven Rostedt (Google)" <rostedt@goodmis.org>
> 
> There's been several times where an event records a function address in
> its field and I needed to filter on that address for a specific function
> name. It required looking up the function in kallsyms, finding its size,
> and doing a compare of "field >= function_start && field < function_end".
> 
> But this would change from boot to boot and is unreliable in scripts.
> Also, it is useful to have this at boot up, where the addresses will not
> be known. For example, on the boot command line:
> 
>   trace_trigger="initcall_finish.traceoff if func.function == acpi_init"
> 
> To implement this, add a ".function" prefix, that will check that the
> field is of size long, and the only operations allowed (so far) are "=="
> and "!=".
> 
> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>

Reviewed-by: Ross Zwisler <zwisler@google.com>

[PATCH 2/2] tracing/selftests: Add test for event filtering on function name

[Steven Rostedt]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

With the new filter logic of passing in the name of a function to match an
instruction pointer (or the address of the function), add a test to make
sure that it is functional.

This is also the first test to test plain filtering. The filtering has
been tested via the trigger logic, which uses the same code, but there was
nothing to test just the event filter, so this test is the first to add
such a case.

Cc: Shuah Khan <shuah@kernel.org>
Cc: Shuah Khan <skhan@linuxfoundation.org>
Cc: linux-kselftest@vger.kernel.org
Suggested-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 .../test.d/filter/event-filter-function.tc    | 58 +++++++++++++++++++
 1 file changed, 58 insertions(+)
 create mode 100644 tools/testing/selftests/ftrace/test.d/filter/event-filter-function.tc

diff --git a/tools/testing/selftests/ftrace/test.d/filter/event-filter-function.tc b/tools/testing/selftests/ftrace/test.d/filter/event-filter-function.tc
new file mode 100644
index 000000000000..e2ff3bf4df80
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/filter/event-filter-function.tc
@@ -0,0 +1,58 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# description: event filter function - test event filtering on functions
+# requires: set_event events/kmem/kmem_cache_free/filter
+# flags: instance
+
+fail() { #msg
+    echo $1
+    exit_fail
+}
+
+echo "Test event filter function name"
+echo 0 > tracing_on
+echo 0 > events/enable
+echo > trace
+echo 'call_site.function == exit_mmap' > events/kmem/kmem_cache_free/filter
+echo 1 > events/kmem/kmem_cache_free/enable
+echo 1 > tracing_on
+ls > /dev/null
+echo 0 > events/kmem/kmem_cache_free/enable
+
+hitcnt=`grep kmem_cache_free trace| grep exit_mmap | wc -l`
+misscnt=`grep kmem_cache_free trace| grep -v exit_mmap | wc -l`
+
+if [ $hitcnt -eq 0 ]; then
+	exit_fail
+fi
+
+if [ $misscnt -gt 0 ]; then
+	exit_fail
+fi
+
+address=`grep ' exit_mmap$' /proc/kallsyms | cut -d' ' -f1`
+
+echo "Test event filter function address"
+echo 0 > tracing_on
+echo 0 > events/enable
+echo > trace
+echo "call_site.function == 0x$address" > events/kmem/kmem_cache_free/filter
+echo 1 > events/kmem/kmem_cache_free/enable
+echo 1 > tracing_on
+sleep 1
+echo 0 > events/kmem/kmem_cache_free/enable
+
+hitcnt=`grep kmem_cache_free trace| grep exit_mmap | wc -l`
+misscnt=`grep kmem_cache_free trace| grep -v exit_mmap | wc -l`
+
+if [ $hitcnt -eq 0 ]; then
+	exit_fail
+fi
+
+if [ $misscnt -gt 0 ]; then
+	exit_fail
+fi
+
+reset_events_filter
+
+exit 0
-- 
2.35.1

Re: [PATCH 2/2] tracing/selftests: Add test for event filtering on function name

[Ross Zwisler]

On Mon, Dec 19, 2022 at 01:31:08PM -0500, Steven Rostedt wrote:
> From: "Steven Rostedt (Google)" <rostedt@goodmis.org>
> 
> With the new filter logic of passing in the name of a function to match an
> instruction pointer (or the address of the function), add a test to make
> sure that it is functional.
> 
> This is also the first test to test plain filtering. The filtering has
> been tested via the trigger logic, which uses the same code, but there was
> nothing to test just the event filter, so this test is the first to add
> such a case.
> 
> Cc: Shuah Khan <shuah@kernel.org>
> Cc: Shuah Khan <skhan@linuxfoundation.org>
> Cc: linux-kselftest@vger.kernel.org
> Suggested-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>

Reviewed-by: Ross Zwisler <zwisler@google.com>

Re: [PATCH 2/2] tracing/selftests: Add test for event filtering on function name

[Shuah Khan]

On 12/19/22 13:57, Ross Zwisler wrote:
> On Mon, Dec 19, 2022 at 01:31:08PM -0500, Steven Rostedt wrote:
>> From: "Steven Rostedt (Google)" <rostedt@goodmis.org>
>>
>> With the new filter logic of passing in the name of a function to match an
>> instruction pointer (or the address of the function), add a test to make
>> sure that it is functional.
>>
>> This is also the first test to test plain filtering. The filtering has
>> been tested via the trigger logic, which uses the same code, but there was
>> nothing to test just the event filter, so this test is the first to add
>> such a case.
>>
>> Cc: Shuah Khan <shuah@kernel.org>
>> Cc: Shuah Khan <skhan@linuxfoundation.org>
>> Cc: linux-kselftest@vger.kernel.org
>> Suggested-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
>> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
> 
> Reviewed-by: Ross Zwisler <zwisler@google.com>

Thank you both. I will apply this after rc1 comes out.

thanks,
-- Shuah

Re: [PATCH 2/2] tracing/selftests: Add test for event filtering on function name

[Steven Rostedt]

On Mon, 19 Dec 2022 15:11:39 -0700
Shuah Khan <skhan@linuxfoundation.org> wrote:

> On 12/19/22 13:57, Ross Zwisler wrote:
> > On Mon, Dec 19, 2022 at 01:31:08PM -0500, Steven Rostedt wrote:  
> >> From: "Steven Rostedt (Google)" <rostedt@goodmis.org>
> >>
> >> With the new filter logic of passing in the name of a function to match an
> >> instruction pointer (or the address of the function), add a test to make
> >> sure that it is functional.
> >>
> >> This is also the first test to test plain filtering. The filtering has
> >> been tested via the trigger logic, which uses the same code, but there was
> >> nothing to test just the event filter, so this test is the first to add
> >> such a case.
> >>
> >> Cc: Shuah Khan <shuah@kernel.org>
> >> Cc: Shuah Khan <skhan@linuxfoundation.org>
> >> Cc: linux-kselftest@vger.kernel.org
> >> Suggested-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
> >> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>  
> > 
> > Reviewed-by: Ross Zwisler <zwisler@google.com>  
> 
> Thank you both. I will apply this after rc1 comes out.

It's dependent on the first patch.

-- Steve

Re: [PATCH 2/2] tracing/selftests: Add test for event filtering on function name

[Shuah Khan]

On 12/19/22 15:35, Steven Rostedt wrote:
> On Mon, 19 Dec 2022 15:11:39 -0700
> Shuah Khan <skhan@linuxfoundation.org> wrote:
> 
>> On 12/19/22 13:57, Ross Zwisler wrote:
>>> On Mon, Dec 19, 2022 at 01:31:08PM -0500, Steven Rostedt wrote:
>>>> From: "Steven Rostedt (Google)" <rostedt@goodmis.org>
>>>>
>>>> With the new filter logic of passing in the name of a function to match an
>>>> instruction pointer (or the address of the function), add a test to make
>>>> sure that it is functional.
>>>>
>>>> This is also the first test to test plain filtering. The filtering has
>>>> been tested via the trigger logic, which uses the same code, but there was
>>>> nothing to test just the event filter, so this test is the first to add
>>>> such a case.
>>>>
>>>> Cc: Shuah Khan <shuah@kernel.org>
>>>> Cc: Shuah Khan <skhan@linuxfoundation.org>
>>>> Cc: linux-kselftest@vger.kernel.org
>>>> Suggested-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
>>>> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
>>>
>>> Reviewed-by: Ross Zwisler <zwisler@google.com>
>>
>> Thank you both. I will apply this after rc1 comes out.
> 
> It's dependent on the first patch.
> 

In which case,

Acked-by: Shuah Khan <skhan@linuxfoundation.org>

thanks,
-- Shuah

Re: [PATCH 2/2] tracing/selftests: Add test for event filtering on function name

[Steven Rostedt]

On Mon, 19 Dec 2022 17:20:39 -0700
Shuah Khan <skhan@linuxfoundation.org> wrote:

> On 12/19/22 15:35, Steven Rostedt wrote:
> > On Mon, 19 Dec 2022 15:11:39 -0700
> > Shuah Khan <skhan@linuxfoundation.org> wrote:
> >   
> >> On 12/19/22 13:57, Ross Zwisler wrote:  
> >>> On Mon, Dec 19, 2022 at 01:31:08PM -0500, Steven Rostedt wrote:  
> >>>> From: "Steven Rostedt (Google)" <rostedt@goodmis.org>
> >>>>
> >>>> With the new filter logic of passing in the name of a function to match an
> >>>> instruction pointer (or the address of the function), add a test to make
> >>>> sure that it is functional.
> >>>>
> >>>> This is also the first test to test plain filtering. The filtering has
> >>>> been tested via the trigger logic, which uses the same code, but there was
> >>>> nothing to test just the event filter, so this test is the first to add
> >>>> such a case.
> >>>>
> >>>> Cc: Shuah Khan <shuah@kernel.org>
> >>>> Cc: Shuah Khan <skhan@linuxfoundation.org>
> >>>> Cc: linux-kselftest@vger.kernel.org
> >>>> Suggested-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
> >>>> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>  
> >>>
> >>> Reviewed-by: Ross Zwisler <zwisler@google.com>  
> >>
> >> Thank you both. I will apply this after rc1 comes out.  
> > 
> > It's dependent on the first patch.
> >   
> 
> In which case,
> 
> Acked-by: Shuah Khan <skhan@linuxfoundation.org>
>

Thanks Shuah!

-- Steve