[PATCH v2 0/2] selftests: livepatch: test if ftrace can trace a livepatched function

[PATCH v2 0/2] selftests: livepatch: test if ftrace can trace a livepatched function

[Filipe Xavier]

This patchset add ftrace helpers functions and
add a new test makes sure that ftrace can trace
a function that was introduced by a livepatch.

Signed-off-by: Filipe Xavier <felipeaggger@gmail.com>
Suggested-by: Marcos Paulo de Souza <mpdesouza@suse.com>
Reviewed-by: Marcos Paulo de Souza <mpdesouza@suse.com>
---
Changes in v2:
- functions.sh: change check traced function to accept a list of functions.
- Link to v1: https://lore.kernel.org/r/20250306-ftrace-sftest-livepatch-v1-0-a6f1dfc30e17@gmail.com

---
Filipe Xavier (2):
      selftests: livepatch: add new ftrace helpers functions
      selftests: livepatch: test if ftrace can trace a livepatched function

 tools/testing/selftests/livepatch/functions.sh   | 49 ++++++++++++++++++++++++
 tools/testing/selftests/livepatch/test-ftrace.sh | 34 ++++++++++++++++
 2 files changed, 83 insertions(+)
---
base-commit: 848e076317446f9c663771ddec142d7c2eb4cb43
change-id: 20250306-ftrace-sftest-livepatch-60d9dc472235

Best regards,
-- 
Filipe Xavier <felipeaggger@gmail.com>

[PATCH v2 1/2] selftests: livepatch: add new ftrace helpers functions

[Filipe Xavier]

Add new ftrace helpers functions cleanup_tracing, trace_function and
check_traced_functions.

Signed-off-by: Filipe Xavier <felipeaggger@gmail.com>
---
 tools/testing/selftests/livepatch/functions.sh | 49 ++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/tools/testing/selftests/livepatch/functions.sh b/tools/testing/selftests/livepatch/functions.sh
index 15601402dee6567837c2c49ba342eb357e410d18..dea9cc10a3f09662c57c2593cff49423302c8a5c 100644
--- a/tools/testing/selftests/livepatch/functions.sh
+++ b/tools/testing/selftests/livepatch/functions.sh
@@ -10,6 +10,7 @@ SYSFS_KERNEL_DIR="/sys/kernel"
 SYSFS_KLP_DIR="$SYSFS_KERNEL_DIR/livepatch"
 SYSFS_DEBUG_DIR="$SYSFS_KERNEL_DIR/debug"
 SYSFS_KPROBES_DIR="$SYSFS_DEBUG_DIR/kprobes"
+SYSFS_TRACING_DIR="$SYSFS_DEBUG_DIR/tracing"
 
 # Kselftest framework requirement - SKIP code is 4
 ksft_skip=4
@@ -62,6 +63,9 @@ function push_config() {
 			awk -F'[: ]' '{print "file " $1 " line " $2 " " $4}')
 	FTRACE_ENABLED=$(sysctl --values kernel.ftrace_enabled)
 	KPROBE_ENABLED=$(cat "$SYSFS_KPROBES_DIR/enabled")
+	TRACING_ON=$(cat "$SYSFS_TRACING_DIR/tracing_on")
+	CURRENT_TRACER=$(cat "$SYSFS_TRACING_DIR/current_tracer")
+	FTRACE_FILTER=$(cat "$SYSFS_TRACING_DIR/set_ftrace_filter")
 }
 
 function pop_config() {
@@ -74,6 +78,17 @@ function pop_config() {
 	if [[ -n "$KPROBE_ENABLED" ]]; then
 		echo "$KPROBE_ENABLED" > "$SYSFS_KPROBES_DIR/enabled"
 	fi
+	if [[ -n "$TRACING_ON" ]]; then
+		echo "$TRACING_ON" > "$SYSFS_TRACING_DIR/tracing_on"
+	fi
+	if [[ -n "$CURRENT_TRACER" ]]; then
+		echo "$CURRENT_TRACER" > "$SYSFS_TRACING_DIR/current_tracer"
+	fi
+	if [[ -n "$FTRACE_FILTER" ]]; then
+		echo "$FTRACE_FILTER" \
+			| sed -e "/#### all functions enabled ####/d"
+			> "$SYSFS_TRACING_DIR/set_ftrace_filter"
+	fi
 }
 
 function set_dynamic_debug() {
@@ -352,3 +367,37 @@ function check_sysfs_value() {
 		die "Unexpected value in $path: $expected_value vs. $value"
 	fi
 }
+
+# cleanup_tracing() - stop and clean up function tracing
+function cleanup_tracing() {
+	echo 0 > "$SYSFS_TRACING_DIR/tracing_on"
+	echo "" > "$SYSFS_TRACING_DIR/set_ftrace_filter"
+	echo "nop" > "$SYSFS_TRACING_DIR/current_tracer"
+	echo "" > "$SYSFS_TRACING_DIR/trace"
+}
+
+# trace_function(function) - start tracing of a function
+#	function - to be traced function
+function trace_function() {
+	local function="$1"; shift
+
+	cleanup_tracing
+
+	echo "function" > "$SYSFS_TRACING_DIR/current_tracer"
+	echo "$function" > "$SYSFS_TRACING_DIR/set_ftrace_filter"
+	echo 1 > "$SYSFS_TRACING_DIR/tracing_on"
+}
+
+# check_traced_functions(functions...) - check whether each function appeared in the trace log
+#	functions - list of functions to be checked
+function check_traced_functions() {
+	local function
+
+	for function in "$@"; do
+		if ! grep -q "$function" "$SYSFS_TRACING_DIR/trace" ; then
+			die "Function ($function) did not appear in the trace"
+		fi
+	done
+
+	cleanup_tracing
+}

-- 
2.46.2

Re: [PATCH v2 1/2] selftests: livepatch: add new ftrace helpers functions

[Joe Lawrence]

On Tue, Mar 18, 2025 at 06:20:35PM -0300, Filipe Xavier wrote:
> Add new ftrace helpers functions cleanup_tracing, trace_function and
> check_traced_functions.
> 
> Signed-off-by: Filipe Xavier <felipeaggger@gmail.com>
> ---
>  tools/testing/selftests/livepatch/functions.sh | 49 ++++++++++++++++++++++++++
>  1 file changed, 49 insertions(+)
> 
> diff --git a/tools/testing/selftests/livepatch/functions.sh b/tools/testing/selftests/livepatch/functions.sh
> index 15601402dee6567837c2c49ba342eb357e410d18..dea9cc10a3f09662c57c2593cff49423302c8a5c 100644
> --- a/tools/testing/selftests/livepatch/functions.sh
> +++ b/tools/testing/selftests/livepatch/functions.sh
> @@ -10,6 +10,7 @@ SYSFS_KERNEL_DIR="/sys/kernel"
>  SYSFS_KLP_DIR="$SYSFS_KERNEL_DIR/livepatch"
>  SYSFS_DEBUG_DIR="$SYSFS_KERNEL_DIR/debug"
>  SYSFS_KPROBES_DIR="$SYSFS_DEBUG_DIR/kprobes"
> +SYSFS_TRACING_DIR="$SYSFS_DEBUG_DIR/tracing"
>  
>  # Kselftest framework requirement - SKIP code is 4
>  ksft_skip=4
> @@ -62,6 +63,9 @@ function push_config() {
>  			awk -F'[: ]' '{print "file " $1 " line " $2 " " $4}')
>  	FTRACE_ENABLED=$(sysctl --values kernel.ftrace_enabled)
>  	KPROBE_ENABLED=$(cat "$SYSFS_KPROBES_DIR/enabled")
> +	TRACING_ON=$(cat "$SYSFS_TRACING_DIR/tracing_on")
> +	CURRENT_TRACER=$(cat "$SYSFS_TRACING_DIR/current_tracer")
> +	FTRACE_FILTER=$(cat "$SYSFS_TRACING_DIR/set_ftrace_filter")
>  }
>  
>  function pop_config() {
> @@ -74,6 +78,17 @@ function pop_config() {
>  	if [[ -n "$KPROBE_ENABLED" ]]; then
>  		echo "$KPROBE_ENABLED" > "$SYSFS_KPROBES_DIR/enabled"
>  	fi
> +	if [[ -n "$TRACING_ON" ]]; then
> +		echo "$TRACING_ON" > "$SYSFS_TRACING_DIR/tracing_on"
> +	fi
> +	if [[ -n "$CURRENT_TRACER" ]]; then
> +		echo "$CURRENT_TRACER" > "$SYSFS_TRACING_DIR/current_tracer"
> +	fi
> +	if [[ -n "$FTRACE_FILTER" ]]; then
> +		echo "$FTRACE_FILTER" \
> +			| sed -e "/#### all functions enabled ####/d"
> +			> "$SYSFS_TRACING_DIR/set_ftrace_filter"
> +	fi
>  }
>  
>  function set_dynamic_debug() {
> @@ -352,3 +367,37 @@ function check_sysfs_value() {
>  		die "Unexpected value in $path: $expected_value vs. $value"
>  	fi
>  }
> +
> +# cleanup_tracing() - stop and clean up function tracing
> +function cleanup_tracing() {
> +	echo 0 > "$SYSFS_TRACING_DIR/tracing_on"
> +	echo "" > "$SYSFS_TRACING_DIR/set_ftrace_filter"
> +	echo "nop" > "$SYSFS_TRACING_DIR/current_tracer"
> +	echo "" > "$SYSFS_TRACING_DIR/trace"
> +}
> +
> +# trace_function(function) - start tracing of a function
> +#	function - to be traced function
> +function trace_function() {
> +	local function="$1"; shift
> +
> +	cleanup_tracing
> +
> +	echo "function" > "$SYSFS_TRACING_DIR/current_tracer"
> +	echo "$function" > "$SYSFS_TRACING_DIR/set_ftrace_filter"
> +	echo 1 > "$SYSFS_TRACING_DIR/tracing_on"
> +}
> +
> +# check_traced_functions(functions...) - check whether each function appeared in the trace log
> +#	functions - list of functions to be checked
> +function check_traced_functions() {
> +	local function
> +
> +	for function in "$@"; do
> +		if ! grep -q "$function" "$SYSFS_TRACING_DIR/trace" ; then

Small suggestion here: grep on "$function" may find partial string
matches, like:

  $ echo 'hamburger' | grep 'ham'
  hamburger

so it's probably safer to user --word-regexp to avoid longer function
names inadvertently matching:

  $ echo 'hamburger' | grep -w 'ham'
  $ echo 'ham' | grep -w 'ham'
  ham

Also, maybe also --fixed-strings for the extra paranoid?  Off the top of
my head, I don't think any C function characters are special regex (like
$ ^ [ ] etc.) so it's probably safe w/o.

-- Joe

Re: [PATCH v2 1/2] selftests: livepatch: add new ftrace helpers functions

[Joe Lawrence]

On Tue, Mar 18, 2025 at 06:20:35PM -0300, Filipe Xavier wrote:
> [ ... snip ... ]
> 
> +	if [[ -n "$FTRACE_FILTER" ]]; then
> +		echo "$FTRACE_FILTER" \
> +			| sed -e "/#### all functions enabled ####/d"
> +			> "$SYSFS_TRACING_DIR/set_ftrace_filter"
> +	fi
  
Also, this may be more stylistic than a functional nit (I don't know for
sure), but shellcheck [1] seemed confused about these lines:

   In tools/testing/selftests/livepatch/functions.sh line 90:
                          > "$SYSFS_TRACING_DIR/set_ftrace_filter"
                          ^-- SC2188 (warning): This redirection doesn't have a command. Move to its command (or use 'true' as no-op).

I wasn't going to comment on these until I saw shellcheck note them, but
I thought shell script convention was typically to end the line with the
redirection/pipe then escape the end of line like:

  commands | commands > \
      tee output.txt

There are a few existing examples of this pattern if you grep the
functions.sh file for '\\$'.

That said, I'm far from certain whether which order is better than the
other.  The only reason for bringing it up is that shellcheck warns on
this patch's usage.

[1] https://www.shellcheck.net/

-- Joe

Re: [PATCH v2 1/2] selftests: livepatch: add new ftrace helpers functions

[Filipe Xavier]

On 3/24/25 3:36 PM, Joe Lawrence wrote:

> On Tue, Mar 18, 2025 at 06:20:35PM -0300, Filipe Xavier wrote:
>> [ ... snip ... ]
>>
>> +	if [[ -n "$FTRACE_FILTER" ]]; then
>> +		echo "$FTRACE_FILTER" \
>> +			| sed -e "/#### all functions enabled ####/d"
>> +			> "$SYSFS_TRACING_DIR/set_ftrace_filter"
>> +	fi
>    
> Also, this may be more stylistic than a functional nit (I don't know for
> sure), but shellcheck [1] seemed confused about these lines:
>
>     In tools/testing/selftests/livepatch/functions.sh line 90:
>                            > "$SYSFS_TRACING_DIR/set_ftrace_filter"
>                            ^-- SC2188 (warning): This redirection doesn't have a command. Move to its command (or use 'true' as no-op).
>
> I wasn't going to comment on these until I saw shellcheck note them, but
> I thought shell script convention was typically to end the line with the
> redirection/pipe then escape the end of line like:
>
>    commands | commands > \
>        tee output.txt
>
> There are a few existing examples of this pattern if you grep the
> functions.sh file for '\\$'.
>
> That said, I'm far from certain whether which order is better than the
> other.  The only reason for bringing it up is that shellcheck warns on
> this patch's usage.
>
> [1] https://www.shellcheck.net/
>
> -- Joe

Thanks for review Joe, I'm sending a new version with all the fixes.

Cheers

Filipe

[PATCH v2 2/2] selftests: livepatch: test if ftrace can trace a livepatched function

[Filipe Xavier]

This new test makes sure that ftrace can trace a
function that was introduced by a livepatch.

Signed-off-by: Filipe Xavier <felipeaggger@gmail.com>
---
 tools/testing/selftests/livepatch/test-ftrace.sh | 34 ++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/tools/testing/selftests/livepatch/test-ftrace.sh b/tools/testing/selftests/livepatch/test-ftrace.sh
index fe14f248913acbec46fb6c0fec38a2fc84209d39..4937c74de0e4d34e4e692f20ee2bbe3cd6f5a232 100755
--- a/tools/testing/selftests/livepatch/test-ftrace.sh
+++ b/tools/testing/selftests/livepatch/test-ftrace.sh
@@ -61,4 +61,38 @@ livepatch: '$MOD_LIVEPATCH': unpatching complete
 % rmmod $MOD_LIVEPATCH"
 
 
+# - verify livepatch can load
+# - check if traces have a patched function
+# - reset trace and unload livepatch
+
+start_test "trace livepatched function and check that the live patch remains in effect"
+
+FUNCTION_NAME="livepatch_cmdline_proc_show"
+
+load_lp $MOD_LIVEPATCH
+trace_function "$FUNCTION_NAME"
+
+if [[ "$(cat /proc/cmdline)" == "$MOD_LIVEPATCH: this has been live patched" ]] ; then
+	log "livepatch: ok"
+fi
+
+check_traced_functions "$FUNCTION_NAME"
+
+disable_lp $MOD_LIVEPATCH
+unload_lp $MOD_LIVEPATCH
+
+check_result "% insmod test_modules/$MOD_LIVEPATCH.ko
+livepatch: enabling patch '$MOD_LIVEPATCH'
+livepatch: '$MOD_LIVEPATCH': initializing patching transition
+livepatch: '$MOD_LIVEPATCH': starting patching transition
+livepatch: '$MOD_LIVEPATCH': completing patching transition
+livepatch: '$MOD_LIVEPATCH': patching complete
+livepatch: ok
+% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH/enabled
+livepatch: '$MOD_LIVEPATCH': initializing unpatching transition
+livepatch: '$MOD_LIVEPATCH': starting unpatching transition
+livepatch: '$MOD_LIVEPATCH': completing unpatching transition
+livepatch: '$MOD_LIVEPATCH': unpatching complete
+% rmmod $MOD_LIVEPATCH"
+
 exit 0

-- 
2.46.2

Re: [PATCH v2 2/2] selftests: livepatch: test if ftrace can trace a livepatched function

[Joe Lawrence]

On Tue, Mar 18, 2025 at 06:20:36PM -0300, Filipe Xavier wrote:
> This new test makes sure that ftrace can trace a
> function that was introduced by a livepatch.
> 
> Signed-off-by: Filipe Xavier <felipeaggger@gmail.com>
> ---
>  tools/testing/selftests/livepatch/test-ftrace.sh | 34 ++++++++++++++++++++++++
>  1 file changed, 34 insertions(+)
> 
> diff --git a/tools/testing/selftests/livepatch/test-ftrace.sh b/tools/testing/selftests/livepatch/test-ftrace.sh
> index fe14f248913acbec46fb6c0fec38a2fc84209d39..4937c74de0e4d34e4e692f20ee2bbe3cd6f5a232 100755
> --- a/tools/testing/selftests/livepatch/test-ftrace.sh
> +++ b/tools/testing/selftests/livepatch/test-ftrace.sh
> @@ -61,4 +61,38 @@ livepatch: '$MOD_LIVEPATCH': unpatching complete
>  % rmmod $MOD_LIVEPATCH"
>  
>  
> +# - verify livepatch can load
> +# - check if traces have a patched function
> +# - reset trace and unload livepatch
> +
> +start_test "trace livepatched function and check that the live patch remains in effect"
> +
> +FUNCTION_NAME="livepatch_cmdline_proc_show"
> +
> +load_lp $MOD_LIVEPATCH
> +trace_function "$FUNCTION_NAME"
> +
> +if [[ "$(cat /proc/cmdline)" == "$MOD_LIVEPATCH: this has been live patched" ]] ; then
> +	log "livepatch: ok"
> +fi
> +
> +check_traced_functions "$FUNCTION_NAME"
> +
> +disable_lp $MOD_LIVEPATCH
> +unload_lp $MOD_LIVEPATCH
> +
> +check_result "% insmod test_modules/$MOD_LIVEPATCH.ko
> +livepatch: enabling patch '$MOD_LIVEPATCH'
> +livepatch: '$MOD_LIVEPATCH': initializing patching transition
> +livepatch: '$MOD_LIVEPATCH': starting patching transition
> +livepatch: '$MOD_LIVEPATCH': completing patching transition
> +livepatch: '$MOD_LIVEPATCH': patching complete
> +livepatch: ok
> +% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH/enabled

A few months ago, 637c730998b8 ("selftests: livepatch: rename
KLP_SYSFS_DIR to SYSFS_KLP_DIR") tweaked the functions.sh::SYSFS_KLP_DIR
and then updated all of the test-*.sh scripts to use the variable
instead of the spelled out "/sys/kernel/livepatch" string.

So if there if there is another patchset version,
s/\/sys\/kernel\/livepatch/$SYSFS_KLP_DIR/g

-- Joe 

Re: [PATCH v2 0/2] selftests: livepatch: test if ftrace can trace a livepatched function

[Miroslav Benes]

On Tue, 18 Mar 2025, Filipe Xavier wrote:

> This patchset add ftrace helpers functions and
> add a new test makes sure that ftrace can trace
> a function that was introduced by a livepatch.
> 
> Signed-off-by: Filipe Xavier <felipeaggger@gmail.com>
> Suggested-by: Marcos Paulo de Souza <mpdesouza@suse.com>
> Reviewed-by: Marcos Paulo de Souza <mpdesouza@suse.com>

Acked-by: Miroslav Benes <mbenes@suse.cz>

M