Re: [PATCH v2] selftests: livepatch: test if ftrace can trace a livepatched function

[Petr Mladek <pmladek@suse.com>]

Hi Filipe,

first, thanks for the test. It is good. I just suggest a lot of clean
up ;-)

On Sat 2025-01-11 15:42:27, Filipe Xavier wrote:
> This new test makes sure that ftrace can trace a
> function that was introduced by a livepatch.
> ---
> Changes in v2:
> - functions.sh: added reset tracing on push and pop_config.
> - test-ftrace.sh: enabled tracing_on before test init.
> - nitpick: added double quotations on filenames and fixed some wording. 
> - Link to v1: https://lore.kernel.org/r/20250102-ftrace-selftest-livepatch-v1-1-84880baefc1b@gmail.com
> ---
>  tools/testing/selftests/livepatch/functions.sh   | 14 ++++++++++
>  tools/testing/selftests/livepatch/test-ftrace.sh | 33 ++++++++++++++++++++++++
>  2 files changed, 47 insertions(+)
> 
> diff --git a/tools/testing/selftests/livepatch/functions.sh b/tools/testing/selftests/livepatch/functions.sh
> index e5d06fb402335d85959bafe099087effc6ddce12..e6c13514002dae5f8d7461f90b8241ab43024ea4 100644
> --- a/tools/testing/selftests/livepatch/functions.sh
> +++ b/tools/testing/selftests/livepatch/functions.sh
> @@ -62,6 +62,9 @@ function push_config() {
>  			awk -F'[: ]' '{print "file " $1 " line " $2 " " $4}')
>  	FTRACE_ENABLED=$(sysctl --values kernel.ftrace_enabled)
>  	KPROBE_ENABLED=$(cat "$SYSFS_KPROBES_DIR/enabled")
> +	TRACING_ON=$(cat "$SYSFS_DEBUG_DIR/tracing/tracing_on")

The $SYSFS_DEBUG_DIR/tracing directory is used many times. It would
make sense to define:

SYSFS_TRACING_DIR="$SYSFS_DEBUG_DIR/tracing"

and use:

	TRACING_ON=$(cat "$SYSFS_TRACING_DIR/tracing_on")

> +	CURRENT_TRACER=$(cat "$SYSFS_DEBUG_DIR/tracing/current_tracer")
> +	FTRACE_FILTER=$(cat "$SYSFS_DEBUG_DIR/tracing/set_ftrace_filter")
>  }
>  
>  function pop_config() {
> @@ -74,6 +77,17 @@ function pop_config() {
>  	if [[ -n "$KPROBE_ENABLED" ]]; then
>  		echo "$KPROBE_ENABLED" > "$SYSFS_KPROBES_DIR/enabled"
>  	fi
> +	if [[ -n "$TRACING_ON" ]]; then
> +		echo "$TRACING_ON" > "$SYSFS_DEBUG_DIR/tracing/tracing_on"
> +	fi
> +	if [[ -n "$CURRENT_TRACER" ]]; then
> +		echo "$CURRENT_TRACER" > "$SYSFS_DEBUG_DIR/tracing/current_tracer"
> +	fi
> +	if [[ "$FTRACE_FILTER" == *"#"* ]]; then
> +		echo > "$SYSFS_DEBUG_DIR/tracing/set_ftrace_filter"

My understanding is that this is handling the situation when the
filter is:

   #### all functions enabled ####

I was curious about the pattern *"#"*. The quotation looked pretty
non-standard to me. It made me to read the bash documentation and
the rules for extressions in [[ ]] looked quite complicated.

I wonder if we could make this more for "dummies". The following
seems to work:

	if [[ -n "$FTRACE_FILTER" ]]; then
		echo "$FTRACE_FILTER"
			| sed -e "/#### all functions enabled ####/d"
			> "$SYSFS_DEBUG_DIR/tracing/set_ftrace_filter"
	fi

Note that the string "#### all functions enabled ####" is a very
stable thing. It has never been modified since it was introduced
by the commit 0c75a3ed633419d75d ("ftrace: state that all functions
are enabled in set_ftrace_filter") in Feb 2009.

The string is used also in ftrace selftests, see
tools/testing/selftests/ftrace/test.d/ftrace/func_set_ftrace_file.tc

> +	elif [[ -n "$FTRACE_FILTER" ]]; then
> +		echo "$FTRACE_FILTER" > "$SYSFS_DEBUG_DIR/tracing/set_ftrace_filter"
> +	fi
>  }
>  
>  function set_dynamic_debug() {
> diff --git a/tools/testing/selftests/livepatch/test-ftrace.sh b/tools/testing/selftests/livepatch/test-ftrace.sh
> index fe14f248913acbec46fb6c0fec38a2fc84209d39..66af5d726c52e48e5177804e182b4ff31784d5ac 100755
> --- a/tools/testing/selftests/livepatch/test-ftrace.sh
> +++ b/tools/testing/selftests/livepatch/test-ftrace.sh
> @@ -61,4 +61,37 @@ livepatch: '$MOD_LIVEPATCH': unpatching complete
>  % rmmod $MOD_LIVEPATCH"
>  
>  
> +# - verify livepatch can load
> +# - check if traces have a patched function
> +# - unload livepatch and reset trace
> +
> +start_test "trace livepatched function and check that the live patch remains in effect"
> +
> +TRACE_FILE="$SYSFS_DEBUG_DIR/tracing/trace"
> +FUNCTION_NAME="livepatch_cmdline_proc_show"
> +
> +load_lp $MOD_LIVEPATCH
> +
> +echo 1 > "$SYSFS_DEBUG_DIR/tracing/tracing_on"
> +echo $FUNCTION_NAME > "$SYSFS_DEBUG_DIR/tracing/set_ftrace_filter"

The above mentioned ftrace selftest shows that the behavior gets
more complicated when "traceoff" is used:

  1. The "traceoff" entries are not cleared by the above command.
  2. The "traceoff" entries might cause that the tracing will
     stop before we finish this test.

> +echo "function" > "$SYSFS_DEBUG_DIR/tracing/current_tracer"
> +echo "" > "$TRACE_FILE"

I suggest to change the ordering and make sure that nothing is running
before we start:

# Start with the clean up
echo 0 > "$SYSFS_TRACING_DIR/tracing_on"
echo '!*:traceoff' > $SYSFS_TRACING_DIR/set_ftrace_filter
echo "" > "$SYSFS_TRACING_DIR/current_tracer"
echo "" > $SYSFS_TRACING_DIR/trace
# Setup and start our tracing
echo "function" > "$SYSFS_TRACING_DIR/current_tracer"
echo $FUNCTION_NAME > "$SYSFS_TRACING_DIR/set_ftrace_filter"
echo 1 > "$SYSFS_TRACING_DIR/tracing_on"

I suggest to this into helper functions which might be used
by more tests:

# cleanup_tracing() - stop and clean up function tracing
function cleanup_tracing() {
	echo 0 > "$SYSFS_TRACING_DIR/tracing_on"
	echo '!*:traceoff' > $SYSFS_TRACING_DIR/set_ftrace_filter
	echo "" > "$SYSFS_TRACING_DIR/current_tracer"
	echo "" > $SYSFS_TRACING_DIR/trace
}

# trace_function(function) - start tracing of a function
#	function - to be traced function
function trace_function() {
	local function="$1"; shift

	cleanup_tracing

	echo "function" > "$SYSFS_TRACING_DIR/current_tracer"
	echo "$functions" > $SYSFS_TRACING_DIR/set_ftrace_filter
	echo 1 > "$SYSFS_TRACING_DIR/tracing_on"
}

# check_traced_function(function) - check whether function appeared in trace log
#	function - to be traced function
function check_trace_function() {
	local function="$1"; shift

	if ! grep -q "$function" $SYSFS_TRACING_DIR/trace" ; then
		die "Function ($function) did not appear in the trace"
	fi
}

As Marcos suggested, it would make sense to add the above mentioned
helper functions in a separate patch.

> +
> +if [[ "$(cat /proc/cmdline)" != "$MOD_LIVEPATCH: this has been live patched" ]] ; then
> +	echo -e "FAIL\n\n"
> +	die "livepatch kselftest(s) failed"
> +fi
> +
> +grep -q $FUNCTION_NAME "$TRACE_FILE"
> +FOUND=$?

We could replace this with the above proposed

	check_traced_function "$FUNCTION_NAME"

Also I would personally stop the tracing after this sub-test finishes.
We might use the cleanup_tracing function here.

	cleanup_tracing

> +
> +disable_lp $MOD_LIVEPATCH
> +unload_lp $MOD_LIVEPATCH
> +
> +if [ "$FOUND" -eq 1 ]; then
> +	echo -e "FAIL\n\n"
> +	die "livepatch kselftest(s) failed"
> +fi

As mentioned by Marcos, we should also print "ok" on success.
I would personally use the "check_result" function here. It
could catch non-expected entries in dmesg.

> +
>  exit 0

Best Regards,
Petr