Re: [PATCH] perf test: Handle perftool-testsuite_probe failure due to broken DWARF

[Arnaldo Carvalho de Melo <acme@kernel.org>]

On Wed, Oct 16, 2024 at 11:19:30AM +0200, vmolnaro@redhat.com wrote:
> From: Veronika Molnarova <vmolnaro@redhat.com>
> 
> Test case test_adding_blacklisted ends in failure if the blacklisted
> probe is of an assembler function with no DWARF available. At the same
> time, probing the blacklisted function with ASM DWARF doesn't test the
> blacklist itself as the failure is a result of the broken DWARF.
> 
> When the broken DWARF output is encountered, check if the probed
> function was compiled by the assembler. If so, the broken DWARF message
> is expected and does not report a perf issue, else report a failure.
> If the ASM DWARF affected the probe, try the next probe on the blacklist.
> If the first 5 probes are defective due to broken DWARF, skip the test
> case.
> 
> Fixes: def5480d63c1e847 ("perf testsuite probe: Add test for blacklisted kprobes handling")
> Signed-off-by: Veronika Molnarova <vmolnaro@redhat.com>
> ---
>  .../base_probe/test_adding_blacklisted.sh     | 74 +++++++++++++++----
>  1 file changed, 59 insertions(+), 15 deletions(-)
> 
> diff --git a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh
> index b5dc10b2a73810b3..f67b3b267ac55269 100755
> --- a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh
> +++ b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh
> @@ -19,35 +19,79 @@
>  TEST_RESULT=0
>  
>  # skip if not supported
> -BLACKFUNC=`head -n 1 /sys/kernel/debug/kprobes/blacklist 2> /dev/null | cut -f2`
> -if [ -z "$BLACKFUNC" ]; then
> +BLACKFUNC_LIST=`head -n 5 /sys/kernel/debug/kprobes/blacklist 2> /dev/null | cut -f2`
> +if [ -z "$BLACKFUNC_LIST" ]; then
>  	print_overall_skipped
>  	exit 0
>  fi
>  
> +# chceck if locate is present to find vmlinux with DWARF debug info

check

> +locate --help 2&> /dev/null
> +if [ $? -eq 0 ]; then
> +	VMLINUX_FILE=$(locate -r '/vmlinux$' | xargs -I{} sh -c 'test -f "{}" && echo "{}"' | grep "$(uname -r)")
> +fi

And that matches the running kernel? Maybe its better to use 'perf
probe' itself to find it:

  root@number:~# perf probe -V icmp_rcv
  Available variables at icmp_rcv
          @<icmp_rcv+0>
                  struct sk_buff* skb
  root@number:~# perf probe -v -V icmp_rcv |& grep "Using.*for symbols"
  Using /usr/lib/debug/lib/modules/6.11.3-200.fc40.x86_64/vmlinux for symbols
  root@number:~# perf probe -v -V icmp_rcv |& grep "Using.*for symbols" | sed -r 's/^Using (.*) for symbols$/\1/'
  /usr/lib/debug/lib/modules/6.11.3-200.fc40.x86_64/vmlinux
  root@number:~#

I'll probably add something like this to 'perf buildid-list':

root@number:~# pahole --running_kernel_vmlinux
/usr/lib/debug/lib/modules/6.11.3-200.fc40.x86_64/vmlinux
root@number:~# readelf -wi `pahole --running_kernel_vmlinux` | head -12
Contents of the .debug_info section:

  Compilation Unit @ offset 0:
   Length:        0x2b082 (32-bit)
   Version:       5
   Unit Type:     DW_UT_compile (1)
   Abbrev Offset: 0
   Pointer Size:  8
 <0><c>: Abbrev Number: 246 (DW_TAG_compile_unit)
    <e>   DW_AT_producer    : (indirect string, offset: 0x4eda9f): GNU C11 14.2.1 20240912 (Red Hat 14.2.1-3) -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -mindirect-branch=thunk-extern -mindirect-branch-register -mindirect-branch-cs-prefix -mfunction-return=thunk-extern -mharden-sls=all -mrecord-mcount -mfentry -march=x86-64 -g -O2 -std=gnu11 -p -fshort-wchar -funsigned-char -fno-common -fno-PIE -fno-strict-aliasing -fcf-protection=branch -falign-jumps=1 -falign-loops=1 -fno-asynchronous-unwind-tables -fno-jump-tables -fpatchable-function-entry=16,16 -fno-delete-null-pointer-checks -fno-allow-store-data-races -fstack-protector-strong -ftrivial-auto-var-init=zero -fno-stack-clash-protection -fmin-function-alignment=16 -fstrict-flex-arrays=3 -fno-strict-overflow -fstack-check=no -fconserve-stack -fno-function-sections -fno-data-sections -fsanitize=bounds-strict -fsanitize=shift
    <12>   DW_AT_language    : 29	(C11)
    <13>   DW_AT_name        : (indirect line string, offset: 0xa546): init/main.c
root@number:~#

> +
>  # remove all previously added probes
>  clear_all_probes
>  
>  
>  ### adding blacklisted function
> -
> -# functions from blacklist should be skipped by perf probe
> -! $CMD_PERF probe $BLACKFUNC > $LOGS_DIR/adding_blacklisted.log 2> $LOGS_DIR/adding_blacklisted.err
> -PERF_EXIT_CODE=$?
> -
>  REGEX_SCOPE_FAIL="Failed to find scope of probe point"
>  REGEX_SKIP_MESSAGE=" is blacklisted function, skip it\."
> -REGEX_NOT_FOUND_MESSAGE="Probe point \'$BLACKFUNC\' not found."
> +REGEX_NOT_FOUND_MESSAGE="Probe point \'$RE_EVENT\' not found."
>  REGEX_ERROR_MESSAGE="Error: Failed to add events."
>  REGEX_INVALID_ARGUMENT="Failed to write event: Invalid argument"
>  REGEX_SYMBOL_FAIL="Failed to find symbol at $RE_ADDRESS"
> -REGEX_OUT_SECTION="$BLACKFUNC is out of \.\w+, skip it"
> -../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err
> -CHECK_EXIT_CODE=$?
> -
> -print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC"
> -(( TEST_RESULT += $? ))
> -
> +REGEX_OUT_SECTION="$RE_EVENT is out of \.\w+, skip it"
> +REGEX_MISSING_DECL_LINE="A function DIE doesn't have decl_line. Maybe broken DWARF?"
> +
> +BLACKFUNC=""
> +SKIP_DWARF=0
> +
> +for BLACKFUNC in $BLACKFUNC_LIST; do
> +	echo "Probing $BLACKFUNC"
> +
> +	# functions from blacklist should be skipped by perf probe
> +	! $CMD_PERF probe $BLACKFUNC > $LOGS_DIR/adding_blacklisted.log 2> $LOGS_DIR/adding_blacklisted.err
> +	PERF_EXIT_CODE=$?
> +
> +	# check for bad DWARF polluting the result
> +	../common/check_all_patterns_found.pl "$REGEX_MISSING_DECL_LINE" >/dev/null < $LOGS_DIR/adding_blacklisted.err
> +
> +	if [ $? -eq 0 ]; then
> +		SKIP_DWARF=1
> +
> +		# confirm that the broken DWARF comes from assembler
> +		if [ -n "$VMLINUX_FILE" ]; then
> +			readelf -wi "$VMLINUX_FILE" |

Can you cache the output of 'readelf -wi' for the whole vmlinux file so
that we don't do it more than once?

Doing it for each assembly function in the blacklist probably will add
up on this test run time.

> +			awk -v probe="$BLACKFUNC" '/DW_AT_language/ { comp_lang = $0 }
> +										$0 ~ probe { if (comp_lang) { print comp_lang }; exit }' |
> +			grep -q "MIPS assembler"
> +
> +			CHECK_EXIT_CODE=$?
> +			if [ $CHECK_EXIT_CODE -ne 0 ]; then
> +				SKIP_DWARF=0 # broken DWARF while available
> +				break
> +			fi
> +		else
> +			echo "Result polluted by broken DWARF, trying another probe"
> +		fi
> +
> +	else
> +		../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err
> +		CHECK_EXIT_CODE=$?
> +
> +		SKIP_DWARF=0
> +		break
> +	fi
> +done
> +
> +if [ $SKIP_DWARF -eq 1 ]; then
> +	print_testcase_skipped "adding blacklisted function $BLACKFUNC"
> +else
> +	print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC"
> +	(( TEST_RESULT += $? ))
> +fi
>  
>  ### listing not-added probe
>  
> -- 
> 2.43.0
>