* [PATCH] perf test: Handle perftool-testsuite_probe failure due to broken DWARF @ 2024-10-16 9:19 vmolnaro 2024-10-16 16:15 ` Arnaldo Carvalho de Melo 0 siblings, 1 reply; 5+ messages in thread From: vmolnaro @ 2024-10-16 9:19 UTC (permalink / raw) To: linux-perf-users, acme, acme Cc: mpetlan, peterz, mingo, namhyung, mark.rutland, alexander.shishkin, jolsa, irogers, adrian.hunter, kan.liang, linux-kernel From: Veronika Molnarova <vmolnaro@redhat.com> Test case test_adding_blacklisted ends in failure if the blacklisted probe is of an assembler function with no DWARF available. At the same time, probing the blacklisted function with ASM DWARF doesn't test the blacklist itself as the failure is a result of the broken DWARF. When the broken DWARF output is encountered, check if the probed function was compiled by the assembler. If so, the broken DWARF message is expected and does not report a perf issue, else report a failure. If the ASM DWARF affected the probe, try the next probe on the blacklist. If the first 5 probes are defective due to broken DWARF, skip the test case. Fixes: def5480d63c1e847 ("perf testsuite probe: Add test for blacklisted kprobes handling") Signed-off-by: Veronika Molnarova <vmolnaro@redhat.com> --- .../base_probe/test_adding_blacklisted.sh | 74 +++++++++++++++---- 1 file changed, 59 insertions(+), 15 deletions(-) diff --git a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh index b5dc10b2a73810b3..f67b3b267ac55269 100755 --- a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh +++ b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh @@ -19,35 +19,79 @@ TEST_RESULT=0 # skip if not supported -BLACKFUNC=`head -n 1 /sys/kernel/debug/kprobes/blacklist 2> /dev/null | cut -f2` -if [ -z "$BLACKFUNC" ]; then +BLACKFUNC_LIST=`head -n 5 /sys/kernel/debug/kprobes/blacklist 2> /dev/null | cut -f2` +if [ -z "$BLACKFUNC_LIST" ]; then print_overall_skipped exit 0 fi +# chceck if locate is present to find vmlinux with DWARF debug info +locate --help 2&> /dev/null +if [ $? -eq 0 ]; then + VMLINUX_FILE=$(locate -r '/vmlinux$' | xargs -I{} sh -c 'test -f "{}" && echo "{}"' | grep "$(uname -r)") +fi + # remove all previously added probes clear_all_probes ### adding blacklisted function - -# functions from blacklist should be skipped by perf probe -! $CMD_PERF probe $BLACKFUNC > $LOGS_DIR/adding_blacklisted.log 2> $LOGS_DIR/adding_blacklisted.err -PERF_EXIT_CODE=$? - REGEX_SCOPE_FAIL="Failed to find scope of probe point" REGEX_SKIP_MESSAGE=" is blacklisted function, skip it\." -REGEX_NOT_FOUND_MESSAGE="Probe point \'$BLACKFUNC\' not found." +REGEX_NOT_FOUND_MESSAGE="Probe point \'$RE_EVENT\' not found." REGEX_ERROR_MESSAGE="Error: Failed to add events." REGEX_INVALID_ARGUMENT="Failed to write event: Invalid argument" REGEX_SYMBOL_FAIL="Failed to find symbol at $RE_ADDRESS" -REGEX_OUT_SECTION="$BLACKFUNC is out of \.\w+, skip it" -../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err -CHECK_EXIT_CODE=$? - -print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC" -(( TEST_RESULT += $? )) - +REGEX_OUT_SECTION="$RE_EVENT is out of \.\w+, skip it" +REGEX_MISSING_DECL_LINE="A function DIE doesn't have decl_line. Maybe broken DWARF?" + +BLACKFUNC="" +SKIP_DWARF=0 + +for BLACKFUNC in $BLACKFUNC_LIST; do + echo "Probing $BLACKFUNC" + + # functions from blacklist should be skipped by perf probe + ! $CMD_PERF probe $BLACKFUNC > $LOGS_DIR/adding_blacklisted.log 2> $LOGS_DIR/adding_blacklisted.err + PERF_EXIT_CODE=$? + + # check for bad DWARF polluting the result + ../common/check_all_patterns_found.pl "$REGEX_MISSING_DECL_LINE" >/dev/null < $LOGS_DIR/adding_blacklisted.err + + if [ $? -eq 0 ]; then + SKIP_DWARF=1 + + # confirm that the broken DWARF comes from assembler + if [ -n "$VMLINUX_FILE" ]; then + readelf -wi "$VMLINUX_FILE" | + awk -v probe="$BLACKFUNC" '/DW_AT_language/ { comp_lang = $0 } + $0 ~ probe { if (comp_lang) { print comp_lang }; exit }' | + grep -q "MIPS assembler" + + CHECK_EXIT_CODE=$? + if [ $CHECK_EXIT_CODE -ne 0 ]; then + SKIP_DWARF=0 # broken DWARF while available + break + fi + else + echo "Result polluted by broken DWARF, trying another probe" + fi + + else + ../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err + CHECK_EXIT_CODE=$? + + SKIP_DWARF=0 + break + fi +done + +if [ $SKIP_DWARF -eq 1 ]; then + print_testcase_skipped "adding blacklisted function $BLACKFUNC" +else + print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC" + (( TEST_RESULT += $? )) +fi ### listing not-added probe -- 2.43.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH] perf test: Handle perftool-testsuite_probe failure due to broken DWARF 2024-10-16 9:19 [PATCH] perf test: Handle perftool-testsuite_probe failure due to broken DWARF vmolnaro @ 2024-10-16 16:15 ` Arnaldo Carvalho de Melo 2024-10-17 13:55 ` Veronika Molnarova 2024-10-17 16:15 ` [PATCH v2] " vmolnaro 0 siblings, 2 replies; 5+ messages in thread From: Arnaldo Carvalho de Melo @ 2024-10-16 16:15 UTC (permalink / raw) To: vmolnaro Cc: linux-perf-users, acme, mpetlan, peterz, mingo, namhyung, mark.rutland, alexander.shishkin, jolsa, irogers, adrian.hunter, kan.liang, linux-kernel On Wed, Oct 16, 2024 at 11:19:30AM +0200, vmolnaro@redhat.com wrote: > From: Veronika Molnarova <vmolnaro@redhat.com> > > Test case test_adding_blacklisted ends in failure if the blacklisted > probe is of an assembler function with no DWARF available. At the same > time, probing the blacklisted function with ASM DWARF doesn't test the > blacklist itself as the failure is a result of the broken DWARF. > > When the broken DWARF output is encountered, check if the probed > function was compiled by the assembler. If so, the broken DWARF message > is expected and does not report a perf issue, else report a failure. > If the ASM DWARF affected the probe, try the next probe on the blacklist. > If the first 5 probes are defective due to broken DWARF, skip the test > case. > > Fixes: def5480d63c1e847 ("perf testsuite probe: Add test for blacklisted kprobes handling") > Signed-off-by: Veronika Molnarova <vmolnaro@redhat.com> > --- > .../base_probe/test_adding_blacklisted.sh | 74 +++++++++++++++---- > 1 file changed, 59 insertions(+), 15 deletions(-) > > diff --git a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh > index b5dc10b2a73810b3..f67b3b267ac55269 100755 > --- a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh > +++ b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh > @@ -19,35 +19,79 @@ > TEST_RESULT=0 > > # skip if not supported > -BLACKFUNC=`head -n 1 /sys/kernel/debug/kprobes/blacklist 2> /dev/null | cut -f2` > -if [ -z "$BLACKFUNC" ]; then > +BLACKFUNC_LIST=`head -n 5 /sys/kernel/debug/kprobes/blacklist 2> /dev/null | cut -f2` > +if [ -z "$BLACKFUNC_LIST" ]; then > print_overall_skipped > exit 0 > fi > > +# chceck if locate is present to find vmlinux with DWARF debug info check > +locate --help 2&> /dev/null > +if [ $? -eq 0 ]; then > + VMLINUX_FILE=$(locate -r '/vmlinux$' | xargs -I{} sh -c 'test -f "{}" && echo "{}"' | grep "$(uname -r)") > +fi And that matches the running kernel? Maybe its better to use 'perf probe' itself to find it: root@number:~# perf probe -V icmp_rcv Available variables at icmp_rcv @<icmp_rcv+0> struct sk_buff* skb root@number:~# perf probe -v -V icmp_rcv |& grep "Using.*for symbols" Using /usr/lib/debug/lib/modules/6.11.3-200.fc40.x86_64/vmlinux for symbols root@number:~# perf probe -v -V icmp_rcv |& grep "Using.*for symbols" | sed -r 's/^Using (.*) for symbols$/\1/' /usr/lib/debug/lib/modules/6.11.3-200.fc40.x86_64/vmlinux root@number:~# I'll probably add something like this to 'perf buildid-list': root@number:~# pahole --running_kernel_vmlinux /usr/lib/debug/lib/modules/6.11.3-200.fc40.x86_64/vmlinux root@number:~# readelf -wi `pahole --running_kernel_vmlinux` | head -12 Contents of the .debug_info section: Compilation Unit @ offset 0: Length: 0x2b082 (32-bit) Version: 5 Unit Type: DW_UT_compile (1) Abbrev Offset: 0 Pointer Size: 8 <0><c>: Abbrev Number: 246 (DW_TAG_compile_unit) <e> DW_AT_producer : (indirect string, offset: 0x4eda9f): GNU C11 14.2.1 20240912 (Red Hat 14.2.1-3) -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -mindirect-branch=thunk-extern -mindirect-branch-register -mindirect-branch-cs-prefix -mfunction-return=thunk-extern -mharden-sls=all -mrecord-mcount -mfentry -march=x86-64 -g -O2 -std=gnu11 -p -fshort-wchar -funsigned-char -fno-common -fno-PIE -fno-strict-aliasing -fcf-protection=branch -falign-jumps=1 -falign-loops=1 -fno-asynchronous-unwind-tables -fno-jump-tables -fpatchable-function-entry=16,16 -fno-delete-null-pointer-checks -fno-allow-store-data-races -fstack-protector-strong -ftrivial-auto-var-init=zero -fno-stack-clash-protection -fmin-function-alignment=16 -fstrict-flex-arrays=3 -fno-strict-overflow -fstack-check=no -fconserve-stack -fno-function-sections -fno-data-sections -fsanitize=bounds-strict -fsanitize=shift <12> DW_AT_language : 29 (C11) <13> DW_AT_name : (indirect line string, offset: 0xa546): init/main.c root@number:~# > + > # remove all previously added probes > clear_all_probes > > > ### adding blacklisted function > - > -# functions from blacklist should be skipped by perf probe > -! $CMD_PERF probe $BLACKFUNC > $LOGS_DIR/adding_blacklisted.log 2> $LOGS_DIR/adding_blacklisted.err > -PERF_EXIT_CODE=$? > - > REGEX_SCOPE_FAIL="Failed to find scope of probe point" > REGEX_SKIP_MESSAGE=" is blacklisted function, skip it\." > -REGEX_NOT_FOUND_MESSAGE="Probe point \'$BLACKFUNC\' not found." > +REGEX_NOT_FOUND_MESSAGE="Probe point \'$RE_EVENT\' not found." > REGEX_ERROR_MESSAGE="Error: Failed to add events." > REGEX_INVALID_ARGUMENT="Failed to write event: Invalid argument" > REGEX_SYMBOL_FAIL="Failed to find symbol at $RE_ADDRESS" > -REGEX_OUT_SECTION="$BLACKFUNC is out of \.\w+, skip it" > -../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err > -CHECK_EXIT_CODE=$? > - > -print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC" > -(( TEST_RESULT += $? )) > - > +REGEX_OUT_SECTION="$RE_EVENT is out of \.\w+, skip it" > +REGEX_MISSING_DECL_LINE="A function DIE doesn't have decl_line. Maybe broken DWARF?" > + > +BLACKFUNC="" > +SKIP_DWARF=0 > + > +for BLACKFUNC in $BLACKFUNC_LIST; do > + echo "Probing $BLACKFUNC" > + > + # functions from blacklist should be skipped by perf probe > + ! $CMD_PERF probe $BLACKFUNC > $LOGS_DIR/adding_blacklisted.log 2> $LOGS_DIR/adding_blacklisted.err > + PERF_EXIT_CODE=$? > + > + # check for bad DWARF polluting the result > + ../common/check_all_patterns_found.pl "$REGEX_MISSING_DECL_LINE" >/dev/null < $LOGS_DIR/adding_blacklisted.err > + > + if [ $? -eq 0 ]; then > + SKIP_DWARF=1 > + > + # confirm that the broken DWARF comes from assembler > + if [ -n "$VMLINUX_FILE" ]; then > + readelf -wi "$VMLINUX_FILE" | Can you cache the output of 'readelf -wi' for the whole vmlinux file so that we don't do it more than once? Doing it for each assembly function in the blacklist probably will add up on this test run time. > + awk -v probe="$BLACKFUNC" '/DW_AT_language/ { comp_lang = $0 } > + $0 ~ probe { if (comp_lang) { print comp_lang }; exit }' | > + grep -q "MIPS assembler" > + > + CHECK_EXIT_CODE=$? > + if [ $CHECK_EXIT_CODE -ne 0 ]; then > + SKIP_DWARF=0 # broken DWARF while available > + break > + fi > + else > + echo "Result polluted by broken DWARF, trying another probe" > + fi > + > + else > + ../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err > + CHECK_EXIT_CODE=$? > + > + SKIP_DWARF=0 > + break > + fi > +done > + > +if [ $SKIP_DWARF -eq 1 ]; then > + print_testcase_skipped "adding blacklisted function $BLACKFUNC" > +else > + print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC" > + (( TEST_RESULT += $? )) > +fi > > ### listing not-added probe > > -- > 2.43.0 > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] perf test: Handle perftool-testsuite_probe failure due to broken DWARF 2024-10-16 16:15 ` Arnaldo Carvalho de Melo @ 2024-10-17 13:55 ` Veronika Molnarova 2024-10-17 16:15 ` [PATCH v2] " vmolnaro 1 sibling, 0 replies; 5+ messages in thread From: Veronika Molnarova @ 2024-10-17 13:55 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: linux-perf-users, acme, mpetlan, peterz, mingo, namhyung, mark.rutland, alexander.shishkin, jolsa, irogers, adrian.hunter, kan.liang, linux-kernel On 10/16/24 18:15, Arnaldo Carvalho de Melo wrote: > On Wed, Oct 16, 2024 at 11:19:30AM +0200, vmolnaro@redhat.com wrote: >> From: Veronika Molnarova <vmolnaro@redhat.com> >> >> Test case test_adding_blacklisted ends in failure if the blacklisted >> probe is of an assembler function with no DWARF available. At the same >> time, probing the blacklisted function with ASM DWARF doesn't test the >> blacklist itself as the failure is a result of the broken DWARF. >> >> When the broken DWARF output is encountered, check if the probed >> function was compiled by the assembler. If so, the broken DWARF message >> is expected and does not report a perf issue, else report a failure. >> If the ASM DWARF affected the probe, try the next probe on the blacklist. >> If the first 5 probes are defective due to broken DWARF, skip the test >> case. >> >> Fixes: def5480d63c1e847 ("perf testsuite probe: Add test for blacklisted kprobes handling") >> Signed-off-by: Veronika Molnarova <vmolnaro@redhat.com> >> --- >> .../base_probe/test_adding_blacklisted.sh | 74 +++++++++++++++---- >> 1 file changed, 59 insertions(+), 15 deletions(-) >> >> diff --git a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh >> index b5dc10b2a73810b3..f67b3b267ac55269 100755 >> --- a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh >> +++ b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh >> @@ -19,35 +19,79 @@ >> TEST_RESULT=0 >> >> # skip if not supported >> -BLACKFUNC=`head -n 1 /sys/kernel/debug/kprobes/blacklist 2> /dev/null | cut -f2` >> -if [ -z "$BLACKFUNC" ]; then >> +BLACKFUNC_LIST=`head -n 5 /sys/kernel/debug/kprobes/blacklist 2> /dev/null | cut -f2` >> +if [ -z "$BLACKFUNC_LIST" ]; then >> print_overall_skipped >> exit 0 >> fi >> >> +# chceck if locate is present to find vmlinux with DWARF debug info > > check > >> +locate --help 2&> /dev/null >> +if [ $? -eq 0 ]; then >> + VMLINUX_FILE=$(locate -r '/vmlinux$' | xargs -I{} sh -c 'test -f "{}" && echo "{}"' | grep "$(uname -r)") >> +fi > > And that matches the running kernel? Maybe its better to use 'perf > probe' itself to find it: > > root@number:~# perf probe -V icmp_rcv > Available variables at icmp_rcv > @<icmp_rcv+0> > struct sk_buff* skb > root@number:~# perf probe -v -V icmp_rcv |& grep "Using.*for symbols" > Using /usr/lib/debug/lib/modules/6.11.3-200.fc40.x86_64/vmlinux for symbols > root@number:~# perf probe -v -V icmp_rcv |& grep "Using.*for symbols" | sed -r 's/^Using (.*) for symbols$/\1/' > /usr/lib/debug/lib/modules/6.11.3-200.fc40.x86_64/vmlinux > root@number:~# > Well, it was the best solution we could find, but using 'perf probe' might be a better solution to it. It seems that 'perf probe' prints the using log line even if the probe doesn't exist: [root@intel-sharkbay-mb-01 ~]# perf probe -v random_probe probe-definition(0): random_probe symbol:random_probe file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments Looking at the vmlinux_path (8 entries long) Using /usr/lib/debug/lib/modules/5.14.0-364.3026_999908015.el9.x86_64/vmlinux for symbols Open Debuginfo file: /usr/lib/debug/.build-id/97/52ebee3cc94ca35fbbaafc9abf3cbacb34d4d1.debug Try to find probe point from debuginfo. Probe point 'random_probe' not found. Error: Failed to add events. Reason: No such device (Code: -19) Thanks, will rework that to grep the vmlinux from the 'perf probe -v' output instead. > I'll probably add something like this to 'perf buildid-list': > > root@number:~# pahole --running_kernel_vmlinux > /usr/lib/debug/lib/modules/6.11.3-200.fc40.x86_64/vmlinux > root@number:~# readelf -wi `pahole --running_kernel_vmlinux` | head -12 > Contents of the .debug_info section: > > Compilation Unit @ offset 0: > Length: 0x2b082 (32-bit) > Version: 5 > Unit Type: DW_UT_compile (1) > Abbrev Offset: 0 > Pointer Size: 8 > <0><c>: Abbrev Number: 246 (DW_TAG_compile_unit) > <e> DW_AT_producer : (indirect string, offset: 0x4eda9f): GNU C11 14.2.1 20240912 (Red Hat 14.2.1-3) -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -mindirect-branch=thunk-extern -mindirect-branch-register -mindirect-branch-cs-prefix -mfunction-return=thunk-extern -mharden-sls=all -mrecord-mcount -mfentry -march=x86-64 -g -O2 -std=gnu11 -p -fshort-wchar -funsigned-char -fno-common -fno-PIE -fno-strict-aliasing -fcf-protection=branch -falign-jumps=1 -falign-loops=1 -fno-asynchronous-unwind-tables -fno-jump-tables -fpatchable-function-entry=16,16 -fno-delete-null-pointer-checks -fno-allow-store-data-races -fstack-protector-strong -ftrivial-auto-var-init=zero -fno-stack-clash-protection -fmin-function-alignment=16 -fstrict-flex-arrays=3 -fno-strict-overflow -fstack-check=no -fconserve-stack -fno-function-sections -fno-data-sections -fsanitize=bounds-strict -fsanitize=shift > <12> DW_AT_language : 29 (C11) > <13> DW_AT_name : (indirect line string, offset: 0xa546): init/main.c > root@number:~# > >> + >> # remove all previously added probes >> clear_all_probes >> >> >> ### adding blacklisted function >> - >> -# functions from blacklist should be skipped by perf probe >> -! $CMD_PERF probe $BLACKFUNC > $LOGS_DIR/adding_blacklisted.log 2> $LOGS_DIR/adding_blacklisted.err >> -PERF_EXIT_CODE=$? >> - >> REGEX_SCOPE_FAIL="Failed to find scope of probe point" >> REGEX_SKIP_MESSAGE=" is blacklisted function, skip it\." >> -REGEX_NOT_FOUND_MESSAGE="Probe point \'$BLACKFUNC\' not found." >> +REGEX_NOT_FOUND_MESSAGE="Probe point \'$RE_EVENT\' not found." >> REGEX_ERROR_MESSAGE="Error: Failed to add events." >> REGEX_INVALID_ARGUMENT="Failed to write event: Invalid argument" >> REGEX_SYMBOL_FAIL="Failed to find symbol at $RE_ADDRESS" >> -REGEX_OUT_SECTION="$BLACKFUNC is out of \.\w+, skip it" >> -../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err >> -CHECK_EXIT_CODE=$? >> - >> -print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC" >> -(( TEST_RESULT += $? )) >> - >> +REGEX_OUT_SECTION="$RE_EVENT is out of \.\w+, skip it" >> +REGEX_MISSING_DECL_LINE="A function DIE doesn't have decl_line. Maybe broken DWARF?" >> + >> +BLACKFUNC="" >> +SKIP_DWARF=0 >> + >> +for BLACKFUNC in $BLACKFUNC_LIST; do >> + echo "Probing $BLACKFUNC" >> + >> + # functions from blacklist should be skipped by perf probe >> + ! $CMD_PERF probe $BLACKFUNC > $LOGS_DIR/adding_blacklisted.log 2> $LOGS_DIR/adding_blacklisted.err >> + PERF_EXIT_CODE=$? >> + >> + # check for bad DWARF polluting the result >> + ../common/check_all_patterns_found.pl "$REGEX_MISSING_DECL_LINE" >/dev/null < $LOGS_DIR/adding_blacklisted.err >> + >> + if [ $? -eq 0 ]; then >> + SKIP_DWARF=1 >> + >> + # confirm that the broken DWARF comes from assembler >> + if [ -n "$VMLINUX_FILE" ]; then >> + readelf -wi "$VMLINUX_FILE" | > > Can you cache the output of 'readelf -wi' for the whole vmlinux file so > that we don't do it more than once? > > Doing it for each assembly function in the blacklist probably will add > up on this test run time. > I am not sure if it is possible to cache the whole output somehow as my tested system takes a minute if I try to print the output of the 'readlef -wi'. To add to that, I do not need the whole vmlinux file as I am exiting as soon I as find the blacklisted_probe in the output. About the test run time, I checked how long it takes to find the probe in the 'readelf -wi' output using awk and it is about 2 seconds per probe: [root@intel-sharkbay-mb-01 ~]# time readelf -wi "$VMLINUX_FILE" | awk -v probe="asm_exc_device_not_available" '/DW_AT_language/ { comp_lang = $0 } $0 ~ probe { if (comp_lang) { print comp_lang }; exit }' <41fc04> DW_AT_language : 29 (C11) real 0m1.509s user 0m1.979s sys 0m0.173s This depends on the probe so some might take longer than that. Doing it repeatedly for multiple probes is not the default behavior anyway, as this only happens if there are multiple assembly functions at the start of the blacklist in a row. Currently, we know only about one of such functions, which is warn_thunk_thunk. If the run time will be the issue, I think we would have to address the issue differently. >> + awk -v probe="$BLACKFUNC" '/DW_AT_language/ { comp_lang = $0 } >> + $0 ~ probe { if (comp_lang) { print comp_lang }; exit }' | >> + grep -q "MIPS assembler" >> + >> + CHECK_EXIT_CODE=$? >> + if [ $CHECK_EXIT_CODE -ne 0 ]; then >> + SKIP_DWARF=0 # broken DWARF while available >> + break >> + fi >> + else >> + echo "Result polluted by broken DWARF, trying another probe" >> + fi >> + >> + else >> + ../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err >> + CHECK_EXIT_CODE=$? >> + >> + SKIP_DWARF=0 >> + break >> + fi >> +done >> + >> +if [ $SKIP_DWARF -eq 1 ]; then >> + print_testcase_skipped "adding blacklisted function $BLACKFUNC" >> +else >> + print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC" >> + (( TEST_RESULT += $? )) >> +fi >> >> ### listing not-added probe >> >> -- >> 2.43.0 >> > ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH v2] perf test: Handle perftool-testsuite_probe failure due to broken DWARF 2024-10-16 16:15 ` Arnaldo Carvalho de Melo 2024-10-17 13:55 ` Veronika Molnarova @ 2024-10-17 16:15 ` vmolnaro 2024-10-23 20:24 ` Arnaldo Carvalho de Melo 1 sibling, 1 reply; 5+ messages in thread From: vmolnaro @ 2024-10-17 16:15 UTC (permalink / raw) To: linux-perf-users, acme, acme Cc: mpetlan, peterz, mingo, namhyung, mark.rutland, alexander.shishkin, jolsa, irogers, adrian.hunter, kan.liang, linux-kernel From: Veronika Molnarova <vmolnaro@redhat.com> Test case test_adding_blacklisted ends in failure if the blacklisted probe is of an assembler function with no DWARF available. At the same time, probing the blacklisted function with ASM DWARF doesn't test the blacklist itself as the failure is a result of the broken DWARF. When the broken DWARF output is encountered, check if the probed function was compiled by the assembler. If so, the broken DWARF message is expected and does not report a perf issue, else report a failure. If the ASM DWARF affected the probe, try the next probe on the blacklist. If the first 5 probes are defective due to broken DWARF, skip the test case. Fixes: def5480d63c1e847 ("perf testsuite probe: Add test for blacklisted kprobes handling") Signed-off-by: Veronika Molnarova <vmolnaro@redhat.com> --- .../base_probe/test_adding_blacklisted.sh | 69 +++++++++++++++---- 1 file changed, 54 insertions(+), 15 deletions(-) diff --git a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh index b5dc10b2a73810b3..bead723e34af3f0e 100755 --- a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh +++ b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh @@ -19,35 +19,74 @@ TEST_RESULT=0 # skip if not supported -BLACKFUNC=`head -n 1 /sys/kernel/debug/kprobes/blacklist 2> /dev/null | cut -f2` -if [ -z "$BLACKFUNC" ]; then +BLACKFUNC_LIST=`head -n 5 /sys/kernel/debug/kprobes/blacklist 2> /dev/null | cut -f2` +if [ -z "$BLACKFUNC_LIST" ]; then print_overall_skipped exit 0 fi +# try to find vmlinux with DWARF debug info +VMLINUX_FILE=$(perf probe -v random_probe |& grep "Using.*for symbols" | sed -r 's/^Using (.*) for symbols$/\1/') + # remove all previously added probes clear_all_probes ### adding blacklisted function - -# functions from blacklist should be skipped by perf probe -! $CMD_PERF probe $BLACKFUNC > $LOGS_DIR/adding_blacklisted.log 2> $LOGS_DIR/adding_blacklisted.err -PERF_EXIT_CODE=$? - REGEX_SCOPE_FAIL="Failed to find scope of probe point" REGEX_SKIP_MESSAGE=" is blacklisted function, skip it\." -REGEX_NOT_FOUND_MESSAGE="Probe point \'$BLACKFUNC\' not found." +REGEX_NOT_FOUND_MESSAGE="Probe point \'$RE_EVENT\' not found." REGEX_ERROR_MESSAGE="Error: Failed to add events." REGEX_INVALID_ARGUMENT="Failed to write event: Invalid argument" REGEX_SYMBOL_FAIL="Failed to find symbol at $RE_ADDRESS" -REGEX_OUT_SECTION="$BLACKFUNC is out of \.\w+, skip it" -../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err -CHECK_EXIT_CODE=$? - -print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC" -(( TEST_RESULT += $? )) - +REGEX_OUT_SECTION="$RE_EVENT is out of \.\w+, skip it" +REGEX_MISSING_DECL_LINE="A function DIE doesn't have decl_line. Maybe broken DWARF?" + +BLACKFUNC="" +SKIP_DWARF=0 + +for BLACKFUNC in $BLACKFUNC_LIST; do + echo "Probing $BLACKFUNC" + + # functions from blacklist should be skipped by perf probe + ! $CMD_PERF probe $BLACKFUNC > $LOGS_DIR/adding_blacklisted.log 2> $LOGS_DIR/adding_blacklisted.err + PERF_EXIT_CODE=$? + + # check for bad DWARF polluting the result + ../common/check_all_patterns_found.pl "$REGEX_MISSING_DECL_LINE" >/dev/null < $LOGS_DIR/adding_blacklisted.err + + if [ $? -eq 0 ]; then + SKIP_DWARF=1 + echo "Result polluted by broken DWARF, trying another probe" + + # confirm that the broken DWARF comes from assembler + if [ -n "$VMLINUX_FILE" ]; then + readelf -wi "$VMLINUX_FILE" | + awk -v probe="$BLACKFUNC" '/DW_AT_language/ { comp_lang = $0 } + $0 ~ probe { if (comp_lang) { print comp_lang }; exit }' | + grep -q "MIPS assembler" + + CHECK_EXIT_CODE=$? + if [ $CHECK_EXIT_CODE -ne 0 ]; then + SKIP_DWARF=0 # broken DWARF while available + break + fi + fi + else + ../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err + CHECK_EXIT_CODE=$? + + SKIP_DWARF=0 + break + fi +done + +if [ $SKIP_DWARF -eq 1 ]; then + print_testcase_skipped "adding blacklisted function $BLACKFUNC" +else + print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC" + (( TEST_RESULT += $? )) +fi ### listing not-added probe -- 2.43.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH v2] perf test: Handle perftool-testsuite_probe failure due to broken DWARF 2024-10-17 16:15 ` [PATCH v2] " vmolnaro @ 2024-10-23 20:24 ` Arnaldo Carvalho de Melo 0 siblings, 0 replies; 5+ messages in thread From: Arnaldo Carvalho de Melo @ 2024-10-23 20:24 UTC (permalink / raw) To: vmolnaro Cc: linux-perf-users, acme, mpetlan, peterz, mingo, namhyung, mark.rutland, alexander.shishkin, jolsa, irogers, adrian.hunter, kan.liang, linux-kernel On Thu, Oct 17, 2024 at 06:15:55PM +0200, vmolnaro@redhat.com wrote: > From: Veronika Molnarova <vmolnaro@redhat.com> > > Test case test_adding_blacklisted ends in failure if the blacklisted > probe is of an assembler function with no DWARF available. At the same > time, probing the blacklisted function with ASM DWARF doesn't test the > blacklist itself as the failure is a result of the broken DWARF. > > When the broken DWARF output is encountered, check if the probed > function was compiled by the assembler. If so, the broken DWARF message > is expected and does not report a perf issue, else report a failure. > If the ASM DWARF affected the probe, try the next probe on the blacklist. > If the first 5 probes are defective due to broken DWARF, skip the test > case. Tested and applied to perf-tools. - Arnaldo > Fixes: def5480d63c1e847 ("perf testsuite probe: Add test for blacklisted kprobes handling") > Signed-off-by: Veronika Molnarova <vmolnaro@redhat.com> > --- > .../base_probe/test_adding_blacklisted.sh | 69 +++++++++++++++---- > 1 file changed, 54 insertions(+), 15 deletions(-) > > diff --git a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh > index b5dc10b2a73810b3..bead723e34af3f0e 100755 > --- a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh > +++ b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh > @@ -19,35 +19,74 @@ > TEST_RESULT=0 > > # skip if not supported > -BLACKFUNC=`head -n 1 /sys/kernel/debug/kprobes/blacklist 2> /dev/null | cut -f2` > -if [ -z "$BLACKFUNC" ]; then > +BLACKFUNC_LIST=`head -n 5 /sys/kernel/debug/kprobes/blacklist 2> /dev/null | cut -f2` > +if [ -z "$BLACKFUNC_LIST" ]; then > print_overall_skipped > exit 0 > fi > > +# try to find vmlinux with DWARF debug info > +VMLINUX_FILE=$(perf probe -v random_probe |& grep "Using.*for symbols" | sed -r 's/^Using (.*) for symbols$/\1/') > + > # remove all previously added probes > clear_all_probes > > > ### adding blacklisted function > - > -# functions from blacklist should be skipped by perf probe > -! $CMD_PERF probe $BLACKFUNC > $LOGS_DIR/adding_blacklisted.log 2> $LOGS_DIR/adding_blacklisted.err > -PERF_EXIT_CODE=$? > - > REGEX_SCOPE_FAIL="Failed to find scope of probe point" > REGEX_SKIP_MESSAGE=" is blacklisted function, skip it\." > -REGEX_NOT_FOUND_MESSAGE="Probe point \'$BLACKFUNC\' not found." > +REGEX_NOT_FOUND_MESSAGE="Probe point \'$RE_EVENT\' not found." > REGEX_ERROR_MESSAGE="Error: Failed to add events." > REGEX_INVALID_ARGUMENT="Failed to write event: Invalid argument" > REGEX_SYMBOL_FAIL="Failed to find symbol at $RE_ADDRESS" > -REGEX_OUT_SECTION="$BLACKFUNC is out of \.\w+, skip it" > -../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err > -CHECK_EXIT_CODE=$? > - > -print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC" > -(( TEST_RESULT += $? )) > - > +REGEX_OUT_SECTION="$RE_EVENT is out of \.\w+, skip it" > +REGEX_MISSING_DECL_LINE="A function DIE doesn't have decl_line. Maybe broken DWARF?" > + > +BLACKFUNC="" > +SKIP_DWARF=0 > + > +for BLACKFUNC in $BLACKFUNC_LIST; do > + echo "Probing $BLACKFUNC" > + > + # functions from blacklist should be skipped by perf probe > + ! $CMD_PERF probe $BLACKFUNC > $LOGS_DIR/adding_blacklisted.log 2> $LOGS_DIR/adding_blacklisted.err > + PERF_EXIT_CODE=$? > + > + # check for bad DWARF polluting the result > + ../common/check_all_patterns_found.pl "$REGEX_MISSING_DECL_LINE" >/dev/null < $LOGS_DIR/adding_blacklisted.err > + > + if [ $? -eq 0 ]; then > + SKIP_DWARF=1 > + echo "Result polluted by broken DWARF, trying another probe" > + > + # confirm that the broken DWARF comes from assembler > + if [ -n "$VMLINUX_FILE" ]; then > + readelf -wi "$VMLINUX_FILE" | > + awk -v probe="$BLACKFUNC" '/DW_AT_language/ { comp_lang = $0 } > + $0 ~ probe { if (comp_lang) { print comp_lang }; exit }' | > + grep -q "MIPS assembler" > + > + CHECK_EXIT_CODE=$? > + if [ $CHECK_EXIT_CODE -ne 0 ]; then > + SKIP_DWARF=0 # broken DWARF while available > + break > + fi > + fi > + else > + ../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err > + CHECK_EXIT_CODE=$? > + > + SKIP_DWARF=0 > + break > + fi > +done > + > +if [ $SKIP_DWARF -eq 1 ]; then > + print_testcase_skipped "adding blacklisted function $BLACKFUNC" > +else > + print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC" > + (( TEST_RESULT += $? )) > +fi > > ### listing not-added probe > > -- > 2.43.0 > ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2024-10-23 20:24 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-10-16 9:19 [PATCH] perf test: Handle perftool-testsuite_probe failure due to broken DWARF vmolnaro 2024-10-16 16:15 ` Arnaldo Carvalho de Melo 2024-10-17 13:55 ` Veronika Molnarova 2024-10-17 16:15 ` [PATCH v2] " vmolnaro 2024-10-23 20:24 ` Arnaldo Carvalho de Melo
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).