[PATCH] perf test: Fix probe testsuite with a new error message

[PATCH] perf test: Fix probe testsuite with a new error message

[Namhyung Kim]

On my system, it's constantly failing because of new error message from
perf probe.  It should update the regex pattern to match the message -
"A function DIE doesn't have decl_line. Maybe broken DWARF?".

  $ sudo head -n 2 /sys/kernel/debug/kprobes/blacklist | cut -f2
  warn_thunk_thunk
  asm_exc_divide_error

  $ sudo perf probe warn_thunk_thunk
  A function DIE doesn't have decl_line. Maybe broken DWARF?
  A function DIE doesn't have decl_line. Maybe broken DWARF?
  Probe point 'warn_thunk_thunk' not found.
    Error: Failed to add events.

  $ sudo perf probe asm_exc_overflow
  Failed to find scope of probe point.
    Error: Failed to add events.

Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Michael Petlan <mpetlan@redhat.com>
Cc: Veronika Molnarova <vmolnaro@redhat.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh
index b5dc10b2a73810b3..01e5e09053c37e46 100755
--- a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh
+++ b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh
@@ -42,7 +42,8 @@ REGEX_ERROR_MESSAGE="Error: Failed to add events."
 REGEX_INVALID_ARGUMENT="Failed to write event: Invalid argument"
 REGEX_SYMBOL_FAIL="Failed to find symbol at $RE_ADDRESS"
 REGEX_OUT_SECTION="$BLACKFUNC is out of \.\w+, skip it"
-../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err
+REGEX_BROKEN_DWARF="A function DIE doesn\'t have decl_line\. Maybe broken DWARF\?"
+../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" "$REGEX_BROKEN_DWARF" < $LOGS_DIR/adding_blacklisted.err
 CHECK_EXIT_CODE=$?
 
 print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC"
-- 
2.47.0.rc0.187.ge670bccf7e-goog

Re: [PATCH] perf test: Fix probe testsuite with a new error message

[Arnaldo Carvalho de Melo]

On Wed, Oct 09, 2024 at 10:16:20PM -0700, Namhyung Kim wrote:
> On my system, it's constantly failing because of new error message from
> perf probe.  It should update the regex pattern to match the message -
> "A function DIE doesn't have decl_line. Maybe broken DWARF?".
> 
>   $ sudo head -n 2 /sys/kernel/debug/kprobes/blacklist | cut -f2
>   warn_thunk_thunk
>   asm_exc_divide_error
> 
>   $ sudo perf probe warn_thunk_thunk
>   A function DIE doesn't have decl_line. Maybe broken DWARF?
>   A function DIE doesn't have decl_line. Maybe broken DWARF?
>   Probe point 'warn_thunk_thunk' not found.
>     Error: Failed to add events.
> 
>   $ sudo perf probe asm_exc_overflow
>   Failed to find scope of probe point.
>     Error: Failed to add events.

We discussed this in the past, I came up with a similar patch, Veronika
rightly pointed out that this may point to a real problem, Masami said
that since these are for DWARF from assembly those are known issues, I
suggested Veronika checked if the CU where the function came from was
generated from Assembly (there are DWARF tags that have that info), IIRC
she said she would try to do it.

I'll try to find out the threads and see what happened.

- Arnaldo
 
> Cc: Masami Hiramatsu <mhiramat@kernel.org>
> Cc: Michael Petlan <mpetlan@redhat.com>
> Cc: Veronika Molnarova <vmolnaro@redhat.com>
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh
> index b5dc10b2a73810b3..01e5e09053c37e46 100755
> --- a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh
> +++ b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh
> @@ -42,7 +42,8 @@ REGEX_ERROR_MESSAGE="Error: Failed to add events."
>  REGEX_INVALID_ARGUMENT="Failed to write event: Invalid argument"
>  REGEX_SYMBOL_FAIL="Failed to find symbol at $RE_ADDRESS"
>  REGEX_OUT_SECTION="$BLACKFUNC is out of \.\w+, skip it"
> -../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err
> +REGEX_BROKEN_DWARF="A function DIE doesn\'t have decl_line\. Maybe broken DWARF\?"
> +../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" "$REGEX_BROKEN_DWARF" < $LOGS_DIR/adding_blacklisted.err
>  CHECK_EXIT_CODE=$?
>  
>  print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC"
> -- 
> 2.47.0.rc0.187.ge670bccf7e-goog

Re: [PATCH] perf test: Fix probe testsuite with a new error message

[Arnaldo Carvalho de Melo]

On Thu, Oct 10, 2024 at 09:48:52AM -0300, Arnaldo Carvalho de Melo wrote:
> On Wed, Oct 09, 2024 at 10:16:20PM -0700, Namhyung Kim wrote:
> > On my system, it's constantly failing because of new error message from
> > perf probe.  It should update the regex pattern to match the message -
> > "A function DIE doesn't have decl_line. Maybe broken DWARF?".
> > 
> >   $ sudo head -n 2 /sys/kernel/debug/kprobes/blacklist | cut -f2
> >   warn_thunk_thunk
> >   asm_exc_divide_error
> > 
> >   $ sudo perf probe warn_thunk_thunk
> >   A function DIE doesn't have decl_line. Maybe broken DWARF?
> >   A function DIE doesn't have decl_line. Maybe broken DWARF?
> >   Probe point 'warn_thunk_thunk' not found.
> >     Error: Failed to add events.
> > 
> >   $ sudo perf probe asm_exc_overflow
> >   Failed to find scope of probe point.
> >     Error: Failed to add events.
> 
> We discussed this in the past, I came up with a similar patch, Veronika
> rightly pointed out that this may point to a real problem, Masami said
> that since these are for DWARF from assembly those are known issues, I
> suggested Veronika checked if the CU where the function came from was
> generated from Assembly (there are DWARF tags that have that info), IIRC
> she said she would try to do it.
> 
> I'll try to find out the threads and see what happened.

https://lore.kernel.org/all/ZvXhJLkJcR99Y2sF@google.com/T/#u

Veronika, was there a v3?

Thanks,

- Arnaldo

Re: [PATCH] perf test: Fix probe testsuite with a new error message

[Veronika Molnarova]

On 10/10/24 14:50, Arnaldo Carvalho de Melo wrote:
> On Thu, Oct 10, 2024 at 09:48:52AM -0300, Arnaldo Carvalho de Melo wrote:
>> On Wed, Oct 09, 2024 at 10:16:20PM -0700, Namhyung Kim wrote:
>>> On my system, it's constantly failing because of new error message from
>>> perf probe.  It should update the regex pattern to match the message -
>>> "A function DIE doesn't have decl_line. Maybe broken DWARF?".
>>>
>>>   $ sudo head -n 2 /sys/kernel/debug/kprobes/blacklist | cut -f2
>>>   warn_thunk_thunk
>>>   asm_exc_divide_error
>>>
>>>   $ sudo perf probe warn_thunk_thunk
>>>   A function DIE doesn't have decl_line. Maybe broken DWARF?
>>>   A function DIE doesn't have decl_line. Maybe broken DWARF?
>>>   Probe point 'warn_thunk_thunk' not found.
>>>     Error: Failed to add events.
>>>
>>>   $ sudo perf probe asm_exc_overflow
>>>   Failed to find scope of probe point.
>>>     Error: Failed to add events.
>>
>> We discussed this in the past, I came up with a similar patch, Veronika
>> rightly pointed out that this may point to a real problem, Masami said
>> that since these are for DWARF from assembly those are known issues, I
>> suggested Veronika checked if the CU where the function came from was
>> generated from Assembly (there are DWARF tags that have that info), IIRC
>> she said she would try to do it.
>>
>> I'll try to find out the threads and see what happened.
> 
> https://lore.kernel.org/all/ZvXhJLkJcR99Y2sF@google.com/T/#u
> 
> Veronika, was there a v3?
> 
> Thanks,
Well the patch was already applied upstream so we are talking about a fix
not v3, right? As Arnaldo mentioned, I got together a fix patch that checks
the CU for the Assembly for blacklisted probe. Michael also suggested the 
possibility to check multiple blacklisted probes if we are not able to check
for the Assembly, so I also added that.

I will send the fix patch soon as part of the perftool upstreaming patchset
to prevent conflicts happening. Hopefully, this can be at the start of the
next week if all of the checks pass.

Thanks,
Veronika
> 
> - Arnaldo
> 

Re: [PATCH] perf test: Fix probe testsuite with a new error message

[Veronika Molnarova]

On 10/10/24 16:40, Arnaldo Carvalho de Melo wrote:
> On Thu, Oct 10, 2024, 10:31 AM Veronika Molnarova <vmolnaro@redhat.com>
> wrote:
> 
>>
>>
>> On 10/10/24 14:50, Arnaldo Carvalho de Melo wrote:
>>> On Thu, Oct 10, 2024 at 09:48:52AM -0300, Arnaldo Carvalho de Melo wrote:
>>>> On Wed, Oct 09, 2024 at 10:16:20PM -0700, Namhyung Kim wrote:
>>>>> On my system, it's constantly failing because of new error message from
>>>>> perf probe.  It should update the regex pattern to match the message -
>>>>> "A function DIE doesn't have decl_line. Maybe broken DWARF?".
>>>>>
>>>>>   $ sudo head -n 2 /sys/kernel/debug/kprobes/blacklist | cut -f2
>>>>>   warn_thunk_thunk
>>>>>   asm_exc_divide_error
>>>>>
>>>>>   $ sudo perf probe warn_thunk_thunk
>>>>>   A function DIE doesn't have decl_line. Maybe broken DWARF?
>>>>>   A function DIE doesn't have decl_line. Maybe broken DWARF?
>>>>>   Probe point 'warn_thunk_thunk' not found.
>>>>>     Error: Failed to add events.
>>>>>
>>>>>   $ sudo perf probe asm_exc_overflow
>>>>>   Failed to find scope of probe point.
>>>>>     Error: Failed to add events.
>>>>
>>>> We discussed this in the past, I came up with a similar patch, Veronika
>>>> rightly pointed out that this may point to a real problem, Masami said
>>>> that since these are for DWARF from assembly those are known issues, I
>>>> suggested Veronika checked if the CU where the function came from was
>>>> generated from Assembly (there are DWARF tags that have that info), IIRC
>>>> she said she would try to do it.
>>>>
>>>> I'll try to find out the threads and see what happened.
>>>
>>> https://lore.kernel.org/all/ZvXhJLkJcR99Y2sF@google.com/T/#u
>>>
>>> Veronika, was there a v3?
>>>
>>> Thanks,
>> Well the patch was already applied upstream so we are talking about a fix
>> not v3, right?
> 
> 
> 
> 
> Can you provide a url for the patch that was applied upstream?
> 
> I just checked at
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh?h=perf-tools-next
> 
> And I'm not seeing it.

Well the patch adding the failing test case that you can see at

https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh?h=perf-tools-next

There was the whole patchset that contained the failing probe test case.
The patchset was applied and merged upstream so there won't be a v3 for 
the failing test case, instead we will send a fix patch resolving the 
mentioned issue. Hopefully this explains the confusion that I think
happened.

We will send the next patchset soon containing also the patch fixing
the probe issue with bad dwarf so if there will still be some confusion 
that should resolve that.

- Veronika

> 
> - Arnaldo
> 
> As Arnaldo mentioned, I got together a fix patch that checks
> 
>> the CU for the Assembly for blacklisted probe. Michael also suggested the
>> possibility to check multiple blacklisted probes if we are not able to
>> check
>> for the Assembly, so I also added that.
>>
>> I will send the fix patch soon as part of the perftool upstreaming patchset
>> to prevent conflicts happening. Hopefully, this can be at the start of the
>> next week if all of the checks pass.
>>
>> Thanks,
>> Veronika
>>>
>>> - Arnaldo
>>>
>>
>>
> 

Re: [PATCH] perf test: Fix probe testsuite with a new error message

[Namhyung Kim]

Hello,

On Tue, Oct 15, 2024 at 12:46:35PM +0200, Veronika Molnarova wrote:
> 
> 
> On 10/10/24 16:40, Arnaldo Carvalho de Melo wrote:
> > On Thu, Oct 10, 2024, 10:31 AM Veronika Molnarova <vmolnaro@redhat.com>
> > wrote:
> > 
> >>
> >>
> >> On 10/10/24 14:50, Arnaldo Carvalho de Melo wrote:
> >>> On Thu, Oct 10, 2024 at 09:48:52AM -0300, Arnaldo Carvalho de Melo wrote:
> >>>> On Wed, Oct 09, 2024 at 10:16:20PM -0700, Namhyung Kim wrote:
> >>>>> On my system, it's constantly failing because of new error message from
> >>>>> perf probe.  It should update the regex pattern to match the message -
> >>>>> "A function DIE doesn't have decl_line. Maybe broken DWARF?".
> >>>>>
> >>>>>   $ sudo head -n 2 /sys/kernel/debug/kprobes/blacklist | cut -f2
> >>>>>   warn_thunk_thunk
> >>>>>   asm_exc_divide_error
> >>>>>
> >>>>>   $ sudo perf probe warn_thunk_thunk
> >>>>>   A function DIE doesn't have decl_line. Maybe broken DWARF?
> >>>>>   A function DIE doesn't have decl_line. Maybe broken DWARF?
> >>>>>   Probe point 'warn_thunk_thunk' not found.
> >>>>>     Error: Failed to add events.
> >>>>>
> >>>>>   $ sudo perf probe asm_exc_overflow
> >>>>>   Failed to find scope of probe point.
> >>>>>     Error: Failed to add events.
> >>>>
> >>>> We discussed this in the past, I came up with a similar patch, Veronika
> >>>> rightly pointed out that this may point to a real problem, Masami said
> >>>> that since these are for DWARF from assembly those are known issues, I
> >>>> suggested Veronika checked if the CU where the function came from was
> >>>> generated from Assembly (there are DWARF tags that have that info), IIRC
> >>>> she said she would try to do it.
> >>>>
> >>>> I'll try to find out the threads and see what happened.
> >>>
> >>> https://lore.kernel.org/all/ZvXhJLkJcR99Y2sF@google.com/T/#u
> >>>
> >>> Veronika, was there a v3?
> >>>
> >>> Thanks,
> >> Well the patch was already applied upstream so we are talking about a fix
> >> not v3, right?
> > 
> > 
> > 
> > 
> > Can you provide a url for the patch that was applied upstream?
> > 
> > I just checked at
> > 
> > https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh?h=perf-tools-next
> > 
> > And I'm not seeing it.
> 
> Well the patch adding the failing test case that you can see at
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh?h=perf-tools-next
> 
> There was the whole patchset that contained the failing probe test case.
> The patchset was applied and merged upstream so there won't be a v3 for 
> the failing test case, instead we will send a fix patch resolving the 
> mentioned issue. Hopefully this explains the confusion that I think
> happened.

Ok, thanks for the explanation.  I see it's merged already and I'm
waiting for the fix only.

> 
> We will send the next patchset soon containing also the patch fixing
> the probe issue with bad dwarf so if there will still be some confusion 
> that should resolve that.
 
It'd be nice if you can send the fix first separately.  As the whole
patchset might take some time for review, I want to fix the existing
problem ASAP.

Thanks,
Namhyung

Re: [PATCH] perf test: Fix probe testsuite with a new error message

[Namhyung Kim]

On Thu, Oct 10, 2024 at 09:48:47AM -0300, Arnaldo Carvalho de Melo wrote:
> On Wed, Oct 09, 2024 at 10:16:20PM -0700, Namhyung Kim wrote:
> > On my system, it's constantly failing because of new error message from
> > perf probe.  It should update the regex pattern to match the message -
> > "A function DIE doesn't have decl_line. Maybe broken DWARF?".
> > 
> >   $ sudo head -n 2 /sys/kernel/debug/kprobes/blacklist | cut -f2
> >   warn_thunk_thunk
> >   asm_exc_divide_error
> > 
> >   $ sudo perf probe warn_thunk_thunk
> >   A function DIE doesn't have decl_line. Maybe broken DWARF?
> >   A function DIE doesn't have decl_line. Maybe broken DWARF?
> >   Probe point 'warn_thunk_thunk' not found.
> >     Error: Failed to add events.
> > 
> >   $ sudo perf probe asm_exc_overflow
> >   Failed to find scope of probe point.
> >     Error: Failed to add events.
> 
> We discussed this in the past, I came up with a similar patch, Veronika
> rightly pointed out that this may point to a real problem, Masami said
> that since these are for DWARF from assembly those are known issues, I
> suggested Veronika checked if the CU where the function came from was
> generated from Assembly (there are DWARF tags that have that info), IIRC
> she said she would try to do it.

Oh ok.  I was lost in the thread. :)

> 
> I'll try to find out the threads and see what happened.

Thanks, I will wait for the next step.
Namhyung

>  
> > Cc: Masami Hiramatsu <mhiramat@kernel.org>
> > Cc: Michael Petlan <mpetlan@redhat.com>
> > Cc: Veronika Molnarova <vmolnaro@redhat.com>
> > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > ---
> >  tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > diff --git a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh
> > index b5dc10b2a73810b3..01e5e09053c37e46 100755
> > --- a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh
> > +++ b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh
> > @@ -42,7 +42,8 @@ REGEX_ERROR_MESSAGE="Error: Failed to add events."
> >  REGEX_INVALID_ARGUMENT="Failed to write event: Invalid argument"
> >  REGEX_SYMBOL_FAIL="Failed to find symbol at $RE_ADDRESS"
> >  REGEX_OUT_SECTION="$BLACKFUNC is out of \.\w+, skip it"
> > -../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err
> > +REGEX_BROKEN_DWARF="A function DIE doesn\'t have decl_line\. Maybe broken DWARF\?"
> > +../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" "$REGEX_BROKEN_DWARF" < $LOGS_DIR/adding_blacklisted.err
> >  CHECK_EXIT_CODE=$?
> >  
> >  print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC"
> > -- 
> > 2.47.0.rc0.187.ge670bccf7e-goog