Issues with Linux perf tool on Raspberry Pi 5

Issues with Linux perf tool on Raspberry Pi 5

[Ian Rogers]

Hi,

I thought I'd report the state of Linux perf tests on Raspberry Pi 5
and so I opened a bug because of the number of failures. I'm happy for
help resolving the issues. The bug is:
https://bugzilla.kernel.org/show_bug.cgi?id=218401

Thanks,
Ian

Re: Issues with Linux perf tool on Raspberry Pi 5

[Namhyung Kim]

Hi Ian,

On Fri, Jan 19, 2024 at 3:40 PM Ian Rogers <irogers@google.com> wrote:
>
> Hi,
>
> I thought I'd report the state of Linux perf tests on Raspberry Pi 5
> and so I opened a bug because of the number of failures. I'm happy for
> help resolving the issues. The bug is:
> https://bugzilla.kernel.org/show_bug.cgi?id=218401

Thanks for doing this.  I've looked at the error log and found
it had a failure in loading BTF.  Maybe the system setup is
more or less broken..?

Thanks,
Namhyung

Re: Issues with Linux perf tool on Raspberry Pi 5

[Ian Rogers]

On Fri, Jan 19, 2024 at 4:55 PM Namhyung Kim <namhyung@gmail.com> wrote:
>
> Hi Ian,
>
> On Fri, Jan 19, 2024 at 3:40 PM Ian Rogers <irogers@google.com> wrote:
> >
> > Hi,
> >
> > I thought I'd report the state of Linux perf tests on Raspberry Pi 5
> > and so I opened a bug because of the number of failures. I'm happy for
> > help resolving the issues. The bug is:
> > https://bugzilla.kernel.org/show_bug.cgi?id=218401
>
> Thanks for doing this.  I've looked at the error log and found
> it had a failure in loading BTF.  Maybe the system setup is
> more or less broken..?

I think so. Looks like the kernel was built with CONFIG_DEBUG_INFO_BTF disabled:
https://forums.raspberrypi.com/viewtopic.php?t=349411
They also package libbpf-tools and those tools fail in the same way.

I think the fix is to update the bcm2712_defconfig:
https://www.raspberrypi.com/documentation/computers/linux_kernel.html
https://github.com/raspberrypi/linux/blob/rpi-6.1.y/arch/arm64/configs/bcm2712_defconfig

It is kind of a shame this option is opt-in rather than opt-out. I
wonder if there is something we should do to diagnose and help users
who may have a BTF disabled kernel.

Thanks,
Ian

Re: Issues with Linux perf tool on Raspberry Pi 5

[Ian Rogers]

On Fri, Jan 19, 2024 at 8:27 PM Ian Rogers <irogers@google.com> wrote:
>
> On Fri, Jan 19, 2024 at 4:55 PM Namhyung Kim <namhyung@gmail.com> wrote:
> >
> > Hi Ian,
> >
> > On Fri, Jan 19, 2024 at 3:40 PM Ian Rogers <irogers@google.com> wrote:
> > >
> > > Hi,
> > >
> > > I thought I'd report the state of Linux perf tests on Raspberry Pi 5
> > > and so I opened a bug because of the number of failures. I'm happy for
> > > help resolving the issues. The bug is:
> > > https://bugzilla.kernel.org/show_bug.cgi?id=218401
> >
> > Thanks for doing this.  I've looked at the error log and found
> > it had a failure in loading BTF.  Maybe the system setup is
> > more or less broken..?
>
> I think so. Looks like the kernel was built with CONFIG_DEBUG_INFO_BTF disabled:
> https://forums.raspberrypi.com/viewtopic.php?t=349411
> They also package libbpf-tools and those tools fail in the same way.
>
> I think the fix is to update the bcm2712_defconfig:
> https://www.raspberrypi.com/documentation/computers/linux_kernel.html
> https://github.com/raspberrypi/linux/blob/rpi-6.1.y/arch/arm64/configs/bcm2712_defconfig
>
> It is kind of a shame this option is opt-in rather than opt-out. I
> wonder if there is something we should do to diagnose and help users
> who may have a BTF disabled kernel.

Rebuilding the kernel with CONFIG_DEBUG_INFO_BTF fixes kernel lock
contention analysis and perf record offcpu profiling tests.

Thanks,
Ian

Re: Issues with Linux perf tool on Raspberry Pi 5

[James Clark]

On 19/01/2024 23:39, Ian Rogers wrote:
> Hi,
> 
> I thought I'd report the state of Linux perf tests on Raspberry Pi 5
> and so I opened a bug because of the number of failures. I'm happy for
> help resolving the issues. The bug is:
> https://bugzilla.kernel.org/show_bug.cgi?id=218401
> 
> Thanks,
> Ian
> 

Hi Ian,

For "test perf probe of function from different CU", Chaitanya has a fix
that she should be sending soon. I already planned to look at "Check
Arm64 callgraphs are complete in fp mode" as I see that failing here so
it's not related to Rpi5.

The others as you mentioned do look to be config related. IMO that's
usually the issue, or it's a build issue where Perf is missing a dependency.

Although those should be a skip so technically there are still bugs.

Thanks
James

Re: Issues with Linux perf tool on Raspberry Pi 5

[Ian Rogers]

On Mon, Jan 22, 2024 at 1:59 AM James Clark <james.clark@arm.com> wrote:
>
> On 19/01/2024 23:39, Ian Rogers wrote:
> > Hi,
> >
> > I thought I'd report the state of Linux perf tests on Raspberry Pi 5
> > and so I opened a bug because of the number of failures. I'm happy for
> > help resolving the issues. The bug is:
> > https://bugzilla.kernel.org/show_bug.cgi?id=218401
> >
> > Thanks,
> > Ian
> >
>
> Hi Ian,
>
> For "test perf probe of function from different CU", Chaitanya has a fix
> that she should be sending soon. I already planned to look at "Check
> Arm64 callgraphs are complete in fp mode" as I see that failing here so
> it's not related to Rpi5.
>
> The others as you mentioned do look to be config related. IMO that's
> usually the issue, or it's a build issue where Perf is missing a dependency.
>
> Although those should be a skip so technically there are still bugs.

Thanks James,

agreed on the fails not being skips still being errors. I'm going to
look to see if we can't at least make the BTF not found errors warn
that the kernel probably wasn't compiled with BTF enabled. I look
forward to testing the Rpi5 fixes :-)

Ian

> Thanks
> James

Re: Issues with Linux perf tool on Raspberry Pi 5

[James Clark]

On 22/01/2024 22:39, Ian Rogers wrote:
> On Mon, Jan 22, 2024 at 1:59 AM James Clark <james.clark@arm.com> wrote:
>>
>> On 19/01/2024 23:39, Ian Rogers wrote:
>>> Hi,
>>>
>>> I thought I'd report the state of Linux perf tests on Raspberry Pi 5
>>> and so I opened a bug because of the number of failures. I'm happy for
>>> help resolving the issues. The bug is:
>>> https://bugzilla.kernel.org/show_bug.cgi?id=218401
>>>
>>> Thanks,
>>> Ian
>>>
>>
>> Hi Ian,
>>
>> For "test perf probe of function from different CU", Chaitanya has a fix
>> that she should be sending soon. I already planned to look at "Check
>> Arm64 callgraphs are complete in fp mode" as I see that failing here so
>> it's not related to Rpi5.
>>

I see that "test dwarf unwind" is completely missing in your output, not
even skipped. This suggests that libdwarf isn't linked in:

#ifdef HAVE_DWARF_UNWIND_SUPPORT
	&suite__dwarf_unwind,
#endif

"Check Arm64 callgraphs are complete in fp mode" relies on a Dwarf
unwinder so I sent a patch to skip this test if one isn't linked in.