Re: [GIT PULL] perf-tools changes for v6.13

[Namhyung Kim <namhyung@kernel.org>]

On Tue, Nov 26, 2024 at 09:59:44PM -0300, Arnaldo Carvalho de Melo wrote:
> On Tue, Nov 26, 2024 at 04:00:12PM -0800, Namhyung Kim wrote:
> > Hello,
> > 
> > On Tue, Nov 26, 2024 at 03:33:10PM -0800, Linus Torvalds wrote:
> > > On Sun, 24 Nov 2024 at 23:17, Namhyung Kim <namhyung@kernel.org> wrote:
> > > >
> > > > Please consider pulling the following changes in perf tools for v6.13.
> > > 
> > > Hmm. Has the default search path for the vmlinux image changed?
> > 
> > I don't think so.
>  
> > > Doing profiling with this, I get
> > > 
> > >     openat(AT_FDCWD, "vmlinux", O_RDONLY)   = 5
> > > 
> > > which is very very wrong for my kernel build - it basically picks up
> > > the vmlinux image from the build directory, which has absolutely
> > > *nothign* to do with the image that is being run at the moment.
> > 
> > Sure, at least it should match the build-id before use.
> 
> I tried it here and at first I reproduced Linus results, which I found
> very strange, then I tried to bisect it, and all the previous versions
> were producing this:
> 
> root@number:/home/acme/git/perf-tools# perf -v ; perf report -vv |& grep -A5 "^Looking at the vmlinux_path"
> perf version 6.12.gb50ecc5aca4d
> Looking at the vmlinux_path (8 entries long)
> symsrc__init: build id mismatch for vmlinux.
> symsrc__init: cannot get elf header.
> overlapping maps:
>  ffffffffc034b75c-ffffffffc034b89b 0 bpf_prog_40ddf486530245f5_sd_devices
>  ffffffffc0000000-ffffffffff000000 7fffc0004000 vmlinux
> root@number:/home/acme/git/perf-tools#
> 
> Which is what I got from the perf-tools branch you asked him to pull.
> 
> The algorithm in perf is similar to the one in pahole (that stole it
> from perf):
> 
> root@number:/home/acme/git/perf-tools# strace -e openat pahole --running_kernel_vmlinux |& tail -12
> openat(AT_FDCWD, "/sys/kernel/notes", O_RDONLY) = 3
> openat(AT_FDCWD, "vmlinux", O_RDONLY)   = 3
> openat(AT_FDCWD, "/boot/vmlinux", O_RDONLY) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/boot/vmlinux-6.11.8-200.fc40.x86_64", O_RDONLY) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/usr/lib/debug/boot/vmlinux-6.11.8-200.fc40.x86_64", O_RDONLY) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/lib/modules/6.11.8-200.fc40.x86_64/build/vmlinux", O_RDONLY) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/usr/lib/debug/lib/modules/6.11.8-200.fc40.x86_64/vmlinux", O_RDONLY) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/usr/lib/debug/boot/vmlinux-6.11.8-200.fc40.x86_64.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/root/.cache/debuginfod_client/55152ddca5da77ca62deb3dd0db105e82b3711e0/debuginfo", O_RDONLY) = -1 ENOENT (No such file or directory)
> pahole: couldn't find a vmlinux that matches the running kernel
> HINT: Maybe you're inside a container or missing a debuginfo package?
> +++ exited with 1 +++
> root@number:/home/acme/git/perf-tools#
> 
> The one in perf does very much the same:
> 
> root@number:/home/acme/git/perf-tools# perf -v ; strace -e openat -o /tmp/strace.output perf report -vv |& grep -A5 "^Looking at the vmlinux_path" ; grep '"vmlinux"' -B1 -A10 /tmp/strace.output 
> perf version 6.12.gb50ecc5aca4d
> Looking at the vmlinux_path (8 entries long)
> symsrc__init: build id mismatch for vmlinux.
> symsrc__init: cannot get elf header.
> overlapping maps:
>  ffffffffc034b75c-ffffffffc034b89b 0 bpf_prog_40ddf486530245f5_sd_devices
>  ffffffffc0000000-ffffffffff000000 7fffc0004000 vmlinux
> openat(AT_FDCWD, "/root/.debug/.build-id/55/152ddca5da77ca62deb3dd0db105e82b3711e0/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "vmlinux", O_RDONLY)   = 142
> openat(AT_FDCWD, "/boot/vmlinux", O_RDONLY) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/boot/vmlinux-6.11.8-200.fc40.x86_64", O_RDONLY) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/usr/lib/debug/boot/vmlinux-6.11.8-200.fc40.x86_64", O_RDONLY) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/lib/modules/6.11.8-200.fc40.x86_64/build/vmlinux", O_RDONLY) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/usr/lib/debug/lib/modules/6.11.8-200.fc40.x86_64/vmlinux", O_RDONLY) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/usr/lib/debug/boot/vmlinux-6.11.8-200.fc40.x86_64.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/root/.debug/.build-id/55/152ddca5da77ca62deb3dd0db105e82b3711e0/kallsyms", O_RDONLY) = 142
> openat(AT_FDCWD, "/sys/kernel/notes", O_RDONLY) = 142
> openat(AT_FDCWD, "/proc/kcore", O_RDONLY) = 142
> openat(AT_FDCWD, "/proc/kallsyms", O_RDONLY) = 142
> root@number:/home/acme/git/perf-tools# 
> root@number:/home/acme/git/perf-tools# perf buildid-list -h -k
> 
>  Usage: perf buildid-list [<options>]
> 
>     -k, --kernel          Show current kernel build id
> 
> root@number:/home/acme/git/perf-tools# perf buildid-list -k
> 55152ddca5da77ca62deb3dd0db105e82b3711e0
> root@number:/home/acme/git/perf-tools# file vmlinux 
> vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=e86cad482822bd4b4caabb01e7cd00161fd51a38, with debug_info, not stripped
> root@number:/home/acme/git/perf-tools#
> 
> And it ends up using /proc/kallsyms
> 
> Today I did some bisects, etc, but even with at first reproducing what
> Linus described, i.e. it used the non-matching vmlinux in the current
> directory to resolve symbols, I couldn't reproduce it after trying to
> bisect it :-\
>  
> > > So now I need to point "perf report" at the actual image, which I
> > > didn't need to do before.
> > 
> > Ok, I'll try to reproduce it and take a look.
> 
> Hope you root cause this, late in this part of the world, going AFK now.

I think it's a bug in perf record since v6.12.  I found the build-id
event in the header area is broken.  Can you verify if this works?

Thanks,
Namhyung

---8<---
From 62ebc7de4c306d99f39af847bee4a2aab05b0f93 Mon Sep 17 00:00:00 2001
From: Namhyung Kim <namhyung@kernel.org>
Date: Tue, 26 Nov 2024 19:13:31 -0800
Subject: [PATCH] perf tools: Fix build-id event recording

The build-id events written at the end of the record session are broken
due to unexpected data.  The write_buildid() writes the fixed length
event first and then variable length filename.

But a recent change made it write more data in the padding area
accidentally.  So readers of the event see zero-filled data for the
next entry and treat it incorrectly.   This resulted in wrong kernel
symbols because the kernel DSO loaded a random vmlinux image in the
path as it didn't have a valid build-id.

Fixes: ae39ba16554e ("perf inject: Fix build ID injection")
Reported-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/build-id.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index 8982f68e7230cd64..e763e8d99a4367d7 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -277,7 +277,7 @@ static int write_buildid(const char *name, size_t name_len, struct build_id *bid
 	struct perf_record_header_build_id b;
 	size_t len;
 
-	len = sizeof(b) + name_len + 1;
+	len = name_len + 1;
 	len = PERF_ALIGN(len, sizeof(u64));
 
 	memset(&b, 0, sizeof(b));
@@ -286,7 +286,7 @@ static int write_buildid(const char *name, size_t name_len, struct build_id *bid
 	misc |= PERF_RECORD_MISC_BUILD_ID_SIZE;
 	b.pid = pid;
 	b.header.misc = misc;
-	b.header.size = len;
+	b.header.size = sizeof(b) + len;
 
 	err = do_write(fd, &b, sizeof(b));
 	if (err < 0)
-- 
2.47.0.338.g60cca15819-goog