* [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
@ 2023-07-26 1:25 Masami Hiramatsu
2023-07-26 10:04 ` Jiri Olsa
0 siblings, 1 reply; 14+ messages in thread
From: Masami Hiramatsu @ 2023-07-26 1:25 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Yonghong Song; +Cc: dwarves, bpf, Steven Rostedt
[-- Attachment #1: Type: text/plain, Size: 1618 bytes --]
Hello,
(I resend this because kconfig was too big)
I found that BTF is not generated for gcc-built kernel with that latest
pahole (v1.25).
When I'm using the distro origin pahole (v1.22) it works. (I also checked
v1.23 and v1.24, both partially generated BTF)
e.g.
# echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events
sh: write error: Invalid argument
# cat /sys/kernel/tracing/error_log
[ 21.595724] trace_fprobe: error: BTF is not available or not supported
Command: f kfree $arg*
^
[ 21.596032] trace_fprobe: error: Invalid $-valiable specified
Command: f kfree $arg*
^
/ # strings /sys/kernel/btf/vmlinux | grep kfree
kfree_on_online
maybe_kfree_parameter
trace_event_raw_rcu_invoke_kfree_bulk_callback
trace_event_data_offsets_rcu_invoke_kfree_bulk_callback
btf_trace_rcu_invoke_kfree_bulk_callback
early_boot_kfree_rcu
__bpf_trace_rcu_invoke_kfree_bulk_callback
perf_trace_rcu_invoke_kfree_bulk_callback
trace_event_raw_event_rcu_invoke_kfree_bulk_callback
trace_raw_output_rcu_invoke_kfree_bulk_callback
__probestub_rcu_invoke_kfree_bulk_callback
__traceiter_rcu_invoke_kfree_bulk_callback
kfree_rcu_cpu_work
kfree_rcu_cpu
kfree_rcu_batch_init
kfree_rcu_scheduler_running
kfree_rcu_shrink_scan
kfree_rcu_shrink_count
kfree_rcu_monitor
kfree_rcu_work
Here is the gcc version which I'm using.
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1)
I also attached the kernel config file.
What is the recommended combination of the tools?
Should I use Clang to build the kernel for BTF?
Thank you,
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
[-- Attachment #2: config.tar.xz --]
[-- Type: application/x-xz, Size: 22156 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole 2023-07-26 1:25 [RESEND] BTF is not generated for gcc-built kernel with the latest pahole Masami Hiramatsu @ 2023-07-26 10:04 ` Jiri Olsa 2023-07-26 11:07 ` Masami Hiramatsu 0 siblings, 1 reply; 14+ messages in thread From: Jiri Olsa @ 2023-07-26 10:04 UTC (permalink / raw) To: Masami Hiramatsu Cc: Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf, Steven Rostedt On Wed, Jul 26, 2023 at 10:25:34AM +0900, Masami Hiramatsu wrote: > Hello, > (I resend this because kconfig was too big) > > I found that BTF is not generated for gcc-built kernel with that latest > pahole (v1.25). hi, I can't reproduce on my setup with your .config does 'bpftool btf dump file ./vmlinux' show any error? is there any error in the kernel build output? > When I'm using the distro origin pahole (v1.22) it works. (I also checked > v1.23 and v1.24, both partially generated BTF) > > e.g. > > # echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events > sh: write error: Invalid argument > > # cat /sys/kernel/tracing/error_log > [ 21.595724] trace_fprobe: error: BTF is not available or not supported > Command: f kfree $arg* > ^ > [ 21.596032] trace_fprobe: error: Invalid $-valiable specified > Command: f kfree $arg* > ^ > > / # strings /sys/kernel/btf/vmlinux | grep kfree hm, if you have this file present, you have BTF in > kfree_on_online > maybe_kfree_parameter > trace_event_raw_rcu_invoke_kfree_bulk_callback > trace_event_data_offsets_rcu_invoke_kfree_bulk_callback > btf_trace_rcu_invoke_kfree_bulk_callback > early_boot_kfree_rcu > __bpf_trace_rcu_invoke_kfree_bulk_callback > perf_trace_rcu_invoke_kfree_bulk_callback > trace_event_raw_event_rcu_invoke_kfree_bulk_callback > trace_raw_output_rcu_invoke_kfree_bulk_callback > __probestub_rcu_invoke_kfree_bulk_callback > __traceiter_rcu_invoke_kfree_bulk_callback > kfree_rcu_cpu_work > kfree_rcu_cpu > kfree_rcu_batch_init > kfree_rcu_scheduler_running > kfree_rcu_shrink_scan > kfree_rcu_shrink_count > kfree_rcu_monitor > kfree_rcu_work > > > Here is the gcc version which I'm using. > > gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1) I tried with gcc (GCC) 13.1.1 20230614 (Red Hat 13.1.1-4) and latest pahole (master branch) > > I also attached the kernel config file. > > What is the recommended combination of the tools? > Should I use Clang to build the kernel for BTF? should work fine with both gcc and clang jirka > > Thank you, > > -- > Masami Hiramatsu (Google) <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole 2023-07-26 10:04 ` Jiri Olsa @ 2023-07-26 11:07 ` Masami Hiramatsu 2023-07-26 14:46 ` Alan Maguire 0 siblings, 1 reply; 14+ messages in thread From: Masami Hiramatsu @ 2023-07-26 11:07 UTC (permalink / raw) To: Jiri Olsa Cc: Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf, Steven Rostedt Hi Jiri, On Wed, 26 Jul 2023 12:04:08 +0200 Jiri Olsa <olsajiri@gmail.com> wrote: > On Wed, Jul 26, 2023 at 10:25:34AM +0900, Masami Hiramatsu wrote: > > Hello, > > (I resend this because kconfig was too big) > > > > I found that BTF is not generated for gcc-built kernel with that latest > > pahole (v1.25). > > hi, > I can't reproduce on my setup with your .config > > does 'bpftool btf dump file ./vmlinux' show any error? > > is there any error in the kernel build output? Yes, here it is. I saw these 2 lines. die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)! die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)! > > > When I'm using the distro origin pahole (v1.22) it works. (I also checked > > v1.23 and v1.24, both partially generated BTF) > > > > e.g. > > > > # echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events > > sh: write error: Invalid argument > > > > # cat /sys/kernel/tracing/error_log > > [ 21.595724] trace_fprobe: error: BTF is not available or not supported > > Command: f kfree $arg* > > ^ > > [ 21.596032] trace_fprobe: error: Invalid $-valiable specified > > Command: f kfree $arg* > > ^ > > > > / # strings /sys/kernel/btf/vmlinux | grep kfree > > hm, if you have this file present, you have BTF in Yes, it seems that the BTF itself is generated, but many entries seems dropped compared with pahole v1.22. So, if a given symbol has BTF, (e.g. kfree_rcu_batch_init) it works. > > > kfree_on_online > > maybe_kfree_parameter > > trace_event_raw_rcu_invoke_kfree_bulk_callback > > trace_event_data_offsets_rcu_invoke_kfree_bulk_callback > > btf_trace_rcu_invoke_kfree_bulk_callback > > early_boot_kfree_rcu > > __bpf_trace_rcu_invoke_kfree_bulk_callback > > perf_trace_rcu_invoke_kfree_bulk_callback > > trace_event_raw_event_rcu_invoke_kfree_bulk_callback > > trace_raw_output_rcu_invoke_kfree_bulk_callback > > __probestub_rcu_invoke_kfree_bulk_callback > > __traceiter_rcu_invoke_kfree_bulk_callback > > kfree_rcu_cpu_work > > kfree_rcu_cpu > > kfree_rcu_batch_init > > kfree_rcu_scheduler_running > > kfree_rcu_shrink_scan > > kfree_rcu_shrink_count > > kfree_rcu_monitor > > kfree_rcu_work > > > > > > Here is the gcc version which I'm using. > > > > gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1) > > I tried with gcc (GCC) 13.1.1 20230614 (Red Hat 13.1.1-4) > and latest pahole (master branch) Curiously, with Clang 16.0.0, it works (but many different errors are shown, see below *). So the combination of gcc/clang and pahole version can affect it. > > > > > I also attached the kernel config file. > > > > What is the recommended combination of the tools? > > Should I use Clang to build the kernel for BTF? > > should work fine with both gcc and clang And I guess it depends on compiler version, doesn't it? Thank you, (*) BTF .btf.vmlinux.bin.o die__process_unit: DW_TAG_label (0xa) @ <0x4b138> not handled! die__process_unit: tag not supported 0xa (label)! die__process_unit: DW_TAG_label (0xa) @ <0x4b241> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b263> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b290> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b2c0> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b2eb> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b317> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b33a> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b35a> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b37d> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b39e> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b3c4> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b3e7> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b40f> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b435> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b457> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b477> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b4a5> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b4d5> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b505> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b530> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b560> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b585> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b5b2> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b5d9> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b605> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b62e> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b652> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b670> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b694> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b6b3> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b6d9> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b705> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b72b> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b753> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b77f> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b7b3> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b7e4> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b811> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b83c> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b869> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b88c> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b8bd> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b8e7> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b90b> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b930> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b960> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b997> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4b9ce> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4ba00> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4ba24> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4ba4d> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4ba89> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4babc> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4bae4> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4bb0b> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4bb2e> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4bb4e> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4bb6d> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4bb8a> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4bba8> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4bbc5> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4bbe1> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4bc01> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4bc1c> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4bc38> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4bc58> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4bc79> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4bc95> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4bcb2> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x7efd6> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x7effe> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x7f11c> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x7f143> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x7f178> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x7f1a4> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x7f1ca> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x7f1fb> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x7f22f> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x7f25a> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x7f28d> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x7f2b7> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x1eea81> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x1eea9d> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x1eeac3> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x1eeaf3> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x1eeb0f> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x1eeb30> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x1eeb59> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x340734> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4c6b4a> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4c6b67> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4c6b8a> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4c6ba5> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4c6bc4> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4c6bea> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4c6c07> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4c6c2a> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4c6c4e> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4c6c79> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4c6c9b> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4c6cc3> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4c6ceb> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x4c6d15> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x30626ac> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x30626cd> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x3062814> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x3062833> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x3062b8b> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f623f6> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f62416> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f62437> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f62458> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f6280b> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f62b09> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f62b37> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f62c45> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f62c60> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f62d69> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f62e8d> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f6305a> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f63ec2> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f63edf> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f63efc> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f63f19> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f63f36> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f63f5b> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f63f80> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f63fa5> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f63fca> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f6fcae> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f6fcc7> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f6fed7> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f6fef0> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f6fdcb> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70352> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f7036f> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70394> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f703b1> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f703d6> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f703f3> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70418> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70435> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f7045a> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f7057b> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f705a6> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f705cf> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f705f8> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70621> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f7064a> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70673> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f7069c> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f706c5> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f706ee> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70716> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f7073e> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70767> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70790> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f707b9> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f707e2> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f7080b> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70834> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70864> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70892> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f708c0> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f708ee> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f7091c> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f7094a> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70978> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f709a6> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f709d4> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70a01> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70a2e> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70a5c> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70a8a> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70ab8> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70ae6> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70b14> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70b42> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70b72> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70ba0> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70bce> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70bfc> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70c2a> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70c58> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70c86> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70cb4> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70ce2> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70d0f> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70d3c> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70d6a> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70d98> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70dc6> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70df4> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70e22> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70e50> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70e71> not handled! die__process_unit: DW_TAG_label (0xa) @ <0x5f70e94> not handled! LD .tmp_vmlinux.kallsyms1 > > jirka > > > > > Thank you, > > > > -- > > Masami Hiramatsu (Google) <mhiramat@kernel.org> > > -- Masami Hiramatsu (Google) <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole 2023-07-26 11:07 ` Masami Hiramatsu @ 2023-07-26 14:46 ` Alan Maguire 2023-07-27 0:38 ` Masami Hiramatsu 0 siblings, 1 reply; 14+ messages in thread From: Alan Maguire @ 2023-07-26 14:46 UTC (permalink / raw) To: Masami Hiramatsu (Google), Jiri Olsa Cc: Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf, Steven Rostedt On 26/07/2023 12:07, Masami Hiramatsu (Google) wrote: > Hi Jiri, > > On Wed, 26 Jul 2023 12:04:08 +0200 > Jiri Olsa <olsajiri@gmail.com> wrote: > >> On Wed, Jul 26, 2023 at 10:25:34AM +0900, Masami Hiramatsu wrote: >>> Hello, >>> (I resend this because kconfig was too big) >>> >>> I found that BTF is not generated for gcc-built kernel with that latest >>> pahole (v1.25). >> >> hi, >> I can't reproduce on my setup with your .config >> >> does 'bpftool btf dump file ./vmlinux' show any error? >> >> is there any error in the kernel build output? > > Yes, here it is. I saw these 2 lines. > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)! > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)! This is strange, looks like some CUs were encoded incorrectly or we are parsing incorrectly. The error originates in die__process() and happens if the dwarf_tag() associated with the DIE isn't an expected unit; it's not even a valid tag value (0) it looks like. I've not built with gcc 13 yet so it's possible that's the reason you're seeing this, I'll try to reproduce it.. > >> >>> When I'm using the distro origin pahole (v1.22) it works. (I also checked >>> v1.23 and v1.24, both partially generated BTF) >>> >>> e.g. >>> >>> # echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events >>> sh: write error: Invalid argument >>> >>> # cat /sys/kernel/tracing/error_log >>> [ 21.595724] trace_fprobe: error: BTF is not available or not supported >>> Command: f kfree $arg* >>> ^ >>> [ 21.596032] trace_fprobe: error: Invalid $-valiable specified >>> Command: f kfree $arg* >>> ^ >>> >>> / # strings /sys/kernel/btf/vmlinux | grep kfree >> >> hm, if you have this file present, you have BTF in > > Yes, it seems that the BTF itself is generated, but many entries seems > dropped compared with pahole v1.22. So, if a given symbol has BTF, (e.g. > kfree_rcu_batch_init) it works. > Yep, BPF generation is more selective about what it emits in 1.25 to avoid cases where a kernel function signature is ambiguous (multiple functions of the same name with different signatures) or where it has unexpected register use. You can observe this via pahole's --verbose option (a lot of outut is emitted): In a built kernel directory (where unstripped vmlinux is present): $ PAHOLE_FLAGS=$(./scripts/pahole_flags) $ PAHOLE=/usr/local/bin/pahole $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out If you want to investigate why a function has been left out, look for "skipping" verbose output like this: skipping addition of 'access_error'(access_error) due to multiple inconsistent function prototypes skipping addition of 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) due to unexpected register used for parameter FWIW I see most of the below in my BTF output on bpf-next, though I am missing a few, possibly due to differing config options since they don't appear in kallsyms either. I don't see the DWARF tag label not handled messages so it's possible that's a symptom of something I suspect however some form of corruption in the DWARF representation may be the reason a lot of these are missing. Would be worth trying to "objdump -g vmlinux >vmlinux.dwarf" (file will be huge tho) I suspect. >> >>> kfree_on_online >>> maybe_kfree_parameter >>> trace_event_raw_rcu_invoke_kfree_bulk_callback >>> trace_event_data_offsets_rcu_invoke_kfree_bulk_callback >>> btf_trace_rcu_invoke_kfree_bulk_callback >>> early_boot_kfree_rcu >>> __bpf_trace_rcu_invoke_kfree_bulk_callback >>> perf_trace_rcu_invoke_kfree_bulk_callback >>> trace_event_raw_event_rcu_invoke_kfree_bulk_callback >>> trace_raw_output_rcu_invoke_kfree_bulk_callback >>> __probestub_rcu_invoke_kfree_bulk_callback >>> __traceiter_rcu_invoke_kfree_bulk_callback >>> kfree_rcu_cpu_work >>> kfree_rcu_cpu >>> kfree_rcu_batch_init >>> kfree_rcu_scheduler_running >>> kfree_rcu_shrink_scan >>> kfree_rcu_shrink_count >>> kfree_rcu_monitor >>> kfree_rcu_work >>> >>> >>> Here is the gcc version which I'm using. >>> >>> gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1) >> >> I tried with gcc (GCC) 13.1.1 20230614 (Red Hat 13.1.1-4) >> and latest pahole (master branch) > > Curiously, with Clang 16.0.0, it works (but many different errors are shown, > see below *). > So the combination of gcc/clang and pahole version can affect it. > >> >>> >>> I also attached the kernel config file. >>> >>> What is the recommended combination of the tools? >>> Should I use Clang to build the kernel for BTF? >> >> should work fine with both gcc and clang > > And I guess it depends on compiler version, doesn't it? > > Thank you, > > > (*) > BTF .btf.vmlinux.bin.o > die__process_unit: DW_TAG_label (0xa) @ <0x4b138> not handled! > die__process_unit: tag not supported 0xa (label)! > die__process_unit: DW_TAG_label (0xa) @ <0x4b241> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b263> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b290> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b2c0> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b2eb> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b317> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b33a> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b35a> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b37d> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b39e> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b3c4> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b3e7> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b40f> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b435> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b457> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b477> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b4a5> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b4d5> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b505> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b530> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b560> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b585> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b5b2> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b5d9> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b605> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b62e> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b652> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b670> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b694> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b6b3> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b6d9> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b705> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b72b> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b753> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b77f> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b7b3> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b7e4> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b811> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b83c> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b869> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b88c> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b8bd> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b8e7> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b90b> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b930> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b960> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b997> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4b9ce> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4ba00> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4ba24> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4ba4d> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4ba89> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4babc> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4bae4> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4bb0b> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4bb2e> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4bb4e> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4bb6d> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4bb8a> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4bba8> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4bbc5> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4bbe1> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4bc01> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4bc1c> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4bc38> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4bc58> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4bc79> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4bc95> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4bcb2> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x7efd6> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x7effe> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x7f11c> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x7f143> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x7f178> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x7f1a4> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x7f1ca> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x7f1fb> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x7f22f> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x7f25a> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x7f28d> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x7f2b7> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x1eea81> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x1eea9d> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x1eeac3> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x1eeaf3> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x1eeb0f> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x1eeb30> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x1eeb59> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x340734> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4c6b4a> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4c6b67> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4c6b8a> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4c6ba5> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4c6bc4> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4c6bea> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4c6c07> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4c6c2a> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4c6c4e> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4c6c79> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4c6c9b> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4c6cc3> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4c6ceb> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x4c6d15> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x30626ac> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x30626cd> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x3062814> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x3062833> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x3062b8b> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f623f6> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f62416> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f62437> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f62458> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f6280b> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f62b09> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f62b37> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f62c45> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f62c60> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f62d69> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f62e8d> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f6305a> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f63ec2> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f63edf> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f63efc> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f63f19> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f63f36> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f63f5b> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f63f80> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f63fa5> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f63fca> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f6fcae> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f6fcc7> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f6fed7> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f6fef0> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f6fdcb> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70352> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f7036f> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70394> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f703b1> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f703d6> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f703f3> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70418> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70435> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f7045a> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f7057b> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f705a6> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f705cf> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f705f8> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70621> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f7064a> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70673> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f7069c> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f706c5> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f706ee> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70716> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f7073e> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70767> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70790> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f707b9> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f707e2> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f7080b> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70834> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70864> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70892> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f708c0> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f708ee> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f7091c> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f7094a> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70978> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f709a6> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f709d4> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70a01> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70a2e> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70a5c> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70a8a> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70ab8> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70ae6> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70b14> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70b42> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70b72> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70ba0> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70bce> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70bfc> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70c2a> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70c58> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70c86> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70cb4> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70ce2> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70d0f> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70d3c> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70d6a> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70d98> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70dc6> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70df4> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70e22> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70e50> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70e71> not handled! > die__process_unit: DW_TAG_label (0xa) @ <0x5f70e94> not handled! > LD .tmp_vmlinux.kallsyms1 > >> >> jirka >> >>> >>> Thank you, >>> >>> -- >>> Masami Hiramatsu (Google) <mhiramat@kernel.org> >> >> > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole 2023-07-26 14:46 ` Alan Maguire @ 2023-07-27 0:38 ` Masami Hiramatsu 2023-07-27 1:51 ` Masami Hiramatsu 2023-07-27 13:10 ` Arnaldo Carvalho de Melo 0 siblings, 2 replies; 14+ messages in thread From: Masami Hiramatsu @ 2023-07-27 0:38 UTC (permalink / raw) To: Alan Maguire Cc: Jiri Olsa, Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf, Steven Rostedt On Wed, 26 Jul 2023 15:46:03 +0100 Alan Maguire <alan.maguire@oracle.com> wrote: > On 26/07/2023 12:07, Masami Hiramatsu (Google) wrote: > > Hi Jiri, > > > > On Wed, 26 Jul 2023 12:04:08 +0200 > > Jiri Olsa <olsajiri@gmail.com> wrote: > > > >> On Wed, Jul 26, 2023 at 10:25:34AM +0900, Masami Hiramatsu wrote: > >>> Hello, > >>> (I resend this because kconfig was too big) > >>> > >>> I found that BTF is not generated for gcc-built kernel with that latest > >>> pahole (v1.25). > >> > >> hi, > >> I can't reproduce on my setup with your .config > >> > >> does 'bpftool btf dump file ./vmlinux' show any error? > >> > >> is there any error in the kernel build output? > > > > Yes, here it is. I saw these 2 lines. > > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)! > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)! > > This is strange, looks like some CUs were encoded incorrectly or we are > parsing incorrectly. The error originates in die__process() and happens > if the dwarf_tag() associated with the DIE isn't an expected unit; it's > not even a valid tag value (0) it looks like. I've not built with gcc 13 > yet so it's possible that's the reason you're seeing this, I'll try to > reproduce it.. And this warning message is not good for debugging. It would better dump the message with DIE index so that we can analyze the DWARF with other tools. > >>> When I'm using the distro origin pahole (v1.22) it works. (I also checked > >>> v1.23 and v1.24, both partially generated BTF) > >>> > >>> e.g. > >>> > >>> # echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events > >>> sh: write error: Invalid argument > >>> > >>> # cat /sys/kernel/tracing/error_log > >>> [ 21.595724] trace_fprobe: error: BTF is not available or not supported > >>> Command: f kfree $arg* > >>> ^ > >>> [ 21.596032] trace_fprobe: error: Invalid $-valiable specified > >>> Command: f kfree $arg* > >>> ^ > >>> > >>> / # strings /sys/kernel/btf/vmlinux | grep kfree > >> > >> hm, if you have this file present, you have BTF in > > > > Yes, it seems that the BTF itself is generated, but many entries seems > > dropped compared with pahole v1.22. So, if a given symbol has BTF, (e.g. > > kfree_rcu_batch_init) it works. > > > > Yep, BPF generation is more selective about what it emits in 1.25 to > avoid cases where a kernel function signature is ambiguous (multiple > functions of the same name with different signatures) or where it has > unexpected register use. You can observe this via pahole's --verbose > option (a lot of outut is emitted): > > In a built kernel directory (where unstripped vmlinux is present): > $ PAHOLE_FLAGS=$(./scripts/pahole_flags) > $ PAHOLE=/usr/local/bin/pahole > $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out So this will generate BTF from vmlinux DWARF info. > If you want to investigate why a function has been left out, look for > "skipping" verbose output like this: > > skipping addition of 'access_error'(access_error) due to multiple > inconsistent function prototypes > skipping addition of > 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) > due to unexpected register used for parameter Ah, that's nice. Let me try. > > FWIW I see most of the below in my BTF output on bpf-next, though I am > missing a few, possibly due to differing config options since they don't > appear in kallsyms either. I don't see the DWARF tag label not handled > messages so it's possible that's a symptom of something > > I suspect however some form of corruption in the DWARF representation > may be the reason a lot of these are missing. Would be worth trying > to "objdump -g vmlinux >vmlinux.dwarf" (file will be huge tho) I suspect. OK, let me dump the dwarf and check what information it has for "kfree". Thank you, -- Masami Hiramatsu (Google) <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole 2023-07-27 0:38 ` Masami Hiramatsu @ 2023-07-27 1:51 ` Masami Hiramatsu 2023-07-31 15:45 ` Alan Maguire 2023-07-27 13:10 ` Arnaldo Carvalho de Melo 1 sibling, 1 reply; 14+ messages in thread From: Masami Hiramatsu @ 2023-07-27 1:51 UTC (permalink / raw) To: Masami Hiramatsu Cc: Alan Maguire, Jiri Olsa, Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf, Steven Rostedt On Thu, 27 Jul 2023 09:38:14 +0900 Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote: > > Yep, BPF generation is more selective about what it emits in 1.25 to > > avoid cases where a kernel function signature is ambiguous (multiple > > functions of the same name with different signatures) or where it has > > unexpected register use. You can observe this via pahole's --verbose > > option (a lot of outut is emitted): > > > > In a built kernel directory (where unstripped vmlinux is present): > > $ PAHOLE_FLAGS=$(./scripts/pahole_flags) > > $ PAHOLE=/usr/local/bin/pahole > > $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out > > So this will generate BTF from vmlinux DWARF info. > > > If you want to investigate why a function has been left out, look for > > "skipping" verbose output like this: > > > > skipping addition of 'access_error'(access_error) due to multiple > > inconsistent function prototypes > > skipping addition of > > 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) > > due to unexpected register used for parameter > > Ah, that's nice. Let me try. $ pahole --version v1.23 $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)! OK, so something failed. $ grep skipping /tmp/pahole.out | wc -l 0 Nothing to be skipped. $ grep -w kfree /tmp/pahole.out | wc -l 0 $ grep -w vfs_read /tmp/pahole.out | wc -l 0 But both kfree and vfs_read are not found. $ perf probe -k ./vmlinux -V kfree Available variables at kfree @<kfree+0> (unknown_type) object $ perf probe -k ./vmlinux -V vfs_read Available variables at vfs_read @<vfs_read+0> char* buf loff_t* pos size_t count struct file* file However, perf probe can find both in the DWARF info. Thank you, -- Masami Hiramatsu (Google) <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole 2023-07-27 1:51 ` Masami Hiramatsu @ 2023-07-31 15:45 ` Alan Maguire 2023-08-01 1:01 ` Masami Hiramatsu 0 siblings, 1 reply; 14+ messages in thread From: Alan Maguire @ 2023-07-31 15:45 UTC (permalink / raw) To: Masami Hiramatsu (Google) Cc: Jiri Olsa, Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf, Steven Rostedt On 27/07/2023 02:51, Masami Hiramatsu (Google) wrote: > On Thu, 27 Jul 2023 09:38:14 +0900 > Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote: > >>> Yep, BPF generation is more selective about what it emits in 1.25 to >>> avoid cases where a kernel function signature is ambiguous (multiple >>> functions of the same name with different signatures) or where it has >>> unexpected register use. You can observe this via pahole's --verbose >>> option (a lot of outut is emitted): >>> >>> In a built kernel directory (where unstripped vmlinux is present): >>> $ PAHOLE_FLAGS=$(./scripts/pahole_flags) >>> $ PAHOLE=/usr/local/bin/pahole >>> $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out >> >> So this will generate BTF from vmlinux DWARF info. >> >>> If you want to investigate why a function has been left out, look for >>> "skipping" verbose output like this: >>> >>> skipping addition of 'access_error'(access_error) due to multiple >>> inconsistent function prototypes >>> skipping addition of >>> 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) >>> due to unexpected register used for parameter >> >> Ah, that's nice. Let me try. > > $ pahole --version > v1.23 > shouldn't this be v1.25? Is it possible pahole is picking up the wrong libdwarves? what does "ldd pahole" say? > $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)! > > OK, so something failed. > > $ grep skipping /tmp/pahole.out | wc -l > 0 > > Nothing to be skipped. > > $ grep -w kfree /tmp/pahole.out | wc -l > 0 > $ grep -w vfs_read /tmp/pahole.out | wc -l > 0 > > But both kfree and vfs_read are not found. > > $ perf probe -k ./vmlinux -V kfree > Available variables at kfree > @<kfree+0> > (unknown_type) object > $ perf probe -k ./vmlinux -V vfs_read > Available variables at vfs_read > @<vfs_read+0> > char* buf > loff_t* pos > size_t count > struct file* file > > However, perf probe can find both in the DWARF info. > > Thank you, > Unfortunately (or fortunately?) I haven't been able to reproduce so far I'm afraid. I used your config and built gcc 13 from source; everything worked as expected, with no warnings or missing functions (aside from the ones skipped due to inconsistent prototypes etc). One other thing I can think of - is it possible libdw[arf]/libelf versions might be having an effect here? I'm using libdwarf.so.1.2, libdw-0.188, libelf-0.188. I can try and match yours. Thanks! Alan ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole 2023-07-31 15:45 ` Alan Maguire @ 2023-08-01 1:01 ` Masami Hiramatsu 2023-08-01 17:36 ` Alan Maguire 0 siblings, 1 reply; 14+ messages in thread From: Masami Hiramatsu @ 2023-08-01 1:01 UTC (permalink / raw) To: Alan Maguire Cc: Jiri Olsa, Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf, Steven Rostedt On Mon, 31 Jul 2023 16:45:24 +0100 Alan Maguire <alan.maguire@oracle.com> wrote: > On 27/07/2023 02:51, Masami Hiramatsu (Google) wrote: > > On Thu, 27 Jul 2023 09:38:14 +0900 > > Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote: > > > >>> Yep, BPF generation is more selective about what it emits in 1.25 to > >>> avoid cases where a kernel function signature is ambiguous (multiple > >>> functions of the same name with different signatures) or where it has > >>> unexpected register use. You can observe this via pahole's --verbose > >>> option (a lot of outut is emitted): > >>> > >>> In a built kernel directory (where unstripped vmlinux is present): > >>> $ PAHOLE_FLAGS=$(./scripts/pahole_flags) > >>> $ PAHOLE=/usr/local/bin/pahole > >>> $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out > >> > >> So this will generate BTF from vmlinux DWARF info. > >> > >>> If you want to investigate why a function has been left out, look for > >>> "skipping" verbose output like this: > >>> > >>> skipping addition of 'access_error'(access_error) due to multiple > >>> inconsistent function prototypes > >>> skipping addition of > >>> 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) > >>> due to unexpected register used for parameter > >> > >> Ah, that's nice. Let me try. > > > > $ pahole --version > > v1.23 > > > > shouldn't this be v1.25? Is it possible pahole is picking up the wrong > libdwarves? what does "ldd pahole" say? Here it is. $ ldd pahole linux-vdso.so.1 (0x00007ffd6b1e2000) libdwarves_reorganize.so.1 => /opt/local/pahole/libdwarves_reorganize.so.1 (0x00007f1ddaad9000) libdwarves.so.1 => /opt/local/pahole/libdwarves.so.1 (0x00007f1ddaa72000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dda82a000) libdw.so.1 => /usr/local/lib/x86_64-linux-gnu/libdw.so.1 (0x00007f1dda78c000) libelf.so.1 => /usr/local/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1dda771000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1dda753000) /lib64/ld-linux-x86-64.so.2 (0x00007f1ddaaef000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dda74e000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1dda723000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dda71e000) Maybe libdwarvers is not related, I could build pahole v1.23 and v1.22 without sync the submodule. I also confirmed the same issue on Ubuntu 22.04's combination, which update pahole from v1.22 to v1.25 recently. (but gcc is still v11.3) gcc-11.3 | clang-16.0 ---------------------------+---------------- v1.22 OK OK (ubuntu) v1.22 OK OK (local) v1.23 NG OK (local) v1.24 NG - (local) v1.25 NG - (ubuntu) v1.25 NG OK (local) So, as far as I checked, there is something wrong between v1.22 and v1.23 which is also related to gcc-11.3. > > > $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)! > > > > OK, so something failed. > > > > $ grep skipping /tmp/pahole.out | wc -l > > 0 > > > > Nothing to be skipped. > > > > $ grep -w kfree /tmp/pahole.out | wc -l > > 0 > > $ grep -w vfs_read /tmp/pahole.out | wc -l > > 0 > > > > But both kfree and vfs_read are not found. > > > > $ perf probe -k ./vmlinux -V kfree > > Available variables at kfree > > @<kfree+0> > > (unknown_type) object > > $ perf probe -k ./vmlinux -V vfs_read > > Available variables at vfs_read > > @<vfs_read+0> > > char* buf > > loff_t* pos > > size_t count > > struct file* file > > > > However, perf probe can find both in the DWARF info. > > > > Thank you, > > > > Unfortunately (or fortunately?) I haven't been able to reproduce so far > I'm afraid. I used your config and built gcc 13 from source; everything > worked as expected, with no warnings or missing functions (aside from > the ones skipped due to inconsistent prototypes etc). Yeah, so I think gcc-11.3 is suspicious too (and it seems fixed in gcc-13). > One other thing I can think of - is it possible libdw[arf]/libelf > versions might be having an effect here? I'm using libdwarf.so.1.2, > libdw-0.188, libelf-0.188. I can try and match yours. Thanks! Both libdw/libelf are 0.181. I didn't install libdwarf. Hmm, I should update the libdw (elfutils) too. Thank you, > > Alan -- Masami Hiramatsu (Google) <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole 2023-08-01 1:01 ` Masami Hiramatsu @ 2023-08-01 17:36 ` Alan Maguire [not found] ` <CA+JHD93Liq95RvfChifmnE7E9mKR42_W7rtpqgY9KAgYyGTZwQ@mail.gmail.com> 2023-08-01 21:45 ` Arnaldo Carvalho de Melo 0 siblings, 2 replies; 14+ messages in thread From: Alan Maguire @ 2023-08-01 17:36 UTC (permalink / raw) To: Masami Hiramatsu (Google) Cc: Jiri Olsa, Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf, Steven Rostedt On 01/08/2023 02:01, Masami Hiramatsu (Google) wrote: > On Mon, 31 Jul 2023 16:45:24 +0100 > Alan Maguire <alan.maguire@oracle.com> wrote: > >> On 27/07/2023 02:51, Masami Hiramatsu (Google) wrote: >>> On Thu, 27 Jul 2023 09:38:14 +0900 >>> Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote: >>> >>>>> Yep, BPF generation is more selective about what it emits in 1.25 to >>>>> avoid cases where a kernel function signature is ambiguous (multiple >>>>> functions of the same name with different signatures) or where it has >>>>> unexpected register use. You can observe this via pahole's --verbose >>>>> option (a lot of outut is emitted): >>>>> >>>>> In a built kernel directory (where unstripped vmlinux is present): >>>>> $ PAHOLE_FLAGS=$(./scripts/pahole_flags) >>>>> $ PAHOLE=/usr/local/bin/pahole >>>>> $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out >>>> >>>> So this will generate BTF from vmlinux DWARF info. >>>> >>>>> If you want to investigate why a function has been left out, look for >>>>> "skipping" verbose output like this: >>>>> >>>>> skipping addition of 'access_error'(access_error) due to multiple >>>>> inconsistent function prototypes >>>>> skipping addition of >>>>> 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) >>>>> due to unexpected register used for parameter >>>> >>>> Ah, that's nice. Let me try. >>> >>> $ pahole --version >>> v1.23 >>> >> >> shouldn't this be v1.25? Is it possible pahole is picking up the wrong >> libdwarves? what does "ldd pahole" say? > > Here it is. > $ ldd pahole > linux-vdso.so.1 (0x00007ffd6b1e2000) > libdwarves_reorganize.so.1 => /opt/local/pahole/libdwarves_reorganize.so.1 (0x00007f1ddaad9000) > libdwarves.so.1 => /opt/local/pahole/libdwarves.so.1 (0x00007f1ddaa72000) > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dda82a000) > libdw.so.1 => /usr/local/lib/x86_64-linux-gnu/libdw.so.1 (0x00007f1dda78c000) > libelf.so.1 => /usr/local/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1dda771000) > libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1dda753000) > /lib64/ld-linux-x86-64.so.2 (0x00007f1ddaaef000) > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dda74e000) > liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1dda723000) > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dda71e000) > > Maybe libdwarvers is not related, I could build pahole v1.23 and v1.22 without > sync the submodule. > > I also confirmed the same issue on Ubuntu 22.04's combination, which > update pahole from v1.22 to v1.25 recently. (but gcc is still v11.3) > > gcc-11.3 | clang-16.0 > ---------------------------+---------------- > v1.22 OK OK > (ubuntu) > v1.22 OK OK > (local) > v1.23 NG OK > (local) > v1.24 NG - > (local) > v1.25 NG - > (ubuntu) > v1.25 NG OK > (local) > > So, as far as I checked, there is something wrong between v1.22 and v1.23 > which is also related to gcc-11.3. > One thing that is notable about gcc 11 is that I believe it's the first gcc release to emit DWARF5 by default. I wonder if it's possible that the kernel emitted DWARF5, but pahole was built with libraries that didn't support it yet? Not sure how that fits with the fact that pahole v1.22 works though. If you have a chance, it might be worth experimenting with your kernel .config to specify DWARF4 to see if that makes a difference. You can check DWARF version associated with CUs with readelf --debug-dump=info vmlinux |grep -A 2 'Compilation Unit' >> >>> $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out >>> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)! >>> >>> OK, so something failed. >>> >>> $ grep skipping /tmp/pahole.out | wc -l >>> 0 >>> >>> Nothing to be skipped. >>> >>> $ grep -w kfree /tmp/pahole.out | wc -l >>> 0 >>> $ grep -w vfs_read /tmp/pahole.out | wc -l >>> 0 >>> >>> But both kfree and vfs_read are not found. >>> >>> $ perf probe -k ./vmlinux -V kfree >>> Available variables at kfree >>> @<kfree+0> >>> (unknown_type) object >>> $ perf probe -k ./vmlinux -V vfs_read >>> Available variables at vfs_read >>> @<vfs_read+0> >>> char* buf >>> loff_t* pos >>> size_t count >>> struct file* file >>> >>> However, perf probe can find both in the DWARF info. >>> >>> Thank you, >>> >> >> Unfortunately (or fortunately?) I haven't been able to reproduce so far >> I'm afraid. I used your config and built gcc 13 from source; everything >> worked as expected, with no warnings or missing functions (aside from >> the ones skipped due to inconsistent prototypes etc). > > Yeah, so I think gcc-11.3 is suspicious too (and it seems fixed in gcc-13). > > >> One other thing I can think of - is it possible libdw[arf]/libelf >> versions might be having an effect here? I'm using libdwarf.so.1.2, >> libdw-0.188, libelf-0.188. I can try and match yours. Thanks! > > Both libdw/libelf are 0.181. I didn't install libdwarf. > Hmm, I should update the libdw (elfutils) too. That might help. Thanks! Alan > Thank you, > >> >> Alan > > ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <CA+JHD93Liq95RvfChifmnE7E9mKR42_W7rtpqgY9KAgYyGTZwQ@mail.gmail.com>]
* Fwd: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole [not found] ` <CA+JHD93Liq95RvfChifmnE7E9mKR42_W7rtpqgY9KAgYyGTZwQ@mail.gmail.com> @ 2023-08-01 18:28 ` Arnaldo Carvalho de Melo 2023-08-02 8:02 ` Masami Hiramatsu 0 siblings, 1 reply; 14+ messages in thread From: Arnaldo Carvalho de Melo @ 2023-08-01 18:28 UTC (permalink / raw) To: Masami Hiramatsu (Google), Alan Maguire Cc: cc: Jiri Olsa, Arnaldo Carvalho de Melo, dwarves, Steven Rostedt, bpf, Yonghong Song Sorry, replied only to Alan :-\ - Arnaldo ---------- Forwarded message --------- From: Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> Date: Tue, Aug 1, 2023 at 3:26 PM Subject: Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole To: Alan Maguire <alan.maguire@oracle.com> On Tue, Aug 1, 2023 at 2:37 PM Alan Maguire <alan.maguire@oracle.com> wrote: > On 01/08/2023 02:01, Masami Hiramatsu (Google) wrote: > > On Mon, 31 Jul 2023 16:45:24 +0100 > > Alan Maguire <alan.maguire@oracle.com> wrote: > >> Unfortunately (or fortunately?) I haven't been able to reproduce so far > >> I'm afraid. I used your config and built gcc 13 from source; everything > >> worked as expected, with no warnings or missing functions (aside from > >> the ones skipped due to inconsistent prototypes etc). > > > > Yeah, so I think gcc-11.3 is suspicious too (and it seems fixed in gcc-13). See below, but this one is interesting, gcc-13 works with CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y? > >> One other thing I can think of - is it possible libdw[arf]/libelf > >> versions might be having an effect here? I'm using libdwarf.so.1.2, > >> libdw-0.188, libelf-0.188. I can try and match yours. Thanks! > > > > Both libdw/libelf are 0.181. I didn't install libdwarf. > > Hmm, I should update the libdw (elfutils) too. > > That might help. Thanks! Probably he needs to tweak these CONFIG_ entries: ⬢[acme@toolbox perf-tools]$ grep DWARF ../build/v6.2-rc5+/.config CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y # CONFIG_DEBUG_INFO_DWARF4 is not set # CONFIG_DEBUG_INFO_DWARF5 is not set ⬢[acme@toolbox perf-tools]$ And make it use CONFIG_DEBUG_INFO_DWARF4=y, CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=n For DWARF5 I need to forward port what I have in: https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=WIP-imported-unit - Arnaldo ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Fwd: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole 2023-08-01 18:28 ` Fwd: " Arnaldo Carvalho de Melo @ 2023-08-02 8:02 ` Masami Hiramatsu 0 siblings, 0 replies; 14+ messages in thread From: Masami Hiramatsu @ 2023-08-02 8:02 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Alan Maguire, cc: Jiri Olsa, Arnaldo Carvalho de Melo, dwarves, Steven Rostedt, bpf, Yonghong Song On Tue, 1 Aug 2023 15:28:38 -0300 Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote: > Sorry, replied only to Alan :-\ > > - Arnaldo > > ---------- Forwarded message --------- > From: Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> > Date: Tue, Aug 1, 2023 at 3:26 PM > Subject: Re: [RESEND] BTF is not generated for gcc-built kernel with > the latest pahole > To: Alan Maguire <alan.maguire@oracle.com> > > > On Tue, Aug 1, 2023 at 2:37 PM Alan Maguire <alan.maguire@oracle.com> wrote: > > On 01/08/2023 02:01, Masami Hiramatsu (Google) wrote: > > > On Mon, 31 Jul 2023 16:45:24 +0100 > > > Alan Maguire <alan.maguire@oracle.com> wrote: > > > >> Unfortunately (or fortunately?) I haven't been able to reproduce so far > > >> I'm afraid. I used your config and built gcc 13 from source; everything > > >> worked as expected, with no warnings or missing functions (aside from > > >> the ones skipped due to inconsistent prototypes etc). > > > > > > Yeah, so I think gcc-11.3 is suspicious too (and it seems fixed in gcc-13). > > See below, but this one is interesting, gcc-13 works with > CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y? I'm using gcc-11.3, and it seems to work. > > > >> One other thing I can think of - is it possible libdw[arf]/libelf > > >> versions might be having an effect here? I'm using libdwarf.so.1.2, > > >> libdw-0.188, libelf-0.188. I can try and match yours. Thanks! > > > > > > Both libdw/libelf are 0.181. I didn't install libdwarf. > > > Hmm, I should update the libdw (elfutils) too. > > > > That might help. Thanks! > > Probably he needs to tweak these CONFIG_ entries: > > ⬢[acme@toolbox perf-tools]$ grep DWARF ../build/v6.2-rc5+/.config > CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y > # CONFIG_DEBUG_INFO_DWARF4 is not set > # CONFIG_DEBUG_INFO_DWARF5 is not set > ⬢[acme@toolbox perf-tools]$ > > And make it use CONFIG_DEBUG_INFO_DWARF4=y, > CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=n OK, interesting. Let me check again with DWARF4. Thank you! > > For DWARF5 I need to forward port what I have in: > > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=WIP-imported-unit > > - Arnaldo -- Masami Hiramatsu (Google) <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole 2023-08-01 17:36 ` Alan Maguire [not found] ` <CA+JHD93Liq95RvfChifmnE7E9mKR42_W7rtpqgY9KAgYyGTZwQ@mail.gmail.com> @ 2023-08-01 21:45 ` Arnaldo Carvalho de Melo 2023-08-02 14:29 ` Masami Hiramatsu 1 sibling, 1 reply; 14+ messages in thread From: Arnaldo Carvalho de Melo @ 2023-08-01 21:45 UTC (permalink / raw) To: Alan Maguire Cc: Masami Hiramatsu (Google), Jiri Olsa, Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf, Steven Rostedt Em Tue, Aug 01, 2023 at 06:36:49PM +0100, Alan Maguire escreveu: > On 01/08/2023 02:01, Masami Hiramatsu (Google) wrote: > > On Mon, 31 Jul 2023 16:45:24 +0100 > > Alan Maguire <alan.maguire@oracle.com> wrote: > > > >> On 27/07/2023 02:51, Masami Hiramatsu (Google) wrote: > >>> On Thu, 27 Jul 2023 09:38:14 +0900 > >>> Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote: > >>> > >>>>> Yep, BPF generation is more selective about what it emits in 1.25 to > >>>>> avoid cases where a kernel function signature is ambiguous (multiple > >>>>> functions of the same name with different signatures) or where it has > >>>>> unexpected register use. You can observe this via pahole's --verbose > >>>>> option (a lot of outut is emitted): > >>>>> > >>>>> In a built kernel directory (where unstripped vmlinux is present): > >>>>> $ PAHOLE_FLAGS=$(./scripts/pahole_flags) > >>>>> $ PAHOLE=/usr/local/bin/pahole > >>>>> $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out > >>>> > >>>> So this will generate BTF from vmlinux DWARF info. > >>>> > >>>>> If you want to investigate why a function has been left out, look for > >>>>> "skipping" verbose output like this: > >>>>> > >>>>> skipping addition of 'access_error'(access_error) due to multiple > >>>>> inconsistent function prototypes > >>>>> skipping addition of > >>>>> 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) > >>>>> due to unexpected register used for parameter > >>>> > >>>> Ah, that's nice. Let me try. > >>> > >>> $ pahole --version > >>> v1.23 > >>> > >> > >> shouldn't this be v1.25? Is it possible pahole is picking up the wrong > >> libdwarves? what does "ldd pahole" say? > > > > Here it is. > > $ ldd pahole > > linux-vdso.so.1 (0x00007ffd6b1e2000) > > libdwarves_reorganize.so.1 => /opt/local/pahole/libdwarves_reorganize.so.1 (0x00007f1ddaad9000) > > libdwarves.so.1 => /opt/local/pahole/libdwarves.so.1 (0x00007f1ddaa72000) > > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dda82a000) > > libdw.so.1 => /usr/local/lib/x86_64-linux-gnu/libdw.so.1 (0x00007f1dda78c000) > > libelf.so.1 => /usr/local/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1dda771000) > > libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1dda753000) > > /lib64/ld-linux-x86-64.so.2 (0x00007f1ddaaef000) > > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dda74e000) > > liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1dda723000) > > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dda71e000) > > > > Maybe libdwarvers is not related, I could build pahole v1.23 and v1.22 without > > sync the submodule. > > > > I also confirmed the same issue on Ubuntu 22.04's combination, which > > update pahole from v1.22 to v1.25 recently. (but gcc is still v11.3) > > > > gcc-11.3 | clang-16.0 > > ---------------------------+---------------- > > v1.22 OK OK > > (ubuntu) > > v1.22 OK OK > > (local) > > v1.23 NG OK > > (local) > > v1.24 NG - > > (local) > > v1.25 NG - > > (ubuntu) > > v1.25 NG OK > > (local) > > > > So, as far as I checked, there is something wrong between v1.22 and v1.23 > > which is also related to gcc-11.3. > > > > One thing that is notable about gcc 11 is that I believe it's the first > gcc release to emit DWARF5 by default. I wonder if it's possible that > the kernel emitted DWARF5, but pahole was built with libraries that > didn't support it yet? Not sure how that fits with the fact that To check that please tweak: ⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y # CONFIG_DEBUG_INFO_DWARF4 is not set # CONFIG_DEBUG_INFO_DWARF5 is not set ⬢[acme@toolbox perf-tools-next]$ i.e. disable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT and enable CONFIG_DEBUG_INFO_DWARF4. - Arnaldo > pahole v1.22 works though. If you have a chance, it might be worth > experimenting with your kernel .config to specify DWARF4 to see if > that makes a difference. > > You can check DWARF version associated with CUs with > > readelf --debug-dump=info vmlinux |grep -A 2 'Compilation Unit' ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole 2023-08-01 21:45 ` Arnaldo Carvalho de Melo @ 2023-08-02 14:29 ` Masami Hiramatsu 0 siblings, 0 replies; 14+ messages in thread From: Masami Hiramatsu @ 2023-08-02 14:29 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Alan Maguire, Masami Hiramatsu (Google), Jiri Olsa, Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf, Steven Rostedt On Tue, 1 Aug 2023 18:45:49 -0300 Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > Em Tue, Aug 01, 2023 at 06:36:49PM +0100, Alan Maguire escreveu: > > On 01/08/2023 02:01, Masami Hiramatsu (Google) wrote: > > > On Mon, 31 Jul 2023 16:45:24 +0100 > > > Alan Maguire <alan.maguire@oracle.com> wrote: > > > > > >> On 27/07/2023 02:51, Masami Hiramatsu (Google) wrote: > > >>> On Thu, 27 Jul 2023 09:38:14 +0900 > > >>> Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote: > > >>> > > >>>>> Yep, BPF generation is more selective about what it emits in 1.25 to > > >>>>> avoid cases where a kernel function signature is ambiguous (multiple > > >>>>> functions of the same name with different signatures) or where it has > > >>>>> unexpected register use. You can observe this via pahole's --verbose > > >>>>> option (a lot of outut is emitted): > > >>>>> > > >>>>> In a built kernel directory (where unstripped vmlinux is present): > > >>>>> $ PAHOLE_FLAGS=$(./scripts/pahole_flags) > > >>>>> $ PAHOLE=/usr/local/bin/pahole > > >>>>> $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out > > >>>> > > >>>> So this will generate BTF from vmlinux DWARF info. > > >>>> > > >>>>> If you want to investigate why a function has been left out, look for > > >>>>> "skipping" verbose output like this: > > >>>>> > > >>>>> skipping addition of 'access_error'(access_error) due to multiple > > >>>>> inconsistent function prototypes > > >>>>> skipping addition of > > >>>>> 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) > > >>>>> due to unexpected register used for parameter > > >>>> > > >>>> Ah, that's nice. Let me try. > > >>> > > >>> $ pahole --version > > >>> v1.23 > > >>> > > >> > > >> shouldn't this be v1.25? Is it possible pahole is picking up the wrong > > >> libdwarves? what does "ldd pahole" say? > > > > > > Here it is. > > > $ ldd pahole > > > linux-vdso.so.1 (0x00007ffd6b1e2000) > > > libdwarves_reorganize.so.1 => /opt/local/pahole/libdwarves_reorganize.so.1 (0x00007f1ddaad9000) > > > libdwarves.so.1 => /opt/local/pahole/libdwarves.so.1 (0x00007f1ddaa72000) > > > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dda82a000) > > > libdw.so.1 => /usr/local/lib/x86_64-linux-gnu/libdw.so.1 (0x00007f1dda78c000) > > > libelf.so.1 => /usr/local/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1dda771000) > > > libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1dda753000) > > > /lib64/ld-linux-x86-64.so.2 (0x00007f1ddaaef000) > > > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dda74e000) > > > liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1dda723000) > > > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dda71e000) > > > > > > Maybe libdwarvers is not related, I could build pahole v1.23 and v1.22 without > > > sync the submodule. > > > > > > I also confirmed the same issue on Ubuntu 22.04's combination, which > > > update pahole from v1.22 to v1.25 recently. (but gcc is still v11.3) > > > > > > gcc-11.3 | clang-16.0 > > > ---------------------------+---------------- > > > v1.22 OK OK > > > (ubuntu) > > > v1.22 OK OK > > > (local) > > > v1.23 NG OK > > > (local) > > > v1.24 NG - > > > (local) > > > v1.25 NG - > > > (ubuntu) > > > v1.25 NG OK > > > (local) > > > > > > So, as far as I checked, there is something wrong between v1.22 and v1.23 > > > which is also related to gcc-11.3. > > > > > > > One thing that is notable about gcc 11 is that I believe it's the first > > gcc release to emit DWARF5 by default. I wonder if it's possible that > > the kernel emitted DWARF5, but pahole was built with libraries that > > didn't support it yet? Not sure how that fits with the fact that > > To check that please tweak: > > ⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config > CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y > # CONFIG_DEBUG_INFO_DWARF4 is not set > # CONFIG_DEBUG_INFO_DWARF5 is not set > ⬢[acme@toolbox perf-tools-next]$ > > i.e. disable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT and enable > CONFIG_DEBUG_INFO_DWARF4. Hm, with CONFIG_DEBUG_INFO_DWARF4, no warning were shown. LD .tmp_vmlinux.btf BTF .btf.vmlinux.bin.o LD .tmp_vmlinux.kallsyms1 And / # strings /sys/kernel/btf/vmlinux | wc -l 89921 / # strings /sys/kernel/btf/vmlinux | grep -w kfree kfree It seems the BTF is correctly generated. (with DWARF5, the number of symbols are about 30000.) Maybe we should check the pahole version and choose the DWARF4 if BTF is selected. Thank you! > > - Arnaldo > > > pahole v1.22 works though. If you have a chance, it might be worth > > experimenting with your kernel .config to specify DWARF4 to see if > > that makes a difference. > > > > You can check DWARF version associated with CUs with > > > > readelf --debug-dump=info vmlinux |grep -A 2 'Compilation Unit' -- Masami Hiramatsu (Google) <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole 2023-07-27 0:38 ` Masami Hiramatsu 2023-07-27 1:51 ` Masami Hiramatsu @ 2023-07-27 13:10 ` Arnaldo Carvalho de Melo 1 sibling, 0 replies; 14+ messages in thread From: Arnaldo Carvalho de Melo @ 2023-07-27 13:10 UTC (permalink / raw) To: Masami Hiramatsu Cc: Alan Maguire, Jiri Olsa, Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf, Steven Rostedt Em Thu, Jul 27, 2023 at 09:38:14AM +0900, Masami Hiramatsu escreveu: > On Wed, 26 Jul 2023 15:46:03 +0100 > Alan Maguire <alan.maguire@oracle.com> wrote: > > > On 26/07/2023 12:07, Masami Hiramatsu (Google) wrote: > > > Hi Jiri, > > > > > > On Wed, 26 Jul 2023 12:04:08 +0200 > > > Jiri Olsa <olsajiri@gmail.com> wrote: > > > > > >> On Wed, Jul 26, 2023 at 10:25:34AM +0900, Masami Hiramatsu wrote: > > >>> Hello, > > >>> (I resend this because kconfig was too big) > > >>> > > >>> I found that BTF is not generated for gcc-built kernel with that latest > > >>> pahole (v1.25). > > >> > > >> hi, > > >> I can't reproduce on my setup with your .config > > >> > > >> does 'bpftool btf dump file ./vmlinux' show any error? > > >> > > >> is there any error in the kernel build output? > > > > > > Yes, here it is. I saw these 2 lines. > > > > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)! > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)! > > > > This is strange, looks like some CUs were encoded incorrectly or we are > > parsing incorrectly. The error originates in die__process() and happens > > if the dwarf_tag() associated with the DIE isn't an expected unit; it's > > not even a valid tag value (0) it looks like. I've not built with gcc 13 > > yet so it's possible that's the reason you're seeing this, I'll try to > > reproduce it.. > > And this warning message is not good for debugging. It would better dump > the message with DIE index so that we can analyze the DWARF with other > tools. Good idea, I'll add that. - Arnaldo ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2023-08-02 14:29 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-07-26 1:25 [RESEND] BTF is not generated for gcc-built kernel with the latest pahole Masami Hiramatsu
2023-07-26 10:04 ` Jiri Olsa
2023-07-26 11:07 ` Masami Hiramatsu
2023-07-26 14:46 ` Alan Maguire
2023-07-27 0:38 ` Masami Hiramatsu
2023-07-27 1:51 ` Masami Hiramatsu
2023-07-31 15:45 ` Alan Maguire
2023-08-01 1:01 ` Masami Hiramatsu
2023-08-01 17:36 ` Alan Maguire
[not found] ` <CA+JHD93Liq95RvfChifmnE7E9mKR42_W7rtpqgY9KAgYyGTZwQ@mail.gmail.com>
2023-08-01 18:28 ` Fwd: " Arnaldo Carvalho de Melo
2023-08-02 8:02 ` Masami Hiramatsu
2023-08-01 21:45 ` Arnaldo Carvalho de Melo
2023-08-02 14:29 ` Masami Hiramatsu
2023-07-27 13:10 ` Arnaldo Carvalho de Melo
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.