* [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
@ 2023-07-26 1:25 Masami Hiramatsu
2023-07-26 10:04 ` Jiri Olsa
0 siblings, 1 reply; 14+ messages in thread
From: Masami Hiramatsu @ 2023-07-26 1:25 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Yonghong Song; +Cc: dwarves, bpf, Steven Rostedt
[-- Attachment #1: Type: text/plain, Size: 1618 bytes --]
Hello,
(I resend this because kconfig was too big)
I found that BTF is not generated for gcc-built kernel with that latest
pahole (v1.25).
When I'm using the distro origin pahole (v1.22) it works. (I also checked
v1.23 and v1.24, both partially generated BTF)
e.g.
# echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events
sh: write error: Invalid argument
# cat /sys/kernel/tracing/error_log
[ 21.595724] trace_fprobe: error: BTF is not available or not supported
Command: f kfree $arg*
^
[ 21.596032] trace_fprobe: error: Invalid $-valiable specified
Command: f kfree $arg*
^
/ # strings /sys/kernel/btf/vmlinux | grep kfree
kfree_on_online
maybe_kfree_parameter
trace_event_raw_rcu_invoke_kfree_bulk_callback
trace_event_data_offsets_rcu_invoke_kfree_bulk_callback
btf_trace_rcu_invoke_kfree_bulk_callback
early_boot_kfree_rcu
__bpf_trace_rcu_invoke_kfree_bulk_callback
perf_trace_rcu_invoke_kfree_bulk_callback
trace_event_raw_event_rcu_invoke_kfree_bulk_callback
trace_raw_output_rcu_invoke_kfree_bulk_callback
__probestub_rcu_invoke_kfree_bulk_callback
__traceiter_rcu_invoke_kfree_bulk_callback
kfree_rcu_cpu_work
kfree_rcu_cpu
kfree_rcu_batch_init
kfree_rcu_scheduler_running
kfree_rcu_shrink_scan
kfree_rcu_shrink_count
kfree_rcu_monitor
kfree_rcu_work
Here is the gcc version which I'm using.
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1)
I also attached the kernel config file.
What is the recommended combination of the tools?
Should I use Clang to build the kernel for BTF?
Thank you,
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
[-- Attachment #2: config.tar.xz --]
[-- Type: application/x-xz, Size: 22156 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
2023-07-26 1:25 [RESEND] BTF is not generated for gcc-built kernel with the latest pahole Masami Hiramatsu
@ 2023-07-26 10:04 ` Jiri Olsa
2023-07-26 11:07 ` Masami Hiramatsu
0 siblings, 1 reply; 14+ messages in thread
From: Jiri Olsa @ 2023-07-26 10:04 UTC (permalink / raw)
To: Masami Hiramatsu
Cc: Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf,
Steven Rostedt
On Wed, Jul 26, 2023 at 10:25:34AM +0900, Masami Hiramatsu wrote:
> Hello,
> (I resend this because kconfig was too big)
>
> I found that BTF is not generated for gcc-built kernel with that latest
> pahole (v1.25).
hi,
I can't reproduce on my setup with your .config
does 'bpftool btf dump file ./vmlinux' show any error?
is there any error in the kernel build output?
> When I'm using the distro origin pahole (v1.22) it works. (I also checked
> v1.23 and v1.24, both partially generated BTF)
>
> e.g.
>
> # echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events
> sh: write error: Invalid argument
>
> # cat /sys/kernel/tracing/error_log
> [ 21.595724] trace_fprobe: error: BTF is not available or not supported
> Command: f kfree $arg*
> ^
> [ 21.596032] trace_fprobe: error: Invalid $-valiable specified
> Command: f kfree $arg*
> ^
>
> / # strings /sys/kernel/btf/vmlinux | grep kfree
hm, if you have this file present, you have BTF in
> kfree_on_online
> maybe_kfree_parameter
> trace_event_raw_rcu_invoke_kfree_bulk_callback
> trace_event_data_offsets_rcu_invoke_kfree_bulk_callback
> btf_trace_rcu_invoke_kfree_bulk_callback
> early_boot_kfree_rcu
> __bpf_trace_rcu_invoke_kfree_bulk_callback
> perf_trace_rcu_invoke_kfree_bulk_callback
> trace_event_raw_event_rcu_invoke_kfree_bulk_callback
> trace_raw_output_rcu_invoke_kfree_bulk_callback
> __probestub_rcu_invoke_kfree_bulk_callback
> __traceiter_rcu_invoke_kfree_bulk_callback
> kfree_rcu_cpu_work
> kfree_rcu_cpu
> kfree_rcu_batch_init
> kfree_rcu_scheduler_running
> kfree_rcu_shrink_scan
> kfree_rcu_shrink_count
> kfree_rcu_monitor
> kfree_rcu_work
>
>
> Here is the gcc version which I'm using.
>
> gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1)
I tried with gcc (GCC) 13.1.1 20230614 (Red Hat 13.1.1-4)
and latest pahole (master branch)
>
> I also attached the kernel config file.
>
> What is the recommended combination of the tools?
> Should I use Clang to build the kernel for BTF?
should work fine with both gcc and clang
jirka
>
> Thank you,
>
> --
> Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
2023-07-26 10:04 ` Jiri Olsa
@ 2023-07-26 11:07 ` Masami Hiramatsu
2023-07-26 14:46 ` Alan Maguire
0 siblings, 1 reply; 14+ messages in thread
From: Masami Hiramatsu @ 2023-07-26 11:07 UTC (permalink / raw)
To: Jiri Olsa
Cc: Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf,
Steven Rostedt
Hi Jiri,
On Wed, 26 Jul 2023 12:04:08 +0200
Jiri Olsa <olsajiri@gmail.com> wrote:
> On Wed, Jul 26, 2023 at 10:25:34AM +0900, Masami Hiramatsu wrote:
> > Hello,
> > (I resend this because kconfig was too big)
> >
> > I found that BTF is not generated for gcc-built kernel with that latest
> > pahole (v1.25).
>
> hi,
> I can't reproduce on my setup with your .config
>
> does 'bpftool btf dump file ./vmlinux' show any error?
>
> is there any error in the kernel build output?
Yes, here it is. I saw these 2 lines.
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
>
> > When I'm using the distro origin pahole (v1.22) it works. (I also checked
> > v1.23 and v1.24, both partially generated BTF)
> >
> > e.g.
> >
> > # echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events
> > sh: write error: Invalid argument
> >
> > # cat /sys/kernel/tracing/error_log
> > [ 21.595724] trace_fprobe: error: BTF is not available or not supported
> > Command: f kfree $arg*
> > ^
> > [ 21.596032] trace_fprobe: error: Invalid $-valiable specified
> > Command: f kfree $arg*
> > ^
> >
> > / # strings /sys/kernel/btf/vmlinux | grep kfree
>
> hm, if you have this file present, you have BTF in
Yes, it seems that the BTF itself is generated, but many entries seems
dropped compared with pahole v1.22. So, if a given symbol has BTF, (e.g.
kfree_rcu_batch_init) it works.
>
> > kfree_on_online
> > maybe_kfree_parameter
> > trace_event_raw_rcu_invoke_kfree_bulk_callback
> > trace_event_data_offsets_rcu_invoke_kfree_bulk_callback
> > btf_trace_rcu_invoke_kfree_bulk_callback
> > early_boot_kfree_rcu
> > __bpf_trace_rcu_invoke_kfree_bulk_callback
> > perf_trace_rcu_invoke_kfree_bulk_callback
> > trace_event_raw_event_rcu_invoke_kfree_bulk_callback
> > trace_raw_output_rcu_invoke_kfree_bulk_callback
> > __probestub_rcu_invoke_kfree_bulk_callback
> > __traceiter_rcu_invoke_kfree_bulk_callback
> > kfree_rcu_cpu_work
> > kfree_rcu_cpu
> > kfree_rcu_batch_init
> > kfree_rcu_scheduler_running
> > kfree_rcu_shrink_scan
> > kfree_rcu_shrink_count
> > kfree_rcu_monitor
> > kfree_rcu_work
> >
> >
> > Here is the gcc version which I'm using.
> >
> > gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1)
>
> I tried with gcc (GCC) 13.1.1 20230614 (Red Hat 13.1.1-4)
> and latest pahole (master branch)
Curiously, with Clang 16.0.0, it works (but many different errors are shown,
see below *).
So the combination of gcc/clang and pahole version can affect it.
>
> >
> > I also attached the kernel config file.
> >
> > What is the recommended combination of the tools?
> > Should I use Clang to build the kernel for BTF?
>
> should work fine with both gcc and clang
And I guess it depends on compiler version, doesn't it?
Thank you,
(*)
BTF .btf.vmlinux.bin.o
die__process_unit: DW_TAG_label (0xa) @ <0x4b138> not handled!
die__process_unit: tag not supported 0xa (label)!
die__process_unit: DW_TAG_label (0xa) @ <0x4b241> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b263> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b290> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b2c0> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b2eb> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b317> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b33a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b35a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b37d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b39e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b3c4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b3e7> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b40f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b435> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b457> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b477> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b4a5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b4d5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b505> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b530> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b560> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b585> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b5b2> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b5d9> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b605> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b62e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b652> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b670> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b694> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b6b3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b6d9> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b705> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b72b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b753> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b77f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b7b3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b7e4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b811> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b83c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b869> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b88c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b8bd> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b8e7> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b90b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b930> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b960> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b997> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b9ce> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4ba00> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4ba24> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4ba4d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4ba89> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4babc> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bae4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bb0b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bb2e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bb4e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bb6d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bb8a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bba8> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bbc5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bbe1> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc01> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc1c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc38> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc58> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc79> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc95> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bcb2> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7efd6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7effe> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f11c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f143> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f178> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f1a4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f1ca> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f1fb> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f22f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f25a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f28d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f2b7> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eea81> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eea9d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eeac3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eeaf3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eeb0f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eeb30> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eeb59> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x340734> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6b4a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6b67> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6b8a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6ba5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6bc4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6bea> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6c07> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6c2a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6c4e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6c79> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6c9b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6cc3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6ceb> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6d15> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x30626ac> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x30626cd> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x3062814> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x3062833> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x3062b8b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f623f6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62416> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62437> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62458> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6280b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62b09> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62b37> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62c45> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62c60> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62d69> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62e8d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6305a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63ec2> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63edf> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63efc> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63f19> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63f36> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63f5b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63f80> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63fa5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63fca> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6fcae> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6fcc7> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6fed7> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6fef0> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6fdcb> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70352> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7036f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70394> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f703b1> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f703d6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f703f3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70418> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70435> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7045a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7057b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f705a6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f705cf> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f705f8> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70621> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7064a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70673> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7069c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f706c5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f706ee> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70716> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7073e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70767> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70790> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f707b9> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f707e2> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7080b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70834> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70864> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70892> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f708c0> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f708ee> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7091c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7094a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70978> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f709a6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f709d4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70a01> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70a2e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70a5c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70a8a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70ab8> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70ae6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70b14> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70b42> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70b72> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70ba0> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70bce> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70bfc> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70c2a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70c58> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70c86> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70cb4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70ce2> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70d0f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70d3c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70d6a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70d98> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70dc6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70df4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70e22> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70e50> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70e71> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70e94> not handled!
LD .tmp_vmlinux.kallsyms1
>
> jirka
>
> >
> > Thank you,
> >
> > --
> > Masami Hiramatsu (Google) <mhiramat@kernel.org>
>
>
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
2023-07-26 11:07 ` Masami Hiramatsu
@ 2023-07-26 14:46 ` Alan Maguire
2023-07-27 0:38 ` Masami Hiramatsu
0 siblings, 1 reply; 14+ messages in thread
From: Alan Maguire @ 2023-07-26 14:46 UTC (permalink / raw)
To: Masami Hiramatsu (Google), Jiri Olsa
Cc: Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf,
Steven Rostedt
On 26/07/2023 12:07, Masami Hiramatsu (Google) wrote:
> Hi Jiri,
>
> On Wed, 26 Jul 2023 12:04:08 +0200
> Jiri Olsa <olsajiri@gmail.com> wrote:
>
>> On Wed, Jul 26, 2023 at 10:25:34AM +0900, Masami Hiramatsu wrote:
>>> Hello,
>>> (I resend this because kconfig was too big)
>>>
>>> I found that BTF is not generated for gcc-built kernel with that latest
>>> pahole (v1.25).
>>
>> hi,
>> I can't reproduce on my setup with your .config
>>
>> does 'bpftool btf dump file ./vmlinux' show any error?
>>
>> is there any error in the kernel build output?
>
> Yes, here it is. I saw these 2 lines.
>
> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
This is strange, looks like some CUs were encoded incorrectly or we are
parsing incorrectly. The error originates in die__process() and happens
if the dwarf_tag() associated with the DIE isn't an expected unit; it's
not even a valid tag value (0) it looks like. I've not built with gcc 13
yet so it's possible that's the reason you're seeing this, I'll try to
reproduce it..
>
>>
>>> When I'm using the distro origin pahole (v1.22) it works. (I also checked
>>> v1.23 and v1.24, both partially generated BTF)
>>>
>>> e.g.
>>>
>>> # echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events
>>> sh: write error: Invalid argument
>>>
>>> # cat /sys/kernel/tracing/error_log
>>> [ 21.595724] trace_fprobe: error: BTF is not available or not supported
>>> Command: f kfree $arg*
>>> ^
>>> [ 21.596032] trace_fprobe: error: Invalid $-valiable specified
>>> Command: f kfree $arg*
>>> ^
>>>
>>> / # strings /sys/kernel/btf/vmlinux | grep kfree
>>
>> hm, if you have this file present, you have BTF in
>
> Yes, it seems that the BTF itself is generated, but many entries seems
> dropped compared with pahole v1.22. So, if a given symbol has BTF, (e.g.
> kfree_rcu_batch_init) it works.
>
Yep, BPF generation is more selective about what it emits in 1.25 to
avoid cases where a kernel function signature is ambiguous (multiple
functions of the same name with different signatures) or where it has
unexpected register use. You can observe this via pahole's --verbose
option (a lot of outut is emitted):
In a built kernel directory (where unstripped vmlinux is present):
$ PAHOLE_FLAGS=$(./scripts/pahole_flags)
$ PAHOLE=/usr/local/bin/pahole
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
If you want to investigate why a function has been left out, look for
"skipping" verbose output like this:
skipping addition of 'access_error'(access_error) due to multiple
inconsistent function prototypes
skipping addition of
'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0)
due to unexpected register used for parameter
FWIW I see most of the below in my BTF output on bpf-next, though I am
missing a few, possibly due to differing config options since they don't
appear in kallsyms either. I don't see the DWARF tag label not handled
messages so it's possible that's a symptom of something
I suspect however some form of corruption in the DWARF representation
may be the reason a lot of these are missing. Would be worth trying
to "objdump -g vmlinux >vmlinux.dwarf" (file will be huge tho) I suspect.
>>
>>> kfree_on_online
>>> maybe_kfree_parameter
>>> trace_event_raw_rcu_invoke_kfree_bulk_callback
>>> trace_event_data_offsets_rcu_invoke_kfree_bulk_callback
>>> btf_trace_rcu_invoke_kfree_bulk_callback
>>> early_boot_kfree_rcu
>>> __bpf_trace_rcu_invoke_kfree_bulk_callback
>>> perf_trace_rcu_invoke_kfree_bulk_callback
>>> trace_event_raw_event_rcu_invoke_kfree_bulk_callback
>>> trace_raw_output_rcu_invoke_kfree_bulk_callback
>>> __probestub_rcu_invoke_kfree_bulk_callback
>>> __traceiter_rcu_invoke_kfree_bulk_callback
>>> kfree_rcu_cpu_work
>>> kfree_rcu_cpu
>>> kfree_rcu_batch_init
>>> kfree_rcu_scheduler_running
>>> kfree_rcu_shrink_scan
>>> kfree_rcu_shrink_count
>>> kfree_rcu_monitor
>>> kfree_rcu_work
>>>
>>>
>>> Here is the gcc version which I'm using.
>>>
>>> gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1)
>>
>> I tried with gcc (GCC) 13.1.1 20230614 (Red Hat 13.1.1-4)
>> and latest pahole (master branch)
>
> Curiously, with Clang 16.0.0, it works (but many different errors are shown,
> see below *).
> So the combination of gcc/clang and pahole version can affect it.
>
>>
>>>
>>> I also attached the kernel config file.
>>>
>>> What is the recommended combination of the tools?
>>> Should I use Clang to build the kernel for BTF?
>>
>> should work fine with both gcc and clang
>
> And I guess it depends on compiler version, doesn't it?
>
> Thank you,
>
>
> (*)
> BTF .btf.vmlinux.bin.o
> die__process_unit: DW_TAG_label (0xa) @ <0x4b138> not handled!
> die__process_unit: tag not supported 0xa (label)!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b241> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b263> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b290> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b2c0> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b2eb> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b317> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b33a> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b35a> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b37d> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b39e> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b3c4> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b3e7> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b40f> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b435> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b457> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b477> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b4a5> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b4d5> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b505> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b530> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b560> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b585> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b5b2> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b5d9> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b605> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b62e> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b652> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b670> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b694> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b6b3> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b6d9> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b705> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b72b> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b753> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b77f> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b7b3> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b7e4> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b811> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b83c> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b869> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b88c> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b8bd> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b8e7> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b90b> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b930> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b960> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b997> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4b9ce> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4ba00> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4ba24> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4ba4d> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4ba89> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4babc> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4bae4> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4bb0b> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4bb2e> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4bb4e> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4bb6d> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4bb8a> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4bba8> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4bbc5> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4bbe1> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4bc01> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4bc1c> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4bc38> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4bc58> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4bc79> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4bc95> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4bcb2> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x7efd6> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x7effe> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x7f11c> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x7f143> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x7f178> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x7f1a4> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x7f1ca> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x7f1fb> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x7f22f> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x7f25a> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x7f28d> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x7f2b7> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x1eea81> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x1eea9d> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x1eeac3> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x1eeaf3> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x1eeb0f> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x1eeb30> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x1eeb59> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x340734> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4c6b4a> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4c6b67> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4c6b8a> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4c6ba5> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4c6bc4> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4c6bea> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4c6c07> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4c6c2a> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4c6c4e> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4c6c79> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4c6c9b> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4c6cc3> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4c6ceb> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x4c6d15> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x30626ac> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x30626cd> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x3062814> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x3062833> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x3062b8b> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f623f6> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f62416> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f62437> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f62458> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f6280b> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f62b09> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f62b37> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f62c45> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f62c60> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f62d69> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f62e8d> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f6305a> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f63ec2> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f63edf> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f63efc> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f63f19> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f63f36> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f63f5b> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f63f80> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f63fa5> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f63fca> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f6fcae> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f6fcc7> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f6fed7> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f6fef0> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f6fdcb> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70352> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f7036f> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70394> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f703b1> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f703d6> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f703f3> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70418> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70435> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f7045a> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f7057b> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f705a6> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f705cf> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f705f8> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70621> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f7064a> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70673> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f7069c> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f706c5> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f706ee> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70716> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f7073e> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70767> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70790> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f707b9> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f707e2> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f7080b> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70834> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70864> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70892> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f708c0> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f708ee> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f7091c> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f7094a> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70978> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f709a6> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f709d4> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70a01> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70a2e> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70a5c> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70a8a> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70ab8> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70ae6> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70b14> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70b42> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70b72> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70ba0> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70bce> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70bfc> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70c2a> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70c58> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70c86> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70cb4> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70ce2> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70d0f> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70d3c> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70d6a> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70d98> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70dc6> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70df4> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70e22> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70e50> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70e71> not handled!
> die__process_unit: DW_TAG_label (0xa) @ <0x5f70e94> not handled!
> LD .tmp_vmlinux.kallsyms1
>
>>
>> jirka
>>
>>>
>>> Thank you,
>>>
>>> --
>>> Masami Hiramatsu (Google) <mhiramat@kernel.org>
>>
>>
>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
2023-07-26 14:46 ` Alan Maguire
@ 2023-07-27 0:38 ` Masami Hiramatsu
2023-07-27 1:51 ` Masami Hiramatsu
2023-07-27 13:10 ` Arnaldo Carvalho de Melo
0 siblings, 2 replies; 14+ messages in thread
From: Masami Hiramatsu @ 2023-07-27 0:38 UTC (permalink / raw)
To: Alan Maguire
Cc: Jiri Olsa, Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf,
Steven Rostedt
On Wed, 26 Jul 2023 15:46:03 +0100
Alan Maguire <alan.maguire@oracle.com> wrote:
> On 26/07/2023 12:07, Masami Hiramatsu (Google) wrote:
> > Hi Jiri,
> >
> > On Wed, 26 Jul 2023 12:04:08 +0200
> > Jiri Olsa <olsajiri@gmail.com> wrote:
> >
> >> On Wed, Jul 26, 2023 at 10:25:34AM +0900, Masami Hiramatsu wrote:
> >>> Hello,
> >>> (I resend this because kconfig was too big)
> >>>
> >>> I found that BTF is not generated for gcc-built kernel with that latest
> >>> pahole (v1.25).
> >>
> >> hi,
> >> I can't reproduce on my setup with your .config
> >>
> >> does 'bpftool btf dump file ./vmlinux' show any error?
> >>
> >> is there any error in the kernel build output?
> >
> > Yes, here it is. I saw these 2 lines.
> >
> > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
> > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
>
> This is strange, looks like some CUs were encoded incorrectly or we are
> parsing incorrectly. The error originates in die__process() and happens
> if the dwarf_tag() associated with the DIE isn't an expected unit; it's
> not even a valid tag value (0) it looks like. I've not built with gcc 13
> yet so it's possible that's the reason you're seeing this, I'll try to
> reproduce it..
And this warning message is not good for debugging. It would better dump
the message with DIE index so that we can analyze the DWARF with other
tools.
> >>> When I'm using the distro origin pahole (v1.22) it works. (I also checked
> >>> v1.23 and v1.24, both partially generated BTF)
> >>>
> >>> e.g.
> >>>
> >>> # echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events
> >>> sh: write error: Invalid argument
> >>>
> >>> # cat /sys/kernel/tracing/error_log
> >>> [ 21.595724] trace_fprobe: error: BTF is not available or not supported
> >>> Command: f kfree $arg*
> >>> ^
> >>> [ 21.596032] trace_fprobe: error: Invalid $-valiable specified
> >>> Command: f kfree $arg*
> >>> ^
> >>>
> >>> / # strings /sys/kernel/btf/vmlinux | grep kfree
> >>
> >> hm, if you have this file present, you have BTF in
> >
> > Yes, it seems that the BTF itself is generated, but many entries seems
> > dropped compared with pahole v1.22. So, if a given symbol has BTF, (e.g.
> > kfree_rcu_batch_init) it works.
> >
>
> Yep, BPF generation is more selective about what it emits in 1.25 to
> avoid cases where a kernel function signature is ambiguous (multiple
> functions of the same name with different signatures) or where it has
> unexpected register use. You can observe this via pahole's --verbose
> option (a lot of outut is emitted):
>
> In a built kernel directory (where unstripped vmlinux is present):
> $ PAHOLE_FLAGS=$(./scripts/pahole_flags)
> $ PAHOLE=/usr/local/bin/pahole
> $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
So this will generate BTF from vmlinux DWARF info.
> If you want to investigate why a function has been left out, look for
> "skipping" verbose output like this:
>
> skipping addition of 'access_error'(access_error) due to multiple
> inconsistent function prototypes
> skipping addition of
> 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0)
> due to unexpected register used for parameter
Ah, that's nice. Let me try.
>
> FWIW I see most of the below in my BTF output on bpf-next, though I am
> missing a few, possibly due to differing config options since they don't
> appear in kallsyms either. I don't see the DWARF tag label not handled
> messages so it's possible that's a symptom of something
>
> I suspect however some form of corruption in the DWARF representation
> may be the reason a lot of these are missing. Would be worth trying
> to "objdump -g vmlinux >vmlinux.dwarf" (file will be huge tho) I suspect.
OK, let me dump the dwarf and check what information it has for "kfree".
Thank you,
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
2023-07-27 0:38 ` Masami Hiramatsu
@ 2023-07-27 1:51 ` Masami Hiramatsu
2023-07-31 15:45 ` Alan Maguire
2023-07-27 13:10 ` Arnaldo Carvalho de Melo
1 sibling, 1 reply; 14+ messages in thread
From: Masami Hiramatsu @ 2023-07-27 1:51 UTC (permalink / raw)
To: Masami Hiramatsu
Cc: Alan Maguire, Jiri Olsa, Arnaldo Carvalho de Melo, Yonghong Song,
dwarves, bpf, Steven Rostedt
On Thu, 27 Jul 2023 09:38:14 +0900
Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:
> > Yep, BPF generation is more selective about what it emits in 1.25 to
> > avoid cases where a kernel function signature is ambiguous (multiple
> > functions of the same name with different signatures) or where it has
> > unexpected register use. You can observe this via pahole's --verbose
> > option (a lot of outut is emitted):
> >
> > In a built kernel directory (where unstripped vmlinux is present):
> > $ PAHOLE_FLAGS=$(./scripts/pahole_flags)
> > $ PAHOLE=/usr/local/bin/pahole
> > $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
>
> So this will generate BTF from vmlinux DWARF info.
>
> > If you want to investigate why a function has been left out, look for
> > "skipping" verbose output like this:
> >
> > skipping addition of 'access_error'(access_error) due to multiple
> > inconsistent function prototypes
> > skipping addition of
> > 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0)
> > due to unexpected register used for parameter
>
> Ah, that's nice. Let me try.
$ pahole --version
v1.23
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
OK, so something failed.
$ grep skipping /tmp/pahole.out | wc -l
0
Nothing to be skipped.
$ grep -w kfree /tmp/pahole.out | wc -l
0
$ grep -w vfs_read /tmp/pahole.out | wc -l
0
But both kfree and vfs_read are not found.
$ perf probe -k ./vmlinux -V kfree
Available variables at kfree
@<kfree+0>
(unknown_type) object
$ perf probe -k ./vmlinux -V vfs_read
Available variables at vfs_read
@<vfs_read+0>
char* buf
loff_t* pos
size_t count
struct file* file
However, perf probe can find both in the DWARF info.
Thank you,
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
2023-07-27 0:38 ` Masami Hiramatsu
2023-07-27 1:51 ` Masami Hiramatsu
@ 2023-07-27 13:10 ` Arnaldo Carvalho de Melo
1 sibling, 0 replies; 14+ messages in thread
From: Arnaldo Carvalho de Melo @ 2023-07-27 13:10 UTC (permalink / raw)
To: Masami Hiramatsu
Cc: Alan Maguire, Jiri Olsa, Arnaldo Carvalho de Melo, Yonghong Song,
dwarves, bpf, Steven Rostedt
Em Thu, Jul 27, 2023 at 09:38:14AM +0900, Masami Hiramatsu escreveu:
> On Wed, 26 Jul 2023 15:46:03 +0100
> Alan Maguire <alan.maguire@oracle.com> wrote:
>
> > On 26/07/2023 12:07, Masami Hiramatsu (Google) wrote:
> > > Hi Jiri,
> > >
> > > On Wed, 26 Jul 2023 12:04:08 +0200
> > > Jiri Olsa <olsajiri@gmail.com> wrote:
> > >
> > >> On Wed, Jul 26, 2023 at 10:25:34AM +0900, Masami Hiramatsu wrote:
> > >>> Hello,
> > >>> (I resend this because kconfig was too big)
> > >>>
> > >>> I found that BTF is not generated for gcc-built kernel with that latest
> > >>> pahole (v1.25).
> > >>
> > >> hi,
> > >> I can't reproduce on my setup with your .config
> > >>
> > >> does 'bpftool btf dump file ./vmlinux' show any error?
> > >>
> > >> is there any error in the kernel build output?
> > >
> > > Yes, here it is. I saw these 2 lines.
> > >
> > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
> > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
> >
> > This is strange, looks like some CUs were encoded incorrectly or we are
> > parsing incorrectly. The error originates in die__process() and happens
> > if the dwarf_tag() associated with the DIE isn't an expected unit; it's
> > not even a valid tag value (0) it looks like. I've not built with gcc 13
> > yet so it's possible that's the reason you're seeing this, I'll try to
> > reproduce it..
>
> And this warning message is not good for debugging. It would better dump
> the message with DIE index so that we can analyze the DWARF with other
> tools.
Good idea, I'll add that.
- Arnaldo
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
2023-07-27 1:51 ` Masami Hiramatsu
@ 2023-07-31 15:45 ` Alan Maguire
2023-08-01 1:01 ` Masami Hiramatsu
0 siblings, 1 reply; 14+ messages in thread
From: Alan Maguire @ 2023-07-31 15:45 UTC (permalink / raw)
To: Masami Hiramatsu (Google)
Cc: Jiri Olsa, Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf,
Steven Rostedt
On 27/07/2023 02:51, Masami Hiramatsu (Google) wrote:
> On Thu, 27 Jul 2023 09:38:14 +0900
> Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:
>
>>> Yep, BPF generation is more selective about what it emits in 1.25 to
>>> avoid cases where a kernel function signature is ambiguous (multiple
>>> functions of the same name with different signatures) or where it has
>>> unexpected register use. You can observe this via pahole's --verbose
>>> option (a lot of outut is emitted):
>>>
>>> In a built kernel directory (where unstripped vmlinux is present):
>>> $ PAHOLE_FLAGS=$(./scripts/pahole_flags)
>>> $ PAHOLE=/usr/local/bin/pahole
>>> $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
>>
>> So this will generate BTF from vmlinux DWARF info.
>>
>>> If you want to investigate why a function has been left out, look for
>>> "skipping" verbose output like this:
>>>
>>> skipping addition of 'access_error'(access_error) due to multiple
>>> inconsistent function prototypes
>>> skipping addition of
>>> 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0)
>>> due to unexpected register used for parameter
>>
>> Ah, that's nice. Let me try.
>
> $ pahole --version
> v1.23
>
shouldn't this be v1.25? Is it possible pahole is picking up the wrong
libdwarves? what does "ldd pahole" say?
> $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
>
> OK, so something failed.
>
> $ grep skipping /tmp/pahole.out | wc -l
> 0
>
> Nothing to be skipped.
>
> $ grep -w kfree /tmp/pahole.out | wc -l
> 0
> $ grep -w vfs_read /tmp/pahole.out | wc -l
> 0
>
> But both kfree and vfs_read are not found.
>
> $ perf probe -k ./vmlinux -V kfree
> Available variables at kfree
> @<kfree+0>
> (unknown_type) object
> $ perf probe -k ./vmlinux -V vfs_read
> Available variables at vfs_read
> @<vfs_read+0>
> char* buf
> loff_t* pos
> size_t count
> struct file* file
>
> However, perf probe can find both in the DWARF info.
>
> Thank you,
>
Unfortunately (or fortunately?) I haven't been able to reproduce so far
I'm afraid. I used your config and built gcc 13 from source; everything
worked as expected, with no warnings or missing functions (aside from
the ones skipped due to inconsistent prototypes etc).
One other thing I can think of - is it possible libdw[arf]/libelf
versions might be having an effect here? I'm using libdwarf.so.1.2,
libdw-0.188, libelf-0.188. I can try and match yours. Thanks!
Alan
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
2023-07-31 15:45 ` Alan Maguire
@ 2023-08-01 1:01 ` Masami Hiramatsu
2023-08-01 17:36 ` Alan Maguire
0 siblings, 1 reply; 14+ messages in thread
From: Masami Hiramatsu @ 2023-08-01 1:01 UTC (permalink / raw)
To: Alan Maguire
Cc: Jiri Olsa, Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf,
Steven Rostedt
On Mon, 31 Jul 2023 16:45:24 +0100
Alan Maguire <alan.maguire@oracle.com> wrote:
> On 27/07/2023 02:51, Masami Hiramatsu (Google) wrote:
> > On Thu, 27 Jul 2023 09:38:14 +0900
> > Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:
> >
> >>> Yep, BPF generation is more selective about what it emits in 1.25 to
> >>> avoid cases where a kernel function signature is ambiguous (multiple
> >>> functions of the same name with different signatures) or where it has
> >>> unexpected register use. You can observe this via pahole's --verbose
> >>> option (a lot of outut is emitted):
> >>>
> >>> In a built kernel directory (where unstripped vmlinux is present):
> >>> $ PAHOLE_FLAGS=$(./scripts/pahole_flags)
> >>> $ PAHOLE=/usr/local/bin/pahole
> >>> $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
> >>
> >> So this will generate BTF from vmlinux DWARF info.
> >>
> >>> If you want to investigate why a function has been left out, look for
> >>> "skipping" verbose output like this:
> >>>
> >>> skipping addition of 'access_error'(access_error) due to multiple
> >>> inconsistent function prototypes
> >>> skipping addition of
> >>> 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0)
> >>> due to unexpected register used for parameter
> >>
> >> Ah, that's nice. Let me try.
> >
> > $ pahole --version
> > v1.23
> >
>
> shouldn't this be v1.25? Is it possible pahole is picking up the wrong
> libdwarves? what does "ldd pahole" say?
Here it is.
$ ldd pahole
linux-vdso.so.1 (0x00007ffd6b1e2000)
libdwarves_reorganize.so.1 => /opt/local/pahole/libdwarves_reorganize.so.1 (0x00007f1ddaad9000)
libdwarves.so.1 => /opt/local/pahole/libdwarves.so.1 (0x00007f1ddaa72000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dda82a000)
libdw.so.1 => /usr/local/lib/x86_64-linux-gnu/libdw.so.1 (0x00007f1dda78c000)
libelf.so.1 => /usr/local/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1dda771000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1dda753000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1ddaaef000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dda74e000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1dda723000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dda71e000)
Maybe libdwarvers is not related, I could build pahole v1.23 and v1.22 without
sync the submodule.
I also confirmed the same issue on Ubuntu 22.04's combination, which
update pahole from v1.22 to v1.25 recently. (but gcc is still v11.3)
gcc-11.3 | clang-16.0
---------------------------+----------------
v1.22 OK OK
(ubuntu)
v1.22 OK OK
(local)
v1.23 NG OK
(local)
v1.24 NG -
(local)
v1.25 NG -
(ubuntu)
v1.25 NG OK
(local)
So, as far as I checked, there is something wrong between v1.22 and v1.23
which is also related to gcc-11.3.
>
> > $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
> > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
> >
> > OK, so something failed.
> >
> > $ grep skipping /tmp/pahole.out | wc -l
> > 0
> >
> > Nothing to be skipped.
> >
> > $ grep -w kfree /tmp/pahole.out | wc -l
> > 0
> > $ grep -w vfs_read /tmp/pahole.out | wc -l
> > 0
> >
> > But both kfree and vfs_read are not found.
> >
> > $ perf probe -k ./vmlinux -V kfree
> > Available variables at kfree
> > @<kfree+0>
> > (unknown_type) object
> > $ perf probe -k ./vmlinux -V vfs_read
> > Available variables at vfs_read
> > @<vfs_read+0>
> > char* buf
> > loff_t* pos
> > size_t count
> > struct file* file
> >
> > However, perf probe can find both in the DWARF info.
> >
> > Thank you,
> >
>
> Unfortunately (or fortunately?) I haven't been able to reproduce so far
> I'm afraid. I used your config and built gcc 13 from source; everything
> worked as expected, with no warnings or missing functions (aside from
> the ones skipped due to inconsistent prototypes etc).
Yeah, so I think gcc-11.3 is suspicious too (and it seems fixed in gcc-13).
> One other thing I can think of - is it possible libdw[arf]/libelf
> versions might be having an effect here? I'm using libdwarf.so.1.2,
> libdw-0.188, libelf-0.188. I can try and match yours. Thanks!
Both libdw/libelf are 0.181. I didn't install libdwarf.
Hmm, I should update the libdw (elfutils) too.
Thank you,
>
> Alan
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
2023-08-01 1:01 ` Masami Hiramatsu
@ 2023-08-01 17:36 ` Alan Maguire
[not found] ` <CA+JHD93Liq95RvfChifmnE7E9mKR42_W7rtpqgY9KAgYyGTZwQ@mail.gmail.com>
2023-08-01 21:45 ` Arnaldo Carvalho de Melo
0 siblings, 2 replies; 14+ messages in thread
From: Alan Maguire @ 2023-08-01 17:36 UTC (permalink / raw)
To: Masami Hiramatsu (Google)
Cc: Jiri Olsa, Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf,
Steven Rostedt
On 01/08/2023 02:01, Masami Hiramatsu (Google) wrote:
> On Mon, 31 Jul 2023 16:45:24 +0100
> Alan Maguire <alan.maguire@oracle.com> wrote:
>
>> On 27/07/2023 02:51, Masami Hiramatsu (Google) wrote:
>>> On Thu, 27 Jul 2023 09:38:14 +0900
>>> Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:
>>>
>>>>> Yep, BPF generation is more selective about what it emits in 1.25 to
>>>>> avoid cases where a kernel function signature is ambiguous (multiple
>>>>> functions of the same name with different signatures) or where it has
>>>>> unexpected register use. You can observe this via pahole's --verbose
>>>>> option (a lot of outut is emitted):
>>>>>
>>>>> In a built kernel directory (where unstripped vmlinux is present):
>>>>> $ PAHOLE_FLAGS=$(./scripts/pahole_flags)
>>>>> $ PAHOLE=/usr/local/bin/pahole
>>>>> $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
>>>>
>>>> So this will generate BTF from vmlinux DWARF info.
>>>>
>>>>> If you want to investigate why a function has been left out, look for
>>>>> "skipping" verbose output like this:
>>>>>
>>>>> skipping addition of 'access_error'(access_error) due to multiple
>>>>> inconsistent function prototypes
>>>>> skipping addition of
>>>>> 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0)
>>>>> due to unexpected register used for parameter
>>>>
>>>> Ah, that's nice. Let me try.
>>>
>>> $ pahole --version
>>> v1.23
>>>
>>
>> shouldn't this be v1.25? Is it possible pahole is picking up the wrong
>> libdwarves? what does "ldd pahole" say?
>
> Here it is.
> $ ldd pahole
> linux-vdso.so.1 (0x00007ffd6b1e2000)
> libdwarves_reorganize.so.1 => /opt/local/pahole/libdwarves_reorganize.so.1 (0x00007f1ddaad9000)
> libdwarves.so.1 => /opt/local/pahole/libdwarves.so.1 (0x00007f1ddaa72000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dda82a000)
> libdw.so.1 => /usr/local/lib/x86_64-linux-gnu/libdw.so.1 (0x00007f1dda78c000)
> libelf.so.1 => /usr/local/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1dda771000)
> libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1dda753000)
> /lib64/ld-linux-x86-64.so.2 (0x00007f1ddaaef000)
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dda74e000)
> liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1dda723000)
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dda71e000)
>
> Maybe libdwarvers is not related, I could build pahole v1.23 and v1.22 without
> sync the submodule.
>
> I also confirmed the same issue on Ubuntu 22.04's combination, which
> update pahole from v1.22 to v1.25 recently. (but gcc is still v11.3)
>
> gcc-11.3 | clang-16.0
> ---------------------------+----------------
> v1.22 OK OK
> (ubuntu)
> v1.22 OK OK
> (local)
> v1.23 NG OK
> (local)
> v1.24 NG -
> (local)
> v1.25 NG -
> (ubuntu)
> v1.25 NG OK
> (local)
>
> So, as far as I checked, there is something wrong between v1.22 and v1.23
> which is also related to gcc-11.3.
>
One thing that is notable about gcc 11 is that I believe it's the first
gcc release to emit DWARF5 by default. I wonder if it's possible that
the kernel emitted DWARF5, but pahole was built with libraries that
didn't support it yet? Not sure how that fits with the fact that
pahole v1.22 works though. If you have a chance, it might be worth
experimenting with your kernel .config to specify DWARF4 to see if
that makes a difference.
You can check DWARF version associated with CUs with
readelf --debug-dump=info vmlinux |grep -A 2 'Compilation Unit'
>>
>>> $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
>>> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
>>>
>>> OK, so something failed.
>>>
>>> $ grep skipping /tmp/pahole.out | wc -l
>>> 0
>>>
>>> Nothing to be skipped.
>>>
>>> $ grep -w kfree /tmp/pahole.out | wc -l
>>> 0
>>> $ grep -w vfs_read /tmp/pahole.out | wc -l
>>> 0
>>>
>>> But both kfree and vfs_read are not found.
>>>
>>> $ perf probe -k ./vmlinux -V kfree
>>> Available variables at kfree
>>> @<kfree+0>
>>> (unknown_type) object
>>> $ perf probe -k ./vmlinux -V vfs_read
>>> Available variables at vfs_read
>>> @<vfs_read+0>
>>> char* buf
>>> loff_t* pos
>>> size_t count
>>> struct file* file
>>>
>>> However, perf probe can find both in the DWARF info.
>>>
>>> Thank you,
>>>
>>
>> Unfortunately (or fortunately?) I haven't been able to reproduce so far
>> I'm afraid. I used your config and built gcc 13 from source; everything
>> worked as expected, with no warnings or missing functions (aside from
>> the ones skipped due to inconsistent prototypes etc).
>
> Yeah, so I think gcc-11.3 is suspicious too (and it seems fixed in gcc-13).
>
>
>> One other thing I can think of - is it possible libdw[arf]/libelf
>> versions might be having an effect here? I'm using libdwarf.so.1.2,
>> libdw-0.188, libelf-0.188. I can try and match yours. Thanks!
>
> Both libdw/libelf are 0.181. I didn't install libdwarf.
> Hmm, I should update the libdw (elfutils) too.
That might help. Thanks!
Alan
> Thank you,
>
>>
>> Alan
>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Fwd: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
[not found] ` <CA+JHD93Liq95RvfChifmnE7E9mKR42_W7rtpqgY9KAgYyGTZwQ@mail.gmail.com>
@ 2023-08-01 18:28 ` Arnaldo Carvalho de Melo
2023-08-02 8:02 ` Masami Hiramatsu
0 siblings, 1 reply; 14+ messages in thread
From: Arnaldo Carvalho de Melo @ 2023-08-01 18:28 UTC (permalink / raw)
To: Masami Hiramatsu (Google), Alan Maguire
Cc: cc: Jiri Olsa, Arnaldo Carvalho de Melo, dwarves, Steven Rostedt,
bpf, Yonghong Song
Sorry, replied only to Alan :-\
- Arnaldo
---------- Forwarded message ---------
From: Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com>
Date: Tue, Aug 1, 2023 at 3:26 PM
Subject: Re: [RESEND] BTF is not generated for gcc-built kernel with
the latest pahole
To: Alan Maguire <alan.maguire@oracle.com>
On Tue, Aug 1, 2023 at 2:37 PM Alan Maguire <alan.maguire@oracle.com> wrote:
> On 01/08/2023 02:01, Masami Hiramatsu (Google) wrote:
> > On Mon, 31 Jul 2023 16:45:24 +0100
> > Alan Maguire <alan.maguire@oracle.com> wrote:
> >> Unfortunately (or fortunately?) I haven't been able to reproduce so far
> >> I'm afraid. I used your config and built gcc 13 from source; everything
> >> worked as expected, with no warnings or missing functions (aside from
> >> the ones skipped due to inconsistent prototypes etc).
> >
> > Yeah, so I think gcc-11.3 is suspicious too (and it seems fixed in gcc-13).
See below, but this one is interesting, gcc-13 works with
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y?
> >> One other thing I can think of - is it possible libdw[arf]/libelf
> >> versions might be having an effect here? I'm using libdwarf.so.1.2,
> >> libdw-0.188, libelf-0.188. I can try and match yours. Thanks!
> >
> > Both libdw/libelf are 0.181. I didn't install libdwarf.
> > Hmm, I should update the libdw (elfutils) too.
>
> That might help. Thanks!
Probably he needs to tweak these CONFIG_ entries:
⬢[acme@toolbox perf-tools]$ grep DWARF ../build/v6.2-rc5+/.config
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
⬢[acme@toolbox perf-tools]$
And make it use CONFIG_DEBUG_INFO_DWARF4=y,
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=n
For DWARF5 I need to forward port what I have in:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=WIP-imported-unit
- Arnaldo
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
2023-08-01 17:36 ` Alan Maguire
[not found] ` <CA+JHD93Liq95RvfChifmnE7E9mKR42_W7rtpqgY9KAgYyGTZwQ@mail.gmail.com>
@ 2023-08-01 21:45 ` Arnaldo Carvalho de Melo
2023-08-02 14:29 ` Masami Hiramatsu
1 sibling, 1 reply; 14+ messages in thread
From: Arnaldo Carvalho de Melo @ 2023-08-01 21:45 UTC (permalink / raw)
To: Alan Maguire
Cc: Masami Hiramatsu (Google), Jiri Olsa, Arnaldo Carvalho de Melo,
Yonghong Song, dwarves, bpf, Steven Rostedt
Em Tue, Aug 01, 2023 at 06:36:49PM +0100, Alan Maguire escreveu:
> On 01/08/2023 02:01, Masami Hiramatsu (Google) wrote:
> > On Mon, 31 Jul 2023 16:45:24 +0100
> > Alan Maguire <alan.maguire@oracle.com> wrote:
> >
> >> On 27/07/2023 02:51, Masami Hiramatsu (Google) wrote:
> >>> On Thu, 27 Jul 2023 09:38:14 +0900
> >>> Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:
> >>>
> >>>>> Yep, BPF generation is more selective about what it emits in 1.25 to
> >>>>> avoid cases where a kernel function signature is ambiguous (multiple
> >>>>> functions of the same name with different signatures) or where it has
> >>>>> unexpected register use. You can observe this via pahole's --verbose
> >>>>> option (a lot of outut is emitted):
> >>>>>
> >>>>> In a built kernel directory (where unstripped vmlinux is present):
> >>>>> $ PAHOLE_FLAGS=$(./scripts/pahole_flags)
> >>>>> $ PAHOLE=/usr/local/bin/pahole
> >>>>> $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
> >>>>
> >>>> So this will generate BTF from vmlinux DWARF info.
> >>>>
> >>>>> If you want to investigate why a function has been left out, look for
> >>>>> "skipping" verbose output like this:
> >>>>>
> >>>>> skipping addition of 'access_error'(access_error) due to multiple
> >>>>> inconsistent function prototypes
> >>>>> skipping addition of
> >>>>> 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0)
> >>>>> due to unexpected register used for parameter
> >>>>
> >>>> Ah, that's nice. Let me try.
> >>>
> >>> $ pahole --version
> >>> v1.23
> >>>
> >>
> >> shouldn't this be v1.25? Is it possible pahole is picking up the wrong
> >> libdwarves? what does "ldd pahole" say?
> >
> > Here it is.
> > $ ldd pahole
> > linux-vdso.so.1 (0x00007ffd6b1e2000)
> > libdwarves_reorganize.so.1 => /opt/local/pahole/libdwarves_reorganize.so.1 (0x00007f1ddaad9000)
> > libdwarves.so.1 => /opt/local/pahole/libdwarves.so.1 (0x00007f1ddaa72000)
> > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dda82a000)
> > libdw.so.1 => /usr/local/lib/x86_64-linux-gnu/libdw.so.1 (0x00007f1dda78c000)
> > libelf.so.1 => /usr/local/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1dda771000)
> > libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1dda753000)
> > /lib64/ld-linux-x86-64.so.2 (0x00007f1ddaaef000)
> > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dda74e000)
> > liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1dda723000)
> > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dda71e000)
> >
> > Maybe libdwarvers is not related, I could build pahole v1.23 and v1.22 without
> > sync the submodule.
> >
> > I also confirmed the same issue on Ubuntu 22.04's combination, which
> > update pahole from v1.22 to v1.25 recently. (but gcc is still v11.3)
> >
> > gcc-11.3 | clang-16.0
> > ---------------------------+----------------
> > v1.22 OK OK
> > (ubuntu)
> > v1.22 OK OK
> > (local)
> > v1.23 NG OK
> > (local)
> > v1.24 NG -
> > (local)
> > v1.25 NG -
> > (ubuntu)
> > v1.25 NG OK
> > (local)
> >
> > So, as far as I checked, there is something wrong between v1.22 and v1.23
> > which is also related to gcc-11.3.
> >
>
> One thing that is notable about gcc 11 is that I believe it's the first
> gcc release to emit DWARF5 by default. I wonder if it's possible that
> the kernel emitted DWARF5, but pahole was built with libraries that
> didn't support it yet? Not sure how that fits with the fact that
To check that please tweak:
⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
⬢[acme@toolbox perf-tools-next]$
i.e. disable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT and enable
CONFIG_DEBUG_INFO_DWARF4.
- Arnaldo
> pahole v1.22 works though. If you have a chance, it might be worth
> experimenting with your kernel .config to specify DWARF4 to see if
> that makes a difference.
>
> You can check DWARF version associated with CUs with
>
> readelf --debug-dump=info vmlinux |grep -A 2 'Compilation Unit'
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Fwd: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
2023-08-01 18:28 ` Fwd: " Arnaldo Carvalho de Melo
@ 2023-08-02 8:02 ` Masami Hiramatsu
0 siblings, 0 replies; 14+ messages in thread
From: Masami Hiramatsu @ 2023-08-02 8:02 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: Alan Maguire, cc: Jiri Olsa, Arnaldo Carvalho de Melo, dwarves,
Steven Rostedt, bpf, Yonghong Song
On Tue, 1 Aug 2023 15:28:38 -0300
Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote:
> Sorry, replied only to Alan :-\
>
> - Arnaldo
>
> ---------- Forwarded message ---------
> From: Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com>
> Date: Tue, Aug 1, 2023 at 3:26 PM
> Subject: Re: [RESEND] BTF is not generated for gcc-built kernel with
> the latest pahole
> To: Alan Maguire <alan.maguire@oracle.com>
>
>
> On Tue, Aug 1, 2023 at 2:37 PM Alan Maguire <alan.maguire@oracle.com> wrote:
> > On 01/08/2023 02:01, Masami Hiramatsu (Google) wrote:
> > > On Mon, 31 Jul 2023 16:45:24 +0100
> > > Alan Maguire <alan.maguire@oracle.com> wrote:
>
> > >> Unfortunately (or fortunately?) I haven't been able to reproduce so far
> > >> I'm afraid. I used your config and built gcc 13 from source; everything
> > >> worked as expected, with no warnings or missing functions (aside from
> > >> the ones skipped due to inconsistent prototypes etc).
> > >
> > > Yeah, so I think gcc-11.3 is suspicious too (and it seems fixed in gcc-13).
>
> See below, but this one is interesting, gcc-13 works with
> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y?
I'm using gcc-11.3, and it seems to work.
>
> > >> One other thing I can think of - is it possible libdw[arf]/libelf
> > >> versions might be having an effect here? I'm using libdwarf.so.1.2,
> > >> libdw-0.188, libelf-0.188. I can try and match yours. Thanks!
> > >
> > > Both libdw/libelf are 0.181. I didn't install libdwarf.
> > > Hmm, I should update the libdw (elfutils) too.
> >
> > That might help. Thanks!
>
> Probably he needs to tweak these CONFIG_ entries:
>
> ⬢[acme@toolbox perf-tools]$ grep DWARF ../build/v6.2-rc5+/.config
> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> # CONFIG_DEBUG_INFO_DWARF4 is not set
> # CONFIG_DEBUG_INFO_DWARF5 is not set
> ⬢[acme@toolbox perf-tools]$
>
> And make it use CONFIG_DEBUG_INFO_DWARF4=y,
> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=n
OK, interesting. Let me check again with DWARF4.
Thank you!
>
> For DWARF5 I need to forward port what I have in:
>
> https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=WIP-imported-unit
>
> - Arnaldo
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
2023-08-01 21:45 ` Arnaldo Carvalho de Melo
@ 2023-08-02 14:29 ` Masami Hiramatsu
0 siblings, 0 replies; 14+ messages in thread
From: Masami Hiramatsu @ 2023-08-02 14:29 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: Alan Maguire, Masami Hiramatsu (Google), Jiri Olsa,
Arnaldo Carvalho de Melo, Yonghong Song, dwarves, bpf,
Steven Rostedt
On Tue, 1 Aug 2023 18:45:49 -0300
Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> Em Tue, Aug 01, 2023 at 06:36:49PM +0100, Alan Maguire escreveu:
> > On 01/08/2023 02:01, Masami Hiramatsu (Google) wrote:
> > > On Mon, 31 Jul 2023 16:45:24 +0100
> > > Alan Maguire <alan.maguire@oracle.com> wrote:
> > >
> > >> On 27/07/2023 02:51, Masami Hiramatsu (Google) wrote:
> > >>> On Thu, 27 Jul 2023 09:38:14 +0900
> > >>> Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:
> > >>>
> > >>>>> Yep, BPF generation is more selective about what it emits in 1.25 to
> > >>>>> avoid cases where a kernel function signature is ambiguous (multiple
> > >>>>> functions of the same name with different signatures) or where it has
> > >>>>> unexpected register use. You can observe this via pahole's --verbose
> > >>>>> option (a lot of outut is emitted):
> > >>>>>
> > >>>>> In a built kernel directory (where unstripped vmlinux is present):
> > >>>>> $ PAHOLE_FLAGS=$(./scripts/pahole_flags)
> > >>>>> $ PAHOLE=/usr/local/bin/pahole
> > >>>>> $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
> > >>>>
> > >>>> So this will generate BTF from vmlinux DWARF info.
> > >>>>
> > >>>>> If you want to investigate why a function has been left out, look for
> > >>>>> "skipping" verbose output like this:
> > >>>>>
> > >>>>> skipping addition of 'access_error'(access_error) due to multiple
> > >>>>> inconsistent function prototypes
> > >>>>> skipping addition of
> > >>>>> 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0)
> > >>>>> due to unexpected register used for parameter
> > >>>>
> > >>>> Ah, that's nice. Let me try.
> > >>>
> > >>> $ pahole --version
> > >>> v1.23
> > >>>
> > >>
> > >> shouldn't this be v1.25? Is it possible pahole is picking up the wrong
> > >> libdwarves? what does "ldd pahole" say?
> > >
> > > Here it is.
> > > $ ldd pahole
> > > linux-vdso.so.1 (0x00007ffd6b1e2000)
> > > libdwarves_reorganize.so.1 => /opt/local/pahole/libdwarves_reorganize.so.1 (0x00007f1ddaad9000)
> > > libdwarves.so.1 => /opt/local/pahole/libdwarves.so.1 (0x00007f1ddaa72000)
> > > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dda82a000)
> > > libdw.so.1 => /usr/local/lib/x86_64-linux-gnu/libdw.so.1 (0x00007f1dda78c000)
> > > libelf.so.1 => /usr/local/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1dda771000)
> > > libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1dda753000)
> > > /lib64/ld-linux-x86-64.so.2 (0x00007f1ddaaef000)
> > > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dda74e000)
> > > liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1dda723000)
> > > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dda71e000)
> > >
> > > Maybe libdwarvers is not related, I could build pahole v1.23 and v1.22 without
> > > sync the submodule.
> > >
> > > I also confirmed the same issue on Ubuntu 22.04's combination, which
> > > update pahole from v1.22 to v1.25 recently. (but gcc is still v11.3)
> > >
> > > gcc-11.3 | clang-16.0
> > > ---------------------------+----------------
> > > v1.22 OK OK
> > > (ubuntu)
> > > v1.22 OK OK
> > > (local)
> > > v1.23 NG OK
> > > (local)
> > > v1.24 NG -
> > > (local)
> > > v1.25 NG -
> > > (ubuntu)
> > > v1.25 NG OK
> > > (local)
> > >
> > > So, as far as I checked, there is something wrong between v1.22 and v1.23
> > > which is also related to gcc-11.3.
> > >
> >
> > One thing that is notable about gcc 11 is that I believe it's the first
> > gcc release to emit DWARF5 by default. I wonder if it's possible that
> > the kernel emitted DWARF5, but pahole was built with libraries that
> > didn't support it yet? Not sure how that fits with the fact that
>
> To check that please tweak:
>
> ⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config
> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> # CONFIG_DEBUG_INFO_DWARF4 is not set
> # CONFIG_DEBUG_INFO_DWARF5 is not set
> ⬢[acme@toolbox perf-tools-next]$
>
> i.e. disable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT and enable
> CONFIG_DEBUG_INFO_DWARF4.
Hm, with CONFIG_DEBUG_INFO_DWARF4, no warning were shown.
LD .tmp_vmlinux.btf
BTF .btf.vmlinux.bin.o
LD .tmp_vmlinux.kallsyms1
And
/ # strings /sys/kernel/btf/vmlinux | wc -l
89921
/ # strings /sys/kernel/btf/vmlinux | grep -w kfree
kfree
It seems the BTF is correctly generated. (with DWARF5, the number of symbols
are about 30000.)
Maybe we should check the pahole version and choose the DWARF4 if BTF is selected.
Thank you!
>
> - Arnaldo
>
> > pahole v1.22 works though. If you have a chance, it might be worth
> > experimenting with your kernel .config to specify DWARF4 to see if
> > that makes a difference.
> >
> > You can check DWARF version associated with CUs with
> >
> > readelf --debug-dump=info vmlinux |grep -A 2 'Compilation Unit'
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2023-08-02 14:29 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-07-26 1:25 [RESEND] BTF is not generated for gcc-built kernel with the latest pahole Masami Hiramatsu
2023-07-26 10:04 ` Jiri Olsa
2023-07-26 11:07 ` Masami Hiramatsu
2023-07-26 14:46 ` Alan Maguire
2023-07-27 0:38 ` Masami Hiramatsu
2023-07-27 1:51 ` Masami Hiramatsu
2023-07-31 15:45 ` Alan Maguire
2023-08-01 1:01 ` Masami Hiramatsu
2023-08-01 17:36 ` Alan Maguire
[not found] ` <CA+JHD93Liq95RvfChifmnE7E9mKR42_W7rtpqgY9KAgYyGTZwQ@mail.gmail.com>
2023-08-01 18:28 ` Fwd: " Arnaldo Carvalho de Melo
2023-08-02 8:02 ` Masami Hiramatsu
2023-08-01 21:45 ` Arnaldo Carvalho de Melo
2023-08-02 14:29 ` Masami Hiramatsu
2023-07-27 13:10 ` Arnaldo Carvalho de Melo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).