* [PATCH bpf-next] scripts/gen-btf.sh: Disable LTO when generating initial .o file
@ 2026-01-05 21:12 Nathan Chancellor
2026-01-05 22:01 ` Ihor Solodrai
0 siblings, 1 reply; 6+ messages in thread
From: Nathan Chancellor @ 2026-01-05 21:12 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Martin KaFai Lau, Eduard Zingerman, Yonghong Song, Ihor Solodrai
Cc: bpf, linux-kernel, llvm, Nathan Chancellor
After commit 600605853f87 ("scripts/gen-btf.sh: Fix .btf.o generation
when compiling for RISCV"), there is an error from llvm-objcopy when
CONFIG_LTO_CLANG is enabled:
llvm-objcopy: error: '.tmp_vmlinux1.btf.o': The file was not recognized as a valid object file
Failed to generate BTF for vmlinux
KBUILD_CFLAGS includes CC_FLAGS_LTO, which makes clang emit an LLVM IR
object, rather than an ELF one as expected by llvm-objcopy.
Most areas of the kernel deal with this by filtering out CC_FLAGS_LTO
from KBUILD_CFLAGS for the particular object or directory but this is
not so easy to do in bash. Just include '-fno-lto' after KBUILD_CFLAGS
to ensure an ELF object is consistently created as the initial .o file.
Fixes: 600605853f87 ("scripts/gen-btf.sh: Fix .btf.o generation when compiling for RISCV")
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
---
scripts/gen-btf.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/gen-btf.sh b/scripts/gen-btf.sh
index d6457661b9b6..08b46b91c04b 100755
--- a/scripts/gen-btf.sh
+++ b/scripts/gen-btf.sh
@@ -87,7 +87,7 @@ gen_btf_o()
# SHF_ALLOC because .BTF will be part of the vmlinux image. --strip-all
# deletes all symbols including __start_BTF and __stop_BTF, which will
# be redefined in the linker script.
- echo "" | ${CC} ${CLANG_FLAGS} ${KBUILD_CFLAGS} -c -x c -o ${btf_data} -
+ echo "" | ${CC} ${CLANG_FLAGS} ${KBUILD_CFLAGS} -fno-lto -c -x c -o ${btf_data} -
${OBJCOPY} --add-section .BTF=${ELF_FILE}.BTF \
--set-section-flags .BTF=alloc,readonly ${btf_data}
${OBJCOPY} --only-section=.BTF --strip-all ${btf_data}
---
base-commit: a069190b590e108223cd841a1c2d0bfb92230ecc
change-id: 20260105-fix-gen-btf-sh-lto-007fe4908070
Best regards,
--
Nathan Chancellor <nathan@kernel.org>
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH bpf-next] scripts/gen-btf.sh: Disable LTO when generating initial .o file 2026-01-05 21:12 [PATCH bpf-next] scripts/gen-btf.sh: Disable LTO when generating initial .o file Nathan Chancellor @ 2026-01-05 22:01 ` Ihor Solodrai 2026-01-05 23:46 ` Nathan Chancellor 0 siblings, 1 reply; 6+ messages in thread From: Ihor Solodrai @ 2026-01-05 22:01 UTC (permalink / raw) To: Nathan Chancellor, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman, Yonghong Song Cc: bpf, linux-kernel, llvm On 1/5/26 1:12 PM, Nathan Chancellor wrote: > After commit 600605853f87 ("scripts/gen-btf.sh: Fix .btf.o generation > when compiling for RISCV"), there is an error from llvm-objcopy when > CONFIG_LTO_CLANG is enabled: > > llvm-objcopy: error: '.tmp_vmlinux1.btf.o': The file was not recognized as a valid object file > Failed to generate BTF for vmlinux > > KBUILD_CFLAGS includes CC_FLAGS_LTO, which makes clang emit an LLVM IR > object, rather than an ELF one as expected by llvm-objcopy. > > Most areas of the kernel deal with this by filtering out CC_FLAGS_LTO > from KBUILD_CFLAGS for the particular object or directory but this is > not so easy to do in bash. Just include '-fno-lto' after KBUILD_CFLAGS > to ensure an ELF object is consistently created as the initial .o file. > > Fixes: 600605853f87 ("scripts/gen-btf.sh: Fix .btf.o generation when compiling for RISCV") > Signed-off-by: Nathan Chancellor <nathan@kernel.org> > --- > scripts/gen-btf.sh | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/scripts/gen-btf.sh b/scripts/gen-btf.sh > index d6457661b9b6..08b46b91c04b 100755 > --- a/scripts/gen-btf.sh > +++ b/scripts/gen-btf.sh > @@ -87,7 +87,7 @@ gen_btf_o() > # SHF_ALLOC because .BTF will be part of the vmlinux image. --strip-all > # deletes all symbols including __start_BTF and __stop_BTF, which will > # be redefined in the linker script. > - echo "" | ${CC} ${CLANG_FLAGS} ${KBUILD_CFLAGS} -c -x c -o ${btf_data} - > + echo "" | ${CC} ${CLANG_FLAGS} ${KBUILD_CFLAGS} -fno-lto -c -x c -o ${btf_data} - > ${OBJCOPY} --add-section .BTF=${ELF_FILE}.BTF \ > --set-section-flags .BTF=alloc,readonly ${btf_data} > ${OBJCOPY} --only-section=.BTF --strip-all ${btf_data} Hi Nathan, thank you for the patch. I'm starting to think it wasn't a good idea to do echo "" | ${CC} ... here, given the number of associated bugs. Before gen-btf.sh was introduced, the .btf.o binary was generated with this [1]: ${OBJCOPY} --only-section=.BTF --set-section-flags .BTF=alloc,readonly \ --strip-all ${1} "${btf_data}" 2>/dev/null I changed to ${CC} on the assumption it's a quicker operation than stripping entire vmlinux. But maybe it's not worth it and we should change back to --strip-all? wdyt? [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/link-vmlinux.sh?h=v6.18#n110 > > --- > base-commit: a069190b590e108223cd841a1c2d0bfb92230ecc > change-id: 20260105-fix-gen-btf-sh-lto-007fe4908070 > > Best regards, > -- > Nathan Chancellor <nathan@kernel.org> > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH bpf-next] scripts/gen-btf.sh: Disable LTO when generating initial .o file 2026-01-05 22:01 ` Ihor Solodrai @ 2026-01-05 23:46 ` Nathan Chancellor 2026-01-06 1:06 ` Ihor Solodrai 0 siblings, 1 reply; 6+ messages in thread From: Nathan Chancellor @ 2026-01-05 23:46 UTC (permalink / raw) To: Ihor Solodrai Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman, Yonghong Song, bpf, linux-kernel, llvm On Mon, Jan 05, 2026 at 02:01:36PM -0800, Ihor Solodrai wrote: > Hi Nathan, thank you for the patch. > > I'm starting to think it wasn't a good idea to do > > echo "" | ${CC} ... > > here, given the number of associated bugs. Yeah, I was wondering if a lack of KBUILD_CPPFLAGS would also be a problem since that contains the endianness flag for some targets. I cannot imagine any more issues than that but I can understand wanting to back out of it. > Before gen-btf.sh was introduced, the .btf.o binary was generated with this [1]: > > ${OBJCOPY} --only-section=.BTF --set-section-flags .BTF=alloc,readonly \ > --strip-all ${1} "${btf_data}" 2>/dev/null > > I changed to ${CC} on the assumption it's a quicker operation than > stripping entire vmlinux. But maybe it's not worth it and we should > change back to --strip-all? wdyt? That certainly seems more robust to me. I see the logic but with '--only-section' and no glob, I would expect that to be a rather quick operation but I am running out of time today to test and benchmark such a change. I will try to do it tomorrow unless someone beats me to it. Cheers, Nathan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH bpf-next] scripts/gen-btf.sh: Disable LTO when generating initial .o file 2026-01-05 23:46 ` Nathan Chancellor @ 2026-01-06 1:06 ` Ihor Solodrai 2026-01-06 21:53 ` Nathan Chancellor 0 siblings, 1 reply; 6+ messages in thread From: Ihor Solodrai @ 2026-01-06 1:06 UTC (permalink / raw) To: Nathan Chancellor Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman, Yonghong Song, bpf, linux-kernel, llvm On 1/5/26 3:46 PM, Nathan Chancellor wrote: > On Mon, Jan 05, 2026 at 02:01:36PM -0800, Ihor Solodrai wrote: >> Hi Nathan, thank you for the patch. >> >> I'm starting to think it wasn't a good idea to do >> >> echo "" | ${CC} ... >> >> here, given the number of associated bugs. > > Yeah, I was wondering if a lack of KBUILD_CPPFLAGS would also be a > problem since that contains the endianness flag for some targets. I > cannot imagine any more issues than that but I can understand wanting to > back out of it. > >> Before gen-btf.sh was introduced, the .btf.o binary was generated with this [1]: >> >> ${OBJCOPY} --only-section=.BTF --set-section-flags .BTF=alloc,readonly \ >> --strip-all ${1} "${btf_data}" 2>/dev/null >> >> I changed to ${CC} on the assumption it's a quicker operation than >> stripping entire vmlinux. But maybe it's not worth it and we should >> change back to --strip-all? wdyt? > > That certainly seems more robust to me. I see the logic but with > '--only-section' and no glob, I would expect that to be a rather quick > operation but I am running out of time today to test and benchmark such > a change. I will try to do it tomorrow unless someone beats me to it. I got curious and did a little experiment. Basically, I ran perf stat on this part of gen-btf.sh: echo "" | ${CC} ${CLANG_FLAGS} ${KBUILD_CFLAGS} -c -x c -o ${btf_data} - ${OBJCOPY} --add-section .BTF=${ELF_FILE}.BTF \ --set-section-flags .BTF=alloc,readonly ${btf_data} ${OBJCOPY} --only-section=.BTF --strip-all ${btf_data} Replacing ${CC} command with: ${OBJCOPY} --strip-all "${ELF_FILE}" ${btf_data} 2>/dev/null for comparison. TL;DR is that using ${CC} is: * about 1.5x faster than GNU objcopy --strip-all .tmp_vmlinux1 * about 16x (!) faster than llvm-objcopy --strip-all .tmp_vmlinux1 With obvious caveats that this is a particular machine (Threadripper PRO 3975WX), toolchain etc: * clang version 21.1.7 * gcc (GCC) 15.2.1 20251211 This is bpf-next (a069190b590e) with BPF CI-like kconfig. Pasting perf stat output below. # llvm-objcopy --strip-all $ perf stat -r 31 -- ./gen-btf.o_strip.sh Performance counter stats for './gen-btf.o_strip.sh' (31 runs): 1,300,945,256 task-clock:u # 0.962 CPUs utilized ( +- 0.10% ) 0 context-switches:u # 0.000 /sec 0 cpu-migrations:u # 0.000 /sec 327,311 page-faults:u # 251.595 K/sec ( +- 0.00% ) 1,532,927,570 instructions:u # 1.33 insn per cycle # 0.03 stalled cycles per insn ( +- 0.00% ) 1,155,639,083 cycles:u # 0.888 GHz ( +- 0.18% ) 53,144,866 stalled-cycles-frontend:u # 4.60% frontend cycles idle ( +- 0.99% ) 297,229,466 branches:u # 228.472 M/sec ( +- 0.00% ) 903,337 branch-misses:u # 0.30% of all branches ( +- 0.02% ) 1.35200 +- 0.00137 seconds time elapsed ( +- 0.10% ) # GNU objcopy --strip-all $ perf stat -r 31 -- ./gen-btf.o_strip.sh Performance counter stats for './gen-btf.o_strip.sh' (31 runs): 119,747,488 task-clock:u # 0.970 CPUs utilized ( +- 0.41% ) 0 context-switches:u # 0.000 /sec 0 cpu-migrations:u # 0.000 /sec 9,186 page-faults:u # 76.711 K/sec ( +- 0.01% ) 132,651,881 instructions:u # 1.68 insn per cycle # 0.08 stalled cycles per insn ( +- 0.00% ) 79,191,259 cycles:u # 0.661 GHz ( +- 1.06% ) 10,136,981 stalled-cycles-frontend:u # 12.80% frontend cycles idle ( +- 2.58% ) 28,422,807 branches:u # 237.356 M/sec ( +- 0.00% ) 354,981 branch-misses:u # 1.25% of all branches ( +- 0.02% ) 0.123415 +- 0.000564 seconds time elapsed ( +- 0.46% ) # echo "" | clang ... $ perf stat -r 31 -- ./gen-btf.o_llvm.sh Performance counter stats for './gen-btf.o_llvm.sh' (31 runs): 62,107,490 task-clock:u # 0.774 CPUs utilized ( +- 0.31% ) 0 context-switches:u # 0.000 /sec 0 cpu-migrations:u # 0.000 /sec 9,755 page-faults:u # 157.066 K/sec ( +- 0.01% ) 88,196,854 instructions:u # 1.18 insn per cycle # 0.19 stalled cycles per insn ( +- 0.00% ) 74,944,793 cycles:u # 1.207 GHz ( +- 0.50% ) 16,494,448 stalled-cycles-frontend:u # 22.01% frontend cycles idle ( +- 0.48% ) 17,914,949 branches:u # 288.451 M/sec ( +- 0.00% ) 459,548 branch-misses:u # 2.57% of all branches ( +- 0.10% ) 0.080237 +- 0.000313 seconds time elapsed ( +- 0.39% ) # echo "" | gcc ... $ perf stat -r 31 -- ./gen-btf.o_gnu.sh Performance counter stats for './gen-btf.o_gnu.sh' (31 runs): 53,683,797 task-clock:u # 0.770 CPUs utilized ( +- 0.33% ) 0 context-switches:u # 0.000 /sec 0 cpu-migrations:u # 0.000 /sec 8,390 page-faults:u # 156.286 K/sec ( +- 0.01% ) 69,398,474 instructions:u # 1.22 insn per cycle # 0.17 stalled cycles per insn ( +- 0.00% ) 56,763,954 cycles:u # 1.057 GHz ( +- 0.39% ) 12,103,546 stalled-cycles-frontend:u # 21.32% frontend cycles idle ( +- 0.47% ) 14,064,366 branches:u # 261.985 M/sec ( +- 0.00% ) 347,383 branch-misses:u # 2.47% of all branches ( +- 0.09% ) 0.069735 +- 0.000253 seconds time elapsed ( +- 0.36% ) > > Cheers, > Nathan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH bpf-next] scripts/gen-btf.sh: Disable LTO when generating initial .o file 2026-01-06 1:06 ` Ihor Solodrai @ 2026-01-06 21:53 ` Nathan Chancellor 2026-01-06 22:01 ` Alexei Starovoitov 0 siblings, 1 reply; 6+ messages in thread From: Nathan Chancellor @ 2026-01-06 21:53 UTC (permalink / raw) To: Ihor Solodrai Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman, Yonghong Song, bpf, linux-kernel, llvm On Mon, Jan 05, 2026 at 05:06:49PM -0800, Ihor Solodrai wrote: > I got curious and did a little experiment. Basically, I ran perf stat > on this part of gen-btf.sh: > > echo "" | ${CC} ${CLANG_FLAGS} ${KBUILD_CFLAGS} -c -x c -o ${btf_data} - > ${OBJCOPY} --add-section .BTF=${ELF_FILE}.BTF \ > --set-section-flags .BTF=alloc,readonly ${btf_data} > ${OBJCOPY} --only-section=.BTF --strip-all ${btf_data} > > Replacing ${CC} command with: > > ${OBJCOPY} --strip-all "${ELF_FILE}" ${btf_data} 2>/dev/null > > for comparison. > > TL;DR is that using ${CC} is: > * about 1.5x faster than GNU objcopy --strip-all .tmp_vmlinux1 > * about 16x (!) faster than llvm-objcopy --strip-all .tmp_vmlinux1 > > With obvious caveats that this is a particular machine (Threadripper > PRO 3975WX), toolchain etc: > * clang version 21.1.7 > * gcc (GCC) 15.2.1 20251211 > > This is bpf-next (a069190b590e) with BPF CI-like kconfig. Oof, that difference between GNU and LLVM's objcopy implementations... At the same time, it was only a little over a second for llvm-objcopy. Maybe that gets worse if more is built into the kernel to the point where it is untenable but maybe it is worth the reduced complexity? That said, my patch is pretty simple (and a follow up for KBUILD_CPPFLAGS if needed would be equally simple), your testing demonstrates that there is some performance improvement, and I cannot imagine there being any other bugs of this nature in this area going forward. I have no real strong opinion, I just need my builds to finish :) Cheers, Nathan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH bpf-next] scripts/gen-btf.sh: Disable LTO when generating initial .o file 2026-01-06 21:53 ` Nathan Chancellor @ 2026-01-06 22:01 ` Alexei Starovoitov 0 siblings, 0 replies; 6+ messages in thread From: Alexei Starovoitov @ 2026-01-06 22:01 UTC (permalink / raw) To: Nathan Chancellor Cc: Ihor Solodrai, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman, Yonghong Song, bpf, LKML, clang-built-linux On Tue, Jan 6, 2026 at 1:53 PM Nathan Chancellor <nathan@kernel.org> wrote: > > On Mon, Jan 05, 2026 at 05:06:49PM -0800, Ihor Solodrai wrote: > > I got curious and did a little experiment. Basically, I ran perf stat > > on this part of gen-btf.sh: > > > > echo "" | ${CC} ${CLANG_FLAGS} ${KBUILD_CFLAGS} -c -x c -o ${btf_data} - > > ${OBJCOPY} --add-section .BTF=${ELF_FILE}.BTF \ > > --set-section-flags .BTF=alloc,readonly ${btf_data} > > ${OBJCOPY} --only-section=.BTF --strip-all ${btf_data} > > > > Replacing ${CC} command with: > > > > ${OBJCOPY} --strip-all "${ELF_FILE}" ${btf_data} 2>/dev/null > > > > for comparison. > > > > TL;DR is that using ${CC} is: > > * about 1.5x faster than GNU objcopy --strip-all .tmp_vmlinux1 > > * about 16x (!) faster than llvm-objcopy --strip-all .tmp_vmlinux1 > > > > With obvious caveats that this is a particular machine (Threadripper > > PRO 3975WX), toolchain etc: > > * clang version 21.1.7 > > * gcc (GCC) 15.2.1 20251211 > > > > This is bpf-next (a069190b590e) with BPF CI-like kconfig. > > Oof, that difference between GNU and LLVM's objcopy implementations... > At the same time, it was only a little over a second for llvm-objcopy. > Maybe that gets worse if more is built into the kernel to the point > where it is untenable but maybe it is worth the reduced complexity? That > said, my patch is pretty simple (and a follow up for KBUILD_CPPFLAGS if > needed would be equally simple), your testing demonstrates that there > is some performance improvement, and I cannot imagine there being any > other bugs of this nature in this area going forward. I have no real > strong opinion, I just need my builds to finish :) Pls resend both patches? Or squash as one ? Sounds like the current one is incomplete. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-01-06 22:01 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-01-05 21:12 [PATCH bpf-next] scripts/gen-btf.sh: Disable LTO when generating initial .o file Nathan Chancellor 2026-01-05 22:01 ` Ihor Solodrai 2026-01-05 23:46 ` Nathan Chancellor 2026-01-06 1:06 ` Ihor Solodrai 2026-01-06 21:53 ` Nathan Chancellor 2026-01-06 22:01 ` Alexei Starovoitov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox