Re: [PATCH bpf-next v2 4/4] resolve_btfids: change in-place update with raw binary output

[Alan Maguire <alan.maguire@oracle.com>]

On 01/12/2025 22:16, Andrii Nakryiko wrote:
> On Thu, Nov 27, 2025 at 10:53 AM Ihor Solodrai <ihor.solodrai@linux.dev> wrote:
>>
>> Currently resolve_btfids updates .BTF_ids section of an ELF file
>> in-place, based on the contents of provided BTF, usually within the
>> same input file, and optionally a BTF base.
>>
>> This patch changes resolve_btfids behavior to enable BTF
>> transformations as part of its main operation. To achieve this
>> in-place ELF write in resolve_btfids is replaced with generation of
>> the following binaries:
>>   * ${1}.btf with .BTF section data
>>   * ${1}.distilled_base.btf with .BTF.base section data (for
>>     out-of-tree modules)
>>   * ${1}.btf_ids with .BTF_ids section data, if it exists in ${1}
>>
>> The execution of resolve_btfids and consumption of its output is
>> orchestrated by scripts/gen-btf.sh introduced in this patch.
>>
>> The rationale for this approach is that updating ELF in-place with
>> libelf API is complicated and bug-prone, especially in the context of
>> the kernel build. On the other hand applying objcopy to manipulate ELF
>> sections is simpler and more reliable.
>>
>> There are two distinct paths for BTF generation and resolve_btfids
>> application in the kernel build: for vmlinux and for kernel modules.
>>
>> For the vmlinux binary a .BTF section is added in a roundabout way to
>> ensure correct linking (details below). The patch doesn't change this
>> approach, only the implementation is a little different.
>>
>> Before this patch it worked like follows:
>>
>>   * pahole consumed .tmp_vmlinux1 [1] and added .BTF section with
>>     llvm-objcopy [2] to it
>>   * then everything except the .BTF section was stripped from .tmp_vmlinux1
>>     into a .tmp_vmlinux1.bpf.o object [1], later linked into vmlinux
>>   * resolve_btfids was executed later on vmlinux.unstripped [3],
>>     updating it in-place
>>
>> After this patch gen-btf.sh implements the following:
>>
>>   * pahole consumes .tmp_vmlinux1 and produces a *detached* file with
>>     raw BTF data
>>   * resolve_btfids consumes .tmp_vmlinux1 and detached BTF to produce
>>     (potentially modified) .BTF, and .BTF_ids sections data
>>   * a .tmp_vmlinux1.bpf.o object is then produced with objcopy copying
>>     BTF output of resolve_btfids
>>   * .BTF_ids data gets embedded into vmlinux.unstripped in
>>     link-vmlinux.sh by objcopy --update-section
>>
>> For the kernel modules creating special .bpf.o file is not necessary,
>> and so embedding of sections data produced by resolve_btfids is
>> straightforward with the objcopy.
>>
>> With this patch an ELF file becomes effectively read-only within
>> resolve_btfids, which allows to delete elf_update() call and satelite
>> code (like compressed_section_fix [4]).
>>
>> Endianness handling of .BTF_ids data is also changed. Previously the
>> "flags" part of the section was bswapped in sets_patch() [5], and then
>> Elf_Type was modified before elf_update() to signal to libelf that
>> bswap may be necessary. With this patch we explicitly bswap entire
>> data buffer on load and on dump.
>>
>> [1] https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git/tree/scripts/link-vmlinux.sh#n115
>> [2] https://git.kernel.org/pub/scm/devel/pahole/pahole.git/tree/btf_encoder.c#n1835
>> [3] https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git/tree/scripts/link-vmlinux.sh#n285
>> [4] https://lore.kernel.org/bpf/20200819092342.259004-1-jolsa@kernel.org/
>> [5] https://lore.kernel.org/bpf/cover.1707223196.git.vmalik@redhat.com/
>>
>> Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
>> ---
>>  MAINTAINERS                          |   1 +
>>  scripts/Makefile.modfinal            |   5 +-
>>  scripts/gen-btf.sh                   | 167 ++++++++++++++++++++
>>  scripts/link-vmlinux.sh              |  42 +-----
>>  tools/bpf/resolve_btfids/main.c      | 218 +++++++++++++++++----------
>>  tools/testing/selftests/bpf/Makefile |   5 +
>>  6 files changed, 317 insertions(+), 121 deletions(-)
>>  create mode 100755 scripts/gen-btf.sh
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 48aabeeed029..5cd34419d952 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -4672,6 +4672,7 @@ F:        net/sched/act_bpf.c
>>  F:     net/sched/cls_bpf.c
>>  F:     samples/bpf/
>>  F:     scripts/bpf_doc.py
>> +F:     scripts/gen-btf.sh
>>  F:     scripts/Makefile.btf
>>  F:     scripts/pahole-version.sh
>>  F:     tools/bpf/
>> diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
>> index 542ba462ed3e..3862fdfa1267 100644
>> --- a/scripts/Makefile.modfinal
>> +++ b/scripts/Makefile.modfinal
>> @@ -38,9 +38,8 @@ quiet_cmd_btf_ko = BTF [M] $@
>>        cmd_btf_ko =                                                     \
>>         if [ ! -f $(objtree)/vmlinux ]; then                            \
>>                 printf "Skipping BTF generation for %s due to unavailability of vmlinux\n" $@ 1>&2; \
>> -       else                                                            \
>> -               LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J $(PAHOLE_FLAGS) $(MODULE_PAHOLE_FLAGS) --btf_base $(objtree)/vmlinux $@; \
>> -               $(RESOLVE_BTFIDS) -b $(objtree)/vmlinux $@;             \
>> +       else    \
>> +               $(srctree)/scripts/gen-btf.sh --btf_base $(objtree)/vmlinux $@; \
>>         fi;
>>
> 
> [...]
> 
>> +if ! is_enabled CONFIG_DEBUG_INFO_BTF; then
>> +       exit 0
>> +fi
>> +
>> +gen_btf_data()
>> +{
>> +       info BTF "${ELF_FILE}"
>> +       btf1="${ELF_FILE}.btf.1"
>> +       ${PAHOLE} -J ${PAHOLE_FLAGS}                    \
>> +               ${BTF_BASE:+--btf_base ${BTF_BASE}}     \
>> +               --btf_encode_detached=${btf1}           \
> 
> please double-check what pahole version has --btf_encode_detached, we
> might need to change minimal supported pahole version because of this
>

yeah, this landed in v1.22 [1]

One thing worth thinking about; are there aspects of the gen_btf.sh
script that could be moved to Makefile.btf to avoid having to compute them
repeatedly for each module? For example computing resolve_btfids 
flags based on CONFIG_WERROR could be done there I think. You could
also determine whether the script is needed at all in Makefile.btf; i.e.

gen-btf-y				=
gen-btf-$(CONFIG_DEBUG_INFO_BTF)	= scripts/gen-btf.sh

export GEN_BTF := $(gen-btf-y)

That would allow you to get rid of the is_enabled() I think.

I'm building this now, but I was wondering if the linking/objcopy changes pose
any risk to kernel address computations in kallsyms or anything like that? IIRC
Stephen ran into some issues with global variable addresses as a consequence of
linking BTF sections [2], but not sure if there are additional concerns here.

[1] https://github.com/acmel/dwarves/releases/tag/v1.22
[2] https://lore.kernel.org/bpf/20250207012045.2129841-2-stephen.s.brennan@oracle.com/