[PATCH dwarves v4 00/10] pahole: faster reproducible BTF encoding

[Ihor Solodrai <ihor.solodrai@pm.me>]

This is v4 of the series aiming to speed up parallel reproducible BTF
encoding. This version mostly addresses feedback from Jiri Olsa on v3.

A notable adition is a patch 10/10, which changes func_states in
btf_encoder from a list to an array.

Testing:

    vmlinux=/home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1 PATH=$(realpath build):$PATH ./tests/tests
      1: Validation of BTF encoding of functions; this may take some time: Ok
      2: Default BTF on a system without BTF: Ok
      3: Flexible arrays accounting: pahole: type 'nft_pipapo_elem' not found
    pahole: type 'tls_rec' not found
    pahole: type 'fuse_direntplus' not found
    pahole: type 'nft_rhash_elem' not found
    pahole: type 'nft_hash_elem' not found
    pahole: type 'nft_bitmap_elem' not found
    pahole: type 'ipt_standard' not found
    pahole: type 'nft_rule_dp_last' not found
    pahole: type 'ip6t_standard' not found
    pahole: type 'ipt_error' not found
    pahole: type 'ip6t_error' not found
    pahole: type 'nft_rbtree_elem' not found
    Ok
      4: Check that pfunct can print btf_decl_tags read from BTF: Ok
      5: Pretty printing of files using DWARF type information: Ok
      6: Parallel reproducible DWARF Loading/Serial BTF encoding: Ok

The warnings about not found types are also present at pahole/next, so
not related to this patchset.


Performance check. This patchset (always reproducible):

     Performance counter stats for '/home/theihor/dev/dwarves/build/pahole -J -j24 --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs,reproducible_build --btf_encode_detached=/dev/null --lang_exclude=rust /home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1' (13 runs):

              5,788.22 msec cpu-clock:u                      #    3.776 CPUs utilized               ( +-  0.17% )

               1.53288 +- 0.00334 seconds time elapsed  ( +-  0.22% )


pahole/next (d444eb6), parallel non-reproducible:

     Performance counter stats for '/home/theihor/dev/dwarves/build/pahole -J -j24 --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs --btf_encode_detached=/dev/null --lang_exclude=rust /home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1' (13 runs):

             10,462.38 msec cpu-clock:u                      #    6.678 CPUs utilized               ( +-  0.15% )

               1.56670 +- 0.00548 seconds time elapsed  ( +-  0.35% )


pahole/next (d444eb6), parallel reproducible:

     Performance counter stats for '/home/theihor/dev/dwarves/build/pahole -J -j24 --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs,reproducible_build --btf_encode_detached=/dev/null --lang_exclude=rust /home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1' (13 runs):

              6,399.88 msec cpu-clock:u                      #    3.164 CPUs utilized               ( +-  0.22% )

               2.02269 +- 0.00359 seconds time elapsed  ( +-  0.18% )


v3: https://lore.kernel.org/dwarves/20241221012245.243845-1-ihor.solodrai@pm.me/
v2: https://lore.kernel.org/dwarves/20241213223641.564002-1-ihor.solodrai@pm.me/
v1: https://lore.kernel.org/dwarves/20241128012341.4081072-1-ihor.solodrai@pm.me/

Alan Maguire (2):
  btf_encoder: simplify function encoding
  btf_encoder: separate elf function, saved function representations

Ihor Solodrai (8):
  btf_encoder: free encoder->secinfo in btf_encoder__delete
  btf_encoder: introduce elf_functions struct type
  btf_encoder: introduce elf_functions_list
  btf_encoder: remove skip_encoding_inconsistent_proto
  dwarf_loader: introduce cu->id
  dwarf_loader: multithreading with a job/worker model
  btf_encoder: clean up global encoders list
  btf_encoder: switch func_states from a list to an array

 btf_encoder.c               | 662 +++++++++++++++++++-----------------
 btf_encoder.h               |   7 +-
 btf_loader.c                |   2 +-
 ctf_loader.c                |   2 +-
 dwarf_loader.c              | 335 ++++++++++++------
 dwarves.c                   |  44 ---
 dwarves.h                   |  21 +-
 pahole.c                    | 230 ++-----------
 pdwtags.c                   |   3 +-
 pfunct.c                    |   3 +-
 tests/reproducible_build.sh |   5 +-
 11 files changed, 624 insertions(+), 690 deletions(-)

-- 
2.47.1