Re: [PATCH v7 00/14] perf build: Reduce build time by nearly half

[Arnaldo Carvalho de Melo <acme@kernel.org>]

On Tue, May 19, 2026 at 11:27:05AM -0700, Namhyung Kim wrote:
> On Mon, May 18, 2026 at 08:46:24AM -0700, Ian Rogers wrote:
> > This patch series refactors Kbuild internals, BPF skeleton generation,
> > Python AST pre-computation, and foundational tooling dependencies across
> > the perf tool build system. By eliminating umbrella target synchronization
> > barriers, decoupling static library prerequisites, parallelizing single-core
> > script generators, and eradicating redundant feature checks, this series
> > unlocks absolute theoretical peak multi-core concurrency during Kbuild startup.
> > 
> > On a 28-core build workstation (make -j28 all from scratch), clean build
> > latency improves by over 44%:
> > 
> >   Before:
> >     real    0m29.006s
> >     user    2m46.019s
> >     sys     0m30.610s
> > 
> >   After:
> >     real    0m16.091s
> >     user    2m40.135s
> >     sys     0m25.740s
> > 
> > Saving 12.9 full seconds time per clean build. Furthermore, nothing to
> > build incremental builds are improved by nearly 7x:
> > 
> >   Before:
> >     real    0m11.528s
> >     user    0m9.633s
> >     sys     0m6.965s
> > 
> >   After:
> >     real    0m1.717s
> >     user    0m1.682s
> >     sys     0m0.960s
> > 
> > Summary of Patches:
> > 
> > 1: Fast-Path Feature Detection
> >   - Refactors test-clang-bpf-co-re.bin and test-bpftool-skeletons.bin feature
> >     checks to group shell pipelines within curly braces and redirect both stdout
> >     and stderr to .make.output before touching $@ purely upon success
> >     (> $(@:.bin=.make.output) 2>&1 && touch $@). Grouping the pipeline ({ cmd1 | cmd2; })
> >     ensures that compiler stderr is successfully captured in .make.output rather
> >     than escaping to the parent shell. This perfectly matches standard Kbuild
> >     feature check conventions and ensures the target files are touched on disk
> >     purely upon success, allowing Kbuild to cache positive detections and avoid
> >     continuous sub-make re-evaluations during incremental builds. Adds
> >     test-bpftool-skeletons.bin to the clean FILES list and explicit source
> >     prerequisite test-clang-bpf-co-re.c.
> 
> I think patch 1 can be separated and needs Ack/Review from BPF folks.
> 
> > 
> > 2-4: Flattening Umbrella Prepare Barriers
> >   - builtin-trace embedded inclusions and pmu-events generation are completely
> >     decoupled from the sequential "prepare" umbrella target, eliminating Make
> >     AST double-parsing overhead and unchoking parallel compilation barriers.
> > 
> > 5-7: Decoupling & Pre-generating BPF Skeletons
> >   - BPF skeleton rules are extracted out of Makefile.perf into bpf_skel.mak.
> >   - Decouples bpftool bootstrap from top-level static libbpf dependencies,
> >     attaching bpf-skel-prepare directly to the umbrella prepare target. This
> >     allows Make to pre-compile bpftool and dump vmlinux.h in the background at
> >     build startup, removing the 7-second serialization bottleneck before BPF
> >     object compilation.
> >   - Ensures benchmark skeleton intermediate .bpf.o files are cleanly removed
> >     during make clean, and adds bpf-skel-prepare to .PHONY.
> > 
> > 8-9: Foundational Linkage Optimization
> >   - Moves static libsymbol library prerequisites out of the prepare step.
> >   - Eliminates redundant libbpf sub-make feature checks during static builds.
> > 
> > 10-11: jevents.py Concurrency & Deduplication
> >   - Splits the massive 2.8 MB big_c_string literal out of pmu-events.c into a
> >     dedicated pmu-events-string.c compilation unit. This slices C compilation
> >     latency in half by compiling string and struct tables simultaneously across
> >     separate CPU cores while preserving zero dynamic ELF relocations. Adds
> >     pmu-events-string.c to .gitignore, declares extern const char big_c_string[];
> >     locally inside output_string_file and output_file when split to prevent linkage
> >     conflicts with empty-pmu-events.c, defers file closures to ensure identical
> >     timestamps, and uses canonical Make 4.0 @: dependency chaining.
> >   - Pre-populates jevents.py JSON ASTs and metric formulas in parallel across
> >     all available CPU cores using ProcessPoolExecutor (accelerating Python
> >     execution by 11x, from 3.3s down to ~290ms). Moves _init_worker to top-level
> >     scope to ensure clean pickling under spawn multiprocessing start methods.
> > 
> > 12: Out-of-Tree Incremental Rebuild Fix
> >   - Prefixes SCRIPTS (perf-archive, perf-iostat) with $(OUTPUT) to prevent
> >     Make from continuously re-executing script installation rules on already
> >     built out-of-tree builds.
> > 
> > 13-14: AST Parsing Optimization & Shell Fork Eradication
> >   - Converts ZENS, ARMS, and INTELS in pmu-events/Build from recursive assignment
> >     (=) to simply expanded assignment (:=) and replaces model_name/vendor_name
> >     with pure GNU Make string functions. This guarantees Make executes directory
> >     probing shell forks exactly once during AST parsing and evaluates path macros
> >     purely in memory, completely eradicating over 7,800 redundant sub-processes
> >     during out-of-tree build evaluation.
> >   - Converts llvm-config shell queries in Makefile.config from recursive assignment
> >     (=) to simply expanded assignment (:=). This eliminates ~185 redundant sub-processes
> >     that were previously executed across object compilation dependency checks.
> > 
> > Changes since v6:
> >   - Rebase/resend as last series failed to apply by Sashiko.
> > 
> > Ian Rogers (14):
> >   tools build: Fix feature checks to touch target files on success
> >   perf trace beauty: Make beauty generated C code standalone .o files
> >   perf build: Decouple pmu-events from prepare umbrella target
> >   perf build: Remove empty archheaders target
> >   perf build: Move BPF skeleton generation out of Makefile.perf
> >   perf build: Encapsulate vmlinux.h and bpftool in bpf_skel.mak
> >   perf build: Pre-generate BPF skeleton tooling during umbrella prepare
> >     phase
> >   perf build: Move libsymbol dependency out of prepare step
> >   perf build: Remove redundant libbpf feature check for static builds
> >   perf pmu-events: Split big_c_string storage into standalone
> >     compilation unit
> >   perf pmu-events: Parallelize JSON and metric pre-computation in
> >     jevents.py
> >   perf build: Prefix SCRIPTS with output directory to fix continuous
> >     rebuilds
> >   perf pmu-events: Convert recursive shell assignments and macros to
> >     Make built-ins
> >   perf build: Convert llvm-config shell queries to simply expanded
> >     variables
> 
> Reviewed-by: Namhyung Kim <namhyung@kernel.org>

So this is for 2-14? I haven't checked if 1 can be left out of an
initial merge by me.

- Arnaldo