From: James Clark <james.clark@linaro.org>
To: Ian Rogers <irogers@google.com>
Cc: Quentin Monnet <qmo@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Andrii Nakryiko <andrii@kernel.org>,
Martin KaFai Lau <martin.lau@linux.dev>,
Eduard Zingerman <eddyz87@gmail.com>,
Kumar Kartikeya Dwivedi <memxor@gmail.com>,
Song Liu <song@kernel.org>,
Yonghong Song <yonghong.song@linux.dev>,
Jiri Olsa <jolsa@kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>,
Arnaldo Carvalho de Melo <acme@kernel.org>,
Namhyung Kim <namhyung@kernel.org>,
Adrian Hunter <adrian.hunter@intel.com>,
Paul Walmsley <pjw@kernel.org>,
Palmer Dabbelt <palmer@dabbelt.com>,
Albert Ou <aou@eecs.berkeley.edu>,
Alexandre Ghiti <alex@ghiti.fr>, Nick Terrell <terrelln@fb.com>,
David Sterba <dsterba@suse.com>,
Nathan Chancellor <nathan@kernel.org>,
Tomas Glozar <tglozar@redhat.com>,
Dmitrii Dolgov <9erthalion6@gmail.com>,
Costa Shulyupin <costa.shul@redhat.com>,
Alexandre Chartre <alexandre.chartre@oracle.com>,
Yuzhuo Jing <yuzhuo@google.com>, Leo Yan <leo.yan@arm.com>,
Ankur Arora <ankur.a.arora@oracle.com>,
Markus Mayer <mmayer@broadcom.com>,
Collin Funk <collin.funk1@gmail.com>,
Howard Chu <howardchu95@gmail.com>,
Dapeng Mi <dapeng1.mi@linux.intel.com>,
Swapnil Sapkal <swapnil.sapkal@amd.com>,
Thomas Falcon <thomas.falcon@intel.com>,
Ricky Ringler <ricky.ringler@proton.me>,
linux-kernel@vger.kernel.org, bpf@vger.kernel.org,
linux-perf-users@vger.kernel.org
Subject: Re: [PATCH v1 00/14] perf build: Reduce build time by one third
Date: Tue, 12 May 2026 10:36:34 +0100 [thread overview]
Message-ID: <3c42ab2a-7698-4459-b1e1-9441a0bf4d9b@linaro.org> (raw)
In-Reply-To: <20260512053539.3410189-1-irogers@google.com>
On 12/05/2026 6:35 am, Ian Rogers wrote:
> This patch series refactors many aspects of the perf build aiming to
> better encapsulate BPF code generation, remove serial build code and
> gain build parallelism. The prepare step that blocks the parallel
> build is reduced to a core 6 smaller dependencies. BPF skeletons are
> made regular dependencies on the targets that use them. Feature tests
> and dependencies are reorgnized. The jevents.py script processes json
> files in parallel and allows the big_c_string to be compiled
> separately.
>
> On a 28-core build workstation (make -j28 all from scratch), clean build
> latency improves by over 36%:
>
> Before:
> real 0m29.006s
> user 2m46.019s
> sys 0m30.610s
>
> After:
> real 0m18.498s
> user 2m32.922s
> sys 0m27.623s
>
I also get similar numbers, even when using ccache:
Before:
real 0m29.584s
user 0m48.993s
sys 0m20.466s
After:
real 0m18.322s
user 0m49.995s
sys 0m17.077s
Tested-by: James Clark <james.clark@linaro.org>
> Summary of Patches:
>
> 1: bpftool Bootstrap Optimization
> - Exempts bpftool bootstrap from non-essential feature tests (LLVM, libbfd,
> libcap), saving 1.1s of sub-make fork overhead during Kbuild startup.
>
> 2-4: Flattening Umbrella Prepare Barriers
> - builtin-trace embedded inclusions and pmu-events generation are completely
> decoupled from the sequential "prepare" umbrella target, eliminating Make
> AST double-parsing overhead and unchoking parallel compilation barriers.
>
> 5-8: Decoupling & Pre-generating BPF Skeletons
> - BPF skeleton rules are extracted out of Makefile.perf into bpf_skel.mak.
> - Decouples bpftool bootstrap from top-level static libbpf dependencies,
> attaching bpf-skel-prepare directly to the umbrella prepare target. This
> allows Make to pre-compile bpftool and dump vmlinux.h in the background at
> build startup, removing the 7-second serialization bottleneck before BPF
> object compilation.
>
> 9-11: Foundational Linkage & Fast-Path Feature Detection
> - Eliminates redundant libbpf sub-make feature checks during static builds.
> - Integrates libdebuginfod directly into test-all.c, allowing Make to skip
> individual feature check sub-make forks during AST parsing on fully
> configured workstations.
>
> 12-13: jevents.py Concurrency & Deduplication
> - Splits the massive 2.8 MB big_c_string literal out of pmu-events.c into a
> dedicated pmu-events-string.c compilation unit. This slices C compilation
> latency in half by compiling string and struct tables simultaneously across
> separate CPU cores while preserving zero dynamic ELF relocations.
> - Pre-populates jevents.py JSON ASTs and metric formulas in parallel across
> all available CPU cores using ProcessPoolExecutor (accelerating Python
> execution by 11x, from 3.3s down to ~290ms).
>
> 14: Out-of-Tree Incremental Rebuild Fix
> - Prefixes SCRIPTS (perf-archive, perf-iostat) with $(OUTPUT) to prevent
> Make from continuously re-executing script installation rules on already
> built out-of-tree builds.
>
> Ian Rogers (14):
> bpftool build: Restrict feature tests during bootstrap compilation
> perf trace beauty: Make beauty generated C code standalone .o files
> perf build: Decouple pmu-events from prepare umbrella target
> perf build: Remove empty archheaders target
> perf build: Move BPF skeleton generation out of Makefile.perf
> perf build: Encapsulate vmlinux.h and bpftool in bpf_skel.mak
> perf build: Move static libbpf dependency out of prepare step
> perf build: Pre-generate BPF skeletons during umbrella prepare phase
> perf build: Move libsymbol dependency out of prepare step
> perf build: Remove redundant libbpf feature check for static builds
> tools build: Integrate libdebuginfod into test-all fast path
> perf pmu-events: Split big_c_string storage into standalone
> compilation unit
> perf pmu-events: Parallelize JSON and metric pre-computation in
> jevents.py
> perf build: Prefix SCRIPTS with output directory to fix continuous
> rebuilds
>
> tools/bpf/bpftool/Makefile | 5 +
> tools/build/Makefile.feature | 6 +-
> tools/build/feature/Makefile | 2 +-
> tools/build/feature/test-all.c | 5 +
> tools/perf/Build | 2 +
> tools/perf/Makefile.config | 6 +-
> tools/perf/Makefile.perf | 427 +-----------------
> tools/perf/bench/Build | 6 +
> .../bpf_skel/bench_uprobe.bpf.c | 0
> tools/perf/bench/uprobe.c | 2 +-
> tools/perf/bpf_skel.mak | 110 +++++
> tools/perf/builtin-trace.c | 30 +-
> tools/perf/pmu-events/Build | 15 +-
> tools/perf/pmu-events/jevents.py | 56 ++-
> tools/perf/trace/beauty/Build | 280 ++++++++++++
> tools/perf/trace/beauty/arch_errno_names.c | 2 +
> tools/perf/trace/beauty/arch_errno_names.sh | 2 +-
> tools/perf/trace/beauty/beauty.h | 60 +++
> tools/perf/trace/beauty/eventfd.c | 6 +-
> tools/perf/trace/beauty/fsconfig.c | 5 +
> tools/perf/trace/beauty/futex_op.c | 6 +-
> tools/perf/trace/beauty/futex_val3.c | 6 +-
> tools/perf/trace/beauty/mmap.c | 24 +-
> tools/perf/trace/beauty/mode_t.c | 6 +-
> tools/perf/trace/beauty/msg_flags.c | 8 +-
> tools/perf/trace/beauty/open_flags.c | 1 +
> tools/perf/trace/beauty/perf_event_open.c | 22 +-
> tools/perf/trace/beauty/pid.c | 5 +-
> tools/perf/trace/beauty/sched_policy.c | 8 +-
> tools/perf/trace/beauty/seccomp.c | 12 +-
> tools/perf/trace/beauty/signum.c | 6 +-
> tools/perf/trace/beauty/socket_type.c | 6 +-
> .../perf/{util => trace/beauty}/syscalltbl.c | 0
> .../perf/{util => trace/beauty}/syscalltbl.h | 0
> tools/perf/trace/beauty/tracepoints/Build | 22 +
> tools/perf/trace/beauty/waitid_options.c | 8 +-
> tools/perf/util/Build | 17 +-
> tools/perf/util/bpf-trace-summary.c | 2 +-
> tools/perf/util/env.c | 4 +-
> tools/perf/util/env.h | 1 +
> 40 files changed, 700 insertions(+), 491 deletions(-)
> rename tools/perf/{util => bench}/bpf_skel/bench_uprobe.bpf.c (100%)
> create mode 100644 tools/perf/bpf_skel.mak
> create mode 100644 tools/perf/trace/beauty/fsconfig.c
> rename tools/perf/{util => trace/beauty}/syscalltbl.c (100%)
> rename tools/perf/{util => trace/beauty}/syscalltbl.h (100%)
>
prev parent reply other threads:[~2026-05-12 9:36 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-05-12 5:35 [PATCH v1 00/14] perf build: Reduce build time by one third Ian Rogers
2026-05-12 5:35 ` [PATCH v1 01/14] bpftool build: Restrict feature tests during bootstrap compilation Ian Rogers
2026-05-12 5:35 ` [PATCH v1 02/14] perf trace beauty: Make beauty generated C code standalone .o files Ian Rogers
2026-05-13 5:21 ` sashiko-bot
2026-05-12 5:35 ` [PATCH v1 03/14] perf build: Decouple pmu-events from prepare umbrella target Ian Rogers
2026-05-12 5:35 ` [PATCH v1 04/14] perf build: Remove empty archheaders target Ian Rogers
2026-05-12 5:35 ` [PATCH v1 05/14] perf build: Move BPF skeleton generation out of Makefile.perf Ian Rogers
2026-05-12 5:35 ` [PATCH v1 06/14] perf build: Encapsulate vmlinux.h and bpftool in bpf_skel.mak Ian Rogers
2026-05-12 5:35 ` [PATCH v1 07/14] perf build: Move static libbpf dependency out of prepare step Ian Rogers
2026-05-12 5:35 ` [PATCH v1 08/14] perf build: Pre-generate BPF skeletons during umbrella prepare phase Ian Rogers
2026-05-12 5:35 ` [PATCH v1 09/14] perf build: Move libsymbol dependency out of prepare step Ian Rogers
2026-05-12 5:35 ` [PATCH v1 10/14] perf build: Remove redundant libbpf feature check for static builds Ian Rogers
2026-05-12 5:35 ` [PATCH v1 11/14] tools build: Integrate libdebuginfod into test-all fast path Ian Rogers
2026-05-12 5:35 ` [PATCH v1 12/14] perf pmu-events: Split big_c_string storage into standalone compilation unit Ian Rogers
2026-05-12 5:35 ` [PATCH v1 13/14] perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py Ian Rogers
2026-05-12 5:35 ` [PATCH v1 14/14] perf build: Prefix SCRIPTS with output directory to fix continuous rebuilds Ian Rogers
2026-05-12 17:46 ` [PATCH v2 00/18] perf build: Reduce build time by nearly half Ian Rogers
2026-05-12 17:46 ` [PATCH v2 01/18] bpftool build: Restrict feature tests during bootstrap compilation Ian Rogers
2026-05-12 17:46 ` [PATCH v2 02/18] tools build: Integrate libdebuginfod into test-all fast path Ian Rogers
2026-05-12 17:46 ` [PATCH v2 03/18] tools build: Fix test-clang-bpf-co-re.bin to generate target file Ian Rogers
2026-05-12 17:46 ` [PATCH v2 04/18] tools scripts: Short-circuit CC_NO_CLANG compiler probe in Makefile.include Ian Rogers
2026-05-12 17:46 ` [PATCH v2 05/18] perf trace beauty: Make beauty generated C code standalone .o files Ian Rogers
2026-05-12 17:46 ` [PATCH v2 06/18] perf build: Decouple pmu-events from prepare umbrella target Ian Rogers
2026-05-12 17:46 ` [PATCH v2 07/18] perf build: Remove empty archheaders target Ian Rogers
2026-05-12 17:46 ` [PATCH v2 08/18] perf build: Move BPF skeleton generation out of Makefile.perf Ian Rogers
2026-05-12 17:46 ` [PATCH v2 09/18] perf build: Encapsulate vmlinux.h and bpftool in bpf_skel.mak Ian Rogers
2026-05-12 17:46 ` [PATCH v2 10/18] perf build: Move static libbpf dependency out of prepare step Ian Rogers
2026-05-12 17:46 ` [PATCH v2 11/18] perf build: Pre-generate BPF skeleton tooling during umbrella prepare phase Ian Rogers
2026-05-12 17:46 ` [PATCH v2 12/18] perf build: Move libsymbol dependency out of prepare step Ian Rogers
2026-05-12 17:46 ` [PATCH v2 13/18] perf build: Remove redundant libbpf feature check for static builds Ian Rogers
2026-05-12 17:46 ` [PATCH v2 14/18] perf pmu-events: Split big_c_string storage into standalone compilation unit Ian Rogers
2026-05-12 17:46 ` [PATCH v2 15/18] perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py Ian Rogers
2026-05-12 17:46 ` [PATCH v2 16/18] perf build: Prefix SCRIPTS with output directory to fix continuous rebuilds Ian Rogers
2026-05-12 17:46 ` [PATCH v2 17/18] perf pmu-events: Convert recursive shell assignments and macros to Make built-ins Ian Rogers
2026-05-12 17:46 ` [PATCH v2 18/18] perf build: Convert llvm-config shell queries to simply expanded variables Ian Rogers
2026-05-12 9:36 ` James Clark [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=3c42ab2a-7698-4459-b1e1-9441a0bf4d9b@linaro.org \
--to=james.clark@linaro.org \
--cc=9erthalion6@gmail.com \
--cc=acme@kernel.org \
--cc=adrian.hunter@intel.com \
--cc=alex@ghiti.fr \
--cc=alexandre.chartre@oracle.com \
--cc=andrii@kernel.org \
--cc=ankur.a.arora@oracle.com \
--cc=aou@eecs.berkeley.edu \
--cc=bpf@vger.kernel.org \
--cc=collin.funk1@gmail.com \
--cc=costa.shul@redhat.com \
--cc=daniel@iogearbox.net \
--cc=dapeng1.mi@linux.intel.com \
--cc=dsterba@suse.com \
--cc=eddyz87@gmail.com \
--cc=howardchu95@gmail.com \
--cc=irogers@google.com \
--cc=jolsa@kernel.org \
--cc=leo.yan@arm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-perf-users@vger.kernel.org \
--cc=martin.lau@linux.dev \
--cc=memxor@gmail.com \
--cc=mingo@redhat.com \
--cc=mmayer@broadcom.com \
--cc=namhyung@kernel.org \
--cc=nathan@kernel.org \
--cc=palmer@dabbelt.com \
--cc=peterz@infradead.org \
--cc=pjw@kernel.org \
--cc=qmo@kernel.org \
--cc=ricky.ringler@proton.me \
--cc=song@kernel.org \
--cc=swapnil.sapkal@amd.com \
--cc=terrelln@fb.com \
--cc=tglozar@redhat.com \
--cc=thomas.falcon@intel.com \
--cc=yonghong.song@linux.dev \
--cc=yuzhuo@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox