Linux Perf Users
 help / color / mirror / Atom feed
From: sashiko-bot@kernel.org
To: "Ian Rogers" <irogers@google.com>
Cc: bpf@vger.kernel.org, linux-perf-users@vger.kernel.org
Subject: Re: [PATCH v4 11/14] perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py
Date: Fri, 15 May 2026 18:32:11 +0000	[thread overview]
Message-ID: <20260515183212.36305C2BCB0@smtp.kernel.org> (raw)
In-Reply-To: <20260515173852.1378571-12-irogers@google.com>

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The use of `ProcessPoolExecutor` without a `max_workers` limit violates GNU Make's jobserver concurrency constraints, causing system oversubscription.
--

commit 09eba7d34c7e309d0f6449fe513b92dfbec5e85c
Author: Ian Rogers <irogers@google.com>

perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py

This commit refactors jevents.py to pre-populate its internal JSON AST cache
in parallel across available CPU cores using ProcessPoolExecutor, reducing
build generation time.

> diff --git a/tools/perf/pmu-events/jevents.py b/tools/perf/pmu-events/jevents.py
> --- a/tools/perf/pmu-events/jevents.py
> +++ b/tools/perf/pmu-events/jevents.py
[ ... ]
> @@ -1459,9 +1471,25 @@ struct pmu_table_entry {
[ ... ]
>    for arch in archs:
>      arch_path = f'{_args.starting_dir}/{arch}'
>      preprocess_arch_std_files(arch_path)
> +    ftw(arch_path, [], collect_json)
> +
> +  with concurrent.futures.ProcessPoolExecutor(initializer=_init_worker, initargs=(_arch_std_events,)) as executor:

Does this bypass GNU Make's jobserver concurrency constraints? By
instantiating ProcessPoolExecutor without a max_workers limit, it will
default to spawning workers for all available CPU cores.

If a user explicitly requests a restricted build with make -j1, or if this
runs alongside numerous other compilation jobs, could it lead to extreme
CPU contention and potential out-of-memory failures?

> +    for path, topic, events in executor.map(_parallel_read_json_events, tasks):
> +      _json_cache[(path, topic)] = events
> +
> +  for arch in archs:
> +    arch_path = f'{_args.starting_dir}/{arch}'
>      ftw(arch_path, [], preprocess_one_file)

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260515173852.1378571-1-irogers@google.com?part=11

  reply	other threads:[~2026-05-15 18:32 UTC|newest]

Thread overview: 118+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-05-12  5:35 [PATCH v1 00/14] perf build: Reduce build time by one third Ian Rogers
2026-05-12  5:35 ` [PATCH v1 01/14] bpftool build: Restrict feature tests during bootstrap compilation Ian Rogers
2026-05-12  5:35 ` [PATCH v1 02/14] perf trace beauty: Make beauty generated C code standalone .o files Ian Rogers
2026-05-13  5:21   ` sashiko-bot
2026-05-12  5:35 ` [PATCH v1 03/14] perf build: Decouple pmu-events from prepare umbrella target Ian Rogers
2026-05-12  5:35 ` [PATCH v1 04/14] perf build: Remove empty archheaders target Ian Rogers
2026-05-12  5:35 ` [PATCH v1 05/14] perf build: Move BPF skeleton generation out of Makefile.perf Ian Rogers
2026-05-13 19:52   ` sashiko-bot
2026-05-12  5:35 ` [PATCH v1 06/14] perf build: Encapsulate vmlinux.h and bpftool in bpf_skel.mak Ian Rogers
2026-05-13 20:09   ` sashiko-bot
2026-05-12  5:35 ` [PATCH v1 07/14] perf build: Move static libbpf dependency out of prepare step Ian Rogers
2026-05-13 20:36   ` sashiko-bot
2026-05-12  5:35 ` [PATCH v1 08/14] perf build: Pre-generate BPF skeletons during umbrella prepare phase Ian Rogers
2026-05-12  5:35 ` [PATCH v1 09/14] perf build: Move libsymbol dependency out of prepare step Ian Rogers
2026-05-13 21:11   ` sashiko-bot
2026-05-12  5:35 ` [PATCH v1 10/14] perf build: Remove redundant libbpf feature check for static builds Ian Rogers
2026-05-12  5:35 ` [PATCH v1 11/14] tools build: Integrate libdebuginfod into test-all fast path Ian Rogers
2026-05-13 21:40   ` sashiko-bot
2026-05-12  5:35 ` [PATCH v1 12/14] perf pmu-events: Split big_c_string storage into standalone compilation unit Ian Rogers
2026-05-13 21:56   ` sashiko-bot
2026-05-12  5:35 ` [PATCH v1 13/14] perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py Ian Rogers
2026-05-13 22:18   ` sashiko-bot
2026-05-12  5:35 ` [PATCH v1 14/14] perf build: Prefix SCRIPTS with output directory to fix continuous rebuilds Ian Rogers
2026-05-12 17:46   ` [PATCH v2 00/18] perf build: Reduce build time by nearly half Ian Rogers
2026-05-12 17:46     ` [PATCH v2 01/18] bpftool build: Restrict feature tests during bootstrap compilation Ian Rogers
2026-05-12 17:46     ` [PATCH v2 02/18] tools build: Integrate libdebuginfod into test-all fast path Ian Rogers
2026-05-13 23:59       ` sashiko-bot
2026-05-12 17:46     ` [PATCH v2 03/18] tools build: Fix test-clang-bpf-co-re.bin to generate target file Ian Rogers
2026-05-14  0:15       ` sashiko-bot
2026-05-12 17:46     ` [PATCH v2 04/18] tools scripts: Short-circuit CC_NO_CLANG compiler probe in Makefile.include Ian Rogers
2026-05-14  0:28       ` sashiko-bot
2026-05-12 17:46     ` [PATCH v2 05/18] perf trace beauty: Make beauty generated C code standalone .o files Ian Rogers
2026-05-14  0:50       ` sashiko-bot
2026-05-12 17:46     ` [PATCH v2 06/18] perf build: Decouple pmu-events from prepare umbrella target Ian Rogers
2026-05-12 17:46     ` [PATCH v2 07/18] perf build: Remove empty archheaders target Ian Rogers
2026-05-12 17:46     ` [PATCH v2 08/18] perf build: Move BPF skeleton generation out of Makefile.perf Ian Rogers
2026-05-14  1:55       ` sashiko-bot
2026-05-12 17:46     ` [PATCH v2 09/18] perf build: Encapsulate vmlinux.h and bpftool in bpf_skel.mak Ian Rogers
2026-05-12 17:46     ` [PATCH v2 10/18] perf build: Move static libbpf dependency out of prepare step Ian Rogers
2026-05-14  3:02       ` sashiko-bot
2026-05-12 17:46     ` [PATCH v2 11/18] perf build: Pre-generate BPF skeleton tooling during umbrella prepare phase Ian Rogers
2026-05-14  3:39       ` sashiko-bot
2026-05-12 17:46     ` [PATCH v2 12/18] perf build: Move libsymbol dependency out of prepare step Ian Rogers
2026-05-12 17:46     ` [PATCH v2 13/18] perf build: Remove redundant libbpf feature check for static builds Ian Rogers
2026-05-12 17:46     ` [PATCH v2 14/18] perf pmu-events: Split big_c_string storage into standalone compilation unit Ian Rogers
2026-05-14  4:35       ` sashiko-bot
2026-05-12 17:46     ` [PATCH v2 15/18] perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py Ian Rogers
2026-05-14  5:06       ` sashiko-bot
2026-05-12 17:46     ` [PATCH v2 16/18] perf build: Prefix SCRIPTS with output directory to fix continuous rebuilds Ian Rogers
2026-05-12 17:46     ` [PATCH v2 17/18] perf pmu-events: Convert recursive shell assignments and macros to Make built-ins Ian Rogers
2026-05-12 17:46     ` [PATCH v2 18/18] perf build: Convert llvm-config shell queries to simply expanded variables Ian Rogers
2026-05-14 16:33     ` [PATCH v3 00/17] perf build: Reduce build time by nearly half Ian Rogers
2026-05-14 16:33       ` [PATCH v3 01/17] bpftool build: Restrict feature tests during bootstrap compilation Ian Rogers
2026-05-14 16:33       ` [PATCH v3 02/17] tools build: Integrate libdebuginfod into test-all fast path Ian Rogers
2026-05-14 20:55         ` sashiko-bot
2026-05-14 16:33       ` [PATCH v3 03/17] tools build: Fix test-clang-bpf-co-re.bin to generate target file Ian Rogers
2026-05-14 21:10         ` sashiko-bot
2026-05-14 16:33       ` [PATCH v3 04/17] perf trace beauty: Make beauty generated C code standalone .o files Ian Rogers
2026-05-14 16:33       ` [PATCH v3 05/17] perf build: Decouple pmu-events from prepare umbrella target Ian Rogers
2026-05-14 16:33       ` [PATCH v3 06/17] perf build: Remove empty archheaders target Ian Rogers
2026-05-14 16:33       ` [PATCH v3 07/17] perf build: Move BPF skeleton generation out of Makefile.perf Ian Rogers
2026-05-14 16:34       ` [PATCH v3 08/17] perf build: Encapsulate vmlinux.h and bpftool in bpf_skel.mak Ian Rogers
2026-05-14 16:34       ` [PATCH v3 09/17] perf build: Move static libbpf dependency out of prepare step Ian Rogers
2026-05-14 23:35         ` sashiko-bot
2026-05-14 16:34       ` [PATCH v3 10/17] perf build: Pre-generate BPF skeleton tooling during umbrella prepare phase Ian Rogers
2026-05-14 23:45         ` sashiko-bot
2026-05-14 16:34       ` [PATCH v3 11/17] perf build: Move libsymbol dependency out of prepare step Ian Rogers
2026-05-14 16:34       ` [PATCH v3 12/17] perf build: Remove redundant libbpf feature check for static builds Ian Rogers
2026-05-14 16:34       ` [PATCH v3 13/17] perf pmu-events: Split big_c_string storage into standalone compilation unit Ian Rogers
2026-05-15  0:39         ` sashiko-bot
2026-05-14 16:34       ` [PATCH v3 14/17] perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py Ian Rogers
2026-05-15  1:00         ` sashiko-bot
2026-05-14 16:34       ` [PATCH v3 15/17] perf build: Prefix SCRIPTS with output directory to fix continuous rebuilds Ian Rogers
2026-05-14 16:34       ` [PATCH v3 16/17] perf pmu-events: Convert recursive shell assignments and macros to Make built-ins Ian Rogers
2026-05-14 16:34       ` [PATCH v3 17/17] perf build: Convert llvm-config shell queries to simply expanded variables Ian Rogers
2026-05-14 22:06       ` [PATCH v3 00/17] perf build: Reduce build time by nearly half Namhyung Kim
2026-05-14 22:23         ` Ian Rogers
2026-05-15 16:20           ` Ian Rogers
2026-05-15 17:38             ` [PATCH v4 00/14] " Ian Rogers
2026-05-15 17:38               ` [PATCH v4 01/14] tools build: Fix feature checks to touch target files on success Ian Rogers
2026-05-15 18:04                 ` sashiko-bot
2026-05-15 17:38               ` [PATCH v4 02/14] perf trace beauty: Make beauty generated C code standalone .o files Ian Rogers
2026-05-15 18:45                 ` Namhyung Kim
2026-05-15 17:38               ` [PATCH v4 03/14] perf build: Decouple pmu-events from prepare umbrella target Ian Rogers
2026-05-15 17:38               ` [PATCH v4 04/14] perf build: Remove empty archheaders target Ian Rogers
2026-05-15 17:38               ` [PATCH v4 05/14] perf build: Move BPF skeleton generation out of Makefile.perf Ian Rogers
2026-05-15 17:38               ` [PATCH v4 06/14] perf build: Encapsulate vmlinux.h and bpftool in bpf_skel.mak Ian Rogers
2026-05-15 17:38               ` [PATCH v4 07/14] perf build: Pre-generate BPF skeleton tooling during umbrella prepare phase Ian Rogers
2026-05-15 17:38               ` [PATCH v4 08/14] perf build: Move libsymbol dependency out of prepare step Ian Rogers
2026-05-15 18:30                 ` sashiko-bot
2026-05-15 17:38               ` [PATCH v4 09/14] perf build: Remove redundant libbpf feature check for static builds Ian Rogers
2026-05-15 17:38               ` [PATCH v4 10/14] perf pmu-events: Split big_c_string storage into standalone compilation unit Ian Rogers
2026-05-15 18:09                 ` sashiko-bot
2026-05-15 17:38               ` [PATCH v4 11/14] perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py Ian Rogers
2026-05-15 18:32                 ` sashiko-bot [this message]
2026-05-15 19:41                 ` Namhyung Kim
2026-05-15 17:38               ` [PATCH v4 12/14] perf build: Prefix SCRIPTS with output directory to fix continuous rebuilds Ian Rogers
2026-05-15 19:48                 ` Namhyung Kim
2026-05-15 17:38               ` [PATCH v4 13/14] perf pmu-events: Convert recursive shell assignments and macros to Make built-ins Ian Rogers
2026-05-15 17:38               ` [PATCH v4 14/14] perf build: Convert llvm-config shell queries to simply expanded variables Ian Rogers
2026-05-15 19:33               ` [PATCH v5 00/14] perf build: Reduce build time by nearly half Ian Rogers
2026-05-15 19:33                 ` [PATCH v5 01/14] tools build: Fix feature checks to touch target files on success Ian Rogers
2026-05-15 19:33                 ` [PATCH v5 02/14] perf trace beauty: Make beauty generated C code standalone .o files Ian Rogers
2026-05-15 19:33                 ` [PATCH v5 03/14] perf build: Decouple pmu-events from prepare umbrella target Ian Rogers
2026-05-15 19:33                 ` [PATCH v5 04/14] perf build: Remove empty archheaders target Ian Rogers
2026-05-15 19:33                 ` [PATCH v5 05/14] perf build: Move BPF skeleton generation out of Makefile.perf Ian Rogers
2026-05-15 19:33                 ` [PATCH v5 06/14] perf build: Encapsulate vmlinux.h and bpftool in bpf_skel.mak Ian Rogers
2026-05-15 19:33                 ` [PATCH v5 07/14] perf build: Pre-generate BPF skeleton tooling during umbrella prepare phase Ian Rogers
2026-05-15 19:33                 ` [PATCH v5 08/14] perf build: Move libsymbol dependency out of prepare step Ian Rogers
2026-05-15 19:53                   ` sashiko-bot
2026-05-15 19:33                 ` [PATCH v5 09/14] perf build: Remove redundant libbpf feature check for static builds Ian Rogers
2026-05-15 19:33                 ` [PATCH v5 10/14] perf pmu-events: Split big_c_string storage into standalone compilation unit Ian Rogers
2026-05-15 19:33                 ` [PATCH v5 11/14] perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py Ian Rogers
2026-05-15 19:33                 ` [PATCH v5 12/14] perf build: Prefix SCRIPTS with output directory to fix continuous rebuilds Ian Rogers
2026-05-15 19:33                 ` [PATCH v5 13/14] perf pmu-events: Convert recursive shell assignments and macros to Make built-ins Ian Rogers
2026-05-15 19:33                 ` [PATCH v5 14/14] perf build: Convert llvm-config shell queries to simply expanded variables Ian Rogers
2026-05-15 18:20           ` [PATCH v3 00/17] perf build: Reduce build time by nearly half Namhyung Kim
2026-05-12  9:36 ` [PATCH v1 00/14] perf build: Reduce build time by one third James Clark

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260515183212.36305C2BCB0@smtp.kernel.org \
    --to=sashiko-bot@kernel.org \
    --cc=bpf@vger.kernel.org \
    --cc=irogers@google.com \
    --cc=linux-perf-users@vger.kernel.org \
    --cc=sashiko-reviews@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox