From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f71.google.com (mail-pj1-f71.google.com [209.85.216.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 91A01439F7C for ; Wed, 22 Jul 2026 17:31:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.71 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784741519; cv=none; b=a062JuQTB5noFULi8ZK+biUlJR8VB4PQ76+Q0vKOnkKolqtHWHfycI6y9AtsPY7b0WJdBj3Z5RAexarpCbC3Srv7m2pO8TTmcHSxQk5pMyDI+ZUCeSzQFdZ9StDJYhBcCgNEm9I9Lf3UMbdUF6vhmFlmUOy7Huiyuqug2xJdH3w= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784741519; c=relaxed/simple; bh=RmiNgHhl4/gTd/JKzR/7AM5zeWlxwdw3HFZKgMxMj+A=; h=Date:Mime-Version:Message-ID:Subject:From:To:Content-Type; b=m0dqXlAH4btdC9Kg/6YzKEO4yC8LYKG9AymYyWN7ArpuVaq7hkDoSa3mmJc3BLy1vaLInX1wrZt2v+sT89OfJn7zYktE71l3TTDu3l6rjhWB5iZp+ZdDQN3bV89wH7qKVLk/h75Uw1BnCMa5D13OdxBe9fVy/7NKi9tN7IfjyNI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=b0yuQnE1; arc=none smtp.client-ip=209.85.216.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="b0yuQnE1" Received: by mail-pj1-f71.google.com with SMTP id 98e67ed59e1d1-385d2703b64so6268574a91.1 for ; Wed, 22 Jul 2026 10:31:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1784741510; x=1785346310; darn=vger.kernel.org; h=content-type:to:from:subject:message-id:mime-version:date:from:to :cc:subject:date:message-id:reply-to:content-type; bh=l18VXGNYcOmQG/0BfithWocT6bT2K2vgr8dII6Ia0aE=; b=b0yuQnE1J0M/OB4xI7Y6EEKVf1KXTFUFHM9DL8MtwH6opeYqiNGyErGsOzH4Nmo52q yEwY6Tn4Smkn4RTA/XKi7zbIl1qMMKl3BYsTvzVB3Bi6PwOj4zUVBS0jHV3hcFSeEFC0 bUoNxEAEVxWE3QEErHv9vVskJNs6q02CgScwn1YlNZE9275ChAtepTCZ7OqAIRIg9nQ7 SkrmdwRDH6ni/qf7qnO/aF1YJJZOQb9UdvQcXQ08LJ/OmUWKIMWuniMZo7dFgi2g/o+x onh1nVbPE86lEcx4fU1VAYIYTZPucTBLpEJFs2rtqAS5twIuT7BZfgF8mYk4lJTBA11h 92Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784741510; x=1785346310; h=content-type:to:from:subject:message-id:mime-version:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=l18VXGNYcOmQG/0BfithWocT6bT2K2vgr8dII6Ia0aE=; b=QhBVgq9ILDMTrUpsf/8Sm622+ezCnQacP/bQl/qBEqPXJMdIVp7CBFHlaA23wUtnhx BNgWnfunB6gXER3TAtJZmKEea4rai6r7Xysn8IasYXGfYAA/vXCd0KGebDbo4BXgV4pR NQ24W0LU2GCDT60R1EJ2w3+N7V1fLRKs3eGjqyPZ0UqKulo5sBG+i9V67WT+Wk72PeSj 8leC2PHBAl8FFFrLhqmQ+w88jLspaacx/gn9teKoPtqkJrgn5Y9WOqBVscyQirUIaoxL wNHwD/BHKmtJEWk8UUrchaEzajb6Buu76xXZPUeMdvW1PZl3FjhDnUqdufjb7NuOGNir eG9A== X-Forwarded-Encrypted: i=1; AHgh+Rp73mWQPyUnKGdWU2WAw/xtDXlWGz+K3HgeghMr3IHr2TSn33PXWHlUapUqniOI9vq+1ow1jePqXz/uO2M=@vger.kernel.org X-Gm-Message-State: AOJu0Yy/l6I1d93hJSnSjR02AExTrdWxocnM/sRKZlFW0NMRvatpjgCz uur0RfuGXf/8CMei7YQwzHIj5FkxFrYKSoP46ot0JTZpvBoBXaR77xiXDueOjTrs6ts9Y2wsexm wQLVNrVLDxQ== X-Received: from dlbtu15.prod.google.com ([2002:a05:7022:3c0f:b0:13d:1bd0:e6cd]) (user=irogers job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90a:d445:b0:38e:7f52:c8a9 with SMTP id 98e67ed59e1d1-38ea379633dmr5048244a91.15.1784741509451; Wed, 22 Jul 2026 10:31:49 -0700 (PDT) Date: Wed, 22 Jul 2026 10:31:38 -0700 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.55.0.229.g6434b31f56-goog Message-ID: <20260722173138.720586-1-irogers@google.com> Subject: [PATCH v1] perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py From: Ian Rogers To: Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Namhyung Kim , Jiri Olsa , Ian Rogers , Adrian Hunter , James Clark , Dapeng Mi , linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Currently, jevents.py parses hundreds of JSON event and metric files sequentially across all CPU architectures during Kbuild startup, taking ~3.5 seconds of single-core execution time. Refactor jevents.py to pre-populate its internal JSON AST cache in parallel across all available CPU cores using ProcessPoolExecutor. First gather all the paths with ftw and collect_json, then spawn _parallel_read_json_events that starts workers to just read the json events. Define the worker process initializer _init_worker so that _arch_std_events is available under spawn multiprocessing semantics. This accelerates the JSON parsing phase by over 10x (from ~3.0s down to ~290ms), reducing overall jevents.py execution time by 3.5x (from ~3.56s down to ~1.03s). Tested-by: James Clark Assisted-by: Gemini:gemini-3.1-pro-preview Signed-off-by: Ian Rogers --- This change was originally part of a larger set of build speed ups, but was taken out to focus on more important fixes. Mailing now separately. The original series at v7 with this change is: https://lore.kernel.org/lkml/20260518154638.2798789-1-irogers@google.com/ Note, the change assumes Python 3.9 for type annotations. The parallel execution APIs were added in Python 3.7. This means the python code should meet the minimum build requirements in: https://www.kernel.org/doc/Documentation/Changes --- tools/perf/pmu-events/jevents.py | 36 ++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/tools/perf/pmu-events/jevents.py b/tools/perf/pmu-events/jevents.py index 5c15485d695a..860027bb71b2 100755 --- a/tools/perf/pmu-events/jevents.py +++ b/tools/perf/pmu-events/jevents.py @@ -464,8 +464,8 @@ class JsonEvent: return f'{make_comment(s)}\t{{ { _bcs.offsets[s] } }},\n' -@lru_cache(maxsize=None) -def read_json_events(path: str, topic: str) -> Sequence[JsonEvent]: +_json_cache = {} +def _read_json_events_impl(path: str, topic: str) -> Sequence[JsonEvent]: """Read json events from the specified file.""" try: events = json.load(open(path), object_hook=JsonEvent) @@ -481,12 +481,16 @@ def read_json_events(path: str, topic: str) -> Sequence[JsonEvent]: if updates: for event in events: if event.metric_name in updates: - # print(f'Updated {event.metric_name} from\n"{event.metric_expr}"\n' - # f'to\n"{updates[event.metric_name]}"') event.metric_expr = updates[event.metric_name] return events +def read_json_events(path: str, topic: str) -> Sequence[JsonEvent]: + key = (path, topic) + if key not in _json_cache: + _json_cache[key] = _read_json_events_impl(path, topic) + return _json_cache[key] + def preprocess_arch_std_files(archpath: str) -> None: """Read in all architecture standard events.""" global _arch_std_events @@ -1446,6 +1450,14 @@ const char *describe_metricgroup(const char *group) } """) +def _parallel_read_json_events(task: Tuple[str, str]) -> Tuple[str, str, Sequence[JsonEvent]]: + path, topic = task + return path, topic, _read_json_events_impl(path, topic) + +def _init_worker(std_events: dict) -> None: + global _arch_std_events + _arch_std_events = std_events + def main() -> None: global _args @@ -1524,9 +1536,25 @@ struct pmu_table_entry { raise IOError(f'Missing architecture directory \'{_args.arch}\'') archs.sort() + import concurrent.futures + tasks = [] + def collect_json(parents: Sequence[str], item: os.DirEntry) -> None: + if len(parents) == 0: + return + if item.is_file() and item.name.endswith('.json') and not item.name.endswith('metricgroups.json'): + tasks.append((item.path, get_topic(item.name))) + for arch in archs: arch_path = f'{_args.starting_dir}/{arch}' preprocess_arch_std_files(arch_path) + ftw(arch_path, [], collect_json) + + with concurrent.futures.ProcessPoolExecutor(initializer=_init_worker, initargs=(_arch_std_events,)) as executor: + for path, topic, events in executor.map(_parallel_read_json_events, tasks): + _json_cache[(path, topic)] = events + + for arch in archs: + arch_path = f'{_args.starting_dir}/{arch}' ftw(arch_path, [], preprocess_one_file) assert _bcs is not None -- 2.55.0.229.g6434b31f56-goog