From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dl1-f70.google.com (mail-dl1-f70.google.com [74.125.82.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B3A1233944 for ; Wed, 23 Sep 2026 18:12:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.70 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790187158; cv=none; b=gHcnxASFeDMDIG9z3wbo0PzzceDu/m2nQEncZJ/CdnSI0oxJPYBmQiOyJkDZt3MnM7B3xVHD0+J8WgFBlu20nlFCPKElKw1ac1Ya6zbcgjIA6bZDy8UjwrQfpWCf8v1SHy2Q1d00UD7itbuEznvWg6tukSd9zv1W/M2zk7kcNGA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790187158; c=relaxed/simple; bh=F++0gZ4CGBsW3dISIfVcqKeKBBNMwGurLHRaRLuMiXQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=aq4WsGJABbKw8CKxpH+ZGJfeinN6vWjnU6tFJmjAjnn3Y9ZEkGU8N7OBbePPgYJSM0jF9SQG4m+eptXZU54tGYgfZZgPhBHbeJUlt0rFMmO5wM4zGxjHkz6+Ao6KRskIeDfNKnTGE5IyjxHq+ODue14YBw901hOKxoU/bX2W0fU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=sie0Fnoc; arc=none smtp.client-ip=74.125.82.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="sie0Fnoc" Received: by mail-dl1-f70.google.com with SMTP id a92af1059eb24-1438fef603cso2362699c88.0 for ; Wed, 23 Sep 2026 11:12:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1790187154; x=1790791954; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=bEI7Eum47JZ1rVrjRmdaIUAF1BiiH65WcxpIXKkjfIQ=; b=sie0FnocFBFQBmUA2hRqT3BCCfILw3jqawmFdQMze4CFRxexfZXe/b7nX2ivneBuqz w26UockhBkXJSSnPeXuh5S3+KnHFBW5xLb2STuGyfc6qvcIwzpwo1YzD2E3OYRDGX2O1 M6dv87cu1sOMXUSMfudvxab4dzhWqDeBWqkXErkWlf/NaVLBX/D4xLxVFB7uTyIEZ9fi 3FCi9DIjzvIDOgxgK1LXjSER4TsSlq4v9dIUDq5RoBerMnMr9hW71ofzx87dWTC87w2W J2RnurkNYdc1AN79aywo2Lki716QBOkHNtaDit3MCIDkPUyjKPimqg7bdPuV/1edxMz4 2g7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790187154; x=1790791954; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=bEI7Eum47JZ1rVrjRmdaIUAF1BiiH65WcxpIXKkjfIQ=; b=ft9U7sCBlNB4bjGJWIfWSzvy7ac19fX1ba7Xa4yvNe481HxQgbtppLefZY3vhuQai2 4wofIQkF7h3U1xwshYBVgAkRcVPJYBTJTWQ3PmZFdkjRDZqoO/x579Qun/Vd7hi4y9yv ycbHTsLbe1+6tVpG5xcNcYEi0ImDm4f0tguXzyXNAqlXvu5kxj795kyELk+dZfLdw/9+ 2TSK9RMt5PuBnTPu4kgNEI7i0IKQaw1OaXLMGiDU1g4bpZNyUT14cNLdDrSTMbQhp5FE w5v4C/5qKfhM8QRj0aGUCegAsddOEUqz1TGZUpcbaWMz/t/I5dx6IPpdFtuwjXhFR7tp hZHQ== X-Forwarded-Encrypted: i=1; AKwUvBy26zfttlPMyO78jKvrhiAhyTkSQPMUlZDMVwygaJ84lf0pUT0Nr4HCYRqrCN17gVjBkEuqDMLCiFT8wtkNtz+V@vger.kernel.org X-Gm-Message-State: AFuF++nvi9eXSF5ZXME6fya6EZjzvWnHflMUr6n1sTEOZxxb5+A46Idy ypTmlsHnFWkCeMWB6QeCgGSh6sbzhdygKy24ZJP3VED5YQSapTtZLqbFolLVC3BAd2w06J3gH72 V1/zP3EghBA== X-Received: from dlbpt9.prod.google.com ([2002:a05:7022:e809:b0:144:bfb6:b5b2]) (user=irogers job=prod-delivery.src-stubby-dispatcher) by 2002:a05:701b:2305:b0:143:3617:c5ae with SMTP id a92af1059eb24-144f90d92b3mr3010379c88.2.1790187153670; Wed, 23 Sep 2026 11:12:33 -0700 (PDT) Date: Wed, 23 Sep 2026 11:11:23 -0700 In-Reply-To: Precedence: bulk X-Mailing-List: linux-perf-users@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.56.0.rc1.310.g51773c2048-goog Message-ID: <20260923181213.3032038-1-irogers@google.com> Subject: [PATCH v3 00/49] perf: Complete transition to standalone Python scripts From: Ian Rogers To: irogers@google.com, acme@kernel.org, alice.mei.rogers@gmail.com, namhyung@kernel.org Cc: adrian.hunter@intel.com, dapeng1.mi@linux.intel.com, james.clark@linaro.org, leo.yan@linux.dev, linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, tmricht@linux.ibm.com Content-Type: text/plain; charset="UTF-8" The perf script command has historically supported running Python and Perl scripts by embedding libpython and libperl. That approach has several drawbacks: - high overhead from constructing Python dictionaries for every event (whether used by the script or not), - complex build dependencies on specific Python/Perl libraries, - threading and interpreter lifecycle complications from embedding the interpreter inside perf, - no unified model for standalone scripts such as ilist.py, treport.py, tracepoint.py, and twatch.py. Parts of this series were previously posted as part of the larger "perf: Reorganize scripting support" series: https://lore.kernel.org/linux-perf-users/20260428071903.1886173-1-irogers@google.com/ with earlier RFC and mailing list discussion at: https://lore.kernel.org/linux-perf-users/20251029053413.355154-1-irogers@google.com/ https://lore.kernel.org/lkml/CAP-5=fWDqE8SYfOLZkg_0=4Ayx6E7O+h7uUp4NDeCFkiN4b7-w@mail.gmail.com/ >From that earlier series, the prerequisite patches (header dependency cleanups, evlist/evsel reference counting, initial perf.data and perf.session C extension wrappers, callchain/brstack/stat/syscall/config accessors, perf.pyi type stubs, LiveSession, and removal of embedded libperl support) have now been merged. We restarted the version number at v1 for the remaining patches, along with additional C extension capabilities, build-time static analysis (mypy and pylint), and review fixes. Running scripts as standalone Python applications using the perf Python C extension module provides a dramatic speedup. For example, with mem-phys-addr.py: Before (using embedded libpython in perf script): ``` $ perf mem record -a sleep 1 $ time perf script tools/perf/scripts/python/mem-phys-addr.py Event: cpu_core/mem-loads-aux/ Memory type count percentage --------------------------------------- ---------- ---------- 0-fff : Reserved 3217 100.0 real 0m3.754s user 0m0.023s sys 0m0.018s ``` After (using standalone Python script with perf module): ``` $ PYTHONPATH=/tmp/perf/python time python3 tools/perf/python/mem-phys-addr.py Event: evsel(cpu_core/mem-loads-aux/) Memory type count percentage --------------------------------------- ---------- ---------- 0-fff : Reserved 3217 100.0 real 0m0.106s user 0m0.021s sys 0m0.020s ``` The 49 patches in this series are organized into four groups: 1. Perf Python C Extension & Existing Script Updates (Patches 01-10): - Refine syscall_name()/syscall_id() argument parsing, expose perf.arch_strerrno() from the auto-generated trace beauty errno tables, and update callchain stubs and session.find_thread() lookup in tools/perf/util/python.c and tools/perf/python/perf.pyi. - Clean up pylint warnings and improve `perf script -l` descriptions in existing standalone scripts (ilist.py, treport.py, tracepoint.py, twatch.py). - Expose destination address location attributes (addr_dso, addr_symbol, addr_sym_offset), branch/transaction fields, sym_offset, context_switch callbacks, and Intel PT call_return / itrace synthesis directly in the perf Python extension (many parts previously posted in https://lore.kernel.org/linux-perf-users/20260428071903.1886173-1-irogers@google.com/). - Allow KeyboardInterrupt to propagate cleanly in LiveSession. 2. Build-Time Python Static Analysis (Patches 11-14): - Clean up mypy and pylint warnings in tools/perf/pmu-events/ and tools/perf/tests/shell/lib/. - Make MYPY and PYLINT build checks enabled by default when mypy and pylint (>= 2.16.0) are installed, with NO_MYPY=1 and NO_PYLINT=1 opt-out controls (matching NO_SHELLCHECK=1), so all Python scripts in tools/perf are continuously type-checked and linted at build time. 3. Porting Legacy Python & Perl Scripts to tools/perf/python/ (Patches 15-45): - Previously sent as part of https://lore.kernel.org/linux-perf-users/20260428071903.1886173-1-irogers@google.com/ and updated with static type annotations, mypy/pylint cleanups, and accompanying shell tests under tools/perf/tests/shell/. Sashiko-driven review and improvement was undertaken while porting so that the ported scripts are in the best shape possible: * Temporarily install tools/perf/python/ scripts during the transition (Patch 15). * General & profiling scripts: stat-cpi.py, mem-phys-addr.py, stackcollapse.py, flamegraph.py, gecko.py, event_analyzing_sample.py (Patches 16-21). * Syscall & I/O scripts (ported from legacy Python and Perl): syscall-counts.py, syscall-counts-by-pid.py, failed-syscalls-by-pid.py, failed-syscalls.py, sctop.py, rw-by-file.py, rw-by-pid.py, rwtop.py (Patches 22-29). * Scheduling, locking, MM & networking scripts: futex-contention.py, task-analyzer.py, sched-migration.py (with SchedGui.py), wakeup-latency.py, compaction-times.py, net_dropmonitor.py, netdev-times.py, check-perf-trace.py (Patches 30-37). * Hardware trace & architecture-specific scripts: arm-cs-trace-disasm.py, powerpc-hcalls.py, intel-pt-events.py (with libxed.py), and migrating the Intel PT virtual LBR test to the Python API (Patches 38-41). * Database export & viewer utilities: export-to-sqlite.py, export-to-postgresql.py, exported-sql-viewer.py, and parallel-perf.py (Patches 42-45). 4. Removing Embedded libpython & Updating Build, CLI, and Docs (Patches 46-49): - Previously sent as part of https://lore.kernel.org/linux-perf-users/20260428071903.1886173-1-irogers@google.com/ : * Remove embedded libpython support (scripting-engines/trace-event-python.c) and delete the legacy tools/perf/scripts/python/ directory (Patch 46). * Replace the libpython feature check with test-python-module.c and update Python script installation paths (Patch 47). * Update `perf script` (builtin-script.c) to execute standalone scripts directly and remove embedded scripting engine support (Patch 48). * Update perf man pages (perf-script.txt, perf-script-python.txt, perf-check.txt) for standalone Python scripts (Patch 49). Changes in v3: - Expose `perf_env__arch_strerrno()` as `perf.arch_strerrno()` in `tools/perf/util/python.c` (and tighten `arch_errno_names.sh` macro matching to `E[A-Z0-9]+` with `EM_SPARC32PLUS` support) so `failed-syscalls-by-pid.py` uses the auto-generated C beauty errno tables instead of maintaining Python errno dictionaries (Patches 01, 24). - Change `static char * const kwlist[]` to `static char *kwlist[]` in `pyrf_session__new()` in `tools/perf/util/python.c` to support Python < 3.13 where `PyArg_ParseTupleAndKeywords()` expects `char **` rather than `char *const *`, and register `psession->tool.event_update` so pipe-mode `PERF_RECORD_EVENT_UPDATE` metadata is processed (Patches 08, 09). - Explicitly annotate `d_hcoal: Union[Event, int]` and `d_mcoal: Union[Event, int]` in `AmdDtlb()` in `tools/perf/pmu-events/amd_metrics.py` so older `mypy` versions do not infer `join(Event, int) = object` (Patch 11). - Pass `--ignore-missing-imports` to `MYPY` and `--ignored-modules=textual,setproctitle` to `PYLINT` in `tools/perf/Makefile.perf` so default static analysis builds succeed on systems without optional third-party Python packages (`python3-textual`, `python3-setproctitle`) installed (Patches 13, 14). - Chain `$(INSTALL)` commands with `&&` in `Makefile.perf` for the `python-scripts` installation rules and invoke `"perf script "` without `-s` from `script_browse()` in `ui/browsers/scripts.c` (Patches 15, 47). - Add Subresource Integrity (`integrity="sha256-..."`) attributes to the CDN `` and `