The Linux Kernel Mailing List
 help / color / mirror / Atom feed
* [PATCH v1 00/14] perf test: Harness improvements
@ 2026-05-13 23:04 Ian Rogers
  2026-05-13 23:04 ` [PATCH v1 01/14] perf jevents.py: Make generated C code more kernel style Ian Rogers
                   ` (14 more replies)
  0 siblings, 15 replies; 138+ messages in thread
From: Ian Rogers @ 2026-05-13 23:04 UTC (permalink / raw)
  To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
	Namhyung Kim, Alexander Shishkin, Jiri Olsa, Adrian Hunter,
	James Clark, linux-kernel, linux-perf-users
  Cc: Ian Rogers

Motivation & Key Enhancements

1. **Test Harness Acceleration & Parallel Polling**

   Previously, when running tests in parallel mode (`perf test -v`),
   child processes writing massive amounts of logging output to pipes
   (such as Granite Rapids PMU metric parsing) would saturate the 64KB
   pipe buffer and block indefinitely. The parent harness only polled
   the pipe of the "current" sequential test waiting to be printed,
   causing severe execution bottlenecks.

   - Refactored the parallel poll loop to drain output pipes from all
     active children simultaneously into dynamic per-child buffers
     (`struct strbuf`). Reaping occurs asynchronously out of order,
     while final console printing remains strictly sequential.

   - Added explicit pipe draining after child process termination to
     prevent losing trailing log data.

   - **Benchmark**: This drops parallel verbose execution time for the
       PMU events suite from ~35 seconds down to ~5.9 seconds (an ~83%
       reduction in latency).

2. **Dynamic Test Suites & Granular PMU Subtests**

   Monolithic test cases (like "Parsing of PMU event table metrics")
   previously evaluated hundreds of tables in a single sequential run,
   making failures difficult to isolate.

   - Added `setup` callbacks and private data pointers (`void *priv`)
     to `struct test_suite` and `struct test_case`, enabling dynamic
     runtime testcase generation.

   - Split the PMU events metric parsing test into individual subtests
     (one pair of real/fake PMU tests per metric table), allowing them
     to execute concurrently and report granular results.

3. **Advanced Triaging & Automated Summary Reporting**

   Triaging failures in highly verbose automated runs previously
   required scrolling through thousands of lines of console output.

   - Introduced a smart, configurable failure snippet processor
     (`--failure-snippet-lines`) that dynamically extracts root-cause
     context lines matching failure keywords (`error`, `fail`, `segv`,
     `abort`) while preserving outline markers.

   - Implemented an automated global execution summary printed at the
     absolute tail of the test run, presenting clear pass/skip/fail
     totals alongside an explicit list of failed test cases for
     effortless cross-referencing.

   - Fixed subtest status column alignment (`: Ok`) for multi-digit
     test indexes.

   - Updated shell script SPDX header parsing to prevent license
     strings from being incorrectly extracted as test descriptions.

4. **JUnit XML Reporting & CI Integration**

   Added a `-j`/`--junit` command-line option to generate standard
   JUnit XML test reports (`test.xml`).

   - Captures individual test suite and subtest execution latency
     alongside XML-escaped failure logs and skip reasons.

   - Guarantees absolute timing precision and immunity to wall-clock
     jumps by measuring durations using
     `clock_gettime(CLOCK_MONOTONIC)` and harvesting `end_time`
     exactly when child processes exit to insulate latencies from
     out-of-order sequential UI printing delays.

   - Added a standalone shell test script to validate generated JUnit
     XML syntax using Python's `ElementTree` parser.

5. **Elimination of External C Compiler Dependencies**

   The Intel PT shell test (`test_intel_pt.sh`) previously compiled
   external C workloads at runtime using `/usr/bin/cc`, which
   frequently breaks in hermetic or minimal CI environments.

   - Created a built-in self-modifying JIT workload (`perf test -w
     jitdump`) and switched the script to use built-in workloads.

   - To guarantee robust multi-architecture compatibility without
     external C compiler dependencies, the JIT workload immediate
     instruction arrays dynamically encode `CHK_BYTE` into opcodes
     across x86, ARM32, ARM64, RISC-V, PowerPC, MIPS, LoongArch, and
     s390x, with clean `#else` fallbacks for unsupported
     architectures.

Ian Rogers (14):
  perf jevents.py: Make generated C code more kernel style
  perf pmu-events: Add API to get metric table name and iterate tables
  perf test: Drain pipe after child finishes to avoid losing output
  perf test: Support dynamic test suites with setup callback and private
    data
  perf test pmu-events: A sub-test per metric table
  perf test: Refactor parallel poll loop to drain all pipes
    simultaneously
  perf test: Show snippet failure output for verbose=1
  perf test: Add summary reporting
  perf test: Fix subtest status alignment for multi-digit indexes
  perf test: Skip shebang and SPDX comments in shell test descriptions
  perf test: Split monolithic 'util' test suite into sub-tests
  perf test: Add -j/--junit option for JUnit XML test reports
  perf test: Add shell test to validate JUnit XML reporting output
  perf test: Remove /usr/bin/cc dependency from Intel PT shell test

 tools/lib/subcmd/run-command.c                |    4 +-
 tools/perf/pmu-events/empty-pmu-events.c      | 8811 +++++++++++------
 tools/perf/pmu-events/jevents.py              |  836 +-
 tools/perf/pmu-events/pmu-events.h            |    4 +
 tools/perf/tests/builtin-test.c               |  587 +-
 tools/perf/tests/pmu-events.c                 |  155 +-
 tools/perf/tests/shell/test_intel_pt.sh       |  169 +-
 .../tests/shell/test_test_junit_output.sh     |   63 +
 tools/perf/tests/tests-scripts.c              |   63 +-
 tools/perf/tests/tests.h                      |    3 +
 tools/perf/tests/util.c                       |   20 +-
 tools/perf/tests/workloads/Build              |    1 +
 tools/perf/tests/workloads/jitdump.c          |  165 +
 tools/perf/util/jitdump.h                     |    3 +-
 14 files changed, 7187 insertions(+), 3697 deletions(-)
 create mode 100755 tools/perf/tests/shell/test_test_junit_output.sh
 create mode 100644 tools/perf/tests/workloads/jitdump.c

-- 
2.54.0.563.g4f69b47b94-goog


^ permalink raw reply	[flat|nested] 138+ messages in thread
end of thread, other threads:[~2026-06-11 15:06 UTC | newest]

Thread overview: 138+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-05-13 23:04 [PATCH v1 00/14] perf test: Harness improvements Ian Rogers
2026-05-13 23:04 ` [PATCH v1 01/14] perf jevents.py: Make generated C code more kernel style Ian Rogers
2026-05-13 23:04 ` [PATCH v1 02/14] perf pmu-events: Add API to get metric table name and iterate tables Ian Rogers
2026-05-13 23:04 ` [PATCH v1 03/14] perf test: Drain pipe after child finishes to avoid losing output Ian Rogers
2026-05-13 23:04 ` [PATCH v1 04/14] perf test: Support dynamic test suites with setup callback and private data Ian Rogers
2026-05-13 23:04 ` [PATCH v1 05/14] perf test pmu-events: A sub-test per metric table Ian Rogers
2026-05-13 23:04 ` [PATCH v1 06/14] perf test: Refactor parallel poll loop to drain all pipes simultaneously Ian Rogers
2026-05-13 23:04 ` [PATCH v1 07/14] perf test: Show snippet failure output for verbose=1 Ian Rogers
2026-05-13 23:04 ` [PATCH v1 08/14] perf test: Add summary reporting Ian Rogers
2026-05-13 23:04 ` [PATCH v1 09/14] perf test: Fix subtest status alignment for multi-digit indexes Ian Rogers
2026-05-13 23:04 ` [PATCH v1 10/14] perf test: Skip shebang and SPDX comments in shell test descriptions Ian Rogers
2026-05-13 23:04 ` [PATCH v1 11/14] perf test: Split monolithic 'util' test suite into sub-tests Ian Rogers
2026-05-13 23:04 ` [PATCH v1 12/14] perf test: Add -j/--junit option for JUnit XML test reports Ian Rogers
2026-05-13 23:04 ` [PATCH v1 13/14] perf test: Add shell test to validate JUnit XML reporting output Ian Rogers
2026-05-13 23:04 ` [PATCH v1 14/14] perf test: Remove /usr/bin/cc dependency from Intel PT shell test Ian Rogers
2026-05-31  5:27 ` [PATCH v2 00/14] perf test: Accelerate parallel test harness and add JUnit XML reporting Ian Rogers
2026-05-31  5:27   ` [PATCH v2 01/14] perf jevents.py: Make generated C code more kernel style Ian Rogers
2026-05-31  5:27   ` [PATCH v2 02/14] perf pmu-events: Add API to get metric table name and iterate tables Ian Rogers
2026-05-31  5:27   ` [PATCH v2 03/14] perf test: Drain pipe after child finishes to avoid losing output Ian Rogers
2026-05-31  5:27   ` [PATCH v2 04/14] perf test: Support dynamic test suites with setup callback and private data Ian Rogers
2026-05-31  5:27   ` [PATCH v2 05/14] perf test pmu-events: A sub-test per metric table Ian Rogers
2026-05-31  5:27   ` [PATCH v2 06/14] perf test: Refactor parallel poll loop to drain all pipes simultaneously Ian Rogers
2026-05-31  5:27   ` [PATCH v2 07/14] perf test: Show snippet failure output for verbose=1 Ian Rogers
2026-05-31  5:27   ` [PATCH v2 08/14] perf test: Add summary reporting Ian Rogers
2026-05-31  5:27   ` [PATCH v2 09/14] perf test: Fix subtest status alignment for multi-digit indexes Ian Rogers
2026-05-31  5:27   ` [PATCH v2 10/14] perf test: Skip shebang and SPDX comments in shell test descriptions Ian Rogers
2026-05-31  5:27   ` [PATCH v2 11/14] perf test: Split monolithic 'util' test suite into sub-tests Ian Rogers
2026-05-31  5:27   ` [PATCH v2 12/14] perf test: Add -j/--junit option for JUnit XML test reports Ian Rogers
2026-05-31  5:27   ` [PATCH v2 13/14] perf test: Add shell test to validate JUnit XML reporting output Ian Rogers
2026-05-31  5:27   ` [PATCH v2 14/14] perf test: Remove /usr/bin/cc dependency from Intel PT shell test Ian Rogers
2026-05-31  6:37   ` [PATCH v3 00/14] perf test: Accelerate parallel test harness and add JUnit XML reporting Ian Rogers
2026-05-31  6:37     ` [PATCH v3 01/14] perf jevents.py: Make generated C code more kernel style Ian Rogers
2026-05-31  6:37     ` [PATCH v3 02/14] perf pmu-events: Add API to get metric table name and iterate tables Ian Rogers
2026-05-31  6:37     ` [PATCH v3 03/14] perf test: Drain pipe after child finishes to avoid losing output Ian Rogers
2026-05-31  6:37     ` [PATCH v3 04/14] perf test: Support dynamic test suites with setup callback and private data Ian Rogers
2026-05-31  6:37     ` [PATCH v3 05/14] perf test pmu-events: A sub-test per metric table Ian Rogers
2026-05-31  6:37     ` [PATCH v3 06/14] perf test: Refactor parallel poll loop to drain all pipes simultaneously Ian Rogers
2026-05-31  6:37     ` [PATCH v3 07/14] perf test: Show snippet failure output for verbose=1 Ian Rogers
2026-05-31  6:37     ` [PATCH v3 08/14] perf test: Add summary reporting Ian Rogers
2026-05-31  6:37     ` [PATCH v3 09/14] perf test: Fix subtest status alignment for multi-digit indexes Ian Rogers
2026-05-31  6:37     ` [PATCH v3 10/14] perf test: Skip shebang and SPDX comments in shell test descriptions Ian Rogers
2026-05-31  6:37     ` [PATCH v3 11/14] perf test: Split monolithic 'util' test suite into sub-tests Ian Rogers
2026-05-31  6:37     ` [PATCH v3 12/14] perf test: Add -j/--junit option for JUnit XML test reports Ian Rogers
2026-05-31  6:37     ` [PATCH v3 13/14] perf test: Add shell test to validate JUnit XML reporting output Ian Rogers
2026-05-31  6:37     ` [PATCH v3 14/14] perf test: Remove /usr/bin/cc dependency from Intel PT shell test Ian Rogers
2026-05-31  8:22     ` [PATCH v4 00/15] perf test: Accelerate parallel test harness and add JUnit XML reporting Ian Rogers
2026-05-31  8:22       ` [PATCH v4 01/15] perf jevents.py: Make generated C code more kernel style Ian Rogers
2026-05-31  8:22       ` [PATCH v4 02/15] perf pmu-events: Add API to get metric table name and iterate tables Ian Rogers
2026-05-31  8:22       ` [PATCH v4 03/15] perf test: Drain pipe after child finishes to avoid losing output Ian Rogers
2026-05-31  8:22       ` [PATCH v4 04/15] perf test: Support dynamic test suites with setup callback and private data Ian Rogers
2026-05-31  8:22       ` [PATCH v4 05/15] perf test pmu-events: A sub-test per metric table Ian Rogers
2026-05-31  8:22       ` [PATCH v4 06/15] tools subcmd: Robust fallback and existence checks for process reaping Ian Rogers
2026-05-31  8:22       ` [PATCH v4 07/15] perf test: Refactor parallel poll loop to drain all pipes simultaneously Ian Rogers
2026-05-31  8:22       ` [PATCH v4 08/15] perf test: Show snippet failure output for verbose=1 Ian Rogers
2026-05-31  8:22       ` [PATCH v4 09/15] perf test: Add summary reporting Ian Rogers
2026-05-31  8:22       ` [PATCH v4 10/15] perf test: Fix subtest status alignment for multi-digit indexes Ian Rogers
2026-05-31  8:22       ` [PATCH v4 11/15] perf test: Skip shebang and SPDX comments in shell test descriptions Ian Rogers
2026-05-31  8:22       ` [PATCH v4 12/15] perf test: Split monolithic 'util' test suite into sub-tests Ian Rogers
2026-05-31  8:22       ` [PATCH v4 13/15] perf test: Add -j/--junit option for JUnit XML test reports Ian Rogers
2026-05-31  8:22       ` [PATCH v4 14/15] perf test: Add shell test to validate JUnit XML reporting output Ian Rogers
2026-05-31  8:22       ` [PATCH v4 15/15] perf test: Remove /usr/bin/cc dependency from Intel PT shell test Ian Rogers
2026-06-01  0:05       ` [PATCH v5 00/15] perf test: Accelerate parallel test harness and add JUnit XML reporting Ian Rogers
2026-06-01  0:05         ` [PATCH 01/15] perf jevents.py: Make generated C code more kernel style Ian Rogers
2026-06-01  0:05         ` [PATCH 02/15] perf pmu-events: Add API to get metric table name and iterate tables Ian Rogers
2026-06-01  0:05         ` [PATCH 03/15] perf test: Drain pipe after child finishes to avoid losing output Ian Rogers
2026-06-01  0:05         ` [PATCH 04/15] perf test: Support dynamic test suites with setup callback and private data Ian Rogers
2026-06-01  0:05         ` [PATCH 05/15] perf test pmu-events: A sub-test per metric table Ian Rogers
2026-06-01  0:05         ` [PATCH 06/15] tools subcmd: Robust fallback and existence checks for process reaping Ian Rogers
2026-06-01  0:05         ` [PATCH 07/15] perf test: Refactor parallel poll loop to drain all pipes simultaneously Ian Rogers
2026-06-01  0:05         ` [PATCH 08/15] perf test: Show snippet failure output for verbose=1 Ian Rogers
2026-06-01  0:05         ` [PATCH 09/15] perf test: Add summary reporting Ian Rogers
2026-06-01  0:05         ` [PATCH 10/15] perf test: Fix subtest status alignment for multi-digit indexes Ian Rogers
2026-06-01  0:05         ` [PATCH 11/15] perf test: Skip shebang and SPDX comments in shell test descriptions Ian Rogers
2026-06-01  0:05         ` [PATCH 12/15] perf test: Split monolithic 'util' test suite into sub-tests Ian Rogers
2026-06-01  0:05         ` [PATCH 13/15] perf test: Add -j/--junit option for JUnit XML test reports Ian Rogers
2026-06-01  0:05         ` [PATCH 14/15] perf test: Add shell test to validate JUnit XML reporting output Ian Rogers
2026-06-01  0:05         ` [PATCH 15/15] perf test: Remove /usr/bin/cc dependency from Intel PT shell test Ian Rogers
2026-06-01  6:13         ` [PATCH v6 00/15] perf test: Accelerate parallel test harness and add JUnit XML reporting Ian Rogers
2026-06-01  6:13           ` [PATCH 01/15] perf jevents.py: Make generated C code more kernel style Ian Rogers
2026-06-01  6:13           ` [PATCH 02/15] perf pmu-events: Add API to get metric table name and iterate tables Ian Rogers
2026-06-01  6:13           ` [PATCH 03/15] perf test: Drain pipe after child finishes to avoid losing output Ian Rogers
2026-06-01  6:13           ` [PATCH 04/15] perf test: Support dynamic test suites with setup callback and private data Ian Rogers
2026-06-01  6:13           ` [PATCH 05/15] perf test pmu-events: A sub-test per metric table Ian Rogers
2026-06-01  6:13           ` [PATCH 06/15] tools subcmd: Robust fallback and existence checks for process reaping Ian Rogers
2026-06-01  6:13           ` [PATCH 07/15] perf test: Refactor parallel poll loop to drain all pipes simultaneously Ian Rogers
2026-06-01  6:13           ` [PATCH 08/15] perf test: Show snippet failure output for verbose=1 Ian Rogers
2026-06-01  6:13           ` [PATCH 09/15] perf test: Add summary reporting Ian Rogers
2026-06-01  6:13           ` [PATCH 10/15] perf test: Fix subtest status alignment for multi-digit indexes Ian Rogers
2026-06-01  6:13           ` [PATCH 11/15] perf test: Skip shebang and SPDX comments in shell test descriptions Ian Rogers
2026-06-01  6:13           ` [PATCH 12/15] perf test: Split monolithic 'util' test suite into sub-tests Ian Rogers
2026-06-01  6:13           ` [PATCH 13/15] perf test: Add -j/--junit option for JUnit XML test reports Ian Rogers
2026-06-01  6:14           ` [PATCH 14/15] perf test: Add shell test to validate JUnit XML reporting output Ian Rogers
2026-06-01  6:14           ` [PATCH 15/15] perf test: Remove /usr/bin/cc dependency from Intel PT shell test Ian Rogers
2026-06-02  7:31           ` [PATCH v7 00/16] perf test: Parallel harness optimizations, summary & JUnit XML Ian Rogers
2026-06-02  7:31             ` [PATCH v7] perf tpebs: Fix concurrent stop races and PID reuse hazards in tpebs_stop Ian Rogers
2026-06-02  7:31             ` [PATCH v7] perf jevents.py: Make generated C code more kernel style Ian Rogers
2026-06-02  7:31             ` [PATCH v7] perf pmu-events: Add API to get metric table name and iterate tables Ian Rogers
2026-06-02  7:31             ` [PATCH v7] perf test: Drain pipe after child finishes to avoid losing output Ian Rogers
2026-06-02  7:31             ` [PATCH v7] perf test: Support dynamic test suites with setup callback and private data Ian Rogers
2026-06-02  7:31             ` [PATCH v7] perf test pmu-events: A sub-test per metric table Ian Rogers
2026-06-02  7:31             ` [PATCH v7] tools subcmd: Robust fallback and existence checks for process reaping Ian Rogers
2026-06-02  7:31             ` [PATCH v7] perf test: Refactor parallel poll loop to drain all pipes simultaneously Ian Rogers
2026-06-02  7:31             ` [PATCH v7] perf test: Show snippet failure output for verbose=1 Ian Rogers
2026-06-02  7:31             ` [PATCH v7] perf test: Add summary reporting Ian Rogers
2026-06-02  7:31             ` [PATCH v7] perf test: Fix subtest status alignment for multi-digit indexes Ian Rogers
2026-06-02  7:31             ` [PATCH v7] perf test: Skip shebang and SPDX comments in shell test descriptions Ian Rogers
2026-06-02  7:31             ` [PATCH v7] perf test: Split monolithic 'util' test suite into sub-tests Ian Rogers
2026-06-02 17:41             ` [PATCH v8 00/18] perf test: Parallel harness optimizations, summary, JUnit XML & PMU fixes Ian Rogers
2026-06-02 17:41               ` [PATCH v8 01/18] perf tpebs: Fix concurrent stop races and PID reuse hazards in tpebs_stop Ian Rogers
2026-06-02 17:41               ` [PATCH v8 02/18] perf jevents.py: Make generated C code more kernel style Ian Rogers
2026-06-02 17:41               ` [PATCH v8 03/18] perf pmu-events: Add API to get metric table name and iterate tables Ian Rogers
2026-06-02 17:41               ` [PATCH v8 04/18] perf test: Drain pipe after child finishes to avoid losing output Ian Rogers
2026-06-02 17:41               ` [PATCH v8 05/18] perf test: Support dynamic test suites with setup callback and private data Ian Rogers
2026-06-02 17:41               ` [PATCH v8 06/18] perf test pmu-events: A sub-test per metric table Ian Rogers
2026-06-02 17:41               ` [PATCH v8 07/18] tools subcmd: Robust fallback and existence checks for process reaping Ian Rogers
2026-06-02 17:41               ` [PATCH v8 08/18] perf test: Refactor parallel poll loop to drain all pipes simultaneously Ian Rogers
2026-06-02 17:41               ` [PATCH v8 09/18] perf test: Show snippet failure output for verbose=1 Ian Rogers
2026-06-02 17:41               ` [PATCH v8 10/18] perf test: Add summary reporting Ian Rogers
2026-06-04 14:38                 ` Arnaldo Carvalho de Melo
2026-06-04 14:40                   ` Arnaldo Carvalho de Melo
2026-06-02 17:41               ` [PATCH v8 11/18] perf test: Fix subtest status alignment for multi-digit indexes Ian Rogers
2026-06-02 17:41               ` [PATCH v8 12/18] perf test: Skip shebang and SPDX comments in shell test descriptions Ian Rogers
2026-06-02 17:41               ` [PATCH v8 13/18] perf test: Split monolithic 'util' test suite into sub-tests Ian Rogers
2026-06-02 17:41               ` [PATCH v8 14/18] perf test: Add -j/--junit option for JUnit XML test reports Ian Rogers
2026-06-02 17:41               ` [PATCH v8 15/18] perf test: Add shell test to validate JUnit XML reporting output Ian Rogers
2026-06-02 17:41               ` [PATCH v8 16/18] perf test: Remove /usr/bin/cc dependency from Intel PT shell test Ian Rogers
2026-06-02 17:41               ` [PATCH v8 17/18] perf pmu: Recognize 'default_core' as a core PMU and document matching Ian Rogers
2026-06-02 17:41               ` [PATCH v8 18/18] perf test: Truncate printed test descriptions dynamically to avoid terminal wrapping Ian Rogers
2026-06-04 14:45                 ` Arnaldo Carvalho de Melo
2026-06-04 14:48                   ` Arnaldo Carvalho de Melo
2026-06-04 16:36               ` [PATCH v9 0/2] perf test & PMU metric resolution improvements Ian Rogers
2026-06-04 16:36                 ` [PATCH v9 1/2] perf pmu: Recognize 'default_core' as a core PMU and document matching Ian Rogers
2026-06-04 16:36                 ` [PATCH v9 2/2] perf test: Truncate printed test descriptions dynamically to avoid terminal wrapping Ian Rogers
2026-06-04 20:26                   ` Arnaldo Carvalho de Melo
2026-06-04 20:47                     ` Ian Rogers
2026-06-04 22:06                       ` [PATCH v10] " Ian Rogers
2026-06-11 15:06                         ` Arnaldo Carvalho de Melo
2026-06-04 20:22                 ` [PATCH v9 0/2] perf test & PMU metric resolution improvements Arnaldo Carvalho de Melo

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox