From: Arnaldo Carvalho de Melo <acme@kernel.org>
To: Namhyung Kim <namhyung@kernel.org>
Cc: Ingo Molnar <mingo@kernel.org>,
Thomas Gleixner <tglx@linutronix.de>,
James Clark <james.clark@linaro.org>,
Jiri Olsa <jolsa@kernel.org>, Ian Rogers <irogers@google.com>,
Adrian Hunter <adrian.hunter@intel.com>,
Clark Williams <williams@redhat.com>,
linux-perf-users@vger.kernel.org,
Arnaldo Carvalho de Melo <acme@redhat.com>,
linux-kernel@vger.kernel.org
Subject: [PATCH v9 00/13] perf coverage: A harness for perf, and the fixes it found
Date: Thu, 19 Nov 2026 22:31:55 -0300 [thread overview]
Message-ID: <20261120013211.1510-1-acme@kernel.org> (raw)
From: Arnaldo Carvalho de Melo <acme@redhat.com>
Hi,
Nothing measured how much of perf the test suite reaches. Patch 1 links
the GTK UI with EXTRA_CFLAGS, without which the coverage build never saw
the UI's counters (the jvmti agent got the same fix upstream); patch 2
adds tools/perf/scripts/coverage.sh: it builds perf with clang source
based coverage, runs the complete perf test suite and reports line and
function coverage, optionally against a baseline. Also as 'make -C
tools/perf coverage', 'coverage-prep' and 'coverage-table'.
The useful output is the list of files with the most uncovered lines,
computed over the whole tree and not only the files the tests already
touch: that is where new tests should try to get more coverage. Namhyung
asked what the table is, so this is what coverage-table shows by default:
make -C tools/perf coverage-table
make: Entering directory '/home/acme/git/perf-tools-next/tools/perf'
tools/perf (no tests/ or bench/), 100+ executable lines:
the 20 biggest uncovered gaps, start here when adding coverage tests.
--all for every file: make -C tools/perf coverage-table COVERAGE_ARGS="--all"
┌──────────────────────────────────────────┬────────┬────────┬────────┬─────────────────────────────┬─────────────┬──────────┐
│ File │ Lines │ Hit │ Miss │ Line Coverage │ Funcs │ FuncCov │
├──────────────────────────────────────────┼────────┼────────┼────────┼─────────────────────────────┼─────────────┼──────────┤
│ ui/browsers/hists.c │ 2730 │ 0 │ 2730 │ ░░░░░░░░░░░░░░░░░░░░ 0.0% │ 0/124 │ 0.00% │
│ builtin-sched.c │ 3737 │ 1542 │ 2195 │ ████████░░░░░░░░░░░░ 41.3% │ 62/141 │ 43.97% │
│ builtin-script.c │ 3212 │ 1118 │ 2094 │ ██████░░░░░░░░░░░░░░ 34.8% │ 51/124 │ 41.13% │
│ util/intel-pt-decoder/intel-pt-decoder.c │ 3208 │ 1201 │ 2007 │ ███████░░░░░░░░░░░░░ 37.4% │ 70/119 │ 58.82% │
│ util/intel-pt.c │ 3237 │ 1426 │ 1811 │ ████████░░░░░░░░░░░░ 44.0% │ 79/142 │ 55.63% │
│ util/python.c │ 2984 │ 1301 │ 1683 │ ████████░░░░░░░░░░░░ 43.6% │ 128/224 │ 57.14% │
│ util/sort.c │ 2421 │ 861 │ 1560 │ ███████░░░░░░░░░░░░░ 35.6% │ 83/246 │ 33.74% │
│ util/session.c │ 2686 │ 1204 │ 1482 │ ████████░░░░░░░░░░░░ 44.8% │ 65/125 │ 52.00% │
│ util/annotate.c │ 2245 │ 866 │ 1379 │ ███████░░░░░░░░░░░░░ 38.6% │ 60/118 │ 50.85% │
│ builtin-kmem.c │ 1337 │ 0 │ 1337 │ ░░░░░░░░░░░░░░░░░░░░ 0.0% │ 0/70 │ 0.00% │
│ util/arm-spe.c │ 1330 │ 0 │ 1330 │ ░░░░░░░░░░░░░░░░░░░░ 0.0% │ 0/50 │ 0.00% │
│ builtin-kvm.c │ 1316 │ 0 │ 1316 │ ░░░░░░░░░░░░░░░░░░░░ 0.0% │ 0/90 │ 0.00% │
│ util/header.c │ 3941 │ 2662 │ 1279 │ █████████████░░░░░░░ 67.6% │ 159/180 │ 88.33% │
│ util/evsel.c │ 3326 │ 2067 │ 1259 │ ████████████░░░░░░░░ 62.2% │ 128/155 │ 82.58% │
│ builtin-record.c │ 2737 │ 1501 │ 1236 │ ██████████░░░░░░░░░░ 54.8% │ 84/120 │ 70.00% │
│ util/probe-event.c │ 2765 │ 1554 │ 1211 │ ███████████░░░░░░░░░ 56.2% │ 81/103 │ 78.64% │
│ builtin-trace.c │ 3657 │ 2451 │ 1206 │ █████████████░░░░░░░ 67.0% │ 154/198 │ 77.78% │
│ builtin-inject.c │ 2023 │ 890 │ 1133 │ ████████░░░░░░░░░░░░ 44.0% │ 37/94 │ 39.36% │
│ builtin-c2c.c │ 2027 │ 1018 │ 1009 │ ██████████░░░░░░░░░░ 50.2% │ 58/156 │ 37.18% │
│ util/auxtrace.c │ 2013 │ 1009 │ 1004 │ ██████████░░░░░░░░░░ 50.1% │ 81/126 │ 64.29% │
├──────────────────────────────────────────┼────────┼────────┼────────┼─────────────────────────────┼─────────────┼──────────┤
│ TOTAL (perf) │ 173244 │ 91059 │ 82185 │ ██████████░░░░░░░░░░ 52.6% │ 5868/9306 │ 63.06% │
└──────────────────────────────────────────┴────────┴────────┴────────┴─────────────────────────────┴─────────────┴──────────┘
make: Leaving directory '/home/acme/git/perf-tools-next/tools/perf'
--all lists every tools/perf file, tests and benches included, most
covered first. The table total is the tools/perf subtotal; the run
summary has the whole report.
The rest is what the first runs found, in the code, the build and the
tests:
- The data type resolver missed absolute addressing, in 32-bit code
and in the 64-bit kernel's sign extended disp32 (patch 3); the
DW_OP_addrx globals it also missed are fixed upstream by
c3991e42f58a3650, so patch 4 only adds the test. A struct pointer
kept in a register at -O2 had no test (patch 7).
- The flexible array member had no test; one showed the struct row at
0.00% while the member said 100.00% (patches 5-6).
- --insn-stat was printed before the loop that accounts the samples,
so --code-with-type showed zeros (patch 8).
- The libbfd probe printed a missing bfd.h error five times per
build-test run (patch 9).
- FEATURE-DUMP didn't record the features checked outside
FEATURE_TESTS (llvm-perf, rust, jvmti), disabling them in the
build-test builds (patch 10).
- The Intel PT virtual LBR check lost every aux sample with a single
data page (patch 11).
- The dlfilter test kept samples whose IP the kernel now zeroes on a
userspace-to-kernel skid (patch 12).
- The LBR inline unwinding check failed instead of reporting Skip when
the AMD PMU rejects branch sampling (patch 13).
With coverage-prep by the developer (clang DWARF 5) and the full suite
as root on this AMD host, 105,466 of 205,823 lines were covered (51.24%;
v5: 50.67% of 205,810), 86,344 of 173,257 in tools/perf/ (49.84%). 168
tests passed, 28 skipped and one failed (20.2, the POLL_HUP regression,
outside this series). The global variable and instruction-stats tests
passed; the Intel PT hardware tests skipped here, so patch 11's 64-page
AUX buffer was only tested on the Intel host below.
The same tree as root on an Intel hybrid host (i7-14700K, Fedora 45)
covered 110,258 of 205,830 lines (53.57%), 91,059 of 173,244 in
tools/perf/ (52.56%). 166 tests passed, 27 skipped and three failed:
20.2, and 143 (branch stacks) and 212 (vfs_getname), which passed on
rerun. The Intel PT tests ran and passed there, exercising patch 11's
64-page AUX buffer.
The AMD numbers are from the v6 set; the later versions change the
build, the tests' diagnostics, the harness and, in patch 3, the operand
parser, so read them as indicative.
The harness and the tests were developed with an AI coding assistant
(OpenCode), which drafted the script and the tests and helped debug the
failures; an independent local model reviewed the script and I verified
its findings before folding them in. I reviewed every change and
reproduced each failure before fixing it.
What changed from v8:
- 1 is GTK only: the jvmti link change went upstream as
321eae8603c76a6e, and the series is rebased onto it.
- 2 checks the path before creating anything and each level before
using it, so the EXIT trap cannot chown a child of another user's
directory (sashiko-bot).
- 3 answers the 32-bit, TYPE and mem-ref questions with the kernel's
sign extended disp32 and a resolving absolute load (Namhyung).
- 3 restores the jump/loop* and movabs filters lost in v7 and adds the
operand parser test.
Best regards,
- Arnaldo
Arnaldo Carvalho de Melo (13):
perf build: Link the GTK UI with EXTRA_CFLAGS
perf scripts: Add a coverage harness for perf
perf annotate: Handle absolute addressing in data type profiling
perf tests: Exercise DW_OP_addrx global variables
perf tests: Exercise the flexible array member path
perf annotate-data: Count flexible array accesses for the whole struct
perf tests: Exercise data type resolution from a register
perf annotate: Print the stats after the annotations
perf tests: Silence the libbfd probe in build-test
perf build: Record the features checked outside FEATURE_TESTS
perf tests: Fix the Intel PT virtual LBR check
perf test: Drop samples without an IP in the dlfilter test
perf test: Skip the LBR inline unwinding check when unsupported
tools/perf/.gitignore | 1 +
tools/perf/Documentation/perf-annotate.txt | 2 +-
tools/perf/Makefile | 11 +-
tools/perf/Makefile.config | 7 +
tools/perf/Makefile.perf | 6 +-
tools/perf/builtin-annotate.c | 19 +-
tools/perf/scripts/coverage.sh | 1153 +++++++++++++++++
tools/perf/tests/Build | 1 +
tools/perf/tests/annotate-ops.c | 93 ++
tools/perf/tests/builtin-test.c | 4 +
tools/perf/tests/make | 2 +-
tools/perf/tests/shell/addr2line_inlines.sh | 20 +-
tools/perf/tests/shell/annotate_stats.sh | 69 +
.../perf/tests/shell/data_type_flex_array.sh | 111 ++
tools/perf/tests/shell/data_type_global.sh | 73 ++
tools/perf/tests/shell/data_type_register.sh | 63 +
tools/perf/tests/shell/script_dlfilter.sh | 2 +-
tools/perf/tests/shell/test_intel_pt.sh | 10 +-
tools/perf/tests/tests.h | 4 +
tools/perf/tests/workloads/Build | 6 +
tools/perf/tests/workloads/flex_array.c | 55 +
tools/perf/tests/workloads/global_data.c | 57 +
tools/perf/tests/workloads/regptr.c | 60 +
tools/perf/ui/browsers/annotate-data.c | 7 +-
tools/perf/util/annotate-arch/annotate-x86.c | 48 +-
tools/perf/util/annotate-data.c | 25 +-
tools/perf/util/annotate-data.h | 8 +
tools/perf/util/annotate.c | 27 +-
tools/perf/util/annotate.h | 7 +
29 files changed, 1922 insertions(+), 29 deletions(-)
create mode 100755 tools/perf/scripts/coverage.sh
create mode 100644 tools/perf/tests/annotate-ops.c
create mode 100755 tools/perf/tests/shell/annotate_stats.sh
create mode 100755 tools/perf/tests/shell/data_type_flex_array.sh
create mode 100755 tools/perf/tests/shell/data_type_global.sh
create mode 100755 tools/perf/tests/shell/data_type_register.sh
create mode 100644 tools/perf/tests/workloads/flex_array.c
create mode 100644 tools/perf/tests/workloads/global_data.c
create mode 100644 tools/perf/tests/workloads/regptr.c
base-commit: 321eae8603c76a6e
v1-head: 3f5465bea8f5a97a
v2-head: 9706c488afffeb6d
v3-head: f2daf1586528671a
v4-head: b9e240a1b80ecd2a
v5-head: a21454e3d055c93b
v6-head: e1315408a335617d
v7-head: 7f8dc594d77ed495
v8-head: 67fde34aeab98a91
v9-head: b1765115908ede31
--
2.56.1
next reply other threads:[~2026-11-20 1:32 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 1:31 Arnaldo Carvalho de Melo [this message]
2026-11-20 1:31 ` [PATCH v9 01/13] perf build: Link the GTK UI with EXTRA_CFLAGS Arnaldo Carvalho de Melo
2026-11-20 1:38 ` sashiko-bot
2026-11-20 1:31 ` [PATCH v9 02/13] perf scripts: Add a coverage harness for perf Arnaldo Carvalho de Melo
2026-11-20 1:40 ` sashiko-bot
2026-11-20 1:31 ` [PATCH v9 03/13] perf annotate: Handle absolute addressing in data type profiling Arnaldo Carvalho de Melo
2026-11-20 1:49 ` sashiko-bot
2026-11-20 1:31 ` [PATCH v9 04/13] perf tests: Exercise DW_OP_addrx global variables Arnaldo Carvalho de Melo
2026-11-20 1:45 ` sashiko-bot
2026-11-20 1:32 ` [PATCH v9 05/13] perf tests: Exercise the flexible array member path Arnaldo Carvalho de Melo
2026-11-20 1:38 ` sashiko-bot
2026-11-20 1:32 ` [PATCH v9 06/13] perf annotate-data: Count flexible array accesses for the whole struct Arnaldo Carvalho de Melo
2026-11-20 1:41 ` sashiko-bot
2026-11-20 1:32 ` [PATCH v9 07/13] perf tests: Exercise data type resolution from a register Arnaldo Carvalho de Melo
2026-11-20 1:39 ` sashiko-bot
2026-11-20 1:32 ` [PATCH v9 08/13] perf annotate: Print the stats after the annotations Arnaldo Carvalho de Melo
2026-11-20 1:37 ` sashiko-bot
2026-11-20 1:32 ` [PATCH v9 09/13] perf tests: Silence the libbfd probe in build-test Arnaldo Carvalho de Melo
2026-11-20 1:35 ` sashiko-bot
2026-11-20 1:32 ` [PATCH v9 10/13] perf build: Record the features checked outside FEATURE_TESTS Arnaldo Carvalho de Melo
2026-11-20 1:40 ` sashiko-bot
2026-11-20 1:32 ` [PATCH v9 11/13] perf tests: Fix the Intel PT virtual LBR check Arnaldo Carvalho de Melo
2026-11-20 1:39 ` sashiko-bot
2026-11-20 1:32 ` [PATCH v9 12/13] perf test: Drop samples without an IP in the dlfilter test Arnaldo Carvalho de Melo
2026-11-20 1:39 ` sashiko-bot
2026-11-20 1:32 ` [PATCH v9 13/13] perf test: Skip the LBR inline unwinding check when unsupported Arnaldo Carvalho de Melo
2026-11-20 1:39 ` sashiko-bot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120013211.1510-1-acme@kernel.org \
--to=acme@kernel.org \
--cc=acme@redhat.com \
--cc=adrian.hunter@intel.com \
--cc=irogers@google.com \
--cc=james.clark@linaro.org \
--cc=jolsa@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-perf-users@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=namhyung@kernel.org \
--cc=tglx@linutronix.de \
--cc=williams@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.