* [PATCH v1 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events
2026-09-29 2:19 [PATCH v1 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
@ 2026-09-29 2:19 ` Ian Rogers
2026-09-29 2:38 ` sashiko-bot
2026-09-29 2:19 ` [PATCH v1 2/9] perf python sctop: Fix offline interval printing and test flakiness Ian Rogers
` (8 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 2:19 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim
Cc: Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark,
linux-perf-users, linux-kernel
On kernels without the fix ("perf/core: Restore header fields in
sideband output callbacks") to restore event_id.header.size in
perf_event_ksymbol_output(), perf_event_bpf_output(), and
perf_event_text_poke_output(), concurrent perf sessions cause those
sideband records to be emitted with header.size inflated by multiple
id_header_size increments while the single id_sample is written
immediately after the event payload. Indexing backwards from
event->header.size reads uninitialized ring-buffer bytes at the end of
the record, causing evlist__event2evsel() to fail with -EFAULT.
Add evsel__event_size() to clamp the effective size used to locate the
trailing id_sample for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and
PERF_RECORD_TEXT_POKE to payload + id_hdr_size while leaving
event->header.size intact for advancing the ring-buffer/file stream.
Fixes: 9aa0bfa370b2 ("perf tools: Handle PERF_RECORD_KSYMBOL")
Fixes: 45178a928a4b ("perf tools: Handle PERF_RECORD_BPF_EVENT")
Fixes: 246eba8e9041 ("perf tools: Add support for PERF_RECORD_TEXT_POKE")
Link: https://lore.kernel.org/r/20260929014206.4175245-1-irogers@google.com
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/util/evlist.c | 6 ++--
tools/perf/util/evsel.c | 59 +++++++++++++++++++++++++++++++++++++++-
tools/perf/util/evsel.h | 1 +
3 files changed, 63 insertions(+), 3 deletions(-)
diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index 9392d912d254..c2402e4791b6 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -973,13 +973,15 @@ static int evlist__event2id(struct evlist *evlist, union perf_event *event, u64
const __u64 *array = event->sample.array;
ssize_t n;
- n = (event->header.size - sizeof(event->header)) >> 3;
-
if (event->header.type == PERF_RECORD_SAMPLE) {
+ n = (event->header.size - sizeof(event->header)) >> 3;
if (evlist__id_pos(evlist) >= n)
return -1;
*id = array[evlist__id_pos(evlist)];
} else {
+ u16 size = evsel__event_size(evlist__first(evlist), event);
+
+ n = (size - sizeof(event->header)) >> 3;
if (evlist__is_pos(evlist) > n)
return -1;
n -= evlist__is_pos(evlist);
diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index 3367242c5764..70da6be798cc 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -3216,7 +3216,7 @@ static int perf_evsel__parse_id_sample(const union perf_event *event,
const __u64 *array = event->sample.array;
bool swapped = evsel->needs_swap;
union u64_swap u;
- int i = ((event->header.size - sizeof(event->header)) / sizeof(u64)) - 1;
+ int i = ((evsel__event_size(evsel, event) - sizeof(event->header)) / sizeof(u64)) - 1;
if (type & PERF_SAMPLE_IDENTIFIER) {
if (i < 0)
@@ -3966,6 +3966,63 @@ u16 evsel__id_hdr_size(const struct evsel *evsel)
return size;
}
+/*
+ * Prior to kernel fix, perf_event_ksymbol_output(), perf_event_bpf_output(),
+ * and perf_event_text_poke_output() in kernel/events/core.c did not save and
+ * restore event_id.header.size across perf_iterate_sb() iterations. When
+ * multiple perf_events had attr.ksymbol, attr.bpf_event, or attr.text_poke
+ * enabled, header.size was incremented by id_header_size for each matching
+ * event while only a single id_sample was written immediately after the event
+ * payload. Clamp the effective size used to locate the trailing id_sample to
+ * payload + id_hdr_size so events recorded on unpatched kernels can be parsed
+ * without reading uninitialized ring-buffer bytes.
+ */
+u16 evsel__event_size(const struct evsel *evsel, const union perf_event *event)
+{
+ u16 size = event->header.size;
+ u16 id_hdr_size;
+ size_t payload;
+
+ if (!evsel->core.attr.sample_id_all)
+ return size;
+
+ switch (event->header.type) {
+ case PERF_RECORD_KSYMBOL: {
+ const char *name = event->ksymbol.name;
+ size_t fixed = offsetof(struct perf_record_ksymbol, name);
+ size_t max_len, len;
+
+ if (size <= fixed)
+ return size;
+ max_len = size - fixed;
+ len = strnlen(name, max_len);
+ if (len == max_len)
+ return size;
+ payload = fixed + PERF_ALIGN(len + 1, sizeof(u64));
+ break;
+ }
+ case PERF_RECORD_BPF_EVENT:
+ payload = sizeof(struct perf_record_bpf_event);
+ break;
+ case PERF_RECORD_TEXT_POKE: {
+ size_t fixed = offsetof(struct perf_record_text_poke_event, bytes);
+
+ if (size < fixed)
+ return size;
+ payload = fixed + PERF_ALIGN((size_t)event->text_poke.old_len +
+ event->text_poke.new_len, sizeof(u64));
+ break;
+ }
+ default:
+ return size;
+ }
+
+ id_hdr_size = evsel__id_hdr_size(evsel);
+ if (payload + id_hdr_size < size)
+ return payload + id_hdr_size;
+ return size;
+}
+
#ifdef HAVE_LIBTRACEEVENT
struct tep_format_field *evsel__field(struct evsel *evsel, const char *name)
{
diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
index 5c5799cee601..174f3414fd3c 100644
--- a/tools/perf/util/evsel.h
+++ b/tools/perf/util/evsel.h
@@ -469,6 +469,7 @@ int evsel__parse_sample_timestamp(struct evsel *evsel, union perf_event *event,
u64 *timestamp);
u16 evsel__id_hdr_size(const struct evsel *evsel);
+u16 evsel__event_size(const struct evsel *evsel, const union perf_event *event);
static inline struct evsel *evsel__next(struct evsel *evsel)
{
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v1 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events
2026-09-29 2:19 ` [PATCH v1 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events Ian Rogers
@ 2026-09-29 2:38 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 2:38 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf evsel: Incorrect PERF_RECORD_TEXT_POKE payload size calculation
--
commit fc6474e9b3305fb280e04ce2b30b2a815e2f3a7e
Author: Ian Rogers <irogers@google.com>
perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events
This commit adds evsel__event_size() to clamp the effective size used to
locate the trailing id_sample for specific events. This prevents user-space
tools from reading uninitialized memory when processing data from unpatched
kernels.
Link: https://lore.kernel.org/r/20260929014206.4175245-1-irogers@google.com
> diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
> index 3367242c57640..70da6be798cca 100644
> --- a/tools/perf/util/evsel.c
> +++ b/tools/perf/util/evsel.c
[ ... ]
> @@ -3966,6 +3966,63 @@ u16 evsel__id_hdr_size(const struct evsel *evsel)
> return size;
> }
>
> +/*
> + * Prior to kernel fix, perf_event_ksymbol_output(), perf_event_bpf_output(),
[ ... ]
> + case PERF_RECORD_TEXT_POKE: {
> + size_t fixed = offsetof(struct perf_record_text_poke_event, bytes);
> +
> + if (size < fixed)
> + return size;
> + payload = fixed + PERF_ALIGN((size_t)event->text_poke.old_len +
> + event->text_poke.new_len, sizeof(u64));
[Severity: Medium]
Does this misapply the 8-byte alignment only to the variable lengths rather
than the entire structure?
The kernel generates PERF_RECORD_TEXT_POKE events by padding the entire
event size to an 8-byte boundary. The payload size before id_sample is
16 + ALIGN(old_len + new_len + 4, 8).
This calculates the user-space payload as
fixed + PERF_ALIGN(old_len + new_len, 8), where fixed = 20. Because 20 is
not a multiple of 8, adding it outside the alignment yields a mathematically
incorrect size.
For example, if old_len = 1 and new_len = 1, the kernel calculates 24 bytes,
but the code here calculates 28 bytes. Calculating the wrong payload size
causes perf to index incorrectly into event->sample.array, leading to parsing
failures or out-of-bounds accesses.
> + break;
> + }
> + default:
> + return size;
> + }
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929021913.56010-1-irogers@google.com?part=1
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v1 2/9] perf python sctop: Fix offline interval printing and test flakiness
2026-09-29 2:19 [PATCH v1 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
2026-09-29 2:19 ` [PATCH v1 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events Ian Rogers
@ 2026-09-29 2:19 ` Ian Rogers
2026-09-29 2:30 ` sashiko-bot
2026-09-29 2:19 ` [PATCH v1 3/9] perf python stat-cpi: Fix live mode signal races " Ian Rogers
` (7 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 2:19 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim
Cc: Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark,
linux-perf-users, linux-kernel
In sctop.py, process_event() updated self.last_print_time on every
sample in offline mode even when skip was true for non-matching comm
events. When filtering by comm with a short interval (such as 'sleep 1'
in test_sctop_python.sh) on a system-wide recording spanning more than
1 second under heavy load, non-matching samples could trigger
print_current_totals() and clear self.syscalls, followed by finally:
printing a trailing empty table. In addition, analyzer.e_machine was
initialized after session.process_events() instead of before.
Fix sctop.py by initializing analyzer.e_machine before
session.process_events(), only advancing self.last_print_time when a
sample is not skipped, and only calling print_current_totals() in
finally: if no table has been printed yet or remaining syscalls are
pending.
In test_sctop_python.sh, use 'perf list tracepoint' instead of
unfiltered 'perf list', record the child workload directly without '-a'
and with '-B -N --no-bpf-event' to avoid system-wide ringbuffer overflow
and synthesis contention, and retry up to 5 times if needed.
Fixes: b83f0bacf5e4 ("perf python: Port sctop to perf module")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/python/sctop.py | 9 ++--
tools/perf/tests/shell/test_sctop_python.sh | 48 +++++++++++----------
2 files changed, 31 insertions(+), 26 deletions(-)
diff --git a/tools/perf/python/sctop.py b/tools/perf/python/sctop.py
index 42e95ecfffad..c524e08e6440 100755
--- a/tools/perf/python/sctop.py
+++ b/tools/perf/python/sctop.py
@@ -37,6 +37,7 @@ class SCTopAnalyzer:
self.offline = offline
self.own_pid = os.getpid()
self.last_print_time: Optional[int] = None
+ self.printed = False
self.session: Optional[perf.session] = None
self.e_machine: Optional[int] = None
@@ -127,7 +128,7 @@ class SCTopAnalyzer:
if not skip and is_enter and 0 <= (syscall_id & ~0x40000000) <= 0xffff:
self.syscalls[syscall_id] += 1
- if self.offline and hasattr(sample, "sample_time"):
+ if not skip and self.offline and hasattr(sample, "sample_time"):
interval_ns = self.interval * (10 ** 9)
if self.last_print_time is None:
self.last_print_time = sample.sample_time
@@ -137,6 +138,7 @@ class SCTopAnalyzer:
def print_current_totals(self):
"""Print current syscall totals."""
+ self.printed = True
# Clear terminal
if not self.offline:
print("\x1b[2J\x1b[H", end="")
@@ -217,8 +219,8 @@ def main():
if args.input:
session = perf.session(perf.data(args.input), sample=analyzer.process_event)
analyzer.session = session
- session.process_events()
analyzer.e_machine = getattr(session, "e_machine", None)
+ session.process_events()
else:
try:
live_session = LiveSession(
@@ -237,7 +239,8 @@ def main():
sys.exit(1)
finally:
if args.input:
- analyzer.print_current_totals()
+ if not analyzer.printed or analyzer.syscalls:
+ analyzer.print_current_totals()
# Break the reference cycle between perf.session and analyzer.process_event
# because perf.session lacks cyclic GC support (tp_traverse).
analyzer.session = None
diff --git a/tools/perf/tests/shell/test_sctop_python.sh b/tools/perf/tests/shell/test_sctop_python.sh
index 007f2584cce6..230eeca96fb8 100755
--- a/tools/perf/tests/shell/test_sctop_python.sh
+++ b/tools/perf/tests/shell/test_sctop_python.sh
@@ -41,37 +41,39 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
echo "Testing sctop.py..."
# Create a perf.data file.
-if perf list | grep -q "raw_syscalls:sys_enter"; then
- perf record -e raw_syscalls:sys_enter -a -o "${temp_data}" \
- -- sleep 0.1 >/dev/null 2>&1 || \
- { echo "Skipping test, perf record failed"; exit 2; }
-else
+if ! perf list tracepoint | grep -q "raw_syscalls:sys_enter"; then
echo "Skipping test, no raw_syscalls:sys_enter event"
exit 2
fi
-if [ ! -s "${temp_data}" ]; then
- echo "Skipping test, perf record failed to create data"
- exit 2
-fi
+passed=0
+for _ in 1 2 3 4 5; do
+ rm -f "${temp_data}" "${temp_out}"
+ if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \
+ -- sh -c "sleep 0.1; sleep 0.05" >/dev/null 2>&1; then
+ echo "Skipping test, perf record failed"
+ exit 2
+ fi
+
+ if [ ! -s "${temp_data}" ]; then
+ continue
+ fi
+
+ # Check that the script executes
+ if perf script sctop -i "${temp_data}" > "${temp_out}" && \
+ grep -E -q "[0-9]+$" "${temp_out}" && \
+ perf script sctop -i "${temp_data}" sleep 1 > "${temp_out}" && \
+ grep -E -q "[0-9]+$" "${temp_out}"; then
+ passed=1
+ break
+ fi
+done
-# Check that the script executes
-if ! perf script sctop -i "${temp_data}" > "${temp_out}"; then
+if [ "$passed" -eq 0 ]; then
echo "sctop.py test failed"
err=1
-elif ! grep -E -q "[0-9]+$" "${temp_out}"; then
- echo "Failed to find metric data rows in default run"
- err=1
-elif ! perf script sctop -i "${temp_data}" sleep 1 > "${temp_out}"; then
- echo "sctop.py comm+interval test failed"
- err=1
else
- if ! grep -E -q "[0-9]+$" "${temp_out}"; then
- echo "Failed to find metric data rows"
- err=1
- else
- echo "sctop test passed."
- fi
+ echo "sctop test passed."
fi
rm -f "${temp_out}"
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v1 2/9] perf python sctop: Fix offline interval printing and test flakiness
2026-09-29 2:19 ` [PATCH v1 2/9] perf python sctop: Fix offline interval printing and test flakiness Ian Rogers
@ 2026-09-29 2:30 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 2:30 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf python: sctop offline interval bucketting broken by comm filter
--
commit b0674a2bbd03e3e74148a12d9d8a3c74b4f35ce5
Author: Ian Rogers <irogers@google.com>
perf python sctop: Fix offline interval printing and test flakiness
This commit addresses sctop offline interval printing by only advancing the
logical clock on non-skipped samples. It also reorders e_machine
initialization and updates the shell test with retries and different
recording parameters.
> diff --git a/tools/perf/python/sctop.py b/tools/perf/python/sctop.py
> index 42e95ecfffad7..c524e08e64405 100755
> --- a/tools/perf/python/sctop.py
> +++ b/tools/perf/python/sctop.py
[ ... ]
> @@ -127,7 +128,7 @@ class SCTopAnalyzer:
> if not skip and is_enter and 0 <= (syscall_id & ~0x40000000) <= 0xffff:
> self.syscalls[syscall_id] += 1
>
> - if self.offline and hasattr(sample, "sample_time"):
> + if not skip and self.offline and hasattr(sample, "sample_time"):
[Severity: Medium]
Does adding 'not skip' to this condition in sctop.py's
SCTopAnalyzer.process_event() introduce a regression by blocking the
offline clock from advancing during periods where the target process is
idle?
By ignoring the sample_time of non-matching events, the script's internal
logical clock fails to advance. This prevents the current interval bucket
from flushing until the target comm executes another syscall, merging
events from multiple intervals into a single bucket and resulting in
incorrect interval sums.
Does this also cause the script to miss printing periodic empty tables
during idle periods, breaking consistency with live mode?
> interval_ns = self.interval * (10 ** 9)
> if self.last_print_time is None:
> self.last_print_time = sample.sample_time
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929021913.56010-1-irogers@google.com?part=2
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v1 3/9] perf python stat-cpi: Fix live mode signal races and test flakiness
2026-09-29 2:19 [PATCH v1 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
2026-09-29 2:19 ` [PATCH v1 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events Ian Rogers
2026-09-29 2:19 ` [PATCH v1 2/9] perf python sctop: Fix offline interval printing and test flakiness Ian Rogers
@ 2026-09-29 2:19 ` Ian Rogers
2026-09-29 2:25 ` sashiko-bot
2026-09-29 2:19 ` [PATCH v1 4/9] perf test: Deflake Intel PT Python shell tests under load Ian Rogers
` (6 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 2:19 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim
Cc: Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark,
linux-perf-users, linux-kernel
In test_stat_cpi_python.sh, 'perf test -w noploop &' defaults to a
1-second duration and can exit under heavy parallel load before
'perf stat -p' and 'perf script stat-cpi' finish starting up. In
addition, the fixed 'sleep 0.5' before sending SIGINT can fire before
Python finishes importing the perf module, opening the live evlist, and
flushing the first interval.
In stat-cpi.py, register SIGINT and SIGTERM handlers before calling
_open_live_evlist() and pass flush=True when printing live output so
redirected stdout is flushed immediately after each interval.
In test_stat_cpi_python.sh, run 'perf test -w noploop 60 &' so the
target workload stays alive until killed, and poll the output file for
'cpi' (up to 5 seconds) before sending SIGINT.
Fixes: 4425182d426b ("perf python: Port stat-cpi to perf module")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/python/stat-cpi.py | 19 ++++++++++---------
.../perf/tests/shell/test_stat_cpi_python.sh | 12 +++++++++---
2 files changed, 19 insertions(+), 12 deletions(-)
diff --git a/tools/perf/python/stat-cpi.py b/tools/perf/python/stat-cpi.py
index 0b7d76876a6c..da92cf560067 100755
--- a/tools/perf/python/stat-cpi.py
+++ b/tools/perf/python/stat-cpi.py
@@ -106,7 +106,8 @@ class StatCpiAnalyzer:
if ins != 0:
cpi = cyc / float(ins)
t_sec = timestamp / 1000000000.0
- print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})")
+ print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})",
+ flush=True)
def read_counters(self, evlist: Any) -> None:
"""Read counters live."""
@@ -173,19 +174,19 @@ class StatCpiAnalyzer:
def run_live(self) -> None:
"""Read counters live."""
- try:
- evlist = self._open_live_evlist()
- except OSError as e:
- print(f"Failed to open events: {e}", file=sys.stderr)
- sys.exit(1)
-
def handle_signal(_signum: int, _frame: Any) -> None:
raise KeyboardInterrupt
signal.signal(signal.SIGINT, signal.default_int_handler)
signal.signal(signal.SIGTERM, handle_signal)
- print("Live mode started. Press Ctrl+C to stop.")
+ try:
+ evlist = self._open_live_evlist()
+ except OSError as e:
+ print(f"Failed to open events: {e}", file=sys.stderr)
+ sys.exit(1)
+
+ print("Live mode started. Press Ctrl+C to stop.", flush=True)
try:
while True:
time.sleep(self.args.interval)
@@ -195,7 +196,7 @@ class StatCpiAnalyzer:
self.data.clear()
self.recorded_pairs.clear()
except KeyboardInterrupt:
- print("\nStopped.")
+ print("\nStopped.", flush=True)
finally:
evlist.close()
diff --git a/tools/perf/tests/shell/test_stat_cpi_python.sh b/tools/perf/tests/shell/test_stat_cpi_python.sh
index fe7562307634..6cb376c92e2f 100755
--- a/tools/perf/tests/shell/test_stat_cpi_python.sh
+++ b/tools/perf/tests/shell/test_stat_cpi_python.sh
@@ -50,7 +50,7 @@ test_live_mode() {
echo "perf stat failed (permissions?), skipping live mode test."
return 0
fi
- perf test -w noploop &
+ perf test -w noploop 60 &
workload_pid=$!
if ! perf stat -e cycles,instructions -p "$workload_pid" -- sleep 0.05 2>/dev/null && \
! perf stat -e cycles:u,instructions:u -p "$workload_pid" -- sleep 0.05 2>/dev/null; then
@@ -61,10 +61,16 @@ test_live_mode() {
fi
ran=1
- # Run live mode for 1 interval in the background, give it a tiny sleep, then interrupt
+ # Run live mode in the background, wait until at least one interval is
+ # printed, then interrupt.
perf script stat-cpi -I 0.1 -p "$workload_pid" > "${temp_out}" &
pid=$!
- sleep 0.5
+ for _ in $(seq 1 50); do
+ if grep -q "cpi" "${temp_out}"; then
+ break
+ fi
+ sleep 0.1
+ done
kill -INT "$pid" 2>/dev/null || true
set +e
wait "$pid"
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v1 3/9] perf python stat-cpi: Fix live mode signal races and test flakiness
2026-09-29 2:19 ` [PATCH v1 3/9] perf python stat-cpi: Fix live mode signal races " Ian Rogers
@ 2026-09-29 2:25 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 2:25 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> In test_stat_cpi_python.sh, 'perf test -w noploop &' defaults to a
> 1-second duration and can exit under heavy parallel load before
> 'perf stat -p' and 'perf script stat-cpi' finish starting up. In
> addition, the fixed 'sleep 0.5' before sending SIGINT can fire before
> Python finishes importing the perf module, opening the live evlist, and
> flushing the first interval.
>
> In stat-cpi.py, register SIGINT and SIGTERM handlers before calling
> _open_live_evlist() and pass flush=True when printing live output so
> redirected stdout is flushed immediately after each interval.
>
> In test_stat_cpi_python.sh, run 'perf test -w noploop 60 &' so the
> target workload stays alive until killed, and poll the output file for
> 'cpi' (up to 5 seconds) before sending SIGINT.
>
> Fixes: 4425182d426b ("perf python: Port stat-cpi to perf module")
> Assisted-by: Antigravity:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929021913.56010-1-irogers@google.com?part=3
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v1 4/9] perf test: Deflake Intel PT Python shell tests under load
2026-09-29 2:19 [PATCH v1 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (2 preceding siblings ...)
2026-09-29 2:19 ` [PATCH v1 3/9] perf python stat-cpi: Fix live mode signal races " Ian Rogers
@ 2026-09-29 2:19 ` Ian Rogers
2026-09-29 2:24 ` sashiko-bot
2026-09-29 2:19 ` [PATCH v1 5/9] perf test: Deflake failed-syscalls " Ian Rogers
` (5 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 2:19 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim
Cc: Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark,
linux-perf-users, linux-kernel
In test_intel_pt_events_python.sh, test_export_to_sqlite_python.sh, and
test_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the
shell builtin 'true' and exits within microseconds of 'uname', sending
SIGCHLD to 'perf record' before the Intel PT AUX buffer is always
flushed under heavy parallel load (~5-10% drop rate).
Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"')
so 'uname' completely exits and flushes its AUX trace before 'sh' exits,
and wrap the record and verification step in a bounded retry loop (up to
5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in
test_export_to_sqlite_python.sh and test_export_to_postgresql_python.sh
to avoid build-id cache and BPF synthesis overhead.
Fixes: d4ce72e9e238 ("perf python: Port intel-pt-events and libxed to perf module")
Fixes: 62d350135e67 ("perf python: Port export-to-sqlite to perf module")
Fixes: b1f968c9656a ("perf python: Port export-to-postgresql to perf module")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
.../shell/test_export_to_postgresql_python.sh | 49 ++++++++++--------
.../shell/test_export_to_sqlite_python.sh | 50 +++++++++++--------
.../shell/test_intel_pt_events_python.sh | 43 +++++++++-------
3 files changed, 79 insertions(+), 63 deletions(-)
diff --git a/tools/perf/tests/shell/test_export_to_postgresql_python.sh b/tools/perf/tests/shell/test_export_to_postgresql_python.sh
index 350813466700..835f48f002e9 100755
--- a/tools/perf/tests/shell/test_export_to_postgresql_python.sh
+++ b/tools/perf/tests/shell/test_export_to_postgresql_python.sh
@@ -61,9 +61,10 @@ test_file_mode() {
fi
# Generate events with callchains and context switches
- if ! perf record -g --switch-events -o "${temp_data}" \
+ if ! perf record -B -N --no-bpf-event -g --switch-events -o "${temp_data}" \
-- perf test -w noploop >/dev/null 2>&1 && \
- ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+ ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping test, perf record failed"
exit 2
fi
@@ -92,29 +93,33 @@ test_file_mode() {
test_intel_pt() {
echo "Testing export-to-postgresql.py with intel_pt..."
- psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true
- rm -f "${temp_data}"
- # Generate some intel_pt events; use a subshell that waits for uname
- if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
- -- sh -c "uname; true" >/dev/null 2>&1; then
- echo "Skipping intel_pt test, intel_pt not available."
- return 0
- fi
+ # Generate some intel_pt events; sleep briefly after uname in the subshell
+ # so uname's AUX buffer is flushed before SIGCHLD stops perf record.
+ passed=0
+ for _ in 1 2 3 4 5; do
+ psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true
+ rm -f "${temp_data}"
+ if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
+ -- sh -c "uname; sleep 0.05" >/dev/null 2>&1; then
+ echo "Skipping intel_pt test, intel_pt not available."
+ return 0
+ fi
- # Run the script with --itrace cr to synthesize call_returns
- if ! perf script export-to-postgresql -i "${temp_data}" \
- -o "${temp_db}" --itrace cr >/dev/null; then
- echo "intel_pt file mode test failed."
- err=1
- else
- # Check DB for calls
- if ! psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \
+ # Run the script with --itrace cr to synthesize call_returns
+ if perf script export-to-postgresql -i "${temp_data}" \
+ -o "${temp_db}" --itrace cr >/dev/null && \
+ psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \
grep -q '[1-9]'; then
- echo "PostgreSQL intel_pt validation failed (no calls found)."
- err=1
- else
- echo "intel_pt test passed (cr validated)."
+ passed=1
+ break
fi
+ done
+
+ if [ "$passed" -eq 0 ]; then
+ echo "PostgreSQL intel_pt validation failed (no calls found)."
+ err=1
+ else
+ echo "intel_pt test passed (cr validated)."
fi
}
diff --git a/tools/perf/tests/shell/test_export_to_sqlite_python.sh b/tools/perf/tests/shell/test_export_to_sqlite_python.sh
index d3c5e22a0754..19ca7c539cf7 100755
--- a/tools/perf/tests/shell/test_export_to_sqlite_python.sh
+++ b/tools/perf/tests/shell/test_export_to_sqlite_python.sh
@@ -47,9 +47,10 @@ test_file_mode() {
echo "Testing export-to-sqlite.py..."
# Generate events with callchains and context switches if supported
- if ! perf record -g --switch-events -o "${temp_data}" \
+ if ! perf record -B -N --no-bpf-event -g --switch-events -o "${temp_data}" \
-- perf test -w noploop >/dev/null 2>&1 && \
- ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+ ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping test, perf record failed"
exit 2
fi
@@ -77,29 +78,34 @@ test_file_mode() {
test_intel_pt() {
echo "Testing export-to-sqlite.py with intel_pt..."
- rm -f "${temp_db}" "${temp_data}"
- # Generate some intel_pt events; use a subshell that waits for uname
- if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
- -- sh -c "uname; true" >/dev/null 2>&1; then
- echo "Skipping intel_pt test, intel_pt not available."
- return 0
- fi
+ query="import sqlite3; c = sqlite3.connect('${temp_db}'); "
+ query="${query}r = c.execute('SELECT COUNT(*) FROM calls').fetchone()[0]; "
+ query="${query}exit(1 if r == 0 else 0)"
+
+ # Generate some intel_pt events; sleep briefly after uname in the subshell
+ # so uname's AUX buffer is flushed before SIGCHLD stops perf record.
+ passed=0
+ for _ in 1 2 3 4 5; do
+ rm -f "${temp_db}" "${temp_data}"
+ if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
+ -- sh -c "uname; sleep 0.05" >/dev/null 2>&1; then
+ echo "Skipping intel_pt test, intel_pt not available."
+ return 0
+ fi
- # Run the script with --itrace cr to synthesize call_returns
- if ! perf script export-to-sqlite -i "${temp_data}" -o "${temp_db}" --itrace cr; then
- echo "intel_pt file mode test failed."
+ # Run the script with --itrace cr to synthesize call_returns
+ if perf script export-to-sqlite -i "${temp_data}" -o "${temp_db}" --itrace cr && \
+ "$PYTHON" -c "$query" >/dev/null 2>&1; then
+ passed=1
+ break
+ fi
+ done
+
+ if [ "$passed" -eq 0 ]; then
+ echo "SQLite intel_pt validation failed (no calls found)."
err=1
else
- # Check DB for calls
- query="import sqlite3; c = sqlite3.connect('${temp_db}'); "
- query="${query}r = c.execute('SELECT COUNT(*) FROM calls').fetchone()[0]; "
- query="${query}exit(1 if r == 0 else 0)"
- if ! "$PYTHON" -c "$query" >/dev/null 2>&1; then
- echo "SQLite intel_pt validation failed (no calls found)."
- err=1
- else
- echo "intel_pt test passed (cr validated)."
- fi
+ echo "intel_pt test passed (cr validated)."
fi
}
diff --git a/tools/perf/tests/shell/test_intel_pt_events_python.sh b/tools/perf/tests/shell/test_intel_pt_events_python.sh
index b5c3173fa2db..9754b9125da1 100755
--- a/tools/perf/tests/shell/test_intel_pt_events_python.sh
+++ b/tools/perf/tests/shell/test_intel_pt_events_python.sh
@@ -30,7 +30,8 @@ cleanup() {
[ -n "${temp_dir}" ] && rm -rf "${temp_dir}"
}
-trap 'cleanup' EXIT TERM INT
+trap 'cleanup' EXIT
+trap 'cleanup; exit 1' TERM INT
temp_dir=$(mktemp -d /tmp/perf.ipt.XXXXXX)
temp_data="${temp_dir}/perf.data"
@@ -39,27 +40,31 @@ temp_out="${temp_dir}/perf.out"
test_intel_pt() {
echo "Testing intel-pt-events.py with intel_pt..."
- rm -f "${temp_data}" "${temp_out}"
- # Generate some intel_pt events; use a subshell that waits for uname so
- # uname's AUX buffer is flushed before the parent workload exits.
- if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
- -- sh -c "uname; true" >/dev/null 2>&1; then
- echo "Skipping intel_pt test, intel_pt not available."
- exit 2
- fi
+ # Generate some intel_pt events; sleep briefly after uname in the subshell
+ # so uname's AUX buffer is flushed before SIGCHLD stops perf record.
+ passed=0
+ for _ in 1 2 3 4 5; do
+ rm -f "${temp_data}" "${temp_out}"
+ if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
+ -- sh -c "uname; sleep 0.05" >/dev/null 2>&1; then
+ echo "Skipping intel_pt test, intel_pt not available."
+ exit 2
+ fi
- # Run the script and check output
- if ! perf script intel-pt-events -i "${temp_data}" > "${temp_out}"; then
- echo "intel-pt-events.py test failed."
+ # Run the script and check output
+ if perf script intel-pt-events -i "${temp_data}" > "${temp_out}" && \
+ grep -q "Intel PT Branch Trace" "${temp_out}" && \
+ grep -q "uname" "${temp_out}"; then
+ passed=1
+ break
+ fi
+ done
+
+ if [ "$passed" -eq 0 ]; then
+ echo "Failed to find expected output: $(cat "${temp_out}" 2>/dev/null)"
err=1
else
- if ! grep -q "Intel PT Branch Trace" "${temp_out}" || \
- ! grep -q "uname" "${temp_out}"; then
- echo "Failed to find expected output: $(cat "${temp_out}")"
- err=1
- else
- echo "intel-pt-events test passed."
- fi
+ echo "intel-pt-events test passed."
fi
rm -f "${temp_out}"
}
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v1 4/9] perf test: Deflake Intel PT Python shell tests under load
2026-09-29 2:19 ` [PATCH v1 4/9] perf test: Deflake Intel PT Python shell tests under load Ian Rogers
@ 2026-09-29 2:24 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 2:24 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> In test_intel_pt_events_python.sh, test_export_to_sqlite_python.sh, and
> test_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the
> shell builtin 'true' and exits within microseconds of 'uname', sending
> SIGCHLD to 'perf record' before the Intel PT AUX buffer is always
> flushed under heavy parallel load (~5-10% drop rate).
>
> Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"')
> so 'uname' completely exits and flushes its AUX trace before 'sh' exits,
> and wrap the record and verification step in a bounded retry loop (up to
> 5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in
> test_export_to_sqlite_python.sh and test_export_to_postgresql_python.sh
> to avoid build-id cache and BPF synthesis overhead.
>
> Fixes: d4ce72e9e238 ("perf python: Port intel-pt-events and libxed to perf module")
> Fixes: 62d350135e67 ("perf python: Port export-to-sqlite to perf module")
> Fixes: b1f968c9656a ("perf python: Port export-to-postgresql to perf module")
> Assisted-by: Antigravity:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929021913.56010-1-irogers@google.com?part=4
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v1 5/9] perf test: Deflake failed-syscalls Python shell tests under load
2026-09-29 2:19 [PATCH v1 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (3 preceding siblings ...)
2026-09-29 2:19 ` [PATCH v1 4/9] perf test: Deflake Intel PT Python shell tests under load Ian Rogers
@ 2026-09-29 2:19 ` Ian Rogers
2026-09-29 2:25 ` sashiko-bot
2026-09-29 2:19 ` [PATCH v1 6/9] perf test: Reduce overhead and contention in Python shell tests Ian Rogers
` (4 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 2:19 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim
Cc: Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark,
linux-perf-users, linux-kernel
When running 'perf test' in parallel under heavy system load, a one-shot
'perf record -e ... -- ls /does_not_exist' without '-B -N --no-bpf-event'
can exit in under 300 microseconds before 'ls's PERF_RECORD_COMM and
ENOENT sys_exit events are captured, while also contending on ~/.debug
build-id caching and BPF event synthesis.
Pass '-B -N --no-bpf-event' to 'perf record', sleep 0.05s in a subshell
after 'ls' so its events are flushed before the parent subshell exits,
and wrap the record and check steps in a bounded retry loop (up to 5
attempts).
Fixes: b76c43d09b06 ("perf python: Port failed-syscalls-by-pid to perf module")
Fixes: 4e3fe6987cba ("perf python: Port failed-syscalls from Perl to perf module")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
.../test_failed_syscalls_by_pid_python.sh | 68 +++++++++++--------
.../shell/test_failed_syscalls_python.sh | 44 +++++++-----
2 files changed, 67 insertions(+), 45 deletions(-)
diff --git a/tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh b/tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh
index 215372d4e1c5..070a7ef6b8ad 100755
--- a/tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh
+++ b/tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh
@@ -41,8 +41,10 @@ test_file_mode() {
echo "Testing failed-syscalls-by-pid.py..."
# Check if syscalls:sys_exit is supported/readable
- if ! perf record -e syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
- if ! perf record -e raw_syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -e syscalls:sys_exit \
+ -o /dev/null -- true >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_exit \
+ -o /dev/null -- true >/dev/null 2>&1; then
echo "Skipping test, no syscalls:sys_exit or raw_syscalls:sys_exit event"
exit 2
else
@@ -52,41 +54,49 @@ test_file_mode() {
EVENT="syscalls:sys_exit"
fi
- # Generate some events by running a command that should fail at least some syscall
- # (e.g. failing stat on non-existent file).
- # Using '|| true' because 'perf record' returns the exit code of 'ls',
- # which fails with ENOENT
- perf record -e "${EVENT}" -o "${temp_data}" -- ls /does_not_exist >/dev/null 2>&1 || true
+ # Generate some events by running a command that fails a syscall
+ # (e.g. failing stat on non-existent file), sleeping briefly in the
+ # subshell so ls's PERF_RECORD_COMM and sys_exit events are flushed.
+ passed=0
+ for _ in 1 2 3 4 5; do
+ rm -f "${temp_data}" "${temp_out}" "${temp_out}.comm" "${temp_out}.pid"
+ perf record -B -N --no-bpf-event -e "${EVENT}" -o "${temp_data}" \
+ -- sh -c "ls /does_not_exist 2>/dev/null; sleep 0.05 || true" \
+ >/dev/null 2>&1 || true
+ if [ ! -s "${temp_data}" ]; then
+ continue
+ fi
+
+ if perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}" && \
+ grep -n -q "err = ENOENT" "${temp_out}" && \
+ perf script failed-syscalls-by-pid -i "${temp_data}" \
+ "ls" > "${temp_out}.comm" && \
+ grep -q "err = ENOENT" "${temp_out}.comm"; then
+ ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' \
+ "${temp_out}.comm" | head -n 1)
+ if [ -n "${ls_pid}" ] && \
+ perf script failed-syscalls-by-pid -i "${temp_data}" \
+ "${ls_pid}" > "${temp_out}.pid" && \
+ grep -q "err = ENOENT" "${temp_out}.pid"; then
+ passed=1
+ break
+ fi
+ fi
+ done
+
if [ ! -s "${temp_data}" ]; then
echo "Skipping test, perf record failed to create data"
exit 2
fi
- # Run the script and check output
- if ! perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}"; then
+ if [ "$passed" -eq 0 ]; then
echo "failed-syscalls-by-pid test failed."
- err=1
- elif ! grep -n -q "err = ENOENT" "${temp_out}"; then
- echo "Failed to find expected failed syscalls"
- cat "${temp_out}"
- err=1
- elif ! perf script failed-syscalls-by-pid -i "${temp_data}" "ls" > "${temp_out}.comm" || \
- ! grep -q "err = ENOENT" "${temp_out}.comm"; then
- echo "failed-syscalls-by-pid comm filter test failed."
- cat "${temp_out}.comm"
+ cat "${temp_out}" 2>/dev/null || true
+ cat "${temp_out}.comm" 2>/dev/null || true
+ cat "${temp_out}.pid" 2>/dev/null || true
err=1
else
- ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' "${temp_out}.comm" | head -n 1)
- if [ -z "${ls_pid}" ] || \
- ! perf script failed-syscalls-by-pid -i "${temp_data}" \
- "${ls_pid}" > "${temp_out}.pid" || \
- ! grep -q "err = ENOENT" "${temp_out}.pid"; then
- echo "failed-syscalls-by-pid PID filter test failed."
- cat "${temp_out}.pid" 2>/dev/null || true
- err=1
- else
- echo "failed-syscalls-by-pid test passed."
- fi
+ echo "failed-syscalls-by-pid test passed."
fi
rm -f "${temp_out}" "${temp_out}.comm" "${temp_out}.pid"
}
diff --git a/tools/perf/tests/shell/test_failed_syscalls_python.sh b/tools/perf/tests/shell/test_failed_syscalls_python.sh
index 861c2ba71c0a..88a42e3de4f3 100755
--- a/tools/perf/tests/shell/test_failed_syscalls_python.sh
+++ b/tools/perf/tests/shell/test_failed_syscalls_python.sh
@@ -41,8 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
echo "Testing failed-syscalls.py..."
# Check if sys_exit event can be recorded
-if ! perf record -e raw_syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
- if ! perf record -e syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
+if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_exit \
+ -o /dev/null -- true >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -e syscalls:sys_exit \
+ -o /dev/null -- true >/dev/null 2>&1; then
echo "Skipping test, no permission or support for sys_exit event"
exit 2
else
@@ -52,28 +54,38 @@ else
EVENT="raw_syscalls:sys_exit"
fi
-# Run perf record with a command that fails a syscall (ls non-existent file).
-# ls exits with non-zero, so perf record returns non-zero exit code of the workload.
-perf record -e "${EVENT}" -o "${temp_data}" \
- -- ls /nonexistent_file_for_test >/dev/null 2>&1 || true
+# Run perf record with a command that fails a syscall (ls non-existent file),
+# sleeping briefly in the subshell so ls's PERF_RECORD_COMM and sys_exit events are flushed.
+passed=0
+for _ in 1 2 3 4 5; do
+ rm -f "${temp_data}" "${temp_out}"
+ perf record -B -N --no-bpf-event -e "${EVENT}" -o "${temp_data}" \
+ -- sh -c "ls /nonexistent_file_for_test 2>/dev/null; sleep 0.05 || true" \
+ >/dev/null 2>&1 || true
+
+ if [ ! -s "${temp_data}" ]; then
+ continue
+ fi
+
+ # Check that the script executes
+ if perf script failed-syscalls -i "${temp_data}" > "${temp_out}" && \
+ grep -q "failed syscalls by comm" "${temp_out}" && \
+ grep -Eq '^ls[[:space:]]+[0-9]+' "${temp_out}"; then
+ passed=1
+ break
+ fi
+done
if [ ! -s "${temp_data}" ]; then
echo "Skipping test, perf record failed to create data"
exit 2
fi
-# Check that the script executes
-if ! perf script failed-syscalls -i "${temp_data}" > "${temp_out}"; then
- echo "failed-syscalls.py test failed"
+if [ "$passed" -eq 0 ]; then
+ echo "Failed to find the metrics table header or expected error"
err=1
else
- if ! grep -q "failed syscalls by comm" "${temp_out}" || \
- ! grep -Eq '^ls[[:space:]]+[0-9]+' "${temp_out}"; then
- echo "Failed to find the metrics table header or expected error"
- err=1
- else
- echo "failed-syscalls test passed."
- fi
+ echo "failed-syscalls test passed."
fi
rm -f "${temp_out}"
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v1 5/9] perf test: Deflake failed-syscalls Python shell tests under load
2026-09-29 2:19 ` [PATCH v1 5/9] perf test: Deflake failed-syscalls " Ian Rogers
@ 2026-09-29 2:25 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 2:25 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] perf test: TOCTOU symlink vulnerability via rm in /tmp
--
commit 2edcd77bcabafd7550e8af4c57dda0ca7cfe9278
Author: Ian Rogers <irogers@google.com>
perf test: Deflake failed-syscalls Python shell tests under load
This commit updates the failed-syscalls python shell tests to handle
heavy system load by adding a bounded retry loop and short sleep intervals.
It also explicitly passes the no-bpf-event flags to perf record to
prevent early exits during BPF event synthesis.
> diff --git a/tools/perf/tests/shell/test_failed_syscalls_python.sh b/tools/perf/tests/shell/test_failed_syscalls_python.sh
> index 861c2ba71c0a2..88a42e3de4f35 100755
> --- a/tools/perf/tests/shell/test_failed_syscalls_python.sh
> +++ b/tools/perf/tests/shell/test_failed_syscalls_python.sh
[ ... ]
> @@ -52,28 +54,38 @@ else
> EVENT="raw_syscalls:sys_exit"
> fi
>
> -# Run perf record with a command that fails a syscall (ls non-existent file).
> -# ls exits with non-zero, so perf record returns non-zero exit code of the workload.
> -perf record -e "${EVENT}" -o "${temp_data}" \
> - -- ls /nonexistent_file_for_test >/dev/null 2>&1 || true
> +# Run perf record with a command that fails a syscall (ls non-existent file),
> +# sleeping briefly in the subshell so ls's PERF_RECORD_COMM and sys_exit events are flushed.
> +passed=0
> +for _ in 1 2 3 4 5; do
> + rm -f "${temp_data}" "${temp_out}"
[Severity: High]
Does this rm command expose the script to a time-of-check to time-of-use
symlink vulnerability?
The files temp_data and temp_out are initially created securely in the
world-writable /tmp directory using mktemp. By explicitly deleting them
here in the retry loop, the filenames are freed.
During the window between this deletion and the subsequent shell
redirection, a local attacker observing /tmp could recreate these names
as symlinks. When the test redirects output or writes to them later, it
will follow the symlink and could overwrite arbitrary files on the
system. This could lead to local privilege escalation if the test is run
with elevated privileges.
> + perf record -B -N --no-bpf-event -e "${EVENT}" -o "${temp_data}" \
> + -- sh -c "ls /nonexistent_file_for_test 2>/dev/null; sleep 0.05 || true" \
> + >/dev/null 2>&1 || true
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929021913.56010-1-irogers@google.com?part=5
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v1 6/9] perf test: Reduce overhead and contention in Python shell tests
2026-09-29 2:19 [PATCH v1 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (4 preceding siblings ...)
2026-09-29 2:19 ` [PATCH v1 5/9] perf test: Deflake failed-syscalls " Ian Rogers
@ 2026-09-29 2:19 ` Ian Rogers
2026-09-29 2:26 ` sashiko-bot
2026-09-29 2:19 ` [PATCH v1 7/9] perf python event_analyzing_sample: Default to in-memory SQLite database Ian Rogers
` (3 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 2:19 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim
Cc: Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark,
linux-perf-users, linux-kernel
When all Python shell tests run concurrently during 'perf test' under
heavy load:
1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in
each test process; use 'perf list tracepoint' when checking for
tracepoint events.
2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF
events across the system and caches build-ids into ~/.debug; add
'-B -N --no-bpf-event' to all Python shell test recordings.
3. Drop '-a' in test_check_perf_trace_python.sh,
test_rw_by_file_python.sh, test_rw_by_pid_python.sh,
test_rwtop_python.sh, and test_syscall_counts_by_pid_python.sh where
only a single child workload ('dd' or 'sleep') is tested, avoiding
system-wide /proc synthesis and ringbuffer contention.
4. Add '-W 1' to 'ping -c 1' in test_net_dropmonitor_python.sh and
test_netdev_times_python.sh, and narrow 'compaction:*' to
'compaction:mm_compaction_begin,compaction:mm_compaction_end' in
test_compaction_times_python.sh.
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/tests/shell/test_check_perf_trace_python.sh | 8 +++++---
tools/perf/tests/shell/test_compaction_times_python.sh | 8 +++++---
.../tests/shell/test_event_analyzing_sample_python.sh | 3 ++-
tools/perf/tests/shell/test_flamegraph_python.sh | 5 +++--
tools/perf/tests/shell/test_futex_contention_python.sh | 7 ++++---
tools/perf/tests/shell/test_gecko_python.sh | 3 ++-
tools/perf/tests/shell/test_mem_phys_addr_python.sh | 8 +++++---
tools/perf/tests/shell/test_net_dropmonitor_python.sh | 9 +++++----
tools/perf/tests/shell/test_netdev_times_python.sh | 6 +++---
tools/perf/tests/shell/test_powerpc_hcalls_python.sh | 4 ++--
tools/perf/tests/shell/test_rw_by_file_python.sh | 5 +++--
tools/perf/tests/shell/test_rw_by_pid_python.sh | 4 ++--
tools/perf/tests/shell/test_rwtop_python.sh | 4 ++--
tools/perf/tests/shell/test_sched_migration_python.sh | 4 ++--
tools/perf/tests/shell/test_stackcollapse_python.sh | 2 +-
.../tests/shell/test_syscall_counts_by_pid_python.sh | 6 +++---
tools/perf/tests/shell/test_syscall_counts_python.sh | 5 +++--
tools/perf/tests/shell/test_task_analyzer.sh | 3 ++-
tools/perf/tests/shell/test_wakeup_latency_python.sh | 7 ++++---
19 files changed, 58 insertions(+), 43 deletions(-)
diff --git a/tools/perf/tests/shell/test_check_perf_trace_python.sh b/tools/perf/tests/shell/test_check_perf_trace_python.sh
index 92cc7b1f0033..baf55945d429 100755
--- a/tools/perf/tests/shell/test_check_perf_trace_python.sh
+++ b/tools/perf/tests/shell/test_check_perf_trace_python.sh
@@ -45,10 +45,11 @@ test_file_mode() {
echo "Testing check-perf-trace.py..."
events=""
- if perf list | grep -q "irq:softirq_entry"; then
+ tp_list=$(perf list tracepoint)
+ if echo "$tp_list" | grep -q "irq:softirq_entry"; then
events="irq:softirq_entry"
fi
- if perf list | grep -q "kmem:kmalloc"; then
+ if echo "$tp_list" | grep -q "kmem:kmalloc"; then
if [ -n "$events" ]; then
events="$events,kmem:kmalloc,kmem:kfree"
else
@@ -62,7 +63,8 @@ test_file_mode() {
fi
# Generate events
- if ! perf record -e "$events" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -e "$events" -o "${temp_data}" \
+ -- sh -c "sleep 0.1" >/dev/null 2>&1; then
echo "Skipping test, perf record failed"
exit 2
fi
diff --git a/tools/perf/tests/shell/test_compaction_times_python.sh b/tools/perf/tests/shell/test_compaction_times_python.sh
index 50397afaafed..3f5a71e52388 100755
--- a/tools/perf/tests/shell/test_compaction_times_python.sh
+++ b/tools/perf/tests/shell/test_compaction_times_python.sh
@@ -44,15 +44,17 @@ test_file_mode() {
echo "Testing compaction-times.py..."
# Check for any compaction events to see if kernel supports it
- if ! perf list | grep -q "compaction:mm_compaction_begin"; then
+ if ! perf list tracepoint | grep -q "compaction:mm_compaction_begin"; then
echo "Skipping test, compaction tracepoints not found"
exit 2
fi
# Generate some events
- # We might not naturally trigger compaction in 0.5s sleep, but the script
+ # We might not naturally trigger compaction in 0.1s sleep, but the script
# should parse the empty or sparse file correctly without crashing.
- if ! perf record -e "compaction:*" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event \
+ -e "compaction:mm_compaction_begin,compaction:mm_compaction_end" \
+ -a -o "${temp_data}" -- sleep 0.1 >/dev/null 2>&1; then
echo "Skipping test, perf record failed"
exit 2
fi
diff --git a/tools/perf/tests/shell/test_event_analyzing_sample_python.sh b/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
index a7a8ded003db..5f0a41080acf 100755
--- a/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
+++ b/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
@@ -39,7 +39,8 @@ test_file_mode() {
echo "Testing event_analyzing_sample.py..."
# Generate some events
- if ! perf record -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping test, perf record failed"
exit 2
fi
diff --git a/tools/perf/tests/shell/test_flamegraph_python.sh b/tools/perf/tests/shell/test_flamegraph_python.sh
index 86313c72f53e..787d6bbdf04a 100755
--- a/tools/perf/tests/shell/test_flamegraph_python.sh
+++ b/tools/perf/tests/shell/test_flamegraph_python.sh
@@ -60,7 +60,8 @@ test_file_mode() {
echo "Testing flamegraph.py..."
# Generate some events with callchains
- if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping test, perf record -g failed (permissions or lack of support)"
exit 2
fi
@@ -75,7 +76,7 @@ test_file_mode() {
# Run the script in pipe ('-') mode and validate JSON output
rm -f "${temp_json}"
- if ! perf record -g -o - -- perf test -w noploop 2>/dev/null | \
+ if ! perf record -B -N --no-bpf-event -g -o - -- perf test -w noploop 2>/dev/null | \
perf script flamegraph -i - -f json -o "${temp_json}" >/dev/null; then
echo "Pipe stdin JSON mode test failed."
err=1
diff --git a/tools/perf/tests/shell/test_futex_contention_python.sh b/tools/perf/tests/shell/test_futex_contention_python.sh
index a8846ed68ac2..41ce8470a75f 100755
--- a/tools/perf/tests/shell/test_futex_contention_python.sh
+++ b/tools/perf/tests/shell/test_futex_contention_python.sh
@@ -89,14 +89,15 @@ EOF
test_file_mode() {
echo "Testing futex-contention.py..."
# Some systems might not have syscalls:sys_enter_futex
- if ! perf list | grep -q syscalls:sys_enter_futex; then
+ if ! perf list tracepoint | grep -q syscalls:sys_enter_futex; then
echo "Skipping file mode test, syscalls:sys_enter_futex not found"
return
fi
# Generate some futex events
- if ! perf record -e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \
- -- sleep 0.5 2>/dev/null; then
+ if ! perf record -B -N --no-bpf-event \
+ -e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \
+ -- sleep 0.1 2>/dev/null; then
echo "Skipping file mode test (record failed)"
return
fi
diff --git a/tools/perf/tests/shell/test_gecko_python.sh b/tools/perf/tests/shell/test_gecko_python.sh
index de1d7fcf88b7..18e9f5cd2ab0 100755
--- a/tools/perf/tests/shell/test_gecko_python.sh
+++ b/tools/perf/tests/shell/test_gecko_python.sh
@@ -39,7 +39,8 @@ test_file_mode() {
echo "Testing gecko.py..."
# Generate some events with callchains
- if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping test, perf record -g failed (permissions or lack of support)"
exit 2
fi
diff --git a/tools/perf/tests/shell/test_mem_phys_addr_python.sh b/tools/perf/tests/shell/test_mem_phys_addr_python.sh
index 451692d5e089..cbb9f572be9d 100755
--- a/tools/perf/tests/shell/test_mem_phys_addr_python.sh
+++ b/tools/perf/tests/shell/test_mem_phys_addr_python.sh
@@ -78,10 +78,12 @@ test_file_mode() {
echo "Testing mem-phys-addr.py file mode..."
# Generate memory access events (try unprivileged user-space first, then system-wide)
- if ! perf record --phys-data -d -o "${temp_data}" \
+ if ! perf record -B -N --no-bpf-event --phys-data -d -o "${temp_data}" \
-- perf test -w datasym >/dev/null 2>&1 && \
- ! perf record -d -o "${temp_data}" -- perf test -w datasym >/dev/null 2>&1 && \
- ! perf record -d -a -o "${temp_data}" -- sleep 0.2 >/dev/null 2>&1; then
+ ! perf record -B -N --no-bpf-event -d -o "${temp_data}" \
+ -- perf test -w datasym >/dev/null 2>&1 && \
+ ! perf record -B -N --no-bpf-event -d -a -o "${temp_data}" \
+ -- sleep 0.1 >/dev/null 2>&1; then
echo "Skipping file mode record test, perf record -d not supported"
return 0
fi
diff --git a/tools/perf/tests/shell/test_net_dropmonitor_python.sh b/tools/perf/tests/shell/test_net_dropmonitor_python.sh
index 05056a897bc9..2e37994418fc 100755
--- a/tools/perf/tests/shell/test_net_dropmonitor_python.sh
+++ b/tools/perf/tests/shell/test_net_dropmonitor_python.sh
@@ -42,11 +42,12 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
echo "Testing net_dropmonitor.py..."
# Create a perf.data file. Force dropping a packet if tracepoint is available!
-if ! perf record -e skb:kfree_skb -o "${temp_data}" -a \
- -- ping -c 1 255.255.255.255 >/dev/null 2>&1; then
- if ! perf record -e skb:kfree_skb -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" -a \
+ -- ping -c 1 -W 1 255.255.255.255 >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" \
-- sleep 0.1 >/dev/null 2>&1; then
- if ! perf record -o "${temp_data}" -- uname >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -o "${temp_data}" \
+ -- uname >/dev/null 2>&1; then
echo "Skipping test, cannot record perf events"
exit 2
fi
diff --git a/tools/perf/tests/shell/test_netdev_times_python.sh b/tools/perf/tests/shell/test_netdev_times_python.sh
index 393bf446efb2..0d8d10a42cab 100755
--- a/tools/perf/tests/shell/test_netdev_times_python.sh
+++ b/tools/perf/tests/shell/test_netdev_times_python.sh
@@ -85,9 +85,9 @@ then
fi
# Create a perf.data file. Force dropping a packet if tracepoint is available!
-if ! perf record -e skb:kfree_skb -a -o "${temp_data}" \
- -- ping -c 1 127.0.0.1 >/dev/null 2>&1; then
- perf record -e cycles -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -a -o "${temp_data}" \
+ -- ping -c 1 -W 1 127.0.0.1 >/dev/null 2>&1; then
+ perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
-- perf test -w noploop >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
fi
diff --git a/tools/perf/tests/shell/test_powerpc_hcalls_python.sh b/tools/perf/tests/shell/test_powerpc_hcalls_python.sh
index 19569c6d3613..b140ccf00ffa 100755
--- a/tools/perf/tests/shell/test_powerpc_hcalls_python.sh
+++ b/tools/perf/tests/shell/test_powerpc_hcalls_python.sh
@@ -67,8 +67,8 @@ if ! grep -q "H_REMOVE.*1.*1500.*1500.*1500" "${temp_out}"; then
fi
# Create a perf.data file if powerpc hcall tracepoints are available on this host.
-if ! perf record -e powerpc:hcall_entry,powerpc:hcall_exit -a -o "${temp_data}" \
- -- perf test -w noploop >/dev/null 2>&1; then
+if ! perf record -B -N --no-bpf-event -e powerpc:hcall_entry,powerpc:hcall_exit \
+ -a -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping live record test, powerpc hcall tracepoints not available"
exit 0
fi
diff --git a/tools/perf/tests/shell/test_rw_by_file_python.sh b/tools/perf/tests/shell/test_rw_by_file_python.sh
index f121597f418f..e3c538c3ace3 100755
--- a/tools/perf/tests/shell/test_rw_by_file_python.sh
+++ b/tools/perf/tests/shell/test_rw_by_file_python.sh
@@ -37,8 +37,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
echo "Testing rw-by-file.py..."
# Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
- perf record -e syscalls:sys_enter_read,syscalls:sys_enter_write -a -o "${temp_data}" \
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
+ perf record -B -N --no-bpf-event -e syscalls:sys_enter_read,syscalls:sys_enter_write \
+ -o "${temp_data}" \
-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
else
diff --git a/tools/perf/tests/shell/test_rw_by_pid_python.sh b/tools/perf/tests/shell/test_rw_by_pid_python.sh
index 41eaa29097a2..cd93c1d9714c 100755
--- a/tools/perf/tests/shell/test_rw_by_pid_python.sh
+++ b/tools/perf/tests/shell/test_rw_by_pid_python.sh
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
echo "Testing rw-by-pid.py..."
# Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
ev="syscalls:sys_enter_read,syscalls:sys_exit_read"
ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write"
- perf record -e "$ev" -a -o "${temp_data}" \
+ perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \
-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
else
diff --git a/tools/perf/tests/shell/test_rwtop_python.sh b/tools/perf/tests/shell/test_rwtop_python.sh
index 05897e384702..d33807923c92 100755
--- a/tools/perf/tests/shell/test_rwtop_python.sh
+++ b/tools/perf/tests/shell/test_rwtop_python.sh
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
echo "Testing rwtop.py..."
# Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
ev="syscalls:sys_enter_read,syscalls:sys_exit_read"
ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write"
- perf record -e "$ev" -a -o "${temp_data}" \
+ perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \
-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
else
diff --git a/tools/perf/tests/shell/test_sched_migration_python.sh b/tools/perf/tests/shell/test_sched_migration_python.sh
index 75465bcd3791..714275b0e492 100755
--- a/tools/perf/tests/shell/test_sched_migration_python.sh
+++ b/tools/perf/tests/shell/test_sched_migration_python.sh
@@ -44,10 +44,10 @@ echo "Testing sched-migration.py..."
ev="sched:sched_switch,sched:sched_migrate_task"
ev="${ev},sched:sched_wakeup_new,sched:sched_wakeup"
has_sched=1
-if ! perf record -e "$ev" -a -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \
-- sleep 0.1 >/dev/null 2>&1; then
has_sched=0
- perf record -e cycles -o "${temp_data}" \
+ perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
-- perf test -w noploop >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
fi
diff --git a/tools/perf/tests/shell/test_stackcollapse_python.sh b/tools/perf/tests/shell/test_stackcollapse_python.sh
index ce0f80407eec..2048734c5e0d 100755
--- a/tools/perf/tests/shell/test_stackcollapse_python.sh
+++ b/tools/perf/tests/shell/test_stackcollapse_python.sh
@@ -37,7 +37,7 @@ echo "Testing stackcollapse.py..."
# Create a perf.data file with callchains. Use a busy workload rather than
# sleep, as an idle system may not generate any samples at all.
-perf record -g -o "${temp_data}" \
+perf record -B -N --no-bpf-event -g -o "${temp_data}" \
-- perf test -w noploop >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
diff --git a/tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh b/tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh
index 45bbdc554c8f..300289fe33b4 100755
--- a/tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh
+++ b/tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh
@@ -43,14 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
test_file_mode() {
echo "Testing syscall-counts-by-pid.py..."
# Some systems might not have raw_syscalls:sys_enter
- if ! perf list | grep -q raw_syscalls:sys_enter; then
+ if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then
echo "Skipping test, raw_syscalls:sys_enter not found"
exit 2
fi
# Generate some syscall events
- perf record -e raw_syscalls:sys_enter -a -o "${temp_data}" \
- -- sleep 0.5 >/dev/null 2>&1 || \
+ perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \
+ -- sh -c "sleep 0.1; sleep 0.05" >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
if ! perf script syscall-counts-by-pid -i "${temp_data}" > "${temp_out}"; then
diff --git a/tools/perf/tests/shell/test_syscall_counts_python.sh b/tools/perf/tests/shell/test_syscall_counts_python.sh
index 310e05d399f9..86463f14391b 100755
--- a/tools/perf/tests/shell/test_syscall_counts_python.sh
+++ b/tools/perf/tests/shell/test_syscall_counts_python.sh
@@ -43,13 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
test_file_mode() {
echo "Testing syscall-counts.py..."
# Some systems might not have raw_syscalls:sys_enter (e.g. stripped kernels or permissions)
- if ! perf list | grep -q raw_syscalls:sys_enter; then
+ if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then
echo "Skipping test, raw_syscalls:sys_enter not found"
exit 2
fi
# Generate some syscall events
- if ! perf record -e raw_syscalls:sys_enter -o "${temp_data}" -- sleep 0.5 2>/dev/null; then
+ if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \
+ -- sh -c "sleep 0.1; sleep 0.05" 2>/dev/null; then
echo "perf record failed (permissions?), skipping file mode test."
exit 2
fi
diff --git a/tools/perf/tests/shell/test_task_analyzer.sh b/tools/perf/tests/shell/test_task_analyzer.sh
index f99a637a6099..f53fa4a27c79 100755
--- a/tools/perf/tests/shell/test_task_analyzer.sh
+++ b/tools/perf/tests/shell/test_task_analyzer.sh
@@ -60,7 +60,8 @@ skip_no_probe_record_support() {
prepare_perf_data() {
# 1s should be sufficient to catch at least some switches
- perf record -e sched:sched_switch -a -o "${perfdata}" -- sleep 1 > /dev/null 2>&1
+ perf record -B -N --no-bpf-event -e sched:sched_switch -a -o "${perfdata}" \
+ -- sleep 1 > /dev/null 2>&1
# check if perf data file got created in above step.
if [ ! -e "${perfdata}" ]; then
printf "FAIL: perf record failed to create \"${perfdata}\" \n"
diff --git a/tools/perf/tests/shell/test_wakeup_latency_python.sh b/tools/perf/tests/shell/test_wakeup_latency_python.sh
index cb2571449168..b43a86996be2 100755
--- a/tools/perf/tests/shell/test_wakeup_latency_python.sh
+++ b/tools/perf/tests/shell/test_wakeup_latency_python.sh
@@ -41,9 +41,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
echo "Testing wakeup-latency.py..."
# Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "sched:sched_wakeup"; then
+if perf list tracepoint | grep -q "sched:sched_wakeup"; then
ev="sched:sched_wakeup,sched:sched_wakeup_new,sched:sched_switch"
- perf record -e "$ev" -a -o "${temp_data}" \
+ perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \
-- sleep 0.1 >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
else
@@ -70,7 +70,8 @@ else
fi
# Also test zero-wakeups / unhandled events path to verify division-by-zero protection
-if perf record -e cycles -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+if perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
if ! perf script wakeup-latency -i "${temp_data}" > "${temp_out}" || \
! grep -q "avg_wakeup_latency (ns): N/A" "${temp_out}"; then
echo "wakeup-latency zero-wakeups guard test failed"
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v1 6/9] perf test: Reduce overhead and contention in Python shell tests
2026-09-29 2:19 ` [PATCH v1 6/9] perf test: Reduce overhead and contention in Python shell tests Ian Rogers
@ 2026-09-29 2:26 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 2:26 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> When all Python shell tests run concurrently during 'perf test' under
> heavy load:
> 1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in
> each test process; use 'perf list tracepoint' when checking for
> tracepoint events.
> 2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF
> events across the system and caches build-ids into ~/.debug; add
> '-B -N --no-bpf-event' to all Python shell test recordings.
> 3. Drop '-a' in test_check_perf_trace_python.sh,
> test_rw_by_file_python.sh, test_rw_by_pid_python.sh,
> test_rwtop_python.sh, and test_syscall_counts_by_pid_python.sh where
> only a single child workload ('dd' or 'sleep') is tested, avoiding
> system-wide /proc synthesis and ringbuffer contention.
> 4. Add '-W 1' to 'ping -c 1' in test_net_dropmonitor_python.sh and
> test_netdev_times_python.sh, and narrow 'compaction:*' to
> 'compaction:mm_compaction_begin,compaction:mm_compaction_end' in
> test_compaction_times_python.sh.
>
> Assisted-by: Antigravity:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929021913.56010-1-irogers@google.com?part=6
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v1 7/9] perf python event_analyzing_sample: Default to in-memory SQLite database
2026-09-29 2:19 [PATCH v1 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (5 preceding siblings ...)
2026-09-29 2:19 ` [PATCH v1 6/9] perf test: Reduce overhead and contention in Python shell tests Ian Rogers
@ 2026-09-29 2:19 ` Ian Rogers
2026-09-29 2:24 ` sashiko-bot
2026-09-29 2:19 ` [PATCH v1 8/9] perf python: Initialize debug output on module load Ian Rogers
` (2 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 2:19 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim
Cc: Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark,
linux-perf-users, linux-kernel
When -d/--db is not specified, event_analyzing_sample.py created a
temporary file in /tmp via tempfile.mkstemp() and deleted it in
trace_end(). As noted during review, creating the database in a shared
/tmp directory does not reserve SQLite's auxiliary sidecar filenames
(-journal or -wal), and a temporary on-disk file is unnecessary when the
caller did not ask to persist the database.
Default to sqlite3.connect(":memory:") when db_path is not provided,
removing the temporary file creation and cleanup logic, and test both
the default in-memory mode and explicit -d file mode in
test_event_analyzing_sample_python.sh.
Fixes: eeb70645a809 ("perf python: Port event_analyzing_sample to perf module")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/python/event_analyzing_sample.py | 35 ++++++-------------
.../test_event_analyzing_sample_python.sh | 8 ++++-
2 files changed, 17 insertions(+), 26 deletions(-)
diff --git a/tools/perf/python/event_analyzing_sample.py b/tools/perf/python/event_analyzing_sample.py
index 15b47cff9fa3..b4efee88d94b 100755
--- a/tools/perf/python/event_analyzing_sample.py
+++ b/tools/perf/python/event_analyzing_sample.py
@@ -17,10 +17,8 @@ from __future__ import annotations
import argparse
import math
-import os
import sqlite3
import struct
-import tempfile
from typing import Any
import perf
@@ -117,16 +115,11 @@ session: Any = None
class _DB:
con: sqlite3.Connection | None = None
- temp_path: str | None = None
def trace_begin(db_path: str | None = None) -> None:
"""Initialize database tables."""
print("In trace_begin:\n")
- if not db_path:
- fd, db_path = tempfile.mkstemp(prefix="perf_events_", suffix=".db")
- os.close(fd)
- _DB.temp_path = db_path
- con = sqlite3.connect(db_path)
+ con = sqlite3.connect(db_path or ":memory:")
try:
# Drop any pre-existing tables so repeated runs do not accumulate duplicate events.
con.execute("drop table if exists gen_events;")
@@ -297,28 +290,20 @@ def show_pebs_ll() -> None:
def trace_end() -> None:
"""Called at the end of trace processing."""
print("In trace_end:\n")
- try:
- if _DB.con:
- try:
- _DB.con.commit()
- show_general_events()
- show_pebs_ll()
- finally:
- _DB.con.close()
- _DB.con = None
- finally:
- if _DB.temp_path and os.path.exists(_DB.temp_path):
- try:
- os.remove(_DB.temp_path)
- except OSError:
- pass
- _DB.temp_path = None
+ if _DB.con:
+ try:
+ _DB.con.commit()
+ show_general_events()
+ show_pebs_ll()
+ finally:
+ _DB.con.close()
+ _DB.con = None
if __name__ == "__main__":
ap = argparse.ArgumentParser(description="Analyze events with SQLite")
ap.add_argument("-i", "--input", default="perf.data", help="Input file name")
ap.add_argument("-d", "--db", "--database", dest="database", default=None,
- help="Database file name (defaults to a temporary file cleaned up on exit)")
+ help="Database file name (defaults to an in-memory database)")
args = ap.parse_args()
try:
diff --git a/tools/perf/tests/shell/test_event_analyzing_sample_python.sh b/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
index 5f0a41080acf..cddb12b67698 100755
--- a/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
+++ b/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
@@ -45,8 +45,14 @@ test_file_mode() {
exit 2
fi
- # Run the script
+ # Run the script with default (:memory:) database and with explicit -d path
if ! perf script event_analyzing_sample -i "${temp_data}" \
+ > "${temp_dir}/perf.mem.out" 2>&1 || \
+ ! grep -q "Statistics about the general events" "${temp_dir}/perf.mem.out" || \
+ grep -q "Error creating/inserting event" "${temp_dir}/perf.mem.out"; then
+ echo "Default in-memory database mode test failed."
+ err=1
+ elif ! perf script event_analyzing_sample -i "${temp_data}" \
-d "${temp_db}" > "${temp_dir}/perf.out" 2>&1; then
echo "File mode test failed."
err=1
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* [PATCH v1 8/9] perf python: Initialize debug output on module load
2026-09-29 2:19 [PATCH v1 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (6 preceding siblings ...)
2026-09-29 2:19 ` [PATCH v1 7/9] perf python event_analyzing_sample: Default to in-memory SQLite database Ian Rogers
@ 2026-09-29 2:19 ` Ian Rogers
2026-09-29 2:24 ` sashiko-bot
2026-09-29 2:19 ` [PATCH v1 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list Ian Rogers
2026-09-29 6:29 ` [PATCH v2 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 2:19 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim
Cc: Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark,
linux-perf-users, linux-kernel
Unlike the perf binary's main(), PyInit_perf() does not call
perf_debug_setup(). As a result, the first warning or error printed from
the perf Python C extension hits debug_file() with _debug_file == NULL
and emits 'debug_file not set' without a trailing newline before the
actual diagnostic message.
Call perf_debug_setup() in PyInit_perf() and add the missing trailing
newline to the fallback warning in debug_file().
Fixes: ec49230cf6dd ("perf debug: Expose debug file")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/util/debug.c | 2 +-
tools/perf/util/python.c | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/tools/perf/util/debug.c b/tools/perf/util/debug.c
index 6b5ffe81f141..b7095519419e 100644
--- a/tools/perf/util/debug.c
+++ b/tools/perf/util/debug.c
@@ -55,7 +55,7 @@ FILE *debug_file(void)
{
if (!_debug_file) {
debug_set_file(stderr);
- pr_warning_once("debug_file not set");
+ pr_warning_once("%s not set\n", __func__);
}
return _debug_file;
}
diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c
index 690a94fb4f62..95140dfaef9c 100644
--- a/tools/perf/util/python.c
+++ b/tools/perf/util/python.c
@@ -5157,6 +5157,7 @@ PyMODINIT_FUNC PyInit_perf(void)
/* The page_size is placed in util object. */
page_size = sysconf(_SC_PAGE_SIZE);
+ perf_debug_setup();
Py_INCREF(&pyrf_evlist__type);
PyModule_AddObject(module, "evlist", (PyObject *)&pyrf_evlist__type);
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* [PATCH v1 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list
2026-09-29 2:19 [PATCH v1 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (7 preceding siblings ...)
2026-09-29 2:19 ` [PATCH v1 8/9] perf python: Initialize debug output on module load Ian Rogers
@ 2026-09-29 2:19 ` Ian Rogers
2026-09-29 2:27 ` sashiko-bot
2026-09-29 6:29 ` [PATCH v2 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 2:19 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim
Cc: Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark,
linux-perf-users, linux-kernel
perf_pmus__print_pmu_events() first counts events via
perf_pmu__num_events() to allocate the aliases array, then populates it
via perf_pmu__for_each_event(). When dynamic tracepoints (such as
kprobes or uprobes during 'perf test' runs) are concurrently created or
removed between the two passes:
- If an event is removed, state.index is smaller than the allocated len,
leaving trailing zeroed entries in aliases[] with name == NULL and
pmu == NULL, which causes qsort(cmp_sevent) or the print loop to crash
with SIGSEGV.
- If a dynamic tracepoint subsystem directory is removed while scanning
/sys/kernel/tracing/events, tp_pmu__for_each_tp_event() returns
-ENOENT and aborts enumeration of all remaining tracepoint subsystems.
- If an event is added, perf_pmus__print_pmu_events__callback() aborts
enumeration when state->index reaches state->aliases_len.
Grow state->aliases dynamically via realloc() when needed, use
state.index as the actual populated length for sorting and printing, and
ignore -ENOENT when a tracepoint subsystem directory disappears during
enumeration.
Fixes: c3245d2093c1 ("perf pmu: Abstract alias/event struct")
Fixes: 45b6e281cb06 ("perf tp_pmu: Add event APIs")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/util/pmus.c | 20 ++++++++++++++++----
tools/perf/util/tp_pmu.c | 8 ++++++--
2 files changed, 22 insertions(+), 6 deletions(-)
diff --git a/tools/perf/util/pmus.c b/tools/perf/util/pmus.c
index e0a4cb2428ca..137abad37c33 100644
--- a/tools/perf/util/pmus.c
+++ b/tools/perf/util/pmus.c
@@ -571,7 +571,7 @@ static int cmp_sevent(const void *a, const void *b)
}
/* Order by event name. */
- return strcmp(as->name, bs->name);
+ return strcmp(as->name ?: "", bs->name ?: "");
}
static bool pmu_alias_is_duplicate(struct sevent *a, struct sevent *b)
@@ -581,7 +581,7 @@ static bool pmu_alias_is_duplicate(struct sevent *a, struct sevent *b)
return false;
/* Don't remove duplicates for different PMUs */
- return strcmp(a->pmu_name, b->pmu_name) == 0;
+ return strcmp(a->pmu_name ?: "", b->pmu_name ?: "") == 0;
}
struct events_callback_state {
@@ -597,8 +597,18 @@ static int perf_pmus__print_pmu_events__callback(void *vstate,
struct sevent *s;
if (state->index >= state->aliases_len) {
- pr_err("Unexpected event %s/%s/\n", info->pmu->name, info->name);
- return 1;
+ size_t new_len = max_t(size_t, 16, state->aliases_len * 2);
+ struct sevent *new_aliases;
+
+ new_aliases = realloc(state->aliases, new_len * sizeof(struct sevent));
+ if (!new_aliases) {
+ pr_err("Unexpected event %s/%s/\n", info->pmu->name, info->name);
+ return 1;
+ }
+ memset(&new_aliases[state->aliases_len], 0,
+ (new_len - state->aliases_len) * sizeof(struct sevent));
+ state->aliases = new_aliases;
+ state->aliases_len = new_len;
}
assert(info->pmu != NULL || info->name != NULL);
s = &state->aliases[state->index];
@@ -654,6 +664,8 @@ void perf_pmus__print_pmu_events(const struct print_callbacks *print_cb, void *p
perf_pmu__for_each_event(pmu, skip_duplicate_pmus, &state,
perf_pmus__print_pmu_events__callback);
}
+ aliases = state.aliases;
+ len = state.index;
qsort(aliases, len, sizeof(struct sevent), cmp_sevent);
for (int j = 0; j < len; j++) {
/* Skip duplicates */
diff --git a/tools/perf/util/tp_pmu.c b/tools/perf/util/tp_pmu.c
index c2be8c9f9084..5ac732e06841 100644
--- a/tools/perf/util/tp_pmu.c
+++ b/tools/perf/util/tp_pmu.c
@@ -151,7 +151,9 @@ static int for_each_event_cb(void *state, const char *sys_name, const char *evt_
static int for_each_event_sys_cb(void *state, const char *sys_name)
{
- return tp_pmu__for_each_tp_event(sys_name, state, for_each_event_cb);
+ int ret = tp_pmu__for_each_tp_event(sys_name, state, for_each_event_cb);
+
+ return ret == -ENOENT ? 0 : ret;
}
int tp_pmu__for_each_event(struct perf_pmu *pmu, void *state, pmu_event_callback cb)
@@ -176,7 +178,9 @@ static int num_events_cb(void *state, const char *sys_name __maybe_unused,
static int num_events_sys_cb(void *state, const char *sys_name)
{
- return tp_pmu__for_each_tp_event(sys_name, state, num_events_cb);
+ int ret = tp_pmu__for_each_tp_event(sys_name, state, num_events_cb);
+
+ return ret == -ENOENT ? 0 : ret;
}
size_t tp_pmu__num_events(struct perf_pmu *pmu __maybe_unused)
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v1 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list
2026-09-29 2:19 ` [PATCH v1 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list Ian Rogers
@ 2026-09-29 2:27 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 2:27 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] perf: missing explicit <stdlib.h> inclusion for realloc()
--
commit 156bff4d99dfcb5bfd8c7cc38a4671357c5113e3
Author: Ian Rogers <irogers@google.com>
perf pmu: Fix race with concurrent tracepoint creation and removal in perf list
This commit modifies the perf tool to handle concurrent addition and removal of
tracepoint events by dynamically reallocating the aliases array and handling
-ENOENT gracefully.
> diff --git a/tools/perf/util/pmus.c b/tools/perf/util/pmus.c
> index e0a4cb2428ca4..137abad37c33b 100644
> --- a/tools/perf/util/pmus.c
> +++ b/tools/perf/util/pmus.c
[ ... ]
> @@ -597,8 +597,18 @@ static int perf_pmus__print_pmu_events__callback(void *vstate,
> struct sevent *s;
>
> if (state->index >= state->aliases_len) {
> - pr_err("Unexpected event %s/%s/\n", info->pmu->name, info->name);
> - return 1;
> + size_t new_len = max_t(size_t, 16, state->aliases_len * 2);
> + struct sevent *new_aliases;
> +
> + new_aliases = realloc(state->aliases, new_len * sizeof(struct sevent));
[Severity: Low]
Does this code in perf_pmus__print_pmu_events__callback() rely on implicit
header inclusions for realloc()?
Since the perf tool can be built against musl libc, which strictly separates
declarations and does not implicitly include <stdlib.h> through other
headers, should <stdlib.h> be explicitly included at the top of
tools/perf/util/pmus.c to prevent compilation failures?
> + if (!new_aliases) {
> + pr_err("Unexpected event %s/%s/\n", info->pmu->name, info->name);
> + return 1;
> + }
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929021913.56010-1-irogers@google.com?part=9
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v2 0/9] perf python: Follow-up fixes and shell test deflaking
2026-09-29 2:19 [PATCH v1 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (8 preceding siblings ...)
2026-09-29 2:19 ` [PATCH v1 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list Ian Rogers
@ 2026-09-29 6:29 ` Ian Rogers
2026-09-29 6:29 ` [PATCH v2 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events Ian Rogers
` (9 more replies)
9 siblings, 10 replies; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
This series contains follow-up fixes for the standalone Python scripts
and their shell tests, particularly when running 'perf test' as root
under heavy parallel load:
- Clamp sample_id size calculation for PERF_RECORD_KSYMBOL,
PERF_RECORD_BPF_EVENT, and PERF_RECORD_TEXT_POKE records to work
around kernels without "perf/core: Restore header fields in sideband
output callbacks" [1].
- Fix offline interval printing in sctop.py and live signal races in
stat-cpi.py.
- Deflake Intel PT, failed-syscalls, and other Python shell tests under
parallel 'perf test' load by scoping tracepoint checks to
'perf list tracepoint', passing '-B -N --no-bpf-event' to 'perf record',
and avoiding unnecessary system-wide ('-a') recordings.
- Default event_analyzing_sample.py to an in-memory SQLite database
(':memory:') when '--db' is not specified.
- Initialize perf debug output ('perf_debug_setup()') when loading the
'perf' Python extension module.
- Fix races in 'perf list' ('perf_pmus__print_pmu_events()' and
'tp_pmu.c') when concurrent tests dynamically create and remove
kprobe/uprobes tracepoints.
v2:
- Fix 8-byte alignment calculation for PERF_RECORD_TEXT_POKE payload in
evsel__event_size() by aligning fixed + old_len + new_len.
- Keep offline interval clock advancing on all samples in sctop.py while
flushing any remaining syscalls at EOF.
- Use 'mktemp -d' private directories in test_sctop_python.sh and
test_failed_syscalls_python.sh so temporary files are not unlinked and
recreated directly in /tmp.
- Explicitly include <stdlib.h> in tools/perf/util/pmus.c.
[1] https://lore.kernel.org/r/20260929014206.4175245-1-irogers@google.com
Ian Rogers (9):
perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke
events
perf python sctop: Fix offline interval printing and test flakiness
perf python stat-cpi: Fix live mode signal races and test flakiness
perf test: Deflake Intel PT Python shell tests under load
perf test: Deflake failed-syscalls Python shell tests under load
perf test: Reduce overhead and contention in Python shell tests
perf python event_analyzing_sample: Default to in-memory SQLite
database
perf python: Initialize debug output on module load
perf pmu: Fix race with concurrent tracepoint creation and removal in
perf list
tools/perf/python/event_analyzing_sample.py | 35 +++-------
tools/perf/python/sctop.py | 7 +-
tools/perf/python/stat-cpi.py | 19 +++---
.../shell/test_check_perf_trace_python.sh | 8 ++-
.../shell/test_compaction_times_python.sh | 8 ++-
.../test_event_analyzing_sample_python.sh | 11 ++-
.../shell/test_export_to_postgresql_python.sh | 49 +++++++------
.../shell/test_export_to_sqlite_python.sh | 50 ++++++++------
.../test_failed_syscalls_by_pid_python.sh | 68 +++++++++++--------
.../shell/test_failed_syscalls_python.sh | 54 +++++++++------
.../tests/shell/test_flamegraph_python.sh | 5 +-
.../shell/test_futex_contention_python.sh | 7 +-
tools/perf/tests/shell/test_gecko_python.sh | 3 +-
.../shell/test_intel_pt_events_python.sh | 43 ++++++------
.../tests/shell/test_mem_phys_addr_python.sh | 8 ++-
.../shell/test_net_dropmonitor_python.sh | 9 +--
.../tests/shell/test_netdev_times_python.sh | 6 +-
.../tests/shell/test_powerpc_hcalls_python.sh | 4 +-
.../tests/shell/test_rw_by_file_python.sh | 5 +-
.../perf/tests/shell/test_rw_by_pid_python.sh | 4 +-
tools/perf/tests/shell/test_rwtop_python.sh | 4 +-
.../shell/test_sched_migration_python.sh | 4 +-
tools/perf/tests/shell/test_sctop_python.sh | 58 ++++++++--------
.../tests/shell/test_stackcollapse_python.sh | 2 +-
.../perf/tests/shell/test_stat_cpi_python.sh | 12 +++-
.../test_syscall_counts_by_pid_python.sh | 6 +-
.../tests/shell/test_syscall_counts_python.sh | 5 +-
tools/perf/tests/shell/test_task_analyzer.sh | 3 +-
.../tests/shell/test_wakeup_latency_python.sh | 7 +-
tools/perf/util/debug.c | 2 +-
tools/perf/util/evlist.c | 6 +-
tools/perf/util/evsel.c | 59 +++++++++++++++-
tools/perf/util/evsel.h | 1 +
tools/perf/util/pmus.c | 21 ++++--
tools/perf/util/python.c | 1 +
tools/perf/util/tp_pmu.c | 8 ++-
36 files changed, 366 insertions(+), 236 deletions(-)
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 58+ messages in thread* [PATCH v2 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events
2026-09-29 6:29 ` [PATCH v2 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
@ 2026-09-29 6:29 ` Ian Rogers
2026-09-29 6:41 ` sashiko-bot
2026-09-29 6:29 ` [PATCH v2 2/9] perf python sctop: Fix offline interval printing and test flakiness Ian Rogers
` (8 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
On kernels without the fix ("perf/core: Restore header fields in
sideband output callbacks") to restore event_id.header.size in
perf_event_ksymbol_output(), perf_event_bpf_output(), and
perf_event_text_poke_output(), concurrent perf sessions cause those
sideband records to be emitted with header.size inflated by multiple
id_header_size increments while the single id_sample is written
immediately after the event payload. Indexing backwards from
event->header.size reads uninitialized ring-buffer bytes at the end of
the record, causing evlist__event2evsel() to fail with -EFAULT.
Add evsel__event_size() to clamp the effective size used to locate the
trailing id_sample for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and
PERF_RECORD_TEXT_POKE to payload + id_hdr_size while leaving
event->header.size intact for advancing the ring-buffer/file stream.
Fixes: 9aa0bfa370b2 ("perf tools: Handle PERF_RECORD_KSYMBOL")
Fixes: 45178a928a4b ("perf tools: Handle PERF_RECORD_BPF_EVENT")
Fixes: 246eba8e9041 ("perf tools: Add support for PERF_RECORD_TEXT_POKE")
Link: https://lore.kernel.org/r/20260929014206.4175245-1-irogers@google.com
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/util/evlist.c | 6 ++--
tools/perf/util/evsel.c | 59 +++++++++++++++++++++++++++++++++++++++-
tools/perf/util/evsel.h | 1 +
3 files changed, 63 insertions(+), 3 deletions(-)
diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index 9392d912d254..c2402e4791b6 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -973,13 +973,15 @@ static int evlist__event2id(struct evlist *evlist, union perf_event *event, u64
const __u64 *array = event->sample.array;
ssize_t n;
- n = (event->header.size - sizeof(event->header)) >> 3;
-
if (event->header.type == PERF_RECORD_SAMPLE) {
+ n = (event->header.size - sizeof(event->header)) >> 3;
if (evlist__id_pos(evlist) >= n)
return -1;
*id = array[evlist__id_pos(evlist)];
} else {
+ u16 size = evsel__event_size(evlist__first(evlist), event);
+
+ n = (size - sizeof(event->header)) >> 3;
if (evlist__is_pos(evlist) > n)
return -1;
n -= evlist__is_pos(evlist);
diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index 3367242c5764..9c5e7510f0c0 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -3216,7 +3216,7 @@ static int perf_evsel__parse_id_sample(const union perf_event *event,
const __u64 *array = event->sample.array;
bool swapped = evsel->needs_swap;
union u64_swap u;
- int i = ((event->header.size - sizeof(event->header)) / sizeof(u64)) - 1;
+ int i = ((evsel__event_size(evsel, event) - sizeof(event->header)) / sizeof(u64)) - 1;
if (type & PERF_SAMPLE_IDENTIFIER) {
if (i < 0)
@@ -3966,6 +3966,63 @@ u16 evsel__id_hdr_size(const struct evsel *evsel)
return size;
}
+/*
+ * Prior to kernel fix, perf_event_ksymbol_output(), perf_event_bpf_output(),
+ * and perf_event_text_poke_output() in kernel/events/core.c did not save and
+ * restore event_id.header.size across perf_iterate_sb() iterations. When
+ * multiple perf_events had attr.ksymbol, attr.bpf_event, or attr.text_poke
+ * enabled, header.size was incremented by id_header_size for each matching
+ * event while only a single id_sample was written immediately after the event
+ * payload. Clamp the effective size used to locate the trailing id_sample to
+ * payload + id_hdr_size so events recorded on unpatched kernels can be parsed
+ * without reading uninitialized ring-buffer bytes.
+ */
+u16 evsel__event_size(const struct evsel *evsel, const union perf_event *event)
+{
+ u16 size = event->header.size;
+ u16 id_hdr_size;
+ size_t payload;
+
+ if (!evsel->core.attr.sample_id_all)
+ return size;
+
+ switch (event->header.type) {
+ case PERF_RECORD_KSYMBOL: {
+ const char *name = event->ksymbol.name;
+ size_t fixed = offsetof(struct perf_record_ksymbol, name);
+ size_t max_len, len;
+
+ if (size <= fixed)
+ return size;
+ max_len = size - fixed;
+ len = strnlen(name, max_len);
+ if (len == max_len)
+ return size;
+ payload = fixed + PERF_ALIGN(len + 1, sizeof(u64));
+ break;
+ }
+ case PERF_RECORD_BPF_EVENT:
+ payload = sizeof(struct perf_record_bpf_event);
+ break;
+ case PERF_RECORD_TEXT_POKE: {
+ size_t fixed = offsetof(struct perf_record_text_poke_event, bytes);
+
+ if (size < fixed)
+ return size;
+ payload = PERF_ALIGN(fixed + (size_t)event->text_poke.old_len +
+ event->text_poke.new_len, sizeof(u64));
+ break;
+ }
+ default:
+ return size;
+ }
+
+ id_hdr_size = evsel__id_hdr_size(evsel);
+ if (payload + id_hdr_size < size)
+ return payload + id_hdr_size;
+ return size;
+}
+
#ifdef HAVE_LIBTRACEEVENT
struct tep_format_field *evsel__field(struct evsel *evsel, const char *name)
{
diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
index 5c5799cee601..174f3414fd3c 100644
--- a/tools/perf/util/evsel.h
+++ b/tools/perf/util/evsel.h
@@ -469,6 +469,7 @@ int evsel__parse_sample_timestamp(struct evsel *evsel, union perf_event *event,
u64 *timestamp);
u16 evsel__id_hdr_size(const struct evsel *evsel);
+u16 evsel__event_size(const struct evsel *evsel, const union perf_event *event);
static inline struct evsel *evsel__next(struct evsel *evsel)
{
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v2 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events
2026-09-29 6:29 ` [PATCH v2 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events Ian Rogers
@ 2026-09-29 6:41 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 6:41 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> On kernels without the fix ("perf/core: Restore header fields in
> sideband output callbacks") to restore event_id.header.size in
> perf_event_ksymbol_output(), perf_event_bpf_output(), and
> perf_event_text_poke_output(), concurrent perf sessions cause those
> sideband records to be emitted with header.size inflated by multiple
> id_header_size increments while the single id_sample is written
> immediately after the event payload. Indexing backwards from
> event->header.size reads uninitialized ring-buffer bytes at the end of
> the record, causing evlist__event2evsel() to fail with -EFAULT.
>
> Add evsel__event_size() to clamp the effective size used to locate the
> trailing id_sample for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and
> PERF_RECORD_TEXT_POKE to payload + id_hdr_size while leaving
> event->header.size intact for advancing the ring-buffer/file stream.
>
> Fixes: 9aa0bfa370b2 ("perf tools: Handle PERF_RECORD_KSYMBOL")
> Fixes: 45178a928a4b ("perf tools: Handle PERF_RECORD_BPF_EVENT")
> Fixes: 246eba8e9041 ("perf tools: Add support for PERF_RECORD_TEXT_POKE")
> Link: https://lore.kernel.org/r/20260929014206.4175245-1-irogers@google.com
> Assisted-by: Antigravity:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929062937.374137-1-irogers@google.com?part=1
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v2 2/9] perf python sctop: Fix offline interval printing and test flakiness
2026-09-29 6:29 ` [PATCH v2 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
2026-09-29 6:29 ` [PATCH v2 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events Ian Rogers
@ 2026-09-29 6:29 ` Ian Rogers
2026-09-29 6:37 ` sashiko-bot
2026-09-29 6:29 ` [PATCH v2 3/9] perf python stat-cpi: Fix live mode signal races " Ian Rogers
` (7 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
In sctop.py:
- If an earlier interval elapsed and printed an empty table before the
target comm ('sleep') executed any syscalls, analyzer.printed became
True and the final partial interval containing the target comm's
syscalls was never flushed at EOF. Flush print_current_totals() in
finally when analyzer.syscalls is non-empty as well as when nothing
has been printed yet.
- Initialize analyzer.e_machine after creating perf.session rather than
when session is still None.
In test_sctop_python.sh:
- Use a private temporary directory via 'mktemp -d'.
- Drop '-a' and pass '-B -N --no-bpf-event' to 'perf record', and sleep
briefly in the subshell ('sh -c "sleep 0.1; sleep 0.05"') with a
bounded retry loop so 'sleep's PERF_RECORD_COMM and sys_enter events
are reliably captured under heavy load.
Fixes: b83f0bacf5e4 ("perf python: Port sctop to perf module")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/python/sctop.py | 7 ++-
tools/perf/tests/shell/test_sctop_python.sh | 58 ++++++++++-----------
2 files changed, 34 insertions(+), 31 deletions(-)
diff --git a/tools/perf/python/sctop.py b/tools/perf/python/sctop.py
index 42e95ecfffad..48320af8b755 100755
--- a/tools/perf/python/sctop.py
+++ b/tools/perf/python/sctop.py
@@ -37,6 +37,7 @@ class SCTopAnalyzer:
self.offline = offline
self.own_pid = os.getpid()
self.last_print_time: Optional[int] = None
+ self.printed = False
self.session: Optional[perf.session] = None
self.e_machine: Optional[int] = None
@@ -137,6 +138,7 @@ class SCTopAnalyzer:
def print_current_totals(self):
"""Print current syscall totals."""
+ self.printed = True
# Clear terminal
if not self.offline:
print("\x1b[2J\x1b[H", end="")
@@ -217,8 +219,8 @@ def main():
if args.input:
session = perf.session(perf.data(args.input), sample=analyzer.process_event)
analyzer.session = session
- session.process_events()
analyzer.e_machine = getattr(session, "e_machine", None)
+ session.process_events()
else:
try:
live_session = LiveSession(
@@ -237,7 +239,8 @@ def main():
sys.exit(1)
finally:
if args.input:
- analyzer.print_current_totals()
+ if not analyzer.printed or analyzer.syscalls:
+ analyzer.print_current_totals()
# Break the reference cycle between perf.session and analyzer.process_event
# because perf.session lacks cyclic GC support (tp_traverse).
analyzer.session = None
diff --git a/tools/perf/tests/shell/test_sctop_python.sh b/tools/perf/tests/shell/test_sctop_python.sh
index 007f2584cce6..b042fc3eefe5 100755
--- a/tools/perf/tests/shell/test_sctop_python.sh
+++ b/tools/perf/tests/shell/test_sctop_python.sh
@@ -27,51 +27,51 @@ if [ ! -f "$script_path" ]; then
fi
err=0
-temp_data=""
-temp_out=""
+temp_dir=$(mktemp -d /tmp/perf-sctop-XXXXXX)
+temp_data="${temp_dir}/perf.data"
+temp_out="${temp_dir}/perf.out"
cleanup() {
- rm -f "${temp_data}" "${temp_out}"
+ rm -rf "${temp_dir}"
}
trap 'cleanup' EXIT TERM INT
-temp_data=$(mktemp /tmp/perf.data.XXXXXX)
-temp_out=$(mktemp /tmp/perf.out.XXXXXX)
-
echo "Testing sctop.py..."
# Create a perf.data file.
-if perf list | grep -q "raw_syscalls:sys_enter"; then
- perf record -e raw_syscalls:sys_enter -a -o "${temp_data}" \
- -- sleep 0.1 >/dev/null 2>&1 || \
- { echo "Skipping test, perf record failed"; exit 2; }
-else
+if ! perf list tracepoint | grep -q "raw_syscalls:sys_enter"; then
echo "Skipping test, no raw_syscalls:sys_enter event"
exit 2
fi
-if [ ! -s "${temp_data}" ]; then
- echo "Skipping test, perf record failed to create data"
- exit 2
-fi
+passed=0
+for _ in 1 2 3 4 5; do
+ rm -f "${temp_data}" "${temp_out}"
+ if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \
+ -- sh -c "sleep 0.1; sleep 0.05" >/dev/null 2>&1; then
+ echo "Skipping test, perf record failed"
+ exit 2
+ fi
-# Check that the script executes
-if ! perf script sctop -i "${temp_data}" > "${temp_out}"; then
+ if [ ! -s "${temp_data}" ]; then
+ continue
+ fi
+
+ # Check that the script executes
+ if perf script sctop -i "${temp_data}" > "${temp_out}" && \
+ grep -E -q "[0-9]+$" "${temp_out}" && \
+ perf script sctop -i "${temp_data}" sleep 1 > "${temp_out}" && \
+ grep -E -q "[0-9]+$" "${temp_out}"; then
+ passed=1
+ break
+ fi
+done
+
+if [ "$passed" -eq 0 ]; then
echo "sctop.py test failed"
err=1
-elif ! grep -E -q "[0-9]+$" "${temp_out}"; then
- echo "Failed to find metric data rows in default run"
- err=1
-elif ! perf script sctop -i "${temp_data}" sleep 1 > "${temp_out}"; then
- echo "sctop.py comm+interval test failed"
- err=1
else
- if ! grep -E -q "[0-9]+$" "${temp_out}"; then
- echo "Failed to find metric data rows"
- err=1
- else
- echo "sctop test passed."
- fi
+ echo "sctop test passed."
fi
rm -f "${temp_out}"
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v2 2/9] perf python sctop: Fix offline interval printing and test flakiness
2026-09-29 6:29 ` [PATCH v2 2/9] perf python sctop: Fix offline interval printing and test flakiness Ian Rogers
@ 2026-09-29 6:37 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 6:37 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> In sctop.py:
> - If an earlier interval elapsed and printed an empty table before the
> target comm ('sleep') executed any syscalls, analyzer.printed became
> True and the final partial interval containing the target comm's
> syscalls was never flushed at EOF. Flush print_current_totals() in
> finally when analyzer.syscalls is non-empty as well as when nothing
> has been printed yet.
> - Initialize analyzer.e_machine after creating perf.session rather than
> when session is still None.
>
> In test_sctop_python.sh:
> - Use a private temporary directory via 'mktemp -d'.
> - Drop '-a' and pass '-B -N --no-bpf-event' to 'perf record', and sleep
> briefly in the subshell ('sh -c "sleep 0.1; sleep 0.05"') with a
> bounded retry loop so 'sleep's PERF_RECORD_COMM and sys_enter events
> are reliably captured under heavy load.
>
> Fixes: b83f0bacf5e4 ("perf python: Port sctop to perf module")
> Assisted-by: Antigravity:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929062937.374137-1-irogers@google.com?part=2
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v2 3/9] perf python stat-cpi: Fix live mode signal races and test flakiness
2026-09-29 6:29 ` [PATCH v2 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
2026-09-29 6:29 ` [PATCH v2 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events Ian Rogers
2026-09-29 6:29 ` [PATCH v2 2/9] perf python sctop: Fix offline interval printing and test flakiness Ian Rogers
@ 2026-09-29 6:29 ` Ian Rogers
2026-09-29 6:37 ` sashiko-bot
2026-09-29 6:29 ` [PATCH v2 4/9] perf test: Deflake Intel PT Python shell tests under load Ian Rogers
` (6 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
In test_stat_cpi_python.sh, 'perf test -w noploop &' defaults to a
1-second duration and can exit under heavy parallel load before
'perf stat -p' and 'perf script stat-cpi' finish starting up. In
addition, the fixed 'sleep 0.5' before sending SIGINT can fire before
Python finishes importing the perf module, opening the live evlist, and
flushing the first interval.
In stat-cpi.py, register SIGINT and SIGTERM handlers before calling
_open_live_evlist() and pass flush=True when printing live output so
redirected stdout is flushed immediately after each interval.
In test_stat_cpi_python.sh, run 'perf test -w noploop 60 &' so the
target workload stays alive until killed, and poll the output file for
'cpi' (up to 5 seconds) before sending SIGINT.
Fixes: 4425182d426b ("perf python: Port stat-cpi to perf module")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/python/stat-cpi.py | 19 ++++++++++---------
.../perf/tests/shell/test_stat_cpi_python.sh | 12 +++++++++---
2 files changed, 19 insertions(+), 12 deletions(-)
diff --git a/tools/perf/python/stat-cpi.py b/tools/perf/python/stat-cpi.py
index 0b7d76876a6c..da92cf560067 100755
--- a/tools/perf/python/stat-cpi.py
+++ b/tools/perf/python/stat-cpi.py
@@ -106,7 +106,8 @@ class StatCpiAnalyzer:
if ins != 0:
cpi = cyc / float(ins)
t_sec = timestamp / 1000000000.0
- print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})")
+ print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})",
+ flush=True)
def read_counters(self, evlist: Any) -> None:
"""Read counters live."""
@@ -173,19 +174,19 @@ class StatCpiAnalyzer:
def run_live(self) -> None:
"""Read counters live."""
- try:
- evlist = self._open_live_evlist()
- except OSError as e:
- print(f"Failed to open events: {e}", file=sys.stderr)
- sys.exit(1)
-
def handle_signal(_signum: int, _frame: Any) -> None:
raise KeyboardInterrupt
signal.signal(signal.SIGINT, signal.default_int_handler)
signal.signal(signal.SIGTERM, handle_signal)
- print("Live mode started. Press Ctrl+C to stop.")
+ try:
+ evlist = self._open_live_evlist()
+ except OSError as e:
+ print(f"Failed to open events: {e}", file=sys.stderr)
+ sys.exit(1)
+
+ print("Live mode started. Press Ctrl+C to stop.", flush=True)
try:
while True:
time.sleep(self.args.interval)
@@ -195,7 +196,7 @@ class StatCpiAnalyzer:
self.data.clear()
self.recorded_pairs.clear()
except KeyboardInterrupt:
- print("\nStopped.")
+ print("\nStopped.", flush=True)
finally:
evlist.close()
diff --git a/tools/perf/tests/shell/test_stat_cpi_python.sh b/tools/perf/tests/shell/test_stat_cpi_python.sh
index fe7562307634..6cb376c92e2f 100755
--- a/tools/perf/tests/shell/test_stat_cpi_python.sh
+++ b/tools/perf/tests/shell/test_stat_cpi_python.sh
@@ -50,7 +50,7 @@ test_live_mode() {
echo "perf stat failed (permissions?), skipping live mode test."
return 0
fi
- perf test -w noploop &
+ perf test -w noploop 60 &
workload_pid=$!
if ! perf stat -e cycles,instructions -p "$workload_pid" -- sleep 0.05 2>/dev/null && \
! perf stat -e cycles:u,instructions:u -p "$workload_pid" -- sleep 0.05 2>/dev/null; then
@@ -61,10 +61,16 @@ test_live_mode() {
fi
ran=1
- # Run live mode for 1 interval in the background, give it a tiny sleep, then interrupt
+ # Run live mode in the background, wait until at least one interval is
+ # printed, then interrupt.
perf script stat-cpi -I 0.1 -p "$workload_pid" > "${temp_out}" &
pid=$!
- sleep 0.5
+ for _ in $(seq 1 50); do
+ if grep -q "cpi" "${temp_out}"; then
+ break
+ fi
+ sleep 0.1
+ done
kill -INT "$pid" 2>/dev/null || true
set +e
wait "$pid"
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v2 3/9] perf python stat-cpi: Fix live mode signal races and test flakiness
2026-09-29 6:29 ` [PATCH v2 3/9] perf python stat-cpi: Fix live mode signal races " Ian Rogers
@ 2026-09-29 6:37 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 6:37 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf python: unhandled KeyboardInterrupt in stat-cpi init
--
commit 1a071f63d2155dda63d4eb450e190a0e0b0b2876
Author: Ian Rogers <irogers@google.com>
perf python stat-cpi: Fix live mode signal races and test flakiness
This commit attempts to fix live mode signal races by registering signal
handlers earlier and flushing output to prevent test flakiness. It also
modifies the shell test to run the workload longer and poll for output
before sending SIGINT.
> diff --git a/tools/perf/python/stat-cpi.py b/tools/perf/python/stat-cpi.py
> index 0b7d76876a6c6..da92cf560067a 100755
> --- a/tools/perf/python/stat-cpi.py
> +++ b/tools/perf/python/stat-cpi.py
[ ... ]
> @@ -173,19 +174,19 @@ class StatCpiAnalyzer:
>
> def run_live(self) -> None:
> """Read counters live."""
> - try:
> - evlist = self._open_live_evlist()
> - except OSError as e:
> - print(f"Failed to open events: {e}", file=sys.stderr)
> - sys.exit(1)
> -
> def handle_signal(_signum: int, _frame: Any) -> None:
> raise KeyboardInterrupt
>
> signal.signal(signal.SIGINT, signal.default_int_handler)
> signal.signal(signal.SIGTERM, handle_signal)
>
> - print("Live mode started. Press Ctrl+C to stop.")
> + try:
> + evlist = self._open_live_evlist()
> + except OSError as e:
> + print(f"Failed to open events: {e}", file=sys.stderr)
> + sys.exit(1)
> +
> + print("Live mode started. Press Ctrl+C to stop.", flush=True)
> try:
> while True:
> time.sleep(self.args.interval)
[Severity: Medium]
Will this crash if a signal is received during _open_live_evlist()?
The signal handlers are registered to raise KeyboardInterrupt before
_open_live_evlist() is called, but the try...except KeyboardInterrupt
block below only wraps the while loop.
If SIGINT or SIGTERM is received during this initialization phase, wouldn't it
result in an unhandled KeyboardInterrupt exception, printing a traceback and
bypassing graceful cleanup?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929062937.374137-1-irogers@google.com?part=3
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v2 4/9] perf test: Deflake Intel PT Python shell tests under load
2026-09-29 6:29 ` [PATCH v2 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (2 preceding siblings ...)
2026-09-29 6:29 ` [PATCH v2 3/9] perf python stat-cpi: Fix live mode signal races " Ian Rogers
@ 2026-09-29 6:29 ` Ian Rogers
2026-09-29 6:35 ` sashiko-bot
2026-09-29 6:29 ` [PATCH v2 5/9] perf test: Deflake failed-syscalls " Ian Rogers
` (5 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
In test_intel_pt_events_python.sh, test_export_to_sqlite_python.sh, and
test_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the
shell builtin 'true' and exits within microseconds of 'uname', sending
SIGCHLD to 'perf record' before the Intel PT AUX buffer is always
flushed under heavy parallel load (~5-10% drop rate).
Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"')
so 'uname' completely exits and flushes its AUX trace before 'sh' exits,
and wrap the record and verification step in a bounded retry loop (up to
5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in
test_export_to_sqlite_python.sh and test_export_to_postgresql_python.sh
to avoid build-id cache and BPF synthesis overhead.
Fixes: d4ce72e9e238 ("perf python: Port intel-pt-events and libxed to perf module")
Fixes: 62d350135e67 ("perf python: Port export-to-sqlite to perf module")
Fixes: b1f968c9656a ("perf python: Port export-to-postgresql to perf module")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
.../shell/test_export_to_postgresql_python.sh | 49 ++++++++++--------
.../shell/test_export_to_sqlite_python.sh | 50 +++++++++++--------
.../shell/test_intel_pt_events_python.sh | 43 +++++++++-------
3 files changed, 79 insertions(+), 63 deletions(-)
diff --git a/tools/perf/tests/shell/test_export_to_postgresql_python.sh b/tools/perf/tests/shell/test_export_to_postgresql_python.sh
index 350813466700..835f48f002e9 100755
--- a/tools/perf/tests/shell/test_export_to_postgresql_python.sh
+++ b/tools/perf/tests/shell/test_export_to_postgresql_python.sh
@@ -61,9 +61,10 @@ test_file_mode() {
fi
# Generate events with callchains and context switches
- if ! perf record -g --switch-events -o "${temp_data}" \
+ if ! perf record -B -N --no-bpf-event -g --switch-events -o "${temp_data}" \
-- perf test -w noploop >/dev/null 2>&1 && \
- ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+ ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping test, perf record failed"
exit 2
fi
@@ -92,29 +93,33 @@ test_file_mode() {
test_intel_pt() {
echo "Testing export-to-postgresql.py with intel_pt..."
- psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true
- rm -f "${temp_data}"
- # Generate some intel_pt events; use a subshell that waits for uname
- if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
- -- sh -c "uname; true" >/dev/null 2>&1; then
- echo "Skipping intel_pt test, intel_pt not available."
- return 0
- fi
+ # Generate some intel_pt events; sleep briefly after uname in the subshell
+ # so uname's AUX buffer is flushed before SIGCHLD stops perf record.
+ passed=0
+ for _ in 1 2 3 4 5; do
+ psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true
+ rm -f "${temp_data}"
+ if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
+ -- sh -c "uname; sleep 0.05" >/dev/null 2>&1; then
+ echo "Skipping intel_pt test, intel_pt not available."
+ return 0
+ fi
- # Run the script with --itrace cr to synthesize call_returns
- if ! perf script export-to-postgresql -i "${temp_data}" \
- -o "${temp_db}" --itrace cr >/dev/null; then
- echo "intel_pt file mode test failed."
- err=1
- else
- # Check DB for calls
- if ! psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \
+ # Run the script with --itrace cr to synthesize call_returns
+ if perf script export-to-postgresql -i "${temp_data}" \
+ -o "${temp_db}" --itrace cr >/dev/null && \
+ psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \
grep -q '[1-9]'; then
- echo "PostgreSQL intel_pt validation failed (no calls found)."
- err=1
- else
- echo "intel_pt test passed (cr validated)."
+ passed=1
+ break
fi
+ done
+
+ if [ "$passed" -eq 0 ]; then
+ echo "PostgreSQL intel_pt validation failed (no calls found)."
+ err=1
+ else
+ echo "intel_pt test passed (cr validated)."
fi
}
diff --git a/tools/perf/tests/shell/test_export_to_sqlite_python.sh b/tools/perf/tests/shell/test_export_to_sqlite_python.sh
index d3c5e22a0754..19ca7c539cf7 100755
--- a/tools/perf/tests/shell/test_export_to_sqlite_python.sh
+++ b/tools/perf/tests/shell/test_export_to_sqlite_python.sh
@@ -47,9 +47,10 @@ test_file_mode() {
echo "Testing export-to-sqlite.py..."
# Generate events with callchains and context switches if supported
- if ! perf record -g --switch-events -o "${temp_data}" \
+ if ! perf record -B -N --no-bpf-event -g --switch-events -o "${temp_data}" \
-- perf test -w noploop >/dev/null 2>&1 && \
- ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+ ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping test, perf record failed"
exit 2
fi
@@ -77,29 +78,34 @@ test_file_mode() {
test_intel_pt() {
echo "Testing export-to-sqlite.py with intel_pt..."
- rm -f "${temp_db}" "${temp_data}"
- # Generate some intel_pt events; use a subshell that waits for uname
- if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
- -- sh -c "uname; true" >/dev/null 2>&1; then
- echo "Skipping intel_pt test, intel_pt not available."
- return 0
- fi
+ query="import sqlite3; c = sqlite3.connect('${temp_db}'); "
+ query="${query}r = c.execute('SELECT COUNT(*) FROM calls').fetchone()[0]; "
+ query="${query}exit(1 if r == 0 else 0)"
+
+ # Generate some intel_pt events; sleep briefly after uname in the subshell
+ # so uname's AUX buffer is flushed before SIGCHLD stops perf record.
+ passed=0
+ for _ in 1 2 3 4 5; do
+ rm -f "${temp_db}" "${temp_data}"
+ if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
+ -- sh -c "uname; sleep 0.05" >/dev/null 2>&1; then
+ echo "Skipping intel_pt test, intel_pt not available."
+ return 0
+ fi
- # Run the script with --itrace cr to synthesize call_returns
- if ! perf script export-to-sqlite -i "${temp_data}" -o "${temp_db}" --itrace cr; then
- echo "intel_pt file mode test failed."
+ # Run the script with --itrace cr to synthesize call_returns
+ if perf script export-to-sqlite -i "${temp_data}" -o "${temp_db}" --itrace cr && \
+ "$PYTHON" -c "$query" >/dev/null 2>&1; then
+ passed=1
+ break
+ fi
+ done
+
+ if [ "$passed" -eq 0 ]; then
+ echo "SQLite intel_pt validation failed (no calls found)."
err=1
else
- # Check DB for calls
- query="import sqlite3; c = sqlite3.connect('${temp_db}'); "
- query="${query}r = c.execute('SELECT COUNT(*) FROM calls').fetchone()[0]; "
- query="${query}exit(1 if r == 0 else 0)"
- if ! "$PYTHON" -c "$query" >/dev/null 2>&1; then
- echo "SQLite intel_pt validation failed (no calls found)."
- err=1
- else
- echo "intel_pt test passed (cr validated)."
- fi
+ echo "intel_pt test passed (cr validated)."
fi
}
diff --git a/tools/perf/tests/shell/test_intel_pt_events_python.sh b/tools/perf/tests/shell/test_intel_pt_events_python.sh
index b5c3173fa2db..9754b9125da1 100755
--- a/tools/perf/tests/shell/test_intel_pt_events_python.sh
+++ b/tools/perf/tests/shell/test_intel_pt_events_python.sh
@@ -30,7 +30,8 @@ cleanup() {
[ -n "${temp_dir}" ] && rm -rf "${temp_dir}"
}
-trap 'cleanup' EXIT TERM INT
+trap 'cleanup' EXIT
+trap 'cleanup; exit 1' TERM INT
temp_dir=$(mktemp -d /tmp/perf.ipt.XXXXXX)
temp_data="${temp_dir}/perf.data"
@@ -39,27 +40,31 @@ temp_out="${temp_dir}/perf.out"
test_intel_pt() {
echo "Testing intel-pt-events.py with intel_pt..."
- rm -f "${temp_data}" "${temp_out}"
- # Generate some intel_pt events; use a subshell that waits for uname so
- # uname's AUX buffer is flushed before the parent workload exits.
- if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
- -- sh -c "uname; true" >/dev/null 2>&1; then
- echo "Skipping intel_pt test, intel_pt not available."
- exit 2
- fi
+ # Generate some intel_pt events; sleep briefly after uname in the subshell
+ # so uname's AUX buffer is flushed before SIGCHLD stops perf record.
+ passed=0
+ for _ in 1 2 3 4 5; do
+ rm -f "${temp_data}" "${temp_out}"
+ if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
+ -- sh -c "uname; sleep 0.05" >/dev/null 2>&1; then
+ echo "Skipping intel_pt test, intel_pt not available."
+ exit 2
+ fi
- # Run the script and check output
- if ! perf script intel-pt-events -i "${temp_data}" > "${temp_out}"; then
- echo "intel-pt-events.py test failed."
+ # Run the script and check output
+ if perf script intel-pt-events -i "${temp_data}" > "${temp_out}" && \
+ grep -q "Intel PT Branch Trace" "${temp_out}" && \
+ grep -q "uname" "${temp_out}"; then
+ passed=1
+ break
+ fi
+ done
+
+ if [ "$passed" -eq 0 ]; then
+ echo "Failed to find expected output: $(cat "${temp_out}" 2>/dev/null)"
err=1
else
- if ! grep -q "Intel PT Branch Trace" "${temp_out}" || \
- ! grep -q "uname" "${temp_out}"; then
- echo "Failed to find expected output: $(cat "${temp_out}")"
- err=1
- else
- echo "intel-pt-events test passed."
- fi
+ echo "intel-pt-events test passed."
fi
rm -f "${temp_out}"
}
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v2 4/9] perf test: Deflake Intel PT Python shell tests under load
2026-09-29 6:29 ` [PATCH v2 4/9] perf test: Deflake Intel PT Python shell tests under load Ian Rogers
@ 2026-09-29 6:35 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 6:35 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> In test_intel_pt_events_python.sh, test_export_to_sqlite_python.sh, and
> test_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the
> shell builtin 'true' and exits within microseconds of 'uname', sending
> SIGCHLD to 'perf record' before the Intel PT AUX buffer is always
> flushed under heavy parallel load (~5-10% drop rate).
>
> Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"')
> so 'uname' completely exits and flushes its AUX trace before 'sh' exits,
> and wrap the record and verification step in a bounded retry loop (up to
> 5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in
> test_export_to_sqlite_python.sh and test_export_to_postgresql_python.sh
> to avoid build-id cache and BPF synthesis overhead.
>
> Fixes: d4ce72e9e238 ("perf python: Port intel-pt-events and libxed to perf module")
> Fixes: 62d350135e67 ("perf python: Port export-to-sqlite to perf module")
> Fixes: b1f968c9656a ("perf python: Port export-to-postgresql to perf module")
> Assisted-by: Antigravity:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929062937.374137-1-irogers@google.com?part=4
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v2 5/9] perf test: Deflake failed-syscalls Python shell tests under load
2026-09-29 6:29 ` [PATCH v2 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (3 preceding siblings ...)
2026-09-29 6:29 ` [PATCH v2 4/9] perf test: Deflake Intel PT Python shell tests under load Ian Rogers
@ 2026-09-29 6:29 ` Ian Rogers
2026-09-29 6:34 ` sashiko-bot
2026-09-29 6:29 ` [PATCH v2 6/9] perf test: Reduce overhead and contention in Python shell tests Ian Rogers
` (4 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
When running 'perf test' in parallel under heavy system load, a one-shot
'perf record -e ... -- ls /does_not_exist' without '-B -N --no-bpf-event'
can exit in under 300 microseconds before 'ls's PERF_RECORD_COMM and
ENOENT sys_exit events are captured, while also contending on ~/.debug
build-id caching and BPF event synthesis.
Pass '-B -N --no-bpf-event' to 'perf record', sleep 0.05s in a subshell
after 'ls' so its events are flushed before the parent subshell exits,
and wrap the record and check steps in a bounded retry loop (up to 5
attempts).
Fixes: b76c43d09b06 ("perf python: Port failed-syscalls-by-pid to perf module")
Fixes: 4e3fe6987cba ("perf python: Port failed-syscalls from Perl to perf module")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
.../test_failed_syscalls_by_pid_python.sh | 68 +++++++++++--------
.../shell/test_failed_syscalls_python.sh | 54 +++++++++------
2 files changed, 71 insertions(+), 51 deletions(-)
diff --git a/tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh b/tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh
index 215372d4e1c5..070a7ef6b8ad 100755
--- a/tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh
+++ b/tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh
@@ -41,8 +41,10 @@ test_file_mode() {
echo "Testing failed-syscalls-by-pid.py..."
# Check if syscalls:sys_exit is supported/readable
- if ! perf record -e syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
- if ! perf record -e raw_syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -e syscalls:sys_exit \
+ -o /dev/null -- true >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_exit \
+ -o /dev/null -- true >/dev/null 2>&1; then
echo "Skipping test, no syscalls:sys_exit or raw_syscalls:sys_exit event"
exit 2
else
@@ -52,41 +54,49 @@ test_file_mode() {
EVENT="syscalls:sys_exit"
fi
- # Generate some events by running a command that should fail at least some syscall
- # (e.g. failing stat on non-existent file).
- # Using '|| true' because 'perf record' returns the exit code of 'ls',
- # which fails with ENOENT
- perf record -e "${EVENT}" -o "${temp_data}" -- ls /does_not_exist >/dev/null 2>&1 || true
+ # Generate some events by running a command that fails a syscall
+ # (e.g. failing stat on non-existent file), sleeping briefly in the
+ # subshell so ls's PERF_RECORD_COMM and sys_exit events are flushed.
+ passed=0
+ for _ in 1 2 3 4 5; do
+ rm -f "${temp_data}" "${temp_out}" "${temp_out}.comm" "${temp_out}.pid"
+ perf record -B -N --no-bpf-event -e "${EVENT}" -o "${temp_data}" \
+ -- sh -c "ls /does_not_exist 2>/dev/null; sleep 0.05 || true" \
+ >/dev/null 2>&1 || true
+ if [ ! -s "${temp_data}" ]; then
+ continue
+ fi
+
+ if perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}" && \
+ grep -n -q "err = ENOENT" "${temp_out}" && \
+ perf script failed-syscalls-by-pid -i "${temp_data}" \
+ "ls" > "${temp_out}.comm" && \
+ grep -q "err = ENOENT" "${temp_out}.comm"; then
+ ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' \
+ "${temp_out}.comm" | head -n 1)
+ if [ -n "${ls_pid}" ] && \
+ perf script failed-syscalls-by-pid -i "${temp_data}" \
+ "${ls_pid}" > "${temp_out}.pid" && \
+ grep -q "err = ENOENT" "${temp_out}.pid"; then
+ passed=1
+ break
+ fi
+ fi
+ done
+
if [ ! -s "${temp_data}" ]; then
echo "Skipping test, perf record failed to create data"
exit 2
fi
- # Run the script and check output
- if ! perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}"; then
+ if [ "$passed" -eq 0 ]; then
echo "failed-syscalls-by-pid test failed."
- err=1
- elif ! grep -n -q "err = ENOENT" "${temp_out}"; then
- echo "Failed to find expected failed syscalls"
- cat "${temp_out}"
- err=1
- elif ! perf script failed-syscalls-by-pid -i "${temp_data}" "ls" > "${temp_out}.comm" || \
- ! grep -q "err = ENOENT" "${temp_out}.comm"; then
- echo "failed-syscalls-by-pid comm filter test failed."
- cat "${temp_out}.comm"
+ cat "${temp_out}" 2>/dev/null || true
+ cat "${temp_out}.comm" 2>/dev/null || true
+ cat "${temp_out}.pid" 2>/dev/null || true
err=1
else
- ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' "${temp_out}.comm" | head -n 1)
- if [ -z "${ls_pid}" ] || \
- ! perf script failed-syscalls-by-pid -i "${temp_data}" \
- "${ls_pid}" > "${temp_out}.pid" || \
- ! grep -q "err = ENOENT" "${temp_out}.pid"; then
- echo "failed-syscalls-by-pid PID filter test failed."
- cat "${temp_out}.pid" 2>/dev/null || true
- err=1
- else
- echo "failed-syscalls-by-pid test passed."
- fi
+ echo "failed-syscalls-by-pid test passed."
fi
rm -f "${temp_out}" "${temp_out}.comm" "${temp_out}.pid"
}
diff --git a/tools/perf/tests/shell/test_failed_syscalls_python.sh b/tools/perf/tests/shell/test_failed_syscalls_python.sh
index 861c2ba71c0a..9aaf05883c50 100755
--- a/tools/perf/tests/shell/test_failed_syscalls_python.sh
+++ b/tools/perf/tests/shell/test_failed_syscalls_python.sh
@@ -27,22 +27,22 @@ if [ ! -f "$script_path" ]; then
fi
err=0
-temp_data=""
-temp_out=""
+temp_dir=$(mktemp -d /tmp/perf-failed-syscalls-XXXXXX)
+temp_data="${temp_dir}/perf.data"
+temp_out="${temp_dir}/perf.out"
cleanup() {
- rm -f "${temp_data}" "${temp_out}"
+ rm -rf "${temp_dir}"
}
trap 'cleanup' EXIT TERM INT
-temp_data=$(mktemp /tmp/perf.data.XXXXXX)
-temp_out=$(mktemp /tmp/perf.out.XXXXXX)
-
echo "Testing failed-syscalls.py..."
# Check if sys_exit event can be recorded
-if ! perf record -e raw_syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
- if ! perf record -e syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
+if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_exit \
+ -o /dev/null -- true >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -e syscalls:sys_exit \
+ -o /dev/null -- true >/dev/null 2>&1; then
echo "Skipping test, no permission or support for sys_exit event"
exit 2
else
@@ -52,28 +52,38 @@ else
EVENT="raw_syscalls:sys_exit"
fi
-# Run perf record with a command that fails a syscall (ls non-existent file).
-# ls exits with non-zero, so perf record returns non-zero exit code of the workload.
-perf record -e "${EVENT}" -o "${temp_data}" \
- -- ls /nonexistent_file_for_test >/dev/null 2>&1 || true
+# Run perf record with a command that fails a syscall (ls non-existent file),
+# sleeping briefly in the subshell so ls's PERF_RECORD_COMM and sys_exit events are flushed.
+passed=0
+for _ in 1 2 3 4 5; do
+ rm -f "${temp_data}" "${temp_out}"
+ perf record -B -N --no-bpf-event -e "${EVENT}" -o "${temp_data}" \
+ -- sh -c "ls /nonexistent_file_for_test 2>/dev/null; sleep 0.05 || true" \
+ >/dev/null 2>&1 || true
+
+ if [ ! -s "${temp_data}" ]; then
+ continue
+ fi
+
+ # Check that the script executes
+ if perf script failed-syscalls -i "${temp_data}" > "${temp_out}" && \
+ grep -q "failed syscalls by comm" "${temp_out}" && \
+ grep -Eq '^ls[[:space:]]+[0-9]+' "${temp_out}"; then
+ passed=1
+ break
+ fi
+done
if [ ! -s "${temp_data}" ]; then
echo "Skipping test, perf record failed to create data"
exit 2
fi
-# Check that the script executes
-if ! perf script failed-syscalls -i "${temp_data}" > "${temp_out}"; then
- echo "failed-syscalls.py test failed"
+if [ "$passed" -eq 0 ]; then
+ echo "Failed to find the metrics table header or expected error"
err=1
else
- if ! grep -q "failed syscalls by comm" "${temp_out}" || \
- ! grep -Eq '^ls[[:space:]]+[0-9]+' "${temp_out}"; then
- echo "Failed to find the metrics table header or expected error"
- err=1
- else
- echo "failed-syscalls test passed."
- fi
+ echo "failed-syscalls test passed."
fi
rm -f "${temp_out}"
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v2 5/9] perf test: Deflake failed-syscalls Python shell tests under load
2026-09-29 6:29 ` [PATCH v2 5/9] perf test: Deflake failed-syscalls " Ian Rogers
@ 2026-09-29 6:34 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 6:34 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> When running 'perf test' in parallel under heavy system load, a one-shot
> 'perf record -e ... -- ls /does_not_exist' without '-B -N --no-bpf-event'
> can exit in under 300 microseconds before 'ls's PERF_RECORD_COMM and
> ENOENT sys_exit events are captured, while also contending on ~/.debug
> build-id caching and BPF event synthesis.
>
> Pass '-B -N --no-bpf-event' to 'perf record', sleep 0.05s in a subshell
> after 'ls' so its events are flushed before the parent subshell exits,
> and wrap the record and check steps in a bounded retry loop (up to 5
> attempts).
>
> Fixes: b76c43d09b06 ("perf python: Port failed-syscalls-by-pid to perf module")
> Fixes: 4e3fe6987cba ("perf python: Port failed-syscalls from Perl to perf module")
> Assisted-by: Antigravity:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929062937.374137-1-irogers@google.com?part=5
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v2 6/9] perf test: Reduce overhead and contention in Python shell tests
2026-09-29 6:29 ` [PATCH v2 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (4 preceding siblings ...)
2026-09-29 6:29 ` [PATCH v2 5/9] perf test: Deflake failed-syscalls " Ian Rogers
@ 2026-09-29 6:29 ` Ian Rogers
2026-09-29 6:35 ` sashiko-bot
2026-09-29 6:29 ` [PATCH v2 7/9] perf python event_analyzing_sample: Default to in-memory SQLite database Ian Rogers
` (3 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
When all Python shell tests run concurrently during 'perf test' under
heavy load:
1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in
each test process; use 'perf list tracepoint' when checking for
tracepoint events.
2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF
events across the system and caches build-ids into ~/.debug; add
'-B -N --no-bpf-event' to all Python shell test recordings.
3. Drop '-a' in test_check_perf_trace_python.sh,
test_rw_by_file_python.sh, test_rw_by_pid_python.sh,
test_rwtop_python.sh, and test_syscall_counts_by_pid_python.sh where
only a single child workload ('dd' or 'sleep') is tested, avoiding
system-wide /proc synthesis and ringbuffer contention.
4. Add '-W 1' to 'ping -c 1' in test_net_dropmonitor_python.sh and
test_netdev_times_python.sh, and narrow 'compaction:*' to
'compaction:mm_compaction_begin,compaction:mm_compaction_end' in
test_compaction_times_python.sh.
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/tests/shell/test_check_perf_trace_python.sh | 8 +++++---
tools/perf/tests/shell/test_compaction_times_python.sh | 8 +++++---
.../tests/shell/test_event_analyzing_sample_python.sh | 3 ++-
tools/perf/tests/shell/test_flamegraph_python.sh | 5 +++--
tools/perf/tests/shell/test_futex_contention_python.sh | 7 ++++---
tools/perf/tests/shell/test_gecko_python.sh | 3 ++-
tools/perf/tests/shell/test_mem_phys_addr_python.sh | 8 +++++---
tools/perf/tests/shell/test_net_dropmonitor_python.sh | 9 +++++----
tools/perf/tests/shell/test_netdev_times_python.sh | 6 +++---
tools/perf/tests/shell/test_powerpc_hcalls_python.sh | 4 ++--
tools/perf/tests/shell/test_rw_by_file_python.sh | 5 +++--
tools/perf/tests/shell/test_rw_by_pid_python.sh | 4 ++--
tools/perf/tests/shell/test_rwtop_python.sh | 4 ++--
tools/perf/tests/shell/test_sched_migration_python.sh | 4 ++--
tools/perf/tests/shell/test_stackcollapse_python.sh | 2 +-
.../tests/shell/test_syscall_counts_by_pid_python.sh | 6 +++---
tools/perf/tests/shell/test_syscall_counts_python.sh | 5 +++--
tools/perf/tests/shell/test_task_analyzer.sh | 3 ++-
tools/perf/tests/shell/test_wakeup_latency_python.sh | 7 ++++---
19 files changed, 58 insertions(+), 43 deletions(-)
diff --git a/tools/perf/tests/shell/test_check_perf_trace_python.sh b/tools/perf/tests/shell/test_check_perf_trace_python.sh
index 92cc7b1f0033..baf55945d429 100755
--- a/tools/perf/tests/shell/test_check_perf_trace_python.sh
+++ b/tools/perf/tests/shell/test_check_perf_trace_python.sh
@@ -45,10 +45,11 @@ test_file_mode() {
echo "Testing check-perf-trace.py..."
events=""
- if perf list | grep -q "irq:softirq_entry"; then
+ tp_list=$(perf list tracepoint)
+ if echo "$tp_list" | grep -q "irq:softirq_entry"; then
events="irq:softirq_entry"
fi
- if perf list | grep -q "kmem:kmalloc"; then
+ if echo "$tp_list" | grep -q "kmem:kmalloc"; then
if [ -n "$events" ]; then
events="$events,kmem:kmalloc,kmem:kfree"
else
@@ -62,7 +63,8 @@ test_file_mode() {
fi
# Generate events
- if ! perf record -e "$events" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -e "$events" -o "${temp_data}" \
+ -- sh -c "sleep 0.1" >/dev/null 2>&1; then
echo "Skipping test, perf record failed"
exit 2
fi
diff --git a/tools/perf/tests/shell/test_compaction_times_python.sh b/tools/perf/tests/shell/test_compaction_times_python.sh
index 50397afaafed..3f5a71e52388 100755
--- a/tools/perf/tests/shell/test_compaction_times_python.sh
+++ b/tools/perf/tests/shell/test_compaction_times_python.sh
@@ -44,15 +44,17 @@ test_file_mode() {
echo "Testing compaction-times.py..."
# Check for any compaction events to see if kernel supports it
- if ! perf list | grep -q "compaction:mm_compaction_begin"; then
+ if ! perf list tracepoint | grep -q "compaction:mm_compaction_begin"; then
echo "Skipping test, compaction tracepoints not found"
exit 2
fi
# Generate some events
- # We might not naturally trigger compaction in 0.5s sleep, but the script
+ # We might not naturally trigger compaction in 0.1s sleep, but the script
# should parse the empty or sparse file correctly without crashing.
- if ! perf record -e "compaction:*" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event \
+ -e "compaction:mm_compaction_begin,compaction:mm_compaction_end" \
+ -a -o "${temp_data}" -- sleep 0.1 >/dev/null 2>&1; then
echo "Skipping test, perf record failed"
exit 2
fi
diff --git a/tools/perf/tests/shell/test_event_analyzing_sample_python.sh b/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
index a7a8ded003db..5f0a41080acf 100755
--- a/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
+++ b/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
@@ -39,7 +39,8 @@ test_file_mode() {
echo "Testing event_analyzing_sample.py..."
# Generate some events
- if ! perf record -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping test, perf record failed"
exit 2
fi
diff --git a/tools/perf/tests/shell/test_flamegraph_python.sh b/tools/perf/tests/shell/test_flamegraph_python.sh
index 86313c72f53e..787d6bbdf04a 100755
--- a/tools/perf/tests/shell/test_flamegraph_python.sh
+++ b/tools/perf/tests/shell/test_flamegraph_python.sh
@@ -60,7 +60,8 @@ test_file_mode() {
echo "Testing flamegraph.py..."
# Generate some events with callchains
- if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping test, perf record -g failed (permissions or lack of support)"
exit 2
fi
@@ -75,7 +76,7 @@ test_file_mode() {
# Run the script in pipe ('-') mode and validate JSON output
rm -f "${temp_json}"
- if ! perf record -g -o - -- perf test -w noploop 2>/dev/null | \
+ if ! perf record -B -N --no-bpf-event -g -o - -- perf test -w noploop 2>/dev/null | \
perf script flamegraph -i - -f json -o "${temp_json}" >/dev/null; then
echo "Pipe stdin JSON mode test failed."
err=1
diff --git a/tools/perf/tests/shell/test_futex_contention_python.sh b/tools/perf/tests/shell/test_futex_contention_python.sh
index a8846ed68ac2..41ce8470a75f 100755
--- a/tools/perf/tests/shell/test_futex_contention_python.sh
+++ b/tools/perf/tests/shell/test_futex_contention_python.sh
@@ -89,14 +89,15 @@ EOF
test_file_mode() {
echo "Testing futex-contention.py..."
# Some systems might not have syscalls:sys_enter_futex
- if ! perf list | grep -q syscalls:sys_enter_futex; then
+ if ! perf list tracepoint | grep -q syscalls:sys_enter_futex; then
echo "Skipping file mode test, syscalls:sys_enter_futex not found"
return
fi
# Generate some futex events
- if ! perf record -e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \
- -- sleep 0.5 2>/dev/null; then
+ if ! perf record -B -N --no-bpf-event \
+ -e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \
+ -- sleep 0.1 2>/dev/null; then
echo "Skipping file mode test (record failed)"
return
fi
diff --git a/tools/perf/tests/shell/test_gecko_python.sh b/tools/perf/tests/shell/test_gecko_python.sh
index de1d7fcf88b7..18e9f5cd2ab0 100755
--- a/tools/perf/tests/shell/test_gecko_python.sh
+++ b/tools/perf/tests/shell/test_gecko_python.sh
@@ -39,7 +39,8 @@ test_file_mode() {
echo "Testing gecko.py..."
# Generate some events with callchains
- if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping test, perf record -g failed (permissions or lack of support)"
exit 2
fi
diff --git a/tools/perf/tests/shell/test_mem_phys_addr_python.sh b/tools/perf/tests/shell/test_mem_phys_addr_python.sh
index 451692d5e089..cbb9f572be9d 100755
--- a/tools/perf/tests/shell/test_mem_phys_addr_python.sh
+++ b/tools/perf/tests/shell/test_mem_phys_addr_python.sh
@@ -78,10 +78,12 @@ test_file_mode() {
echo "Testing mem-phys-addr.py file mode..."
# Generate memory access events (try unprivileged user-space first, then system-wide)
- if ! perf record --phys-data -d -o "${temp_data}" \
+ if ! perf record -B -N --no-bpf-event --phys-data -d -o "${temp_data}" \
-- perf test -w datasym >/dev/null 2>&1 && \
- ! perf record -d -o "${temp_data}" -- perf test -w datasym >/dev/null 2>&1 && \
- ! perf record -d -a -o "${temp_data}" -- sleep 0.2 >/dev/null 2>&1; then
+ ! perf record -B -N --no-bpf-event -d -o "${temp_data}" \
+ -- perf test -w datasym >/dev/null 2>&1 && \
+ ! perf record -B -N --no-bpf-event -d -a -o "${temp_data}" \
+ -- sleep 0.1 >/dev/null 2>&1; then
echo "Skipping file mode record test, perf record -d not supported"
return 0
fi
diff --git a/tools/perf/tests/shell/test_net_dropmonitor_python.sh b/tools/perf/tests/shell/test_net_dropmonitor_python.sh
index 05056a897bc9..2e37994418fc 100755
--- a/tools/perf/tests/shell/test_net_dropmonitor_python.sh
+++ b/tools/perf/tests/shell/test_net_dropmonitor_python.sh
@@ -42,11 +42,12 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
echo "Testing net_dropmonitor.py..."
# Create a perf.data file. Force dropping a packet if tracepoint is available!
-if ! perf record -e skb:kfree_skb -o "${temp_data}" -a \
- -- ping -c 1 255.255.255.255 >/dev/null 2>&1; then
- if ! perf record -e skb:kfree_skb -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" -a \
+ -- ping -c 1 -W 1 255.255.255.255 >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" \
-- sleep 0.1 >/dev/null 2>&1; then
- if ! perf record -o "${temp_data}" -- uname >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -o "${temp_data}" \
+ -- uname >/dev/null 2>&1; then
echo "Skipping test, cannot record perf events"
exit 2
fi
diff --git a/tools/perf/tests/shell/test_netdev_times_python.sh b/tools/perf/tests/shell/test_netdev_times_python.sh
index 393bf446efb2..0d8d10a42cab 100755
--- a/tools/perf/tests/shell/test_netdev_times_python.sh
+++ b/tools/perf/tests/shell/test_netdev_times_python.sh
@@ -85,9 +85,9 @@ then
fi
# Create a perf.data file. Force dropping a packet if tracepoint is available!
-if ! perf record -e skb:kfree_skb -a -o "${temp_data}" \
- -- ping -c 1 127.0.0.1 >/dev/null 2>&1; then
- perf record -e cycles -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -a -o "${temp_data}" \
+ -- ping -c 1 -W 1 127.0.0.1 >/dev/null 2>&1; then
+ perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
-- perf test -w noploop >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
fi
diff --git a/tools/perf/tests/shell/test_powerpc_hcalls_python.sh b/tools/perf/tests/shell/test_powerpc_hcalls_python.sh
index 19569c6d3613..b140ccf00ffa 100755
--- a/tools/perf/tests/shell/test_powerpc_hcalls_python.sh
+++ b/tools/perf/tests/shell/test_powerpc_hcalls_python.sh
@@ -67,8 +67,8 @@ if ! grep -q "H_REMOVE.*1.*1500.*1500.*1500" "${temp_out}"; then
fi
# Create a perf.data file if powerpc hcall tracepoints are available on this host.
-if ! perf record -e powerpc:hcall_entry,powerpc:hcall_exit -a -o "${temp_data}" \
- -- perf test -w noploop >/dev/null 2>&1; then
+if ! perf record -B -N --no-bpf-event -e powerpc:hcall_entry,powerpc:hcall_exit \
+ -a -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping live record test, powerpc hcall tracepoints not available"
exit 0
fi
diff --git a/tools/perf/tests/shell/test_rw_by_file_python.sh b/tools/perf/tests/shell/test_rw_by_file_python.sh
index f121597f418f..e3c538c3ace3 100755
--- a/tools/perf/tests/shell/test_rw_by_file_python.sh
+++ b/tools/perf/tests/shell/test_rw_by_file_python.sh
@@ -37,8 +37,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
echo "Testing rw-by-file.py..."
# Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
- perf record -e syscalls:sys_enter_read,syscalls:sys_enter_write -a -o "${temp_data}" \
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
+ perf record -B -N --no-bpf-event -e syscalls:sys_enter_read,syscalls:sys_enter_write \
+ -o "${temp_data}" \
-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
else
diff --git a/tools/perf/tests/shell/test_rw_by_pid_python.sh b/tools/perf/tests/shell/test_rw_by_pid_python.sh
index 41eaa29097a2..cd93c1d9714c 100755
--- a/tools/perf/tests/shell/test_rw_by_pid_python.sh
+++ b/tools/perf/tests/shell/test_rw_by_pid_python.sh
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
echo "Testing rw-by-pid.py..."
# Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
ev="syscalls:sys_enter_read,syscalls:sys_exit_read"
ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write"
- perf record -e "$ev" -a -o "${temp_data}" \
+ perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \
-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
else
diff --git a/tools/perf/tests/shell/test_rwtop_python.sh b/tools/perf/tests/shell/test_rwtop_python.sh
index 05897e384702..d33807923c92 100755
--- a/tools/perf/tests/shell/test_rwtop_python.sh
+++ b/tools/perf/tests/shell/test_rwtop_python.sh
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
echo "Testing rwtop.py..."
# Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
ev="syscalls:sys_enter_read,syscalls:sys_exit_read"
ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write"
- perf record -e "$ev" -a -o "${temp_data}" \
+ perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \
-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
else
diff --git a/tools/perf/tests/shell/test_sched_migration_python.sh b/tools/perf/tests/shell/test_sched_migration_python.sh
index 75465bcd3791..714275b0e492 100755
--- a/tools/perf/tests/shell/test_sched_migration_python.sh
+++ b/tools/perf/tests/shell/test_sched_migration_python.sh
@@ -44,10 +44,10 @@ echo "Testing sched-migration.py..."
ev="sched:sched_switch,sched:sched_migrate_task"
ev="${ev},sched:sched_wakeup_new,sched:sched_wakeup"
has_sched=1
-if ! perf record -e "$ev" -a -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \
-- sleep 0.1 >/dev/null 2>&1; then
has_sched=0
- perf record -e cycles -o "${temp_data}" \
+ perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
-- perf test -w noploop >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
fi
diff --git a/tools/perf/tests/shell/test_stackcollapse_python.sh b/tools/perf/tests/shell/test_stackcollapse_python.sh
index ce0f80407eec..2048734c5e0d 100755
--- a/tools/perf/tests/shell/test_stackcollapse_python.sh
+++ b/tools/perf/tests/shell/test_stackcollapse_python.sh
@@ -37,7 +37,7 @@ echo "Testing stackcollapse.py..."
# Create a perf.data file with callchains. Use a busy workload rather than
# sleep, as an idle system may not generate any samples at all.
-perf record -g -o "${temp_data}" \
+perf record -B -N --no-bpf-event -g -o "${temp_data}" \
-- perf test -w noploop >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
diff --git a/tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh b/tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh
index 45bbdc554c8f..300289fe33b4 100755
--- a/tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh
+++ b/tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh
@@ -43,14 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
test_file_mode() {
echo "Testing syscall-counts-by-pid.py..."
# Some systems might not have raw_syscalls:sys_enter
- if ! perf list | grep -q raw_syscalls:sys_enter; then
+ if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then
echo "Skipping test, raw_syscalls:sys_enter not found"
exit 2
fi
# Generate some syscall events
- perf record -e raw_syscalls:sys_enter -a -o "${temp_data}" \
- -- sleep 0.5 >/dev/null 2>&1 || \
+ perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \
+ -- sh -c "sleep 0.1; sleep 0.05" >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
if ! perf script syscall-counts-by-pid -i "${temp_data}" > "${temp_out}"; then
diff --git a/tools/perf/tests/shell/test_syscall_counts_python.sh b/tools/perf/tests/shell/test_syscall_counts_python.sh
index 310e05d399f9..86463f14391b 100755
--- a/tools/perf/tests/shell/test_syscall_counts_python.sh
+++ b/tools/perf/tests/shell/test_syscall_counts_python.sh
@@ -43,13 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
test_file_mode() {
echo "Testing syscall-counts.py..."
# Some systems might not have raw_syscalls:sys_enter (e.g. stripped kernels or permissions)
- if ! perf list | grep -q raw_syscalls:sys_enter; then
+ if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then
echo "Skipping test, raw_syscalls:sys_enter not found"
exit 2
fi
# Generate some syscall events
- if ! perf record -e raw_syscalls:sys_enter -o "${temp_data}" -- sleep 0.5 2>/dev/null; then
+ if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \
+ -- sh -c "sleep 0.1; sleep 0.05" 2>/dev/null; then
echo "perf record failed (permissions?), skipping file mode test."
exit 2
fi
diff --git a/tools/perf/tests/shell/test_task_analyzer.sh b/tools/perf/tests/shell/test_task_analyzer.sh
index f99a637a6099..f53fa4a27c79 100755
--- a/tools/perf/tests/shell/test_task_analyzer.sh
+++ b/tools/perf/tests/shell/test_task_analyzer.sh
@@ -60,7 +60,8 @@ skip_no_probe_record_support() {
prepare_perf_data() {
# 1s should be sufficient to catch at least some switches
- perf record -e sched:sched_switch -a -o "${perfdata}" -- sleep 1 > /dev/null 2>&1
+ perf record -B -N --no-bpf-event -e sched:sched_switch -a -o "${perfdata}" \
+ -- sleep 1 > /dev/null 2>&1
# check if perf data file got created in above step.
if [ ! -e "${perfdata}" ]; then
printf "FAIL: perf record failed to create \"${perfdata}\" \n"
diff --git a/tools/perf/tests/shell/test_wakeup_latency_python.sh b/tools/perf/tests/shell/test_wakeup_latency_python.sh
index cb2571449168..b43a86996be2 100755
--- a/tools/perf/tests/shell/test_wakeup_latency_python.sh
+++ b/tools/perf/tests/shell/test_wakeup_latency_python.sh
@@ -41,9 +41,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
echo "Testing wakeup-latency.py..."
# Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "sched:sched_wakeup"; then
+if perf list tracepoint | grep -q "sched:sched_wakeup"; then
ev="sched:sched_wakeup,sched:sched_wakeup_new,sched:sched_switch"
- perf record -e "$ev" -a -o "${temp_data}" \
+ perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \
-- sleep 0.1 >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
else
@@ -70,7 +70,8 @@ else
fi
# Also test zero-wakeups / unhandled events path to verify division-by-zero protection
-if perf record -e cycles -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+if perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
if ! perf script wakeup-latency -i "${temp_data}" > "${temp_out}" || \
! grep -q "avg_wakeup_latency (ns): N/A" "${temp_out}"; then
echo "wakeup-latency zero-wakeups guard test failed"
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v2 6/9] perf test: Reduce overhead and contention in Python shell tests
2026-09-29 6:29 ` [PATCH v2 6/9] perf test: Reduce overhead and contention in Python shell tests Ian Rogers
@ 2026-09-29 6:35 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 6:35 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> When all Python shell tests run concurrently during 'perf test' under
> heavy load:
> 1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in
> each test process; use 'perf list tracepoint' when checking for
> tracepoint events.
> 2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF
> events across the system and caches build-ids into ~/.debug; add
> '-B -N --no-bpf-event' to all Python shell test recordings.
> 3. Drop '-a' in test_check_perf_trace_python.sh,
> test_rw_by_file_python.sh, test_rw_by_pid_python.sh,
> test_rwtop_python.sh, and test_syscall_counts_by_pid_python.sh where
> only a single child workload ('dd' or 'sleep') is tested, avoiding
> system-wide /proc synthesis and ringbuffer contention.
> 4. Add '-W 1' to 'ping -c 1' in test_net_dropmonitor_python.sh and
> test_netdev_times_python.sh, and narrow 'compaction:*' to
> 'compaction:mm_compaction_begin,compaction:mm_compaction_end' in
> test_compaction_times_python.sh.
>
> Assisted-by: Antigravity:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929062937.374137-1-irogers@google.com?part=6
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v2 7/9] perf python event_analyzing_sample: Default to in-memory SQLite database
2026-09-29 6:29 ` [PATCH v2 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (5 preceding siblings ...)
2026-09-29 6:29 ` [PATCH v2 6/9] perf test: Reduce overhead and contention in Python shell tests Ian Rogers
@ 2026-09-29 6:29 ` Ian Rogers
2026-09-29 6:37 ` sashiko-bot
2026-09-29 6:29 ` [PATCH v2 8/9] perf python: Initialize debug output on module load Ian Rogers
` (2 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
When -d/--db is not specified, event_analyzing_sample.py created a
temporary file in /tmp via tempfile.mkstemp() and deleted it in
trace_end(). As noted during review, creating the database in a shared
/tmp directory does not reserve SQLite's auxiliary sidecar filenames
(-journal or -wal), and a temporary on-disk file is unnecessary when the
caller did not ask to persist the database.
Default to sqlite3.connect(":memory:") when db_path is not provided,
removing the temporary file creation and cleanup logic, and test both
the default in-memory mode and explicit -d file mode in
test_event_analyzing_sample_python.sh.
Fixes: eeb70645a809 ("perf python: Port event_analyzing_sample to perf module")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/python/event_analyzing_sample.py | 35 ++++++-------------
.../test_event_analyzing_sample_python.sh | 8 ++++-
2 files changed, 17 insertions(+), 26 deletions(-)
diff --git a/tools/perf/python/event_analyzing_sample.py b/tools/perf/python/event_analyzing_sample.py
index 15b47cff9fa3..b4efee88d94b 100755
--- a/tools/perf/python/event_analyzing_sample.py
+++ b/tools/perf/python/event_analyzing_sample.py
@@ -17,10 +17,8 @@ from __future__ import annotations
import argparse
import math
-import os
import sqlite3
import struct
-import tempfile
from typing import Any
import perf
@@ -117,16 +115,11 @@ session: Any = None
class _DB:
con: sqlite3.Connection | None = None
- temp_path: str | None = None
def trace_begin(db_path: str | None = None) -> None:
"""Initialize database tables."""
print("In trace_begin:\n")
- if not db_path:
- fd, db_path = tempfile.mkstemp(prefix="perf_events_", suffix=".db")
- os.close(fd)
- _DB.temp_path = db_path
- con = sqlite3.connect(db_path)
+ con = sqlite3.connect(db_path or ":memory:")
try:
# Drop any pre-existing tables so repeated runs do not accumulate duplicate events.
con.execute("drop table if exists gen_events;")
@@ -297,28 +290,20 @@ def show_pebs_ll() -> None:
def trace_end() -> None:
"""Called at the end of trace processing."""
print("In trace_end:\n")
- try:
- if _DB.con:
- try:
- _DB.con.commit()
- show_general_events()
- show_pebs_ll()
- finally:
- _DB.con.close()
- _DB.con = None
- finally:
- if _DB.temp_path and os.path.exists(_DB.temp_path):
- try:
- os.remove(_DB.temp_path)
- except OSError:
- pass
- _DB.temp_path = None
+ if _DB.con:
+ try:
+ _DB.con.commit()
+ show_general_events()
+ show_pebs_ll()
+ finally:
+ _DB.con.close()
+ _DB.con = None
if __name__ == "__main__":
ap = argparse.ArgumentParser(description="Analyze events with SQLite")
ap.add_argument("-i", "--input", default="perf.data", help="Input file name")
ap.add_argument("-d", "--db", "--database", dest="database", default=None,
- help="Database file name (defaults to a temporary file cleaned up on exit)")
+ help="Database file name (defaults to an in-memory database)")
args = ap.parse_args()
try:
diff --git a/tools/perf/tests/shell/test_event_analyzing_sample_python.sh b/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
index 5f0a41080acf..cddb12b67698 100755
--- a/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
+++ b/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
@@ -45,8 +45,14 @@ test_file_mode() {
exit 2
fi
- # Run the script
+ # Run the script with default (:memory:) database and with explicit -d path
if ! perf script event_analyzing_sample -i "${temp_data}" \
+ > "${temp_dir}/perf.mem.out" 2>&1 || \
+ ! grep -q "Statistics about the general events" "${temp_dir}/perf.mem.out" || \
+ grep -q "Error creating/inserting event" "${temp_dir}/perf.mem.out"; then
+ echo "Default in-memory database mode test failed."
+ err=1
+ elif ! perf script event_analyzing_sample -i "${temp_data}" \
-d "${temp_db}" > "${temp_dir}/perf.out" 2>&1; then
echo "File mode test failed."
err=1
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* [PATCH v2 8/9] perf python: Initialize debug output on module load
2026-09-29 6:29 ` [PATCH v2 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (6 preceding siblings ...)
2026-09-29 6:29 ` [PATCH v2 7/9] perf python event_analyzing_sample: Default to in-memory SQLite database Ian Rogers
@ 2026-09-29 6:29 ` Ian Rogers
2026-09-29 6:37 ` sashiko-bot
2026-09-29 6:29 ` [PATCH v2 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list Ian Rogers
2026-09-29 6:58 ` [PATCH v3 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
Unlike the perf binary's main(), PyInit_perf() does not call
perf_debug_setup(). As a result, the first warning or error printed from
the perf Python C extension hits debug_file() with _debug_file == NULL
and emits 'debug_file not set' without a trailing newline before the
actual diagnostic message.
Call perf_debug_setup() in PyInit_perf() and add the missing trailing
newline to the fallback warning in debug_file().
Fixes: ec49230cf6dd ("perf debug: Expose debug file")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/util/debug.c | 2 +-
tools/perf/util/python.c | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/tools/perf/util/debug.c b/tools/perf/util/debug.c
index 6b5ffe81f141..b7095519419e 100644
--- a/tools/perf/util/debug.c
+++ b/tools/perf/util/debug.c
@@ -55,7 +55,7 @@ FILE *debug_file(void)
{
if (!_debug_file) {
debug_set_file(stderr);
- pr_warning_once("debug_file not set");
+ pr_warning_once("%s not set\n", __func__);
}
return _debug_file;
}
diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c
index 690a94fb4f62..95140dfaef9c 100644
--- a/tools/perf/util/python.c
+++ b/tools/perf/util/python.c
@@ -5157,6 +5157,7 @@ PyMODINIT_FUNC PyInit_perf(void)
/* The page_size is placed in util object. */
page_size = sysconf(_SC_PAGE_SIZE);
+ perf_debug_setup();
Py_INCREF(&pyrf_evlist__type);
PyModule_AddObject(module, "evlist", (PyObject *)&pyrf_evlist__type);
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* [PATCH v2 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list
2026-09-29 6:29 ` [PATCH v2 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (7 preceding siblings ...)
2026-09-29 6:29 ` [PATCH v2 8/9] perf python: Initialize debug output on module load Ian Rogers
@ 2026-09-29 6:29 ` Ian Rogers
2026-09-29 6:38 ` sashiko-bot
2026-09-29 6:58 ` [PATCH v3 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
perf_pmus__print_pmu_events() first counts events via
perf_pmu__num_events() to allocate the aliases array, then populates it
via perf_pmu__for_each_event(). When dynamic tracepoints (such as
kprobes or uprobes during 'perf test' runs) are concurrently created or
removed between the two passes:
- If an event is removed, state.index is smaller than the allocated len,
leaving trailing zeroed entries in aliases[] with name == NULL and
pmu == NULL, which causes qsort(cmp_sevent) or the print loop to crash
with SIGSEGV.
- If a dynamic tracepoint subsystem directory is removed while scanning
/sys/kernel/tracing/events, tp_pmu__for_each_tp_event() returns
-ENOENT and aborts enumeration of all remaining tracepoint subsystems.
- If an event is added, perf_pmus__print_pmu_events__callback() aborts
enumeration when state->index reaches state->aliases_len.
Grow state->aliases dynamically via realloc() when needed, use
state.index as the actual populated length for sorting and printing, and
ignore -ENOENT when a tracepoint subsystem directory disappears during
enumeration.
Fixes: c3245d2093c1 ("perf pmu: Abstract alias/event struct")
Fixes: 45b6e281cb06 ("perf tp_pmu: Add event APIs")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/util/pmus.c | 21 +++++++++++++++++----
tools/perf/util/tp_pmu.c | 8 ++++++--
2 files changed, 23 insertions(+), 6 deletions(-)
diff --git a/tools/perf/util/pmus.c b/tools/perf/util/pmus.c
index e0a4cb2428ca..1355b317f3ed 100644
--- a/tools/perf/util/pmus.c
+++ b/tools/perf/util/pmus.c
@@ -8,6 +8,7 @@
#include <sys/types.h>
#include <ctype.h>
#include <pthread.h>
+#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include "cpumap.h"
@@ -571,7 +572,7 @@ static int cmp_sevent(const void *a, const void *b)
}
/* Order by event name. */
- return strcmp(as->name, bs->name);
+ return strcmp(as->name ?: "", bs->name ?: "");
}
static bool pmu_alias_is_duplicate(struct sevent *a, struct sevent *b)
@@ -581,7 +582,7 @@ static bool pmu_alias_is_duplicate(struct sevent *a, struct sevent *b)
return false;
/* Don't remove duplicates for different PMUs */
- return strcmp(a->pmu_name, b->pmu_name) == 0;
+ return strcmp(a->pmu_name ?: "", b->pmu_name ?: "") == 0;
}
struct events_callback_state {
@@ -597,8 +598,18 @@ static int perf_pmus__print_pmu_events__callback(void *vstate,
struct sevent *s;
if (state->index >= state->aliases_len) {
- pr_err("Unexpected event %s/%s/\n", info->pmu->name, info->name);
- return 1;
+ size_t new_len = max_t(size_t, 16, state->aliases_len * 2);
+ struct sevent *new_aliases;
+
+ new_aliases = realloc(state->aliases, new_len * sizeof(struct sevent));
+ if (!new_aliases) {
+ pr_err("Unexpected event %s/%s/\n", info->pmu->name, info->name);
+ return 1;
+ }
+ memset(&new_aliases[state->aliases_len], 0,
+ (new_len - state->aliases_len) * sizeof(struct sevent));
+ state->aliases = new_aliases;
+ state->aliases_len = new_len;
}
assert(info->pmu != NULL || info->name != NULL);
s = &state->aliases[state->index];
@@ -654,6 +665,8 @@ void perf_pmus__print_pmu_events(const struct print_callbacks *print_cb, void *p
perf_pmu__for_each_event(pmu, skip_duplicate_pmus, &state,
perf_pmus__print_pmu_events__callback);
}
+ aliases = state.aliases;
+ len = state.index;
qsort(aliases, len, sizeof(struct sevent), cmp_sevent);
for (int j = 0; j < len; j++) {
/* Skip duplicates */
diff --git a/tools/perf/util/tp_pmu.c b/tools/perf/util/tp_pmu.c
index c2be8c9f9084..5ac732e06841 100644
--- a/tools/perf/util/tp_pmu.c
+++ b/tools/perf/util/tp_pmu.c
@@ -151,7 +151,9 @@ static int for_each_event_cb(void *state, const char *sys_name, const char *evt_
static int for_each_event_sys_cb(void *state, const char *sys_name)
{
- return tp_pmu__for_each_tp_event(sys_name, state, for_each_event_cb);
+ int ret = tp_pmu__for_each_tp_event(sys_name, state, for_each_event_cb);
+
+ return ret == -ENOENT ? 0 : ret;
}
int tp_pmu__for_each_event(struct perf_pmu *pmu, void *state, pmu_event_callback cb)
@@ -176,7 +178,9 @@ static int num_events_cb(void *state, const char *sys_name __maybe_unused,
static int num_events_sys_cb(void *state, const char *sys_name)
{
- return tp_pmu__for_each_tp_event(sys_name, state, num_events_cb);
+ int ret = tp_pmu__for_each_tp_event(sys_name, state, num_events_cb);
+
+ return ret == -ENOENT ? 0 : ret;
}
size_t tp_pmu__num_events(struct perf_pmu *pmu __maybe_unused)
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v2 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list
2026-09-29 6:29 ` [PATCH v2 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list Ian Rogers
@ 2026-09-29 6:38 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 6:38 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> perf_pmus__print_pmu_events() first counts events via
> perf_pmu__num_events() to allocate the aliases array, then populates it
> via perf_pmu__for_each_event(). When dynamic tracepoints (such as
> kprobes or uprobes during 'perf test' runs) are concurrently created or
> removed between the two passes:
> - If an event is removed, state.index is smaller than the allocated len,
> leaving trailing zeroed entries in aliases[] with name == NULL and
> pmu == NULL, which causes qsort(cmp_sevent) or the print loop to crash
> with SIGSEGV.
> - If a dynamic tracepoint subsystem directory is removed while scanning
> /sys/kernel/tracing/events, tp_pmu__for_each_tp_event() returns
> -ENOENT and aborts enumeration of all remaining tracepoint subsystems.
> - If an event is added, perf_pmus__print_pmu_events__callback() aborts
> enumeration when state->index reaches state->aliases_len.
>
> Grow state->aliases dynamically via realloc() when needed, use
> state.index as the actual populated length for sorting and printing, and
> ignore -ENOENT when a tracepoint subsystem directory disappears during
> enumeration.
>
> Fixes: c3245d2093c1 ("perf pmu: Abstract alias/event struct")
> Fixes: 45b6e281cb06 ("perf tp_pmu: Add event APIs")
> Assisted-by: Antigravity:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929062937.374137-1-irogers@google.com?part=9
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v3 0/9] perf python: Follow-up fixes and shell test deflaking
2026-09-29 6:29 ` [PATCH v2 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (8 preceding siblings ...)
2026-09-29 6:29 ` [PATCH v2 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list Ian Rogers
@ 2026-09-29 6:58 ` Ian Rogers
2026-09-29 6:58 ` [PATCH v3 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events Ian Rogers
` (9 more replies)
9 siblings, 10 replies; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
This series contains follow-up fixes for the standalone Python scripts
and their shell tests, particularly when running 'perf test' as root
under heavy parallel load:
- Clamp sample_id size calculation for PERF_RECORD_KSYMBOL,
PERF_RECORD_BPF_EVENT, and PERF_RECORD_TEXT_POKE records to work
around kernels without "perf/core: Restore header fields in sideband
output callbacks" [1].
- Fix offline interval printing in sctop.py and live signal races in
stat-cpi.py.
- Deflake Intel PT, failed-syscalls, and other Python shell tests under
parallel 'perf test' load by scoping tracepoint checks to
'perf list tracepoint', passing '-B -N --no-bpf-event' to 'perf record',
and avoiding unnecessary system-wide ('-a') recordings.
- Default event_analyzing_sample.py to an in-memory SQLite database
(':memory:') when '--db' is not specified.
- Initialize perf debug output ('perf_debug_setup()') when loading the
'perf' Python extension module.
- Fix races in 'perf list' ('perf_pmus__print_pmu_events()' and
'tp_pmu.c') when concurrent tests dynamically create and remove
kprobe/uprobes tracepoints.
v3:
- Wrap _open_live_evlist() inside the try/except KeyboardInterrupt block
in stat-cpi.py so signals arriving during event initialization are
caught and cleaned up gracefully.
v2:
- Fix 8-byte alignment calculation for PERF_RECORD_TEXT_POKE payload in
evsel__event_size() by aligning fixed + old_len + new_len.
- Keep offline interval clock advancing on all samples in sctop.py while
flushing any remaining syscalls at EOF.
- Use 'mktemp -d' private directories in test_sctop_python.sh and
test_failed_syscalls_python.sh so temporary files are not unlinked and
recreated directly in /tmp.
- Explicitly include <stdlib.h> in tools/perf/util/pmus.c.
[1] https://lore.kernel.org/r/20260929014206.4175245-1-irogers@google.com
Ian Rogers (9):
perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke
events
perf python sctop: Fix offline interval printing and test flakiness
perf python stat-cpi: Fix live mode signal races and test flakiness
perf test: Deflake Intel PT Python shell tests under load
perf test: Deflake failed-syscalls Python shell tests under load
perf test: Reduce overhead and contention in Python shell tests
perf python event_analyzing_sample: Default to in-memory SQLite
database
perf python: Initialize debug output on module load
perf pmu: Fix race with concurrent tracepoint creation and removal in
perf list
tools/perf/python/event_analyzing_sample.py | 35 +++-------
tools/perf/python/sctop.py | 7 +-
tools/perf/python/stat-cpi.py | 32 ++++++---
.../shell/test_check_perf_trace_python.sh | 8 ++-
.../shell/test_compaction_times_python.sh | 8 ++-
.../test_event_analyzing_sample_python.sh | 11 ++-
.../shell/test_export_to_postgresql_python.sh | 49 +++++++------
.../shell/test_export_to_sqlite_python.sh | 50 ++++++++------
.../test_failed_syscalls_by_pid_python.sh | 68 +++++++++++--------
.../shell/test_failed_syscalls_python.sh | 54 +++++++++------
.../tests/shell/test_flamegraph_python.sh | 5 +-
.../shell/test_futex_contention_python.sh | 7 +-
tools/perf/tests/shell/test_gecko_python.sh | 3 +-
.../shell/test_intel_pt_events_python.sh | 43 ++++++------
.../tests/shell/test_mem_phys_addr_python.sh | 8 ++-
.../shell/test_net_dropmonitor_python.sh | 9 +--
.../tests/shell/test_netdev_times_python.sh | 6 +-
.../tests/shell/test_powerpc_hcalls_python.sh | 4 +-
.../tests/shell/test_rw_by_file_python.sh | 5 +-
.../perf/tests/shell/test_rw_by_pid_python.sh | 4 +-
tools/perf/tests/shell/test_rwtop_python.sh | 4 +-
.../shell/test_sched_migration_python.sh | 4 +-
tools/perf/tests/shell/test_sctop_python.sh | 58 ++++++++--------
.../tests/shell/test_stackcollapse_python.sh | 2 +-
.../perf/tests/shell/test_stat_cpi_python.sh | 12 +++-
.../test_syscall_counts_by_pid_python.sh | 6 +-
.../tests/shell/test_syscall_counts_python.sh | 5 +-
tools/perf/tests/shell/test_task_analyzer.sh | 3 +-
.../tests/shell/test_wakeup_latency_python.sh | 7 +-
tools/perf/util/debug.c | 2 +-
tools/perf/util/evlist.c | 6 +-
tools/perf/util/evsel.c | 59 +++++++++++++++-
tools/perf/util/evsel.h | 1 +
tools/perf/util/pmus.c | 21 ++++--
tools/perf/util/python.c | 1 +
tools/perf/util/tp_pmu.c | 8 ++-
36 files changed, 378 insertions(+), 237 deletions(-)
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 58+ messages in thread* [PATCH v3 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events
2026-09-29 6:58 ` [PATCH v3 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
@ 2026-09-29 6:58 ` Ian Rogers
2026-09-29 7:08 ` sashiko-bot
2026-09-29 6:58 ` [PATCH v3 2/9] perf python sctop: Fix offline interval printing and test flakiness Ian Rogers
` (8 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
On kernels without the fix ("perf/core: Restore header fields in
sideband output callbacks") to restore event_id.header.size in
perf_event_ksymbol_output(), perf_event_bpf_output(), and
perf_event_text_poke_output(), concurrent perf sessions cause those
sideband records to be emitted with header.size inflated by multiple
id_header_size increments while the single id_sample is written
immediately after the event payload. Indexing backwards from
event->header.size reads uninitialized ring-buffer bytes at the end of
the record, causing evlist__event2evsel() to fail with -EFAULT.
Add evsel__event_size() to clamp the effective size used to locate the
trailing id_sample for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and
PERF_RECORD_TEXT_POKE to payload + id_hdr_size while leaving
event->header.size intact for advancing the ring-buffer/file stream.
Fixes: 9aa0bfa370b2 ("perf tools: Handle PERF_RECORD_KSYMBOL")
Fixes: 45178a928a4b ("perf tools: Handle PERF_RECORD_BPF_EVENT")
Fixes: 246eba8e9041 ("perf tools: Add support for PERF_RECORD_TEXT_POKE")
Link: https://lore.kernel.org/r/20260929014206.4175245-1-irogers@google.com
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/util/evlist.c | 6 ++--
tools/perf/util/evsel.c | 59 +++++++++++++++++++++++++++++++++++++++-
tools/perf/util/evsel.h | 1 +
3 files changed, 63 insertions(+), 3 deletions(-)
diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index 9392d912d254..c2402e4791b6 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -973,13 +973,15 @@ static int evlist__event2id(struct evlist *evlist, union perf_event *event, u64
const __u64 *array = event->sample.array;
ssize_t n;
- n = (event->header.size - sizeof(event->header)) >> 3;
-
if (event->header.type == PERF_RECORD_SAMPLE) {
+ n = (event->header.size - sizeof(event->header)) >> 3;
if (evlist__id_pos(evlist) >= n)
return -1;
*id = array[evlist__id_pos(evlist)];
} else {
+ u16 size = evsel__event_size(evlist__first(evlist), event);
+
+ n = (size - sizeof(event->header)) >> 3;
if (evlist__is_pos(evlist) > n)
return -1;
n -= evlist__is_pos(evlist);
diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index 3367242c5764..9c5e7510f0c0 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -3216,7 +3216,7 @@ static int perf_evsel__parse_id_sample(const union perf_event *event,
const __u64 *array = event->sample.array;
bool swapped = evsel->needs_swap;
union u64_swap u;
- int i = ((event->header.size - sizeof(event->header)) / sizeof(u64)) - 1;
+ int i = ((evsel__event_size(evsel, event) - sizeof(event->header)) / sizeof(u64)) - 1;
if (type & PERF_SAMPLE_IDENTIFIER) {
if (i < 0)
@@ -3966,6 +3966,63 @@ u16 evsel__id_hdr_size(const struct evsel *evsel)
return size;
}
+/*
+ * Prior to kernel fix, perf_event_ksymbol_output(), perf_event_bpf_output(),
+ * and perf_event_text_poke_output() in kernel/events/core.c did not save and
+ * restore event_id.header.size across perf_iterate_sb() iterations. When
+ * multiple perf_events had attr.ksymbol, attr.bpf_event, or attr.text_poke
+ * enabled, header.size was incremented by id_header_size for each matching
+ * event while only a single id_sample was written immediately after the event
+ * payload. Clamp the effective size used to locate the trailing id_sample to
+ * payload + id_hdr_size so events recorded on unpatched kernels can be parsed
+ * without reading uninitialized ring-buffer bytes.
+ */
+u16 evsel__event_size(const struct evsel *evsel, const union perf_event *event)
+{
+ u16 size = event->header.size;
+ u16 id_hdr_size;
+ size_t payload;
+
+ if (!evsel->core.attr.sample_id_all)
+ return size;
+
+ switch (event->header.type) {
+ case PERF_RECORD_KSYMBOL: {
+ const char *name = event->ksymbol.name;
+ size_t fixed = offsetof(struct perf_record_ksymbol, name);
+ size_t max_len, len;
+
+ if (size <= fixed)
+ return size;
+ max_len = size - fixed;
+ len = strnlen(name, max_len);
+ if (len == max_len)
+ return size;
+ payload = fixed + PERF_ALIGN(len + 1, sizeof(u64));
+ break;
+ }
+ case PERF_RECORD_BPF_EVENT:
+ payload = sizeof(struct perf_record_bpf_event);
+ break;
+ case PERF_RECORD_TEXT_POKE: {
+ size_t fixed = offsetof(struct perf_record_text_poke_event, bytes);
+
+ if (size < fixed)
+ return size;
+ payload = PERF_ALIGN(fixed + (size_t)event->text_poke.old_len +
+ event->text_poke.new_len, sizeof(u64));
+ break;
+ }
+ default:
+ return size;
+ }
+
+ id_hdr_size = evsel__id_hdr_size(evsel);
+ if (payload + id_hdr_size < size)
+ return payload + id_hdr_size;
+ return size;
+}
+
#ifdef HAVE_LIBTRACEEVENT
struct tep_format_field *evsel__field(struct evsel *evsel, const char *name)
{
diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
index 5c5799cee601..174f3414fd3c 100644
--- a/tools/perf/util/evsel.h
+++ b/tools/perf/util/evsel.h
@@ -469,6 +469,7 @@ int evsel__parse_sample_timestamp(struct evsel *evsel, union perf_event *event,
u64 *timestamp);
u16 evsel__id_hdr_size(const struct evsel *evsel);
+u16 evsel__event_size(const struct evsel *evsel, const union perf_event *event);
static inline struct evsel *evsel__next(struct evsel *evsel)
{
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v3 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events
2026-09-29 6:58 ` [PATCH v3 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events Ian Rogers
@ 2026-09-29 7:08 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 7:08 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> On kernels without the fix ("perf/core: Restore header fields in
> sideband output callbacks") to restore event_id.header.size in
> perf_event_ksymbol_output(), perf_event_bpf_output(), and
> perf_event_text_poke_output(), concurrent perf sessions cause those
> sideband records to be emitted with header.size inflated by multiple
> id_header_size increments while the single id_sample is written
> immediately after the event payload. Indexing backwards from
> event->header.size reads uninitialized ring-buffer bytes at the end of
> the record, causing evlist__event2evsel() to fail with -EFAULT.
>
> Add evsel__event_size() to clamp the effective size used to locate the
> trailing id_sample for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and
> PERF_RECORD_TEXT_POKE to payload + id_hdr_size while leaving
> event->header.size intact for advancing the ring-buffer/file stream.
>
> Fixes: 9aa0bfa370b2 ("perf tools: Handle PERF_RECORD_KSYMBOL")
> Fixes: 45178a928a4b ("perf tools: Handle PERF_RECORD_BPF_EVENT")
> Fixes: 246eba8e9041 ("perf tools: Add support for PERF_RECORD_TEXT_POKE")
> Link: https://lore.kernel.org/r/20260929014206.4175245-1-irogers@google.com
> Assisted-by: Antigravity:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929065839.407062-1-irogers@google.com?part=1
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v3 2/9] perf python sctop: Fix offline interval printing and test flakiness
2026-09-29 6:58 ` [PATCH v3 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
2026-09-29 6:58 ` [PATCH v3 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events Ian Rogers
@ 2026-09-29 6:58 ` Ian Rogers
2026-09-29 7:06 ` sashiko-bot
2026-09-29 6:58 ` [PATCH v3 3/9] perf python stat-cpi: Fix live mode signal races " Ian Rogers
` (7 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
In sctop.py:
- If an earlier interval elapsed and printed an empty table before the
target comm ('sleep') executed any syscalls, analyzer.printed became
True and the final partial interval containing the target comm's
syscalls was never flushed at EOF. Flush print_current_totals() in
finally when analyzer.syscalls is non-empty as well as when nothing
has been printed yet.
- Initialize analyzer.e_machine after creating perf.session rather than
when session is still None.
In test_sctop_python.sh:
- Use a private temporary directory via 'mktemp -d'.
- Drop '-a' and pass '-B -N --no-bpf-event' to 'perf record', and sleep
briefly in the subshell ('sh -c "sleep 0.1; sleep 0.05"') with a
bounded retry loop so 'sleep's PERF_RECORD_COMM and sys_enter events
are reliably captured under heavy load.
Fixes: b83f0bacf5e4 ("perf python: Port sctop to perf module")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/python/sctop.py | 7 ++-
tools/perf/tests/shell/test_sctop_python.sh | 58 ++++++++++-----------
2 files changed, 34 insertions(+), 31 deletions(-)
diff --git a/tools/perf/python/sctop.py b/tools/perf/python/sctop.py
index 42e95ecfffad..48320af8b755 100755
--- a/tools/perf/python/sctop.py
+++ b/tools/perf/python/sctop.py
@@ -37,6 +37,7 @@ class SCTopAnalyzer:
self.offline = offline
self.own_pid = os.getpid()
self.last_print_time: Optional[int] = None
+ self.printed = False
self.session: Optional[perf.session] = None
self.e_machine: Optional[int] = None
@@ -137,6 +138,7 @@ class SCTopAnalyzer:
def print_current_totals(self):
"""Print current syscall totals."""
+ self.printed = True
# Clear terminal
if not self.offline:
print("\x1b[2J\x1b[H", end="")
@@ -217,8 +219,8 @@ def main():
if args.input:
session = perf.session(perf.data(args.input), sample=analyzer.process_event)
analyzer.session = session
- session.process_events()
analyzer.e_machine = getattr(session, "e_machine", None)
+ session.process_events()
else:
try:
live_session = LiveSession(
@@ -237,7 +239,8 @@ def main():
sys.exit(1)
finally:
if args.input:
- analyzer.print_current_totals()
+ if not analyzer.printed or analyzer.syscalls:
+ analyzer.print_current_totals()
# Break the reference cycle between perf.session and analyzer.process_event
# because perf.session lacks cyclic GC support (tp_traverse).
analyzer.session = None
diff --git a/tools/perf/tests/shell/test_sctop_python.sh b/tools/perf/tests/shell/test_sctop_python.sh
index 007f2584cce6..b042fc3eefe5 100755
--- a/tools/perf/tests/shell/test_sctop_python.sh
+++ b/tools/perf/tests/shell/test_sctop_python.sh
@@ -27,51 +27,51 @@ if [ ! -f "$script_path" ]; then
fi
err=0
-temp_data=""
-temp_out=""
+temp_dir=$(mktemp -d /tmp/perf-sctop-XXXXXX)
+temp_data="${temp_dir}/perf.data"
+temp_out="${temp_dir}/perf.out"
cleanup() {
- rm -f "${temp_data}" "${temp_out}"
+ rm -rf "${temp_dir}"
}
trap 'cleanup' EXIT TERM INT
-temp_data=$(mktemp /tmp/perf.data.XXXXXX)
-temp_out=$(mktemp /tmp/perf.out.XXXXXX)
-
echo "Testing sctop.py..."
# Create a perf.data file.
-if perf list | grep -q "raw_syscalls:sys_enter"; then
- perf record -e raw_syscalls:sys_enter -a -o "${temp_data}" \
- -- sleep 0.1 >/dev/null 2>&1 || \
- { echo "Skipping test, perf record failed"; exit 2; }
-else
+if ! perf list tracepoint | grep -q "raw_syscalls:sys_enter"; then
echo "Skipping test, no raw_syscalls:sys_enter event"
exit 2
fi
-if [ ! -s "${temp_data}" ]; then
- echo "Skipping test, perf record failed to create data"
- exit 2
-fi
+passed=0
+for _ in 1 2 3 4 5; do
+ rm -f "${temp_data}" "${temp_out}"
+ if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \
+ -- sh -c "sleep 0.1; sleep 0.05" >/dev/null 2>&1; then
+ echo "Skipping test, perf record failed"
+ exit 2
+ fi
-# Check that the script executes
-if ! perf script sctop -i "${temp_data}" > "${temp_out}"; then
+ if [ ! -s "${temp_data}" ]; then
+ continue
+ fi
+
+ # Check that the script executes
+ if perf script sctop -i "${temp_data}" > "${temp_out}" && \
+ grep -E -q "[0-9]+$" "${temp_out}" && \
+ perf script sctop -i "${temp_data}" sleep 1 > "${temp_out}" && \
+ grep -E -q "[0-9]+$" "${temp_out}"; then
+ passed=1
+ break
+ fi
+done
+
+if [ "$passed" -eq 0 ]; then
echo "sctop.py test failed"
err=1
-elif ! grep -E -q "[0-9]+$" "${temp_out}"; then
- echo "Failed to find metric data rows in default run"
- err=1
-elif ! perf script sctop -i "${temp_data}" sleep 1 > "${temp_out}"; then
- echo "sctop.py comm+interval test failed"
- err=1
else
- if ! grep -E -q "[0-9]+$" "${temp_out}"; then
- echo "Failed to find metric data rows"
- err=1
- else
- echo "sctop test passed."
- fi
+ echo "sctop test passed."
fi
rm -f "${temp_out}"
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v3 2/9] perf python sctop: Fix offline interval printing and test flakiness
2026-09-29 6:58 ` [PATCH v3 2/9] perf python sctop: Fix offline interval printing and test flakiness Ian Rogers
@ 2026-09-29 7:06 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 7:06 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> In sctop.py:
> - If an earlier interval elapsed and printed an empty table before the
> target comm ('sleep') executed any syscalls, analyzer.printed became
> True and the final partial interval containing the target comm's
> syscalls was never flushed at EOF. Flush print_current_totals() in
> finally when analyzer.syscalls is non-empty as well as when nothing
> has been printed yet.
> - Initialize analyzer.e_machine after creating perf.session rather than
> when session is still None.
>
> In test_sctop_python.sh:
> - Use a private temporary directory via 'mktemp -d'.
> - Drop '-a' and pass '-B -N --no-bpf-event' to 'perf record', and sleep
> briefly in the subshell ('sh -c "sleep 0.1; sleep 0.05"') with a
> bounded retry loop so 'sleep's PERF_RECORD_COMM and sys_enter events
> are reliably captured under heavy load.
>
> Fixes: b83f0bacf5e4 ("perf python: Port sctop to perf module")
> Assisted-by: Antigravity:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929065839.407062-1-irogers@google.com?part=2
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v3 3/9] perf python stat-cpi: Fix live mode signal races and test flakiness
2026-09-29 6:58 ` [PATCH v3 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
2026-09-29 6:58 ` [PATCH v3 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events Ian Rogers
2026-09-29 6:58 ` [PATCH v3 2/9] perf python sctop: Fix offline interval printing and test flakiness Ian Rogers
@ 2026-09-29 6:58 ` Ian Rogers
2026-09-29 7:04 ` sashiko-bot
2026-09-29 6:58 ` [PATCH v3 4/9] perf test: Deflake Intel PT Python shell tests under load Ian Rogers
` (6 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
In test_stat_cpi_python.sh, 'perf test -w noploop &' defaults to a
1-second duration and can exit under heavy parallel load before
'perf stat -p' and 'perf script stat-cpi' finish starting up. In
addition, the fixed 'sleep 0.5' before sending SIGINT can fire before
Python finishes importing the perf module, opening the live evlist, and
flushing the first interval.
In stat-cpi.py, register SIGINT and SIGTERM handlers before calling
_open_live_evlist() and pass flush=True when printing live output so
redirected stdout is flushed immediately after each interval.
In test_stat_cpi_python.sh, run 'perf test -w noploop 60 &' so the
target workload stays alive until killed, and poll the output file for
'cpi' (up to 5 seconds) before sending SIGINT.
Fixes: 4425182d426b ("perf python: Port stat-cpi to perf module")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/python/stat-cpi.py | 32 +++++++++++++------
.../perf/tests/shell/test_stat_cpi_python.sh | 12 +++++--
2 files changed, 31 insertions(+), 13 deletions(-)
diff --git a/tools/perf/python/stat-cpi.py b/tools/perf/python/stat-cpi.py
index 0b7d76876a6c..b4ec87b07938 100755
--- a/tools/perf/python/stat-cpi.py
+++ b/tools/perf/python/stat-cpi.py
@@ -106,7 +106,8 @@ class StatCpiAnalyzer:
if ins != 0:
cpi = cyc / float(ins)
t_sec = timestamp / 1000000000.0
- print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})")
+ print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})",
+ flush=True)
def read_counters(self, evlist: Any) -> None:
"""Read counters live."""
@@ -151,6 +152,7 @@ class StatCpiAnalyzer:
last_err: Optional[OSError] = None
for events, tmap in candidates:
+ evlist = None
try:
evlist = perf.parse_events(events, None, tmap)
for evsel in evlist:
@@ -161,32 +163,41 @@ class StatCpiAnalyzer:
evlist.enable()
return evlist
except PermissionError as e:
+ if evlist is not None:
+ evlist.close()
last_err = e
except OSError as e:
+ if evlist is not None:
+ evlist.close()
if e.errno == 13:
last_err = e
else:
raise
+ except BaseException:
+ if evlist is not None:
+ evlist.close()
+ raise
if last_err is not None:
raise last_err
raise RuntimeError("Failed to open events")
def run_live(self) -> None:
"""Read counters live."""
- try:
- evlist = self._open_live_evlist()
- except OSError as e:
- print(f"Failed to open events: {e}", file=sys.stderr)
- sys.exit(1)
-
def handle_signal(_signum: int, _frame: Any) -> None:
raise KeyboardInterrupt
signal.signal(signal.SIGINT, signal.default_int_handler)
signal.signal(signal.SIGTERM, handle_signal)
- print("Live mode started. Press Ctrl+C to stop.")
+ evlist = None
try:
+ try:
+ evlist = self._open_live_evlist()
+ except OSError as e:
+ print(f"Failed to open events: {e}", file=sys.stderr)
+ sys.exit(1)
+
+ print("Live mode started. Press Ctrl+C to stop.", flush=True)
while True:
time.sleep(self.args.interval)
timestamp = time.time_ns()
@@ -195,9 +206,10 @@ class StatCpiAnalyzer:
self.data.clear()
self.recorded_pairs.clear()
except KeyboardInterrupt:
- print("\nStopped.")
+ print("\nStopped.", flush=True)
finally:
- evlist.close()
+ if evlist is not None:
+ evlist.close()
def main() -> None:
"""Main function."""
diff --git a/tools/perf/tests/shell/test_stat_cpi_python.sh b/tools/perf/tests/shell/test_stat_cpi_python.sh
index fe7562307634..6cb376c92e2f 100755
--- a/tools/perf/tests/shell/test_stat_cpi_python.sh
+++ b/tools/perf/tests/shell/test_stat_cpi_python.sh
@@ -50,7 +50,7 @@ test_live_mode() {
echo "perf stat failed (permissions?), skipping live mode test."
return 0
fi
- perf test -w noploop &
+ perf test -w noploop 60 &
workload_pid=$!
if ! perf stat -e cycles,instructions -p "$workload_pid" -- sleep 0.05 2>/dev/null && \
! perf stat -e cycles:u,instructions:u -p "$workload_pid" -- sleep 0.05 2>/dev/null; then
@@ -61,10 +61,16 @@ test_live_mode() {
fi
ran=1
- # Run live mode for 1 interval in the background, give it a tiny sleep, then interrupt
+ # Run live mode in the background, wait until at least one interval is
+ # printed, then interrupt.
perf script stat-cpi -I 0.1 -p "$workload_pid" > "${temp_out}" &
pid=$!
- sleep 0.5
+ for _ in $(seq 1 50); do
+ if grep -q "cpi" "${temp_out}"; then
+ break
+ fi
+ sleep 0.1
+ done
kill -INT "$pid" 2>/dev/null || true
set +e
wait "$pid"
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v3 3/9] perf python stat-cpi: Fix live mode signal races and test flakiness
2026-09-29 6:58 ` [PATCH v3 3/9] perf python stat-cpi: Fix live mode signal races " Ian Rogers
@ 2026-09-29 7:04 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 7:04 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> In test_stat_cpi_python.sh, 'perf test -w noploop &' defaults to a
> 1-second duration and can exit under heavy parallel load before
> 'perf stat -p' and 'perf script stat-cpi' finish starting up. In
> addition, the fixed 'sleep 0.5' before sending SIGINT can fire before
> Python finishes importing the perf module, opening the live evlist, and
> flushing the first interval.
>
> In stat-cpi.py, register SIGINT and SIGTERM handlers before calling
> _open_live_evlist() and pass flush=True when printing live output so
> redirected stdout is flushed immediately after each interval.
>
> In test_stat_cpi_python.sh, run 'perf test -w noploop 60 &' so the
> target workload stays alive until killed, and poll the output file for
> 'cpi' (up to 5 seconds) before sending SIGINT.
>
> Fixes: 4425182d426b ("perf python: Port stat-cpi to perf module")
> Assisted-by: Antigravity:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929065839.407062-1-irogers@google.com?part=3
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v3 4/9] perf test: Deflake Intel PT Python shell tests under load
2026-09-29 6:58 ` [PATCH v3 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (2 preceding siblings ...)
2026-09-29 6:58 ` [PATCH v3 3/9] perf python stat-cpi: Fix live mode signal races " Ian Rogers
@ 2026-09-29 6:58 ` Ian Rogers
2026-09-29 7:03 ` sashiko-bot
2026-09-29 6:58 ` [PATCH v3 5/9] perf test: Deflake failed-syscalls " Ian Rogers
` (5 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
In test_intel_pt_events_python.sh, test_export_to_sqlite_python.sh, and
test_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the
shell builtin 'true' and exits within microseconds of 'uname', sending
SIGCHLD to 'perf record' before the Intel PT AUX buffer is always
flushed under heavy parallel load (~5-10% drop rate).
Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"')
so 'uname' completely exits and flushes its AUX trace before 'sh' exits,
and wrap the record and verification step in a bounded retry loop (up to
5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in
test_export_to_sqlite_python.sh and test_export_to_postgresql_python.sh
to avoid build-id cache and BPF synthesis overhead.
Fixes: d4ce72e9e238 ("perf python: Port intel-pt-events and libxed to perf module")
Fixes: 62d350135e67 ("perf python: Port export-to-sqlite to perf module")
Fixes: b1f968c9656a ("perf python: Port export-to-postgresql to perf module")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
.../shell/test_export_to_postgresql_python.sh | 49 ++++++++++--------
.../shell/test_export_to_sqlite_python.sh | 50 +++++++++++--------
.../shell/test_intel_pt_events_python.sh | 43 +++++++++-------
3 files changed, 79 insertions(+), 63 deletions(-)
diff --git a/tools/perf/tests/shell/test_export_to_postgresql_python.sh b/tools/perf/tests/shell/test_export_to_postgresql_python.sh
index 350813466700..835f48f002e9 100755
--- a/tools/perf/tests/shell/test_export_to_postgresql_python.sh
+++ b/tools/perf/tests/shell/test_export_to_postgresql_python.sh
@@ -61,9 +61,10 @@ test_file_mode() {
fi
# Generate events with callchains and context switches
- if ! perf record -g --switch-events -o "${temp_data}" \
+ if ! perf record -B -N --no-bpf-event -g --switch-events -o "${temp_data}" \
-- perf test -w noploop >/dev/null 2>&1 && \
- ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+ ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping test, perf record failed"
exit 2
fi
@@ -92,29 +93,33 @@ test_file_mode() {
test_intel_pt() {
echo "Testing export-to-postgresql.py with intel_pt..."
- psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true
- rm -f "${temp_data}"
- # Generate some intel_pt events; use a subshell that waits for uname
- if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
- -- sh -c "uname; true" >/dev/null 2>&1; then
- echo "Skipping intel_pt test, intel_pt not available."
- return 0
- fi
+ # Generate some intel_pt events; sleep briefly after uname in the subshell
+ # so uname's AUX buffer is flushed before SIGCHLD stops perf record.
+ passed=0
+ for _ in 1 2 3 4 5; do
+ psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true
+ rm -f "${temp_data}"
+ if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
+ -- sh -c "uname; sleep 0.05" >/dev/null 2>&1; then
+ echo "Skipping intel_pt test, intel_pt not available."
+ return 0
+ fi
- # Run the script with --itrace cr to synthesize call_returns
- if ! perf script export-to-postgresql -i "${temp_data}" \
- -o "${temp_db}" --itrace cr >/dev/null; then
- echo "intel_pt file mode test failed."
- err=1
- else
- # Check DB for calls
- if ! psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \
+ # Run the script with --itrace cr to synthesize call_returns
+ if perf script export-to-postgresql -i "${temp_data}" \
+ -o "${temp_db}" --itrace cr >/dev/null && \
+ psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \
grep -q '[1-9]'; then
- echo "PostgreSQL intel_pt validation failed (no calls found)."
- err=1
- else
- echo "intel_pt test passed (cr validated)."
+ passed=1
+ break
fi
+ done
+
+ if [ "$passed" -eq 0 ]; then
+ echo "PostgreSQL intel_pt validation failed (no calls found)."
+ err=1
+ else
+ echo "intel_pt test passed (cr validated)."
fi
}
diff --git a/tools/perf/tests/shell/test_export_to_sqlite_python.sh b/tools/perf/tests/shell/test_export_to_sqlite_python.sh
index d3c5e22a0754..19ca7c539cf7 100755
--- a/tools/perf/tests/shell/test_export_to_sqlite_python.sh
+++ b/tools/perf/tests/shell/test_export_to_sqlite_python.sh
@@ -47,9 +47,10 @@ test_file_mode() {
echo "Testing export-to-sqlite.py..."
# Generate events with callchains and context switches if supported
- if ! perf record -g --switch-events -o "${temp_data}" \
+ if ! perf record -B -N --no-bpf-event -g --switch-events -o "${temp_data}" \
-- perf test -w noploop >/dev/null 2>&1 && \
- ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+ ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping test, perf record failed"
exit 2
fi
@@ -77,29 +78,34 @@ test_file_mode() {
test_intel_pt() {
echo "Testing export-to-sqlite.py with intel_pt..."
- rm -f "${temp_db}" "${temp_data}"
- # Generate some intel_pt events; use a subshell that waits for uname
- if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
- -- sh -c "uname; true" >/dev/null 2>&1; then
- echo "Skipping intel_pt test, intel_pt not available."
- return 0
- fi
+ query="import sqlite3; c = sqlite3.connect('${temp_db}'); "
+ query="${query}r = c.execute('SELECT COUNT(*) FROM calls').fetchone()[0]; "
+ query="${query}exit(1 if r == 0 else 0)"
+
+ # Generate some intel_pt events; sleep briefly after uname in the subshell
+ # so uname's AUX buffer is flushed before SIGCHLD stops perf record.
+ passed=0
+ for _ in 1 2 3 4 5; do
+ rm -f "${temp_db}" "${temp_data}"
+ if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
+ -- sh -c "uname; sleep 0.05" >/dev/null 2>&1; then
+ echo "Skipping intel_pt test, intel_pt not available."
+ return 0
+ fi
- # Run the script with --itrace cr to synthesize call_returns
- if ! perf script export-to-sqlite -i "${temp_data}" -o "${temp_db}" --itrace cr; then
- echo "intel_pt file mode test failed."
+ # Run the script with --itrace cr to synthesize call_returns
+ if perf script export-to-sqlite -i "${temp_data}" -o "${temp_db}" --itrace cr && \
+ "$PYTHON" -c "$query" >/dev/null 2>&1; then
+ passed=1
+ break
+ fi
+ done
+
+ if [ "$passed" -eq 0 ]; then
+ echo "SQLite intel_pt validation failed (no calls found)."
err=1
else
- # Check DB for calls
- query="import sqlite3; c = sqlite3.connect('${temp_db}'); "
- query="${query}r = c.execute('SELECT COUNT(*) FROM calls').fetchone()[0]; "
- query="${query}exit(1 if r == 0 else 0)"
- if ! "$PYTHON" -c "$query" >/dev/null 2>&1; then
- echo "SQLite intel_pt validation failed (no calls found)."
- err=1
- else
- echo "intel_pt test passed (cr validated)."
- fi
+ echo "intel_pt test passed (cr validated)."
fi
}
diff --git a/tools/perf/tests/shell/test_intel_pt_events_python.sh b/tools/perf/tests/shell/test_intel_pt_events_python.sh
index b5c3173fa2db..9754b9125da1 100755
--- a/tools/perf/tests/shell/test_intel_pt_events_python.sh
+++ b/tools/perf/tests/shell/test_intel_pt_events_python.sh
@@ -30,7 +30,8 @@ cleanup() {
[ -n "${temp_dir}" ] && rm -rf "${temp_dir}"
}
-trap 'cleanup' EXIT TERM INT
+trap 'cleanup' EXIT
+trap 'cleanup; exit 1' TERM INT
temp_dir=$(mktemp -d /tmp/perf.ipt.XXXXXX)
temp_data="${temp_dir}/perf.data"
@@ -39,27 +40,31 @@ temp_out="${temp_dir}/perf.out"
test_intel_pt() {
echo "Testing intel-pt-events.py with intel_pt..."
- rm -f "${temp_data}" "${temp_out}"
- # Generate some intel_pt events; use a subshell that waits for uname so
- # uname's AUX buffer is flushed before the parent workload exits.
- if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
- -- sh -c "uname; true" >/dev/null 2>&1; then
- echo "Skipping intel_pt test, intel_pt not available."
- exit 2
- fi
+ # Generate some intel_pt events; sleep briefly after uname in the subshell
+ # so uname's AUX buffer is flushed before SIGCHLD stops perf record.
+ passed=0
+ for _ in 1 2 3 4 5; do
+ rm -f "${temp_data}" "${temp_out}"
+ if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
+ -- sh -c "uname; sleep 0.05" >/dev/null 2>&1; then
+ echo "Skipping intel_pt test, intel_pt not available."
+ exit 2
+ fi
- # Run the script and check output
- if ! perf script intel-pt-events -i "${temp_data}" > "${temp_out}"; then
- echo "intel-pt-events.py test failed."
+ # Run the script and check output
+ if perf script intel-pt-events -i "${temp_data}" > "${temp_out}" && \
+ grep -q "Intel PT Branch Trace" "${temp_out}" && \
+ grep -q "uname" "${temp_out}"; then
+ passed=1
+ break
+ fi
+ done
+
+ if [ "$passed" -eq 0 ]; then
+ echo "Failed to find expected output: $(cat "${temp_out}" 2>/dev/null)"
err=1
else
- if ! grep -q "Intel PT Branch Trace" "${temp_out}" || \
- ! grep -q "uname" "${temp_out}"; then
- echo "Failed to find expected output: $(cat "${temp_out}")"
- err=1
- else
- echo "intel-pt-events test passed."
- fi
+ echo "intel-pt-events test passed."
fi
rm -f "${temp_out}"
}
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v3 4/9] perf test: Deflake Intel PT Python shell tests under load
2026-09-29 6:58 ` [PATCH v3 4/9] perf test: Deflake Intel PT Python shell tests under load Ian Rogers
@ 2026-09-29 7:03 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 7:03 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> In test_intel_pt_events_python.sh, test_export_to_sqlite_python.sh, and
> test_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the
> shell builtin 'true' and exits within microseconds of 'uname', sending
> SIGCHLD to 'perf record' before the Intel PT AUX buffer is always
> flushed under heavy parallel load (~5-10% drop rate).
>
> Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"')
> so 'uname' completely exits and flushes its AUX trace before 'sh' exits,
> and wrap the record and verification step in a bounded retry loop (up to
> 5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in
> test_export_to_sqlite_python.sh and test_export_to_postgresql_python.sh
> to avoid build-id cache and BPF synthesis overhead.
>
> Fixes: d4ce72e9e238 ("perf python: Port intel-pt-events and libxed to perf module")
> Fixes: 62d350135e67 ("perf python: Port export-to-sqlite to perf module")
> Fixes: b1f968c9656a ("perf python: Port export-to-postgresql to perf module")
> Assisted-by: Antigravity:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929065839.407062-1-irogers@google.com?part=4
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v3 5/9] perf test: Deflake failed-syscalls Python shell tests under load
2026-09-29 6:58 ` [PATCH v3 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (3 preceding siblings ...)
2026-09-29 6:58 ` [PATCH v3 4/9] perf test: Deflake Intel PT Python shell tests under load Ian Rogers
@ 2026-09-29 6:58 ` Ian Rogers
2026-09-29 7:04 ` sashiko-bot
2026-09-29 6:58 ` [PATCH v3 6/9] perf test: Reduce overhead and contention in Python shell tests Ian Rogers
` (4 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
When running 'perf test' in parallel under heavy system load, a one-shot
'perf record -e ... -- ls /does_not_exist' without '-B -N --no-bpf-event'
can exit in under 300 microseconds before 'ls's PERF_RECORD_COMM and
ENOENT sys_exit events are captured, while also contending on ~/.debug
build-id caching and BPF event synthesis.
Pass '-B -N --no-bpf-event' to 'perf record', sleep 0.05s in a subshell
after 'ls' so its events are flushed before the parent subshell exits,
and wrap the record and check steps in a bounded retry loop (up to 5
attempts).
Fixes: b76c43d09b06 ("perf python: Port failed-syscalls-by-pid to perf module")
Fixes: 4e3fe6987cba ("perf python: Port failed-syscalls from Perl to perf module")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
.../test_failed_syscalls_by_pid_python.sh | 68 +++++++++++--------
.../shell/test_failed_syscalls_python.sh | 54 +++++++++------
2 files changed, 71 insertions(+), 51 deletions(-)
diff --git a/tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh b/tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh
index 215372d4e1c5..070a7ef6b8ad 100755
--- a/tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh
+++ b/tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh
@@ -41,8 +41,10 @@ test_file_mode() {
echo "Testing failed-syscalls-by-pid.py..."
# Check if syscalls:sys_exit is supported/readable
- if ! perf record -e syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
- if ! perf record -e raw_syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -e syscalls:sys_exit \
+ -o /dev/null -- true >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_exit \
+ -o /dev/null -- true >/dev/null 2>&1; then
echo "Skipping test, no syscalls:sys_exit or raw_syscalls:sys_exit event"
exit 2
else
@@ -52,41 +54,49 @@ test_file_mode() {
EVENT="syscalls:sys_exit"
fi
- # Generate some events by running a command that should fail at least some syscall
- # (e.g. failing stat on non-existent file).
- # Using '|| true' because 'perf record' returns the exit code of 'ls',
- # which fails with ENOENT
- perf record -e "${EVENT}" -o "${temp_data}" -- ls /does_not_exist >/dev/null 2>&1 || true
+ # Generate some events by running a command that fails a syscall
+ # (e.g. failing stat on non-existent file), sleeping briefly in the
+ # subshell so ls's PERF_RECORD_COMM and sys_exit events are flushed.
+ passed=0
+ for _ in 1 2 3 4 5; do
+ rm -f "${temp_data}" "${temp_out}" "${temp_out}.comm" "${temp_out}.pid"
+ perf record -B -N --no-bpf-event -e "${EVENT}" -o "${temp_data}" \
+ -- sh -c "ls /does_not_exist 2>/dev/null; sleep 0.05 || true" \
+ >/dev/null 2>&1 || true
+ if [ ! -s "${temp_data}" ]; then
+ continue
+ fi
+
+ if perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}" && \
+ grep -n -q "err = ENOENT" "${temp_out}" && \
+ perf script failed-syscalls-by-pid -i "${temp_data}" \
+ "ls" > "${temp_out}.comm" && \
+ grep -q "err = ENOENT" "${temp_out}.comm"; then
+ ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' \
+ "${temp_out}.comm" | head -n 1)
+ if [ -n "${ls_pid}" ] && \
+ perf script failed-syscalls-by-pid -i "${temp_data}" \
+ "${ls_pid}" > "${temp_out}.pid" && \
+ grep -q "err = ENOENT" "${temp_out}.pid"; then
+ passed=1
+ break
+ fi
+ fi
+ done
+
if [ ! -s "${temp_data}" ]; then
echo "Skipping test, perf record failed to create data"
exit 2
fi
- # Run the script and check output
- if ! perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}"; then
+ if [ "$passed" -eq 0 ]; then
echo "failed-syscalls-by-pid test failed."
- err=1
- elif ! grep -n -q "err = ENOENT" "${temp_out}"; then
- echo "Failed to find expected failed syscalls"
- cat "${temp_out}"
- err=1
- elif ! perf script failed-syscalls-by-pid -i "${temp_data}" "ls" > "${temp_out}.comm" || \
- ! grep -q "err = ENOENT" "${temp_out}.comm"; then
- echo "failed-syscalls-by-pid comm filter test failed."
- cat "${temp_out}.comm"
+ cat "${temp_out}" 2>/dev/null || true
+ cat "${temp_out}.comm" 2>/dev/null || true
+ cat "${temp_out}.pid" 2>/dev/null || true
err=1
else
- ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' "${temp_out}.comm" | head -n 1)
- if [ -z "${ls_pid}" ] || \
- ! perf script failed-syscalls-by-pid -i "${temp_data}" \
- "${ls_pid}" > "${temp_out}.pid" || \
- ! grep -q "err = ENOENT" "${temp_out}.pid"; then
- echo "failed-syscalls-by-pid PID filter test failed."
- cat "${temp_out}.pid" 2>/dev/null || true
- err=1
- else
- echo "failed-syscalls-by-pid test passed."
- fi
+ echo "failed-syscalls-by-pid test passed."
fi
rm -f "${temp_out}" "${temp_out}.comm" "${temp_out}.pid"
}
diff --git a/tools/perf/tests/shell/test_failed_syscalls_python.sh b/tools/perf/tests/shell/test_failed_syscalls_python.sh
index 861c2ba71c0a..9aaf05883c50 100755
--- a/tools/perf/tests/shell/test_failed_syscalls_python.sh
+++ b/tools/perf/tests/shell/test_failed_syscalls_python.sh
@@ -27,22 +27,22 @@ if [ ! -f "$script_path" ]; then
fi
err=0
-temp_data=""
-temp_out=""
+temp_dir=$(mktemp -d /tmp/perf-failed-syscalls-XXXXXX)
+temp_data="${temp_dir}/perf.data"
+temp_out="${temp_dir}/perf.out"
cleanup() {
- rm -f "${temp_data}" "${temp_out}"
+ rm -rf "${temp_dir}"
}
trap 'cleanup' EXIT TERM INT
-temp_data=$(mktemp /tmp/perf.data.XXXXXX)
-temp_out=$(mktemp /tmp/perf.out.XXXXXX)
-
echo "Testing failed-syscalls.py..."
# Check if sys_exit event can be recorded
-if ! perf record -e raw_syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
- if ! perf record -e syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
+if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_exit \
+ -o /dev/null -- true >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -e syscalls:sys_exit \
+ -o /dev/null -- true >/dev/null 2>&1; then
echo "Skipping test, no permission or support for sys_exit event"
exit 2
else
@@ -52,28 +52,38 @@ else
EVENT="raw_syscalls:sys_exit"
fi
-# Run perf record with a command that fails a syscall (ls non-existent file).
-# ls exits with non-zero, so perf record returns non-zero exit code of the workload.
-perf record -e "${EVENT}" -o "${temp_data}" \
- -- ls /nonexistent_file_for_test >/dev/null 2>&1 || true
+# Run perf record with a command that fails a syscall (ls non-existent file),
+# sleeping briefly in the subshell so ls's PERF_RECORD_COMM and sys_exit events are flushed.
+passed=0
+for _ in 1 2 3 4 5; do
+ rm -f "${temp_data}" "${temp_out}"
+ perf record -B -N --no-bpf-event -e "${EVENT}" -o "${temp_data}" \
+ -- sh -c "ls /nonexistent_file_for_test 2>/dev/null; sleep 0.05 || true" \
+ >/dev/null 2>&1 || true
+
+ if [ ! -s "${temp_data}" ]; then
+ continue
+ fi
+
+ # Check that the script executes
+ if perf script failed-syscalls -i "${temp_data}" > "${temp_out}" && \
+ grep -q "failed syscalls by comm" "${temp_out}" && \
+ grep -Eq '^ls[[:space:]]+[0-9]+' "${temp_out}"; then
+ passed=1
+ break
+ fi
+done
if [ ! -s "${temp_data}" ]; then
echo "Skipping test, perf record failed to create data"
exit 2
fi
-# Check that the script executes
-if ! perf script failed-syscalls -i "${temp_data}" > "${temp_out}"; then
- echo "failed-syscalls.py test failed"
+if [ "$passed" -eq 0 ]; then
+ echo "Failed to find the metrics table header or expected error"
err=1
else
- if ! grep -q "failed syscalls by comm" "${temp_out}" || \
- ! grep -Eq '^ls[[:space:]]+[0-9]+' "${temp_out}"; then
- echo "Failed to find the metrics table header or expected error"
- err=1
- else
- echo "failed-syscalls test passed."
- fi
+ echo "failed-syscalls test passed."
fi
rm -f "${temp_out}"
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v3 5/9] perf test: Deflake failed-syscalls Python shell tests under load
2026-09-29 6:58 ` [PATCH v3 5/9] perf test: Deflake failed-syscalls " Ian Rogers
@ 2026-09-29 7:04 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 7:04 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> When running 'perf test' in parallel under heavy system load, a one-shot
> 'perf record -e ... -- ls /does_not_exist' without '-B -N --no-bpf-event'
> can exit in under 300 microseconds before 'ls's PERF_RECORD_COMM and
> ENOENT sys_exit events are captured, while also contending on ~/.debug
> build-id caching and BPF event synthesis.
>
> Pass '-B -N --no-bpf-event' to 'perf record', sleep 0.05s in a subshell
> after 'ls' so its events are flushed before the parent subshell exits,
> and wrap the record and check steps in a bounded retry loop (up to 5
> attempts).
>
> Fixes: b76c43d09b06 ("perf python: Port failed-syscalls-by-pid to perf module")
> Fixes: 4e3fe6987cba ("perf python: Port failed-syscalls from Perl to perf module")
> Assisted-by: Antigravity:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929065839.407062-1-irogers@google.com?part=5
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v3 6/9] perf test: Reduce overhead and contention in Python shell tests
2026-09-29 6:58 ` [PATCH v3 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (4 preceding siblings ...)
2026-09-29 6:58 ` [PATCH v3 5/9] perf test: Deflake failed-syscalls " Ian Rogers
@ 2026-09-29 6:58 ` Ian Rogers
2026-09-29 7:04 ` sashiko-bot
2026-09-29 6:58 ` [PATCH v3 7/9] perf python event_analyzing_sample: Default to in-memory SQLite database Ian Rogers
` (3 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
When all Python shell tests run concurrently during 'perf test' under
heavy load:
1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in
each test process; use 'perf list tracepoint' when checking for
tracepoint events.
2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF
events across the system and caches build-ids into ~/.debug; add
'-B -N --no-bpf-event' to all Python shell test recordings.
3. Drop '-a' in test_check_perf_trace_python.sh,
test_rw_by_file_python.sh, test_rw_by_pid_python.sh,
test_rwtop_python.sh, and test_syscall_counts_by_pid_python.sh where
only a single child workload ('dd' or 'sleep') is tested, avoiding
system-wide /proc synthesis and ringbuffer contention.
4. Add '-W 1' to 'ping -c 1' in test_net_dropmonitor_python.sh and
test_netdev_times_python.sh, and narrow 'compaction:*' to
'compaction:mm_compaction_begin,compaction:mm_compaction_end' in
test_compaction_times_python.sh.
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/tests/shell/test_check_perf_trace_python.sh | 8 +++++---
tools/perf/tests/shell/test_compaction_times_python.sh | 8 +++++---
.../tests/shell/test_event_analyzing_sample_python.sh | 3 ++-
tools/perf/tests/shell/test_flamegraph_python.sh | 5 +++--
tools/perf/tests/shell/test_futex_contention_python.sh | 7 ++++---
tools/perf/tests/shell/test_gecko_python.sh | 3 ++-
tools/perf/tests/shell/test_mem_phys_addr_python.sh | 8 +++++---
tools/perf/tests/shell/test_net_dropmonitor_python.sh | 9 +++++----
tools/perf/tests/shell/test_netdev_times_python.sh | 6 +++---
tools/perf/tests/shell/test_powerpc_hcalls_python.sh | 4 ++--
tools/perf/tests/shell/test_rw_by_file_python.sh | 5 +++--
tools/perf/tests/shell/test_rw_by_pid_python.sh | 4 ++--
tools/perf/tests/shell/test_rwtop_python.sh | 4 ++--
tools/perf/tests/shell/test_sched_migration_python.sh | 4 ++--
tools/perf/tests/shell/test_stackcollapse_python.sh | 2 +-
.../tests/shell/test_syscall_counts_by_pid_python.sh | 6 +++---
tools/perf/tests/shell/test_syscall_counts_python.sh | 5 +++--
tools/perf/tests/shell/test_task_analyzer.sh | 3 ++-
tools/perf/tests/shell/test_wakeup_latency_python.sh | 7 ++++---
19 files changed, 58 insertions(+), 43 deletions(-)
diff --git a/tools/perf/tests/shell/test_check_perf_trace_python.sh b/tools/perf/tests/shell/test_check_perf_trace_python.sh
index 92cc7b1f0033..baf55945d429 100755
--- a/tools/perf/tests/shell/test_check_perf_trace_python.sh
+++ b/tools/perf/tests/shell/test_check_perf_trace_python.sh
@@ -45,10 +45,11 @@ test_file_mode() {
echo "Testing check-perf-trace.py..."
events=""
- if perf list | grep -q "irq:softirq_entry"; then
+ tp_list=$(perf list tracepoint)
+ if echo "$tp_list" | grep -q "irq:softirq_entry"; then
events="irq:softirq_entry"
fi
- if perf list | grep -q "kmem:kmalloc"; then
+ if echo "$tp_list" | grep -q "kmem:kmalloc"; then
if [ -n "$events" ]; then
events="$events,kmem:kmalloc,kmem:kfree"
else
@@ -62,7 +63,8 @@ test_file_mode() {
fi
# Generate events
- if ! perf record -e "$events" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -e "$events" -o "${temp_data}" \
+ -- sh -c "sleep 0.1" >/dev/null 2>&1; then
echo "Skipping test, perf record failed"
exit 2
fi
diff --git a/tools/perf/tests/shell/test_compaction_times_python.sh b/tools/perf/tests/shell/test_compaction_times_python.sh
index 50397afaafed..3f5a71e52388 100755
--- a/tools/perf/tests/shell/test_compaction_times_python.sh
+++ b/tools/perf/tests/shell/test_compaction_times_python.sh
@@ -44,15 +44,17 @@ test_file_mode() {
echo "Testing compaction-times.py..."
# Check for any compaction events to see if kernel supports it
- if ! perf list | grep -q "compaction:mm_compaction_begin"; then
+ if ! perf list tracepoint | grep -q "compaction:mm_compaction_begin"; then
echo "Skipping test, compaction tracepoints not found"
exit 2
fi
# Generate some events
- # We might not naturally trigger compaction in 0.5s sleep, but the script
+ # We might not naturally trigger compaction in 0.1s sleep, but the script
# should parse the empty or sparse file correctly without crashing.
- if ! perf record -e "compaction:*" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event \
+ -e "compaction:mm_compaction_begin,compaction:mm_compaction_end" \
+ -a -o "${temp_data}" -- sleep 0.1 >/dev/null 2>&1; then
echo "Skipping test, perf record failed"
exit 2
fi
diff --git a/tools/perf/tests/shell/test_event_analyzing_sample_python.sh b/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
index a7a8ded003db..5f0a41080acf 100755
--- a/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
+++ b/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
@@ -39,7 +39,8 @@ test_file_mode() {
echo "Testing event_analyzing_sample.py..."
# Generate some events
- if ! perf record -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping test, perf record failed"
exit 2
fi
diff --git a/tools/perf/tests/shell/test_flamegraph_python.sh b/tools/perf/tests/shell/test_flamegraph_python.sh
index 86313c72f53e..787d6bbdf04a 100755
--- a/tools/perf/tests/shell/test_flamegraph_python.sh
+++ b/tools/perf/tests/shell/test_flamegraph_python.sh
@@ -60,7 +60,8 @@ test_file_mode() {
echo "Testing flamegraph.py..."
# Generate some events with callchains
- if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping test, perf record -g failed (permissions or lack of support)"
exit 2
fi
@@ -75,7 +76,7 @@ test_file_mode() {
# Run the script in pipe ('-') mode and validate JSON output
rm -f "${temp_json}"
- if ! perf record -g -o - -- perf test -w noploop 2>/dev/null | \
+ if ! perf record -B -N --no-bpf-event -g -o - -- perf test -w noploop 2>/dev/null | \
perf script flamegraph -i - -f json -o "${temp_json}" >/dev/null; then
echo "Pipe stdin JSON mode test failed."
err=1
diff --git a/tools/perf/tests/shell/test_futex_contention_python.sh b/tools/perf/tests/shell/test_futex_contention_python.sh
index a8846ed68ac2..41ce8470a75f 100755
--- a/tools/perf/tests/shell/test_futex_contention_python.sh
+++ b/tools/perf/tests/shell/test_futex_contention_python.sh
@@ -89,14 +89,15 @@ EOF
test_file_mode() {
echo "Testing futex-contention.py..."
# Some systems might not have syscalls:sys_enter_futex
- if ! perf list | grep -q syscalls:sys_enter_futex; then
+ if ! perf list tracepoint | grep -q syscalls:sys_enter_futex; then
echo "Skipping file mode test, syscalls:sys_enter_futex not found"
return
fi
# Generate some futex events
- if ! perf record -e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \
- -- sleep 0.5 2>/dev/null; then
+ if ! perf record -B -N --no-bpf-event \
+ -e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \
+ -- sleep 0.1 2>/dev/null; then
echo "Skipping file mode test (record failed)"
return
fi
diff --git a/tools/perf/tests/shell/test_gecko_python.sh b/tools/perf/tests/shell/test_gecko_python.sh
index de1d7fcf88b7..18e9f5cd2ab0 100755
--- a/tools/perf/tests/shell/test_gecko_python.sh
+++ b/tools/perf/tests/shell/test_gecko_python.sh
@@ -39,7 +39,8 @@ test_file_mode() {
echo "Testing gecko.py..."
# Generate some events with callchains
- if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping test, perf record -g failed (permissions or lack of support)"
exit 2
fi
diff --git a/tools/perf/tests/shell/test_mem_phys_addr_python.sh b/tools/perf/tests/shell/test_mem_phys_addr_python.sh
index 451692d5e089..cbb9f572be9d 100755
--- a/tools/perf/tests/shell/test_mem_phys_addr_python.sh
+++ b/tools/perf/tests/shell/test_mem_phys_addr_python.sh
@@ -78,10 +78,12 @@ test_file_mode() {
echo "Testing mem-phys-addr.py file mode..."
# Generate memory access events (try unprivileged user-space first, then system-wide)
- if ! perf record --phys-data -d -o "${temp_data}" \
+ if ! perf record -B -N --no-bpf-event --phys-data -d -o "${temp_data}" \
-- perf test -w datasym >/dev/null 2>&1 && \
- ! perf record -d -o "${temp_data}" -- perf test -w datasym >/dev/null 2>&1 && \
- ! perf record -d -a -o "${temp_data}" -- sleep 0.2 >/dev/null 2>&1; then
+ ! perf record -B -N --no-bpf-event -d -o "${temp_data}" \
+ -- perf test -w datasym >/dev/null 2>&1 && \
+ ! perf record -B -N --no-bpf-event -d -a -o "${temp_data}" \
+ -- sleep 0.1 >/dev/null 2>&1; then
echo "Skipping file mode record test, perf record -d not supported"
return 0
fi
diff --git a/tools/perf/tests/shell/test_net_dropmonitor_python.sh b/tools/perf/tests/shell/test_net_dropmonitor_python.sh
index 05056a897bc9..2e37994418fc 100755
--- a/tools/perf/tests/shell/test_net_dropmonitor_python.sh
+++ b/tools/perf/tests/shell/test_net_dropmonitor_python.sh
@@ -42,11 +42,12 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
echo "Testing net_dropmonitor.py..."
# Create a perf.data file. Force dropping a packet if tracepoint is available!
-if ! perf record -e skb:kfree_skb -o "${temp_data}" -a \
- -- ping -c 1 255.255.255.255 >/dev/null 2>&1; then
- if ! perf record -e skb:kfree_skb -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" -a \
+ -- ping -c 1 -W 1 255.255.255.255 >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" \
-- sleep 0.1 >/dev/null 2>&1; then
- if ! perf record -o "${temp_data}" -- uname >/dev/null 2>&1; then
+ if ! perf record -B -N --no-bpf-event -o "${temp_data}" \
+ -- uname >/dev/null 2>&1; then
echo "Skipping test, cannot record perf events"
exit 2
fi
diff --git a/tools/perf/tests/shell/test_netdev_times_python.sh b/tools/perf/tests/shell/test_netdev_times_python.sh
index 393bf446efb2..0d8d10a42cab 100755
--- a/tools/perf/tests/shell/test_netdev_times_python.sh
+++ b/tools/perf/tests/shell/test_netdev_times_python.sh
@@ -85,9 +85,9 @@ then
fi
# Create a perf.data file. Force dropping a packet if tracepoint is available!
-if ! perf record -e skb:kfree_skb -a -o "${temp_data}" \
- -- ping -c 1 127.0.0.1 >/dev/null 2>&1; then
- perf record -e cycles -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -a -o "${temp_data}" \
+ -- ping -c 1 -W 1 127.0.0.1 >/dev/null 2>&1; then
+ perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
-- perf test -w noploop >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
fi
diff --git a/tools/perf/tests/shell/test_powerpc_hcalls_python.sh b/tools/perf/tests/shell/test_powerpc_hcalls_python.sh
index 19569c6d3613..b140ccf00ffa 100755
--- a/tools/perf/tests/shell/test_powerpc_hcalls_python.sh
+++ b/tools/perf/tests/shell/test_powerpc_hcalls_python.sh
@@ -67,8 +67,8 @@ if ! grep -q "H_REMOVE.*1.*1500.*1500.*1500" "${temp_out}"; then
fi
# Create a perf.data file if powerpc hcall tracepoints are available on this host.
-if ! perf record -e powerpc:hcall_entry,powerpc:hcall_exit -a -o "${temp_data}" \
- -- perf test -w noploop >/dev/null 2>&1; then
+if ! perf record -B -N --no-bpf-event -e powerpc:hcall_entry,powerpc:hcall_exit \
+ -a -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
echo "Skipping live record test, powerpc hcall tracepoints not available"
exit 0
fi
diff --git a/tools/perf/tests/shell/test_rw_by_file_python.sh b/tools/perf/tests/shell/test_rw_by_file_python.sh
index f121597f418f..e3c538c3ace3 100755
--- a/tools/perf/tests/shell/test_rw_by_file_python.sh
+++ b/tools/perf/tests/shell/test_rw_by_file_python.sh
@@ -37,8 +37,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
echo "Testing rw-by-file.py..."
# Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
- perf record -e syscalls:sys_enter_read,syscalls:sys_enter_write -a -o "${temp_data}" \
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
+ perf record -B -N --no-bpf-event -e syscalls:sys_enter_read,syscalls:sys_enter_write \
+ -o "${temp_data}" \
-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
else
diff --git a/tools/perf/tests/shell/test_rw_by_pid_python.sh b/tools/perf/tests/shell/test_rw_by_pid_python.sh
index 41eaa29097a2..cd93c1d9714c 100755
--- a/tools/perf/tests/shell/test_rw_by_pid_python.sh
+++ b/tools/perf/tests/shell/test_rw_by_pid_python.sh
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
echo "Testing rw-by-pid.py..."
# Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
ev="syscalls:sys_enter_read,syscalls:sys_exit_read"
ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write"
- perf record -e "$ev" -a -o "${temp_data}" \
+ perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \
-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
else
diff --git a/tools/perf/tests/shell/test_rwtop_python.sh b/tools/perf/tests/shell/test_rwtop_python.sh
index 05897e384702..d33807923c92 100755
--- a/tools/perf/tests/shell/test_rwtop_python.sh
+++ b/tools/perf/tests/shell/test_rwtop_python.sh
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
echo "Testing rwtop.py..."
# Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
ev="syscalls:sys_enter_read,syscalls:sys_exit_read"
ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write"
- perf record -e "$ev" -a -o "${temp_data}" \
+ perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \
-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
else
diff --git a/tools/perf/tests/shell/test_sched_migration_python.sh b/tools/perf/tests/shell/test_sched_migration_python.sh
index 75465bcd3791..714275b0e492 100755
--- a/tools/perf/tests/shell/test_sched_migration_python.sh
+++ b/tools/perf/tests/shell/test_sched_migration_python.sh
@@ -44,10 +44,10 @@ echo "Testing sched-migration.py..."
ev="sched:sched_switch,sched:sched_migrate_task"
ev="${ev},sched:sched_wakeup_new,sched:sched_wakeup"
has_sched=1
-if ! perf record -e "$ev" -a -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \
-- sleep 0.1 >/dev/null 2>&1; then
has_sched=0
- perf record -e cycles -o "${temp_data}" \
+ perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
-- perf test -w noploop >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
fi
diff --git a/tools/perf/tests/shell/test_stackcollapse_python.sh b/tools/perf/tests/shell/test_stackcollapse_python.sh
index ce0f80407eec..2048734c5e0d 100755
--- a/tools/perf/tests/shell/test_stackcollapse_python.sh
+++ b/tools/perf/tests/shell/test_stackcollapse_python.sh
@@ -37,7 +37,7 @@ echo "Testing stackcollapse.py..."
# Create a perf.data file with callchains. Use a busy workload rather than
# sleep, as an idle system may not generate any samples at all.
-perf record -g -o "${temp_data}" \
+perf record -B -N --no-bpf-event -g -o "${temp_data}" \
-- perf test -w noploop >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
diff --git a/tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh b/tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh
index 45bbdc554c8f..300289fe33b4 100755
--- a/tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh
+++ b/tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh
@@ -43,14 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
test_file_mode() {
echo "Testing syscall-counts-by-pid.py..."
# Some systems might not have raw_syscalls:sys_enter
- if ! perf list | grep -q raw_syscalls:sys_enter; then
+ if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then
echo "Skipping test, raw_syscalls:sys_enter not found"
exit 2
fi
# Generate some syscall events
- perf record -e raw_syscalls:sys_enter -a -o "${temp_data}" \
- -- sleep 0.5 >/dev/null 2>&1 || \
+ perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \
+ -- sh -c "sleep 0.1; sleep 0.05" >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
if ! perf script syscall-counts-by-pid -i "${temp_data}" > "${temp_out}"; then
diff --git a/tools/perf/tests/shell/test_syscall_counts_python.sh b/tools/perf/tests/shell/test_syscall_counts_python.sh
index 310e05d399f9..86463f14391b 100755
--- a/tools/perf/tests/shell/test_syscall_counts_python.sh
+++ b/tools/perf/tests/shell/test_syscall_counts_python.sh
@@ -43,13 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
test_file_mode() {
echo "Testing syscall-counts.py..."
# Some systems might not have raw_syscalls:sys_enter (e.g. stripped kernels or permissions)
- if ! perf list | grep -q raw_syscalls:sys_enter; then
+ if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then
echo "Skipping test, raw_syscalls:sys_enter not found"
exit 2
fi
# Generate some syscall events
- if ! perf record -e raw_syscalls:sys_enter -o "${temp_data}" -- sleep 0.5 2>/dev/null; then
+ if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \
+ -- sh -c "sleep 0.1; sleep 0.05" 2>/dev/null; then
echo "perf record failed (permissions?), skipping file mode test."
exit 2
fi
diff --git a/tools/perf/tests/shell/test_task_analyzer.sh b/tools/perf/tests/shell/test_task_analyzer.sh
index f99a637a6099..f53fa4a27c79 100755
--- a/tools/perf/tests/shell/test_task_analyzer.sh
+++ b/tools/perf/tests/shell/test_task_analyzer.sh
@@ -60,7 +60,8 @@ skip_no_probe_record_support() {
prepare_perf_data() {
# 1s should be sufficient to catch at least some switches
- perf record -e sched:sched_switch -a -o "${perfdata}" -- sleep 1 > /dev/null 2>&1
+ perf record -B -N --no-bpf-event -e sched:sched_switch -a -o "${perfdata}" \
+ -- sleep 1 > /dev/null 2>&1
# check if perf data file got created in above step.
if [ ! -e "${perfdata}" ]; then
printf "FAIL: perf record failed to create \"${perfdata}\" \n"
diff --git a/tools/perf/tests/shell/test_wakeup_latency_python.sh b/tools/perf/tests/shell/test_wakeup_latency_python.sh
index cb2571449168..b43a86996be2 100755
--- a/tools/perf/tests/shell/test_wakeup_latency_python.sh
+++ b/tools/perf/tests/shell/test_wakeup_latency_python.sh
@@ -41,9 +41,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
echo "Testing wakeup-latency.py..."
# Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "sched:sched_wakeup"; then
+if perf list tracepoint | grep -q "sched:sched_wakeup"; then
ev="sched:sched_wakeup,sched:sched_wakeup_new,sched:sched_switch"
- perf record -e "$ev" -a -o "${temp_data}" \
+ perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \
-- sleep 0.1 >/dev/null 2>&1 || \
{ echo "Skipping test, perf record failed"; exit 2; }
else
@@ -70,7 +70,8 @@ else
fi
# Also test zero-wakeups / unhandled events path to verify division-by-zero protection
-if perf record -e cycles -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+if perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
+ -- perf test -w noploop >/dev/null 2>&1; then
if ! perf script wakeup-latency -i "${temp_data}" > "${temp_out}" || \
! grep -q "avg_wakeup_latency (ns): N/A" "${temp_out}"; then
echo "wakeup-latency zero-wakeups guard test failed"
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v3 6/9] perf test: Reduce overhead and contention in Python shell tests
2026-09-29 6:58 ` [PATCH v3 6/9] perf test: Reduce overhead and contention in Python shell tests Ian Rogers
@ 2026-09-29 7:04 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 7:04 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> When all Python shell tests run concurrently during 'perf test' under
> heavy load:
> 1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in
> each test process; use 'perf list tracepoint' when checking for
> tracepoint events.
> 2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF
> events across the system and caches build-ids into ~/.debug; add
> '-B -N --no-bpf-event' to all Python shell test recordings.
> 3. Drop '-a' in test_check_perf_trace_python.sh,
> test_rw_by_file_python.sh, test_rw_by_pid_python.sh,
> test_rwtop_python.sh, and test_syscall_counts_by_pid_python.sh where
> only a single child workload ('dd' or 'sleep') is tested, avoiding
> system-wide /proc synthesis and ringbuffer contention.
> 4. Add '-W 1' to 'ping -c 1' in test_net_dropmonitor_python.sh and
> test_netdev_times_python.sh, and narrow 'compaction:*' to
> 'compaction:mm_compaction_begin,compaction:mm_compaction_end' in
> test_compaction_times_python.sh.
>
> Assisted-by: Antigravity:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929065839.407062-1-irogers@google.com?part=6
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH v3 7/9] perf python event_analyzing_sample: Default to in-memory SQLite database
2026-09-29 6:58 ` [PATCH v3 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (5 preceding siblings ...)
2026-09-29 6:58 ` [PATCH v3 6/9] perf test: Reduce overhead and contention in Python shell tests Ian Rogers
@ 2026-09-29 6:58 ` Ian Rogers
2026-09-29 7:05 ` sashiko-bot
2026-09-29 6:58 ` [PATCH v3 8/9] perf python: Initialize debug output on module load Ian Rogers
` (2 subsequent siblings)
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
When -d/--db is not specified, event_analyzing_sample.py created a
temporary file in /tmp via tempfile.mkstemp() and deleted it in
trace_end(). As noted during review, creating the database in a shared
/tmp directory does not reserve SQLite's auxiliary sidecar filenames
(-journal or -wal), and a temporary on-disk file is unnecessary when the
caller did not ask to persist the database.
Default to sqlite3.connect(":memory:") when db_path is not provided,
removing the temporary file creation and cleanup logic, and test both
the default in-memory mode and explicit -d file mode in
test_event_analyzing_sample_python.sh.
Fixes: eeb70645a809 ("perf python: Port event_analyzing_sample to perf module")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/python/event_analyzing_sample.py | 35 ++++++-------------
.../test_event_analyzing_sample_python.sh | 8 ++++-
2 files changed, 17 insertions(+), 26 deletions(-)
diff --git a/tools/perf/python/event_analyzing_sample.py b/tools/perf/python/event_analyzing_sample.py
index 15b47cff9fa3..b4efee88d94b 100755
--- a/tools/perf/python/event_analyzing_sample.py
+++ b/tools/perf/python/event_analyzing_sample.py
@@ -17,10 +17,8 @@ from __future__ import annotations
import argparse
import math
-import os
import sqlite3
import struct
-import tempfile
from typing import Any
import perf
@@ -117,16 +115,11 @@ session: Any = None
class _DB:
con: sqlite3.Connection | None = None
- temp_path: str | None = None
def trace_begin(db_path: str | None = None) -> None:
"""Initialize database tables."""
print("In trace_begin:\n")
- if not db_path:
- fd, db_path = tempfile.mkstemp(prefix="perf_events_", suffix=".db")
- os.close(fd)
- _DB.temp_path = db_path
- con = sqlite3.connect(db_path)
+ con = sqlite3.connect(db_path or ":memory:")
try:
# Drop any pre-existing tables so repeated runs do not accumulate duplicate events.
con.execute("drop table if exists gen_events;")
@@ -297,28 +290,20 @@ def show_pebs_ll() -> None:
def trace_end() -> None:
"""Called at the end of trace processing."""
print("In trace_end:\n")
- try:
- if _DB.con:
- try:
- _DB.con.commit()
- show_general_events()
- show_pebs_ll()
- finally:
- _DB.con.close()
- _DB.con = None
- finally:
- if _DB.temp_path and os.path.exists(_DB.temp_path):
- try:
- os.remove(_DB.temp_path)
- except OSError:
- pass
- _DB.temp_path = None
+ if _DB.con:
+ try:
+ _DB.con.commit()
+ show_general_events()
+ show_pebs_ll()
+ finally:
+ _DB.con.close()
+ _DB.con = None
if __name__ == "__main__":
ap = argparse.ArgumentParser(description="Analyze events with SQLite")
ap.add_argument("-i", "--input", default="perf.data", help="Input file name")
ap.add_argument("-d", "--db", "--database", dest="database", default=None,
- help="Database file name (defaults to a temporary file cleaned up on exit)")
+ help="Database file name (defaults to an in-memory database)")
args = ap.parse_args()
try:
diff --git a/tools/perf/tests/shell/test_event_analyzing_sample_python.sh b/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
index 5f0a41080acf..cddb12b67698 100755
--- a/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
+++ b/tools/perf/tests/shell/test_event_analyzing_sample_python.sh
@@ -45,8 +45,14 @@ test_file_mode() {
exit 2
fi
- # Run the script
+ # Run the script with default (:memory:) database and with explicit -d path
if ! perf script event_analyzing_sample -i "${temp_data}" \
+ > "${temp_dir}/perf.mem.out" 2>&1 || \
+ ! grep -q "Statistics about the general events" "${temp_dir}/perf.mem.out" || \
+ grep -q "Error creating/inserting event" "${temp_dir}/perf.mem.out"; then
+ echo "Default in-memory database mode test failed."
+ err=1
+ elif ! perf script event_analyzing_sample -i "${temp_data}" \
-d "${temp_db}" > "${temp_dir}/perf.out" 2>&1; then
echo "File mode test failed."
err=1
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* [PATCH v3 8/9] perf python: Initialize debug output on module load
2026-09-29 6:58 ` [PATCH v3 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (6 preceding siblings ...)
2026-09-29 6:58 ` [PATCH v3 7/9] perf python event_analyzing_sample: Default to in-memory SQLite database Ian Rogers
@ 2026-09-29 6:58 ` Ian Rogers
2026-09-29 7:04 ` sashiko-bot
2026-09-29 6:58 ` [PATCH v3 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list Ian Rogers
2026-09-29 18:44 ` [PATCH v3 0/9] perf python: Follow-up fixes and shell test deflaking Arnaldo Carvalho de Melo
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
Unlike the perf binary's main(), PyInit_perf() does not call
perf_debug_setup(). As a result, the first warning or error printed from
the perf Python C extension hits debug_file() with _debug_file == NULL
and emits 'debug_file not set' without a trailing newline before the
actual diagnostic message.
Call perf_debug_setup() in PyInit_perf() and add the missing trailing
newline to the fallback warning in debug_file().
Fixes: ec49230cf6dd ("perf debug: Expose debug file")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/util/debug.c | 2 +-
tools/perf/util/python.c | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/tools/perf/util/debug.c b/tools/perf/util/debug.c
index 6b5ffe81f141..b7095519419e 100644
--- a/tools/perf/util/debug.c
+++ b/tools/perf/util/debug.c
@@ -55,7 +55,7 @@ FILE *debug_file(void)
{
if (!_debug_file) {
debug_set_file(stderr);
- pr_warning_once("debug_file not set");
+ pr_warning_once("%s not set\n", __func__);
}
return _debug_file;
}
diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c
index 690a94fb4f62..95140dfaef9c 100644
--- a/tools/perf/util/python.c
+++ b/tools/perf/util/python.c
@@ -5157,6 +5157,7 @@ PyMODINIT_FUNC PyInit_perf(void)
/* The page_size is placed in util object. */
page_size = sysconf(_SC_PAGE_SIZE);
+ perf_debug_setup();
Py_INCREF(&pyrf_evlist__type);
PyModule_AddObject(module, "evlist", (PyObject *)&pyrf_evlist__type);
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* [PATCH v3 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list
2026-09-29 6:58 ` [PATCH v3 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (7 preceding siblings ...)
2026-09-29 6:58 ` [PATCH v3 8/9] perf python: Initialize debug output on module load Ian Rogers
@ 2026-09-29 6:58 ` Ian Rogers
2026-09-29 7:06 ` sashiko-bot
2026-09-29 18:44 ` [PATCH v3 0/9] perf python: Follow-up fixes and shell test deflaking Arnaldo Carvalho de Melo
9 siblings, 1 reply; 58+ messages in thread
From: Ian Rogers @ 2026-09-29 6:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz
perf_pmus__print_pmu_events() first counts events via
perf_pmu__num_events() to allocate the aliases array, then populates it
via perf_pmu__for_each_event(). When dynamic tracepoints (such as
kprobes or uprobes during 'perf test' runs) are concurrently created or
removed between the two passes:
- If an event is removed, state.index is smaller than the allocated len,
leaving trailing zeroed entries in aliases[] with name == NULL and
pmu == NULL, which causes qsort(cmp_sevent) or the print loop to crash
with SIGSEGV.
- If a dynamic tracepoint subsystem directory is removed while scanning
/sys/kernel/tracing/events, tp_pmu__for_each_tp_event() returns
-ENOENT and aborts enumeration of all remaining tracepoint subsystems.
- If an event is added, perf_pmus__print_pmu_events__callback() aborts
enumeration when state->index reaches state->aliases_len.
Grow state->aliases dynamically via realloc() when needed, use
state.index as the actual populated length for sorting and printing, and
ignore -ENOENT when a tracepoint subsystem directory disappears during
enumeration.
Fixes: c3245d2093c1 ("perf pmu: Abstract alias/event struct")
Fixes: 45b6e281cb06 ("perf tp_pmu: Add event APIs")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/util/pmus.c | 21 +++++++++++++++++----
tools/perf/util/tp_pmu.c | 8 ++++++--
2 files changed, 23 insertions(+), 6 deletions(-)
diff --git a/tools/perf/util/pmus.c b/tools/perf/util/pmus.c
index e0a4cb2428ca..1355b317f3ed 100644
--- a/tools/perf/util/pmus.c
+++ b/tools/perf/util/pmus.c
@@ -8,6 +8,7 @@
#include <sys/types.h>
#include <ctype.h>
#include <pthread.h>
+#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include "cpumap.h"
@@ -571,7 +572,7 @@ static int cmp_sevent(const void *a, const void *b)
}
/* Order by event name. */
- return strcmp(as->name, bs->name);
+ return strcmp(as->name ?: "", bs->name ?: "");
}
static bool pmu_alias_is_duplicate(struct sevent *a, struct sevent *b)
@@ -581,7 +582,7 @@ static bool pmu_alias_is_duplicate(struct sevent *a, struct sevent *b)
return false;
/* Don't remove duplicates for different PMUs */
- return strcmp(a->pmu_name, b->pmu_name) == 0;
+ return strcmp(a->pmu_name ?: "", b->pmu_name ?: "") == 0;
}
struct events_callback_state {
@@ -597,8 +598,18 @@ static int perf_pmus__print_pmu_events__callback(void *vstate,
struct sevent *s;
if (state->index >= state->aliases_len) {
- pr_err("Unexpected event %s/%s/\n", info->pmu->name, info->name);
- return 1;
+ size_t new_len = max_t(size_t, 16, state->aliases_len * 2);
+ struct sevent *new_aliases;
+
+ new_aliases = realloc(state->aliases, new_len * sizeof(struct sevent));
+ if (!new_aliases) {
+ pr_err("Unexpected event %s/%s/\n", info->pmu->name, info->name);
+ return 1;
+ }
+ memset(&new_aliases[state->aliases_len], 0,
+ (new_len - state->aliases_len) * sizeof(struct sevent));
+ state->aliases = new_aliases;
+ state->aliases_len = new_len;
}
assert(info->pmu != NULL || info->name != NULL);
s = &state->aliases[state->index];
@@ -654,6 +665,8 @@ void perf_pmus__print_pmu_events(const struct print_callbacks *print_cb, void *p
perf_pmu__for_each_event(pmu, skip_duplicate_pmus, &state,
perf_pmus__print_pmu_events__callback);
}
+ aliases = state.aliases;
+ len = state.index;
qsort(aliases, len, sizeof(struct sevent), cmp_sevent);
for (int j = 0; j < len; j++) {
/* Skip duplicates */
diff --git a/tools/perf/util/tp_pmu.c b/tools/perf/util/tp_pmu.c
index c2be8c9f9084..5ac732e06841 100644
--- a/tools/perf/util/tp_pmu.c
+++ b/tools/perf/util/tp_pmu.c
@@ -151,7 +151,9 @@ static int for_each_event_cb(void *state, const char *sys_name, const char *evt_
static int for_each_event_sys_cb(void *state, const char *sys_name)
{
- return tp_pmu__for_each_tp_event(sys_name, state, for_each_event_cb);
+ int ret = tp_pmu__for_each_tp_event(sys_name, state, for_each_event_cb);
+
+ return ret == -ENOENT ? 0 : ret;
}
int tp_pmu__for_each_event(struct perf_pmu *pmu, void *state, pmu_event_callback cb)
@@ -176,7 +178,9 @@ static int num_events_cb(void *state, const char *sys_name __maybe_unused,
static int num_events_sys_cb(void *state, const char *sys_name)
{
- return tp_pmu__for_each_tp_event(sys_name, state, num_events_cb);
+ int ret = tp_pmu__for_each_tp_event(sys_name, state, num_events_cb);
+
+ return ret == -ENOENT ? 0 : ret;
}
size_t tp_pmu__num_events(struct perf_pmu *pmu __maybe_unused)
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH v3 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list
2026-09-29 6:58 ` [PATCH v3 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list Ian Rogers
@ 2026-09-29 7:06 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-09-29 7:06 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> perf_pmus__print_pmu_events() first counts events via
> perf_pmu__num_events() to allocate the aliases array, then populates it
> via perf_pmu__for_each_event(). When dynamic tracepoints (such as
> kprobes or uprobes during 'perf test' runs) are concurrently created or
> removed between the two passes:
> - If an event is removed, state.index is smaller than the allocated len,
> leaving trailing zeroed entries in aliases[] with name == NULL and
> pmu == NULL, which causes qsort(cmp_sevent) or the print loop to crash
> with SIGSEGV.
> - If a dynamic tracepoint subsystem directory is removed while scanning
> /sys/kernel/tracing/events, tp_pmu__for_each_tp_event() returns
> -ENOENT and aborts enumeration of all remaining tracepoint subsystems.
> - If an event is added, perf_pmus__print_pmu_events__callback() aborts
> enumeration when state->index reaches state->aliases_len.
>
> Grow state->aliases dynamically via realloc() when needed, use
> state.index as the actual populated length for sorting and printing, and
> ignore -ENOENT when a tracepoint subsystem directory disappears during
> enumeration.
>
> Fixes: c3245d2093c1 ("perf pmu: Abstract alias/event struct")
> Fixes: 45b6e281cb06 ("perf tp_pmu: Add event APIs")
> Assisted-by: Antigravity:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260929065839.407062-1-irogers@google.com?part=9
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH v3 0/9] perf python: Follow-up fixes and shell test deflaking
2026-09-29 6:58 ` [PATCH v3 0/9] perf python: Follow-up fixes and shell test deflaking Ian Rogers
` (8 preceding siblings ...)
2026-09-29 6:58 ` [PATCH v3 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list Ian Rogers
@ 2026-09-29 18:44 ` Arnaldo Carvalho de Melo
9 siblings, 0 replies; 58+ messages in thread
From: Arnaldo Carvalho de Melo @ 2026-09-29 18:44 UTC (permalink / raw)
To: Ian Rogers
Cc: namhyung, adrian.hunter, james.clark, jolsa, linux-kernel,
linux-perf-users, mingo, peterz
On Mon, Sep 28, 2026 at 11:58:30PM -0700, Ian Rogers wrote:
> This series contains follow-up fixes for the standalone Python scripts
> and their shell tests, particularly when running 'perf test' as root
> under heavy parallel load:
Thanks, applied to perf-tools-next, for v7.4.
- Arnaldo
^ permalink raw reply [flat|nested] 58+ messages in thread