[PATCH v2 0/7] Perf stat --null/offline CPU segv related fixes/tests

[PATCH v2 0/7] Perf stat --null/offline CPU segv related fixes/tests

[Ian Rogers]

Ingo reported [1] that `perf stat --null` was segfaulting. Fix the
underlying issue and add a test to the "perf stat tests". Do some
related fixing/cleanup in the perf util cpumap code.

Thomas reported an issue fixed by the same patches [2] but caused by
giving perf stat an offline CPU. Add test coverage for that and
improve the "error" message that reports "success".

Ingo further pointed at broken signal handling in repeat mode [3]. I
observed we weren't giving the best exit code, 0 rather than the
expected 128+<signal number>. Add a patch fixing this.

[1] https://lore.kernel.org/linux-perf-users/aSwt7yzFjVJCEmVp@gmail.com/
[2] https://lore.kernel.org/linux-perf-users/94313b82-888b-4f42-9fb0-4585f9e90080@linux.ibm.com/
[3] https://lore.kernel.org/lkml/aS5wjmbAM9ka3M2g@gmail.com/

Ian Rogers (7):
  perf stat: Allow no events to open if this is a "--null" run
  libperf cpumap: Fix perf_cpu_map__max for an empty/NULL map
  perf cpumap: Add "any" CPU handling to cpu_map__snprint_mask
  perf tests stat: Add "--null" coverage
  perf stat: When no events, don't report an error if there is none
  perf tests stat: Add test for error for an offline CPU
  perf stat: Improve handling of termination by signal

 tools/lib/perf/cpumap.c        | 10 +++++----
 tools/perf/builtin-stat.c      | 29 ++++++++++++++++++-------
 tools/perf/tests/shell/stat.sh | 39 ++++++++++++++++++++++++++++++++++
 tools/perf/util/cpumap.c       |  9 ++++++--
 4 files changed, 73 insertions(+), 14 deletions(-)

-- 
2.52.0.177.g9f829587af-goog

[PATCH v2 1/7] perf stat: Allow no events to open if this is a "--null" run

[Ian Rogers]

It is intended that a "--null" run doesn't open any events.

Fixes: 2cc7aa995ce9 ("perf stat: Refactor retry/skip/fatal error handling")
Tested-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/builtin-stat.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c
index 5c06e9b61821..6410115ed9c5 100644
--- a/tools/perf/builtin-stat.c
+++ b/tools/perf/builtin-stat.c
@@ -923,7 +923,7 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
 			goto err_out;
 		}
 	}
-	if (!has_supported_counters) {
+	if (!has_supported_counters && !stat_config.null_run) {
 		evsel__open_strerror(evlist__first(evsel_list), &target, open_err,
 				     msg, sizeof(msg));
 		ui__error("No supported events found.\n%s\n", msg);
-- 
2.52.0.177.g9f829587af-goog

[PATCH v2 2/7] libperf cpumap: Fix perf_cpu_map__max for an empty/NULL map

[Ian Rogers]

Passing an empty map to perf_cpu_map__max triggered a SEGV. Explicitly
test for the empty map.

Reported-by: Ingo Molnar <mingo@kernel.org>
Closes: https://lore.kernel.org/linux-perf-users/aSwt7yzFjVJCEmVp@gmail.com/
Tested-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/lib/perf/cpumap.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/tools/lib/perf/cpumap.c b/tools/lib/perf/cpumap.c
index 7e88417ba84d..4160e7d2e120 100644
--- a/tools/lib/perf/cpumap.c
+++ b/tools/lib/perf/cpumap.c
@@ -368,10 +368,12 @@ struct perf_cpu perf_cpu_map__max(const struct perf_cpu_map *map)
 		.cpu = -1
 	};
 
-	// cpu_map__trim_new() qsort()s it, cpu_map__default_new() sorts it as well.
-	return __perf_cpu_map__nr(map) > 0
-		? __perf_cpu_map__cpu(map, __perf_cpu_map__nr(map) - 1)
-		: result;
+	if (!map)
+		return result;
+
+	// The CPUs are always sorted and nr is always > 0 as 0 length map is
+	// encoded as NULL.
+	return __perf_cpu_map__cpu(map, __perf_cpu_map__nr(map) - 1);
 }
 
 /** Is 'b' a subset of 'a'. */
-- 
2.52.0.177.g9f829587af-goog

[PATCH v2 3/7] perf cpumap: Add "any" CPU handling to cpu_map__snprint_mask

[Ian Rogers]

If the perf_cpu_map is empty or is just the any CPU value, then early
return. Don't process the "any" CPU when creating the bitmap.

Tested-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/util/cpumap.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/cpumap.c b/tools/perf/util/cpumap.c
index 89570397a4b3..a80845038a5e 100644
--- a/tools/perf/util/cpumap.c
+++ b/tools/perf/util/cpumap.c
@@ -684,16 +684,21 @@ size_t cpu_map__snprint_mask(struct perf_cpu_map *map, char *buf, size_t size)
 	unsigned char *bitmap;
 	struct perf_cpu c, last_cpu = perf_cpu_map__max(map);
 
-	if (buf == NULL)
+	if (buf == NULL || size == 0)
 		return 0;
 
+	if (last_cpu.cpu < 0) {
+		buf[0] = '\0';
+		return 0;
+	}
+
 	bitmap = zalloc(last_cpu.cpu / 8 + 1);
 	if (bitmap == NULL) {
 		buf[0] = '\0';
 		return 0;
 	}
 
-	perf_cpu_map__for_each_cpu(c, idx, map)
+	perf_cpu_map__for_each_cpu_skip_any(c, idx, map)
 		bitmap[c.cpu / 8] |= 1 << (c.cpu % 8);
 
 	for (int cpu = last_cpu.cpu / 4 * 4; cpu >= 0; cpu -= 4) {
-- 
2.52.0.177.g9f829587af-goog

[PATCH v2 4/7] perf tests stat: Add "--null" coverage

[Ian Rogers]

Ensure "--null" does a minimal run.

Reported-by: Ingo Molnar <mingo@kernel.org>
Closes: https://lore.kernel.org/linux-perf-users/aSwt7yzFjVJCEmVp@gmail.com/
Tested-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/tests/shell/stat.sh | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/tools/perf/tests/shell/stat.sh b/tools/perf/tests/shell/stat.sh
index 985adc02749e..7885e875caf4 100755
--- a/tools/perf/tests/shell/stat.sh
+++ b/tools/perf/tests/shell/stat.sh
@@ -16,6 +16,17 @@ test_default_stat() {
   echo "Basic stat command test [Success]"
 }
 
+test_null_stat() {
+  echo "Null stat command test"
+  if ! perf stat --null true 2>&1 | grep -E -q "Performance counter stats for 'true':"
+  then
+    echo "Null stat command test [Failed]"
+    err=1
+    return
+  fi
+  echo "Null stat command test [Success]"
+}
+
 test_stat_record_report() {
   echo "stat record and report test"
   if ! perf stat record -e task-clock -o - true | perf stat report -i - 2>&1 | \
@@ -212,6 +223,7 @@ test_hybrid() {
 }
 
 test_default_stat
+test_null_stat
 test_stat_record_report
 test_stat_record_script
 test_stat_repeat_weak_groups
-- 
2.52.0.177.g9f829587af-goog

[PATCH v2 5/7] perf stat: When no events, don't report an error if there is none

[Ian Rogers]

Events may fail to open as no supported CPUs were specified on the
command line. In this case a confusing "error" message of "success"
can be reported. Let's skip the error in that case.

Before:
```
$ perf stat -C2048 -e cycles -- true
WARNING: A requested CPU in '2048' is not supported by PMU 'cpu' (CPUs 0-7) for event 'cycles'
Error:
No supported events found.
The sys_perf_event_open() syscall returned with 0 (Success) for event (cpu/unknown-hardware/).
"dmesg | grep -i perf" may provide additional information.
```

After:
```
$ perf stat -C2048 -e cycles -- true
WARNING: A requested CPU in '2048' is not supported by PMU 'cpu' (CPUs 0-7) for event 'cycles'
Error:
No supported events found.
```

Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/builtin-stat.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c
index 6410115ed9c5..bd3c3de8d200 100644
--- a/tools/perf/builtin-stat.c
+++ b/tools/perf/builtin-stat.c
@@ -924,8 +924,10 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
 		}
 	}
 	if (!has_supported_counters && !stat_config.null_run) {
-		evsel__open_strerror(evlist__first(evsel_list), &target, open_err,
-				     msg, sizeof(msg));
+		if (open_err) {
+			evsel__open_strerror(evlist__first(evsel_list), &target, open_err,
+					     msg, sizeof(msg));
+		}
 		ui__error("No supported events found.\n%s\n", msg);
 
 		if (child_pid != -1)
-- 
2.52.0.177.g9f829587af-goog

[PATCH v2 6/7] perf tests stat: Add test for error for an offline CPU

[Ian Rogers]

Add a test that if an offline CPU is requested perf stat will fail.

```
$ perf test -vv "perf stat tests"
101: perf stat tests:
--- start ---
test child forked, pid 46965
Basic stat command test
Basic stat command test [Success]
Null stat command test
Null stat command test [Success]
Offline CPU stat command test (cpu 8)
Offline CPU stat command test [Success]
stat record and report test
stat record and report test [Success]
stat record and script test
stat record and script test [Success]
stat repeat weak groups test
stat repeat weak groups test [Success]
Topdown event group test
Topdown event group test [Success]
Topdown weak groups test
Topdown weak groups test [Skipped event parsing failed]
cputype test
cputype test [Success]
hybrid test
hybrid test [Success]
---- end(0) ----
101: perf stat tests                                                 : Ok
```

Reported-by: Thomas Richter <tmricht@linux.ibm.com>
Closes: https://lore.kernel.org/linux-perf-users/94313b82-888b-4f42-9fb0-4585f9e90080@linux.ibm.com/
Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/tests/shell/stat.sh | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/tools/perf/tests/shell/stat.sh b/tools/perf/tests/shell/stat.sh
index 7885e875caf4..0b2f0f88ca16 100755
--- a/tools/perf/tests/shell/stat.sh
+++ b/tools/perf/tests/shell/stat.sh
@@ -27,6 +27,32 @@ test_null_stat() {
   echo "Null stat command test [Success]"
 }
 
+find_offline_cpu() {
+  for i in $(seq 1 4096)
+  do
+    if [[ ! -f /sys/devices/system/cpu/cpu$i/online || \
+          $(cat /sys/devices/system/cpu/cpu$i/online) == "0" ]]
+    then
+      echo $i
+      return
+    fi
+  done
+  echo "Failed to find offline CPU"
+  exit 1
+}
+
+test_offline_cpu_stat() {
+  cpu=$(find_offline_cpu)
+  echo "Offline CPU stat command test (cpu $cpu)"
+  if ! perf stat "-C$cpu" -e cycles true 2>&1 | grep -E -q "No supported events found."
+  then
+    echo "Offline CPU stat command test [Failed]"
+    err=1
+    return
+  fi
+  echo "Offline CPU stat command test [Success]"
+}
+
 test_stat_record_report() {
   echo "stat record and report test"
   if ! perf stat record -e task-clock -o - true | perf stat report -i - 2>&1 | \
@@ -224,6 +250,7 @@ test_hybrid() {
 
 test_default_stat
 test_null_stat
+test_offline_cpu_stat
 test_stat_record_report
 test_stat_record_script
 test_stat_repeat_weak_groups
-- 
2.52.0.177.g9f829587af-goog

[PATCH v2 7/7] perf stat: Improve handling of termination by signal

[Ian Rogers]

When interrupting perf stat in repeat mode with a signal the signal is
passed to the child process but the repeat doesn't terminate:
```
$ perf stat -v --null --repeat 10 sleep 1
Control descriptor is not initialized
[ perf stat: executing run #1 ... ]
[ perf stat: executing run #2 ... ]
^Csleep: Interrupt
[ perf stat: executing run #3 ... ]
[ perf stat: executing run #4 ... ]
[ perf stat: executing run #5 ... ]
[ perf stat: executing run #6 ... ]
[ perf stat: executing run #7 ... ]
[ perf stat: executing run #8 ... ]
[ perf stat: executing run #9 ... ]
[ perf stat: executing run #10 ... ]

 Performance counter stats for 'sleep 1' (10 runs):

            0.9500 +- 0.0512 seconds time elapsed  ( +-  5.39% )

0.01user 0.02system 0:09.53elapsed 0%CPU (0avgtext+0avgdata 18940maxresident)k
29944inputs+0outputs (0major+2629minor)pagefaults 0swaps
```

Terminate the repeated run and give a reasonable exit value:
```
$ perf stat -v --null --repeat 10 sleep 1
Control descriptor is not initialized
[ perf stat: executing run #1 ... ]
[ perf stat: executing run #2 ... ]
[ perf stat: executing run #3 ... ]
^Csleep: Interrupt

 Performance counter stats for 'sleep 1' (10 runs):

             0.680 +- 0.321 seconds time elapsed  ( +- 47.16% )

Command exited with non-zero status 130
0.00user 0.01system 0:02.05elapsed 0%CPU (0avgtext+0avgdata 70688maxresident)k
0inputs+0outputs (0major+5002minor)pagefaults 0swaps
```

Note, this also changes the exit value for non-repeat runs when
interrupted by a signal.

Reported-by: Ingo Molnar <mingo@kernel.org>
Closes: https://lore.kernel.org/lkml/aS5wjmbAM9ka3M2g@gmail.com/
Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/builtin-stat.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c
index bd3c3de8d200..ab40d85fb125 100644
--- a/tools/perf/builtin-stat.c
+++ b/tools/perf/builtin-stat.c
@@ -1007,10 +1007,20 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
 			goto err_out;
 		}
 
-		if (WIFSIGNALED(status))
+		if (WIFSIGNALED(status)) {
+			/*
+			 * We want to indicate failure to stop a repeat run,
+			 * hence negative. We want the value to be the exit code
+			 * of perf, which for termination by a signal is 128
+			 * plus the signal number.
+			 */
+			err = 0 - (128 + WTERMSIG(status));
 			psignal(WTERMSIG(status), argv[0]);
+		} else {
+			err = WEXITSTATUS(status);
+		}
 	} else {
-		status = dispatch_events(forks, timeout, interval, &times);
+		err = dispatch_events(forks, timeout, interval, &times);
 	}
 
 	disable_counters();
@@ -1050,7 +1060,7 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
 	if (!STAT_RECORD)
 		evlist__close(evsel_list);
 
-	return WEXITSTATUS(status);
+	return err;
 
 err_out:
 	if (forks)
@@ -2969,7 +2979,7 @@ int cmd_stat(int argc, const char **argv)
 			evlist__reset_prev_raw_counts(evsel_list);
 
 		status = run_perf_stat(argc, argv, run_idx);
-		if (status == -1)
+		if (status < 0)
 			break;
 
 		if (forever && !interval) {
@@ -3039,5 +3049,6 @@ int cmd_stat(int argc, const char **argv)
 
 	evlist__close_control(stat_config.ctl_fd, stat_config.ctl_fd_ack, &stat_config.ctl_fd_close);
 
-	return status;
+	/* Only the low byte of status becomes the exit code. */
+	return abs(status);
 }
-- 
2.52.0.177.g9f829587af-goog

Re: [PATCH v2 0/7] Perf stat --null/offline CPU segv related fixes/tests

[Thomas Richter]

On 12/3/25 22:46, Ian Rogers wrote:
> Ingo reported [1] that `perf stat --null` was segfaulting. Fix the
> underlying issue and add a test to the "perf stat tests". Do some
> related fixing/cleanup in the perf util cpumap code.
> 
> Thomas reported an issue fixed by the same patches [2] but caused by
> giving perf stat an offline CPU. Add test coverage for that and
> improve the "error" message that reports "success".
> 
> Ingo further pointed at broken signal handling in repeat mode [3]. I
> observed we weren't giving the best exit code, 0 rather than the
> expected 128+<signal number>. Add a patch fixing this.
> 
> [1] https://lore.kernel.org/linux-perf-users/aSwt7yzFjVJCEmVp@gmail.com/
> [2] https://lore.kernel.org/linux-perf-users/94313b82-888b-4f42-9fb0-4585f9e90080@linux.ibm.com/
> [3] https://lore.kernel.org/lkml/aS5wjmbAM9ka3M2g@gmail.com/
> 
> Ian Rogers (7):
>   perf stat: Allow no events to open if this is a "--null" run
>   libperf cpumap: Fix perf_cpu_map__max for an empty/NULL map
>   perf cpumap: Add "any" CPU handling to cpu_map__snprint_mask
>   perf tests stat: Add "--null" coverage
>   perf stat: When no events, don't report an error if there is none
>   perf tests stat: Add test for error for an offline CPU
>   perf stat: Improve handling of termination by signal
> 
>  tools/lib/perf/cpumap.c        | 10 +++++----
>  tools/perf/builtin-stat.c      | 29 ++++++++++++++++++-------
>  tools/perf/tests/shell/stat.sh | 39 ++++++++++++++++++++++++++++++++++
>  tools/perf/util/cpumap.c       |  9 ++++++--
>  4 files changed, 73 insertions(+), 14 deletions(-)
> 

For the whole series:

Tested-by: Thomas Richter <tmricht@linux.ibm.com>

Thanks for fixing this, Ian

-- 
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
IBM Deutschland Research & Development GmbH

Vorsitzender des Aufsichtsrats: Wolfgang Wendt

Geschäftsführung: David Faller

Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294

Re: [PATCH v2 0/7] Perf stat --null/offline CPU segv related fixes/tests

[Namhyung Kim]

On Wed, 03 Dec 2025 13:46:59 -0800, Ian Rogers wrote:
> Ingo reported [1] that `perf stat --null` was segfaulting. Fix the
> underlying issue and add a test to the "perf stat tests". Do some
> related fixing/cleanup in the perf util cpumap code.
> 
> Thomas reported an issue fixed by the same patches [2] but caused by
> giving perf stat an offline CPU. Add test coverage for that and
> improve the "error" message that reports "success".
> 
> [...]
Applied to perf-tools-next, thanks!

Best regards,
Namhyung

Re: [PATCH v2 0/7] Perf stat --null/offline CPU segv related fixes/tests

[Ingo Molnar]

* Ian Rogers <irogers@google.com> wrote:

> Ingo reported [1] that `perf stat --null` was segfaulting. Fix the
> underlying issue and add a test to the "perf stat tests". Do some
> related fixing/cleanup in the perf util cpumap code.
>
> Thomas reported an issue fixed by the same patches [2] but caused by
> giving perf stat an offline CPU. Add test coverage for that and
> improve the "error" message that reports "success".
>
> Ingo further pointed at broken signal handling in repeat mode [3]. I
> observed we weren't giving the best exit code, 0 rather than the
> expected 128+<signal number>. Add a patch fixing this.
>
> [1] https://lore.kernel.org/linux-perf-users/aSwt7yzFjVJCEmVp@gmail.com/
> [2] https://lore.kernel.org/linux-perf-users/94313b82-888b-4f42-9fb0-4585f9e90080@linux.ibm.com/
> [3] https://lore.kernel.org/lkml/aS5wjmbAM9ka3M2g@gmail.com/
>
> Ian Rogers (7):
>   perf stat: Allow no events to open if this is a "--null" run
>   libperf cpumap: Fix perf_cpu_map__max for an empty/NULL map
>   perf cpumap: Add "any" CPU handling to cpu_map__snprint_mask
>   perf tests stat: Add "--null" coverage
>   perf stat: When no events, don't report an error if there is none
>   perf tests stat: Add test for error for an offline CPU
>   perf stat: Improve handling of termination by signal
>
>  tools/lib/perf/cpumap.c        | 10 +++++----
>  tools/perf/builtin-stat.c      | 29 ++++++++++++++++++-------
>  tools/perf/tests/shell/stat.sh | 39 ++++++++++++++++++++++++++++++++++
>  tools/perf/util/cpumap.c       |  9 ++++++--
>  4 files changed, 73 insertions(+), 14 deletions(-)

A belated:

  Tested-by: Ingo Molnar <mingo@kernel.org>

And thank you a lot for doing these QoL fixes!

	Ingo

Re: [PATCH v2 0/7] Perf stat --null/offline CPU segv related fixes/tests

[Ingo Molnar]

* Ingo Molnar <mingo@kernel.org> wrote:

> * Ian Rogers <irogers@google.com> wrote:
>
> > Ingo reported [1] that `perf stat --null` was segfaulting. Fix the
> > underlying issue and add a test to the "perf stat tests". Do some
> > related fixing/cleanup in the perf util cpumap code.
> >
> > Thomas reported an issue fixed by the same patches [2] but caused by
> > giving perf stat an offline CPU. Add test coverage for that and
> > improve the "error" message that reports "success".
> >
> > Ingo further pointed at broken signal handling in repeat mode [3]. I
> > observed we weren't giving the best exit code, 0 rather than the
> > expected 128+<signal number>. Add a patch fixing this.
> >
> > [1] https://lore.kernel.org/linux-perf-users/aSwt7yzFjVJCEmVp@gmail.com/
> > [2] https://lore.kernel.org/linux-perf-users/94313b82-888b-4f42-9fb0-4585f9e90080@linux.ibm.com/
> > [3] https://lore.kernel.org/lkml/aS5wjmbAM9ka3M2g@gmail.com/
> >
> > Ian Rogers (7):
> >   perf stat: Allow no events to open if this is a "--null" run
> >   libperf cpumap: Fix perf_cpu_map__max for an empty/NULL map
> >   perf cpumap: Add "any" CPU handling to cpu_map__snprint_mask
> >   perf tests stat: Add "--null" coverage
> >   perf stat: When no events, don't report an error if there is none
> >   perf tests stat: Add test for error for an offline CPU
> >   perf stat: Improve handling of termination by signal
> >
> >  tools/lib/perf/cpumap.c        | 10 +++++----
> >  tools/perf/builtin-stat.c      | 29 ++++++++++++++++++-------
> >  tools/perf/tests/shell/stat.sh | 39 ++++++++++++++++++++++++++++++++++
> >  tools/perf/util/cpumap.c       |  9 ++++++--
> >  4 files changed, 73 insertions(+), 14 deletions(-)
>
> A belated:
>
>   Tested-by: Ingo Molnar <mingo@kernel.org>
>
> And thank you a lot for doing these QoL fixes!

There's one more perf stat QoL bug I'd like to report - I frequently
do repeated runs of perf stat --repeat and grep the output, to get
a feel for the run-to-run stability of a particular benchmark:

  starship:~/tip> while :; do perf stat --null --repeat 3 sleep 0.1 2>&1 | grep elapsed; done 
         0.1017997 +- 0.0000771 seconds time elapsed  ( +-  0.08% )
         0.1017627 +- 0.0000795 seconds time elapsed  ( +-  0.08% )
         0.1018106 +- 0.0000650 seconds time elapsed  ( +-  0.06% )
         0.1017844 +- 0.0000601 seconds time elapsed  ( +-  0.06% )
          0.101883 +- 0.000169 seconds time elapsed  ( +-  0.17% ) <====
         0.1017757 +- 0.0000532 seconds time elapsed  ( +-  0.05% )
         0.1017991 +- 0.0000720 seconds time elapsed  ( +-  0.07% )
         0.1018024 +- 0.0000704 seconds time elapsed  ( +-  0.07% )
         0.1018074 +- 0.0000946 seconds time elapsed  ( +-  0.09% )
         0.1019797 +- 0.0000524 seconds time elapsed  ( +-  0.05% )
         0.1018407 +- 0.0000658 seconds time elapsed  ( +-  0.06% )
         0.1017907 +- 0.0000605 seconds time elapsed  ( +-  0.06% )
         0.1018328 +- 0.0000868 seconds time elapsed  ( +-  0.09% )
         0.1017469 +- 0.0000285 seconds time elapsed  ( +-  0.03% )
         0.1019589 +- 0.0000549 seconds time elapsed  ( +-  0.05% )
         0.1018465 +- 0.0000891 seconds time elapsed  ( +-  0.09% )
          0.101868 +- 0.000117 seconds time elapsed  ( +-  0.12% ) <====
         0.1017705 +- 0.0000590 seconds time elapsed  ( +-  0.06% )
         0.1017728 +- 0.0000718 seconds time elapsed  ( +-  0.07% )
         0.1017821 +- 0.0000419 seconds time elapsed  ( +-  0.04% )
         0.1018328 +- 0.0000581 seconds time elapsed  ( +-  0.06% )
         0.1017836 +- 0.0000853 seconds time elapsed  ( +-  0.08% )
         0.1018124 +- 0.0000765 seconds time elapsed  ( +-  0.08% )
         0.1018706 +- 0.0000639 seconds time elapsed  ( +-  0.06% )

Note the two outliers, which happen due to some misguided
output optimization feature in perf shortening zero-ended
numbers unnecessarily, and adding noise to the grepped
output's vertical alignment.

Those two lines should be:

         0.1017844 +- 0.0000601 seconds time elapsed  ( +-  0.06% )
         0.1018830 +- 0.0001690 seconds time elapsed  ( +-  0.17% ) <====
         0.1017757 +- 0.0000532 seconds time elapsed  ( +-  0.05% )

         0.1018465 +- 0.0000891 seconds time elapsed  ( +-  0.09% )
         0.1018680 +- 0.0001170 seconds time elapsed  ( +-  0.12% ) <====
         0.1017705 +- 0.0000590 seconds time elapsed  ( +-  0.06% )

(The zeroes are printed fully, to full precision.)

Basically random chance causing an apparent lack of significant
numbers doesn't mean the tool should strip them from the output.

Thanks,

	Ingo

Re: [PATCH v2 0/7] Perf stat --null/offline CPU segv related fixes/tests

[Ian Rogers]

On Sat, Dec 6, 2025 at 3:20 AM Ingo Molnar <mingo@kernel.org> wrote:
>
> * Ingo Molnar <mingo@kernel.org> wrote:
>
> > * Ian Rogers <irogers@google.com> wrote:
> >
> > > Ingo reported [1] that `perf stat --null` was segfaulting. Fix the
> > > underlying issue and add a test to the "perf stat tests". Do some
> > > related fixing/cleanup in the perf util cpumap code.
> > >
> > > Thomas reported an issue fixed by the same patches [2] but caused by
> > > giving perf stat an offline CPU. Add test coverage for that and
> > > improve the "error" message that reports "success".
> > >
> > > Ingo further pointed at broken signal handling in repeat mode [3]. I
> > > observed we weren't giving the best exit code, 0 rather than the
> > > expected 128+<signal number>. Add a patch fixing this.
> > >
> > > [1] https://lore.kernel.org/linux-perf-users/aSwt7yzFjVJCEmVp@gmail.com/
> > > [2] https://lore.kernel.org/linux-perf-users/94313b82-888b-4f42-9fb0-4585f9e90080@linux.ibm.com/
> > > [3] https://lore.kernel.org/lkml/aS5wjmbAM9ka3M2g@gmail.com/
> > >
> > > Ian Rogers (7):
> > >   perf stat: Allow no events to open if this is a "--null" run
> > >   libperf cpumap: Fix perf_cpu_map__max for an empty/NULL map
> > >   perf cpumap: Add "any" CPU handling to cpu_map__snprint_mask
> > >   perf tests stat: Add "--null" coverage
> > >   perf stat: When no events, don't report an error if there is none
> > >   perf tests stat: Add test for error for an offline CPU
> > >   perf stat: Improve handling of termination by signal
> > >
> > >  tools/lib/perf/cpumap.c        | 10 +++++----
> > >  tools/perf/builtin-stat.c      | 29 ++++++++++++++++++-------
> > >  tools/perf/tests/shell/stat.sh | 39 ++++++++++++++++++++++++++++++++++
> > >  tools/perf/util/cpumap.c       |  9 ++++++--
> > >  4 files changed, 73 insertions(+), 14 deletions(-)
> >
> > A belated:
> >
> >   Tested-by: Ingo Molnar <mingo@kernel.org>
> >
> > And thank you a lot for doing these QoL fixes!
>
> There's one more perf stat QoL bug I'd like to report - I frequently
> do repeated runs of perf stat --repeat and grep the output, to get
> a feel for the run-to-run stability of a particular benchmark:
>
>   starship:~/tip> while :; do perf stat --null --repeat 3 sleep 0.1 2>&1 | grep elapsed; done
>          0.1017997 +- 0.0000771 seconds time elapsed  ( +-  0.08% )
>          0.1017627 +- 0.0000795 seconds time elapsed  ( +-  0.08% )
>          0.1018106 +- 0.0000650 seconds time elapsed  ( +-  0.06% )
>          0.1017844 +- 0.0000601 seconds time elapsed  ( +-  0.06% )
>           0.101883 +- 0.000169 seconds time elapsed  ( +-  0.17% ) <====
>          0.1017757 +- 0.0000532 seconds time elapsed  ( +-  0.05% )
>          0.1017991 +- 0.0000720 seconds time elapsed  ( +-  0.07% )
>          0.1018024 +- 0.0000704 seconds time elapsed  ( +-  0.07% )
>          0.1018074 +- 0.0000946 seconds time elapsed  ( +-  0.09% )
>          0.1019797 +- 0.0000524 seconds time elapsed  ( +-  0.05% )
>          0.1018407 +- 0.0000658 seconds time elapsed  ( +-  0.06% )
>          0.1017907 +- 0.0000605 seconds time elapsed  ( +-  0.06% )
>          0.1018328 +- 0.0000868 seconds time elapsed  ( +-  0.09% )
>          0.1017469 +- 0.0000285 seconds time elapsed  ( +-  0.03% )
>          0.1019589 +- 0.0000549 seconds time elapsed  ( +-  0.05% )
>          0.1018465 +- 0.0000891 seconds time elapsed  ( +-  0.09% )
>           0.101868 +- 0.000117 seconds time elapsed  ( +-  0.12% ) <====
>          0.1017705 +- 0.0000590 seconds time elapsed  ( +-  0.06% )
>          0.1017728 +- 0.0000718 seconds time elapsed  ( +-  0.07% )
>          0.1017821 +- 0.0000419 seconds time elapsed  ( +-  0.04% )
>          0.1018328 +- 0.0000581 seconds time elapsed  ( +-  0.06% )
>          0.1017836 +- 0.0000853 seconds time elapsed  ( +-  0.08% )
>          0.1018124 +- 0.0000765 seconds time elapsed  ( +-  0.08% )
>          0.1018706 +- 0.0000639 seconds time elapsed  ( +-  0.06% )
>
> Note the two outliers, which happen due to some misguided
> output optimization feature in perf shortening zero-ended
> numbers unnecessarily, and adding noise to the grepped
> output's vertical alignment.
>
> Those two lines should be:
>
>          0.1017844 +- 0.0000601 seconds time elapsed  ( +-  0.06% )
>          0.1018830 +- 0.0001690 seconds time elapsed  ( +-  0.17% ) <====
>          0.1017757 +- 0.0000532 seconds time elapsed  ( +-  0.05% )
>
>          0.1018465 +- 0.0000891 seconds time elapsed  ( +-  0.09% )
>          0.1018680 +- 0.0001170 seconds time elapsed  ( +-  0.12% ) <====
>          0.1017705 +- 0.0000590 seconds time elapsed  ( +-  0.06% )
>
> (The zeroes are printed fully, to full precision.)
>
> Basically random chance causing an apparent lack of significant
> numbers doesn't mean the tool should strip them from the output.

Ugh. I'm not sure why the output is like that. Searching back through
history it seems you are to blame :-)
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/commit/tools/perf/builtin-stat.c?h=perf-tools-next&id=bc22de9bcdb2249150fb5b3c48fdc4f6bedd3ad7

Perhaps we can drop the precision printf flag and just make the flags
fixed. I think this output is obscure enough that nobody cares about
minor changes.

Thanks,
Ian

> Thanks,
>
>         Ingo

Re: [PATCH v2 0/7] Perf stat --null/offline CPU segv related fixes/tests

[Ingo Molnar]

* Ian Rogers <irogers@google.com> wrote:

> > There's one more perf stat QoL bug I'd like to report - I frequently
> > do repeated runs of perf stat --repeat and grep the output, to get
> > a feel for the run-to-run stability of a particular benchmark:
> >
> >   starship:~/tip> while :; do perf stat --null --repeat 3 sleep 0.1 2>&1 | grep elapsed; done
> >          0.1017997 +- 0.0000771 seconds time elapsed  ( +-  0.08% )
> >          0.1017627 +- 0.0000795 seconds time elapsed  ( +-  0.08% )
> >          0.1018106 +- 0.0000650 seconds time elapsed  ( +-  0.06% )
> >          0.1017844 +- 0.0000601 seconds time elapsed  ( +-  0.06% )
> >           0.101883 +- 0.000169 seconds time elapsed  ( +-  0.17% ) <====
> >          0.1017757 +- 0.0000532 seconds time elapsed  ( +-  0.05% )
> >          0.1017991 +- 0.0000720 seconds time elapsed  ( +-  0.07% )
> >          0.1018024 +- 0.0000704 seconds time elapsed  ( +-  0.07% )
> >          0.1018074 +- 0.0000946 seconds time elapsed  ( +-  0.09% )
> >          0.1019797 +- 0.0000524 seconds time elapsed  ( +-  0.05% )
> >          0.1018407 +- 0.0000658 seconds time elapsed  ( +-  0.06% )
> >          0.1017907 +- 0.0000605 seconds time elapsed  ( +-  0.06% )
> >          0.1018328 +- 0.0000868 seconds time elapsed  ( +-  0.09% )
> >          0.1017469 +- 0.0000285 seconds time elapsed  ( +-  0.03% )
> >          0.1019589 +- 0.0000549 seconds time elapsed  ( +-  0.05% )
> >          0.1018465 +- 0.0000891 seconds time elapsed  ( +-  0.09% )
> >           0.101868 +- 0.000117 seconds time elapsed  ( +-  0.12% ) <====
> >          0.1017705 +- 0.0000590 seconds time elapsed  ( +-  0.06% )
> >          0.1017728 +- 0.0000718 seconds time elapsed  ( +-  0.07% )
> >          0.1017821 +- 0.0000419 seconds time elapsed  ( +-  0.04% )
> >          0.1018328 +- 0.0000581 seconds time elapsed  ( +-  0.06% )
> >          0.1017836 +- 0.0000853 seconds time elapsed  ( +-  0.08% )
> >          0.1018124 +- 0.0000765 seconds time elapsed  ( +-  0.08% )
> >          0.1018706 +- 0.0000639 seconds time elapsed  ( +-  0.06% )
> >
> > Note the two outliers, which happen due to some misguided
> > output optimization feature in perf shortening zero-ended
> > numbers unnecessarily, and adding noise to the grepped
> > output's vertical alignment.
> >
> > Those two lines should be:
> >
> >          0.1017844 +- 0.0000601 seconds time elapsed  ( +-  0.06% )
> >          0.1018830 +- 0.0001690 seconds time elapsed  ( +-  0.17% ) <====
> >          0.1017757 +- 0.0000532 seconds time elapsed  ( +-  0.05% )
> >
> >          0.1018465 +- 0.0000891 seconds time elapsed  ( +-  0.09% )
> >          0.1018680 +- 0.0001170 seconds time elapsed  ( +-  0.12% ) <====
> >          0.1017705 +- 0.0000590 seconds time elapsed  ( +-  0.06% )
> >
> > (The zeroes are printed fully, to full precision.)
> >
> > Basically random chance causing an apparent lack of significant
> > numbers doesn't mean the tool should strip them from the output.
>
> Ugh. I'm not sure why the output is like that.

Yeah, I'm sure some dim-witted kernel developer requested it,
without thinking through all the consequences.

> [...] Searching back through history it seems you are to blame :-)
> https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/commit/tools/perf/builtin-stat.c?h=perf-tools-next&id=bc22de9bcdb2249150fb5b3c48fdc4f6bedd3ad7

Uhm, never mind, what a brilliant feature! ;-)

> Perhaps we can drop the precision printf flag and just make the flags
> fixed. I think this output is obscure enough that nobody cares about
> minor changes.

Yeah, sounds good.

Thanks,

	Ingo

Re: [PATCH v2 0/7] Perf stat --null/offline CPU segv related fixes/tests

[Ian Rogers]

On Tue, Dec 9, 2025 at 12:44 AM Ingo Molnar <mingo@kernel.org> wrote:
>
>
> * Ian Rogers <irogers@google.com> wrote:
>
> > > There's one more perf stat QoL bug I'd like to report - I frequently
> > > do repeated runs of perf stat --repeat and grep the output, to get
> > > a feel for the run-to-run stability of a particular benchmark:
> > >
> > >   starship:~/tip> while :; do perf stat --null --repeat 3 sleep 0.1 2>&1 | grep elapsed; done
> > >          0.1017997 +- 0.0000771 seconds time elapsed  ( +-  0.08% )
> > >          0.1017627 +- 0.0000795 seconds time elapsed  ( +-  0.08% )
> > >          0.1018106 +- 0.0000650 seconds time elapsed  ( +-  0.06% )
> > >          0.1017844 +- 0.0000601 seconds time elapsed  ( +-  0.06% )
> > >           0.101883 +- 0.000169 seconds time elapsed  ( +-  0.17% ) <====
> > >          0.1017757 +- 0.0000532 seconds time elapsed  ( +-  0.05% )
> > >          0.1017991 +- 0.0000720 seconds time elapsed  ( +-  0.07% )
> > >          0.1018024 +- 0.0000704 seconds time elapsed  ( +-  0.07% )
> > >          0.1018074 +- 0.0000946 seconds time elapsed  ( +-  0.09% )
> > >          0.1019797 +- 0.0000524 seconds time elapsed  ( +-  0.05% )
> > >          0.1018407 +- 0.0000658 seconds time elapsed  ( +-  0.06% )
> > >          0.1017907 +- 0.0000605 seconds time elapsed  ( +-  0.06% )
> > >          0.1018328 +- 0.0000868 seconds time elapsed  ( +-  0.09% )
> > >          0.1017469 +- 0.0000285 seconds time elapsed  ( +-  0.03% )
> > >          0.1019589 +- 0.0000549 seconds time elapsed  ( +-  0.05% )
> > >          0.1018465 +- 0.0000891 seconds time elapsed  ( +-  0.09% )
> > >           0.101868 +- 0.000117 seconds time elapsed  ( +-  0.12% ) <====
> > >          0.1017705 +- 0.0000590 seconds time elapsed  ( +-  0.06% )
> > >          0.1017728 +- 0.0000718 seconds time elapsed  ( +-  0.07% )
> > >          0.1017821 +- 0.0000419 seconds time elapsed  ( +-  0.04% )
> > >          0.1018328 +- 0.0000581 seconds time elapsed  ( +-  0.06% )
> > >          0.1017836 +- 0.0000853 seconds time elapsed  ( +-  0.08% )
> > >          0.1018124 +- 0.0000765 seconds time elapsed  ( +-  0.08% )
> > >          0.1018706 +- 0.0000639 seconds time elapsed  ( +-  0.06% )
> > >
> > > Note the two outliers, which happen due to some misguided
> > > output optimization feature in perf shortening zero-ended
> > > numbers unnecessarily, and adding noise to the grepped
> > > output's vertical alignment.
> > >
> > > Those two lines should be:
> > >
> > >          0.1017844 +- 0.0000601 seconds time elapsed  ( +-  0.06% )
> > >          0.1018830 +- 0.0001690 seconds time elapsed  ( +-  0.17% ) <====
> > >          0.1017757 +- 0.0000532 seconds time elapsed  ( +-  0.05% )
> > >
> > >          0.1018465 +- 0.0000891 seconds time elapsed  ( +-  0.09% )
> > >          0.1018680 +- 0.0001170 seconds time elapsed  ( +-  0.12% ) <====
> > >          0.1017705 +- 0.0000590 seconds time elapsed  ( +-  0.06% )
> > >
> > > (The zeroes are printed fully, to full precision.)
> > >
> > > Basically random chance causing an apparent lack of significant
> > > numbers doesn't mean the tool should strip them from the output.
> >
> > Ugh. I'm not sure why the output is like that.
>
> Yeah, I'm sure some dim-witted kernel developer requested it,
> without thinking through all the consequences.
>
> > [...] Searching back through history it seems you are to blame :-)
> > https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/commit/tools/perf/builtin-stat.c?h=perf-tools-next&id=bc22de9bcdb2249150fb5b3c48fdc4f6bedd3ad7
>
> Uhm, never mind, what a brilliant feature! ;-)
>
> > Perhaps we can drop the precision printf flag and just make the flags
> > fixed. I think this output is obscure enough that nobody cares about
> > minor changes.
>
> Yeah, sounds good.

Cool. I sent:
https://lore.kernel.org/lkml/20251209173610.2033973-1-irogers@google.com/

Thanks,
Ian

> Thanks,
>
>         Ingo
>