* [PATCH] perf stat: Fix crash on arm64
@ 2026-02-05 11:46 Breno Leitao
2026-02-05 13:32 ` Dmitry Ilvokhin
2026-02-05 16:59 ` Ian Rogers
0 siblings, 2 replies; 14+ messages in thread
From: Breno Leitao @ 2026-02-05 11:46 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Ian Rogers, Adrian Hunter, James Clark
Cc: linux-perf-users, linux-kernel, kernel-team, Denis Yaroshevskiy,
Breno Leitao
Perf stat is crashing on arm64 hosts with the following issue:
# make -C tools/perf DEBUG=1
# perf stat sleep 1
perf: util/evsel.c:2034: get_group_fd: Assertion `!(!leader->core.fd)' failed.
[1] 1220794 IOT instruction (core dumped) ./perf stat
The sorting function introduced by commit a745c0831c15c ("perf stat:
Sort default events/metrics") compares events based on their individual
properties. This can cause events from different groups to be
interleaved, resulting in group members appearing before their leaders
in the sorted evlist.
When the iterator opens events in list order, a group member may be
processed before its leader has been opened.
For example, CPU_CYCLES (idx=32) with leader STALL_SLOT_BACKEND (idx=37)
could be sorted before its leader, causing the crash when CPU_CYCLES
tries to get its group fd from the not-yet-opened leader.
Fix this by comparing events based on their leader's attributes instead
of their own attributes when the events are in different groups. This
ensures all members of a group share the same sort key as their leader,
keeping groups together and guaranteeing leaders are opened before their
members.
Reported-by: Denis Yaroshevskiy <dyaroshev@meta.com>
Fixes: a745c0831c15c ("perf stat: Sort default events/metrics")
Signed-off-by: Breno Leitao <leitao@debian.org>
---
Cc; linux-arm-kernel@lists.infradead.org
---
tools/perf/builtin-stat.c | 26 +++++++++++++++++---------
1 file changed, 17 insertions(+), 9 deletions(-)
diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c
index ab40d85fb1259..3a423ca31d8d3 100644
--- a/tools/perf/builtin-stat.c
+++ b/tools/perf/builtin-stat.c
@@ -1938,25 +1938,33 @@ static int default_evlist_evsel_cmp(void *priv __maybe_unused,
const struct evsel *lhs = container_of(lhs_core, struct evsel, core);
const struct perf_evsel *rhs_core = container_of(r, struct perf_evsel, node);
const struct evsel *rhs = container_of(rhs_core, struct evsel, core);
+ const struct evsel *lhs_leader = evsel__leader(lhs);
+ const struct evsel *rhs_leader = evsel__leader(rhs);
- if (evsel__leader(lhs) == evsel__leader(rhs)) {
+ if (lhs_leader == rhs_leader) {
/* Within the same group, respect the original order. */
return lhs_core->idx - rhs_core->idx;
}
+ /*
+ * Compare using leader's attributes so that all members of a group
+ * stay together. This ensures leaders are opened before their members.
+ */
+
/* Sort default metrics evsels first, and default show events before those. */
- if (lhs->default_metricgroup != rhs->default_metricgroup)
- return lhs->default_metricgroup ? -1 : 1;
+ if (lhs_leader->default_metricgroup != rhs_leader->default_metricgroup)
+ return lhs_leader->default_metricgroup ? -1 : 1;
- if (lhs->default_show_events != rhs->default_show_events)
- return lhs->default_show_events ? -1 : 1;
+ if (lhs_leader->default_show_events != rhs_leader->default_show_events)
+ return lhs_leader->default_show_events ? -1 : 1;
/* Sort by PMU type (prefers legacy types first). */
- if (lhs->pmu != rhs->pmu)
- return lhs->pmu->type - rhs->pmu->type;
+ if (lhs_leader->pmu != rhs_leader->pmu)
+ return lhs_leader->pmu->type - rhs_leader->pmu->type;
- /* Sort by name. */
- return strcmp(evsel__name((struct evsel *)lhs), evsel__name((struct evsel *)rhs));
+ /* Sort by leader's name. */
+ return strcmp(evsel__name((struct evsel *)lhs_leader),
+ evsel__name((struct evsel *)rhs_leader));
}
/*
---
base-commit: 5fd0a1df5d05ad066e5618ccdd3d0fa6cb686c27
change-id: 20260205-perf_stat-a0a2a37e21c5
Best regards,
--
Breno Leitao <leitao@debian.org>
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH] perf stat: Fix crash on arm64
2026-02-05 11:46 [PATCH] perf stat: Fix crash on arm64 Breno Leitao
@ 2026-02-05 13:32 ` Dmitry Ilvokhin
2026-02-05 16:59 ` Ian Rogers
1 sibling, 0 replies; 14+ messages in thread
From: Dmitry Ilvokhin @ 2026-02-05 13:32 UTC (permalink / raw)
To: Breno Leitao
Cc: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Ian Rogers, Adrian Hunter, James Clark, linux-perf-users,
linux-kernel, kernel-team, Denis Yaroshevskiy
On Thu, Feb 05, 2026 at 03:46:31AM -0800, Breno Leitao wrote:
> Perf stat is crashing on arm64 hosts with the following issue:
>
> # make -C tools/perf DEBUG=1
> # perf stat sleep 1
> perf: util/evsel.c:2034: get_group_fd: Assertion `!(!leader->core.fd)' failed.
> [1] 1220794 IOT instruction (core dumped) ./perf stat
>
> The sorting function introduced by commit a745c0831c15c ("perf stat:
> Sort default events/metrics") compares events based on their individual
> properties. This can cause events from different groups to be
> interleaved, resulting in group members appearing before their leaders
> in the sorted evlist.
>
> When the iterator opens events in list order, a group member may be
> processed before its leader has been opened.
>
> For example, CPU_CYCLES (idx=32) with leader STALL_SLOT_BACKEND (idx=37)
> could be sorted before its leader, causing the crash when CPU_CYCLES
> tries to get its group fd from the not-yet-opened leader.
>
> Fix this by comparing events based on their leader's attributes instead
> of their own attributes when the events are in different groups. This
> ensures all members of a group share the same sort key as their leader,
> keeping groups together and guaranteeing leaders are opened before their
> members.
>
> Reported-by: Denis Yaroshevskiy <dyaroshev@meta.com>
> Fixes: a745c0831c15c ("perf stat: Sort default events/metrics")
> Signed-off-by: Breno Leitao <leitao@debian.org>
> ---
> Cc; linux-arm-kernel@lists.infradead.org
> ---
> tools/perf/builtin-stat.c | 26 +++++++++++++++++---------
> 1 file changed, 17 insertions(+), 9 deletions(-)
>
> diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c
> index ab40d85fb1259..3a423ca31d8d3 100644
> --- a/tools/perf/builtin-stat.c
> +++ b/tools/perf/builtin-stat.c
> @@ -1938,25 +1938,33 @@ static int default_evlist_evsel_cmp(void *priv __maybe_unused,
> const struct evsel *lhs = container_of(lhs_core, struct evsel, core);
> const struct perf_evsel *rhs_core = container_of(r, struct perf_evsel, node);
> const struct evsel *rhs = container_of(rhs_core, struct evsel, core);
> + const struct evsel *lhs_leader = evsel__leader(lhs);
> + const struct evsel *rhs_leader = evsel__leader(rhs);
>
> - if (evsel__leader(lhs) == evsel__leader(rhs)) {
> + if (lhs_leader == rhs_leader) {
> /* Within the same group, respect the original order. */
> return lhs_core->idx - rhs_core->idx;
> }
>
> + /*
> + * Compare using leader's attributes so that all members of a group
> + * stay together. This ensures leaders are opened before their members.
> + */
> +
> /* Sort default metrics evsels first, and default show events before those. */
> - if (lhs->default_metricgroup != rhs->default_metricgroup)
> - return lhs->default_metricgroup ? -1 : 1;
> + if (lhs_leader->default_metricgroup != rhs_leader->default_metricgroup)
> + return lhs_leader->default_metricgroup ? -1 : 1;
>
> - if (lhs->default_show_events != rhs->default_show_events)
> - return lhs->default_show_events ? -1 : 1;
> + if (lhs_leader->default_show_events != rhs_leader->default_show_events)
> + return lhs_leader->default_show_events ? -1 : 1;
>
> /* Sort by PMU type (prefers legacy types first). */
> - if (lhs->pmu != rhs->pmu)
> - return lhs->pmu->type - rhs->pmu->type;
> + if (lhs_leader->pmu != rhs_leader->pmu)
> + return lhs_leader->pmu->type - rhs_leader->pmu->type;
>
> - /* Sort by name. */
> - return strcmp(evsel__name((struct evsel *)lhs), evsel__name((struct evsel *)rhs));
> + /* Sort by leader's name. */
> + return strcmp(evsel__name((struct evsel *)lhs_leader),
> + evsel__name((struct evsel *)rhs_leader));
> }
>
> /*
>
> ---
> base-commit: 5fd0a1df5d05ad066e5618ccdd3d0fa6cb686c27
> change-id: 20260205-perf_stat-a0a2a37e21c5
>
> Best regards,
> --
> Breno Leitao <leitao@debian.org>
>
Tested-by: Dmitry Ilvokhin <d@ilvokhin.com>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] perf stat: Fix crash on arm64
2026-02-05 11:46 [PATCH] perf stat: Fix crash on arm64 Breno Leitao
2026-02-05 13:32 ` Dmitry Ilvokhin
@ 2026-02-05 16:59 ` Ian Rogers
2026-02-05 17:39 ` Leo Yan
2026-02-06 12:01 ` Breno Leitao
1 sibling, 2 replies; 14+ messages in thread
From: Ian Rogers @ 2026-02-05 16:59 UTC (permalink / raw)
To: Breno Leitao
Cc: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Adrian Hunter, James Clark, linux-perf-users, linux-kernel,
kernel-team, Denis Yaroshevskiy
On Thu, Feb 5, 2026 at 3:46 AM Breno Leitao <leitao@debian.org> wrote:
>
> Perf stat is crashing on arm64 hosts with the following issue:
>
> # make -C tools/perf DEBUG=1
> # perf stat sleep 1
> perf: util/evsel.c:2034: get_group_fd: Assertion `!(!leader->core.fd)' failed.
> [1] 1220794 IOT instruction (core dumped) ./perf stat
>
> The sorting function introduced by commit a745c0831c15c ("perf stat:
> Sort default events/metrics") compares events based on their individual
> properties. This can cause events from different groups to be
> interleaved, resulting in group members appearing before their leaders
> in the sorted evlist.
Hi, sorry for the issue. I can see what you're saying but why is this
an arm64 issue? The legacy Default metrics are common to all
architectures:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/pmu-events/arch/common/common/metrics.json?h=perf-tools-next
> When the iterator opens events in list order, a group member may be
> processed before its leader has been opened.
>
> For example, CPU_CYCLES (idx=32) with leader STALL_SLOT_BACKEND (idx=37)
> could be sorted before its leader, causing the crash when CPU_CYCLES
> tries to get its group fd from the not-yet-opened leader.
Which metric is this?
> Fix this by comparing events based on their leader's attributes instead
> of their own attributes when the events are in different groups. This
> ensures all members of a group share the same sort key as their leader,
> keeping groups together and guaranteeing leaders are opened before their
> members.
This makes sense but I'm not understanding why this problem wasn't
seen previously. I'm guessing that in a metric like
backend_cycles_idle:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/pmu-events/arch/common/common/metrics.json?h=perf-tools-next#n63
```
"BriefDescription": "Backend stalls per cycle",
"MetricExpr": "stalled\\-cycles\\-backend / cpu\\-cycles",
"MetricGroup": "Default",
"MetricName": "backend_cycles_idle",
"MetricThreshold": "backend_cycles_idle > 0.2",
"DefaultShowEvents": "1"
```
The PMUs for cpu-cycles and stalled-cycles differ? This may mean we
also need to be smarting in determining PMUs for legacy events.
It'd be interesting to see what events are coming from the kernel, e.g.:
```
$ ls /sys/bus/event_source/devices/*/events
/sys/bus/event_source/devices/cpu_atom/events:
branch-instructions cache-misses instructions ref-cycles
topdown-fe-bound
branch-misses cache-references mem-loads topdown-bad-spec
topdown-retiring
bus-cycles cpu-cycles mem-stores topdown-be-bound
...
```
and the cpuid to match it up with the json.
```
$ perf stat -v sleep 1 2>&1 |head -1
Using CPUID GenuineIntel-6-B7-1
$ ./tools/perf/pmu-events/models.py x86 GenuineIntel-6-B7-1
tools/perf/pmu-events/arch/
alderlake
```
this information is in the verbose output too:
```
$ perf stat -vv sleep 1
...
------------------------------------------------------------
perf_event_attr:
type 1 (PERF_TYPE_SOFTWARE)
size 136
config 0x2 (PERF_COUNT_SW_PAGE_FAULTS)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
sys_perf_event_open: pid 608809 cpu -1 group_fd -1 flags 0x8 = 7
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 136
config 0xa00000001
(cpu_atom/PERF_COUNT_HW_INSTRUCTIONS/)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
sys_perf_event_open: pid 608809 cpu -1 group_fd -1 flags 0x8 = 8
...
```
Thanks,
Ian
> Reported-by: Denis Yaroshevskiy <dyaroshev@meta.com>
> Fixes: a745c0831c15c ("perf stat: Sort default events/metrics")
> Signed-off-by: Breno Leitao <leitao@debian.org>
> ---
> Cc; linux-arm-kernel@lists.infradead.org
> ---
> tools/perf/builtin-stat.c | 26 +++++++++++++++++---------
> 1 file changed, 17 insertions(+), 9 deletions(-)
>
> diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c
> index ab40d85fb1259..3a423ca31d8d3 100644
> --- a/tools/perf/builtin-stat.c
> +++ b/tools/perf/builtin-stat.c
> @@ -1938,25 +1938,33 @@ static int default_evlist_evsel_cmp(void *priv __maybe_unused,
> const struct evsel *lhs = container_of(lhs_core, struct evsel, core);
> const struct perf_evsel *rhs_core = container_of(r, struct perf_evsel, node);
> const struct evsel *rhs = container_of(rhs_core, struct evsel, core);
> + const struct evsel *lhs_leader = evsel__leader(lhs);
> + const struct evsel *rhs_leader = evsel__leader(rhs);
>
> - if (evsel__leader(lhs) == evsel__leader(rhs)) {
> + if (lhs_leader == rhs_leader) {
> /* Within the same group, respect the original order. */
> return lhs_core->idx - rhs_core->idx;
> }
>
> + /*
> + * Compare using leader's attributes so that all members of a group
> + * stay together. This ensures leaders are opened before their members.
> + */
> +
> /* Sort default metrics evsels first, and default show events before those. */
> - if (lhs->default_metricgroup != rhs->default_metricgroup)
> - return lhs->default_metricgroup ? -1 : 1;
> + if (lhs_leader->default_metricgroup != rhs_leader->default_metricgroup)
> + return lhs_leader->default_metricgroup ? -1 : 1;
>
> - if (lhs->default_show_events != rhs->default_show_events)
> - return lhs->default_show_events ? -1 : 1;
> + if (lhs_leader->default_show_events != rhs_leader->default_show_events)
> + return lhs_leader->default_show_events ? -1 : 1;
>
> /* Sort by PMU type (prefers legacy types first). */
> - if (lhs->pmu != rhs->pmu)
> - return lhs->pmu->type - rhs->pmu->type;
> + if (lhs_leader->pmu != rhs_leader->pmu)
> + return lhs_leader->pmu->type - rhs_leader->pmu->type;
>
> - /* Sort by name. */
> - return strcmp(evsel__name((struct evsel *)lhs), evsel__name((struct evsel *)rhs));
> + /* Sort by leader's name. */
> + return strcmp(evsel__name((struct evsel *)lhs_leader),
> + evsel__name((struct evsel *)rhs_leader));
> }
>
> /*
>
> ---
> base-commit: 5fd0a1df5d05ad066e5618ccdd3d0fa6cb686c27
> change-id: 20260205-perf_stat-a0a2a37e21c5
>
> Best regards,
> --
> Breno Leitao <leitao@debian.org>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] perf stat: Fix crash on arm64
2026-02-05 16:59 ` Ian Rogers
@ 2026-02-05 17:39 ` Leo Yan
2026-02-05 17:52 ` Leo Yan
2026-02-06 12:01 ` Breno Leitao
1 sibling, 1 reply; 14+ messages in thread
From: Leo Yan @ 2026-02-05 17:39 UTC (permalink / raw)
To: Ian Rogers
Cc: Breno Leitao, Peter Zijlstra, Ingo Molnar,
Arnaldo Carvalho de Melo, Namhyung Kim, Mark Rutland,
Alexander Shishkin, Jiri Olsa, Adrian Hunter, James Clark,
linux-perf-users, linux-kernel, kernel-team, Denis Yaroshevskiy
Hi Ian,
On Thu, Feb 05, 2026 at 08:59:07AM -0800, Ian Rogers wrote:
> On Thu, Feb 5, 2026 at 3:46 AM Breno Leitao <leitao@debian.org> wrote:
> >
> > Perf stat is crashing on arm64 hosts with the following issue:
> >
> > # make -C tools/perf DEBUG=1
> > # perf stat sleep 1
> > perf: util/evsel.c:2034: get_group_fd: Assertion `!(!leader->core.fd)' failed.
> > [1] 1220794 IOT instruction (core dumped) ./perf stat
> >
> > The sorting function introduced by commit a745c0831c15c ("perf stat:
> > Sort default events/metrics") compares events based on their individual
> > properties. This can cause events from different groups to be
> > interleaved, resulting in group members appearing before their leaders
> > in the sorted evlist.
>
> Hi, sorry for the issue. I can see what you're saying but why is this
> an arm64 issue? The legacy Default metrics are common to all
> architectures:
> https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/pmu-events/arch/common/common/metrics.json?h=perf-tools-next
Since you are mentioning common metrics, I found the common metrics does
not work on Arm64 platform (I built with NO_JEVENTS=1 or enabled jevnts
but both don't work).
The latest perf will have no any output if the CPU type is missed in
json and rallback to common metrics. The failure path is:
add_default_events()
metricgroup__parse_groups()
pmu_metrics_table__find() => return NULL
In my case, pmu_metrics_table__find() always return NULL, as a result,
`perf stat sleep 1` directly bail out without any output.
I expect Breno's env might have the corresponding CPU json files, this
is possible different from my test machine.
Thanks,
Leo
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] perf stat: Fix crash on arm64
2026-02-05 17:39 ` Leo Yan
@ 2026-02-05 17:52 ` Leo Yan
2026-02-05 18:22 ` Ian Rogers
0 siblings, 1 reply; 14+ messages in thread
From: Leo Yan @ 2026-02-05 17:52 UTC (permalink / raw)
To: Ian Rogers
Cc: Breno Leitao, Peter Zijlstra, Ingo Molnar,
Arnaldo Carvalho de Melo, Namhyung Kim, Mark Rutland,
Alexander Shishkin, Jiri Olsa, Adrian Hunter, James Clark,
linux-perf-users, linux-kernel, kernel-team, Denis Yaroshevskiy
On Thu, Feb 05, 2026 at 05:39:18PM +0000, Leo Yan wrote:
> > > The sorting function introduced by commit a745c0831c15c ("perf stat:
> > > Sort default events/metrics") compares events based on their individual
> > > properties. This can cause events from different groups to be
> > > interleaved, resulting in group members appearing before their leaders
> > > in the sorted evlist.
> >
> > Hi, sorry for the issue. I can see what you're saying but why is this
> > an arm64 issue? The legacy Default metrics are common to all
> > architectures:
> > https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/pmu-events/arch/common/common/metrics.json?h=perf-tools-next
>
> Since you are mentioning common metrics, I found the common metrics does
> not work on Arm64 platform (I built with NO_JEVENTS=1 or enabled jevnts
> but both don't work).
>
> The latest perf will have no any output if the CPU type is missed in
> json and rallback to common metrics. The failure path is:
>
> add_default_events()
> metricgroup__parse_groups()
> pmu_metrics_table__find() => return NULL
>
> In my case, pmu_metrics_table__find() always return NULL, as a result,
> `perf stat sleep 1` directly bail out without any output.
>
> I expect Breno's env might have the corresponding CPU json files, this
> is possible different from my test machine.
On my local env, I need a fix:
diff --git a/tools/perf/pmu-events/empty-pmu-events.c b/tools/perf/pmu-events/empty-pmu-events.c
index e4d00f6b2b5d..f74acc206856 100644
--- a/tools/perf/pmu-events/empty-pmu-events.c
+++ b/tools/perf/pmu-events/empty-pmu-events.c
@@ -3237,14 +3237,6 @@ const struct pmu_events_table *perf_pmu__default_core_events_table(void)
return NULL;
}
-const struct pmu_metrics_table *pmu_metrics_table__find(void)
-{
- struct perf_cpu cpu = {-1};
- const struct pmu_events_map *map = map_for_cpu(cpu);
-
- return map ? &map->metric_table : NULL;
-}
-
const struct pmu_metrics_table *pmu_metrics_table__default(void)
{
int i = 0;
@@ -3261,6 +3253,17 @@ const struct pmu_metrics_table *pmu_metrics_table__default(void)
return NULL;
}
+const struct pmu_metrics_table *pmu_metrics_table__find(void)
+{
+ struct perf_cpu cpu = {-1};
+ const struct pmu_events_map *map = map_for_cpu(cpu);
+
+ if (map)
+ return &map->metric_table;
+
+ return pmu_metrics_table__default();
+}
+
I have no deep understanding for jevents, seems to me, Breno's issue is
a different one from me. Please kindly confirm.
Thanks,
Leo
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH] perf stat: Fix crash on arm64
2026-02-05 17:52 ` Leo Yan
@ 2026-02-05 18:22 ` Ian Rogers
2026-03-11 10:21 ` Breno Leitao
0 siblings, 1 reply; 14+ messages in thread
From: Ian Rogers @ 2026-02-05 18:22 UTC (permalink / raw)
To: Leo Yan
Cc: Breno Leitao, Peter Zijlstra, Ingo Molnar,
Arnaldo Carvalho de Melo, Namhyung Kim, Mark Rutland,
Alexander Shishkin, Jiri Olsa, Adrian Hunter, James Clark,
linux-perf-users, linux-kernel, kernel-team, Denis Yaroshevskiy
On Thu, Feb 5, 2026 at 9:52 AM Leo Yan <leo.yan@arm.com> wrote:
>
> On Thu, Feb 05, 2026 at 05:39:18PM +0000, Leo Yan wrote:
>
> > > > The sorting function introduced by commit a745c0831c15c ("perf stat:
> > > > Sort default events/metrics") compares events based on their individual
> > > > properties. This can cause events from different groups to be
> > > > interleaved, resulting in group members appearing before their leaders
> > > > in the sorted evlist.
> > >
> > > Hi, sorry for the issue. I can see what you're saying but why is this
> > > an arm64 issue? The legacy Default metrics are common to all
> > > architectures:
> > > https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/pmu-events/arch/common/common/metrics.json?h=perf-tools-next
> >
> > Since you are mentioning common metrics, I found the common metrics does
> > not work on Arm64 platform (I built with NO_JEVENTS=1 or enabled jevnts
> > but both don't work).
> >
> > The latest perf will have no any output if the CPU type is missed in
> > json and rallback to common metrics. The failure path is:
> >
> > add_default_events()
> > metricgroup__parse_groups()
> > pmu_metrics_table__find() => return NULL
> >
The return is correct but the early return is wrong, the metric code
was updated to always consider the default table and skip a NULL
table:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/metricgroup.c?h=perf-tools-next#n430
I'll send a patch for the early return.
> > In my case, pmu_metrics_table__find() always return NULL, as a result,
> > `perf stat sleep 1` directly bail out without any output.
> >
> > I expect Breno's env might have the corresponding CPU json files, this
> > is possible different from my test machine.
>
> On my local env, I need a fix:
>
> diff --git a/tools/perf/pmu-events/empty-pmu-events.c b/tools/perf/pmu-events/empty-pmu-events.c
> index e4d00f6b2b5d..f74acc206856 100644
> --- a/tools/perf/pmu-events/empty-pmu-events.c
> +++ b/tools/perf/pmu-events/empty-pmu-events.c
> @@ -3237,14 +3237,6 @@ const struct pmu_events_table *perf_pmu__default_core_events_table(void)
> return NULL;
> }
>
> -const struct pmu_metrics_table *pmu_metrics_table__find(void)
> -{
> - struct perf_cpu cpu = {-1};
> - const struct pmu_events_map *map = map_for_cpu(cpu);
> -
> - return map ? &map->metric_table : NULL;
> -}
> -
> const struct pmu_metrics_table *pmu_metrics_table__default(void)
> {
> int i = 0;
> @@ -3261,6 +3253,17 @@ const struct pmu_metrics_table *pmu_metrics_table__default(void)
> return NULL;
> }
>
> +const struct pmu_metrics_table *pmu_metrics_table__find(void)
> +{
> + struct perf_cpu cpu = {-1};
> + const struct pmu_events_map *map = map_for_cpu(cpu);
> +
> + if (map)
> + return &map->metric_table;
> +
> + return pmu_metrics_table__default();
> +}
> +
>
> I have no deep understanding for jevents, seems to me, Breno's issue is
> a different one from me. Please kindly confirm.
I think it is a different issue, they have metrics while you don't.
Your report does highlight we're missing a NO_JEVENTS=1 build-test,
but the build is working for me. I'll send out two patches for these
issues.
Thanks,
Ian
> Thanks,
> Leo
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] perf stat: Fix crash on arm64
2026-02-05 16:59 ` Ian Rogers
2026-02-05 17:39 ` Leo Yan
@ 2026-02-06 12:01 ` Breno Leitao
1 sibling, 0 replies; 14+ messages in thread
From: Breno Leitao @ 2026-02-06 12:01 UTC (permalink / raw)
To: Ian Rogers
Cc: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Adrian Hunter, James Clark, linux-perf-users, linux-kernel,
kernel-team, Denis Yaroshevskiy
Hello Ian, thanks for the quick reply!
On Thu, Feb 05, 2026 at 08:59:07AM -0800, Ian Rogers wrote:
> On Thu, Feb 5, 2026 at 3:46 AM Breno Leitao <leitao@debian.org> wrote:
> >
> > Perf stat is crashing on arm64 hosts with the following issue:
> >
> > # make -C tools/perf DEBUG=1
> > # perf stat sleep 1
> > perf: util/evsel.c:2034: get_group_fd: Assertion `!(!leader->core.fd)' failed.
> > [1] 1220794 IOT instruction (core dumped) ./perf stat
> >
> > The sorting function introduced by commit a745c0831c15c ("perf stat:
> > Sort default events/metrics") compares events based on their individual
> > properties. This can cause events from different groups to be
> > interleaved, resulting in group members appearing before their leaders
> > in the sorted evlist.
>
> Hi, sorry for the issue. I can see what you're saying but why is this
> an arm64 issue?
Sorry, It's not ARM64-specific - the bug is in the generic sort code.
It just happens to manifest on ARM64.
> The legacy Default metrics are common to all
> architectures:
> https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/pmu-events/arch/common/common/metrics.json?h=perf-tools-next
>
> > When the iterator opens events in list order, a group member may be
> > processed before its leader has been opened.
> >
> > For example, CPU_CYCLES (idx=32) with leader STALL_SLOT_BACKEND (idx=37)
> > could be sorted before its leader, causing the crash when CPU_CYCLES
> > tries to get its group fd from the not-yet-opened leader.
>
> Which metric is this?
These are ARM neoverse metrics, they can be found in
tools/perf/pmu-events/arch/arm64/arm/neoverse-n*
> > Fix this by comparing events based on their leader's attributes instead
> > of their own attributes when the events are in different groups. This
> > ensures all members of a group share the same sort key as their leader,
> > keeping groups together and guaranteeing leaders are opened before their
> > members.
>
> This makes sense but I'm not understanding why this problem wasn't
> seen previously. I'm guessing that in a metric like
> backend_cycles_idle:
> https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/pmu-events/arch/common/common/metrics.json?h=perf-tools-next#n63
> ```
> "BriefDescription": "Backend stalls per cycle",
> "MetricExpr": "stalled\\-cycles\\-backend / cpu\\-cycles",
> "MetricGroup": "Default",
> "MetricName": "backend_cycles_idle",
> "MetricThreshold": "backend_cycles_idle > 0.2",
> "DefaultShowEvents": "1"
> ```
I was able to limit this to the following json:
[
{
"ArchStdEvent": "backend_bound",
"MetricExpr": "(100 * ((STALL_SLOT_BACKEND / (CPU_CYCLES * #slots)) - ((BR_MIS_PRED * 3) / CPU_CYCLES)))"
},
{
"ArchStdEvent": "frontend_bound",
"MetricExpr": "(100 * (((STALL_SLOT_FRONTEND) / (CPU_CYCLES * #slots)) - (BR_MIS_PRED / CPU_CYCLES)))"
}
]
and then
# ./tools/perf/perf stat -v sleep 0.01
Using CPUID 0x00000000410fd4f0
metric expr 100 * (STALL_SLOT_BACKEND / (CPU_CYCLES * #slots) - BR_MIS_PRED * 3 / CPU_CYCLES) for backend_bound
metric expr 100 * (STALL_SLOT_FRONTEND / (CPU_CYCLES * #slots) - BR_MIS_PRED / CPU_CYCLES) for frontend_bound
metric expr (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) / (duration_time * 1e9) for CPUs_utilized
metric expr stalled\-cycles\-backend / cpu\-cycles for backend_cycles_idle
metric expr stalled\-cycles\-backend / cpu\-cycles for backend_cycles_idle
metric expr branches / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for branch_frequency
metric expr branch\-misses / branches for branch_miss_rate
metric expr branch\-misses / branches for branch_miss_rate
metric expr software@context\-switches\,name\=context\-switches@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for cs_per_second
metric expr cpu\-cycles / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for cycles_frequency
metric expr stalled\-cycles\-frontend / cpu\-cycles for frontend_cycles_idle
metric expr stalled\-cycles\-frontend / cpu\-cycles for frontend_cycles_idle
metric expr instructions / cpu\-cycles for insn_per_cycle
metric expr instructions / cpu\-cycles for insn_per_cycle
metric expr software@cpu\-migrations\,name\=cpu\-migrations@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for migrations_per_second
metric expr software@page\-faults\,name\=page\-faults@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for page_faults_per_second
metric expr max(stalled\-cycles\-frontend, stalled\-cycles\-backend) / instructions for stalled_cycles_per_instruction
hwmon_pmu: failure to open '/sys/class/hwmon/hwmon4/name'
hwmon_pmu: failure to open '/sys/class/hwmon/hwmon5/name'
hwmon_pmu: failure to open '/sys/class/hwmon/hwmon3/name'
found event software@context-switches,name=context-switches@
found event duration_time
found event software@page-faults,name=page-faults@
found event software@task-clock,name=task-clock@
found event cpu-cycles
found event branches
found event software@cpu-migrations,name=cpu-migrations@
Parsing metric events 'software/context-switches,name=context-switches,metric-id=software!3context!1switches!0name!2context!1switches!3/,software/page-faults,name=page-faults,metric-id=software!3page!1faults!0name!2page!1faults!3/,software/task-clock,name=task-clock,metric-id=software!3task!1clock!0name!2task!1clock!3/,cpu-cycles/metric-id=cpu!1cycles/,branches/metric-id=branches/,software/cpu-migrations,name=cpu-migrations,metric-id=software!3cpu!1migrations!0name!2cpu!1migrations!3/,duration_time'
cpu-cycles -> armv8_pmuv3_0/metric-id=cpu!1cycles,cpu-cycles/
branches -> armv8_pmuv3_0/metric-id=branches,branches/
duration_time -> tool/duration_time/
found event STALL_SLOT_FRONTEND
found event duration_time
found event BR_MIS_PRED
found event CPU_CYCLES
Parsing metric events '{STALL_SLOT_FRONTEND/metric-id=STALL_SLOT_FRONTEND/,BR_MIS_PRED/metric-id=BR_MIS_PRED/,CPU_CYCLES/metric-id=CPU_CYCLES/}:W,duration_time'
STALL_SLOT_FRONTEND -> armv8_pmuv3_0/metric-id=STALL_SLOT_FRONTEND,STALL_SLOT_FRONTEND/
BR_MIS_PRED -> armv8_pmuv3_0/metric-id=BR_MIS_PRED,BR_MIS_PRED/
CPU_CYCLES -> armv8_pmuv3_0/metric-id=CPU_CYCLES,CPU_CYCLES/
duration_time -> tool/duration_time/
Matched metric-id STALL_SLOT_FRONTEND to STALL_SLOT_FRONTEND
Matched metric-id BR_MIS_PRED to BR_MIS_PRED
Matched metric-id CPU_CYCLES to CPU_CYCLES
Matched metric-id duration_time to duration_time
found event STALL_SLOT_BACKEND
found event duration_time
found event BR_MIS_PRED
found event CPU_CYCLES
Parsing metric events '{STALL_SLOT_BACKEND/metric-id=STALL_SLOT_BACKEND/,BR_MIS_PRED/metric-id=BR_MIS_PRED/,CPU_CYCLES/metric-id=CPU_CYCLES/}:W,duration_time'
STALL_SLOT_BACKEND -> armv8_pmuv3_0/metric-id=STALL_SLOT_BACKEND,STALL_SLOT_BACKEND/
BR_MIS_PRED -> armv8_pmuv3_0/metric-id=BR_MIS_PRED,BR_MIS_PRED/
CPU_CYCLES -> armv8_pmuv3_0/metric-id=CPU_CYCLES,CPU_CYCLES/
duration_time -> tool/duration_time/
Matched metric-id STALL_SLOT_BACKEND to STALL_SLOT_BACKEND
Matched metric-id BR_MIS_PRED to BR_MIS_PRED
Matched metric-id CPU_CYCLES to CPU_CYCLES
Matched metric-id duration_time to duration_time
found event duration_time
found event stalled-cycles-backend
found event instructions
found event stalled-cycles-frontend
Parsing metric events '{stalled-cycles-backend/metric-id=stalled!1cycles!1backend/,instructions/metric-id=instructions/,stalled-cycles-frontend/metric-id=stalled!1cycles!1frontend/}:W,duration_time'
stalled-cycles-backend -> armv8_pmuv3_0/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
instructions -> armv8_pmuv3_0/metric-id=instructions,instructions/
stalled-cycles-frontend -> armv8_pmuv3_0/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
duration_time -> tool/duration_time/
Matched metric-id stalled-cycles-backend to stalled-cycles-backend
Matched metric-id instructions to instructions
Matched metric-id stalled-cycles-frontend to stalled-cycles-frontend
Matched metric-id duration_time to duration_time
Matched metric-id software@page-faults,name=page-faults@ to page-faults
Matched metric-id software@task-clock,name=task-clock@ to task-clock
Matched metric-id software@task-clock,name=task-clock@ to task-clock
Matched metric-id software@cpu-migrations,name=cpu-migrations@ to cpu-migrations
found event duration_time
found event cpu-cycles
found event instructions
Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,instructions/metric-id=instructions/}:W,duration_time'
cpu-cycles -> armv8_pmuv3_0/metric-id=cpu!1cycles,cpu-cycles/
instructions -> armv8_pmuv3_0/metric-id=instructions,instructions/
duration_time -> tool/duration_time/
Matched metric-id cpu-cycles to cpu-cycles
Matched metric-id instructions to instructions
Matched metric-id duration_time to duration_time
found event duration_time
found event cpu-cycles
found event stalled-cycles-frontend
Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,stalled-cycles-frontend/metric-id=stalled!1cycles!1frontend/}:W,duration_time'
cpu-cycles -> armv8_pmuv3_0/metric-id=cpu!1cycles,cpu-cycles/
stalled-cycles-frontend -> armv8_pmuv3_0/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
duration_time -> tool/duration_time/
Matched metric-id cpu-cycles to cpu-cycles
Matched metric-id stalled-cycles-frontend to stalled-cycles-frontend
Matched metric-id duration_time to duration_time
Matched metric-id software@task-clock,name=task-clock@ to task-clock
Matched metric-id cpu-cycles to cpu-cycles
Matched metric-id software@context-switches,name=context-switches@ to context-switches
Matched metric-id software@task-clock,name=task-clock@ to task-clock
found event duration_time
found event branch-misses
found event branches
Parsing metric events '{branch-misses/metric-id=branch!1misses/,branches/metric-id=branches/}:W,duration_time'
branch-misses -> armv8_pmuv3_0/metric-id=branch!1misses,branch-misses/
branches -> armv8_pmuv3_0/metric-id=branches,branches/
duration_time -> tool/duration_time/
Matched metric-id branch-misses to branch-misses
Matched metric-id branches to branches
Matched metric-id duration_time to duration_time
Matched metric-id software@task-clock,name=task-clock@ to task-clock
Matched metric-id branches to branches
found event duration_time
found event cpu-cycles
found event stalled-cycles-backend
Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,stalled-cycles-backend/metric-id=stalled!1cycles!1backend/}:W,duration_time'
cpu-cycles -> armv8_pmuv3_0/metric-id=cpu!1cycles,cpu-cycles/
stalled-cycles-backend -> armv8_pmuv3_0/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
duration_time -> tool/duration_time/
Matched metric-id cpu-cycles to cpu-cycles
Matched metric-id stalled-cycles-backend to stalled-cycles-backend
Matched metric-id duration_time to duration_time
Matched metric-id software@task-clock,name=task-clock@ to task-clock
Matched metric-id duration_time to duration_time
copying metric event for cgroup 'root': context-switches (idx=0)
copying metric event for cgroup 'root': page-faults (idx=1)
copying metric event for cgroup 'root': task-clock (idx=2)
copying metric event for cgroup 'root': cpu-cycles (idx=3)
copying metric event for cgroup 'root': branches (idx=4)
copying metric event for cgroup 'root': cpu-migrations (idx=5)
copying metric event for cgroup 'root': STALL_SLOT_FRONTEND (idx=7)
copying metric event for cgroup 'root': stalled-cycles-backend (idx=29)
copying metric event for cgroup 'root': STALL_SLOT_BACKEND (idx=11)
copying metric event for cgroup 'root': stalled-cycles-backend (idx=15)
copying metric event for cgroup 'root': instructions (idx=20)
copying metric event for cgroup 'root': stalled-cycles-frontend (idx=23)
copying metric event for cgroup 'root': branch-misses (idx=25)
copying metric event for cgroup 'root': context-switches (idx=6)
copying metric event for cgroup 'root': page-faults (idx=8)
copying metric event for cgroup 'root': task-clock (idx=9)
copying metric event for cgroup 'root': cpu-cycles (idx=13)
copying metric event for cgroup 'root': branches (idx=12)
copying metric event for cgroup 'root': cpu-migrations (idx=7)
copying metric event for cgroup 'root': STALL_SLOT_FRONTEND (idx=25)
copying metric event for cgroup 'root': stalled-cycles-backend (idx=19)
copying metric event for cgroup 'root': STALL_SLOT_BACKEND (idx=29)
copying metric event for cgroup 'root': stalled-cycles-backend (idx=20)
copying metric event for cgroup 'root': instructions (idx=15)
copying metric event for cgroup 'root': stalled-cycles-frontend (idx=17)
copying metric event for cgroup 'root': branch-misses (idx=10)
Control descriptor is not initialized
perf: util/evsel.c:2034: get_group_fd: Assertion `!(!leader->core.fd)' failed.
[1] 832866 IOT instruction (core dumped) ./tools/perf/perf stat -v sleep 0.01
> The PMUs for cpu-cycles and stalled-cycles differ? This may mean we
> also need to be smarting in determining PMUs for legacy events.
>
> It'd be interesting to see what events are coming from the kernel, e.g.:
> ```
> $ ls /sys/bus/event_source/devices/*/events
# ls /sys/bus/event_source/devices/*/events
/sys/bus/event_source/devices/armv8_pmuv3_0/events:
br_mis_pred cti_trigout7 l1d_tlb_refill l2d_tlb_refill mem_access_checked_wr stall_backend_mem
br_mis_pred_retired dtlb_walk l1i_cache l3d_cache memory_error stall_frontend
br_pred exc_return l1i_cache_lmiss l3d_cache_allocate op_retired stall_slot
br_retired exc_taken l1i_cache_refill l3d_cache_lmiss_rd op_spec stall_slot_backend
bus_access inst_retired l1i_tlb l3d_cache_refill remote_access stall_slot_frontend
bus_cycles inst_spec l1i_tlb_refill ld_align_lat sample_collision trb_wrap
cid_write_retired itlb_walk l2d_cache ldst_align_lat sample_feed trcextout0
cnt_cycles l1d_cache l2d_cache_allocate ll_cache_miss_rd sample_filtrate trcextout1
cpu_cycles l1d_cache_lmiss_rd l2d_cache_lmiss_rd ll_cache_rd sample_pop trcextout2
cti_trigout4 l1d_cache_refill l2d_cache_refill mem_access st_align_lat trcextout3
cti_trigout5 l1d_cache_wb l2d_cache_wb mem_access_checked stall ttbr_write_retired
cti_trigout6 l1d_tlb l2d_tlb mem_access_checked_rd stall_backend
/sys/bus/event_source/devices/cs_etm/events:
autofdo
/sys/bus/event_source/devices/nvidia_cnvlink_pmu_0/events:
cycles rd_bytes_rem rd_cum_outs_rem rd_req_rem total_bytes_rem total_req_rem wr_bytes_rem wr_req_rem
rd_bytes_loc rd_cum_outs_loc rd_req_loc total_bytes_loc total_req_loc wr_bytes_loc wr_req_loc
/sys/bus/event_source/devices/nvidia_nvlink_c2c0_pmu_0/events:
cycles rd_bytes_rem rd_cum_outs_rem rd_req_rem total_bytes_rem total_req_rem wr_bytes_rem wr_req_rem
rd_bytes_loc rd_cum_outs_loc rd_req_loc total_bytes_loc total_req_loc wr_bytes_loc wr_req_loc
/sys/bus/event_source/devices/nvidia_nvlink_c2c1_pmu_0/events:
cycles rd_bytes_rem rd_cum_outs_rem rd_req_rem total_bytes_rem total_req_rem wr_bytes_rem wr_req_rem
rd_bytes_loc rd_cum_outs_loc rd_req_loc total_bytes_loc total_req_loc wr_bytes_loc wr_req_loc
/sys/bus/event_source/devices/nvidia_pcie_pmu_0/events:
cycles rd_bytes_rem rd_cum_outs_rem rd_req_rem total_bytes_rem total_req_rem wr_bytes_rem wr_req_rem
rd_bytes_loc rd_cum_outs_loc rd_req_loc total_bytes_loc total_req_loc wr_bytes_loc wr_req_loc
/sys/bus/event_source/devices/nvidia_scf_pmu_0/events:
bus_cycles gmem_rd_data scf_cache socket_1_rd_access socket_2_wb_data
cmem_rd_access gmem_rd_outstanding scf_cache_allocate socket_1_rd_data socket_2_wr_access
cmem_rd_data gmem_wb_access scf_cache_refill socket_1_rd_outstanding socket_2_wr_data
cmem_rd_outstanding gmem_wb_data scf_cache_wb socket_1_wb_access socket_3_rd_access
cmem_wb_access gmem_wr_access socket_0_rd_access socket_1_wb_data socket_3_rd_data
cmem_wb_data gmem_wr_data socket_0_rd_data socket_1_wr_access socket_3_rd_outstanding
cmem_wr_access gmem_wr_total_bytes socket_0_rd_outstanding socket_1_wr_data socket_3_wb_access
cmem_wr_data remote_socket_rd_access socket_0_wb_access socket_2_rd_access socket_3_wb_data
cmem_wr_total_bytes remote_socket_rd_data socket_0_wb_data socket_2_rd_data socket_3_wr_access
cycles remote_socket_rd_outstanding socket_0_wr_access socket_2_rd_outstanding socket_3_wr_data
gmem_rd_access remote_socket_wr_total_bytes socket_0_wr_data socket_2_wb_access
/sys/bus/event_source/devices/smmuv3_pmcg_11002/events:
config_cache_miss config_struct_access cycles pcie_ats_trans_rq tlb_miss transaction trans_table_walk_access
/sys/bus/event_source/devices/smmuv3_pmcg_11042/events:
cycles pcie_ats_trans_passed tlb_miss transaction
/sys/bus/event_source/devices/smmuv3_pmcg_11062/events:
cycles pcie_ats_trans_passed tlb_miss transaction
/sys/bus/event_source/devices/smmuv3_pmcg_11082/events:
cycles pcie_ats_trans_passed tlb_miss transaction
/sys/bus/event_source/devices/smmuv3_pmcg_110a2/events:
cycles pcie_ats_trans_passed tlb_miss transaction
/sys/bus/event_source/devices/smmuv3_pmcg_12002/events:
config_cache_miss config_struct_access cycles pcie_ats_trans_rq tlb_miss transaction trans_table_walk_access
/sys/bus/event_source/devices/smmuv3_pmcg_12042/events:
cycles pcie_ats_trans_passed tlb_miss transaction
/sys/bus/event_source/devices/smmuv3_pmcg_12062/events:
cycles pcie_ats_trans_passed tlb_miss transaction
/sys/bus/event_source/devices/smmuv3_pmcg_12082/events:
cycles pcie_ats_trans_passed tlb_miss transaction
/sys/bus/event_source/devices/smmuv3_pmcg_120a2/events:
cycles pcie_ats_trans_passed tlb_miss transaction
/sys/bus/event_source/devices/smmuv3_pmcg_15002/events:
config_cache_miss config_struct_access cycles pcie_ats_trans_rq tlb_miss transaction trans_table_walk_access
/sys/bus/event_source/devices/smmuv3_pmcg_15042/events:
cycles pcie_ats_trans_passed tlb_miss transaction
/sys/bus/event_source/devices/smmuv3_pmcg_15062/events:
cycles pcie_ats_trans_passed tlb_miss transaction
/sys/bus/event_source/devices/smmuv3_pmcg_15082/events:
cycles pcie_ats_trans_passed tlb_miss transaction
/sys/bus/event_source/devices/smmuv3_pmcg_150a2/events:
cycles pcie_ats_trans_passed tlb_miss transaction
/sys/bus/event_source/devices/smmuv3_pmcg_16002/events:
config_cache_miss config_struct_access cycles pcie_ats_trans_rq tlb_miss transaction trans_table_walk_access
/sys/bus/event_source/devices/smmuv3_pmcg_16042/events:
cycles pcie_ats_trans_passed tlb_miss transaction
/sys/bus/event_source/devices/smmuv3_pmcg_16062/events:
cycles pcie_ats_trans_passed tlb_miss transaction
/sys/bus/event_source/devices/smmuv3_pmcg_5002/events:
config_cache_miss config_struct_access cycles pcie_ats_trans_rq tlb_miss transaction trans_table_walk_access
/sys/bus/event_source/devices/smmuv3_pmcg_5042/events:
cycles pcie_ats_trans_passed tlb_miss transaction
/sys/bus/event_source/devices/smmuv3_pmcg_5062/events:
cycles pcie_ats_trans_passed tlb_miss transaction
> ```
> and the cpuid to match it up with the json.
> ```
> $ perf stat -v sleep 1 2>&1 |head -1
# perf stat -v sleep 1 2>&1 |head -1
Using CPUID 0x00000000410fd4f0
> this information is in the verbose output too:
> ```
> $ perf stat -vv sleep 1
------------------------------------------------------------
perf_event_attr:
type 4294967294 (tool)
size 144
config 0x1 (duration_time)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
------------------------------------------------------------
perf_event_attr:
type 4294967294 (tool)
size 144
config 0x1 (duration_time)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
------------------------------------------------------------
perf_event_attr:
type 4294967294 (tool)
size 144
config 0x1 (duration_time)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
------------------------------------------------------------
perf_event_attr:
type 4294967294 (tool)
size 144
config 0x1 (duration_time)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
------------------------------------------------------------
perf_event_attr:
type 4294967294 (tool)
size 144
config 0x1 (duration_time)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
------------------------------------------------------------
perf_event_attr:
type 4294967294 (tool)
size 144
config 0x1 (duration_time)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
------------------------------------------------------------
perf_event_attr:
type 1 (PERF_TYPE_SOFTWARE)
size 144
config 0x3 (PERF_COUNT_SW_CONTEXT_SWITCHES)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd -1 flags 0x8 = 3
------------------------------------------------------------
perf_event_attr:
type 1 (PERF_TYPE_SOFTWARE)
size 144
config 0x4 (PERF_COUNT_SW_CPU_MIGRATIONS)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd -1 flags 0x8 = 4
------------------------------------------------------------
perf_event_attr:
type 1 (PERF_TYPE_SOFTWARE)
size 144
config 0x2 (PERF_COUNT_SW_PAGE_FAULTS)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd -1 flags 0x8 = 5
------------------------------------------------------------
perf_event_attr:
type 1 (PERF_TYPE_SOFTWARE)
size 144
config 0x1 (PERF_COUNT_SW_TASK_CLOCK)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd -1 flags 0x8 = 7
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 144
config 0x5 (PERF_COUNT_HW_BRANCH_MISSES)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd -1 flags 0x8 = 8
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 144
config 0x4 (PERF_COUNT_HW_BRANCH_INSTRUCTIONS)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
inherit 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd 8 flags 0x8 = 9
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 144
config 0x4 (PERF_COUNT_HW_BRANCH_INSTRUCTIONS)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd -1 flags 0x8 = 10
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 144
config 0 (PERF_COUNT_HW_CPU_CYCLES)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd -1 flags 0x8 = 11
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 144
config 0 (PERF_COUNT_HW_CPU_CYCLES)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd -1 flags 0x8 = 12
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 144
config 0x1 (PERF_COUNT_HW_INSTRUCTIONS)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
inherit 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd 12 flags 0x8 = 13
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 144
config 0 (PERF_COUNT_HW_CPU_CYCLES)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd -1 flags 0x8 = 14
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 144
config 0x7 (PERF_COUNT_HW_STALLED_CYCLES_FRONTEND)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
inherit 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd 14 flags 0x8 = 15
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 144
config 0 (PERF_COUNT_HW_CPU_CYCLES)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd -1 flags 0x8 = 16
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 144
config 0x8 (PERF_COUNT_HW_STALLED_CYCLES_BACKEND)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
inherit 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd 16 flags 0x8 = 17
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 144
config 0x8 (PERF_COUNT_HW_STALLED_CYCLES_BACKEND)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd -1 flags 0x8 = 18
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 144
config 0x1 (PERF_COUNT_HW_INSTRUCTIONS)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
inherit 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd 18 flags 0x8 = 19
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 144
config 0x7 (PERF_COUNT_HW_STALLED_CYCLES_FRONTEND)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
inherit 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd 18 flags 0x8 = 20
------------------------------------------------------------
perf_event_attr:
type 4294967294 (tool)
size 144
config 0x1 (duration_time)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
------------------------------------------------------------
perf_event_attr:
type 4294967294 (tool)
size 144
config 0x1 (duration_time)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
------------------------------------------------------------
perf_event_attr:
type 4294967294 (tool)
size 144
config 0x1 (duration_time)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
------------------------------------------------------------
perf_event_attr:
type 4294967294 (tool)
size 144
config 0x1 (duration_time)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
------------------------------------------------------------
perf_event_attr:
type 10 (armv8_pmuv3_0)
size 144
config 0x10 (br_mis_pred)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
disabled 1
inherit 1
enable_on_exec 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd -1 flags 0x8 = 21
------------------------------------------------------------
perf_event_attr:
type 10 (armv8_pmuv3_0)
size 144
config 0x3b (op_spec)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
inherit 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd 21 flags 0x8 = 22
------------------------------------------------------------
perf_event_attr:
type 10 (armv8_pmuv3_0)
size 144
config 0x3f (stall_slot)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
inherit 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd 21 flags 0x8 = 23
------------------------------------------------------------
perf_event_attr:
type 10 (armv8_pmuv3_0)
size 144
config 0x11 (cpu_cycles)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
inherit 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd 21 flags 0x8 = 24
------------------------------------------------------------
perf_event_attr:
type 10 (armv8_pmuv3_0)
size 144
config 0x3a (op_retired)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
inherit 1
------------------------------------------------------------
sys_perf_event_open: pid 865887 cpu -1 group_fd 21 flags 0x8 = 25
------------------------------------------------------------
perf_event_attr:
type 10 (armv8_pmuv3_0)
size 144
config 0x11 (cpu_cycles)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
inherit 1
------------------------------------------------------------
Thanks for your help,
--breno
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] perf stat: Fix crash on arm64
2026-02-05 18:22 ` Ian Rogers
@ 2026-03-11 10:21 ` Breno Leitao
2026-03-11 11:50 ` Leo Yan
0 siblings, 1 reply; 14+ messages in thread
From: Breno Leitao @ 2026-03-11 10:21 UTC (permalink / raw)
To: Ian Rogers, leo.yan, acme
Cc: Leo Yan, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Adrian Hunter, James Clark, linux-perf-users, linux-kernel,
kernel-team, Denis Yaroshevskiy
On Thu, Feb 05, 2026 at 10:22:27AM -0800, Ian Rogers wrote:
> I think it is a different issue, they have metrics while you don't.
> Your report does highlight we're missing a NO_JEVENTS=1 build-test,
> but the build is working for me. I'll send out two patches for these
> issues.
Hi Ian, Leo, Acme
I wanted to follow up on this. Are there any next steps I should take?
Thanks,
--breno
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] perf stat: Fix crash on arm64
2026-03-11 10:21 ` Breno Leitao
@ 2026-03-11 11:50 ` Leo Yan
2026-03-23 14:21 ` Arnaldo Carvalho de Melo
2026-03-23 17:06 ` Ian Rogers
0 siblings, 2 replies; 14+ messages in thread
From: Leo Yan @ 2026-03-11 11:50 UTC (permalink / raw)
To: Breno Leitao
Cc: Ian Rogers, acme, Peter Zijlstra, Ingo Molnar, Namhyung Kim,
Mark Rutland, Alexander Shishkin, Jiri Olsa, Adrian Hunter,
James Clark, linux-perf-users, linux-kernel, kernel-team,
Denis Yaroshevskiy
Hi Breno,
On Wed, Mar 11, 2026 at 03:21:00AM -0700, Breno Leitao wrote:
> On Thu, Feb 05, 2026 at 10:22:27AM -0800, Ian Rogers wrote:
> > I think it is a different issue, they have metrics while you don't.
> > Your report does highlight we're missing a NO_JEVENTS=1 build-test,
> > but the build is working for me. I'll send out two patches for these
> > issues.
>
> Hi Ian, Leo, Acme
>
> I wanted to follow up on this. Are there any next steps I should take?
Sorry for not tracking this issue.
I can reproduce the issue on my Orion6 with setting a _dummy_ CPUID:
$ export PERF_CPUID=0x00000000410fd490
$ perf stat -C 5 -vvv
...
Aborted
perf: util/evsel.c:2156: get_group_fd: Assertion `!(!leader->core.fd)' failed
Because we are working on different hardwares, I am a bit suspect I
reproduced the issue with difference sequence as yours. Anyway, I do
see that an event can be opened prior to its leader event, see the log
below.
Thus, your patch seems make sense to me as we need to ensure the leader
event to be opened first. Ian, how about you think?
I have a side question (maybe this is for James :): why other arch (or
platforms) don't have reproduce the issue? I am wandering if any error
or special stuffs in Arm CPU's json files.
Thanks,
Leo
---8<---
Opening: BR_MIS_PRED
------------------------------------------------------------
perf_event_attr:
type 11 (armv8_pmuv3_1)
size 144
config 0x10 (br_mis_pred)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
disabled 1
inherit 1
------------------------------------------------------------
evsel=BR_MIS_PRED (supported=1) leader=BR_MIS_PRED
sys_perf_event_open: pid -1 cpu 5 group_fd -1 flags 0x8
sys_perf_event_open failed, error -2
Warning:
BR_MIS_PRED event is not supported by the kernel.
The BR_MIS_PRED event is not supported.
Opening: OP_SPEC
------------------------------------------------------------
perf_event_attr:
type 11 (armv8_pmuv3_1)
size 144
config 0x3b (op_spec)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
inherit 1
------------------------------------------------------------
evsel=OP_SPEC (supported=1) leader=BR_MIS_PRED
leader->core.fd=0xaaab19edd950
broken group leader for OP_SPEC
Warning:
skipping event OP_SPEC that kernel failed to open.
The sys_perf_event_open() syscall failed for event (OP_SPEC): Success
"dmesg | grep -i perf" may provide additional information.
Opening: STALL_SLOT
------------------------------------------------------------
perf_event_attr:
type 11 (armv8_pmuv3_1)
size 144
config 0x3f (stall_slot)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
inherit 1
------------------------------------------------------------
evsel=STALL_SLOT (supported=1) leader=BR_MIS_PRED
leader->core.fd=0xaaab19edd950
broken group leader for STALL_SLOT
Warning:
skipping event STALL_SLOT that kernel failed to open.
The sys_perf_event_open() syscall failed for event (STALL_SLOT): Success
"dmesg | grep -i perf" may provide additional information.
Opening: CPU_CYCLES
------------------------------------------------------------
perf_event_attr:
type 11 (armv8_pmuv3_1)
size 144
config 0x11 (cpu_cycles)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
inherit 1
------------------------------------------------------------
evsel=CPU_CYCLES (supported=1) leader=BR_MIS_PRED
leader->core.fd=0xaaab19edd950
broken group leader for CPU_CYCLES
Warning:
skipping event CPU_CYCLES that kernel failed to open.
The sys_perf_event_open() syscall failed for event (CPU_CYCLES): Success
"dmesg | grep -i perf" may provide additional information.
Opening: OP_RETIRED
------------------------------------------------------------
perf_event_attr:
type 11 (armv8_pmuv3_1)
size 144
config 0x3a (op_retired)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
inherit 1
------------------------------------------------------------
evsel=OP_RETIRED (supported=1) leader=BR_MIS_PRED
leader->core.fd=0xaaab19edd950
broken group leader for OP_RETIRED
Warning:
skipping event OP_RETIRED that kernel failed to open.
The sys_perf_event_open() syscall failed for event (OP_RETIRED): Success
"dmesg | grep -i perf" may provide additional information.
Opening: BR_MIS_PRED
------------------------------------------------------------
perf_event_attr:
type 11 (armv8_pmuv3_1)
size 144
config 0x10 (br_mis_pred)
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
inherit 1
------------------------------------------------------------
evsel=BR_MIS_PRED (supported=1) leader=STALL_SLOT_BACKEND
leader->core.fd=(nil)
perf: util/evsel.c:2156: get_group_fd: Assertion `!(!leader->core.fd)' failed.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] perf stat: Fix crash on arm64
2026-03-11 11:50 ` Leo Yan
@ 2026-03-23 14:21 ` Arnaldo Carvalho de Melo
2026-03-23 14:36 ` Arnaldo Carvalho de Melo
2026-03-23 17:06 ` Ian Rogers
1 sibling, 1 reply; 14+ messages in thread
From: Arnaldo Carvalho de Melo @ 2026-03-23 14:21 UTC (permalink / raw)
To: Leo Yan
Cc: Breno Leitao, Ian Rogers, Peter Zijlstra, Ingo Molnar,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Adrian Hunter, James Clark, linux-perf-users, linux-kernel,
kernel-team, Denis Yaroshevskiy
On Wed, Mar 11, 2026 at 11:50:11AM +0000, Leo Yan wrote:
> On Wed, Mar 11, 2026 at 03:21:00AM -0700, Breno Leitao wrote:
> > On Thu, Feb 05, 2026 at 10:22:27AM -0800, Ian Rogers wrote:
> > > I think it is a different issue, they have metrics while you don't.
> > > Your report does highlight we're missing a NO_JEVENTS=1 build-test,
> > > but the build is working for me. I'll send out two patches for these
> > > issues.
> > I wanted to follow up on this. Are there any next steps I should take?
> I can reproduce the issue on my Orion6 with setting a _dummy_ CPUID:
> $ export PERF_CPUID=0x00000000410fd490
> $ perf stat -C 5 -vvv
> ...
> Aborted
> perf: util/evsel.c:2156: get_group_fd: Assertion `!(!leader->core.fd)' failed
> Because we are working on different hardwares, I am a bit suspect I
> reproduced the issue with difference sequence as yours. Anyway, I do
> see that an event can be opened prior to its leader event, see the log
> below.
> Thus, your patch seems make sense to me as we need to ensure the leader
> event to be opened first. Ian, how about you think?
> I have a side question (maybe this is for James :): why other arch (or
> platforms) don't have reproduce the issue? I am wandering if any error
> or special stuffs in Arm CPU's json files.
I managed to reproduce using Leo's method:
(gdb) run stat -C 1
Starting program: /root/bin/perf stat -C 1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Failure to read '#slots'Failure to read '#slots'Failure to read '#slots'Failure to read '#slots'
Program received signal SIGSEGV, Segmentation fault.
metricgroup__copy_metric_events (evlist=0x555555c85790, cgrp=0x0, new_metric_events=0x555555c86250, old_metric_events=0x555555c86d60) at util/metricgroup.c:1662
1662 evsel = evlist__find_evsel(evlist, old_me->evsel->core.idx);
(gdb) bt
#0 metricgroup__copy_metric_events (evlist=0x555555c85790, cgrp=0x0, new_metric_events=0x555555c86250, old_metric_events=0x555555c86d60) at util/metricgroup.c:1662
#1 0x00005555555db404 in add_default_events () at builtin-stat.c:2102
#2 0x00005555555de000 in cmd_stat (argc=0, argv=0x7ffffffffc30) at builtin-stat.c:2830
#3 0x00005555556778e0 in run_builtin (p=0x555555bd5558 <commands+360>, argc=3, argv=0x7ffffffffc30) at perf.c:348
#4 0x0000555555677b5c in handle_internal_command (argc=3, argv=0x7ffffffffc30) at perf.c:398
#5 0x0000555555677d20 in run_argv (argcp=0x7ffffffffa4c, argv=0x7ffffffffa40) at perf.c:442
#6 0x0000555555678064 in main (argc=3, argv=0x7ffffffffc30) at perf.c:549
(gdb)
acme@raspberrypi:~/git/perf-tools $ git log --no-merges --oneline -15 tools/
493ad070cbcb (HEAD -> perf-tools, tag: perf-tools-fixes-for-v7.0-2-2026-03-23, x1/perf-tools, number/perf-tools) tools headers: Synchronize linux/build_bug.h with the kernel sources
0a8b2a0857ed tools headers UAPI: Sync x86's asm/kvm.h with the kernel sources
3c71ae8ec9ad tools headers UAPI: Sync linux/kvm.h with the kernel sources
4ddd7588fae6 tools arch x86: Sync the msr-index.h copy with the kernel sources
563d39928db6 perf kvm stat: Fix relative paths for including headers
72a8b9c060d3 perf parse-events: Fix big-endian 'overwrite' by writing correct union member
8dd1d9a33532 perf metricgroup: Fix metricgroup__has_metric_or_groups()
5d4c6c132ea9 selftests/hid: fix compilation when bpf_wq and hid_device are not exported
81f86728a980 tools headers: Skip arm64 cputype.h check
e2715ea5fb35 bootconfig: Add bootconfig tests about braces
d2ea4ff1ce50 KVM: selftests: Verify SEV+ guests can read and write EFER, CR0, CR4, and CR8
146c9ab38b48 powerpc/selftests/copyloops: extend selftest to exercise __copy_tofrom_user_power7_vmx
68e76fc12df0 selftests: rtnetlink: add neighbour update test
35b16a7a2c4f (tag: perf-tools-fixes-for-v7.0-1-2026-03-13, five/perf-tools) perf synthetic-events: Fix stale build ID in module MMAP2 records
3e745694b032 selftests: kvm: add a test that VMX validates controls on RSM
acme@raspberrypi:~/git/perf-tools $
Below goes the full -vvv log FWIW.
- Arnaldo
root@raspberrypi:~# export PERF_CPUID=0x00000000410fd490
root@raspberrypi:~# perf stat -C 1 -vvv
Using CPUID 0x00000000410fd490
metric expr 100 * (STALL_SLOT_BACKEND / (CPU_CYCLES * #slots) - BR_MIS_PRED * 3 / CPU_CYCLES) for backend_bound
parsing metric: 100 * (STALL_SLOT_BACKEND / (CPU_CYCLES * #slots) - BR_MIS_PRED * 3 / CPU_CYCLES)
Failure to read '#slots'literal: #slots = nan
syntax error
metric expr 100 * ((1 - OP_RETIRED / OP_SPEC) * (1 - (STALL_SLOT if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT - CPU_CYCLES) / (CPU_CYCLES * #slots)) + BR_MIS_PRED * 4 / CPU_CYCLES) for bad_speculation
parsing metric: 100 * ((1 - OP_RETIRED / OP_SPEC) * (1 - (STALL_SLOT if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT - CPU_CYCLES) / (CPU_CYCLES * #slots)) + BR_MIS_PRED * 4 / CPU_CYCLES)
Failure to read '#slots'literal: #slots = nan
syntax error
metric expr 100 * ((STALL_SLOT_FRONTEND if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT_FRONTEND - CPU_CYCLES) / (CPU_CYCLES * #slots) - BR_MIS_PRED / CPU_CYCLES) for frontend_bound
parsing metric: 100 * ((STALL_SLOT_FRONTEND if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT_FRONTEND - CPU_CYCLES) / (CPU_CYCLES * #slots) - BR_MIS_PRED / CPU_CYCLES)
Failure to read '#slots'literal: #slots = nan
syntax error
metric expr 100 * (OP_RETIRED / OP_SPEC * (1 - (STALL_SLOT if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT - CPU_CYCLES) / (CPU_CYCLES * #slots))) for retiring
parsing metric: 100 * (OP_RETIRED / OP_SPEC * (1 - (STALL_SLOT if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT - CPU_CYCLES) / (CPU_CYCLES * #slots)))
Failure to read '#slots'literal: #slots = nan
syntax error
metric expr (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) / (duration_time * 1e9) for CPUs_utilized
parsing metric: (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) / (duration_time * 1e9)
literal: #target_cpu = 1.000000
metric expr stalled\-cycles\-backend / cpu\-cycles for backend_cycles_idle
parsing metric: backend_cycles_idle
metric expr stalled\-cycles\-backend / cpu\-cycles for backend_cycles_idle
parsing metric: stalled\-cycles\-backend / cpu\-cycles
metric expr branches / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for branch_frequency
parsing metric: branches / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
literal: #target_cpu = 1.000000
metric expr branch\-misses / branches for branch_miss_rate
parsing metric: branch_miss_rate
metric expr branch\-misses / branches for branch_miss_rate
parsing metric: branch\-misses / branches
metric expr software@context\-switches\,name\=context\-switches@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for cs_per_second
parsing metric: software@context\-switches\,name\=context\-switches@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
literal: #target_cpu = 1.000000
metric expr cpu\-cycles / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for cycles_frequency
parsing metric: cpu\-cycles / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
literal: #target_cpu = 1.000000
metric expr stalled\-cycles\-frontend / cpu\-cycles for frontend_cycles_idle
parsing metric: frontend_cycles_idle
metric expr stalled\-cycles\-frontend / cpu\-cycles for frontend_cycles_idle
parsing metric: stalled\-cycles\-frontend / cpu\-cycles
metric expr instructions / cpu\-cycles for insn_per_cycle
parsing metric: insn_per_cycle
metric expr instructions / cpu\-cycles for insn_per_cycle
parsing metric: instructions / cpu\-cycles
metric expr software@cpu\-migrations\,name\=cpu\-migrations@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for migrations_per_second
parsing metric: software@cpu\-migrations\,name\=cpu\-migrations@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
literal: #target_cpu = 1.000000
metric expr software@page\-faults\,name\=page\-faults@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for page_faults_per_second
parsing metric: software@page\-faults\,name\=page\-faults@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
literal: #target_cpu = 1.000000
metric expr max(stalled\-cycles\-frontend, stalled\-cycles\-backend) / instructions for stalled_cycles_per_instruction
parsing metric: max(stalled\-cycles\-frontend, stalled\-cycles\-backend) / instructions
found event software@context-switches,name=context-switches@
found event duration_time
found event software@page-faults,name=page-faults@
found event cpu-cycles
found event branches
found event software@cpu-migrations,name=cpu-migrations@
found event software@cpu-clock,name=cpu-clock@
Parsing metric events 'software/context-switches,name=context-switches,metric-id=software!3context!1switches!0name!2context!1switches!3/,software/page-faults,name=page-faults,metric-id=software!3page!1faults!0name!2page!1faults!3/,cpu-cycles/metric-id=cpu!1cycles/,branches/metric-id=branches/,software/cpu-migrations,name=cpu-migrations,metric-id=software!3cpu!1migrations!0name!2cpu!1migrations!3/,software/cpu-clock,name=cpu-clock,metric-id=software!3cpu!1clock!0name!2cpu!1clock!3/,duration_time'
Attempt to add: software/context-switches,name=context-switches,metric-id=software!3context!1switches!0name!2context!1switches!3/
..after resolving event: software/config=0x3,name=context-switches,metric-id=software!3context!1switches!0name!2context!1switches!3/
Attempt to add: software/page-faults,name=page-faults,metric-id=software!3page!1faults!0name!2page!1faults!3/
..after resolving event: software/config=0x2,name=page-faults,metric-id=software!3page!1faults!0name!2page!1faults!3/
hwmon_pmu: not a hwmon file 'cpu-cycles'
Attempt to add: armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
..after resolving event: armv8_cortex_a76/metric-id=cpu!1cycles,legacy-hardware-config=0/
cpu-cycles -> armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'branches'
Attempt to add: armv8_cortex_a76/metric-id=branches,branches/
..after resolving event: armv8_cortex_a76/metric-id=branches,legacy-hardware-config=0x4/
branches -> armv8_cortex_a76/metric-id=branches,branches/
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
Attempt to add: software/cpu-migrations,name=cpu-migrations,metric-id=software!3cpu!1migrations!0name!2cpu!1migrations!3/
..after resolving event: software/config=0x4,name=cpu-migrations,metric-id=software!3cpu!1migrations!0name!2cpu!1migrations!3/
Attempt to add: software/cpu-clock,name=cpu-clock,metric-id=software!3cpu!1clock!0name!2cpu!1clock!3/
..after resolving event: software/config=0,name=cpu-clock,metric-id=software!3cpu!1clock!0name!2cpu!1clock!3/
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
found event duration_time
Parsing metric events 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id duration_time to duration_time
Events in 'frontend_bound' fully contained within 'retiring'
Events in 'bad_speculation' fully contained within 'retiring'
Events in 'backend_bound' fully contained within 'retiring'
found event duration_time
found event stalled-cycles-backend
found event instructions
found event stalled-cycles-frontend
Parsing metric events '{stalled-cycles-backend/metric-id=stalled!1cycles!1backend/,instructions/metric-id=instructions/,stalled-cycles-frontend/metric-id=stalled!1cycles!1frontend/}:W,duration_time'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
Attempt to add: armv8_cortex_a76/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
..after resolving event: armv8_cortex_a76/metric-id=stalled!1cycles!1backend,legacy-hardware-config=0x8/
stalled-cycles-backend -> armv8_cortex_a76/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'instructions'
Attempt to add: armv8_cortex_a76/metric-id=instructions,instructions/
..after resolving event: armv8_cortex_a76/metric-id=instructions,legacy-hardware-config=0x1/
instructions -> armv8_cortex_a76/metric-id=instructions,instructions/
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
Attempt to add: armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
..after resolving event: armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,legacy-hardware-config=0x7/
stalled-cycles-frontend -> armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id stalled-cycles-backend to stalled-cycles-backend
Matched metric-id instructions to instructions
Matched metric-id stalled-cycles-frontend to stalled-cycles-frontend
Matched metric-id duration_time to duration_time
Matched metric-id software@page-faults,name=page-faults@ to page-faults
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
Matched metric-id software@cpu-migrations,name=cpu-migrations@ to cpu-migrations
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
found event duration_time
found event cpu-cycles
found event instructions
Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,instructions/metric-id=instructions/}:W,duration_time'
hwmon_pmu: not a hwmon file 'cpu-cycles'
Attempt to add: armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
..after resolving event: armv8_cortex_a76/metric-id=cpu!1cycles,legacy-hardware-config=0/
cpu-cycles -> armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'instructions'
Attempt to add: armv8_cortex_a76/metric-id=instructions,instructions/
..after resolving event: armv8_cortex_a76/metric-id=instructions,legacy-hardware-config=0x1/
instructions -> armv8_cortex_a76/metric-id=instructions,instructions/
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id cpu-cycles to cpu-cycles
Matched metric-id instructions to instructions
Matched metric-id duration_time to duration_time
found event duration_time
found event cpu-cycles
found event stalled-cycles-frontend
Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,stalled-cycles-frontend/metric-id=stalled!1cycles!1frontend/}:W,duration_time'
hwmon_pmu: not a hwmon file 'cpu-cycles'
Attempt to add: armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
..after resolving event: armv8_cortex_a76/metric-id=cpu!1cycles,legacy-hardware-config=0/
cpu-cycles -> armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
Attempt to add: armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
..after resolving event: armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,legacy-hardware-config=0x7/
stalled-cycles-frontend -> armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id cpu-cycles to cpu-cycles
Matched metric-id stalled-cycles-frontend to stalled-cycles-frontend
Matched metric-id duration_time to duration_time
Matched metric-id cpu-cycles to cpu-cycles
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
Matched metric-id software@context-switches,name=context-switches@ to context-switches
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
found event duration_time
found event branch-misses
found event branches
Parsing metric events '{branch-misses/metric-id=branch!1misses/,branches/metric-id=branches/}:W,duration_time'
hwmon_pmu: not a hwmon file 'branch-misses'
Attempt to add: armv8_cortex_a76/metric-id=branch!1misses,branch-misses/
..after resolving event: armv8_cortex_a76/metric-id=branch!1misses,legacy-hardware-config=0x5/
branch-misses -> armv8_cortex_a76/metric-id=branch!1misses,branch-misses/
hwmon_pmu: not a hwmon file 'branch-misses'
hwmon_pmu: not a hwmon file 'branch-misses'
hwmon_pmu: not a hwmon file 'branch-misses'
hwmon_pmu: not a hwmon file 'branch-misses'
hwmon_pmu: not a hwmon file 'branch-misses'
hwmon_pmu: not a hwmon file 'branches'
Attempt to add: armv8_cortex_a76/metric-id=branches,branches/
..after resolving event: armv8_cortex_a76/metric-id=branches,legacy-hardware-config=0x4/
branches -> armv8_cortex_a76/metric-id=branches,branches/
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id branch-misses to branch-misses
Matched metric-id branches to branches
Matched metric-id duration_time to duration_time
Matched metric-id branches to branches
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
found event duration_time
found event cpu-cycles
found event stalled-cycles-backend
Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,stalled-cycles-backend/metric-id=stalled!1cycles!1backend/}:W,duration_time'
hwmon_pmu: not a hwmon file 'cpu-cycles'
Attempt to add: armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
..after resolving event: armv8_cortex_a76/metric-id=cpu!1cycles,legacy-hardware-config=0/
cpu-cycles -> armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
Attempt to add: armv8_cortex_a76/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
..after resolving event: armv8_cortex_a76/metric-id=stalled!1cycles!1backend,legacy-hardware-config=0x8/
stalled-cycles-backend -> armv8_cortex_a76/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id cpu-cycles to cpu-cycles
Matched metric-id stalled-cycles-backend to stalled-cycles-backend
Matched metric-id duration_time to duration_time
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
Matched metric-id duration_time to duration_time
Segmentation fault
root@raspberrypi:~#
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] perf stat: Fix crash on arm64
2026-03-23 14:21 ` Arnaldo Carvalho de Melo
@ 2026-03-23 14:36 ` Arnaldo Carvalho de Melo
2026-03-23 15:21 ` Leo Yan
0 siblings, 1 reply; 14+ messages in thread
From: Arnaldo Carvalho de Melo @ 2026-03-23 14:36 UTC (permalink / raw)
To: Leo Yan
Cc: Breno Leitao, Ian Rogers, Peter Zijlstra, Ingo Molnar,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Adrian Hunter, James Clark, linux-perf-users, linux-kernel,
kernel-team, Denis Yaroshevskiy
On Mon, Mar 23, 2026 at 11:21:48AM -0300, Arnaldo Carvalho de Melo wrote:
> On Wed, Mar 11, 2026 at 11:50:11AM +0000, Leo Yan wrote:
> > On Wed, Mar 11, 2026 at 03:21:00AM -0700, Breno Leitao wrote:
> > > On Thu, Feb 05, 2026 at 10:22:27AM -0800, Ian Rogers wrote:
> > > > I think it is a different issue, they have metrics while you don't.
> > > > Your report does highlight we're missing a NO_JEVENTS=1 build-test,
> > > > but the build is working for me. I'll send out two patches for these
> > > > issues.
>
> > > I wanted to follow up on this. Are there any next steps I should take?
>
> > I can reproduce the issue on my Orion6 with setting a _dummy_ CPUID:
>
> > $ export PERF_CPUID=0x00000000410fd490
> > $ perf stat -C 5 -vvv
> > ...
> > Aborted
> > perf: util/evsel.c:2156: get_group_fd: Assertion `!(!leader->core.fd)' failed
>
> > Because we are working on different hardwares, I am a bit suspect I
> > reproduced the issue with difference sequence as yours. Anyway, I do
> > see that an event can be opened prior to its leader event, see the log
> > below.
>
> > Thus, your patch seems make sense to me as we need to ensure the leader
> > event to be opened first. Ian, how about you think?
>
> > I have a side question (maybe this is for James :): why other arch (or
> > platforms) don't have reproduce the issue? I am wandering if any error
> > or special stuffs in Arm CPU's json files.
>
> I managed to reproduce using Leo's method:
But then I applied Breno's patch and got the same SEGV as below.
- Arnaldo
> (gdb) run stat -C 1
> Starting program: /root/bin/perf stat -C 1
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
> Failure to read '#slots'Failure to read '#slots'Failure to read '#slots'Failure to read '#slots'
> Program received signal SIGSEGV, Segmentation fault.
> metricgroup__copy_metric_events (evlist=0x555555c85790, cgrp=0x0, new_metric_events=0x555555c86250, old_metric_events=0x555555c86d60) at util/metricgroup.c:1662
> 1662 evsel = evlist__find_evsel(evlist, old_me->evsel->core.idx);
> (gdb) bt
> #0 metricgroup__copy_metric_events (evlist=0x555555c85790, cgrp=0x0, new_metric_events=0x555555c86250, old_metric_events=0x555555c86d60) at util/metricgroup.c:1662
> #1 0x00005555555db404 in add_default_events () at builtin-stat.c:2102
> #2 0x00005555555de000 in cmd_stat (argc=0, argv=0x7ffffffffc30) at builtin-stat.c:2830
> #3 0x00005555556778e0 in run_builtin (p=0x555555bd5558 <commands+360>, argc=3, argv=0x7ffffffffc30) at perf.c:348
> #4 0x0000555555677b5c in handle_internal_command (argc=3, argv=0x7ffffffffc30) at perf.c:398
> #5 0x0000555555677d20 in run_argv (argcp=0x7ffffffffa4c, argv=0x7ffffffffa40) at perf.c:442
> #6 0x0000555555678064 in main (argc=3, argv=0x7ffffffffc30) at perf.c:549
> (gdb)
>
> acme@raspberrypi:~/git/perf-tools $ git log --no-merges --oneline -15 tools/
> 493ad070cbcb (HEAD -> perf-tools, tag: perf-tools-fixes-for-v7.0-2-2026-03-23, x1/perf-tools, number/perf-tools) tools headers: Synchronize linux/build_bug.h with the kernel sources
> 0a8b2a0857ed tools headers UAPI: Sync x86's asm/kvm.h with the kernel sources
> 3c71ae8ec9ad tools headers UAPI: Sync linux/kvm.h with the kernel sources
> 4ddd7588fae6 tools arch x86: Sync the msr-index.h copy with the kernel sources
> 563d39928db6 perf kvm stat: Fix relative paths for including headers
> 72a8b9c060d3 perf parse-events: Fix big-endian 'overwrite' by writing correct union member
> 8dd1d9a33532 perf metricgroup: Fix metricgroup__has_metric_or_groups()
> 5d4c6c132ea9 selftests/hid: fix compilation when bpf_wq and hid_device are not exported
> 81f86728a980 tools headers: Skip arm64 cputype.h check
> e2715ea5fb35 bootconfig: Add bootconfig tests about braces
> d2ea4ff1ce50 KVM: selftests: Verify SEV+ guests can read and write EFER, CR0, CR4, and CR8
> 146c9ab38b48 powerpc/selftests/copyloops: extend selftest to exercise __copy_tofrom_user_power7_vmx
> 68e76fc12df0 selftests: rtnetlink: add neighbour update test
> 35b16a7a2c4f (tag: perf-tools-fixes-for-v7.0-1-2026-03-13, five/perf-tools) perf synthetic-events: Fix stale build ID in module MMAP2 records
> 3e745694b032 selftests: kvm: add a test that VMX validates controls on RSM
> acme@raspberrypi:~/git/perf-tools $
>
> Below goes the full -vvv log FWIW.
>
> - Arnaldo
>
> root@raspberrypi:~# export PERF_CPUID=0x00000000410fd490
> root@raspberrypi:~# perf stat -C 1 -vvv
> Using CPUID 0x00000000410fd490
> metric expr 100 * (STALL_SLOT_BACKEND / (CPU_CYCLES * #slots) - BR_MIS_PRED * 3 / CPU_CYCLES) for backend_bound
> parsing metric: 100 * (STALL_SLOT_BACKEND / (CPU_CYCLES * #slots) - BR_MIS_PRED * 3 / CPU_CYCLES)
> Failure to read '#slots'literal: #slots = nan
> syntax error
> metric expr 100 * ((1 - OP_RETIRED / OP_SPEC) * (1 - (STALL_SLOT if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT - CPU_CYCLES) / (CPU_CYCLES * #slots)) + BR_MIS_PRED * 4 / CPU_CYCLES) for bad_speculation
> parsing metric: 100 * ((1 - OP_RETIRED / OP_SPEC) * (1 - (STALL_SLOT if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT - CPU_CYCLES) / (CPU_CYCLES * #slots)) + BR_MIS_PRED * 4 / CPU_CYCLES)
> Failure to read '#slots'literal: #slots = nan
> syntax error
> metric expr 100 * ((STALL_SLOT_FRONTEND if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT_FRONTEND - CPU_CYCLES) / (CPU_CYCLES * #slots) - BR_MIS_PRED / CPU_CYCLES) for frontend_bound
> parsing metric: 100 * ((STALL_SLOT_FRONTEND if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT_FRONTEND - CPU_CYCLES) / (CPU_CYCLES * #slots) - BR_MIS_PRED / CPU_CYCLES)
> Failure to read '#slots'literal: #slots = nan
> syntax error
> metric expr 100 * (OP_RETIRED / OP_SPEC * (1 - (STALL_SLOT if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT - CPU_CYCLES) / (CPU_CYCLES * #slots))) for retiring
> parsing metric: 100 * (OP_RETIRED / OP_SPEC * (1 - (STALL_SLOT if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT - CPU_CYCLES) / (CPU_CYCLES * #slots)))
> Failure to read '#slots'literal: #slots = nan
> syntax error
> metric expr (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) / (duration_time * 1e9) for CPUs_utilized
> parsing metric: (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) / (duration_time * 1e9)
> literal: #target_cpu = 1.000000
> metric expr stalled\-cycles\-backend / cpu\-cycles for backend_cycles_idle
> parsing metric: backend_cycles_idle
> metric expr stalled\-cycles\-backend / cpu\-cycles for backend_cycles_idle
> parsing metric: stalled\-cycles\-backend / cpu\-cycles
> metric expr branches / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for branch_frequency
> parsing metric: branches / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
> literal: #target_cpu = 1.000000
> metric expr branch\-misses / branches for branch_miss_rate
> parsing metric: branch_miss_rate
> metric expr branch\-misses / branches for branch_miss_rate
> parsing metric: branch\-misses / branches
> metric expr software@context\-switches\,name\=context\-switches@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for cs_per_second
> parsing metric: software@context\-switches\,name\=context\-switches@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
> literal: #target_cpu = 1.000000
> metric expr cpu\-cycles / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for cycles_frequency
> parsing metric: cpu\-cycles / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
> literal: #target_cpu = 1.000000
> metric expr stalled\-cycles\-frontend / cpu\-cycles for frontend_cycles_idle
> parsing metric: frontend_cycles_idle
> metric expr stalled\-cycles\-frontend / cpu\-cycles for frontend_cycles_idle
> parsing metric: stalled\-cycles\-frontend / cpu\-cycles
> metric expr instructions / cpu\-cycles for insn_per_cycle
> parsing metric: insn_per_cycle
> metric expr instructions / cpu\-cycles for insn_per_cycle
> parsing metric: instructions / cpu\-cycles
> metric expr software@cpu\-migrations\,name\=cpu\-migrations@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for migrations_per_second
> parsing metric: software@cpu\-migrations\,name\=cpu\-migrations@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
> literal: #target_cpu = 1.000000
> metric expr software@page\-faults\,name\=page\-faults@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for page_faults_per_second
> parsing metric: software@page\-faults\,name\=page\-faults@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
> literal: #target_cpu = 1.000000
> metric expr max(stalled\-cycles\-frontend, stalled\-cycles\-backend) / instructions for stalled_cycles_per_instruction
> parsing metric: max(stalled\-cycles\-frontend, stalled\-cycles\-backend) / instructions
> found event software@context-switches,name=context-switches@
> found event duration_time
> found event software@page-faults,name=page-faults@
> found event cpu-cycles
> found event branches
> found event software@cpu-migrations,name=cpu-migrations@
> found event software@cpu-clock,name=cpu-clock@
> Parsing metric events 'software/context-switches,name=context-switches,metric-id=software!3context!1switches!0name!2context!1switches!3/,software/page-faults,name=page-faults,metric-id=software!3page!1faults!0name!2page!1faults!3/,cpu-cycles/metric-id=cpu!1cycles/,branches/metric-id=branches/,software/cpu-migrations,name=cpu-migrations,metric-id=software!3cpu!1migrations!0name!2cpu!1migrations!3/,software/cpu-clock,name=cpu-clock,metric-id=software!3cpu!1clock!0name!2cpu!1clock!3/,duration_time'
> Attempt to add: software/context-switches,name=context-switches,metric-id=software!3context!1switches!0name!2context!1switches!3/
> ..after resolving event: software/config=0x3,name=context-switches,metric-id=software!3context!1switches!0name!2context!1switches!3/
> Attempt to add: software/page-faults,name=page-faults,metric-id=software!3page!1faults!0name!2page!1faults!3/
> ..after resolving event: software/config=0x2,name=page-faults,metric-id=software!3page!1faults!0name!2page!1faults!3/
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> Attempt to add: armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
> ..after resolving event: armv8_cortex_a76/metric-id=cpu!1cycles,legacy-hardware-config=0/
> cpu-cycles -> armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'branches'
> Attempt to add: armv8_cortex_a76/metric-id=branches,branches/
> ..after resolving event: armv8_cortex_a76/metric-id=branches,legacy-hardware-config=0x4/
> branches -> armv8_cortex_a76/metric-id=branches,branches/
> hwmon_pmu: not a hwmon file 'branches'
> hwmon_pmu: not a hwmon file 'branches'
> hwmon_pmu: not a hwmon file 'branches'
> hwmon_pmu: not a hwmon file 'branches'
> hwmon_pmu: not a hwmon file 'branches'
> Attempt to add: software/cpu-migrations,name=cpu-migrations,metric-id=software!3cpu!1migrations!0name!2cpu!1migrations!3/
> ..after resolving event: software/config=0x4,name=cpu-migrations,metric-id=software!3cpu!1migrations!0name!2cpu!1migrations!3/
> Attempt to add: software/cpu-clock,name=cpu-clock,metric-id=software!3cpu!1clock!0name!2cpu!1clock!3/
> ..after resolving event: software/config=0,name=cpu-clock,metric-id=software!3cpu!1clock!0name!2cpu!1clock!3/
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> Attempt to add: tool/duration_time/
> ..after resolving event: tool/config=0x1/
> duration_time -> tool/duration_time/
> found event duration_time
> Parsing metric events 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> Attempt to add: tool/duration_time/
> ..after resolving event: tool/config=0x1/
> duration_time -> tool/duration_time/
> Matched metric-id duration_time to duration_time
> Events in 'frontend_bound' fully contained within 'retiring'
> Events in 'bad_speculation' fully contained within 'retiring'
> Events in 'backend_bound' fully contained within 'retiring'
> found event duration_time
> found event stalled-cycles-backend
> found event instructions
> found event stalled-cycles-frontend
> Parsing metric events '{stalled-cycles-backend/metric-id=stalled!1cycles!1backend/,instructions/metric-id=instructions/,stalled-cycles-frontend/metric-id=stalled!1cycles!1frontend/}:W,duration_time'
> hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
> Attempt to add: armv8_cortex_a76/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
> ..after resolving event: armv8_cortex_a76/metric-id=stalled!1cycles!1backend,legacy-hardware-config=0x8/
> stalled-cycles-backend -> armv8_cortex_a76/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
> hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
> hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
> hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
> hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
> hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
> hwmon_pmu: not a hwmon file 'instructions'
> Attempt to add: armv8_cortex_a76/metric-id=instructions,instructions/
> ..after resolving event: armv8_cortex_a76/metric-id=instructions,legacy-hardware-config=0x1/
> instructions -> armv8_cortex_a76/metric-id=instructions,instructions/
> hwmon_pmu: not a hwmon file 'instructions'
> hwmon_pmu: not a hwmon file 'instructions'
> hwmon_pmu: not a hwmon file 'instructions'
> hwmon_pmu: not a hwmon file 'instructions'
> hwmon_pmu: not a hwmon file 'instructions'
> hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
> Attempt to add: armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
> ..after resolving event: armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,legacy-hardware-config=0x7/
> stalled-cycles-frontend -> armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
> hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
> hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
> hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
> hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
> hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> Attempt to add: tool/duration_time/
> ..after resolving event: tool/config=0x1/
> duration_time -> tool/duration_time/
> Matched metric-id stalled-cycles-backend to stalled-cycles-backend
> Matched metric-id instructions to instructions
> Matched metric-id stalled-cycles-frontend to stalled-cycles-frontend
> Matched metric-id duration_time to duration_time
> Matched metric-id software@page-faults,name=page-faults@ to page-faults
> Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
> Matched metric-id software@cpu-migrations,name=cpu-migrations@ to cpu-migrations
> Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
> found event duration_time
> found event cpu-cycles
> found event instructions
> Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,instructions/metric-id=instructions/}:W,duration_time'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> Attempt to add: armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
> ..after resolving event: armv8_cortex_a76/metric-id=cpu!1cycles,legacy-hardware-config=0/
> cpu-cycles -> armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'instructions'
> Attempt to add: armv8_cortex_a76/metric-id=instructions,instructions/
> ..after resolving event: armv8_cortex_a76/metric-id=instructions,legacy-hardware-config=0x1/
> instructions -> armv8_cortex_a76/metric-id=instructions,instructions/
> hwmon_pmu: not a hwmon file 'instructions'
> hwmon_pmu: not a hwmon file 'instructions'
> hwmon_pmu: not a hwmon file 'instructions'
> hwmon_pmu: not a hwmon file 'instructions'
> hwmon_pmu: not a hwmon file 'instructions'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> Attempt to add: tool/duration_time/
> ..after resolving event: tool/config=0x1/
> duration_time -> tool/duration_time/
> Matched metric-id cpu-cycles to cpu-cycles
> Matched metric-id instructions to instructions
> Matched metric-id duration_time to duration_time
> found event duration_time
> found event cpu-cycles
> found event stalled-cycles-frontend
> Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,stalled-cycles-frontend/metric-id=stalled!1cycles!1frontend/}:W,duration_time'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> Attempt to add: armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
> ..after resolving event: armv8_cortex_a76/metric-id=cpu!1cycles,legacy-hardware-config=0/
> cpu-cycles -> armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
> Attempt to add: armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
> ..after resolving event: armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,legacy-hardware-config=0x7/
> stalled-cycles-frontend -> armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
> hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
> hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
> hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
> hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
> hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> Attempt to add: tool/duration_time/
> ..after resolving event: tool/config=0x1/
> duration_time -> tool/duration_time/
> Matched metric-id cpu-cycles to cpu-cycles
> Matched metric-id stalled-cycles-frontend to stalled-cycles-frontend
> Matched metric-id duration_time to duration_time
> Matched metric-id cpu-cycles to cpu-cycles
> Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
> Matched metric-id software@context-switches,name=context-switches@ to context-switches
> Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
> found event duration_time
> found event branch-misses
> found event branches
> Parsing metric events '{branch-misses/metric-id=branch!1misses/,branches/metric-id=branches/}:W,duration_time'
> hwmon_pmu: not a hwmon file 'branch-misses'
> Attempt to add: armv8_cortex_a76/metric-id=branch!1misses,branch-misses/
> ..after resolving event: armv8_cortex_a76/metric-id=branch!1misses,legacy-hardware-config=0x5/
> branch-misses -> armv8_cortex_a76/metric-id=branch!1misses,branch-misses/
> hwmon_pmu: not a hwmon file 'branch-misses'
> hwmon_pmu: not a hwmon file 'branch-misses'
> hwmon_pmu: not a hwmon file 'branch-misses'
> hwmon_pmu: not a hwmon file 'branch-misses'
> hwmon_pmu: not a hwmon file 'branch-misses'
> hwmon_pmu: not a hwmon file 'branches'
> Attempt to add: armv8_cortex_a76/metric-id=branches,branches/
> ..after resolving event: armv8_cortex_a76/metric-id=branches,legacy-hardware-config=0x4/
> branches -> armv8_cortex_a76/metric-id=branches,branches/
> hwmon_pmu: not a hwmon file 'branches'
> hwmon_pmu: not a hwmon file 'branches'
> hwmon_pmu: not a hwmon file 'branches'
> hwmon_pmu: not a hwmon file 'branches'
> hwmon_pmu: not a hwmon file 'branches'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> Attempt to add: tool/duration_time/
> ..after resolving event: tool/config=0x1/
> duration_time -> tool/duration_time/
> Matched metric-id branch-misses to branch-misses
> Matched metric-id branches to branches
> Matched metric-id duration_time to duration_time
> Matched metric-id branches to branches
> Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
> found event duration_time
> found event cpu-cycles
> found event stalled-cycles-backend
> Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,stalled-cycles-backend/metric-id=stalled!1cycles!1backend/}:W,duration_time'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> Attempt to add: armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
> ..after resolving event: armv8_cortex_a76/metric-id=cpu!1cycles,legacy-hardware-config=0/
> cpu-cycles -> armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'cpu-cycles'
> hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
> Attempt to add: armv8_cortex_a76/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
> ..after resolving event: armv8_cortex_a76/metric-id=stalled!1cycles!1backend,legacy-hardware-config=0x8/
> stalled-cycles-backend -> armv8_cortex_a76/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
> hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
> hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
> hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
> hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
> hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
> Attempt to add: tool/duration_time/
> ..after resolving event: tool/config=0x1/
> duration_time -> tool/duration_time/
> Matched metric-id cpu-cycles to cpu-cycles
> Matched metric-id stalled-cycles-backend to stalled-cycles-backend
> Matched metric-id duration_time to duration_time
> Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
> Matched metric-id duration_time to duration_time
> Segmentation fault
> root@raspberrypi:~#
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] perf stat: Fix crash on arm64
2026-03-23 14:36 ` Arnaldo Carvalho de Melo
@ 2026-03-23 15:21 ` Leo Yan
0 siblings, 0 replies; 14+ messages in thread
From: Leo Yan @ 2026-03-23 15:21 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: Breno Leitao, Ian Rogers, Peter Zijlstra, Ingo Molnar,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Adrian Hunter, James Clark, linux-perf-users, linux-kernel,
kernel-team, Denis Yaroshevskiy
Hi Arnaldo,
On Mon, Mar 23, 2026 at 11:36:22AM -0300, Arnaldo Carvalho de Melo wrote:
[...]
> > I managed to reproduce using Leo's method:
>
> But then I applied Breno's patch and got the same SEGV as below.
Are you sure ?
When my replied previously, I tried Breno and confirmed the Segmentation
fault can be dismissed. Do you want me to test again?
Thanks,
Leo
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] perf stat: Fix crash on arm64
2026-03-11 11:50 ` Leo Yan
2026-03-23 14:21 ` Arnaldo Carvalho de Melo
@ 2026-03-23 17:06 ` Ian Rogers
2026-03-24 11:00 ` Breno Leitao
1 sibling, 1 reply; 14+ messages in thread
From: Ian Rogers @ 2026-03-23 17:06 UTC (permalink / raw)
To: Leo Yan
Cc: Breno Leitao, acme, Peter Zijlstra, Ingo Molnar, Namhyung Kim,
Mark Rutland, Alexander Shishkin, Jiri Olsa, Adrian Hunter,
James Clark, linux-perf-users, linux-kernel, kernel-team,
Denis Yaroshevskiy
On Wed, Mar 11, 2026 at 4:50 AM Leo Yan <leo.yan@arm.com> wrote:
>
> Hi Breno,
>
> On Wed, Mar 11, 2026 at 03:21:00AM -0700, Breno Leitao wrote:
> > On Thu, Feb 05, 2026 at 10:22:27AM -0800, Ian Rogers wrote:
> > > I think it is a different issue, they have metrics while you don't.
> > > Your report does highlight we're missing a NO_JEVENTS=1 build-test,
> > > but the build is working for me. I'll send out two patches for these
> > > issues.
> >
> > Hi Ian, Leo, Acme
> >
> > I wanted to follow up on this. Are there any next steps I should take?
>
> Sorry for not tracking this issue.
>
> I can reproduce the issue on my Orion6 with setting a _dummy_ CPUID:
>
> $ export PERF_CPUID=0x00000000410fd490
> $ perf stat -C 5 -vvv
> ...
> Aborted
> perf: util/evsel.c:2156: get_group_fd: Assertion `!(!leader->core.fd)' failed
>
> Because we are working on different hardwares, I am a bit suspect I
> reproduced the issue with difference sequence as yours. Anyway, I do
> see that an event can be opened prior to its leader event, see the log
> below.
>
> Thus, your patch seems make sense to me as we need to ensure the leader
> event to be opened first. Ian, how about you think?
Because so many things depend on the event ordering, the patch makes
me very nervous, particularly how it will change architectural
requirement handling. Ordering the Intel slots and metric events is a
challenge. There is also how handling uncore events, which are
deliberately parsed out-of-order, will change. I've got a feeling the
test coverage for this isn't adequate, and finding the bugs requires
running on particular machines. It will also require examining the
default perf stat output to ensure this isn't broken; hopefully the
code is somewhat robust.
Ideally, the impact of the change on all these issues would be in the
commit message, but it's more realistic to cover each issue with
testing. I'll try to ask an AI buddy to help with this. Could you
rebase and send a v2? I'm curious to see what sashiko will throw up.
Thanks,
Ian
> I have a side question (maybe this is for James :): why other arch (or
> platforms) don't have reproduce the issue? I am wandering if any error
> or special stuffs in Arm CPU's json files.
>
> Thanks,
> Leo
>
>
> ---8<---
>
> Opening: BR_MIS_PRED
> ------------------------------------------------------------
> perf_event_attr:
> type 11 (armv8_pmuv3_1)
> size 144
> config 0x10 (br_mis_pred)
> sample_type IDENTIFIER
> read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
> disabled 1
> inherit 1
> ------------------------------------------------------------
> evsel=BR_MIS_PRED (supported=1) leader=BR_MIS_PRED
> sys_perf_event_open: pid -1 cpu 5 group_fd -1 flags 0x8
> sys_perf_event_open failed, error -2
> Warning:
> BR_MIS_PRED event is not supported by the kernel.
> The BR_MIS_PRED event is not supported.
> Opening: OP_SPEC
> ------------------------------------------------------------
> perf_event_attr:
> type 11 (armv8_pmuv3_1)
> size 144
> config 0x3b (op_spec)
> sample_type IDENTIFIER
> read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
> inherit 1
> ------------------------------------------------------------
> evsel=OP_SPEC (supported=1) leader=BR_MIS_PRED
> leader->core.fd=0xaaab19edd950
> broken group leader for OP_SPEC
> Warning:
> skipping event OP_SPEC that kernel failed to open.
> The sys_perf_event_open() syscall failed for event (OP_SPEC): Success
> "dmesg | grep -i perf" may provide additional information.
>
> Opening: STALL_SLOT
> ------------------------------------------------------------
> perf_event_attr:
> type 11 (armv8_pmuv3_1)
> size 144
> config 0x3f (stall_slot)
> sample_type IDENTIFIER
> read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
> inherit 1
> ------------------------------------------------------------
> evsel=STALL_SLOT (supported=1) leader=BR_MIS_PRED
> leader->core.fd=0xaaab19edd950
> broken group leader for STALL_SLOT
> Warning:
> skipping event STALL_SLOT that kernel failed to open.
> The sys_perf_event_open() syscall failed for event (STALL_SLOT): Success
> "dmesg | grep -i perf" may provide additional information.
>
> Opening: CPU_CYCLES
> ------------------------------------------------------------
> perf_event_attr:
> type 11 (armv8_pmuv3_1)
> size 144
> config 0x11 (cpu_cycles)
> sample_type IDENTIFIER
> read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
> inherit 1
> ------------------------------------------------------------
> evsel=CPU_CYCLES (supported=1) leader=BR_MIS_PRED
> leader->core.fd=0xaaab19edd950
> broken group leader for CPU_CYCLES
> Warning:
> skipping event CPU_CYCLES that kernel failed to open.
> The sys_perf_event_open() syscall failed for event (CPU_CYCLES): Success
> "dmesg | grep -i perf" may provide additional information.
>
> Opening: OP_RETIRED
> ------------------------------------------------------------
> perf_event_attr:
> type 11 (armv8_pmuv3_1)
> size 144
> config 0x3a (op_retired)
> sample_type IDENTIFIER
> read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
> inherit 1
> ------------------------------------------------------------
> evsel=OP_RETIRED (supported=1) leader=BR_MIS_PRED
> leader->core.fd=0xaaab19edd950
> broken group leader for OP_RETIRED
> Warning:
> skipping event OP_RETIRED that kernel failed to open.
> The sys_perf_event_open() syscall failed for event (OP_RETIRED): Success
> "dmesg | grep -i perf" may provide additional information.
>
> Opening: BR_MIS_PRED
> ------------------------------------------------------------
> perf_event_attr:
> type 11 (armv8_pmuv3_1)
> size 144
> config 0x10 (br_mis_pred)
> sample_type IDENTIFIER
> read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
> inherit 1
> ------------------------------------------------------------
> evsel=BR_MIS_PRED (supported=1) leader=STALL_SLOT_BACKEND
> leader->core.fd=(nil)
> perf: util/evsel.c:2156: get_group_fd: Assertion `!(!leader->core.fd)' failed.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] perf stat: Fix crash on arm64
2026-03-23 17:06 ` Ian Rogers
@ 2026-03-24 11:00 ` Breno Leitao
0 siblings, 0 replies; 14+ messages in thread
From: Breno Leitao @ 2026-03-24 11:00 UTC (permalink / raw)
To: Ian Rogers
Cc: Leo Yan, acme, Peter Zijlstra, Ingo Molnar, Namhyung Kim,
Mark Rutland, Alexander Shishkin, Jiri Olsa, Adrian Hunter,
James Clark, linux-perf-users, linux-kernel, kernel-team,
Denis Yaroshevskiy
On Mon, Mar 23, 2026 at 10:06:37AM -0700, Ian Rogers wrote:
> On Wed, Mar 11, 2026 at 4:50 AM Leo Yan <leo.yan@arm.com> wrote:
> >
> > Hi Breno,
> >
> > On Wed, Mar 11, 2026 at 03:21:00AM -0700, Breno Leitao wrote:
> > > On Thu, Feb 05, 2026 at 10:22:27AM -0800, Ian Rogers wrote:
> > > > I think it is a different issue, they have metrics while you don't.
> > > > Your report does highlight we're missing a NO_JEVENTS=1 build-test,
> > > > but the build is working for me. I'll send out two patches for these
> > > > issues.
> > >
> > > Hi Ian, Leo, Acme
> > >
> > > I wanted to follow up on this. Are there any next steps I should take?
> >
> > Sorry for not tracking this issue.
> >
> > I can reproduce the issue on my Orion6 with setting a _dummy_ CPUID:
> >
> > $ export PERF_CPUID=0x00000000410fd490
> > $ perf stat -C 5 -vvv
> > ...
> > Aborted
> > perf: util/evsel.c:2156: get_group_fd: Assertion `!(!leader->core.fd)' failed
> >
> > Because we are working on different hardwares, I am a bit suspect I
> > reproduced the issue with difference sequence as yours. Anyway, I do
> > see that an event can be opened prior to its leader event, see the log
> > below.
> >
> > Thus, your patch seems make sense to me as we need to ensure the leader
> > event to be opened first. Ian, how about you think?
>
> Because so many things depend on the event ordering, the patch makes
> me very nervous, particularly how it will change architectural
> requirement handling. Ordering the Intel slots and metric events is a
> challenge. There is also how handling uncore events, which are
> deliberately parsed out-of-order, will change. I've got a feeling the
> test coverage for this isn't adequate, and finding the bugs requires
> running on particular machines. It will also require examining the
> default perf stat output to ensure this isn't broken; hopefully the
> code is somewhat robust.
Sure thing. please let me know if there is any action on my side, and
I am happy to help.
> Ideally, the impact of the change on all these issues would be in the
> commit message, but it's more realistic to cover each issue with
> testing. I'll try to ask an AI buddy to help with this. Could you
> rebase and send a v2? I'm curious to see what sashiko will throw up.
Ack. I will resping it, so, we can check what sashiko thinks about it.
Thanks,
--breno
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2026-03-24 11:00 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-02-05 11:46 [PATCH] perf stat: Fix crash on arm64 Breno Leitao
2026-02-05 13:32 ` Dmitry Ilvokhin
2026-02-05 16:59 ` Ian Rogers
2026-02-05 17:39 ` Leo Yan
2026-02-05 17:52 ` Leo Yan
2026-02-05 18:22 ` Ian Rogers
2026-03-11 10:21 ` Breno Leitao
2026-03-11 11:50 ` Leo Yan
2026-03-23 14:21 ` Arnaldo Carvalho de Melo
2026-03-23 14:36 ` Arnaldo Carvalho de Melo
2026-03-23 15:21 ` Leo Yan
2026-03-23 17:06 ` Ian Rogers
2026-03-24 11:00 ` Breno Leitao
2026-02-06 12:01 ` Breno Leitao
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox