* [PATCH v2 1/3] perf test: Check test suite description properly
@ 2025-07-01 20:10 Namhyung Kim
2025-07-01 20:10 ` [PATCH v2 2/3] perf test: Add libsubcmd help tests Namhyung Kim
` (3 more replies)
0 siblings, 4 replies; 15+ messages in thread
From: Namhyung Kim @ 2025-07-01 20:10 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Ian Rogers, Kan Liang
Cc: Jiri Olsa, Adrian Hunter, Peter Zijlstra, Ingo Molnar, LKML,
linux-perf-users
Currently perf test checks the given string with descriptions for both
test suites and cases (subtests). But sometimes it's confusing since
the subtests don't contain the important keyword.
I think it's better to check the suite level and run the whole suite
together if it matches description in the suite.
Before:
$ perf test hwmon
(no output)
After:
$ perf test hwmon
10: Hwmon PMU :
10.1: Basic parsing test : Ok
10.2: Parsing without PMU name : Ok
10.3: Parsing with PMU name : Ok
And keep the existing behavior when it only matches test description only.
$ perf test "Equal cpu map"
39.5: Equal cpu map : Ok
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
tools/perf/tests/builtin-test.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index 80375ca39a37a256..846c9b3a732c9b3a 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -539,6 +539,7 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[],
for (struct test_suite **t = suites; *t; t++, curr_suite++) {
int curr_test_case;
+ bool suite_matched = false;
if (!perf_test__matches(test_description(*t, -1), curr_suite, argc, argv)) {
/*
@@ -556,6 +557,8 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[],
}
if (skip)
continue;
+ } else {
+ suite_matched = true;
}
if (intlist__find(skiplist, curr_suite + 1)) {
@@ -567,10 +570,10 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[],
for (unsigned int run = 0; run < runs_per_test; run++) {
test_suite__for_each_test_case(*t, curr_test_case) {
- if (!perf_test__matches(test_description(*t, curr_test_case),
+ if (!suite_matched &&
+ !perf_test__matches(test_description(*t, curr_test_case),
curr_suite, argc, argv))
continue;
-
err = start_test(*t, curr_suite, curr_test_case,
&child_tests[child_test_num++],
width, pass);
--
2.50.0.727.gbf7dc18ff4-goog
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH v2 2/3] perf test: Add libsubcmd help tests
2025-07-01 20:10 [PATCH v2 1/3] perf test: Check test suite description properly Namhyung Kim
@ 2025-07-01 20:10 ` Namhyung Kim
2025-07-01 20:10 ` [PATCH v2 3/3] perf tools: Fix use-after-free in help_unknown_cmd() Namhyung Kim
` (2 subsequent siblings)
3 siblings, 0 replies; 15+ messages in thread
From: Namhyung Kim @ 2025-07-01 20:10 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Ian Rogers, Kan Liang
Cc: Jiri Olsa, Adrian Hunter, Peter Zijlstra, Ingo Molnar, LKML,
linux-perf-users
Add a set of tests for subcmd routines. Currently it fails the last one
since there's a bug. It'll be fixed by the next commit.
$ perf test subcmd
69: libsubcmd help tests :
69.1: Load subcmd names : Ok
69.2: Uniquify subcmd names : Ok
69.3: Exclude duplicate subcmd names : FAILED!
Reviewed-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
tools/perf/tests/Build | 1 +
tools/perf/tests/builtin-test.c | 1 +
tools/perf/tests/subcmd-help.c | 108 ++++++++++++++++++++++++++++++++
tools/perf/tests/tests.h | 2 +
4 files changed, 112 insertions(+)
create mode 100644 tools/perf/tests/subcmd-help.c
diff --git a/tools/perf/tests/Build b/tools/perf/tests/Build
index d6c35dd0de3bd6ce..3e8394be15aea4fb 100644
--- a/tools/perf/tests/Build
+++ b/tools/perf/tests/Build
@@ -69,6 +69,7 @@ perf-test-y += symbols.o
perf-test-y += util.o
perf-test-y += hwmon_pmu.o
perf-test-y += tool_pmu.o
+perf-test-y += subcmd-help.o
ifeq ($(SRCARCH),$(filter $(SRCARCH),x86 arm arm64 powerpc))
perf-test-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index 846c9b3a732c9b3a..e242d56523cee171 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -139,6 +139,7 @@ static struct test_suite *generic_tests[] = {
&suite__event_groups,
&suite__symbols,
&suite__util,
+ &suite__subcmd_help,
NULL,
};
diff --git a/tools/perf/tests/subcmd-help.c b/tools/perf/tests/subcmd-help.c
new file mode 100644
index 0000000000000000..2280b4c0e5e7083b
--- /dev/null
+++ b/tools/perf/tests/subcmd-help.c
@@ -0,0 +1,108 @@
+// SPDX-License-Identifier: GPL-2.0
+#include "tests.h"
+#include <linux/compiler.h>
+#include <subcmd/help.h>
+
+static int test__load_cmdnames(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
+{
+ struct cmdnames cmds = {};
+
+ add_cmdname(&cmds, "aaa", 3);
+ add_cmdname(&cmds, "foo", 3);
+ add_cmdname(&cmds, "xyz", 3);
+
+ TEST_ASSERT_VAL("cannot find cmd", is_in_cmdlist(&cmds, "aaa") == 1);
+ TEST_ASSERT_VAL("wrong cmd", is_in_cmdlist(&cmds, "bar") == 0);
+ TEST_ASSERT_VAL("case sensitive", is_in_cmdlist(&cmds, "XYZ") == 0);
+
+ clean_cmdnames(&cmds);
+ return TEST_OK;
+}
+
+static int test__uniq_cmdnames(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
+{
+ struct cmdnames cmds = {};
+
+ /* uniq() assumes it's sorted */
+ add_cmdname(&cmds, "aaa", 3);
+ add_cmdname(&cmds, "aaa", 3);
+ add_cmdname(&cmds, "bbb", 3);
+
+ TEST_ASSERT_VAL("invalid original size", cmds.cnt == 3);
+ /* uniquify command names (to remove second 'aaa') */
+ uniq(&cmds);
+ TEST_ASSERT_VAL("invalid final size", cmds.cnt == 2);
+
+ TEST_ASSERT_VAL("cannot find cmd", is_in_cmdlist(&cmds, "aaa") == 1);
+ TEST_ASSERT_VAL("cannot find cmd", is_in_cmdlist(&cmds, "bbb") == 1);
+ TEST_ASSERT_VAL("wrong cmd", is_in_cmdlist(&cmds, "ccc") == 0);
+
+ clean_cmdnames(&cmds);
+ return TEST_OK;
+}
+
+static int test__exclude_cmdnames(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
+{
+ struct cmdnames cmds1 = {};
+ struct cmdnames cmds2 = {};
+
+ add_cmdname(&cmds1, "aaa", 3);
+ add_cmdname(&cmds1, "bbb", 3);
+ add_cmdname(&cmds1, "ccc", 3);
+ add_cmdname(&cmds1, "ddd", 3);
+ add_cmdname(&cmds1, "eee", 3);
+ add_cmdname(&cmds1, "fff", 3);
+ add_cmdname(&cmds1, "ggg", 3);
+ add_cmdname(&cmds1, "hhh", 3);
+ add_cmdname(&cmds1, "iii", 3);
+ add_cmdname(&cmds1, "jjj", 3);
+
+ add_cmdname(&cmds2, "bbb", 3);
+ add_cmdname(&cmds2, "eee", 3);
+ add_cmdname(&cmds2, "jjj", 3);
+
+ TEST_ASSERT_VAL("invalid original size", cmds1.cnt == 10);
+ TEST_ASSERT_VAL("invalid original size", cmds2.cnt == 3);
+
+ /* remove duplicate command names in cmds1 */
+ exclude_cmds(&cmds1, &cmds2);
+
+ TEST_ASSERT_VAL("invalid excluded size", cmds1.cnt == 7);
+ TEST_ASSERT_VAL("invalid excluded size", cmds2.cnt == 3);
+
+ /* excluded commands should not belong to cmds1 */
+ TEST_ASSERT_VAL("cannot find cmd", is_in_cmdlist(&cmds1, "aaa") == 1);
+ TEST_ASSERT_VAL("wrong cmd", is_in_cmdlist(&cmds1, "bbb") == 0);
+ TEST_ASSERT_VAL("cannot find cmd", is_in_cmdlist(&cmds1, "ccc") == 1);
+ TEST_ASSERT_VAL("cannot find cmd", is_in_cmdlist(&cmds1, "ddd") == 1);
+ TEST_ASSERT_VAL("wrong cmd", is_in_cmdlist(&cmds1, "eee") == 0);
+ TEST_ASSERT_VAL("cannot find cmd", is_in_cmdlist(&cmds1, "fff") == 1);
+ TEST_ASSERT_VAL("cannot find cmd", is_in_cmdlist(&cmds1, "ggg") == 1);
+ TEST_ASSERT_VAL("cannot find cmd", is_in_cmdlist(&cmds1, "hhh") == 1);
+ TEST_ASSERT_VAL("cannot find cmd", is_in_cmdlist(&cmds1, "iii") == 1);
+ TEST_ASSERT_VAL("wrong cmd", is_in_cmdlist(&cmds1, "jjj") == 0);
+
+ /* they should be only in cmds2 */
+ TEST_ASSERT_VAL("cannot find cmd", is_in_cmdlist(&cmds2, "bbb") == 1);
+ TEST_ASSERT_VAL("cannot find cmd", is_in_cmdlist(&cmds2, "eee") == 1);
+ TEST_ASSERT_VAL("cannot find cmd", is_in_cmdlist(&cmds2, "jjj") == 1);
+
+ clean_cmdnames(&cmds1);
+ clean_cmdnames(&cmds2);
+ return TEST_OK;
+}
+
+static struct test_case tests__subcmd_help[] = {
+ TEST_CASE("Load subcmd names", load_cmdnames),
+ TEST_CASE("Uniquify subcmd names", uniq_cmdnames),
+ TEST_CASE("Exclude duplicate subcmd names", exclude_cmdnames),
+ { .name = NULL, }
+};
+
+struct test_suite suite__subcmd_help = {
+ .desc = "libsubcmd help tests",
+ .test_cases = tests__subcmd_help,
+};
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index 4c128a9594413b32..97e62db8764a0537 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -3,6 +3,7 @@
#define TESTS_H
#include <stdbool.h>
+#include "util/debug.h"
enum {
TEST_OK = 0,
@@ -177,6 +178,7 @@ DECLARE_SUITE(sigtrap);
DECLARE_SUITE(event_groups);
DECLARE_SUITE(symbols);
DECLARE_SUITE(util);
+DECLARE_SUITE(subcmd_help);
/*
* PowerPC and S390 do not support creation of instruction breakpoints using the
--
2.50.0.727.gbf7dc18ff4-goog
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH v2 3/3] perf tools: Fix use-after-free in help_unknown_cmd()
2025-07-01 20:10 [PATCH v2 1/3] perf test: Check test suite description properly Namhyung Kim
2025-07-01 20:10 ` [PATCH v2 2/3] perf test: Add libsubcmd help tests Namhyung Kim
@ 2025-07-01 20:10 ` Namhyung Kim
2025-09-09 9:49 ` perf --help triggers an assertion Guilherme Amadio
2025-07-01 22:38 ` [PATCH v2 1/3] perf test: Check test suite description properly Ian Rogers
2025-07-02 16:43 ` Namhyung Kim
3 siblings, 1 reply; 15+ messages in thread
From: Namhyung Kim @ 2025-07-01 20:10 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Ian Rogers, Kan Liang
Cc: Jiri Olsa, Adrian Hunter, Peter Zijlstra, Ingo Molnar, LKML,
linux-perf-users
Currently perf aborts when it finds an invalid command. I guess it
depends on the environment as I have some custom commands in the path.
$ perf bad-command
perf: 'bad-command' is not a perf-command. See 'perf --help'.
Aborted (core dumped)
It's because the exclude_cmds() in libsubcmd has a use-after-free when
it removes some entries. After copying one to another entry, it keeps
the pointer in the both position. And the next copy operation will free
the later one but it's the same entry in the previous one.
For example, let's say cmds = { A, B, C, D, E } and excludes = { B, E }.
ci cj ei cmds-name excludes
-----------+--------------------
0 0 0 | A B : cmp < 0, ci == cj
1 1 0 | B B : cmp == 0
2 1 1 | C E : cmp < 0, ci != cj
At this point, it frees cmds->names[1] and cmds->names[1] is assigned to
cmds->names[2].
3 2 1 | D E : cmp < 0, ci != cj
Now it frees cmds->names[2] but it's the same as cmds->names[1]. So
accessing cmds->names[1] will be invalid.
This makes the subcmd tests succeed.
$ perf test subcmd
69: libsubcmd help tests :
69.1: Load subcmd names : Ok
69.2: Uniquify subcmd names : Ok
69.3: Exclude duplicate subcmd names : Ok
Fixes: 657a3efee43a ("libsubcmd: Avoid SEGV/use-after-free when commands aren't excluded")
Reviewed-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
tools/lib/subcmd/help.c | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/tools/lib/subcmd/help.c b/tools/lib/subcmd/help.c
index 8561b0f01a247690..9ef569492560efd7 100644
--- a/tools/lib/subcmd/help.c
+++ b/tools/lib/subcmd/help.c
@@ -9,6 +9,7 @@
#include <sys/stat.h>
#include <unistd.h>
#include <dirent.h>
+#include <assert.h>
#include "subcmd-util.h"
#include "help.h"
#include "exec-cmd.h"
@@ -82,10 +83,11 @@ void exclude_cmds(struct cmdnames *cmds, struct cmdnames *excludes)
ci++;
cj++;
} else {
- zfree(&cmds->names[cj]);
- cmds->names[cj++] = cmds->names[ci++];
+ cmds->names[cj++] = cmds->names[ci];
+ cmds->names[ci++] = NULL;
}
} else if (cmp == 0) {
+ zfree(&cmds->names[ci]);
ci++;
ei++;
} else if (cmp > 0) {
@@ -94,12 +96,12 @@ void exclude_cmds(struct cmdnames *cmds, struct cmdnames *excludes)
}
if (ci != cj) {
while (ci < cmds->cnt) {
- zfree(&cmds->names[cj]);
- cmds->names[cj++] = cmds->names[ci++];
+ cmds->names[cj++] = cmds->names[ci];
+ cmds->names[ci++] = NULL;
}
}
for (ci = cj; ci < cmds->cnt; ci++)
- zfree(&cmds->names[ci]);
+ assert(cmds->names[ci] == NULL);
cmds->cnt = cj;
}
--
2.50.0.727.gbf7dc18ff4-goog
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v2 1/3] perf test: Check test suite description properly
2025-07-01 20:10 [PATCH v2 1/3] perf test: Check test suite description properly Namhyung Kim
2025-07-01 20:10 ` [PATCH v2 2/3] perf test: Add libsubcmd help tests Namhyung Kim
2025-07-01 20:10 ` [PATCH v2 3/3] perf tools: Fix use-after-free in help_unknown_cmd() Namhyung Kim
@ 2025-07-01 22:38 ` Ian Rogers
2025-07-02 16:43 ` Namhyung Kim
3 siblings, 0 replies; 15+ messages in thread
From: Ian Rogers @ 2025-07-01 22:38 UTC (permalink / raw)
To: Namhyung Kim
Cc: Arnaldo Carvalho de Melo, Kan Liang, Jiri Olsa, Adrian Hunter,
Peter Zijlstra, Ingo Molnar, LKML, linux-perf-users
On Tue, Jul 1, 2025 at 1:10 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> Currently perf test checks the given string with descriptions for both
> test suites and cases (subtests). But sometimes it's confusing since
> the subtests don't contain the important keyword.
>
> I think it's better to check the suite level and run the whole suite
> together if it matches description in the suite.
>
> Before:
> $ perf test hwmon
> (no output)
>
> After:
> $ perf test hwmon
> 10: Hwmon PMU :
> 10.1: Basic parsing test : Ok
> 10.2: Parsing without PMU name : Ok
> 10.3: Parsing with PMU name : Ok
>
> And keep the existing behavior when it only matches test description only.
>
> $ perf test "Equal cpu map"
> 39.5: Equal cpu map : Ok
>
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Reviewed-by: Ian Rogers <irogers@google.com>
Thanks,
Ian
> ---
> tools/perf/tests/builtin-test.c | 7 +++++--
> 1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
> index 80375ca39a37a256..846c9b3a732c9b3a 100644
> --- a/tools/perf/tests/builtin-test.c
> +++ b/tools/perf/tests/builtin-test.c
> @@ -539,6 +539,7 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[],
>
> for (struct test_suite **t = suites; *t; t++, curr_suite++) {
> int curr_test_case;
> + bool suite_matched = false;
>
> if (!perf_test__matches(test_description(*t, -1), curr_suite, argc, argv)) {
> /*
> @@ -556,6 +557,8 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[],
> }
> if (skip)
> continue;
> + } else {
> + suite_matched = true;
> }
>
> if (intlist__find(skiplist, curr_suite + 1)) {
> @@ -567,10 +570,10 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[],
>
> for (unsigned int run = 0; run < runs_per_test; run++) {
> test_suite__for_each_test_case(*t, curr_test_case) {
> - if (!perf_test__matches(test_description(*t, curr_test_case),
> + if (!suite_matched &&
> + !perf_test__matches(test_description(*t, curr_test_case),
> curr_suite, argc, argv))
> continue;
> -
> err = start_test(*t, curr_suite, curr_test_case,
> &child_tests[child_test_num++],
> width, pass);
> --
> 2.50.0.727.gbf7dc18ff4-goog
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 1/3] perf test: Check test suite description properly
2025-07-01 20:10 [PATCH v2 1/3] perf test: Check test suite description properly Namhyung Kim
` (2 preceding siblings ...)
2025-07-01 22:38 ` [PATCH v2 1/3] perf test: Check test suite description properly Ian Rogers
@ 2025-07-02 16:43 ` Namhyung Kim
3 siblings, 0 replies; 15+ messages in thread
From: Namhyung Kim @ 2025-07-02 16:43 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Ian Rogers, Kan Liang, Namhyung Kim
Cc: Jiri Olsa, Adrian Hunter, Peter Zijlstra, Ingo Molnar, LKML,
linux-perf-users
On Tue, 01 Jul 2025 13:10:25 -0700, Namhyung Kim wrote:
> Currently perf test checks the given string with descriptions for both
> test suites and cases (subtests). But sometimes it's confusing since
> the subtests don't contain the important keyword.
>
> I think it's better to check the suite level and run the whole suite
> together if it matches description in the suite.
>
> [...]
Applied to perf-tools-next, thanks!
Best regards,
Namhyung
^ permalink raw reply [flat|nested] 15+ messages in thread
* perf --help triggers an assertion
2025-07-01 20:10 ` [PATCH v2 3/3] perf tools: Fix use-after-free in help_unknown_cmd() Namhyung Kim
@ 2025-09-09 9:49 ` Guilherme Amadio
2025-09-09 18:31 ` Ian Rogers
0 siblings, 1 reply; 15+ messages in thread
From: Guilherme Amadio @ 2025-09-09 9:49 UTC (permalink / raw)
To: namhyung
Cc: acme, adrian.hunter, irogers, jolsa, kan.liang, linux-kernel,
linux-perf-users, mingo, peterz
Hi Namhyung,
I was updating perf's package in Gentoo Linux and noticed some problems
which were not there before. I tested with the version below and the problem
still seems to be there. perf --help triggers an assertion (see below).
Looking in the list, it seems related to the patch below:
https://lore.kernel.org/linux-perf-users/20250701201027.1171561-3-namhyung@kernel.org/
Cheers,
-Guilherme
The problem:
gentoo perf $ ./perf --help
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) ./perf --help
Some extra information:
gentoo perf $ ./perf version
perf version 6.17.rc5.gf777d1112ee5
gentoo perf $ ./perf test subcmd
69: libsubcmd help tests :
69.1: Load subcmd names : Ok
69.2: Uniquify subcmd names : Ok
69.3: Exclude duplicate subcmd names : Ok
gentoo perf $ ./perf bad-command
perf: 'bad-command' is not a perf-command. See 'perf --help'.
gentoo perf $ ./perf --help
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) ./perf --help
gentoo perf $ gdb run --args ./perf --help
GNU gdb (Gentoo 16.3 vanilla) 16.3
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./perf...
(gdb) run
Starting program: /home/amadio/src/linux/tools/perf/perf --help
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1 0x00007ffff74c1656 in __pthread_kill_internal (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:89
#2 0x00007ffff74c166d in __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:100
#3 0x00007ffff747509c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x00007ffff747637e in __GI_abort () at abort.c:77
#5 0x00007ffff746e023 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x555555974d03 "cmds->names[ci] == NULL",
file=file@entry=0x555555974cfc "help.c", line=line@entry=104, function=function@entry=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds")
at assert.c:118
#6 0x00007ffff746e075 in __assert_fail (assertion=0x555555974d03 "cmds->names[ci] == NULL", file=0x555555974cfc "help.c", line=104,
function=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds") at assert.c:127
#7 0x0000555555693813 in exclude_cmds (cmds=0x55555614e5e0 <other_cmds>, excludes=0x55555614e5c0 <main_cmds>) at help.c:104
#8 0x0000555555693eb3 in load_command_list (prefix=0x555555954071 "perf-", main_cmds=0x55555614e5c0 <main_cmds>,
other_cmds=0x55555614e5e0 <other_cmds>) at help.c:252
#9 0x00005555555e9987 in cmd_help (argc=1, argv=0x7fffffffd1a0) at builtin-help.c:458
#10 0x0000555555685d45 in run_builtin (p=0x555556130de0 <commands+192>, argc=1, argv=0x7fffffffd1a0) at perf.c:349
#11 0x0000555555685fe1 in handle_internal_command (argc=1, argv=0x7fffffffd1a0) at perf.c:401
#12 0x0000555555686142 in run_argv (argcp=0x7fffffffcfac, argv=0x7fffffffcfa0) at perf.c:445
#13 0x0000555555686493 in main (argc=1, argv=0x7fffffffd1a0) at perf.c:553
(gdb) quit
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: perf --help triggers an assertion
2025-09-09 9:49 ` perf --help triggers an assertion Guilherme Amadio
@ 2025-09-09 18:31 ` Ian Rogers
2025-09-10 12:52 ` Guilherme Amadio
0 siblings, 1 reply; 15+ messages in thread
From: Ian Rogers @ 2025-09-09 18:31 UTC (permalink / raw)
To: Guilherme Amadio
Cc: namhyung, acme, adrian.hunter, jolsa, kan.liang, linux-kernel,
linux-perf-users, mingo, peterz
On Tue, Sep 9, 2025 at 2:49 AM Guilherme Amadio <amadio@gentoo.org> wrote:
>
> Hi Namhyung,
>
> I was updating perf's package in Gentoo Linux and noticed some problems
> which were not there before. I tested with the version below and the problem
> still seems to be there. perf --help triggers an assertion (see below).
> Looking in the list, it seems related to the patch below:
>
> https://lore.kernel.org/linux-perf-users/20250701201027.1171561-3-namhyung@kernel.org/
>
> Cheers,
> -Guilherme
>
> The problem:
>
> gentoo perf $ ./perf --help
> perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> Aborted (core dumped) ./perf --help
>
> Some extra information:
>
> gentoo perf $ ./perf version
> perf version 6.17.rc5.gf777d1112ee5
> gentoo perf $ ./perf test subcmd
> 69: libsubcmd help tests :
> 69.1: Load subcmd names : Ok
> 69.2: Uniquify subcmd names : Ok
> 69.3: Exclude duplicate subcmd names : Ok
> gentoo perf $ ./perf bad-command
> perf: 'bad-command' is not a perf-command. See 'perf --help'.
> gentoo perf $ ./perf --help
> perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> Aborted (core dumped) ./perf --help
> gentoo perf $ gdb run --args ./perf --help
> GNU gdb (Gentoo 16.3 vanilla) 16.3
> Copyright (C) 2024 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "x86_64-pc-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <https://bugs.gentoo.org/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
>
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from ./perf...
> (gdb) run
> Starting program: /home/amadio/src/linux/tools/perf/perf --help
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/usr/lib64/libthread_db.so.1".
> perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
>
> Program received signal SIGABRT, Aborted.
> __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> 44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
> (gdb) bt
> #0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> #1 0x00007ffff74c1656 in __pthread_kill_internal (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:89
> #2 0x00007ffff74c166d in __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:100
> #3 0x00007ffff747509c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> #4 0x00007ffff747637e in __GI_abort () at abort.c:77
> #5 0x00007ffff746e023 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x555555974d03 "cmds->names[ci] == NULL",
> file=file@entry=0x555555974cfc "help.c", line=line@entry=104, function=function@entry=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds")
> at assert.c:118
> #6 0x00007ffff746e075 in __assert_fail (assertion=0x555555974d03 "cmds->names[ci] == NULL", file=0x555555974cfc "help.c", line=104,
> function=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds") at assert.c:127
> #7 0x0000555555693813 in exclude_cmds (cmds=0x55555614e5e0 <other_cmds>, excludes=0x55555614e5c0 <main_cmds>) at help.c:104
> #8 0x0000555555693eb3 in load_command_list (prefix=0x555555954071 "perf-", main_cmds=0x55555614e5c0 <main_cmds>,
> other_cmds=0x55555614e5e0 <other_cmds>) at help.c:252
> #9 0x00005555555e9987 in cmd_help (argc=1, argv=0x7fffffffd1a0) at builtin-help.c:458
> #10 0x0000555555685d45 in run_builtin (p=0x555556130de0 <commands+192>, argc=1, argv=0x7fffffffd1a0) at perf.c:349
> #11 0x0000555555685fe1 in handle_internal_command (argc=1, argv=0x7fffffffd1a0) at perf.c:401
> #12 0x0000555555686142 in run_argv (argcp=0x7fffffffcfac, argv=0x7fffffffcfa0) at perf.c:445
> #13 0x0000555555686493 in main (argc=1, argv=0x7fffffffd1a0) at perf.c:553
> (gdb) quit
Thanks Guilherme,
I tried to reproduce the same version with various options: DEBUG=1
-UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various
directories with "perf-" prefixed files. I'm afraid I wasn't able to
reproduce. The assert is trying to avoid a memory leak, so
non-critical, and I couldn't in a quick inspection eye-ball an issue.
Without getting a reproduction I don't think I can make progress with
the issue.
Thanks,
Ian
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: perf --help triggers an assertion
2025-09-09 18:31 ` Ian Rogers
@ 2025-09-10 12:52 ` Guilherme Amadio
2025-09-10 15:04 ` Ian Rogers
2025-09-11 20:02 ` Arnaldo Carvalho de Melo
0 siblings, 2 replies; 15+ messages in thread
From: Guilherme Amadio @ 2025-09-10 12:52 UTC (permalink / raw)
To: Ian Rogers
Cc: namhyung, acme, adrian.hunter, jolsa, kan.liang, linux-kernel,
linux-perf-users, mingo, peterz
On Tue, Sep 09, 2025 at 11:31:51AM -0700, Ian Rogers wrote:
> On Tue, Sep 9, 2025 at 2:49 AM Guilherme Amadio <amadio@gentoo.org> wrote:
> >
> > Hi Namhyung,
> >
> > I was updating perf's package in Gentoo Linux and noticed some problems
> > which were not there before. I tested with the version below and the problem
> > still seems to be there. perf --help triggers an assertion (see below).
> > Looking in the list, it seems related to the patch below:
> >
> > https://lore.kernel.org/linux-perf-users/20250701201027.1171561-3-namhyung@kernel.org/
> >
> > Cheers,
> > -Guilherme
> >
> > The problem:
> >
> > gentoo perf $ ./perf --help
> > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > Aborted (core dumped) ./perf --help
> >
> > Some extra information:
> >
> > gentoo perf $ ./perf version
> > perf version 6.17.rc5.gf777d1112ee5
> > gentoo perf $ ./perf test subcmd
> > 69: libsubcmd help tests :
> > 69.1: Load subcmd names : Ok
> > 69.2: Uniquify subcmd names : Ok
> > 69.3: Exclude duplicate subcmd names : Ok
> > gentoo perf $ ./perf bad-command
> > perf: 'bad-command' is not a perf-command. See 'perf --help'.
> > gentoo perf $ ./perf --help
> > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > Aborted (core dumped) ./perf --help
> > gentoo perf $ gdb run --args ./perf --help
> > GNU gdb (Gentoo 16.3 vanilla) 16.3
> > Copyright (C) 2024 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law.
> > Type "show copying" and "show warranty" for details.
> > This GDB was configured as "x86_64-pc-linux-gnu".
> > Type "show configuration" for configuration details.
> > For bug reporting instructions, please see:
> > <https://bugs.gentoo.org/>.
> > Find the GDB manual and other documentation resources online at:
> > <http://www.gnu.org/software/gdb/documentation/>.
> >
> > For help, type "help".
> > Type "apropos word" to search for commands related to "word"...
> > Reading symbols from ./perf...
> > (gdb) run
> > Starting program: /home/amadio/src/linux/tools/perf/perf --help
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/usr/lib64/libthread_db.so.1".
> > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> >
> > Program received signal SIGABRT, Aborted.
> > __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > 44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
> > (gdb) bt
> > #0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > #1 0x00007ffff74c1656 in __pthread_kill_internal (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:89
> > #2 0x00007ffff74c166d in __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:100
> > #3 0x00007ffff747509c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> > #4 0x00007ffff747637e in __GI_abort () at abort.c:77
> > #5 0x00007ffff746e023 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x555555974d03 "cmds->names[ci] == NULL",
> > file=file@entry=0x555555974cfc "help.c", line=line@entry=104, function=function@entry=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds")
> > at assert.c:118
> > #6 0x00007ffff746e075 in __assert_fail (assertion=0x555555974d03 "cmds->names[ci] == NULL", file=0x555555974cfc "help.c", line=104,
> > function=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds") at assert.c:127
> > #7 0x0000555555693813 in exclude_cmds (cmds=0x55555614e5e0 <other_cmds>, excludes=0x55555614e5c0 <main_cmds>) at help.c:104
> > #8 0x0000555555693eb3 in load_command_list (prefix=0x555555954071 "perf-", main_cmds=0x55555614e5c0 <main_cmds>,
> > other_cmds=0x55555614e5e0 <other_cmds>) at help.c:252
> > #9 0x00005555555e9987 in cmd_help (argc=1, argv=0x7fffffffd1a0) at builtin-help.c:458
> > #10 0x0000555555685d45 in run_builtin (p=0x555556130de0 <commands+192>, argc=1, argv=0x7fffffffd1a0) at perf.c:349
> > #11 0x0000555555685fe1 in handle_internal_command (argc=1, argv=0x7fffffffd1a0) at perf.c:401
> > #12 0x0000555555686142 in run_argv (argcp=0x7fffffffcfac, argv=0x7fffffffcfa0) at perf.c:445
> > #13 0x0000555555686493 in main (argc=1, argv=0x7fffffffd1a0) at perf.c:553
> > (gdb) quit
>
> Thanks Guilherme,
>
> I tried to reproduce the same version with various options: DEBUG=1
> -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various
> directories with "perf-" prefixed files. I'm afraid I wasn't able to
> reproduce. The assert is trying to avoid a memory leak, so
> non-critical, and I couldn't in a quick inspection eye-ball an issue.
> Without getting a reproduction I don't think I can make progress with
> the issue.
I do not have any special setup on my machine (if you consider Gentoo
not special, of course). I just did a git bisect and arrived at commit
9401d1771dad99bfc795dd2ae0c292343fd1f78d, which is the commit I linked
above. I used
$ make -C tools/perf clean && make -B -j16 -C tools/perf DEBUG=1 WERROR=0 NO_SHELLCHECK=1
then called perf --help each time (but perf help record or equivalent
also triggers the assertion). I'm using GCC 14 on a 3950X. The features
I have enabled and installed dependencies are below. This patch has been
backported to at least 6.12 stable, and I can reproduce this with
v6.12.45.
gentoo linux $ tools/perf/perf version --build-options
perf version 6.16.g45effee4b85a
aio: [ on ] # HAVE_AIO_SUPPORT
bpf: [ on ] # HAVE_LIBBPF_SUPPORT
bpf_skeletons: [ on ] # HAVE_BPF_SKEL
debuginfod: [ on ] # HAVE_DEBUGINFOD_SUPPORT
dwarf: [ on ] # HAVE_LIBDW_SUPPORT
dwarf_getlocations: [ on ] # HAVE_LIBDW_SUPPORT
dwarf-unwind: [ on ] # HAVE_DWARF_UNWIND_SUPPORT
auxtrace: [ on ] # HAVE_AUXTRACE_SUPPORT
libbfd: [ OFF ] # HAVE_LIBBFD_SUPPORT ( tip: Deprecated, license incompatibility, use BUILD_NONDISTRO=1 and install binutils-dev[el] )
libcapstone: [ on ] # HAVE_LIBCAPSTONE_SUPPORT
libcrypto: [ on ] # HAVE_LIBCRYPTO_SUPPORT
libdw-dwarf-unwind: [ on ] # HAVE_LIBDW_SUPPORT
libelf: [ on ] # HAVE_LIBELF_SUPPORT
libnuma: [ on ] # HAVE_LIBNUMA_SUPPORT
libopencsd: [ OFF ] # HAVE_CSTRACE_SUPPORT
libperl: [ on ] # HAVE_LIBPERL_SUPPORT
libpfm4: [ on ] # HAVE_LIBPFM
libpython: [ on ] # HAVE_LIBPYTHON_SUPPORT
libslang: [ on ] # HAVE_SLANG_SUPPORT
libtraceevent: [ on ] # HAVE_LIBTRACEEVENT
libunwind: [ OFF ] # HAVE_LIBUNWIND_SUPPORT ( tip: Deprecated, use LIBUNWIND=1 and install libunwind-dev[el] to build with it )
lzma: [ on ] # HAVE_LZMA_SUPPORT
numa_num_possible_cpus: [ on ] # HAVE_LIBNUMA_SUPPORT
zlib: [ on ] # HAVE_ZLIB_SUPPORT
zstd: [ on ] # HAVE_ZSTD_SUPPORT
And here are the version of all dependencies I have installed:
* dependency graph for dev-util/perf-6.16-r1
`-- dev-util/perf-6.16-r1
`-- dev-util/babeltrace-1.5.11
`-- dev-libs/libbpf-1.6.2
`-- dev-util/bpftool-7.6.0
`-- dev-util/pahole-1.30
`-- llvm-core/clang-18.1.8-r6
`-- llvm-core/llvm-18.1.8-r6
`-- llvm-core/clang-19.1.7
`-- llvm-core/llvm-19.1.7
`-- llvm-core/clang-20.1.8
`-- llvm-core/llvm-20.1.8
`-- llvm-core/clang-21.1.0
`-- llvm-core/llvm-21.1.0
`-- sys-libs/libcap-2.76
`-- dev-libs/capstone-5.0.6
`-- dev-libs/openssl-3.5.2
`-- x11-libs/gtk+-2.24.33-r3
`-- virtual/jre-21
`-- dev-libs/libpfm-4.13.0
`-- dev-libs/libtraceevent-1.8.4
`-- dev-libs/libtracefs-1.8.2
`-- app-arch/xz-utils-5.8.1-r1
`-- sys-process/numactl-2.0.19
`-- dev-lang/perl-5.42.0
`-- dev-lang/python-0.3.13.7
`-- dev-lang/python-0.3.14.0_rc2
`-- dev-lang/python-3.11.13_p1
`-- dev-lang/python-3.12.11_p1
`-- dev-lang/python-3.13.9999
`-- dev-lang/python-3.14.0_rc2
`-- sys-libs/slang-2.3.3-r1
`-- dev-debug/systemtap-5.3
`-- dev-util/google-perftools-2.17.2
`-- sys-libs/libunwind-1.8.2-r1
`-- app-arch/zstd-1.5.7-r1
`-- dev-libs/elfutils-0.193
`-- sys-libs/binutils-libs-2.45-r1
`-- sys-libs/zlib-1.3.1-r1
`-- virtual/libcrypt-2-r1
`-- sys-kernel/linux-headers-6.16
`-- virtual/jdk-21
`-- app-arch/tar-1.35
`-- dev-python/setuptools-80.9.0-r1
`-- app-alternatives/yacc-1-r2
`-- app-alternatives/lex-0-r1
`-- sys-apps/which-2.23
`-- virtual/pkgconfig-3
`-- app-text/asciidoc-10.2.1
`-- app-text/sgml-common-0.6.3-r7
`-- app-text/xmlto-0.0.28-r11
`-- sys-process/time-1.9-r1
I hope that this helps with reproducing the problem, but if not, let me
know which additional information you'd like to have and I will send it.
My running kernel is 6.16.4, and glibc is version 2.42.
Best regards,
-Guilherme
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: perf --help triggers an assertion
2025-09-10 12:52 ` Guilherme Amadio
@ 2025-09-10 15:04 ` Ian Rogers
2025-12-07 19:08 ` Guilherme Amadio
2025-09-11 20:02 ` Arnaldo Carvalho de Melo
1 sibling, 1 reply; 15+ messages in thread
From: Ian Rogers @ 2025-09-10 15:04 UTC (permalink / raw)
To: Guilherme Amadio
Cc: namhyung, acme, adrian.hunter, jolsa, kan.liang, linux-kernel,
linux-perf-users, mingo, peterz
On Wed, Sep 10, 2025 at 5:52 AM Guilherme Amadio <amadio@gentoo.org> wrote:
>
> On Tue, Sep 09, 2025 at 11:31:51AM -0700, Ian Rogers wrote:
> > On Tue, Sep 9, 2025 at 2:49 AM Guilherme Amadio <amadio@gentoo.org> wrote:
> > >
> > > Hi Namhyung,
> > >
> > > I was updating perf's package in Gentoo Linux and noticed some problems
> > > which were not there before. I tested with the version below and the problem
> > > still seems to be there. perf --help triggers an assertion (see below).
> > > Looking in the list, it seems related to the patch below:
> > >
> > > https://lore.kernel.org/linux-perf-users/20250701201027.1171561-3-namhyung@kernel.org/
> > >
> > > Cheers,
> > > -Guilherme
> > >
> > > The problem:
> > >
> > > gentoo perf $ ./perf --help
> > > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > > Aborted (core dumped) ./perf --help
> > >
> > > Some extra information:
> > >
> > > gentoo perf $ ./perf version
> > > perf version 6.17.rc5.gf777d1112ee5
> > > gentoo perf $ ./perf test subcmd
> > > 69: libsubcmd help tests :
> > > 69.1: Load subcmd names : Ok
> > > 69.2: Uniquify subcmd names : Ok
> > > 69.3: Exclude duplicate subcmd names : Ok
> > > gentoo perf $ ./perf bad-command
> > > perf: 'bad-command' is not a perf-command. See 'perf --help'.
> > > gentoo perf $ ./perf --help
> > > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > > Aborted (core dumped) ./perf --help
> > > gentoo perf $ gdb run --args ./perf --help
> > > GNU gdb (Gentoo 16.3 vanilla) 16.3
> > > Copyright (C) 2024 Free Software Foundation, Inc.
> > > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> > > This is free software: you are free to change and redistribute it.
> > > There is NO WARRANTY, to the extent permitted by law.
> > > Type "show copying" and "show warranty" for details.
> > > This GDB was configured as "x86_64-pc-linux-gnu".
> > > Type "show configuration" for configuration details.
> > > For bug reporting instructions, please see:
> > > <https://bugs.gentoo.org/>.
> > > Find the GDB manual and other documentation resources online at:
> > > <http://www.gnu.org/software/gdb/documentation/>.
> > >
> > > For help, type "help".
> > > Type "apropos word" to search for commands related to "word"...
> > > Reading symbols from ./perf...
> > > (gdb) run
> > > Starting program: /home/amadio/src/linux/tools/perf/perf --help
> > > [Thread debugging using libthread_db enabled]
> > > Using host libthread_db library "/usr/lib64/libthread_db.so.1".
> > > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > >
> > > Program received signal SIGABRT, Aborted.
> > > __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > > 44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
> > > (gdb) bt
> > > #0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > > #1 0x00007ffff74c1656 in __pthread_kill_internal (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:89
> > > #2 0x00007ffff74c166d in __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:100
> > > #3 0x00007ffff747509c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> > > #4 0x00007ffff747637e in __GI_abort () at abort.c:77
> > > #5 0x00007ffff746e023 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x555555974d03 "cmds->names[ci] == NULL",
> > > file=file@entry=0x555555974cfc "help.c", line=line@entry=104, function=function@entry=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds")
> > > at assert.c:118
> > > #6 0x00007ffff746e075 in __assert_fail (assertion=0x555555974d03 "cmds->names[ci] == NULL", file=0x555555974cfc "help.c", line=104,
> > > function=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds") at assert.c:127
> > > #7 0x0000555555693813 in exclude_cmds (cmds=0x55555614e5e0 <other_cmds>, excludes=0x55555614e5c0 <main_cmds>) at help.c:104
> > > #8 0x0000555555693eb3 in load_command_list (prefix=0x555555954071 "perf-", main_cmds=0x55555614e5c0 <main_cmds>,
> > > other_cmds=0x55555614e5e0 <other_cmds>) at help.c:252
> > > #9 0x00005555555e9987 in cmd_help (argc=1, argv=0x7fffffffd1a0) at builtin-help.c:458
> > > #10 0x0000555555685d45 in run_builtin (p=0x555556130de0 <commands+192>, argc=1, argv=0x7fffffffd1a0) at perf.c:349
> > > #11 0x0000555555685fe1 in handle_internal_command (argc=1, argv=0x7fffffffd1a0) at perf.c:401
> > > #12 0x0000555555686142 in run_argv (argcp=0x7fffffffcfac, argv=0x7fffffffcfa0) at perf.c:445
> > > #13 0x0000555555686493 in main (argc=1, argv=0x7fffffffd1a0) at perf.c:553
> > > (gdb) quit
> >
> > Thanks Guilherme,
> >
> > I tried to reproduce the same version with various options: DEBUG=1
> > -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various
> > directories with "perf-" prefixed files. I'm afraid I wasn't able to
> > reproduce. The assert is trying to avoid a memory leak, so
> > non-critical, and I couldn't in a quick inspection eye-ball an issue.
> > Without getting a reproduction I don't think I can make progress with
> > the issue.
>
> I do not have any special setup on my machine (if you consider Gentoo
> not special, of course). I just did a git bisect and arrived at commit
> 9401d1771dad99bfc795dd2ae0c292343fd1f78d, which is the commit I linked
> above. I used
>
> $ make -C tools/perf clean && make -B -j16 -C tools/perf DEBUG=1 WERROR=0 NO_SHELLCHECK=1
>
> then called perf --help each time (but perf help record or equivalent
> also triggers the assertion). I'm using GCC 14 on a 3950X. The features
> I have enabled and installed dependencies are below. This patch has been
> backported to at least 6.12 stable, and I can reproduce this with
> v6.12.45.
>
> gentoo linux $ tools/perf/perf version --build-options
> perf version 6.16.g45effee4b85a
> aio: [ on ] # HAVE_AIO_SUPPORT
> bpf: [ on ] # HAVE_LIBBPF_SUPPORT
> bpf_skeletons: [ on ] # HAVE_BPF_SKEL
> debuginfod: [ on ] # HAVE_DEBUGINFOD_SUPPORT
> dwarf: [ on ] # HAVE_LIBDW_SUPPORT
> dwarf_getlocations: [ on ] # HAVE_LIBDW_SUPPORT
> dwarf-unwind: [ on ] # HAVE_DWARF_UNWIND_SUPPORT
> auxtrace: [ on ] # HAVE_AUXTRACE_SUPPORT
> libbfd: [ OFF ] # HAVE_LIBBFD_SUPPORT ( tip: Deprecated, license incompatibility, use BUILD_NONDISTRO=1 and install binutils-dev[el] )
> libcapstone: [ on ] # HAVE_LIBCAPSTONE_SUPPORT
> libcrypto: [ on ] # HAVE_LIBCRYPTO_SUPPORT
> libdw-dwarf-unwind: [ on ] # HAVE_LIBDW_SUPPORT
> libelf: [ on ] # HAVE_LIBELF_SUPPORT
> libnuma: [ on ] # HAVE_LIBNUMA_SUPPORT
> libopencsd: [ OFF ] # HAVE_CSTRACE_SUPPORT
> libperl: [ on ] # HAVE_LIBPERL_SUPPORT
> libpfm4: [ on ] # HAVE_LIBPFM
> libpython: [ on ] # HAVE_LIBPYTHON_SUPPORT
> libslang: [ on ] # HAVE_SLANG_SUPPORT
> libtraceevent: [ on ] # HAVE_LIBTRACEEVENT
> libunwind: [ OFF ] # HAVE_LIBUNWIND_SUPPORT ( tip: Deprecated, use LIBUNWIND=1 and install libunwind-dev[el] to build with it )
> lzma: [ on ] # HAVE_LZMA_SUPPORT
> numa_num_possible_cpus: [ on ] # HAVE_LIBNUMA_SUPPORT
> zlib: [ on ] # HAVE_ZLIB_SUPPORT
> zstd: [ on ] # HAVE_ZSTD_SUPPORT
>
> And here are the version of all dependencies I have installed:
>
> * dependency graph for dev-util/perf-6.16-r1
> `-- dev-util/perf-6.16-r1
> `-- dev-util/babeltrace-1.5.11
> `-- dev-libs/libbpf-1.6.2
> `-- dev-util/bpftool-7.6.0
> `-- dev-util/pahole-1.30
> `-- llvm-core/clang-18.1.8-r6
> `-- llvm-core/llvm-18.1.8-r6
> `-- llvm-core/clang-19.1.7
> `-- llvm-core/llvm-19.1.7
> `-- llvm-core/clang-20.1.8
> `-- llvm-core/llvm-20.1.8
> `-- llvm-core/clang-21.1.0
> `-- llvm-core/llvm-21.1.0
> `-- sys-libs/libcap-2.76
> `-- dev-libs/capstone-5.0.6
> `-- dev-libs/openssl-3.5.2
> `-- x11-libs/gtk+-2.24.33-r3
> `-- virtual/jre-21
> `-- dev-libs/libpfm-4.13.0
> `-- dev-libs/libtraceevent-1.8.4
> `-- dev-libs/libtracefs-1.8.2
> `-- app-arch/xz-utils-5.8.1-r1
> `-- sys-process/numactl-2.0.19
> `-- dev-lang/perl-5.42.0
> `-- dev-lang/python-0.3.13.7
> `-- dev-lang/python-0.3.14.0_rc2
> `-- dev-lang/python-3.11.13_p1
> `-- dev-lang/python-3.12.11_p1
> `-- dev-lang/python-3.13.9999
> `-- dev-lang/python-3.14.0_rc2
> `-- sys-libs/slang-2.3.3-r1
> `-- dev-debug/systemtap-5.3
> `-- dev-util/google-perftools-2.17.2
> `-- sys-libs/libunwind-1.8.2-r1
> `-- app-arch/zstd-1.5.7-r1
> `-- dev-libs/elfutils-0.193
> `-- sys-libs/binutils-libs-2.45-r1
> `-- sys-libs/zlib-1.3.1-r1
> `-- virtual/libcrypt-2-r1
> `-- sys-kernel/linux-headers-6.16
> `-- virtual/jdk-21
> `-- app-arch/tar-1.35
> `-- dev-python/setuptools-80.9.0-r1
> `-- app-alternatives/yacc-1-r2
> `-- app-alternatives/lex-0-r1
> `-- sys-apps/which-2.23
> `-- virtual/pkgconfig-3
> `-- app-text/asciidoc-10.2.1
> `-- app-text/sgml-common-0.6.3-r7
> `-- app-text/xmlto-0.0.28-r11
> `-- sys-process/time-1.9-r1
>
> I hope that this helps with reproducing the problem, but if not, let me
> know which additional information you'd like to have and I will send it.
> My running kernel is 6.16.4, and glibc is version 2.42.
Thanks Guilherme,
There is also:
https://lore.kernel.org/lkml/CADHxFxSBKuKUWDvki9QDqc5jf1Y+k2EpvOnDsg8O09pZ10n2NA@mail.gmail.com/
which may give me a reproduction. The fix there doesn't look right.
I'll try to find time to look into it again.
Thanks,
Ian
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: perf --help triggers an assertion
2025-09-10 12:52 ` Guilherme Amadio
2025-09-10 15:04 ` Ian Rogers
@ 2025-09-11 20:02 ` Arnaldo Carvalho de Melo
2025-09-11 21:12 ` Namhyung Kim
2025-09-12 11:27 ` Guilherme Amadio
1 sibling, 2 replies; 15+ messages in thread
From: Arnaldo Carvalho de Melo @ 2025-09-11 20:02 UTC (permalink / raw)
To: Guilherme Amadio
Cc: Ian Rogers, namhyung, adrian.hunter, jolsa, kan.liang,
linux-kernel, linux-perf-users, mingo, peterz
On Wed, Sep 10, 2025 at 02:52:25PM +0200, Guilherme Amadio wrote:
> On Tue, Sep 09, 2025 at 11:31:51AM -0700, Ian Rogers wrote:
> > On Tue, Sep 9, 2025 at 2:49 AM Guilherme Amadio <amadio@gentoo.org> wrote:
> > > I was updating perf's package in Gentoo Linux and noticed some problems
> > > which were not there before. I tested with the version below and the problem
> > > still seems to be there. perf --help triggers an assertion (see below).
> > > Looking in the list, it seems related to the patch below:
> > > https://lore.kernel.org/linux-perf-users/20250701201027.1171561-3-namhyung@kernel.org/
<SNIP>
> > I tried to reproduce the same version with various options: DEBUG=1
> > -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various
> > directories with "perf-" prefixed files. I'm afraid I wasn't able to
> > reproduce. The assert is trying to avoid a memory leak, so
> > non-critical, and I couldn't in a quick inspection eye-ball an issue.
> > Without getting a reproduction I don't think I can make progress with
> > the issue.
> I do not have any special setup on my machine (if you consider Gentoo
> not special, of course). I just did a git bisect and arrived at commit
> 9401d1771dad99bfc795dd2ae0c292343fd1f78d, which is the commit I linked
> above. I used>
⬢ [acme@toolbx perf-tools-next]$ git show 9401d1771dad99bfc795dd2ae0c292343fd1f78d
fatal: bad object 9401d1771dad99bfc795dd2ae0c292343fd1f78d
⬢ [acme@toolbx perf-tools-next]$
Looking for the patch title I got to this one:
commit 1fdf938168c4d26fa279d4f204768690d1f9c4ae
Author: Namhyung Kim <namhyung@kernel.org>
Date: Tue Jul 1 13:10:27 2025 -0700
perf tools: Fix use-after-free in help_unknown_cmd()
?
- Arnaldo
> $ make -C tools/perf clean && make -B -j16 -C tools/perf DEBUG=1 WERROR=0 NO_SHELLCHECK=1
>
> then called perf --help each time (but perf help record or equivalent
> also triggers the assertion). I'm using GCC 14 on a 3950X. The features
> I have enabled and installed dependencies are below. This patch has been
> backported to at least 6.12 stable, and I can reproduce this with
> v6.12.45.
>
> gentoo linux $ tools/perf/perf version --build-options
> perf version 6.16.g45effee4b85a
> aio: [ on ] # HAVE_AIO_SUPPORT
> bpf: [ on ] # HAVE_LIBBPF_SUPPORT
> bpf_skeletons: [ on ] # HAVE_BPF_SKEL
> debuginfod: [ on ] # HAVE_DEBUGINFOD_SUPPORT
> dwarf: [ on ] # HAVE_LIBDW_SUPPORT
> dwarf_getlocations: [ on ] # HAVE_LIBDW_SUPPORT
> dwarf-unwind: [ on ] # HAVE_DWARF_UNWIND_SUPPORT
> auxtrace: [ on ] # HAVE_AUXTRACE_SUPPORT
> libbfd: [ OFF ] # HAVE_LIBBFD_SUPPORT ( tip: Deprecated, license incompatibility, use BUILD_NONDISTRO=1 and install binutils-dev[el] )
> libcapstone: [ on ] # HAVE_LIBCAPSTONE_SUPPORT
> libcrypto: [ on ] # HAVE_LIBCRYPTO_SUPPORT
> libdw-dwarf-unwind: [ on ] # HAVE_LIBDW_SUPPORT
> libelf: [ on ] # HAVE_LIBELF_SUPPORT
> libnuma: [ on ] # HAVE_LIBNUMA_SUPPORT
> libopencsd: [ OFF ] # HAVE_CSTRACE_SUPPORT
> libperl: [ on ] # HAVE_LIBPERL_SUPPORT
> libpfm4: [ on ] # HAVE_LIBPFM
> libpython: [ on ] # HAVE_LIBPYTHON_SUPPORT
> libslang: [ on ] # HAVE_SLANG_SUPPORT
> libtraceevent: [ on ] # HAVE_LIBTRACEEVENT
> libunwind: [ OFF ] # HAVE_LIBUNWIND_SUPPORT ( tip: Deprecated, use LIBUNWIND=1 and install libunwind-dev[el] to build with it )
> lzma: [ on ] # HAVE_LZMA_SUPPORT
> numa_num_possible_cpus: [ on ] # HAVE_LIBNUMA_SUPPORT
> zlib: [ on ] # HAVE_ZLIB_SUPPORT
> zstd: [ on ] # HAVE_ZSTD_SUPPORT
>
> And here are the version of all dependencies I have installed:
>
> * dependency graph for dev-util/perf-6.16-r1
> `-- dev-util/perf-6.16-r1
> `-- dev-util/babeltrace-1.5.11
> `-- dev-libs/libbpf-1.6.2
> `-- dev-util/bpftool-7.6.0
> `-- dev-util/pahole-1.30
> `-- llvm-core/clang-18.1.8-r6
> `-- llvm-core/llvm-18.1.8-r6
> `-- llvm-core/clang-19.1.7
> `-- llvm-core/llvm-19.1.7
> `-- llvm-core/clang-20.1.8
> `-- llvm-core/llvm-20.1.8
> `-- llvm-core/clang-21.1.0
> `-- llvm-core/llvm-21.1.0
> `-- sys-libs/libcap-2.76
> `-- dev-libs/capstone-5.0.6
> `-- dev-libs/openssl-3.5.2
> `-- x11-libs/gtk+-2.24.33-r3
> `-- virtual/jre-21
> `-- dev-libs/libpfm-4.13.0
> `-- dev-libs/libtraceevent-1.8.4
> `-- dev-libs/libtracefs-1.8.2
> `-- app-arch/xz-utils-5.8.1-r1
> `-- sys-process/numactl-2.0.19
> `-- dev-lang/perl-5.42.0
> `-- dev-lang/python-0.3.13.7
> `-- dev-lang/python-0.3.14.0_rc2
> `-- dev-lang/python-3.11.13_p1
> `-- dev-lang/python-3.12.11_p1
> `-- dev-lang/python-3.13.9999
> `-- dev-lang/python-3.14.0_rc2
> `-- sys-libs/slang-2.3.3-r1
> `-- dev-debug/systemtap-5.3
> `-- dev-util/google-perftools-2.17.2
> `-- sys-libs/libunwind-1.8.2-r1
> `-- app-arch/zstd-1.5.7-r1
> `-- dev-libs/elfutils-0.193
> `-- sys-libs/binutils-libs-2.45-r1
> `-- sys-libs/zlib-1.3.1-r1
> `-- virtual/libcrypt-2-r1
> `-- sys-kernel/linux-headers-6.16
> `-- virtual/jdk-21
> `-- app-arch/tar-1.35
> `-- dev-python/setuptools-80.9.0-r1
> `-- app-alternatives/yacc-1-r2
> `-- app-alternatives/lex-0-r1
> `-- sys-apps/which-2.23
> `-- virtual/pkgconfig-3
> `-- app-text/asciidoc-10.2.1
> `-- app-text/sgml-common-0.6.3-r7
> `-- app-text/xmlto-0.0.28-r11
> `-- sys-process/time-1.9-r1
>
> I hope that this helps with reproducing the problem, but if not, let me
> know which additional information you'd like to have and I will send it.
> My running kernel is 6.16.4, and glibc is version 2.42.
>
> Best regards,
> -Guilherme
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: perf --help triggers an assertion
2025-09-11 20:02 ` Arnaldo Carvalho de Melo
@ 2025-09-11 21:12 ` Namhyung Kim
2025-09-12 11:27 ` Guilherme Amadio
1 sibling, 0 replies; 15+ messages in thread
From: Namhyung Kim @ 2025-09-11 21:12 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: Guilherme Amadio, Ian Rogers, adrian.hunter, jolsa, kan.liang,
linux-kernel, linux-perf-users, mingo, peterz
On Thu, Sep 11, 2025 at 05:02:52PM -0300, Arnaldo Carvalho de Melo wrote:
> On Wed, Sep 10, 2025 at 02:52:25PM +0200, Guilherme Amadio wrote:
> > On Tue, Sep 09, 2025 at 11:31:51AM -0700, Ian Rogers wrote:
> > > On Tue, Sep 9, 2025 at 2:49 AM Guilherme Amadio <amadio@gentoo.org> wrote:
> > > > I was updating perf's package in Gentoo Linux and noticed some problems
> > > > which were not there before. I tested with the version below and the problem
> > > > still seems to be there. perf --help triggers an assertion (see below).
> > > > Looking in the list, it seems related to the patch below:
>
> > > > https://lore.kernel.org/linux-perf-users/20250701201027.1171561-3-namhyung@kernel.org/
>
> <SNIP>
>
> > > I tried to reproduce the same version with various options: DEBUG=1
> > > -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various
> > > directories with "perf-" prefixed files. I'm afraid I wasn't able to
> > > reproduce. The assert is trying to avoid a memory leak, so
> > > non-critical, and I couldn't in a quick inspection eye-ball an issue.
> > > Without getting a reproduction I don't think I can make progress with
> > > the issue.
>
> > I do not have any special setup on my machine (if you consider Gentoo
> > not special, of course). I just did a git bisect and arrived at commit
> > 9401d1771dad99bfc795dd2ae0c292343fd1f78d, which is the commit I linked
> > above. I used>
>
> ⬢ [acme@toolbx perf-tools-next]$ git show 9401d1771dad99bfc795dd2ae0c292343fd1f78d
> fatal: bad object 9401d1771dad99bfc795dd2ae0c292343fd1f78d
> ⬢ [acme@toolbx perf-tools-next]$
>
> Looking for the patch title I got to this one:
>
> commit 1fdf938168c4d26fa279d4f204768690d1f9c4ae
> Author: Namhyung Kim <namhyung@kernel.org>
> Date: Tue Jul 1 13:10:27 2025 -0700
>
> perf tools: Fix use-after-free in help_unknown_cmd()
Sorry for the trouble. The hupu's patch should fix the problem.
Thanks,
Namhyung
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: perf --help triggers an assertion
2025-09-11 20:02 ` Arnaldo Carvalho de Melo
2025-09-11 21:12 ` Namhyung Kim
@ 2025-09-12 11:27 ` Guilherme Amadio
1 sibling, 0 replies; 15+ messages in thread
From: Guilherme Amadio @ 2025-09-12 11:27 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: Ian Rogers, namhyung, adrian.hunter, jolsa, kan.liang,
linux-kernel, linux-perf-users, mingo, peterz
On Thu, Sep 11, 2025 at 05:02:52PM -0300, Arnaldo Carvalho de Melo wrote:
> On Wed, Sep 10, 2025 at 02:52:25PM +0200, Guilherme Amadio wrote:
> > On Tue, Sep 09, 2025 at 11:31:51AM -0700, Ian Rogers wrote:
> > > On Tue, Sep 9, 2025 at 2:49 AM Guilherme Amadio <amadio@gentoo.org> wrote:
> > > > I was updating perf's package in Gentoo Linux and noticed some problems
> > > > which were not there before. I tested with the version below and the problem
> > > > still seems to be there. perf --help triggers an assertion (see below).
> > > > Looking in the list, it seems related to the patch below:
>
> > > > https://lore.kernel.org/linux-perf-users/20250701201027.1171561-3-namhyung@kernel.org/
>
> <SNIP>
>
> > > I tried to reproduce the same version with various options: DEBUG=1
> > > -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various
> > > directories with "perf-" prefixed files. I'm afraid I wasn't able to
> > > reproduce. The assert is trying to avoid a memory leak, so
> > > non-critical, and I couldn't in a quick inspection eye-ball an issue.
> > > Without getting a reproduction I don't think I can make progress with
> > > the issue.
>
> > I do not have any special setup on my machine (if you consider Gentoo
> > not special, of course). I just did a git bisect and arrived at commit
> > 9401d1771dad99bfc795dd2ae0c292343fd1f78d, which is the commit I linked
> > above. I used>
>
> ⬢ [acme@toolbx perf-tools-next]$ git show 9401d1771dad99bfc795dd2ae0c292343fd1f78d
> fatal: bad object 9401d1771dad99bfc795dd2ae0c292343fd1f78d
> ⬢ [acme@toolbx perf-tools-next]$
>
> Looking for the patch title I got to this one:
>
> commit 1fdf938168c4d26fa279d4f204768690d1f9c4ae
> Author: Namhyung Kim <namhyung@kernel.org>
> Date: Tue Jul 1 13:10:27 2025 -0700
>
> perf tools: Fix use-after-free in help_unknown_cmd()
>
> ?
Sorry, I used the tag I arrived at with git bisect, which was probably cherry-picked:
gentoo linux $ git remote -v
amadio git@github.com:amadio/linux (fetch)
amadio git@github.com:amadio/linux (push)
perf-tools git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools.git (fetch)
perf-tools git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools.git (push)
perf-tools-next git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git (fetch)
perf-tools-next git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git (push)
stable git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git (fetch)
stable git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git (push)
torvalds git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git (fetch)
torvalds git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git (push)
gentoo linux $ git show 9401d1771dad99bfc795dd2ae0c292343fd1f78d | head
commit 9401d1771dad99bfc795dd2ae0c292343fd1f78d
Author: Namhyung Kim <namhyung@kernel.org>
Date: Tue Jul 1 13:10:27 2025 -0700
perf tools: Fix use-after-free in help_unknown_cmd()
[ Upstream commit 1fdf938168c4d26fa279d4f204768690d1f9c4ae ]
Currently perf aborts when it finds an invalid command. I guess it
depends on the environment as I have some custom commands in the path.
gentoo linux $ git tag --contains 9401d1771dad99bfc795dd2ae0c292343fd1f78d
v6.16.1
v6.16.2
v6.16.3
v6.16.4
v6.16.5
v6.16.6
v6.16.7
gentoo linux $ git tag --contains 1fdf938168c4d26fa279d4f204768690d1f9c4ae
perf-tools-fixes-for-v6.17-2025-08-27
perf-tools-fixes-for-v6.17-2025-09-05
perf-tools-for-v6.17-2025-08-01
v6.17-rc1
v6.17-rc2
v6.17-rc3
v6.17-rc4
v6.17-rc5
Cheers,
-Guilherme
>
> - Arnaldo
>
> > $ make -C tools/perf clean && make -B -j16 -C tools/perf DEBUG=1 WERROR=0 NO_SHELLCHECK=1
> >
> > then called perf --help each time (but perf help record or equivalent
> > also triggers the assertion). I'm using GCC 14 on a 3950X. The features
> > I have enabled and installed dependencies are below. This patch has been
> > backported to at least 6.12 stable, and I can reproduce this with
> > v6.12.45.
> >
> > gentoo linux $ tools/perf/perf version --build-options
> > perf version 6.16.g45effee4b85a
> > aio: [ on ] # HAVE_AIO_SUPPORT
> > bpf: [ on ] # HAVE_LIBBPF_SUPPORT
> > bpf_skeletons: [ on ] # HAVE_BPF_SKEL
> > debuginfod: [ on ] # HAVE_DEBUGINFOD_SUPPORT
> > dwarf: [ on ] # HAVE_LIBDW_SUPPORT
> > dwarf_getlocations: [ on ] # HAVE_LIBDW_SUPPORT
> > dwarf-unwind: [ on ] # HAVE_DWARF_UNWIND_SUPPORT
> > auxtrace: [ on ] # HAVE_AUXTRACE_SUPPORT
> > libbfd: [ OFF ] # HAVE_LIBBFD_SUPPORT ( tip: Deprecated, license incompatibility, use BUILD_NONDISTRO=1 and install binutils-dev[el] )
> > libcapstone: [ on ] # HAVE_LIBCAPSTONE_SUPPORT
> > libcrypto: [ on ] # HAVE_LIBCRYPTO_SUPPORT
> > libdw-dwarf-unwind: [ on ] # HAVE_LIBDW_SUPPORT
> > libelf: [ on ] # HAVE_LIBELF_SUPPORT
> > libnuma: [ on ] # HAVE_LIBNUMA_SUPPORT
> > libopencsd: [ OFF ] # HAVE_CSTRACE_SUPPORT
> > libperl: [ on ] # HAVE_LIBPERL_SUPPORT
> > libpfm4: [ on ] # HAVE_LIBPFM
> > libpython: [ on ] # HAVE_LIBPYTHON_SUPPORT
> > libslang: [ on ] # HAVE_SLANG_SUPPORT
> > libtraceevent: [ on ] # HAVE_LIBTRACEEVENT
> > libunwind: [ OFF ] # HAVE_LIBUNWIND_SUPPORT ( tip: Deprecated, use LIBUNWIND=1 and install libunwind-dev[el] to build with it )
> > lzma: [ on ] # HAVE_LZMA_SUPPORT
> > numa_num_possible_cpus: [ on ] # HAVE_LIBNUMA_SUPPORT
> > zlib: [ on ] # HAVE_ZLIB_SUPPORT
> > zstd: [ on ] # HAVE_ZSTD_SUPPORT
> >
> > And here are the version of all dependencies I have installed:
> >
> > * dependency graph for dev-util/perf-6.16-r1
> > `-- dev-util/perf-6.16-r1
> > `-- dev-util/babeltrace-1.5.11
> > `-- dev-libs/libbpf-1.6.2
> > `-- dev-util/bpftool-7.6.0
> > `-- dev-util/pahole-1.30
> > `-- llvm-core/clang-18.1.8-r6
> > `-- llvm-core/llvm-18.1.8-r6
> > `-- llvm-core/clang-19.1.7
> > `-- llvm-core/llvm-19.1.7
> > `-- llvm-core/clang-20.1.8
> > `-- llvm-core/llvm-20.1.8
> > `-- llvm-core/clang-21.1.0
> > `-- llvm-core/llvm-21.1.0
> > `-- sys-libs/libcap-2.76
> > `-- dev-libs/capstone-5.0.6
> > `-- dev-libs/openssl-3.5.2
> > `-- x11-libs/gtk+-2.24.33-r3
> > `-- virtual/jre-21
> > `-- dev-libs/libpfm-4.13.0
> > `-- dev-libs/libtraceevent-1.8.4
> > `-- dev-libs/libtracefs-1.8.2
> > `-- app-arch/xz-utils-5.8.1-r1
> > `-- sys-process/numactl-2.0.19
> > `-- dev-lang/perl-5.42.0
> > `-- dev-lang/python-0.3.13.7
> > `-- dev-lang/python-0.3.14.0_rc2
> > `-- dev-lang/python-3.11.13_p1
> > `-- dev-lang/python-3.12.11_p1
> > `-- dev-lang/python-3.13.9999
> > `-- dev-lang/python-3.14.0_rc2
> > `-- sys-libs/slang-2.3.3-r1
> > `-- dev-debug/systemtap-5.3
> > `-- dev-util/google-perftools-2.17.2
> > `-- sys-libs/libunwind-1.8.2-r1
> > `-- app-arch/zstd-1.5.7-r1
> > `-- dev-libs/elfutils-0.193
> > `-- sys-libs/binutils-libs-2.45-r1
> > `-- sys-libs/zlib-1.3.1-r1
> > `-- virtual/libcrypt-2-r1
> > `-- sys-kernel/linux-headers-6.16
> > `-- virtual/jdk-21
> > `-- app-arch/tar-1.35
> > `-- dev-python/setuptools-80.9.0-r1
> > `-- app-alternatives/yacc-1-r2
> > `-- app-alternatives/lex-0-r1
> > `-- sys-apps/which-2.23
> > `-- virtual/pkgconfig-3
> > `-- app-text/asciidoc-10.2.1
> > `-- app-text/sgml-common-0.6.3-r7
> > `-- app-text/xmlto-0.0.28-r11
> > `-- sys-process/time-1.9-r1
> >
> > I hope that this helps with reproducing the problem, but if not, let me
> > know which additional information you'd like to have and I will send it.
> > My running kernel is 6.16.4, and glibc is version 2.42.
> >
> > Best regards,
> > -Guilherme
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: perf --help triggers an assertion
2025-09-10 15:04 ` Ian Rogers
@ 2025-12-07 19:08 ` Guilherme Amadio
2025-12-07 21:54 ` Ian Rogers
0 siblings, 1 reply; 15+ messages in thread
From: Guilherme Amadio @ 2025-12-07 19:08 UTC (permalink / raw)
To: Ian Rogers
Cc: namhyung, acme, adrian.hunter, jolsa, kan.liang, linux-kernel,
linux-perf-users, mingo, peterz
Hi Ian,
On Wed, Sep 10, 2025 at 08:04:42AM -0700, Ian Rogers wrote:
> On Wed, Sep 10, 2025 at 5:52 AM Guilherme Amadio <amadio@gentoo.org> wrote:
> >
> > On Tue, Sep 09, 2025 at 11:31:51AM -0700, Ian Rogers wrote:
> > > On Tue, Sep 9, 2025 at 2:49 AM Guilherme Amadio <amadio@gentoo.org> wrote:
> > > >
> > > > Hi Namhyung,
> > > >
> > > > I was updating perf's package in Gentoo Linux and noticed some problems
> > > > which were not there before. I tested with the version below and the problem
> > > > still seems to be there. perf --help triggers an assertion (see below).
> > > > Looking in the list, it seems related to the patch below:
> > > >
> > > > https://lore.kernel.org/linux-perf-users/20250701201027.1171561-3-namhyung@kernel.org/
> > > >
> > > > Cheers,
> > > > -Guilherme
> > > >
> > > > The problem:
> > > >
> > > > gentoo perf $ ./perf --help
> > > > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > > > Aborted (core dumped) ./perf --help
> > > >
> > > > Some extra information:
> > > >
> > > > gentoo perf $ ./perf version
> > > > perf version 6.17.rc5.gf777d1112ee5
> > > > gentoo perf $ ./perf test subcmd
> > > > 69: libsubcmd help tests :
> > > > 69.1: Load subcmd names : Ok
> > > > 69.2: Uniquify subcmd names : Ok
> > > > 69.3: Exclude duplicate subcmd names : Ok
> > > > gentoo perf $ ./perf bad-command
> > > > perf: 'bad-command' is not a perf-command. See 'perf --help'.
> > > > gentoo perf $ ./perf --help
> > > > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > > > Aborted (core dumped) ./perf --help
> > > > gentoo perf $ gdb run --args ./perf --help
> > > > GNU gdb (Gentoo 16.3 vanilla) 16.3
> > > > Copyright (C) 2024 Free Software Foundation, Inc.
> > > > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> > > > This is free software: you are free to change and redistribute it.
> > > > There is NO WARRANTY, to the extent permitted by law.
> > > > Type "show copying" and "show warranty" for details.
> > > > This GDB was configured as "x86_64-pc-linux-gnu".
> > > > Type "show configuration" for configuration details.
> > > > For bug reporting instructions, please see:
> > > > <https://bugs.gentoo.org/>.
> > > > Find the GDB manual and other documentation resources online at:
> > > > <http://www.gnu.org/software/gdb/documentation/>.
> > > >
> > > > For help, type "help".
> > > > Type "apropos word" to search for commands related to "word"...
> > > > Reading symbols from ./perf...
> > > > (gdb) run
> > > > Starting program: /home/amadio/src/linux/tools/perf/perf --help
> > > > [Thread debugging using libthread_db enabled]
> > > > Using host libthread_db library "/usr/lib64/libthread_db.so.1".
> > > > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > > >
> > > > Program received signal SIGABRT, Aborted.
> > > > __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > > > 44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
> > > > (gdb) bt
> > > > #0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > > > #1 0x00007ffff74c1656 in __pthread_kill_internal (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:89
> > > > #2 0x00007ffff74c166d in __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:100
> > > > #3 0x00007ffff747509c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> > > > #4 0x00007ffff747637e in __GI_abort () at abort.c:77
> > > > #5 0x00007ffff746e023 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x555555974d03 "cmds->names[ci] == NULL",
> > > > file=file@entry=0x555555974cfc "help.c", line=line@entry=104, function=function@entry=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds")
> > > > at assert.c:118
> > > > #6 0x00007ffff746e075 in __assert_fail (assertion=0x555555974d03 "cmds->names[ci] == NULL", file=0x555555974cfc "help.c", line=104,
> > > > function=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds") at assert.c:127
> > > > #7 0x0000555555693813 in exclude_cmds (cmds=0x55555614e5e0 <other_cmds>, excludes=0x55555614e5c0 <main_cmds>) at help.c:104
> > > > #8 0x0000555555693eb3 in load_command_list (prefix=0x555555954071 "perf-", main_cmds=0x55555614e5c0 <main_cmds>,
> > > > other_cmds=0x55555614e5e0 <other_cmds>) at help.c:252
> > > > #9 0x00005555555e9987 in cmd_help (argc=1, argv=0x7fffffffd1a0) at builtin-help.c:458
> > > > #10 0x0000555555685d45 in run_builtin (p=0x555556130de0 <commands+192>, argc=1, argv=0x7fffffffd1a0) at perf.c:349
> > > > #11 0x0000555555685fe1 in handle_internal_command (argc=1, argv=0x7fffffffd1a0) at perf.c:401
> > > > #12 0x0000555555686142 in run_argv (argcp=0x7fffffffcfac, argv=0x7fffffffcfa0) at perf.c:445
> > > > #13 0x0000555555686493 in main (argc=1, argv=0x7fffffffd1a0) at perf.c:553
> > > > (gdb) quit
> > >
> > > Thanks Guilherme,
> > >
> > > I tried to reproduce the same version with various options: DEBUG=1
> > > -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various
> > > directories with "perf-" prefixed files. I'm afraid I wasn't able to
> > > reproduce. The assert is trying to avoid a memory leak, so
> > > non-critical, and I couldn't in a quick inspection eye-ball an issue.
> > > Without getting a reproduction I don't think I can make progress with
> > > the issue.
I had a look at this with gdb again while I was updating perf to 6.18 in Gentoo.
BTW, I've followed your advice and turned off BUILD_NONDISTRO:
https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b92c5b8bd89c70c40a84f86013c4ecf321f3132f
It looks like the logic to exclude commands sometimes doesn't work as
expected, and you reach a line where there is an assertion for
cmds->names[ci] == NULL, but it's not actually NULL. The difference with
other setups is probably the installation of perf-read-vdso32 into a
default path in our packaging. In gdb I saw that when running perf --help,
I arrive at exclude_cmds() with the following:
Breakpoint 1, exclude_cmds (cmds=cmds@entry=0x55555605bd90 <other_cmds>, excludes=excludes@entry=0x55555605bdb0 <main_cmds>) at help.c:74
74 {
(gdb) p *cmds
$1 = {alloc = 24, cnt = 1, names = 0x5555568fe180}
(gdb) p *excludes
$2 = {alloc = 24, cnt = 1, names = 0x5555568fe0c0}
(gdb) p *cmds->names[0]
$3 = {len = 11, name = 0x5555568db788 "read-vdso32"}
(gdb) p *excludes->names[0]
$4 = {len = 7, name = 0x555556890198 "archive"}
I have /usr/bin/perf and /usr/bin/perf-read-vdso32, as well as this:
$ ls /usr/libexec/perf-core/
dlfilters libperf-gtk.so perf-archive perf-iostat scripts tests
If I remove /usr/bin/perf-read-vdso32, then perf --help works again:
gentoo ~ $ sudo rm /usr/bin/perf-read-vdso32
gentoo ~ $ perf version
perf version 6.18
gentoo ~ $ perf --help | head -3
usage: perf [--version] [--help] [OPTIONS] COMMAND [ARGS]
gentoo ~ $ touch ~/bin/perf-read-vdso32
gentoo ~ $ chmod +x ~/bin/perf-read-vdso32
gentoo ~ $ perf --help
perf: help.c:107: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) perf --help
So, I think that if you don't have perf-read-vdso32 installed by default,
you will likely be able to reproduce the problem like shown above. So,
the assumption that cmds->names[0] == NULL is not correct, since we have
the "read-vdso32" command at position 0, and the excludes list only
contained "archive". The patch below is my attempt to fix the issue:
diff --git a/tools/lib/subcmd/help.c b/tools/lib/subcmd/help.c
index ddaeb4eb3e24..0c4ba87422e5 100644
--- a/tools/lib/subcmd/help.c
+++ b/tools/lib/subcmd/help.c
@@ -102,10 +102,10 @@ void exclude_cmds(struct cmdnames *cmds, struct cmdnames *excludes)
cmds->names[cj++] = cmds->names[ci];
cmds->names[ci++] = NULL;
}
+ for (ci = cj; ci < cmds->cnt; ci++)
+ assert(cmds->names[ci] == NULL);
+ cmds->cnt = cj;
}
- for (ci = cj; ci < cmds->cnt; ci++)
- assert(cmds->names[ci] == NULL);
- cmds->cnt = cj;
}
So, if there are no actual command exclusions, at the end of the while loop
we get out with ei > excludes->cnt, and ci == cj, like the example above,
in which ci == cj == 0, and ei == 1. So the assumption that
cmds->names[0] == NULL is wrong, since there's a valid "read-vdso32"
command at position 0, and setting cmds->cnt = cj; is also wrong, since
cnt should be 1 and is set to 0. The patch moves the asserts and the
update of cmds->cnt inside the if(ci != cj), which indicates something
was actually excluded. If cmds was not modified, then the initial value
of cmds->cnt was already correct. Please let me know if you'd like me
to submit this properly as a patch, but since it's so simple, please
feel free to just take this and apply yourself.
Best regards,
-Guilherme
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: perf --help triggers an assertion
2025-12-07 19:08 ` Guilherme Amadio
@ 2025-12-07 21:54 ` Ian Rogers
2025-12-11 14:35 ` Guilherme Amadio
0 siblings, 1 reply; 15+ messages in thread
From: Ian Rogers @ 2025-12-07 21:54 UTC (permalink / raw)
To: Guilherme Amadio, Jayaramappa, Srilakshmi
Cc: namhyung, acme, adrian.hunter, jolsa, linux-kernel,
linux-perf-users, mingo, peterz
On Sun, Dec 7, 2025 at 11:08 AM Guilherme Amadio <amadio@gentoo.org> wrote:
>
> Hi Ian,
>
> On Wed, Sep 10, 2025 at 08:04:42AM -0700, Ian Rogers wrote:
> > On Wed, Sep 10, 2025 at 5:52 AM Guilherme Amadio <amadio@gentoo.org> wrote:
> > >
> > > On Tue, Sep 09, 2025 at 11:31:51AM -0700, Ian Rogers wrote:
> > > > On Tue, Sep 9, 2025 at 2:49 AM Guilherme Amadio <amadio@gentoo.org> wrote:
> > > > >
> > > > > Hi Namhyung,
> > > > >
> > > > > I was updating perf's package in Gentoo Linux and noticed some problems
> > > > > which were not there before. I tested with the version below and the problem
> > > > > still seems to be there. perf --help triggers an assertion (see below).
> > > > > Looking in the list, it seems related to the patch below:
> > > > >
> > > > > https://lore.kernel.org/linux-perf-users/20250701201027.1171561-3-namhyung@kernel.org/
> > > > >
> > > > > Cheers,
> > > > > -Guilherme
> > > > >
> > > > > The problem:
> > > > >
> > > > > gentoo perf $ ./perf --help
> > > > > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > > > > Aborted (core dumped) ./perf --help
> > > > >
> > > > > Some extra information:
> > > > >
> > > > > gentoo perf $ ./perf version
> > > > > perf version 6.17.rc5.gf777d1112ee5
> > > > > gentoo perf $ ./perf test subcmd
> > > > > 69: libsubcmd help tests :
> > > > > 69.1: Load subcmd names : Ok
> > > > > 69.2: Uniquify subcmd names : Ok
> > > > > 69.3: Exclude duplicate subcmd names : Ok
> > > > > gentoo perf $ ./perf bad-command
> > > > > perf: 'bad-command' is not a perf-command. See 'perf --help'.
> > > > > gentoo perf $ ./perf --help
> > > > > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > > > > Aborted (core dumped) ./perf --help
> > > > > gentoo perf $ gdb run --args ./perf --help
> > > > > GNU gdb (Gentoo 16.3 vanilla) 16.3
> > > > > Copyright (C) 2024 Free Software Foundation, Inc.
> > > > > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> > > > > This is free software: you are free to change and redistribute it.
> > > > > There is NO WARRANTY, to the extent permitted by law.
> > > > > Type "show copying" and "show warranty" for details.
> > > > > This GDB was configured as "x86_64-pc-linux-gnu".
> > > > > Type "show configuration" for configuration details.
> > > > > For bug reporting instructions, please see:
> > > > > <https://bugs.gentoo.org/>.
> > > > > Find the GDB manual and other documentation resources online at:
> > > > > <http://www.gnu.org/software/gdb/documentation/>.
> > > > >
> > > > > For help, type "help".
> > > > > Type "apropos word" to search for commands related to "word"...
> > > > > Reading symbols from ./perf...
> > > > > (gdb) run
> > > > > Starting program: /home/amadio/src/linux/tools/perf/perf --help
> > > > > [Thread debugging using libthread_db enabled]
> > > > > Using host libthread_db library "/usr/lib64/libthread_db.so.1".
> > > > > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > > > >
> > > > > Program received signal SIGABRT, Aborted.
> > > > > __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > > > > 44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
> > > > > (gdb) bt
> > > > > #0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > > > > #1 0x00007ffff74c1656 in __pthread_kill_internal (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:89
> > > > > #2 0x00007ffff74c166d in __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:100
> > > > > #3 0x00007ffff747509c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> > > > > #4 0x00007ffff747637e in __GI_abort () at abort.c:77
> > > > > #5 0x00007ffff746e023 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x555555974d03 "cmds->names[ci] == NULL",
> > > > > file=file@entry=0x555555974cfc "help.c", line=line@entry=104, function=function@entry=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds")
> > > > > at assert.c:118
> > > > > #6 0x00007ffff746e075 in __assert_fail (assertion=0x555555974d03 "cmds->names[ci] == NULL", file=0x555555974cfc "help.c", line=104,
> > > > > function=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds") at assert.c:127
> > > > > #7 0x0000555555693813 in exclude_cmds (cmds=0x55555614e5e0 <other_cmds>, excludes=0x55555614e5c0 <main_cmds>) at help.c:104
> > > > > #8 0x0000555555693eb3 in load_command_list (prefix=0x555555954071 "perf-", main_cmds=0x55555614e5c0 <main_cmds>,
> > > > > other_cmds=0x55555614e5e0 <other_cmds>) at help.c:252
> > > > > #9 0x00005555555e9987 in cmd_help (argc=1, argv=0x7fffffffd1a0) at builtin-help.c:458
> > > > > #10 0x0000555555685d45 in run_builtin (p=0x555556130de0 <commands+192>, argc=1, argv=0x7fffffffd1a0) at perf.c:349
> > > > > #11 0x0000555555685fe1 in handle_internal_command (argc=1, argv=0x7fffffffd1a0) at perf.c:401
> > > > > #12 0x0000555555686142 in run_argv (argcp=0x7fffffffcfac, argv=0x7fffffffcfa0) at perf.c:445
> > > > > #13 0x0000555555686493 in main (argc=1, argv=0x7fffffffd1a0) at perf.c:553
> > > > > (gdb) quit
> > > >
> > > > Thanks Guilherme,
> > > >
> > > > I tried to reproduce the same version with various options: DEBUG=1
> > > > -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various
> > > > directories with "perf-" prefixed files. I'm afraid I wasn't able to
> > > > reproduce. The assert is trying to avoid a memory leak, so
> > > > non-critical, and I couldn't in a quick inspection eye-ball an issue.
> > > > Without getting a reproduction I don't think I can make progress with
> > > > the issue.
>
> I had a look at this with gdb again while I was updating perf to 6.18 in Gentoo.
>
> BTW, I've followed your advice and turned off BUILD_NONDISTRO:
> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b92c5b8bd89c70c40a84f86013c4ecf321f3132f
>
> It looks like the logic to exclude commands sometimes doesn't work as
> expected, and you reach a line where there is an assertion for
> cmds->names[ci] == NULL, but it's not actually NULL. The difference with
> other setups is probably the installation of perf-read-vdso32 into a
> default path in our packaging. In gdb I saw that when running perf --help,
> I arrive at exclude_cmds() with the following:
>
> Breakpoint 1, exclude_cmds (cmds=cmds@entry=0x55555605bd90 <other_cmds>, excludes=excludes@entry=0x55555605bdb0 <main_cmds>) at help.c:74
> 74 {
> (gdb) p *cmds
> $1 = {alloc = 24, cnt = 1, names = 0x5555568fe180}
> (gdb) p *excludes
> $2 = {alloc = 24, cnt = 1, names = 0x5555568fe0c0}
> (gdb) p *cmds->names[0]
> $3 = {len = 11, name = 0x5555568db788 "read-vdso32"}
> (gdb) p *excludes->names[0]
> $4 = {len = 7, name = 0x555556890198 "archive"}
>
> I have /usr/bin/perf and /usr/bin/perf-read-vdso32, as well as this:
>
> $ ls /usr/libexec/perf-core/
> dlfilters libperf-gtk.so perf-archive perf-iostat scripts tests
>
> If I remove /usr/bin/perf-read-vdso32, then perf --help works again:
>
> gentoo ~ $ sudo rm /usr/bin/perf-read-vdso32
> gentoo ~ $ perf version
> perf version 6.18
> gentoo ~ $ perf --help | head -3
>
> usage: perf [--version] [--help] [OPTIONS] COMMAND [ARGS]
>
> gentoo ~ $ touch ~/bin/perf-read-vdso32
> gentoo ~ $ chmod +x ~/bin/perf-read-vdso32
> gentoo ~ $ perf --help
> perf: help.c:107: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> Aborted (core dumped) perf --help
>
> So, I think that if you don't have perf-read-vdso32 installed by default,
> you will likely be able to reproduce the problem like shown above. So,
> the assumption that cmds->names[0] == NULL is not correct, since we have
> the "read-vdso32" command at position 0, and the excludes list only
> contained "archive". The patch below is my attempt to fix the issue:
>
> diff --git a/tools/lib/subcmd/help.c b/tools/lib/subcmd/help.c
> index ddaeb4eb3e24..0c4ba87422e5 100644
> --- a/tools/lib/subcmd/help.c
> +++ b/tools/lib/subcmd/help.c
> @@ -102,10 +102,10 @@ void exclude_cmds(struct cmdnames *cmds, struct cmdnames *excludes)
> cmds->names[cj++] = cmds->names[ci];
> cmds->names[ci++] = NULL;
> }
> + for (ci = cj; ci < cmds->cnt; ci++)
> + assert(cmds->names[ci] == NULL);
> + cmds->cnt = cj;
> }
> - for (ci = cj; ci < cmds->cnt; ci++)
> - assert(cmds->names[ci] == NULL);
> - cmds->cnt = cj;
> }
>
> So, if there are no actual command exclusions, at the end of the while loop
> we get out with ei > excludes->cnt, and ci == cj, like the example above,
> in which ci == cj == 0, and ei == 1. So the assumption that
> cmds->names[0] == NULL is wrong, since there's a valid "read-vdso32"
> command at position 0, and setting cmds->cnt = cj; is also wrong, since
> cnt should be 1 and is set to 0. The patch moves the asserts and the
> update of cmds->cnt inside the if(ci != cj), which indicates something
> was actually excluded. If cmds was not modified, then the initial value
> of cmds->cnt was already correct. Please let me know if you'd like me
> to submit this properly as a patch, but since it's so simple, please
> feel free to just take this and apply yourself.
Thanks Guilherme,
This code is the gift that keeps on giving in terms of crashes. We did
have a recent and related patch sent to the list:
https://lore.kernel.org/lkml/20251202213632.2873731-1-sjayaram@akamai.com/
I need some time to get my head around the issue and the fixes. I
wandered into this code originally due to address/leak sanitizer
issues.
Thanks,
Ian
> Best regards,
> -Guilherme
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: perf --help triggers an assertion
2025-12-07 21:54 ` Ian Rogers
@ 2025-12-11 14:35 ` Guilherme Amadio
0 siblings, 0 replies; 15+ messages in thread
From: Guilherme Amadio @ 2025-12-11 14:35 UTC (permalink / raw)
To: Ian Rogers
Cc: Jayaramappa, Srilakshmi, namhyung, acme, adrian.hunter, jolsa,
linux-kernel, linux-perf-users, mingo, peterz
On Sun, Dec 07, 2025 at 01:54:19PM -0800, Ian Rogers wrote:
> On Sun, Dec 7, 2025 at 11:08 AM Guilherme Amadio <amadio@gentoo.org> wrote:
> >
> > Hi Ian,
> >
> > On Wed, Sep 10, 2025 at 08:04:42AM -0700, Ian Rogers wrote:
> > > On Wed, Sep 10, 2025 at 5:52 AM Guilherme Amadio <amadio@gentoo.org> wrote:
> > > >
> > > > On Tue, Sep 09, 2025 at 11:31:51AM -0700, Ian Rogers wrote:
> > > > > On Tue, Sep 9, 2025 at 2:49 AM Guilherme Amadio <amadio@gentoo.org> wrote:
> > > > > >
> > > > > > Hi Namhyung,
> > > > > >
> > > > > > I was updating perf's package in Gentoo Linux and noticed some problems
> > > > > > which were not there before. I tested with the version below and the problem
> > > > > > still seems to be there. perf --help triggers an assertion (see below).
> > > > > > Looking in the list, it seems related to the patch below:
> > > > > >
> > > > > > https://lore.kernel.org/linux-perf-users/20250701201027.1171561-3-namhyung@kernel.org/
> > > > > >
> > > > > > Cheers,
> > > > > > -Guilherme
> > > > > >
> > > > > > The problem:
> > > > > >
> > > > > > gentoo perf $ ./perf --help
> > > > > > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > > > > > Aborted (core dumped) ./perf --help
> > > > > >
> > > > > > Some extra information:
> > > > > >
> > > > > > gentoo perf $ ./perf version
> > > > > > perf version 6.17.rc5.gf777d1112ee5
> > > > > > gentoo perf $ ./perf test subcmd
> > > > > > 69: libsubcmd help tests :
> > > > > > 69.1: Load subcmd names : Ok
> > > > > > 69.2: Uniquify subcmd names : Ok
> > > > > > 69.3: Exclude duplicate subcmd names : Ok
> > > > > > gentoo perf $ ./perf bad-command
> > > > > > perf: 'bad-command' is not a perf-command. See 'perf --help'.
> > > > > > gentoo perf $ ./perf --help
> > > > > > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > > > > > Aborted (core dumped) ./perf --help
> > > > > > gentoo perf $ gdb run --args ./perf --help
> > > > > > GNU gdb (Gentoo 16.3 vanilla) 16.3
> > > > > > Copyright (C) 2024 Free Software Foundation, Inc.
> > > > > > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> > > > > > This is free software: you are free to change and redistribute it.
> > > > > > There is NO WARRANTY, to the extent permitted by law.
> > > > > > Type "show copying" and "show warranty" for details.
> > > > > > This GDB was configured as "x86_64-pc-linux-gnu".
> > > > > > Type "show configuration" for configuration details.
> > > > > > For bug reporting instructions, please see:
> > > > > > <https://bugs.gentoo.org/>.
> > > > > > Find the GDB manual and other documentation resources online at:
> > > > > > <http://www.gnu.org/software/gdb/documentation/>.
> > > > > >
> > > > > > For help, type "help".
> > > > > > Type "apropos word" to search for commands related to "word"...
> > > > > > Reading symbols from ./perf...
> > > > > > (gdb) run
> > > > > > Starting program: /home/amadio/src/linux/tools/perf/perf --help
> > > > > > [Thread debugging using libthread_db enabled]
> > > > > > Using host libthread_db library "/usr/lib64/libthread_db.so.1".
> > > > > > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > > > > >
> > > > > > Program received signal SIGABRT, Aborted.
> > > > > > __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > > > > > 44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
> > > > > > (gdb) bt
> > > > > > #0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > > > > > #1 0x00007ffff74c1656 in __pthread_kill_internal (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:89
> > > > > > #2 0x00007ffff74c166d in __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:100
> > > > > > #3 0x00007ffff747509c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> > > > > > #4 0x00007ffff747637e in __GI_abort () at abort.c:77
> > > > > > #5 0x00007ffff746e023 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x555555974d03 "cmds->names[ci] == NULL",
> > > > > > file=file@entry=0x555555974cfc "help.c", line=line@entry=104, function=function@entry=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds")
> > > > > > at assert.c:118
> > > > > > #6 0x00007ffff746e075 in __assert_fail (assertion=0x555555974d03 "cmds->names[ci] == NULL", file=0x555555974cfc "help.c", line=104,
> > > > > > function=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds") at assert.c:127
> > > > > > #7 0x0000555555693813 in exclude_cmds (cmds=0x55555614e5e0 <other_cmds>, excludes=0x55555614e5c0 <main_cmds>) at help.c:104
> > > > > > #8 0x0000555555693eb3 in load_command_list (prefix=0x555555954071 "perf-", main_cmds=0x55555614e5c0 <main_cmds>,
> > > > > > other_cmds=0x55555614e5e0 <other_cmds>) at help.c:252
> > > > > > #9 0x00005555555e9987 in cmd_help (argc=1, argv=0x7fffffffd1a0) at builtin-help.c:458
> > > > > > #10 0x0000555555685d45 in run_builtin (p=0x555556130de0 <commands+192>, argc=1, argv=0x7fffffffd1a0) at perf.c:349
> > > > > > #11 0x0000555555685fe1 in handle_internal_command (argc=1, argv=0x7fffffffd1a0) at perf.c:401
> > > > > > #12 0x0000555555686142 in run_argv (argcp=0x7fffffffcfac, argv=0x7fffffffcfa0) at perf.c:445
> > > > > > #13 0x0000555555686493 in main (argc=1, argv=0x7fffffffd1a0) at perf.c:553
> > > > > > (gdb) quit
> > > > >
> > > > > Thanks Guilherme,
> > > > >
> > > > > I tried to reproduce the same version with various options: DEBUG=1
> > > > > -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various
> > > > > directories with "perf-" prefixed files. I'm afraid I wasn't able to
> > > > > reproduce. The assert is trying to avoid a memory leak, so
> > > > > non-critical, and I couldn't in a quick inspection eye-ball an issue.
> > > > > Without getting a reproduction I don't think I can make progress with
> > > > > the issue.
> >
> > I had a look at this with gdb again while I was updating perf to 6.18 in Gentoo.
> >
> > BTW, I've followed your advice and turned off BUILD_NONDISTRO:
> > https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b92c5b8bd89c70c40a84f86013c4ecf321f3132f
> >
> > It looks like the logic to exclude commands sometimes doesn't work as
> > expected, and you reach a line where there is an assertion for
> > cmds->names[ci] == NULL, but it's not actually NULL. The difference with
> > other setups is probably the installation of perf-read-vdso32 into a
> > default path in our packaging. In gdb I saw that when running perf --help,
> > I arrive at exclude_cmds() with the following:
> >
> > Breakpoint 1, exclude_cmds (cmds=cmds@entry=0x55555605bd90 <other_cmds>, excludes=excludes@entry=0x55555605bdb0 <main_cmds>) at help.c:74
> > 74 {
> > (gdb) p *cmds
> > $1 = {alloc = 24, cnt = 1, names = 0x5555568fe180}
> > (gdb) p *excludes
> > $2 = {alloc = 24, cnt = 1, names = 0x5555568fe0c0}
> > (gdb) p *cmds->names[0]
> > $3 = {len = 11, name = 0x5555568db788 "read-vdso32"}
> > (gdb) p *excludes->names[0]
> > $4 = {len = 7, name = 0x555556890198 "archive"}
> >
> > I have /usr/bin/perf and /usr/bin/perf-read-vdso32, as well as this:
> >
> > $ ls /usr/libexec/perf-core/
> > dlfilters libperf-gtk.so perf-archive perf-iostat scripts tests
> >
> > If I remove /usr/bin/perf-read-vdso32, then perf --help works again:
> >
> > gentoo ~ $ sudo rm /usr/bin/perf-read-vdso32
> > gentoo ~ $ perf version
> > perf version 6.18
> > gentoo ~ $ perf --help | head -3
> >
> > usage: perf [--version] [--help] [OPTIONS] COMMAND [ARGS]
> >
> > gentoo ~ $ touch ~/bin/perf-read-vdso32
> > gentoo ~ $ chmod +x ~/bin/perf-read-vdso32
> > gentoo ~ $ perf --help
> > perf: help.c:107: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > Aborted (core dumped) perf --help
> >
> > So, I think that if you don't have perf-read-vdso32 installed by default,
> > you will likely be able to reproduce the problem like shown above. So,
> > the assumption that cmds->names[0] == NULL is not correct, since we have
> > the "read-vdso32" command at position 0, and the excludes list only
> > contained "archive". The patch below is my attempt to fix the issue:
> >
> > diff --git a/tools/lib/subcmd/help.c b/tools/lib/subcmd/help.c
> > index ddaeb4eb3e24..0c4ba87422e5 100644
> > --- a/tools/lib/subcmd/help.c
> > +++ b/tools/lib/subcmd/help.c
> > @@ -102,10 +102,10 @@ void exclude_cmds(struct cmdnames *cmds, struct cmdnames *excludes)
> > cmds->names[cj++] = cmds->names[ci];
> > cmds->names[ci++] = NULL;
> > }
> > + for (ci = cj; ci < cmds->cnt; ci++)
> > + assert(cmds->names[ci] == NULL);
> > + cmds->cnt = cj;
> > }
> > - for (ci = cj; ci < cmds->cnt; ci++)
> > - assert(cmds->names[ci] == NULL);
> > - cmds->cnt = cj;
> > }
> >
> > So, if there are no actual command exclusions, at the end of the while loop
> > we get out with ei > excludes->cnt, and ci == cj, like the example above,
> > in which ci == cj == 0, and ei == 1. So the assumption that
> > cmds->names[0] == NULL is wrong, since there's a valid "read-vdso32"
> > command at position 0, and setting cmds->cnt = cj; is also wrong, since
> > cnt should be 1 and is set to 0. The patch moves the asserts and the
> > update of cmds->cnt inside the if(ci != cj), which indicates something
> > was actually excluded. If cmds was not modified, then the initial value
> > of cmds->cnt was already correct. Please let me know if you'd like me
> > to submit this properly as a patch, but since it's so simple, please
> > feel free to just take this and apply yourself.
>
> Thanks Guilherme,
>
> This code is the gift that keeps on giving in terms of crashes. We did
> have a recent and related patch sent to the list:
> https://lore.kernel.org/lkml/20251202213632.2873731-1-sjayaram@akamai.com/
> I need some time to get my head around the issue and the fixes. I
> wandered into this code originally due to address/leak sanitizer
> issues.
Thank you, Ian,
I've applied Sri's patch in Gentoo. It seems to be a better fix than
what I proposed above, so please feel free to add my Tested-by: to Sri's patch.
https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a77ba80e013b62ca44a3123167aa3ee376e7faeb
Best regards,
-Guilherme
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2025-12-11 14:35 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-07-01 20:10 [PATCH v2 1/3] perf test: Check test suite description properly Namhyung Kim
2025-07-01 20:10 ` [PATCH v2 2/3] perf test: Add libsubcmd help tests Namhyung Kim
2025-07-01 20:10 ` [PATCH v2 3/3] perf tools: Fix use-after-free in help_unknown_cmd() Namhyung Kim
2025-09-09 9:49 ` perf --help triggers an assertion Guilherme Amadio
2025-09-09 18:31 ` Ian Rogers
2025-09-10 12:52 ` Guilherme Amadio
2025-09-10 15:04 ` Ian Rogers
2025-12-07 19:08 ` Guilherme Amadio
2025-12-07 21:54 ` Ian Rogers
2025-12-11 14:35 ` Guilherme Amadio
2025-09-11 20:02 ` Arnaldo Carvalho de Melo
2025-09-11 21:12 ` Namhyung Kim
2025-09-12 11:27 ` Guilherme Amadio
2025-07-01 22:38 ` [PATCH v2 1/3] perf test: Check test suite description properly Ian Rogers
2025-07-02 16:43 ` Namhyung Kim
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).