Re: perf --help triggers an assertion

Guilherme Amadio
2025-09-10T12:52:32Z

On Tue, Sep 09, 2025 at 11:31:51AM -0700, Ian Rogers wrote:
> On Tue, Sep 9, 2025 at 2:49 AM Guilherme Amadio <amadio@gentoo.org> wrote:
> >
> > Hi Namhyung,
> >
> > I was updating perf's package in Gentoo Linux and noticed some problems
> > which were not there before. I tested with the version below and the problem
> > still seems to be there. perf --help triggers an assertion (see below).
> > Looking in the list, it seems related to the patch below:
> >
> > https://lore.kernel.org/linux-perf-users/20250701201027.1171561-3-namhyung@kernel.org/
> >
> > Cheers,
> > -Guilherme
> >
> > The problem:
> >
> > gentoo perf $ ./perf --help
> > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > Aborted (core dumped) ./perf --help
> >
> > Some extra information:
> >
> > gentoo perf $ ./perf version
> > perf version 6.17.rc5.gf777d1112ee5
> > gentoo perf $ ./perf test subcmd
> > 69: libsubcmd help tests :
> > 69.1: Load subcmd names : Ok
> > 69.2: Uniquify subcmd names : Ok
> > 69.3: Exclude duplicate subcmd names : Ok
> > gentoo perf $ ./perf bad-command
> > perf: 'bad-command' is not a perf-command. See 'perf --help'.
> > gentoo perf $ ./perf --help
> > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > Aborted (core dumped) ./perf --help
> > gentoo perf $ gdb run --args ./perf --help
> > GNU gdb (Gentoo 16.3 vanilla) 16.3
> > Copyright (C) 2024 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law.
> > Type "show copying" and "show warranty" for details.
> > This GDB was configured as "x86_64-pc-linux-gnu".
> > Type "show configuration" for configuration details.
> > For bug reporting instructions, please see:
> > <https://bugs.gentoo.org/>.
> > Find the GDB manual and other documentation resources online at:
> > <http://www.gnu.org/software/gdb/documentation/>.
> >
> > For help, type "help".
> > Type "apropos word" to search for commands related to "word"...
> > Reading symbols from ./perf...
> > (gdb) run
> > Starting program: /home/amadio/src/linux/tools/perf/perf --help
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/usr/lib64/libthread_db.so.1".
> > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> >
> > Program received signal SIGABRT, Aborted.
> > __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > 44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
> > (gdb) bt
> > #0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > #1 0x00007ffff74c1656 in __pthread_kill_internal (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:89
> > #2 0x00007ffff74c166d in __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:100
> > #3 0x00007ffff747509c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> > #4 0x00007ffff747637e in __GI_abort () at abort.c:77
> > #5 0x00007ffff746e023 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x555555974d03 "cmds->names[ci] == NULL",
> > file=file@entry=0x555555974cfc "help.c", line=line@entry=104, function=function@entry=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds")
> > at assert.c:118
> > #6 0x00007ffff746e075 in __assert_fail (assertion=0x555555974d03 "cmds->names[ci] == NULL", file=0x555555974cfc "help.c", line=104,
> > function=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds") at assert.c:127
> > #7 0x0000555555693813 in exclude_cmds (cmds=0x55555614e5e0 <other_cmds>, excludes=0x55555614e5c0 <main_cmds>) at help.c:104
> > #8 0x0000555555693eb3 in load_command_list (prefix=0x555555954071 "perf-", main_cmds=0x55555614e5c0 <main_cmds>,
> > other_cmds=0x55555614e5e0 <other_cmds>) at help.c:252
> > #9 0x00005555555e9987 in cmd_help (argc=1, argv=0x7fffffffd1a0) at builtin-help.c:458
> > #10 0x0000555555685d45 in run_builtin (p=0x555556130de0 <commands+192>, argc=1, argv=0x7fffffffd1a0) at perf.c:349
> > #11 0x0000555555685fe1 in handle_internal_command (argc=1, argv=0x7fffffffd1a0) at perf.c:401
> > #12 0x0000555555686142 in run_argv (argcp=0x7fffffffcfac, argv=0x7fffffffcfa0) at perf.c:445
> > #13 0x0000555555686493 in main (argc=1, argv=0x7fffffffd1a0) at perf.c:553
> > (gdb) quit
>
> Thanks Guilherme,
>
> I tried to reproduce the same version with various options: DEBUG=1
> -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various
> directories with "perf-" prefixed files. I'm afraid I wasn't able to
> reproduce. The assert is trying to avoid a memory leak, so
> non-critical, and I couldn't in a quick inspection eye-ball an issue.
> Without getting a reproduction I don't think I can make progress with
> the issue.

I do not have any special setup on my machine (if you consider Gentoo
not special, of course). I just did a git bisect and arrived at commit
9401d1771dad99bfc795dd2ae0c292343fd1f78d, which is the commit I linked
above. I used

$ make -C tools/perf clean && make -B -j16 -C tools/perf DEBUG=1 WERROR=0 NO_SHELLCHECK=1

then called perf --help each time (but perf help record or equivalent
also triggers the assertion). I'm using GCC 14 on a 3950X. The features
I have enabled and installed dependencies are below. This patch has been
backported to at least 6.12 stable, and I can reproduce this with
v6.12.45.

gentoo linux $ tools/perf/perf version --build-options
perf version 6.16.g45effee4b85a
aio: [ on ] # HAVE_AIO_SUPPORT
bpf: [ on ] # HAVE_LIBBPF_SUPPORT
bpf_skeletons: [ on ] # HAVE_BPF_SKEL
debuginfod: [ on ] # HAVE_DEBUGINFOD_SUPPORT
dwarf: [ on ] # HAVE_LIBDW_SUPPORT
dwarf_getlocations: [ on ] # HAVE_LIBDW_SUPPORT
dwarf-unwind: [ on ] # HAVE_DWARF_UNWIND_SUPPORT
auxtrace: [ on ] # HAVE_AUXTRACE_SUPPORT
libbfd: [ OFF ] # HAVE_LIBBFD_SUPPORT ( tip: Deprecated, license incompatibility, use BUILD_NONDISTRO=1 and install binutils-dev[el] )
libcapstone: [ on ] # HAVE_LIBCAPSTONE_SUPPORT
libcrypto: [ on ] # HAVE_LIBCRYPTO_SUPPORT
libdw-dwarf-unwind: [ on ] # HAVE_LIBDW_SUPPORT
libelf: [ on ] # HAVE_LIBELF_SUPPORT
libnuma: [ on ] # HAVE_LIBNUMA_SUPPORT
libopencsd: [ OFF ] # HAVE_CSTRACE_SUPPORT
libperl: [ on ] # HAVE_LIBPERL_SUPPORT
libpfm4: [ on ] # HAVE_LIBPFM
libpython: [ on ] # HAVE_LIBPYTHON_SUPPORT
libslang: [ on ] # HAVE_SLANG_SUPPORT
libtraceevent: [ on ] # HAVE_LIBTRACEEVENT
libunwind: [ OFF ] # HAVE_LIBUNWIND_SUPPORT ( tip: Deprecated, use LIBUNWIND=1 and install libunwind-dev[el] to build with it )
lzma: [ on ] # HAVE_LZMA_SUPPORT
numa_num_possible_cpus: [ on ] # HAVE_LIBNUMA_SUPPORT
zlib: [ on ] # HAVE_ZLIB_SUPPORT
zstd: [ on ] # HAVE_ZSTD_SUPPORT

And here are the version of all dependencies I have installed:

* dependency graph for dev-util/perf-6.16-r1
`-- dev-util/perf-6.16-r1
`-- dev-util/babeltrace-1.5.11
`-- dev-libs/libbpf-1.6.2
`-- dev-util/bpftool-7.6.0
`-- dev-util/pahole-1.30
`-- llvm-core/clang-18.1.8-r6
`-- llvm-core/llvm-18.1.8-r6
`-- llvm-core/clang-19.1.7
`-- llvm-core/llvm-19.1.7
`-- llvm-core/clang-20.1.8
`-- llvm-core/llvm-20.1.8
`-- llvm-core/clang-21.1.0
`-- llvm-core/llvm-21.1.0
`-- sys-libs/libcap-2.76
`-- dev-libs/capstone-5.0.6
`-- dev-libs/openssl-3.5.2
`-- x11-libs/gtk+-2.24.33-r3
`-- virtual/jre-21
`-- dev-libs/libpfm-4.13.0
`-- dev-libs/libtraceevent-1.8.4
`-- dev-libs/libtracefs-1.8.2
`-- app-arch/xz-utils-5.8.1-r1
`-- sys-process/numactl-2.0.19
`-- dev-lang/perl-5.42.0
`-- dev-lang/python-0.3.13.7
`-- dev-lang/python-0.3.14.0_rc2
`-- dev-lang/python-3.11.13_p1
`-- dev-lang/python-3.12.11_p1
`-- dev-lang/python-3.13.9999
`-- dev-lang/python-3.14.0_rc2
`-- sys-libs/slang-2.3.3-r1
`-- dev-debug/systemtap-5.3
`-- dev-util/google-perftools-2.17.2
`-- sys-libs/libunwind-1.8.2-r1
`-- app-arch/zstd-1.5.7-r1
`-- dev-libs/elfutils-0.193
`-- sys-libs/binutils-libs-2.45-r1
`-- sys-libs/zlib-1.3.1-r1
`-- virtual/libcrypt-2-r1
`-- sys-kernel/linux-headers-6.16
`-- virtual/jdk-21
`-- app-arch/tar-1.35
`-- dev-python/setuptools-80.9.0-r1
`-- app-alternatives/yacc-1-r2
`-- app-alternatives/lex-0-r1
`-- sys-apps/which-2.23
`-- virtual/pkgconfig-3
`-- app-text/asciidoc-10.2.1
`-- app-text/sgml-common-0.6.3-r7
`-- app-text/xmlto-0.0.28-r11
`-- sys-process/time-1.9-r1

I hope that this helps with reproducing the problem, but if not, let me
know which additional information you'd like to have and I will send it.
My running kernel is 6.16.4, and glibc is version 2.42.

Best regards,
-Guilherme