Re: perf --help triggers an assertion

[Guilherme Amadio <amadio@gentoo.org>]

On Tue, Sep 09, 2025 at 11:31:51AM -0700, Ian Rogers wrote:
> On Tue, Sep 9, 2025 at 2:49 AM Guilherme Amadio <amadio@gentoo.org> wrote:
> >
> > Hi Namhyung,
> >
> > I was updating perf's package in Gentoo Linux and noticed some problems
> > which were not there before. I tested with the version below and the problem
> > still seems to be there. perf --help triggers an assertion (see below).
> > Looking in the list, it seems related to the patch below:
> >
> > https://lore.kernel.org/linux-perf-users/20250701201027.1171561-3-namhyung@kernel.org/
> >
> > Cheers,
> > -Guilherme
> >
> > The problem:
> >
> > gentoo perf $ ./perf --help
> > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > Aborted                    (core dumped) ./perf --help
> >
> > Some extra information:
> >
> > gentoo perf $ ./perf version
> > perf version 6.17.rc5.gf777d1112ee5
> > gentoo perf $ ./perf test subcmd
> >  69: libsubcmd help tests                                            :
> >  69.1: Load subcmd names                                             : Ok
> >  69.2: Uniquify subcmd names                                         : Ok
> >  69.3: Exclude duplicate subcmd names                                : Ok
> > gentoo perf $ ./perf bad-command
> > perf: 'bad-command' is not a perf-command. See 'perf --help'.
> > gentoo perf $ ./perf --help
> > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > Aborted                    (core dumped) ./perf --help
> > gentoo perf $ gdb run --args ./perf --help
> > GNU gdb (Gentoo 16.3 vanilla) 16.3
> > Copyright (C) 2024 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law.
> > Type "show copying" and "show warranty" for details.
> > This GDB was configured as "x86_64-pc-linux-gnu".
> > Type "show configuration" for configuration details.
> > For bug reporting instructions, please see:
> > <https://bugs.gentoo.org/>.
> > Find the GDB manual and other documentation resources online at:
> >     <http://www.gnu.org/software/gdb/documentation/>.
> >
> > For help, type "help".
> > Type "apropos word" to search for commands related to "word"...
> > Reading symbols from ./perf...
> > (gdb) run
> > Starting program: /home/amadio/src/linux/tools/perf/perf --help
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/usr/lib64/libthread_db.so.1".
> > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> >
> > Program received signal SIGABRT, Aborted.
> > __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > 44            return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
> > (gdb) bt
> > #0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > #1  0x00007ffff74c1656 in __pthread_kill_internal (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:89
> > #2  0x00007ffff74c166d in __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:100
> > #3  0x00007ffff747509c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> > #4  0x00007ffff747637e in __GI_abort () at abort.c:77
> > #5  0x00007ffff746e023 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x555555974d03 "cmds->names[ci] == NULL",
> >     file=file@entry=0x555555974cfc "help.c", line=line@entry=104, function=function@entry=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds")
> >     at assert.c:118
> > #6  0x00007ffff746e075 in __assert_fail (assertion=0x555555974d03 "cmds->names[ci] == NULL", file=0x555555974cfc "help.c", line=104,
> >     function=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds") at assert.c:127
> > #7  0x0000555555693813 in exclude_cmds (cmds=0x55555614e5e0 <other_cmds>, excludes=0x55555614e5c0 <main_cmds>) at help.c:104
> > #8  0x0000555555693eb3 in load_command_list (prefix=0x555555954071 "perf-", main_cmds=0x55555614e5c0 <main_cmds>,
> >     other_cmds=0x55555614e5e0 <other_cmds>) at help.c:252
> > #9  0x00005555555e9987 in cmd_help (argc=1, argv=0x7fffffffd1a0) at builtin-help.c:458
> > #10 0x0000555555685d45 in run_builtin (p=0x555556130de0 <commands+192>, argc=1, argv=0x7fffffffd1a0) at perf.c:349
> > #11 0x0000555555685fe1 in handle_internal_command (argc=1, argv=0x7fffffffd1a0) at perf.c:401
> > #12 0x0000555555686142 in run_argv (argcp=0x7fffffffcfac, argv=0x7fffffffcfa0) at perf.c:445
> > #13 0x0000555555686493 in main (argc=1, argv=0x7fffffffd1a0) at perf.c:553
> > (gdb) quit
> 
> Thanks Guilherme,
> 
> I tried to reproduce the same version with various options: DEBUG=1
> -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various
> directories with "perf-" prefixed files. I'm afraid I wasn't able to
> reproduce. The assert is trying to avoid a memory leak, so
> non-critical, and I couldn't in a quick inspection eye-ball an issue.
> Without getting a reproduction I don't think I can make progress with
> the issue.

I do not have any special setup on my machine (if you consider Gentoo
not special, of course). I just did a git bisect and arrived at commit
9401d1771dad99bfc795dd2ae0c292343fd1f78d, which is the commit I linked
above. I used

$ make -C tools/perf clean && make -B -j16 -C tools/perf DEBUG=1 WERROR=0 NO_SHELLCHECK=1

then called perf --help each time (but perf help record or equivalent
also triggers the assertion). I'm using GCC 14 on a 3950X. The features
I have enabled and installed dependencies are below. This patch has been
backported to at least 6.12 stable, and I can reproduce this with
v6.12.45.

gentoo linux $ tools/perf/perf version --build-options
perf version 6.16.g45effee4b85a
                   aio: [ on  ]  # HAVE_AIO_SUPPORT
                   bpf: [ on  ]  # HAVE_LIBBPF_SUPPORT
         bpf_skeletons: [ on  ]  # HAVE_BPF_SKEL
            debuginfod: [ on  ]  # HAVE_DEBUGINFOD_SUPPORT
                 dwarf: [ on  ]  # HAVE_LIBDW_SUPPORT
    dwarf_getlocations: [ on  ]  # HAVE_LIBDW_SUPPORT
          dwarf-unwind: [ on  ]  # HAVE_DWARF_UNWIND_SUPPORT
              auxtrace: [ on  ]  # HAVE_AUXTRACE_SUPPORT
                libbfd: [ OFF ]  # HAVE_LIBBFD_SUPPORT ( tip: Deprecated, license incompatibility, use BUILD_NONDISTRO=1 and install binutils-dev[el] )
           libcapstone: [ on  ]  # HAVE_LIBCAPSTONE_SUPPORT
             libcrypto: [ on  ]  # HAVE_LIBCRYPTO_SUPPORT
    libdw-dwarf-unwind: [ on  ]  # HAVE_LIBDW_SUPPORT
                libelf: [ on  ]  # HAVE_LIBELF_SUPPORT
               libnuma: [ on  ]  # HAVE_LIBNUMA_SUPPORT
            libopencsd: [ OFF ]  # HAVE_CSTRACE_SUPPORT
               libperl: [ on  ]  # HAVE_LIBPERL_SUPPORT
               libpfm4: [ on  ]  # HAVE_LIBPFM
             libpython: [ on  ]  # HAVE_LIBPYTHON_SUPPORT
              libslang: [ on  ]  # HAVE_SLANG_SUPPORT
         libtraceevent: [ on  ]  # HAVE_LIBTRACEEVENT
             libunwind: [ OFF ]  # HAVE_LIBUNWIND_SUPPORT ( tip: Deprecated, use LIBUNWIND=1 and install libunwind-dev[el] to build with it )
                  lzma: [ on  ]  # HAVE_LZMA_SUPPORT
numa_num_possible_cpus: [ on  ]  # HAVE_LIBNUMA_SUPPORT
                  zlib: [ on  ]  # HAVE_ZLIB_SUPPORT
                  zstd: [ on  ]  # HAVE_ZSTD_SUPPORT

And here are the version of all dependencies I have installed:

 * dependency graph for dev-util/perf-6.16-r1
 `--  dev-util/perf-6.16-r1
   `--  dev-util/babeltrace-1.5.11
   `--  dev-libs/libbpf-1.6.2
   `--  dev-util/bpftool-7.6.0
   `--  dev-util/pahole-1.30
   `--  llvm-core/clang-18.1.8-r6
   `--  llvm-core/llvm-18.1.8-r6
   `--  llvm-core/clang-19.1.7
   `--  llvm-core/llvm-19.1.7
   `--  llvm-core/clang-20.1.8
   `--  llvm-core/llvm-20.1.8
   `--  llvm-core/clang-21.1.0
   `--  llvm-core/llvm-21.1.0
   `--  sys-libs/libcap-2.76
   `--  dev-libs/capstone-5.0.6
   `--  dev-libs/openssl-3.5.2
   `--  x11-libs/gtk+-2.24.33-r3
   `--  virtual/jre-21
   `--  dev-libs/libpfm-4.13.0
   `--  dev-libs/libtraceevent-1.8.4
   `--  dev-libs/libtracefs-1.8.2
   `--  app-arch/xz-utils-5.8.1-r1
   `--  sys-process/numactl-2.0.19
   `--  dev-lang/perl-5.42.0
   `--  dev-lang/python-0.3.13.7
   `--  dev-lang/python-0.3.14.0_rc2
   `--  dev-lang/python-3.11.13_p1
   `--  dev-lang/python-3.12.11_p1
   `--  dev-lang/python-3.13.9999
   `--  dev-lang/python-3.14.0_rc2
   `--  sys-libs/slang-2.3.3-r1
   `--  dev-debug/systemtap-5.3
   `--  dev-util/google-perftools-2.17.2
   `--  sys-libs/libunwind-1.8.2-r1
   `--  app-arch/zstd-1.5.7-r1
   `--  dev-libs/elfutils-0.193
   `--  sys-libs/binutils-libs-2.45-r1
   `--  sys-libs/zlib-1.3.1-r1
   `--  virtual/libcrypt-2-r1
   `--  sys-kernel/linux-headers-6.16
   `--  virtual/jdk-21
   `--  app-arch/tar-1.35
   `--  dev-python/setuptools-80.9.0-r1
   `--  app-alternatives/yacc-1-r2
   `--  app-alternatives/lex-0-r1
   `--  sys-apps/which-2.23
   `--  virtual/pkgconfig-3
   `--  app-text/asciidoc-10.2.1
   `--  app-text/sgml-common-0.6.3-r7
   `--  app-text/xmlto-0.0.28-r11
   `--  sys-process/time-1.9-r1

I hope that this helps with reproducing the problem, but if not, let me
know which additional information you'd like to have and I will send it.
My running kernel is 6.16.4, and glibc is version 2.42.

Best regards,
-Guilherme