Re: perf --help triggers an assertion

[Guilherme Amadio <amadio@gentoo.org>]

On Sun, Dec 07, 2025 at 01:54:19PM -0800, Ian Rogers wrote:
> On Sun, Dec 7, 2025 at 11:08 AM Guilherme Amadio <amadio@gentoo.org> wrote:
> >
> > Hi Ian,
> >
> > On Wed, Sep 10, 2025 at 08:04:42AM -0700, Ian Rogers wrote:
> > > On Wed, Sep 10, 2025 at 5:52 AM Guilherme Amadio <amadio@gentoo.org> wrote:
> > > >
> > > > On Tue, Sep 09, 2025 at 11:31:51AM -0700, Ian Rogers wrote:
> > > > > On Tue, Sep 9, 2025 at 2:49 AM Guilherme Amadio <amadio@gentoo.org> wrote:
> > > > > >
> > > > > > Hi Namhyung,
> > > > > >
> > > > > > I was updating perf's package in Gentoo Linux and noticed some problems
> > > > > > which were not there before. I tested with the version below and the problem
> > > > > > still seems to be there. perf --help triggers an assertion (see below).
> > > > > > Looking in the list, it seems related to the patch below:
> > > > > >
> > > > > > https://lore.kernel.org/linux-perf-users/20250701201027.1171561-3-namhyung@kernel.org/
> > > > > >
> > > > > > Cheers,
> > > > > > -Guilherme
> > > > > >
> > > > > > The problem:
> > > > > >
> > > > > > gentoo perf $ ./perf --help
> > > > > > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > > > > > Aborted                    (core dumped) ./perf --help
> > > > > >
> > > > > > Some extra information:
> > > > > >
> > > > > > gentoo perf $ ./perf version
> > > > > > perf version 6.17.rc5.gf777d1112ee5
> > > > > > gentoo perf $ ./perf test subcmd
> > > > > >  69: libsubcmd help tests                                            :
> > > > > >  69.1: Load subcmd names                                             : Ok
> > > > > >  69.2: Uniquify subcmd names                                         : Ok
> > > > > >  69.3: Exclude duplicate subcmd names                                : Ok
> > > > > > gentoo perf $ ./perf bad-command
> > > > > > perf: 'bad-command' is not a perf-command. See 'perf --help'.
> > > > > > gentoo perf $ ./perf --help
> > > > > > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > > > > > Aborted                    (core dumped) ./perf --help
> > > > > > gentoo perf $ gdb run --args ./perf --help
> > > > > > GNU gdb (Gentoo 16.3 vanilla) 16.3
> > > > > > Copyright (C) 2024 Free Software Foundation, Inc.
> > > > > > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> > > > > > This is free software: you are free to change and redistribute it.
> > > > > > There is NO WARRANTY, to the extent permitted by law.
> > > > > > Type "show copying" and "show warranty" for details.
> > > > > > This GDB was configured as "x86_64-pc-linux-gnu".
> > > > > > Type "show configuration" for configuration details.
> > > > > > For bug reporting instructions, please see:
> > > > > > <https://bugs.gentoo.org/>.
> > > > > > Find the GDB manual and other documentation resources online at:
> > > > > >     <http://www.gnu.org/software/gdb/documentation/>.
> > > > > >
> > > > > > For help, type "help".
> > > > > > Type "apropos word" to search for commands related to "word"...
> > > > > > Reading symbols from ./perf...
> > > > > > (gdb) run
> > > > > > Starting program: /home/amadio/src/linux/tools/perf/perf --help
> > > > > > [Thread debugging using libthread_db enabled]
> > > > > > Using host libthread_db library "/usr/lib64/libthread_db.so.1".
> > > > > > perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > > > > >
> > > > > > Program received signal SIGABRT, Aborted.
> > > > > > __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > > > > > 44            return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
> > > > > > (gdb) bt
> > > > > > #0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > > > > > #1  0x00007ffff74c1656 in __pthread_kill_internal (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:89
> > > > > > #2  0x00007ffff74c166d in __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:100
> > > > > > #3  0x00007ffff747509c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> > > > > > #4  0x00007ffff747637e in __GI_abort () at abort.c:77
> > > > > > #5  0x00007ffff746e023 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x555555974d03 "cmds->names[ci] == NULL",
> > > > > >     file=file@entry=0x555555974cfc "help.c", line=line@entry=104, function=function@entry=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds")
> > > > > >     at assert.c:118
> > > > > > #6  0x00007ffff746e075 in __assert_fail (assertion=0x555555974d03 "cmds->names[ci] == NULL", file=0x555555974cfc "help.c", line=104,
> > > > > >     function=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds") at assert.c:127
> > > > > > #7  0x0000555555693813 in exclude_cmds (cmds=0x55555614e5e0 <other_cmds>, excludes=0x55555614e5c0 <main_cmds>) at help.c:104
> > > > > > #8  0x0000555555693eb3 in load_command_list (prefix=0x555555954071 "perf-", main_cmds=0x55555614e5c0 <main_cmds>,
> > > > > >     other_cmds=0x55555614e5e0 <other_cmds>) at help.c:252
> > > > > > #9  0x00005555555e9987 in cmd_help (argc=1, argv=0x7fffffffd1a0) at builtin-help.c:458
> > > > > > #10 0x0000555555685d45 in run_builtin (p=0x555556130de0 <commands+192>, argc=1, argv=0x7fffffffd1a0) at perf.c:349
> > > > > > #11 0x0000555555685fe1 in handle_internal_command (argc=1, argv=0x7fffffffd1a0) at perf.c:401
> > > > > > #12 0x0000555555686142 in run_argv (argcp=0x7fffffffcfac, argv=0x7fffffffcfa0) at perf.c:445
> > > > > > #13 0x0000555555686493 in main (argc=1, argv=0x7fffffffd1a0) at perf.c:553
> > > > > > (gdb) quit
> > > > >
> > > > > Thanks Guilherme,
> > > > >
> > > > > I tried to reproduce the same version with various options: DEBUG=1
> > > > > -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various
> > > > > directories with "perf-" prefixed files. I'm afraid I wasn't able to
> > > > > reproduce. The assert is trying to avoid a memory leak, so
> > > > > non-critical, and I couldn't in a quick inspection eye-ball an issue.
> > > > > Without getting a reproduction I don't think I can make progress with
> > > > > the issue.
> >
> > I had a look at this with gdb again while I was updating perf to 6.18 in Gentoo.
> >
> > BTW, I've followed your advice and turned off BUILD_NONDISTRO:
> > https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b92c5b8bd89c70c40a84f86013c4ecf321f3132f
> >
> > It looks like the logic to exclude commands sometimes doesn't work as
> > expected, and you reach a line where there is an assertion for
> > cmds->names[ci] == NULL, but it's not actually NULL. The difference with
> > other setups is probably the installation of perf-read-vdso32 into a
> > default path in our packaging. In gdb I saw that when running perf --help,
> > I arrive at exclude_cmds() with the following:
> >
> > Breakpoint 1, exclude_cmds (cmds=cmds@entry=0x55555605bd90 <other_cmds>, excludes=excludes@entry=0x55555605bdb0 <main_cmds>) at help.c:74
> > 74      {
> > (gdb) p *cmds
> > $1 = {alloc = 24, cnt = 1, names = 0x5555568fe180}
> > (gdb) p *excludes
> > $2 = {alloc = 24, cnt = 1, names = 0x5555568fe0c0}
> > (gdb) p *cmds->names[0]
> > $3 = {len = 11, name = 0x5555568db788 "read-vdso32"}
> > (gdb) p *excludes->names[0]
> > $4 = {len = 7, name = 0x555556890198 "archive"}
> >
> > I have /usr/bin/perf and /usr/bin/perf-read-vdso32, as well as this:
> >
> > $ ls /usr/libexec/perf-core/
> > dlfilters  libperf-gtk.so  perf-archive  perf-iostat  scripts  tests
> >
> > If I remove /usr/bin/perf-read-vdso32, then perf --help works again:
> >
> > gentoo ~ $ sudo rm /usr/bin/perf-read-vdso32
> > gentoo ~ $ perf version
> > perf version 6.18
> > gentoo ~ $ perf --help | head -3
> >
> >  usage: perf [--version] [--help] [OPTIONS] COMMAND [ARGS]
> >
> > gentoo ~ $ touch ~/bin/perf-read-vdso32
> > gentoo ~ $ chmod +x ~/bin/perf-read-vdso32
> > gentoo ~ $ perf --help
> > perf: help.c:107: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
> > Aborted                    (core dumped) perf --help
> >
> > So, I think that if you don't have perf-read-vdso32 installed by default,
> > you will likely be able to reproduce the problem like shown above. So,
> > the assumption that cmds->names[0] == NULL is not correct, since we have
> > the "read-vdso32" command at position 0, and the excludes list only
> > contained "archive". The patch below is my attempt to fix the issue:
> >
> > diff --git a/tools/lib/subcmd/help.c b/tools/lib/subcmd/help.c
> > index ddaeb4eb3e24..0c4ba87422e5 100644
> > --- a/tools/lib/subcmd/help.c
> > +++ b/tools/lib/subcmd/help.c
> > @@ -102,10 +102,10 @@ void exclude_cmds(struct cmdnames *cmds, struct cmdnames *excludes)
> >                         cmds->names[cj++] = cmds->names[ci];
> >                         cmds->names[ci++] = NULL;
> >                 }
> > +               for (ci = cj; ci < cmds->cnt; ci++)
> > +                       assert(cmds->names[ci] == NULL);
> > +               cmds->cnt = cj;
> >         }
> > -       for (ci = cj; ci < cmds->cnt; ci++)
> > -               assert(cmds->names[ci] == NULL);
> > -       cmds->cnt = cj;
> >  }
> >
> > So, if there are no actual command exclusions, at the end of the while loop
> > we get out with ei > excludes->cnt, and ci == cj, like the example above,
> > in which ci == cj == 0, and ei == 1. So the assumption that
> > cmds->names[0] == NULL is wrong, since there's a valid "read-vdso32"
> > command at position 0, and setting cmds->cnt = cj; is also wrong, since
> > cnt should be 1 and is set to 0. The patch moves the asserts and the
> > update of cmds->cnt inside the if(ci != cj), which indicates something
> > was actually excluded. If cmds was not modified, then the initial value
> > of cmds->cnt was already correct. Please let me know if you'd like me
> > to submit this properly as a patch, but since it's so simple, please
> > feel free to just take this and apply yourself.
> 
> Thanks Guilherme,
> 
> This code is the gift that keeps on giving in terms of crashes. We did
> have a recent and related patch sent to the list:
> https://lore.kernel.org/lkml/20251202213632.2873731-1-sjayaram@akamai.com/
> I need some time to get my head around the issue and the fixes. I
> wandered into this code originally due to address/leak sanitizer
> issues.

Thank you, Ian,

I've applied Sri's patch in Gentoo. It seems to be a better fix than
what I proposed above, so please feel free to add my Tested-by: to Sri's patch. 

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a77ba80e013b62ca44a3123167aa3ee376e7faeb

Best regards,
-Guilherme