[PATCH] perf comm str: Fix perf top coredump due to concurrent read and write

[PATCH] perf comm str: Fix perf top coredump due to concurrent read and write

[Fei Lang]

(gdb) bt
    __strcmp_evex () at ../sysdeps/x86_64/multiarch/strcmp-evex.S:314
    sort.comm_collapse () at util/sort.c:202
    hist_entry__collapse at util/hist.c:1312
    hists__collapse_insert_entry at util/hist.c:1620
    hists__collapse_resort at util/hist.c:1704
    perf_top__resort_hists at builtin-top.c:303
    perf_top__print_sym_table at builtin-top.c:350
    display_thread at builtin-top.c:700

Link:https://bugzilla.kernel.org/show_bug.cgi?id=220096

Fixes: <3178f58b9894> ("perf comm str: Avoid sort during insert")
Signed-off-by: Fei Lang <langfei@huawei.com>
---
 tools/perf/util/comm.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/comm.c b/tools/perf/util/comm.c
index 8aa456d7c2cd..0438870d31d2 100644
--- a/tools/perf/util/comm.c
+++ b/tools/perf/util/comm.c
@@ -209,13 +209,16 @@ struct comm *comm__new(const char *str, u64 timestamp, bool exec)
 int comm__override(struct comm *comm, const char *str, u64 timestamp, bool exec)
 {
 	struct comm_str *new, *old = comm->comm_str;
+	struct comm_strs *comm_strs = comm_strs__get();
 
 	new = comm_strs__findnew(str);
 	if (!new)
 		return -ENOMEM;
 
+	down_write(&comm_strs->lock);
 	comm_str__put(old);
 	comm->comm_str = new;
+	up_write(&comm_strs->lock);
 	comm->start = timestamp;
 	if (exec)
 		comm->exec = true;
@@ -225,11 +228,22 @@ int comm__override(struct comm *comm, const char *str, u64 timestamp, bool exec)
 
 void comm__free(struct comm *comm)
 {
+	struct comm_strs *comm_strs = comm_strs__get();
+
+	down_write(&comm_strs->lock);
 	comm_str__put(comm->comm_str);
 	free(comm);
+	up_write(&comm_strs->lock);
 }
 
 const char *comm__str(const struct comm *comm)
 {
-	return comm_str__str(comm->comm_str);
+	struct comm_strs *comm_strs = comm_strs__get();
+	char *p;
+
+	down_read(&comm_strs->lock);
+	p = comm_str__str(comm->comm_str);
+	up_read(&comm_strs->lock);
+
+	return p;
 }
-- 
2.33.0

Re: [PATCH] perf comm str: Fix perf top coredump due to concurrent read and write

[Ian Rogers]

On Mon, May 19, 2025 at 4:56 AM Fei Lang <langfei@huawei.com> wrote:
>
> (gdb) bt
>     __strcmp_evex () at ../sysdeps/x86_64/multiarch/strcmp-evex.S:314
>     sort.comm_collapse () at util/sort.c:202
>     hist_entry__collapse at util/hist.c:1312
>     hists__collapse_insert_entry at util/hist.c:1620
>     hists__collapse_resort at util/hist.c:1704
>     perf_top__resort_hists at builtin-top.c:303
>     perf_top__print_sym_table at builtin-top.c:350
>     display_thread at builtin-top.c:700
>
> Link:https://bugzilla.kernel.org/show_bug.cgi?id=220096
>
> Fixes: <3178f58b9894> ("perf comm str: Avoid sort during insert")
> Signed-off-by: Fei Lang <langfei@huawei.com>
> ---
>  tools/perf/util/comm.c | 16 +++++++++++++++-
>  1 file changed, 15 insertions(+), 1 deletion(-)
>
> diff --git a/tools/perf/util/comm.c b/tools/perf/util/comm.c
> index 8aa456d7c2cd..0438870d31d2 100644
> --- a/tools/perf/util/comm.c
> +++ b/tools/perf/util/comm.c
> @@ -209,13 +209,16 @@ struct comm *comm__new(const char *str, u64 timestamp, bool exec)
>  int comm__override(struct comm *comm, const char *str, u64 timestamp, bool exec)
>  {
>         struct comm_str *new, *old = comm->comm_str;
> +       struct comm_strs *comm_strs = comm_strs__get();
>
>         new = comm_strs__findnew(str);
>         if (!new)
>                 return -ENOMEM;
>
> +       down_write(&comm_strs->lock);

comm_strs are a uniq-ified set of strs to avoid memory overhead from
comm events. A comm_str is reference counted and immutable. Using the
comm_str lock on the comm struct isn't something I agree with as we
already have thread__comm_lock.

From the bug report $rdi is non-zero but comm_strs are immutable and
reference counted, perhaps address sanitizer and reference count
checking can point to the problem (add -fsanitize=address to your
cflags). I put together some thread safety patches to see if the
problem can be caught, but nothing that looks particularly likely:
https://lore.kernel.org/lkml/20250519224645.1810891-1-irogers@google.com/
I couldn't repro the problem locally.

Thanks,
Ian




>         comm_str__put(old);
>         comm->comm_str = new;
> +       up_write(&comm_strs->lock);
>         comm->start = timestamp;
>         if (exec)
>                 comm->exec = true;
> @@ -225,11 +228,22 @@ int comm__override(struct comm *comm, const char *str, u64 timestamp, bool exec)
>
>  void comm__free(struct comm *comm)
>  {
> +       struct comm_strs *comm_strs = comm_strs__get();
> +
> +       down_write(&comm_strs->lock);
>         comm_str__put(comm->comm_str);
>         free(comm);
> +       up_write(&comm_strs->lock);
>  }
>
>  const char *comm__str(const struct comm *comm)
>  {
> -       return comm_str__str(comm->comm_str);
> +       struct comm_strs *comm_strs = comm_strs__get();
> +       char *p;
> +
> +       down_read(&comm_strs->lock);
> +       p = comm_str__str(comm->comm_str);
> +       up_read(&comm_strs->lock);
> +
> +       return p;
>  }
> --
> 2.33.0
>

Re: [PATCH] perf comm str: Fix perf top coredump due to concurrent read and write

[Arnaldo Carvalho de Melo]

On Mon, May 19, 2025 at 03:48:37PM -0700, Ian Rogers wrote:
> On Mon, May 19, 2025 at 4:56 AM Fei Lang <langfei@huawei.com> wrote:
> >
> > (gdb) bt
> >     __strcmp_evex () at ../sysdeps/x86_64/multiarch/strcmp-evex.S:314
> >     sort.comm_collapse () at util/sort.c:202
> >     hist_entry__collapse at util/hist.c:1312
> >     hists__collapse_insert_entry at util/hist.c:1620
> >     hists__collapse_resort at util/hist.c:1704
> >     perf_top__resort_hists at builtin-top.c:303
> >     perf_top__print_sym_table at builtin-top.c:350
> >     display_thread at builtin-top.c:700
> >
> > Link:https://bugzilla.kernel.org/show_bug.cgi?id=220096
> >
> > Fixes: <3178f58b9894> ("perf comm str: Avoid sort during insert")
> > Signed-off-by: Fei Lang <langfei@huawei.com>
> > ---
> >  tools/perf/util/comm.c | 16 +++++++++++++++-
> >  1 file changed, 15 insertions(+), 1 deletion(-)
> >
> > diff --git a/tools/perf/util/comm.c b/tools/perf/util/comm.c
> > index 8aa456d7c2cd..0438870d31d2 100644
> > --- a/tools/perf/util/comm.c
> > +++ b/tools/perf/util/comm.c
> > @@ -209,13 +209,16 @@ struct comm *comm__new(const char *str, u64 timestamp, bool exec)
> >  int comm__override(struct comm *comm, const char *str, u64 timestamp, bool exec)
> >  {
> >         struct comm_str *new, *old = comm->comm_str;
> > +       struct comm_strs *comm_strs = comm_strs__get();
> >
> >         new = comm_strs__findnew(str);
> >         if (!new)
> >                 return -ENOMEM;
> >
> > +       down_write(&comm_strs->lock);
> 
> comm_strs are a uniq-ified set of strs to avoid memory overhead from
> comm events. A comm_str is reference counted and immutable. Using the
> comm_str lock on the comm struct isn't something I agree with as we
> already have thread__comm_lock.
> 
> >From the bug report $rdi is non-zero but comm_strs are immutable and
> reference counted, perhaps address sanitizer and reference count
> checking can point to the problem (add -fsanitize=address to your
> cflags). I put together some thread safety patches to see if the
> problem can be caught, but nothing that looks particularly likely:
> https://lore.kernel.org/lkml/20250519224645.1810891-1-irogers@google.com/
> I couldn't repro the problem locally.

I couldn't repro it here as well, and without your thread safety
patches, that I have applied on my notebook and I'm merging with this
workstation repo to push out.

- Arnaldo