[PATCH] perf report: Add 'tgid' sort key

[PATCH] perf report: Add 'tgid' sort key

[Namhyung Kim]

Sometimes we need to analyze the data in process level but current sort
keys only work on thread level.  Let's add 'tgid' sort key for that as
'pid' is already taken for thread.

This will look mostly the same, but it only uses tgid instead of tid.
Here's an example of a process with two threads (thloop).

  $ perf record -- perf test -w thloop

  $ perf report --stdio -s tgid,pid -H
  ...
  #
  #    Overhead  Tgid:Command / Pid:Command
  # ...........  ..........................
  #
     100.00%     2018407:perf
         50.34%     2018407:perf
         49.66%     2018409:perf

Suggested-by: Stephane Eranian <eranian@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/Documentation/perf-report.txt |  1 +
 tools/perf/util/hist.h                   |  1 +
 tools/perf/util/sort.c                   | 35 ++++++++++++++++++++++++
 tools/perf/util/sort.h                   |  1 +
 4 files changed, 38 insertions(+)

diff --git a/tools/perf/Documentation/perf-report.txt b/tools/perf/Documentation/perf-report.txt
index 87f86451940623f3..4050ec4038425bf0 100644
--- a/tools/perf/Documentation/perf-report.txt
+++ b/tools/perf/Documentation/perf-report.txt
@@ -79,6 +79,7 @@ OPTIONS
 
 	- comm: command (name) of the task which can be read via /proc/<pid>/comm
 	- pid: command and tid of the task
+	- tgid: command and tgid of the task
 	- dso: name of library or module executed at the time of sample
 	- dso_size: size of library or module executed at the time of sample
 	- symbol: name of function executed at the time of sample
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index 46c8373e314657fa..c164e178e0a48a8e 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -38,6 +38,7 @@ enum hist_column {
 	HISTC_TIME,
 	HISTC_DSO,
 	HISTC_THREAD,
+	HISTC_TGID,
 	HISTC_COMM,
 	HISTC_CGROUP_ID,
 	HISTC_CGROUP,
diff --git a/tools/perf/util/sort.c b/tools/perf/util/sort.c
index 3dd33721823f365d..5987438174967fd6 100644
--- a/tools/perf/util/sort.c
+++ b/tools/perf/util/sort.c
@@ -141,6 +141,40 @@ struct sort_entry sort_thread = {
 	.se_width_idx	= HISTC_THREAD,
 };
 
+/* --sort tgid */
+
+static int64_t
+sort__tgid_cmp(struct hist_entry *left, struct hist_entry *right)
+{
+	return thread__pid(right->thread) - thread__pid(left->thread);
+}
+
+static int hist_entry__tgid_snprintf(struct hist_entry *he, char *bf,
+				       size_t size, unsigned int width)
+{
+	int tgid = thread__pid(he->thread);
+	const char *comm = NULL;
+
+	if (thread__pid(he->thread) == thread__tid(he->thread)) {
+		comm = thread__comm_str(he->thread);
+	} else {
+		struct maps *maps = thread__maps(he->thread);
+		struct thread *leader = machine__find_thread(maps__machine(maps),
+							     tgid, tgid);
+		if (leader)
+			comm = thread__comm_str(leader);
+	}
+	width = max(7U, width) - 8;
+	return repsep_snprintf(bf, size, "%7d:%-*.*s", tgid, width, width, comm ?: "");
+}
+
+struct sort_entry sort_tgid = {
+	.se_header	= "   Tgid:Command",
+	.se_cmp		= sort__tgid_cmp,
+	.se_snprintf	= hist_entry__tgid_snprintf,
+	.se_width_idx	= HISTC_TGID,
+};
+
 /* --sort simd */
 
 static int64_t
@@ -2501,6 +2535,7 @@ static void sort_dimension_add_dynamic_header(struct sort_dimension *sd)
 
 static struct sort_dimension common_sort_dimensions[] = {
 	DIM(SORT_PID, "pid", sort_thread),
+	DIM(SORT_TGID, "tgid", sort_tgid),
 	DIM(SORT_COMM, "comm", sort_comm),
 	DIM(SORT_DSO, "dso", sort_dso),
 	DIM(SORT_SYM, "symbol", sort_sym),
diff --git a/tools/perf/util/sort.h b/tools/perf/util/sort.h
index a8572574e1686be6..6044eb1d61447c0d 100644
--- a/tools/perf/util/sort.h
+++ b/tools/perf/util/sort.h
@@ -72,6 +72,7 @@ enum sort_type {
 	SORT_ANNOTATE_DATA_TYPE_OFFSET,
 	SORT_SYM_OFFSET,
 	SORT_ANNOTATE_DATA_TYPE_CACHELINE,
+	SORT_TGID,
 
 	/* branch stack specific sort keys */
 	__SORT_BRANCH_STACK,
-- 
2.48.1.502.g6dc24dfdaf-goog

Re: [PATCH] perf report: Add 'tgid' sort key

[Ian Rogers]

On Wed, Feb 5, 2025 at 4:01 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> Sometimes we need to analyze the data in process level but current sort
> keys only work on thread level.  Let's add 'tgid' sort key for that as
> 'pid' is already taken for thread.
>
> This will look mostly the same, but it only uses tgid instead of tid.
> Here's an example of a process with two threads (thloop).
>
>   $ perf record -- perf test -w thloop
>
>   $ perf report --stdio -s tgid,pid -H
>   ...
>   #
>   #    Overhead  Tgid:Command / Pid:Command
>   # ...........  ..........................
>   #
>      100.00%     2018407:perf
>          50.34%     2018407:perf
>          49.66%     2018409:perf
>
> Suggested-by: Stephane Eranian <eranian@google.com>
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>

Reviewed-by: Ian Rogers <irogers@google.com>

Thanks,
Ian

Re: [PATCH] perf report: Add 'tgid' sort key

[Arnaldo Carvalho de Melo]

On Wed, Feb 05, 2025 at 04:01:37PM -0800, Namhyung Kim wrote:
> Sometimes we need to analyze the data in process level but current sort
> keys only work on thread level.  Let's add 'tgid' sort key for that as
> 'pid' is already taken for thread.
> 
> This will look mostly the same, but it only uses tgid instead of tid.
> Here's an example of a process with two threads (thloop).
> 
>   $ perf record -- perf test -w thloop

Unrelated, but when building perf with DEBUG=1 and trying to test the
above I noticed:

root@number:~# perf record -- perf test -w thloop
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.404 MB perf.data (7968 samples) ]
perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
Aborted (core dumped)
root@number:~# perf record -- perf test -w offcpu
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.040 MB perf.data (23 samples) ]
perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
Aborted (core dumped)
root@number:~#

I have:

⬢ [acme@toolbox perf-tools-next]$ git log --oneline perf-tools-next/perf-tools-next..
9de1ed6fa3b73cb1 (HEAD -> perf-tools-next) perf report: Add 'tgid' sort key
23e98ede2a353530 perf trace: Add --summary-mode option
e6d6104625a3790b perf tools: Get rid of now-unused rb_resort.h
173ec14e72ef4ed7 perf trace: Convert syscall_stats to hashmap
66edfb5d404e743d perf trace: Allocate syscall stats only if summary is on
ca6637e1ea08e6f4 perf parse-events filter: Use evsel__find_pmu()
bd1ac4a678f7f2c8 perf bench evlist-open-close: Reduce scope of 2 variables
cd59081880e89df8 perf test: Add direct off-cpu test
56cbd794c0c46ba9 perf record --off-cpu: Add --off-cpu-thresh option
28d9b19c5455556f perf record --off-cpu: Dump the remaining samples in BPF's stack trace map
2bc05b02743b50a7 perf script: Display off-cpu samples correctly
bfa457a621596947 perf record --off-cpu: Disable perf_event's callchain collection
eca732cc42d20266 perf evsel: Assemble offcpu samples
74ce50e40c569e90 perf record --off-cpu: Dump off-cpu samples in BPF
e75f8ce63bfa6cb9 perf record --off-cpu: Preparation of off-cpu BPF program
0ffab9d26971c91c perf record --off-cpu: Parse off-cpu event
efc3fe2070853b7d perf evsel: Expose evsel__is_offcpu_event() for future use
⬢ [acme@toolbox perf-tools-next]$

locally, that is the stuff I've been testing lately, doubt it is related
to these patches, I'll investigate later, have to go AFK, so FWIW as a
heads up.

- Arnaldo

>   $ perf report --stdio -s tgid,pid -H
>   ...
>   #
>   #    Overhead  Tgid:Command / Pid:Command
>   # ...........  ..........................
>   #
>      100.00%     2018407:perf
>          50.34%     2018407:perf
>          49.66%     2018409:perf
> 
> Suggested-by: Stephane Eranian <eranian@google.com>
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/Documentation/perf-report.txt |  1 +
>  tools/perf/util/hist.h                   |  1 +
>  tools/perf/util/sort.c                   | 35 ++++++++++++++++++++++++
>  tools/perf/util/sort.h                   |  1 +
>  4 files changed, 38 insertions(+)
> 
> diff --git a/tools/perf/Documentation/perf-report.txt b/tools/perf/Documentation/perf-report.txt
> index 87f86451940623f3..4050ec4038425bf0 100644
> --- a/tools/perf/Documentation/perf-report.txt
> +++ b/tools/perf/Documentation/perf-report.txt
> @@ -79,6 +79,7 @@ OPTIONS
>  
>  	- comm: command (name) of the task which can be read via /proc/<pid>/comm
>  	- pid: command and tid of the task
> +	- tgid: command and tgid of the task
>  	- dso: name of library or module executed at the time of sample
>  	- dso_size: size of library or module executed at the time of sample
>  	- symbol: name of function executed at the time of sample
> diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
> index 46c8373e314657fa..c164e178e0a48a8e 100644
> --- a/tools/perf/util/hist.h
> +++ b/tools/perf/util/hist.h
> @@ -38,6 +38,7 @@ enum hist_column {
>  	HISTC_TIME,
>  	HISTC_DSO,
>  	HISTC_THREAD,
> +	HISTC_TGID,
>  	HISTC_COMM,
>  	HISTC_CGROUP_ID,
>  	HISTC_CGROUP,
> diff --git a/tools/perf/util/sort.c b/tools/perf/util/sort.c
> index 3dd33721823f365d..5987438174967fd6 100644
> --- a/tools/perf/util/sort.c
> +++ b/tools/perf/util/sort.c
> @@ -141,6 +141,40 @@ struct sort_entry sort_thread = {
>  	.se_width_idx	= HISTC_THREAD,
>  };
>  
> +/* --sort tgid */
> +
> +static int64_t
> +sort__tgid_cmp(struct hist_entry *left, struct hist_entry *right)
> +{
> +	return thread__pid(right->thread) - thread__pid(left->thread);
> +}
> +
> +static int hist_entry__tgid_snprintf(struct hist_entry *he, char *bf,
> +				       size_t size, unsigned int width)
> +{
> +	int tgid = thread__pid(he->thread);
> +	const char *comm = NULL;
> +
> +	if (thread__pid(he->thread) == thread__tid(he->thread)) {
> +		comm = thread__comm_str(he->thread);
> +	} else {
> +		struct maps *maps = thread__maps(he->thread);
> +		struct thread *leader = machine__find_thread(maps__machine(maps),
> +							     tgid, tgid);
> +		if (leader)
> +			comm = thread__comm_str(leader);
> +	}
> +	width = max(7U, width) - 8;
> +	return repsep_snprintf(bf, size, "%7d:%-*.*s", tgid, width, width, comm ?: "");
> +}
> +
> +struct sort_entry sort_tgid = {
> +	.se_header	= "   Tgid:Command",
> +	.se_cmp		= sort__tgid_cmp,
> +	.se_snprintf	= hist_entry__tgid_snprintf,
> +	.se_width_idx	= HISTC_TGID,
> +};
> +
>  /* --sort simd */
>  
>  static int64_t
> @@ -2501,6 +2535,7 @@ static void sort_dimension_add_dynamic_header(struct sort_dimension *sd)
>  
>  static struct sort_dimension common_sort_dimensions[] = {
>  	DIM(SORT_PID, "pid", sort_thread),
> +	DIM(SORT_TGID, "tgid", sort_tgid),
>  	DIM(SORT_COMM, "comm", sort_comm),
>  	DIM(SORT_DSO, "dso", sort_dso),
>  	DIM(SORT_SYM, "symbol", sort_sym),
> diff --git a/tools/perf/util/sort.h b/tools/perf/util/sort.h
> index a8572574e1686be6..6044eb1d61447c0d 100644
> --- a/tools/perf/util/sort.h
> +++ b/tools/perf/util/sort.h
> @@ -72,6 +72,7 @@ enum sort_type {
>  	SORT_ANNOTATE_DATA_TYPE_OFFSET,
>  	SORT_SYM_OFFSET,
>  	SORT_ANNOTATE_DATA_TYPE_CACHELINE,
> +	SORT_TGID,
>  
>  	/* branch stack specific sort keys */
>  	__SORT_BRANCH_STACK,
> -- 
> 2.48.1.502.g6dc24dfdaf-goog

Re: [PATCH] perf report: Add 'tgid' sort key

[Arnaldo Carvalho de Melo]

On Wed, Feb 12, 2025 at 10:05:27PM +0100, Arnaldo Carvalho de Melo wrote:
> On Wed, Feb 05, 2025 at 04:01:37PM -0800, Namhyung Kim wrote:
> > Sometimes we need to analyze the data in process level but current sort
> > keys only work on thread level.  Let's add 'tgid' sort key for that as
> > 'pid' is already taken for thread.
> > 
> > This will look mostly the same, but it only uses tgid instead of tid.
> > Here's an example of a process with two threads (thloop).
> > 
> >   $ perf record -- perf test -w thloop
> 
> Unrelated, but when building perf with DEBUG=1 and trying to test the
> above I noticed:
> 
> root@number:~# perf record -- perf test -w thloop
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.404 MB perf.data (7968 samples) ]
> perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> Aborted (core dumped)
> root@number:~# perf record -- perf test -w offcpu
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.040 MB perf.data (23 samples) ]
> perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> Aborted (core dumped)
> root@number:~#
> 
> I have:
> 
> ⬢ [acme@toolbox perf-tools-next]$ git log --oneline perf-tools-next/perf-tools-next..
> 9de1ed6fa3b73cb1 (HEAD -> perf-tools-next) perf report: Add 'tgid' sort key
> 23e98ede2a353530 perf trace: Add --summary-mode option
> e6d6104625a3790b perf tools: Get rid of now-unused rb_resort.h
> 173ec14e72ef4ed7 perf trace: Convert syscall_stats to hashmap
> 66edfb5d404e743d perf trace: Allocate syscall stats only if summary is on
> ca6637e1ea08e6f4 perf parse-events filter: Use evsel__find_pmu()
> bd1ac4a678f7f2c8 perf bench evlist-open-close: Reduce scope of 2 variables
> cd59081880e89df8 perf test: Add direct off-cpu test
> 56cbd794c0c46ba9 perf record --off-cpu: Add --off-cpu-thresh option
> 28d9b19c5455556f perf record --off-cpu: Dump the remaining samples in BPF's stack trace map
> 2bc05b02743b50a7 perf script: Display off-cpu samples correctly
> bfa457a621596947 perf record --off-cpu: Disable perf_event's callchain collection
> eca732cc42d20266 perf evsel: Assemble offcpu samples
> 74ce50e40c569e90 perf record --off-cpu: Dump off-cpu samples in BPF
> e75f8ce63bfa6cb9 perf record --off-cpu: Preparation of off-cpu BPF program
> 0ffab9d26971c91c perf record --off-cpu: Parse off-cpu event
> efc3fe2070853b7d perf evsel: Expose evsel__is_offcpu_event() for future use
> ⬢ [acme@toolbox perf-tools-next]$
> 
> locally, that is the stuff I've been testing lately, doubt it is related
> to these patches, I'll investigate later, have to go AFK, so FWIW as a
> heads up.

Had time to extract this, now going really AFK:

[New Thread 0x7fffdf24c6c0 (LWP 580622)]
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.403 MB perf.data (7948 samples) ]
[Thread 0x7fffdf24c6c0 (LWP 580622) exited]
perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.

Thread 1 "perf" received signal SIGABRT, Aborted.
Downloading 4.06 K source file /usr/src/debug/glibc-2.39-37.fc40.x86_64/nptl/pthread_kill.c
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44                                                                      
44	      return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78
#2  0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007ffff6e36902 in __GI_abort () at abort.c:79
#4  0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev) <= map__end(map)", 
    file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96
#5  0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95, 
    function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105
#6  0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95
#7  0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538
#8  0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176
#9  0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238
#10 0x00000000006388af in perf_session__destroy_kernel_maps (session=0xf97f60) at util/session.c:105
#11 0x0000000000638df0 in perf_session__delete (session=0xf97f60) at util/session.c:248
#12 0x0000000000431f18 in __cmd_record (rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888
#13 0x00000000004351fb in cmd_record (argc=4, argv=0x7fffffffde60) at builtin-record.c:4286
#14 0x00000000004bd4d4 in run_builtin (p=0xecddc0 <commands+288>, argc=6, argv=0x7fffffffde60) at perf.c:351
#15 0x00000000004bd77b in handle_internal_command (argc=6, argv=0x7fffffffde60) at perf.c:404
#16 0x00000000004bd8d4 in run_argv (argcp=0x7fffffffdc4c, argv=0x7fffffffdc40) at perf.c:448
#17 0x00000000004bdc1d in main (argc=6, argv=0x7fffffffde60) at perf.c:556
(gdb)
 
> - Arnaldo
> 
> >   $ perf report --stdio -s tgid,pid -H
> >   ...
> >   #
> >   #    Overhead  Tgid:Command / Pid:Command
> >   # ...........  ..........................
> >   #
> >      100.00%     2018407:perf
> >          50.34%     2018407:perf
> >          49.66%     2018409:perf
> > 
> > Suggested-by: Stephane Eranian <eranian@google.com>
> > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > ---
> >  tools/perf/Documentation/perf-report.txt |  1 +
> >  tools/perf/util/hist.h                   |  1 +
> >  tools/perf/util/sort.c                   | 35 ++++++++++++++++++++++++
> >  tools/perf/util/sort.h                   |  1 +
> >  4 files changed, 38 insertions(+)
> > 
> > diff --git a/tools/perf/Documentation/perf-report.txt b/tools/perf/Documentation/perf-report.txt
> > index 87f86451940623f3..4050ec4038425bf0 100644
> > --- a/tools/perf/Documentation/perf-report.txt
> > +++ b/tools/perf/Documentation/perf-report.txt
> > @@ -79,6 +79,7 @@ OPTIONS
> >  
> >  	- comm: command (name) of the task which can be read via /proc/<pid>/comm
> >  	- pid: command and tid of the task
> > +	- tgid: command and tgid of the task
> >  	- dso: name of library or module executed at the time of sample
> >  	- dso_size: size of library or module executed at the time of sample
> >  	- symbol: name of function executed at the time of sample
> > diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
> > index 46c8373e314657fa..c164e178e0a48a8e 100644
> > --- a/tools/perf/util/hist.h
> > +++ b/tools/perf/util/hist.h
> > @@ -38,6 +38,7 @@ enum hist_column {
> >  	HISTC_TIME,
> >  	HISTC_DSO,
> >  	HISTC_THREAD,
> > +	HISTC_TGID,
> >  	HISTC_COMM,
> >  	HISTC_CGROUP_ID,
> >  	HISTC_CGROUP,
> > diff --git a/tools/perf/util/sort.c b/tools/perf/util/sort.c
> > index 3dd33721823f365d..5987438174967fd6 100644
> > --- a/tools/perf/util/sort.c
> > +++ b/tools/perf/util/sort.c
> > @@ -141,6 +141,40 @@ struct sort_entry sort_thread = {
> >  	.se_width_idx	= HISTC_THREAD,
> >  };
> >  
> > +/* --sort tgid */
> > +
> > +static int64_t
> > +sort__tgid_cmp(struct hist_entry *left, struct hist_entry *right)
> > +{
> > +	return thread__pid(right->thread) - thread__pid(left->thread);
> > +}
> > +
> > +static int hist_entry__tgid_snprintf(struct hist_entry *he, char *bf,
> > +				       size_t size, unsigned int width)
> > +{
> > +	int tgid = thread__pid(he->thread);
> > +	const char *comm = NULL;
> > +
> > +	if (thread__pid(he->thread) == thread__tid(he->thread)) {
> > +		comm = thread__comm_str(he->thread);
> > +	} else {
> > +		struct maps *maps = thread__maps(he->thread);
> > +		struct thread *leader = machine__find_thread(maps__machine(maps),
> > +							     tgid, tgid);
> > +		if (leader)
> > +			comm = thread__comm_str(leader);
> > +	}
> > +	width = max(7U, width) - 8;
> > +	return repsep_snprintf(bf, size, "%7d:%-*.*s", tgid, width, width, comm ?: "");
> > +}
> > +
> > +struct sort_entry sort_tgid = {
> > +	.se_header	= "   Tgid:Command",
> > +	.se_cmp		= sort__tgid_cmp,
> > +	.se_snprintf	= hist_entry__tgid_snprintf,
> > +	.se_width_idx	= HISTC_TGID,
> > +};
> > +
> >  /* --sort simd */
> >  
> >  static int64_t
> > @@ -2501,6 +2535,7 @@ static void sort_dimension_add_dynamic_header(struct sort_dimension *sd)
> >  
> >  static struct sort_dimension common_sort_dimensions[] = {
> >  	DIM(SORT_PID, "pid", sort_thread),
> > +	DIM(SORT_TGID, "tgid", sort_tgid),
> >  	DIM(SORT_COMM, "comm", sort_comm),
> >  	DIM(SORT_DSO, "dso", sort_dso),
> >  	DIM(SORT_SYM, "symbol", sort_sym),
> > diff --git a/tools/perf/util/sort.h b/tools/perf/util/sort.h
> > index a8572574e1686be6..6044eb1d61447c0d 100644
> > --- a/tools/perf/util/sort.h
> > +++ b/tools/perf/util/sort.h
> > @@ -72,6 +72,7 @@ enum sort_type {
> >  	SORT_ANNOTATE_DATA_TYPE_OFFSET,
> >  	SORT_SYM_OFFSET,
> >  	SORT_ANNOTATE_DATA_TYPE_CACHELINE,
> > +	SORT_TGID,
> >  
> >  	/* branch stack specific sort keys */
> >  	__SORT_BRANCH_STACK,
> > -- 
> > 2.48.1.502.g6dc24dfdaf-goog

Re: [PATCH] perf report: Add 'tgid' sort key

[Ian Rogers]

On Wed, Feb 12, 2025 at 1:07 PM Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
>
> On Wed, Feb 12, 2025 at 10:05:27PM +0100, Arnaldo Carvalho de Melo wrote:
> > On Wed, Feb 05, 2025 at 04:01:37PM -0800, Namhyung Kim wrote:
> > > Sometimes we need to analyze the data in process level but current sort
> > > keys only work on thread level.  Let's add 'tgid' sort key for that as
> > > 'pid' is already taken for thread.
> > >
> > > This will look mostly the same, but it only uses tgid instead of tid.
> > > Here's an example of a process with two threads (thloop).
> > >
> > >   $ perf record -- perf test -w thloop
> >
> > Unrelated, but when building perf with DEBUG=1 and trying to test the
> > above I noticed:
> >
> > root@number:~# perf record -- perf test -w thloop
> > [ perf record: Woken up 1 times to write data ]
> > [ perf record: Captured and wrote 0.404 MB perf.data (7968 samples) ]
> > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > Aborted (core dumped)
> > root@number:~# perf record -- perf test -w offcpu
> > [ perf record: Woken up 1 times to write data ]
> > [ perf record: Captured and wrote 0.040 MB perf.data (23 samples) ]
> > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > Aborted (core dumped)
> > root@number:~#
> >
> > I have:
> >
> > ⬢ [acme@toolbox perf-tools-next]$ git log --oneline perf-tools-next/perf-tools-next..
> > 9de1ed6fa3b73cb1 (HEAD -> perf-tools-next) perf report: Add 'tgid' sort key
> > 23e98ede2a353530 perf trace: Add --summary-mode option
> > e6d6104625a3790b perf tools: Get rid of now-unused rb_resort.h
> > 173ec14e72ef4ed7 perf trace: Convert syscall_stats to hashmap
> > 66edfb5d404e743d perf trace: Allocate syscall stats only if summary is on
> > ca6637e1ea08e6f4 perf parse-events filter: Use evsel__find_pmu()
> > bd1ac4a678f7f2c8 perf bench evlist-open-close: Reduce scope of 2 variables
> > cd59081880e89df8 perf test: Add direct off-cpu test
> > 56cbd794c0c46ba9 perf record --off-cpu: Add --off-cpu-thresh option
> > 28d9b19c5455556f perf record --off-cpu: Dump the remaining samples in BPF's stack trace map
> > 2bc05b02743b50a7 perf script: Display off-cpu samples correctly
> > bfa457a621596947 perf record --off-cpu: Disable perf_event's callchain collection
> > eca732cc42d20266 perf evsel: Assemble offcpu samples
> > 74ce50e40c569e90 perf record --off-cpu: Dump off-cpu samples in BPF
> > e75f8ce63bfa6cb9 perf record --off-cpu: Preparation of off-cpu BPF program
> > 0ffab9d26971c91c perf record --off-cpu: Parse off-cpu event
> > efc3fe2070853b7d perf evsel: Expose evsel__is_offcpu_event() for future use
> > ⬢ [acme@toolbox perf-tools-next]$
> >
> > locally, that is the stuff I've been testing lately, doubt it is related
> > to these patches, I'll investigate later, have to go AFK, so FWIW as a
> > heads up.
>
> Had time to extract this, now going really AFK:
>
> [New Thread 0x7fffdf24c6c0 (LWP 580622)]
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.403 MB perf.data (7948 samples) ]
> [Thread 0x7fffdf24c6c0 (LWP 580622) exited]
> perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
>
> Thread 1 "perf" received signal SIGABRT, Aborted.
> Downloading 4.06 K source file /usr/src/debug/glibc-2.39-37.fc40.x86_64/nptl/pthread_kill.c
> __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> 44            return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
> (gdb) bt
> #0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> #1  0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78
> #2  0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> #3  0x00007ffff6e36902 in __GI_abort () at abort.c:79
> #4  0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev) <= map__end(map)",
>     file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96
> #5  0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95,
>     function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105
> #6  0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95
> #7  0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538
> #8  0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176
> #9  0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238
> #10 0x00000000006388af in perf_session__destroy_kernel_maps (session=0xf97f60) at util/session.c:105
> #11 0x0000000000638df0 in perf_session__delete (session=0xf97f60) at util/session.c:248
> #12 0x0000000000431f18 in __cmd_record (rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888
> #13 0x00000000004351fb in cmd_record (argc=4, argv=0x7fffffffde60) at builtin-record.c:4286
> #14 0x00000000004bd4d4 in run_builtin (p=0xecddc0 <commands+288>, argc=6, argv=0x7fffffffde60) at perf.c:351
> #15 0x00000000004bd77b in handle_internal_command (argc=6, argv=0x7fffffffde60) at perf.c:404
> #16 0x00000000004bd8d4 in run_argv (argcp=0x7fffffffdc4c, argv=0x7fffffffdc40) at perf.c:448
> #17 0x00000000004bdc1d in main (argc=6, argv=0x7fffffffde60) at perf.c:556
> (gdb)

So my guess would be that something modified a map and broke the
invariants of the maps_by_addresss/maps_by_name. It should be possible
to add more check_invariants to work out where this happens.

Thanks,
Ian

> > - Arnaldo
> >
> > >   $ perf report --stdio -s tgid,pid -H
> > >   ...
> > >   #
> > >   #    Overhead  Tgid:Command / Pid:Command
> > >   # ...........  ..........................
> > >   #
> > >      100.00%     2018407:perf
> > >          50.34%     2018407:perf
> > >          49.66%     2018409:perf
> > >
> > > Suggested-by: Stephane Eranian <eranian@google.com>
> > > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > > ---
> > >  tools/perf/Documentation/perf-report.txt |  1 +
> > >  tools/perf/util/hist.h                   |  1 +
> > >  tools/perf/util/sort.c                   | 35 ++++++++++++++++++++++++
> > >  tools/perf/util/sort.h                   |  1 +
> > >  4 files changed, 38 insertions(+)
> > >
> > > diff --git a/tools/perf/Documentation/perf-report.txt b/tools/perf/Documentation/perf-report.txt
> > > index 87f86451940623f3..4050ec4038425bf0 100644
> > > --- a/tools/perf/Documentation/perf-report.txt
> > > +++ b/tools/perf/Documentation/perf-report.txt
> > > @@ -79,6 +79,7 @@ OPTIONS
> > >
> > >     - comm: command (name) of the task which can be read via /proc/<pid>/comm
> > >     - pid: command and tid of the task
> > > +   - tgid: command and tgid of the task
> > >     - dso: name of library or module executed at the time of sample
> > >     - dso_size: size of library or module executed at the time of sample
> > >     - symbol: name of function executed at the time of sample
> > > diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
> > > index 46c8373e314657fa..c164e178e0a48a8e 100644
> > > --- a/tools/perf/util/hist.h
> > > +++ b/tools/perf/util/hist.h
> > > @@ -38,6 +38,7 @@ enum hist_column {
> > >     HISTC_TIME,
> > >     HISTC_DSO,
> > >     HISTC_THREAD,
> > > +   HISTC_TGID,
> > >     HISTC_COMM,
> > >     HISTC_CGROUP_ID,
> > >     HISTC_CGROUP,
> > > diff --git a/tools/perf/util/sort.c b/tools/perf/util/sort.c
> > > index 3dd33721823f365d..5987438174967fd6 100644
> > > --- a/tools/perf/util/sort.c
> > > +++ b/tools/perf/util/sort.c
> > > @@ -141,6 +141,40 @@ struct sort_entry sort_thread = {
> > >     .se_width_idx   = HISTC_THREAD,
> > >  };
> > >
> > > +/* --sort tgid */
> > > +
> > > +static int64_t
> > > +sort__tgid_cmp(struct hist_entry *left, struct hist_entry *right)
> > > +{
> > > +   return thread__pid(right->thread) - thread__pid(left->thread);
> > > +}
> > > +
> > > +static int hist_entry__tgid_snprintf(struct hist_entry *he, char *bf,
> > > +                                  size_t size, unsigned int width)
> > > +{
> > > +   int tgid = thread__pid(he->thread);
> > > +   const char *comm = NULL;
> > > +
> > > +   if (thread__pid(he->thread) == thread__tid(he->thread)) {
> > > +           comm = thread__comm_str(he->thread);
> > > +   } else {
> > > +           struct maps *maps = thread__maps(he->thread);
> > > +           struct thread *leader = machine__find_thread(maps__machine(maps),
> > > +                                                        tgid, tgid);
> > > +           if (leader)
> > > +                   comm = thread__comm_str(leader);
> > > +   }
> > > +   width = max(7U, width) - 8;
> > > +   return repsep_snprintf(bf, size, "%7d:%-*.*s", tgid, width, width, comm ?: "");
> > > +}
> > > +
> > > +struct sort_entry sort_tgid = {
> > > +   .se_header      = "   Tgid:Command",
> > > +   .se_cmp         = sort__tgid_cmp,
> > > +   .se_snprintf    = hist_entry__tgid_snprintf,
> > > +   .se_width_idx   = HISTC_TGID,
> > > +};
> > > +
> > >  /* --sort simd */
> > >
> > >  static int64_t
> > > @@ -2501,6 +2535,7 @@ static void sort_dimension_add_dynamic_header(struct sort_dimension *sd)
> > >
> > >  static struct sort_dimension common_sort_dimensions[] = {
> > >     DIM(SORT_PID, "pid", sort_thread),
> > > +   DIM(SORT_TGID, "tgid", sort_tgid),
> > >     DIM(SORT_COMM, "comm", sort_comm),
> > >     DIM(SORT_DSO, "dso", sort_dso),
> > >     DIM(SORT_SYM, "symbol", sort_sym),
> > > diff --git a/tools/perf/util/sort.h b/tools/perf/util/sort.h
> > > index a8572574e1686be6..6044eb1d61447c0d 100644
> > > --- a/tools/perf/util/sort.h
> > > +++ b/tools/perf/util/sort.h
> > > @@ -72,6 +72,7 @@ enum sort_type {
> > >     SORT_ANNOTATE_DATA_TYPE_OFFSET,
> > >     SORT_SYM_OFFSET,
> > >     SORT_ANNOTATE_DATA_TYPE_CACHELINE,
> > > +   SORT_TGID,
> > >
> > >     /* branch stack specific sort keys */
> > >     __SORT_BRANCH_STACK,
> > > --
> > > 2.48.1.502.g6dc24dfdaf-goog

Re: [PATCH] perf report: Add 'tgid' sort key

[Ian Rogers]

On Wed, Feb 12, 2025 at 1:59 PM Ian Rogers <irogers@google.com> wrote:
>
> On Wed, Feb 12, 2025 at 1:07 PM Arnaldo Carvalho de Melo
> <acme@kernel.org> wrote:
> >
> > On Wed, Feb 12, 2025 at 10:05:27PM +0100, Arnaldo Carvalho de Melo wrote:
> > > On Wed, Feb 05, 2025 at 04:01:37PM -0800, Namhyung Kim wrote:
> > > > Sometimes we need to analyze the data in process level but current sort
> > > > keys only work on thread level.  Let's add 'tgid' sort key for that as
> > > > 'pid' is already taken for thread.
> > > >
> > > > This will look mostly the same, but it only uses tgid instead of tid.
> > > > Here's an example of a process with two threads (thloop).
> > > >
> > > >   $ perf record -- perf test -w thloop
> > >
> > > Unrelated, but when building perf with DEBUG=1 and trying to test the
> > > above I noticed:
> > >
> > > root@number:~# perf record -- perf test -w thloop
> > > [ perf record: Woken up 1 times to write data ]
> > > [ perf record: Captured and wrote 0.404 MB perf.data (7968 samples) ]
> > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > > Aborted (core dumped)
> > > root@number:~# perf record -- perf test -w offcpu
> > > [ perf record: Woken up 1 times to write data ]
> > > [ perf record: Captured and wrote 0.040 MB perf.data (23 samples) ]
> > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > > Aborted (core dumped)
> > > root@number:~#
> > >
> > > I have:
> > >
> > > ⬢ [acme@toolbox perf-tools-next]$ git log --oneline perf-tools-next/perf-tools-next..
> > > 9de1ed6fa3b73cb1 (HEAD -> perf-tools-next) perf report: Add 'tgid' sort key
> > > 23e98ede2a353530 perf trace: Add --summary-mode option
> > > e6d6104625a3790b perf tools: Get rid of now-unused rb_resort.h
> > > 173ec14e72ef4ed7 perf trace: Convert syscall_stats to hashmap
> > > 66edfb5d404e743d perf trace: Allocate syscall stats only if summary is on
> > > ca6637e1ea08e6f4 perf parse-events filter: Use evsel__find_pmu()
> > > bd1ac4a678f7f2c8 perf bench evlist-open-close: Reduce scope of 2 variables
> > > cd59081880e89df8 perf test: Add direct off-cpu test
> > > 56cbd794c0c46ba9 perf record --off-cpu: Add --off-cpu-thresh option
> > > 28d9b19c5455556f perf record --off-cpu: Dump the remaining samples in BPF's stack trace map
> > > 2bc05b02743b50a7 perf script: Display off-cpu samples correctly
> > > bfa457a621596947 perf record --off-cpu: Disable perf_event's callchain collection
> > > eca732cc42d20266 perf evsel: Assemble offcpu samples
> > > 74ce50e40c569e90 perf record --off-cpu: Dump off-cpu samples in BPF
> > > e75f8ce63bfa6cb9 perf record --off-cpu: Preparation of off-cpu BPF program
> > > 0ffab9d26971c91c perf record --off-cpu: Parse off-cpu event
> > > efc3fe2070853b7d perf evsel: Expose evsel__is_offcpu_event() for future use
> > > ⬢ [acme@toolbox perf-tools-next]$
> > >
> > > locally, that is the stuff I've been testing lately, doubt it is related
> > > to these patches, I'll investigate later, have to go AFK, so FWIW as a
> > > heads up.
> >
> > Had time to extract this, now going really AFK:
> >
> > [New Thread 0x7fffdf24c6c0 (LWP 580622)]
> > [ perf record: Woken up 1 times to write data ]
> > [ perf record: Captured and wrote 0.403 MB perf.data (7948 samples) ]
> > [Thread 0x7fffdf24c6c0 (LWP 580622) exited]
> > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> >
> > Thread 1 "perf" received signal SIGABRT, Aborted.
> > Downloading 4.06 K source file /usr/src/debug/glibc-2.39-37.fc40.x86_64/nptl/pthread_kill.c
> > __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > 44            return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
> > (gdb) bt
> > #0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > #1  0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78
> > #2  0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> > #3  0x00007ffff6e36902 in __GI_abort () at abort.c:79
> > #4  0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev) <= map__end(map)",
> >     file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96
> > #5  0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95,
> >     function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105
> > #6  0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95
> > #7  0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538
> > #8  0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176
> > #9  0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238
> > #10 0x00000000006388af in perf_session__destroy_kernel_maps (session=0xf97f60) at util/session.c:105
> > #11 0x0000000000638df0 in perf_session__delete (session=0xf97f60) at util/session.c:248
> > #12 0x0000000000431f18 in __cmd_record (rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888
> > #13 0x00000000004351fb in cmd_record (argc=4, argv=0x7fffffffde60) at builtin-record.c:4286
> > #14 0x00000000004bd4d4 in run_builtin (p=0xecddc0 <commands+288>, argc=6, argv=0x7fffffffde60) at perf.c:351
> > #15 0x00000000004bd77b in handle_internal_command (argc=6, argv=0x7fffffffde60) at perf.c:404
> > #16 0x00000000004bd8d4 in run_argv (argcp=0x7fffffffdc4c, argv=0x7fffffffdc40) at perf.c:448
> > #17 0x00000000004bdc1d in main (argc=6, argv=0x7fffffffde60) at perf.c:556
> > (gdb)
>
> So my guess would be that something modified a map and broke the
> invariants of the maps_by_addresss/maps_by_name. It should be possible
> to add more check_invariants to work out where this happens.
>
> Thanks,
> Ian

I also suspect this is a regression. If you could bisect to find the
cause then the fix is probably to not modify a map but clone it,
change it and then reinsert it into the maps - the insert is called
maps__fixup_overlap_and_insert so that maps don't overlap one another
like the invariant check is detecting. Fwiw, in the older rbtree code,
invariant breakages like this would be silently ignored, so we may
have a latent bug :-(

Thanks,
Ian

> > > - Arnaldo
> > >
> > > >   $ perf report --stdio -s tgid,pid -H
> > > >   ...
> > > >   #
> > > >   #    Overhead  Tgid:Command / Pid:Command
> > > >   # ...........  ..........................
> > > >   #
> > > >      100.00%     2018407:perf
> > > >          50.34%     2018407:perf
> > > >          49.66%     2018409:perf
> > > >
> > > > Suggested-by: Stephane Eranian <eranian@google.com>
> > > > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > > > ---
> > > >  tools/perf/Documentation/perf-report.txt |  1 +
> > > >  tools/perf/util/hist.h                   |  1 +
> > > >  tools/perf/util/sort.c                   | 35 ++++++++++++++++++++++++
> > > >  tools/perf/util/sort.h                   |  1 +
> > > >  4 files changed, 38 insertions(+)
> > > >
> > > > diff --git a/tools/perf/Documentation/perf-report.txt b/tools/perf/Documentation/perf-report.txt
> > > > index 87f86451940623f3..4050ec4038425bf0 100644
> > > > --- a/tools/perf/Documentation/perf-report.txt
> > > > +++ b/tools/perf/Documentation/perf-report.txt
> > > > @@ -79,6 +79,7 @@ OPTIONS
> > > >
> > > >     - comm: command (name) of the task which can be read via /proc/<pid>/comm
> > > >     - pid: command and tid of the task
> > > > +   - tgid: command and tgid of the task
> > > >     - dso: name of library or module executed at the time of sample
> > > >     - dso_size: size of library or module executed at the time of sample
> > > >     - symbol: name of function executed at the time of sample
> > > > diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
> > > > index 46c8373e314657fa..c164e178e0a48a8e 100644
> > > > --- a/tools/perf/util/hist.h
> > > > +++ b/tools/perf/util/hist.h
> > > > @@ -38,6 +38,7 @@ enum hist_column {
> > > >     HISTC_TIME,
> > > >     HISTC_DSO,
> > > >     HISTC_THREAD,
> > > > +   HISTC_TGID,
> > > >     HISTC_COMM,
> > > >     HISTC_CGROUP_ID,
> > > >     HISTC_CGROUP,
> > > > diff --git a/tools/perf/util/sort.c b/tools/perf/util/sort.c
> > > > index 3dd33721823f365d..5987438174967fd6 100644
> > > > --- a/tools/perf/util/sort.c
> > > > +++ b/tools/perf/util/sort.c
> > > > @@ -141,6 +141,40 @@ struct sort_entry sort_thread = {
> > > >     .se_width_idx   = HISTC_THREAD,
> > > >  };
> > > >
> > > > +/* --sort tgid */
> > > > +
> > > > +static int64_t
> > > > +sort__tgid_cmp(struct hist_entry *left, struct hist_entry *right)
> > > > +{
> > > > +   return thread__pid(right->thread) - thread__pid(left->thread);
> > > > +}
> > > > +
> > > > +static int hist_entry__tgid_snprintf(struct hist_entry *he, char *bf,
> > > > +                                  size_t size, unsigned int width)
> > > > +{
> > > > +   int tgid = thread__pid(he->thread);
> > > > +   const char *comm = NULL;
> > > > +
> > > > +   if (thread__pid(he->thread) == thread__tid(he->thread)) {
> > > > +           comm = thread__comm_str(he->thread);
> > > > +   } else {
> > > > +           struct maps *maps = thread__maps(he->thread);
> > > > +           struct thread *leader = machine__find_thread(maps__machine(maps),
> > > > +                                                        tgid, tgid);
> > > > +           if (leader)
> > > > +                   comm = thread__comm_str(leader);
> > > > +   }
> > > > +   width = max(7U, width) - 8;
> > > > +   return repsep_snprintf(bf, size, "%7d:%-*.*s", tgid, width, width, comm ?: "");
> > > > +}
> > > > +
> > > > +struct sort_entry sort_tgid = {
> > > > +   .se_header      = "   Tgid:Command",
> > > > +   .se_cmp         = sort__tgid_cmp,
> > > > +   .se_snprintf    = hist_entry__tgid_snprintf,
> > > > +   .se_width_idx   = HISTC_TGID,
> > > > +};
> > > > +
> > > >  /* --sort simd */
> > > >
> > > >  static int64_t
> > > > @@ -2501,6 +2535,7 @@ static void sort_dimension_add_dynamic_header(struct sort_dimension *sd)
> > > >
> > > >  static struct sort_dimension common_sort_dimensions[] = {
> > > >     DIM(SORT_PID, "pid", sort_thread),
> > > > +   DIM(SORT_TGID, "tgid", sort_tgid),
> > > >     DIM(SORT_COMM, "comm", sort_comm),
> > > >     DIM(SORT_DSO, "dso", sort_dso),
> > > >     DIM(SORT_SYM, "symbol", sort_sym),
> > > > diff --git a/tools/perf/util/sort.h b/tools/perf/util/sort.h
> > > > index a8572574e1686be6..6044eb1d61447c0d 100644
> > > > --- a/tools/perf/util/sort.h
> > > > +++ b/tools/perf/util/sort.h
> > > > @@ -72,6 +72,7 @@ enum sort_type {
> > > >     SORT_ANNOTATE_DATA_TYPE_OFFSET,
> > > >     SORT_SYM_OFFSET,
> > > >     SORT_ANNOTATE_DATA_TYPE_CACHELINE,
> > > > +   SORT_TGID,
> > > >
> > > >     /* branch stack specific sort keys */
> > > >     __SORT_BRANCH_STACK,
> > > > --
> > > > 2.48.1.502.g6dc24dfdaf-goog

Re: [PATCH] perf report: Add 'tgid' sort key

[Namhyung Kim]

On Wed, Feb 12, 2025 at 02:10:42PM -0800, Ian Rogers wrote:
> On Wed, Feb 12, 2025 at 1:59 PM Ian Rogers <irogers@google.com> wrote:
> >
> > On Wed, Feb 12, 2025 at 1:07 PM Arnaldo Carvalho de Melo
> > <acme@kernel.org> wrote:
> > >
> > > On Wed, Feb 12, 2025 at 10:05:27PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > On Wed, Feb 05, 2025 at 04:01:37PM -0800, Namhyung Kim wrote:
> > > > > Sometimes we need to analyze the data in process level but current sort
> > > > > keys only work on thread level.  Let's add 'tgid' sort key for that as
> > > > > 'pid' is already taken for thread.
> > > > >
> > > > > This will look mostly the same, but it only uses tgid instead of tid.
> > > > > Here's an example of a process with two threads (thloop).
> > > > >
> > > > >   $ perf record -- perf test -w thloop
> > > >
> > > > Unrelated, but when building perf with DEBUG=1 and trying to test the
> > > > above I noticed:
> > > >
> > > > root@number:~# perf record -- perf test -w thloop
> > > > [ perf record: Woken up 1 times to write data ]
> > > > [ perf record: Captured and wrote 0.404 MB perf.data (7968 samples) ]
> > > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > > > Aborted (core dumped)
> > > > root@number:~# perf record -- perf test -w offcpu
> > > > [ perf record: Woken up 1 times to write data ]
> > > > [ perf record: Captured and wrote 0.040 MB perf.data (23 samples) ]
> > > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > > > Aborted (core dumped)
> > > > root@number:~#
> > > >
> > > > I have:
> > > >
> > > > ⬢ [acme@toolbox perf-tools-next]$ git log --oneline perf-tools-next/perf-tools-next..
> > > > 9de1ed6fa3b73cb1 (HEAD -> perf-tools-next) perf report: Add 'tgid' sort key
> > > > 23e98ede2a353530 perf trace: Add --summary-mode option
> > > > e6d6104625a3790b perf tools: Get rid of now-unused rb_resort.h
> > > > 173ec14e72ef4ed7 perf trace: Convert syscall_stats to hashmap
> > > > 66edfb5d404e743d perf trace: Allocate syscall stats only if summary is on
> > > > ca6637e1ea08e6f4 perf parse-events filter: Use evsel__find_pmu()
> > > > bd1ac4a678f7f2c8 perf bench evlist-open-close: Reduce scope of 2 variables
> > > > cd59081880e89df8 perf test: Add direct off-cpu test
> > > > 56cbd794c0c46ba9 perf record --off-cpu: Add --off-cpu-thresh option
> > > > 28d9b19c5455556f perf record --off-cpu: Dump the remaining samples in BPF's stack trace map
> > > > 2bc05b02743b50a7 perf script: Display off-cpu samples correctly
> > > > bfa457a621596947 perf record --off-cpu: Disable perf_event's callchain collection
> > > > eca732cc42d20266 perf evsel: Assemble offcpu samples
> > > > 74ce50e40c569e90 perf record --off-cpu: Dump off-cpu samples in BPF
> > > > e75f8ce63bfa6cb9 perf record --off-cpu: Preparation of off-cpu BPF program
> > > > 0ffab9d26971c91c perf record --off-cpu: Parse off-cpu event
> > > > efc3fe2070853b7d perf evsel: Expose evsel__is_offcpu_event() for future use
> > > > ⬢ [acme@toolbox perf-tools-next]$
> > > >
> > > > locally, that is the stuff I've been testing lately, doubt it is related
> > > > to these patches, I'll investigate later, have to go AFK, so FWIW as a
> > > > heads up.
> > >
> > > Had time to extract this, now going really AFK:
> > >
> > > [New Thread 0x7fffdf24c6c0 (LWP 580622)]
> > > [ perf record: Woken up 1 times to write data ]
> > > [ perf record: Captured and wrote 0.403 MB perf.data (7948 samples) ]
> > > [Thread 0x7fffdf24c6c0 (LWP 580622) exited]
> > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > >
> > > Thread 1 "perf" received signal SIGABRT, Aborted.
> > > Downloading 4.06 K source file /usr/src/debug/glibc-2.39-37.fc40.x86_64/nptl/pthread_kill.c
> > > __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > > 44            return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
> > > (gdb) bt
> > > #0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > > #1  0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78
> > > #2  0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> > > #3  0x00007ffff6e36902 in __GI_abort () at abort.c:79
> > > #4  0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev) <= map__end(map)",
> > >     file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96
> > > #5  0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95,
> > >     function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105
> > > #6  0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95
> > > #7  0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538
> > > #8  0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176
> > > #9  0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238
> > > #10 0x00000000006388af in perf_session__destroy_kernel_maps (session=0xf97f60) at util/session.c:105
> > > #11 0x0000000000638df0 in perf_session__delete (session=0xf97f60) at util/session.c:248
> > > #12 0x0000000000431f18 in __cmd_record (rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888
> > > #13 0x00000000004351fb in cmd_record (argc=4, argv=0x7fffffffde60) at builtin-record.c:4286
> > > #14 0x00000000004bd4d4 in run_builtin (p=0xecddc0 <commands+288>, argc=6, argv=0x7fffffffde60) at perf.c:351
> > > #15 0x00000000004bd77b in handle_internal_command (argc=6, argv=0x7fffffffde60) at perf.c:404
> > > #16 0x00000000004bd8d4 in run_argv (argcp=0x7fffffffdc4c, argv=0x7fffffffdc40) at perf.c:448
> > > #17 0x00000000004bdc1d in main (argc=6, argv=0x7fffffffde60) at perf.c:556
> > > (gdb)
> >
> > So my guess would be that something modified a map and broke the
> > invariants of the maps_by_addresss/maps_by_name. It should be possible
> > to add more check_invariants to work out where this happens.
> >
> > Thanks,
> > Ian
> 
> I also suspect this is a regression. If you could bisect to find the
> cause then the fix is probably to not modify a map but clone it,
> change it and then reinsert it into the maps - the insert is called
> maps__fixup_overlap_and_insert so that maps don't overlap one another
> like the invariant check is detecting. Fwiw, in the older rbtree code,
> invariant breakages like this would be silently ignored, so we may
> have a latent bug :-(

Sorry, I also cannot reproduce it on my machine.  But I think it's
unrelated to this change since you saw it during record.  It'd be nice
if you could bisect.

And I think I forgot to call thread__put() for the leader thread. :)
Will update in v2.

Thanks,
Namhyung

Re: [PATCH] perf report: Add 'tgid' sort key

[Arnaldo Carvalho de Melo]

On Wed, Feb 12, 2025 at 02:10:42PM -0800, Ian Rogers wrote:
> On Wed, Feb 12, 2025 at 1:59 PM Ian Rogers <irogers@google.com> wrote:
> > On Wed, Feb 12, 2025 at 1:07 PM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > > On Wed, Feb 12, 2025 at 10:05:27PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > On Wed, Feb 05, 2025 at 04:01:37PM -0800, Namhyung Kim wrote:
> > > > > Sometimes we need to analyze the data in process level but current sort
> > > > > keys only work on thread level.  Let's add 'tgid' sort key for that as
> > > > > 'pid' is already taken for thread.

> > > > > This will look mostly the same, but it only uses tgid instead of tid.
> > > > > Here's an example of a process with two threads (thloop).

> > > > >   $ perf record -- perf test -w thloop

> > > > Unrelated, but when building perf with DEBUG=1 and trying to test the
> > > > above I noticed:

> > > > root@number:~# perf record -- perf test -w thloop
> > > > [ perf record: Woken up 1 times to write data ]
> > > > [ perf record: Captured and wrote 0.404 MB perf.data (7968 samples) ]
> > > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > > > Aborted (core dumped)
> > > > root@number:~# perf record -- perf test -w offcpu
> > > > [ perf record: Woken up 1 times to write data ]
> > > > [ perf record: Captured and wrote 0.040 MB perf.data (23 samples) ]
> > > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > > > Aborted (core dumped)
> > > > root@number:~#

> > > > I have:

> > > > ⬢ [acme@toolbox perf-tools-next]$ git log --oneline perf-tools-next/perf-tools-next..
> > > > 9de1ed6fa3b73cb1 (HEAD -> perf-tools-next) perf report: Add 'tgid' sort key
> > > > 23e98ede2a353530 perf trace: Add --summary-mode option
> > > > e6d6104625a3790b perf tools: Get rid of now-unused rb_resort.h
> > > > 173ec14e72ef4ed7 perf trace: Convert syscall_stats to hashmap
> > > > 66edfb5d404e743d perf trace: Allocate syscall stats only if summary is on
> > > > ca6637e1ea08e6f4 perf parse-events filter: Use evsel__find_pmu()
> > > > bd1ac4a678f7f2c8 perf bench evlist-open-close: Reduce scope of 2 variables
> > > > cd59081880e89df8 perf test: Add direct off-cpu test
> > > > 56cbd794c0c46ba9 perf record --off-cpu: Add --off-cpu-thresh option
> > > > 28d9b19c5455556f perf record --off-cpu: Dump the remaining samples in BPF's stack trace map
> > > > 2bc05b02743b50a7 perf script: Display off-cpu samples correctly
> > > > bfa457a621596947 perf record --off-cpu: Disable perf_event's callchain collection
> > > > eca732cc42d20266 perf evsel: Assemble offcpu samples
> > > > 74ce50e40c569e90 perf record --off-cpu: Dump off-cpu samples in BPF
> > > > e75f8ce63bfa6cb9 perf record --off-cpu: Preparation of off-cpu BPF program
> > > > 0ffab9d26971c91c perf record --off-cpu: Parse off-cpu event
> > > > efc3fe2070853b7d perf evsel: Expose evsel__is_offcpu_event() for future use
> > > > ⬢ [acme@toolbox perf-tools-next]$

> > > > locally, that is the stuff I've been testing lately, doubt it is related
> > > > to these patches, I'll investigate later, have to go AFK, so FWIW as a
> > > > heads up.

> > > Had time to extract this, now going really AFK:

> > > [New Thread 0x7fffdf24c6c0 (LWP 580622)]
> > > [ perf record: Woken up 1 times to write data ]
> > > [ perf record: Captured and wrote 0.403 MB perf.data (7948 samples) ]
> > > [Thread 0x7fffdf24c6c0 (LWP 580622) exited]
> > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.

> > > Thread 1 "perf" received signal SIGABRT, Aborted.
> > > Downloading 4.06 K source file /usr/src/debug/glibc-2.39-37.fc40.x86_64/nptl/pthread_kill.c
> > > __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > > 44            return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
> > > (gdb) bt
> > > #0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > > #1  0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78
> > > #2  0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> > > #3  0x00007ffff6e36902 in __GI_abort () at abort.c:79
> > > #4  0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev) <= map__end(map)",
> > >     file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96
> > > #5  0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95,
> > >     function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105
> > > #6  0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95
> > > #7  0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538
> > > #8  0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176
> > > #9  0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238
> > > #10 0x00000000006388af in perf_session__destroy_kernel_maps (session=0xf97f60) at util/session.c:105
> > > #11 0x0000000000638df0 in perf_session__delete (session=0xf97f60) at util/session.c:248
> > > #12 0x0000000000431f18 in __cmd_record (rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888
> > > #13 0x00000000004351fb in cmd_record (argc=4, argv=0x7fffffffde60) at builtin-record.c:4286
> > > #14 0x00000000004bd4d4 in run_builtin (p=0xecddc0 <commands+288>, argc=6, argv=0x7fffffffde60) at perf.c:351
> > > #15 0x00000000004bd77b in handle_internal_command (argc=6, argv=0x7fffffffde60) at perf.c:404
> > > #16 0x00000000004bd8d4 in run_argv (argcp=0x7fffffffdc4c, argv=0x7fffffffdc40) at perf.c:448
> > > #17 0x00000000004bdc1d in main (argc=6, argv=0x7fffffffde60) at perf.c:556
> > > (gdb)

> > So my guess would be that something modified a map and broke the
> > invariants of the maps_by_addresss/maps_by_name. It should be possible
> > to add more check_invariants to work out where this happens.
 
> I also suspect this is a regression. If you could bisect to find the

I bisected it to:

⬢ [acme@toolbox perf-tools-next]$ git bisect good
876e80cf83d10585df6ee1e353cfbf562f9a930e is the first bad commit
commit 876e80cf83d10585df6ee1e353cfbf562f9a930e
Author: Namhyung Kim <namhyung@kernel.org>
Date:   Wed Dec 18 14:04:53 2024 -0800

    perf tools: Fixup end address of modules
    
    In machine__create_module(), it reads /proc/modules to get a list of
    modules in the system.  The file shows the start address (of text) and
    the size of the module so it uses the info to reconstruct system memory
    maps for symbol resolution.
    
    But module memory consists of multiple segments and they can be
    scaterred.  Currently perf tools assume they are contiguous and see some
    overlaps.  This can confuse the tool when it finds a map containing a
    given address.
    
    As we mostly care about the function symbols in the text segment, it can
    fixup the size or end address of modules when there's an overlap.  We
    can use maps__fixup_end() which updates the end address using the start
    address of the next map.
    
    Ideally it should be able to track other segments (like data/rodata),
    but that would require some changes in /proc/modules IMHO.
    
    Reported-by: Blake Jones <blakejones@google.com>
    Signed-off-by: Namhyung Kim <namhyung@kernel.org>
    Acked-by: Ian Rogers <irogers@google.com>
    Cc: Adrian Hunter <adrian.hunter@intel.com>
    Cc: Daniel Gomez <da.gomez@samsung.com>
    Cc: Ingo Molnar <mingo@kernel.org>
    Cc: Jiri Olsa <jolsa@kernel.org>
    Cc: Kan Liang <kan.liang@linux.intel.com>
    Cc: Luis Chamberlain <mcgrof@kernel.org>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Petr Pavlu <petr.pavlu@suse.com>
    Cc: Sami Tolvanen <samitolvanen@google.com>
    Link: https://lore.kernel.org/r/20241218220453.203069-1-namhyung@kernel.org
    Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>

 tools/perf/util/machine.c | 2 ++
 1 file changed, 2 insertions(+)
⬢ [acme@toolbox perf-tools-next]$

If we simply revert this it gets back working:

⬢ [acme@toolbox perf-tools-next]$ git revert 876e80cf83d10585df6ee1e353cfbf562f9a930e
Auto-merging tools/perf/util/machine.c
[perf-tools-next 1ab31115859a0944] Revert "perf tools: Fixup end address of modules"
 1 file changed, 2 deletions(-)
# rm -rf build dir, rebuild it

root@number:~# perf record -- perf test -w thloop
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.403 MB perf.data (7960 samples) ]
root@number:~# 

No time today to try to dig deeper, so just reporting the bisection
result.

- Arnaldo

> cause then the fix is probably to not modify a map but clone it,
> change it and then reinsert it into the maps - the insert is called
> maps__fixup_overlap_and_insert so that maps don't overlap one another
> like the invariant check is detecting. Fwiw, in the older rbtree code,
> invariant breakages like this would be silently ignored, so we may
> have a latent bug :-(
> 
> Thanks,
> Ian
> 
> > > > - Arnaldo
> > > >
> > > > >   $ perf report --stdio -s tgid,pid -H
> > > > >   ...
> > > > >   #
> > > > >   #    Overhead  Tgid:Command / Pid:Command
> > > > >   # ...........  ..........................
> > > > >   #
> > > > >      100.00%     2018407:perf
> > > > >          50.34%     2018407:perf
> > > > >          49.66%     2018409:perf
> > > > >
> > > > > Suggested-by: Stephane Eranian <eranian@google.com>
> > > > > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > > > > ---
> > > > >  tools/perf/Documentation/perf-report.txt |  1 +
> > > > >  tools/perf/util/hist.h                   |  1 +
> > > > >  tools/perf/util/sort.c                   | 35 ++++++++++++++++++++++++
> > > > >  tools/perf/util/sort.h                   |  1 +
> > > > >  4 files changed, 38 insertions(+)
> > > > >
> > > > > diff --git a/tools/perf/Documentation/perf-report.txt b/tools/perf/Documentation/perf-report.txt
> > > > > index 87f86451940623f3..4050ec4038425bf0 100644
> > > > > --- a/tools/perf/Documentation/perf-report.txt
> > > > > +++ b/tools/perf/Documentation/perf-report.txt
> > > > > @@ -79,6 +79,7 @@ OPTIONS
> > > > >
> > > > >     - comm: command (name) of the task which can be read via /proc/<pid>/comm
> > > > >     - pid: command and tid of the task
> > > > > +   - tgid: command and tgid of the task
> > > > >     - dso: name of library or module executed at the time of sample
> > > > >     - dso_size: size of library or module executed at the time of sample
> > > > >     - symbol: name of function executed at the time of sample
> > > > > diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
> > > > > index 46c8373e314657fa..c164e178e0a48a8e 100644
> > > > > --- a/tools/perf/util/hist.h
> > > > > +++ b/tools/perf/util/hist.h
> > > > > @@ -38,6 +38,7 @@ enum hist_column {
> > > > >     HISTC_TIME,
> > > > >     HISTC_DSO,
> > > > >     HISTC_THREAD,
> > > > > +   HISTC_TGID,
> > > > >     HISTC_COMM,
> > > > >     HISTC_CGROUP_ID,
> > > > >     HISTC_CGROUP,
> > > > > diff --git a/tools/perf/util/sort.c b/tools/perf/util/sort.c
> > > > > index 3dd33721823f365d..5987438174967fd6 100644
> > > > > --- a/tools/perf/util/sort.c
> > > > > +++ b/tools/perf/util/sort.c
> > > > > @@ -141,6 +141,40 @@ struct sort_entry sort_thread = {
> > > > >     .se_width_idx   = HISTC_THREAD,
> > > > >  };
> > > > >
> > > > > +/* --sort tgid */
> > > > > +
> > > > > +static int64_t
> > > > > +sort__tgid_cmp(struct hist_entry *left, struct hist_entry *right)
> > > > > +{
> > > > > +   return thread__pid(right->thread) - thread__pid(left->thread);
> > > > > +}
> > > > > +
> > > > > +static int hist_entry__tgid_snprintf(struct hist_entry *he, char *bf,
> > > > > +                                  size_t size, unsigned int width)
> > > > > +{
> > > > > +   int tgid = thread__pid(he->thread);
> > > > > +   const char *comm = NULL;
> > > > > +
> > > > > +   if (thread__pid(he->thread) == thread__tid(he->thread)) {
> > > > > +           comm = thread__comm_str(he->thread);
> > > > > +   } else {
> > > > > +           struct maps *maps = thread__maps(he->thread);
> > > > > +           struct thread *leader = machine__find_thread(maps__machine(maps),
> > > > > +                                                        tgid, tgid);
> > > > > +           if (leader)
> > > > > +                   comm = thread__comm_str(leader);
> > > > > +   }
> > > > > +   width = max(7U, width) - 8;
> > > > > +   return repsep_snprintf(bf, size, "%7d:%-*.*s", tgid, width, width, comm ?: "");
> > > > > +}
> > > > > +
> > > > > +struct sort_entry sort_tgid = {
> > > > > +   .se_header      = "   Tgid:Command",
> > > > > +   .se_cmp         = sort__tgid_cmp,
> > > > > +   .se_snprintf    = hist_entry__tgid_snprintf,
> > > > > +   .se_width_idx   = HISTC_TGID,
> > > > > +};
> > > > > +
> > > > >  /* --sort simd */
> > > > >
> > > > >  static int64_t
> > > > > @@ -2501,6 +2535,7 @@ static void sort_dimension_add_dynamic_header(struct sort_dimension *sd)
> > > > >
> > > > >  static struct sort_dimension common_sort_dimensions[] = {
> > > > >     DIM(SORT_PID, "pid", sort_thread),
> > > > > +   DIM(SORT_TGID, "tgid", sort_tgid),
> > > > >     DIM(SORT_COMM, "comm", sort_comm),
> > > > >     DIM(SORT_DSO, "dso", sort_dso),
> > > > >     DIM(SORT_SYM, "symbol", sort_sym),
> > > > > diff --git a/tools/perf/util/sort.h b/tools/perf/util/sort.h
> > > > > index a8572574e1686be6..6044eb1d61447c0d 100644
> > > > > --- a/tools/perf/util/sort.h
> > > > > +++ b/tools/perf/util/sort.h
> > > > > @@ -72,6 +72,7 @@ enum sort_type {
> > > > >     SORT_ANNOTATE_DATA_TYPE_OFFSET,
> > > > >     SORT_SYM_OFFSET,
> > > > >     SORT_ANNOTATE_DATA_TYPE_CACHELINE,
> > > > > +   SORT_TGID,
> > > > >
> > > > >     /* branch stack specific sort keys */
> > > > >     __SORT_BRANCH_STACK,
> > > > > --
> > > > > 2.48.1.502.g6dc24dfdaf-goog

Re: [PATCH] perf report: Add 'tgid' sort key

[Arnaldo Carvalho de Melo]

On Fri, Feb 14, 2025 at 11:22:39PM +0100, Arnaldo Carvalho de Melo wrote:
> On Wed, Feb 12, 2025 at 02:10:42PM -0800, Ian Rogers wrote:
> > On Wed, Feb 12, 2025 at 1:59 PM Ian Rogers <irogers@google.com> wrote:
> > > On Wed, Feb 12, 2025 at 1:07 PM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > > > On Wed, Feb 12, 2025 at 10:05:27PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > On Wed, Feb 05, 2025 at 04:01:37PM -0800, Namhyung Kim wrote:
> > > > > > Sometimes we need to analyze the data in process level but current sort
> > > > > > keys only work on thread level.  Let's add 'tgid' sort key for that as
> > > > > > 'pid' is already taken for thread.
> 
> > > > > > This will look mostly the same, but it only uses tgid instead of tid.
> > > > > > Here's an example of a process with two threads (thloop).
> 
> > > > > >   $ perf record -- perf test -w thloop
> 
> > > > > Unrelated, but when building perf with DEBUG=1 and trying to test the
> > > > > above I noticed:
> 
> > > > > root@number:~# perf record -- perf test -w thloop
> > > > > [ perf record: Woken up 1 times to write data ]
> > > > > [ perf record: Captured and wrote 0.404 MB perf.data (7968 samples) ]
> > > > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > > > > Aborted (core dumped)
> > > > > root@number:~# perf record -- perf test -w offcpu
> > > > > [ perf record: Woken up 1 times to write data ]
> > > > > [ perf record: Captured and wrote 0.040 MB perf.data (23 samples) ]
> > > > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > > > > Aborted (core dumped)
> > > > > root@number:~#
> 
> > > > > I have:
> 
> > > > > ⬢ [acme@toolbox perf-tools-next]$ git log --oneline perf-tools-next/perf-tools-next..
> > > > > 9de1ed6fa3b73cb1 (HEAD -> perf-tools-next) perf report: Add 'tgid' sort key
> > > > > 23e98ede2a353530 perf trace: Add --summary-mode option
> > > > > e6d6104625a3790b perf tools: Get rid of now-unused rb_resort.h
> > > > > 173ec14e72ef4ed7 perf trace: Convert syscall_stats to hashmap
> > > > > 66edfb5d404e743d perf trace: Allocate syscall stats only if summary is on
> > > > > ca6637e1ea08e6f4 perf parse-events filter: Use evsel__find_pmu()
> > > > > bd1ac4a678f7f2c8 perf bench evlist-open-close: Reduce scope of 2 variables
> > > > > cd59081880e89df8 perf test: Add direct off-cpu test
> > > > > 56cbd794c0c46ba9 perf record --off-cpu: Add --off-cpu-thresh option
> > > > > 28d9b19c5455556f perf record --off-cpu: Dump the remaining samples in BPF's stack trace map
> > > > > 2bc05b02743b50a7 perf script: Display off-cpu samples correctly
> > > > > bfa457a621596947 perf record --off-cpu: Disable perf_event's callchain collection
> > > > > eca732cc42d20266 perf evsel: Assemble offcpu samples
> > > > > 74ce50e40c569e90 perf record --off-cpu: Dump off-cpu samples in BPF
> > > > > e75f8ce63bfa6cb9 perf record --off-cpu: Preparation of off-cpu BPF program
> > > > > 0ffab9d26971c91c perf record --off-cpu: Parse off-cpu event
> > > > > efc3fe2070853b7d perf evsel: Expose evsel__is_offcpu_event() for future use
> > > > > ⬢ [acme@toolbox perf-tools-next]$
> 
> > > > > locally, that is the stuff I've been testing lately, doubt it is related
> > > > > to these patches, I'll investigate later, have to go AFK, so FWIW as a
> > > > > heads up.
> 
> > > > Had time to extract this, now going really AFK:
> 
> > > > [New Thread 0x7fffdf24c6c0 (LWP 580622)]
> > > > [ perf record: Woken up 1 times to write data ]
> > > > [ perf record: Captured and wrote 0.403 MB perf.data (7948 samples) ]
> > > > [Thread 0x7fffdf24c6c0 (LWP 580622) exited]
> > > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> 
> > > > Thread 1 "perf" received signal SIGABRT, Aborted.
> > > > Downloading 4.06 K source file /usr/src/debug/glibc-2.39-37.fc40.x86_64/nptl/pthread_kill.c
> > > > __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > > > 44            return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
> > > > (gdb) bt
> > > > #0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> > > > #1  0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78
> > > > #2  0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> > > > #3  0x00007ffff6e36902 in __GI_abort () at abort.c:79
> > > > #4  0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev) <= map__end(map)",
> > > >     file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96
> > > > #5  0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95,
> > > >     function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105
> > > > #6  0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95
> > > > #7  0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538
> > > > #8  0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176
> > > > #9  0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238
> > > > #10 0x00000000006388af in perf_session__destroy_kernel_maps (session=0xf97f60) at util/session.c:105
> > > > #11 0x0000000000638df0 in perf_session__delete (session=0xf97f60) at util/session.c:248
> > > > #12 0x0000000000431f18 in __cmd_record (rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888
> > > > #13 0x00000000004351fb in cmd_record (argc=4, argv=0x7fffffffde60) at builtin-record.c:4286
> > > > #14 0x00000000004bd4d4 in run_builtin (p=0xecddc0 <commands+288>, argc=6, argv=0x7fffffffde60) at perf.c:351
> > > > #15 0x00000000004bd77b in handle_internal_command (argc=6, argv=0x7fffffffde60) at perf.c:404
> > > > #16 0x00000000004bd8d4 in run_argv (argcp=0x7fffffffdc4c, argv=0x7fffffffdc40) at perf.c:448
> > > > #17 0x00000000004bdc1d in main (argc=6, argv=0x7fffffffde60) at perf.c:556
> > > > (gdb)
> 
> > > So my guess would be that something modified a map and broke the
> > > invariants of the maps_by_addresss/maps_by_name. It should be possible
> > > to add more check_invariants to work out where this happens.
>  
> > I also suspect this is a regression. If you could bisect to find the
> 
> I bisected it to:
> 
> ⬢ [acme@toolbox perf-tools-next]$ git bisect good
> 876e80cf83d10585df6ee1e353cfbf562f9a930e is the first bad commit
> commit 876e80cf83d10585df6ee1e353cfbf562f9a930e
> Author: Namhyung Kim <namhyung@kernel.org>
> Date:   Wed Dec 18 14:04:53 2024 -0800
> 
>     perf tools: Fixup end address of modules
>     
>     In machine__create_module(), it reads /proc/modules to get a list of
>     modules in the system.  The file shows the start address (of text) and
>     the size of the module so it uses the info to reconstruct system memory
>     maps for symbol resolution.
>     
>     But module memory consists of multiple segments and they can be
>     scaterred.  Currently perf tools assume they are contiguous and see some
>     overlaps.  This can confuse the tool when it finds a map containing a
>     given address.
>     
>     As we mostly care about the function symbols in the text segment, it can
>     fixup the size or end address of modules when there's an overlap.  We
>     can use maps__fixup_end() which updates the end address using the start
>     address of the next map.
>     
>     Ideally it should be able to track other segments (like data/rodata),
>     but that would require some changes in /proc/modules IMHO.
>     
>     Reported-by: Blake Jones <blakejones@google.com>
>     Signed-off-by: Namhyung Kim <namhyung@kernel.org>
>     Acked-by: Ian Rogers <irogers@google.com>
>     Cc: Adrian Hunter <adrian.hunter@intel.com>
>     Cc: Daniel Gomez <da.gomez@samsung.com>
>     Cc: Ingo Molnar <mingo@kernel.org>
>     Cc: Jiri Olsa <jolsa@kernel.org>
>     Cc: Kan Liang <kan.liang@linux.intel.com>
>     Cc: Luis Chamberlain <mcgrof@kernel.org>
>     Cc: Peter Zijlstra <peterz@infradead.org>
>     Cc: Petr Pavlu <petr.pavlu@suse.com>
>     Cc: Sami Tolvanen <samitolvanen@google.com>
>     Link: https://lore.kernel.org/r/20241218220453.203069-1-namhyung@kernel.org
>     Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
> 
>  tools/perf/util/machine.c | 2 ++
>  1 file changed, 2 insertions(+)
> ⬢ [acme@toolbox perf-tools-next]$
> 
> If we simply revert this it gets back working:
> 
> ⬢ [acme@toolbox perf-tools-next]$ git revert 876e80cf83d10585df6ee1e353cfbf562f9a930e
> Auto-merging tools/perf/util/machine.c
> [perf-tools-next 1ab31115859a0944] Revert "perf tools: Fixup end address of modules"
>  1 file changed, 2 deletions(-)
> # rm -rf build dir, rebuild it
> 
> root@number:~# perf record -- perf test -w thloop
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.403 MB perf.data (7960 samples) ]
> root@number:~# 
> 
> No time today to try to dig deeper, so just reporting the bisection
> result.

So, using 'perf probe' + 'perf trace' to trace 'perf record':

root@number:~# perf probe -x ~/bin/perf maps__insert maps 'map->dso->name:string'
Target program is compiled without optimization. Skipping prologue.
Probe on address 0x634e4e to force probing at the function entry.

Added new event:
  probe_perf:maps_insert (on maps__insert in /home/acme/bin/perf with maps name=map->dso->name:string)

You can now use it in all perf tools, such as:

	perf record -e probe_perf:maps_insert -aR sleep 1

root@number:~# perf probe -x ~/bin/perf maps__fixup_end maps 
Target program is compiled without optimization. Skipping prologue.
Probe on address 0x636c14 to force probing at the function entry.

Added new event:
  probe_perf:maps_fixup_end (on maps__fixup_end in /home/acme/bin/perf with maps)

You can now use it in all perf tools, such as:

	perf record -e probe_perf:maps_fixup_end -aR sleep 1

root@number:~# perf probe -l
  probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps)
  probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name)
root@number:~# perf trace -e probe_perf:maps* perf record sleep 1
     0.000 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[kernel.kallsyms]")
     0.040 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[overlay]")
     0.053 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[rfcomm]")
<SNIP>
     2.736 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[nvme_auth]")
     2.757 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[video]")
     2.773 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[wmi]")
     2.789 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[pinctrl_alderlake]")
     2.804 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[ip6_tables]")
     2.821 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[ip_tables]")
     2.838 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[fuse]")
    66.799 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[kernel.kallsyms]")
   192.465 perf/1346602 probe_perf:maps_fixup_end(__probe_ip: 6515745, maps: 155203024)
[ perf record: Woken up 1 times to write data ]
  1327.967 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_prog_e8932b6bae2b9745_restrict_filesystems")
  1328.015 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_prog_40ddf486530245f5_sd_devices")
  1328.025 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_prog_6deef7357e7b4530_sd_fw_egress")
<SNIP>
  1328.296 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_prog_6deef7357e7b4530_sd_fw_egress")
  1328.308 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_prog_6deef7357e7b4530_sd_fw_ingress")
  1328.320 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_prog_6deef7357e7b4530_sd_fw_egress")
  1328.331 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_prog_6deef7357e7b4530_sd_fw_ingress")
  1328.343 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_prog_be31ae23198a0378_sd_devices")
  1328.354 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_trampoline_6442522522")
[ perf record: Captured and wrote 0.036 MB perf.data (19 samples) ]
perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
root@number:~#


So the call to maps_fixup_end() will set maps->end_broken to false,
since it fixed up the map ends, etc, but then we insert more maps with
broken ends:

#6  0x0000000000633d52 in check_invariants (maps=0xf967c0) at util/maps.c:95
95						assert(map__end(prev) <= map__end(map));
(gdb) p prev->dso->name
$1 = 0xfc47ab "bpf_trampoline_6442522522"
(gdb) p map->dso->name
$2 = 0xfe5dcb "bpf_prog_40ddf486530245f5_sd_devices"
(gdb) p /x prev->start
$3 = 0xffffffffc0147640
(gdb) p /x map->start
$4 = 0xffffffffc014774c
(gdb) p /x prev->end
$5 = 0xffffffffc0148640
(gdb) p /x map->end
$6 = 0xffffffffc014788b
(gdb) p prev->start < map->start
$7 = 1
(gdb) p prev->end < map->start
$8 = 0
(gdb)

If we ask for backtraces we see where maps are added after fixup_end:

    59.088 perf/1448114 probe_perf:maps_insert(__probe_ip: 6508126, maps: 267654608, name: "[kernel.kallsyms]")
                                       maps__insert (/home/acme/bin/perf)
                                       machine__update_kernel_mmap (/home/acme/bin/perf)
                                       machine__create_kernel_maps (/home/acme/bin/perf)
                                       perf_session__create_kernel_maps (/home/acme/bin/perf)
                                       __perf_session__new (/home/acme/bin/perf)
                                       perf_session__new (/home/acme/bin/perf)
                                       __cmd_record (/home/acme/bin/perf)
                                       cmd_record (/home/acme/bin/perf)
   185.548 perf/1448114 probe_perf:maps_fixup_end(__probe_ip: 6515745, maps: 267654608)
                                       maps__fixup_end (/home/acme/bin/perf)
                                       machine__create_kernel_maps (/home/acme/bin/perf)
                                       perf_session__create_kernel_maps (/home/acme/bin/perf)
                                       __perf_session__new (/home/acme/bin/perf)
                                       perf_session__new (/home/acme/bin/perf)
                                       __cmd_record (/home/acme/bin/perf)
                                       cmd_record (/home/acme/bin/perf)
                                       run_builtin (/home/acme/bin/perf)
sleep: missing operand
Try 'sleep --help' for more information.
[ perf record: Woken up 1 times to write data ]
   320.675 perf/1448114 probe_perf:maps_insert(__probe_ip: 6508126, maps: 267654608, name: "bpf_prog_e8932b6bae2b9745_restrict_filesystems")
                                       maps__insert (/home/acme/bin/perf)
                                       machine__process_ksymbol_register (/home/acme/bin/perf)
                                       machine__process_ksymbol (/home/acme/bin/perf)
                                       perf_event__process_ksymbol (/home/acme/bin/perf)
                                       machines__deliver_event (/home/acme/bin/perf)
                                       perf_session__deliver_event (/home/acme/bin/perf)
                                       perf_session__process_event (/home/acme/bin/perf)
                                       process_simple (/home/acme/bin/perf)


Now looking at machine__process_ksymbol_register()...

- Arnaldo

Re: [PATCH] perf report: Add 'tgid' sort key

[Arnaldo Carvalho de Melo]

On Tue, Feb 18, 2025 at 09:36:52PM +0100, Arnaldo Carvalho de Melo wrote:
> So the call to maps_fixup_end() will set maps->end_broken to false,
> since it fixed up the map ends, etc, but then we insert more maps with
> broken ends:
 
> #6  0x0000000000633d52 in check_invariants (maps=0xf967c0) at util/maps.c:95
> 95						assert(map__end(prev) <= map__end(map));
> (gdb) p prev->dso->name
> $1 = 0xfc47ab "bpf_trampoline_6442522522"

So the above map is created overlapping a previously existing map:

root@number:~# perf probe -l
  probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps)
  probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end)
root@number:~# 

root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep
<SNIP>
   319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8)
   319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8)
   319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c)
   319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec)
   319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2)
   319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640)
[ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ]
perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
root@number:~# 

So a PERF_RECORD_KSYMBOL processing will add a map for
"bpf_trampoline_6442522522" that has its start after before the
"bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after
"bpf_prog_40ddf486530245f5_sd_devices", overlapping it.

machine__process_ksymbol_register() does:

713			map__set_start(map, event->ksymbol.addr);
714			map__set_end(map, map__start(map) + event->ksymbol.len);
715			err = maps__insert(machine__kernel_maps(machine), map);

And:

(gdb) p /x event->ksymbol.addr
$2 = 0xffffffffc0147a2c
(gdb) p event->ksymbol.len
$3 = 306

Thread 1 "perf" hit Breakpoint 1, machine__process_ksymbol_register (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:688
688	{
(gdb) bt
#0  machine__process_ksymbol_register (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:688
#1  0x00000000006294ca in machine__process_ksymbol (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:779
#2  0x00000000005ce2fd in perf_event__process_ksymbol (tool=0xec8ce0 <record>, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, machine=0xf96158) at util/event.c:296
#3  0x000000000063b6e4 in machines__deliver_event (machines=0xf96158, evlist=0xf521f0, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, tool=0xec8ce0 <record>, file_offset=31392, 
    file_path=0xf96850 "perf.data") at util/session.c:1334
#4  0x000000000063b8c9 in perf_session__deliver_event (session=0xf95f40, event=0x7ffff7fb9aa0, tool=0xec8ce0 <record>, file_offset=31392, file_path=0xf96850 "perf.data")
    at util/session.c:1367
#5  0x000000000063c6bd in perf_session__process_event (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1626
#6  0x000000000063de3d in process_simple (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:2203
#7  0x000000000063daf4 in reader__read_event (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2132
#8  0x000000000063dcee in reader__process_events (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2181
#9  0x000000000063df8b in __perf_session__process_events (session=0xf95f40) at util/session.c:2226
#10 0x000000000063e988 in perf_session__process_events (session=0xf95f40) at util/session.c:2390
#11 0x000000000042d98b in process_buildids (rec=0xec8ce0 <record>) at builtin-record.c:1475
#12 0x000000000042e963 in record__finish_output (rec=0xec8ce0 <record>) at builtin-record.c:1798
#13 0x0000000000431c46 in __cmd_record (rec=0xec8ce0 <record>, argc=2, argv=0x7fffffffde80) at builtin-record.c:2841
#14 0x000000000043513f in cmd_record (argc=2, argv=0x7fffffffde80) at builtin-record.c:4260
#15 0x00000000004bcf65 in run_builtin (p=0xecbd60 <commands+288>, argc=3, argv=0x7fffffffde80) at perf.c:351
#16 0x00000000004bd20c in handle_internal_command (argc=3, argv=0x7fffffffde80) at perf.c:404
#17 0x00000000004bd365 in run_argv (argcp=0x7fffffffdc6c, argv=0x7fffffffdc60) at perf.c:448
#18 0x00000000004bd6ae in main (argc=3, argv=0x7fffffffde80) at perf.c:556
(gdb)


So, this one liner "refixes" the "modules" ends when processing the
records to find the build ids, unsure if it is the best solution tho:

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 5db1aedf48df92d2..5c4603d08ab5f2cb 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -1797,6 +1797,8 @@ record__finish_output(struct record *rec)
 	if (!rec->no_buildid) {
 		process_buildids(rec);
 
+		maps__fixup_end(machine__kernel_maps(&rec->session->machines.host));
+
 		if (rec->buildid_all)
 			perf_session__dsos_hit_all(rec->session);
 	}

Re: [PATCH] perf report: Add 'tgid' sort key

[Namhyung Kim]

Hi Arnaldo,

On Tue, Feb 18, 2025 at 10:01:33PM +0100, Arnaldo Carvalho de Melo wrote:
> On Tue, Feb 18, 2025 at 09:36:52PM +0100, Arnaldo Carvalho de Melo wrote:
> > So the call to maps_fixup_end() will set maps->end_broken to false,
> > since it fixed up the map ends, etc, but then we insert more maps with
> > broken ends:
>  
> > #6  0x0000000000633d52 in check_invariants (maps=0xf967c0) at util/maps.c:95
> > 95						assert(map__end(prev) <= map__end(map));
> > (gdb) p prev->dso->name
> > $1 = 0xfc47ab "bpf_trampoline_6442522522"
> 
> So the above map is created overlapping a previously existing map:
> 
> root@number:~# perf probe -l
>   probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps)
>   probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end)
> root@number:~# 
> 
> root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep
> <SNIP>
>    319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8)
>    319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8)
>    319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c)
>    319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec)
>    319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2)
>    319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640)
> [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ]
> perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> root@number:~# 
> 
> So a PERF_RECORD_KSYMBOL processing will add a map for
> "bpf_trampoline_6442522522" that has its start after before the
> "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after
> "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
> 
> machine__process_ksymbol_register() does:
> 
> 713			map__set_start(map, event->ksymbol.addr);
> 714			map__set_end(map, map__start(map) + event->ksymbol.len);
> 715			err = maps__insert(machine__kernel_maps(machine), map);
> 
> And:
> 
> (gdb) p /x event->ksymbol.addr
> $2 = 0xffffffffc0147a2c
> (gdb) p event->ksymbol.len
> $3 = 306

Hmm.. so I think the situation is like below.

                   (bpf_trampoline_6442522522)
            +---------------------------------------+   
	    |                                       |
	    |       +------------------------+      |
	    |       | (bpf_prog_40ddf486...) | <----+----  adding this
	    |       |                        |      |
	    |       |                        |      |
	    |   c0147a2c                            |
	    |                                       |
        c0147640                                 c0148640

And it failed to add bpf_prog_40ddf486... in check_invariants() because
the end address is smaller than the previous map.

> 
> Thread 1 "perf" hit Breakpoint 1, machine__process_ksymbol_register (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:688
> 688	{
> (gdb) bt
> #0  machine__process_ksymbol_register (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:688
> #1  0x00000000006294ca in machine__process_ksymbol (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:779
> #2  0x00000000005ce2fd in perf_event__process_ksymbol (tool=0xec8ce0 <record>, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, machine=0xf96158) at util/event.c:296
> #3  0x000000000063b6e4 in machines__deliver_event (machines=0xf96158, evlist=0xf521f0, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, tool=0xec8ce0 <record>, file_offset=31392, 
>     file_path=0xf96850 "perf.data") at util/session.c:1334
> #4  0x000000000063b8c9 in perf_session__deliver_event (session=0xf95f40, event=0x7ffff7fb9aa0, tool=0xec8ce0 <record>, file_offset=31392, file_path=0xf96850 "perf.data")
>     at util/session.c:1367
> #5  0x000000000063c6bd in perf_session__process_event (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1626
> #6  0x000000000063de3d in process_simple (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:2203
> #7  0x000000000063daf4 in reader__read_event (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2132
> #8  0x000000000063dcee in reader__process_events (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2181
> #9  0x000000000063df8b in __perf_session__process_events (session=0xf95f40) at util/session.c:2226
> #10 0x000000000063e988 in perf_session__process_events (session=0xf95f40) at util/session.c:2390
> #11 0x000000000042d98b in process_buildids (rec=0xec8ce0 <record>) at builtin-record.c:1475
> #12 0x000000000042e963 in record__finish_output (rec=0xec8ce0 <record>) at builtin-record.c:1798
> #13 0x0000000000431c46 in __cmd_record (rec=0xec8ce0 <record>, argc=2, argv=0x7fffffffde80) at builtin-record.c:2841
> #14 0x000000000043513f in cmd_record (argc=2, argv=0x7fffffffde80) at builtin-record.c:4260
> #15 0x00000000004bcf65 in run_builtin (p=0xecbd60 <commands+288>, argc=3, argv=0x7fffffffde80) at perf.c:351
> #16 0x00000000004bd20c in handle_internal_command (argc=3, argv=0x7fffffffde80) at perf.c:404
> #17 0x00000000004bd365 in run_argv (argcp=0x7fffffffdc6c, argv=0x7fffffffdc60) at perf.c:448
> #18 0x00000000004bd6ae in main (argc=3, argv=0x7fffffffde80) at perf.c:556
> (gdb)
> 
> 
> So, this one liner "refixes" the "modules" ends when processing the
> records to find the build ids, unsure if it is the best solution tho:

I think it "fixes" the problem by not clearing maps->ends_broken during
the sample processing.  So check_invariants() will not check the end
addresses of overlapping bpf_trampoline and bpf_prog.

I'm curious how other commands (like perf report) are affected.  I think
the original concern was the output of `perf buildid-list -m`.

Thanks,
Namhyung

> 
> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> index 5db1aedf48df92d2..5c4603d08ab5f2cb 100644
> --- a/tools/perf/builtin-record.c
> +++ b/tools/perf/builtin-record.c
> @@ -1797,6 +1797,8 @@ record__finish_output(struct record *rec)
>  	if (!rec->no_buildid) {
>  		process_buildids(rec);
>  
> +		maps__fixup_end(machine__kernel_maps(&rec->session->machines.host));
> +
>  		if (rec->buildid_all)
>  			perf_session__dsos_hit_all(rec->session);
>  	}
> 
> 

Re: [PATCH] perf report: Add 'tgid' sort key

[Arnaldo Carvalho de Melo]

On Tue, Feb 18, 2025 at 02:03:01PM -0800, Namhyung Kim wrote:
> On Tue, Feb 18, 2025 at 10:01:33PM +0100, Arnaldo Carvalho de Melo wrote:
> > On Tue, Feb 18, 2025 at 09:36:52PM +0100, Arnaldo Carvalho de Melo wrote:
> > > So the call to maps_fixup_end() will set maps->end_broken to false,
> > > since it fixed up the map ends, etc, but then we insert more maps with
> > > broken ends:
> >  
> > > #6  0x0000000000633d52 in check_invariants (maps=0xf967c0) at util/maps.c:95
> > > 95						assert(map__end(prev) <= map__end(map));
> > > (gdb) p prev->dso->name
> > > $1 = 0xfc47ab "bpf_trampoline_6442522522"
> > 
> > So the above map is created overlapping a previously existing map:
> > 
> > root@number:~# perf probe -l
> >   probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps)
> >   probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end)
> > root@number:~# 
> > 
> > root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep
> > <SNIP>
> >    319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8)
> >    319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8)
> >    319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c)
> >    319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec)
> >    319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2)
> >    319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640)
> > [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ]
> > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > root@number:~# 
> > 
> > So a PERF_RECORD_KSYMBOL processing will add a map for
> > "bpf_trampoline_6442522522" that has its start after before the
> > "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after
> > "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
> > 
> > machine__process_ksymbol_register() does:
> > 
> > 713			map__set_start(map, event->ksymbol.addr);
> > 714			map__set_end(map, map__start(map) + event->ksymbol.len);
> > 715			err = maps__insert(machine__kernel_maps(machine), map);
> > 
> > And:
> > 
> > (gdb) p /x event->ksymbol.addr
> > $2 = 0xffffffffc0147a2c
> > (gdb) p event->ksymbol.len
> > $3 = 306
> 
> Hmm.. so I think the situation is like below.
> 
>                    (bpf_trampoline_6442522522)
>             +---------------------------------------+   
> 	    |                                       |
> 	    |       +------------------------+      |
> 	    |       | (bpf_prog_40ddf486...) | <----+----  adding this
> 	    |       |                        |      |
> 	    |       |                        |      |
> 	    |   c0147a2c                            |
> 	    |                                       |
>         c0147640                                 c0148640
> 
> And it failed to add bpf_prog_40ddf486... in check_invariants() because
> the end address is smaller than the previous map.

No, it didn't fail to add, it managed to do it which left the kernel
maps in a broken state, with overlappings while it had a cleared
ends_broken, then, later, when the checks_invariant is finally called at
perf record exit time:

> > > > #4  0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev) <= map__end(map)",
> > > >     file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96
> > > > #5  0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95,
> > > >     function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105
> > > > #6  0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95
> > > > #7  0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538
> > > > #8  0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176
> > > > #9  0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238
> > > > #10 0x00000000006388af in perf_session__destroy_kernel_maps (session=0xf97f60) at util/session.c:105
> > > > #11 0x0000000000638df0 in perf_session__delete (session=0xf97f60) at util/session.c:248
> > > > #12 0x0000000000431f18 in __cmd_record (rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888

is when we detect the problem, but I see what you mean, I'm trying to
figure out why this isn't caught here:

machine__process_ksymbol_register() ->
	int maps__insert(struct maps *maps, struct map *map)
	{       
		int ret;

		down_write(maps__lock(maps));
		ret = __maps__insert(maps, map);
		check_invariants(maps);
		up_write(maps__lock(maps));
		return ret;
	}

Some more tracing:

root@number:~# perf probe -d probe_perf:* ; perf probe -qx ~/bin/perf check_invariants maps 'maps->maps_by_address_sorted' ; perf probe -qx ~/bin/perf maps__insert maps 'map->dso->name:string' 'map->start' 'map->end' ; perf probe -qx ~/bin/perf maps__fixup_end maps ; perf probe -l
Removed event: probe_perf:check_invariants
Removed event: probe_perf:maps_fixup_end
Removed event: probe_perf:maps_insert
  probe_perf:check_invariants (on check_invariants:1@util/maps.c in /home/acme/bin/perf with maps maps_by_address_sorted)
  probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps)
  probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end)
root@number:~#

And then:

root@number:~# perf trace --lib -e probe_perf:maps_*,probe_perf:check_invariants/max-stack=32/ perf record sleep
<SNIP>
   316.283 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec)
   316.284 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0)
                                       check_invariants (/home/acme/bin/perf)
                                       maps__insert (/home/acme/bin/perf)
                                       machine__process_ksymbol_register (/home/acme/bin/perf)
                                       machine__process_ksymbol (/home/acme/bin/perf)
                                       perf_event__process_ksymbol (/home/acme/bin/perf)
                                       machines__deliver_event (/home/acme/bin/perf)
                                       perf_session__deliver_event (/home/acme/bin/perf)
                                       perf_session__process_event (/home/acme/bin/perf)
                                       process_simple (/home/acme/bin/perf)
                                       reader__read_event (/home/acme/bin/perf)
                                       reader__process_events (/home/acme/bin/perf)
                                       __perf_session__process_events (/home/acme/bin/perf)
                                       perf_session__process_events (/home/acme/bin/perf)
                                       process_buildids (/home/acme/bin/perf)
                                       record__finish_output (/home/acme/bin/perf)
                                       __cmd_record (/home/acme/bin/perf)
                                       cmd_record (/home/acme/bin/perf)
                                       run_builtin (/home/acme/bin/perf)
                                       handle_internal_command (/home/acme/bin/perf)
                                       run_argv (/home/acme/bin/perf)
                                       main (/home/acme/bin/perf)
                                       __libc_start_call_main (/usr/lib64/libc.so.6)
                                       __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
                                       _start (/home/acme/bin/perf)
   316.296 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2)
   316.298 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0)
                                       check_invariants (/home/acme/bin/perf)
                                       maps__insert (/home/acme/bin/perf)
                                       machine__process_ksymbol_register (/home/acme/bin/perf)
                                       machine__process_ksymbol (/home/acme/bin/perf)
                                       perf_event__process_ksymbol (/home/acme/bin/perf)
                                       machines__deliver_event (/home/acme/bin/perf)
                                       perf_session__deliver_event (/home/acme/bin/perf)
                                       perf_session__process_event (/home/acme/bin/perf)
                                       process_simple (/home/acme/bin/perf)
                                       reader__read_event (/home/acme/bin/perf)
                                       reader__process_events (/home/acme/bin/perf)
                                       __perf_session__process_events (/home/acme/bin/perf)
                                       perf_session__process_events (/home/acme/bin/perf)
                                       process_buildids (/home/acme/bin/perf)
                                       record__finish_output (/home/acme/bin/perf)
                                       __cmd_record (/home/acme/bin/perf)
                                       cmd_record (/home/acme/bin/perf)
                                       run_builtin (/home/acme/bin/perf)
                                       handle_internal_command (/home/acme/bin/perf)
                                       run_argv (/home/acme/bin/perf)
                                       main (/home/acme/bin/perf)
                                       __libc_start_call_main (/usr/lib64/libc.so.6)
                                       __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
                                       _start (/home/acme/bin/perf)
   316.310 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640)
   316.311 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0)
                                       check_invariants (/home/acme/bin/perf)
                                       maps__insert (/home/acme/bin/perf)
                                       machine__process_ksymbol_register (/home/acme/bin/perf)
                                       machine__process_ksymbol (/home/acme/bin/perf)
                                       perf_event__process_ksymbol (/home/acme/bin/perf)
                                       machines__deliver_event (/home/acme/bin/perf)
                                       perf_session__deliver_event (/home/acme/bin/perf)
                                       perf_session__process_event (/home/acme/bin/perf)
                                       process_simple (/home/acme/bin/perf)
                                       reader__read_event (/home/acme/bin/perf)
                                       reader__process_events (/home/acme/bin/perf)
                                       __perf_session__process_events (/home/acme/bin/perf)
                                       perf_session__process_events (/home/acme/bin/perf)
                                       process_buildids (/home/acme/bin/perf)
                                       record__finish_output (/home/acme/bin/perf)
                                       __cmd_record (/home/acme/bin/perf)
                                       cmd_record (/home/acme/bin/perf)
                                       run_builtin (/home/acme/bin/perf)
                                       handle_internal_command (/home/acme/bin/perf)
                                       run_argv (/home/acme/bin/perf)
                                       main (/home/acme/bin/perf)
                                       __libc_start_call_main (/usr/lib64/libc.so.6)
                                       __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
                                       _start (/home/acme/bin/perf)
   316.369 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342a6950 maps_by_address_sorted=0x1)
                                       check_invariants (/home/acme/bin/perf)
                                       __maps__insert_sorted (/home/acme/bin/perf)
                                       __maps__fixup_overlap_and_insert (/home/acme/bin/perf)
                                       maps__fixup_overlap_and_insert (/home/acme/bin/perf)
                                       thread__insert_map (/home/acme/bin/perf)
                                       machine__process_mmap2_event (/home/acme/bin/perf)
                                       perf_event__process_mmap2 (/home/acme/bin/perf)
                                       build_id__process_mmap2 (/home/acme/bin/perf)
                                       machines__deliver_event (/home/acme/bin/perf)
                                       perf_session__deliver_event (/home/acme/bin/perf)
                                       ordered_events__deliver_event (/home/acme/bin/perf)
                                       do_flush (/home/acme/bin/perf)
                                       __ordered_events__flush (/home/acme/bin/perf)
                                       ordered_events__flush (/home/acme/bin/perf)
                                       __perf_session__process_events (/home/acme/bin/perf)
                                       perf_session__process_events (/home/acme/bin/perf)
                                       process_buildids (/home/acme/bin/perf)
                                       record__finish_output (/home/acme/bin/perf)
                                       __cmd_record (/home/acme/bin/perf)
                                       cmd_record (/home/acme/bin/perf)
                                       run_builtin (/home/acme/bin/perf)
                                       handle_internal_command (/home/acme/bin/perf)
                                       run_argv (/home/acme/bin/perf)
                                       main (/home/acme/bin/perf)
                                       __libc_start_call_main (/usr/lib64/libc.so.6)
                                       __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
                                       _start (/home/acme/bin/perf)
<SNIP>
[ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ]
  1195.433 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x1)
                                       check_invariants (/home/acme/bin/perf)
                                       maps__remove (/home/acme/bin/perf)
                                       machine__destroy_kernel_maps (/home/acme/bin/perfperf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
)
                                       machines__destroy_kernel_maps (/home/acme/bin/perf)
                                       perf_session__destroy_kernel_maps (/home/acme/bin/perf)
                                       perf_session__delete (/home/acme/bin/perf)
                                       __cmd_record (/home/acme/bin/perf)
                                       cmd_record (/home/acme/bin/perf)
                                       run_builtin (/home/acme/bin/perf)
                                       handle_internal_command (/home/acme/bin/perf)
                                       run_argv (/home/acme/bin/perf)
                                       main (/home/acme/bin/perf)
                                       __libc_start_call_main (/usr/lib64/libc.so.6)
                                       __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
                                       _start (/home/acme/bin/perf)
root@number:~#

check_invariants() doesn't check the ends because the
maps_byh_address_sorted is not set, I'll soon disappear into a call, but
the above should help as a checkpoint, I'll be back.

- Arnaldo

 
> > Thread 1 "perf" hit Breakpoint 1, machine__process_ksymbol_register (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:688
> > 688	{
> > (gdb) bt
> > #0  machine__process_ksymbol_register (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:688
> > #1  0x00000000006294ca in machine__process_ksymbol (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:779
> > #2  0x00000000005ce2fd in perf_event__process_ksymbol (tool=0xec8ce0 <record>, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, machine=0xf96158) at util/event.c:296
> > #3  0x000000000063b6e4 in machines__deliver_event (machines=0xf96158, evlist=0xf521f0, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, tool=0xec8ce0 <record>, file_offset=31392, 
> >     file_path=0xf96850 "perf.data") at util/session.c:1334
> > #4  0x000000000063b8c9 in perf_session__deliver_event (session=0xf95f40, event=0x7ffff7fb9aa0, tool=0xec8ce0 <record>, file_offset=31392, file_path=0xf96850 "perf.data")
> >     at util/session.c:1367
> > #5  0x000000000063c6bd in perf_session__process_event (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1626
> > #6  0x000000000063de3d in process_simple (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:2203
> > #7  0x000000000063daf4 in reader__read_event (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2132
> > #8  0x000000000063dcee in reader__process_events (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2181
> > #9  0x000000000063df8b in __perf_session__process_events (session=0xf95f40) at util/session.c:2226
> > #10 0x000000000063e988 in perf_session__process_events (session=0xf95f40) at util/session.c:2390
> > #11 0x000000000042d98b in process_buildids (rec=0xec8ce0 <record>) at builtin-record.c:1475
> > #12 0x000000000042e963 in record__finish_output (rec=0xec8ce0 <record>) at builtin-record.c:1798
> > #13 0x0000000000431c46 in __cmd_record (rec=0xec8ce0 <record>, argc=2, argv=0x7fffffffde80) at builtin-record.c:2841
> > #14 0x000000000043513f in cmd_record (argc=2, argv=0x7fffffffde80) at builtin-record.c:4260
> > #15 0x00000000004bcf65 in run_builtin (p=0xecbd60 <commands+288>, argc=3, argv=0x7fffffffde80) at perf.c:351
> > #16 0x00000000004bd20c in handle_internal_command (argc=3, argv=0x7fffffffde80) at perf.c:404
> > #17 0x00000000004bd365 in run_argv (argcp=0x7fffffffdc6c, argv=0x7fffffffdc60) at perf.c:448
> > #18 0x00000000004bd6ae in main (argc=3, argv=0x7fffffffde80) at perf.c:556
> > (gdb)
> 
> > So, this one liner "refixes" the "modules" ends when processing the
> > records to find the build ids, unsure if it is the best solution tho:

> I think it "fixes" the problem by not clearing maps->ends_broken during
> the sample processing.  So check_invariants() will not check the end
> addresses of overlapping bpf_trampoline and bpf_prog.

You mean my one-liner?

I meant "refixes" as in maps__fixup_end() will fixup the overlapping of
the bpf_trampoline and bpf_prog and will re-clear maps->ends_broken
(needlessly, it was already cleared by the first call to
maps__fixup_end() after loading modules, at the start of the session).

Then check_invariants() _will_, check again, because maps->ends_broken
is cleared (was cleared twice even), the end addresses and there will
not be any overlapping, no?

- Arnaldo
 
> I'm curious how other commands (like perf report) are affected.  I think
> the original concern was the output of `perf buildid-list -m`.
> 
> Thanks,
> Namhyung
> 
> > 
> > diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> > index 5db1aedf48df92d2..5c4603d08ab5f2cb 100644
> > --- a/tools/perf/builtin-record.c
> > +++ b/tools/perf/builtin-record.c
> > @@ -1797,6 +1797,8 @@ record__finish_output(struct record *rec)
> >  	if (!rec->no_buildid) {
> >  		process_buildids(rec);
> >  
> > +		maps__fixup_end(machine__kernel_maps(&rec->session->machines.host));
> > +
> >  		if (rec->buildid_all)
> >  			perf_session__dsos_hit_all(rec->session);
> >  	}
> > 
> > 

Re: [PATCH] perf report: Add 'tgid' sort key

[Arnaldo Carvalho de Melo]

On Wed, Feb 19, 2025 at 03:37:10PM +0100, Arnaldo Carvalho de Melo wrote:
> On Tue, Feb 18, 2025 at 02:03:01PM -0800, Namhyung Kim wrote:
> > On Tue, Feb 18, 2025 at 10:01:33PM +0100, Arnaldo Carvalho de Melo wrote:
> > > On Tue, Feb 18, 2025 at 09:36:52PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > So the call to maps_fixup_end() will set maps->end_broken to false,
> > > > since it fixed up the map ends, etc, but then we insert more maps with
> > > > broken ends:
> > >  
> > > > #6  0x0000000000633d52 in check_invariants (maps=0xf967c0) at util/maps.c:95
> > > > 95						assert(map__end(prev) <= map__end(map));
> > > > (gdb) p prev->dso->name
> > > > $1 = 0xfc47ab "bpf_trampoline_6442522522"
> > > 
> > > So the above map is created overlapping a previously existing map:
> > > 
> > > root@number:~# perf probe -l
> > >   probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps)
> > >   probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end)
> > > root@number:~# 
> > > 
> > > root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep
> > > <SNIP>
> > >    319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8)
> > >    319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8)
> > >    319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c)
> > >    319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec)
> > >    319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2)
> > >    319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640)
> > > [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ]
> > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > > root@number:~# 
> > > 
> > > So a PERF_RECORD_KSYMBOL processing will add a map for
> > > "bpf_trampoline_6442522522" that has its start after before the
> > > "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after
> > > "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
> > > 
> > > machine__process_ksymbol_register() does:
> > > 
> > > 713			map__set_start(map, event->ksymbol.addr);
> > > 714			map__set_end(map, map__start(map) + event->ksymbol.len);
> > > 715			err = maps__insert(machine__kernel_maps(machine), map);
> > > 
> > > And:
> > > 
> > > (gdb) p /x event->ksymbol.addr
> > > $2 = 0xffffffffc0147a2c
> > > (gdb) p event->ksymbol.len
> > > $3 = 306
> > 
> > Hmm.. so I think the situation is like below.
> > 
> >                    (bpf_trampoline_6442522522)
> >             +---------------------------------------+   
> > 	    |                                       |
> > 	    |       +------------------------+      |
> > 	    |       | (bpf_prog_40ddf486...) | <----+----  adding this
> > 	    |       |                        |      |
> > 	    |       |                        |      |
> > 	    |   c0147a2c                            |
> > 	    |                                       |
> >         c0147640                                 c0148640
> > 
> > And it failed to add bpf_prog_40ddf486... in check_invariants() because
> > the end address is smaller than the previous map.
> 
> No, it didn't fail to add, it managed to do it which left the kernel
> maps in a broken state, with overlappings while it had a cleared
> ends_broken, then, later, when the checks_invariant is finally called at
> perf record exit time:

Nope, __maps__insert() should notice that the ends are broken and set
it:

        if (nr_maps == 1) {
                /* If there's just 1 entry then maps are sorted. */
                maps__set_maps_by_address_sorted(maps, true);
                maps__set_maps_by_name_sorted(maps, maps_by_name != NULL);
        } else {
                /* Sorted if maps were already sorted and this map starts after the last one. */
                maps__set_maps_by_address_sorted(maps,
                        maps__maps_by_address_sorted(maps) &&
                        map__end(maps_by_address[nr_maps - 2]) <= map__start(new));
                maps__set_maps_by_name_sorted(maps, false);
        }
        if (map__end(new) < map__start(new))
                RC_CHK_ACCESS(maps)->ends_broken = true;


humm, RC_CHK_ACCESS(maps)->ends_broken should be set for the case we
have and I think it isn't being... Then the bpf trampoline map that is
the last entry to be added is before the last entry and thus
maps_by_address_sorted is set to false, ends_broken continues false and
at the end maps_by_address_sorted is set to true and the last
check_invariants triggerrs the asserts...
 
> > > > > #4  0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev) <= map__end(map)",
> > > > >     file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96
> > > > > #5  0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95,
> > > > >     function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105
> > > > > #6  0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95
> > > > > #7  0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538
> > > > > #8  0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176
> > > > > #9  0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238
> > > > > #10 0x00000000006388af in perf_session__destroy_kernel_maps (session=0xf97f60) at util/session.c:105
> > > > > #11 0x0000000000638df0 in perf_session__delete (session=0xf97f60) at util/session.c:248
> > > > > #12 0x0000000000431f18 in __cmd_record (rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888
> 
> is when we detect the problem, but I see what you mean, I'm trying to
> figure out why this isn't caught here:
> 
> machine__process_ksymbol_register() ->
> 	int maps__insert(struct maps *maps, struct map *map)
> 	{       
> 		int ret;
> 
> 		down_write(maps__lock(maps));
> 		ret = __maps__insert(maps, map);
> 		check_invariants(maps);
> 		up_write(maps__lock(maps));
> 		return ret;
> 	}
> 
> Some more tracing:
> 
> root@number:~# perf probe -d probe_perf:* ; perf probe -qx ~/bin/perf check_invariants maps 'maps->maps_by_address_sorted' ; perf probe -qx ~/bin/perf maps__insert maps 'map->dso->name:string' 'map->start' 'map->end' ; perf probe -qx ~/bin/perf maps__fixup_end maps ; perf probe -l
> Removed event: probe_perf:check_invariants
> Removed event: probe_perf:maps_fixup_end
> Removed event: probe_perf:maps_insert
>   probe_perf:check_invariants (on check_invariants:1@util/maps.c in /home/acme/bin/perf with maps maps_by_address_sorted)
>   probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps)
>   probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end)
> root@number:~#
> 
> And then:
> 
> root@number:~# perf trace --lib -e probe_perf:maps_*,probe_perf:check_invariants/max-stack=32/ perf record sleep
> <SNIP>
>    316.283 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec)
>    316.284 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0)
>                                        check_invariants (/home/acme/bin/perf)
>                                        maps__insert (/home/acme/bin/perf)
>                                        machine__process_ksymbol_register (/home/acme/bin/perf)
>                                        machine__process_ksymbol (/home/acme/bin/perf)
>                                        perf_event__process_ksymbol (/home/acme/bin/perf)
>                                        machines__deliver_event (/home/acme/bin/perf)
>                                        perf_session__deliver_event (/home/acme/bin/perf)
>                                        perf_session__process_event (/home/acme/bin/perf)
>                                        process_simple (/home/acme/bin/perf)
>                                        reader__read_event (/home/acme/bin/perf)
>                                        reader__process_events (/home/acme/bin/perf)
>                                        __perf_session__process_events (/home/acme/bin/perf)
>                                        perf_session__process_events (/home/acme/bin/perf)
>                                        process_buildids (/home/acme/bin/perf)
>                                        record__finish_output (/home/acme/bin/perf)
>                                        __cmd_record (/home/acme/bin/perf)
>                                        cmd_record (/home/acme/bin/perf)
>                                        run_builtin (/home/acme/bin/perf)
>                                        handle_internal_command (/home/acme/bin/perf)
>                                        run_argv (/home/acme/bin/perf)
>                                        main (/home/acme/bin/perf)
>                                        __libc_start_call_main (/usr/lib64/libc.so.6)
>                                        __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
>                                        _start (/home/acme/bin/perf)
>    316.296 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2)
>    316.298 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0)
>                                        check_invariants (/home/acme/bin/perf)
>                                        maps__insert (/home/acme/bin/perf)
>                                        machine__process_ksymbol_register (/home/acme/bin/perf)
>                                        machine__process_ksymbol (/home/acme/bin/perf)
>                                        perf_event__process_ksymbol (/home/acme/bin/perf)
>                                        machines__deliver_event (/home/acme/bin/perf)
>                                        perf_session__deliver_event (/home/acme/bin/perf)
>                                        perf_session__process_event (/home/acme/bin/perf)
>                                        process_simple (/home/acme/bin/perf)
>                                        reader__read_event (/home/acme/bin/perf)
>                                        reader__process_events (/home/acme/bin/perf)
>                                        __perf_session__process_events (/home/acme/bin/perf)
>                                        perf_session__process_events (/home/acme/bin/perf)
>                                        process_buildids (/home/acme/bin/perf)
>                                        record__finish_output (/home/acme/bin/perf)
>                                        __cmd_record (/home/acme/bin/perf)
>                                        cmd_record (/home/acme/bin/perf)
>                                        run_builtin (/home/acme/bin/perf)
>                                        handle_internal_command (/home/acme/bin/perf)
>                                        run_argv (/home/acme/bin/perf)
>                                        main (/home/acme/bin/perf)
>                                        __libc_start_call_main (/usr/lib64/libc.so.6)
>                                        __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
>                                        _start (/home/acme/bin/perf)
>    316.310 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640)
>    316.311 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0)
>                                        check_invariants (/home/acme/bin/perf)
>                                        maps__insert (/home/acme/bin/perf)
>                                        machine__process_ksymbol_register (/home/acme/bin/perf)
>                                        machine__process_ksymbol (/home/acme/bin/perf)
>                                        perf_event__process_ksymbol (/home/acme/bin/perf)
>                                        machines__deliver_event (/home/acme/bin/perf)
>                                        perf_session__deliver_event (/home/acme/bin/perf)
>                                        perf_session__process_event (/home/acme/bin/perf)
>                                        process_simple (/home/acme/bin/perf)
>                                        reader__read_event (/home/acme/bin/perf)
>                                        reader__process_events (/home/acme/bin/perf)
>                                        __perf_session__process_events (/home/acme/bin/perf)
>                                        perf_session__process_events (/home/acme/bin/perf)
>                                        process_buildids (/home/acme/bin/perf)
>                                        record__finish_output (/home/acme/bin/perf)
>                                        __cmd_record (/home/acme/bin/perf)
>                                        cmd_record (/home/acme/bin/perf)
>                                        run_builtin (/home/acme/bin/perf)
>                                        handle_internal_command (/home/acme/bin/perf)
>                                        run_argv (/home/acme/bin/perf)
>                                        main (/home/acme/bin/perf)
>                                        __libc_start_call_main (/usr/lib64/libc.so.6)
>                                        __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
>                                        _start (/home/acme/bin/perf)
>    316.369 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342a6950 maps_by_address_sorted=0x1)
>                                        check_invariants (/home/acme/bin/perf)
>                                        __maps__insert_sorted (/home/acme/bin/perf)
>                                        __maps__fixup_overlap_and_insert (/home/acme/bin/perf)
>                                        maps__fixup_overlap_and_insert (/home/acme/bin/perf)
>                                        thread__insert_map (/home/acme/bin/perf)
>                                        machine__process_mmap2_event (/home/acme/bin/perf)
>                                        perf_event__process_mmap2 (/home/acme/bin/perf)
>                                        build_id__process_mmap2 (/home/acme/bin/perf)
>                                        machines__deliver_event (/home/acme/bin/perf)
>                                        perf_session__deliver_event (/home/acme/bin/perf)
>                                        ordered_events__deliver_event (/home/acme/bin/perf)
>                                        do_flush (/home/acme/bin/perf)
>                                        __ordered_events__flush (/home/acme/bin/perf)
>                                        ordered_events__flush (/home/acme/bin/perf)
>                                        __perf_session__process_events (/home/acme/bin/perf)
>                                        perf_session__process_events (/home/acme/bin/perf)
>                                        process_buildids (/home/acme/bin/perf)
>                                        record__finish_output (/home/acme/bin/perf)
>                                        __cmd_record (/home/acme/bin/perf)
>                                        cmd_record (/home/acme/bin/perf)
>                                        run_builtin (/home/acme/bin/perf)
>                                        handle_internal_command (/home/acme/bin/perf)
>                                        run_argv (/home/acme/bin/perf)
>                                        main (/home/acme/bin/perf)
>                                        __libc_start_call_main (/usr/lib64/libc.so.6)
>                                        __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
>                                        _start (/home/acme/bin/perf)
> <SNIP>
> [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ]
>   1195.433 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x1)
>                                        check_invariants (/home/acme/bin/perf)
>                                        maps__remove (/home/acme/bin/perf)
>                                        machine__destroy_kernel_maps (/home/acme/bin/perfperf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> )
>                                        machines__destroy_kernel_maps (/home/acme/bin/perf)
>                                        perf_session__destroy_kernel_maps (/home/acme/bin/perf)
>                                        perf_session__delete (/home/acme/bin/perf)
>                                        __cmd_record (/home/acme/bin/perf)
>                                        cmd_record (/home/acme/bin/perf)
>                                        run_builtin (/home/acme/bin/perf)
>                                        handle_internal_command (/home/acme/bin/perf)
>                                        run_argv (/home/acme/bin/perf)
>                                        main (/home/acme/bin/perf)
>                                        __libc_start_call_main (/usr/lib64/libc.so.6)
>                                        __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
>                                        _start (/home/acme/bin/perf)
> root@number:~#
> 
> check_invariants() doesn't check the ends because the
> maps_byh_address_sorted is not set, I'll soon disappear into a call, but
> the above should help as a checkpoint, I'll be back.
> 
> - Arnaldo
> 
>  
> > > Thread 1 "perf" hit Breakpoint 1, machine__process_ksymbol_register (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:688
> > > 688	{
> > > (gdb) bt
> > > #0  machine__process_ksymbol_register (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:688
> > > #1  0x00000000006294ca in machine__process_ksymbol (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:779
> > > #2  0x00000000005ce2fd in perf_event__process_ksymbol (tool=0xec8ce0 <record>, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, machine=0xf96158) at util/event.c:296
> > > #3  0x000000000063b6e4 in machines__deliver_event (machines=0xf96158, evlist=0xf521f0, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, tool=0xec8ce0 <record>, file_offset=31392, 
> > >     file_path=0xf96850 "perf.data") at util/session.c:1334
> > > #4  0x000000000063b8c9 in perf_session__deliver_event (session=0xf95f40, event=0x7ffff7fb9aa0, tool=0xec8ce0 <record>, file_offset=31392, file_path=0xf96850 "perf.data")
> > >     at util/session.c:1367
> > > #5  0x000000000063c6bd in perf_session__process_event (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1626
> > > #6  0x000000000063de3d in process_simple (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:2203
> > > #7  0x000000000063daf4 in reader__read_event (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2132
> > > #8  0x000000000063dcee in reader__process_events (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2181
> > > #9  0x000000000063df8b in __perf_session__process_events (session=0xf95f40) at util/session.c:2226
> > > #10 0x000000000063e988 in perf_session__process_events (session=0xf95f40) at util/session.c:2390
> > > #11 0x000000000042d98b in process_buildids (rec=0xec8ce0 <record>) at builtin-record.c:1475
> > > #12 0x000000000042e963 in record__finish_output (rec=0xec8ce0 <record>) at builtin-record.c:1798
> > > #13 0x0000000000431c46 in __cmd_record (rec=0xec8ce0 <record>, argc=2, argv=0x7fffffffde80) at builtin-record.c:2841
> > > #14 0x000000000043513f in cmd_record (argc=2, argv=0x7fffffffde80) at builtin-record.c:4260
> > > #15 0x00000000004bcf65 in run_builtin (p=0xecbd60 <commands+288>, argc=3, argv=0x7fffffffde80) at perf.c:351
> > > #16 0x00000000004bd20c in handle_internal_command (argc=3, argv=0x7fffffffde80) at perf.c:404
> > > #17 0x00000000004bd365 in run_argv (argcp=0x7fffffffdc6c, argv=0x7fffffffdc60) at perf.c:448
> > > #18 0x00000000004bd6ae in main (argc=3, argv=0x7fffffffde80) at perf.c:556
> > > (gdb)
> > 
> > > So, this one liner "refixes" the "modules" ends when processing the
> > > records to find the build ids, unsure if it is the best solution tho:
> 
> > I think it "fixes" the problem by not clearing maps->ends_broken during
> > the sample processing.  So check_invariants() will not check the end
> > addresses of overlapping bpf_trampoline and bpf_prog.
> 
> You mean my one-liner?
> 
> I meant "refixes" as in maps__fixup_end() will fixup the overlapping of
> the bpf_trampoline and bpf_prog and will re-clear maps->ends_broken
> (needlessly, it was already cleared by the first call to
> maps__fixup_end() after loading modules, at the start of the session).
> 
> Then check_invariants() _will_, check again, because maps->ends_broken
> is cleared (was cleared twice even), the end addresses and there will
> not be any overlapping, no?
> 
> - Arnaldo
>  
> > I'm curious how other commands (like perf report) are affected.  I think
> > the original concern was the output of `perf buildid-list -m`.
> > 
> > Thanks,
> > Namhyung
> > 
> > > 
> > > diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> > > index 5db1aedf48df92d2..5c4603d08ab5f2cb 100644
> > > --- a/tools/perf/builtin-record.c
> > > +++ b/tools/perf/builtin-record.c
> > > @@ -1797,6 +1797,8 @@ record__finish_output(struct record *rec)
> > >  	if (!rec->no_buildid) {
> > >  		process_buildids(rec);
> > >  
> > > +		maps__fixup_end(machine__kernel_maps(&rec->session->machines.host));
> > > +
> > >  		if (rec->buildid_all)
> > >  			perf_session__dsos_hit_all(rec->session);
> > >  	}
> > > 
> > > 

Re: [PATCH] perf report: Add 'tgid' sort key

[Namhyung Kim]

On Wed, Feb 19, 2025 at 03:47:44PM +0100, Arnaldo Carvalho de Melo wrote:
> On Wed, Feb 19, 2025 at 03:37:10PM +0100, Arnaldo Carvalho de Melo wrote:
> > On Tue, Feb 18, 2025 at 02:03:01PM -0800, Namhyung Kim wrote:
> > > On Tue, Feb 18, 2025 at 10:01:33PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > On Tue, Feb 18, 2025 at 09:36:52PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > So the call to maps_fixup_end() will set maps->end_broken to false,
> > > > > since it fixed up the map ends, etc, but then we insert more maps with
> > > > > broken ends:
> > > >  
> > > > > #6  0x0000000000633d52 in check_invariants (maps=0xf967c0) at util/maps.c:95
> > > > > 95						assert(map__end(prev) <= map__end(map));
> > > > > (gdb) p prev->dso->name
> > > > > $1 = 0xfc47ab "bpf_trampoline_6442522522"
> > > > 
> > > > So the above map is created overlapping a previously existing map:
> > > > 
> > > > root@number:~# perf probe -l
> > > >   probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps)
> > > >   probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end)
> > > > root@number:~# 
> > > > 
> > > > root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep
> > > > <SNIP>
> > > >    319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8)
> > > >    319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8)
> > > >    319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c)
> > > >    319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec)
> > > >    319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2)
> > > >    319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640)
> > > > [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ]
> > > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > > > root@number:~# 
> > > > 
> > > > So a PERF_RECORD_KSYMBOL processing will add a map for
> > > > "bpf_trampoline_6442522522" that has its start after before the
> > > > "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after
> > > > "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
> > > > 
> > > > machine__process_ksymbol_register() does:
> > > > 
> > > > 713			map__set_start(map, event->ksymbol.addr);
> > > > 714			map__set_end(map, map__start(map) + event->ksymbol.len);
> > > > 715			err = maps__insert(machine__kernel_maps(machine), map);
> > > > 
> > > > And:
> > > > 
> > > > (gdb) p /x event->ksymbol.addr
> > > > $2 = 0xffffffffc0147a2c
> > > > (gdb) p event->ksymbol.len
> > > > $3 = 306
> > > 
> > > Hmm.. so I think the situation is like below.
> > > 
> > >                    (bpf_trampoline_6442522522)
> > >             +---------------------------------------+   
> > > 	    |                                       |
> > > 	    |       +------------------------+      |
> > > 	    |       | (bpf_prog_40ddf486...) | <----+----  adding this
> > > 	    |       |                        |      |
> > > 	    |       |                        |      |
> > > 	    |   c0147a2c                            |
> > > 	    |                                       |
> > >         c0147640                                 c0148640
> > > 
> > > And it failed to add bpf_prog_40ddf486... in check_invariants() because
> > > the end address is smaller than the previous map.
> > 
> > No, it didn't fail to add, it managed to do it which left the kernel
> > maps in a broken state, with overlappings while it had a cleared
> > ends_broken, then, later, when the checks_invariant is finally called at
> > perf record exit time:
> 
> Nope, __maps__insert() should notice that the ends are broken and set
> it:
> 
>         if (nr_maps == 1) {
>                 /* If there's just 1 entry then maps are sorted. */
>                 maps__set_maps_by_address_sorted(maps, true);
>                 maps__set_maps_by_name_sorted(maps, maps_by_name != NULL);
>         } else {
>                 /* Sorted if maps were already sorted and this map starts after the last one. */
>                 maps__set_maps_by_address_sorted(maps,
>                         maps__maps_by_address_sorted(maps) &&
>                         map__end(maps_by_address[nr_maps - 2]) <= map__start(new));
>                 maps__set_maps_by_name_sorted(maps, false);
>         }
>         if (map__end(new) < map__start(new))
>                 RC_CHK_ACCESS(maps)->ends_broken = true;
> 
> 
> humm, RC_CHK_ACCESS(maps)->ends_broken should be set for the case we
> have and I think it isn't being... Then the bpf trampoline map that is
> the last entry to be added is before the last entry and thus
> maps_by_address_sorted is set to false, ends_broken continues false and
> at the end maps_by_address_sorted is set to true and the last
> check_invariants triggerrs the asserts...

Right, probably it needs to set the ends_broken when the end address of
the new map is smaller than the previous (but the start address is
bigger) and fixup the end address when it sorts the maps by address.

Thanks,
Namhyung

>  
> > > > > > #4  0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev) <= map__end(map)",
> > > > > >     file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96
> > > > > > #5  0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95,
> > > > > >     function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105
> > > > > > #6  0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95
> > > > > > #7  0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538
> > > > > > #8  0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176
> > > > > > #9  0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238
> > > > > > #10 0x00000000006388af in perf_session__destroy_kernel_maps (session=0xf97f60) at util/session.c:105
> > > > > > #11 0x0000000000638df0 in perf_session__delete (session=0xf97f60) at util/session.c:248
> > > > > > #12 0x0000000000431f18 in __cmd_record (rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888
> > 
> > is when we detect the problem, but I see what you mean, I'm trying to
> > figure out why this isn't caught here:
> > 
> > machine__process_ksymbol_register() ->
> > 	int maps__insert(struct maps *maps, struct map *map)
> > 	{       
> > 		int ret;
> > 
> > 		down_write(maps__lock(maps));
> > 		ret = __maps__insert(maps, map);
> > 		check_invariants(maps);
> > 		up_write(maps__lock(maps));
> > 		return ret;
> > 	}
> > 
> > Some more tracing:
> > 
> > root@number:~# perf probe -d probe_perf:* ; perf probe -qx ~/bin/perf check_invariants maps 'maps->maps_by_address_sorted' ; perf probe -qx ~/bin/perf maps__insert maps 'map->dso->name:string' 'map->start' 'map->end' ; perf probe -qx ~/bin/perf maps__fixup_end maps ; perf probe -l
> > Removed event: probe_perf:check_invariants
> > Removed event: probe_perf:maps_fixup_end
> > Removed event: probe_perf:maps_insert
> >   probe_perf:check_invariants (on check_invariants:1@util/maps.c in /home/acme/bin/perf with maps maps_by_address_sorted)
> >   probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps)
> >   probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end)
> > root@number:~#
> > 
> > And then:
> > 
> > root@number:~# perf trace --lib -e probe_perf:maps_*,probe_perf:check_invariants/max-stack=32/ perf record sleep
> > <SNIP>
> >    316.283 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec)
> >    316.284 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0)
> >                                        check_invariants (/home/acme/bin/perf)
> >                                        maps__insert (/home/acme/bin/perf)
> >                                        machine__process_ksymbol_register (/home/acme/bin/perf)
> >                                        machine__process_ksymbol (/home/acme/bin/perf)
> >                                        perf_event__process_ksymbol (/home/acme/bin/perf)
> >                                        machines__deliver_event (/home/acme/bin/perf)
> >                                        perf_session__deliver_event (/home/acme/bin/perf)
> >                                        perf_session__process_event (/home/acme/bin/perf)
> >                                        process_simple (/home/acme/bin/perf)
> >                                        reader__read_event (/home/acme/bin/perf)
> >                                        reader__process_events (/home/acme/bin/perf)
> >                                        __perf_session__process_events (/home/acme/bin/perf)
> >                                        perf_session__process_events (/home/acme/bin/perf)
> >                                        process_buildids (/home/acme/bin/perf)
> >                                        record__finish_output (/home/acme/bin/perf)
> >                                        __cmd_record (/home/acme/bin/perf)
> >                                        cmd_record (/home/acme/bin/perf)
> >                                        run_builtin (/home/acme/bin/perf)
> >                                        handle_internal_command (/home/acme/bin/perf)
> >                                        run_argv (/home/acme/bin/perf)
> >                                        main (/home/acme/bin/perf)
> >                                        __libc_start_call_main (/usr/lib64/libc.so.6)
> >                                        __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
> >                                        _start (/home/acme/bin/perf)
> >    316.296 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2)
> >    316.298 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0)
> >                                        check_invariants (/home/acme/bin/perf)
> >                                        maps__insert (/home/acme/bin/perf)
> >                                        machine__process_ksymbol_register (/home/acme/bin/perf)
> >                                        machine__process_ksymbol (/home/acme/bin/perf)
> >                                        perf_event__process_ksymbol (/home/acme/bin/perf)
> >                                        machines__deliver_event (/home/acme/bin/perf)
> >                                        perf_session__deliver_event (/home/acme/bin/perf)
> >                                        perf_session__process_event (/home/acme/bin/perf)
> >                                        process_simple (/home/acme/bin/perf)
> >                                        reader__read_event (/home/acme/bin/perf)
> >                                        reader__process_events (/home/acme/bin/perf)
> >                                        __perf_session__process_events (/home/acme/bin/perf)
> >                                        perf_session__process_events (/home/acme/bin/perf)
> >                                        process_buildids (/home/acme/bin/perf)
> >                                        record__finish_output (/home/acme/bin/perf)
> >                                        __cmd_record (/home/acme/bin/perf)
> >                                        cmd_record (/home/acme/bin/perf)
> >                                        run_builtin (/home/acme/bin/perf)
> >                                        handle_internal_command (/home/acme/bin/perf)
> >                                        run_argv (/home/acme/bin/perf)
> >                                        main (/home/acme/bin/perf)
> >                                        __libc_start_call_main (/usr/lib64/libc.so.6)
> >                                        __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
> >                                        _start (/home/acme/bin/perf)
> >    316.310 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640)
> >    316.311 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0)
> >                                        check_invariants (/home/acme/bin/perf)
> >                                        maps__insert (/home/acme/bin/perf)
> >                                        machine__process_ksymbol_register (/home/acme/bin/perf)
> >                                        machine__process_ksymbol (/home/acme/bin/perf)
> >                                        perf_event__process_ksymbol (/home/acme/bin/perf)
> >                                        machines__deliver_event (/home/acme/bin/perf)
> >                                        perf_session__deliver_event (/home/acme/bin/perf)
> >                                        perf_session__process_event (/home/acme/bin/perf)
> >                                        process_simple (/home/acme/bin/perf)
> >                                        reader__read_event (/home/acme/bin/perf)
> >                                        reader__process_events (/home/acme/bin/perf)
> >                                        __perf_session__process_events (/home/acme/bin/perf)
> >                                        perf_session__process_events (/home/acme/bin/perf)
> >                                        process_buildids (/home/acme/bin/perf)
> >                                        record__finish_output (/home/acme/bin/perf)
> >                                        __cmd_record (/home/acme/bin/perf)
> >                                        cmd_record (/home/acme/bin/perf)
> >                                        run_builtin (/home/acme/bin/perf)
> >                                        handle_internal_command (/home/acme/bin/perf)
> >                                        run_argv (/home/acme/bin/perf)
> >                                        main (/home/acme/bin/perf)
> >                                        __libc_start_call_main (/usr/lib64/libc.so.6)
> >                                        __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
> >                                        _start (/home/acme/bin/perf)
> >    316.369 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342a6950 maps_by_address_sorted=0x1)
> >                                        check_invariants (/home/acme/bin/perf)
> >                                        __maps__insert_sorted (/home/acme/bin/perf)
> >                                        __maps__fixup_overlap_and_insert (/home/acme/bin/perf)
> >                                        maps__fixup_overlap_and_insert (/home/acme/bin/perf)
> >                                        thread__insert_map (/home/acme/bin/perf)
> >                                        machine__process_mmap2_event (/home/acme/bin/perf)
> >                                        perf_event__process_mmap2 (/home/acme/bin/perf)
> >                                        build_id__process_mmap2 (/home/acme/bin/perf)
> >                                        machines__deliver_event (/home/acme/bin/perf)
> >                                        perf_session__deliver_event (/home/acme/bin/perf)
> >                                        ordered_events__deliver_event (/home/acme/bin/perf)
> >                                        do_flush (/home/acme/bin/perf)
> >                                        __ordered_events__flush (/home/acme/bin/perf)
> >                                        ordered_events__flush (/home/acme/bin/perf)
> >                                        __perf_session__process_events (/home/acme/bin/perf)
> >                                        perf_session__process_events (/home/acme/bin/perf)
> >                                        process_buildids (/home/acme/bin/perf)
> >                                        record__finish_output (/home/acme/bin/perf)
> >                                        __cmd_record (/home/acme/bin/perf)
> >                                        cmd_record (/home/acme/bin/perf)
> >                                        run_builtin (/home/acme/bin/perf)
> >                                        handle_internal_command (/home/acme/bin/perf)
> >                                        run_argv (/home/acme/bin/perf)
> >                                        main (/home/acme/bin/perf)
> >                                        __libc_start_call_main (/usr/lib64/libc.so.6)
> >                                        __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
> >                                        _start (/home/acme/bin/perf)
> > <SNIP>
> > [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ]
> >   1195.433 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x1)
> >                                        check_invariants (/home/acme/bin/perf)
> >                                        maps__remove (/home/acme/bin/perf)
> >                                        machine__destroy_kernel_maps (/home/acme/bin/perfperf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > )
> >                                        machines__destroy_kernel_maps (/home/acme/bin/perf)
> >                                        perf_session__destroy_kernel_maps (/home/acme/bin/perf)
> >                                        perf_session__delete (/home/acme/bin/perf)
> >                                        __cmd_record (/home/acme/bin/perf)
> >                                        cmd_record (/home/acme/bin/perf)
> >                                        run_builtin (/home/acme/bin/perf)
> >                                        handle_internal_command (/home/acme/bin/perf)
> >                                        run_argv (/home/acme/bin/perf)
> >                                        main (/home/acme/bin/perf)
> >                                        __libc_start_call_main (/usr/lib64/libc.so.6)
> >                                        __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
> >                                        _start (/home/acme/bin/perf)
> > root@number:~#
> > 
> > check_invariants() doesn't check the ends because the
> > maps_byh_address_sorted is not set, I'll soon disappear into a call, but
> > the above should help as a checkpoint, I'll be back.
> > 
> > - Arnaldo
> > 
> >  
> > > > Thread 1 "perf" hit Breakpoint 1, machine__process_ksymbol_register (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:688
> > > > 688	{
> > > > (gdb) bt
> > > > #0  machine__process_ksymbol_register (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:688
> > > > #1  0x00000000006294ca in machine__process_ksymbol (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:779
> > > > #2  0x00000000005ce2fd in perf_event__process_ksymbol (tool=0xec8ce0 <record>, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, machine=0xf96158) at util/event.c:296
> > > > #3  0x000000000063b6e4 in machines__deliver_event (machines=0xf96158, evlist=0xf521f0, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, tool=0xec8ce0 <record>, file_offset=31392, 
> > > >     file_path=0xf96850 "perf.data") at util/session.c:1334
> > > > #4  0x000000000063b8c9 in perf_session__deliver_event (session=0xf95f40, event=0x7ffff7fb9aa0, tool=0xec8ce0 <record>, file_offset=31392, file_path=0xf96850 "perf.data")
> > > >     at util/session.c:1367
> > > > #5  0x000000000063c6bd in perf_session__process_event (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1626
> > > > #6  0x000000000063de3d in process_simple (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:2203
> > > > #7  0x000000000063daf4 in reader__read_event (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2132
> > > > #8  0x000000000063dcee in reader__process_events (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2181
> > > > #9  0x000000000063df8b in __perf_session__process_events (session=0xf95f40) at util/session.c:2226
> > > > #10 0x000000000063e988 in perf_session__process_events (session=0xf95f40) at util/session.c:2390
> > > > #11 0x000000000042d98b in process_buildids (rec=0xec8ce0 <record>) at builtin-record.c:1475
> > > > #12 0x000000000042e963 in record__finish_output (rec=0xec8ce0 <record>) at builtin-record.c:1798
> > > > #13 0x0000000000431c46 in __cmd_record (rec=0xec8ce0 <record>, argc=2, argv=0x7fffffffde80) at builtin-record.c:2841
> > > > #14 0x000000000043513f in cmd_record (argc=2, argv=0x7fffffffde80) at builtin-record.c:4260
> > > > #15 0x00000000004bcf65 in run_builtin (p=0xecbd60 <commands+288>, argc=3, argv=0x7fffffffde80) at perf.c:351
> > > > #16 0x00000000004bd20c in handle_internal_command (argc=3, argv=0x7fffffffde80) at perf.c:404
> > > > #17 0x00000000004bd365 in run_argv (argcp=0x7fffffffdc6c, argv=0x7fffffffdc60) at perf.c:448
> > > > #18 0x00000000004bd6ae in main (argc=3, argv=0x7fffffffde80) at perf.c:556
> > > > (gdb)
> > > 
> > > > So, this one liner "refixes" the "modules" ends when processing the
> > > > records to find the build ids, unsure if it is the best solution tho:
> > 
> > > I think it "fixes" the problem by not clearing maps->ends_broken during
> > > the sample processing.  So check_invariants() will not check the end
> > > addresses of overlapping bpf_trampoline and bpf_prog.
> > 
> > You mean my one-liner?
> > 
> > I meant "refixes" as in maps__fixup_end() will fixup the overlapping of
> > the bpf_trampoline and bpf_prog and will re-clear maps->ends_broken
> > (needlessly, it was already cleared by the first call to
> > maps__fixup_end() after loading modules, at the start of the session).
> > 
> > Then check_invariants() _will_, check again, because maps->ends_broken
> > is cleared (was cleared twice even), the end addresses and there will
> > not be any overlapping, no?
> > 
> > - Arnaldo
> >  
> > > I'm curious how other commands (like perf report) are affected.  I think
> > > the original concern was the output of `perf buildid-list -m`.
> > > 
> > > Thanks,
> > > Namhyung
> > > 
> > > > 
> > > > diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> > > > index 5db1aedf48df92d2..5c4603d08ab5f2cb 100644
> > > > --- a/tools/perf/builtin-record.c
> > > > +++ b/tools/perf/builtin-record.c
> > > > @@ -1797,6 +1797,8 @@ record__finish_output(struct record *rec)
> > > >  	if (!rec->no_buildid) {
> > > >  		process_buildids(rec);
> > > >  
> > > > +		maps__fixup_end(machine__kernel_maps(&rec->session->machines.host));
> > > > +
> > > >  		if (rec->buildid_all)
> > > >  			perf_session__dsos_hit_all(rec->session);
> > > >  	}
> > > > 
> > > > 

Re: [PATCH] perf report: Add 'tgid' sort key

[Ian Rogers]

On Wed, Feb 19, 2025 at 1:10 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> On Wed, Feb 19, 2025 at 03:47:44PM +0100, Arnaldo Carvalho de Melo wrote:
> > On Wed, Feb 19, 2025 at 03:37:10PM +0100, Arnaldo Carvalho de Melo wrote:
> > > On Tue, Feb 18, 2025 at 02:03:01PM -0800, Namhyung Kim wrote:
> > > > On Tue, Feb 18, 2025 at 10:01:33PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > On Tue, Feb 18, 2025 at 09:36:52PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > So the call to maps_fixup_end() will set maps->end_broken to false,
> > > > > > since it fixed up the map ends, etc, but then we insert more maps with
> > > > > > broken ends:
> > > > >
> > > > > > #6  0x0000000000633d52 in check_invariants (maps=0xf967c0) at util/maps.c:95
> > > > > > 95                                            assert(map__end(prev) <= map__end(map));
> > > > > > (gdb) p prev->dso->name
> > > > > > $1 = 0xfc47ab "bpf_trampoline_6442522522"
> > > > >
> > > > > So the above map is created overlapping a previously existing map:
> > > > >
> > > > > root@number:~# perf probe -l
> > > > >   probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps)
> > > > >   probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end)
> > > > > root@number:~#
> > > > >
> > > > > root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep
> > > > > <SNIP>
> > > > >    319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8)
> > > > >    319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8)
> > > > >    319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c)
> > > > >    319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec)
> > > > >    319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2)
> > > > >    319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640)
> > > > > [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ]
> > > > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > > > > root@number:~#
> > > > >
> > > > > So a PERF_RECORD_KSYMBOL processing will add a map for
> > > > > "bpf_trampoline_6442522522" that has its start after before the
> > > > > "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after
> > > > > "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
> > > > >
> > > > > machine__process_ksymbol_register() does:
> > > > >
> > > > > 713                     map__set_start(map, event->ksymbol.addr);
> > > > > 714                     map__set_end(map, map__start(map) + event->ksymbol.len);
> > > > > 715                     err = maps__insert(machine__kernel_maps(machine), map);
> > > > >
> > > > > And:
> > > > >
> > > > > (gdb) p /x event->ksymbol.addr
> > > > > $2 = 0xffffffffc0147a2c
> > > > > (gdb) p event->ksymbol.len
> > > > > $3 = 306
> > > >
> > > > Hmm.. so I think the situation is like below.
> > > >
> > > >                    (bpf_trampoline_6442522522)
> > > >             +---------------------------------------+
> > > >       |                                       |
> > > >       |       +------------------------+      |
> > > >       |       | (bpf_prog_40ddf486...) | <----+----  adding this
> > > >       |       |                        |      |
> > > >       |       |                        |      |
> > > >       |   c0147a2c                            |
> > > >       |                                       |
> > > >         c0147640                                 c0148640
> > > >
> > > > And it failed to add bpf_prog_40ddf486... in check_invariants() because
> > > > the end address is smaller than the previous map.
> > >
> > > No, it didn't fail to add, it managed to do it which left the kernel
> > > maps in a broken state, with overlappings while it had a cleared
> > > ends_broken, then, later, when the checks_invariant is finally called at
> > > perf record exit time:
> >
> > Nope, __maps__insert() should notice that the ends are broken and set
> > it:
> >
> >         if (nr_maps == 1) {
> >                 /* If there's just 1 entry then maps are sorted. */
> >                 maps__set_maps_by_address_sorted(maps, true);
> >                 maps__set_maps_by_name_sorted(maps, maps_by_name != NULL);
> >         } else {
> >                 /* Sorted if maps were already sorted and this map starts after the last one. */
> >                 maps__set_maps_by_address_sorted(maps,
> >                         maps__maps_by_address_sorted(maps) &&
> >                         map__end(maps_by_address[nr_maps - 2]) <= map__start(new));
> >                 maps__set_maps_by_name_sorted(maps, false);
> >         }
> >         if (map__end(new) < map__start(new))
> >                 RC_CHK_ACCESS(maps)->ends_broken = true;
> >
> >
> > humm, RC_CHK_ACCESS(maps)->ends_broken should be set for the case we
> > have and I think it isn't being... Then the bpf trampoline map that is
> > the last entry to be added is before the last entry and thus
> > maps_by_address_sorted is set to false, ends_broken continues false and
> > at the end maps_by_address_sorted is set to true and the last
> > check_invariants triggerrs the asserts...
>
> Right, probably it needs to set the ends_broken when the end address of
> the new map is smaller than the previous (but the start address is
> bigger) and fixup the end address when it sorts the maps by address.

Ugh, I get git blamed for ends_broken and I was wondering what the heck it is:
https://lore.kernel.org/all/20240210031746.4057262-2-irogers@google.com/
My memory is that when the rb-tree was built the maps put in it could
be broken and ends_broken was to capture we were in this state as the
sorting would get broken, invariants be off, etc.. The rb-tree
constructing code would then call maps__fixup_end. Having the caller
call maps__fixup_end seems error prone, as does the whole
"ends_broken" thing - remember I was in the code to fix memory leaks
so modifying the maps API wasn't front of mind. I added ends_broken,
the original rb-tree had no notion of it, because I was trying to get
the invariants right for the testing I could do and ends_broken was
the pragmatic thing to do for odd cases like kernel modules before
maps__fixup_end is called.

The maps API has evolved and we have a pretty robust, but possibly not
fast, maps__fixup_overlap_and_insert:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/maps.h?h=perf-tools-next#n69
I think ideally we'd make maps__insert uphold the invariants and not
have ends_broken. I'm worried that making ends_broken more load
bearing isn't the right thing to do, we may even be able to not have
the variable for the "ifndef NDEBUG" case, which making it load
bearing would completely defeat.

So I think the fix here should be to understand the maps construction
code for the modules, try to work out why maps__fixup_end wasn't
called, perhaps migrate the code to maps__fixup_overlap_and_insert or
add a missed maps__fixup_end call.

Given the blame I kind of feel responsible for this, but the real
issue is adding the invariant checks has caught a latent bug that the
rb-tree code would have just ignored and possibly been broken as a
consequence. I lack bandwidth and a reproduction so thank you for
digging into this.

Thanks,
Ian

Re: [PATCH] perf report: Add 'tgid' sort key

[Namhyung Kim]

On Thu, Feb 20, 2025 at 09:12:46AM -0800, Ian Rogers wrote:
> On Wed, Feb 19, 2025 at 1:10 PM Namhyung Kim <namhyung@kernel.org> wrote:
> >
> > On Wed, Feb 19, 2025 at 03:47:44PM +0100, Arnaldo Carvalho de Melo wrote:
> > > On Wed, Feb 19, 2025 at 03:37:10PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > On Tue, Feb 18, 2025 at 02:03:01PM -0800, Namhyung Kim wrote:
> > > > > On Tue, Feb 18, 2025 at 10:01:33PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > On Tue, Feb 18, 2025 at 09:36:52PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > > So the call to maps_fixup_end() will set maps->end_broken to false,
> > > > > > > since it fixed up the map ends, etc, but then we insert more maps with
> > > > > > > broken ends:
> > > > > >
> > > > > > > #6  0x0000000000633d52 in check_invariants (maps=0xf967c0) at util/maps.c:95
> > > > > > > 95                                            assert(map__end(prev) <= map__end(map));
> > > > > > > (gdb) p prev->dso->name
> > > > > > > $1 = 0xfc47ab "bpf_trampoline_6442522522"
> > > > > >
> > > > > > So the above map is created overlapping a previously existing map:
> > > > > >
> > > > > > root@number:~# perf probe -l
> > > > > >   probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps)
> > > > > >   probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end)
> > > > > > root@number:~#
> > > > > >
> > > > > > root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep
> > > > > > <SNIP>
> > > > > >    319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8)
> > > > > >    319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8)
> > > > > >    319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c)
> > > > > >    319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec)
> > > > > >    319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2)
> > > > > >    319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640)
> > > > > > [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ]
> > > > > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > > > > > root@number:~#
> > > > > >
> > > > > > So a PERF_RECORD_KSYMBOL processing will add a map for
> > > > > > "bpf_trampoline_6442522522" that has its start after before the
> > > > > > "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after
> > > > > > "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
> > > > > >
> > > > > > machine__process_ksymbol_register() does:
> > > > > >
> > > > > > 713                     map__set_start(map, event->ksymbol.addr);
> > > > > > 714                     map__set_end(map, map__start(map) + event->ksymbol.len);
> > > > > > 715                     err = maps__insert(machine__kernel_maps(machine), map);
> > > > > >
> > > > > > And:
> > > > > >
> > > > > > (gdb) p /x event->ksymbol.addr
> > > > > > $2 = 0xffffffffc0147a2c
> > > > > > (gdb) p event->ksymbol.len
> > > > > > $3 = 306
> > > > >
> > > > > Hmm.. so I think the situation is like below.
> > > > >
> > > > >              (bpf_trampoline_6442522522)
> > > > >       +---------------------------------------+
> > > > >       |                                       |
> > > > >       |       +------------------------+      |
> > > > >       |       | (bpf_prog_40ddf486...) | <----+----  adding this
> > > > >       |       |                        |      |
> > > > >       |       |                        |      |
> > > > >       |   c0147a2c                            |
> > > > >       |                                       |
> > > > >   c0147640                                 c0148640
> > > > >
> > > > > And it failed to add bpf_prog_40ddf486... in check_invariants() because
> > > > > the end address is smaller than the previous map.
> > > >
> > > > No, it didn't fail to add, it managed to do it which left the kernel
> > > > maps in a broken state, with overlappings while it had a cleared
> > > > ends_broken, then, later, when the checks_invariant is finally called at
> > > > perf record exit time:
> > >
> > > Nope, __maps__insert() should notice that the ends are broken and set
> > > it:
> > >
> > >         if (nr_maps == 1) {
> > >                 /* If there's just 1 entry then maps are sorted. */
> > >                 maps__set_maps_by_address_sorted(maps, true);
> > >                 maps__set_maps_by_name_sorted(maps, maps_by_name != NULL);
> > >         } else {
> > >                 /* Sorted if maps were already sorted and this map starts after the last one. */
> > >                 maps__set_maps_by_address_sorted(maps,
> > >                         maps__maps_by_address_sorted(maps) &&
> > >                         map__end(maps_by_address[nr_maps - 2]) <= map__start(new));
> > >                 maps__set_maps_by_name_sorted(maps, false);
> > >         }
> > >         if (map__end(new) < map__start(new))
> > >                 RC_CHK_ACCESS(maps)->ends_broken = true;
> > >
> > >
> > > humm, RC_CHK_ACCESS(maps)->ends_broken should be set for the case we
> > > have and I think it isn't being... Then the bpf trampoline map that is
> > > the last entry to be added is before the last entry and thus
> > > maps_by_address_sorted is set to false, ends_broken continues false and
> > > at the end maps_by_address_sorted is set to true and the last
> > > check_invariants triggerrs the asserts...
> >
> > Right, probably it needs to set the ends_broken when the end address of
> > the new map is smaller than the previous (but the start address is
> > bigger) and fixup the end address when it sorts the maps by address.
> 
> Ugh, I get git blamed for ends_broken and I was wondering what the heck it is:
> https://lore.kernel.org/all/20240210031746.4057262-2-irogers@google.com/
> My memory is that when the rb-tree was built the maps put in it could
> be broken and ends_broken was to capture we were in this state as the
> sorting would get broken, invariants be off, etc.. The rb-tree
> constructing code would then call maps__fixup_end. Having the caller
> call maps__fixup_end seems error prone, as does the whole
> "ends_broken" thing - remember I was in the code to fix memory leaks
> so modifying the maps API wasn't front of mind. I added ends_broken,
> the original rb-tree had no notion of it, because I was trying to get
> the invariants right for the testing I could do and ends_broken was
> the pragmatic thing to do for odd cases like kernel modules before
> maps__fixup_end is called.
> 
> The maps API has evolved and we have a pretty robust, but possibly not
> fast, maps__fixup_overlap_and_insert:
> https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/maps.h?h=perf-tools-next#n69
> I think ideally we'd make maps__insert uphold the invariants and not
> have ends_broken. I'm worried that making ends_broken more load
> bearing isn't the right thing to do, we may even be able to not have
> the variable for the "ifndef NDEBUG" case, which making it load
> bearing would completely defeat.
> 
> So I think the fix here should be to understand the maps construction
> code for the modules, try to work out why maps__fixup_end wasn't
> called, perhaps migrate the code to maps__fixup_overlap_and_insert or
> add a missed maps__fixup_end call.

IIUC module size in /proc/modules are wrong due to the reason in the
commit 876e80cf83d10585 ("perf tools: Fixup end address of modules") and
it called maps__fixup_end() for that.

But the problem is some BPF maps processed at real-time during the
build-id processing at the end of perf record.  One map is inside of
another and check_invariants() didn't expect such maps and crashed.

Maybe we can fix maps__insert() to check such condition and fix it
everytime.  But it means it needs to sort the maps which would add big
overhead we had before.  So I just wanted to set the flag quickly and
to fix the end address when it calls maps__find() or similar later.

Thanks,
Namhyung

> 
> Given the blame I kind of feel responsible for this, but the real
> issue is adding the invariant checks has caught a latent bug that the
> rb-tree code would have just ignored and possibly been broken as a
> consequence. I lack bandwidth and a reproduction so thank you for
> digging into this.
> 
> Thanks,
> Ian

Re: [PATCH] perf report: Add 'tgid' sort key

[Ian Rogers]

On Thu, Feb 20, 2025 at 11:04 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> On Thu, Feb 20, 2025 at 09:12:46AM -0800, Ian Rogers wrote:
> > On Wed, Feb 19, 2025 at 1:10 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > >
> > > On Wed, Feb 19, 2025 at 03:47:44PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > On Wed, Feb 19, 2025 at 03:37:10PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > On Tue, Feb 18, 2025 at 02:03:01PM -0800, Namhyung Kim wrote:
> > > > > > On Tue, Feb 18, 2025 at 10:01:33PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > > On Tue, Feb 18, 2025 at 09:36:52PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > > > So the call to maps_fixup_end() will set maps->end_broken to false,
> > > > > > > > since it fixed up the map ends, etc, but then we insert more maps with
> > > > > > > > broken ends:
> > > > > > >
> > > > > > > > #6  0x0000000000633d52 in check_invariants (maps=0xf967c0) at util/maps.c:95
> > > > > > > > 95                                            assert(map__end(prev) <= map__end(map));
> > > > > > > > (gdb) p prev->dso->name
> > > > > > > > $1 = 0xfc47ab "bpf_trampoline_6442522522"
> > > > > > >
> > > > > > > So the above map is created overlapping a previously existing map:
> > > > > > >
> > > > > > > root@number:~# perf probe -l
> > > > > > >   probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps)
> > > > > > >   probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end)
> > > > > > > root@number:~#
> > > > > > >
> > > > > > > root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep
> > > > > > > <SNIP>
> > > > > > >    319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8)
> > > > > > >    319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8)
> > > > > > >    319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c)
> > > > > > >    319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec)
> > > > > > >    319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2)
> > > > > > >    319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640)
> > > > > > > [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ]
> > > > > > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > > > > > > root@number:~#
> > > > > > >
> > > > > > > So a PERF_RECORD_KSYMBOL processing will add a map for
> > > > > > > "bpf_trampoline_6442522522" that has its start after before the
> > > > > > > "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after
> > > > > > > "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
> > > > > > >
> > > > > > > machine__process_ksymbol_register() does:
> > > > > > >
> > > > > > > 713                     map__set_start(map, event->ksymbol.addr);
> > > > > > > 714                     map__set_end(map, map__start(map) + event->ksymbol.len);
> > > > > > > 715                     err = maps__insert(machine__kernel_maps(machine), map);
> > > > > > >
> > > > > > > And:
> > > > > > >
> > > > > > > (gdb) p /x event->ksymbol.addr
> > > > > > > $2 = 0xffffffffc0147a2c
> > > > > > > (gdb) p event->ksymbol.len
> > > > > > > $3 = 306
> > > > > >
> > > > > > Hmm.. so I think the situation is like below.
> > > > > >
> > > > > >              (bpf_trampoline_6442522522)
> > > > > >       +---------------------------------------+
> > > > > >       |                                       |
> > > > > >       |       +------------------------+      |
> > > > > >       |       | (bpf_prog_40ddf486...) | <----+----  adding this
> > > > > >       |       |                        |      |
> > > > > >       |       |                        |      |
> > > > > >       |   c0147a2c                            |
> > > > > >       |                                       |
> > > > > >   c0147640                                 c0148640
> > > > > >
> > > > > > And it failed to add bpf_prog_40ddf486... in check_invariants() because
> > > > > > the end address is smaller than the previous map.
> > > > >
> > > > > No, it didn't fail to add, it managed to do it which left the kernel
> > > > > maps in a broken state, with overlappings while it had a cleared
> > > > > ends_broken, then, later, when the checks_invariant is finally called at
> > > > > perf record exit time:
> > > >
> > > > Nope, __maps__insert() should notice that the ends are broken and set
> > > > it:
> > > >
> > > >         if (nr_maps == 1) {
> > > >                 /* If there's just 1 entry then maps are sorted. */
> > > >                 maps__set_maps_by_address_sorted(maps, true);
> > > >                 maps__set_maps_by_name_sorted(maps, maps_by_name != NULL);
> > > >         } else {
> > > >                 /* Sorted if maps were already sorted and this map starts after the last one. */
> > > >                 maps__set_maps_by_address_sorted(maps,
> > > >                         maps__maps_by_address_sorted(maps) &&
> > > >                         map__end(maps_by_address[nr_maps - 2]) <= map__start(new));
> > > >                 maps__set_maps_by_name_sorted(maps, false);
> > > >         }
> > > >         if (map__end(new) < map__start(new))
> > > >                 RC_CHK_ACCESS(maps)->ends_broken = true;
> > > >
> > > >
> > > > humm, RC_CHK_ACCESS(maps)->ends_broken should be set for the case we
> > > > have and I think it isn't being... Then the bpf trampoline map that is
> > > > the last entry to be added is before the last entry and thus
> > > > maps_by_address_sorted is set to false, ends_broken continues false and
> > > > at the end maps_by_address_sorted is set to true and the last
> > > > check_invariants triggerrs the asserts...
> > >
> > > Right, probably it needs to set the ends_broken when the end address of
> > > the new map is smaller than the previous (but the start address is
> > > bigger) and fixup the end address when it sorts the maps by address.
> >
> > Ugh, I get git blamed for ends_broken and I was wondering what the heck it is:
> > https://lore.kernel.org/all/20240210031746.4057262-2-irogers@google.com/
> > My memory is that when the rb-tree was built the maps put in it could
> > be broken and ends_broken was to capture we were in this state as the
> > sorting would get broken, invariants be off, etc.. The rb-tree
> > constructing code would then call maps__fixup_end. Having the caller
> > call maps__fixup_end seems error prone, as does the whole
> > "ends_broken" thing - remember I was in the code to fix memory leaks
> > so modifying the maps API wasn't front of mind. I added ends_broken,
> > the original rb-tree had no notion of it, because I was trying to get
> > the invariants right for the testing I could do and ends_broken was
> > the pragmatic thing to do for odd cases like kernel modules before
> > maps__fixup_end is called.
> >
> > The maps API has evolved and we have a pretty robust, but possibly not
> > fast, maps__fixup_overlap_and_insert:
> > https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/maps.h?h=perf-tools-next#n69
> > I think ideally we'd make maps__insert uphold the invariants and not
> > have ends_broken. I'm worried that making ends_broken more load
> > bearing isn't the right thing to do, we may even be able to not have
> > the variable for the "ifndef NDEBUG" case, which making it load
> > bearing would completely defeat.
> >
> > So I think the fix here should be to understand the maps construction
> > code for the modules, try to work out why maps__fixup_end wasn't
> > called, perhaps migrate the code to maps__fixup_overlap_and_insert or
> > add a missed maps__fixup_end call.
>
> IIUC module size in /proc/modules are wrong due to the reason in the
> commit 876e80cf83d10585 ("perf tools: Fixup end address of modules") and
> it called maps__fixup_end() for that.
>
> But the problem is some BPF maps processed at real-time during the
> build-id processing at the end of perf record.  One map is inside of
> another and check_invariants() didn't expect such maps and crashed.

I thought the real-time processing had to use
maps__fixup_overlap_and_insert (rather than maps__insert) as mmap
events only give us VMA data and two mmaps may have been merged.
Shouldn't doing this change be the simplest fix?

Thanks,
Ian

> Maybe we can fix maps__insert() to check such condition and fix it
> everytime.  But it means it needs to sort the maps which would add big
> overhead we had before.  So I just wanted to set the flag quickly and
> to fix the end address when it calls maps__find() or similar later.
>
> Thanks,
> Namhyung
>
> >
> > Given the blame I kind of feel responsible for this, but the real
> > issue is adding the invariant checks has caught a latent bug that the
> > rb-tree code would have just ignored and possibly been broken as a
> > consequence. I lack bandwidth and a reproduction so thank you for
> > digging into this.
> >
> > Thanks,
> > Ian

Re: [PATCH] perf report: Add 'tgid' sort key

[Namhyung Kim]

On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
> On Thu, Feb 20, 2025 at 11:04 PM Namhyung Kim <namhyung@kernel.org> wrote:
> >
> > On Thu, Feb 20, 2025 at 09:12:46AM -0800, Ian Rogers wrote:
> > > On Wed, Feb 19, 2025 at 1:10 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > > >
> > > > On Wed, Feb 19, 2025 at 03:47:44PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > On Wed, Feb 19, 2025 at 03:37:10PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > On Tue, Feb 18, 2025 at 02:03:01PM -0800, Namhyung Kim wrote:
> > > > > > > On Tue, Feb 18, 2025 at 10:01:33PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > > > On Tue, Feb 18, 2025 at 09:36:52PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > > > > So the call to maps_fixup_end() will set maps->end_broken to false,
> > > > > > > > > since it fixed up the map ends, etc, but then we insert more maps with
> > > > > > > > > broken ends:
> > > > > > > >
> > > > > > > > > #6  0x0000000000633d52 in check_invariants (maps=0xf967c0) at util/maps.c:95
> > > > > > > > > 95                                            assert(map__end(prev) <= map__end(map));
> > > > > > > > > (gdb) p prev->dso->name
> > > > > > > > > $1 = 0xfc47ab "bpf_trampoline_6442522522"
> > > > > > > >
> > > > > > > > So the above map is created overlapping a previously existing map:
> > > > > > > >
> > > > > > > > root@number:~# perf probe -l
> > > > > > > >   probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps)
> > > > > > > >   probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end)
> > > > > > > > root@number:~#
> > > > > > > >
> > > > > > > > root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep
> > > > > > > > <SNIP>
> > > > > > > >    319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8)
> > > > > > > >    319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8)
> > > > > > > >    319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c)
> > > > > > > >    319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec)
> > > > > > > >    319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2)
> > > > > > > >    319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640)
> > > > > > > > [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ]
> > > > > > > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > > > > > > > root@number:~#
> > > > > > > >
> > > > > > > > So a PERF_RECORD_KSYMBOL processing will add a map for
> > > > > > > > "bpf_trampoline_6442522522" that has its start after before the
> > > > > > > > "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after
> > > > > > > > "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
> > > > > > > >
> > > > > > > > machine__process_ksymbol_register() does:
> > > > > > > >
> > > > > > > > 713                     map__set_start(map, event->ksymbol.addr);
> > > > > > > > 714                     map__set_end(map, map__start(map) + event->ksymbol.len);
> > > > > > > > 715                     err = maps__insert(machine__kernel_maps(machine), map);
> > > > > > > >
> > > > > > > > And:
> > > > > > > >
> > > > > > > > (gdb) p /x event->ksymbol.addr
> > > > > > > > $2 = 0xffffffffc0147a2c
> > > > > > > > (gdb) p event->ksymbol.len
> > > > > > > > $3 = 306
> > > > > > >
> > > > > > > Hmm.. so I think the situation is like below.
> > > > > > >
> > > > > > >              (bpf_trampoline_6442522522)
> > > > > > >       +---------------------------------------+
> > > > > > >       |                                       |
> > > > > > >       |       +------------------------+      |
> > > > > > >       |       | (bpf_prog_40ddf486...) | <----+----  adding this
> > > > > > >       |       |                        |      |
> > > > > > >       |       |                        |      |
> > > > > > >       |   c0147a2c                            |
> > > > > > >       |                                       |
> > > > > > >   c0147640                                 c0148640
> > > > > > >
> > > > > > > And it failed to add bpf_prog_40ddf486... in check_invariants() because
> > > > > > > the end address is smaller than the previous map.
> > > > > >
> > > > > > No, it didn't fail to add, it managed to do it which left the kernel
> > > > > > maps in a broken state, with overlappings while it had a cleared
> > > > > > ends_broken, then, later, when the checks_invariant is finally called at
> > > > > > perf record exit time:
> > > > >
> > > > > Nope, __maps__insert() should notice that the ends are broken and set
> > > > > it:
> > > > >
> > > > >         if (nr_maps == 1) {
> > > > >                 /* If there's just 1 entry then maps are sorted. */
> > > > >                 maps__set_maps_by_address_sorted(maps, true);
> > > > >                 maps__set_maps_by_name_sorted(maps, maps_by_name != NULL);
> > > > >         } else {
> > > > >                 /* Sorted if maps were already sorted and this map starts after the last one. */
> > > > >                 maps__set_maps_by_address_sorted(maps,
> > > > >                         maps__maps_by_address_sorted(maps) &&
> > > > >                         map__end(maps_by_address[nr_maps - 2]) <= map__start(new));
> > > > >                 maps__set_maps_by_name_sorted(maps, false);
> > > > >         }
> > > > >         if (map__end(new) < map__start(new))
> > > > >                 RC_CHK_ACCESS(maps)->ends_broken = true;
> > > > >
> > > > >
> > > > > humm, RC_CHK_ACCESS(maps)->ends_broken should be set for the case we
> > > > > have and I think it isn't being... Then the bpf trampoline map that is
> > > > > the last entry to be added is before the last entry and thus
> > > > > maps_by_address_sorted is set to false, ends_broken continues false and
> > > > > at the end maps_by_address_sorted is set to true and the last
> > > > > check_invariants triggerrs the asserts...
> > > >
> > > > Right, probably it needs to set the ends_broken when the end address of
> > > > the new map is smaller than the previous (but the start address is
> > > > bigger) and fixup the end address when it sorts the maps by address.
> > >
> > > Ugh, I get git blamed for ends_broken and I was wondering what the heck it is:
> > > https://lore.kernel.org/all/20240210031746.4057262-2-irogers@google.com/
> > > My memory is that when the rb-tree was built the maps put in it could
> > > be broken and ends_broken was to capture we were in this state as the
> > > sorting would get broken, invariants be off, etc.. The rb-tree
> > > constructing code would then call maps__fixup_end. Having the caller
> > > call maps__fixup_end seems error prone, as does the whole
> > > "ends_broken" thing - remember I was in the code to fix memory leaks
> > > so modifying the maps API wasn't front of mind. I added ends_broken,
> > > the original rb-tree had no notion of it, because I was trying to get
> > > the invariants right for the testing I could do and ends_broken was
> > > the pragmatic thing to do for odd cases like kernel modules before
> > > maps__fixup_end is called.
> > >
> > > The maps API has evolved and we have a pretty robust, but possibly not
> > > fast, maps__fixup_overlap_and_insert:
> > > https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/maps.h?h=perf-tools-next#n69
> > > I think ideally we'd make maps__insert uphold the invariants and not
> > > have ends_broken. I'm worried that making ends_broken more load
> > > bearing isn't the right thing to do, we may even be able to not have
> > > the variable for the "ifndef NDEBUG" case, which making it load
> > > bearing would completely defeat.
> > >
> > > So I think the fix here should be to understand the maps construction
> > > code for the modules, try to work out why maps__fixup_end wasn't
> > > called, perhaps migrate the code to maps__fixup_overlap_and_insert or
> > > add a missed maps__fixup_end call.
> >
> > IIUC module size in /proc/modules are wrong due to the reason in the
> > commit 876e80cf83d10585 ("perf tools: Fixup end address of modules") and
> > it called maps__fixup_end() for that.
> >
> > But the problem is some BPF maps processed at real-time during the
> > build-id processing at the end of perf record.  One map is inside of
> > another and check_invariants() didn't expect such maps and crashed.
> 
> I thought the real-time processing had to use
> maps__fixup_overlap_and_insert (rather than maps__insert) as mmap
> events only give us VMA data and two mmaps may have been merged.
> Shouldn't doing this change be the simplest fix?

Make sense.  How about this?

Thanks,
Namhyung


---8<---
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 316f0879e5e08d66..d80b34717090db44 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -717,7 +717,7 @@ static int machine__process_ksymbol_register(struct machine *machine,
 
 		map__set_start(map, event->ksymbol.addr);
 		map__set_end(map, map__start(map) + event->ksymbol.len);
-		err = maps__insert(machine__kernel_maps(machine), map);
+		err = maps__fixup_overlap_and_insert(machine__kernel_maps(machine), map);
 		if (err) {
 			err = -ENOMEM;
 			goto out;
@@ -1459,8 +1459,6 @@ static int machine__create_modules(struct machine *machine)
 	if (modules__parse(modules, machine, machine__create_module))
 		return -1;
 
-	maps__fixup_end(machine__kernel_maps(machine));
-
 	if (!machine__set_modules_path(machine))
 		return 0;
 
@@ -1554,6 +1552,8 @@ int machine__create_kernel_maps(struct machine *machine)
 		}
 	}
 
+	maps__fixup_end(machine__kernel_maps(machine));
+
 out_put:
 	dso__put(kernel);
 	return ret;

Re: [PATCH] perf report: Add 'tgid' sort key

[Ian Rogers]

On Mon, Feb 24, 2025 at 6:51 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
> > On Thu, Feb 20, 2025 at 11:04 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > >
> > > On Thu, Feb 20, 2025 at 09:12:46AM -0800, Ian Rogers wrote:
> > > > On Wed, Feb 19, 2025 at 1:10 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > > > >
> > > > > On Wed, Feb 19, 2025 at 03:47:44PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > On Wed, Feb 19, 2025 at 03:37:10PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > > On Tue, Feb 18, 2025 at 02:03:01PM -0800, Namhyung Kim wrote:
> > > > > > > > On Tue, Feb 18, 2025 at 10:01:33PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > > > > On Tue, Feb 18, 2025 at 09:36:52PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > > > > > So the call to maps_fixup_end() will set maps->end_broken to false,
> > > > > > > > > > since it fixed up the map ends, etc, but then we insert more maps with
> > > > > > > > > > broken ends:
> > > > > > > > >
> > > > > > > > > > #6  0x0000000000633d52 in check_invariants (maps=0xf967c0) at util/maps.c:95
> > > > > > > > > > 95                                            assert(map__end(prev) <= map__end(map));
> > > > > > > > > > (gdb) p prev->dso->name
> > > > > > > > > > $1 = 0xfc47ab "bpf_trampoline_6442522522"
> > > > > > > > >
> > > > > > > > > So the above map is created overlapping a previously existing map:
> > > > > > > > >
> > > > > > > > > root@number:~# perf probe -l
> > > > > > > > >   probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps)
> > > > > > > > >   probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end)
> > > > > > > > > root@number:~#
> > > > > > > > >
> > > > > > > > > root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep
> > > > > > > > > <SNIP>
> > > > > > > > >    319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8)
> > > > > > > > >    319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8)
> > > > > > > > >    319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c)
> > > > > > > > >    319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec)
> > > > > > > > >    319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2)
> > > > > > > > >    319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640)
> > > > > > > > > [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ]
> > > > > > > > > perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
> > > > > > > > > root@number:~#
> > > > > > > > >
> > > > > > > > > So a PERF_RECORD_KSYMBOL processing will add a map for
> > > > > > > > > "bpf_trampoline_6442522522" that has its start after before the
> > > > > > > > > "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after
> > > > > > > > > "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
> > > > > > > > >
> > > > > > > > > machine__process_ksymbol_register() does:
> > > > > > > > >
> > > > > > > > > 713                     map__set_start(map, event->ksymbol.addr);
> > > > > > > > > 714                     map__set_end(map, map__start(map) + event->ksymbol.len);
> > > > > > > > > 715                     err = maps__insert(machine__kernel_maps(machine), map);
> > > > > > > > >
> > > > > > > > > And:
> > > > > > > > >
> > > > > > > > > (gdb) p /x event->ksymbol.addr
> > > > > > > > > $2 = 0xffffffffc0147a2c
> > > > > > > > > (gdb) p event->ksymbol.len
> > > > > > > > > $3 = 306
> > > > > > > >
> > > > > > > > Hmm.. so I think the situation is like below.
> > > > > > > >
> > > > > > > >              (bpf_trampoline_6442522522)
> > > > > > > >       +---------------------------------------+
> > > > > > > >       |                                       |
> > > > > > > >       |       +------------------------+      |
> > > > > > > >       |       | (bpf_prog_40ddf486...) | <----+----  adding this
> > > > > > > >       |       |                        |      |
> > > > > > > >       |       |                        |      |
> > > > > > > >       |   c0147a2c                            |
> > > > > > > >       |                                       |
> > > > > > > >   c0147640                                 c0148640
> > > > > > > >
> > > > > > > > And it failed to add bpf_prog_40ddf486... in check_invariants() because
> > > > > > > > the end address is smaller than the previous map.
> > > > > > >
> > > > > > > No, it didn't fail to add, it managed to do it which left the kernel
> > > > > > > maps in a broken state, with overlappings while it had a cleared
> > > > > > > ends_broken, then, later, when the checks_invariant is finally called at
> > > > > > > perf record exit time:
> > > > > >
> > > > > > Nope, __maps__insert() should notice that the ends are broken and set
> > > > > > it:
> > > > > >
> > > > > >         if (nr_maps == 1) {
> > > > > >                 /* If there's just 1 entry then maps are sorted. */
> > > > > >                 maps__set_maps_by_address_sorted(maps, true);
> > > > > >                 maps__set_maps_by_name_sorted(maps, maps_by_name != NULL);
> > > > > >         } else {
> > > > > >                 /* Sorted if maps were already sorted and this map starts after the last one. */
> > > > > >                 maps__set_maps_by_address_sorted(maps,
> > > > > >                         maps__maps_by_address_sorted(maps) &&
> > > > > >                         map__end(maps_by_address[nr_maps - 2]) <= map__start(new));
> > > > > >                 maps__set_maps_by_name_sorted(maps, false);
> > > > > >         }
> > > > > >         if (map__end(new) < map__start(new))
> > > > > >                 RC_CHK_ACCESS(maps)->ends_broken = true;
> > > > > >
> > > > > >
> > > > > > humm, RC_CHK_ACCESS(maps)->ends_broken should be set for the case we
> > > > > > have and I think it isn't being... Then the bpf trampoline map that is
> > > > > > the last entry to be added is before the last entry and thus
> > > > > > maps_by_address_sorted is set to false, ends_broken continues false and
> > > > > > at the end maps_by_address_sorted is set to true and the last
> > > > > > check_invariants triggerrs the asserts...
> > > > >
> > > > > Right, probably it needs to set the ends_broken when the end address of
> > > > > the new map is smaller than the previous (but the start address is
> > > > > bigger) and fixup the end address when it sorts the maps by address.
> > > >
> > > > Ugh, I get git blamed for ends_broken and I was wondering what the heck it is:
> > > > https://lore.kernel.org/all/20240210031746.4057262-2-irogers@google.com/
> > > > My memory is that when the rb-tree was built the maps put in it could
> > > > be broken and ends_broken was to capture we were in this state as the
> > > > sorting would get broken, invariants be off, etc.. The rb-tree
> > > > constructing code would then call maps__fixup_end. Having the caller
> > > > call maps__fixup_end seems error prone, as does the whole
> > > > "ends_broken" thing - remember I was in the code to fix memory leaks
> > > > so modifying the maps API wasn't front of mind. I added ends_broken,
> > > > the original rb-tree had no notion of it, because I was trying to get
> > > > the invariants right for the testing I could do and ends_broken was
> > > > the pragmatic thing to do for odd cases like kernel modules before
> > > > maps__fixup_end is called.
> > > >
> > > > The maps API has evolved and we have a pretty robust, but possibly not
> > > > fast, maps__fixup_overlap_and_insert:
> > > > https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/maps.h?h=perf-tools-next#n69
> > > > I think ideally we'd make maps__insert uphold the invariants and not
> > > > have ends_broken. I'm worried that making ends_broken more load
> > > > bearing isn't the right thing to do, we may even be able to not have
> > > > the variable for the "ifndef NDEBUG" case, which making it load
> > > > bearing would completely defeat.
> > > >
> > > > So I think the fix here should be to understand the maps construction
> > > > code for the modules, try to work out why maps__fixup_end wasn't
> > > > called, perhaps migrate the code to maps__fixup_overlap_and_insert or
> > > > add a missed maps__fixup_end call.
> > >
> > > IIUC module size in /proc/modules are wrong due to the reason in the
> > > commit 876e80cf83d10585 ("perf tools: Fixup end address of modules") and
> > > it called maps__fixup_end() for that.
> > >
> > > But the problem is some BPF maps processed at real-time during the
> > > build-id processing at the end of perf record.  One map is inside of
> > > another and check_invariants() didn't expect such maps and crashed.
> >
> > I thought the real-time processing had to use
> > maps__fixup_overlap_and_insert (rather than maps__insert) as mmap
> > events only give us VMA data and two mmaps may have been merged.
> > Shouldn't doing this change be the simplest fix?
>
> Make sense.  How about this?

Lgtm, I have no way to test the issue. Why does maps__fixup_end need
to get pushed later?

Thanks,
Ian

> Thanks,
> Namhyung
>
>
> ---8<---
> diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
> index 316f0879e5e08d66..d80b34717090db44 100644
> --- a/tools/perf/util/machine.c
> +++ b/tools/perf/util/machine.c
> @@ -717,7 +717,7 @@ static int machine__process_ksymbol_register(struct machine *machine,
>
>                 map__set_start(map, event->ksymbol.addr);
>                 map__set_end(map, map__start(map) + event->ksymbol.len);
> -               err = maps__insert(machine__kernel_maps(machine), map);
> +               err = maps__fixup_overlap_and_insert(machine__kernel_maps(machine), map);
>                 if (err) {
>                         err = -ENOMEM;
>                         goto out;
> @@ -1459,8 +1459,6 @@ static int machine__create_modules(struct machine *machine)
>         if (modules__parse(modules, machine, machine__create_module))
>                 return -1;
>
> -       maps__fixup_end(machine__kernel_maps(machine));
> -
>         if (!machine__set_modules_path(machine))
>                 return 0;
>
> @@ -1554,6 +1552,8 @@ int machine__create_kernel_maps(struct machine *machine)
>                 }
>         }
>
> +       maps__fixup_end(machine__kernel_maps(machine));
> +
>  out_put:
>         dso__put(kernel);
>         return ret;

Re: [PATCH] perf report: Add 'tgid' sort key

[Namhyung Kim]

On Mon, Feb 24, 2025 at 08:40:37PM -0800, Ian Rogers wrote:
> On Mon, Feb 24, 2025 at 6:51 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
[SNIP]
> > > I thought the real-time processing had to use
> > > maps__fixup_overlap_and_insert (rather than maps__insert) as mmap
> > > events only give us VMA data and two mmaps may have been merged.
> > > Shouldn't doing this change be the simplest fix?
> >
> > Make sense.  How about this?
> 
> Lgtm, I have no way to test the issue. Why does maps__fixup_end need
> to get pushed later?

I just noticed it would add extra kernel maps after modules.  I think it
should fixup end address of the kernel maps after adding all maps first.

Arnaldo, can you please test this?

Thanks,
Namhyung

> 
> Thanks,
> Ian
> 
> > Thanks,
> > Namhyung
> >
> >
> > ---8<---
> > diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
> > index 316f0879e5e08d66..d80b34717090db44 100644
> > --- a/tools/perf/util/machine.c
> > +++ b/tools/perf/util/machine.c
> > @@ -717,7 +717,7 @@ static int machine__process_ksymbol_register(struct machine *machine,
> >
> >                 map__set_start(map, event->ksymbol.addr);
> >                 map__set_end(map, map__start(map) + event->ksymbol.len);
> > -               err = maps__insert(machine__kernel_maps(machine), map);
> > +               err = maps__fixup_overlap_and_insert(machine__kernel_maps(machine), map);
> >                 if (err) {
> >                         err = -ENOMEM;
> >                         goto out;
> > @@ -1459,8 +1459,6 @@ static int machine__create_modules(struct machine *machine)
> >         if (modules__parse(modules, machine, machine__create_module))
> >                 return -1;
> >
> > -       maps__fixup_end(machine__kernel_maps(machine));
> > -
> >         if (!machine__set_modules_path(machine))
> >                 return 0;
> >
> > @@ -1554,6 +1552,8 @@ int machine__create_kernel_maps(struct machine *machine)
> >                 }
> >         }
> >
> > +       maps__fixup_end(machine__kernel_maps(machine));
> > +
> >  out_put:
> >         dso__put(kernel);
> >         return ret;

Re: [PATCH] perf report: Add 'tgid' sort key

[Arnaldo Carvalho de Melo]

On Mon, Feb 24, 2025 at 11:51:35PM -0800, Namhyung Kim wrote:
> On Mon, Feb 24, 2025 at 08:40:37PM -0800, Ian Rogers wrote:
> > On Mon, Feb 24, 2025 at 6:51 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > > On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
> [SNIP]
> > > > I thought the real-time processing had to use
> > > > maps__fixup_overlap_and_insert (rather than maps__insert) as mmap
> > > > events only give us VMA data and two mmaps may have been merged.
> > > > Shouldn't doing this change be the simplest fix?
> > >
> > > Make sense.  How about this?
> > 
> > Lgtm, I have no way to test the issue. Why does maps__fixup_end need
> > to get pushed later?
> 
> I just noticed it would add extra kernel maps after modules.  I think it
> should fixup end address of the kernel maps after adding all maps first.
> 
> Arnaldo, can you please test this?

Trying it now.

- Arnaldo

Re: [PATCH] perf report: Add 'tgid' sort key

[Arnaldo Carvalho de Melo]

On Tue, Feb 25, 2025 at 08:07:18PM +0100, Arnaldo Carvalho de Melo wrote:
> On Mon, Feb 24, 2025 at 11:51:35PM -0800, Namhyung Kim wrote:
> > On Mon, Feb 24, 2025 at 08:40:37PM -0800, Ian Rogers wrote:
> > > On Mon, Feb 24, 2025 at 6:51 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > > > On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
> > [SNIP]
> > > > > I thought the real-time processing had to use
> > > > > maps__fixup_overlap_and_insert (rather than maps__insert) as mmap
> > > > > events only give us VMA data and two mmaps may have been merged.
> > > > > Shouldn't doing this change be the simplest fix?

> > > > Make sense.  How about this?

> > > Lgtm, I have no way to test the issue. Why does maps__fixup_end need
> > > to get pushed later?

> > I just noticed it would add extra kernel maps after modules.  I think it
> > should fixup end address of the kernel maps after adding all maps first.

> > Arnaldo, can you please test this?
 
> Trying it now.

Now we have something different:

root@number:~# perf record sleep
sleep: missing operand
Try 'sleep --help' for more information.
[ perf record: Woken up 1 times to write data ]
perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed.
Aborted (core dumped)
root@number:~#

Re: [PATCH] perf report: Add 'tgid' sort key

[Arnaldo Carvalho de Melo]

On Tue, Feb 25, 2025 at 08:11:17PM +0100, Arnaldo Carvalho de Melo wrote:
> On Tue, Feb 25, 2025 at 08:07:18PM +0100, Arnaldo Carvalho de Melo wrote:
> > On Mon, Feb 24, 2025 at 11:51:35PM -0800, Namhyung Kim wrote:
> > > On Mon, Feb 24, 2025 at 08:40:37PM -0800, Ian Rogers wrote:
> > > > On Mon, Feb 24, 2025 at 6:51 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > > > > On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
> > > [SNIP]
> > > > > > I thought the real-time processing had to use
> > > > > > maps__fixup_overlap_and_insert (rather than maps__insert) as mmap
> > > > > > events only give us VMA data and two mmaps may have been merged.
> > > > > > Shouldn't doing this change be the simplest fix?
> 
> > > > > Make sense.  How about this?
> 
> > > > Lgtm, I have no way to test the issue. Why does maps__fixup_end need
> > > > to get pushed later?
> 
> > > I just noticed it would add extra kernel maps after modules.  I think it
> > > should fixup end address of the kernel maps after adding all maps first.
> 
> > > Arnaldo, can you please test this?
>  
> > Trying it now.
> 
> Now we have something different:
> 
> root@number:~# perf record sleep
> sleep: missing operand
> Try 'sleep --help' for more information.
> [ perf record: Woken up 1 times to write data ]
> perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed.
> Aborted (core dumped)
> root@number:~#

__maps__insert() does:

        if (dso && dso__kernel(dso)) {
                struct kmap *kmap = map__kmap(new);

                if (kmap)
                        kmap->kmaps = maps;
                else
                        pr_err("Internal error: kernel dso with non kernel map\n");
        }

while maps__fixup_overlap_and_insert() doesn't.

It calls __maps__insert_sorted() that probably should do what
__maps__insert() does?

- Arnaldo

Re: [PATCH] perf report: Add 'tgid' sort key

[Arnaldo Carvalho de Melo]

On Tue, Feb 25, 2025 at 08:25:35PM +0100, Arnaldo Carvalho de Melo wrote:
> On Tue, Feb 25, 2025 at 08:11:17PM +0100, Arnaldo Carvalho de Melo wrote:
> > On Tue, Feb 25, 2025 at 08:07:18PM +0100, Arnaldo Carvalho de Melo wrote:
> > > On Mon, Feb 24, 2025 at 11:51:35PM -0800, Namhyung Kim wrote:
> > > > On Mon, Feb 24, 2025 at 08:40:37PM -0800, Ian Rogers wrote:
> > > > > On Mon, Feb 24, 2025 at 6:51 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > > > > > On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
> > > > [SNIP]
> > > > > > > I thought the real-time processing had to use
> > > > > > > maps__fixup_overlap_and_insert (rather than maps__insert) as mmap
> > > > > > > events only give us VMA data and two mmaps may have been merged.
> > > > > > > Shouldn't doing this change be the simplest fix?
> > 
> > > > > > Make sense.  How about this?
> > 
> > > > > Lgtm, I have no way to test the issue. Why does maps__fixup_end need
> > > > > to get pushed later?
> > 
> > > > I just noticed it would add extra kernel maps after modules.  I think it
> > > > should fixup end address of the kernel maps after adding all maps first.
> > 
> > > > Arnaldo, can you please test this?
> >  
> > > Trying it now.
> > 
> > Now we have something different:
> > 
> > root@number:~# perf record sleep
> > sleep: missing operand
> > Try 'sleep --help' for more information.
> > [ perf record: Woken up 1 times to write data ]
> > perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed.
> > Aborted (core dumped)
> > root@number:~#
> 
> __maps__insert() does:
> 
>         if (dso && dso__kernel(dso)) {
>                 struct kmap *kmap = map__kmap(new);
> 
>                 if (kmap)
>                         kmap->kmaps = maps;
>                 else
>                         pr_err("Internal error: kernel dso with non kernel map\n");
>         }
> 
> while maps__fixup_overlap_and_insert() doesn't.
> 
> It calls __maps__insert_sorted() that probably should do what
> __maps__insert() does?

Ok, so I did the following patch but this case fails:

@@ -910,6 +928,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
                                 */
                                map__put(maps_by_address[i]);
                                maps_by_address[i] = map__get(new);
+                               map__set_kmap(new, maps);
                                check_invariants(maps);
                                return err;
                        }

With:

perf: util/maps.c:110: check_invariants: Assertion `refcount_read(map__refcnt(map)) > 1' failed.

As:

106				/*
107				 * Maps by name maps should be in maps_by_address, so
108				 * the reference count should be higher.
109				 */
110				assert(refcount_read(map__refcnt(map)) > 1);

Since it is just replacing the map in the maps_by_address and not
touching on the maps_by_name? Thus the refcount is just 1:

[ perf record: Woken up 1 times to write data ]
perf: util/maps.c:110: check_invariants: Assertion `refcount_read(map__refcnt(map)) > 1' failed.

Thread 1 "perf" received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44	      return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78
#2  0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007ffff6e36902 in __GI_abort () at abort.c:79
#4  0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bcdd0 "refcount_read(map__refcnt(map)) > 1", 
    file=file@entry=0x7bcc53 "util/maps.c", line=line@entry=110, function=function@entry=0x7bd010 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96
#5  0x00007ffff6e47047 in __assert_fail (assertion=0x7bcdd0 "refcount_read(map__refcnt(map)) > 1", file=0x7bcc53 "util/maps.c", line=110, 
    function=0x7bd010 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105
#6  0x0000000000633e3b in check_invariants (maps=0xf947c0) at util/maps.c:110
#7  0x00000000006362b2 in __maps__fixup_overlap_and_insert (maps=0xf947c0, new=0xfc27e0) at util/maps.c:932
#8  0x000000000063636c in maps__fixup_overlap_and_insert (maps=0xf947c0, new=0xfc27e0) at util/maps.c:954
#9  0x000000000062920a in machine__process_ksymbol_register (machine=0xf94158, event=0x7ffff7fbaba8, sample=0x7fffffffa860) at util/machine.c:715
#10 0x00000000006294ca in machine__process_ksymbol (machine=0xf94158, event=0x7ffff7fbaba8, sample=0x7fffffffa860) at util/machine.c:779
#11 0x00000000005ce2fd in perf_event__process_ksymbol (tool=0xec6ce0 <record>, event=0x7ffff7fbaba8, sample=0x7fffffffa860, machine=0xf94158) at util/event.c:296
#12 0x000000000063b76c in machines__deliver_event (machines=0xf94158, evlist=0xf501f0, event=0x7ffff7fbaba8, sample=0x7fffffffa860, tool=0xec6ce0 <record>, file_offset=35752, 
    file_path=0xf94850 "perf.data") at util/session.c:1334
#13 0x000000000063b951 in perf_session__deliver_event (session=0xf93f40, event=0x7ffff7fbaba8, tool=0xec6ce0 <record>, file_offset=35752, file_path=0xf94850 "perf.data")
    at util/session.c:1367
#14 0x000000000063c745 in perf_session__process_event (session=0xf93f40, event=0x7ffff7fbaba8, file_offset=35752, file_path=0xf94850 "perf.data") at util/session.c:1626
#15 0x000000000063dec5 in process_simple (session=0xf93f40, event=0x7ffff7fbaba8, file_offset=35752, file_path=0xf94850 "perf.data") at util/session.c:2203
#16 0x000000000063db7c in reader__read_event (rd=0x7fffffffafa0, session=0xf93f40, prog=0x7fffffffaf70) at util/session.c:2132
#17 0x000000000063dd76 in reader__process_events (rd=0x7fffffffafa0, session=0xf93f40, prog=0x7fffffffaf70) at util/session.c:2181
#18 0x000000000063e013 in __perf_session__process_events (session=0xf93f40) at util/session.c:2226
#19 0x000000000063ea10 in perf_session__process_events (session=0xf93f40) at util/session.c:2390
#20 0x000000000042d98b in process_buildids (rec=0xec6ce0 <record>) at builtin-record.c:1475
#21 0x000000000042e963 in record__finish_output (rec=0xec6ce0 <record>) at builtin-record.c:1798
#22 0x0000000000431c46 in __cmd_record (rec=0xec6ce0 <record>, argc=2, argv=0x7fffffffde80) at builtin-record.c:2841
#23 0x000000000043513f in cmd_record (argc=2, argv=0x7fffffffde80) at builtin-record.c:4260
#24 0x00000000004bcf65 in run_builtin (p=0xec9d60 <commands+288>, argc=3, argv=0x7fffffffde80) at perf.c:351
#25 0x00000000004bd20c in handle_internal_command (argc=3, argv=0x7fffffffde80) at perf.c:404
#26 0x00000000004bd365 in run_argv (argcp=0x7fffffffdc6c, argv=0x7fffffffdc60) at perf.c:448
#27 0x00000000004bd6ae in main (argc=3, argv=0x7fffffffde80) at perf.c:556
(gdb)
#6  0x0000000000633e3b in check_invariants (maps=0xf947c0) at util/maps.c:110
110				assert(refcount_read(map__refcnt(map)) > 1);
(gdb) p map
$2 = (struct map *) 0xfe6060
(gdb) p map->dso
$3 = (struct dso *) 0xfe5ea0
(gdb) p map->dso->name
$4 = 0xfe602b "bpf_prog_6deef7357e7b4530_sd_fw_ingress"
(gdb) p refcount_read(map__refcnt(map))
$5 = 1
(gdb) fr 7
#7  0x00000000006362b2 in __maps__fixup_overlap_and_insert (maps=0xf947c0, new=0xfc27e0) at util/maps.c:932
932					check_invariants(maps);
(gdb) p next
$6 = (struct map *) 0xfe4bc0
(gdb) p next->dso
$7 = (struct dso *) 0xfe4a00
(gdb) p next->dso->name
$8 = 0xfe4b8b "bpf_prog_6deef7357e7b4530_sd_fw_egress"
(gdb) 
(gdb) p new->dso->name
$9 = 0xfc27ab "bpf_trampoline_6442522521"
(gdb) p /x map__start(next)
$12 = 0xffffffffc012b158
(gdb) p /x map__start(new)
$13 = 0xffffffffc0129640
(gdb) p /x map__end(next)
$14 = 0xffffffffc012b198
(gdb) p /x map__end(new)
$15 = 0xffffffffc012a640
(gdb) 

So its again that case of overlapping maps...

Ah, the patch below is on top of Namhyungs.

- Arnaldo

diff --git a/tools/perf/util/maps.c b/tools/perf/util/maps.c
index 09c9cc326c08d435..e413602afaeb2e83 100644
--- a/tools/perf/util/maps.c
+++ b/tools/perf/util/maps.c
@@ -428,11 +428,29 @@ static unsigned int maps__by_name_index(const struct maps *maps, const struct ma
 	return -1;
 }
 
+static void map__set_kmap(struct map *map, struct maps *maps)
+{
+	struct dso *dso;
+
+	if (map == NULL)
+		return;
+
+	dso = map__dso(map);
+
+	if (dso && dso__kernel(dso)) {
+                struct kmap *kmap = map__kmap(map);
+
+                if (kmap)
+                        kmap->kmaps = maps;
+                else
+                        pr_err("Internal error: kernel dso with non kernel map\n");
+        }
+}
+
 static int __maps__insert(struct maps *maps, struct map *new)
 {
 	struct map **maps_by_address = maps__maps_by_address(maps);
 	struct map **maps_by_name = maps__maps_by_name(maps);
-	const struct dso *dso = map__dso(new);
 	unsigned int nr_maps = maps__nr_maps(maps);
 	unsigned int nr_allocate = RC_CHK_ACCESS(maps)->nr_maps_allocated;
 
@@ -483,14 +501,9 @@ static int __maps__insert(struct maps *maps, struct map *new)
 	}
 	if (map__end(new) < map__start(new))
 		RC_CHK_ACCESS(maps)->ends_broken = true;
-	if (dso && dso__kernel(dso)) {
-		struct kmap *kmap = map__kmap(new);
 
-		if (kmap)
-			kmap->kmaps = maps;
-		else
-			pr_err("Internal error: kernel dso with non kernel map\n");
-	}
+	map__set_kmap(new, maps);
+
 	return 0;
 }
 
@@ -784,7 +797,12 @@ static int __maps__insert_sorted(struct maps *maps, unsigned int first_after_ind
 			maps_by_name[nr_maps + 1] = map__get(new2);
 	}
 	RC_CHK_ACCESS(maps)->nr_maps = nr_maps + to_add;
+
 	maps__set_maps_by_name_sorted(maps, false);
+
+	map__set_kmap(new1, maps);
+	map__set_kmap(new2, maps);
+
 	check_invariants(maps);
 	return 0;
 }
@@ -910,6 +928,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
 				 */
 				map__put(maps_by_address[i]);
 				maps_by_address[i] = map__get(new);
+				map__set_kmap(new, maps);
 				check_invariants(maps);
 				return err;
 			}

Re: [PATCH] perf report: Add 'tgid' sort key

[Namhyung Kim]

On Tue, Feb 25, 2025 at 08:48:56PM +0100, Arnaldo Carvalho de Melo wrote:
> On Tue, Feb 25, 2025 at 08:25:35PM +0100, Arnaldo Carvalho de Melo wrote:
> > On Tue, Feb 25, 2025 at 08:11:17PM +0100, Arnaldo Carvalho de Melo wrote:
> > > On Tue, Feb 25, 2025 at 08:07:18PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > On Mon, Feb 24, 2025 at 11:51:35PM -0800, Namhyung Kim wrote:
> > > > > On Mon, Feb 24, 2025 at 08:40:37PM -0800, Ian Rogers wrote:
> > > > > > On Mon, Feb 24, 2025 at 6:51 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > > > > > > On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
> > > > > [SNIP]
> > > > > > > > I thought the real-time processing had to use
> > > > > > > > maps__fixup_overlap_and_insert (rather than maps__insert) as mmap
> > > > > > > > events only give us VMA data and two mmaps may have been merged.
> > > > > > > > Shouldn't doing this change be the simplest fix?
> > > 
> > > > > > > Make sense.  How about this?
> > > 
> > > > > > Lgtm, I have no way to test the issue. Why does maps__fixup_end need
> > > > > > to get pushed later?
> > > 
> > > > > I just noticed it would add extra kernel maps after modules.  I think it
> > > > > should fixup end address of the kernel maps after adding all maps first.
> > > 
> > > > > Arnaldo, can you please test this?
> > >  
> > > > Trying it now.
> > > 
> > > Now we have something different:
> > > 
> > > root@number:~# perf record sleep
> > > sleep: missing operand
> > > Try 'sleep --help' for more information.
> > > [ perf record: Woken up 1 times to write data ]
> > > perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed.
> > > Aborted (core dumped)
> > > root@number:~#
> > 
> > __maps__insert() does:
> > 
> >         if (dso && dso__kernel(dso)) {
> >                 struct kmap *kmap = map__kmap(new);
> > 
> >                 if (kmap)
> >                         kmap->kmaps = maps;
> >                 else
> >                         pr_err("Internal error: kernel dso with non kernel map\n");
> >         }
> > 
> > while maps__fixup_overlap_and_insert() doesn't.
> > 
> > It calls __maps__insert_sorted() that probably should do what
> > __maps__insert() does?
> 
> Ok, so I did the following patch but this case fails:
> 
> @@ -910,6 +928,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
>                                  */
>                                 map__put(maps_by_address[i]);
>                                 maps_by_address[i] = map__get(new);
> +                               map__set_kmap(new, maps);
>                                 check_invariants(maps);
>                                 return err;
>                         }
> 
> With:
> 
> perf: util/maps.c:110: check_invariants: Assertion `refcount_read(map__refcnt(map)) > 1' failed.
> 
> As:
> 
> 106				/*
> 107				 * Maps by name maps should be in maps_by_address, so
> 108				 * the reference count should be higher.
> 109				 */
> 110				assert(refcount_read(map__refcnt(map)) > 1);
> 
> Since it is just replacing the map in the maps_by_address and not
> touching on the maps_by_name? Thus the refcount is just 1:

Sounds like it.  Can you add this on top?

Thanks,
Namhyung


---8<---
diff --git a/tools/perf/util/maps.c b/tools/perf/util/maps.c
index 09c9cc326c08d435..3aee0c9e8d421cef 100644
--- a/tools/perf/util/maps.c
+++ b/tools/perf/util/maps.c
@@ -797,7 +797,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
 {
 	int err = 0;
 	FILE *fp = debug_file();
-	unsigned int i;
+	unsigned int i, ni;
 
 	if (!maps__maps_by_address_sorted(maps))
 		__maps__sort_by_address(maps);
@@ -808,6 +808,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
 	 */
 	for (i = first_ending_after(maps, new); i < maps__nr_maps(maps); ) {
 		struct map **maps_by_address = maps__maps_by_address(maps);
+		struct map **maps_by_name = maps__maps_by_name(maps);
 		struct map *pos = maps_by_address[i];
 		struct map *before = NULL, *after = NULL;
 
@@ -827,6 +828,9 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
 			map__fprintf(pos, fp);
 		}
 
+		if (maps_by_name)
+			ni = maps__by_name_index(maps, pos);
+
 		/*
 		 * Now check if we need to create new maps for areas not
 		 * overlapped by the new map:
@@ -871,6 +875,11 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
 		if (before) {
 			map__put(maps_by_address[i]);
 			maps_by_address[i] = before;
+			if (maps_by_name) {
+				map__put(maps_by_name[ni]);
+				maps_by_name[ni] = map__get(before);
+			}
+
 			/* Maps are still ordered, go to next one. */
 			i++;
 			if (after) {
@@ -892,6 +901,10 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
 			 */
 			map__put(maps_by_address[i]);
 			maps_by_address[i] = map__get(new);
+			if (maps_by_name) {
+				map__put(maps_by_name[ni]);
+				maps_by_name[ni] = map__get(new);
+			}
 			err = __maps__insert_sorted(maps, i + 1, after, NULL);
 			map__put(after);
 			check_invariants(maps);
@@ -910,6 +923,10 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
 				 */
 				map__put(maps_by_address[i]);
 				maps_by_address[i] = map__get(new);
+				if (maps_by_name) {
+					map__put(maps_by_name[ni]);
+					maps_by_name[ni] = map__get(new);
+				}
 				check_invariants(maps);
 				return err;
 			}

Re: [PATCH] perf report: Add 'tgid' sort key

[Arnaldo Carvalho de Melo]

On Wed, Feb 26, 2025 at 11:34:13AM -0800, Namhyung Kim wrote:
> On Tue, Feb 25, 2025 at 08:48:56PM +0100, Arnaldo Carvalho de Melo wrote:
> > On Tue, Feb 25, 2025 at 08:25:35PM +0100, Arnaldo Carvalho de Melo wrote:
> > > On Tue, Feb 25, 2025 at 08:11:17PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > On Tue, Feb 25, 2025 at 08:07:18PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > On Mon, Feb 24, 2025 at 11:51:35PM -0800, Namhyung Kim wrote:
> > > > > > On Mon, Feb 24, 2025 at 08:40:37PM -0800, Ian Rogers wrote:
> > > > > > > On Mon, Feb 24, 2025 at 6:51 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > > > > > > > On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
> > > > > > [SNIP]
> > > > > > > > > I thought the real-time processing had to use
> > > > > > > > > maps__fixup_overlap_and_insert (rather than maps__insert) as mmap
> > > > > > > > > events only give us VMA data and two mmaps may have been merged.
> > > > > > > > > Shouldn't doing this change be the simplest fix?
> > > > 
> > > > > > > > Make sense.  How about this?
> > > > 
> > > > > > > Lgtm, I have no way to test the issue. Why does maps__fixup_end need
> > > > > > > to get pushed later?
> > > > 
> > > > > > I just noticed it would add extra kernel maps after modules.  I think it
> > > > > > should fixup end address of the kernel maps after adding all maps first.
> > > > 
> > > > > > Arnaldo, can you please test this?
> > > >  
> > > > > Trying it now.
> > > > 
> > > > Now we have something different:
> > > > 
> > > > root@number:~# perf record sleep
> > > > sleep: missing operand
> > > > Try 'sleep --help' for more information.
> > > > [ perf record: Woken up 1 times to write data ]
> > > > perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed.
> > > > Aborted (core dumped)
> > > > root@number:~#
> > > 
> > > __maps__insert() does:
> > > 
> > >         if (dso && dso__kernel(dso)) {
> > >                 struct kmap *kmap = map__kmap(new);
> > > 
> > >                 if (kmap)
> > >                         kmap->kmaps = maps;
> > >                 else
> > >                         pr_err("Internal error: kernel dso with non kernel map\n");
> > >         }
> > > 
> > > while maps__fixup_overlap_and_insert() doesn't.
> > > 
> > > It calls __maps__insert_sorted() that probably should do what
> > > __maps__insert() does?
> > 
> > Ok, so I did the following patch but this case fails:
> > 
> > @@ -910,6 +928,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
> >                                  */
> >                                 map__put(maps_by_address[i]);
> >                                 maps_by_address[i] = map__get(new);
> > +                               map__set_kmap(new, maps);
> >                                 check_invariants(maps);
> >                                 return err;
> >                         }
> > 
> > With:
> > 
> > perf: util/maps.c:110: check_invariants: Assertion `refcount_read(map__refcnt(map)) > 1' failed.
> > 
> > As:
> > 
> > 106				/*
> > 107				 * Maps by name maps should be in maps_by_address, so
> > 108				 * the reference count should be higher.
> > 109				 */
> > 110				assert(refcount_read(map__refcnt(map)) > 1);
> > 
> > Since it is just replacing the map in the maps_by_address and not
> > touching on the maps_by_name? Thus the refcount is just 1:
> 
> Sounds like it.  Can you add this on top?

Trying, but somehow its not applying cleanly, checking:

⬢ [acme@toolbox perf-tools]$ patch -p1 < ~/wb/1.patch 
patching file tools/perf/util/maps.c
Hunk #1 succeeded at 815 (offset 18 lines).
Hunk #2 succeeded at 826 (offset 18 lines).
Hunk #3 succeeded at 846 (offset 18 lines).
Hunk #4 succeeded at 893 (offset 18 lines).
Hunk #5 succeeded at 919 (offset 18 lines).
Hunk #6 FAILED at 923.
1 out of 6 hunks FAILED -- saving rejects to file tools/perf/util/maps.c.rej
⬢ [acme@toolbox perf-tools]$

⬢ [acme@toolbox perf-tools]$ git log --oneline -5
4a9f5d76130b707f (HEAD -> perf-tools) wip: acme
d5ba0f5af35937c7 wip: namhyung
42367eca7604e16e (perf-tools/tmp.perf-tools, perf-tools/perf-tools) tools: Remove redundant quiet setup
293f324ce96d7001 tools: Unify top-level quiet infrastructure
9fae5884bb0e3480 (tag: perf-tools-fixes-for-v6.14-2025-01-30) perf cpumap: Fix die and cluster IDs
⬢ [acme@toolbox perf-tools]$
 
> Thanks,
> Namhyung
> 
> 
> ---8<---
> diff --git a/tools/perf/util/maps.c b/tools/perf/util/maps.c
> index 09c9cc326c08d435..3aee0c9e8d421cef 100644
> --- a/tools/perf/util/maps.c
> +++ b/tools/perf/util/maps.c
> @@ -797,7 +797,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
>  {
>  	int err = 0;
>  	FILE *fp = debug_file();
> -	unsigned int i;
> +	unsigned int i, ni;
>  
>  	if (!maps__maps_by_address_sorted(maps))
>  		__maps__sort_by_address(maps);
> @@ -808,6 +808,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
>  	 */
>  	for (i = first_ending_after(maps, new); i < maps__nr_maps(maps); ) {
>  		struct map **maps_by_address = maps__maps_by_address(maps);
> +		struct map **maps_by_name = maps__maps_by_name(maps);
>  		struct map *pos = maps_by_address[i];
>  		struct map *before = NULL, *after = NULL;
>  
> @@ -827,6 +828,9 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
>  			map__fprintf(pos, fp);
>  		}
>  
> +		if (maps_by_name)
> +			ni = maps__by_name_index(maps, pos);
> +
>  		/*
>  		 * Now check if we need to create new maps for areas not
>  		 * overlapped by the new map:
> @@ -871,6 +875,11 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
>  		if (before) {
>  			map__put(maps_by_address[i]);
>  			maps_by_address[i] = before;
> +			if (maps_by_name) {
> +				map__put(maps_by_name[ni]);
> +				maps_by_name[ni] = map__get(before);
> +			}
> +
>  			/* Maps are still ordered, go to next one. */
>  			i++;
>  			if (after) {
> @@ -892,6 +901,10 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
>  			 */
>  			map__put(maps_by_address[i]);
>  			maps_by_address[i] = map__get(new);
> +			if (maps_by_name) {
> +				map__put(maps_by_name[ni]);
> +				maps_by_name[ni] = map__get(new);
> +			}
>  			err = __maps__insert_sorted(maps, i + 1, after, NULL);
>  			map__put(after);
>  			check_invariants(maps);
> @@ -910,6 +923,10 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
>  				 */
>  				map__put(maps_by_address[i]);
>  				maps_by_address[i] = map__get(new);
> +				if (maps_by_name) {
> +					map__put(maps_by_name[ni]);
> +					maps_by_name[ni] = map__get(new);
> +				}
>  				check_invariants(maps);
>  				return err;
>  			}

Re: [PATCH] perf report: Add 'tgid' sort key

[Arnaldo Carvalho de Melo]

On Wed, Feb 26, 2025 at 06:38:00PM -0300, Arnaldo Carvalho de Melo wrote:
> On Wed, Feb 26, 2025 at 11:34:13AM -0800, Namhyung Kim wrote:
> > On Tue, Feb 25, 2025 at 08:48:56PM +0100, Arnaldo Carvalho de Melo wrote:
> > > On Tue, Feb 25, 2025 at 08:25:35PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > On Tue, Feb 25, 2025 at 08:11:17PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > On Tue, Feb 25, 2025 at 08:07:18PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > On Mon, Feb 24, 2025 at 11:51:35PM -0800, Namhyung Kim wrote:
> > > > > > > On Mon, Feb 24, 2025 at 08:40:37PM -0800, Ian Rogers wrote:
> > > > > > > > On Mon, Feb 24, 2025 at 6:51 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > > > > > > > > On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
> > > > > > > [SNIP]
> > > > > > > > > > I thought the real-time processing had to use
> > > > > > > > > > maps__fixup_overlap_and_insert (rather than maps__insert) as mmap
> > > > > > > > > > events only give us VMA data and two mmaps may have been merged.
> > > > > > > > > > Shouldn't doing this change be the simplest fix?
> > > > > 
> > > > > > > > > Make sense.  How about this?
> > > > > 
> > > > > > > > Lgtm, I have no way to test the issue. Why does maps__fixup_end need
> > > > > > > > to get pushed later?
> > > > > 
> > > > > > > I just noticed it would add extra kernel maps after modules.  I think it
> > > > > > > should fixup end address of the kernel maps after adding all maps first.
> > > > > 
> > > > > > > Arnaldo, can you please test this?
> > > > >  
> > > > > > Trying it now.
> > > > > 
> > > > > Now we have something different:
> > > > > 
> > > > > root@number:~# perf record sleep
> > > > > sleep: missing operand
> > > > > Try 'sleep --help' for more information.
> > > > > [ perf record: Woken up 1 times to write data ]
> > > > > perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed.
> > > > > Aborted (core dumped)
> > > > > root@number:~#
> > > > 
> > > > __maps__insert() does:
> > > > 
> > > >         if (dso && dso__kernel(dso)) {
> > > >                 struct kmap *kmap = map__kmap(new);
> > > > 
> > > >                 if (kmap)
> > > >                         kmap->kmaps = maps;
> > > >                 else
> > > >                         pr_err("Internal error: kernel dso with non kernel map\n");
> > > >         }
> > > > 
> > > > while maps__fixup_overlap_and_insert() doesn't.
> > > > 
> > > > It calls __maps__insert_sorted() that probably should do what
> > > > __maps__insert() does?
> > > 
> > > Ok, so I did the following patch but this case fails:
> > > 
> > > @@ -910,6 +928,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
> > >                                  */
> > >                                 map__put(maps_by_address[i]);
> > >                                 maps_by_address[i] = map__get(new);
> > > +                               map__set_kmap(new, maps);
> > >                                 check_invariants(maps);
> > >                                 return err;
> > >                         }
> > > 
> > > With:
> > > 
> > > perf: util/maps.c:110: check_invariants: Assertion `refcount_read(map__refcnt(map)) > 1' failed.
> > > 
> > > As:
> > > 
> > > 106				/*
> > > 107				 * Maps by name maps should be in maps_by_address, so
> > > 108				 * the reference count should be higher.
> > > 109				 */
> > > 110				assert(refcount_read(map__refcnt(map)) > 1);
> > > 
> > > Since it is just replacing the map in the maps_by_address and not
> > > touching on the maps_by_name? Thus the refcount is just 1:
> > 
> > Sounds like it.  Can you add this on top?
> 
> Trying, but somehow its not applying cleanly, checking:
> 
> ⬢ [acme@toolbox perf-tools]$ patch -p1 < ~/wb/1.patch 
> patching file tools/perf/util/maps.c
> Hunk #1 succeeded at 815 (offset 18 lines).
> Hunk #2 succeeded at 826 (offset 18 lines).
> Hunk #3 succeeded at 846 (offset 18 lines).
> Hunk #4 succeeded at 893 (offset 18 lines).
> Hunk #5 succeeded at 919 (offset 18 lines).
> Hunk #6 FAILED at 923.
> 1 out of 6 hunks FAILED -- saving rejects to file tools/perf/util/maps.c.rej
> ⬢ [acme@toolbox perf-tools]$
> 
> ⬢ [acme@toolbox perf-tools]$ git log --oneline -5
> 4a9f5d76130b707f (HEAD -> perf-tools) wip: acme
> d5ba0f5af35937c7 wip: namhyung
> 42367eca7604e16e (perf-tools/tmp.perf-tools, perf-tools/perf-tools) tools: Remove redundant quiet setup
> 293f324ce96d7001 tools: Unify top-level quiet infrastructure
> 9fae5884bb0e3480 (tag: perf-tools-fixes-for-v6.14-2025-01-30) perf cpumap: Fix die and cluster IDs
> ⬢ [acme@toolbox perf-tools]$

⬢ [acme@toolbox perf-tools]$ cat tools/perf/util/maps.c.rej
--- tools/perf/util/maps.c
+++ tools/perf/util/maps.c
@@ -923,6 +936,10 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
 				 */
 				map__put(maps_by_address[i]);
 				maps_by_address[i] = map__get(new);
+				if (maps_by_name) {
+					map__put(maps_by_name[ni]);
+					maps_by_name[ni] = map__get(new);
+				}
 				check_invariants(maps);
 				return err;
 			}
⬢ [acme@toolbox perf-tools]$

Fixing this up by hand

Re: [PATCH] perf report: Add 'tgid' sort key

[Arnaldo Carvalho de Melo]

On Wed, Feb 26, 2025 at 06:38:50PM -0300, Arnaldo Carvalho de Melo wrote:
> On Wed, Feb 26, 2025 at 06:38:00PM -0300, Arnaldo Carvalho de Melo wrote:
> > On Wed, Feb 26, 2025 at 11:34:13AM -0800, Namhyung Kim wrote:
> > > On Tue, Feb 25, 2025 at 08:48:56PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > On Tue, Feb 25, 2025 at 08:25:35PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > On Tue, Feb 25, 2025 at 08:11:17PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > On Tue, Feb 25, 2025 at 08:07:18PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > > On Mon, Feb 24, 2025 at 11:51:35PM -0800, Namhyung Kim wrote:
> > > > > > > > On Mon, Feb 24, 2025 at 08:40:37PM -0800, Ian Rogers wrote:
> > > > > > > > > On Mon, Feb 24, 2025 at 6:51 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > > > > > > > > > On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
> > > > > > > > [SNIP]
> > > > > > > > > > > I thought the real-time processing had to use
> > > > > > > > > > > maps__fixup_overlap_and_insert (rather than maps__insert) as mmap
> > > > > > > > > > > events only give us VMA data and two mmaps may have been merged.
> > > > > > > > > > > Shouldn't doing this change be the simplest fix?
> > > > > > 
> > > > > > > > > > Make sense.  How about this?
> > > > > > 
> > > > > > > > > Lgtm, I have no way to test the issue. Why does maps__fixup_end need
> > > > > > > > > to get pushed later?
> > > > > > 
> > > > > > > > I just noticed it would add extra kernel maps after modules.  I think it
> > > > > > > > should fixup end address of the kernel maps after adding all maps first.
> > > > > > 
> > > > > > > > Arnaldo, can you please test this?
> > > > > >  
> > > > > > > Trying it now.
> > > > > > 
> > > > > > Now we have something different:
> > > > > > 
> > > > > > root@number:~# perf record sleep
> > > > > > sleep: missing operand
> > > > > > Try 'sleep --help' for more information.
> > > > > > [ perf record: Woken up 1 times to write data ]
> > > > > > perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed.
> > > > > > Aborted (core dumped)
> > > > > > root@number:~#
> > > > > 
> > > > > __maps__insert() does:
> > > > > 
> > > > >         if (dso && dso__kernel(dso)) {
> > > > >                 struct kmap *kmap = map__kmap(new);
> > > > > 
> > > > >                 if (kmap)
> > > > >                         kmap->kmaps = maps;
> > > > >                 else
> > > > >                         pr_err("Internal error: kernel dso with non kernel map\n");
> > > > >         }
> > > > > 
> > > > > while maps__fixup_overlap_and_insert() doesn't.
> > > > > 
> > > > > It calls __maps__insert_sorted() that probably should do what
> > > > > __maps__insert() does?
> > > > 
> > > > Ok, so I did the following patch but this case fails:
> > > > 
> > > > @@ -910,6 +928,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
> > > >                                  */
> > > >                                 map__put(maps_by_address[i]);
> > > >                                 maps_by_address[i] = map__get(new);
> > > > +                               map__set_kmap(new, maps);
> > > >                                 check_invariants(maps);
> > > >                                 return err;
> > > >                         }
> > > > 
> > > > With:
> > > > 
> > > > perf: util/maps.c:110: check_invariants: Assertion `refcount_read(map__refcnt(map)) > 1' failed.
> > > > 
> > > > As:
> > > > 
> > > > 106				/*
> > > > 107				 * Maps by name maps should be in maps_by_address, so
> > > > 108				 * the reference count should be higher.
> > > > 109				 */
> > > > 110				assert(refcount_read(map__refcnt(map)) > 1);
> > > > 
> > > > Since it is just replacing the map in the maps_by_address and not
> > > > touching on the maps_by_name? Thus the refcount is just 1:
> > > 
> > > Sounds like it.  Can you add this on top?
> > 
> > Trying, but somehow its not applying cleanly, checking:
> > 
> > ⬢ [acme@toolbox perf-tools]$ patch -p1 < ~/wb/1.patch 
> > patching file tools/perf/util/maps.c
> > Hunk #1 succeeded at 815 (offset 18 lines).
> > Hunk #2 succeeded at 826 (offset 18 lines).
> > Hunk #3 succeeded at 846 (offset 18 lines).
> > Hunk #4 succeeded at 893 (offset 18 lines).
> > Hunk #5 succeeded at 919 (offset 18 lines).
> > Hunk #6 FAILED at 923.
> > 1 out of 6 hunks FAILED -- saving rejects to file tools/perf/util/maps.c.rej
> > ⬢ [acme@toolbox perf-tools]$
> > 
> > ⬢ [acme@toolbox perf-tools]$ git log --oneline -5
> > 4a9f5d76130b707f (HEAD -> perf-tools) wip: acme
> > d5ba0f5af35937c7 wip: namhyung
> > 42367eca7604e16e (perf-tools/tmp.perf-tools, perf-tools/perf-tools) tools: Remove redundant quiet setup
> > 293f324ce96d7001 tools: Unify top-level quiet infrastructure
> > 9fae5884bb0e3480 (tag: perf-tools-fixes-for-v6.14-2025-01-30) perf cpumap: Fix die and cluster IDs
> > ⬢ [acme@toolbox perf-tools]$
> 
> ⬢ [acme@toolbox perf-tools]$ cat tools/perf/util/maps.c.rej
> --- tools/perf/util/maps.c
> +++ tools/perf/util/maps.c
> @@ -923,6 +936,10 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
>  				 */
>  				map__put(maps_by_address[i]);
>  				maps_by_address[i] = map__get(new);
> +				if (maps_by_name) {
> +					map__put(maps_by_name[ni]);
> +					maps_by_name[ni] = map__get(new);
> +				}
>  				check_invariants(maps);
>  				return err;
>  			}
> ⬢ [acme@toolbox perf-tools]$
> 
> Fixing this up by hand

I see, I had tried this after sending that patch:

     map__set_kmap(new, maps);

Before check_invariants(), but that doesn't make sense, I should've
dropped that, doing it now.

- Arnaldo

Re: [PATCH] perf report: Add 'tgid' sort key

[Arnaldo Carvalho de Melo]

On Wed, Feb 26, 2025 at 06:40:29PM -0300, Arnaldo Carvalho de Melo wrote:
> On Wed, Feb 26, 2025 at 06:38:50PM -0300, Arnaldo Carvalho de Melo wrote:
> > On Wed, Feb 26, 2025 at 06:38:00PM -0300, Arnaldo Carvalho de Melo wrote:
> > > On Wed, Feb 26, 2025 at 11:34:13AM -0800, Namhyung Kim wrote:
> > > > On Tue, Feb 25, 2025 at 08:48:56PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > On Tue, Feb 25, 2025 at 08:25:35PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > On Tue, Feb 25, 2025 at 08:11:17PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > > On Tue, Feb 25, 2025 at 08:07:18PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > > > On Mon, Feb 24, 2025 at 11:51:35PM -0800, Namhyung Kim wrote:
> > > > > > > > > On Mon, Feb 24, 2025 at 08:40:37PM -0800, Ian Rogers wrote:
> > > > > > > > > > On Mon, Feb 24, 2025 at 6:51 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > > > > > > > > > > On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
> > > > > > > > > [SNIP]
> > > > > > > > > > > > I thought the real-time processing had to use
> > > > > > > > > > > > maps__fixup_overlap_and_insert (rather than maps__insert) as mmap
> > > > > > > > > > > > events only give us VMA data and two mmaps may have been merged.
> > > > > > > > > > > > Shouldn't doing this change be the simplest fix?
> > > > > > > 
> > > > > > > > > > > Make sense.  How about this?
> > > > > > > 
> > > > > > > > > > Lgtm, I have no way to test the issue. Why does maps__fixup_end need
> > > > > > > > > > to get pushed later?
> > > > > > > 
> > > > > > > > > I just noticed it would add extra kernel maps after modules.  I think it
> > > > > > > > > should fixup end address of the kernel maps after adding all maps first.
> > > > > > > 
> > > > > > > > > Arnaldo, can you please test this?
> > > > > > >  
> > > > > > > > Trying it now.
> > > > > > > 
> > > > > > > Now we have something different:
> > > > > > > 
> > > > > > > root@number:~# perf record sleep
> > > > > > > sleep: missing operand
> > > > > > > Try 'sleep --help' for more information.
> > > > > > > [ perf record: Woken up 1 times to write data ]
> > > > > > > perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed.
> > > > > > > Aborted (core dumped)
> > > > > > > root@number:~#
> > > > > > 
> > > > > > __maps__insert() does:
> > > > > > 
> > > > > >         if (dso && dso__kernel(dso)) {
> > > > > >                 struct kmap *kmap = map__kmap(new);
> > > > > > 
> > > > > >                 if (kmap)
> > > > > >                         kmap->kmaps = maps;
> > > > > >                 else
> > > > > >                         pr_err("Internal error: kernel dso with non kernel map\n");
> > > > > >         }
> > > > > > 
> > > > > > while maps__fixup_overlap_and_insert() doesn't.
> > > > > > 
> > > > > > It calls __maps__insert_sorted() that probably should do what
> > > > > > __maps__insert() does?
> > > > > 
> > > > > Ok, so I did the following patch but this case fails:
> > > > > 
> > > > > @@ -910,6 +928,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
> > > > >                                  */
> > > > >                                 map__put(maps_by_address[i]);
> > > > >                                 maps_by_address[i] = map__get(new);
> > > > > +                               map__set_kmap(new, maps);
> > > > >                                 check_invariants(maps);
> > > > >                                 return err;
> > > > >                         }
> > > > > 
> > > > > With:
> > > > > 
> > > > > perf: util/maps.c:110: check_invariants: Assertion `refcount_read(map__refcnt(map)) > 1' failed.
> > > > > 
> > > > > As:
> > > > > 
> > > > > 106				/*
> > > > > 107				 * Maps by name maps should be in maps_by_address, so
> > > > > 108				 * the reference count should be higher.
> > > > > 109				 */
> > > > > 110				assert(refcount_read(map__refcnt(map)) > 1);
> > > > > 
> > > > > Since it is just replacing the map in the maps_by_address and not
> > > > > touching on the maps_by_name? Thus the refcount is just 1:

> > > > Sounds like it.  Can you add this on top?

> > > Trying, but somehow its not applying cleanly, checking:

> > > ⬢ [acme@toolbox perf-tools]$ patch -p1 < ~/wb/1.patch 
> > > patching file tools/perf/util/maps.c
> > > Hunk #1 succeeded at 815 (offset 18 lines).
> > > Hunk #2 succeeded at 826 (offset 18 lines).
> > > Hunk #3 succeeded at 846 (offset 18 lines).
> > > Hunk #4 succeeded at 893 (offset 18 lines).
> > > Hunk #5 succeeded at 919 (offset 18 lines).
> > > Hunk #6 FAILED at 923.
> > > 1 out of 6 hunks FAILED -- saving rejects to file tools/perf/util/maps.c.rej
> > > ⬢ [acme@toolbox perf-tools]$

> > > ⬢ [acme@toolbox perf-tools]$ git log --oneline -5
> > > 4a9f5d76130b707f (HEAD -> perf-tools) wip: acme
> > > d5ba0f5af35937c7 wip: namhyung
> > > 42367eca7604e16e (perf-tools/tmp.perf-tools, perf-tools/perf-tools) tools: Remove redundant quiet setup
> > > 293f324ce96d7001 tools: Unify top-level quiet infrastructure
> > > 9fae5884bb0e3480 (tag: perf-tools-fixes-for-v6.14-2025-01-30) perf cpumap: Fix die and cluster IDs
> > > ⬢ [acme@toolbox perf-tools]$

> > ⬢ [acme@toolbox perf-tools]$ cat tools/perf/util/maps.c.rej
> > --- tools/perf/util/maps.c
> > +++ tools/perf/util/maps.c
> > @@ -923,6 +936,10 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
> >  				 */
> >  				map__put(maps_by_address[i]);
> >  				maps_by_address[i] = map__get(new);
> > +				if (maps_by_name) {
> > +					map__put(maps_by_name[ni]);
> > +					maps_by_name[ni] = map__get(new);
> > +				}
> >  				check_invariants(maps);
> >  				return err;
> >  			}
> > ⬢ [acme@toolbox perf-tools]$

> > Fixing this up by hand
 
> I see, I had tried this after sending that patch:
 
>      map__set_kmap(new, maps);
 
> Before check_invariants(), but that doesn't make sense, I should've
> dropped that, doing it now.

Nope, it still triggers:

 930                 } else {
 931                         struct map *next = NULL;
 932 
 933                         if (i + 1 < maps__nr_maps(maps))
 934                                 next = maps_by_address[i + 1];
 935 
 936                         if (!next  || map__start(next) >= map__end(new)) {
 937                                 /*
 938                                  * Replace existing mapping and end knowing
 939                                  * there aren't later overlapping or any
 940                                  * mappings.
 941                                  */
 942                                 map__put(maps_by_address[i]);
 943                                 maps_by_address[i] = map__get(new);
 944                                 if (maps_by_name) {
 945                                         map__put(maps_by_name[ni]);
 946                                         maps_by_name[ni] = map__get(new);
 947                                 }
 948                                 check_invariants(maps);
 949                                 return err;
 950                         }
 951                         __maps__remove(maps, pos);
 952                         check_invariants(maps);
 953                         /*
 954                          * Maps are ordered but no need to increase `i` as the
 955                          * later maps were moved down.
 956                          */
 957                 }


perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed.

Thread 1 "perf" received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44	      return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78
#2  0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007ffff6e36902 in __GI_abort () at abort.c:79
#4  0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bfd08 "RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)", 
    file=file@entry=0x7bfc53 "util/maps.c", line=line@entry=80, function=function@entry=0x7c0010 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96
#5  0x00007ffff6e47047 in __assert_fail (assertion=0x7bfd08 "RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)", file=0x7bfc53 "util/maps.c", line=80, 
    function=0x7c0010 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105
#6  0x0000000000633c74 in check_invariants (maps=0xf977c0) at util/maps.c:80
#7  0x00000000006363a6 in __maps__fixup_overlap_and_insert (maps=0xf977c0, new=0xfc57e0) at util/maps.c:948
#8  0x0000000000636460 in maps__fixup_overlap_and_insert (maps=0xf977c0, new=0xfc57e0) at util/maps.c:970
#9  0x000000000062920a in machine__process_ksymbol_register (machine=0xf97158, event=0x7ffff7fbaba8, sample=0x7fffffffa860) at util/machine.c:715
#10 0x00000000006294ca in machine__process_ksymbol (machine=0xf97158, event=0x7ffff7fbaba8, sample=0x7fffffffa860) at util/machine.c:779
#11 0x00000000005ce2fd in perf_event__process_ksymbol (tool=0xec9ce0 <record>, event=0x7ffff7fbaba8, sample=0x7fffffffa860, machine=0xf97158) at util/event.c:296
#12 0x000000000063b860 in machines__deliver_event (machines=0xf97158, evlist=0xf531f0, event=0x7ffff7fbaba8, sample=0x7fffffffa860, tool=0xec9ce0 <record>, file_offset=35752, 
    file_path=0xf97850 "perf.data") at util/session.c:1334
#13 0x000000000063ba45 in perf_session__deliver_event (session=0xf96f40, event=0x7ffff7fbaba8, tool=0xec9ce0 <record>, file_offset=35752, file_path=0xf97850 "perf.data")
    at util/session.c:1367
#14 0x000000000063c839 in perf_session__process_event (session=0xf96f40, event=0x7ffff7fbaba8, file_offset=35752, file_path=0xf97850 "perf.data") at util/session.c:1626
#15 0x000000000063dfb9 in process_simple (session=0xf96f40, event=0x7ffff7fbaba8, file_offset=35752, file_path=0xf97850 "perf.data") at util/session.c:2203
#16 0x000000000063dc70 in reader__read_event (rd=0x7fffffffafa0, session=0xf96f40, prog=0x7fffffffaf70) at util/session.c:2132
#17 0x000000000063de6a in reader__process_events (rd=0x7fffffffafa0, session=0xf96f40, prog=0x7fffffffaf70) at util/session.c:2181
#18 0x000000000063e107 in __perf_session__process_events (session=0xf96f40) at util/session.c:2226
#19 0x000000000063eb04 in perf_session__process_events (session=0xf96f40) at util/session.c:2390
#20 0x000000000042d98b in process_buildids (rec=0xec9ce0 <record>) at builtin-record.c:1475
#21 0x000000000042e963 in record__finish_output (rec=0xec9ce0 <record>) at builtin-record.c:1798
#22 0x0000000000431c46 in __cmd_record (rec=0xec9ce0 <record>, argc=2, argv=0x7fffffffde80) at builtin-record.c:2841
#23 0x000000000043513f in cmd_record (argc=2, argv=0x7fffffffde80) at builtin-record.c:4260
#24 0x00000000004bcf65 in run_builtin (p=0xeccd60 <commands+288>, argc=3, argv=0x7fffffffde80) at perf.c:351
#25 0x00000000004bd20c in handle_internal_command (argc=3, argv=0x7fffffffde80) at perf.c:404
#26 0x00000000004bd365 in run_argv (argcp=0x7fffffffdc6c, argv=0x7fffffffdc60) at perf.c:448
#27 0x00000000004bd6ae in main (argc=3, argv=0x7fffffffde80) at perf.c:556
(gdb)

humm, it seems that thing I removed may be needed after all...

Yeah, please check if adding this, on top of your latest (second) patch,
makese sense:

diff --git a/tools/perf/util/maps.c b/tools/perf/util/maps.c
index a710720e8bcfd305..776bbdaafdb32269 100644
--- a/tools/perf/util/maps.c
+++ b/tools/perf/util/maps.c
@@ -945,6 +945,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
                                        map__put(maps_by_name[ni]);
                                        maps_by_name[ni] = map__get(new);
                                }
+                               map__set_kmap(new, maps);
                                check_invariants(maps);
                                return err;
                        }
⬢ [acme@toolbox perf-tools]$


With your two patches and my two patches finally I get:

root@number:~# perf record sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.035 MB perf.data (14 samples) ]
root@number:~#

on a perf-tools/perf-tools build with DEBUG=1

- Arnaldo

Re: [PATCH] perf report: Add 'tgid' sort key

[Namhyung Kim]

On Wed, Feb 26, 2025 at 06:49:31PM -0300, Arnaldo Carvalho de Melo wrote:
> On Wed, Feb 26, 2025 at 06:40:29PM -0300, Arnaldo Carvalho de Melo wrote:
> > On Wed, Feb 26, 2025 at 06:38:50PM -0300, Arnaldo Carvalho de Melo wrote:
> > > On Wed, Feb 26, 2025 at 06:38:00PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > On Wed, Feb 26, 2025 at 11:34:13AM -0800, Namhyung Kim wrote:
> > > > > On Tue, Feb 25, 2025 at 08:48:56PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > On Tue, Feb 25, 2025 at 08:25:35PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > > On Tue, Feb 25, 2025 at 08:11:17PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > > > On Tue, Feb 25, 2025 at 08:07:18PM +0100, Arnaldo Carvalho de Melo wrote:
> > > > > > > > > On Mon, Feb 24, 2025 at 11:51:35PM -0800, Namhyung Kim wrote:
> > > > > > > > > > On Mon, Feb 24, 2025 at 08:40:37PM -0800, Ian Rogers wrote:
> > > > > > > > > > > On Mon, Feb 24, 2025 at 6:51 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > > > > > > > > > > > On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
> > > > > > > > > > [SNIP]
> > > > > > > > > > > > > I thought the real-time processing had to use
> > > > > > > > > > > > > maps__fixup_overlap_and_insert (rather than maps__insert) as mmap
> > > > > > > > > > > > > events only give us VMA data and two mmaps may have been merged.
> > > > > > > > > > > > > Shouldn't doing this change be the simplest fix?
> > > > > > > > 
> > > > > > > > > > > > Make sense.  How about this?
> > > > > > > > 
> > > > > > > > > > > Lgtm, I have no way to test the issue. Why does maps__fixup_end need
> > > > > > > > > > > to get pushed later?
> > > > > > > > 
> > > > > > > > > > I just noticed it would add extra kernel maps after modules.  I think it
> > > > > > > > > > should fixup end address of the kernel maps after adding all maps first.
> > > > > > > > 
> > > > > > > > > > Arnaldo, can you please test this?
> > > > > > > >  
> > > > > > > > > Trying it now.
> > > > > > > > 
> > > > > > > > Now we have something different:
> > > > > > > > 
> > > > > > > > root@number:~# perf record sleep
> > > > > > > > sleep: missing operand
> > > > > > > > Try 'sleep --help' for more information.
> > > > > > > > [ perf record: Woken up 1 times to write data ]
> > > > > > > > perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed.
> > > > > > > > Aborted (core dumped)
> > > > > > > > root@number:~#
> > > > > > > 
> > > > > > > __maps__insert() does:
> > > > > > > 
> > > > > > >         if (dso && dso__kernel(dso)) {
> > > > > > >                 struct kmap *kmap = map__kmap(new);
> > > > > > > 
> > > > > > >                 if (kmap)
> > > > > > >                         kmap->kmaps = maps;
> > > > > > >                 else
> > > > > > >                         pr_err("Internal error: kernel dso with non kernel map\n");
> > > > > > >         }
> > > > > > > 
> > > > > > > while maps__fixup_overlap_and_insert() doesn't.
> > > > > > > 
> > > > > > > It calls __maps__insert_sorted() that probably should do what
> > > > > > > __maps__insert() does?
> > > > > > 
> > > > > > Ok, so I did the following patch but this case fails:
> > > > > > 
> > > > > > @@ -910,6 +928,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
> > > > > >                                  */
> > > > > >                                 map__put(maps_by_address[i]);
> > > > > >                                 maps_by_address[i] = map__get(new);
> > > > > > +                               map__set_kmap(new, maps);
> > > > > >                                 check_invariants(maps);
> > > > > >                                 return err;
> > > > > >                         }
> > > > > > 
> > > > > > With:
> > > > > > 
> > > > > > perf: util/maps.c:110: check_invariants: Assertion `refcount_read(map__refcnt(map)) > 1' failed.
> > > > > > 
> > > > > > As:
> > > > > > 
> > > > > > 106				/*
> > > > > > 107				 * Maps by name maps should be in maps_by_address, so
> > > > > > 108				 * the reference count should be higher.
> > > > > > 109				 */
> > > > > > 110				assert(refcount_read(map__refcnt(map)) > 1);
> > > > > > 
> > > > > > Since it is just replacing the map in the maps_by_address and not
> > > > > > touching on the maps_by_name? Thus the refcount is just 1:
> 
> > > > > Sounds like it.  Can you add this on top?
> 
> > > > Trying, but somehow its not applying cleanly, checking:
> 
> > > > ⬢ [acme@toolbox perf-tools]$ patch -p1 < ~/wb/1.patch 
> > > > patching file tools/perf/util/maps.c
> > > > Hunk #1 succeeded at 815 (offset 18 lines).
> > > > Hunk #2 succeeded at 826 (offset 18 lines).
> > > > Hunk #3 succeeded at 846 (offset 18 lines).
> > > > Hunk #4 succeeded at 893 (offset 18 lines).
> > > > Hunk #5 succeeded at 919 (offset 18 lines).
> > > > Hunk #6 FAILED at 923.
> > > > 1 out of 6 hunks FAILED -- saving rejects to file tools/perf/util/maps.c.rej
> > > > ⬢ [acme@toolbox perf-tools]$
> 
> > > > ⬢ [acme@toolbox perf-tools]$ git log --oneline -5
> > > > 4a9f5d76130b707f (HEAD -> perf-tools) wip: acme
> > > > d5ba0f5af35937c7 wip: namhyung
> > > > 42367eca7604e16e (perf-tools/tmp.perf-tools, perf-tools/perf-tools) tools: Remove redundant quiet setup
> > > > 293f324ce96d7001 tools: Unify top-level quiet infrastructure
> > > > 9fae5884bb0e3480 (tag: perf-tools-fixes-for-v6.14-2025-01-30) perf cpumap: Fix die and cluster IDs
> > > > ⬢ [acme@toolbox perf-tools]$
> 
> > > ⬢ [acme@toolbox perf-tools]$ cat tools/perf/util/maps.c.rej
> > > --- tools/perf/util/maps.c
> > > +++ tools/perf/util/maps.c
> > > @@ -923,6 +936,10 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
> > >  				 */
> > >  				map__put(maps_by_address[i]);
> > >  				maps_by_address[i] = map__get(new);
> > > +				if (maps_by_name) {
> > > +					map__put(maps_by_name[ni]);
> > > +					maps_by_name[ni] = map__get(new);
> > > +				}
> > >  				check_invariants(maps);
> > >  				return err;
> > >  			}
> > > ⬢ [acme@toolbox perf-tools]$
> 
> > > Fixing this up by hand
>  
> > I see, I had tried this after sending that patch:
>  
> >      map__set_kmap(new, maps);
>  
> > Before check_invariants(), but that doesn't make sense, I should've
> > dropped that, doing it now.
> 
> Nope, it still triggers:
> 
>  930                 } else {
>  931                         struct map *next = NULL;
>  932 
>  933                         if (i + 1 < maps__nr_maps(maps))
>  934                                 next = maps_by_address[i + 1];
>  935 
>  936                         if (!next  || map__start(next) >= map__end(new)) {
>  937                                 /*
>  938                                  * Replace existing mapping and end knowing
>  939                                  * there aren't later overlapping or any
>  940                                  * mappings.
>  941                                  */
>  942                                 map__put(maps_by_address[i]);
>  943                                 maps_by_address[i] = map__get(new);
>  944                                 if (maps_by_name) {
>  945                                         map__put(maps_by_name[ni]);
>  946                                         maps_by_name[ni] = map__get(new);
>  947                                 }
>  948                                 check_invariants(maps);
>  949                                 return err;
>  950                         }
>  951                         __maps__remove(maps, pos);
>  952                         check_invariants(maps);
>  953                         /*
>  954                          * Maps are ordered but no need to increase `i` as the
>  955                          * later maps were moved down.
>  956                          */
>  957                 }
> 
> 
> perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed.
> 
> Thread 1 "perf" received signal SIGABRT, Aborted.
> __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> 44	      return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
> (gdb) bt
> #0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> #1  0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78
> #2  0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> #3  0x00007ffff6e36902 in __GI_abort () at abort.c:79
> #4  0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bfd08 "RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)", 
>     file=file@entry=0x7bfc53 "util/maps.c", line=line@entry=80, function=function@entry=0x7c0010 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96
> #5  0x00007ffff6e47047 in __assert_fail (assertion=0x7bfd08 "RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)", file=0x7bfc53 "util/maps.c", line=80, 
>     function=0x7c0010 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105
> #6  0x0000000000633c74 in check_invariants (maps=0xf977c0) at util/maps.c:80
> #7  0x00000000006363a6 in __maps__fixup_overlap_and_insert (maps=0xf977c0, new=0xfc57e0) at util/maps.c:948
> #8  0x0000000000636460 in maps__fixup_overlap_and_insert (maps=0xf977c0, new=0xfc57e0) at util/maps.c:970
> #9  0x000000000062920a in machine__process_ksymbol_register (machine=0xf97158, event=0x7ffff7fbaba8, sample=0x7fffffffa860) at util/machine.c:715
> #10 0x00000000006294ca in machine__process_ksymbol (machine=0xf97158, event=0x7ffff7fbaba8, sample=0x7fffffffa860) at util/machine.c:779
> #11 0x00000000005ce2fd in perf_event__process_ksymbol (tool=0xec9ce0 <record>, event=0x7ffff7fbaba8, sample=0x7fffffffa860, machine=0xf97158) at util/event.c:296
> #12 0x000000000063b860 in machines__deliver_event (machines=0xf97158, evlist=0xf531f0, event=0x7ffff7fbaba8, sample=0x7fffffffa860, tool=0xec9ce0 <record>, file_offset=35752, 
>     file_path=0xf97850 "perf.data") at util/session.c:1334
> #13 0x000000000063ba45 in perf_session__deliver_event (session=0xf96f40, event=0x7ffff7fbaba8, tool=0xec9ce0 <record>, file_offset=35752, file_path=0xf97850 "perf.data")
>     at util/session.c:1367
> #14 0x000000000063c839 in perf_session__process_event (session=0xf96f40, event=0x7ffff7fbaba8, file_offset=35752, file_path=0xf97850 "perf.data") at util/session.c:1626
> #15 0x000000000063dfb9 in process_simple (session=0xf96f40, event=0x7ffff7fbaba8, file_offset=35752, file_path=0xf97850 "perf.data") at util/session.c:2203
> #16 0x000000000063dc70 in reader__read_event (rd=0x7fffffffafa0, session=0xf96f40, prog=0x7fffffffaf70) at util/session.c:2132
> #17 0x000000000063de6a in reader__process_events (rd=0x7fffffffafa0, session=0xf96f40, prog=0x7fffffffaf70) at util/session.c:2181
> #18 0x000000000063e107 in __perf_session__process_events (session=0xf96f40) at util/session.c:2226
> #19 0x000000000063eb04 in perf_session__process_events (session=0xf96f40) at util/session.c:2390
> #20 0x000000000042d98b in process_buildids (rec=0xec9ce0 <record>) at builtin-record.c:1475
> #21 0x000000000042e963 in record__finish_output (rec=0xec9ce0 <record>) at builtin-record.c:1798
> #22 0x0000000000431c46 in __cmd_record (rec=0xec9ce0 <record>, argc=2, argv=0x7fffffffde80) at builtin-record.c:2841
> #23 0x000000000043513f in cmd_record (argc=2, argv=0x7fffffffde80) at builtin-record.c:4260
> #24 0x00000000004bcf65 in run_builtin (p=0xeccd60 <commands+288>, argc=3, argv=0x7fffffffde80) at perf.c:351
> #25 0x00000000004bd20c in handle_internal_command (argc=3, argv=0x7fffffffde80) at perf.c:404
> #26 0x00000000004bd365 in run_argv (argcp=0x7fffffffdc6c, argv=0x7fffffffdc60) at perf.c:448
> #27 0x00000000004bd6ae in main (argc=3, argv=0x7fffffffde80) at perf.c:556
> (gdb)
> 
> humm, it seems that thing I removed may be needed after all...
> 
> Yeah, please check if adding this, on top of your latest (second) patch,
> makese sense:
> 
> diff --git a/tools/perf/util/maps.c b/tools/perf/util/maps.c
> index a710720e8bcfd305..776bbdaafdb32269 100644
> --- a/tools/perf/util/maps.c
> +++ b/tools/perf/util/maps.c
> @@ -945,6 +945,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
>                                         map__put(maps_by_name[ni]);
>                                         maps_by_name[ni] = map__get(new);
>                                 }
> +                               map__set_kmap(new, maps);
>                                 check_invariants(maps);
>                                 return err;
>                         }
> ⬢ [acme@toolbox perf-tools]$
> 
> 
> With your two patches and my two patches finally I get:
> 
> root@number:~# perf record sleep 1
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.035 MB perf.data (14 samples) ]
> root@number:~#
> 
> on a perf-tools/perf-tools build with DEBUG=1

Great!  Can you please send the final version as a formal patch?

Thanks,
Namhyung