[PATCH 1/3] perf sort: Keep output fields in the same level

[PATCH 1/3] perf sort: Keep output fields in the same level

[Namhyung Kim]

This is useful for hierarchy output mode where the first level is
considered as output fields.  We want them in the same level so that it
can show only the remaining groups in the hierarchy.

Before:
  $ perf report -s overhead,sample,period,comm,dso -H --stdio
  ...
  #          Overhead  Samples / Period / Command / Shared Object
  # .................  ..........................................
  #
     100.00%           4035
        100.00%           3835883066
           100.00%           perf
               99.37%           perf
                0.50%           ld-linux-x86-64.so.2
                0.06%           [unknown]
                0.04%           libc.so.6
                0.02%           libLLVM-16.so.1

After:
  $ perf report -s overhead,sample,period,comm,dso -H --stdio
  ...
  #    Overhead       Samples        Period  Command / Shared Object
  # .......................................  .......................
  #
     100.00%          4035    3835883066     perf
         99.37%          4005    3811826223     perf
          0.50%            19      19210014     ld-linux-x86-64.so.2
          0.06%             8       2367089     [unknown]
          0.04%             2       1720336     libc.so.6
          0.02%             1        759404     libLLVM-16.so.1

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/sort.c | 44 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/tools/perf/util/sort.c b/tools/perf/util/sort.c
index f08fbc4bf0a2ce29..6b49d64854f5f986 100644
--- a/tools/perf/util/sort.c
+++ b/tools/perf/util/sort.c
@@ -3720,6 +3720,34 @@ int sort_dimension__add(struct perf_hpp_list *list, const char *tok,
 	return -ESRCH;
 }
 
+/* This should match with sort_dimension__add() above */
+static bool is_hpp_sort_key(const char *key)
+{
+	unsigned i;
+
+	for (i = 0; i < ARRAY_SIZE(arch_specific_sort_keys); i++) {
+		if (!strcmp(arch_specific_sort_keys[i], key) &&
+		    !arch_support_sort_key(key)) {
+			return false;
+		}
+	}
+
+	for (i = 0; i < ARRAY_SIZE(common_sort_dimensions); i++) {
+		struct sort_dimension *sd = &common_sort_dimensions[i];
+
+		if (sd->name && !strncasecmp(key, sd->name, strlen(key)))
+			return false;
+	}
+
+	for (i = 0; i < ARRAY_SIZE(hpp_sort_dimensions); i++) {
+		struct hpp_dimension *hd = &hpp_sort_dimensions[i];
+
+		if (!strncasecmp(key, hd->name, strlen(key)))
+			return true;
+	}
+	return false;
+}
+
 static int setup_sort_list(struct perf_hpp_list *list, char *str,
 			   struct evlist *evlist)
 {
@@ -3727,7 +3755,9 @@ static int setup_sort_list(struct perf_hpp_list *list, char *str,
 	int ret = 0;
 	int level = 0;
 	int next_level = 1;
+	int prev_level = 0;
 	bool in_group = false;
+	bool prev_was_hpp = false;
 
 	do {
 		tok = str;
@@ -3748,6 +3778,19 @@ static int setup_sort_list(struct perf_hpp_list *list, char *str,
 		}
 
 		if (*tok) {
+			if (is_hpp_sort_key(tok)) {
+				/* keep output (hpp) sort keys in the same level */
+				if (prev_was_hpp) {
+					bool next_same = (level == next_level);
+
+					level = prev_level;
+					next_level = next_same ? level : level+1;
+				}
+				prev_was_hpp = true;
+			} else {
+				prev_was_hpp = false;
+			}
+
 			ret = sort_dimension__add(list, tok, evlist, level);
 			if (ret == -EINVAL) {
 				if (!cacheline_size() && !strncasecmp(tok, "dcacheline", strlen(tok)))
@@ -3759,6 +3802,7 @@ static int setup_sort_list(struct perf_hpp_list *list, char *str,
 				ui__error("Unknown --sort key: `%s'", tok);
 				break;
 			}
+			prev_level = level;
 		}
 
 		level = next_level;
-- 
2.49.0.rc0.332.g42c0ae87b1-goog

[PATCH 2/3] perf report: Allow hierarchy mode for --children

[Namhyung Kim]

It was prohibited because the output fields in the children mode were
not handled properly with hierarchy.  But we can have the output fields
in the same level, it can allow them together.

For example, latency mode adds more output fields by default and now
they are displayed properly.

  $ perf record --latency -g -- perf test -w thloop

  $ perf report -H --stdio
  # To display the perf.data header info, please use --header/--header-only options.
  #
  #
  # Total Lost Samples: 0
  #
  # Samples: 2K of event 'cycles:Pu'
  # Event count (approx.): 8266456478
  #
  #       Children  Latency  Overhead   Latency  Command / Shared Object / Symbol
  # ...........................................  ........................................................
  #
       0.08%    0.16%   100.00%   100.00%        perf
          0.08%    0.16%     0.24%     0.47%        ld-linux-x86-64.so.2
             0.12%    0.24%     0.12%     0.24%        [.] _dl_relocate_object
             0.08%    0.16%     0.08%     0.16%        [.] _dl_lookup_symbol_x
             0.03%    0.06%     0.03%     0.06%        [.] strcmp
             0.00%    0.01%     0.00%     0.01%        [.] _dl_start
             0.00%    0.00%     0.00%     0.00%        [.] _dl_start_user
             0.00%    0.00%     0.00%     0.00%        [.] _dl_sysdep_start
             0.00%    0.00%     0.00%     0.00%        [.] _start
             0.00%    0.00%     0.00%     0.00%        [.] dl_main
          0.03%    0.06%     0.03%     0.06%        libLLVM-16.so.1
             0.03%    0.06%     0.03%     0.06%        [.] llvm::StringMapImpl::RehashTable(unsigned int)
             0.00%    0.00%     0.00%     0.00%        [.] 0x00007f137ccd18e8
          0.00%    0.00%    99.66%    99.31%        perf
            99.66%   99.31%    99.66%    99.31%        [.] test_loop
              |
              |--49.86%--0x7f137b633d68
              |          0x55dbdbbb7d2c
              ...

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-report.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index c1226da6c610258f..fc776e9d7fdfa273 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -1673,8 +1673,6 @@ int cmd_report(int argc, const char **argv)
 
 	if (symbol_conf.report_hierarchy) {
 		/* disable incompatible options */
-		symbol_conf.cumulate_callchain = false;
-
 		if (field_order) {
 			pr_err("Error: --hierarchy and --fields options cannot be used together\n");
 			parse_options_usage(report_usage, options, "F", 1);
-- 
2.49.0.rc0.332.g42c0ae87b1-goog

[PATCH 3/3] perf report: Disable children column for data type profiling

[Namhyung Kim]

I've realized that it doesn't make sense to accumulate the samples to
parent in the callchain when data type profiling is enabled.  Because it
won't have the same data type access in the parent.  Otherwise it'd see
something like this:

  $ perf report -s type --stdio -g none
  # To display the perf.data header info, please use --header/--header-only options.
  #
  #
  # Total Lost Samples: 0
  #
  # Samples: 2K of event 'cycles:Pu'
  # Event count (approx.): 8266456478
  #
  # Children  Latency      Self   Latency  Data Type
  # ........  .......  ........  ........  .........
  #
     698.97%   697.72%    99.80%    99.61%  (unknown)
       0.09%    0.18%     0.09%     0.18%  Elf64_Rela
       0.05%    0.10%     0.05%     0.10%  unsigned char
       0.05%    0.10%     0.05%     0.10%  struct exit_function_list
       0.00%    0.01%     0.00%     0.01%  struct rtld_global

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-report.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index fc776e9d7fdfa273..b030ce72e13ea8d1 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -1719,6 +1719,9 @@ int cmd_report(int argc, const char **argv)
 		report.data_type = true;
 		annotate_opts.annotate_src = false;
 
+		/* disable incompatible options */
+		symbol_conf.cumulate_callchain = false;
+
 #ifndef HAVE_LIBDW_SUPPORT
 		pr_err("Error: Data type profiling is disabled due to missing DWARF support\n");
 		goto error;
-- 
2.49.0.rc0.332.g42c0ae87b1-goog

Re: [PATCH 1/3] perf sort: Keep output fields in the same level

[Namhyung Kim]

On Fri, Mar 07, 2025 at 12:08:27AM -0800, Namhyung Kim wrote:
> This is useful for hierarchy output mode where the first level is
> considered as output fields.  We want them in the same level so that it
> can show only the remaining groups in the hierarchy.
> 
> Before:
>   $ perf report -s overhead,sample,period,comm,dso -H --stdio
>   ...
>   #          Overhead  Samples / Period / Command / Shared Object
>   # .................  ..........................................
>   #
>      100.00%           4035
>         100.00%           3835883066
>            100.00%           perf
>                99.37%           perf
>                 0.50%           ld-linux-x86-64.so.2
>                 0.06%           [unknown]
>                 0.04%           libc.so.6
>                 0.02%           libLLVM-16.so.1
> 
> After:
>   $ perf report -s overhead,sample,period,comm,dso -H --stdio
>   ...
>   #    Overhead       Samples        Period  Command / Shared Object
>   # .......................................  .......................
>   #
>      100.00%          4035    3835883066     perf
>          99.37%          4005    3811826223     perf
>           0.50%            19      19210014     ld-linux-x86-64.so.2
>           0.06%             8       2367089     [unknown]
>           0.04%             2       1720336     libc.so.6
>           0.02%             1        759404     libLLVM-16.so.1
> 
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>

Ping!  Anybody interested in this change? :)

Thanks,
Namhyung

> ---
>  tools/perf/util/sort.c | 44 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 44 insertions(+)
> 
> diff --git a/tools/perf/util/sort.c b/tools/perf/util/sort.c
> index f08fbc4bf0a2ce29..6b49d64854f5f986 100644
> --- a/tools/perf/util/sort.c
> +++ b/tools/perf/util/sort.c
> @@ -3720,6 +3720,34 @@ int sort_dimension__add(struct perf_hpp_list *list, const char *tok,
>  	return -ESRCH;
>  }
>  
> +/* This should match with sort_dimension__add() above */
> +static bool is_hpp_sort_key(const char *key)
> +{
> +	unsigned i;
> +
> +	for (i = 0; i < ARRAY_SIZE(arch_specific_sort_keys); i++) {
> +		if (!strcmp(arch_specific_sort_keys[i], key) &&
> +		    !arch_support_sort_key(key)) {
> +			return false;
> +		}
> +	}
> +
> +	for (i = 0; i < ARRAY_SIZE(common_sort_dimensions); i++) {
> +		struct sort_dimension *sd = &common_sort_dimensions[i];
> +
> +		if (sd->name && !strncasecmp(key, sd->name, strlen(key)))
> +			return false;
> +	}
> +
> +	for (i = 0; i < ARRAY_SIZE(hpp_sort_dimensions); i++) {
> +		struct hpp_dimension *hd = &hpp_sort_dimensions[i];
> +
> +		if (!strncasecmp(key, hd->name, strlen(key)))
> +			return true;
> +	}
> +	return false;
> +}
> +
>  static int setup_sort_list(struct perf_hpp_list *list, char *str,
>  			   struct evlist *evlist)
>  {
> @@ -3727,7 +3755,9 @@ static int setup_sort_list(struct perf_hpp_list *list, char *str,
>  	int ret = 0;
>  	int level = 0;
>  	int next_level = 1;
> +	int prev_level = 0;
>  	bool in_group = false;
> +	bool prev_was_hpp = false;
>  
>  	do {
>  		tok = str;
> @@ -3748,6 +3778,19 @@ static int setup_sort_list(struct perf_hpp_list *list, char *str,
>  		}
>  
>  		if (*tok) {
> +			if (is_hpp_sort_key(tok)) {
> +				/* keep output (hpp) sort keys in the same level */
> +				if (prev_was_hpp) {
> +					bool next_same = (level == next_level);
> +
> +					level = prev_level;
> +					next_level = next_same ? level : level+1;
> +				}
> +				prev_was_hpp = true;
> +			} else {
> +				prev_was_hpp = false;
> +			}
> +
>  			ret = sort_dimension__add(list, tok, evlist, level);
>  			if (ret == -EINVAL) {
>  				if (!cacheline_size() && !strncasecmp(tok, "dcacheline", strlen(tok)))
> @@ -3759,6 +3802,7 @@ static int setup_sort_list(struct perf_hpp_list *list, char *str,
>  				ui__error("Unknown --sort key: `%s'", tok);
>  				break;
>  			}
> +			prev_level = level;
>  		}
>  
>  		level = next_level;
> -- 
> 2.49.0.rc0.332.g42c0ae87b1-goog
> 

Re: [PATCH 1/3] perf sort: Keep output fields in the same level

[Ingo Molnar]

* Namhyung Kim <namhyung@kernel.org> wrote:

> On Fri, Mar 07, 2025 at 12:08:27AM -0800, Namhyung Kim wrote:
> > This is useful for hierarchy output mode where the first level is
> > considered as output fields.  We want them in the same level so that it
> > can show only the remaining groups in the hierarchy.
> > 
> > Before:
> >   $ perf report -s overhead,sample,period,comm,dso -H --stdio
> >   ...
> >   #          Overhead  Samples / Period / Command / Shared Object
> >   # .................  ..........................................
> >   #
> >      100.00%           4035
> >         100.00%           3835883066
> >            100.00%           perf
> >                99.37%           perf
> >                 0.50%           ld-linux-x86-64.so.2
> >                 0.06%           [unknown]
> >                 0.04%           libc.so.6
> >                 0.02%           libLLVM-16.so.1
> > 
> > After:
> >   $ perf report -s overhead,sample,period,comm,dso -H --stdio
> >   ...
> >   #    Overhead       Samples        Period  Command / Shared Object
> >   # .......................................  .......................
> >   #
> >      100.00%          4035    3835883066     perf
> >          99.37%          4005    3811826223     perf
> >           0.50%            19      19210014     ld-linux-x86-64.so.2
> >           0.06%             8       2367089     [unknown]
> >           0.04%             2       1720336     libc.so.6
> >           0.02%             1        759404     libLLVM-16.so.1
> > 
> > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> 
> Ping!  Anybody interested in this change? :)

Oh yes, all such pieces of intelligent organization of textual output 
of profiling data are worth their weight in gold in my book. :-)

  Acked-by: Ingo Molnar <mingo@kernel.org>

1)

On a related note, does anyone know why perf stat output alignment 
sucks so much these days:

  starship:~/tip> perf stat --null --repeat 20 perf stat --null --repeat 3 perf bench sched messaging 2>&1 | grep elapsed
           0.11620 +- 0.00327 seconds time elapsed  ( +-  2.81% )
          0.120813 +- 0.000570 seconds time elapsed  ( +-  0.47% )
          0.122280 +- 0.000443 seconds time elapsed  ( +-  0.36% )
          0.119813 +- 0.000752 seconds time elapsed  ( +-  0.63% )
           0.12190 +- 0.00134 seconds time elapsed  ( +-  1.10% )
          0.119862 +- 0.000542 seconds time elapsed  ( +-  0.45% )
          0.120075 +- 0.000608 seconds time elapsed  ( +-  0.51% )
          0.120350 +- 0.000273 seconds time elapsed  ( +-  0.23% )
           0.12203 +- 0.00114 seconds time elapsed  ( +-  0.93% )
           0.12229 +- 0.00114 seconds time elapsed  ( +-  0.93% )
           0.12032 +- 0.00115 seconds time elapsed  ( +-  0.95% )
          0.121241 +- 0.000463 seconds time elapsed  ( +-  0.38% )
          0.119404 +- 0.000333 seconds time elapsed  ( +-  0.28% )
          0.119945 +- 0.000766 seconds time elapsed  ( +-  0.64% )
          0.121215 +- 0.000879 seconds time elapsed  ( +-  0.72% )
           0.12001 +- 0.00109 seconds time elapsed  ( +-  0.91% )
           0.12193 +- 0.00182 seconds time elapsed  ( +-  1.49% )
          0.119184 +- 0.000794 seconds time elapsed  ( +-  0.67% )
          0.120062 +- 0.000439 seconds time elapsed  ( +-  0.37% )
          0.120834 +- 0.000760 seconds time elapsed  ( +-  0.63% )
          0.369473 +- 0.000992 seconds time elapsed  ( +-  0.27% )

... see how the vertical alignment of the output goes randomly wacko - 
I presume because there's a trailing zero in the output number and the 
code for some inexplicable reason decides to shorten it to make the 
life of developers harder? ;-)

2)

It's also incredibly hard to Ctrl-C a 'perf stat --repeat' instance:

 starship:~/tip> perf stat --null --repeat 20 perf stat --null --repeat 3 perf bench sched messaging
 # Running 'sched/messaging' benchmark:
 # 20 sender and receiver processes per group
 # 10 groups == 400 processes run

 ...
 Ctrl-C

 # Running 'sched/messaging' benchmark:
 perf: pollperf: perf: pollperf: pollpollperf: pollperf: pollperf: : Interrupted system call
 : Interrupted system call
 poll: Interrupted system call
 perf: pollperf: : Interrupted system call
 perf: pollperf: pollpollperf: : Interrupted system call
 pollperf: pollperf: perf: perf: pollpollpollperf: : Interrupted system call
 pollperf: poll: Interrupted system call
 : Interrupted system call
 : Interrupted system call
 : Interrupted system call
 perf: poll: Interrupted system call
 perf: perf: pollpoll: Interrupted system call
 : Interrupted system call
 perf: perf: perf: perf: perf: perf: : Interrupted system call
 pollpollpollpollpollpoll: Interrupted system call
 : Interrupted system call
 : Interrupted system call
 perf: perf: pollperf: perf: perf: perf: perf: perf: pollperf: : Interrupted system call
 pollpollpoll: Interrupted system call

Note how the perf stat instance actually *hangs*. I have to Ctrl-Z it, 
and kill -9 %1 it the hard way to clean up:

 pollpollpoll: Interrupted system call
 �
 [1]+  Stopped                 perf stat --null --repeat 20 perf stat --null --repeat 3 perf bench sched messaging
 starship:~/tip> kill -9 %1

 [1]+  Stopped                 perf stat --null --repeat 20 perf stat --null --repeat 3 perf bench sched messaging
 starship:~/tip> kill -9 %1

Does anyone use this thing for actual benchmarking work? ;-)

3)

It would also be nice to be able to Ctrl-C out of a 'perf top' instance 
that freezes the output or so. Or prints a snapshot in ASCII. Anything 
but what it does currently: it just exits and clears the xterm screen 
of all useful information...

I have to use 'f' (how many people know about that feature?) and copy & 
paste anything interesting from the screen the hard way.

4)

It would also be nice to have an export function to save current 'perf 
top' profiling data and have it available for 'perf report' et al 
analysis. Ie. frequently I just see an interesting snapshot and decide 
that it's good for further analysis, freeze the screen and are left 
with very few options to keep it for further look and reference.

5)

Would anyone be interested in an OpenGL-ish version of perf top, with 
its own low level shader for font-atlas based text output and vertex 
based polygon graphics, double buffering, full screen support with very 
little Xorg interaction for the GX pathway, etc? It should be *far* 
faster and lower overhead than the current ncurses->xterm->Wayland 
levels of indirection... and it could open up a new world of on-screen 
profiling information as well. Basically a very simple self-sustained 
OpenGL game engine for the key low level graphics primitives of modern 
GFX hardware. I could whip up a prototype if there's interest.

Thanks,

	Ingo

Re: [PATCH 1/3] perf sort: Keep output fields in the same level

[Namhyung Kim]

Hi Ingo,

On Thu, Mar 20, 2025 at 10:32:39AM +0100, Ingo Molnar wrote:
> 
> * Namhyung Kim <namhyung@kernel.org> wrote:
> 
> > On Fri, Mar 07, 2025 at 12:08:27AM -0800, Namhyung Kim wrote:
> > > This is useful for hierarchy output mode where the first level is
> > > considered as output fields.  We want them in the same level so that it
> > > can show only the remaining groups in the hierarchy.
> > > 
> > > Before:
> > >   $ perf report -s overhead,sample,period,comm,dso -H --stdio
> > >   ...
> > >   #          Overhead  Samples / Period / Command / Shared Object
> > >   # .................  ..........................................
> > >   #
> > >      100.00%           4035
> > >         100.00%           3835883066
> > >            100.00%           perf
> > >                99.37%           perf
> > >                 0.50%           ld-linux-x86-64.so.2
> > >                 0.06%           [unknown]
> > >                 0.04%           libc.so.6
> > >                 0.02%           libLLVM-16.so.1
> > > 
> > > After:
> > >   $ perf report -s overhead,sample,period,comm,dso -H --stdio
> > >   ...
> > >   #    Overhead       Samples        Period  Command / Shared Object
> > >   # .......................................  .......................
> > >   #
> > >      100.00%          4035    3835883066     perf
> > >          99.37%          4005    3811826223     perf
> > >           0.50%            19      19210014     ld-linux-x86-64.so.2
> > >           0.06%             8       2367089     [unknown]
> > >           0.04%             2       1720336     libc.so.6
> > >           0.02%             1        759404     libLLVM-16.so.1
> > > 
> > > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > 
> > Ping!  Anybody interested in this change? :)
> 
> Oh yes, all such pieces of intelligent organization of textual output 
> of profiling data are worth their weight in gold in my book. :-)
> 
>   Acked-by: Ingo Molnar <mingo@kernel.org>

Thanks a lot!

> 
> 1)
> 
> On a related note, does anyone know why perf stat output alignment 
> sucks so much these days:
> 
>   starship:~/tip> perf stat --null --repeat 20 perf stat --null --repeat 3 perf bench sched messaging 2>&1 | grep elapsed
>            0.11620 +- 0.00327 seconds time elapsed  ( +-  2.81% )
>           0.120813 +- 0.000570 seconds time elapsed  ( +-  0.47% )
>           0.122280 +- 0.000443 seconds time elapsed  ( +-  0.36% )
>           0.119813 +- 0.000752 seconds time elapsed  ( +-  0.63% )
>            0.12190 +- 0.00134 seconds time elapsed  ( +-  1.10% )
>           0.119862 +- 0.000542 seconds time elapsed  ( +-  0.45% )
>           0.120075 +- 0.000608 seconds time elapsed  ( +-  0.51% )
>           0.120350 +- 0.000273 seconds time elapsed  ( +-  0.23% )
>            0.12203 +- 0.00114 seconds time elapsed  ( +-  0.93% )
>            0.12229 +- 0.00114 seconds time elapsed  ( +-  0.93% )
>            0.12032 +- 0.00115 seconds time elapsed  ( +-  0.95% )
>           0.121241 +- 0.000463 seconds time elapsed  ( +-  0.38% )
>           0.119404 +- 0.000333 seconds time elapsed  ( +-  0.28% )
>           0.119945 +- 0.000766 seconds time elapsed  ( +-  0.64% )
>           0.121215 +- 0.000879 seconds time elapsed  ( +-  0.72% )
>            0.12001 +- 0.00109 seconds time elapsed  ( +-  0.91% )
>            0.12193 +- 0.00182 seconds time elapsed  ( +-  1.49% )
>           0.119184 +- 0.000794 seconds time elapsed  ( +-  0.67% )
>           0.120062 +- 0.000439 seconds time elapsed  ( +-  0.37% )
>           0.120834 +- 0.000760 seconds time elapsed  ( +-  0.63% )
>           0.369473 +- 0.000992 seconds time elapsed  ( +-  0.27% )
> 
> ... see how the vertical alignment of the output goes randomly wacko - 
> I presume because there's a trailing zero in the output number and the 
> code for some inexplicable reason decides to shorten it to make the 
> life of developers harder? ;-)

It seems it calcuates the precision when printing the result and doesn't
care much about the repeatations.

> 
> 2)
> 
> It's also incredibly hard to Ctrl-C a 'perf stat --repeat' instance:
> 
>  starship:~/tip> perf stat --null --repeat 20 perf stat --null --repeat 3 perf bench sched messaging
>  # Running 'sched/messaging' benchmark:
>  # 20 sender and receiver processes per group
>  # 10 groups == 400 processes run
> 
>  ...
>  Ctrl-C
> 
>  # Running 'sched/messaging' benchmark:
>  perf: pollperf: perf: pollperf: pollpollperf: pollperf: pollperf: : Interrupted system call
>  : Interrupted system call
>  poll: Interrupted system call
>  perf: pollperf: : Interrupted system call
>  perf: pollperf: pollpollperf: : Interrupted system call
>  pollperf: pollperf: perf: perf: pollpollpollperf: : Interrupted system call
>  pollperf: poll: Interrupted system call
>  : Interrupted system call
>  : Interrupted system call
>  : Interrupted system call
>  perf: poll: Interrupted system call
>  perf: perf: pollpoll: Interrupted system call
>  : Interrupted system call
>  perf: perf: perf: perf: perf: perf: : Interrupted system call
>  pollpollpollpollpollpoll: Interrupted system call
>  : Interrupted system call
>  : Interrupted system call
>  perf: perf: pollperf: perf: perf: perf: perf: perf: pollperf: : Interrupted system call
>  pollpollpoll: Interrupted system call
> 
> Note how the perf stat instance actually *hangs*. I have to Ctrl-Z it, 
> and kill -9 %1 it the hard way to clean up:
> 
>  pollpollpoll: Interrupted system call
>  �
>  [1]+  Stopped                 perf stat --null --repeat 20 perf stat --null --repeat 3 perf bench sched messaging
>  starship:~/tip> kill -9 %1
> 
>  [1]+  Stopped                 perf stat --null --repeat 20 perf stat --null --repeat 3 perf bench sched messaging
>  starship:~/tip> kill -9 %1
> 
> Does anyone use this thing for actual benchmarking work? ;-)

Oh, ok.  I'll take a look.

> 
> 3)
> 
> It would also be nice to be able to Ctrl-C out of a 'perf top' instance 
> that freezes the output or so. Or prints a snapshot in ASCII. Anything 
> but what it does currently: it just exits and clears the xterm screen 
> of all useful information...
> 
> I have to use 'f' (how many people know about that feature?) and copy & 
> paste anything interesting from the screen the hard way.

Actually I was not aware of 'f' key. :)  I think you can use 'P' to save
the current screen to a file.

> 
> 4)
> 
> It would also be nice to have an export function to save current 'perf 
> top' profiling data and have it available for 'perf report' et al 
> analysis. Ie. frequently I just see an interesting snapshot and decide 
> that it's good for further analysis, freeze the screen and are left 
> with very few options to keep it for further look and reference.

Hmm.. interesting.  I'll think about it but it seems doable.

> 
> 5)
> 
> Would anyone be interested in an OpenGL-ish version of perf top, with 
> its own low level shader for font-atlas based text output and vertex 
> based polygon graphics, double buffering, full screen support with very 
> little Xorg interaction for the GX pathway, etc? It should be *far* 
> faster and lower overhead than the current ncurses->xterm->Wayland 
> levels of indirection... and it could open up a new world of on-screen 
> profiling information as well. Basically a very simple self-sustained 
> OpenGL game engine for the key low level graphics primitives of modern 
> GFX hardware. I could whip up a prototype if there's interest.

I'm not familiar with the fancy GFX things but it'd be interesting to
see.  Not sure how well we can manage it in the future though, as we
have unloved GTK2 UI as well.

Thanks,
Namhyung

Re: [PATCH 1/3] perf sort: Keep output fields in the same level

[Namhyung Kim]

On Fri, 07 Mar 2025 00:08:27 -0800, Namhyung Kim wrote:
> This is useful for hierarchy output mode where the first level is
> considered as output fields.  We want them in the same level so that it
> can show only the remaining groups in the hierarchy.
> 
> Before:
>   $ perf report -s overhead,sample,period,comm,dso -H --stdio
>   ...
>   #          Overhead  Samples / Period / Command / Shared Object
>   # .................  ..........................................
>   #
>      100.00%           4035
>         100.00%           3835883066
>            100.00%           perf
>                99.37%           perf
>                 0.50%           ld-linux-x86-64.so.2
>                 0.06%           [unknown]
>                 0.04%           libc.so.6
>                 0.02%           libLLVM-16.so.1
> 
> [...]
Applied to perf-tools-next, thanks!

Best regards,
Namhyung

Re: [PATCH 1/3] perf sort: Keep output fields in the same level

[Ingo Molnar]

* Namhyung Kim <namhyung@kernel.org> wrote:

> > 3)
> > 
> > It would also be nice to be able to Ctrl-C out of a 'perf top' instance 
> > that freezes the output or so. Or prints a snapshot in ASCII. Anything 
> > but what it does currently: it just exits and clears the xterm screen 
> > of all useful information...
> > 
> > I have to use 'f' (how many people know about that feature?) and copy & 
> > paste anything interesting from the screen the hard way.
> 
> Actually I was not aware of 'f' key. :)  I think you can use 'P' to save
> the current screen to a file.

And I was not aware of the 'P' key. :-)

Would be nice if it also printed column and meta information, i.e. if 
it included these lines:

  Samples: 1M of event 'cycles:P', 4000 Hz, Event count (approx.): 758731362801 lost: 0/0 drop: 0/0
  Overhead  Shared Object            Symbol

or so?

BTW., the approximate 'event count' is a pretty uninformative value 
these days, right? Might as well not clutter the screen with it.

Thanks,

	Ingo

Re: [PATCH 1/3] perf sort: Keep output fields in the same level

[Namhyung Kim]

On Mon, Mar 24, 2025 at 08:28:31AM +0100, Ingo Molnar wrote:
> 
> * Namhyung Kim <namhyung@kernel.org> wrote:
> 
> > > 3)
> > > 
> > > It would also be nice to be able to Ctrl-C out of a 'perf top' instance 
> > > that freezes the output or so. Or prints a snapshot in ASCII. Anything 
> > > but what it does currently: it just exits and clears the xterm screen 
> > > of all useful information...
> > > 
> > > I have to use 'f' (how many people know about that feature?) and copy & 
> > > paste anything interesting from the screen the hard way.
> > 
> > Actually I was not aware of 'f' key. :)  I think you can use 'P' to save
> > the current screen to a file.
> 
> And I was not aware of the 'P' key. :-)
> 
> Would be nice if it also printed column and meta information, i.e. if 
> it included these lines:
> 
>   Samples: 1M of event 'cycles:P', 4000 Hz, Event count (approx.): 758731362801 lost: 0/0 drop: 0/0
>   Overhead  Shared Object            Symbol
> 
> or so?

Yep, maybe we can do the same as the stdio output.

  #
  #
  # Total Lost Samples: 0
  #
  # Samples: 7  of event 'cycles'
  # Event count (approx.): 3641468
  #
  # Overhead  Command  Shared Object     Symbol                            
  # ........  .......  ................  ..................................
  #

> 
> BTW., the approximate 'event count' is a pretty uninformative value 
> these days, right? Might as well not clutter the screen with it.

Maybe.  I don't look at the number so far but also don't feel the need
for removal.

Thanks,
Namhyung

Re: [PATCH 1/3] perf sort: Keep output fields in the same level

[Namhyung Kim]

On Thu, Mar 20, 2025 at 10:32:39AM +0100, Ingo Molnar wrote:
> 4)
> 
> It would also be nice to have an export function to save current 'perf 
> top' profiling data and have it available for 'perf report' et al 
> analysis. Ie. frequently I just see an interesting snapshot and decide 
> that it's good for further analysis, freeze the screen and are left 
> with very few options to keep it for further look and reference.

I've thought about this but there are some limitations.  Basically perf
top and perf report aggregates sammples into hist_entry's using the
given sort keys.  That means other information in the samples are gone
already.

For perf top, it only has dso and symbol - plus number of samples and
the total periods.  Other informations like CPU, PID and TIME are not
available so it cannot reconstruct samples precisely.  So it's only
possible to use the same sort keys or a subset of them IMHO.

Do you think it'd be useful even with this?

Thanks,
Namhyung

Re: [PATCH 1/3] perf sort: Keep output fields in the same level

[Namhyung Kim]

On Thu, Mar 20, 2025 at 10:32:39AM +0100, Ingo Molnar wrote:
> 
> * Namhyung Kim <namhyung@kernel.org> wrote:
> 
> > On Fri, Mar 07, 2025 at 12:08:27AM -0800, Namhyung Kim wrote:
> > > This is useful for hierarchy output mode where the first level is
> > > considered as output fields.  We want them in the same level so that it
> > > can show only the remaining groups in the hierarchy.
> > > 
> > > Before:
> > >   $ perf report -s overhead,sample,period,comm,dso -H --stdio
> > >   ...
> > >   #          Overhead  Samples / Period / Command / Shared Object
> > >   # .................  ..........................................
> > >   #
> > >      100.00%           4035
> > >         100.00%           3835883066
> > >            100.00%           perf
> > >                99.37%           perf
> > >                 0.50%           ld-linux-x86-64.so.2
> > >                 0.06%           [unknown]
> > >                 0.04%           libc.so.6
> > >                 0.02%           libLLVM-16.so.1
> > > 
> > > After:
> > >   $ perf report -s overhead,sample,period,comm,dso -H --stdio
> > >   ...
> > >   #    Overhead       Samples        Period  Command / Shared Object
> > >   # .......................................  .......................
> > >   #
> > >      100.00%          4035    3835883066     perf
> > >          99.37%          4005    3811826223     perf
> > >           0.50%            19      19210014     ld-linux-x86-64.so.2
> > >           0.06%             8       2367089     [unknown]
> > >           0.04%             2       1720336     libc.so.6
> > >           0.02%             1        759404     libLLVM-16.so.1
> > > 
> > > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > 
> > Ping!  Anybody interested in this change? :)
> 
> Oh yes, all such pieces of intelligent organization of textual output 
> of profiling data are worth their weight in gold in my book. :-)
> 
>   Acked-by: Ingo Molnar <mingo@kernel.org>
> 
> 1)
> 
> On a related note, does anyone know why perf stat output alignment 
> sucks so much these days:
> 
>   starship:~/tip> perf stat --null --repeat 20 perf stat --null --repeat 3 perf bench sched messaging 2>&1 | grep elapsed
>            0.11620 +- 0.00327 seconds time elapsed  ( +-  2.81% )
>           0.120813 +- 0.000570 seconds time elapsed  ( +-  0.47% )
>           0.122280 +- 0.000443 seconds time elapsed  ( +-  0.36% )
>           0.119813 +- 0.000752 seconds time elapsed  ( +-  0.63% )
>            0.12190 +- 0.00134 seconds time elapsed  ( +-  1.10% )
>           0.119862 +- 0.000542 seconds time elapsed  ( +-  0.45% )
>           0.120075 +- 0.000608 seconds time elapsed  ( +-  0.51% )
>           0.120350 +- 0.000273 seconds time elapsed  ( +-  0.23% )
>            0.12203 +- 0.00114 seconds time elapsed  ( +-  0.93% )
>            0.12229 +- 0.00114 seconds time elapsed  ( +-  0.93% )
>            0.12032 +- 0.00115 seconds time elapsed  ( +-  0.95% )
>           0.121241 +- 0.000463 seconds time elapsed  ( +-  0.38% )
>           0.119404 +- 0.000333 seconds time elapsed  ( +-  0.28% )
>           0.119945 +- 0.000766 seconds time elapsed  ( +-  0.64% )
>           0.121215 +- 0.000879 seconds time elapsed  ( +-  0.72% )
>            0.12001 +- 0.00109 seconds time elapsed  ( +-  0.91% )
>            0.12193 +- 0.00182 seconds time elapsed  ( +-  1.49% )
>           0.119184 +- 0.000794 seconds time elapsed  ( +-  0.67% )
>           0.120062 +- 0.000439 seconds time elapsed  ( +-  0.37% )
>           0.120834 +- 0.000760 seconds time elapsed  ( +-  0.63% )
>           0.369473 +- 0.000992 seconds time elapsed  ( +-  0.27% )
> 
> ... see how the vertical alignment of the output goes randomly wacko - 
> I presume because there's a trailing zero in the output number and the 
> code for some inexplicable reason decides to shorten it to make the 
> life of developers harder? ;-)

I found this commit. :)  I think it didn't consider nested perf stat
with --repeat options.

commit bc22de9bcdb2249150fb5b3c48fdc4f6bedd3ad7
Author: Jiri Olsa <jolsa@kernel.org>
Date:   Mon Apr 23 11:08:20 2018 +0200

    perf stat: Display time in precision based on std deviation
    
    Ingo suggested to display elapsed time for multirun workload (perf stat
    -e) with precision based on the precision of the standard deviation.
    
    In his own words:
    
      > This output is a slightly bit misleading:
    
      >  Performance counter stats for 'make -j128' (10 runs):
      >      27.988995256 seconds time elapsed                  ( +-  0.39% )
    
      > The 9 significant digits in the result, while only 1 is valid, suggests accuracy
      > where none exists.
    
      > It would be better if 'perf stat' would display elapsed time with a precision
      > adjusted to stddev, it should display at most 2 more significant digits than
      > the stddev inaccuracy.
    
      > I.e. in the above case 0.39% is 0.109, so we only have accuracy for 1 digit, and
      > so we should only display 3:
    
      >        27.988 seconds time elapsed                       ( +-  0.39% )
    
    Plus a suggestion about the output, which is small enough and connected
    with the above change that I merged both changes together.
    
      > Small output style nit - I think it would be nice if with --repeat the stddev was
      > also displayed in absolute values, besides percentage:
      >
      >       27.988 +- 0.109 seconds time elapsed   ( +- 0.39% )
    
    The output is now:
    
       Performance counter stats for './perf bench sched pipe' (5 runs):
       SNIP
               13.3667 +- 0.0256 seconds time elapsed  ( +-  0.19% )
    
    Suggested-by: Ingo Molnar <mingo@kernel.org>
    Signed-off-by: Jiri Olsa <jolsa@kernel.org>
    Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
    Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
    Cc: Andi Kleen <ak@linux.intel.com>
    Cc: David Ahern <dsahern@gmail.com>
    Cc: Namhyung Kim <namhyung@kernel.org>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Link: http://lkml.kernel.org/r/20180423090823.32309-7-jolsa@kernel.org
    Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>

> 
> 2)
> 
> It's also incredibly hard to Ctrl-C a 'perf stat --repeat' instance:
> 
>  starship:~/tip> perf stat --null --repeat 20 perf stat --null --repeat 3 perf bench sched messaging
>  # Running 'sched/messaging' benchmark:
>  # 20 sender and receiver processes per group
>  # 10 groups == 400 processes run
> 
>  ...
>  Ctrl-C
> 
>  # Running 'sched/messaging' benchmark:
>  perf: pollperf: perf: pollperf: pollpollperf: pollperf: pollperf: : Interrupted system call
>  : Interrupted system call
>  poll: Interrupted system call
>  perf: pollperf: : Interrupted system call
>  perf: pollperf: pollpollperf: : Interrupted system call
>  pollperf: pollperf: perf: perf: pollpollpollperf: : Interrupted system call
>  pollperf: poll: Interrupted system call
>  : Interrupted system call
>  : Interrupted system call
>  : Interrupted system call
>  perf: poll: Interrupted system call
>  perf: perf: pollpoll: Interrupted system call
>  : Interrupted system call
>  perf: perf: perf: perf: perf: perf: : Interrupted system call
>  pollpollpollpollpollpoll: Interrupted system call
>  : Interrupted system call
>  : Interrupted system call
>  perf: perf: pollperf: perf: perf: perf: perf: perf: pollperf: : Interrupted system call
>  pollpollpoll: Interrupted system call
> 
> Note how the perf stat instance actually *hangs*. I have to Ctrl-Z it, 
> and kill -9 %1 it the hard way to clean up:

It looks like the problem is in the perf bench sched messaging not in
perf stat.  If the signal delivered before all groups are created, it
may wait for children indefinitely.

Thanks,
Namhyung

> 
>  pollpollpoll: Interrupted system call
>  �
>  [1]+  Stopped                 perf stat --null --repeat 20 perf stat --null --repeat 3 perf bench sched messaging
>  starship:~/tip> kill -9 %1
> 
>  [1]+  Stopped                 perf stat --null --repeat 20 perf stat --null --repeat 3 perf bench sched messaging
>  starship:~/tip> kill -9 %1
> 
> Does anyone use this thing for actual benchmarking work? ;-)