[PATCH 1/3 v2] perf stat: don't report a null stalled cycles per insn metric

[PATCH 1/3 v2] perf stat: don't report a null stalled cycles per insn metric

[Kim Phillips]

For data collected on machines with front end stalled cycles supported,
such as found on modern AMD CPU families, commit 146540fb545b ("perf
stat: Always separate stalled cycles per insn") introduces a new line
in CSV output with a leading comma that upsets some automated scripts.
Scripts have to use "-e ex_ret_instr" to work around this issue, after
upgrading to a version of perf with that commit.

We could add "if (have_frontend_stalled && !config->csv_sep)"
to the not (total && avg) else clause, to emphasize that CSV users
are usually scripts, and are written to do only what is needed, i.e.,
they wouldn't typically invoke "perf stat" without specifying an
explicit event list.

But - let alone CSV output - why should users now tolerate a constant
0-reporting extra line in regular terminal output?:

BEFORE:

$ sudo perf stat --all-cpus -einstructions,cycles -- sleep 1

 Performance counter stats for 'system wide':

       181,110,981      instructions              #    0.58  insn per cycle
                                                  #    0.00  stalled cycles per insn
       309,876,469      cycles

       1.002202582 seconds time elapsed

The user would not like to see the now permanent
"0.00  stalled cycles per insn" line fixture, as it gives
no useful information.

So this patch removes the printing of the zeroed stalled cycles
line altogether, almost reverting the very original commit fb4605ba47e7
("perf stat: Check for frontend stalled for metrics"), which seems
like it was written to normalize --metric-only column output
of common Intel machines at the time: modern Intel machines
have ceased to support the genericised frontend stalled metrics AFAICT.

AFTER:

$ sudo perf stat --all-cpus -einstructions,cycles -- sleep 1

 Performance counter stats for 'system wide':

       244,071,432      instructions              #    0.69  insn per cycle
       355,353,490      cycles

       1.001862516 seconds time elapsed

Output behaviour when stalled cycles is indeed measured is not affected
(BEFORE == AFTER):

$ sudo perf stat --all-cpus -einstructions,cycles,stalled-cycles-frontend -- sleep 1

 Performance counter stats for 'system wide':

       247,227,799      instructions              #    0.63  insn per cycle
                                                  #    0.26  stalled cycles per insn
       394,745,636      cycles
        63,194,485      stalled-cycles-frontend   #   16.01% frontend cycles idle

       1.002079770 seconds time elapsed

Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Cong Wang <xiyou.wangcong@gmail.com>
Cc: Jin Yao <yao.jin@linux.intel.com>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Kim Phillips <kim.phillips@amd.com>
Cc: Song Liu <songliubraving@fb.com>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: linux-perf-users@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Acked-by: Jiri Olsa <jolsa@redhat.com>
Acked-by: Andi Kleen <ak@linux.intel.com>
Fixes: 146540fb545b ("perf stat: Always separate stalled cycles per insn")
Signed-off-by: Kim Phillips <kim.phillips@amd.com>
---
v2: Added Acked-bys from Jiri Olsa:

https://lore.kernel.org/lkml/20200120092928.GD608405@krava/

and Andi Kleen:

https://lore.kernel.org/lkml/20200120115720.GQ302770@tassilo.jf.intel.com/

 tools/perf/util/stat-shadow.c | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/tools/perf/util/stat-shadow.c b/tools/perf/util/stat-shadow.c
index 2c41d47f6f83..90d23cc3c8d4 100644
--- a/tools/perf/util/stat-shadow.c
+++ b/tools/perf/util/stat-shadow.c
@@ -18,7 +18,6 @@
  * AGGR_NONE: Use matching CPU
  * AGGR_THREAD: Not supported?
  */
-static bool have_frontend_stalled;
 
 struct runtime_stat rt_stat;
 struct stats walltime_nsecs_stats;
@@ -144,7 +143,6 @@ void runtime_stat__exit(struct runtime_stat *st)
 
 void perf_stat__init_shadow_stats(void)
 {
-	have_frontend_stalled = pmu_have_event("cpu", "stalled-cycles-frontend");
 	runtime_stat__init(&rt_stat);
 }
 
@@ -853,10 +851,6 @@ void perf_stat__print_shadow_stats(struct perf_stat_config *config,
 			print_metric(config, ctxp, NULL, "%7.2f ",
 					"stalled cycles per insn",
 					ratio);
-		} else if (have_frontend_stalled) {
-			out->new_line(config, ctxp);
-			print_metric(config, ctxp, NULL, "%7.2f ",
-				     "stalled cycles per insn", 0);
 		}
 	} else if (perf_evsel__match(evsel, HARDWARE, HW_BRANCH_MISSES)) {
 		if (runtime_stat_n(st, STAT_BRANCHES, ctx, cpu) != 0)
-- 
2.25.0

[PATCH 2/3 v2] perf symbols: Update the list of kernel idle symbols

[Kim Phillips]

"acpi_idle_do_entry", "acpi_processor_ffh_cstate_enter", and "idle_cpu"
appear in 'perf top' output, at least on AMD systems.

Add them to perf's idle_symbols list, so they don't dominate 'perf top'
output.

Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Cong Wang <xiyou.wangcong@gmail.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Jin Yao <yao.jin@linux.intel.com>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Kim Phillips <kim.phillips@amd.com>
Cc: Song Liu <songliubraving@fb.com>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: linux-perf-users@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Acked-by: Jiri Olsa <jolsa@redhat.com>
Signed-off-by: Kim Phillips <kim.phillips@amd.com>
---
v2: Added Jiri Olsa's acked-by:

https://lore.kernel.org/lkml/20200120092844.GC608405@krava/

 tools/perf/util/symbol.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index 3b379b1296f1..f3120c4f47ad 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -635,9 +635,12 @@ int modules__parse(const char *filename, void *arg,
 static bool symbol__is_idle(const char *name)
 {
 	const char * const idle_symbols[] = {
+		"acpi_idle_do_entry",
+		"acpi_processor_ffh_cstate_enter",
 		"arch_cpu_idle",
 		"cpu_idle",
 		"cpu_startup_entry",
+		"idle_cpu",
 		"intel_idle",
 		"default_idle",
 		"native_safe_halt",
-- 
2.25.0

[PATCH 3/3 v2] perf symbols: convert symbol__is_idle to use strlist

[Kim Phillips]

Use the more optimized strlist implementation to do the idle function
lookup.

Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Cong Wang <xiyou.wangcong@gmail.com>
Cc: Jin Yao <yao.jin@linux.intel.com>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Kim Phillips <kim.phillips@amd.com>
Cc: Song Liu <songliubraving@fb.com>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: linux-perf-users@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Kim Phillips <kim.phillips@amd.com>
---
v2: new this series, based on Jiri's comment:

https://lore.kernel.org/lkml/20200120092844.GC608405@krava/

...and this time with the Cc list intact.

 tools/perf/util/symbol.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index f3120c4f47ad..1077013d8ce2 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -654,13 +654,17 @@ static bool symbol__is_idle(const char *name)
 		NULL
 	};
 	int i;
+	static struct strlist *idle_symbols_list;
 
-	for (i = 0; idle_symbols[i]; i++) {
-		if (!strcmp(idle_symbols[i], name))
-			return true;
-	}
+	if (idle_symbols_list)
+		return strlist__has_entry(idle_symbols_list, name);
 
-	return false;
+	idle_symbols_list = strlist__new(NULL, NULL);
+
+	for (i = 0; idle_symbols[i]; i++)
+		strlist__add(idle_symbols_list, idle_symbols[i]);
+
+	return strlist__has_entry(idle_symbols_list, name);
 }
 
 static int map__process_kallsym_symbol(void *arg, const char *name,
-- 
2.25.0

Re: [PATCH 3/3 v2] perf symbols: convert symbol__is_idle to use strlist

[Song Liu]

> On Feb 10, 2020, at 8:31 AM, Kim Phillips <kim.phillips@amd.com> wrote:
> 
> Use the more optimized strlist implementation to do the idle function
> lookup.
> 
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Andi Kleen <ak@linux.intel.com>
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
> Cc: Jiri Olsa <jolsa@kernel.org>
> Cc: Jiri Olsa <jolsa@redhat.com>
> Cc: Namhyung Kim <namhyung@kernel.org>
> Cc: Cong Wang <xiyou.wangcong@gmail.com>
> Cc: Jin Yao <yao.jin@linux.intel.com>
> Cc: Kan Liang <kan.liang@linux.intel.com>
> Cc: Kim Phillips <kim.phillips@amd.com>
> Cc: Song Liu <songliubraving@fb.com>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Cc: linux-perf-users@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> Signed-off-by: Kim Phillips <kim.phillips@amd.com>
> ---
> v2: new this series, based on Jiri's comment:
> 
> https://lore.kernel.org/lkml/20200120092844.GC608405@krava/
> 
> ...and this time with the Cc list intact.
> 
> tools/perf/util/symbol.c | 14 +++++++++-----
> 1 file changed, 9 insertions(+), 5 deletions(-)
> 
> diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
> index f3120c4f47ad..1077013d8ce2 100644
> --- a/tools/perf/util/symbol.c
> +++ b/tools/perf/util/symbol.c
> @@ -654,13 +654,17 @@ static bool symbol__is_idle(const char *name)
> 		NULL
> 	};
> 	int i;
> +	static struct strlist *idle_symbols_list;

nit, probably just personal preference:

Maybe move idle_symbols_list out of the function and add the logic 
to symbol__init()?

Other than this:

Acked-by: Song Liu <songliubraving@fb.com>

> 
> -	for (i = 0; idle_symbols[i]; i++) {
> -		if (!strcmp(idle_symbols[i], name))
> -			return true;
> -	}
> +	if (idle_symbols_list)
> +		return strlist__has_entry(idle_symbols_list, name);
> 
> -	return false;
> +	idle_symbols_list = strlist__new(NULL, NULL);
> +
> +	for (i = 0; idle_symbols[i]; i++)
> +		strlist__add(idle_symbols_list, idle_symbols[i]);
> +
> +	return strlist__has_entry(idle_symbols_list, name);
> }
> 
> static int map__process_kallsym_symbol(void *arg, const char *name,
> -- 
> 2.25.0
> 

Re: [PATCH 2/3 v2] perf symbols: Update the list of kernel idle symbols

[Song Liu]

> On Feb 7, 2020, at 3:06 PM, Kim Phillips <kim.phillips@amd.com> wrote:
> 
> "acpi_idle_do_entry", "acpi_processor_ffh_cstate_enter", and "idle_cpu"
> appear in 'perf top' output, at least on AMD systems.
> 
> Add them to perf's idle_symbols list, so they don't dominate 'perf top'
> output.
> 
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
> Cc: Jiri Olsa <jolsa@kernel.org>
> Cc: Namhyung Kim <namhyung@kernel.org>
> Cc: Cong Wang <xiyou.wangcong@gmail.com>
> Cc: Andi Kleen <ak@linux.intel.com>
> Cc: Jin Yao <yao.jin@linux.intel.com>
> Cc: Kan Liang <kan.liang@linux.intel.com>
> Cc: Kim Phillips <kim.phillips@amd.com>
> Cc: Song Liu <songliubraving@fb.com>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Cc: linux-perf-users@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> Acked-by: Jiri Olsa <jolsa@redhat.com>
> Signed-off-by: Kim Phillips <kim.phillips@amd.com>

Acked-by: Song Liu <songliubraving@fb.com>

Re: [PATCH 1/3 v2] perf stat: don't report a null stalled cycles per insn metric

[Song Liu]

> On Feb 7, 2020, at 3:06 PM, Kim Phillips <kim.phillips@amd.com> wrote:
> 
> For data collected on machines with front end stalled cycles supported,
> such as found on modern AMD CPU families, commit 146540fb545b ("perf
> stat: Always separate stalled cycles per insn") introduces a new line
> in CSV output with a leading comma that upsets some automated scripts.
> Scripts have to use "-e ex_ret_instr" to work around this issue, after
> upgrading to a version of perf with that commit.
> 
> We could add "if (have_frontend_stalled && !config->csv_sep)"
> to the not (total && avg) else clause, to emphasize that CSV users
> are usually scripts, and are written to do only what is needed, i.e.,
> they wouldn't typically invoke "perf stat" without specifying an
> explicit event list.
> 
> But - let alone CSV output - why should users now tolerate a constant
> 0-reporting extra line in regular terminal output?:
> 
> BEFORE:
> 
> $ sudo perf stat --all-cpus -einstructions,cycles -- sleep 1
> 
> Performance counter stats for 'system wide':
> 
>       181,110,981      instructions              #    0.58  insn per cycle
>                                                  #    0.00  stalled cycles per insn
>       309,876,469      cycles
> 
>       1.002202582 seconds time elapsed
> 
> The user would not like to see the now permanent
> "0.00  stalled cycles per insn" line fixture, as it gives
> no useful information.
> 
> So this patch removes the printing of the zeroed stalled cycles
> line altogether, almost reverting the very original commit fb4605ba47e7
> ("perf stat: Check for frontend stalled for metrics"), which seems
> like it was written to normalize --metric-only column output
> of common Intel machines at the time: modern Intel machines
> have ceased to support the genericised frontend stalled metrics AFAICT.
> 
> AFTER:
> 
> $ sudo perf stat --all-cpus -einstructions,cycles -- sleep 1
> 
> Performance counter stats for 'system wide':
> 
>       244,071,432      instructions              #    0.69  insn per cycle
>       355,353,490      cycles
> 
>       1.001862516 seconds time elapsed
> 
> Output behaviour when stalled cycles is indeed measured is not affected
> (BEFORE == AFTER):
> 
> $ sudo perf stat --all-cpus -einstructions,cycles,stalled-cycles-frontend -- sleep 1
> 
> Performance counter stats for 'system wide':
> 
>       247,227,799      instructions              #    0.63  insn per cycle
>                                                  #    0.26  stalled cycles per insn
>       394,745,636      cycles
>        63,194,485      stalled-cycles-frontend   #   16.01% frontend cycles idle
> 
>       1.002079770 seconds time elapsed
> 
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
> Cc: Jiri Olsa <jolsa@kernel.org>
> Cc: Namhyung Kim <namhyung@kernel.org>
> Cc: Cong Wang <xiyou.wangcong@gmail.com>
> Cc: Jin Yao <yao.jin@linux.intel.com>
> Cc: Kan Liang <kan.liang@linux.intel.com>
> Cc: Kim Phillips <kim.phillips@amd.com>
> Cc: Song Liu <songliubraving@fb.com>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Cc: linux-perf-users@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> Acked-by: Jiri Olsa <jolsa@redhat.com>
> Acked-by: Andi Kleen <ak@linux.intel.com>
> Fixes: 146540fb545b ("perf stat: Always separate stalled cycles per insn")
> Signed-off-by: Kim Phillips <kim.phillips@amd.com>

Acked-by: Song Liu <songliubraving@fb.com>

Thanks for improving perf on AMD systems! 

Re: [PATCH 3/3 v2] perf symbols: convert symbol__is_idle to use strlist

[Arnaldo Carvalho de Melo]

Em Mon, Feb 10, 2020 at 06:51:53PM +0000, Song Liu escreveu:
> 
> 
> > On Feb 10, 2020, at 8:31 AM, Kim Phillips <kim.phillips@amd.com> wrote:
> > 
> > Use the more optimized strlist implementation to do the idle function
> > lookup.
> > 
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > Cc: Ingo Molnar <mingo@redhat.com>
> > Cc: Andi Kleen <ak@linux.intel.com>
> > Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> > Cc: Mark Rutland <mark.rutland@arm.com>
> > Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
> > Cc: Jiri Olsa <jolsa@kernel.org>
> > Cc: Jiri Olsa <jolsa@redhat.com>
> > Cc: Namhyung Kim <namhyung@kernel.org>
> > Cc: Cong Wang <xiyou.wangcong@gmail.com>
> > Cc: Jin Yao <yao.jin@linux.intel.com>
> > Cc: Kan Liang <kan.liang@linux.intel.com>
> > Cc: Kim Phillips <kim.phillips@amd.com>
> > Cc: Song Liu <songliubraving@fb.com>
> > Cc: Davidlohr Bueso <dave@stgolabs.net>
> > Cc: linux-perf-users@vger.kernel.org
> > Cc: linux-kernel@vger.kernel.org
> > Signed-off-by: Kim Phillips <kim.phillips@amd.com>
> > ---
> > v2: new this series, based on Jiri's comment:
> > 
> > https://lore.kernel.org/lkml/20200120092844.GC608405@krava/
> > 
> > ...and this time with the Cc list intact.
> > 
> > tools/perf/util/symbol.c | 14 +++++++++-----
> > 1 file changed, 9 insertions(+), 5 deletions(-)
> > 
> > diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
> > index f3120c4f47ad..1077013d8ce2 100644
> > --- a/tools/perf/util/symbol.c
> > +++ b/tools/perf/util/symbol.c
> > @@ -654,13 +654,17 @@ static bool symbol__is_idle(const char *name)
> > 		NULL
> > 	};
> > 	int i;
> > +	static struct strlist *idle_symbols_list;
> 
> nit, probably just personal preference:
> 
> Maybe move idle_symbols_list out of the function and add the logic 
> to symbol__init()?
> 
> Other than this:
> 
> Acked-by: Song Liu <songliubraving@fb.com>

I applied it as is, improvements can be made on top of it.

- Arnaldo
 
> > 
> > -	for (i = 0; idle_symbols[i]; i++) {
> > -		if (!strcmp(idle_symbols[i], name))
> > -			return true;
> > -	}
> > +	if (idle_symbols_list)
> > +		return strlist__has_entry(idle_symbols_list, name);
> > 
> > -	return false;
> > +	idle_symbols_list = strlist__new(NULL, NULL);
> > +
> > +	for (i = 0; idle_symbols[i]; i++)
> > +		strlist__add(idle_symbols_list, idle_symbols[i]);
> > +
> > +	return strlist__has_entry(idle_symbols_list, name);
> > }
> > 
> > static int map__process_kallsym_symbol(void *arg, const char *name,
> > -- 
> > 2.25.0
> > 
> 

-- 

- Arnaldo