[PATCH] perf test: Update all metrics test like metricgroups test

[PATCH] perf test: Update all metrics test like metricgroups test

[Namhyung Kim]

Like in the metricgroup tests, it should check the permission first and
then skip relevant failures accordingly.

Also it needs to try again with the system wide flag properly.  On the
second round, check if the result has the metric name because other
failure cases are checked in the first round already.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/shell/stat_all_metrics.sh | 87 ++++++++++++++++------
 1 file changed, 66 insertions(+), 21 deletions(-)

diff --git a/tools/perf/tests/shell/stat_all_metrics.sh b/tools/perf/tests/shell/stat_all_metrics.sh
index 54774525e18a7b9a..73e9347e88a964bd 100755
--- a/tools/perf/tests/shell/stat_all_metrics.sh
+++ b/tools/perf/tests/shell/stat_all_metrics.sh
@@ -2,42 +2,87 @@
 # perf all metrics test
 # SPDX-License-Identifier: GPL-2.0
 
+ParanoidAndNotRoot()
+{
+  [ "$(id -u)" != 0 ] && [ "$(cat /proc/sys/kernel/perf_event_paranoid)" -gt $1 ]
+}
+
+system_wide_flag="-a"
+if ParanoidAndNotRoot 0
+then
+  system_wide_flag=""
+fi
+
 err=0
 for m in $(perf list --raw-dump metrics); do
   echo "Testing $m"
-  result=$(perf stat -M "$m" true 2>&1)
-  if [[ "$result" =~ ${m:0:50} ]] || [[ "$result" =~ "<not supported>" ]]
+  result=$(perf stat -M "$m" $system_wide_flag -- sleep 0.01 2>&1)
+  result_err=$?
+  if [[ $result_err -gt 0 ]]
   then
-    continue
+    if [[ "$result" =~ \
+          "Access to performance monitoring and observability operations is limited" ]]
+    then
+      echo "Permission failure"
+      echo $result
+      if [[ $err -eq 0 ]]
+      then
+        err=2 # Skip
+      fi
+      continue
+    elif [[ "$result" =~ "in per-thread mode, enable system wide" ]]
+    then
+      echo "Permissions - need system wide mode"
+      echo $result
+      if [[ $err -eq 0 ]]
+      then
+        err=2 # Skip
+      fi
+      continue
+    elif [[ "$result" =~ "<not supported>" ]]
+    then
+      echo "Not supported events"
+      echo $result
+      if [[ $err -eq 0 ]]
+      then
+        err=2 # Skip
+      fi
+      continue
+    elif [[ "$result" =~ "FP_ARITH" || "$result" =~ "AMX" ]]
+    then
+      echo "FP issues"
+      echo $result
+      if [[ $err -eq 0 ]]
+      then
+        err=2 # Skip
+      fi
+      continue
+    elif [[ "$result" =~ "PMM" ]]
+    then
+      echo "Optane memory issues"
+      echo $result
+      if [[ $err -eq 0 ]]
+      then
+        err=2 # Skip
+      fi
+      continue
+    fi
   fi
-  # Failed so try system wide.
-  result=$(perf stat -M "$m" -a sleep 0.01 2>&1)
+
   if [[ "$result" =~ ${m:0:50} ]]
   then
     continue
   fi
-  # Failed again, possibly the workload was too small so retry with something
-  # longer.
-  result=$(perf stat -M "$m" perf bench internals synthesize 2>&1)
+
+  # Failed, possibly the workload was too small so retry with something longer.
+  result=$(perf stat -M "$m" $system_wide_flag -- perf bench internals synthesize 2>&1)
   if [[ "$result" =~ ${m:0:50} ]]
   then
     continue
   fi
   echo "Metric '$m' not printed in:"
   echo "$result"
-  if [[ "$err" != "1" ]]
-  then
-    err=2
-    if [[ "$result" =~ "FP_ARITH" || "$result" =~ "AMX" ]]
-    then
-      echo "Skip, not fail, for FP issues"
-    elif [[ "$result" =~ "PMM" ]]
-    then
-      echo "Skip, not fail, for Optane memory issues"
-    else
-      err=1
-    fi
-  fi
+  err=1
 done
 
 exit "$err"
-- 
2.47.0.rc1.288.g06298d1525-goog

Re: [PATCH] perf test: Update all metrics test like metricgroups test

[Ian Rogers]

On Fri, Oct 18, 2024 at 1:43 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> Like in the metricgroup tests, it should check the permission first and
> then skip relevant failures accordingly.
>
> Also it needs to try again with the system wide flag properly.  On the
> second round, check if the result has the metric name because other
> failure cases are checked in the first round already.
>
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>

Reviewed-by: Ian Rogers <irogers@google.com>

Thanks,
Ian

> ---
>  tools/perf/tests/shell/stat_all_metrics.sh | 87 ++++++++++++++++------
>  1 file changed, 66 insertions(+), 21 deletions(-)
>
> diff --git a/tools/perf/tests/shell/stat_all_metrics.sh b/tools/perf/tests/shell/stat_all_metrics.sh
> index 54774525e18a7b9a..73e9347e88a964bd 100755
> --- a/tools/perf/tests/shell/stat_all_metrics.sh
> +++ b/tools/perf/tests/shell/stat_all_metrics.sh
> @@ -2,42 +2,87 @@
>  # perf all metrics test
>  # SPDX-License-Identifier: GPL-2.0
>
> +ParanoidAndNotRoot()
> +{
> +  [ "$(id -u)" != 0 ] && [ "$(cat /proc/sys/kernel/perf_event_paranoid)" -gt $1 ]
> +}
> +
> +system_wide_flag="-a"
> +if ParanoidAndNotRoot 0
> +then
> +  system_wide_flag=""
> +fi
> +
>  err=0
>  for m in $(perf list --raw-dump metrics); do
>    echo "Testing $m"
> -  result=$(perf stat -M "$m" true 2>&1)
> -  if [[ "$result" =~ ${m:0:50} ]] || [[ "$result" =~ "<not supported>" ]]
> +  result=$(perf stat -M "$m" $system_wide_flag -- sleep 0.01 2>&1)
> +  result_err=$?
> +  if [[ $result_err -gt 0 ]]
>    then
> -    continue
> +    if [[ "$result" =~ \
> +          "Access to performance monitoring and observability operations is limited" ]]
> +    then
> +      echo "Permission failure"
> +      echo $result
> +      if [[ $err -eq 0 ]]
> +      then
> +        err=2 # Skip
> +      fi
> +      continue
> +    elif [[ "$result" =~ "in per-thread mode, enable system wide" ]]
> +    then
> +      echo "Permissions - need system wide mode"
> +      echo $result
> +      if [[ $err -eq 0 ]]
> +      then
> +        err=2 # Skip
> +      fi
> +      continue
> +    elif [[ "$result" =~ "<not supported>" ]]
> +    then
> +      echo "Not supported events"
> +      echo $result
> +      if [[ $err -eq 0 ]]
> +      then
> +        err=2 # Skip
> +      fi
> +      continue
> +    elif [[ "$result" =~ "FP_ARITH" || "$result" =~ "AMX" ]]
> +    then
> +      echo "FP issues"
> +      echo $result
> +      if [[ $err -eq 0 ]]
> +      then
> +        err=2 # Skip
> +      fi
> +      continue
> +    elif [[ "$result" =~ "PMM" ]]
> +    then
> +      echo "Optane memory issues"
> +      echo $result
> +      if [[ $err -eq 0 ]]
> +      then
> +        err=2 # Skip
> +      fi
> +      continue
> +    fi
>    fi
> -  # Failed so try system wide.
> -  result=$(perf stat -M "$m" -a sleep 0.01 2>&1)
> +
>    if [[ "$result" =~ ${m:0:50} ]]
>    then
>      continue
>    fi
> -  # Failed again, possibly the workload was too small so retry with something
> -  # longer.
> -  result=$(perf stat -M "$m" perf bench internals synthesize 2>&1)
> +
> +  # Failed, possibly the workload was too small so retry with something longer.
> +  result=$(perf stat -M "$m" $system_wide_flag -- perf bench internals synthesize 2>&1)
>    if [[ "$result" =~ ${m:0:50} ]]
>    then
>      continue
>    fi
>    echo "Metric '$m' not printed in:"
>    echo "$result"
> -  if [[ "$err" != "1" ]]
> -  then
> -    err=2
> -    if [[ "$result" =~ "FP_ARITH" || "$result" =~ "AMX" ]]
> -    then
> -      echo "Skip, not fail, for FP issues"
> -    elif [[ "$result" =~ "PMM" ]]
> -    then
> -      echo "Skip, not fail, for Optane memory issues"
> -    else
> -      err=1
> -    fi
> -  fi
> +  err=1
>  done
>
>  exit "$err"
> --
> 2.47.0.rc1.288.g06298d1525-goog
>

Re: [PATCH] perf test: Update all metrics test like metricgroups test

[Namhyung Kim]

On Fri, 18 Oct 2024 13:43:06 -0700, Namhyung Kim wrote:

> Like in the metricgroup tests, it should check the permission first and
> then skip relevant failures accordingly.
> 
> Also it needs to try again with the system wide flag properly.  On the
> second round, check if the result has the metric name because other
> failure cases are checked in the first round already.
> 
> [...]

Applied to perf-tools-next, thanks!

Best regards,
Namhyung