Re: [PATCH v2] perf test stat_bpf_counter.sh: Stabilize the test results

[Namhyung Kim <namhyung@kernel.org>]

Hello,

On Tue, Jun 25, 2024 at 11:20:01AM +0200, vmolnaro@redhat.com wrote:
> From: Veronika Molnarova <vmolnaro@redhat.com>
> 
> The test has been failing for some time when two separate runs of
> perf benchmarks are recorded for cycles events and their counts are
> compared, while once the recording was done with option --bpf-counters
> and once without it. It is expected that the count of the samples
> should be within a certain range, firstly the difference was set to be
> within 10%, which was then later raised to 20%. However, the test case
> keeps failing on certain architectures as recording the provided
> benchmark can produce completely different counts based on the
> current load of the system.
> 
> Sampling two separate runs on intel-eaglestream-spr-13 of "perf stat
> --no-big-num -e cycles -- perf bench sched messaging -g 1 -l 100 -t":
> 
>  Performance counter stats for 'perf bench sched messaging -g 1 -l 100 -t':
> 
>          396782898      cycles
> 
>        0.010051983 seconds time elapsed
> 
>        0.008664000 seconds user
>        0.097058000 seconds sys
> 
>  Performance counter stats for 'perf bench sched messaging -g 1 -l 100 -t':
> 
>         1431133032      cycles
> 
>        0.021803714 seconds time elapsed
> 
>        0.023377000 seconds user
>        0.349918000 seconds sys
> 
> , which is ranging from 400mil to 1400mil samples.
> 
> Instead of recording the cycles use instructions event, which provides
> more stable values. At the same time change the tested workload to one
> of the provided testing workloads by perf that is not based on a
> scheduler, which can provide another dependency on the current load.
> 
> Sampling instructions event with the new workload provide much more
> stable results on intel-eaglestream-spr-13 of "perf stat --no-big-num
> -e instructions -- perf test -w brstack":
> 
>  Performance counter stats for 'perf test -w brstack':
> 
>           64584494      instructions
> 
>        0.009173945 seconds time elapsed
> 
>        0.007262000 seconds user
>        0.002071000 seconds sys
> 
>  Performance counter stats for 'perf test -w brstack':
> 
>           64672669      instructions
> 
>        0.008888135 seconds time elapsed
> 
>        0.005018000 seconds user
>        0.004018000 seconds sys
> 
> Signed-off-by: Veronika Molnarova <vmolnaro@redhat.com>

Acked-by: Namhyung Kim <namhyung@kernel.org>

Thanks,
Namhyung


> ---
>  tools/perf/tests/shell/stat_bpf_counters.sh | 36 ++++++++++-----------
>  1 file changed, 18 insertions(+), 18 deletions(-)
> 
> diff --git a/tools/perf/tests/shell/stat_bpf_counters.sh b/tools/perf/tests/shell/stat_bpf_counters.sh
> index 61f8149d854e..f250b7d6f773 100755
> --- a/tools/perf/tests/shell/stat_bpf_counters.sh
> +++ b/tools/perf/tests/shell/stat_bpf_counters.sh
> @@ -4,7 +4,7 @@
>  
>  set -e
>  
> -workload="perf bench sched messaging -g 1 -l 100 -t"
> +workload="perf test -w brstack"
>  
>  # check whether $2 is within +/- 20% of $1
>  compare_number()
> @@ -25,15 +25,15 @@ compare_number()
>  
>  check_counts()
>  {
> -	base_cycles=$1
> -	bpf_cycles=$2
> +	base_instructions=$1
> +	bpf_instructions=$2
>  
> -	if [ "$base_cycles" = "<not" ]; then
> -		echo "Skipping: cycles event not counted"
> +	if [ "$base_instructions" = "<not" ]; then
> +		echo "Skipping: instructions event not counted"
>  		exit 2
>  	fi
> -	if [ "$bpf_cycles" = "<not" ]; then
> -		echo "Failed: cycles not counted with --bpf-counters"
> +	if [ "$bpf_instructions" = "<not" ]; then
> +		echo "Failed: instructions not counted with --bpf-counters"
>  		exit 1
>  	fi
>  }
> @@ -41,29 +41,29 @@ check_counts()
>  test_bpf_counters()
>  {
>  	printf "Testing --bpf-counters "
> -	base_cycles=$(perf stat --no-big-num -e cycles -- $workload 2>&1 | awk '/cycles/ {print $1}')
> -	bpf_cycles=$(perf stat --no-big-num --bpf-counters -e cycles -- $workload  2>&1 | awk '/cycles/ {print $1}')
> -	check_counts $base_cycles $bpf_cycles
> -	compare_number $base_cycles $bpf_cycles
> +	base_instructions=$(perf stat --no-big-num -e instructions -- $workload 2>&1 | awk '/instructions/ {print $1}')
> +	bpf_instructions=$(perf stat --no-big-num --bpf-counters -e instructions -- $workload  2>&1 | awk '/instructions/ {print $1}')
> +	check_counts $base_instructions $bpf_instructions
> +	compare_number $base_instructions $bpf_instructions
>  	echo "[Success]"
>  }
>  
>  test_bpf_modifier()
>  {
>  	printf "Testing bpf event modifier "
> -	stat_output=$(perf stat --no-big-num -e cycles/name=base_cycles/,cycles/name=bpf_cycles/b -- $workload 2>&1)
> -	base_cycles=$(echo "$stat_output"| awk '/base_cycles/ {print $1}')
> -	bpf_cycles=$(echo "$stat_output"| awk '/bpf_cycles/ {print $1}')
> -	check_counts $base_cycles $bpf_cycles
> -	compare_number $base_cycles $bpf_cycles
> +	stat_output=$(perf stat --no-big-num -e instructions/name=base_instructions/,instructions/name=bpf_instructions/b -- $workload 2>&1)
> +	base_instructions=$(echo "$stat_output"| awk '/base_instructions/ {print $1}')
> +	bpf_instructions=$(echo "$stat_output"| awk '/bpf_instructions/ {print $1}')
> +	check_counts $base_instructions $bpf_instructions
> +	compare_number $base_instructions $bpf_instructions
>  	echo "[Success]"
>  }
>  
>  # skip if --bpf-counters is not supported
> -if ! perf stat -e cycles --bpf-counters true > /dev/null 2>&1; then
> +if ! perf stat -e instructions --bpf-counters true > /dev/null 2>&1; then
>  	if [ "$1" = "-v" ]; then
>  		echo "Skipping: --bpf-counters not supported"
> -		perf --no-pager stat -e cycles --bpf-counters true || true
> +		perf --no-pager stat -e instructions --bpf-counters true || true
>  	fi
>  	exit 2
>  fi
> -- 
> 2.43.0
>