From: Thomas Richter <tmricht@linux.ibm.com>
To: Anubhav Shelat <ashelat@redhat.com>,
mpetlan@redhat.com, acme@kernel.org, namhyung@kernel.org,
irogers@google.com, linux-perf-users@vger.kernel.org
Cc: peterz@infradead.org, mingo@redhat.com, mark.rutland@arm.com,
alexander.shishkin@linux.intel.com, jolsa@kernel.org,
adrian.hunter@intel.com, kan.liang@linux.intel.com,
dapeng1.mi@linux.intel.com, james.clark@linaro.org
Subject: Re: [PATCH] Revert "perf test: Allow tolerance for leader sampling test"
Date: Fri, 24 Oct 2025 10:40:01 +0200 [thread overview]
Message-ID: <5b02372a-f0be-4d3a-a875-c5ea65f2bafe@linux.ibm.com> (raw)
In-Reply-To: <20251023132406.78359-2-ashelat@redhat.com>
On 10/23/25 15:24, Anubhav Shelat wrote:
> This reverts commit 1c5721ca89a1c8ae71082d3a102b39fd1ec0a205.
>
> The throttling bug has been fixed in 9734e25fbf5a perf: Fix the throttle
> logic for a group. So this commit can be reverted.
>
> Signed-off-by: Anubhav Shelat <ashelat@redhat.com>
> ---
> tools/perf/tests/shell/record.sh | 33 ++++++--------------------------
> 1 file changed, 6 insertions(+), 27 deletions(-)
>
> diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh
> index 0f5841c479e7..13e0d6ef66c9 100755
> --- a/tools/perf/tests/shell/record.sh
> +++ b/tools/perf/tests/shell/record.sh
> @@ -267,43 +267,22 @@ test_leader_sampling() {
> err=1
> return
> fi
> - perf script -i "${perfdata}" | grep brstack > $script_output
> - # Check if the two instruction counts are equal in each record.
> - # However, the throttling code doesn't consider event grouping. During throttling, only the
> - # leader is stopped, causing the slave's counts significantly higher. To temporarily solve this,
> - # let's set the tolerance rate to 80%.
> - # TODO: Revert the code for tolerance once the throttling mechanism is fixed.
> index=0
> - valid_counts=0
> - invalid_counts=0
> - tolerance_rate=0.8
> + perf script -i "${perfdata}" | grep brstack > "${script_output}"
> while IFS= read -r line
> do
> + # Check if the two instruction counts are equal in each record
> cycles=$(echo $line | awk '{for(i=1;i<=NF;i++) if($i=="cycles:") print $(i-1)}')
> if [ $(($index%2)) -ne 0 ] && [ ${cycles}x != ${prev_cycles}x ]
> then
> - invalid_counts=$(($invalid_counts+1))
> - else
> - valid_counts=$(($valid_counts+1))
> + echo "Leader sampling [Failed inconsistent cycles count]"
> + err=1
> + return
> fi
> index=$(($index+1))
> prev_cycles=$cycles
> done < "${script_output}"
> - total_counts=$(bc <<< "$invalid_counts+$valid_counts")
> - if (( $(bc <<< "$total_counts <= 0") ))
> - then
> - echo "Leader sampling [No sample generated]"
> - err=1
> - return
> - fi
> - isok=$(bc <<< "scale=2; if (($invalid_counts/$total_counts) < (1-$tolerance_rate)) { 0 } else { 1 };")
> - if [ $isok -eq 1 ]
> - then
> - echo "Leader sampling [Failed inconsistent cycles count]"
> - err=1
> - else
> - echo "Basic leader sampling test [Success]"
> - fi
> + echo "Basic leader sampling test [Success]"
> }
>
> test_topdown_leader_sampling() {
I disagree here. Reverting this patch cause the test case to fail very often on s390.
The test fails about every 2nd run, because the is one run-away value out of many.
Here is an example:
# ./perf record -e "{cycles,cycles}:Su" -- perf test -w brstack
[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.015 MB perf.data (74 samples) ]
[root@b83lp65 perf]# perf script | grep brstack
perf 136408 340637.903395: 1377000 cycles: 1171606 brstack_foo+0x1e (/root/linux/tools/perf/perf)
perf 136408 340637.903396: 1377000 cycles: 1171664 brstack_bench+0x24 (/root/linux/tools/perf/perf)
perf 136408 340637.903396: 1377000 cycles: 11716d4 brstack_bench+0x94 (/root/linux/tools/perf/perf)
perf 136408 340637.903397: 1377000 cycles: 11716d4 brstack_bench+0x94 (/root/linux/tools/perf/perf)
perf 136408 340637.903398: 1377000 cycles: 11716e8 brstack_bench+0xa8 (/root/linux/tools/perf/perf)
perf 136408 340637.903398: 1377000 cycles: 1171606 brstack_foo+0x1e (/root/linux/tools/perf/perf)
perf 136408 340637.903399: 1377000 cycles: 11715cc brstack_bar+0x34 (/root/linux/tools/perf/perf)
perf 136408 340637.910844: 1377000 cycles: 11716e4 brstack_bench+0xa4 (/root/linux/tools/perf/perf)
perf 136408 340637.910844: 39843371 cycles: 11716e4 brstack_bench+0xa4 (/root/linux/tools/perf/perf)
perf 136408 340637.910845: 1377000 cycles: 1171632 brstack_foo+0x4a (/root/linux/tools/perf/perf)
perf 136408 340637.910846: 1377000 cycles: 1171692 brstack_bench+0x52 (/root/linux/tools/perf/perf)
perf 136408 340637.910847: 1377000 cycles: 11716ee brstack_bench+0xae (/root/linux/tools/perf/perf)
perf 136408 340637.910847: 1377000 cycles: 11715cc brstack_bar+0x34 (/root/linux/tools/perf/perf)
perf 136408 340637.910848: 1377000 cycles: 1171598 brstack_bar+0x0 (/root/linux/tools/perf/perf)
perf 136408 340637.910848: 1377000 cycles: 11715e8 brstack_foo+0x0 (/root/linux/tools/perf/perf)
perf 136408 340637.910849: 1377000 cycles: 11717ae brstack+0x86 (/root/linux/tools/perf/perf)
perf 136408 340637.910850: 1377000 cycles: 11715cc brstack_bar+0x34 (/root/linux/tools/perf/perf)
perf 136408 340637.910850: 1377000 cycles: 11716ee brstack_bench+0xae (/root/linux/tools/perf/perf)
perf 136408 340637.910851: 1377000 cycles: 11716ee brstack_bench+0xae (/root/linux/tools/perf/perf)
perf 136408 340637.910851: 1377000 cycles: 117159e brstack_bar+0x6 (/root/linux/tools/perf/perf)
perf 136408 340637.910852: 1377000 cycles: 1171598 brstack_bar+0x0 (/root/linux/tools/perf/perf)
perf 136408 340637.910853: 1377000 cycles: 117179e brstack+0x76 (/root/linux/tools/perf/perf)
perf 136408 340637.910853: 1377000 cycles: 1171606 brstack_foo+0x1e (/root/linux/tools/perf/perf)
perf 136408 340637.910854: 1377000 cycles: 11716d4 brstack_bench+0x94 (/root/linux/tools/perf/perf)
perf 136408 340637.910855: 1377000 cycles: 1171612 brstack_foo+0x2a (/root/linux/tools/perf/perf)
perf 136408 340637.910855: 1377000 cycles: 11715cc brstack_bar+0x34 (/root/linux/tools/perf/perf)
perf 136408 340637.910856: 1377000 cycles: 1171598 brstack_bar+0x0 (/root/linux/tools/perf/perf)
perf 136408 340637.910856: 1377000 cycles: 11715cc brstack_bar+0x34 (/root/linux/tools/perf/perf)
perf 136408 340637.910857: 1377000 cycles: 1171632 brstack_foo+0x4a (/root/linux/tools/perf/perf)
.... many more lines with identical cycles value.
I have contacted our hardware/firmware team, but have not gotten a response back.
I still think this has to do with s390 LPAR running under hyperviser control and I do not know what
happens when the hipervisor kicks in.
I agree with James Clark that this should be handled transperently by the hipervisor, that means
stopping the LPAR should stop the CPU measurement unit, before giving control to a different lpar.
But what happens when the hipervisor just kicks in and returns to the same LPAR again? Or does
some admin work on behalf of this LPAR. As long as I can not answer this question, I would like
to keep some ratio to handle run-away values.
As said before, this happens in roughly 50% of the runs...
Here is a run where the test succeeds without a run-away value:
# ./perf record -e "{cycles,cycles}:Su" -- perf test -w brstack
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.015 MB perf.data (70 samples) ]
# perf script | grep brstack
perf 136455 341212.430466: 1377000 cycles: 117159e brstack_bar+0x6 (/root/linux/tools/perf/perf)
perf 136455 341212.430467: 1377000 cycles: 11715cc brstack_bar+0x34 (/root/linux/tools/perf/perf)
perf 136455 341212.430468: 1377000 cycles: 1171612 brstack_foo+0x2a (/root/linux/tools/perf/perf)
perf 136455 341212.430468: 1377000 cycles: 1171656 brstack_bench+0x16 (/root/linux/tools/perf/perf)
perf 136455 341212.430469: 1377000 cycles: 11716ee brstack_bench+0xae (/root/linux/tools/perf/perf)
perf 136455 341212.430469: 1377000 cycles: 11715cc brstack_bar+0x34 (/root/linux/tools/perf/perf)
perf 136455 341212.430470: 1377000 cycles: 11715e8 brstack_foo+0x0 (/root/linux/tools/perf/perf)
perf 136455 341212.430471: 1377000 cycles: 11716c6 brstack_bench+0x86 (/root/linux/tools/perf/perf)
perf 136455 341212.430471: 1377000 cycles: 1171622 brstack_foo+0x3a (/root/linux/tools/perf/perf)
perf 136455 341212.430472: 1377000 cycles: 11715cc brstack_bar+0x34 (/root/linux/tools/perf/perf)
perf 136455 341212.430472: 1377000 cycles: 1171692 brstack_bench+0x52 (/root/linux/tools/perf/perf)
perf 136455 341212.430473: 1377000 cycles: 117170c brstack_bench+0xcc (/root/linux/tools/perf/perf)
perf 136455 341212.430474: 1377000 cycles: 11716e4 brstack_bench+0xa4 (/root/linux/tools/perf/perf)
perf 136455 341212.430474: 1377000 cycles: 117169a brstack_bench+0x5a (/root/linux/tools/perf/perf)
perf 136455 341212.430475: 1377000 cycles: 11715cc brstack_bar+0x34 (/root/linux/tools/perf/perf)
perf 136455 341212.430475: 1377000 cycles: 11715cc brstack_bar+0x34 (/root/linux/tools/perf/perf)
perf 136455 341212.430476: 1377000 cycles: 11716e4 brstack_bench+0xa4 (/root/linux/tools/perf/perf)
perf 136455 341212.430476: 1377000 cycles: 11715cc brstack_bar+0x34 (/root/linux/tools/perf/perf)
perf 136455 341212.430477: 1377000 cycles: 11715e8 brstack_foo+0x0 (/root/linux/tools/perf/perf)
perf 136455 341212.430478: 1377000 cycles: 117170c brstack_bench+0xcc (/root/linux/tools/perf/perf)
perf 136455 341212.430478: 1377000 cycles: 117170c brstack_bench+0xcc (/root/linux/tools/perf/perf)
perf 136455 341212.430479: 1377000 cycles: 11716ee brstack_bench+0xae (/root/linux/tools/perf/perf)
perf 136455 341212.430479: 1377000 cycles: 1171598 brstack_bar+0x0 (/root/linux/tools/perf/perf)
perf 136455 341212.430480: 1377000 cycles: 11715cc brstack_bar+0x34 (/root/linux/tools/perf/perf)
perf 136455 341212.430481: 1377000 cycles: 11715fc brstack_foo+0x14 (/root/linux/tools/perf/perf)
perf 136455 341212.430481: 1377000 cycles: 11715cc brstack_bar+0x34 (/root/linux/tools/perf/perf)
perf 136455 341212.430482: 1377000 cycles: 11716ee brstack_bench+0xae (/root/linux/tools/perf/perf)
perf 136455 341212.430482: 1377000 cycles: 1171664 brstack_bench+0x24 (/root/linux/tools/perf/perf)
perf 136455 341212.430483: 1377000 cycles: 117170c brstack_bench+0xcc (/root/linux/tools/perf/perf)
perf 136455 341212.430484: 1377000 cycles: 117170c brstack_bench+0xcc (/root/linux/tools/perf/perf)
perf 136455 341212.430484: 1377000 cycles: 11715cc brstack_bar+0x34 (/root/linux/tools/perf/perf)
#
So please let this code in otherwise we need an s390 specific check for success.
Thanks a lot
--
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Wolfgang Wendt
Geschäftsführung: David Faller
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
next prev parent reply other threads:[~2025-10-24 8:40 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-10-23 13:24 [PATCH] Revert "perf test: Allow tolerance for leader sampling test" Anubhav Shelat
2025-10-24 8:40 ` Thomas Richter [this message]
2025-10-24 17:21 ` Anubhav Shelat
2025-10-27 10:27 ` Thomas Richter
2025-10-28 11:30 ` James Clark
2025-10-28 12:55 ` Thomas Richter
2025-10-28 15:23 ` James Clark
2025-10-29 7:37 ` Thomas Richter
2025-10-29 9:25 ` James Clark
2025-11-11 11:22 ` Thomas Richter
2025-11-11 14:03 ` James Clark
2025-11-12 10:47 ` Thomas Richter
2025-11-13 11:58 ` James Clark
2025-10-30 13:52 ` Anubhav Shelat
2025-10-30 14:19 ` James Clark
2025-10-30 15:22 ` Anubhav Shelat
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5b02372a-f0be-4d3a-a875-c5ea65f2bafe@linux.ibm.com \
--to=tmricht@linux.ibm.com \
--cc=acme@kernel.org \
--cc=adrian.hunter@intel.com \
--cc=alexander.shishkin@linux.intel.com \
--cc=ashelat@redhat.com \
--cc=dapeng1.mi@linux.intel.com \
--cc=irogers@google.com \
--cc=james.clark@linaro.org \
--cc=jolsa@kernel.org \
--cc=kan.liang@linux.intel.com \
--cc=linux-perf-users@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mingo@redhat.com \
--cc=mpetlan@redhat.com \
--cc=namhyung@kernel.org \
--cc=peterz@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).