From: Anubhav Shelat <ashelat@redhat.com>
To: Thomas Richter <tmricht@linux.ibm.com>
Cc: James Clark <james.clark@linaro.org>,
mpetlan@redhat.com, acme@kernel.org, namhyung@kernel.org,
irogers@google.com, linux-perf-users@vger.kernel.org,
peterz@infradead.org, mingo@redhat.com, mark.rutland@arm.com,
alexander.shishkin@linux.intel.com, jolsa@kernel.org,
adrian.hunter@intel.com, kan.liang@linux.intel.com,
dapeng1.mi@linux.intel.com
Subject: Re: [PATCH] Revert "perf test: Allow tolerance for leader sampling test"
Date: Thu, 30 Oct 2025 13:52:48 +0000 [thread overview]
Message-ID: <CA+G8DhKTpZLzRbZvGiS6YZxOXecEcrtOUcNdHOpjDM9Cz0qzxg@mail.gmail.com> (raw)
In-Reply-To: <2e756e75-7dc9-4838-8651-ca1a0f056966@linux.ibm.com>
> On 10/28/25 12:30, James Clark wrote:
> > You mentioned on the other thread that it's failing "differently", can you expand on that? I'm wondering why you sent the revert patch then?
The issue was that for every (leader, sibling) event pair there was an
18 cycle difference between the sibling and the leader. Initially I
thought that this was fixed by the group throttle fixing patch
(9734e25fbf5a), but I just got a VM running 6.12.0-145.el10.aarch64
which is up to date with v6.16 and it has that bug.
I want to revert the patch since the bug it exists for has been fixed,
and as you mentioned it could be silencing other bugs. Since I'm able
to reprouce Thomas's issue on an arm machine it seems like this is not
a hardware issue inherent to s390x and there's no reason for this
patch to be in the kernel anymore.
> > As I mentioned before I'm not seeing any issues. Can you share the kernel version that you tested on and your kernel config? And can you share the same outputs that I asked Thomas for below please.
For the system experiencing the 18 cycle difference bug:
Linux ampere-mtsnow-altramax-03-vm-13.lab.eng.rdu2.redhat.com
6.12.0-145.el10.aarch64 #1 SMP PREEMPT_DYNAMIC Fri Oct 24 12:39:51 UTC
2025 aarch64 GNU/Linux
[root@ampere-mtsnow-altramax-03-vm-13 ~]# perf script | head -30
perf 91202 168884.094165: 146 cycles:
ffff9e94ba40 _start+0x0 (/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094165: 164 cycles:
ffff9e94ba40 _start+0x0 (/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094188: 74 cycles:
ffffc6fe45649728 [unknown] ([unknown])
perf 91202 168884.094188: 74 cycles:
ffffc6fe45649728 [unknown] ([unknown])
perf 91202 168884.094213: 134 cycles:
ffff9e94ba40 _start+0x0 (/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094213: 152 cycles:
ffff9e94ba40 _start+0x0 (/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094233: 136 cycles:
ffff9e9473cc _dl_start+0x0 (/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094233: 154 cycles:
ffff9e9473cc _dl_start+0x0 (/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094252: 293 cycles:
ffffc6fe45648e18 [unknown] ([unknown])
perf 91202 168884.094252: 293 cycles:
ffffc6fe45648e18 [unknown] ([unknown])
perf 91202 168884.094273: 275 cycles:
ffff9e9473fc _dl_start+0x30 (/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094273: 293 cycles:
ffff9e9473fc _dl_start+0x30 (/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094291: 56 cycles:
ffffc6fe45648e18 [unknown] ([unknown])
perf 91202 168884.094291: 56 cycles:
ffffc6fe45648e18 [unknown] ([unknown])
perf 91202 168884.094312: 1138 cycles:
ffff9e947400 _dl_start+0x34 (/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094312: 1156 cycles:
ffff9e947400 _dl_start+0x34 (/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094336: 414 cycles:
ffff9e947454 _dl_start+0x88 (/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094336: 432 cycles:
ffff9e947454 _dl_start+0x88 (/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094355: 692 cycles:
ffff9e947444 _dl_start+0x78 (/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094355: 710 cycles:
ffff9e947444 _dl_start+0x78 (/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094379: 1328 cycles:
ffffc6fe45648e18 [unknown] ([unknown])
perf 91202 168884.094379: 1328 cycles:
ffffc6fe45648e18 [unknown] ([unknown])
perf 91202 168884.094403: 3871 cycles:
ffff9e941550 __GI___tunables_init+0xd0
(/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094403: 3889 cycles:
ffff9e941550 __GI___tunables_init+0xd0
(/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094425: 7787 cycles:
ffff9e94153c __GI___tunables_init+0xbc
(/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094425: 7805 cycles:
ffff9e94153c __GI___tunables_init+0xbc
(/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094464: 17320 cycles:
ffff9e94c794 __set_vma_name+0x54 (/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094464: 17338 cycles:
ffff9e94c794 __set_vma_name+0x54 (/usr/lib/ld-linux-aarch64.so.1)
perf 91202 168884.094606: 38889 cycles:
ffffc6fe45649bcc [unknown] ([unknown])
perf 91202 168884.094606: 38889 cycles:
ffffc6fe45649bcc [unknown] ([unknown])
For the first mismatching sample:
[root@ampere-mtsnow-altramax-03-vm-13 ~]# perf report -D | grep
ffff9e94ba40 -A 8
168884094165520 0x6a0 [0x68]: PERF_RECORD_SAMPLE(IP, 0x2):
91202/91202: 0xffff9e94ba40 period: 1 addr: 0
... sample_read:
.... group nr 2
..... id 000000000000002c, value 0000000000000092, lost 0
..... id 0000000000000034, value 00000000000000a4, lost 0
... thread: perf:91202
...... dso: /usr/lib/ld-linux-aarch64.so.1
... thread: perf:91202
...... dso: /usr/lib/ld-linux-aarch64.so.1
--
168884094213520 0x770 [0x68]: PERF_RECORD_SAMPLE(IP, 0x2):
91202/91202: 0xffff9e94ba40 period: 1 addr: 0
... sample_read:
.... group nr 2
..... id 000000000000002c, value 0000000000000162, lost 0
..... id 0000000000000034, value 0000000000000186, lost 0
... thread: perf:91202
...... dso: /usr/lib/ld-linux-aarch64.so.1
... thread: perf:91202
...... dso: /usr/lib/ld-linux-aarch64.so.1
For the first matching sample:
[root@ampere-mtsnow-altramax-03-vm-13 ~]# perf report -D | grep
ffffc6fe45649728 -A 8
168884094188520 0x708 [0x68]: PERF_RECORD_SAMPLE(IP, 0x1):
91202/91202: 0xffffc6fe45649728 period: 1 addr: 0
... sample_read:
.... group nr 2
..... id 000000000000002c, value 00000000000000dc, lost 0
..... id 0000000000000034, value 00000000000000ee, lost 0
... thread: perf:91202
...... dso: <not found>
... thread: perf:91202
...... dso: <not found>
I also was able to get a bare metal arm machine with the same kernel
version: Linux ampere-mtsnow-altra-01.khw.eng.rdu2.dc.redhat.com
6.12.0-145.el10.aarch64 #1 SMP PREEMPT_DYNAMIC Fri Oct 24 12:39:51 UTC
2025 aarch64 GNU/Linux. On this machine I wasn't able to reproduce any
issue. I'm trying to get another vm to reproduce Thomas' issue again.
Anubhav
next prev parent reply other threads:[~2025-10-30 13:53 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-10-23 13:24 [PATCH] Revert "perf test: Allow tolerance for leader sampling test" Anubhav Shelat
2025-10-24 8:40 ` Thomas Richter
2025-10-24 17:21 ` Anubhav Shelat
2025-10-27 10:27 ` Thomas Richter
2025-10-28 11:30 ` James Clark
2025-10-28 12:55 ` Thomas Richter
2025-10-28 15:23 ` James Clark
2025-10-29 7:37 ` Thomas Richter
2025-10-29 9:25 ` James Clark
2025-11-11 11:22 ` Thomas Richter
2025-11-11 14:03 ` James Clark
2025-11-12 10:47 ` Thomas Richter
2025-11-13 11:58 ` James Clark
2025-10-30 13:52 ` Anubhav Shelat [this message]
2025-10-30 14:19 ` James Clark
2025-10-30 15:22 ` Anubhav Shelat
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CA+G8DhKTpZLzRbZvGiS6YZxOXecEcrtOUcNdHOpjDM9Cz0qzxg@mail.gmail.com \
--to=ashelat@redhat.com \
--cc=acme@kernel.org \
--cc=adrian.hunter@intel.com \
--cc=alexander.shishkin@linux.intel.com \
--cc=dapeng1.mi@linux.intel.com \
--cc=irogers@google.com \
--cc=james.clark@linaro.org \
--cc=jolsa@kernel.org \
--cc=kan.liang@linux.intel.com \
--cc=linux-perf-users@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mingo@redhat.com \
--cc=mpetlan@redhat.com \
--cc=namhyung@kernel.org \
--cc=peterz@infradead.org \
--cc=tmricht@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).