From: James Clark <james.clark@linaro.org>
To: Anubhav Shelat <ashelat@redhat.com>,
Thomas Richter <tmricht@linux.ibm.com>
Cc: mpetlan@redhat.com, acme@kernel.org, namhyung@kernel.org,
irogers@google.com, linux-perf-users@vger.kernel.org,
peterz@infradead.org, mingo@redhat.com, mark.rutland@arm.com,
alexander.shishkin@linux.intel.com, jolsa@kernel.org,
adrian.hunter@intel.com, kan.liang@linux.intel.com,
dapeng1.mi@linux.intel.com
Subject: Re: [PATCH] Revert "perf test: Allow tolerance for leader sampling test"
Date: Thu, 30 Oct 2025 14:19:18 +0000 [thread overview]
Message-ID: <6b1d742e-ae54-4c24-a4c0-11565729cb90@linaro.org> (raw)
In-Reply-To: <CA+G8DhKTpZLzRbZvGiS6YZxOXecEcrtOUcNdHOpjDM9Cz0qzxg@mail.gmail.com>
On 30/10/2025 1:52 pm, Anubhav Shelat wrote:
>> On 10/28/25 12:30, James Clark wrote:
>>> You mentioned on the other thread that it's failing "differently", can you expand on that? I'm wondering why you sent the revert patch then?
> The issue was that for every (leader, sibling) event pair there was an
> 18 cycle difference between the sibling and the leader. Initially I
Isn't that the same throttling bug? A difference in cycle counts.
> thought that this was fixed by the group throttle fixing patch
> (9734e25fbf5a), but I just got a VM running 6.12.0-145.el10.aarch64
> which is up to date with v6.16 and it has that bug.
I'm not really sure what "v6.12 up to date with v6.16" means.
> I want to revert the patch since the bug it exists for has been fixed,
How do you know it's fixed if you are still seeing differences in cycle
counts? That seems like not fixed to me. (Although I'm not able to
reproduce it).
> and as you mentioned it could be silencing other bugs. Since I'm able
> to reprouce Thomas's issue on an arm machine it seems like this is not
> a hardware issue inherent to s390x and there's no reason for this
> patch to be in the kernel anymore.
>
>>> As I mentioned before I'm not seeing any issues. Can you share the kernel version that you tested on and your kernel config? And can you share the same outputs that I asked Thomas for below please.
> For the system experiencing the 18 cycle difference bug:
> Linux ampere-mtsnow-altramax-03-vm-13.lab.eng.rdu2.redhat.com
> 6.12.0-145.el10.aarch64 #1 SMP PREEMPT_DYNAMIC Fri Oct 24 12:39:51 UTC
> 2025 aarch64 GNU/Linux
Right, but this is an old kernel. Does the issue reproduce for you on
the latest kernel? Can we just ignore old kernels from this point on, I
think that's what got confusing in the last thread. Let's simplify it
and say it has to be tested on the latest kernel on bare metal. If it
has to be in a VM because that's the only place it reproduces then both
the host and guest have to be the same and latest version. Otherwise
we're just putting effort into some bug that's already been fixed.
> [root@ampere-mtsnow-altramax-03-vm-13 ~]# perf script | head -30
> perf 91202 168884.094165: 146 cycles:
> ffff9e94ba40 _start+0x0 (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094165: 164 cycles:
> ffff9e94ba40 _start+0x0 (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094188: 74 cycles:
> ffffc6fe45649728 [unknown] ([unknown])
> perf 91202 168884.094188: 74 cycles:
> ffffc6fe45649728 [unknown] ([unknown])
> perf 91202 168884.094213: 134 cycles:
> ffff9e94ba40 _start+0x0 (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094213: 152 cycles:
> ffff9e94ba40 _start+0x0 (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094233: 136 cycles:
> ffff9e9473cc _dl_start+0x0 (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094233: 154 cycles:
> ffff9e9473cc _dl_start+0x0 (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094252: 293 cycles:
> ffffc6fe45648e18 [unknown] ([unknown])
> perf 91202 168884.094252: 293 cycles:
> ffffc6fe45648e18 [unknown] ([unknown])
> perf 91202 168884.094273: 275 cycles:
> ffff9e9473fc _dl_start+0x30 (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094273: 293 cycles:
> ffff9e9473fc _dl_start+0x30 (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094291: 56 cycles:
> ffffc6fe45648e18 [unknown] ([unknown])
> perf 91202 168884.094291: 56 cycles:
> ffffc6fe45648e18 [unknown] ([unknown])
> perf 91202 168884.094312: 1138 cycles:
> ffff9e947400 _dl_start+0x34 (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094312: 1156 cycles:
> ffff9e947400 _dl_start+0x34 (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094336: 414 cycles:
> ffff9e947454 _dl_start+0x88 (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094336: 432 cycles:
> ffff9e947454 _dl_start+0x88 (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094355: 692 cycles:
> ffff9e947444 _dl_start+0x78 (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094355: 710 cycles:
> ffff9e947444 _dl_start+0x78 (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094379: 1328 cycles:
> ffffc6fe45648e18 [unknown] ([unknown])
> perf 91202 168884.094379: 1328 cycles:
> ffffc6fe45648e18 [unknown] ([unknown])
> perf 91202 168884.094403: 3871 cycles:
> ffff9e941550 __GI___tunables_init+0xd0
> (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094403: 3889 cycles:
> ffff9e941550 __GI___tunables_init+0xd0
> (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094425: 7787 cycles:
> ffff9e94153c __GI___tunables_init+0xbc
> (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094425: 7805 cycles:
> ffff9e94153c __GI___tunables_init+0xbc
> (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094464: 17320 cycles:
> ffff9e94c794 __set_vma_name+0x54 (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094464: 17338 cycles:
> ffff9e94c794 __set_vma_name+0x54 (/usr/lib/ld-linux-aarch64.so.1)
> perf 91202 168884.094606: 38889 cycles:
> ffffc6fe45649bcc [unknown] ([unknown])
> perf 91202 168884.094606: 38889 cycles:
> ffffc6fe45649bcc [unknown] ([unknown])
>
> For the first mismatching sample:
> [root@ampere-mtsnow-altramax-03-vm-13 ~]# perf report -D | grep
> ffff9e94ba40 -A 8
> 168884094165520 0x6a0 [0x68]: PERF_RECORD_SAMPLE(IP, 0x2):
> 91202/91202: 0xffff9e94ba40 period: 1 addr: 0
> ... sample_read:
> .... group nr 2
> ..... id 000000000000002c, value 0000000000000092, lost 0
> ..... id 0000000000000034, value 00000000000000a4, lost 0
> ... thread: perf:91202
> ...... dso: /usr/lib/ld-linux-aarch64.so.1
> ... thread: perf:91202
> ...... dso: /usr/lib/ld-linux-aarch64.so.1
> --
> 168884094213520 0x770 [0x68]: PERF_RECORD_SAMPLE(IP, 0x2):
> 91202/91202: 0xffff9e94ba40 period: 1 addr: 0
> ... sample_read:
> .... group nr 2
> ..... id 000000000000002c, value 0000000000000162, lost 0
> ..... id 0000000000000034, value 0000000000000186, lost 0
> ... thread: perf:91202
> ...... dso: /usr/lib/ld-linux-aarch64.so.1
> ... thread: perf:91202
> ...... dso: /usr/lib/ld-linux-aarch64.so.1
>
> For the first matching sample:
> [root@ampere-mtsnow-altramax-03-vm-13 ~]# perf report -D | grep
> ffffc6fe45649728 -A 8
> 168884094188520 0x708 [0x68]: PERF_RECORD_SAMPLE(IP, 0x1):
> 91202/91202: 0xffffc6fe45649728 period: 1 addr: 0
> ... sample_read:
> .... group nr 2
> ..... id 000000000000002c, value 00000000000000dc, lost 0
> ..... id 0000000000000034, value 00000000000000ee, lost 0
> ... thread: perf:91202
> ...... dso: <not found>
> ... thread: perf:91202
> ...... dso: <not found>
>
>
> I also was able to get a bare metal arm machine with the same kernel
> version: Linux ampere-mtsnow-altra-01.khw.eng.rdu2.dc.redhat.com
> 6.12.0-145.el10.aarch64 #1 SMP PREEMPT_DYNAMIC Fri Oct 24 12:39:51 UTC
> 2025 aarch64 GNU/Linux. On this machine I wasn't able to reproduce any
> issue. I'm trying to get another vm to reproduce Thomas' issue again.
>
> Anubhav
>
next prev parent reply other threads:[~2025-10-30 14:19 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-10-23 13:24 [PATCH] Revert "perf test: Allow tolerance for leader sampling test" Anubhav Shelat
2025-10-24 8:40 ` Thomas Richter
2025-10-24 17:21 ` Anubhav Shelat
2025-10-27 10:27 ` Thomas Richter
2025-10-28 11:30 ` James Clark
2025-10-28 12:55 ` Thomas Richter
2025-10-28 15:23 ` James Clark
2025-10-29 7:37 ` Thomas Richter
2025-10-29 9:25 ` James Clark
2025-11-11 11:22 ` Thomas Richter
2025-11-11 14:03 ` James Clark
2025-11-12 10:47 ` Thomas Richter
2025-11-13 11:58 ` James Clark
2025-10-30 13:52 ` Anubhav Shelat
2025-10-30 14:19 ` James Clark [this message]
2025-10-30 15:22 ` Anubhav Shelat
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6b1d742e-ae54-4c24-a4c0-11565729cb90@linaro.org \
--to=james.clark@linaro.org \
--cc=acme@kernel.org \
--cc=adrian.hunter@intel.com \
--cc=alexander.shishkin@linux.intel.com \
--cc=ashelat@redhat.com \
--cc=dapeng1.mi@linux.intel.com \
--cc=irogers@google.com \
--cc=jolsa@kernel.org \
--cc=kan.liang@linux.intel.com \
--cc=linux-perf-users@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mingo@redhat.com \
--cc=mpetlan@redhat.com \
--cc=namhyung@kernel.org \
--cc=peterz@infradead.org \
--cc=tmricht@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).