IBS perf test failures on 9950x3d

IBS perf test failures on 9950x3d

[Arnaldo Carvalho de Melo]

Ravi,

	Does this look familiar?

⬢ [acme@toolbx perf-tools-next]$ git log --oneline -10
8cdf00b843ea3ca0 (HEAD -> perf-tools-next) perf record: Fix a asan runtime error in util/maps.c
208c0e16834472bb perf record: Add 8-byte aligned event type PERF_RECORD_COMPRESSED2
bcfab08db7fb38bf perf intel-tpebs: Filter non-workload samples
1c5721ca89a1c8ae (x1/perf-tools-next, x1/HEAD, five/perf-tools-next, five/HEAD) perf test: Allow tolerance for leader sampling test
cb422594d62066a5 perf test: Add stat uniquifying test
137359b7895f61cd perf parse-events: Use wildcard processing to set an event to merge into
7d45f402d3117e0b perf evlist: Make uniquifying counter names consistent
ef60b8f5724da364 perf trace: Support --summary-mode=cgroup
39922dc53c268429 perf report: Add 'tgid' sort key
b922881712743035 perf test: Update sysfs path for core PMU caps
⬢ [acme@toolbx perf-tools-next]$

I'll push my local branch soon, but I think this predates it.

If it needs a specific kernel, then the test needs to cope with that,
etc.

- Arnaldo


root@number:~# perf test -vvvvv 115
115: AMD IBS sample period:
--- start ---
test child forked, pid 1949350
Using CPUID AuthenticAMD-26-44-0

IBS config tests:
-----------------
Fetch PMU tests:
mprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission denied0xffff            : Fail (nr samples: 0)
mprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission denied0x1000            : Fail (nr samples: 0)
mprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission denied0xff              : Ok   (nr samples: 5)
mprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission denied0x1               : Ok   (nr samples: 101)
0x0               : Ok  
0x10000           : Ok  
Op PMU tests:
0x0               : Ok  
0x1               : Fail
0x8               : Fail
mprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission denied0x9               : Ok   (nr samples: 101)
mprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission denied0xf               : Ok   (nr samples: 101)
mprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission denied0x1000            : Fail (nr samples: 0)
mprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission denied0xffff            : Fail (nr samples: 0)
0x10000           : Ok  
mprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission denied0x100000          : Fail (nr samples: 0)
mprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission denied0xf00000          : Fail (nr samples: 0)
mprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission denied0xf0ffff          : Fail (nr samples: 0)
mprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission denied0x1f0ffff         : Fail (nr samples: 0)
mprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission deniedmprotect() failed. Permission denied0x7f0ffff         : Fail (nr samples: 0)
0x8f0ffff         : Ok  
0x17f0ffff        : Ok  

IBS sample period constraint tests:
-----------------------------------
Fetch PMU test:
freq 0, sample_freq         0: Ok  
freq 0, sample_freq         1: Fail
freq 0, sample_freq        15: Fail
mprotect() failed. Permission deniedfreq 0, sample_freq        16: Ok   (nr samples: 202)
mprotect() failed. Permission deniedfreq 0, sample_freq        17: Ok   (nr samples: 101)
mprotect() failed. Permission deniedfreq 0, sample_freq       143: Ok   (nr samples: 101)
mprotect() failed. Permission deniedfreq 0, sample_freq       144: Ok   (nr samples: 101)
mprotect() failed. Permission deniedfreq 0, sample_freq       145: Ok   (nr samples: 101)
mprotect() failed. Permission deniedfreq 0, sample_freq      1234: Ok   (nr samples: 4)
mprotect() failed. Permission deniedfreq 0, sample_freq      4103: Fail (nr samples: 0)
mprotect() failed. Permission deniedfreq 0, sample_freq     65520: Fail (nr samples: 0)
mprotect() failed. Permission deniedfreq 0, sample_freq     65535: Fail (nr samples: 0)
mprotect() failed. Permission deniedfreq 0, sample_freq     65552: Fail (nr samples: 0)
mprotect() failed. Permission deniedfreq 0, sample_freq   8388607: Fail (nr samples: 0)
mprotect() failed. Permission deniedfreq 0, sample_freq 268435455: Fail (nr samples: 0)
freq 1, sample_freq         0: Ok  
mprotect() failed. Permission deniedfreq 1, sample_freq         1: Ok   (nr samples: 3)
mprotect() failed. Permission deniedfreq 1, sample_freq        15: Ok   (nr samples: 3)
mprotect() failed. Permission deniedfreq 1, sample_freq        16: Ok   (nr samples: 3)
mprotect() failed. Permission deniedfreq 1, sample_freq        17: Ok   (nr samples: 3)
mprotect() failed. Permission deniedfreq 1, sample_freq       143: Ok   (nr samples: 3)
mprotect() failed. Permission deniedfreq 1, sample_freq       144: Ok   (nr samples: 3)
mprotect() failed. Permission deniedfreq 1, sample_freq       145: Ok   (nr samples: 3)
mprotect() failed. Permission deniedfreq 1, sample_freq      1234: Ok   (nr samples: 4)
mprotect() failed. Permission deniedfreq 1, sample_freq      4103: Ok   (nr samples: 4)
mprotect() failed. Permission deniedfreq 1, sample_freq     65520: Ok   (nr samples: 8)
mprotect() failed. Permission deniedfreq 1, sample_freq     65535: Ok   (nr samples: 8)
mprotect() failed. Permission deniedfreq 1, sample_freq     65552: Ok   (nr samples: 8)
freq 1, sample_freq   8388607: Ok  
Op PMU test:
freq 0, sample_freq         0: Ok  
freq 0, sample_freq         1: Fail
freq 0, sample_freq        15: Fail
freq 0, sample_freq        16: Fail
freq 0, sample_freq        17: Fail
freq 0, sample_freq       143: Fail
mprotect() failed. Permission deniedfreq 0, sample_freq       144: Ok   (nr samples: 101)
mprotect() failed. Permission deniedfreq 0, sample_freq       145: Ok   (nr samples: 101)
mprotect() failed. Permission deniedfreq 0, sample_freq      1234: Ok   (nr samples: 101)
mprotect() failed. Permission deniedfreq 0, sample_freq      4103: Ok   (nr samples: 6)
mprotect() failed. Permission deniedfreq 0, sample_freq     65520: Fail (nr samples: 0)
mprotect() failed. Permission deniedfreq 0, sample_freq     65535: Fail (nr samples: 0)
mprotect() failed. Permission deniedfreq 0, sample_freq     65552: Fail (nr samples: 0)
mprotect() failed. Permission deniedfreq 0, sample_freq   8388607: Fail (nr samples: 0)
mprotect() failed. Permission deniedfreq 0, sample_freq 268435455: Fail (nr samples: 0)
freq 1, sample_freq         0: Ok  
mprotect() failed. Permission deniedfreq 1, sample_freq         1: Fail (nr samples: 3)
mprotect() failed. Permission deniedfreq 1, sample_freq        15: Fail (nr samples: 3)
mprotect() failed. Permission deniedfreq 1, sample_freq        16: Fail (nr samples: 3)
mprotect() failed. Permission deniedfreq 1, sample_freq        17: Fail (nr samples: 3)
mprotect() failed. Permission deniedfreq 1, sample_freq       143: Fail (nr samples: 4)
mprotect() failed. Permission deniedfreq 1, sample_freq       144: Fail (nr samples: 4)
mprotect() failed. Permission deniedfreq 1, sample_freq       145: Fail (nr samples: 4)
mprotect() failed. Permission deniedfreq 1, sample_freq      1234: Fail (nr samples: 4)
mprotect() failed. Permission deniedfreq 1, sample_freq      4103: Fail (nr samples: 5)
mprotect() failed. Permission deniedfreq 1, sample_freq     65520: Fail (nr samples: 9)
mprotect() failed. Permission deniedfreq 1, sample_freq     65535: Fail (nr samples: 9)
mprotect() failed. Permission deniedfreq 1, sample_freq     65552: Fail (nr samples: 9)
freq 1, sample_freq   8388607: Ok  

IBS ioctl() tests:
------------------
Fetch PMU tests
ioctl(period = 0x0      ): Ok
ioctl(period = 0x1      ): Fail
ioctl(period = 0xf      ): Fail
ioctl(period = 0x10     ): Ok
ioctl(period = 0x11     ): Fail
ioctl(period = 0x1f     ): Fail
ioctl(period = 0x20     ): Ok
ioctl(period = 0x80     ): Ok
ioctl(period = 0x8f     ): Fail
ioctl(period = 0x90     ): Ok
ioctl(period = 0x91     ): Fail
ioctl(period = 0x100    ): Ok
ioctl(period = 0xfff0   ): Ok
ioctl(period = 0xffff   ): Fail
ioctl(period = 0x10000  ): Ok
ioctl(period = 0x1fff0  ): Ok
ioctl(period = 0x1fff5  ): Fail
ioctl(freq   = 0x0      ): Ok
ioctl(freq   = 0x1      ): Ok
ioctl(freq   = 0xf      ): Ok
ioctl(freq   = 0x10     ): Ok
ioctl(freq   = 0x11     ): Ok
ioctl(freq   = 0x1f     ): Ok
ioctl(freq   = 0x20     ): Ok
ioctl(freq   = 0x80     ): Ok
ioctl(freq   = 0x8f     ): Ok
ioctl(freq   = 0x90     ): Ok
ioctl(freq   = 0x91     ): Ok
ioctl(freq   = 0x100    ): Ok
Op PMU tests
ioctl(period = 0x0      ): Ok
ioctl(period = 0x1      ): Fail
ioctl(period = 0xf      ): Fail
ioctl(period = 0x10     ): Fail
ioctl(period = 0x11     ): Fail
ioctl(period = 0x1f     ): Fail
ioctl(period = 0x20     ): Fail
ioctl(period = 0x80     ): Fail
ioctl(period = 0x8f     ): Fail
ioctl(period = 0x90     ): Ok
ioctl(period = 0x91     ): Fail
ioctl(period = 0x100    ): Ok
ioctl(period = 0xfff0   ): Ok
ioctl(period = 0xffff   ): Fail
ioctl(period = 0x10000  ): Ok
ioctl(period = 0x1fff0  ): Ok
ioctl(period = 0x1fff5  ): Fail
ioctl(freq   = 0x0      ): Ok
ioctl(freq   = 0x1      ): Ok
ioctl(freq   = 0xf      ): Ok
ioctl(freq   = 0x10     ): Ok
ioctl(freq   = 0x11     ): Ok
ioctl(freq   = 0x1f     ): Ok
ioctl(freq   = 0x20     ): Ok
ioctl(freq   = 0x80     ): Ok
ioctl(freq   = 0x8f     ): Ok
ioctl(freq   = 0x90     ): Ok
ioctl(freq   = 0x91     ): Ok
ioctl(freq   = 0x100    ): Ok

IBS freq (negative) tests:
--------------------------
freq 1, sample_freq 200000: Fail

IBS L3MissOnly test: (takes a while)
--------------------
Fetch L3MissOnly: Fail (nr_samples: 299)
Op L3MissOnly:    Ok   (nr_samples: 444)
---- end(-1) ----
115: AMD IBS sample period                                           : FAILED!
root@number:~# 
root@number:~# 
root@number:~# 
root@number:~# uname -r
6.14.5-300.fc42.x86_64
root@number:~# grep -m1 "model name" /proc/cpuinfo 
model name	: AMD Ryzen 9 9950X3D 16-Core Processor
root@number:~# 

Re: IBS perf test failures on 9950x3d

[Ravi Bangoria]

Hi Arnaldo,

> If it needs a specific kernel, then the test needs to cope with that,
> etc.

1) Bunch of IBS kernel fixes went in recently (v6.15-rc1):

   https://lore.kernel.org/r/20250115054438.1021-1-ravi.bangoria@amd.com

   Since you are running on 6.14.5-300.fc42.x86_64, the failures are
   genuine and indicating that kernel should be fixed.

   I'm wondering how would the test cope with the buggy kernel. Or
   probably 'perf test' a wrong place to put such selftests?

2) "mprotect() failed. Permission denied" seems to be because of
   SELinux enforced mode? I'll prepare and post a fix but, even
   with the error, it's just part of the dummy workload function
   so wouldn't break any test functionality.

Thanks,
Ravi

Re: IBS perf test failures on 9950x3d

[Namhyung Kim]

Hi Ravi,

On Sat, May 17, 2025 at 05:33:19PM +0530, Ravi Bangoria wrote:
> Hi Arnaldo,
> 
> > If it needs a specific kernel, then the test needs to cope with that,
> > etc.
> 
> 1) Bunch of IBS kernel fixes went in recently (v6.15-rc1):
> 
>    https://lore.kernel.org/r/20250115054438.1021-1-ravi.bangoria@amd.com
> 
>    Since you are running on 6.14.5-300.fc42.x86_64, the failures are
>    genuine and indicating that kernel should be fixed.
> 
>    I'm wondering how would the test cope with the buggy kernel. Or
>    probably 'perf test' a wrong place to put such selftests?

Maybe... some kernel internal changes can be tested in the selftest.

And 'perf test' should work with any kernel versions.  Is the test
checking any invalid configs?  Probably then it should move to the
selftest so that it can be tied to the fixed kernels and 'perf test'
should only use valid configs.

Thanks,
Namhyung

> 
> 2) "mprotect() failed. Permission denied" seems to be because of
>    SELinux enforced mode? I'll prepare and post a fix but, even
>    with the error, it's just part of the dummy workload function
>    so wouldn't break any test functionality.
> 
> Thanks,
> Ravi

Re: IBS perf test failures on 9950x3d

[Ravi Bangoria]

Hi Namhyung,

>>> If it needs a specific kernel, then the test needs to cope with that,
>>> etc.
>>
>> 1) Bunch of IBS kernel fixes went in recently (v6.15-rc1):
>>
>>    https://lore.kernel.org/r/20250115054438.1021-1-ravi.bangoria@amd.com
>>
>>    Since you are running on 6.14.5-300.fc42.x86_64, the failures are
>>    genuine and indicating that kernel should be fixed.
>>
>>    I'm wondering how would the test cope with the buggy kernel. Or
>>    probably 'perf test' a wrong place to put such selftests?
> 
> Maybe... some kernel internal changes can be tested in the selftest.
> 
> And 'perf test' should work with any kernel versions.  Is the test
> checking any invalid configs?  Probably then it should move to the
> selftest so that it can be tied to the fixed kernels and 'perf test'
> should only use valid configs.

Sure. I'll prepare a patch to move this test from 'perf test' to
tools/testing/selftest/.

Thanks,
Ravi

Re: IBS perf test failures on 9950x3d

[Arnaldo Carvalho de Melo]

On Sun, May 18, 2025 at 11:22:24AM -0700, Namhyung Kim wrote:
> On Sat, May 17, 2025 at 05:33:19PM +0530, Ravi Bangoria wrote:
> > Hi Arnaldo,

> > > If it needs a specific kernel, then the test needs to cope with that,
> > > etc.

> > 1) Bunch of IBS kernel fixes went in recently (v6.15-rc1):

> >    https://lore.kernel.org/r/20250115054438.1021-1-ravi.bangoria@amd.com

> >    Since you are running on 6.14.5-300.fc42.x86_64, the failures are
> >    genuine and indicating that kernel should be fixed.

Telling that to the user and possibly skipping the test if viable using
uname to get the version and if less than v6.15-rc1 print "Skip (needs
v6.15-rc1 or newer)" may be an option.

Then if the bug somehow reappears, people running 'perf test' will flag
it.

> >    I'm wondering how would the test cope with the buggy kernel. Or
> >    probably 'perf test' a wrong place to put such selftests?
 
> Maybe... some kernel internal changes can be tested in the selftest.
 
> And 'perf test' should work with any kernel versions.  Is the test
> checking any invalid configs?  Probably then it should move to the
> selftest so that it can be tied to the fixed kernels and 'perf test'
> should only use valid configs.

Right, but then there are cases where something changed and then
'perf test' should adapt to accept new behaviour while keeping what made
sense for older kernels.

The fix Leo sent recently is one such case.

https://lore.kernel.org/all/20250519082755.1669187-1-leo.yan@arm.com/T/#u

And I think having it in 'perf test' as well may make the feature to be
tested more widely, both by those who run selftests as well as by people
trying just 'perf test'.

- Arnaldo

Re: IBS perf test failures on 9950x3d

[Ravi Bangoria]

Hi Arnaldo,

> Telling that to the user and possibly skipping the test if viable using
> uname to get the version and if less than v6.15-rc1 print "Skip (needs
> v6.15-rc1 or newer)" may be an option.
> 
> Then if the bug somehow reappears, people running 'perf test' will flag
> it.

Sure. Attaching a patch at the end.

> And I think having it in 'perf test' as well may make the feature to be
> tested more widely, both by those who run selftests as well as by people
> trying just 'perf test'.

Yeah, that was precisely the reason I did it in 'perf test'.

Thanks,
Ravi

---
From fb8c9f77231a9361a5055c45e75aed3d2414f685 Mon Sep 17 00:00:00 2001
From: Ravi Bangoria <ravi.bangoria@amd.com>
Date: Tue, 20 May 2025 05:32:48 +0000
Subject: [PATCH] perf test amd: Skip amd-ibs-period test on kernel < v6.15

Bunch of IBS kernel fixes went in v6.15-rc1 [1]. The amd-ibs-period test
will fail Without those kernel patches. Skip the test on system running
kernel older than v6.15 to distinguish genuine new failures vs known
failure due to old kernel.

Since all the related IBS fixes went in -rc1 itself, the ">= 6.15" check
will work for any custom compiled v6.15-* kernel as well.

Link: https://lore.kernel.org/r/20250115054438.1021-1-ravi.bangoria@amd.com [1]
Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Closes: https://lore.kernel.org/r/aCfuGXUnNIbnYo_r@x1
Suggested-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Signed-off-by: Ravi Bangoria <ravi.bangoria@amd.com>
---
 tools/perf/arch/x86/tests/amd-ibs-period.c | 29 ++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/tools/perf/arch/x86/tests/amd-ibs-period.c b/tools/perf/arch/x86/tests/amd-ibs-period.c
index 7c77d86f369d..223e059e04de 100644
--- a/tools/perf/arch/x86/tests/amd-ibs-period.c
+++ b/tools/perf/arch/x86/tests/amd-ibs-period.c
@@ -3,6 +3,7 @@
 #include <sys/syscall.h>
 #include <sys/mman.h>
 #include <sys/ioctl.h>
+#include <sys/utsname.h>
 #include <string.h>
 
 #include "arch-tests.h"
@@ -912,6 +913,29 @@ static unsigned int get_perf_event_max_sample_rate(void)
 	return max_sample_rate;
 }
 
+/*
+ * Bunch of IBS sample period fixes that this test exercise went in v6.15.
+ * Skip the test on older kernels to distinguish between test failure due
+ * to a new bug vs known failure due to older kernel.
+ */
+static bool kernel_v6_15_or_newer(void)
+{
+	struct utsname utsname;
+	char *endptr = NULL;
+	long major, minor;
+
+	if (uname(&utsname) < 0) {
+		pr_debug("uname() failed. [%m]");
+		return false;
+	}
+
+	major = strtol(utsname.release, &endptr, 10);
+	endptr++;
+	minor = strtol(endptr, NULL, 10);
+
+	return major >= 6 && minor >= 15;
+}
+
 int test__amd_ibs_period(struct test_suite *test __maybe_unused,
 			 int subtest __maybe_unused)
 {
@@ -931,6 +955,11 @@ int test__amd_ibs_period(struct test_suite *test __maybe_unused,
 	if (!x86__is_amd_cpu() || !fetch_pmu || !op_pmu)
 		return TEST_SKIP;
 
+	if (!kernel_v6_15_or_newer()) {
+		pr_debug("Need v6.15 or newer kernel. Skipping.\n");
+		return TEST_SKIP;
+	}
+
 	perf_exe(perf, sizeof(perf));
 
 	if (sched_affine(0))
-- 
2.43.0

Re: IBS perf test failures on 9950x3d

[Arnaldo Carvalho de Melo]

On Tue, May 20, 2025 at 04:31:29PM +0530, Ravi Bangoria wrote:
> > Telling that to the user and possibly skipping the test if viable using
> > uname to get the version and if less than v6.15-rc1 print "Skip (needs
> > v6.15-rc1 or newer)" may be an option.
 
> > Then if the bug somehow reappears, people running 'perf test' will flag
> > it.
 
> Sure. Attaching a patch at the end.

Thanks, I'm applying it.

But now that I think about it, it may well be the case that, say, RHEL
backports the fixes in v6.15 for an enterprise kernel that has an
"older" version, so probably the best is for execute the test, if it
fails, then do the version check to decide if it is an unexpected
failure.

But this can be done on top, lets make progress and apply your patch.

> > And I think having it in 'perf test' as well may make the feature to be
> > tested more widely, both by those who run selftests as well as by people
> > trying just 'perf test'.
> 
> Yeah, that was precisely the reason I did it in 'perf test'.

:-)

- Arnaldo

Re: IBS perf test failures on 9950x3d

[Namhyung Kim]

On Tue, May 20, 2025 at 05:47:50PM -0300, Arnaldo Carvalho de Melo wrote:
> On Tue, May 20, 2025 at 04:31:29PM +0530, Ravi Bangoria wrote:
> > > Telling that to the user and possibly skipping the test if viable using
> > > uname to get the version and if less than v6.15-rc1 print "Skip (needs
> > > v6.15-rc1 or newer)" may be an option.
>  
> > > Then if the bug somehow reappears, people running 'perf test' will flag
> > > it.
>  
> > Sure. Attaching a patch at the end.
> 
> Thanks, I'm applying it.
> 
> But now that I think about it, it may well be the case that, say, RHEL
> backports the fixes in v6.15 for an enterprise kernel that has an
> "older" version, so probably the best is for execute the test, if it
> fails, then do the version check to decide if it is an unexpected
> failure.

Yeah, that was my concern too.  I'm fine with having it in perf test and
skip older kernels though.

> 
> But this can be done on top, lets make progress and apply your patch.
> 
> > > And I think having it in 'perf test' as well may make the feature to be
> > > tested more widely, both by those who run selftests as well as by people
> > > trying just 'perf test'.
> > 
> > Yeah, that was precisely the reason I did it in 'perf test'.

+1.

Thanks,
Namhyung