Re: perf test failures in linux-next on s390

[Thomas Richter <tmricht@linux.ibm.com>]

[-- Attachment #1: Type: text/plain, Size: 12355 bytes --]

On 6/14/23 16:57, Ian Rogers wrote:
> On Wed, Jun 14, 2023 at 1:32 AM Thomas Richter <tmricht@linux.ibm.com> wrote:
>>
>> On 6/13/23 16:32, Ian Rogers wrote:
>>> On Tue, Jun 13, 2023 at 5:54 AM Thomas Richter <tmricht@linux.ibm.com> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I have run the perf test suite on the current 6.4rc6 kernel and see just one error:
>>>> # ./perf test 2>&1 | fgrep FAILED
>>>> fgrep: warning: fgrep is obsolescent; using grep -F
>>>>  42.3: BPF prologue generation                                       : FAILED!
>>>> #
>>>>
>>>> However when I download the linux-next tree and build kernel and perf
>>>> tool with the same kernel config file, I get a bunch of failing test cases,
>>>> many with perf tool dumping core:
>>>>
>>>> # perf test 2>&1 | fgrep FAILED
>>>> fgrep: warning: fgrep is obsolescent; using grep -F
>>>>   6.1: Test event parsing                                            : FAILED!
>>>>  10.3: Parsing of PMU event table metrics                            : FAILED!
>>>>  10.4: Parsing of PMU event table metrics with fake PMUs             : FAILED!
>>>>  17: Setup struct perf_event_attr                                    : FAILED!
>>>>  24: Number of exit events of a simple workload                      : FAILED! core-dump
>>>>  28: Use a dummy software event to keep tracking                     : FAILED!
>>>>  35: Track with sched_switch                                         : FAILED!
>>>>  42.3: BPF prologue generation                                       : FAILED!
>>>>  66: Parse and process metrics                                       : FAILED!
>>>>  68: Event expansion for cgroups                                     : FAILED!
>>>>  69.2: Perf time to TSC                                              : FAILED! core-dump
>>>>  74: build id cache operations                                       : FAILED! core-dump
>>>>  81: kernel lock contention analysis test                            : FAILED!
>>>>  86: Zstd perf.data compression/decompression                        : FAILED! core-dump
>>>>  87: perf record tests                                               : FAILED! core-dump
>>>>  94: perf all metricgroups test                                      : FAILED!
>>>>  95: perf all metrics test                                           : FAILED!
>>>> 106: Test java symbol                                                : FAILED! core-dump
>>>> #
>>>>
>>>> I am afraid this will show up pretty soon in the linux tree.
>>>> I am going to look into each failure in the next few days.
>>>>
>>>> What I already found out is that many test cases now fail due to the
>>>> event/PMU rework, here is one example:
>>>>
>>>> # perf test -Fvvvv 95
>>>> 95: perf all metrics test
>>>> --- start ---
>>>> Testing cpi
>>>> ....
>>>> Metric 'transaction' not printed in:
>>>> Error:
>>>> The TX_NC_TABORT event is not supported.
>>>> ---- end ----
>>>> perf all metrics test: FAILED!
>>>> # ls -l /sys/devices/cpum_cf/events/TX_NC_TABORT
>>>> -r--r--r--. 1 root root 4096 Jun 13 13:49 /sys/devices/cpum_cf/events/TX_NC_TABORT
>>>> #
>>>>
>>>> As can be seen, the event is definitely there and supported.
>>>> This same test case succeeds in the linux tree!
>>>>
>>>> Hopefully I can sort out some of the failures before this code show up
>>>> in the linux tree.
>>>
>>> Thanks Thomas, to be clear this is what is in
>>> perf-tools-next/linux-next and not 6.4?
>>
>> Ian,
>>
>> thanks for your help.
>> Correct, I am talking about the linux-next repo. The linux repo is fine.
>>
>>>
>>> Rather than try to do more complicated cases like the metrics tests,
>>> it makes sense to dig into why event parsing is failing. Test 6 first
>>> of all, could you give output?
>>>
>>> Thanks,
>>> Ian
>>>
>> We discussed some aspects of this about two weeks ago, but last week
>> I was on vacation and now I resumed my work on linux-next.
>> We run the linux-next perf test suite every night and I am concerned
>> and would like to get this sorted out before it hits Linux 6.5.
>>
>> Here is the output on my linux-next tree built yesterday:
>> # uname -a
>> Linux a35lp67.lnxne.boe 6.4.0-rc6-next-20230613d-perf #2 \
>>               SMP Tue Jun 13 15:18:43 CEST 2023 s390x GNU/Linux
>> # ./perf test -F 6
>>   6: Parse event definition strings  :
>>   6.1: Test event parsing            :Segmentation fault (core dumped)
>> #
>> # gdb perf
>>   ....
>>   (gdb) r test -F 6
>>    6: Parse event definition strings                                  :
>>   6.1: Test event parsing                                            :
>> Program received signal SIGSEGV, Segmentation fault.
>> __GI_strcmp () at ../sysdeps/s390/strcmp-vx.S:47
>> (gdb) where
>> #0  __GI_strcmp () at ../sysdeps/s390/strcmp-vx.S:47
>> #1  0x000000000110a18c in test__term_equal_term (evlist=0x152ea80) at tests/parse-events.c:1580
>> #2  0x000000000110a96a in test_event (e=0x14dc758 <test.events+1416>) at tests/parse-events.c:2209
>> #3  0x000000000110ac58 in test_events (events=0x14dc1d0 <test.events>, cnt=61) at tests/parse-events.c:2260
>> #4  0x000000000110ad52 in test__events2 (test=0x1500758 <suite.parse_events>, subtest=0)
>>     at tests/parse-events.c:2272
>> #5  0x00000000010f6fac in run_test (test=0x1500758 <suite.parse_events>, subtest=0) at tests/builtin-test.c:236
>> #6  0x00000000010f7142 in test_and_print (t=0x1500758 <suite.parse_events>, subtest=0) at tests/builtin-test.c:265
>> #7  0x00000000010f7b1e in __cmd_test (argc=1, argv=0x3ffffffa320, skiplist=0x0) at tests/builtin-test.c:436
>> #8  0x00000000010f8404 in cmd_test (argc=1, argv=0x3ffffffa320) at tests/builtin-test.c:559
>> #9  0x00000000011473fc in run_builtin (p=0x14f60e8 <commands+600>, argc=3, argv=0x3ffffffa320) at perf.c:323
>> #10 0x000000000114776e in handle_internal_command (argc=3, argv=0x3ffffffa320) at perf.c:377
>> #11 0x0000000001147980 in run_argv (argcp=0x3ffffff9f94, argv=0x3ffffff9f88) at perf.c:421
>> #12 0x0000000001147d48 in main (argc=3, argv=0x3ffffffa320) at perf.c:537
>> (gdb)
>>
>> To be honest, I am no expert on the yacc/bison/flex tool chain.
>> I understand a little bit about them, but that is it.
>>
>> When I look at the output of perf test -Fvvvv 6 on linux-next, some things seem odd,
>> I marked them with 3 question masks ???:
>>
>> # ./perf test -Fvvv 6
>>   6: Parse event definition strings     :
>>   6.1: Test event parsing               :
>> --- start ---
>> running test 0 'syscalls:sys_enter_openat'
>> Using CPUID IBM,3931,704,A01,3.7,002f
>> running test 1 'syscalls:*'
>> running test 2 'r1a'
>> running test 3 '1:1'
>> running test 4 'instructions'
>> No PMU found for 'instructions'FAILED tests/parse-events.c:143 wrong number of entries
>> Event test failure: test 4 'instructions'running test 5 'cycles/period=100000,config2/'
>> ??? What is wrong here?
>> ??? Output on linux 6.4.0rc3:
>> ??? # ./perf stat -e instructions -- true
>> ???
>> ??? Performance counter stats for 'true':
>> ???
>> ???         2,965,720      instructions
>> ???
>> ???        0.002026832 seconds time elapsed
>> ???
>> ???        0.000056000 seconds user
>> ???        0.002048000 seconds sys
>> ??? #
>> ??? This is fine and works as expected. The s390 PMU for counters
>> ??? has a direct mapping for this. So we end up in the s390 PMU
>> ??? to retrieve the value.
>> ???
>> ??? Output on linux-next
>> ???# ./perf stat -e instructions -- true
>> ???
>> ??? Performance counter stats for 'true':
>> ???
>> ???              0.65 msec task-clock                       #    0.250 CPUs utilized
>> ???                 0      context-switches                 #    0.000 /sec
>> ???                 0      cpu-migrations                   #    0.000 /sec
>> ???                49      page-faults                      #   75.375 K/sec
>> ???         3,367,228      cycles                           #    5.180 GHz
>> ???         2,880,270      instructions                     #    0.86  insn per cycle
>> ???   <not supported>      branches
>> ???   <not supported>      branch-misses
>> ???
>> ???       0.002599176 seconds time elapsed
>> ???
>> ???       0.000053000 seconds user
>> ???       0.002650000 seconds sys
>> ???
>> ???#
>> ??? Somehow we end up in a different PMU. The output is the same as if
>> ??? I do not specify an event at all. To reach the s390 specific PMU
>> ??? I have to add it explicitly as in:
>> ???# ./perf stat -e cpum_cf/instructions/ -- true
>> ???
>> ??? Performance counter stats for 'true':
>> ???
>> ???         2,814,522      cpum_cf/instructions/
>> ???
>> ???       0.001899881 seconds time elapsed
>> ???
>> ???       0.000050000 seconds user
>> ???       0.001928000 seconds sys
>> ???
>> ???]#
>> No PMU found for 'cycles/period=100000,config2/'FAILED tests/parse-events.c:157 wrong number of entries
>> Event test failure: test 5 'cycles/period=100000,config2/'running test 6 'faults'
>> ...
>> ??? Similar output for basicly all events.
>>
>> No PMU found for 'cycles'running test 59 'cycles/name=name/'
>> No PMU found for 'name'Segmentation fault (core dumped)
>>
>> Hope this helps.
>>
>> PS: Should we keep the linux-perf-use mailing list as addressee? Not sure
>> if everybody else is interested in this?
> 
> Smaller list is okay. Could you send me a zip of the sysfs
> (/sys/devices) ? At least one issue is that the code didn't find a
> core PMU. On non-hybrid x86 this would be /sys/devices/cpu, I think we
> spoke about this before for s390 and there are >1. The issue here is
> that the test found 0, and we're trying to use PMUs in the code now as
> a way to sort events. There's code/comment in util/pmu.c:
> 
> '''
> /**
>  * is_sysfs_pmu_core() - PMU CORE devices have different name other than cpu in
>  *         sysfs on some platforms like ARM or Intel hybrid. Looking for
>  *         possible the cpus file in sysfs files to identify whether this is a
>  *         core device.
>  * @name: The PMU name such as "cpu_atom".
>  */
> static int is_sysfs_pmu_core(const char *name)
> {
> char path[PATH_MAX];
> 
> if (!perf_pmu__pathname_scnprintf(path, sizeof(path), name, "cpus"))
> return 0;
> return file_available(path);
> }
> ...
> bool is_pmu_core(const char *name)
> {
> return !strcmp(name, "cpu") || is_sysfs_pmu_core(name);
> }
> '''
> 
> Thanks,
> Ian
> 

Thanks for refreshing my memory. With s390 core PMU named as
/sys/devices/cpum_cf, this fix was missing:

diff --git a/tools/perf/util/pmu.c b/tools/perf/util/pmu.c
index fe64ad292d36..6142e4710a2f 100644
--- a/tools/perf/util/pmu.c
+++ b/tools/perf/util/pmu.c
@@ -1419,7 +1419,7 @@ void perf_pmu__del_formats(struct list_head *formats)
 
 bool is_pmu_core(const char *name)
 {
-       return !strcmp(name, "cpu") || is_sysfs_pmu_core(name);
+       return !strcmp(name, "cpu") || !strcmp(name, "cpum_cf") || is_sysfs_pmu_core(name);
 }
 
 bool perf_pmu__supports_legacy_cache(const struct perf_pmu *pmu)

With that fix applied, the test succeeds:

# ./perf test -F 6
  6: Parse event definition strings                                  :
  6.1: Test event parsing                                            : Ok
  6.2: Parsing of all PMU events from sysfs                          : Ok
  6.3: Parsing of given PMU events from sysfs                        : Ok
  6.4: Parsing of aliased events from sysfs                          : Skip (no aliases in sysfs)
  6.5: Parsing of aliased events                                     : Ok
  6.6: Parsing of terms (event modifiers)                            : Ok
#

I have tried to send my /sys/devices tree before as zip or tgz but that
mail got deleted by your mailer because it contained compressed data.

So I send you an ls -lR from our 5 PMUs on s390 as text attachment and
here are the type values:
# for i in  /sys/devices/cpum_[sc]f* /sys/devices/pai_*; do echo PMU $i PMU_TYPE $(cat $i/type); done
PMU /sys/devices/cpum_cf PMU_TYPE 8
PMU /sys/devices/cpum_cf_diag PMU_TYPE 9
PMU /sys/devices/cpum_sf PMU_TYPE 4
PMU /sys/devices/pai_crypto PMU_TYPE 10
PMU /sys/devices/pai_ext PMU_TYPE 11
# 

For the /sys/devices/PMU tree see attachment sysfs-s390.txt

Thanks a lot for you help.
-- 
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: David Faller
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294

[-- Attachment #2: sysfs-s390.txt --]
[-- Type: text/plain, Size: 19744 bytes --]

/sys/devices/cpum_cf:
total 0
drwxr-xr-x 2 root root    0 Jun 14 13:03 events
drwxr-xr-x 2 root root    0 Jun 14 13:03 format
-rw-r--r-- 1 root root 4096 Jun 15 10:48 perf_event_mux_interval_ms
lrwxrwxrwx 1 root root    0 Jun 15 10:48 subsystem -> ../../bus/event_source
-r--r--r-- 1 root root 4096 Jun 14 13:03 type
-rw-r--r-- 1 root root 4096 Jun 15 10:48 uevent

/sys/devices/cpum_cf/events:
total 0
-r--r--r-- 1 root root 4096 Jun 14 13:03 AES_BLOCKED_CYCLES
-r--r--r-- 1 root root 4096 Jun 14 13:03 AES_BLOCKED_FUNCTIONS
-r--r--r-- 1 root root 4096 Jun 14 13:03 AES_CYCLES
-r--r--r-- 1 root root 4096 Jun 14 13:03 AES_FUNCTIONS
-r--r--r-- 1 root root 4096 Jun 14 13:03 BCD_DFP_EXECUTION_SLOTS
-r--r--r-- 1 root root 4096 Jun 14 13:03 CPU_CYCLES
-r--r--r-- 1 root root 4096 Jun 14 13:03 CRSTE_1MB_WRITES
-r--r--r-- 1 root root 4096 Jun 14 13:03 DCW_OFF_DRAWER
-r--r--r-- 1 root root 4096 Jun 14 13:03 DCW_OFF_DRAWER_MEMORY
-r--r--r-- 1 root root 4096 Jun 14 13:03 DCW_ON_CHIP
-r--r--r-- 1 root root 4096 Jun 14 13:03 DCW_ON_CHIP_CHIP_HIT
-r--r--r-- 1 root root 4096 Jun 14 13:03 DCW_ON_CHIP_DRAWER_HIT
-r--r--r-- 1 root root 4096 Jun 14 13:03 DCW_ON_CHIP_IV
-r--r--r-- 1 root root 4096 Jun 14 13:03 DCW_ON_CHIP_MEMORY
-r--r--r-- 1 root root 4096 Jun 14 13:03 DCW_ON_DRAWER
-r--r--r-- 1 root root 4096 Jun 14 13:03 DCW_ON_DRAWER_MEMORY
-r--r--r-- 1 root root 4096 Jun 14 13:03 DCW_ON_MODULE
-r--r--r-- 1 root root 4096 Jun 14 13:03 DCW_ON_MODULE_MEMORY
-r--r--r-- 1 root root 4096 Jun 14 13:03 DCW_REQ
-r--r--r-- 1 root root 4096 Jun 14 13:03 DCW_REQ_CHIP_HIT
-r--r--r-- 1 root root 4096 Jun 14 13:03 DCW_REQ_DRAWER_HIT
-r--r--r-- 1 root root 4096 Jun 14 13:03 DCW_REQ_IV
-r--r--r-- 1 root root 4096 Jun 14 13:03 DEA_BLOCKED_CYCLES
-r--r--r-- 1 root root 4096 Jun 14 13:03 DEA_BLOCKED_FUNCTIONS
-r--r--r-- 1 root root 4096 Jun 14 13:03 DEA_CYCLES
-r--r--r-- 1 root root 4096 Jun 14 13:03 DEA_FUNCTIONS
-r--r--r-- 1 root root 4096 Jun 14 13:03 DECIMAL_INSTRUCTIONS
-r--r--r-- 1 root root 4096 Jun 14 13:03 DFLT_ACCESS
-r--r--r-- 1 root root 4096 Jun 14 13:03 DFLT_CC
-r--r--r-- 1 root root 4096 Jun 14 13:03 DFLT_CCFINISH
-r--r--r-- 1 root root 4096 Jun 14 13:03 DFLT_CYCLES
-r--r--r-- 1 root root 4096 Jun 14 13:03 DTLB2_GPAGE_WRITES
-r--r--r-- 1 root root 4096 Jun 14 13:03 DTLB2_MISSES
-r--r--r-- 1 root root 4096 Jun 14 13:03 DTLB2_WRITES
-r--r--r-- 1 root root 4096 Jun 14 13:03 ECC_BLOCKED_CYCLES_COUNT
-r--r--r-- 1 root root 4096 Jun 14 13:03 ECC_BLOCKED_FUNCTION_COUNT
-r--r--r-- 1 root root 4096 Jun 14 13:03 ECC_CYCLES_COUNT
-r--r--r-- 1 root root 4096 Jun 14 13:03 ECC_FUNCTION_COUNT
-r--r--r-- 1 root root 4096 Jun 14 13:03 ICW_OFF_DRAWER
-r--r--r-- 1 root root 4096 Jun 14 13:03 ICW_OFF_DRAWER_MEMORY
-r--r--r-- 1 root root 4096 Jun 14 13:03 ICW_ON_CHIP
-r--r--r-- 1 root root 4096 Jun 14 13:03 ICW_ON_CHIP_CHIP_HIT
-r--r--r-- 1 root root 4096 Jun 14 13:03 ICW_ON_CHIP_DRAWER_HIT
-r--r--r-- 1 root root 4096 Jun 14 13:03 ICW_ON_CHIP_IV
-r--r--r-- 1 root root 4096 Jun 14 13:03 ICW_ON_CHIP_MEMORY
-r--r--r-- 1 root root 4096 Jun 14 13:03 ICW_ON_DRAWER
-r--r--r-- 1 root root 4096 Jun 14 13:03 ICW_ON_DRAWER_MEMORY
-r--r--r-- 1 root root 4096 Jun 14 13:03 ICW_ON_MODULE
-r--r--r-- 1 root root 4096 Jun 14 13:03 ICW_ON_MODULE_MEMORY
-r--r--r-- 1 root root 4096 Jun 14 13:03 ICW_REQ
-r--r--r-- 1 root root 4096 Jun 14 13:03 ICW_REQ_CHIP_HIT
-r--r--r-- 1 root root 4096 Jun 14 13:03 ICW_REQ_DRAWER_HIT
-r--r--r-- 1 root root 4096 Jun 14 13:03 ICW_REQ_IV
-r--r--r-- 1 root root 4096 Jun 14 13:03 IDCW_OFF_DRAWER_CHIP_HIT
-r--r--r-- 1 root root 4096 Jun 14 13:03 IDCW_OFF_DRAWER_DRAWER_HIT
-r--r--r-- 1 root root 4096 Jun 14 13:03 IDCW_OFF_DRAWER_IV
-r--r--r-- 1 root root 4096 Jun 14 13:03 IDCW_ON_DRAWER_CHIP_HIT
-r--r--r-- 1 root root 4096 Jun 14 13:03 IDCW_ON_DRAWER_DRAWER_HIT
-r--r--r-- 1 root root 4096 Jun 14 13:03 IDCW_ON_DRAWER_IV
-r--r--r-- 1 root root 4096 Jun 14 13:03 IDCW_ON_MODULE_CHIP_HIT
-r--r--r-- 1 root root 4096 Jun 14 13:03 IDCW_ON_MODULE_DRAWER_HIT
-r--r--r-- 1 root root 4096 Jun 14 13:03 IDCW_ON_MODULE_IV
-r--r--r-- 1 root root 4096 Jun 14 13:03 INSTRUCTIONS
-r--r--r-- 1 root root 4096 Jun 14 13:03 ITLB2_MISSES
-r--r--r-- 1 root root 4096 Jun 14 13:03 ITLB2_WRITES
-r--r--r-- 1 root root 4096 Jun 14 13:03 L1C_TLB2_MISSES
-r--r--r-- 1 root root 4096 Jun 14 13:03 L1D_DIR_WRITES
-r--r--r-- 1 root root 4096 Jun 14 13:03 L1D_PENALTY_CYCLES
-r--r--r-- 1 root root 4096 Jun 14 13:03 L1D_RO_EXCL_WRITES
-r--r--r-- 1 root root 4096 Jun 14 13:03 L1I_DIR_WRITES
-r--r--r-- 1 root root 4096 Jun 14 13:03 L1I_PENALTY_CYCLES
-r--r--r-- 1 root root 4096 Jun 14 13:03 LAST_HOST_TRANSLATIONS
-r--r--r-- 1 root root 4096 Jun 14 13:03 MT_DIAG_CYCLES_ONE_THR_ACTIVE
-r--r--r-- 1 root root 4096 Jun 14 13:03 MT_DIAG_CYCLES_TWO_THR_ACTIVE
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_COMPLETIONS
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_HOLD_LOCK
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_INVOCATIONS
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_WAIT_LOCK
-r--r--r-- 1 root root 4096 Jun 14 13:03 PRNG_BLOCKED_CYCLES
-r--r--r-- 1 root root 4096 Jun 14 13:03 PRNG_BLOCKED_FUNCTIONS
-r--r--r-- 1 root root 4096 Jun 14 13:03 PRNG_CYCLES
-r--r--r-- 1 root root 4096 Jun 14 13:03 PRNG_FUNCTIONS
-r--r--r-- 1 root root 4096 Jun 14 13:03 PROBLEM_STATE_CPU_CYCLES
-r--r--r-- 1 root root 4096 Jun 14 13:03 PROBLEM_STATE_INSTRUCTIONS
-r--r--r-- 1 root root 4096 Jun 14 13:03 SHA_BLOCKED_CYCLES
-r--r--r-- 1 root root 4096 Jun 14 13:03 SHA_BLOCKED_FUNCTIONS
-r--r--r-- 1 root root 4096 Jun 14 13:03 SHA_CYCLES
-r--r--r-- 1 root root 4096 Jun 14 13:03 SHA_FUNCTIONS
-r--r--r-- 1 root root 4096 Jun 14 13:03 SORTL
-r--r--r-- 1 root root 4096 Jun 14 13:03 TLB2_CRSTE_WRITES
-r--r--r-- 1 root root 4096 Jun 14 13:03 TLB2_ENGINES_BUSY
-r--r--r-- 1 root root 4096 Jun 14 13:03 TLB2_PTE_WRITES
-r--r--r-- 1 root root 4096 Jun 14 13:03 TX_C_TABORT_NO_SPECIAL
-r--r--r-- 1 root root 4096 Jun 14 13:03 TX_C_TABORT_SPECIAL
-r--r--r-- 1 root root 4096 Jun 14 13:03 TX_C_TEND
-r--r--r-- 1 root root 4096 Jun 14 13:03 TX_NC_TABORT
-r--r--r-- 1 root root 4096 Jun 14 13:03 TX_NC_TEND
-r--r--r-- 1 root root 4096 Jun 14 13:03 VX_BCD_EXECUTION_SLOTS

/sys/devices/cpum_cf/format:
total 0
-r--r--r-- 1 root root 4096 Jun 14 13:03 event

/sys/devices/cpum_cf_diag:
total 0
drwxr-xr-x 2 root root    0 Jun 14 15:43 events
drwxr-xr-x 2 root root    0 Jun 14 15:43 format
-rw-r--r-- 1 root root 4096 Jun 15 10:53 perf_event_mux_interval_ms
lrwxrwxrwx 1 root root    0 Jun 15 10:53 subsystem -> ../../bus/event_source
-r--r--r-- 1 root root 4096 Jun 14 15:43 type
-rw-r--r-- 1 root root 4096 Jun 15 10:53 uevent

/sys/devices/cpum_cf_diag/events:
total 0
-r--r--r-- 1 root root 4096 Jun 14 15:43 CF_DIAG

/sys/devices/cpum_cf_diag/format:
total 0
-r--r--r-- 1 root root 4096 Jun 14 15:43 event

/sys/devices/cpum_sf:
total 0
drwxr-xr-x 2 root root    0 Jun 14 13:03 events
drwxr-xr-x 2 root root    0 Jun 14 13:03 format
-rw-r--r-- 1 root root 4096 Jun 15 10:48 perf_event_mux_interval_ms
lrwxrwxrwx 1 root root    0 Jun 15 10:48 subsystem -> ../../bus/event_source
-r--r--r-- 1 root root 4096 Jun 14 13:03 type
-rw-r--r-- 1 root root 4096 Jun 15 10:48 uevent

/sys/devices/cpum_sf/events:
total 0
-r--r--r-- 1 root root 4096 Jun 14 13:03 SF_CYCLES_BASIC
-r--r--r-- 1 root root 4096 Jun 14 13:03 SF_CYCLES_BASIC_DIAG

/sys/devices/cpum_sf/format:
total 0
-r--r--r-- 1 root root 4096 Jun 14 13:03 event

/sys/devices/pai_crypto:
total 0
drwxr-xr-x 2 root root    0 Jun 14 12:12 events
drwxr-xr-x 2 root root    0 Jun 14 13:03 format
-rw-r--r-- 1 root root 4096 Jun 15 10:49 perf_event_mux_interval_ms
lrwxrwxrwx 1 root root    0 Jun 15 10:49 subsystem -> ../../bus/event_source
-r--r--r-- 1 root root 4096 Jun 14 13:03 type
-rw-r--r-- 1 root root 4096 Jun 15 10:49 uevent

/sys/devices/pai_crypto/events:
total 0
-r--r--r-- 1 root root 4096 Jun 14 15:43 CRYPTO_ALL
-r--r--r-- 1 root root 4096 Jun 15 10:33 IBM_RESERVED_155
-r--r--r-- 1 root root 4096 Jun 15 10:33 IBM_RESERVED_156
-r--r--r-- 1 root root 4096 Jun 15 10:33 KDSA_ECDSA_SIGN_P256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KDSA_ECDSA_SIGN_P384
-r--r--r-- 1 root root 4096 Jun 15 10:33 KDSA_ECDSA_SIGN_P521
-r--r--r-- 1 root root 4096 Jun 15 10:33 KDSA_ECDSA_VERIFY_P256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KDSA_ECDSA_VERIFY_P384
-r--r--r-- 1 root root 4096 Jun 15 10:33 KDSA_ECDSA_VERIFY_P521
-r--r--r-- 1 root root 4096 Jun 14 15:43 KDSA_EDDSA_SIGN_ED25519
-r--r--r-- 1 root root 4096 Jun 15 10:33 KDSA_EDDSA_SIGN_ED448
-r--r--r-- 1 root root 4096 Jun 15 10:33 KDSA_EDDSA_VERIFY_ED25519
-r--r--r-- 1 root root 4096 Jun 15 10:33 KDSA_EDDSA_VERIFY_ED448
-r--r--r-- 1 root root 4096 Jun 14 15:43 KDSA_ENCRYPTED_ECDSA_SIGN_P256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KDSA_ENCRYPTED_ECDSA_SIGN_P384
-r--r--r-- 1 root root 4096 Jun 14 15:43 KDSA_ENCRYPTED_ECDSA_SIGN_P521
-r--r--r-- 1 root root 4096 Jun 15 10:33 KDSA_ENCRYPTED_EDDSA_SIGN_ED25519
-r--r--r-- 1 root root 4096 Jun 15 10:33 KDSA_ENCRYPTED_EDDSA_SIGN_ED448
-r--r--r-- 1 root root 4096 Jun 15 10:33 KIMD_GHASH
-r--r--r-- 1 root root 4096 Jun 15 10:33 KIMD_SHA_1
-r--r--r-- 1 root root 4096 Jun 15 10:33 KIMD_SHA_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KIMD_SHA3_224
-r--r--r-- 1 root root 4096 Jun 15 10:33 KIMD_SHA3_256
-r--r--r-- 1 root root 4096 Jun 14 15:43 KIMD_SHA3_384
-r--r--r-- 1 root root 4096 Jun 15 10:33 KIMD_SHA3_512
-r--r--r-- 1 root root 4096 Jun 15 10:33 KIMD_SHA_512
-r--r--r-- 1 root root 4096 Jun 15 10:33 KIMD_SHAKE_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KIMD_SHAKE_256
-r--r--r-- 1 root root 4096 Jun 14 13:03 KLMD_SHA_1
-r--r--r-- 1 root root 4096 Jun 15 10:33 KLMD_SHA_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KLMD_SHA3_224
-r--r--r-- 1 root root 4096 Jun 15 10:33 KLMD_SHA3_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KLMD_SHA3_384
-r--r--r-- 1 root root 4096 Jun 14 15:43 KLMD_SHA3_512
-r--r--r-- 1 root root 4096 Jun 15 10:33 KLMD_SHA_512
-r--r--r-- 1 root root 4096 Jun 14 15:43 KLMD_SHAKE_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KLMD_SHAKE_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMAC_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMAC_AES_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMAC_AES_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMAC_DEA
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMAC_ENCRYPTED_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMAC_ENCRYPTED_AES_192
-r--r--r-- 1 root root 4096 Jun 14 15:43 KMAC_ENCRYPTED_AES_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMAC_ENCRYPTED_DEA
-r--r--r-- 1 root root 4096 Jun 14 15:43 KMAC_ENCRYPTED_TDEA_128
-r--r--r-- 1 root root 4096 Jun 14 15:43 KMAC_ENCRYPTED_TDEA_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMAC_TDEA_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMAC_TDEA_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KM_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KM_AES_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KM_AES_256
-r--r--r-- 1 root root 4096 Jun 14 15:43 KMA_GCM_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMA_GCM_AES_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMA_GCM_AES_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMA_GCM_ENCRYPTED_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMA_GCM_ENCRYPTED_AES_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMA_GCM_ENCRYPTED_AES_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMC_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMC_AES_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMC_AES_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMC_DEA
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMC_ENCRYPTED_AES_128
-r--r--r-- 1 root root 4096 Jun 14 15:43 KMC_ENCRYPTED_AES_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMC_ENCRYPTED_AES_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMC_ENCRYPTED_DEA
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMC_ENCRYPTED_TDEA_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMC_ENCRYPTED_TDEA_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMC_PRNG
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMC_TDEA_128
-r--r--r-- 1 root root 4096 Jun 14 15:43 KMC_TDEA_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMCTR_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMCTR_AES_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMCTR_AES_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMCTR_DEA
-r--r--r-- 1 root root 4096 Jun 14 15:43 KMCTR_ENCRYPTED_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMCTR_ENCRYPTED_AES_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMCTR_ENCRYPTED_AES_256
-r--r--r-- 1 root root 4096 Jun 14 15:43 KMCTR_ENCRYPTED_DEA
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMCTR_ENCRYPTED_TDEA_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMCTR_ENCRYPTED_TDEA_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMCTR_TDEA_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMCTR_TDEA_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KM_DEA
-r--r--r-- 1 root root 4096 Jun 15 10:33 KM_ENCRYPTED_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KM_ENCRYPTED_AES_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KM_ENCRYPTED_AES_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KM_ENCRYPTED_DEA
-r--r--r-- 1 root root 4096 Jun 15 10:33 KM_ENCRYPTED_TDEA_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KM_ENCRYPTED_TDEA_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMF_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMF_AES_192
-r--r--r-- 1 root root 4096 Jun 14 15:43 KMF_AES_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMF_DEA
-r--r--r-- 1 root root 4096 Jun 14 15:43 KMF_ENCRYPTED_AES_128
-r--r--r-- 1 root root 4096 Jun 14 15:43 KMF_ENCRYPTED_AES_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMF_ENCRYPTED_AES_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMF_ENCRYPTED_DEA
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMF_ENCRYPTED_TDEA_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMF_ENCRYPTED_TDEA_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMF_TDEA_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMF_TDEA_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMO_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMO_AES_192
-r--r--r-- 1 root root 4096 Jun 14 15:43 KMO_AES_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMO_DEA
-r--r--r-- 1 root root 4096 Jun 14 15:43 KMO_ENCRYPTED_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMO_ENCRYPTED_AES_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMO_ENCRYPTED_AES_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMO_ENCRYPTED_DEA
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMO_ENCRYPTED_TDEA_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMO_ENCRYPTED_TDEA_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMO_TDEA_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KMO_TDEA_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 KM_TDEA_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KM_TDEA_192
-r--r--r-- 1 root root 4096 Jun 14 15:43 KM_XTS_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KM_XTS_AES_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 KM_XTS_ENCRYPTED_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 KM_XTS_ENCRYPTED_AES_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_COMPUTE_LAST_BLOCK_CMAC_USING_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_COMPUTE_LAST_BLOCK_CMAC_USING_AES_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_COMPUTE_LAST_BLOCK_CMAC_USING_AES_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_COMPUTE_LAST_BLOCK_CMAC_USING_DEA
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_COMPUTE_LAST_BLOCK_CMAC_USING_ENCRYPTED_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_COMPUTE_LAST_BLOCK_CMAC_USING_ENCRYPTED_AES_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_COMPUTE_LAST_BLOCK_CMAC_USING_ENCRYPTED_AES_256A
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_COMPUTE_LAST_BLOCK_CMAC_USING_ENCRYPTED_DEA
-r--r--r-- 1 root root 4096 Jun 14 15:43 PCC_COMPUTE_LAST_BLOCK_CMAC_USING_ENCRYPTED_TDEA_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_COMPUTE_LAST_BLOCK_CMAC_USING_ENCRYPTED_TDEA_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_COMPUTE_LAST_BLOCK_CMAC_USING_TDEA_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_COMPUTE_LAST_BLOCK_CMAC_USING_TDEA_192
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_COMPUTE_XTS_PARAMETER_USING_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_COMPUTE_XTS_PARAMETER_USING_AES_256
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_COMPUTE_XTS_PARAMETER_USING_ENCRYPTED_AES_128
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_COMPUTE_XTS_PARAMETER_USING_ENCRYPTED_AES_256
-r--r--r-- 1 root root 4096 Jun 14 15:43 PCC_SCALAR_MULTIPLY_ED25519
-r--r--r-- 1 root root 4096 Jun 14 15:43 PCC_SCALAR_MULTIPLY_ED448
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_SCALAR_MULTIPLY_P256
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_SCALAR_MULTIPLY_P384
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_SCALAR_MULTIPLY_P521
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_SCALAR_MULTIPLY_X25519
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCC_SCALAR_MULTIPLY_X448
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCKMO_ENCRYPT_AES_128_KEY
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCKMO_ENCRYPT_AES_192_KEY
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCKMO_ENCRYPT_AES_256_KEY
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCKMO_ENCRYPT_DEA_KEY
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCKMO_ENCRYPT_ECC_ED25519_KEY
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCKMO_ENCRYPT_ECC_ED448_KEY
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCKMO_ENCRYPT_ECC_P256_KEY
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCKMO_ENCRYPT_ECC_P384_KEY
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCKMO_ENCRYPT_ECC_P521_KEY
-r--r--r-- 1 root root 4096 Jun 15 10:33 PCKMO_ENCRYPT_TDEA_128_KEY
-r--r--r-- 1 root root 4096 Jun 14 15:43 PCKMO_ENCRYPT_TDEA_192_KEY
-r--r--r-- 1 root root 4096 Jun 15 10:33 PRNO_SHA_512_DRNG
-r--r--r-- 1 root root 4096 Jun 15 10:33 PRNO_TRNG
-r--r--r-- 1 root root 4096 Jun 15 10:33 PRNO_TRNG_QUERY_RAW_TO_CONDITIONED_RATIO

/sys/devices/pai_crypto/format:
total 0
-r--r--r-- 1 root root 4096 Jun 14 13:03 event

/sys/devices/pai_ext:
total 0
drwxr-xr-x 2 root root    0 Jun 14 13:03 events
drwxr-xr-x 2 root root    0 Jun 14 13:03 format
-rw-r--r-- 1 root root 4096 Jun 15 10:49 perf_event_mux_interval_ms
lrwxrwxrwx 1 root root    0 Jun 15 10:49 subsystem -> ../../bus/event_source
-r--r--r-- 1 root root 4096 Jun 14 13:03 type
-rw-r--r-- 1 root root 4096 Jun 15 10:49 uevent

/sys/devices/pai_ext/events:
total 0
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_1MFRAME
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_2GFRAME
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_ACCESSEXCEPT
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_ADD
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_ALL
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_AVGPOOL2D
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_BATCHNORM
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_CONVOLUTION
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_DIV
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_EXP
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_GRUACT
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_IBM_RESERVED_9
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_LARGEDIM
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_LOG
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_LSTMACT
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_MATMUL_OP
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_MATMUL_OP_BCAST23
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_MAX
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_MAXPOOL2D
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_MIN
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_MUL
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_RELU
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_SIGMOID
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_SMALLBATCH
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_SMALLTENSOR
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_SOFTMAX
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_SUB
-r--r--r-- 1 root root 4096 Jun 14 13:03 NNPA_TANH

/sys/devices/pai_ext/format:
total 0
-r--r--r-- 1 root root 4096 Jun 14 13:03 event