perf record dumps core all the time on s390 (6.6.0rc1)

perf record dumps core all the time on s390 (6.6.0rc1)

[Thomas Richter]

Using Linux kernel version 6.6.0rc1, the command

  # perf record -e cycles

dumps core immediately on s390 running on a z/VM virtual machine.
It works on an s390 LPAR (same kernel).

This issue showed up during my vacation on 25-Aug-2023 in linux-next
and since Monday 11-Sep-2023 it shows up in linux 6.6.0rc1.

The event cycles is treated as a software event and it fails in:

 (gdb) n
 1014           pmu->events_table = perf_pmu__find_events_table(pmu);
 (gdb) p *pmu
  $4 = {name = 0x152a150 "software", alias_name = 0x0, id = 0x0, type = 1,
          selectable = false, is_core = false,
  is_uncore = false, auxtrace = false, formats_checked = false,
  config_masks_present = false,
  config_masks_computed = false, max_precise = -1, default_config = 0x0, cpus = 0x0,
  format = {next = 0x152a0c8, prev = 0x152a0c8},
  aliases = {next = 0x152a0d8, prev = 0x152a0d8}, events_table = 0x0, sysfs_aliases = 0,
  loaded_json_aliases = 0, sysfs_aliases_loaded = false,
  cpu_aliases_added = false, caps_initialized = false,
  nr_caps = 0, caps = {next = 0x152a100, prev = 0x152a100},
  list = {next = 0x0, prev = 0x0}, config_masks = {0, 0, 0, 0},
  missing_features = {exclude_guest = false}}
 (gdb) n

 Program received signal SIGSEGV, Segmentation fault.
 0x0000000001344558 in perf_pmu__find_events_table (pmu=0x152a090) at pmu-events/pmu-events.c:1970
 1970           for (i = 0; i < table->num_pmus; i++) {
 (gdb) p table
 $5 = (const struct pmu_events_table *) 0x0
 (gdb)

(gdb) where
 #0  perf_pmu__find_events_table (pmu=pmu@entry=0x2aa003d48d0)
        at pmu-events/pmu-events.c:1970
 #1  0x000002aa00180cee in perf_pmu__lookup (pmus=0x2aa0039baf8 <other_pmus>,
        dirfd=dirfd@entry=3, lookup_name=lookup_name@entry=0x2aa003dca83 "software")
        at util/pmu.c:1014
 #2  0x000002aa0018151e in perf_pmu__find2 (name=0x2aa003dca83 "software",
        dirfd=<optimized out>) at util/pmus.c:150
 #3  pmu_read_sysfs (core_only=false) at util/pmus.c:198
 #4  0x000002aa00181e04 in pmu_read_sysfs (core_only=false) at util/pmus.c:179
 #5  perf_pmus__find_by_type (type=0) at util/pmus.c:238
 #6  perf_pmus__find_by_type (type=type@entry=0) at util/pmus.c:231
 #7  0x000002aa0012ee4a in parse_events_add_numeric (parse_state=0x3ffffff7718,
        parse_state@entry=0x100000000,
        list=0x2aa003d47b0, list@entry=<error reading variable:
                value has been optimized out>, type=type@entry=0,
    config=<optimized out>, head_config=<optimized out>, head_config@entry=0x0, wildcard=true)
    at util/parse-events.c:1347
 #8  0x000002aa0017b7c2 in parse_events_parse (_parse_state=0x100000000,
        _parse_state@entry=0x3ffffff7718, scanner=<optimized out>) at util/parse-events.y:418
 #9  0x000002aa0012ca6e in parse_events__scanner (parse_state=0x3ffffff7718, input=0x0,
        str=0x3ffffffa58a "cycles") at util/parse-events.c:1822
 #10 __parse_events (evlist=0x3ffffff7830, str=str@entry=0x3ffffffa58a "cycles",
        pmu_filter=<optimized out>, err=err@entry=0x3ffffff7830, fake_pmu=fake_pmu@entry=0x0,
        warn_if_reordered=true) at util/parse-events.c:2094
 ....rest omitted.....


Variable table is not assigned any value and remains a NULL pointer.
This happens because on a z/VM vertual machines (on s390) there
is not CPU measurement facility and the directories
   /sys/devices/cpum_cf
   /sys/devices/cpum_sf
do not exist. Therefore the event cycles as treated as software event
and this default does work with the latest pmu-events rework.

Note: This core-dump will also happen on an LPAR when the CPU Measurement
facility is not configured. Then above PMU directory do not exist on
an LPAR.

The other PMU for s390 are
   /sys/devices/pai_crypto
   /sys/devices/pai_ext
do exist on z/VM and LPAR on s390 (when configured).

Do you have an idea how to fix this?
Since the pmu-event subtree was completely redesigned, I would
like some guidance on how to proceed.

Thanks a lot for your help.

-- 
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: David Faller
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294

Re: perf record dumps core all the time on s390 (6.6.0rc1) part2

[Thomas Richter]

On 9/13/23 11:17, Thomas Richter wrote:
> Using Linux kernel version 6.6.0rc1, the command
> 
>   # perf record -e cycles
> 
> dumps core immediately on s390 running on a z/VM virtual machine.
> It works on an s390 LPAR (same kernel).
> 
> This issue showed up during my vacation on 25-Aug-2023 in linux-next
> and since Monday 11-Sep-2023 it shows up in linux 6.6.0rc1.
> 
> The event cycles is treated as a software event and it fails in:
> 
>  (gdb) n
>  1014           pmu->events_table = perf_pmu__find_events_table(pmu);
>  (gdb) p *pmu
>   $4 = {name = 0x152a150 "software", alias_name = 0x0, id = 0x0, type = 1,
>           selectable = false, is_core = false,
>   is_uncore = false, auxtrace = false, formats_checked = false,
>   config_masks_present = false,
>   config_masks_computed = false, max_precise = -1, default_config = 0x0, cpus = 0x0,
>   format = {next = 0x152a0c8, prev = 0x152a0c8},
>   aliases = {next = 0x152a0d8, prev = 0x152a0d8}, events_table = 0x0, sysfs_aliases = 0,
>   loaded_json_aliases = 0, sysfs_aliases_loaded = false,
>   cpu_aliases_added = false, caps_initialized = false,
>   nr_caps = 0, caps = {next = 0x152a100, prev = 0x152a100},
>   list = {next = 0x0, prev = 0x0}, config_masks = {0, 0, 0, 0},
>   missing_features = {exclude_guest = false}}
>  (gdb) n
> 
>  Program received signal SIGSEGV, Segmentation fault.
>  0x0000000001344558 in perf_pmu__find_events_table (pmu=0x152a090) at pmu-events/pmu-events.c:1970
>  1970           for (i = 0; i < table->num_pmus; i++) {
>  (gdb) p table
>  $5 = (const struct pmu_events_table *) 0x0
>  (gdb)
> 
> (gdb) where
>  #0  perf_pmu__find_events_table (pmu=pmu@entry=0x2aa003d48d0)
>         at pmu-events/pmu-events.c:1970
>  #1  0x000002aa00180cee in perf_pmu__lookup (pmus=0x2aa0039baf8 <other_pmus>,
>         dirfd=dirfd@entry=3, lookup_name=lookup_name@entry=0x2aa003dca83 "software")
>         at util/pmu.c:1014
>  #2  0x000002aa0018151e in perf_pmu__find2 (name=0x2aa003dca83 "software",
>         dirfd=<optimized out>) at util/pmus.c:150
>  #3  pmu_read_sysfs (core_only=false) at util/pmus.c:198
>  #4  0x000002aa00181e04 in pmu_read_sysfs (core_only=false) at util/pmus.c:179
>  #5  perf_pmus__find_by_type (type=0) at util/pmus.c:238
>  #6  perf_pmus__find_by_type (type=type@entry=0) at util/pmus.c:231
>  #7  0x000002aa0012ee4a in parse_events_add_numeric (parse_state=0x3ffffff7718,
>         parse_state@entry=0x100000000,
>         list=0x2aa003d47b0, list@entry=<error reading variable:
>                 value has been optimized out>, type=type@entry=0,
>     config=<optimized out>, head_config=<optimized out>, head_config@entry=0x0, wildcard=true)
>     at util/parse-events.c:1347
>  #8  0x000002aa0017b7c2 in parse_events_parse (_parse_state=0x100000000,
>         _parse_state@entry=0x3ffffff7718, scanner=<optimized out>) at util/parse-events.y:418
>  #9  0x000002aa0012ca6e in parse_events__scanner (parse_state=0x3ffffff7718, input=0x0,
>         str=0x3ffffffa58a "cycles") at util/parse-events.c:1822
>  #10 __parse_events (evlist=0x3ffffff7830, str=str@entry=0x3ffffffa58a "cycles",
>         pmu_filter=<optimized out>, err=err@entry=0x3ffffff7830, fake_pmu=fake_pmu@entry=0x0,
>         warn_if_reordered=true) at util/parse-events.c:2094
>  ....rest omitted.....
> 
> 
> Variable table is not assigned any value and remains a NULL pointer.
> This happens because on a z/VM vertual machines (on s390) there
> is not CPU measurement facility and the directories
>    /sys/devices/cpum_cf
>    /sys/devices/cpum_sf
> do not exist. Therefore the event cycles as treated as software event
> and this default does work with the latest pmu-events rework.
> 
> Note: This core-dump will also happen on an LPAR when the CPU Measurement
> facility is not configured. Then above PMU directory do not exist on
> an LPAR.
> 
> The other PMU for s390 are
>    /sys/devices/pai_crypto
>    /sys/devices/pai_ext
> do exist on z/VM and LPAR on s390 (when configured).
> 
> Do you have an idea how to fix this?
> Since the pmu-event subtree was completely redesigned, I would
> like some guidance on how to proceed.
> 
> Thanks a lot for your help.
> 


When I use software events like cpu-clock or cs, the commands

 # ./perf stat -e cs -- true
 Segmentation fault (core dumped)
 # ./perf stat -e cpu-clock-- true
 Segmentation fault (core dumped)
 #

also dump core. This should not happen as these events are defined
even when no hardware PMU is available.
Debugging this reveals this call chain:

  perf_pmus__find_by_type(type=1)
  +--> pmu_read_sysfs(core_only=false)
       +--> perf_pmu__find2(dirfd=3, name=0x152a113 "software")
            +--> perf_pmu__lookup(pmus=0x14f0568 <other_pmus>, dirfd=3,
                                  lookup_name=0x152a113 "software")
                 +--> perf_pmu__find_events_table (pmu=0x1532130)

Now the pmu is "software" and it tries to find a proper table
generated by the pmu-event generation process for s390:

 # cd pmu-events/
 # ./jevents.py  s390 all /root/linux/tools/perf/pmu-events/arch |\
        grep -E '^const struct pmu_table_entry'
 const struct pmu_table_entry pmu_events__cf_z10[] = {
 const struct pmu_table_entry pmu_events__cf_z13[] = {
 const struct pmu_table_entry pmu_metrics__cf_z13[] = {
 const struct pmu_table_entry pmu_events__cf_z14[] = {
 const struct pmu_table_entry pmu_metrics__cf_z14[] = {
 const struct pmu_table_entry pmu_events__cf_z15[] = {
 const struct pmu_table_entry pmu_metrics__cf_z15[] = {
 const struct pmu_table_entry pmu_events__cf_z16[] = {
 const struct pmu_table_entry pmu_metrics__cf_z16[] = {
 const struct pmu_table_entry pmu_events__cf_z196[] = {
 const struct pmu_table_entry pmu_events__cf_zec12[] = {
 const struct pmu_table_entry pmu_metrics__cf_zec12[] = {
 const struct pmu_table_entry pmu_events__test_soc_cpu[] = {
 const struct pmu_table_entry pmu_metrics__test_soc_cpu[] = {
 const struct pmu_table_entry pmu_events__test_soc_sys[] = {
 #

However event "software" is not listed, as can be seen in the
generated const struct pmu_events_map pmu_events_map[].

So in function perf_pmu__find_events_table(), there variable
table is initialized to NULL, but never set to a proper
value. The function scans all generated &pmu_events_map[]
tables, but no table matches, because the tables are
s390 CPU Measurement unit specific:


        i = 0;
        for (;;) {
                const struct pmu_events_map *map = &pmu_events_map[i++];
                if (!map->arch)
                        break;

        --> the maps are there because the build generated them

                if (!strcmp_cpuid_str(map->cpuid, cpuid)) {
                        table = &map->event_table;
                        break;
                }
        --> Since no matching CPU string the table var remains 0x0
        }
        free(cpuid);
        if (!pmu)
                return table;

        --> The pmu is "software" so it exists and no return

        for (i = 0; i < table->num_pmus; i++) {
                const struct pmu_table_entry *table_pmu = &table->pmus[i];
        --> and here perf dies because table is 0x0

To me something is missing, either generate a pmu_events_map[] for software
events or return and handle these events somewhere else.

The events cpu-clock or cs are completely software driven and do not
need any CPU measurement hardware at all.
Something is very strange and I am confused.
Does this work on other platforms?

Thanks 

-- 
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: David Faller
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294