Re: [REGRESSION] Perf (userspace) broken on big.LITTLE systems since v6.5

[Marc Zyngier <maz@kernel.org>]

On Thu, 23 Nov 2023 14:23:10 +0000,
Mark Rutland <mark.rutland@arm.com> wrote:
> 
> On Tue, Nov 21, 2023 at 03:24:25PM +0000, Marc Zyngier wrote:
> > On Tue, 21 Nov 2023 13:40:31 +0000,
> > Marc Zyngier <maz@kernel.org> wrote:
> > > 
> > > [Adding key people on Cc]
> > > 
> > > On Tue, 21 Nov 2023 12:08:48 +0000,
> > > Hector Martin <marcan@marcan.st> wrote:
> > > > 
> > > > Perf broke on all Apple ARM64 systems (tested almost everything), and
> > > > according to maz also on Juno (so, probably all big.LITTLE) since v6.5.
> > > 
> > > I can confirm that at least on 6.7-rc2, perf is pretty busted on any
> > > asymmetric ARM platform. It isn't clear what criteria is used to pick
> > > the PMU, but nothing works anymore.
> > > 
> > > The saving grace in my case is that Debian still ships a 6.1 perftool
> > > package, but that's obviously not going to last.
> > > 
> > > I'm happy to test potential fixes.
> > 
> > At Mark's request, I've dumped a couple of perf (as of -rc2) runs with
> > -vvv.  And it is quite entertaining (this is taskset to an 'icestorm'
> > CPU):
> 
> Looking at this with fresh(er) eyes, I think there's a userspace bug here,
> regardless of whether one believes it's correct to convert a named-pmu event to
> a PERF_TYPE_HARDWARE event directed at that PMU.
> 
> It looks like the userspace tool is dropping the extended type ID after an
> initial probe, and requests events with plain PERF_TYPE_HARDWARE (without an
> extended type ID), which explains why we seem to get events from one PMU only.
> 
> More detail below...
> 
> Marc, if you have time, could you run the same commands (on the same kernel)
> with a perf tool build from v6.4?

Here you go:

<quote>
$ sudo taskset -c 0 ./perf stat -vvv -e apple_icestorm_pmu/cycles/ -e  apple_firestorm_pmu/cycles/ -e cycles ls >/dev/null
Using CPUID 0x00000000610f0280
Attempting to add event pmu 'apple_icestorm_pmu' with 'cycles,' that may result in non-fatal errors
After aliases, add event pmu 'apple_icestorm_pmu' with 'event,' that may result in non-fatal errors
Attempting to add event pmu 'apple_firestorm_pmu' with 'cycles,' that may result in non-fatal errors
After aliases, add event pmu 'apple_firestorm_pmu' with 'event,' that may result in non-fatal errors
Control descriptor is not initialized
------------------------------------------------------------
perf_event_attr:
  type                             10
  size                             136
  config                           0x2
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
  disabled                         1
  inherit                          1
  enable_on_exec                   1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid 1624462  cpu -1  group_fd -1  flags 0x8 = 3
------------------------------------------------------------
perf_event_attr:
  type                             11
  size                             136
  config                           0x2
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
  disabled                         1
  inherit                          1
  enable_on_exec                   1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid 1624462  cpu -1  group_fd -1  flags 0x8 = 4
------------------------------------------------------------
perf_event_attr:
  size                             136
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
  disabled                         1
  inherit                          1
  enable_on_exec                   1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid 1624462  cpu -1  group_fd -1  flags 0x8 = 5
apple_icestorm_pmu/cycles/: -1: 1492180 724333 724333
apple_firestorm_pmu/cycles/: -1: 0 724333 0
cycles: -1: 0 724333 0
apple_icestorm_pmu/cycles/: 1492180 724333 724333
apple_firestorm_pmu/cycles/: 0 724333 0
cycles: 0 724333 0

 Performance counter stats for 'ls':

         1,492,180      apple_icestorm_pmu/cycles/                                            
     <not counted>      apple_firestorm_pmu/cycles/                                             (0.00%)
     <not counted>      cycles                                                                  (0.00%)

       0.000001917 seconds time elapsed

       0.000000000 seconds user
       0.000000000 seconds sys
</quote>

and on the other cluster:

<quote>
$ sudo taskset -c 2 ./perf stat -vvv -e apple_icestorm_pmu/cycles/ -e  apple_firestorm_pmu/cycles/ -e cycles ls >/dev/null
Using CPUID 0x00000000610f0280
Attempting to add event pmu 'apple_icestorm_pmu' with 'cycles,' that may result in non-fatal errors
After aliases, add event pmu 'apple_icestorm_pmu' with 'event,' that may result in non-fatal errors
Attempting to add event pmu 'apple_firestorm_pmu' with 'cycles,' that may result in non-fatal errors
After aliases, add event pmu 'apple_firestorm_pmu' with 'event,' that may result in non-fatal errors
Control descriptor is not initialized
------------------------------------------------------------
perf_event_attr:
  type                             10
  size                             136
  config                           0x2
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
  disabled                         1
  inherit                          1
  enable_on_exec                   1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid 1624466  cpu -1  group_fd -1  flags 0x8 = 3
------------------------------------------------------------
perf_event_attr:
  type                             11
  size                             136
  config                           0x2
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
  disabled                         1
  inherit                          1
  enable_on_exec                   1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid 1624466  cpu -1  group_fd -1  flags 0x8 = 4
------------------------------------------------------------
perf_event_attr:
  size                             136
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
  disabled                         1
  inherit                          1
  enable_on_exec                   1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid 1624466  cpu -1  group_fd -1  flags 0x8 = 5
apple_icestorm_pmu/cycles/: -1: 0 593209 0
apple_firestorm_pmu/cycles/: -1: 1038247 593209 593209
cycles: -1: 1037870 593209 593209
apple_icestorm_pmu/cycles/: 0 593209 0
apple_firestorm_pmu/cycles/: 1038247 593209 593209
cycles: 1037870 593209 593209

 Performance counter stats for 'ls':

     <not counted>      apple_icestorm_pmu/cycles/                                              (0.00%)
         1,038,247      apple_firestorm_pmu/cycles/                                           
         1,037,870      cycles                                                                

       0.000001500 seconds time elapsed

       0.000000000 seconds user
       0.000000000 seconds sys
</quote>

For the record, this is on a 6.6-rc6 kernel, userspace perf as of v6.4.0.

	M.

-- 
Without deviation from the norm, progress is not possible.