armv8pmu: Pending overflow interrupt is discarded when perf event is disabled

["Ashley, William" <wash@amazon.com>]

[-- Attachment #1: Type: text/plain, Size: 1695 bytes --]

An issue [1] was opened in the rr-debugger project reporting occasional missed
perf event overflow signals on arm64. I've been digging into this and think I
understand what's happening, but wanted to confirm my understanding.

The attached example application, derived from an rr-debugger test case, reports
when the value of a counter doesn't increase by the expected period +/- some
tolerance. When it is ping-ponged between cores (e.g. with taskset) at a high
frequency, it frequently reports increases of ~2x the expected. I've confirmed
this same behavior on kernels 5.4, 5.10, 6.1 and 6.5.

I found armv8pmu_disable_intens [2] that is called as part of event
de-scheduling and contains
    /* Clear the overflow flag in case an interrupt is pending. */
    write_pmovsclr(mask);
which results in any pending overflow interrupt being dropped. I added some
debug output here and indeed there is a correlation of this bit being high at
the point of the above code and the reproducer identifying a missed signal.

This behavior does not occur with pseudo-NMIs (irqchip.gicv3_pseudo_nmi=1)
enabled.

When an event is not being explicitly torn down (e.g. being closed), this seems
like an undesirable behavior. I haven't attempted to demo it yet, but I suspect
an application disabling an event temporarily could occasionally see the same
missed overflow signals. Is my understanding here correct? Does anyone have
thoughts on how this could be addressed without creating other issues?

[1] https://github.com/rr-debugger/rr/issues/3607
[2] https://github.com/torvalds/linux/blob/c42d9eeef8e5ba9292eda36fd8e3c11f35ee065c/drivers/perf/arm_pmuv3.c#L652C20-L652C43



[-- Attachment #2: counter-overflow-test-aarch64.c --]
[-- Type: application/octet-stream, Size: 2103 bytes --]

#include <errno.h>
#include <fcntl.h>
#include <linux/perf_event.h>
#include <signal.h>
#include <inttypes.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <stdbool.h>

#define CHECK(condition) \
  do { \
    if (!(condition)) { \
      fprintf(stderr, "CHECK '%s' failed at %s:%d (errno=%d)\n", #condition, __FILE__, __LINE__, errno); \
      abort(); \
    } \
  } while (0)

const static uint64_t period = 10000;
const static uint32_t period_tolerance = 1000;

static int counter_fd;
static uint64_t signal_count = 0;
static uint64_t last_value = 0;

void sighandler(int sig) {
  signal_count++;

  uint64_t cur_value;
  CHECK(sizeof(cur_value) == read(counter_fd, &cur_value, sizeof(cur_value)));
  uint64_t diff = cur_value - last_value;
  if (diff > period + period_tolerance || diff < period - period_tolerance) {
    printf("Signal #%" PRIu64 ": delta of %" PRIu64 " is outside %" PRIu64 " +/- %" PRIu32 "\n",
           signal_count, diff, period, period_tolerance);
  }
  last_value = cur_value;
}

int main(int argc, const char** argv) {
  struct perf_event_attr perf_attr;
  memset(&perf_attr, 0, sizeof(perf_attr));
  perf_attr.type = PERF_TYPE_RAW;
  perf_attr.size = sizeof(perf_attr);
  perf_attr.config = 0x21; // Branches
  perf_attr.exclude_kernel = 1;
  perf_attr.sample_period = 0x1000000000000000LL; // Max period

  counter_fd = syscall(__NR_perf_event_open, &perf_attr, 0, -1, -1, 0);
  CHECK(counter_fd >= 0);
  CHECK(0 == fcntl(counter_fd, F_SETOWN, getpid()));
  CHECK(0 == fcntl(counter_fd, F_SETFL, O_ASYNC));
  signal(SIGIO, sighandler);

  printf("Pid %d running with period %" PRIu64 " tolerance %" PRIu32 "\n",
         getpid(), period, period_tolerance);

  CHECK(0 == ioctl(counter_fd, PERF_EVENT_IOC_DISABLE, 0));
  CHECK(0 == ioctl(counter_fd, PERF_EVENT_IOC_RESET, 0));
  CHECK(0 == ioctl(counter_fd, PERF_EVENT_IOC_PERIOD, &period));
  CHECK(0 == ioctl(counter_fd, PERF_EVENT_IOC_ENABLE, 0));

  // This is where we create the branches being counted
  while (true) {
  }
}