[PATCH v5 00/24] ARM64 PMU Partitioning

[Colton Lewis <coltonlewis@google.com>]

This series creates a new PMU scheme on ARM, a partitioned PMU that
allows reserving a subset of counters for more direct guest access,
significantly reducing overhead. More details, including performance
benchmarks, can be read in the v1 cover letter linked below.

An overview of what this series accomplishes was presented at KVM
Forum 2025. Slides [1] and video [2] are linked below.

The long duration between v4 and v5 is due to time spent on this
project being monopolized preparing this feature for internal
production. As a result, there are too many improvements to fully list
here, but I will cover the notable ones.

v5:

* Rebase onto v6.18-rc7. This required pulling some reorganization
  patches from Anish and Sean that were dependencies from previous
  versions based on kvm/queue but never made it to upstream.

* Ensure FGTs (fine-grained traps) are correctly programmed at vCPU
  load using kvm_vcpu_load_fgt() and helpers introduced by Oliver
  Upton.

* Cleanly separate concerns of whether the partitioned PMU is enabled
  for the guest and whether FGT should be enabled. This allows that
  the capability can be VM-scoped while the implementation detail of
  whether FGT and context switching are in effect can remain
  vCPU-scoped.

* Shrink the uAPI change. Instead of a cap and corresponding ioctl,
  the feature can be controlled by just a cap with an argument. The
  cap is now also VM-scoped and enforces ordering that it should be
  decided before vCPUs are created. Whether the cap is enabled is now
  tracked by the new flag KVM_ARCH_ARM_PARTITIONED_PMU_ENABLED.

* Improve log messages when partitioning in the PMUv3 driver.

* Introduce a global variable armv8pmu_hpmn_max in the PMUv3 driver so
  KVM code can read if a value was set before the PMU is probed. This
  is needed to properly test if we have the capability before vCPUs
  are created.

* Make it possible for a VMM to filter the HPMN0 feature bit.

* Fix event filter problems with PMEVTYPER handling in
  writethrough_pmevtyper() and kvm_pmu_apply_event_filter() by using
  kvm_pmu_event_mask() in the right spots. And if an event is
  filtered, write the physical register with the appropriate exclude
  bits set but keep the virtual register exactly what the guest wrote.

* Fix register access problems with the PMU register fast path handler
  by lifting some static PMU access checks from sys_regs.c to use them
  in the fast path too and make bit masking more strict for better ARM
  compliance.

* Fix the readability and logic of programming the MDCR_EL2 register
  when entering the guest. Make sure to set the HPME bit to allow host
  counters to count guest events. Set TPM and TPMCR by default and
  clear them if partitioning is enabled rather than the previous
  inverted logic of leaving them clear and setting them if
  partitioning is not enabled. Make the HPMN field computation more
  clear.

* As part of lazy context switching, do a load when the guest is
  switching to physical access to ensure any previous writes that only
  reached the virtual registers reach the physical ones as well and
  are not clobbered by the next vcpu_put().

* Other fixes and improvements that are too small to mention or left
  out from my personal notes.

v4:
https://lore.kernel.org/kvmarm/20250714225917.1396543-1-coltonlewis@google.com/

v3:
https://lore.kernel.org/kvm/20250626200459.1153955-1-coltonlewis@google.com/

v2:
https://lore.kernel.org/kvm/20250620221326.1261128-1-coltonlewis@google.com/

v1:
https://lore.kernel.org/kvm/20250602192702.2125115-1-coltonlewis@google.com/

[1] https://gitlab.com/qemu-project/kvm-forum/-/raw/main/_attachments/2025/Optimizing__itvHkhc.pdf
[2] https://www.youtube.com/watch?v=YRzZ8jMIA6M&list=PLW3ep1uCIRfxwmllXTOA2txfDWN6vUOHp&index=9

Anish Ghulati (1):
  KVM: arm64: Move arm_{psci,hypercalls}.h to an internal KVM path

Colton Lewis (20):
  arm64: cpufeature: Add cpucap for HPMN0
  KVM: arm64: Reorganize PMU functions
  perf: arm_pmuv3: Introduce method to partition the PMU
  perf: arm_pmuv3: Generalize counter bitmasks
  perf: arm_pmuv3: Keep out of guest counter partition
  KVM: arm64: Set up FGT for Partitioned PMU
  KVM: arm64: Writethrough trapped PMEVTYPER register
  KVM: arm64: Use physical PMSELR for PMXEVTYPER if partitioned
  KVM: arm64: Writethrough trapped PMOVS register
  KVM: arm64: Write fast path PMU register handlers
  KVM: arm64: Setup MDCR_EL2 to handle a partitioned PMU
  KVM: arm64: Account for partitioning in PMCR_EL0 access
  KVM: arm64: Context swap Partitioned PMU guest registers
  KVM: arm64: Enforce PMU event filter at vcpu_load()
  KVM: arm64: Implement lazy PMU context swaps
  perf: arm_pmuv3: Handle IRQs for Partitioned PMU guest counters
  KVM: arm64: Inject recorded guest interrupts
  KVM: arm64: Add KVM_CAP to partition the PMU
  KVM: selftests: Add find_bit to KVM library
  KVM: arm64: selftests: Add test case for partitioned PMU

Marc Zyngier (1):
  KVM: arm64: Reorganize PMU includes

Sean Christopherson (2):
  KVM: arm64: Include KVM headers to get forward declarations
  KVM: arm64: Move ARM specific headers in include/kvm to arch directory

 Documentation/virt/kvm/api.rst                |  24 +
 arch/arm/include/asm/arm_pmuv3.h              |  28 +
 arch/arm64/include/asm/arm_pmuv3.h            |  61 +-
 .../arm64/include/asm/kvm_arch_timer.h        |   2 +
 arch/arm64/include/asm/kvm_host.h             |  24 +-
 .../arm64/include/asm/kvm_pmu.h               | 142 ++++
 arch/arm64/include/asm/kvm_types.h            |   7 +-
 .../arm64/include/asm/kvm_vgic.h              |   0
 arch/arm64/kernel/cpufeature.c                |   8 +
 arch/arm64/kvm/Makefile                       |   2 +-
 arch/arm64/kvm/arch_timer.c                   |   5 +-
 arch/arm64/kvm/arm.c                          |  23 +-
 {include => arch/arm64}/kvm/arm_hypercalls.h  |   0
 {include => arch/arm64}/kvm/arm_psci.h        |   0
 arch/arm64/kvm/config.c                       |  34 +-
 arch/arm64/kvm/debug.c                        |  31 +-
 arch/arm64/kvm/guest.c                        |   2 +-
 arch/arm64/kvm/handle_exit.c                  |   2 +-
 arch/arm64/kvm/hyp/Makefile                   |   6 +-
 arch/arm64/kvm/hyp/include/hyp/switch.h       | 211 ++++-
 arch/arm64/kvm/hyp/nvhe/switch.c              |   4 +-
 arch/arm64/kvm/hyp/vhe/switch.c               |   4 +-
 arch/arm64/kvm/hypercalls.c                   |   4 +-
 arch/arm64/kvm/pmu-direct.c                   | 464 +++++++++++
 arch/arm64/kvm/pmu-emul.c                     | 678 +---------------
 arch/arm64/kvm/pmu.c                          | 726 ++++++++++++++++++
 arch/arm64/kvm/psci.c                         |   4 +-
 arch/arm64/kvm/pvtime.c                       |   2 +-
 arch/arm64/kvm/reset.c                        |   3 +-
 arch/arm64/kvm/sys_regs.c                     | 110 +--
 arch/arm64/kvm/trace_arm.h                    |   2 +-
 arch/arm64/kvm/trng.c                         |   2 +-
 arch/arm64/kvm/vgic/vgic-debug.c              |   2 +-
 arch/arm64/kvm/vgic/vgic-init.c               |   2 +-
 arch/arm64/kvm/vgic/vgic-irqfd.c              |   2 +-
 arch/arm64/kvm/vgic/vgic-kvm-device.c         |   2 +-
 arch/arm64/kvm/vgic/vgic-mmio-v2.c            |   2 +-
 arch/arm64/kvm/vgic/vgic-mmio-v3.c            |   2 +-
 arch/arm64/kvm/vgic/vgic-mmio.c               |   4 +-
 arch/arm64/kvm/vgic/vgic-v2.c                 |   2 +-
 arch/arm64/kvm/vgic/vgic-v3-nested.c          |   3 +-
 arch/arm64/kvm/vgic/vgic-v3.c                 |   2 +-
 arch/arm64/kvm/vgic/vgic-v5.c                 |   2 +-
 arch/arm64/tools/cpucaps                      |   1 +
 arch/arm64/tools/sysreg                       |   6 +-
 drivers/perf/arm_pmuv3.c                      | 137 +++-
 include/linux/perf/arm_pmu.h                  |   1 +
 include/linux/perf/arm_pmuv3.h                |  14 +-
 include/uapi/linux/kvm.h                      |   1 +
 tools/include/uapi/linux/kvm.h                |   1 +
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/arm64/vpmu_counter_access.c |  77 +-
 tools/testing/selftests/kvm/lib/find_bit.c    |   1 +
 53 files changed, 2049 insertions(+), 831 deletions(-)
 rename include/kvm/arm_arch_timer.h => arch/arm64/include/asm/kvm_arch_timer.h (98%)
 rename include/kvm/arm_pmu.h => arch/arm64/include/asm/kvm_pmu.h (61%)
 rename include/kvm/arm_vgic.h => arch/arm64/include/asm/kvm_vgic.h (100%)
 rename {include => arch/arm64}/kvm/arm_hypercalls.h (100%)
 rename {include => arch/arm64}/kvm/arm_psci.h (100%)
 create mode 100644 arch/arm64/kvm/pmu-direct.c
 create mode 100644 tools/testing/selftests/kvm/lib/find_bit.c


base-commit: ac3fd01e4c1efce8f2c054cdeb2ddd2fc0fb150d
--
2.52.0.239.gd5f0c6e74e-goog