Kernel KVM virtualization development
 help / color / mirror / Atom feed
* [PATCH v10 00/33] ARM64 PMU Partitioning
@ 2026-11-19 18:51 Colton Lewis
  2026-11-19 18:51 ` [PATCH v10 01/33] arm64: cpufeature: Add cpucap for HPMN0 Colton Lewis
                   ` (32 more replies)
  0 siblings, 33 replies; 36+ messages in thread
From: Colton Lewis @ 2026-11-19 18:51 UTC (permalink / raw)
  To: Marc Zyngier, Oliver Upton
  Cc: Catalin Marinas, Will Deacon, Mark Rutland, Peter Zijlstra,
	Ingo Molnar, Paolo Bonzini, Russell King, Jonathan Corbet,
	Fuad Tabba, Joey Gouly, Suzuki K Poulose, Zenghui Yu, Shuah Khan,
	James Clark, Alexandru Elisei, Mingwei Zhang, Ganapatrao Kulkarni,
	kvmarm, linux-arm-kernel, kvm, linux-perf-users, linux-kselftest,
	linux-doc, linux-kernel

This series creates a new PMU scheme on ARM, a partitioned PMU that
allows reserving a subset of counters for more direct guest access,
significantly reducing overhead. More details, including performance
benchmarks, can be read in the v1 cover letter linked below.

PMU partitioning is controlled via the KVM API using
`KVM_ARM_VCPU_PMU_V3_ENABLE_PARTITION` and
`KVM_ARM_VCPU_PMU_V3_SET_NR_COUNTERS` vCPU device attributes. When
partitioning is enabled for a VM, userspace must explicitly configure a
guest event counter count strictly less than the maximum general-purpose
counters implemented by the PMU (leaving at least one general-purpose
counter for the host) prior to calling `KVM_ARM_VCPU_PMU_V3_INIT`. The
number of general-purpose counters implemented by each host PMU is
exposed in sysfs at
`/sys/bus/event_source/devices/<pmu>/caps/num_counters`. A QEMU patch
demonstrating how to use the uAPI is sent separately.

An overview of what this series accomplishes was presented at KVM
Forum 2025. Slides [1] and video [2] are linked below.

v10:

* Rebase on top of v7.4-rc3.

* Add nested virtualization (`FEAT_NV` / `FEAT_NV2`) support covering
  both an Emulated L2 PMU (`vHPMN == N_L1`, `TPM == 1`, where L1 keeps
  all `N_L1` counters and emulates L2 via L1 `perf` events) and a
  Sub-partitioned L2 PMU (`vHPMN < N_L1`, `TPM == 0`, creating a
  three-way split across `0..vHPMN-1` for L2, `vHPMN..N_L1-1` for L1,
  and `N_L1..N_phys-1` for L0) (Oliver Upton). Because ARM64 provides
  only a single physical `MDCR_EL2.HPMN` field, hardware can split the
  PMU into two counter ranges at a time rather than three. Key design
  decisions and supporting changes:

  - Keep physical `MDCR_EL2.HPMN = N_L1` at all times (`vHPMN` boundary
    and event filtering): Lowering physical `HPMN` to `vHPMN` while
    running L2 was considered and rejected because it would move L1's
    hypervisor counters (`vHPMN..N_L1-1`) into the same physical
    `>= HPMN` range as L0's host counters (`N_L1..N_phys-1`), where a
    single set of physical `MDCR_EL2.{HPME,HPMD,HLP}` bits governs both
    and breaks host/guest isolation; it would also fail for
    `vHPMN == 0` on hardware without `FEAT_HPMN0`. Instead, when
    `vHPMN < N_L1`, L0 keeps physical `HPMN = N_L1` and sets
    `MDCR_EL2.{TPM,TPMCR}` in L2, forwarding `vHPMN..N_L1-1` traps to
    L1 and handling `0..vHPMN-1` plus `PMCCNTR_EL0` in L0 directly
    against physical registers. This preserves L0 isolation and
    supports `vHPMN == 0` on all hardware; L2 events still count
    directly in hardware without L0 `perf_event` scheduling, at the
    cost of trapping L2 PMU accesses to L0 and rewriting per-counter
    filter registers on `L1 <-> L2` transitions. In L1
    (`is_hyp_ctxt()`), L0 emulates `vHPMN`, `vMDCR_EL2.HPMD`, and
    `vMDCR_EL2.HCCD` via `PMEVTYPER<n>_EL0` and `PMCCFILTR_EL0`.

  - Dynamically multiplex `vPMCR_EL0.E` and `vMDCR_EL2.HPME`: Because
    both `0..vHPMN-1` and `vHPMN..N_L1-1` sit below physical
    `HPMN = N_L1`, hardware gates both with physical `PMCR_EL0.E`.
    Rather than unconditionally trapping `PMCR_EL0` and `PMCNTEN*` for
    all `FEAT_NV` VMs (which would penalize Emulated L2 PMU where
    `vHPMN == N_L1`), L0 dynamically enables `MDCR_EL2.TPMCR` and
    `HDFG{R,W}TR_EL2.PMCNTEN` only when `vHPMN < N_L1`. L0 keeps
    architectural `PMCR_EL0` and `PMCNTENSET_EL0` in memory, sets
    physical `PMCR_EL0.E` to `vPMCR_EL0.E || vMDCR_EL2.HPME`, masks
    disabled sub-ranges out of physical `PMCNTENSET_EL0`, and gates
    overflow delivery per domain. Trapped `PMSWINC_EL0` writes (also
    enabled via `HDFGWTR_EL2` when `vHPMN < N_L1`) flush live registers
    and reuse `kvm_pmu_software_increment()` to prevent cross-partition
    chaining.

  - Bias 32-bit counters on mismatched `vPMCR_EL0.LP != vMDCR_EL2.HLP`
    (`FEAT_PMUv3p5`): When `LP == HLP`, physical `PMCR_EL0.LP` matches
    both domains directly and `PMEVCNTR<n>_EL0` stays untrapped in L1.
    When `vHPMN < N_L1` and `LP != HLP`, rather than ignoring the
    mismatch (breaking 32-bit overflow interrupts) or falling back to
    `perf_event` emulation mid-flight (requiring reservation teardown),
    L0 forces physical `PMCR_EL0.LP = 1` (64-bit overflow), dynamically
    traps `HDFG{R,W}TR_EL2.PMEVCNTRn_EL0`, and biases counters in the
    32-bit sub-range `2^32` events below 64-bit overflow
    (`0xffffffff00000000 | low32`) while tracking their upper-32-bit
    offsets and re-biasing outside interrupt/NMI context. Because
    `HDFG{R,W}TR_EL2.PMEVCNTRn_EL0` is a single trap bit covering all
    event counters (`0..30`), L1 counter reads/writes trap across both
    sub-ranges while `LP != HLP`, with zero overhead when `LP == HLP`.

  - Baseline nested support (both Emulated and Sub-partitioned L2):
    separate implemented and accessible counter masks by using
    `nr_pmu_counters` directly in `kvm_pmu_implemented_counter_mask()`,
    masking trapped guest reads of `PMCNTEN*`, `PMINTEN*`, and `PMOVS*`
    with `kvm_pmu_accessible_counter_mask()`, and ignoring
    `MDCR_EL2.HPMN` in userspace `KVM_GET_ONE_REG` / `KVM_SET_ONE_REG`
    (`set_pmreg()`, `get_pmreg()`, `get_pmcr()`) so saving/restoring a
    nested vCPU paused below EL2 preserves all VM-implemented counters
    and true `PMCR_EL0.N`; preserve host counter reservations across
    `IN_NESTED_ERET` / `IN_NESTED_EXCEPTION` transitions to avoid host
    `perf` reschedules on emulated L2 traps; read live hardware
    `PMSELR_EL0` in `check_mdcr_hpmn()` when untrapped; and forward
    fine-grained traps for selector-indexed `PMXEVTYPER_EL0` and
    `PMXEVCNTR_EL0` to L1.

* Expose the number of general-purpose counters implemented by the PMU
  in sysfs under `/sys/bus/event_source/devices/<pmu>/caps/num_counters`
  so userspace VMMs can query host PMU counter capacity when configuring
  a partitioned PMU (James Clark).

* Make `__vcpu_rmw_sys_reg()` atomic on the local CPU using
  `try_cmpxchg_local()` to prevent read-modify-write races with GICv3
  pseudo-NMI PMU interrupt handlers (`kvm_pmu_handle_guest_irq()`),
  which are not masked by `local_irq_save()` (Sashiko AI Review).

* Construct `mdcr_el2` in a local variable before assigning
  `vcpu->arch.mdcr_el2` once in `kvm_arm_setup_mdcr_el2()`, and mask
  GICv3 pseudo-NMIs (`local_daif_save()`) across hardware PMU context
  switches and `MDCR_EL2` updates (Sashiko AI Review).

* Prioritize kernel-owned pinned event groups (`is_kernel_event()`)
  ahead of userspace pinned events in `perf_event_groups_cmp()` /
  `perf_less_group_idx()` so kernel-owned per-CPU pinned events (such as
  the hardlockup NMI watchdog if toggled at runtime) are scheduled before
  userspace pinned events when `perf_pmu_resched_update()` reschedules
  events out of guest-reserved counters (James Clark, Sashiko AI Review).

* Move `kvm_vcpu_reload_pmu()` from `arch/arm64/kvm/pmu-emul.c` to
  `arch/arm64/kvm/pmu.c` in `KVM: arm64: Reorganize PMU functions`.

* Add `kvm_pmu_is_partitioned()`, `kvm_pmu_guest_counter_mask()`,
  `kvm_pmu_handle_guest_irq()`, `kvm_pmu_host_start()`, and
  `kvm_pmu_host_stop()` stubs to `arch/arm/include/asm/arm_pmuv3.h` to
  fix 32-bit `ARCH=arm` builds (kernel test robot).

* Allow identical VM-wide PMU device attributes
  (`KVM_ARM_VCPU_PMU_V3_SET_PMU`, `KVM_ARM_VCPU_PMU_V3_SET_NR_COUNTERS`,
  and `KVM_ARM_VCPU_PMU_V3_ENABLE_PARTITION`) to be applied across all
  vCPUs when userspace initializes vCPUs sequentially (`CPU_FOREACH`),
  validate `kvm_arm_pmu_part_hpmn(n) < max_counters` (requiring
  `max_counters >= 2` for `n == 0` without `FEAT_HPMN0`), and restrict
  `kvm->arch.supported_cpus` to `arm_pmu->supported_cpus` when
  partitioning is enabled.

* Centralize partitioned PMU hardware register dispatch under
  `kvm_pmu_direct_reg_read()` / `kvm_pmu_direct_reg_write()` with
  preemption disabled (`kvm_pmu_part_on_cpu()`), guard lazy ownership
  transitions in `kvm_pmu_part_overflow_status()` with
  `scoped_guard(preempt)`, avoid matching `reset_unknown()` poison bits on
  `PMCNTENSET_EL0` / `PMINTENSET_EL1` during `KVM_REQ_RELOAD_PMU`, and
  schedule `KVM_REQ_RELOAD_PMU` on `KVM_SET_ONE_REG` writes so
  `vcpu_put()` does not overwrite userspace register writes (Sashiko AI
  Review).

* Support configuring 0 guest event counters (cycle counter only) on
  Partitioned PMU hosts without `FEAT_HPMN0` whenever `max_counters >= 2`.
  Writing `MDCR_EL2.HPMN = 0` without `FEAT_HPMN0` is architecturally
  `CONSTRAINED UNPREDICTABLE`, which would otherwise prevent userspace
  from creating a cycle-counter-only partitioned guest on CPUs lacking
  `FEAT_HPMN0`. Instead, KVM programs physical `MDCR_EL2.HPMN = 1` via
  `kvm_arm_pmu_part_hpmn()`, reserves and quiesces physical counter 0
  alongside `PMCCNTR_EL0` so host perf continues using counters `1..N-1`,
  keeps `MDCR_EL2.TPMCR = 1` so trapped `PMCR_EL0` reads report `N == 0`,
  and enables fine-grained traps on `PMEVCNTRn_EL0`, `PMCNTEN`,
  `PMINTEN`, and `PMSWINC_EL0` so counter 0 stays completely hidden
  (`UNDEFINED` / RAZ / WI) while `PMCCNTR_EL0` remains untrapped on
  `FEAT_FGT` hosts. Also clamp `MDCR_EL2.HPMN` against `nr_pmu_counters`
  in `kvm_vcpu_read_pmcr()` and `kvm_pmu_direct_pmcr_read()`, and extend
  `vpmu_counter_access` to verify `PMCCFILTR_EL0`, live `PMCCNTR_EL0`
  counting, and trapped `UNDEFINED` / RAZ / WI behavior on counter 0 when
  `n == 0`.

Tool-assisted development note (generated-content.rst):
Several v10 additions (nested virtualization sub-partitioning,
cycle-counter-only support without `FEAT_HPMN0`, the userspace `HPMN`
masking fix, and cover letter text) were drafted and verified with AI
coding assistance (Gemini) and pre-reviewed with Sashiko AI Review. All
patches were reviewed, built (`arm64`, `CONFIG_ARM_PMU=n`, 32-bit `arm`),
and tested on bare-metal Neoverse V2 and AmpereOne hosts (and under QEMU
TCG for the userspace `HPMN` fix).

v9:
https://lore.kernel.org/kvmarm/20260924172928.2110956-1-coltonlewis@google.com/

v8:
https://lore.kernel.org/kvmarm/20260612192909.1153907-1-coltonlewis@google.com/

v7:
https://lore.kernel.org/kvmarm/20260504211813.1804997-1-coltonlewis@google.com/

v6:
https://lore.kernel.org/kvmarm/20260209221414.2169465-1-coltonlewis@google.com/

v5:
https://lore.kernel.org/kvmarm/20251209205121.1871534-1-coltonlewis@google.com/

v4:
https://lore.kernel.org/kvmarm/20250714225917.1396543-1-coltonlewis@google.com/

v3:
https://lore.kernel.org/kvm/20250626200459.1153955-1-coltonlewis@google.com/

v2:
https://lore.kernel.org/kvm/20250620221326.1261128-1-coltonlewis@google.com/

v1:
https://lore.kernel.org/kvm/20250602192702.2125115-1-coltonlewis@google.com/

[1] https://gitlab.com/qemu-project/kvm-forum/-/raw/main/_attachments/2025/Optimizing__itvHkhc.pdf
[2] https://www.youtube.com/watch?v=YRzZ8jMIA6M&list=PLW3ep1uCIRfxwmllXTOA2txfDWN6vUOHp&index=9

Colton Lewis (32):
  arm64: cpufeature: Add cpucap for HPMN0
  KVM: arm64: Reorganize PMU functions
  KVM: arm64: Use nr_pmu_counters in kvm_pmu_implemented_counter_mask()
  KVM: arm64: Ignore HPMN for userspace PMU register access
  perf: arm_pmuv3: Generalize counter bitmasks
  perf: arm_pmuv3: Move counter allocation mask to per-CPU struct
    pmu_hw_events
  perf: arm_pmuv3: Check cntr_mask before using pmccntr
  perf: arm_pmuv3: Allocate counter indices from high to low
  perf: arm_pmuv3: Expose number of general-purpose counters in sysfs
  KVM: arm64: Add initial scaffolding for Partitioned PMU
  KVM: arm64: Set up FGT for Partitioned PMU
  KVM: arm64: Make __vcpu_rmw_sys_reg() atomic
  KVM: arm64: Add Partitioned PMU register trap handlers
  KVM: arm64: Construct mdcr_el2 in a local variable in
    kvm_arm_setup_mdcr_el2()
  KVM: arm64: Set up MDCR_EL2 to handle a Partitioned PMU
  KVM: arm64: Context swap Partitioned PMU guest registers
  KVM: arm64: Enforce PMU event filter at vcpu_load()
  perf: Add perf_pmu_resched_update()
  perf/core: Prioritize kernel-owned pinned events when scheduling
    groups
  KVM: arm64: Allow kvm_vcpu_pmu_resync_el0() to resync filters in
    process context
  KVM: arm64: Apply dynamic guest counter reservations
  KVM: arm64: Implement lazy PMU context swaps
  perf: arm_pmuv3: Handle IRQs for Partitioned PMU guest counters
  KVM: arm64: Detect overflows for the Partitioned PMU
  KVM: arm64: Multiplex PMCR_EL0.E and MDCR_EL2.HPME for sub-partitioned
    L1
  KVM: arm64: Emulate mismatched LP and HLP overflow widths for
    sub-partitioned L1
  KVM: arm64: Emulate trapped PMSWINC_EL0 writes for Partitioned PMU
  KVM: arm64: Forward FGT traps for PMXEVTYPER_EL0 and PMXEVCNTR_EL0
  KVM: arm64: Support zero event counters on Partitioned PMU without
    HPMN0
  KVM: arm64: Add vCPU device attr to partition the PMU
  KVM: selftests: Add find_bit to KVM library
  KVM: arm64: selftests: Add test case for Partitioned PMU

Marc Zyngier (1):
  KVM: arm64: Reorganize PMU includes

 Documentation/virt/kvm/devices/vcpu.rst       |   57 +-
 arch/arm/include/asm/arm_pmuv3.h              |   12 +
 arch/arm64/include/asm/arm_pmuv3.h            |    7 +-
 arch/arm64/include/asm/kvm_host.h             |   32 +-
 arch/arm64/include/asm/kvm_nested.h           |    2 +-
 arch/arm64/include/asm/kvm_types.h            |    6 +-
 arch/arm64/include/uapi/asm/kvm.h             |    2 +
 arch/arm64/kernel/cpufeature.c                |   11 +-
 arch/arm64/kvm/Makefile                       |    2 +-
 arch/arm64/kvm/arm.c                          |    4 +-
 arch/arm64/kvm/config.c                       |   76 +-
 arch/arm64/kvm/debug.c                        |   76 +-
 arch/arm64/kvm/emulate-nested.c               |   25 +-
 arch/arm64/kvm/nested.c                       |   39 +-
 arch/arm64/kvm/pmu-direct.c                   | 1230 +++++++++++++++++
 arch/arm64/kvm/pmu-emul.c                     |  846 ++----------
 arch/arm64/kvm/pmu.c                          |  831 ++++++++++-
 arch/arm64/kvm/sys_regs.c                     |  199 ++-
 arch/arm64/tools/cpucaps                      |    1 +
 arch/arm64/tools/sysreg                       |    6 +-
 drivers/perf/arm_pmu.c                        |   18 +-
 drivers/perf/arm_pmuv3.c                      |  121 +-
 include/kvm/arm_pmu.h                         |  117 ++
 include/linux/perf/arm_pmu.h                  |    2 +
 include/linux/perf/arm_pmuv3.h                |   14 +-
 include/linux/perf_event.h                    |    3 +
 kernel/events/core.c                          |   61 +-
 tools/include/perf/arm_pmuv3.h                |   12 +-
 tools/testing/selftests/kvm/Makefile.kvm      |    1 +
 .../selftests/kvm/arm64/vpmu_counter_access.c |  217 ++-
 tools/testing/selftests/kvm/lib/find_bit.c    |    2 +
 31 files changed, 3032 insertions(+), 1000 deletions(-)
 create mode 100644 arch/arm64/kvm/pmu-direct.c
 create mode 100644 tools/testing/selftests/kvm/lib/find_bit.c


base-commit: aebb7274476403040d582c61478df8f4a7bf9cd3
-- 
2.56.1.708.g77570a6a1d-goog

^ permalink raw reply	[flat|nested] 36+ messages in thread

end of thread, other threads:[~2026-11-19 19:26 UTC | newest]

Thread overview: 36+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-19 18:51 [PATCH v10 00/33] ARM64 PMU Partitioning Colton Lewis
2026-11-19 18:51 ` [PATCH v10 01/33] arm64: cpufeature: Add cpucap for HPMN0 Colton Lewis
2026-11-19 18:51 ` [PATCH v10 02/33] KVM: arm64: Reorganize PMU includes Colton Lewis
2026-11-19 18:51 ` [PATCH v10 03/33] KVM: arm64: Reorganize PMU functions Colton Lewis
2026-11-19 18:51 ` [PATCH v10 04/33] KVM: arm64: Use nr_pmu_counters in kvm_pmu_implemented_counter_mask() Colton Lewis
2026-11-19 18:51 ` [PATCH v10 05/33] KVM: arm64: Ignore HPMN for userspace PMU register access Colton Lewis
2026-11-19 18:51 ` [PATCH v10 06/33] perf: arm_pmuv3: Generalize counter bitmasks Colton Lewis
2026-11-19 18:51 ` [PATCH v10 07/33] perf: arm_pmuv3: Move counter allocation mask to per-CPU struct pmu_hw_events Colton Lewis
2026-11-19 18:51 ` [PATCH v10 08/33] perf: arm_pmuv3: Check cntr_mask before using pmccntr Colton Lewis
2026-11-19 18:51 ` [PATCH v10 09/33] perf: arm_pmuv3: Allocate counter indices from high to low Colton Lewis
2026-11-19 18:51 ` [PATCH v10 10/33] perf: arm_pmuv3: Expose number of general-purpose counters in sysfs Colton Lewis
2026-11-19 18:51 ` [PATCH v10 11/33] KVM: arm64: Add initial scaffolding for Partitioned PMU Colton Lewis
2026-11-19 18:51 ` [PATCH v10 12/33] KVM: arm64: Set up FGT " Colton Lewis
2026-11-19 18:51 ` [PATCH v10 13/33] KVM: arm64: Make __vcpu_rmw_sys_reg() atomic Colton Lewis
2026-11-19 18:51 ` [PATCH v10 14/33] KVM: arm64: Add Partitioned PMU register trap handlers Colton Lewis
2026-11-19 19:21   ` sashiko-bot
2026-11-19 18:51 ` [PATCH v10 15/33] KVM: arm64: Construct mdcr_el2 in a local variable in kvm_arm_setup_mdcr_el2() Colton Lewis
2026-11-19 18:51 ` [PATCH v10 16/33] KVM: arm64: Set up MDCR_EL2 to handle a Partitioned PMU Colton Lewis
2026-11-19 18:51 ` [PATCH v10 17/33] KVM: arm64: Context swap Partitioned PMU guest registers Colton Lewis
2026-11-19 18:52 ` [PATCH v10 18/33] KVM: arm64: Enforce PMU event filter at vcpu_load() Colton Lewis
2026-11-19 18:52 ` [PATCH v10 19/33] perf: Add perf_pmu_resched_update() Colton Lewis
2026-11-19 18:52 ` [PATCH v10 20/33] perf/core: Prioritize kernel-owned pinned events when scheduling groups Colton Lewis
2026-11-19 18:52 ` [PATCH v10 21/33] KVM: arm64: Allow kvm_vcpu_pmu_resync_el0() to resync filters in process context Colton Lewis
2026-11-19 18:52 ` [PATCH v10 22/33] KVM: arm64: Apply dynamic guest counter reservations Colton Lewis
2026-11-19 18:52 ` [PATCH v10 23/33] KVM: arm64: Implement lazy PMU context swaps Colton Lewis
2026-11-19 18:52 ` [PATCH v10 24/33] perf: arm_pmuv3: Handle IRQs for Partitioned PMU guest counters Colton Lewis
2026-11-19 18:52 ` [PATCH v10 25/33] KVM: arm64: Detect overflows for the Partitioned PMU Colton Lewis
2026-11-19 18:52 ` [PATCH v10 26/33] KVM: arm64: Multiplex PMCR_EL0.E and MDCR_EL2.HPME for sub-partitioned L1 Colton Lewis
2026-11-19 18:52 ` [PATCH v10 27/33] KVM: arm64: Emulate mismatched LP and HLP overflow widths " Colton Lewis
2026-11-19 19:26   ` sashiko-bot
2026-11-19 18:52 ` [PATCH v10 28/33] KVM: arm64: Emulate trapped PMSWINC_EL0 writes for Partitioned PMU Colton Lewis
2026-11-19 18:52 ` [PATCH v10 29/33] KVM: arm64: Forward FGT traps for PMXEVTYPER_EL0 and PMXEVCNTR_EL0 Colton Lewis
2026-11-19 18:52 ` [PATCH v10 30/33] KVM: arm64: Support zero event counters on Partitioned PMU without HPMN0 Colton Lewis
2026-11-19 18:52 ` [PATCH v10 31/33] KVM: arm64: Add vCPU device attr to partition the PMU Colton Lewis
2026-11-19 18:52 ` [PATCH v10 32/33] KVM: selftests: Add find_bit to KVM library Colton Lewis
2026-11-19 18:52 ` [PATCH v10 33/33] KVM: arm64: selftests: Add test case for Partitioned PMU Colton Lewis

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox