Re: [PATCH RFC] KVM: arm64: PMU: Use multiple host PMUs

[Oliver Upton <oliver.upton@linux.dev>]

Hi Akihiko,

On Wed, Mar 19, 2025 at 03:33:46PM +0900, Akihiko Odaki wrote:
> Problem
> -------
> 
> arch/arm64/kvm/pmu-emul.c used to have a comment saying the follows:
> > The observant among you will notice that the supported_cpus
> > mask does not get updated for the default PMU even though it
> > is quite possible the selected instance supports only a
> > subset of cores in the system. This is intentional, and
> > upholds the preexisting behavior on heterogeneous systems
> > where vCPUs can be scheduled on any core but the guest
> > counters could stop working.
> 
> Despite the reference manual says counters may not continuously
> incrementing, Windows is not robust enough to handle stopped PMCCNTR_EL0
> and crashes with a division-by-zero error and it also crashes when the
> PMU is not present.
> 
> To avoid such a problem, the userspace should pin the vCPU threads to
> pCPUs supported by one host PMU when initializing the vCPUs or specify
> the host PMU to use with KVM_ARM_VCPU_PMU_V3_SET_PMU after the
> initialization. However, QEMU/libvirt can pin vCPU threads only after the
> vCPUs are initialized. It also limits the pCPUs the guest can use even
> for VMMs that support proper pinning.
> 
> Solution
> --------
> 
> Ideally, Windows should fix the division-by-zero error and QEMU/libvirt
> should support pinning better, but neither of them are going to happen
> anytime soon.
> 
> To allow running Windows on QEMU/libvirt or with heterogeneous cores,
> combine all host PMUs necessary to cover the cores vCPUs can run and
> keep PMCCNTR_EL0 working.

I'm extremely uneasy about making this a generalized solution. PMUs are
deeply tied to the microarchitecture of a particular implementation, and
that isn't something we can abstract away from the guest in KVM.

For example, you could have an event ID that counts on only a subset of
cores, or better yet an event that counts something completely different
depending on where a vCPU lands.

I do appreciate the issue that you're trying to solve.

The good news though is that the fixed PMU cycle counter is the only
thing guaranteed to be present in any PMUv3 implementation. Since
that's the only counter Windows actually needs, perhaps we could
special-case this in KVM.

I have the following (completely untested) patch, do you want to give it
a try? There's still going to be observable differences between PMUs
(e.g. CPU frequency) but at least it should get things booting.

Thanks,
Oliver

---
diff --git a/arch/arm64/kvm/pmu-emul.c b/arch/arm64/kvm/pmu-emul.c
index a1bc10d7116a..913a7bab50b5 100644
--- a/arch/arm64/kvm/pmu-emul.c
+++ b/arch/arm64/kvm/pmu-emul.c
@@ -724,14 +724,21 @@ static void kvm_pmu_create_perf_event(struct kvm_pmc *pmc)
 		return;
 
 	memset(&attr, 0, sizeof(struct perf_event_attr));
-	attr.type = arm_pmu->pmu.type;
+
+	if (pmc->idx == ARMV8_PMU_CYCLE_IDX) {
+		attr.type = PERF_TYPE_HARDWARE;
+		attr.config = PERF_COUNT_HW_CPU_CYCLES;
+	} else {
+		attr.type = arm_pmu->pmu.type;
+		attr.config = eventsel;
+	}
+
 	attr.size = sizeof(attr);
 	attr.pinned = 1;
 	attr.disabled = !kvm_pmu_counter_is_enabled(pmc);
 	attr.exclude_user = !kvm_pmc_counts_at_el0(pmc);
 	attr.exclude_hv = 1; /* Don't count EL2 events */
 	attr.exclude_host = 1; /* Don't count host events */
-	attr.config = eventsel;
 
 	/*
 	 * Filter events at EL1 (i.e. vEL2) when in a hyp context based on the