[RFC PATCH 0/2] KVM: x86/pmu: Let userspace disable SW accounting of emulated instructions

[Luka Absandze <absandze@amazon.de>]

On hosts that run guests with an *emulated* vPMU -- guest PMU MSR
accesses trap into KVM and each guest counter is backed by a host
perf_event -- the per-emulated-instruction PMU accounting added by commit
9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions") is
pathologically expensive.

Whenever a guest programs a counter for retired instructions (0xc0) or
retired branches (0xc2) -- routine for any profiler-style workload --
every instruction KVM emulates triggers a software increment plus a
batched KVM_REQ_PMU reprogram. The reprogram is drained on the next
VM-entry regardless of which exit preceded it, so ordinary exits (most
importantly the guest's 1kHz timer tick) absorb a full
reprogram_counter() cycle: ctx->mutex, ctx_resched(), and a burst of
serialized PMU-MSR writes. That inflates timer-interrupt handling into
the hundreds of microseconds and, under load, can escalate to a guest
CSD lockup.

This series adds KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING, a VM-scoped
capability that takes a bitmask of PERF_COUNT_HW_* event IDs and makes
KVM skip the software accounting of emulated instructions for those
events. It is opt-in and defaults off; the only cost to an opted-in
guest is that instructions-retired / branches-retired counters
undercount by the instructions KVM emulates in host context, i.e. the
behavior that predates that accounting. Hardware-executed guest
instructions are still counted by the backing perf_event and its
overflow/PMI path is untouched.

  Patch 1 adds the capability and the accounting short-circuit.
  Patch 2 documents it in Documentation/virt/kvm/api.rst.

Measured benefit
================

Host: AMD EPYC 7R13 "Milan", KVM, v7.2-rc4 + this series. Guest: QEMU,
-cpu host,pmu=on, 8 vCPUs. The guest workload keeps an instructions-
retired counter resident on every vCPU and periodically reprograms the
PMU, driving the accounting storm above, while the vCPUs are kept busy
so the 1kHz tick fires. The guest's local-APIC timer interrupt handler
(__sysvec_apic_timer_interrupt) is traced with ftrace function_graph
for 30s (~160k samples per run). The capability is toggled per-VM via
KVM_ENABLE_CAP; nothing else changes between runs.

  Timer-interrupt handler duration:

    metric   accounting ON (baseline)   accounting OFF (CAP)   speedup
    ------   ------------------------   --------------------   -------
    mean            209.8 us                   3.16 us           66x
    p50             256.4 us                   3.14 us           82x
    p99             463.3 us                   4.45 us          104x
    p99.9           915.2 us                   5.03 us          182x
    max             929.7 us                  22.55 us           41x

Some workloads genuinely need the extra precision that the emulated-
instruction accounting provides, and for them the default behavior is
the right trade-off. But for many others the accuracy gained is not
worth the cost it imposes on every exit, and this capability lets those
users opt out of paying it.

Luka Absandze (2):
  KVM: x86/pmu: Add CAP to disable SW accounting of emulated
    instructions
  KVM: Documentation: Document KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING

 Documentation/virt/kvm/api.rst  | 24 ++++++++++++++++++++++++
 arch/x86/include/asm/kvm_host.h |  7 +++++++
 arch/x86/kvm/pmu.c              |  6 ++++++
 arch/x86/kvm/pmu.h              |  9 +++++++++
 arch/x86/kvm/x86.c              | 10 ++++++++++
 include/uapi/linux/kvm.h        |  1 +
 6 files changed, 57 insertions(+)

-- 
2.47.3