Re: [PATCH 0/8] KVM: Add support for hardware-switched mediated PMU

["Mi, Dapeng" <dapeng1.mi@linux.intel.com>]

On 8/10/2026 6:14 PM, Sandipan Das wrote:
> Mediated PMU currently saves and restores the guest PMU state in
> software on every world switch. On Zen 5 and later processors, it is
> possible to offload this to the hardware, which automatically saves and
> restores the guest counter state to and from a vendor save area (VMCB)
> during world switches. This feature is called PMC virtualization and
> detailed information about it can be found in Volume 2 of the AMD64
> Architecture Programmer's Manual under Section 15.39.
>
> This series extends the mediated PMU framework with a hardware-switched
> mode and adds the AMD (SVM) implementation, including nested support.
>
> Performance
> -----------
> The difference in the world switch overhead between software-switched
> and hardware-switched modes is measured with the x86/vmexit test from
> kvm-unit-tests on a system with an AMD EPYC 9755 processor. The values
> are in cycles per operation (lower is better), averaged over 5 runs.
>
> ------------------------------------------------------------------------
> test                               sw-switched  hw-switched         gain
> ------------------------------------------------------------------------
> cpuid                                     5333         2592       +51.4%
> vmcall                                   15858         7619       +52.0%
> wbinvd                                    5256         2517       +52.1%
> invd                                      1857         2147       -15.6%
> mov_from_cr8                                18           18         0.0%
> mov_to_cr8                                 157          156        +0.6%
> inl_from_pmtimer                         25934        21176       +18.3%
> inl_from_qemu                            25881        21156       +18.3%
> inl_from_kernel                           5521         2772       +49.8%
> outl_to_kernel                            5522         2778       +49.7%
> mov_dr                                     114          114         0.0%
> self_ipi_sti_nop                          1020         1015        +0.5%
>   ipi self_ipi_sti_nop                     310          307        +1.0%
>   eoi self_ipi_sti_nop                     242          241        +0.4%
> self_ipi_sti_hlt                          2714         2704        +0.4%
>   ipi self_ipi_sti_hlt                     310          307        +1.0%
>   eoi self_ipi_sti_hlt                     242          240        +0.8%
> self_ipi_tpr                              1272         1263        +0.7%
>   ipi self_ipi_tpr                         309          307        +0.6%
>   eoi self_ipi_tpr                         241          240        +0.4%
> self_ipi_tpr_sti_nop                      1304         1295        +0.7%
>   ipi self_ipi_tpr_sti_nop                 309          307        +0.6%
>   eoi self_ipi_tpr_sti_nop                 242          241        +0.4%
> self_ipi_tpr_sti_hlt                      3003         2991        +0.4%
>   ipi self_ipi_tpr_sti_hlt                 309          307        +0.6%
>   eoi self_ipi_tpr_sti_hlt                 242          240        +0.8%
> x2apic_self_ipi_sti_nop                    899          895        +0.4%
>   ipi x2apic_self_ipi_sti_nop              191          190        +0.5%
>   eoi x2apic_self_ipi_sti_nop              242          241        +0.4%
> x2apic_self_ipi_sti_hlt                   2592         2587        +0.2%
>   ipi x2apic_self_ipi_sti_hlt              191          190        +0.5%
>   eoi x2apic_self_ipi_sti_hlt              242          241        +0.4%
> x2apic_self_ipi_tpr                       1159         1151        +0.7%
>   ipi x2apic_self_ipi_tpr                  193          193         0.0%
>   eoi x2apic_self_ipi_tpr                  241          240        +0.4%
> x2apic_self_ipi_tpr_sti_nop               1189         1180        +0.8%
>   ipi x2apic_self_ipi_tpr_sti_nop          193          193         0.0%
>   eoi x2apic_self_ipi_tpr_sti_nop          242          241        +0.4%
> x2apic_self_ipi_tpr_sti_hlt               2887         2875        +0.4%
>   ipi x2apic_self_ipi_tpr_sti_hlt          194          192        +1.0%
>   eoi x2apic_self_ipi_tpr_sti_hlt          242          239        +1.2%
> ple_round_robin                             19           19         0.0%
> wr_kernel_gs_base                           73           73         0.0%
> wr_ibpb_msr                                563          557        +1.1%
> wr_tsc_adjust_msr                         5352         2603       +51.4%
> rd_tsc_adjust_msr                         5330         2586       +51.5%
> toggle_cr0_wp                            11319         5810       +48.7%
> toggle_cr4_pge                           11743         6242       +46.8%
> mmio-no-eventfd:pci-mem                  31890        26835       +15.9%
> mmio-wildcard-eventfd:pci-mem             6461         3666       +43.3%
> mmio-datamatch-eventfd:pci-mem            6449         3671       +43.1%
> portio-no-eventfd:pci-io                 25218        21455       +14.9%
> portio-wildcard-eventfd:pci-io            5512         2774       +49.7%
> portio-datamatch-eventfd:pci-io           5524         2780       +49.7%
> ------------------------------------------------------------------------

Just curious what causes the gain difference between these test cases?
Different vm-exit/vm-entry number in each test case? Suppose the cost for
saving/restoring PMU MSRs on each vm-exit/vm-entry should be same or close.


>
> The largest gains are for wbinvd (+52.1%), vmcall (+52.0%), and cpuid
> (+51.4%), followed by the TSC-adjust MSR accesses (~+51%), the
> kernel-handled port I/O paths (~+50%), and the control-register toggles
> (~+47-49%). The wildcard and datamatch MMIO / port-I/O paths improve by
> ~+43-50%, while the slower no-eventfd MMIO / port-I/O and pmtimer/qemu
> port-I/O paths improve by ~+15-18%.
>
> The only meaningful regression is invd (-15.6%). The self-IPI and EOI
> subtests move by at most 1.2% and wr_kernel_gs_base is unchanged, all
> within run-to-run noise.
>
> Design
> ------
>   * The entity responsible for switching the guest PMU state is tracked
>     per-VM via a new set of mediated PMU capability flags in kvm_arch.
>     In software-switched mode the state resides in struct kvm_pmu; in
>     hardware-switched mode it resides in the vendor save area.
>   * Event filtering, instruction emulation, and intercepted RDPMC still
>     need the state in struct kvm_pmu, so new vendor state-sync PMU ops
>     synchronize specific MSRs between struct kvm_pmu and the vendor
>     save area on demand.
>   * On SVM, the feature depends on VNMI for guest PMI delivery and is
>     controlled by the new "vpmc" kvm_amd module parameter, enabled by
>     default when the host supports it and mediated PMU is enabled.
>
> Limitations
> -----------
>   * Not supported for SEV-ES and SEV-SNP guests: the guest PMU state
>     resides in an encrypted VMSA that is inaccessible to the event
>     filtering logic.
>   * With VNMI and no AVIC, overflow interrupts are always delivered as
>     NMIs because the APIC is emulated and the LVTPC is ignored. This
>     shows up as expected behavioral differences in the x86/pmu test of
>     kvm-unit-tests.
>
> Overview
> --------
> Patches 1-3 extend the mediated PMU framework with hardware-switched
> mode: the capability flags, the vendor state-sync ops, and the generic
> support. Patches 4-6 add the SVM enablement: the PerfCtrVirt feature
> bit, the VMCB save-area fields, and the SVM implementation. Patch 7
> adds nested (nSVM) support to use the feature with L2 guests. Patch 8
> explicitly disables the feature for SEV-ES and SEV-SNP guests.
>
> Tested with kvm-unit-tests (KUT) and perf fuzzer. KUT's x86/pmu test
> requires patching to accept overflow interrupts as NMIs (limitation
> because of VNMI ignoring the LVTPC). It is recommended to apply the
> following patches before testing.
> https://lore.kernel.org/kvm/3bfbf15c22652ba00cf4a16fe9e0a3bfe7071f97.1784096302.git.sandipan.das@amd.com/
> https://lore.kernel.org/kvm/b075ec13e95167da1f2229626713c91d291bb75b.1784097178.git.sandipan.das@amd.com/
> https://lore.kernel.org/all/106bc3a4660653cb7750f3ae07a4dbca42c2762c.1786356229.git.sandipan.das@amd.com/
>
> base-commit: 6cf46b37b0356b3408f8ba949e10462cc71b533d
>              (tag: kvm-x86-next-2026.08.07)
>
> Previous versions can be found at:
> rfc: https://lore.kernel.org/kvm/cover.1762960531.git.sandipan.das@amd.com/
>
> Changes in v1:
>   * Introduce mediated PMU capabilities for hardware PMU virtualization
>     features instead of perf PMU capabilities.
>   * Make VNMI a hard requirement since AVIC can be inhibited.
>   * Introduce new PMU ops to access VMCB PMU state instead of relying
>     on host-initiated {get,set}_msr() ops.
>   * Make VMCB PMU state accessible via KVM_{GET,SET}_MSRS for migration.
>   * Implement support for nested SVM.
>   * Disable support for SEV-ES and SEV-SNP guests explicitly.
>
> Sandipan Das (8):
>   KVM: x86/pmu: Add mediated PMU capability flags
>   KVM: x86/pmu: Add PMU ops for vendor state sync
>   KVM: x86/pmu: Add support for hardware-switched PMU
>   x86/cpufeatures: Add PerfCtrVirt feature bit
>   KVM: SVM: Add VMCB fields for PMC virtualization
>   KVM: SVM: Add support for hardware-switched PMU
>   KVM: nSVM: Add support for hardware-switched PMU
>   KVM: SEV: Disallow the use of hardware-switched PMU
>
>  arch/x86/include/asm/cpufeatures.h     |   1 +
>  arch/x86/include/asm/kvm-x86-pmu-ops.h |   3 +
>  arch/x86/include/asm/kvm_host.h        |  11 +++
>  arch/x86/include/asm/svm.h             |  13 ++-
>  arch/x86/kvm/cpuid.c                   |   1 +
>  arch/x86/kvm/pmu.c                     | 102 +++++++++++++++++----
>  arch/x86/kvm/pmu.h                     |  29 ++++++
>  arch/x86/kvm/svm/nested.c              |  45 +++++++++-
>  arch/x86/kvm/svm/pmu.c                 | 117 +++++++++++++++++++++++++
>  arch/x86/kvm/svm/sev.c                 |   3 +
>  arch/x86/kvm/svm/svm.c                 |  58 +++++++++++-
>  arch/x86/kvm/svm/svm.h                 |  19 ++++
>  arch/x86/kvm/vmx/pmu_intel.c           |   2 +
>  arch/x86/kvm/vmx/vmx.c                 |   3 +-
>  arch/x86/kvm/x86.c                     |   2 +
>  15 files changed, 384 insertions(+), 25 deletions(-)
>