Re: [PATCH 0/8] KVM: Add support for hardware-switched mediated PMU

["Mi, Dapeng" <dapeng1.mi@linux.intel.com>]

On 8/11/2026 4:37 PM, Sandipan Das wrote:
> On 11-08-2026 13:19, Mi, Dapeng wrote:
>> On 8/10/2026 6:14 PM, Sandipan Das wrote:
>>> Mediated PMU currently saves and restores the guest PMU state in
>>> software on every world switch. On Zen 5 and later processors, it is
>>> possible to offload this to the hardware, which automatically saves and
>>> restores the guest counter state to and from a vendor save area (VMCB)
>>> during world switches. This feature is called PMC virtualization and
>>> detailed information about it can be found in Volume 2 of the AMD64
>>> Architecture Programmer's Manual under Section 15.39.
>>>
>>> This series extends the mediated PMU framework with a hardware-switched
>>> mode and adds the AMD (SVM) implementation, including nested support.
>>>
>>> Performance
>>> -----------
>>> The difference in the world switch overhead between software-switched
>>> and hardware-switched modes is measured with the x86/vmexit test from
>>> kvm-unit-tests on a system with an AMD EPYC 9755 processor. The values
>>> are in cycles per operation (lower is better), averaged over 5 runs.
>>>
>>> ------------------------------------------------------------------------
>>> test                               sw-switched  hw-switched         gain
>>> ------------------------------------------------------------------------
>>> cpuid                                     5333         2592       +51.4%
>>> vmcall                                   15858         7619       +52.0%
>>> wbinvd                                    5256         2517       +52.1%
>>> invd                                      1857         2147       -15.6%
>>> mov_from_cr8                                18           18         0.0%
>>> mov_to_cr8                                 157          156        +0.6%
>>> inl_from_pmtimer                         25934        21176       +18.3%
>>> inl_from_qemu                            25881        21156       +18.3%
>>> inl_from_kernel                           5521         2772       +49.8%
>>> outl_to_kernel                            5522         2778       +49.7%
>>> mov_dr                                     114          114         0.0%
>>> self_ipi_sti_nop                          1020         1015        +0.5%
>>>   ipi self_ipi_sti_nop                     310          307        +1.0%
>>>   eoi self_ipi_sti_nop                     242          241        +0.4%
>>> self_ipi_sti_hlt                          2714         2704        +0.4%
>>>   ipi self_ipi_sti_hlt                     310          307        +1.0%
>>>   eoi self_ipi_sti_hlt                     242          240        +0.8%
>>> self_ipi_tpr                              1272         1263        +0.7%
>>>   ipi self_ipi_tpr                         309          307        +0.6%
>>>   eoi self_ipi_tpr                         241          240        +0.4%
>>> self_ipi_tpr_sti_nop                      1304         1295        +0.7%
>>>   ipi self_ipi_tpr_sti_nop                 309          307        +0.6%
>>>   eoi self_ipi_tpr_sti_nop                 242          241        +0.4%
>>> self_ipi_tpr_sti_hlt                      3003         2991        +0.4%
>>>   ipi self_ipi_tpr_sti_hlt                 309          307        +0.6%
>>>   eoi self_ipi_tpr_sti_hlt                 242          240        +0.8%
>>> x2apic_self_ipi_sti_nop                    899          895        +0.4%
>>>   ipi x2apic_self_ipi_sti_nop              191          190        +0.5%
>>>   eoi x2apic_self_ipi_sti_nop              242          241        +0.4%
>>> x2apic_self_ipi_sti_hlt                   2592         2587        +0.2%
>>>   ipi x2apic_self_ipi_sti_hlt              191          190        +0.5%
>>>   eoi x2apic_self_ipi_sti_hlt              242          241        +0.4%
>>> x2apic_self_ipi_tpr                       1159         1151        +0.7%
>>>   ipi x2apic_self_ipi_tpr                  193          193         0.0%
>>>   eoi x2apic_self_ipi_tpr                  241          240        +0.4%
>>> x2apic_self_ipi_tpr_sti_nop               1189         1180        +0.8%
>>>   ipi x2apic_self_ipi_tpr_sti_nop          193          193         0.0%
>>>   eoi x2apic_self_ipi_tpr_sti_nop          242          241        +0.4%
>>> x2apic_self_ipi_tpr_sti_hlt               2887         2875        +0.4%
>>>   ipi x2apic_self_ipi_tpr_sti_hlt          194          192        +1.0%
>>>   eoi x2apic_self_ipi_tpr_sti_hlt          242          239        +1.2%
>>> ple_round_robin                             19           19         0.0%
>>> wr_kernel_gs_base                           73           73         0.0%
>>> wr_ibpb_msr                                563          557        +1.1%
>>> wr_tsc_adjust_msr                         5352         2603       +51.4%
>>> rd_tsc_adjust_msr                         5330         2586       +51.5%
>>> toggle_cr0_wp                            11319         5810       +48.7%
>>> toggle_cr4_pge                           11743         6242       +46.8%
>>> mmio-no-eventfd:pci-mem                  31890        26835       +15.9%
>>> mmio-wildcard-eventfd:pci-mem             6461         3666       +43.3%
>>> mmio-datamatch-eventfd:pci-mem            6449         3671       +43.1%
>>> portio-no-eventfd:pci-io                 25218        21455       +14.9%
>>> portio-wildcard-eventfd:pci-io            5512         2774       +49.7%
>>> portio-datamatch-eventfd:pci-io           5524         2780       +49.7%
>>> ------------------------------------------------------------------------
>> Just curious what causes the gain difference between these test cases?
>> Different vm-exit/vm-entry number in each test case? Suppose the cost for
>> saving/restoring PMU MSRs on each vm-exit/vm-entry should be same or close.
>>
> In this case, the cost is actually lower when hardware handles save-restore
> of PMU MSRs. Hardware can use fastpaths and optimizations which an usual
> RDMSR/WRMSR sequence cannot.

Thanks. My question is why the different test case show so much difference
on the gains, e.g., cpuid has +51%, while invd has -15%. Per my
understanding, the cost of saving/restoring PMU MSRs for most vm-exits
(exclude the fastpath vm-exit) should be fixed even for different vm-exit
reasons. Is it true? 


>
>>> The largest gains are for wbinvd (+52.1%), vmcall (+52.0%), and cpuid
>>> (+51.4%), followed by the TSC-adjust MSR accesses (~+51%), the
>>> kernel-handled port I/O paths (~+50%), and the control-register toggles
>>> (~+47-49%). The wildcard and datamatch MMIO / port-I/O paths improve by
>>> ~+43-50%, while the slower no-eventfd MMIO / port-I/O and pmtimer/qemu
>>> port-I/O paths improve by ~+15-18%.
>>>
>>> The only meaningful regression is invd (-15.6%). The self-IPI and EOI
>>> subtests move by at most 1.2% and wr_kernel_gs_base is unchanged, all
>>> within run-to-run noise.
>>>
>>> Design
>>> ------
>>>   * The entity responsible for switching the guest PMU state is tracked
>>>     per-VM via a new set of mediated PMU capability flags in kvm_arch.
>>>     In software-switched mode the state resides in struct kvm_pmu; in
>>>     hardware-switched mode it resides in the vendor save area.
>>>   * Event filtering, instruction emulation, and intercepted RDPMC still
>>>     need the state in struct kvm_pmu, so new vendor state-sync PMU ops
>>>     synchronize specific MSRs between struct kvm_pmu and the vendor
>>>     save area on demand.
>>>   * On SVM, the feature depends on VNMI for guest PMI delivery and is
>>>     controlled by the new "vpmc" kvm_amd module parameter, enabled by
>>>     default when the host supports it and mediated PMU is enabled.
>>>
>>> Limitations
>>> -----------
>>>   * Not supported for SEV-ES and SEV-SNP guests: the guest PMU state
>>>     resides in an encrypted VMSA that is inaccessible to the event
>>>     filtering logic.
>>>   * With VNMI and no AVIC, overflow interrupts are always delivered as
>>>     NMIs because the APIC is emulated and the LVTPC is ignored. This
>>>     shows up as expected behavioral differences in the x86/pmu test of
>>>     kvm-unit-tests.
>>>
>>> Overview
>>> --------
>>> Patches 1-3 extend the mediated PMU framework with hardware-switched
>>> mode: the capability flags, the vendor state-sync ops, and the generic
>>> support. Patches 4-6 add the SVM enablement: the PerfCtrVirt feature
>>> bit, the VMCB save-area fields, and the SVM implementation. Patch 7
>>> adds nested (nSVM) support to use the feature with L2 guests. Patch 8
>>> explicitly disables the feature for SEV-ES and SEV-SNP guests.
>>>
>>> Tested with kvm-unit-tests (KUT) and perf fuzzer. KUT's x86/pmu test
>>> requires patching to accept overflow interrupts as NMIs (limitation
>>> because of VNMI ignoring the LVTPC). It is recommended to apply the
>>> following patches before testing.
>>> https://lore.kernel.org/kvm/3bfbf15c22652ba00cf4a16fe9e0a3bfe7071f97.1784096302.git.sandipan.das@amd.com/
>>> https://lore.kernel.org/kvm/b075ec13e95167da1f2229626713c91d291bb75b.1784097178.git.sandipan.das@amd.com/
>>> https://lore.kernel.org/all/106bc3a4660653cb7750f3ae07a4dbca42c2762c.1786356229.git.sandipan.das@amd.com/
>>>
>>> base-commit: 6cf46b37b0356b3408f8ba949e10462cc71b533d
>>>              (tag: kvm-x86-next-2026.08.07)
>>>
>>> Previous versions can be found at:
>>> rfc: https://lore.kernel.org/kvm/cover.1762960531.git.sandipan.das@amd.com/
>>>
>>> Changes in v1:
>>>   * Introduce mediated PMU capabilities for hardware PMU virtualization
>>>     features instead of perf PMU capabilities.
>>>   * Make VNMI a hard requirement since AVIC can be inhibited.
>>>   * Introduce new PMU ops to access VMCB PMU state instead of relying
>>>     on host-initiated {get,set}_msr() ops.
>>>   * Make VMCB PMU state accessible via KVM_{GET,SET}_MSRS for migration.
>>>   * Implement support for nested SVM.
>>>   * Disable support for SEV-ES and SEV-SNP guests explicitly.
>>>
>>> Sandipan Das (8):
>>>   KVM: x86/pmu: Add mediated PMU capability flags
>>>   KVM: x86/pmu: Add PMU ops for vendor state sync
>>>   KVM: x86/pmu: Add support for hardware-switched PMU
>>>   x86/cpufeatures: Add PerfCtrVirt feature bit
>>>   KVM: SVM: Add VMCB fields for PMC virtualization
>>>   KVM: SVM: Add support for hardware-switched PMU
>>>   KVM: nSVM: Add support for hardware-switched PMU
>>>   KVM: SEV: Disallow the use of hardware-switched PMU
>>>
>>>  arch/x86/include/asm/cpufeatures.h     |   1 +
>>>  arch/x86/include/asm/kvm-x86-pmu-ops.h |   3 +
>>>  arch/x86/include/asm/kvm_host.h        |  11 +++
>>>  arch/x86/include/asm/svm.h             |  13 ++-
>>>  arch/x86/kvm/cpuid.c                   |   1 +
>>>  arch/x86/kvm/pmu.c                     | 102 +++++++++++++++++----
>>>  arch/x86/kvm/pmu.h                     |  29 ++++++
>>>  arch/x86/kvm/svm/nested.c              |  45 +++++++++-
>>>  arch/x86/kvm/svm/pmu.c                 | 117 +++++++++++++++++++++++++
>>>  arch/x86/kvm/svm/sev.c                 |   3 +
>>>  arch/x86/kvm/svm/svm.c                 |  58 +++++++++++-
>>>  arch/x86/kvm/svm/svm.h                 |  19 ++++
>>>  arch/x86/kvm/vmx/pmu_intel.c           |   2 +
>>>  arch/x86/kvm/vmx/vmx.c                 |   3 +-
>>>  arch/x86/kvm/x86.c                     |   2 +
>>>  15 files changed, 384 insertions(+), 25 deletions(-)
>>>