[PATCH v2 00/36] KVM: VMX APIC timer virtualization support

[isaku.yamahata@intel.com]

From: Isaku Yamahata <isaku.yamahata@intel.com>

This patch series is v2 of VMX APIC timer virtualization support.

qemu patches doesn't need update.
https://lore.kernel.org/kvm/e41ffc4af96b8b4ee3a65f8da1f52dac7f3f4913.1770116952.git.isaku.yamahata@intel.com/

v2 patch for kvm-unit-tests update will be posted.
(v1 https://lore.kernel.org/kvm/7acdd9974effabe5dc461aa755eacf9fb0697467.1770116601.git.isaku.yamahata@intel.com/)

Changes v1 -> v2:
https://lore.kernel.org/kvm/cover.1770116050.git.isaku.yamahata@intel.com/
- Rebased to kvm-x86 and v7.0-rcX.
- Fixed an issue found by syzbot.  Require in-kernel APIC for nested
  VMX APIC timer virtualization.
- Add test case for nested VMX APIC timer virtualization with in-kernel
  APIC.
- Avoid compile time error by clearing the feature bit explicitly instead
  of clearing the bit of KVM_OPTIONAL_VMX_TERTIARY_VM_EXEC_CONTROL.
- Fix compile errors. EXPORT_SYMBOL_FOR_KVM_INTERNAL.
- selftests: Use GUEST_ASSERT_EQ() where meaningful.
- selftests: Use safe_halt(), sti(), cli() instead of raw inline assembly.
- selftests: Require serialize instruction for tests that uses it.

Background
==========
X86 provides the TSC deadline timer as the primary local timer
interrupt source.  Currently, KVM intercepts the guest programming of
the timer and emulates it using either the host OS timer or the VMX
preemption timer.


Problem
=======
VMM emulation causes high latency.  Some workloads require lower
latency, such as gaming applications, while there have been efforts to
reduce latency in the past, a hardware extension can reduce it further
by eliminating VM Exits.


Solution
========
Hardware Extension
------------------
The APIC timer virtualization [1] allows the guest to directly access
the TSC DEADLINE MSR and receive timer interrupts without VM Exits.

It introduces
- A feature bit in the tertiary processor-based VM-execution controls
- Guest deadline: 64-bit physical deadline (host TSC value)
- Guest deadline shadow: 64-bit virtual deadline (virtualized TSC
  value with TSC offset and multiplier)
- Virtual timer vector: interrupt vector to inject on timeout.

Implementation
--------------
Add hooks to the LAPIC timer emulation and implement them in the VMX
backend.  Enable the feature when available, falling back to
software/preemption timer in the following cases
One-shot or periodic APIC timer:
  The hardware supports only the TSC deadline timer
Masked the timer interrupt in LVTT:
  The hardware doesn't respect the emulated LVTT and always generates an
  interrupt on timeout.
vCPU blocking/unblocking:
  The hardware generates an interrupt while the vCPU is running.  The KVM
  must wake up from vCPU blocking by getting the latest TSC
  deadline and setting a software timer before blocking the vCPU.
VM Entry to L2 vCPU:
  If the L1 timer interrupt fires while the L2 vCPU is running, the
  expected behavior is a VM Exit from L2 to L1, followed by an interrupt
  injection into the L1 vCPU.

nVMX Support
------------
Support nVMX to address the benchmark result below.  Emulate related
MSRs and VMCS individually.
MSRs: capability reporting registers of primary/tertiary processor-based
      VM-execution controls.
VMCS fields: primary/tertiary VM-execution controls, guest deadline,
             guest deadline shadow, and virtual timer vector.

Patch Organization
------------------
The patch is organized into 5 parts as follows.

Patches  1- 8: VMX support (feature probe, hooks to KVM LAPIC, VMX hooks)
Patches  9-18: nVMX support (implement emulation of MSR and VMCS fields)
Patches 19-23: Expose the feature to the user
Patches 24-31: KVM selftests
Patches 32   : Documentation update

https://lore.kernel.org/kvm/e41ffc4af96b8b4ee3a65f8da1f52dac7f3f4913.1770116952.git.isaku.yamahata@intel.com/
(KVM unit tests turned out test case issue. It needs fixes.)
https://lore.kernel.org/kvm/7acdd9974effabe5dc461aa755eacf9fb0697467.1770116601.git.isaku.yamahata@intel.com/


Test
====
The following tests were conducted:  The newly added test case as a
part of KVM selftests, KVM unit tests, and cyclic test included in
rt-tests [2].  Selftests and KVM unit tests were run on platforms with
and without APIC timer virtualization.


Benchmark Results
=================
cyclictest
----------
10-minute run of
cyclictest --quiet --nsecs --smp --mlockall --priority=95 --policy=fifo
# of vCPU: host 256, L1 and L2: 16

Legends:
L1 or L2: cyclic test run as L1/L2 process
Y: feature enabled
N: feature disabled

Run in
|       APIC timer virtualization
|       |       nested APIC timer virtualization
|       |       |       min reduction %
|       |       |       |       avg reduction %
|       |       |       |       |
L1	N	-
L1	Y	-	21%	21% (compared to L1 N)

L2	N	N
L2	Y	N	4%	-2% (compared to L2 N N)
L2	Y	Y	75%	51% (compared to L2 N N)

Micro benchmark: Timer latency
------------------------------
10-minute run of custom micro benchmark, timer_latency.
# of vCPU: host 256, L1 and L2: 16

Legends:
L1: the benchmark run in L0 Linux.
L2: the benchmark run in L1 Linux.
Y: feature enabled
N: feature disabled

Run as
|       APIC timer virtualization
|       |       nested APIC timer virtualization
|       |       |       HLT or busy
|       |       |       |       min reduction %
|       |       |       |       |       avg reduction %
|       |       |       |       |       |
L1	N	-	HLT
L1	Y	-	HLT	49%	24% (compared to L1 N HLT)

L1	N	-	busy
L1	Y	-	busy	63%	61% (compared to L1 N busy)

L2	N	N	HLT
L2	Y	N	HLT	-19%	-3% (compared to L2 N N HLT)
L2	Y	Y	HLT	99%	27% (compared to L2 N N HLT)

L2	N	N	busy
L2	Y	N	busy	-5%	-4% (compared to L2 N N busy)
L2	Y	Y	busy	99%	97% (compared to L2 N N busy)


[1] Intel Architecture Instruction Set Extensions and Future Features
September 2025 319433-059
Chapter 8 APIC-TIMER VIRTUALIZATION
https://cdrdv2.intel.com/v1/dl/getContent/671368

[2] rt-tests
https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git/


Isaku Yamahata (29):
  KVM: x86/lapic: Wire DEADLINE MSR update to guest virtual TSC deadline
  KVM: VMX: Update APIC timer virtualization on apicv changed
  KVM: nVMX: Disallow/allow guest APIC timer virtualization switch
    to/from L2
  KVM: nVMX: Pass struct msr_data to VMX MSRs emulation
  KVM: nVMX: Supports VMX tertiary controls and GUEST_APIC_TIMER bit
  KVM: nVMX: Add alignment check for struct vmcs12
  KVM: nVMX: Add tertiary VM-execution control VMCS support
  KVM: nVMX: Update intercept on TSC deadline MSR
  KVM: nVMX: Handle virtual timer vector VMCS field
  KVM: VMX: Make vmx_calc_deadline_l1_to_host() non-static
  KVM: nVMX: Enable guest deadline and its shadow VMCS field
  KVM: nVMX: Add VM entry checks related to APIC timer virtualization
  KVM: nVMX: Add check vmread/vmwrite on tertiary control
  KVM: nVMX: Add check VMCS index for guest timer virtualization
  KVM: VMX: Advertise tertiary controls to the user space
  KVM: VMX: Enable APIC timer virtualization
  KVM: nVMX: Introduce module parameter for nested APIC timer
    virtualization
  KVM: selftests: Add a test to measure local timer latency
  KVM: selftests: Add nVMX support to timer_latency test case
  KVM: selftests: Add test for nVMX MSR_IA32_VMX_PROCBASED_CTLS3
  KVM: selftests: Add test vmx_set_nested_state_test with EVMCS disabled
  KVM: selftests: Add tests nested state of APIC timer virtualization
  KVM: selftests: Add VMCS access test to APIC timer virtualization
  KVM: selftests: Add serialize() helper and X86_FEATURE_SERIALIZE
  KVM: selftests: Test cases for L1 APIC timer virtualization
  KVM: selftests: Add tests for nVMX to vmx_apic_timer_virt
  KVM: selftests: Add a global option to disable in-kernel irqchip
  KVM: selftests: Test VMX apic timer virt with inkernel apic disable
  Documentation: KVM: x86: Update documentation of struct vmcs12

Yang Zhong (7):
  KVM: VMX: Detect APIC timer virtualization bit
  KVM: x86: Implement APIC virt timer helpers with callbacks
  KVM: x86/lapic: Start/stop sw/hv timer on vCPU un/block
  KVM: x86/lapic: Add a trace point for guest virtual timer
  KVM: VMX: Implement the hooks for VMX guest virtual deadline timer
  KVM: VMX: dump_vmcs() support the guest virt timer
  KVM: VMX: Introduce module parameter for APIC virt timer support

 Documentation/virt/kvm/x86/nested-vmx.rst     |  14 +-
 arch/x86/include/asm/kvm-x86-ops.h            |   5 +
 arch/x86/include/asm/kvm_host.h               |   6 +
 arch/x86/include/asm/vmx.h                    |   6 +
 arch/x86/include/asm/vmxfeatures.h            |   1 +
 arch/x86/kvm/lapic.c                          | 157 +++-
 arch/x86/kvm/lapic.h                          |  18 +
 arch/x86/kvm/trace.h                          |  16 +
 arch/x86/kvm/vmx/capabilities.h               |   8 +
 arch/x86/kvm/vmx/hyperv.c                     |  17 +
 arch/x86/kvm/vmx/main.c                       |   5 +
 arch/x86/kvm/vmx/nested.c                     | 212 +++++-
 arch/x86/kvm/vmx/nested.h                     |  39 +-
 arch/x86/kvm/vmx/vmcs12.c                     |   8 +
 arch/x86/kvm/vmx/vmcs12.h                     |  13 +-
 arch/x86/kvm/vmx/vmcs_shadow_fields.h         |   1 +
 arch/x86/kvm/vmx/vmx.c                        | 148 +++-
 arch/x86/kvm/vmx/vmx.h                        |   7 +-
 arch/x86/kvm/vmx/x86_ops.h                    |   5 +
 arch/x86/kvm/x86.c                            |   8 +-
 arch/x86/kvm/x86.h                            |   2 +-
 tools/testing/selftests/kvm/Makefile.kvm      |   4 +
 .../testing/selftests/kvm/include/x86/apic.h  |   1 +
 .../selftests/kvm/include/x86/kvm_util_arch.h |   1 +
 .../selftests/kvm/include/x86/processor.h     |   7 +
 tools/testing/selftests/kvm/include/x86/vmx.h |  14 +
 .../testing/selftests/kvm/lib/x86/processor.c |   5 +-
 .../selftests/kvm/x86/nested_set_state_test.c | 250 +++++++
 .../testing/selftests/kvm/x86/timer_latency.c | 704 ++++++++++++++++++
 .../kvm/x86/vmx_apic_timer_virt_test.c        | 510 +++++++++++++
 .../kvm/x86/vmx_apic_timer_virt_vmcs_test.c   | 464 ++++++++++++
 .../testing/selftests/kvm/x86/vmx_msrs_test.c |  53 ++
 .../kvm/x86/vmx_tertiary_ctls_test.c          | 168 +++++
 33 files changed, 2853 insertions(+), 24 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86/timer_latency.c
 create mode 100644 tools/testing/selftests/kvm/x86/vmx_apic_timer_virt_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/vmx_apic_timer_virt_vmcs_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/vmx_tertiary_ctls_test.c


base-commit: 5128b972fb2801ad9aca54d990a75611ab5283a9
-- 
2.45.2