Re: Questions about TSC virtualization in KVM

[Sean Christopherson <seanjc@google.com>]

Please don't top post.  https://people.kernel.org/tglx/notes-about-netiquette

On Wed, Oct 25, 2023, Yifei Ma wrote:
> > On Oct 24, 2023, at 10:33 AM, Sean Christopherson <seanjc@google.com> wrote:
> > 
> > On Tue, Oct 24, 2023, Yifei Ma wrote:
> >> Hi KVM community,
> >> 
> >>   I am trying to figure out how TSC is virtualized in KVM-VMX world.
> >>   According to the kernel documentation, reading TSC register through MSR
> >>   can be trapped into KVM and VMX. I am trying to figure out the KVM code
> >>   handing this trap.
> > 
> > Key word "can".  KVM chooses not to intercept RDMSR to MSR_IA32_TSC because
> > hardware handles the necessary offset and scaling.  KVM does still emulate reads
> > in kvm_get_msr_common(), e.g. if KVM is forced to emulate a RDMSR, but that's a
> > very, very uncommon path.
> > 
> > Ditto for the RDTSC instruction, which isn't subject to MSR intercpetion bitmaps
> > and has a dedicated control.  KVM will emulate RDTSC if KVM is already emulating,
> > but otherwise the guest can execute RDTSC without triggering a VM-Exit.
> > 
> > Modern CPUs provide both a offset and a scaling factor for VMX guests, i.e. the
> > CPU itself virtualizes guest TSC.  See the RDMSR and RDTSC bullet points in the
> > "CHANGES TO INSTRUCTION BEHAVIOR IN VMX NON-ROOT OPERATION" section of the SDM
> > for details.
> 
> I went through the SDM virtual machine extensions chapter, and some KVM
> patches and it helped me a lot. My understanding is:
> 
> If the RDTSC existing flied in the VMCS is not set, then the rdtsc from
> non-root model won’t cause VM-exit. In this case, the TSC returned to
> non-root is the value of the physical TSC * scaling + offset, if scaling and
> offset are set by KVM.

Yes.  Note, if hardware supports TSC offsetting and/or TSC scaling, they are
enabled by KVM.  KVM simply uses an initial offset of '0' and a multiplier that
makes the guest TSC "run" at the same frequency as the host.

> The TSC offset and scaling of a vCPU can be set from root-mode through KVM
> APIs using command KVM_VCPU_TSC_CTRL & KVM_SET_TSC_KHZ , and they are written
> to the vCPU’s VMCS fields. Next time, non-root mode calls rdtsc, the VMX
> hardware will add the offset & scaling to the physical TSC.

Yes, with caveats.  The guest can write MSR_IA32_TSC and/or MSR_IA32_TSC_ADJUST,
which KVM emulates by modifying TSC_OFFSET.  If the CPU doesn't have a constant
TSC, KVM will adjust TSC_OFFSET before the next VM-Enter to try and keep guest
TSC consistent and monotonic.  If the CPU doesn't support TSC scaling, KVM will
manually scale the guest TSC prior to every VM-Enter by again adjusting TSC_OFFSET
to "catch up" to what the guest TSC _should_ be given the guest TSC frequency.