Re: RFC: VMX: initialize TSC offset relative to vm creation time

["David S. Ahern" <daahern@cisco.com>]

Marcelo:

Do you have a similar patch/idea for AMD?

Same program as before. Sets affinity to run on vcpu 0, call
gettimeofday(). Repeat for vcpu 1. ... Repeat for vcpu max. sleep(1).
Repeat sequence.

So in the following example output the process calls sleep with affinity
set to vcpu3, and on wake sets it to vcpu0 and then calls gettimeofday.
The result is a backward jump in time going from vcpu3 to vcpu0 and then
a forward jump from vcpu0 to vcpu1:

cpu 0: 1223902798.704804 *
cpu 1: 1223902799.824095
cpu 2: 1223902799.824139
cpu 3: 1223902799.824198

(sleep 1)

cpu 0: 1223902799.714804 *
cpu 1: 1223902800.834148
cpu 2: 1223902800.834190
cpu 3: 1223902800.834231

(sleep 1)

cpu 0: 1223902800.724863 *
cpu 1: 1223902801.844156
cpu 2: 1223902801.844234
cpu 3: 1223902801.844278

...

david

Marcelo Tosatti wrote:
> VMX initializes the TSC offset for each vcpu at different times, and
> also reinitializes it for vcpus other than 0 on APIC SIPI message.
> 
> This bug causes the TSC's to appear unsynchronized in the guest, even if
> the host is good.
> 
> Older Linux kernels don't handle the situation very well, so
> gettimeofday is likely to go backwards in time:
> 
> http://www.mail-archive.com/kvm@vger.kernel.org/msg02955.html
> http://sourceforge.net/tracker/index.php?func=detail&aid=2025534&group_id=180599&atid=893831
> 
> Fix it by initializating the offset of each vcpu relative to vm creation
> time, and moving it from vmx_vcpu_reset to vmx_vcpu_setup, out of the
> APIC MP init path.
> 
> 
> Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> 
> 
> Index: kvm.tip/arch/x86/kvm/vmx.c
> ===================================================================
> --- kvm.tip.orig/arch/x86/kvm/vmx.c
> +++ kvm.tip/arch/x86/kvm/vmx.c
> @@ -850,11 +850,8 @@ static u64 guest_read_tsc(void)
>   * writes 'guest_tsc' into guest's timestamp counter "register"
>   * guest_tsc = host_tsc + tsc_offset ==> tsc_offset = guest_tsc - host_tsc
>   */
> -static void guest_write_tsc(u64 guest_tsc)
> +static void guest_write_tsc(u64 guest_tsc, u64 host_tsc)
>  {
> -	u64 host_tsc;
> -
> -	rdtscll(host_tsc);
>  	vmcs_write64(TSC_OFFSET, guest_tsc - host_tsc);
>  }
>  
> @@ -918,6 +915,7 @@ static int vmx_set_msr(struct kvm_vcpu *
>  {
>  	struct vcpu_vmx *vmx = to_vmx(vcpu);
>  	struct kvm_msr_entry *msr;
> +	u64 host_tsc;
>  	int ret = 0;
>  
>  	switch (msr_index) {
> @@ -943,7 +941,8 @@ static int vmx_set_msr(struct kvm_vcpu *
>  		vmcs_writel(GUEST_SYSENTER_ESP, data);
>  		break;
>  	case MSR_IA32_TIME_STAMP_COUNTER:
> -		guest_write_tsc(data);
> +		rdtscll(host_tsc);
> +		guest_write_tsc(data, host_tsc);
>  		break;
>  	case MSR_P6_PERFCTR0:
>  	case MSR_P6_PERFCTR1:
> @@ -2202,6 +2201,7 @@ static int vmx_vcpu_setup(struct vcpu_vm
>  	vmcs_writel(CR0_GUEST_HOST_MASK, ~0UL);
>  	vmcs_writel(CR4_GUEST_HOST_MASK, KVM_GUEST_CR4_MASK);
>  
> +	guest_write_tsc(0, vmx->vcpu.kvm->arch.vm_init_tsc);
>  
>  	return 0;
>  }
> @@ -2292,8 +2292,6 @@ static int vmx_vcpu_reset(struct kvm_vcp
>  	vmcs_write32(GUEST_INTERRUPTIBILITY_INFO, 0);
>  	vmcs_write32(GUEST_PENDING_DBG_EXCEPTIONS, 0);
>  
> -	guest_write_tsc(0);
> -
>  	/* Special registers */
>  	vmcs_write64(GUEST_IA32_DEBUGCTL, 0);
>  
> Index: kvm.tip/arch/x86/kvm/x86.c
> ===================================================================
> --- kvm.tip.orig/arch/x86/kvm/x86.c
> +++ kvm.tip/arch/x86/kvm/x86.c
> @@ -4250,6 +4250,8 @@ struct  kvm *kvm_arch_create_vm(void)
>  	INIT_LIST_HEAD(&kvm->arch.active_mmu_pages);
>  	INIT_LIST_HEAD(&kvm->arch.assigned_dev_head);
>  
> +	rdtscll(kvm->arch.vm_init_tsc);
> +
>  	return kvm;
>  }
>  
> Index: kvm.tip/include/asm-x86/kvm_host.h
> ===================================================================
> --- kvm.tip.orig/include/asm-x86/kvm_host.h
> +++ kvm.tip/include/asm-x86/kvm_host.h
> @@ -377,6 +377,7 @@ struct kvm_arch{
>  
>  	struct page *ept_identity_pagetable;
>  	bool ept_identity_pagetable_done;
> +	u64 vm_init_tsc;
>  };
>  
>  struct kvm_vm_stat {
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html