Re: RFC: VMX: initialize TSC offset relative to vm creation time

["David S. Ahern" <daahern@cisco.com>]

Hi Marcelo:

Dramatic improvement. The following is an example with kvm-75 and this
patch.  Without cpu affinity from a kvm perspective (vcpu-to-pcpu):

cpu 0: 1221107886.020298
cpu 1: 1221107886.020290 *
cpu 2: 1221107886.020555
cpu 3: 1221107886.020549 *

cpu 0: 1221107887.030244
cpu 1: 1221107887.030236 *
cpu 2: 1221107887.030498
cpu 3: 1221107887.030493 *

cpu 0: 1221107888.040248
cpu 1: 1221107888.040262
cpu 2: 1221107888.040314
cpu 3: 1221107888.040470

cpu 0: 1221107889.050305
cpu 1: 1221107889.050300 *
cpu 2: 1221107889.050354
cpu 3: 1221107889.050394

cpu 0: 1221107890.060384
cpu 1: 1221107890.060489
cpu 2: 1221107890.060753
cpu 3: 1221107890.060918

cpu 0: 1221107891.083559
cpu 1: 1221107891.083558 *
cpu 2: 1221107891.083614
cpu 3: 1221107891.083613 *

cpu 0: 1221107892.091705
cpu 1: 1221107892.091699 *
cpu 2: 1221107892.092998
cpu 3: 1221107892.093011

Setting vcpu-pcpu affinity well after guest startup, tracking is a bit
better (fewer time travels).

I do not believe there's a way to set affinity as kvm/qemu threads are
spawned (short of modifying qemu).

As before, RHEL3 guest. DL380G5 host.

david


Marcelo Tosatti wrote:
> VMX initializes the TSC offset for each vcpu at different times, and
> also reinitializes it for vcpus other than 0 on APIC SIPI message.
> 
> This bug causes the TSC's to appear unsynchronized in the guest, even if
> the host is good.
> 
> Older Linux kernels don't handle the situation very well, so
> gettimeofday is likely to go backwards in time:
> 
> http://www.mail-archive.com/kvm@vger.kernel.org/msg02955.html
> http://sourceforge.net/tracker/index.php?func=detail&aid=2025534&group_id=180599&atid=893831
> 
> Fix it by initializating the offset of each vcpu relative to vm creation
> time, and moving it from vmx_vcpu_reset to vmx_vcpu_setup, out of the
> APIC MP init path.
> 
> 
> Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> 
> 
> Index: kvm.tip/arch/x86/kvm/vmx.c
> ===================================================================
> --- kvm.tip.orig/arch/x86/kvm/vmx.c
> +++ kvm.tip/arch/x86/kvm/vmx.c
> @@ -850,11 +850,8 @@ static u64 guest_read_tsc(void)
>   * writes 'guest_tsc' into guest's timestamp counter "register"
>   * guest_tsc = host_tsc + tsc_offset ==> tsc_offset = guest_tsc - host_tsc
>   */
> -static void guest_write_tsc(u64 guest_tsc)
> +static void guest_write_tsc(u64 guest_tsc, u64 host_tsc)
>  {
> -	u64 host_tsc;
> -
> -	rdtscll(host_tsc);
>  	vmcs_write64(TSC_OFFSET, guest_tsc - host_tsc);
>  }
>  
> @@ -918,6 +915,7 @@ static int vmx_set_msr(struct kvm_vcpu *
>  {
>  	struct vcpu_vmx *vmx = to_vmx(vcpu);
>  	struct kvm_msr_entry *msr;
> +	u64 host_tsc;
>  	int ret = 0;
>  
>  	switch (msr_index) {
> @@ -943,7 +941,8 @@ static int vmx_set_msr(struct kvm_vcpu *
>  		vmcs_writel(GUEST_SYSENTER_ESP, data);
>  		break;
>  	case MSR_IA32_TIME_STAMP_COUNTER:
> -		guest_write_tsc(data);
> +		rdtscll(host_tsc);
> +		guest_write_tsc(data, host_tsc);
>  		break;
>  	case MSR_P6_PERFCTR0:
>  	case MSR_P6_PERFCTR1:
> @@ -2202,6 +2201,7 @@ static int vmx_vcpu_setup(struct vcpu_vm
>  	vmcs_writel(CR0_GUEST_HOST_MASK, ~0UL);
>  	vmcs_writel(CR4_GUEST_HOST_MASK, KVM_GUEST_CR4_MASK);
>  
> +	guest_write_tsc(0, vmx->vcpu.kvm->arch.vm_init_tsc);
>  
>  	return 0;
>  }
> @@ -2292,8 +2292,6 @@ static int vmx_vcpu_reset(struct kvm_vcp
>  	vmcs_write32(GUEST_INTERRUPTIBILITY_INFO, 0);
>  	vmcs_write32(GUEST_PENDING_DBG_EXCEPTIONS, 0);
>  
> -	guest_write_tsc(0);
> -
>  	/* Special registers */
>  	vmcs_write64(GUEST_IA32_DEBUGCTL, 0);
>  
> Index: kvm.tip/arch/x86/kvm/x86.c
> ===================================================================
> --- kvm.tip.orig/arch/x86/kvm/x86.c
> +++ kvm.tip/arch/x86/kvm/x86.c
> @@ -4250,6 +4250,8 @@ struct  kvm *kvm_arch_create_vm(void)
>  	INIT_LIST_HEAD(&kvm->arch.active_mmu_pages);
>  	INIT_LIST_HEAD(&kvm->arch.assigned_dev_head);
>  
> +	rdtscll(kvm->arch.vm_init_tsc);
> +
>  	return kvm;
>  }
>  
> Index: kvm.tip/include/asm-x86/kvm_host.h
> ===================================================================
> --- kvm.tip.orig/include/asm-x86/kvm_host.h
> +++ kvm.tip/include/asm-x86/kvm_host.h
> @@ -377,6 +377,7 @@ struct kvm_arch{
>  
>  	struct page *ept_identity_pagetable;
>  	bool ept_identity_pagetable_done;
> +	u64 vm_init_tsc;
>  };
>  
>  struct kvm_vm_stat {
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html