Re: Nested VMX - L1 hangs on running L2

[Bandan Das <bandan.das@stratus.com>]

On  0, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> 
> On Fri, Jul 08, 2011 at 02:40:53PM -0400, Bandan Das wrote:
> > I have already discussed this a bit with Nadav but hoping someone 
> > else has any other ideas/clues/suggestions/comments. With recent versions of the 
> > kernel (The last I tried is 3.0-rc5 with nVMX patches already merged), my L1 guest 
> > always hangs when I start L2. 
> > 
> > My setup : The host, L1 and L2 all are FC15 with the host running 3.0-rc5. When L1 is up 
> > and running, I start L2 from L1. Within a minute or two, both L1 and L2 hang. Although, if
> > if I run tracing on the host, I see :
> > 
>
... 
> Using guests TSC value when performing TSC adjustments is wrong. Can
> you please try the following patch, which skips TSC adjustments if
> vcpu is in guest mode.
> 
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 2b76ae3..44c90d1 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -1096,6 +1096,9 @@ static int kvm_guest_time_update(struct kvm_vcpu *v)
>  	s64 kernel_ns, max_kernel_ns;
>  	u64 tsc_timestamp;
>  
> +	if (is_guest_mode(v))
> +		return 0;
> +
>  	/* Keep irq disabled to prevent changes to the clock */
>  	local_irq_save(flags);
>  	kvm_get_msr(v, MSR_IA32_TSC, &tsc_timestamp);
> @@ -2214,6 +2217,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
>  		tsc_delta = !vcpu->arch.last_guest_tsc ? 0 :
>  			     tsc - vcpu->arch.last_guest_tsc;
>  
> +		if (is_guest_mode(vcpu))
> +			tsc_delta = 0;
> +
>  		if (tsc_delta < 0)
>  			mark_tsc_unstable("KVM discovered backwards TSC");
>  		if (check_tsc_unstable()) {
> @@ -2234,7 +2240,8 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
>  {
>  	kvm_x86_ops->vcpu_put(vcpu);
>  	kvm_put_guest_fpu(vcpu);
> -	kvm_get_msr(vcpu, MSR_IA32_TSC, &vcpu->arch.last_guest_tsc);
> +	if (!is_guest_mode(vcpu))
> +		kvm_get_msr(vcpu, MSR_IA32_TSC, &vcpu->arch.last_guest_tsc);
>  }
>  
>  static int is_efer_nx(void)
> @@ -5717,7 +5724,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
>  	if (hw_breakpoint_active())
>  		hw_breakpoint_restore();
>  
> -	kvm_get_msr(vcpu, MSR_IA32_TSC, &vcpu->arch.last_guest_tsc);
> +	if (!is_guest_mode(vcpu))
> +		kvm_get_msr(vcpu, MSR_IA32_TSC, &vcpu->arch.last_guest_tsc);
>  
>  	vcpu->mode = OUTSIDE_GUEST_MODE;
>  	smp_wmb();

Hi Marcelo,

Thanks for looking at this and for the explanation of the cause. Your patch does solve
my problem. I have been running my L2 guest for a few hours now without any hang, while
usually, I would get a hang within minutes of booting L2. 

Thanks again!
Bandan