[PATCH] KVM: x86: Fix guest time accounting with VIRT_CPU_ACCOUNTING_GEN

[PATCH] KVM: x86: Fix guest time accounting with VIRT_CPU_ACCOUNTING_GEN

[Wei Yang]

move guest_exit() after local_irq_eanbled() so that the timer interrupt
hits we account that tick as spent in the guest.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Wei Yang <w90p710@gmail.com>
---
 arch/x86/kvm/x86.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 2e302e977dac..04a2913f9226 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -8044,7 +8044,6 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 
 	++vcpu->stat.exits;
 
-	guest_exit_irqoff();
 	if (lapic_in_kernel(vcpu)) {
 		s64 delta = vcpu->arch.apic->lapic_timer.advance_expire_delta;
 		if (delta != S64_MIN) {
@@ -8054,6 +8053,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 	}
 
 	local_irq_enable();
+	guest_exit();
 	preempt_enable();
 
 	vcpu->srcu_idx = srcu_read_lock(&vcpu->kvm->srcu);
-- 
2.14.1.40.g8e62ba1

Re: [PATCH] KVM: x86: Fix guest time accounting with VIRT_CPU_ACCOUNTING_GEN

[Sean Christopherson]

On Tue, Jul 09, 2019 at 12:47:51AM +0800, Wei Yang wrote:
> move guest_exit() after local_irq_eanbled() so that the timer interrupt
> hits we account that tick as spent in the guest.
> 
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Signed-off-by: Wei Yang <w90p710@gmail.com>
> ---
>  arch/x86/kvm/x86.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 2e302e977dac..04a2913f9226 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -8044,7 +8044,6 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
>  
>  	++vcpu->stat.exits;
>  
> -	guest_exit_irqoff();
>  	if (lapic_in_kernel(vcpu)) {
>  		s64 delta = vcpu->arch.apic->lapic_timer.advance_expire_delta;
>  		if (delta != S64_MIN) {
> @@ -8054,6 +8053,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
>  	}
>  
>  	local_irq_enable();
> +	guest_exit();

The tracing invoked by trace_kvm_wait_lapic_expire() needs to be done
after guest exit, otherwise it will violate the RCU quiescent state.  See
commits:

  8b89fe1f6c43 ("kvm: x86: move tracepoints outside extended quiescent state")
  ec0671d5684a ("KVM: LAPIC: Delay trace_kvm_wait_lapic_expire tracepoint to after vmexit")

Is this an actual issue in practice, or was this prompted by code
inspection?

On SVM, this patch is essentially a nop as irqs are temporarily enabled by
svm_handle_exit_irqoff().  On VMX, this only applies to ticks that occur
between VM-Exit and local_irq_enable(), which is a fairly small window all
things considered.  Toggling irqs off and back on in guest_exit() seems
like a waste of cycles, and could introduce other inaccuracies on VMX,
e.g. if non-tick interrupts are taken and cause the tick to expire.

>  	preempt_enable();
>  
>  	vcpu->srcu_idx = srcu_read_lock(&vcpu->kvm->srcu);
> -- 
> 2.14.1.40.g8e62ba1
> 

Re: [PATCH] KVM: x86: Fix guest time accounting with VIRT_CPU_ACCOUNTING_GEN

[Sean Christopherson]

On Wed, Jul 10, 2019 at 08:42:30PM +0800, Zhang Yang wrote:
> Yes, In fact we had a trouble with the small window,  the system monitor
> found that the host machine's sys is very high now and then.  Based on our's
> tracing, we found that timer interrupt always hit after local_irq_enable, the
> guest tick would be always accounted to system time. after applying this
> patch, the phenomenon disappears.

Fun.  Rather than switching to guest_exit(), I think it'd be better to
unconditionally open an irq window in order to minimize the overhead.
I'll send a patch, it's easier to explain with code :-)