From mboxrd@z Thu Jan  1 00:00:00 1970
From: Radim Krcmar <rkrcmar@redhat.com>
Subject: Re: [patch 2/2] KVM: x86: add option to advance tscdeadline hrtimer
 expiration
Date: Fri, 12 Dec 2014 19:35:45 +0100
Message-ID: <20141212183545.GB14673@potion.brq.redhat.com>
References: <20141210205749.035440781@redhat.com>
 <20141210205904.415174860@redhat.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Cc: kvm@vger.kernel.org, Luiz Capitulino <lcapitulino@redhat.com>,
	Rik van Riel <riel@redhat.com>,
	Paolo Bonzini <pbonzini@redhat.com>
To: Marcelo Tosatti <mtosatti@redhat.com>
Return-path: <kvm-owner@vger.kernel.org>
Received: from mx1.redhat.com ([209.132.183.28]:42184 "EHLO mx1.redhat.com"
	rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
	id S1753175AbaLLSfw (ORCPT <rfc822;kvm@vger.kernel.org>);
	Fri, 12 Dec 2014 13:35:52 -0500
Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27])
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id sBCIZqMq003740
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL)
	for <kvm@vger.kernel.org>; Fri, 12 Dec 2014 13:35:52 -0500
Content-Disposition: inline
In-Reply-To: <20141210205904.415174860@redhat.com>
Sender: kvm-owner@vger.kernel.org
List-ID: <kvm.vger.kernel.org>

2014-12-10 15:57-0500, Marcelo Tosatti:
> For the hrtimer which emulates the tscdeadline timer in the guest,
> add an option to advance expiration, and busy spin on VM-entry waiting
> for the actual expiration time to elapse.
> 
> This allows achieving low latencies in cyclictest (or any scenario 
> which requires strict timing regarding timer expiration).
> 
> Reduces cyclictest avg latency by 50%.
> 
> Note: this option requires tuning to find the appropriate value 
> for a particular hardware/guest combination. One method is to measure the 
> average delay between apic_timer_fn and VM-entry. 
> Another method is to start with 1000ns, and increase the value
> in say 500ns increments until avg cyclictest numbers stop decreasing.
> 
> Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> 
> Index: kvm/arch/x86/kvm/lapic.c
> ===================================================================
> --- kvm.orig/arch/x86/kvm/lapic.c
> +++ kvm/arch/x86/kvm/lapic.c
> @@ -1087,11 +1089,58 @@ static void apic_timer_expired(struct kv
>  
>  	if (waitqueue_active(q))
>  		wake_up_interruptible(q);
> +
> +	if (ktimer->timer_mode_mask == APIC_LVT_TIMER_TSCDEADLINE)

timer_mode != timer_mode_mask.  (Please use apic_lvtt_tscdeadline().)

(So the code never waited for tsc_deadline >= guest_tsc ...
 I suppose it was possible to achieve lower latencies thanks to that.)

> +void wait_lapic_expire(struct kvm_vcpu *vcpu)
> +{
> +	struct kvm_lapic *apic = vcpu->arch.apic;
> +	u64 guest_tsc, tsc_deadline;
> +
> +	if (!kvm_vcpu_has_lapic(vcpu))
> +		return;
> +
> +	if (!apic_lvtt_tscdeadline(apic))
> +		return;

(It is better to check expired_tscdeadline here and zero it later.)

> +
> +	if (!lapic_timer_int_injected(vcpu))
> +		return;
> +
> +	tsc_deadline = apic->lapic_timer.expired_tscdeadline;
> +	guest_tsc = kvm_x86_ops->read_l1_tsc(vcpu, native_read_tsc());
> +
> +	while (guest_tsc < tsc_deadline) {
> +		int delay = min(tsc_deadline - guest_tsc, 1000ULL);
> +
> +		ndelay(delay);

The delay is in nanoseconds, but you feed it a difference in TSC.
(And usually overestimate the time;  cpu_relax() loop seems easiest.)