From mboxrd@z Thu Jan 1 00:00:00 1970 From: Marcelo Tosatti Subject: Re: KVM: x86: limit difference between kvmclock updates Date: Wed, 15 May 2013 16:01:13 -0300 Message-ID: <20130515190113.GA11349@amt.cnet> References: <20130509232141.GA7642@amt.cnet> <20130514090513.GB20995@redhat.com> <20130514131257.GA19277@amt.cnet> <20130515174154.GD24814@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: kvm-devel , Glauber Costa To: Gleb Natapov Return-path: Received: from mx1.redhat.com ([209.132.183.28]:22113 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932945Ab3EOTBo (ORCPT ); Wed, 15 May 2013 15:01:44 -0400 Content-Disposition: inline In-Reply-To: <20130515174154.GD24814@redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: On Wed, May 15, 2013 at 08:41:54PM +0300, Gleb Natapov wrote: > On Tue, May 14, 2013 at 10:12:57AM -0300, Marcelo Tosatti wrote: > > On Tue, May 14, 2013 at 12:05:13PM +0300, Gleb Natapov wrote: > > > On Thu, May 09, 2013 at 08:21:41PM -0300, Marcelo Tosatti wrote: > > > > > > > > kvmclock updates which are isolated to a given vcpu, such as vcpu->cpu > > > > migration, should not allow system_timestamp from the rest of the vcpus > > > > to remain static. Otherwise ntp frequency correction applies to one > > > > vcpu's system_timestamp but not the others. > > > > > > > > So in those cases, request a kvmclock update for all vcpus. The worst > > > > case for a remote vcpu to update its kvmclock is then bounded by maximum > > > > nohz sleep latency. > > > > > > > Does this mean that when one vcpu is migrated all others are kicked out > > > from a guest mode? > > > > Yes, those which are in guest mode. For guests with large number of > > vcpus this is a problem, but i can't see a simpler method to fix the bug > > for now. > > > > Yes, this aspect must be improved (however, the bug incurs on timers in > > the guest taking tens of milliseconds with vcpu->pcpu pinning, which can > > be unacceptable). > Not sure I understand. With vcpu->pcpu pinning there will be no > migration. Do you mean "without" here? With vcpu->pcpu pinning there is no guarantee of kvm_arch_vcpu_load therefore no KVM_REQ_UPDATE_CLOCK. This is the problem. > If vcpu->kvm->arch.use_master_clock is false we kick vcpus on each > vcpu_load. When is it false? When - the host does not use TSC clocksource or - the vcpus TSCs are out of sync > I applied the patch since it fixes the real problem, but we need to > evaluate how it affects scalability. I'll look into ways to reduce the IPIs.