Re: KVM timekeeping and TSC virtualization

["David S. Ahern" <daahern@cisco.com>]

On 08/20/10 17:24, Zachary Amsden wrote:
> On 08/20/2010 03:26 AM, David S. Ahern wrote:
>>
>> On 08/20/10 02:07, Zachary Amsden wrote:
>>   
>>> This patch set implements full TSC virtualization, with both
>>> trapping and passthrough modes, and intelligent mode switching.
>>> As a result, TSC will never go backwards, we are stable against
>>> guest re-calibration attempts, VM reset, and migration.  For guests
>>> which require it, the TSC khz can even be preserved on migration
>>> to a new host.
>>>
>>> The TSC will never be trapped on UP systems unless the host TSC
>>> actually runs faster than the guest; other conditions, including
>>> bad hardware and changing speeds are accomodated by using catchup
>>> mode to keep the guest passthrough TSC in line with the host clock.
>>>      
>> What's the overhead of trapping TSC reads for Nehalem-type processors?
>>
>> gettimeofday() in guests is the biggest performance problem with KVM for
>> me, especially for older OSes like RHEL4 which is a supported OS for
>> another 2 years. Even with RHEL5, 32-bit, I had to force kvmclock off to
>> get the VM to run reliably:
>>
>> http://article.gmane.org/gmane.comp.emulators.kvm.devel/51017/match=kvmclock+rhel5.5
>>
>>    
> 
> Correctness is the biggest timekeeping problem with KVM for me.  The
> fact that you had to force kvmclock off is evidence of that.  Slightly
> slower applications are fine.  Broken ones are not acceptable.

I have been concerned with speed and correctness for a while:

http://www.mail-archive.com/kvm@vger.kernel.org/msg02955.html
http://www.mail-archive.com/kvm@vger.kernel.org/msg07231.html

> 
> TSC will not be trapped with kvmclock, and the bug you hit with RHEL5
> kvmclock has since been fixed.  As you can see, it is not a simple and
> straightforward issue to get all the issues sorted out.

kvmclock is for guests running RHEL5.5+some update and or some guest
running a very recent linux kernel. There's a lot of products running on
OS'es older than that.

> 
> Also, TSC will not be trapped with UP VMs, only SMP.  If you seriously
> believe RHEL4 will perform better as an SMP guest than several instances
> of coordinated UP guests, you would worry about this issue.  I don't. 
> The amount of upstream scalability and performance work done since that
> timeframe is enormous, to the point that it's entirely plausible that
> KVM governed UP RHEL4 guests as a cluster are faster than a RHEL4 SMP host.

Products built on RHEL3, RHEL4 or earlier RHEL5 were developed in the
past, and performance expectations set for that version based on SMP -
be it bare metal or virtual. You can't expect a product to be redesigned
to run on KVM.

> 
> So the answer is - it depends.  Hardware is always getting faster, and
> trap / exit cost is going down.   Right now, it is anywhere from a few
> hundred to multiple thousands of cycles, depending on your hardware.  I
> don't have an exact benchmark number I can quote, although in a couple
> of hours, I probably will.  I'll guess 3,000 cycles.
> 
> I agree, gettimeofday is a huge issue, for poorly written applications. 

I understand it is not a simple problem, and "poorly written
applications" is a bit of reach don't you think? There are a number of
workloads that depend on time stamps; that does not make them poorly
designed.

> Not that this means we won't speed it up, in fact, I have already done
> quite a bit of work on ways to reduce the exit cost.  Let's, however,
> get things correct before trying to make them aggressively fast.
> 
> Zach

I have also looked at time keeping and performance of getimeofday on a
certain proprietary hypervisor. KVM lags severely here and workloads
dependent on timestamps are dramatically impacted. Evaluations and
decisions are made today based on current designs - both KVM and
product. Severe performance deltas raise a lot of flags.

David