From mboxrd@z Thu Jan 1 00:00:00 1970 From: Zachary Amsden Subject: Re: KVM timekeeping and TSC virtualization Date: Mon, 23 Aug 2010 19:47:53 -1000 Message-ID: <4C735D09.6080506@redhat.com> References: <1282291669-25709-1-git-send-email-zamsden@redhat.com> <4C6E8284.6020200@cisco.com> <4C6F0EB8.7050906@redhat.com> <4C707E23.80209@cisco.com> <4C73240F.5010902@redhat.com> <4C7336C9.8000006@cisco.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: kvm@vger.kernel.org To: "David S. Ahern" Return-path: Received: from mx1.redhat.com ([209.132.183.28]:28700 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753526Ab0HXFsW (ORCPT ); Tue, 24 Aug 2010 01:48:22 -0400 In-Reply-To: <4C7336C9.8000006@cisco.com> Sender: kvm-owner@vger.kernel.org List-ID: On 08/23/2010 05:04 PM, David S. Ahern wrote: > > On 08/23/10 19:44, Zachary Amsden wrote: > >>> I have also looked at time keeping and performance of getimeofday on a >>> certain proprietary hypervisor. KVM lags severely here and workloads >>> dependent on timestamps are dramatically impacted. Evaluations and >>> decisions are made today based on current designs - both KVM and >>> product. Severe performance deltas raise a lot of flags. >>> >>> >> This is laughably incorrect. >> > Uh, right. > I've heard the rumor that TSC is orders of magnitude faster under VMware than under KVM from three people now, and I thought you were part of that camp. Needless to say, they are either laughably incorrect, or possess some great secret knowledge of how to make things under virtualization go faster than bare metal. I also have a magical talking unicorn, which, btw, is invisible. Extraordinary claims require extraordinary proof (the proof of my unicorn is too complex to fit in the margin of this e-mail, however, I assure you he is real). > >> Gettimeofday is faster on KVM than anything else using TSC based clock >> because it passes the TSC through directly. VMware traps the TSC and >> is actually slower. >> > Yes, it does trap the TSC to ensure it is increasing. My question > regarding trapping on KVM was about to what to expect in terms of > overhead. Furthermore, if you add trapping on KVM are TSC reads still > faster on KVM? > > >> Can you please define your "severe performance delta" and tell us your >> benchmark methodology? I'd like to help you figure out how it is flawed. >> > I sent you the link in the last response. Here it is again: > http://www.mail-archive.com/kvm@vger.kernel.org/msg07231.html > > TSC - fast, but has horrible time drifts > > PIT - horribly slow > > ACPI PM - horribly slow > > HPET - did not exist in Nov. 2008, and since has not been reliable in my > tests with RHEL4 and RHEL5 > > kvmclock - does not exist for RHEL4 and not usable on RHEL5 until the > update of 5.5 with the fix (I have not retried RHEL5 with the latest > maintenance kernel to verify it is stable in my use cases). > > Take the program from the link above. Run it in a RHEL4& RHEL5 guest > running on VMware for all the clock sources. Somewhere I have the data > for these comparisons -- KVM, VMware and bare metal. Same hardware, same > OS. The PIT and acpi-PM clock sources are faster on VMware than bare metal. > > > My point is that kvmclock is Red Hat's answer for the future -- RHEL6, > RHEL5.Y (whenever it proves reliable). What about the present? What > about products based on other distributions newer than RHEL5 but > pre-kvmclock? > It should be obvious from this patchset... PIT or TSC. KVM did not have an in-kernel PIT implementation circa 2008, so this data is quite old. It's much faster now and will continue to get faster as exit cost goes down and the emulation gets further optimized. Plus, now we have an error-free TSC. > There are a lot of moving windows of what to use as a clock source, not > just per major number (RHEL4, RHEL5) but minor number (e.g., TSC > stability on RHEL4 -- e.g., > https://bugzilla.redhat.com/show_bug.cgi?id=491154) and further > maintenance releases (kvmclock requiring RHEL5.5+). That is not very > friendly to a product making a transition to virtualization - and with > the same code base running bare metal or in a VM. > If you have old software running on broken hardware you do not get hardware performance and error-free time virtualization. With any vendor. Period. With this patchset, KVM now has a much stronger guarantee: If you have old guest software running on broken hardware, using SMP virtual machines, you do not get hardware performance and error-free time virtualization. However, if you have new guest software, non-broken hardware, or can simply run UP guests instead of SMP, you can have hardware performance, and it is now error free. Alternatively, you can sacrifice some accuracy and have hardware performance, even for SMP guests, if you can tolerate some minor cross-CPU TSC variation. No other vendor I know of can make that guarantee. Zach