From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH 0/2] Improve hpet accuracy Date: Sun, 8 Jun 2008 15:18:09 -0600 Message-ID: <20080608151809703.00000002776@djm-pc> References: Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1964054289==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Mime-version: 1.0 Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell , Keir Fraser Cc: xen-devel , Ben Guthro List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format --===============1964054289== Content-Type: multipart/alternative; boundary=-------50bc178350bc1783 This is a multi-part message in MIME format ---------50bc178350bc1783 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy> A disadvantage of the si= mulated mode is that it can return the same value > for the counter in consecutive calls. It also occurs to me that if the granularity is good enough, an easy fix to this problem might be to always increment the returned value by at least one. Then time is always at least increasing rather than stopped -----Original Message----- From: Dave Winchell [mailto:dwinchell@virtualiron.com] Sent: Sunday, June 08, 2008 2:32 PM To: dan.magenheimer@oracle.com; Keir Fraser Cc: Ben Guthro; xen-devel; Dave Winchell Subject: RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy Hi Dan, > While I am fully supportive of offering hardware hpet as an option > for hvm guests (let's call it hwhpet=3D1 for shorthand), I am very > surprised by your preliminary results; the most obvious conclusion > is that Xen system time is losing time at the rate of 1000 PPM > though its possible there's a bug somewhere else in the "time > stack". Your Windows result is jaw-dropping and inexplicable, > though I have to admit ignorance of how Windows manages time. I think xen system time is fine. You have to add the interrupt delivery policies decribed in the write-up for the patch to get accurate timekeeping in the guest. The windows policy is obvious and results in a large improvement in accuracy. The Linux policy is more subtle, but is required to go from .1% to .03%. > I think with my recent patch and hpet=3D1 (essentially the same as > your emulated hpet), hvm guest time should track Xen system time. > I wonder if domain0 (which if I understand correctly is directly > using Xen system time) is also seeing an error of .1%? Also > I wonder for the skew you are seeing (in both hvm guests and > domain0) is time moving too fast or two slow? I don't recall the direction. I can look it up in my notes at work tomorrow. > Although hwhpet=3D1 is a fine alternative in many cases, it may > be unavailable on some systems and may cause significant performance > issues on others. So I think we will still need to track down > the poor accuracy when hwhpet=3D0. Our patch is accurate to < .03% using the physical hpet mode or the simulated mode. > And if for some reason > Xen system time can't be made accurate enough (< 0.05%), then > I think we should consider building Xen system time itself on > top of hardware hpet instead of TSC... at least when Xen discovers > a capable hpet. In our experience, Xen system time is accurate enough now. > One more thought... do you know the accuracy of the TSC crystals > on your test systems? I posted a patch awhile ago that was > intended to test that, though I guess it was only testing skew > of different TSCs on the same system, not TSCs against an > external time source. I do not know the tsc accuracy. > Or maybe there's a computation error somewhere in the hvm hpet > scaling code? Hmmm... Regards, Dave -----Original Message----- From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] Sent: Fri 6/6/2008 4:29 PM To: Dave Winchell; Keir Fraser Cc: Ben Guthro; xen-devel Subject: RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy Dave -- Thanks much for posting the preliminary results! While I am fully supportive of offering hardware hpet as an option for hvm guests (let's call it hwhpet=3D1 for shorthand), I am very surprised by your preliminary results; the most obvious conclusion is that Xen system time is losing time at the rate of 1000 PPM though its possible there's a bug somewhere else in the "time stack". Your Windows result is jaw-dropping and inexplicable, though I have to admit ignorance of how Windows manages time. I think with my recent patch and hpet=3D1 (essentially the same as your emulated hpet), hvm guest time should track Xen system time. I wonder if domain0 (which if I understand correctly is directly using Xen system time) is also seeing an error of .1%? Also I wonder for the skew you are seeing (in both hvm guests and domain0) is time moving too fast or two slow? Although hwhpet=3D1 is a fine alternative in many cases, it may be unavailable on some systems and may cause significant performance issues on others. So I think we will still need to track down the poor accuracy when hwhpet=3D0. And if for some reason Xen system time can't be made accurate enough (< 0.05%), then I think we should consider building Xen system time itself on top of hardware hpet instead of TSC... at least when Xen discovers a capable hpet. One more thought... do you know the accuracy of the TSC crystals on your test systems? I posted a patch awhile ago that was intended to test that, though I guess it was only testing skew of different TSCs on the same system, not TSCs against an external time source. Or maybe there's a computation error somewhere in the hvm hpet scaling code? Hmmm... Thanks, Dan > -----Original Message----- > From: Dave Winchell [mailto:dwinchell@virtualiron.com] > Sent: Friday, June 06, 2008 1:33 PM > To: dan.magenheimer@oracle.com; Keir Fraser > Cc: Ben Guthro; xen-devel; Dave Winchell > Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy > > > Dan, Keir: > > Preliminary tests results indicate an error of .1% for Linux 64 bit > guests configured > for hpet with xen-unstable as is. As we have discussed many times, the > ntp requirement is .05%. > Tests on the patch we just submitted for hpet have indicated errors of > .0012% > on this platform under similar test conditions and .03% on > other platforms. > > Windows vista64 has an error of 11% using hpet with the > xen-unstable bits. > In an overnight test with our hpet patch, the Windows vista > error was .008%. > > The tests are with two or three guests on a physical node, all under > load, and with > the ratio of vcpus to phys cpus > 1. > > I will continue to run tests over the next few days. > > thanks, > Dave > > > Dan Magenheimer wrote: > > > Hi Dave and Ben -- > > > > When running tests on xen-unstable (without your patch), > please ensure > > that hpet=3D1 is set in the hvm config and also I think that when hpe= t > > is the clocksource on RHEL4-32, the clock IS resilient to > missed ticks > > so timer_mode should be 2 (vs when pit is the clocksource > on RHEL4-32, > > all clock ticks must be delivered and so timer_mode should be 0). > > > > Per > > > http://lists.xensource.com/archives/html/xen-devel/2008-06/msg > 00098.html it's > > my intent to clean this up, but I won't get to it until next week. > > > > Thanks, > > Dan > > > > -----Original Message----- > > *From:* xen-devel-bounces@lists.xensource.com > > [mailto:xen-devel-bounces@lists.xensource.com]*On > Behalf Of *Dave > > Winchell > > *Sent:* Friday, June 06, 2008 4:46 AM > > *To:* Keir Fraser; Ben Guthro; xen-devel > > *Cc:* dan.magenheimer@oracle.com; Dave Winchell > > *Subject:* RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy > > > > Keir, > > > > I think the changes are required. We'll run some tests > today today so > > that we have some data to talk about. > > > > -Dave > > > > > > -----Original Message----- > > From: xen-devel-bounces@lists.xensource.com on behalf > of Keir Fraser > > Sent: Fri 6/6/2008 4:58 AM > > To: Ben Guthro; xen-devel > > Cc: dan.magenheimer@oracle.com > > Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy > > > > Are these patches needed now the timers are built on Xen system > > time rather > > than host TSC? Dan has reported much better > time-keeping with his > > patch > > checked in, and it=B9s for sure a lot less invasive than > this patchset. > > > > > > -- Keir > > > > On 5/6/08 15:59, "Ben Guthro" wrote: > > > > > > > > 1. Introduction > > > > > > This patch improves the hpet based guest clock in > terms of drift and > > > monotonicity. > > > Prior to this work the drift with hpet was greater > than 2%, far > > above the .05% > > > limit > > > for ntp to synchronize. With this code, the drift ranges from > > .001% to .0033% > > > depending > > > on guest and physical platform. > > > > > > Using hpet allows guest operating systems to provide monotonic > > time to their > > > applications. Time sources other than hpet are not > monotonic because > > > of their reliance on tsc, which is not synchronized > across physical > > > processors. > > > > > > Windows 2k864 and many Linux guests are supported with two > > policies, one for > > > guests > > > that handle missed clock interrupts and the other for guests > > that require the > > > correct number of interrupts. > > > > > > Guests may use hpet for the timing source even if the physical > > platform has no > > > visible > > > hpet. Migration is supported between physical machines which > > differ in > > > physical > > > hpet visibility. > > > > > > Most of the changes are in hpet.c. Two general facilities are > > added to track > > > interrupt > > > progress. The ideas here and the facilities would be useful in > > vpt.c, for > > > other time > > > sources, though no attempt is made here to improve vpt.c. > > > > > > The following sections discuss hpet dependencies, interrupt > > delivery policies, > > > live migration, > > > test results, and relation to recent work with monotonic time. > > > > > > > > > 2. Virtual Hpet dependencies > > > > > > The virtual hpet depends on the ability to read the > physical or > > simulated > > > (see discussion below) hpet. For timekeeping, the > virtual hpet > > also depends > > > on two new interrupt notification facilities to implement its > > policies for > > > interrupt delivery. > > > > > > 2.1. Two modes of low-level hpet main counter reads. > > > > > > In this implementation, the virtual hpet reads with > > read_64_main_counter(), > > > exported by > > > time.c, either the real physical hpet main counter register > > directly or a > > > "simulated" > > > hpet main counter. > > > > > > The simulated mode uses a monotonic version of get_s_time() > > (NOW()), where the > > > last > > > time value is returned whenever the current time value is less > > than the last > > > time > > > value. In simulated mode, since it is layered on s_time, the > > underlying > > > hardware > > > can be hpet or some other device. The frequency of the main > > counter in > > > simulated > > > mode is the same as the standard physical hpet frequency, > > allowing live > > > migration > > > between nodes that are configured differently. > > > > > > If the physical platform does not have an hpet > device, or if xen > > is configured > > > not > > > to use the device, then the simulated method is used. If there > > is a physical > > > hpet device, > > > and xen has initialized it, then either simulated or physical > > mode can be > > > used. > > > This is governed by a boot time option, hpet-avoid. > Setting this > > option to 1 > > > gives the > > > simulated mode and 0 the physical mode. The default > is physical > > mode. > > > > > > A disadvantage of the physical mode is that may take longer to > > read the device > > > than in simulated mode. On some platforms the cost is > about the > > same (less > > > than 250 nsec) for > > > physical and simulated modes, while on others physical cost is > > much higher > > > than simulated. > > > A disadvantage of the simulated mode is that it can return the > > same value > > > for the counter in consecutive calls. > > > > > > 2.2. Interrupt notification facilities. > > > > > > Two interrupt notification facilities are introduced, one is > > > hvm_isa_irq_assert_cb() > > > and the other hvm_register_intr_en_notif(). > > > > > > The vhpet uses hvm_isa_irq_assert_cb to deliver interrupts to > > the vioapic. > > > hvm_isa_irq_assert_cb allows a callback to be passed along to > > > vioapic_deliver() > > > and this callback is called with a mask of the vcpus > which will > > get the > > > interrupt. This callback is made before any vcpus receive an > > interrupt. > > > > > > Vhpet uses hvm_register_intr_en_notif() to register a handler > > for a particular > > > vector that will be called when that vector is injected in > > > [vmx,svm]_intr_assist() > > > and also when the guest finishes handling the interrupt. Here > > finished is > > > defined > > > as the point when the guest re-enables interrupts or > lowers the > > tpr value. > > > EOI is not used as the end of interrupt as this is sometimes > > returned before > > > the interrupt handler has done its work. A flag is > passed to the > > handler > > > indicating > > > whether this is the injection point (post =3D 1) or the > interrupt > > finished (post > > > =3D 0) point. > > > The need for the finished point callback is discussed in the > > missed ticks > > > policy section. > > > > > > To prevent a possible early trigger of the finished callback, > > intr_en_notif > > > logic > > > has a two stage arm, the first at injection > > (hvm_intr_en_notif_arm()) and the > > > second when > > > interrupts are seen to be disabled > (hvm_intr_en_notif_disarm()). > > Once fully > > > armed, re-enabling > > > interrupts will cause hvm_intr_en_notif_disarm() to > make the end > > of interrupt > > > callback. hvm_intr_en_notif_arm() and > hvm_intr_en_notif_disarm() > > are called by > > > [vmx,svm]_intr_assist(). > > > > > > 3. Interrupt delivery policies > > > > > > The existing hpet interrupt delivery is preserved. > This includes > > > vcpu round robin delivery used by Linux and broadcast delivery > > used by > > > Windows. > > > > > > There are two policies for interrupt delivery, one for Windows > > 2k8-64 and the > > > other > > > for Linux. The Linux policy takes advantage of the > (guest) Linux > > missed tick > > > and offset > > > calculations and does not attempt to deliver the > right number of > > interrupts. > > > The Windows policy delivers the correct number of interrupts, > > even if > > > sometimes much > > > closer to each other than the period. The policies are similar > > to those in > > > vpt.c, though > > > there are some important differences. > > > > > > Policies are selected with an HVMOP_set_param > hypercall with index > > > HVM_PARAM_TIMER_MODE. > > > Two new values are added, > HVM_HPET_guest_computes_missed_ticks and > > > HVM_HPET_guest_does_not_compute_missed_ticks. The reason that > > two new ones > > > are added is that > > > in some guests (32bit Linux) a no-missed policy is needed for > > clock sources > > > other than hpet > > > and a missed ticks policy for hpet. It was felt that > there would > > be less > > > confusion by simply > > > introducing the two hpet policies. > > > > > > 3.1. The missed ticks policy > > > > > > The Linux clock interrupt handler for hpet calculates missed > > ticks and offset > > > using the hpet > > > main counter. The algorithm works well when the time since the > > last interrupt > > > is greater than > > > or equal to a period and poorly otherwise. > > > > > > The missed ticks policy ensures that no two clock > interrupts are > > delivered to > > > the guest at > > > a time interval less than a period. A time stamp (hpet main > > counter value) is > > > recorded (by a > > > callback registered with hvm_register_intr_en_notif) > when Linux > > finishes > > > handling the clock > > > interrupt. Then, ensuing interrupts are delivered to > the vioapic > > only if the > > > current main > > > counter value is a period greater than when the last interrupt > > was handled. > > > > > > Tests showed a significant improvement in clock drift with end > > of interrupt > > > time stamps > > > versus beginning of interrupt[1]. It is believed that > the reason > > for the > > > improvement > > > is that the clock interrupt handler goes for a > spinlock and can > > be therefore > > > delayed in its > > > processing. Furthermore, the main counter is read by the guest > > under the lock. > > > The net > > > effect is that if we time stamp injection, we can get the > > difference in time > > > between successive interrupt handler lock acquisitions to be > > less than the > > > period. > > > > > > 3.2. The no-missed ticks policy > > > > > > Windows 2k864 keeps very poor time with the missed > ticks policy. > > So the > > > no-missed ticks policy > > > was developed. In the no-missed ticks policy we deliver the > > correct number of > > > interrupts, > > > even if they are spaced less than a period apart > (when catching up). > > > > > > Windows 2k864 uses a broadcast mode in the interrupt routing > > such that > > > all vcpus get the clock interrupt. The best Windows drift > > performance was > > > achieved when the > > > policy code ensured that all the previous interrupts (on the > > various vcpus) > > > had been injected > > > before injecting the next interrupt to the vioapic.. > > > > > > The policy code works as follows. It uses the > > hvm_isa_irq_assert_cb() to > > > record > > > the vcpus to be interrupted in h->hpet.pending_mask. Then, in > > the callback > > > registered > > > with hvm_register_intr_en_notif() at post=3D1 time it clears th= e > > current vcpu in > > > the pending_mask. > > > When the pending_mask is clear it decrements > > hpet.intr_pending_nr and if > > > intr_pending_nr is still > > > non-zero posts another interrupt to the ioapic with > > hvm_isa_irq_assert_cb(). > > > Intr_pending_nr is incremented in > > hpet_route_decision_not_missed_ticks(). > > > > > > The missed ticks policy intr_en_notif callback also uses the > > pending_mask > > > method. So even though > > > Linux does not broadcast its interrupts, the code could handle > > it if it did. > > > In this case the end of interrupt time stamp is made when the > > pending_mask is > > > clear. > > > > > > 4. Live Migration > > > > > > Live migration with hpet preserves the current offset of the > > guest clock with > > > respect > > > to ntp. This is accomplished by migrating all of the state in > > the h->hpet data > > > structure > > > in the usual way. The hp->mc_offset is recalculated on the > > receiving node so > > > that the > > > guest sees a continuous hpet main counter. > > > > > > Code as been added to xc_domain_save.c to send a small message > > after the > > > domain context is sent. The contents of the message is the > > physical tsc > > > timestamp, last_tsc, > > > read just before the message is sent. When the > last_tsc message > > is received in > > > xc_domain_restore.c, > > > another physical tsc timestamp, cur_tsc, is read. The two > > timestamps are > > > loaded into the domain > > > structure as last_tsc_sender and first_tsc_receiver with > > hypercalls. Then > > > xc_domain_hvm_setcontext > > > is called so that hpet_load has access to these time stamps. > > Hpet_load uses > > > the timestamps > > > to account for the time spent saving and loading the domain > > context. With this > > > technique, > > > the only neglected time is the time spent sending a small > > network message. > > > > > > 5. Test Results > > > > > > Some recent test results are: > > > > > > 5.1 Linux 4u664 and Windows 2k864 load test. > > > Duration: 70 hours. > > > Test date: 6/2/08 > > > Loads: usex -b48 on Linux; burn-in on Windows > > > Guest vcpus: 8 for Linux; 2 for Windows > > > Hardware: 8 physical cpu AMD > > > Clock drift : Linux: .0012% Windows: .009% > > > > > > 5.2 Linux 4u664, Linux 4u464 , and Windows 2k864 no-load test > > > Duration: 23 hours. > > > Test date: 6/3/08 > > > Loads: none > > > Guest vcpus: 8 for each Linux; 2 for Windows > > > Hardware: 4 physical cpu AMD > > > Clock drift : Linux: .033% Windows: .019% > > > > > > 6. Relation to recent work in xen-unstable > > > > > > There is a similarity between hvm_get_guest_time() in > > xen-unstable and > > > read_64_main_counter() > > > in this code. However, read_64_main_counter() is more tuned to > > the needs of > > > hpet.c. It has no > > > "set" operation, only the get. It isolates the mode, > physical or > > simulated, in > > > read_64_main_counter() > > > itself. It uses no vcpu or domain state as it is a physical > > entity, in either > > > mode. And it provides a real > > > physical mode for every read for those applications > that desire > > this. > > > > > > 7. Conclusion > > > > > > The virtual hpet is improved by this patch in terms > of accuracy and > > > monotonicity. > > > Tests performed to date verify this and more testing > is under way. > > > > > > 8. Future Work > > > > > > Testing with Windows Vista will be performed soon. The reason > > for accuracy > > > variations > > > on different platforms using the physical hpet device will be > > investigated. > > > Additional overhead measurements on simulated vs physical hpet > > mode will be > > > made. > > > > > > Footnotes: > > > > > > 1. I don't recall the accuracy improvement with end > of interrupt > > stamping, but > > > it was > > > significant, perhaps better than two to one improvement. It > > would be a very > > > simple matter > > > to re-measure the improvement as the facility can call back at > > injection time > > > as well. > > > > > > > > > Signed-off-by: Dave Winchell > > > > > > Signed-off-by: Ben Guthro > > > > > > > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xensource.com > > > http://lists.xensource.com/xen-devel > > > > > > > > ---------50bc178350bc1783 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy</TITLE= > <META http-equiv=3DContent-Type content=3D"text/html; charset=3Diso-8859-1"= > <META content=3D"MSHTML 6.00.2900.3314" name=3DGENERATOR></HEAD> <BODY> <DIV><FONT size=3D2>> A disadvantage of the simulated mode is that it ca= n = return the<SPAN class=3D421351521-08062008> </SPAN>s<SPAN = class=3D421351521-08062008>am</SPAN>e value<BR>> for the counter in cons= ecutive = calls.</FONT><BR></DIV> <DIV><SPAN class=3D421351521-08062008><FONT face=3DArial color=3D#0000ff si= ze=3D2>It = also occurs to me that if the granularity is good enough, an easy = fix</FONT></SPAN></DIV> <DIV><SPAN class=3D421351521-08062008><FONT face=3DArial color=3D#0000ff si= ze=3D2>to = this problem might be to always increment the returned value</FONT></SPAN><= /DIV> <DIV><SPAN class=3D421351521-08062008><FONT face=3DArial color=3D#0000ff si= ze=3D2>by at = least one.  Then time is always at least increasing = rather</FONT></SPAN></DIV> <DIV><SPAN class=3D421351521-08062008><FONT face=3DArial color=3D#0000ff si= ze=3D2>than = stopped</FONT></SPAN></DIV> <BLOCKQUOTE dir=3Dltr = style=3D"PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px soli= d; MARGIN-RIGHT: 0px"> <DIV class=3DOutlookMessageHeader dir=3Dltr align=3Dleft><FONT face=3DTah= oma = size=3D2>-----Original Message-----<BR><B>From:</B> Dave Winchell = [mailto:dwinchell@virtualiron.com]<BR><B>Sent:</B> Sunday, June 08, 2008 = 2:32 = PM<BR><B>To:</B> dan.magenheimer@oracle.com; Keir Fraser<BR><B>Cc:</B> Be= n = Guthro; xen-devel; Dave Winchell<BR><B>Subject:</B> RE: [Xen-devel] [PATC= H = 0/2] Improve hpet accuracy<BR><BR></FONT></DIV><!-- Converted from text/p= lain format --> <P><FONT size=3D2>Hi Dan,<BR><BR>> While I am fully supportive of offe= ring = hardware hpet as an option<BR>> for hvm guests (let's call it hwhpet=3D= 1 for = shorthand), I am very<BR>> surprised by your preliminary results; the = most = obvious conclusion<BR>> is that Xen system time is losing time at the = rate = of 1000 PPM<BR>> though its possible there's a bug somewhere else in t= he = "time<BR>> stack".  Your Windows result is jaw-dropping and = inexplicable,<BR>> though I have to admit ignorance of how Windows man= ages = time.<BR><BR>I think xen system time is fine. You have to add the = interrupt<BR>delivery policies decribed in the write-up for the patch to = = get<BR>accurate timekeeping in the guest.<BR><BR>The windows policy is ob= vious = and results in a large improvement<BR>in accuracy. The Linux policy is mo= re = subtle, but is required to go<BR>from .1% to .03%.<BR><BR>> I think wi= th my = recent patch and hpet=3D1 (essentially the same as<BR>> your emulated = hpet), = hvm guest time should track Xen system time.<BR>> I wonder if domain0 = = (which if I understand correctly is directly<BR>> using Xen system tim= e) is = also seeing an error of .1%?  Also<BR>> I wonder for the skew you= are = seeing (in both hvm guests and<BR>> domain0) is time moving too fast o= r two = slow?<BR><BR>I don't recall the direction. I can look it up in my notes a= t = work<BR>tomorrow.<BR><BR>> Although hwhpet=3D1 is a fine alternative i= n many = cases, it may<BR>> be unavailable on some systems and may cause signif= icant = performance<BR>> issues on others.  So I think we will still need= to = track down<BR>> the poor accuracy when hwhpet=3D0.<BR><BR>Our patch is= = accurate to < .03% using the physical hpet mode or<BR>the simulated = mode.<BR><BR>> And if for some reason<BR>> Xen system time can't be= made = accurate enough (< 0.05%), then<BR>> I think we should consider bui= lding = Xen system time itself on<BR>> top of hardware hpet instead of TSC... = at = least when Xen discovers<BR>> a capable hpet.<BR><BR>In our experience= , Xen = system time is accurate enough now.<BR><BR>> One more thought... do yo= u = know the accuracy of the TSC crystals<BR>> on your test systems? = I = posted a patch awhile ago that was<BR>> intended to test that, though = I = guess it was only testing skew<BR>> of different TSCs on the same syst= em, = not TSCs against an<BR>> external time source.<BR><BR>I do not know th= e tsc = accuracy.<BR><BR>> Or maybe there's a computation error somewhere in t= he = hvm hpet<BR>> scaling code?  = Hmmm...<BR><BR><BR>Regards,<BR>Dave<BR><BR><BR>-----Original = Message-----<BR>From: Dan Magenheimer [<A = href=3D"mailto:dan.magenheimer@oracle.com">mailto:dan.magenheimer@oracle.= com</A>]<BR>Sent: = Fri 6/6/2008 4:29 PM<BR>To: Dave Winchell; Keir Fraser<BR>Cc: Ben Guthro;= = xen-devel<BR>Subject: RE: [Xen-devel] [PATCH 0/2] Improve hpet = accuracy<BR><BR>Dave --<BR><BR>Thanks much for posting the preliminary = results!<BR><BR>While I am fully supportive of offering hardware hpet as = an = option<BR>for hvm guests (let's call it hwhpet=3D1 for shorthand), I am = very<BR>surprised by your preliminary results; the most obvious = conclusion<BR>is that Xen system time is losing time at the rate of 1000 = = PPM<BR>though its possible there's a bug somewhere else in the = "time<BR>stack".  Your Windows result is jaw-dropping and = inexplicable,<BR>though I have to admit ignorance of how Windows manages = = time.<BR><BR><BR>I think with my recent patch and hpet=3D1 (essentially t= he same = as<BR>your emulated hpet), hvm guest time should track Xen system time.<B= R>I = wonder if domain0 (which if I understand correctly is directly<BR>using X= en = system time) is also seeing an error of .1%?  Also<BR>I wonder for t= he = skew you are seeing (in both hvm guests and<BR>domain0) is time moving to= o = fast or two slow?<BR><BR>Although hwhpet=3D1 is a fine alternative in man= y = cases, it may<BR>be unavailable on some systems and may cause significant= = performance<BR>issues on others.  So I think we will still need to t= rack = down<BR>the poor accuracy when hwhpet=3D0.  And if for some reason<B= R>Xen = system time can't be made accurate enough (< 0.05%), then<BR>I think w= e = should consider building Xen system time itself on<BR>top of hardware hpe= t = instead of TSC... at least when Xen discovers<BR>a capable hpet.<BR><BR>O= ne = more thought... do you know the accuracy of the TSC crystals<BR>on your t= est = systems?  I posted a patch awhile ago that was<BR>intended to test t= hat, = though I guess it was only testing skew<BR>of different TSCs on the same = = system, not TSCs against an<BR>external time source.<BR><BR>Or maybe ther= e's a = computation error somewhere in the hvm hpet<BR>scaling code?  = Hmmm...<BR><BR>Thanks,<BR>Dan<BR><BR>> -----Original Message-----<BR>&= gt; = From: Dave Winchell [<A = href=3D"mailto:dwinchell@virtualiron.com">mailto:dwinchell@virtualiron.co= m</A>]<BR>> = Sent: Friday, June 06, 2008 1:33 PM<BR>> To: dan.magenheimer@oracle.co= m; = Keir Fraser<BR>> Cc: Ben Guthro; xen-devel; Dave Winchell<BR>> Subj= ect: = Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy<BR>><BR>><BR>>= Dan, = Keir:<BR>><BR>> Preliminary tests results indicate an error of .1% = for = Linux 64 bit<BR>> guests configured<BR>> for hpet with xen-unstable= as = is. As we have discussed many times, the<BR>> ntp requirement is = .05%.<BR>> Tests on the patch we just submitted for hpet have indicate= d = errors of<BR>> .0012%<BR>> on this platform under similar test = conditions and .03% on<BR>> other platforms.<BR>><BR>> Windows = vista64 has an error of 11% using hpet with the<BR>> xen-unstable = bits.<BR>> In an overnight test with our hpet patch, the Windows = vista<BR>> error was .008%.<BR>><BR>> The tests are with two or = three = guests on a physical node, all under<BR>> load, and with<BR>> the r= atio = of vcpus to phys cpus > 1.<BR>><BR>> I will continue to run test= s = over the next few days.<BR>><BR>> thanks,<BR>> = Dave<BR>><BR>><BR>> Dan Magenheimer wrote:<BR>><BR>> > = Hi = Dave and Ben --<BR>> ><BR>> > When running tests on xen-unsta= ble = (without your patch),<BR>> please ensure<BR>> > that hpet=3D1 is= set in = the hvm config and also I think that when hpet<BR>> > is the clocks= ource = on RHEL4-32, the clock IS resilient to<BR>> missed ticks<BR>> > = so = timer_mode should be 2 (vs when pit is the clocksource<BR>> on = RHEL4-32,<BR>> > all clock ticks must be delivered and so timer_mod= e = should be 0).<BR>> ><BR>> > Per<BR>> ><BR>> <A = href=3D"http://lists.xensource.com/archives/html/xen-devel/2008-06/msg">h= ttp://lists.xensource.com/archives/html/xen-devel/2008-06/msg</A><BR>> = 00098.html it's<BR>> > my intent to clean this up, but I won't get = to it = until next week.<BR>> ><BR>> > Thanks,<BR>> > Dan<BR>&g= t; = ><BR>> >     -----Original Message-----<BR>&= gt; = >     *From:* = xen-devel-bounces@lists.xensource.com<BR>> >    = ; [<A = href=3D"mailto:xen-devel-bounces@lists.xensource.com">mailto:xen-devel-bo= unces@lists.xensource.com</A>]*On<BR>> = Behalf Of *Dave<BR>> >     Winchell<BR>> = >     *Sent:* Friday, June 06, 2008 4:46 AM<BR>>= ; = >     *To:* Keir Fraser; Ben Guthro; xen-devel<BR>= > = >     *Cc:* dan.magenheimer@oracle.com; Dave = Winchell<BR>> >     *Subject:* RE: [Xen-devel] = = [PATCH 0/2] Improve hpet accuracy<BR>> ><BR>> = >     Keir,<BR>> ><BR>> = >     I think the changes are required. We'll run = some = tests<BR>> today today so<BR>> >     that we= have = some data to talk about.<BR>> ><BR>> >    = ; = -Dave<BR>> ><BR>> ><BR>> >     = -----Original Message-----<BR>> >     From: = xen-devel-bounces@lists.xensource.com on behalf<BR>> of Keir Fraser<BR= >> = >     Sent: Fri 6/6/2008 4:58 AM<BR>> = >     To: Ben Guthro; xen-devel<BR>> = >     Cc: dan.magenheimer@oracle.com<BR>> = >     Subject: Re: [Xen-devel] [PATCH 0/2] Improve= hpet = accuracy<BR>> ><BR>> >     Are these patc= hes = needed now the timers are built on Xen system<BR>> = >     time rather<BR>> >   &n= bsp; = than host TSC? Dan has reported much better<BR>> time-keeping with = his<BR>> >     patch<BR>> = >     checked in, and it=B9s for sure a lot less i= nvasive = than<BR>> this patchset.<BR>> ><BR>> ><BR>> = >      -- Keir<BR>> ><BR>> = >     On 5/6/08 15:59, "Ben Guthro" = <bguthro@virtualiron.com> wrote:<BR>> ><BR>> = >     ><BR>> >     &g= t; 1. = Introduction<BR>> >     ><BR>> = >     > This patch improves the hpet based gues= t = clock in<BR>> terms of drift and<BR>> >     = > = monotonicity.<BR>> >     > Prior to this wor= k the = drift with hpet was greater<BR>> than 2%, far<BR>> = >     above the .05%<BR>> = >     > limit<BR>> >   &nb= sp; = > for ntp to synchronize. With this code, the drift ranges from<BR>>= ; = >     .001% to .0033%<BR>> = >     > depending<BR>> = >     > on guest and physical platform.<BR>>= = >     ><BR>> >     &g= t; = Using hpet allows guest operating systems to provide monotonic<BR>> = >     time to their<BR>> = >     > applications. Time sources other than h= pet = are not<BR>> monotonic because<BR>> >     &g= t; of = their reliance on tsc, which is not synchronized<BR>> across = physical<BR>> >     > processors.<BR>> = >     ><BR>> >     &g= t; = Windows 2k864 and many Linux guests are supported with two<BR>> = >     policies, one for<BR>> = >     > guests<BR>> >   &n= bsp; = > that handle missed clock interrupts and the other for guests<BR>>= = >     that require the<BR>> = >     > correct number of interrupts.<BR>> = >     ><BR>> >     &g= t; = Guests may use hpet for the timing source even if the physical<BR>> = >     platform has no<BR>> = >     > visible<BR>> >   &= nbsp; = > hpet. Migration is supported between physical machines which<BR>>= = >     differ in<BR>> >   &nbs= p; = > physical<BR>> >     > hpet = visibility.<BR>> >     ><BR>> = >     > Most of the changes are in hpet.c. Two = = general facilities are<BR>> >     added to = track<BR>> >     > interrupt<BR>> = >     > progress. The ideas here and the facili= ties = would be useful in<BR>> >     vpt.c, for<BR>>= ; = >     > other time<BR>> = >     > sources, though no attempt is made here= to = improve vpt.c.<BR>> >     ><BR>> = >     > The following sections discuss hpet = dependencies, interrupt<BR>> >     delivery = policies,<BR>> >     > live migration,<BR>&g= t; = >     > test results, and relation to recent wo= rk = with monotonic time.<BR>> >     ><BR>> = >     ><BR>> >     &g= t; 2. = Virtual Hpet dependencies<BR>> >     ><BR>&g= t; = >     > The virtual hpet depends on the ability= to = read the<BR>> physical or<BR>> >     = simulated<BR>> >     > (see discussion below= ) = hpet.  For timekeeping, the<BR>> virtual hpet<BR>> = >     also depends<BR>> >   &= nbsp; = > on two new interrupt notification facilities to implement its<BR>>= ; = >     policies for<BR>> >   &= nbsp; = > interrupt delivery.<BR>> >     ><BR>>= ; = >     > 2.1. Two modes of low-level hpet main = counter reads.<BR>> >     ><BR>> = >     > In this implementation, the virtual hpe= t = reads with<BR>> >     = read_64_main_counter(),<BR>> >     > exporte= d = by<BR>> >     > time.c, either the real phys= ical = hpet main counter register<BR>> >     directly = or = a<BR>> >     > "simulated"<BR>> = >     > hpet main counter.<BR>> = >     ><BR>> >     &g= t; = The simulated mode uses a monotonic version of get_s_time()<BR>> = >     (NOW()), where the<BR>> = >     > last<BR>> >   &nbs= p; = > time value is returned whenever the current time value is less<BR>&g= t; = >     than the last<BR>> = >     > time<BR>> >   &nbs= p; = > value. In simulated mode, since it is layered on s_time, the<BR>>= = >     underlying<BR>> >   &nb= sp; = > hardware<BR>> >     > can be hpet or so= me = other device. The frequency of the main<BR>> >   &nb= sp; = counter in<BR>> >     > simulated<BR>> = >     > mode is the same as the standard physic= al = hpet frequency,<BR>> >     allowing live<BR>>= ; = >     > migration<BR>> = >     > between nodes that are configured = differently.<BR>> >     ><BR>> = >     > If the physical platform does not have = an = hpet<BR>> device, or if xen<BR>> >     is = configured<BR>> >     > not<BR>> = >     > to use the device, then the simulated m= ethod = is used. If there<BR>> >     is a physical<BR>&= gt; = >     > hpet device,<BR>> = >     > and xen has initialized it, then either= = simulated or physical<BR>> >     mode can be<BR= >> = >     > used.<BR>> >   &nb= sp; = > This is governed by a boot time option, hpet-avoid.<BR>> Setting = = this<BR>> >     option to 1<BR>> = >     > gives the<BR>> = >     > simulated mode and 0 the physical mode.= The = default<BR>> is physical<BR>> >     mode.<BR= >> = >     ><BR>> >     &g= t; A = disadvantage of the physical mode is that may take longer to<BR>> = >     read the device<BR>> = >     > than in simulated mode. On some platfor= ms = the cost is<BR>> about the<BR>> >     same = (less<BR>> >     > than 250 nsec) for<BR>>= ; = >     > physical and simulated modes, while on = = others physical cost is<BR>> >     much = higher<BR>> >     > than simulated.<BR>> = = >     > A disadvantage of the simulated mode is= that = it can return the<BR>> >     same value<BR>>= = >     > for the counter in consecutive = calls.<BR>> >     ><BR>> = >     > 2.2. Interrupt notification = facilities.<BR>> >     ><BR>> = >     > Two interrupt notification facilities a= re = introduced, one is<BR>> >     > = hvm_isa_irq_assert_cb()<BR>> >     > and the= = other hvm_register_intr_en_notif().<BR>> >     = = ><BR>> >     > The vhpet uses = hvm_isa_irq_assert_cb to deliver interrupts to<BR>> = >     the vioapic.<BR>> >   &= nbsp; = > hvm_isa_irq_assert_cb allows a callback to be passed along to<BR>>= ; = >     > vioapic_deliver()<BR>> = >     > and this callback is called with a mask= of = the vcpus<BR>> which will<BR>> >     get = the<BR>> >     > interrupt. This callback is= made = before any vcpus receive an<BR>> >     = interrupt.<BR>> >     ><BR>> = >     > Vhpet uses hvm_register_intr_en_notif()= to = register a handler<BR>> >     for a = particular<BR>> >     > vector that will be = = called when that vector is injected in<BR>> >   &nbs= p; = > [vmx,svm]_intr_assist()<BR>> >     > an= d = also when the guest finishes handling the interrupt. Here<BR>> = >     finished is<BR>> >   &n= bsp; = > defined<BR>> >     > as the point when = the = guest re-enables interrupts or<BR>> lowers the<BR>> = >     tpr value.<BR>> >   &nb= sp; = > EOI is not used as the end of interrupt as this is sometimes<BR>>= = >     returned before<BR>> = >     > the interrupt handler has done its work= . A = flag is<BR>> passed to the<BR>> >     = handler<BR>> >     > indicating<BR>> = >     > whether this is the injection point (po= st =3D = 1) or the<BR>> interrupt<BR>> >     finished= = (post<BR>> >     > =3D 0) point.<BR>> = >     > The need for the finished point callbac= k is = discussed in the<BR>> >     missed ticks<BR>>= ; = >     > policy section.<BR>> = >     ><BR>> >     &g= t; To = prevent a possible early trigger of the finished callback,<BR>> = >     intr_en_notif<BR>> = >     > logic<BR>> >   &nb= sp; = > has a two stage arm, the first at injection<BR>> = >     (hvm_intr_en_notif_arm()) and the<BR>> = >     > second when<BR>> = >     > interrupts are seen to be disabled<BR>&= gt; = (hvm_intr_en_notif_disarm()).<BR>> >     Once = fully<BR>> >     > armed, re-enabling<BR>>= ; = >     > interrupts will cause = hvm_intr_en_notif_disarm() to<BR>> make the end<BR>> = >     of interrupt<BR>> >   &= nbsp; = > callback. hvm_intr_en_notif_arm() and<BR>> = hvm_intr_en_notif_disarm()<BR>> >     are calle= d = by<BR>> >     > [vmx,svm]_intr_assist().<BR>= > = >     ><BR>> >     &g= t; 3. = Interrupt delivery policies<BR>> >     ><BR>= > = >     > The existing hpet interrupt delivery is= = preserved.<BR>> This includes<BR>> >     >= ; = vcpu round robin delivery used by Linux and broadcast delivery<BR>> = >     used by<BR>> >    = > = Windows.<BR>> >     ><BR>> = >     > There are two policies for interrupt = delivery, one for Windows<BR>> >     2k8-64 and= = the<BR>> >     > other<BR>> = >     > for Linux. The Linux policy takes advan= tage = of the<BR>> (guest) Linux<BR>> >     missed = = tick<BR>> >     > and offset<BR>> = >     > calculations and does not attempt to de= liver = the<BR>> right number of<BR>> >     = interrupts.<BR>> >     > The Windows policy = = delivers the correct number of interrupts,<BR>> = >     even if<BR>> >    = > = sometimes much<BR>> >     > closer to each o= ther = than the period. The policies are similar<BR>> >   &= nbsp; = to those in<BR>> >     > vpt.c, though<BR>&g= t; = >     > there are some important = differences.<BR>> >     ><BR>> = >     > Policies are selected with an = HVMOP_set_param<BR>> hypercall with index<BR>> = >     > HVM_PARAM_TIMER_MODE.<BR>> = >     > Two new values are added,<BR>> = HVM_HPET_guest_computes_missed_ticks and<BR>> >   &n= bsp; = > HVM_HPET_guest_does_not_compute_missed_ticks.  The reason = that<BR>> >     two new ones<BR>> = >     > are added is that<BR>> = >     > in some guests (32bit Linux) a no-misse= d = policy is needed for<BR>> >     clock = sources<BR>> >     > other than hpet<BR>>= = >     > and a missed ticks policy for hpet. It = was = felt that<BR>> there would<BR>> >     be = less<BR>> >     > confusion by simply<BR>>= ; = >     > introducing the two hpet policies.<BR>&= gt; = >     ><BR>> >     &g= t; = 3.1. The missed ticks policy<BR>> >     ><BR= >> = >     > The Linux clock interrupt handler for h= pet = calculates missed<BR>> >     ticks and = offset<BR>> >     > using the hpet<BR>> = >     > main counter. The algorithm works well = when = the time since the<BR>> >     last interrupt<BR= >> = >     > is greater than<BR>> = >     > or equal to a period and poorly = otherwise.<BR>> >     ><BR>> = >     > The missed ticks policy ensures that no= two = clock<BR>> interrupts are<BR>> >     deliver= ed = to<BR>> >     > the guest at<BR>> = >     > a time interval less than a period. A t= ime = stamp (hpet main<BR>> >     counter value) = is<BR>> >     > recorded (by a<BR>> = >     > callback registered with = hvm_register_intr_en_notif)<BR>> when Linux<BR>> = >     finishes<BR>> >    = ; = > handling the clock<BR>> >     > interru= pt. = Then, ensuing interrupts are delivered to<BR>> the vioapic<BR>> = >     only if the<BR>> >   &n= bsp; = > current main<BR>> >     > counter value= is a = period greater than when the last interrupt<BR>> = >     was handled.<BR>> >   &= nbsp; = ><BR>> >     > Tests showed a significant= = improvement in clock drift with end<BR>> >     = of = interrupt<BR>> >     > time stamps<BR>> = >     > versus beginning of interrupt[1]. It is= = believed that<BR>> the reason<BR>> >     for= = the<BR>> >     > improvement<BR>> = >     > is that the clock interrupt handler goe= s for = a<BR>> spinlock and can<BR>> >     be = therefore<BR>> >     > delayed in its<BR>>= ; = >     > processing. Furthermore, the main count= er is = read by the guest<BR>> >     under the lock.<BR= >> = >     > The net<BR>> >   &= nbsp; = > effect is that if we time stamp injection, we can get the<BR>> = >     difference in time<BR>> = >     > between successive interrupt handler lo= ck = acquisitions to be<BR>> >     less than the<BR>= > = >     > period.<BR>> >   &= nbsp; = ><BR>> >     > 3.2. The no-missed ticks = policy<BR>> >     ><BR>> = >     > Windows 2k864 keeps very poor time with= the = missed<BR>> ticks policy.<BR>> >     So = the<BR>> >     > no-missed ticks policy<BR>&= gt; = >     > was developed. In the no-missed ticks p= olicy = we deliver the<BR>> >     correct number of<BR>= > = >     > interrupts,<BR>> = >     > even if they are spaced less than a per= iod = apart<BR>> (when catching up).<BR>> >     = ><BR>> >     > Windows 2k864 uses a broad= cast = mode in the interrupt routing<BR>> >     such = that<BR>> >     > all vcpus get the clock = interrupt. The best Windows drift<BR>> >     = performance was<BR>> >     > achieved when = the<BR>> >     > policy code ensured that al= l the = previous interrupts (on the<BR>> >     various = = vcpus)<BR>> >     > had been injected<BR>>= ; = >     > before injecting the next interrupt to = the = vioapic..<BR>> >     ><BR>> = >     > The policy code works as follows. It us= es = the<BR>> >     hvm_isa_irq_assert_cb() to<BR>&g= t; = >     > record<BR>> >   &n= bsp; = > the vcpus to be interrupted in h->hpet.pending_mask. Then, in<BR>= > = >     the callback<BR>> >   &= nbsp; = > registered<BR>> >     > with = hvm_register_intr_en_notif() at post=3D1 time it clears the<BR>> = >     current vcpu in<BR>> = >     > the pending_mask.<BR>> = >     > When the pending_mask is clear it = decrements<BR>> >     hpet.intr_pending_nr and = = if<BR>> >     > intr_pending_nr is still<BR>= > = >     > non-zero posts another interrupt to the= = ioapic with<BR>> >     = hvm_isa_irq_assert_cb().<BR>> >     > = Intr_pending_nr is incremented in<BR>> >     = hpet_route_decision_not_missed_ticks().<BR>> >   &nb= sp; = ><BR>> >     > The missed ticks policy = intr_en_notif callback also uses the<BR>> >    = = pending_mask<BR>> >     > method. So even = though<BR>> >     > Linux does not broadcast= its = interrupts, the code could handle<BR>> >     it= if = it did.<BR>> >     > In this case the end of= = interrupt time stamp is made when the<BR>> >    = ; = pending_mask is<BR>> >     > clear.<BR>> = = >     ><BR>> >     &g= t; 4. = Live Migration<BR>> >     ><BR>> = >     > Live migration with hpet preserves the = = current offset of the<BR>> >     guest clock = with<BR>> >     > respect<BR>> = >     > to ntp. This is accomplished by migrati= ng = all of the state in<BR>> >     the h->hpet = data<BR>> >     > structure<BR>> = >     > in the usual way. The hp->mc_offset = is = recalculated on the<BR>> >     receiving node = so<BR>> >     > that the<BR>> = >     > guest sees a continuous hpet main = counter.<BR>> >     ><BR>> = >     > Code as been added to xc_domain_save.c = to = send a small message<BR>> >     after the<BR>&g= t; = >     > domain context is sent. The contents of= the = message is the<BR>> >     physical tsc<BR>> = = >     > timestamp, last_tsc,<BR>> = >     > read just before the message is sent. W= hen = the<BR>> last_tsc message<BR>> >     is rece= ived = in<BR>> >     > xc_domain_restore.c,<BR>>= = >     > another physical tsc timestamp, cur_tsc= , is = read. The two<BR>> >     timestamps are<BR>>= = >     > loaded into the domain<BR>> = >     > structure as last_tsc_sender and = first_tsc_receiver with<BR>> >     hypercalls. = = Then<BR>> >     > = xc_domain_hvm_setcontext<BR>> >     > is cal= led = so that hpet_load has access to these time stamps.<BR>> = >     Hpet_load uses<BR>> = >     > the timestamps<BR>> = >     > to account for the time spent saving an= d = loading the domain<BR>> >     context. With = this<BR>> >     > technique,<BR>> = >     > the only neglected time is the time spe= nt = sending a small<BR>> >     network message.<BR>= > = >     ><BR>> >     &g= t; 5. = Test Results<BR>> >     ><BR>> = >     > Some recent test results are:<BR>> = >     ><BR>> >     &g= t; = 5.1 Linux 4u664 and Windows 2k864 load test.<BR>> = >     >       = Duration: 70 hours.<BR>> >     = >       Test date: 6/2/08<BR>> = >     >       Loa= ds: = usex -b48 on Linux; burn-in on Windows<BR>> >   &nbs= p; = >       Guest vcpus: 8 for Linux; 2 for = = Windows<BR>> >     = >       Hardware: 8 physical cpu AMD<BR>= > = >     >       Clo= ck = drift : Linux: .0012% Windows: .009%<BR>> >    = = ><BR>> >     > 5.2 Linux 4u664, Linux 4u4= 64 , = and Windows 2k864 no-load test<BR>> >     = >       Duration: 23 hours.<BR>> = >     >       Tes= t = date: 6/3/08<BR>> >     = >       Loads: none<BR>> = >     >       Gue= st = vcpus: 8 for each Linux; 2 for Windows<BR>> >   &nbs= p; = >       Hardware: 4 physical cpu AMD<BR>= > = >     >       Clo= ck = drift : Linux: .033% Windows: .019%<BR>> >     = = ><BR>> >     > 6. Relation to recent work= in = xen-unstable<BR>> >     ><BR>> = >     > There is a similarity between = hvm_get_guest_time() in<BR>> >     xen-unstable= = and<BR>> >     > read_64_main_counter()<BR>&= gt; = >     > in this code. However, = read_64_main_counter() is more tuned to<BR>> >   &nb= sp; = the needs of<BR>> >     > hpet.c. It has = no<BR>> >     > "set" operation, only the ge= t. It = isolates the mode,<BR>> physical or<BR>> >   &nbs= p; = simulated, in<BR>> >     > = read_64_main_counter()<BR>> >     > itself. = It = uses no vcpu or domain state as it is a physical<BR>> = >     entity, in either<BR>> = >     > mode. And it provides a real<BR>> = >     > physical mode for every read for those = = applications<BR>> that desire<BR>> >     = this.<BR>> >     ><BR>> = >     > 7. Conclusion<BR>> = >     ><BR>> >     &g= t; = The virtual hpet is improved by this patch in terms<BR>> of accuracy = and<BR>> >     > monotonicity.<BR>> = >     > Tests performed to date verify this and= more = testing<BR>> is under way.<BR>> >     = ><BR>> >     > 8. Future Work<BR>> = >     ><BR>> >     &g= t; = Testing with Windows Vista will be performed soon. The reason<BR>> = >     for accuracy<BR>> >   &= nbsp; = > variations<BR>> >     > on different = platforms using the physical hpet device will be<BR>> = >     investigated.<BR>> = >     > Additional overhead measurements on = simulated vs physical hpet<BR>> >     mode will= = be<BR>> >     > made.<BR>> = >     ><BR>> >     &g= t; = Footnotes:<BR>> >     ><BR>> = >     > 1. I don't recall the accuracy improvem= ent = with end<BR>> of interrupt<BR>> >     stampi= ng, = but<BR>> >     > it was<BR>> = >     > significant, perhaps better than two to= one = improvement. It<BR>> >     would be a very<BR>&= gt; = >     > simple matter<BR>> = >     > to re-measure the improvement as the = facility can call back at<BR>> >     injection = = time<BR>> >     > as well.<BR>> = >     ><BR>> >     = ><BR>> >     > Signed-off-by: Dave Winche= ll = <dwinchell@virtualiron.com><BR>> >     &g= t; = <<A = href=3D"mailto:dwinchell@virtualiron.com">mailto:dwinchell@virtualiron.co= m</A>><BR>> = >     > Signed-off-by: Ben Guthro = <bguthro@virtualiron.com><BR>> >     >= = <<A = href=3D"mailto:bguthro@virtualiron.com">mailto:bguthro@virtualiron.com</A= >><BR>> = >     ><BR>> >     = ><BR>> >     > = _______________________________________________<BR>> = >     > Xen-devel mailing list<BR>> = >     > Xen-devel@lists.xensource.com<BR>> = >     > <A = href=3D"http://lists.xensource.com/xen-devel">http://lists.xensource.com/= xen-devel</A><BR>> = ><BR>> ><BR>> = ><BR>><BR>><BR><BR><BR></FONT></P></BLOCKQUOTE></BODY></HTML> ---------50bc178350bc1783-- --===============1964054289== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============1964054289==--