From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH 0/2] Improve hpet accuracy Date: Fri, 6 Jun 2008 09:53:23 -0600 Message-ID: <20080606095323843.00000002776@djm-pc> References: Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1744561359==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Mime-version: 1.0 Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell , Keir Fraser , Ben Guthro , xen-devel List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format --===============1744561359== Content-Type: multipart/alternative; boundary=-------6d2a2b166d2a2b16 This is a multi-part message in MIME format ---------6d2a2b166d2a2b16 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracyHi Dave and Ben -- When running tests on xen-unstable (without your patch), please ensure that= hpet=3D1 is set in the hvm config and also I think that when hpet is the c= locksource on RHEL4-32, the clock IS resilient to missed ticks so timer_mod= e should be 2 (vs when pit is the clocksource on RHEL4-32, all clock ticks = must be delivered and so timer_mode should be 0). Per http://lists.xensource.com/archives/html/xen-devel/2008-06/msg00098.htm= l it's my intent to clean this up, but I won't get to it until next week. Thanks, Dan -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lis= ts.xensource.com]On Behalf Of Dave Winchell Sent: Friday, June 06, 2008 4:46 AM To: Keir Fraser; Ben Guthro; xen-devel Cc: dan.magenheimer@oracle.com; Dave Winchell Subject: RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy Keir, I think the changes are required. We'll run some tests today today so that we have some data to talk about. -Dave -----Original Message----- From: xen-devel-bounces@lists.xensource.com on behalf of Keir Fraser Sent: Fri 6/6/2008 4:58 AM To: Ben Guthro; xen-devel Cc: dan.magenheimer@oracle.com Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy Are these patches needed now the timers are built on Xen system time rath= er than host TSC? Dan has reported much better time-keeping with his patch checked in, and it=B9s for sure a lot less invasive than this patchset. -- Keir On 5/6/08 15:59, "Ben Guthro" wrote: > > 1. Introduction > > This patch improves the hpet based guest clock in terms of drift and > monotonicity. > Prior to this work the drift with hpet was greater than 2%, far above t= he .05% > limit > for ntp to synchronize. With this code, the drift ranges from .001% to = .0033% > depending > on guest and physical platform. > > Using hpet allows guest operating systems to provide monotonic time to = their > applications. Time sources other than hpet are not monotonic because > of their reliance on tsc, which is not synchronized across physical > processors. > > Windows 2k864 and many Linux guests are supported with two policies, on= e for > guests > that handle missed clock interrupts and the other for guests that requi= re the > correct number of interrupts. > > Guests may use hpet for the timing source even if the physical platform= has no > visible > hpet. Migration is supported between physical machines which differ in > physical > hpet visibility. > > Most of the changes are in hpet.c. Two general facilities are added to = track > interrupt > progress. The ideas here and the facilities would be useful in vpt.c, f= or > other time > sources, though no attempt is made here to improve vpt.c. > > The following sections discuss hpet dependencies, interrupt delivery po= licies, > live migration, > test results, and relation to recent work with monotonic time. > > > 2. Virtual Hpet dependencies > > The virtual hpet depends on the ability to read the physical or simulat= ed > (see discussion below) hpet. For timekeeping, the virtual hpet also de= pends > on two new interrupt notification facilities to implement its policies = for > interrupt delivery. > > 2.1. Two modes of low-level hpet main counter reads. > > In this implementation, the virtual hpet reads with read_64_main_counte= r(), > exported by > time.c, either the real physical hpet main counter register directly or= a > "simulated" > hpet main counter. > > The simulated mode uses a monotonic version of get_s_time() (NOW()), wh= ere the > last > time value is returned whenever the current time value is less than the= last > time > value. In simulated mode, since it is layered on s_time, the underlying= > hardware > can be hpet or some other device. The frequency of the main counter in > simulated > mode is the same as the standard physical hpet frequency, allowing live= > migration > between nodes that are configured differently. > > If the physical platform does not have an hpet device, or if xen is con= figured > not > to use the device, then the simulated method is used. If there is a phy= sical > hpet device, > and xen has initialized it, then either simulated or physical mode can = be > used. > This is governed by a boot time option, hpet-avoid. Setting this option= to 1 > gives the > simulated mode and 0 the physical mode. The default is physical mode. > > A disadvantage of the physical mode is that may take longer to read the= device > than in simulated mode. On some platforms the cost is about the same (l= ess > than 250 nsec) for > physical and simulated modes, while on others physical cost is much hig= her > than simulated. > A disadvantage of the simulated mode is that it can return the same val= ue > for the counter in consecutive calls. > > 2.2. Interrupt notification facilities. > > Two interrupt notification facilities are introduced, one is > hvm_isa_irq_assert_cb() > and the other hvm_register_intr_en_notif(). > > The vhpet uses hvm_isa_irq_assert_cb to deliver interrupts to the vioap= ic. > hvm_isa_irq_assert_cb allows a callback to be passed along to > vioapic_deliver() > and this callback is called with a mask of the vcpus which will get the= > interrupt. This callback is made before any vcpus receive an interrupt.= > > Vhpet uses hvm_register_intr_en_notif() to register a handler for a par= ticular > vector that will be called when that vector is injected in > [vmx,svm]_intr_assist() > and also when the guest finishes handling the interrupt. Here finished = is > defined > as the point when the guest re-enables interrupts or lowers the tpr val= ue. > EOI is not used as the end of interrupt as this is sometimes returned b= efore > the interrupt handler has done its work. A flag is passed to the handle= r > indicating > whether this is the injection point (post =3D 1) or the interrupt finis= hed (post > =3D 0) point. > The need for the finished point callback is discussed in the missed tic= ks > policy section. > > To prevent a possible early trigger of the finished callback, intr_en_n= otif > logic > has a two stage arm, the first at injection (hvm_intr_en_notif_arm()) a= nd the > second when > interrupts are seen to be disabled (hvm_intr_en_notif_disarm()). Once f= ully > armed, re-enabling > interrupts will cause hvm_intr_en_notif_disarm() to make the end of int= errupt > callback. hvm_intr_en_notif_arm() and hvm_intr_en_notif_disarm() are ca= lled by > [vmx,svm]_intr_assist(). > > 3. Interrupt delivery policies > > The existing hpet interrupt delivery is preserved. This includes > vcpu round robin delivery used by Linux and broadcast delivery used by > Windows. > > There are two policies for interrupt delivery, one for Windows 2k8-64 a= nd the > other > for Linux. The Linux policy takes advantage of the (guest) Linux missed= tick > and offset > calculations and does not attempt to deliver the right number of interr= upts. > The Windows policy delivers the correct number of interrupts, even if > sometimes much > closer to each other than the period. The policies are similar to those= in > vpt.c, though > there are some important differences. > > Policies are selected with an HVMOP_set_param hypercall with index > HVM_PARAM_TIMER_MODE. > Two new values are added, HVM_HPET_guest_computes_missed_ticks and > HVM_HPET_guest_does_not_compute_missed_ticks. The reason that two new = ones > are added is that > in some guests (32bit Linux) a no-missed policy is needed for clock sou= rces > other than hpet > and a missed ticks policy for hpet. It was felt that there would be les= s > confusion by simply > introducing the two hpet policies. > > 3.1. The missed ticks policy > > The Linux clock interrupt handler for hpet calculates missed ticks and = offset > using the hpet > main counter. The algorithm works well when the time since the last int= errupt > is greater than > or equal to a period and poorly otherwise. > > The missed ticks policy ensures that no two clock interrupts are delive= red to > the guest at > a time interval less than a period. A time stamp (hpet main counter val= ue) is > recorded (by a > callback registered with hvm_register_intr_en_notif) when Linux finishe= s > handling the clock > interrupt. Then, ensuing interrupts are delivered to the vioapic only i= f the > current main > counter value is a period greater than when the last interrupt was hand= led. > > Tests showed a significant improvement in clock drift with end of inter= rupt > time stamps > versus beginning of interrupt[1]. It is believed that the reason for th= e > improvement > is that the clock interrupt handler goes for a spinlock and can be ther= efore > delayed in its > processing. Furthermore, the main counter is read by the guest under th= e lock. > The net > effect is that if we time stamp injection, we can get the difference in= time > between successive interrupt handler lock acquisitions to be less than = the > period. > > 3.2. The no-missed ticks policy > > Windows 2k864 keeps very poor time with the missed ticks policy. So the= > no-missed ticks policy > was developed. In the no-missed ticks policy we deliver the correct num= ber of > interrupts, > even if they are spaced less than a period apart (when catching up). > > Windows 2k864 uses a broadcast mode in the interrupt routing such that > all vcpus get the clock interrupt. The best Windows drift performance w= as > achieved when the > policy code ensured that all the previous interrupts (on the various vc= pus) > had been injected > before injecting the next interrupt to the vioapic.. > > The policy code works as follows. It uses the hvm_isa_irq_assert_cb() t= o > record > the vcpus to be interrupted in h->hpet.pending_mask. Then, in the callb= ack > registered > with hvm_register_intr_en_notif() at post=3D1 time it clears the curren= t vcpu in > the pending_mask. > When the pending_mask is clear it decrements hpet.intr_pending_nr and i= f > intr_pending_nr is still > non-zero posts another interrupt to the ioapic with hvm_isa_irq_assert_= cb(). > Intr_pending_nr is incremented in hpet_route_decision_not_missed_ticks(= ). > > The missed ticks policy intr_en_notif callback also uses the pending_ma= sk > method. So even though > Linux does not broadcast its interrupts, the code could handle it if it= did. > In this case the end of interrupt time stamp is made when the pending_m= ask is > clear. > > 4. Live Migration > > Live migration with hpet preserves the current offset of the guest cloc= k with > respect > to ntp. This is accomplished by migrating all of the state in the h->hp= et data > structure > in the usual way. The hp->mc_offset is recalculated on the receiving no= de so > that the > guest sees a continuous hpet main counter. > > Code as been added to xc_domain_save.c to send a small message after th= e > domain context is sent. The contents of the message is the physical tsc= > timestamp, last_tsc, > read just before the message is sent. When the last_tsc message is rece= ived in > xc_domain_restore.c, > another physical tsc timestamp, cur_tsc, is read. The two timestamps ar= e > loaded into the domain > structure as last_tsc_sender and first_tsc_receiver with hypercalls. Th= en > xc_domain_hvm_setcontext > is called so that hpet_load has access to these time stamps. Hpet_load = uses > the timestamps > to account for the time spent saving and loading the domain context. Wi= th this > technique, > the only neglected time is the time spent sending a small network messa= ge. > > 5. Test Results > > Some recent test results are: > > 5.1 Linux 4u664 and Windows 2k864 load test. > Duration: 70 hours. > Test date: 6/2/08 > Loads: usex -b48 on Linux; burn-in on Windows > Guest vcpus: 8 for Linux; 2 for Windows > Hardware: 8 physical cpu AMD > Clock drift : Linux: .0012% Windows: .009% > > 5.2 Linux 4u664, Linux 4u464 , and Windows 2k864 no-load test > Duration: 23 hours. > Test date: 6/3/08 > Loads: none > Guest vcpus: 8 for each Linux; 2 for Windows > Hardware: 4 physical cpu AMD > Clock drift : Linux: .033% Windows: .019% > > 6. Relation to recent work in xen-unstable > > There is a similarity between hvm_get_guest_time() in xen-unstable and > read_64_main_counter() > in this code. However, read_64_main_counter() is more tuned to the need= s of > hpet.c. It has no > "set" operation, only the get. It isolates the mode, physical or simula= ted, in > read_64_main_counter() > itself. It uses no vcpu or domain state as it is a physical entity, in = either > mode. And it provides a real > physical mode for every read for those applications that desire this. > > 7. Conclusion > > The virtual hpet is improved by this patch in terms of accuracy and > monotonicity. > Tests performed to date verify this and more testing is under way. > > 8. Future Work > > Testing with Windows Vista will be performed soon. The reason for accur= acy > variations > on different platforms using the physical hpet device will be investiga= ted. > Additional overhead measurements on simulated vs physical hpet mode wil= l be > made. > > Footnotes: > > 1. I don't recall the accuracy improvement with end of interrupt stampi= ng, but > it was > significant, perhaps better than two to one improvement. It would be a = very > simple matter > to re-measure the improvement as the facility can call back at injectio= n time > as well. > > > Signed-off-by: Dave Winchell > > Signed-off-by: Ben Guthro > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel ---------6d2a2b166d2a2b16 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy</TITLE= > <META http-equiv=3DContent-Type content=3D"text/html; charset=3Diso-8859-1"= > <META content=3D"MSHTML 6.00.2900.3314" name=3DGENERATOR></HEAD> <BODY> <DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN class=3D093354715-06= 062008>Hi = Dave and Ben --</SPAN></FONT></DIV> <DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN = class=3D093354715-06062008></SPAN></FONT> </DIV> <DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN class=3D093354715-06= 062008>When = running tests on xen-unstable (without your patch), please ensure that hpet= =3D1 is = set </SPAN></FONT><FONT face=3DArial color=3D#0000ff size=3D2><SPAN = class=3D093354715-06062008>in the hvm config and also I think that when hpe= t is = the clocksource on RHEL4-32, the clock IS resilient to missed ticks so = timer_mode should be 2 (vs when pit is the clocksource on </SPAN></FONT><FO= NT = face=3DArial color=3D#0000ff size=3D2><SPAN class=3D093354715-06062008>RHEL= 4-32, all = clock ticks must be delivered and so timer_mode should be = 0).</SPAN></FONT></DIV> <DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN = class=3D093354715-06062008></SPAN></FONT> </DIV> <DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN class=3D093354715-06= 062008>Per <A = href=3D"http://lists.xensource.com/archives/html/xen-devel/2008-06/msg00098= .html">http://lists.xensource.com/archives/html/xen-devel/2008-06/msg00098.= html</A> it's = my i</SPAN></FONT><FONT face=3DArial color=3D#0000ff size=3D2><SPAN = class=3D093354715-06062008>ntent to clean this up, but I won't get to it un= til = next week.</SPAN></FONT></DIV> <DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN = class=3D093354715-06062008></SPAN></FONT> </DIV> <DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN = class=3D093354715-06062008>Thanks,</SPAN></FONT></DIV> <DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN = class=3D093354715-06062008>Dan</SPAN></FONT></DIV> <BLOCKQUOTE dir=3Dltr = style=3D"PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px soli= d; MARGIN-RIGHT: 0px"> <DIV class=3DOutlookMessageHeader dir=3Dltr align=3Dleft><FONT face=3DTah= oma = size=3D2>-----Original Message-----<BR><B>From:</B> = xen-devel-bounces@lists.xensource.com = [mailto:xen-devel-bounces@lists.xensource.com]<B>On Behalf Of </B>Dave = Winchell<BR><B>Sent:</B> Friday, June 06, 2008 4:46 AM<BR><B>To:</B> Keir= = Fraser; Ben Guthro; xen-devel<BR><B>Cc:</B> dan.magenheimer@oracle.com; D= ave = Winchell<BR><B>Subject:</B> RE: [Xen-devel] [PATCH 0/2] Improve hpet = accuracy<BR><BR></FONT></DIV><!-- Converted from text/plain format --> <P><FONT size=3D2>Keir,<BR><BR>I think the changes are required. We'll ru= n some = tests today today so<BR>that we have some data to talk = about.<BR><BR>-Dave<BR><BR><BR>-----Original Message-----<BR>From: = xen-devel-bounces@lists.xensource.com on behalf of Keir Fraser<BR>Sent: F= ri = 6/6/2008 4:58 AM<BR>To: Ben Guthro; xen-devel<BR>Cc: = dan.magenheimer@oracle.com<BR>Subject: Re: [Xen-devel] [PATCH 0/2] Improv= e = hpet accuracy<BR><BR>Are these patches needed now the timers are built on= Xen = system time rather<BR>than host TSC? Dan has reported much better time-ke= eping = with his patch<BR>checked in, and it=B9s for sure a lot less invasive tha= n this = patchset.<BR><BR><BR> -- Keir<BR><BR>On 5/6/08 15:59, "Ben Guthro" = <bguthro@virtualiron.com> wrote:<BR><BR>><BR>> 1. = Introduction<BR>><BR>> This patch improves the hpet based guest clo= ck in = terms of drift and<BR>> monotonicity.<BR>> Prior to this work the d= rift = with hpet was greater than 2%, far above the .05%<BR>> limit<BR>> f= or = ntp to synchronize. With this code, the drift ranges from .001% to = .0033%<BR>> depending<BR>> on guest and physical = platform.<BR>><BR>> Using hpet allows guest operating systems to pr= ovide = monotonic time to their<BR>> applications. Time sources other than hpe= t are = not monotonic because<BR>> of their reliance on tsc, which is not = synchronized across physical<BR>> processors.<BR>><BR>> Windows = 2k864 = and many Linux guests are supported with two policies, one for<BR>> = guests<BR>> that handle missed clock interrupts and the other for gues= ts = that require the<BR>> correct number of interrupts.<BR>><BR>> Gu= ests = may use hpet for the timing source even if the physical platform has = no<BR>> visible<BR>> hpet. Migration is supported between physical = = machines which differ in<BR>> physical<BR>> hpet = visibility.<BR>><BR>> Most of the changes are in hpet.c. Two genera= l = facilities are added to track<BR>> interrupt<BR>> progress. The ide= as = here and the facilities would be useful in vpt.c, for<BR>> other = time<BR>> sources, though no attempt is made here to improve = vpt.c.<BR>><BR>> The following sections discuss hpet dependencies, = = interrupt delivery policies,<BR>> live migration,<BR>> test results= , and = relation to recent work with monotonic time.<BR>><BR>><BR>> 2. = Virtual Hpet dependencies<BR>><BR>> The virtual hpet depends on the= = ability to read the physical or simulated<BR>> (see discussion below) = = hpet.  For timekeeping, the virtual hpet also depends<BR>> on two= new = interrupt notification facilities to implement its policies for<BR>> = interrupt delivery.<BR>><BR>> 2.1. Two modes of low-level hpet main= = counter reads.<BR>><BR>> In this implementation, the virtual hpet r= eads = with read_64_main_counter(),<BR>> exported by<BR>> time.c, either t= he = real physical hpet main counter register directly or a<BR>> = "simulated"<BR>> hpet main counter.<BR>><BR>> The simulated mode= uses = a monotonic version of get_s_time() (NOW()), where the<BR>> last<BR>&g= t; = time value is returned whenever the current time value is less than the = last<BR>> time<BR>> value. In simulated mode, since it is layered o= n = s_time, the underlying<BR>> hardware<BR>> can be hpet or some other= = device. The frequency of the main counter in<BR>> simulated<BR>> mo= de is = the same as the standard physical hpet frequency, allowing live<BR>> = migration<BR>> between nodes that are configured = differently.<BR>><BR>> If the physical platform does not have an hp= et = device, or if xen is configured<BR>> not<BR>> to use the device, th= en = the simulated method is used. If there is a physical<BR>> hpet = device,<BR>> and xen has initialized it, then either simulated or phys= ical = mode can be<BR>> used.<BR>> This is governed by a boot time option,= = hpet-avoid. Setting this option to 1<BR>> gives the<BR>> simulated = mode = and 0 the physical mode. The default is physical mode.<BR>><BR>> A = = disadvantage of the physical mode is that may take longer to read the = device<BR>> than in simulated mode. On some platforms the cost is abou= t the = same (less<BR>> than 250 nsec) for<BR>> physical and simulated mode= s, = while on others physical cost is much higher<BR>> than simulated.<BR>&= gt; A = disadvantage of the simulated mode is that it can return the same = value<BR>> for the counter in consecutive calls.<BR>><BR>> 2.2. = = Interrupt notification facilities.<BR>><BR>> Two interrupt notifica= tion = facilities are introduced, one is<BR>> hvm_isa_irq_assert_cb()<BR>>= and = the other hvm_register_intr_en_notif().<BR>><BR>> The vhpet uses = hvm_isa_irq_assert_cb to deliver interrupts to the vioapic.<BR>> = hvm_isa_irq_assert_cb allows a callback to be passed along to<BR>> = vioapic_deliver()<BR>> and this callback is called with a mask of the = vcpus = which will get the<BR>> interrupt. This callback is made before any vc= pus = receive an interrupt.<BR>><BR>> Vhpet uses hvm_register_intr_en_not= if() = to register a handler for a particular<BR>> vector that will be called= when = that vector is injected in<BR>> [vmx,svm]_intr_assist()<BR>> and al= so = when the guest finishes handling the interrupt. Here finished is<BR>> = = defined<BR>> as the point when the guest re-enables interrupts or lowe= rs = the tpr value.<BR>> EOI is not used as the end of interrupt as this is= = sometimes returned before<BR>> the interrupt handler has done its work= . A = flag is passed to the handler<BR>> indicating<BR>> whether this is = the = injection point (post =3D 1) or the interrupt finished (post<BR>> =3D = 0) = point.<BR>> The need for the finished point callback is discussed in t= he = missed ticks<BR>> policy section.<BR>><BR>> To prevent a possibl= e = early trigger of the finished callback, intr_en_notif<BR>> logic<BR>&g= t; = has a two stage arm, the first at injection (hvm_intr_en_notif_arm()) and= = the<BR>> second when<BR>> interrupts are seen to be disabled = (hvm_intr_en_notif_disarm()). Once fully<BR>> armed, re-enabling<BR>&g= t; = interrupts will cause hvm_intr_en_notif_disarm() to make the end of = interrupt<BR>> callback. hvm_intr_en_notif_arm() and = hvm_intr_en_notif_disarm() are called by<BR>> = [vmx,svm]_intr_assist().<BR>><BR>> 3. Interrupt delivery = policies<BR>><BR>> The existing hpet interrupt delivery is preserve= d. = This includes<BR>> vcpu round robin delivery used by Linux and broadca= st = delivery used by<BR>> Windows.<BR>><BR>> There are two policies = for = interrupt delivery, one for Windows 2k8-64 and the<BR>> other<BR>> = for = Linux. The Linux policy takes advantage of the (guest) Linux missed = tick<BR>> and offset<BR>> calculations and does not attempt to deli= ver = the right number of interrupts.<BR>> The Windows policy delivers the = correct number of interrupts, even if<BR>> sometimes much<BR>> clos= er to = each other than the period. The policies are similar to those in<BR>> = = vpt.c, though<BR>> there are some important differences.<BR>><BR>&g= t; = Policies are selected with an HVMOP_set_param hypercall with index<BR>>= ; = HVM_PARAM_TIMER_MODE.<BR>> Two new values are added, = HVM_HPET_guest_computes_missed_ticks and<BR>> = HVM_HPET_guest_does_not_compute_missed_ticks.  The reason that two n= ew = ones<BR>> are added is that<BR>> in some guests (32bit Linux) a = no-missed policy is needed for clock sources<BR>> other than hpet<BR>&= gt; = and a missed ticks policy for hpet. It was felt that there would be = less<BR>> confusion by simply<BR>> introducing the two hpet = policies.<BR>><BR>> 3.1. The missed ticks policy<BR>><BR>> Th= e = Linux clock interrupt handler for hpet calculates missed ticks and = offset<BR>> using the hpet<BR>> main counter. The algorithm works w= ell = when the time since the last interrupt<BR>> is greater than<BR>> or= = equal to a period and poorly otherwise.<BR>><BR>> The missed ticks = = policy ensures that no two clock interrupts are delivered to<BR>> the = guest = at<BR>> a time interval less than a period. A time stamp (hpet main co= unter = value) is<BR>> recorded (by a<BR>> callback registered with = hvm_register_intr_en_notif) when Linux finishes<BR>> handling the = clock<BR>> interrupt. Then, ensuing interrupts are delivered to the vi= oapic = only if the<BR>> current main<BR>> counter value is a period greate= r = than when the last interrupt was handled.<BR>><BR>> Tests showed a = = significant improvement in clock drift with end of interrupt<BR>> time= = stamps<BR>> versus beginning of interrupt[1]. It is believed that the = = reason for the<BR>> improvement<BR>> is that the clock interrupt ha= ndler = goes for a spinlock and can be therefore<BR>> delayed in its<BR>> = processing. Furthermore, the main counter is read by the guest under the = = lock.<BR>> The net<BR>> effect is that if we time stamp injection, = we = can get the difference in time<BR>> between successive interrupt handl= er = lock acquisitions to be less than the<BR>> period.<BR>><BR>> 3.2= . The = no-missed ticks policy<BR>><BR>> Windows 2k864 keeps very poor time= with = the missed ticks policy. So the<BR>> no-missed ticks policy<BR>> wa= s = developed. In the no-missed ticks policy we deliver the correct number = of<BR>> interrupts,<BR>> even if they are spaced less than a period= = apart (when catching up).<BR>><BR>> Windows 2k864 uses a broadcast = mode = in the interrupt routing such that<BR>> all vcpus get the clock interr= upt. = The best Windows drift performance was<BR>> achieved when the<BR>> = = policy code ensured that all the previous interrupts (on the various = vcpus)<BR>> had been injected<BR>> before injecting the next interr= upt = to the vioapic..<BR>><BR>> The policy code works as follows. It use= s the = hvm_isa_irq_assert_cb() to<BR>> record<BR>> the vcpus to be interru= pted = in h->hpet.pending_mask. Then, in the callback<BR>> registered<BR>&= gt; = with hvm_register_intr_en_notif() at post=3D1 time it clears the current = vcpu = in<BR>> the pending_mask.<BR>> When the pending_mask is clear it = decrements hpet.intr_pending_nr and if<BR>> intr_pending_nr is = still<BR>> non-zero posts another interrupt to the ioapic with = hvm_isa_irq_assert_cb().<BR>> Intr_pending_nr is incremented in = hpet_route_decision_not_missed_ticks().<BR>><BR>> The missed ticks = = policy intr_en_notif callback also uses the pending_mask<BR>> method. = So = even though<BR>> Linux does not broadcast its interrupts, the code cou= ld = handle it if it did.<BR>> In this case the end of interrupt time stamp= is = made when the pending_mask is<BR>> clear.<BR>><BR>> 4. Live = Migration<BR>><BR>> Live migration with hpet preserves the current = = offset of the guest clock with<BR>> respect<BR>> to ntp. This is = accomplished by migrating all of the state in the h->hpet data<BR>>= = structure<BR>> in the usual way. The hp->mc_offset is recalculated = on = the receiving node so<BR>> that the<BR>> guest sees a continuous hp= et = main counter.<BR>><BR>> Code as been added to xc_domain_save.c to s= end a = small message after the<BR>> domain context is sent. The contents of t= he = message is the physical tsc<BR>> timestamp, last_tsc,<BR>> read jus= t = before the message is sent. When the last_tsc message is received in<BR>&= gt; = xc_domain_restore.c,<BR>> another physical tsc timestamp, cur_tsc, is = read. = The two timestamps are<BR>> loaded into the domain<BR>> structure a= s = last_tsc_sender and first_tsc_receiver with hypercalls. Then<BR>> = xc_domain_hvm_setcontext<BR>> is called so that hpet_load has access t= o = these time stamps. Hpet_load uses<BR>> the timestamps<BR>> to accou= nt = for the time spent saving and loading the domain context. With this<BR>&g= t; = technique,<BR>> the only neglected time is the time spent sending a sm= all = network message.<BR>><BR>> 5. Test Results<BR>><BR>> Some rec= ent = test results are:<BR>><BR>> 5.1 Linux 4u664 and Windows 2k864 load = = test.<BR>>       Duration: 70 = hours.<BR>>       Test date: = 6/2/08<BR>>       Loads: usex -b48 on Li= nux; = burn-in on Windows<BR>>       Guest vcpu= s: 8 = for Linux; 2 for Windows<BR>>       Hard= ware: = 8 physical cpu AMD<BR>>       Clock drif= t : = Linux: .0012% Windows: .009%<BR>><BR>> 5.2 Linux 4u664, Linux 4u464= , = and Windows 2k864 no-load test<BR>>      = ; = Duration: 23 hours.<BR>>       Test date= : = 6/3/08<BR>>       Loads: = none<BR>>       Guest vcpus: 8 for each = = Linux; 2 for Windows<BR>>       Hardware= : 4 = physical cpu AMD<BR>>       Clock drift = : = Linux: .033% Windows: .019%<BR>><BR>> 6. Relation to recent work in= = xen-unstable<BR>><BR>> There is a similarity between = hvm_get_guest_time() in xen-unstable and<BR>> = read_64_main_counter()<BR>> in this code. However, read_64_main_counte= r() = is more tuned to the needs of<BR>> hpet.c. It has no<BR>> "set" = operation, only the get. It isolates the mode, physical or simulated, = in<BR>> read_64_main_counter()<BR>> itself. It uses no vcpu or doma= in = state as it is a physical entity, in either<BR>> mode. And it provides= a = real<BR>> physical mode for every read for those applications that des= ire = this.<BR>><BR>> 7. Conclusion<BR>><BR>> The virtual hpet is = improved by this patch in terms of accuracy and<BR>> monotonicity.<BR>= > = Tests performed to date verify this and more testing is under = way.<BR>><BR>> 8. Future Work<BR>><BR>> Testing with Windows = Vista = will be performed soon. The reason for accuracy<BR>> variations<BR>>= ; on = different platforms using the physical hpet device will be = investigated.<BR>> Additional overhead measurements on simulated vs = physical hpet mode will be<BR>> made.<BR>><BR>> = Footnotes:<BR>><BR>> 1. I don't recall the accuracy improvement wit= h end = of interrupt stamping, but<BR>> it was<BR>> significant, perhaps be= tter = than two to one improvement. It would be a very<BR>> simple matter<BR>= > = to re-measure the improvement as the facility can call back at injection = = time<BR>> as well.<BR>><BR>><BR>> Signed-off-by: Dave Winchel= l = <dwinchell@virtualiron.com><BR>> <<A = href=3D"mailto:dwinchell@virtualiron.com">mailto:dwinchell@virtualiron.co= m</A>><BR>> = Signed-off-by: Ben Guthro <bguthro@virtualiron.com><BR>> <<A = = href=3D"mailto:bguthro@virtualiron.com">mailto:bguthro@virtualiron.com</A= >><BR>><BR>><BR>> = _______________________________________________<BR>> Xen-devel mailing= = list<BR>> Xen-devel@lists.xensource.com<BR>> <A = href=3D"http://lists.xensource.com/xen-devel">http://lists.xensource.com/= xen-devel</A><BR><BR><BR><BR></FONT></P></BLOCKQUOTE></BODY></HTML> ---------6d2a2b166d2a2b16-- --===============1744561359== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============1744561359==--