From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH 0/2] Improve hpet accuracy Date: Fri, 06 Jun 2008 09:58:39 +0100 Message-ID: References: <4847FF61.5050302@virtualiron.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0099531770==" Return-path: In-Reply-To: <4847FF61.5050302@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Mime-version: 1.0 Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Ben Guthro , xen-devel Cc: "dan.magenheimer@oracle.com" List-Id: xen-devel@lists.xenproject.org > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============0099531770== Content-type: multipart/alternative; boundary="B_3295591122_19675835" > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3295591122_19675835 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable Are these patches needed now the timers are built on Xen system time rather than host TSC? Dan has reported much better time-keeping with his patch checked in, and it=B9s for sure a lot less invasive than this patchset. -- Keir On 5/6/08 15:59, "Ben Guthro" wrote: >=20 > 1. Introduction >=20 > This patch improves the hpet based guest clock in terms of drift and > monotonicity. > Prior to this work the drift with hpet was greater than 2%, far above the= .05% > limit > for ntp to synchronize. With this code, the drift ranges from .001% to .0= 033% > depending > on guest and physical platform. >=20 > Using hpet allows guest operating systems to provide monotonic time to th= eir > applications. Time sources other than hpet are not monotonic because > of their reliance on tsc, which is not synchronized across physical > processors. >=20 > Windows 2k864 and many Linux guests are supported with two policies, one = for > guests > that handle missed clock interrupts and the other for guests that require= the > correct number of interrupts. >=20 > Guests may use hpet for the timing source even if the physical platform h= as no > visible > hpet. Migration is supported between physical machines which differ in > physical > hpet visibility. >=20 > Most of the changes are in hpet.c. Two general facilities are added to tr= ack > interrupt > progress. The ideas here and the facilities would be useful in vpt.c, for > other time > sources, though no attempt is made here to improve vpt.c. >=20 > The following sections discuss hpet dependencies, interrupt delivery poli= cies, > live migration, > test results, and relation to recent work with monotonic time. >=20 >=20 > 2. Virtual Hpet dependencies >=20 > The virtual hpet depends on the ability to read the physical or simulated > (see discussion below) hpet. For timekeeping, the virtual hpet also depe= nds > on two new interrupt notification facilities to implement its policies fo= r > interrupt delivery. >=20 > 2.1. Two modes of low-level hpet main counter reads. >=20 > In this implementation, the virtual hpet reads with read_64_main_counter(= ), > exported by > time.c, either the real physical hpet main counter register directly or a > "simulated" > hpet main counter. >=20 > The simulated mode uses a monotonic version of get_s_time() (NOW()), wher= e the > last > time value is returned whenever the current time value is less than the l= ast > time > value. In simulated mode, since it is layered on s_time, the underlying > hardware > can be hpet or some other device. The frequency of the main counter in > simulated > mode is the same as the standard physical hpet frequency, allowing live > migration > between nodes that are configured differently. >=20 > If the physical platform does not have an hpet device, or if xen is confi= gured > not > to use the device, then the simulated method is used. If there is a physi= cal > hpet device, > and xen has initialized it, then either simulated or physical mode can be > used. > This is governed by a boot time option, hpet-avoid. Setting this option t= o 1 > gives the > simulated mode and 0 the physical mode. The default is physical mode. >=20 > A disadvantage of the physical mode is that may take longer to read the d= evice > than in simulated mode. On some platforms the cost is about the same (les= s > than 250 nsec) for > physical and simulated modes, while on others physical cost is much highe= r > than simulated. > A disadvantage of the simulated mode is that it can return the same value > for the counter in consecutive calls. >=20 > 2.2. Interrupt notification facilities. >=20 > Two interrupt notification facilities are introduced, one is > hvm_isa_irq_assert_cb() > and the other hvm_register_intr_en_notif(). >=20 > The vhpet uses hvm_isa_irq_assert_cb to deliver interrupts to the vioapic= . > hvm_isa_irq_assert_cb allows a callback to be passed along to > vioapic_deliver() > and this callback is called with a mask of the vcpus which will get the > interrupt. This callback is made before any vcpus receive an interrupt. >=20 > Vhpet uses hvm_register_intr_en_notif() to register a handler for a parti= cular > vector that will be called when that vector is injected in > [vmx,svm]_intr_assist() > and also when the guest finishes handling the interrupt. Here finished is > defined > as the point when the guest re-enables interrupts or lowers the tpr value= . > EOI is not used as the end of interrupt as this is sometimes returned bef= ore > the interrupt handler has done its work. A flag is passed to the handler > indicating > whether this is the injection point (post =3D 1) or the interrupt finished = (post > =3D 0) point. > The need for the finished point callback is discussed in the missed ticks > policy section. >=20 > To prevent a possible early trigger of the finished callback, intr_en_not= if > logic > has a two stage arm, the first at injection (hvm_intr_en_notif_arm()) and= the > second when > interrupts are seen to be disabled (hvm_intr_en_notif_disarm()). Once ful= ly > armed, re-enabling > interrupts will cause hvm_intr_en_notif_disarm() to make the end of inter= rupt > callback. hvm_intr_en_notif_arm() and hvm_intr_en_notif_disarm() are call= ed by > [vmx,svm]_intr_assist(). >=20 > 3. Interrupt delivery policies >=20 > The existing hpet interrupt delivery is preserved. This includes > vcpu round robin delivery used by Linux and broadcast delivery used by > Windows. >=20 > There are two policies for interrupt delivery, one for Windows 2k8-64 and= the > other > for Linux. The Linux policy takes advantage of the (guest) Linux missed t= ick > and offset > calculations and does not attempt to deliver the right number of interrup= ts. > The Windows policy delivers the correct number of interrupts, even if > sometimes much > closer to each other than the period. The policies are similar to those i= n > vpt.c, though > there are some important differences. >=20 > Policies are selected with an HVMOP_set_param hypercall with index > HVM_PARAM_TIMER_MODE. > Two new values are added, HVM_HPET_guest_computes_missed_ticks and > HVM_HPET_guest_does_not_compute_missed_ticks. The reason that two new on= es > are added is that > in some guests (32bit Linux) a no-missed policy is needed for clock sourc= es > other than hpet > and a missed ticks policy for hpet. It was felt that there would be less > confusion by simply > introducing the two hpet policies. >=20 > 3.1. The missed ticks policy >=20 > The Linux clock interrupt handler for hpet calculates missed ticks and of= fset > using the hpet > main counter. The algorithm works well when the time since the last inter= rupt > is greater than > or equal to a period and poorly otherwise. >=20 > The missed ticks policy ensures that no two clock interrupts are delivere= d to > the guest at > a time interval less than a period. A time stamp (hpet main counter value= ) is > recorded (by a > callback registered with hvm_register_intr_en_notif) when Linux finishes > handling the clock > interrupt. Then, ensuing interrupts are delivered to the vioapic only if = the > current main > counter value is a period greater than when the last interrupt was handle= d. >=20 > Tests showed a significant improvement in clock drift with end of interru= pt > time stamps > versus beginning of interrupt[1]. It is believed that the reason for the > improvement > is that the clock interrupt handler goes for a spinlock and can be theref= ore > delayed in its > processing. Furthermore, the main counter is read by the guest under the = lock. > The net > effect is that if we time stamp injection, we can get the difference in t= ime > between successive interrupt handler lock acquisitions to be less than th= e > period. >=20 > 3.2. The no-missed ticks policy >=20 > Windows 2k864 keeps very poor time with the missed ticks policy. So the > no-missed ticks policy > was developed. In the no-missed ticks policy we deliver the correct numbe= r of > interrupts, > even if they are spaced less than a period apart (when catching up). >=20 > Windows 2k864 uses a broadcast mode in the interrupt routing such that > all vcpus get the clock interrupt. The best Windows drift performance was > achieved when the > policy code ensured that all the previous interrupts (on the various vcpu= s) > had been injected > before injecting the next interrupt to the vioapic.. >=20 > The policy code works as follows. It uses the hvm_isa_irq_assert_cb() to > record > the vcpus to be interrupted in h->hpet.pending_mask. Then, in the callbac= k > registered > with hvm_register_intr_en_notif() at post=3D1 time it clears the current vc= pu in > the pending_mask. > When the pending_mask is clear it decrements hpet.intr_pending_nr and if > intr_pending_nr is still > non-zero posts another interrupt to the ioapic with hvm_isa_irq_assert_cb= (). > Intr_pending_nr is incremented in hpet_route_decision_not_missed_ticks(). >=20 > The missed ticks policy intr_en_notif callback also uses the pending_mask > method. So even though > Linux does not broadcast its interrupts, the code could handle it if it d= id. > In this case the end of interrupt time stamp is made when the pending_mas= k is > clear. >=20 > 4. Live Migration >=20 > Live migration with hpet preserves the current offset of the guest clock = with > respect > to ntp. This is accomplished by migrating all of the state in the h->hpet= data > structure > in the usual way. The hp->mc_offset is recalculated on the receiving node= so > that the > guest sees a continuous hpet main counter. >=20 > Code as been added to xc_domain_save.c to send a small message after the > domain context is sent. The contents of the message is the physical tsc > timestamp, last_tsc, > read just before the message is sent. When the last_tsc message is receiv= ed in > xc_domain_restore.c, > another physical tsc timestamp, cur_tsc, is read. The two timestamps are > loaded into the domain > structure as last_tsc_sender and first_tsc_receiver with hypercalls. Then > xc_domain_hvm_setcontext > is called so that hpet_load has access to these time stamps. Hpet_load us= es > the timestamps > to account for the time spent saving and loading the domain context. With= this > technique, > the only neglected time is the time spent sending a small network message= . >=20 > 5. Test Results >=20 > Some recent test results are: >=20 > 5.1 Linux 4u664 and Windows 2k864 load test. > Duration: 70 hours. > Test date: 6/2/08 > Loads: usex -b48 on Linux; burn-in on Windows > Guest vcpus: 8 for Linux; 2 for Windows > Hardware: 8 physical cpu AMD > Clock drift : Linux: .0012% Windows: .009% >=20 > 5.2 Linux 4u664, Linux 4u464 , and Windows 2k864 no-load test > Duration: 23 hours. > Test date: 6/3/08 > Loads: none > Guest vcpus: 8 for each Linux; 2 for Windows > Hardware: 4 physical cpu AMD > Clock drift : Linux: .033% Windows: .019% >=20 > 6. Relation to recent work in xen-unstable >=20 > There is a similarity between hvm_get_guest_time() in xen-unstable and > read_64_main_counter() > in this code. However, read_64_main_counter() is more tuned to the needs = of > hpet.c. It has no > "set" operation, only the get. It isolates the mode, physical or simulate= d, in > read_64_main_counter() > itself. It uses no vcpu or domain state as it is a physical entity, in ei= ther > mode. And it provides a real > physical mode for every read for those applications that desire this. >=20 > 7. Conclusion >=20 > The virtual hpet is improved by this patch in terms of accuracy and > monotonicity. > Tests performed to date verify this and more testing is under way. >=20 > 8. Future Work >=20 > Testing with Windows Vista will be performed soon. The reason for accurac= y > variations > on different platforms using the physical hpet device will be investigate= d. > Additional overhead measurements on simulated vs physical hpet mode will = be > made. >=20 > Footnotes: >=20 > 1. I don't recall the accuracy improvement with end of interrupt stamping= , but > it was > significant, perhaps better than two to one improvement. It would be a ve= ry > simple matter > to re-measure the improvement as the facility can call back at injection = time > as well. >=20 >=20 > Signed-off-by: Dave Winchell > > Signed-off-by: Ben Guthro > >=20 >=20 > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel --B_3295591122_19675835 Content-type: text/html; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy Are t= hese patches needed now the timers are built on Xen system time rather than = host TSC? Dan has reported much better time-keeping with his patch checked i= n, and it’s for sure a lot less invasive than this patchset.

 -- Keir

On 5/6/08 15:59, "Ben Guthro" <bguthro@virtualiron.com> wro= te:


1. Introduction

This patch improves the hpet based guest clock in terms of drift and monoto= nicity.
Prior to this work the drift with hpet was greater than 2%, far above the .= 05% limit
for ntp to synchronize. With this code, the drift ranges from .001% to .003= 3% depending
on guest and physical platform.

Using hpet allows guest operating systems to provide monotonic time to thei= r
applications. Time sources other than hpet are not monotonic because
of their reliance on tsc, which is not synchronized across physical process= ors.

Windows 2k864 and many Linux guests are supported with two policies, one fo= r guests
that handle missed clock interrupts and the other for guests that require t= he
correct number of interrupts.

Guests may use hpet for the timing source even if the physical platform has= no visible
hpet. Migration is supported between physical machines which differ in phys= ical
hpet visibility.

Most of the changes are in hpet.c. Two general facilities are added to trac= k interrupt
progress. The ideas here and the facilities would be useful in vpt.c, for o= ther time
sources, though no attempt is made here to improve vpt.c.

The following sections discuss hpet dependencies, interrupt delivery polici= es, live migration,
test results, and relation to recent work with monotonic time.


2. Virtual Hpet dependencies

The virtual hpet depends on the ability to read the physical or simulated (see discussion below) hpet.  For timekeeping, the virtual hpet also d= epends
on two new interrupt notification facilities to implement its policies for<= BR> interrupt delivery.

2.1. Two modes of low-level hpet main counter reads.

In this implementation, the virtual hpet reads with read_64_main_counter(),= exported by
time.c, either the real physical hpet main counter register directly or a &= quot;simulated"
hpet main counter.

The simulated mode uses a monotonic version of get_s_time() (NOW()), where = the last
time value is returned whenever the current time value is less than the las= t time
value. In simulated mode, since it is layered on s_time, the underlying har= dware
can be hpet or some other device. The frequency of the main counter in simu= lated
mode is the same as the standard physical hpet frequency, allowing live mig= ration
between nodes that are configured differently.

If the physical platform does not have an hpet device, or if xen is configu= red not
to use the device, then the simulated method is used. If there is a physica= l hpet device,
and xen has initialized it, then either simulated or physical mode can be u= sed.
This is governed by a boot time option, hpet-avoid. Setting this option to = 1 gives the
simulated mode and 0 the physical mode. The default is physical mode.

A disadvantage of the physical mode is that may take longer to read the dev= ice
than in simulated mode. On some platforms the cost is about the same (less = than 250 nsec) for
physical and simulated modes, while on others physical cost is much higher = than simulated.
A disadvantage of the simulated mode is that it can return the same value for the counter in consecutive calls.

2.2. Interrupt notification facilities.

Two interrupt notification facilities are introduced, one is hvm_isa_irq_as= sert_cb()
and the other hvm_register_intr_en_notif().

The vhpet uses hvm_isa_irq_assert_cb to deliver interrupts to the vioapic.<= BR> hvm_isa_irq_assert_cb allows a callback to be passed along to vioapic_deliv= er()
and this callback is called with a mask of the vcpus which will get the
interrupt. This callback is made before any vcpus receive an interrupt.

Vhpet uses hvm_register_intr_en_notif() to register a handler for a particu= lar
vector that will be called when that vector is injected in [vmx,svm]_intr_a= ssist()
and also when the guest finishes handling the interrupt. Here finished is d= efined
as the point when the guest re-enables interrupts or lowers the tpr value.<= BR> EOI is not used as the end of interrupt as this is sometimes returned befor= e
the interrupt handler has done its work. A flag is passed to the handler in= dicating
whether this is the injection point (post =3D 1) or the interrupt finished (p= ost =3D 0) point.
The need for the finished point callback is discussed in the missed ticks p= olicy section.

To prevent a possible early trigger of the finished callback, intr_en_notif= logic
has a two stage arm, the first at injection (hvm_intr_en_notif_arm()) and t= he second when
interrupts are seen to be disabled (hvm_intr_en_notif_disarm()). Once fully= armed, re-enabling
interrupts will cause hvm_intr_en_notif_disarm() to make the end of interru= pt
callback. hvm_intr_en_notif_arm() and hvm_intr_en_notif_disarm() are called= by
[vmx,svm]_intr_assist().

3. Interrupt delivery policies

The existing hpet interrupt delivery is preserved. This includes
vcpu round robin delivery used by Linux and broadcast delivery used by Wind= ows.

There are two policies for interrupt delivery, one for Windows 2k8-64 and t= he other
for Linux. The Linux policy takes advantage of the (guest) Linux missed tic= k and offset
calculations and does not attempt to deliver the right number of interrupts= .
The Windows policy delivers the correct number of interrupts, even if somet= imes much
closer to each other than the period. The policies are similar to those in = vpt.c, though
there are some important differences.

Policies are selected with an HVMOP_set_param hypercall with index HVM_PARA= M_TIMER_MODE.
Two new values are added, HVM_HPET_guest_computes_missed_ticks and
HVM_HPET_guest_does_not_compute_missed_ticks.  The reason that two new= ones are added is that
in some guests (32bit Linux) a no-missed policy is needed for clock sources= other than hpet
and a missed ticks policy for hpet. It was felt that there would be less co= nfusion by simply
introducing the two hpet policies.

3.1. The missed ticks policy

The Linux clock interrupt handler for hpet calculates missed ticks and offs= et using the hpet
main counter. The algorithm works well when the time since the last interru= pt is greater than
or equal to a period and poorly otherwise.

The missed ticks policy ensures that no two clock interrupts are delivered = to the guest at
a time interval less than a period. A time stamp (hpet main counter value) = is recorded (by a
callback registered with hvm_register_intr_en_notif) when Linux finishes ha= ndling the clock
interrupt. Then, ensuing interrupts are delivered to the vioapic only if th= e current main
counter value is a period greater than when the last interrupt was handled.=

Tests showed a significant improvement in clock drift with end of interrupt= time stamps
versus beginning of interrupt[1]. It is believed that the reason for the im= provement
is that the clock interrupt handler goes for a spinlock and can be therefor= e delayed in its
processing. Furthermore, the main counter is read by the guest under the lo= ck. The net
effect is that if we time stamp injection, we can get the difference in tim= e
between successive interrupt handler lock acquisitions to be less than the = period.

3.2. The no-missed ticks policy

Windows 2k864 keeps very poor time with the missed ticks policy. So the no-= missed ticks policy
was developed. In the no-missed ticks policy we deliver the correct number = of interrupts,
even if they are spaced less than a period apart (when catching up).

Windows 2k864 uses a broadcast mode in the interrupt routing such that
all vcpus get the clock interrupt. The best Windows drift performance was a= chieved when the
policy code ensured that all the previous interrupts (on the various vcpus)= had been injected
before injecting the next interrupt to the vioapic..

The policy code works as follows. It uses the hvm_isa_irq_assert_cb() to re= cord
the vcpus to be interrupted in h->hpet.pending_mask. Then, in the callba= ck registered
with hvm_register_intr_en_notif() at post=3D1 time it clears the current vcpu= in the pending_mask.
When the pending_mask is clear it decrements hpet.intr_pending_nr and if in= tr_pending_nr is still
non-zero posts another interrupt to the ioapic with hvm_isa_irq_assert_cb()= .
Intr_pending_nr is incremented in hpet_route_decision_not_missed_ticks().
The missed ticks policy intr_en_notif callback also uses the pending_mask m= ethod. So even though
Linux does not broadcast its interrupts, the code could handle it if it did= .
In this case the end of interrupt time stamp is made when the pending_mask = is clear.

4. Live Migration

Live migration with hpet preserves the current offset of the guest clock wi= th respect
to ntp. This is accomplished by migrating all of the state in the h->hpe= t data structure
in the usual way. The hp->mc_offset is recalculated on the receiving nod= e so that the
guest sees a continuous hpet main counter.

Code as been added to xc_domain_save.c to send a small message after the domain context is sent. The contents of the message is the physical tsc tim= estamp, last_tsc,
read just before the message is sent. When the last_tsc message is received= in xc_domain_restore.c,
another physical tsc timestamp, cur_tsc, is read. The two timestamps are lo= aded into the domain
structure as last_tsc_sender and first_tsc_receiver with hypercalls. Then x= c_domain_hvm_setcontext
is called so that hpet_load has access to these time stamps. Hpet_load uses= the timestamps
to account for the time spent saving and loading the domain context. With t= his technique,
the only neglected time is the time spent sending a small network message.<= BR>
5. Test Results

Some recent test results are:

5.1 Linux 4u664 and Windows 2k864 load test.
      Duration: 70 hours.
      Test date: 6/2/08
      Loads: usex -b48 on Linux; burn-in on W= indows
      Guest vcpus: 8 for Linux; 2 for Windows=
      Hardware: 8 physical cpu AMD
      Clock drift : Linux: .0012% Windows: .0= 09%

5.2 Linux 4u664, Linux 4u464 , and Windows 2k864 no-load test
      Duration: 23 hours.
      Test date: 6/3/08
      Loads: none
      Guest vcpus: 8 for each Linux; 2 for Wi= ndows
      Hardware: 4 physical cpu AMD
      Clock drift : Linux: .033% Windows: .01= 9%

6. Relation to recent work in xen-unstable

There is a similarity between hvm_get_guest_time() in xen-unstable and read= _64_main_counter()
in this code. However, read_64_main_counter() is more tuned to the needs of= hpet.c. It has no
"set" operation, only the get. It isolates the mode, physical or = simulated, in read_64_main_counter()
itself. It uses no vcpu or domain state as it is a physical entity, in eith= er mode. And it provides a real
physical mode for every read for those applications that desire this.

7. Conclusion

The virtual hpet is improved by this patch in terms of accuracy and monoton= icity.
Tests performed to date verify this and more testing is under way.

8. Future Work

Testing with Windows Vista will be performed soon. The reason for accuracy = variations
on different platforms using the physical hpet device will be investigated.=
Additional overhead measurements on simulated vs physical hpet mode will be= made.

Footnotes:

1. I don't recall the accuracy improvement with end of interrupt stamping, = but it was
significant, perhaps better than two to one improvement. It would be a very= simple matter
to re-measure the improvement as the facility can call back at injection ti= me as well.


Signed-off-by: Dave Winchell <dwinchell@virtualiron.com> <mailto:dwinchell@virtualiron.com> =
Signed-off-by: Ben Guthro <bguthro@virtualiron.com> <mailto:bguthro@virtualiron.com>

____________________= ___________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/x= en-devel

 --B_3295591122_19675835-- --===============0099531770== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============0099531770==--