From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Fri, 1 Feb 2008 15:31:11 -0700 Message-ID: <20080201153111687.00000002384@djm-pc> References: <47A0EFC7.3020102@virtualiron.com> Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: multipart/mixed; boundary=-------3c0352933c035293 Return-path: In-Reply-To: <47A0EFC7.3020102@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "xen-devel@lists.xensource.com" , Deepak Patel List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format ---------3c0352933c035293 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable OK, Deepak repeated the test without ntpd and using ntpdate -b before the test. The attached graph shows his results: el5u1-64 (best=3D~0.07%), el4u5-64 (middle=3D~0.2%), and el4u5-32 (worst=3D~0.3%). We will continue to look at LTP to try to isolate. Thanks, Dan P.S. elXuY is essentially RHEL XuY with some patches. > -----Original Message----- > From: Dave Winchell [mailto:dwinchell@virtualiron.com] > Sent: Wednesday, January 30, 2008 2:45 PM > To: Deepak Patel > Cc: dan.magenheimer@oracle.com; Keir Fraser; > xen-devel@lists.xensource.com; akira.ijuin@oracle.com; Dave Winchell > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that = > disables pending > missed ticks > = > = > Dan, Deeepak, > = > It may be that the underlying clock error is too great for ntp > to handle. It would be useful if you did not run ntpd > and, instead did ntpdate -b at the start of the test > for each guest. Then capture the data as you have been doing. > If the drift is greater than .05%, then we need to address that. > = > Another option is, when running ntpd, to enable loop statistics in > /etc/ntp.conf > by adding this to the file: > = > statistics loopstats > statsdir /var/lib/ntp/ > = > Then you will see loop data in that directory. > Correlating the data in the loopstats files with the > peaks in skew would be interesting. You will see entries of the form > = > 54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511 10 > = > Where the second to last column is the Allan Deviation. When that > gets over 1000, ntpd is working pretty hard. However, I have = > not seen ntpd > completely loose it like you have. > = > I'm on vacation until Monday, and won't be reading > email. > = > Thanks for all your work on this! > = > -Dave > = > Deepak Patel wrote: > = > > > >> > >> Is the graph for RHEL5u1-64? (I've never tested this one.) > > > > > > I do not know which graph was attached with this. But I saw this > > behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I > > was running ltp tests continuously. > > > >> What was the behaviour of the other guests running? > > > > > > All pvm guests are fine. But behavior of most of the hvm guests were > > as described. > > > >> If they had spikes, were they at the same wall time? > > > > > > No. They are not at the same wall time. > > > >> Were the other guests running ltp as well? > >> > > Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp > > continuously. > > > >> How are you measuring skew? > > > > > > I was collecting output of "ntpdate -q every = > 300 seconds > > (5 minutes) and have created graph based on that. > > > >> > >> Are you running ntpd? > >> > > Yes. ntp was running on all the guests. > > > > I am investigating what causes this spikes and let everyone = > know what > > are my findings. > > > > Thanks, > > Deepak > > > >> Anything that you can discover that would be in sync with > >> the spikes would be very helpful! > >> > >> The code that I test with is our product code, which is based > >> on 3.1. So it is possible that something in 3.2 other than vpt.c > >> is the cause. I can test with 3.2, if necessary. > >> > >> thanks, > >> Dave > >> > >> > >> > >> Dan Magenheimer wrote: > >> > >>> Hi Dave (Keir, see suggestion below) -- > >>> > >>> Thanks! > >>> > >>> Turning off vhpet certainly helps a lot (though see below). > >>> > >>> I wonder if timekeeping with vhpet is so bad that it should be > >>> turned off by default (in 3.1, 3.2, and unstable) until it is > >>> fixed? (I have a patch that defaults it off, can post it if > >>> there is agreement on the above point.) The whole point of an > >>> HPET is to provide more precise timekeeping and if vhpet is > >>> worse than vpit, it can only confuse users. Comments? > >>> > >>> > >>> In your testing, are you just measuring % skew over a long > >>> period of time? > >>> We are graphing the skew continuously and > >>> seeing periodic behavior that is unsettling, even with pit. > >>> See attached. Though your algorithm recovers, the "cliffs" > >>> could still cause real user problems. I wonder if there is > >>> anything that can be done to make the "recovery" more > >>> responsive? > >>> > >>> We are looking into what part(s) of LTP is causing the cliffs. > >>> > >>> Thanks, > >>> Dan > >>> > >>> > >>> > >>>> -----Original Message----- > >>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>> Sent: Monday, January 28, 2008 8:21 AM > >>>> To: dan.magenheimer@oracle.com > >>>> Cc: Keir Fraser; xen-devel@lists.xensource.com; > >>>> deepak.patel@oracle.com; > >>>> akira.ijuin@oracle.com; Dave Winchell > >>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables > >>>> pending > >>>> missed ticks > >>>> > >>>> > >>>> Dan, > >>>> > >>>> I guess I'm a bit out of date calling for clock=3D usage. > >>>> Looking at linux 2.6.20.4 sources, I think you should specify > >>>> "clocksource=3Dpit nohpet" on the linux guest bootline. > >>>> > >>>> You can leave the xen and dom0 bootlines as they are. > >>>> The xen and guest clocksources do not need to be the same. > >>>> In my tests, xen is using the hpet for its timekeeping and > >>>> that appears to be the default. > >>>> > >>>> When you boot the guests you should see > >>>> time.c: Using PIT/TSC based timekeeping. > >>>> on the rh4u5-64 guest, and something similar on the others. > >>>> > >>>> > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer > >>>> > 14.318MHz HPET.) > >>>> > >>>> This appears to be the xen state, which is fine. > >>>> I was wrongly assuming that this was the guest state. > >>>> You might want to look in your guest logs and see what they were > >>>> picking > >>>> for a clock source. > >>>> > >>>> Regards, > >>>> Dave > >>>> > >>>> > >>>> > >>>> > >>>> Dan Magenheimer wrote: > >>>> > >>>> > >>>> > >>>>> Thanks, I hadn't realized that! No wonder we didn't = > see the same > >>>>> improvement you saw! > >>>>> > >>>>> > >>>>> > >>>>>> Try specifying clock=3Dpit on the linux boot line... > >>>>>> > >>>>> > >>>>> > >>>>> I'm confused... do you mean "clocksource=3Dpit" on the Xen > >>>> > >>>> > >>>> command line or > >>>> > >>>> > >>>>> "nohpet" / "clock=3Dpit" / "clocksource=3Dpit" on the guest (or > >>>> > >>>> > >>>> dom0?) command > >>>> > >>>> > >>>>> line? Or both places? Since the tests take awhile, it > >>>> > >>>> > >>>> would be nice > >>>> > >>>> > >>>>> to get this right the first time. Do the Xen and guest > >>>> > >>>> > >>>> clocksources need > >>>> > >>>> > >>>>> to be the same? > >>>>> > >>>>> Thanks, > >>>>> Dan > >>>>> > >>>>> -----Original Message----- > >>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>> *Sent:* Sunday, January 27, 2008 2:22 PM > >>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser > >>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; > >>>>> akira.ijuin@oracle.com; Dave Winchell > >>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode = > that disables > >>>>> pending missed ticks > >>>>> > >>>>> Hi Dan, > >>>>> > >>>>> Hpet timer does have a fairly large error, as I was = > trying this > >>>>> one recently. > >>>>> I don't remember what I got for error, but 1% sounds > >>>> > >>>> > >>>> about right. > >>>> > >>>> > >>>>> The problem is that hpet is not built on top of vpt.c, > >>>> > >>>> > >>>> the module > >>>> > >>>> > >>>>> Keir and I did > >>>>> all the recent work in, for its periodic timer needs. Try > >>>>> specifying clock=3Dpit > >>>>> on the linux boot line. If it still picks the hpet, which it > >>>>> might, let me know > >>>>> and I'll tell you how to get around this. > >>>>> > >>>>> Regards, > >>>>> Dave > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>>> -------------------------------------------------------------- > >>>> ---------- > >>>> > >>>> > >>>>> *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > >>>>> *Sent:* Fri 1/25/2008 6:50 PM > >>>>> *To:* Dave Winchell; Keir Fraser > >>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; > >>>>> akira.ijuin@oracle.com > >>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode > >>>> > >>>> > >>>> that disables > >>>> > >>>> > >>>>> pending missed ticks > >>>>> > >>>>> Sorry for the very late followup on this but we finally > >>>> > >>>> > >>>> were able > >>>> > >>>> > >>>>> to get our testing set up again on stable 3.1 bits and have > >>>>> seen some very bad results on 3.1.3-rc1, on the order of 1%. > >>>>> > >>>>> Test enviroment was a 4-socket dual core machine with 24GB of > >>>>> memory running six two-vcpu 2GB domains, four hvm = > plus two pv. > >>>>> All six guests were running LTP simultaneously. The four hvm > >>>>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and = > RHEL4u5-64. > >>>>> Timer_mode was set to 2 for 64-bit guests and 0 for > >>>> > >>>> > >>>> 32-bit guests. > >>>> > >>>> > >>>>> All four hvm guests experienced skew around -1%, = > even the 32-bit > >>>>> guest. Less intensive testing didn't exhibit much = > skew at all. > >>>>> > >>>>> A representative graph is attached. > >>>>> > >>>>> Dave, I wonder if some portion of your patches = > didn't end up in > >>>>> the xen trees? > >>>>> > >>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer > >>>>> 14.318MHz HPET.) > >>>>> > >>>>> Thanks, > >>>>> Dan > >>>>> > >>>>> P.S. Many thanks to Deepak and Akira for running tests. > >>>>> > >>>>> > -----Original Message----- > >>>>> > From: xen-devel-bounces@lists.xensource.com > >>>>> > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > >>>>> > Dave Winchell > >>>>> > Sent: Wednesday, January 09, 2008 9:53 AM > >>>>> > To: Keir Fraser > >>>>> > Cc: dan.magenheimer@oracle.com; > >>>> > >>>> > >>>> xen-devel@lists.xensource.com; Dave > >>>> > >>>> > >>>>> > Winchell > >>>>> > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>>>> > disables pending > >>>>> > missed ticks > >>>>> > > >>>>> > > >>>>> > Hi Keir, > >>>>> > > >>>>> > The latest change, c/s 16690, looks fine. > >>>>> > I agree that the code in c/s 16690 is equivalent to > >>>>> > the code I submitted. Also, your version is more > >>>>> > concise. > >>>>> > > >>>>> > The error tests confirm the equivalence. With > >>>> > >>>> > >>>> overnight cpu loads, > >>>> > >>>> > >>>>> > the checked in version was accurate to +.048% for sles > >>>>> > and +.038% for red hat. My version was +.046% and = > +.032% in a > >>>>> > 2 hour test. > >>>>> > I don't think the difference is significant. > >>>>> > > >>>>> > i/o loads produced errors of +.01%. > >>>>> > > >>>>> > Thanks for all your efforts on this issue. > >>>>> > > >>>>> > Regards, > >>>>> > Dave > >>>>> > > >>>>> > > >>>>> > > >>>>> > Keir Fraser wrote: > >>>>> > > >>>>> > >Applied as c/s 16690, although the checked-in patch is > >>>>> > smaller. I think the > >>>>> > >only important fix is to pt_intr_post() and the = > only bit of > >>>>> > the patch I > >>>>> > >totally omitted was the change to = > pt_process_missed_ticks(). > >>>>> > I don't think > >>>>> > >that change can be important, but let's see what > >>>> > >>>> > >>>> happens to the > >>>> > >>>> > >>>>> error > >>>>> > >percentage... > >>>>> > > > >>>>> > > -- Keir > >>>>> > > > >>>>> > >On 4/1/08 23:24, "Dave Winchell" > >>>> > >>>> > >>>> wrote: > >>>> > >>>> > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > >>Hi Dan and Keir, > >>>>> > >> > >>>>> > >>Attached is a patch that fixes some issues with the > >>>> > >>>> > >>>> SYNC policy > >>>> > >>>> > >>>>> > >>(no_missed_ticks_pending). > >>>>> > >>I have not tried to make the change the minimal one, but, > >>>>> > rather, just > >>>>> > >>ported into > >>>>> > >>the new code what I know to work well. The error for > >>>>> > >>no_missed_ticks_pending goes from > >>>>> > >>over 3% to .03% with this change according to my testing. > >>>>> > >> > >>>>> > >>Regards, > >>>>> > >>Dave > >>>>> > >> > >>>>> > >>Dan Magenheimer wrote: > >>>>> > >> > >>>>> > >> > >>>>> > >> > >>>>> > >>>Hi Dave -- > >>>>> > >>> > >>>>> > >>>Did you get your correction ported? If so, it would be > >>>>> > nice to see this get > >>>>> > >>>into 3.1.3. > >>>>> > >>> > >>>>> > >>>Note that I just did some very limited testing with > >>>>> > timer_mode=3D2(=3DSYNC=3Dno > >>>>> > >>>missed ticks pending) > >>>>> > >>>on tip of xen-3.1-testing (64-bit Linux hv = > guest) and the > >>>>> > worst error I've > >>>>> > >>>seen so far > >>>>> > >>>is 0.012%. But I haven't tried any exotic = > loads, just LTP. > >>>>> > >>> > >>>>> > >>>Thanks, > >>>>> > >>>Dan > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>>>-----Original Message----- > >>>>> > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM > >>>>> > >>>>To: dan.magenheimer@oracle.com > >>>>> > >>>>Cc: Keir Fraser; Shan, Haitao; > >>>>> > xen-devel@lists.xensource.com; Dong, > >>>>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell > >>>>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>>>> > >>>>disables pending > >>>>> > >>>>missed ticks > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>>Dan, > >>>>> > >>>> > >>>>> > >>>>I did some testing with the constant tsc offset > >>>> > >>>> > >>>> SYNC method > >>>> > >>>> > >>>>> > >>>>(now called > >>>>> > >>>>no_missed_ticks_pending) > >>>>> > >>>>and found the error to be very high, much larger > >>>> > >>>> > >>>> than 1 %, as > >>>> > >>>> > >>>>> > >>>>I recall. > >>>>> > >>>>I have not had a chance to submit a correction. I > >>>> > >>>> > >>>> will try to > >>>> > >>>> > >>>>> > >>>>do it later > >>>>> > >>>>this week or the first week in January. My version of > >>>>> constant tsc > >>>>> > >>>>offset SYNC method > >>>>> > >>>>produces .02 % error, so I just need to port = > that into the > >>>>> > >>>>current code. > >>>>> > >>>> > >>>>> > >>>>The error you got for both of those kernels is > >>>> > >>>> > >>>> what I would > >>>> > >>>> > >>>>> expect > >>>>> > >>>>for the default mode, delay_for_missed_ticks. > >>>>> > >>>> > >>>>> > >>>>I'll let Keir answer on how to set the time mode. > >>>>> > >>>> > >>>>> > >>>>Regards, > >>>>> > >>>>Dave > >>>>> > >>>> > >>>>> > >>>>Dan Magenheimer wrote: > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>>>Anyone make measurements on the final patch? > >>>>> > >>>>> > >>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and = > saw a loss of > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>about 0.2% with no load. This was = > xen-unstable tip today > >>>>> > >>>>with no options specified. 32-bit was about 0.01%. > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>>>I think I missed something... how do I run the various > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>accounting choices and which ones are known to be > >>>> > >>>> > >>>> appropriate > >>>> > >>>> > >>>>> > >>>>for which kernels? > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>>>Thanks, > >>>>> > >>>>>Dan > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>-----Original Message----- > >>>>> > >>>>>>From: xen-devel-bounces@lists.xensource.com > >>>>> > > >>>>> > >>>>>>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>Keir Fraser > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM > >>>>> > >>>>>>To: Dave Winchell > >>>>> > >>>>>>Cc: Shan, Haitao; = > xen-devel@lists.xensource.com; Dong, > >>>>> > Eddie; Jiang, > >>>>> > >>>>>>Yunhong > >>>>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer = > mode that > >>>>> > >>>>>>disables pending > >>>>> > >>>>>>missed ticks > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>Please take a look at xen-unstable changeset 16545. > >>>>> > >>>>>> > >>>>> > >>>>>>-- Keir > >>>>> > >>>>>> > >>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>> wrote: > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>Keir, > >>>>> > >>>>>>> > >>>>> > >>>>>>>The accuracy data I've collected for i/o = > loads for the > >>>>> > >>>>>>>various time protocols follows. In = > addition, the data > >>>>> > >>>>>>>for cpu loads is shown. > >>>>> > >>>>>>> > >>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 > >>>> > >>>> > >>>> processor AMD > >>>> > >>>> > >>>>> box. > >>>>> > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. > >>>>> > >>>>>>>The cpu load is usex -e36 on each guest. > >>>>> > >>>>>>>(usex is available at > >>>>> http://people.redhat.com/anderson/usex.) > >>>>> > >>>>>>>i/o load is 8 instances of dd if=3D/dev/hda6 > >>>> > >>>> > >>>> of=3D/dev/null. > >>>> > >>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. > >>>>> > >>>>>>>Also, these are run on 4 cpu AMD box. > >>>>> > >>>>>>>In addition, there is an idle rh-32bit guest. > >>>>> > >>>>>>>All three guests are 8vcpu. > >>>>> > >>>>>>> > >>>>> > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 > >>>>> > >>>>>>>except that the redhat-64 guest has 4 = > instances of dd. > >>>>> > >>>>>>> > >>>>> > >>>>>>>Date Duration Protocol sles, rhat error load > >>>>> > >>>>>>> > >>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 = > sec -.006%, > >>>>> > +.005% cpu > >>>>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 = > sec, -.001%, > >>>>> > +.012% cpu > >>>>> > >>>>>>> > >>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, > >>>>> -.004% cpu > >>>>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, > >>>> > >>>> > >>>> -.005%, -.005% cpu > >>>> > >>>> > >>>>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, > >>>> > >>>> > >>>> -.008%, -.003% cpu > >>>> > >>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, > >>>> > >>>> > >>>> -.040% cpu > >>>> > >>>> > >>>>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 = > sec, -.034%, > >>>>> > -.031% cpu > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, > >>>>> > -.09% i/o-8 > >>>>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% > >>>>> > -.14% i/o-8 > >>>>> > >>>>>>> > >>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, > >>>>> > -.022% i/o-8 > >>>>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, > >>>> > >>>> > >>>> -.017%, -.018% i/o-8 > >>>> > >>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, > >>>>> > -.029% i/o-8 > >>>>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 = > sec, -.023%, > >>>>> > -.031% i/o-8 > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, > >>>> > >>>> > >>>> -.04% i/o-32 > >>>> > >>>> > >>>>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, > >>>>> > -.005% i/o-32 > >>>>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, > >>>> > >>>> > >>>> -.11% i/o-32 > >>>> > >>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, > >>>>> > .003% i/o-4/32 > >>>>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, > >>>>> > .01% i/o-4/32 > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>Overhead measurements: > >>>>> > >>>>>>> > >>>>> > >>>>>>>Progress in terms of number of passes = > through a fixed > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>system workload > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. > >>>>> > >>>>>>>The workload was usex -b48. > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min > >>>>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min > >>>>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min > >>>>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>Conclusions: > >>>>> > >>>>>>> > >>>>> > >>>>>>>The only protocol which meets the .05% accuracy > >>>>> > requirement for ntp > >>>>> > >>>>>>>tracking under the loads > >>>>> > >>>>>>>above is the SYNC protocol. The worst case > >>>> > >>>> > >>>> accuracies for > >>>> > >>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>SYNC, MIXED, > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>and ASYNC > >>>>> > >>>>>>>are .022%, .12%, and .14%, respectively. > >>>>> > >>>>>>> > >>>>> > >>>>>>>We could reduce the cost of the SYNC method by only > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>scheduling the extra > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>wakeups if a certain number > >>>>> > >>>>>>>of ticks are missed. > >>>>> > >>>>>>> > >>>>> > >>>>>>>Regards, > >>>>> > >>>>>>>Dave > >>>>> > >>>>>>> > >>>>> > >>>>>>>Keir Fraser wrote: > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>> wrote: > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>>>Since I had a high error (~.03%) for the > >>>> > >>>> > >>>> ASYNC method a > >>>> > >>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>couple of days ago, > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>I ran another ASYNC test. I think there may have > >>>>> > been something > >>>>> > >>>>>>>>>wrong with the code I used a couple of = > days ago for > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>ASYNC. It may have been > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>missing the immediate delivery of interrupt > >>>> > >>>> > >>>> after context > >>>> > >>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>switch in. > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>My results indicate that either SYNC or ASYNC give > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>acceptable accuracy, > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>each running consistently around or under > >>>> > >>>> > >>>> .01%. MIXED has > >>>> > >>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>a fairly high > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>error of > >>>>> > >>>>>>>>>greater than .03%. Probably too close to .05% ntp > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>threshold for comfort. > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I > >>>> > >>>> > >>>> plan to leave > >>>> > >>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>SYNC running > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>over the weekend. If you'd rather I can = > leave MIXED > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>running instead. > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>It may be too early to pick the protocol and > >>>> > >>>> > >>>> I can run > >>>> > >>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>more overnight tests > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>next week. > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>I'm a bit worried about any unwanted side > >>>> > >>>> > >>>> effects of the > >>>> > >>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>SYNC+run_timer > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will > >>>> > >>>> > >>>> cause higher > >>>> > >>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>system-wide CPU > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>contention. I find it easier to think through the > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>implications of ASYNC. I'm > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>surprised that MIXED loses time, and is less > >>>> > >>>> > >>>> accurate than > >>>> > >>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>ASYNC. Perhaps it > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>delivers more timer interrupts than the other > >>>> > >>>> > >>>> approaches, > >>>> > >>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>and each interrupt > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>event causes a small accumulated error? > >>>>> > >>>>>>>> > >>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as > >>>> > >>>> > >>>> favourites and > >>>> > >>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>if the latter is > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>actually more accurate then I can simply revert the > >>>>> > changeset that > >>>>> > >>>>>>>>implemented MIXED. > >>>>> > >>>>>>>> > >>>>> > >>>>>>>>Perhaps rather than running more of the same > >>>> > >>>> > >>>> workloads you > >>>> > >>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>could try idle > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated > >>>> > >>>> > >>>> large disc reads > >>>> > >>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>to /dev/null)? We > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>don't have any data on workloads that aren't > >>>> > >>>> > >>>> CPU bound, so > >>>> > >>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>that's really an > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>obvious place to put any further effort imo. > >>>>> > >>>>>>>> > >>>>> > >>>>>>>>-- Keir > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>_______________________________________________ > >>>>> > >>>>>>Xen-devel mailing list > >>>>> > >>>>>>Xen-devel@lists.xensource.com > >>>>> > >>>>>>http://lists.xensource.com/xen-devel > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c > >>>>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 = > 2007 +0000 > >>>>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 = > 2008 -0500 > >>>>> > >>@@ -58,7 +58,7 @@ static void = > pt_process_missed_ticks(stru > >>>>> > >> > >>>>> > >> missed_ticks =3D missed_ticks / (s_time_t) > >>>> > >>>> > >>>> pt->period + 1; > >>>> > >>>> > >>>>> > >> if ( mode_is(pt->vcpu->domain, > >>>> > >>>> > >>>> no_missed_ticks_pending) ) > >>>> > >>>> > >>>>> > >>- pt->do_not_freeze =3D !pt->pending_intr_nr; > >>>>> > >>+ pt->do_not_freeze =3D 1; > >>>>> > >> else > >>>>> > >> pt->pending_intr_nr +=3D missed_ticks; > >>>>> > >> pt->scheduled +=3D missed_ticks * pt->period; > >>>>> > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) > >>>>> > >> > >>>>> > >> pt_lock(pt); > >>>>> > >> > >>>>> > >>- pt->pending_intr_nr++; > >>>>> > >>+ if ( mode_is(pt->vcpu->domain, > >>>> > >>>> > >>>> no_missed_ticks_pending) ) { > >>>> > >>>> > >>>>> > >>+ pt->pending_intr_nr =3D 1; > >>>>> > >>+ pt->do_not_freeze =3D 0; > >>>>> > >>+ } > >>>>> > >>+ else > >>>>> > >>+ pt->pending_intr_nr++; > >>>>> > >> > >>>>> > >> if ( !pt->one_shot ) > >>>>> > >> { > >>>>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct = > vcpu *v, struct > >>>>> > >> return; > >>>>> > >> } > >>>>> > >> > >>>>> > >>- pt->do_not_freeze =3D 0; > >>>>> > >>- > >>>>> > >> if ( pt->one_shot ) > >>>>> > >> { > >>>>> > >> pt->enabled =3D 0; > >>>>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu > >>>> > >>>> > >>>> *v, struct > >>>> > >>>> > >>>>> > >> pt->last_plt_gtime =3D hvm_get_guest_time(v); > >>>>> > >> pt->pending_intr_nr =3D 0; /* 'collapse' all > >>>>> > missed ticks */ > >>>>> > >> } > >>>>> > >>+ else if ( mode_is(v->domain, = > no_missed_ticks_pending) ) { > >>>>> > >>+ pt->pending_intr_nr--; > >>>>> > >>+ pt->last_plt_gtime =3D hvm_get_guest_time(v); > >>>>> > >>+ } > >>>>> > >> else > >>>>> > >> { > >>>>> > >> pt->last_plt_gtime +=3D pt->period_cycles; > >>>>> > >> > >>>>> > >> > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > >>>>> > > >>>>> > _______________________________________________ > >>>>> > Xen-devel mailing list > >>>>> > Xen-devel@lists.xensource.com > >>>>> > http://lists.xensource.com/xen-devel > >>>>> > > >>>>> > >>>>> > >>>> > >>>> > >> > = > ---------3c0352933c035293 Content-Type: application/octet-stream; name="hvm-compare.png" Content-Disposition: attachment; filename="hvm-compare.png" Content-Transfer-Encoding: base64 iVBORw0KGgoAAAANSUhEUgAAAaMAAAGkEAAAAADpa8nTAAAACXBIWXMAAABIAAAASABGyWs+ AAAACXZwQWcAAAGjAAABpADnhMYnAAAhMElEQVR42u2dv46syNnG0SdL1soROpYsqyXLQnJi TeJFljrcgGCs1YRIG05goQk3Yy4BzR0QOyOZC+AWuAVugVs43zxbU8ufgaZoKKoKnl9w+nQ3 00B1PV1Vb71/vO+EkJV4pi+AEPehjAhZDWW0E56XpqavgeiCMtoJyujIUEY7QRkdGcpoJyij I0MZ7QRklKa+73lBIAQVBHEs300S3//+Pc+jqKqiyPN8P03LMgw9b05+aRoE7Wd+/15VcYy/ al+JorJMErwWRXWNM3leGFYV3uufsWnwWtMkCT7T8+K4PappcFT3DHFclqZb1RYoo50Q3bj4 AN0xzyEAz5MdF51YvBKGOAZd3ffzvCzRZXH0OBAmBIe/xCdUle/jE8QrSSLO7Pvo8nkuRFwU +F8QiL8XghPHC1lDaDhz/6g4xlHfv9e177f3IWRGKKOdQHeVopFdXgokz0WHRHcVv/Dt/+TR Y9S152WZ+H+ShKGQgDiL+Iy6xpmjqH2l/Xz5KD8d/8dV5HlRtK/IRylljFTy/GHYjqfnhjLa ia4Y5P/ltC6OIYG223b/d2tVlWVyPJP4vhiBgBRZXyj9/7UjYl+UEHlRYFI5PApnKD/BVNF0 u9oBm2EnxmQkZIDxRvzW35JRXZcDmqZ71NezyGdzMhoeXxRiZRSGXRm1x/Qx3a52wGbYiTEZ 4fc/zzGlE7/1t2SEV/qU5dho1J4F8lwyGonjyxJrKrHmGRNb9wxEQhntxJiMxOoiDOVEbOmk Dqurdm2ENUsc99dGkMOcjPpro+6Zx2QUx+3aKE3bKeS5oYx2YlxGGE/ksn+5jIRs5i11t2WE 91tLHUYjYWIQ14b/d68GZ4ClTqyMpDHi7FBGOzEuI0ykur/uS2UkTN63943mZCT2p9p9oywL PoApQRjehcTaM8ozQK6mW9UWKKNT89VIQe6BjXhqKKNtYCOeGspoG9iIpwbecqav4QhQRoSs hjIiZDWUESGroYwIWQ1lRMhqKCNCVkMZEbIayoiQ1VBGhKyGMiJkNTvICAmb4FifJP1ITUKO wg4yQs4aZBxoozwJORY7yEhGdyKu0vTtEqID7R27K542XJqQI2FMRk3zn//8/MHbJ+/vJSGW 8f4u+2fywevr1OremIxeX//vgz/84Y9//NOf/vGPf//75eWf//z5518++PXXdJTHx+fndBGX y7Lj0/R6XXb8w8PTk94zLL+L5+fHR71nWH4XT08PD8v+Qv+3rdJOLy/XTy4f/PDD66shGU2t jZLkchGv/ve/v/4aRUHw5z//618//fTzz3/5C1Lmfv/+44/RB74v80Rfr7/8kqZFoW7v+/Zt 6bUuDWK7Xt/e9J5h+V0gBYneMyy/i7e363XZX4iM4TrvYnk7XS5TRjJjljopoxbkBc2yNEXm NuRvu16RGRQJBsX7f/vb21ue4xdCPP/rXyGzOH5/F8/f3mTW63sbljLSdReU0WrG943mGraq IBmMUkIsafrLL8iOBrGJI0QjtI1xvcZx9BviPN8+wDOZEFFOEuQZ0hQ5QtftZS3/spejP8xb /xmWd1kb72L6R9OYCXrZ7xOWe5CVFFaSQARi8jeNaFj5Bf74o1g2yub+8UeMfWEon889jlVP oIzUoIw0sXyY7wI5ZFkUhSFWS1NyWtKwbTmS6XN+fY0yUoMy0sQ6GUmaBl+QGKHEJK377pKG rWus4OaPa0uXAMpIDcpIE+/v0lSwDRBUlsGcEYZyuresFhymePNH5Xl37Ktr/X6C+iva6T9D 08xNwF24i5cXac4aYkxG+n6fmibPYWyYLvU4hdp4RM7K9NzjgDISoFhwGOr5DaTYzskJZSTO cd/4Mm20EMgK3eRcnFRG9ybXzTJsAU9LiaPROTmtjMQm633Xx+zWpM+JZYSqcNt/Kid1Z+TE Mlq/nzA2mrUV8sh5OLWM1nZ5OAxx9CEnl1EUrTV7w+JHIZFTy2gLh52q6stoD+8FYhunlhE8 w7f+zCy7z/5HXObUMqoqJvYiW3BqGcFXbpvPga8eJ3PnhTLaiKJASAacXrFWEoGEbeNWlQgM lFIbPtfv5Ux0cnIZ6fCBgzSlTKQ8pKymnsvIKHk1eA7a47uiI7ZxchkhQtbUfc6B3BJSOJBd kvRlhiwUHMXswEoZiSxhe5yrKPaT7LaI8Y4yMo/InWehjJ6f0UX2OFdVjaUjWUdd7925Odkz Cfoqeuz4u6eY1G1pZJDk+fLo2nXcE89LtsTKSd2eMsIqw/Vf86aJY9aIMsnpZYTR4wh+cWpp V4geKKPvWCEhq912v+ZLsomTI0AZfZLnQbCV8XuYFY8cHcrod5oGiSFdH0dQbYej4d5QRj2Q 6AQTPFPn3wKRLbbdqN0jf+vZoYxG2CIQ3BZXVexquP7DYD+U0QhbZP6pKjtkJOC+kl4ooxGY QIssgzIaIUm2WU24vxtF1KCMRthqUb69o9E6OMrqgjIaIU2PuZaABc/ewBCXoYxGcDd8Yg5k n5BCQuA7Ypbk1FOEXpi+QjehjDRfQRzbGhM0DGQXham/RuBOPcfxKF1tk0XSFJTRCFW13SrC VhHdS2vIHwbCIzoXfiBTgfJD2R4JymiUrRfjR+w6Y3RlUpZ92cAHvS8789/zVlBGo2yd0H59 muPjgYoe5r/pbaCMRrHNVH1UjhDpBXaWUZoGgef5vojVbJok8T7oR27aICPusOzDUcboXWWU 576P09V1GCKVCEoZNx+EYTcJsA0yiuOtTb/H3Inaitbs7ia7yqgsZWcqS+/j8z1PnFw8a48y L6PtgwuYIP8WiD922QxjaG2EyV1XPFJQwA4ZcfTYF2TEMH0N92NARnWdJL6PXYQpGV0u3z6I ov52354c14/BXorCpZUS9hYF6KuXi0YZeT3wStOkKUwKaDCbR6Ptr4GONqq4WGht19EIxoSu c4y9a6Mt/RjkJ1JGahRFGLo2pd5VRig53H1ur6WOJm+T1LVrrb+rjKKoP8mzd99oez8GsgTX 7Hb0YphAR25v0/dEdEEZTbD9tIIyOi6U0QQ66vCRo0IZTcANWLM0jUvh7pTRBE0TBNsuc/Pc pe1F06D93ZkPUEaTbF3qxK1devNUlTtCooxucJRoGFfRUVJUD5TRzSsJAnhNJcn4hIwiIwLK SIGqkllxxgiCMBT/Q1B0nsN7bkxgdU13oPuw39xAGW165fAMB0JeyJMDhLT+94HpK3QTuIvZ bTeljDTSNJCPTDQFcYUhi6TcA4Rkcwk3ymh3soxVw5fTNO36VP4smb6mFspoR+R9IdKTQrof mQFPPEOxUZFoUr6//8YCZbQj7fyePhJb08oKYRYC8byNUx1/XtcY2dqceWIqvizMkjIyAnIj mb4GIhjKRsiqnTRiG7ibVHkMysgQdH11ibrO825o6RDKaEfwOyf/XxS3vhbiFpTRrnS3EWlm OA6UkTHSlCkgXWPqh48yMgYsSiLVGHGFKa9/K2V0vbbmyKPRNy0URRDQ9O0SQTD84UNfvV4t lNGRR6Phl9A0cUxjgzuU5dgPvJWj0ZFlNAa87TgmuUKef10hUUZW3CHymjMznrtQRkYQU7v+ r1qSMB7JVSgjg8hiZEJOWcaJnatQRhYgCiyX5d//LkYpGVkjq3i31byJnVBG1tA0P/0kW0A8 ynaQj7KeeRQJWVFctkAZWcQyIwPibLqPrdzkV6r6SNZCGVmEHG2WIkel4eil+ihl2H9ESTfT LeIKlJFFpKlNGXAQh4NH96oN7Q9lZBE2V5yVoiJjUEYWYXOWUJR0M30N9kIZWQU9GdyEMrIK rkLchDKyCrsdgpAV1vQ12AllZBVZZnNELLOQT0EZWcWZ791lKCOrOPO9uwxlZBVlabtZeetS nsfAgIyaxvdlgECSeB/0U8OfWUb2ewxQRGMYkFGaep44aZIgVxuKbnR/g88sI5q83WR3GWVZ WUoZyUe80h5BGdlNXXNEGrKzjIS7i5BPVzxSUIAyshv8EJq+BtvYVUaYviEUYE5GDw/XD2Re /3MlRPSMmXbIEtq6E+irDw8aZeT1EFXSxOu3ZfT09PaBLJdxrikEC7a4QVvOBX316WnH0SiK +rLi2ugr9k/q8A2da4Ywj5F9I1rqpnFBRnnO2kx9jMqI+0ZfkelKiEvQi8Eyooh2MPegjCxj +guxibE81mdGWUb7Wc0oI9PXME+W0cjQZVZGTZNlYdja14JAd2gZZWT6GshSZmSUpmGItE/S MgNreZbFcRTps9WcXUauZPKuKk7sJDMymnqzrikjPeS5K3dvczqwvVlkYqiqPWbE55bRue/e VZRkVBSIECoKrI30TznO3ZFcu/ui4FpOUUZBkGXwNUCT6c+k5lpHOvfd1zWzBSnKCB5vTSP8 3vR7ILvWkba+e3szp5IpFEcjYZ9rxaSTc8uoLbPiDvCLNH0NZlGSUZ5jVYTBO0n0/1aeW0Zu OKf2Ofs3pmypqyph4N7DCeTsX4p7MiL0qbMOBu65x4yMvC/o/608u4zcDCM/t+F71osBZJnv I5EFHvWbN88uIzcndTr9WuxHaVIXhm3RXe4b6cZNGZ0b5X2jsf/rgTIyfQVkKUoyksmCORrt wb31yE1z5u9NSUZp6vt5XpZ57vv6m+rMXweIYzcX603jpvy3QNHgnaZBgJC9PTp4WV6vUXTe qQ0D99wCffV65b6RZVBG7qHoDOT73TSNeqGMXJXReTeOFV1TkX9BovuSKCOb67/ewlX5r2ex wVs/Z5cRg7PdQ3E02tMGc3YZnf3+XURJRlUVRTB4c1K3B+7ev7urOn133pHRsBKEXtztRme/ //PW4WOghHWU5Xn3zFxFSUZNAz8Gz9vDh4Eyoledeyg6A4WhcAZCDlXdl0QZuSqjsuzWqToT iq6p0lJX176v+5IoI1dl1DRnjTlS3DeSS0dmBtoD/V70ZFuUZBTHSYLxqK6ZGWgP9I/4ZFsU TQyyNAtqteq+JMpIVmx3D1eno2tRNnhXVVnuM/OljODk6WZKX66NhvRkBCsdHpHLW/clUUYY /4PgrF3SRZSjX4XXcRTpN2lSRqCqzht24B6KBu+22h4N3nvh4grJnVqBW983Dd7WEoauTezO 6lW32OC9zhZTFLD5SaeipkkS2P+SpNv4lJGEEztXUDR4Sx/vdQbvshSpuuoahcdQnwKfB3N6 d8VFGbXEsWvj0TlRNnjX9XqDdxz3BeJ54uRl2Z0qUkYtReGal9o5d452DZTAZE5O6pqmKx4p KEAZdXHNMeicueqU89QhUALTsDUjEqaFmMzVNTzFKSMV0tTNjdhzoRwogTrk2H5d4lPXL+mC 53LIx6dNy+hy+fZB9Mm5VwdVxVqwNoLECgL01ctFOVBifQnlMGxlhP0nro1UCAKXjMhIxWb6 GvZnQYKtbj3y+0AN2XZSR0udGlnmUnu4JPntUNw3wheJTdi1gRL9XODcN1IB/nXnXLi7g/K+ Ebo/1jZ0Td0fJjmxnV33jdSgjL7iTkLic+ZjWLRvVFV7TC4oo69g/Sg2Cc659rAdJRnBrlaW MFJ7nn4PXsponKZBdm9Mr6MIq1UUtQYUlnkUc3gjXA97RyxaaQeIRYYFD0QRLJ0IUSgVMX31 x0PZ4C1N3QyUsBGII1UE2wzYwcOIVhTbj2VnNIcojkb47YOpm/FGRwEmI7T0Hgk8j49itT2s iuDbxQRbxyNN45jrq3UoWuqqShi785z7RscDSaW5wbuGGRnF8Zh/cV3rjLmnjPYHpqOtdgXP WC1wRkYIG/d92IGEjSfPMZsOAp3O+5SRCRCwfm5f+jUoTOrqOsuiSNQih2lVd/wLZWSG89bK Ww/LhJFPMNMwfQ2uQhmRT7Zr9/Ol8qeMyCd1fcaN022gjMjvMCvevVBG5He28pdERLPpe9kX JRnBOrdfw1BGptgqKPN8MUeKMkLsaxxnGeONjsw505FswYLo1zyHnPTPnykjUzAn3r0oywid O459fw8ZPT8zKsYEW+4c1fVZ8uuhr6LHjr/bkRES4cMlaA/HVFzW46OLtX3cZ9t5wFlci9BX Hx+V4o18f6+VESd15kAGKNPX4CaKkzqxMoJTKotWHpmtp+xnqb23YN9IxEqyaOWR2VpGIpFn 04gJnnw8Hoqp8LE6Qq7TPSw5lJE59KTzxFym+1hV4htWfaxr2y2ISjIKwyzb73eEMjKHncES UkaIdzN9LePQGYh0sHvnqK5lKgPRQ+yR1a5lwtSgjMzhVgA4shvhMc9Faua90mN/RWuZsPug jMzhdtvLTXskvsTjPsmygZKMtioTptoYLn+VbnOMnSMpJ5l4Rz7Gcf+5nMKW5Vq57VomTLUR KCNzHEFGqsjRSo5e9+cZ2blMmAqUkUnOJKPtYJkw0uN8eRSG3FNJimXCSA+ORvf4jiqORu3/ 6FN3bCije1D0YhBCQrkwhu0dmz2m7faztA0UK0ogMW0ce94eHZwyMkkc2+gOtDdRtGxip7g2 QmkWVDna4xYoI5PY6VVnOwuCyLerOXAbysgklNE9zMjIG0H3JVFGJtFZcsct0lR9Yjcjo60L 8BYF9p9kYRfY/SBMEdwloYxMwoT4kqpSNzQoTOqky4SsuHc/de15OF1Zeh4+E2HpyLCJci/t UZSRSdj69zAjI/gvCPucGDek6fs+IB8pIzyKf8Xz7lH8Is3B1r+HGRlhvMC4UVUooYxJ2Lrt 1zAU6yvsPnXFIwUF+EWapCzPkl9uHvW+PiMjESIhIo7w2DRrfK6aJk0xMUSMPZyLpmR0uXz7 IPrkqGkw7OQYoRLbcMsOUFWyf6KvXi4zljrxJAzlCLHEUje08CH4T7xTFGHI0chOKKPlKI1G iDISh60bjTAxFP8TsbRcG9kIZbScGRlhMocyG1I8yON9/8mw0hKTOmGbo6XORvTvDLqCek+c tdTBQuf7YksOclqzUsHaCKlRfD9NRRpA7hvZB0cjSdOobsAuSrC1ZEPqfigjs1BGy2GeOjKA oRLLoYzIAIZKtKgGTFBGZAB9vFvqWm1kpozIAMpoOczFQAbYncd7f1QSbjMXAxnAUIk+KvZp 5mIgA9j+Y9yWEnMxkAFs/zFuT+2Yi4EMgPOX6WtwDeZiIAPqmn4MS9k5F4MKlJFpKKNxsmyq 9ytO6mR5wCxjKvzjQxmNU9fwaRD/9lGstuf7Is9+FHHf6PiwqsQtiuJrCjLFanvSuLAubE8N ysg0+vcGj4ZitT05lWO1vTPASd1SFKvtJYmYFSaJ/iamjExDGd3ma2lm5Wp7MjEWTQzHR5Ya JuPIWrEtrLZHvtA0e221HwXF0ajNoaD/kigj8yDpDGNg1VE0eIchBjK4qOrv4mV5vSKNnumm OTdZRpegabqTXvTV61XJ4C2XVHVNg/dZ4AppmmGZZRq8yQTIH8gVkhqLDd76E6VTRraAFZLp a3ADRRNDWwmCBu8zwdp7aigbvKuKBu/zgZhn09fgAgyUIDcJAhq+52HYHrkJp3UqME8duUld c1o3z4I8dU3DtdEZEWVLyS1mZZRlQSC6NYp57ZEmnTKyiywTIZtkmhkZZRniXlsfhn2cgSgj m8B2h+lrsJ0ZGQVBPxFtVQWB7kuijGyDZoY5Zi11X16mpe50YDyi2fsWszLqNx996s5JmvI7 ucWMjOK433zrSiirQRnZBwL5UGmCNrtxZmRUVb4vsxdX1doSympQRjZS10WRplEUBFE0DBMg swbvqpJ5GGDu3mPniDKym7rOsjCMIhgeypJrJqC4/Qpfur0ajDJyAfSJNEWmKPzMIg1onp83 OonOQGQT8K3FMaZ8AGtqMVqV5RkmgDvIqGvdy/MgQKUksQ+BEpiYLCZJd6SjjNwH8kHVPhB9 AoklCV4piqONW5plVNdIgyJlhKKXOF1Z+j62dZME+xHYleimz6CMjgvkBVOFGLfSFDUajjBa aZaRmD9LGaHh5GmR+cfzxMnhrdf/G9PNQvQjVlfS+gfcLd68w6SuFYmUjXitK572HcrovGCc CkPMTVxzPrJSRk9Pbx/IaFuaVM9GVaVpENjtVy6s1wB99elpQxmlaTdKVo4py2X08HD9IP3k CHNnshz9TtBrqGvZP9FXHx52HI24NiLqJIn+vB9bseukjpY6og5WSqavQZVdZcR9I6IOygGZ vgZV6MVArMWdKCfKiFiLO/tIlBGxFndWR5QRsRZ3kqlQRsRiXFkdUUbEYooCwev2b8BTRsRq mgabJLYHVlBGxHqQHdFuIVFGxAEgJBFTWxQ2rpYoI+IIKFQnshOJUAqbVkyUEXEQ9BER7ocI WvOCooyIw9R1WWYZQtLNZiaijMghkHKSGR72XUFRRuRQICo1y7pTvj2SplBG5LC0khKEoe9H PbZaV1FG5LQ0TZtGed0nUUbk5MBMEUXrAtYpI0K+I8Prmr+njAj5GJHWBaxTRoR8X5vOy0oZ PT6ySCLZl/vTeaGvPj5aKKPnZ2SjNHV+ckbuXx2hr6LHjr/LSR05EetWR1ZO6igjsj9rVkeU ESG/sSbZMWVEyG+s2TuijAj5DaTzujfUgjIi5JOqujehF2VEyO/k+X32OsqIkA6IULrnrygj Qn6naYJg+cSOMiKkR553i9apQRkRMmB59BFlRMgAhPEt+wvKiJAvxPGy8YgyIuQLSx1VKSNC RljmYUcZETJCXYehevItyoiQUYoCybfUxiTKiJBJVIs47yCjpvF+/6w0DQLP8/0kwU5x0ySJ 94F4JqGMiC1UVRyrHKdZRnWd52EoZZTnvo/TYd6Jy0sSeNTCQb27b0wZEXtQs9hplhEkgRFH PpNJXssSr3meOLl41v0bU41GSB+10PIdJnV9kcjTBkH3dSkocTxlRGzBitEIDGVU10ni+yhB SBkR24ljlYjYTWWUpl4HKYauXJoGxyQJLPLTMrpcvn0gi2fYXYOaHJspgVSV7J/oq5fLjqMR jAlx3G5qcW1EbAf1keaP2nVSl6Zh2H2HljpiO2q9cVcZRVF30sd9I2I/mLzNH0UvBkJuQhkR shrfnz+GMiLkJhyNCFmNSuQRZUTITaYlonIMZUTId7UU+ZQRITdR6Y+UESE3UUlvQhkRMgNl RMhq5mOOKCNCZuBoRMhq5oN1KCNCZpjfOaKMCJkhTYti7gjKiJCbzG/AUkaEzDDfIykjQmYo y7mkj5QRIbPMmbwpI0Jm6ecQ+QplRMgsc34MVsro+bkslxaxJUQftyZ16KvosePvGpTR42Oa ckQi9nDLjwF99fHRQhlRQsQu5vwYrJzUUUbELub8GCgjQmaZ82OgjAiZZa5PUkaEzDLnx0AZ EaLAbT8GyogQBW77MVBGhChw24+BMiJEAU7qCFnN7XwMlBEhCtz2Y6CMCFHgth8DZUSIAlUV hnk+9S5lRIgSKPWdpt0qxS2UESHKpGkQQDJDMVFGhCygabIsjqOoP8GzUEbv7y8vus8xl0x2 PXU9Pvy7dRf6z9A0de3+Xby8vL+Pv2NMRm9v16vuc6iUxV2HSqlD++9C/xn2mHvov4vr9e1t /B3KaBWUkRqUkSJN43n9574vuljTJIn3QZJ0J0CUkT13QRmpoVlGdZ3nYdiXUZp6nuhiSRKG TQNDYpK071NG9twFZaSGZhmhkTDitK9kWVlKGclHvNIekSSXy7KzLO+y374tvZOlX8V0w251 huV3sbzL6m+n5T+a+r/t5e10uXQHgi6bTeq6IqkqRBEK+XRfl4IClJGuu6CM1LBcRpi+wbx5 W0avrz/8cPng+snLSzrD4+Pz89wxfS6XZcen6fW67PiHh6cnvWdYfhfPzyIDoL4zLL+Lp6eH h2V/of/bVmmnlxfZP9FXf/jh9XUzGWHV0yIV3coFp29FMyWjpnl9TT54++T9vSTEMt7fZf9E X319ndol1DAaRVFXZlNrI0KOg5a10edH37TUEXIcdpDR+L4RIceB0yxCVkMZEbIayoiQ1VBG hKzGiIx0GR267rF5HgSeFwQy8CpNfX/dGXHV+IwwFJEt4i58X+6cDc94D7evert2KwrZUjra SW52CF+H4VUP2+0+xHVGkfjM7dtpuDd6+9s2IiMdJvC+e2xRCP/ysvR95HrBTSPELo7vP2OS RBG8MyAmfD1JEscIRxNNOTzjfffgefhbWD1xpuFVb9VudY29PV3tVJb9FL7Dq+63232kKTxl 0LXhdqarnUBVqXzbRmSkY0O27x4bRfJXI03xmyifrzljnsv4SnH98i76Z5DP76Fp2nbB/4ZX vVW7RZEcjXS0E8Kvu8+HV91vt/vw/W6sq652wjcSBH0H6/Fv24CMptyDtvzk9nPFa+3z9WcU Y0V7ruEZ1n15+GtMFvAL2L/qrdotSYpi2OG2bCfpLCYykA6vethu97ZRdxqnp53EvQx/Vsa/ bcpoIXnu+1k237BrqOs0RRfU0T3yHGOFThnlOaY9mHAhsbwuGUFAdS2mbbpkVNdiQkcZbSqj qgoCOcjrlBHm3uiC23cPzO3RMXTKSAJTw9er3kpGsp2+XvV2/StJvjpeWyMjfc6qXffY7ef8 +FuMQ8O72G5tlKZy0S26x/ZzfjFlbC1QOtopTUUWIMioO6ZutzZqLbJY6utaG8Hg0+Yzsm5t pM9Ztfsrtb0FKgy7ItJhqcN8H5+ByYpeC5RsKR3tFEWYcGFSJyx2Oix14hxoJ3RnPe2UZV2L o4WWOl37Rt1fn+33Q7q/42hCfG1b7xshXyc+U16nrn2jtqX07K/hKtHtxq562G73nUN+hr52 iuPuFd7+tunFQMhqKCNCVkMZEbIayoiQ1VBGhKyGMiJkNZQRIauhjAzRT0OGnah7HVdu5RdV 28f/WlfuK3Gsv0KRu1BGRul2c5XO/BXs1d96V0WaKgIuS/2p5t2FMjLKeq/CLFtfq0FtHITD zV7t4hqUkVG+eiNXFSqOCtcW4X4i3E2mXGjCUHjwyb+Ct1mbFFp8fvuZ3bTQ8l0xvcQ7whUJ 5YPxLorbd516Wn9nMoQyMspXGYmQNBH8Jh/xrvApls6YLdILWXR3/DUe4SHeZlCXn4lXp32u cSZIVp5DSBb+aULIWcZp3RSUkVHGZdR9RzzCs1gcJUIDvv79sJrUUEZiROmHv/dlFATSH1pE O8FBtjuNYw72adgwRlGVUT9OaOzv52TUPWpcRn3LIaSL+FXfj2OR94AymoYNYxRVGVWVHE/6 trd1MpKJTeRoJOOk6lpkUcAZmwZBCMNrJX3YMEZRlVE3GG6Yd0eMFUtkhOmaWANJGUFASFol 5CPMGtKcAcMDHrk2moYyMoq6jISl7msomrTULZGRDDkrCiEM2OoQ2TtlqRNCpaVuGsrIcbbY N1KD+0bTUEaOc9uLYTuKgkXepqGMnCdN702hsgT61N2CMiJkNZQRIav5fwHBRRvFdXjTAAAA PHRFWHRDb21tZW50ACBJbWFnZSBnZW5lcmF0ZWQgYnkgRVNQIEdob3N0c2NyaXB0IChkZXZp Y2U9cG5tcmF3KQqV01S1AAAAInRFWHRwczpIaVJlc0JvdW5kaW5nQm94ADQxOXg0MjArNzIr MTg0PNQ4sQAAABx0RVh0cHM6TGV2ZWwAQWRvYmUtMy4wIEVQU0YtMy4wCptwu+MAAAAASUVO RK5CYII= ---------3c0352933c035293 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ---------3c0352933c035293--