From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Tue, 29 Jan 2008 15:34:58 -0700 Message-ID: <20080129153458531.00000002384@djm-pc> References: <479DF2EF.8090601@virtualiron.com> Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: multipart/mixed; boundary=-------6184412361844123 Return-path: In-Reply-To: <479DF2EF.8090601@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "akira.ijuin@oracle.com" , "xen-devel@lists.xensource.com" , "deepak.patel@oracle.com" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format ---------6184412361844123 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Hi Dave (Keir, see suggestion below) -- Thanks! Turning off vhpet certainly helps a lot (though see below). I wonder if timekeeping with vhpet is so bad that it should be turned off by default (in 3.1, 3.2, and unstable) until it is fixed? (I have a patch that defaults it off, can post it if there is agreement on the above point.) The whole point of an HPET is to provide more precise timekeeping and if vhpet is worse than vpit, it can only confuse users. Comments? In your testing, are you just measuring % skew over a long period of time? We are graphing the skew continuously and seeing periodic behavior that is unsettling, even with pit. See attached. Though your algorithm recovers, the "cliffs" could still cause real user problems. I wonder if there is anything that can be done to make the "recovery" more responsive? We are looking into what part(s) of LTP is causing the cliffs. Thanks, Dan > -----Original Message----- > From: Dave Winchell [mailto:dwinchell@virtualiron.com] > Sent: Monday, January 28, 2008 8:21 AM > To: dan.magenheimer@oracle.com > Cc: Keir Fraser; xen-devel@lists.xensource.com; = > deepak.patel@oracle.com; > akira.ijuin@oracle.com; Dave Winchell > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that = > disables pending > missed ticks > = > = > Dan, > = > I guess I'm a bit out of date calling for clock=3D usage. > Looking at linux 2.6.20.4 sources, I think you should specify > "clocksource=3Dpit nohpet" on the linux guest bootline. > = > You can leave the xen and dom0 bootlines as they are. > The xen and guest clocksources do not need to be the same. > In my tests, xen is using the hpet for its timekeeping and > that appears to be the default. > = > When you boot the guests you should see > time.c: Using PIT/TSC based timekeeping. > on the rh4u5-64 guest, and something similar on the others. > = > > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer > > 14.318MHz HPET.) > = > This appears to be the xen state, which is fine. > I was wrongly assuming that this was the guest state. > You might want to look in your guest logs and see what they = > were picking > for a clock source. > = > Regards, > Dave > = > = > = > = > Dan Magenheimer wrote: > = > > Thanks, I hadn't realized that! No wonder we didn't see the same > > improvement you saw! > > > >> Try specifying clock=3Dpit on the linux boot line... > > > > I'm confused... do you mean "clocksource=3Dpit" on the Xen = > command line or > > "nohpet" / "clock=3Dpit" / "clocksource=3Dpit" on the guest (or = > dom0?) command > > line? Or both places? Since the tests take awhile, it = > would be nice > > to get this right the first time. Do the Xen and guest = > clocksources need > > to be the same? > > > > Thanks, > > Dan > > > > -----Original Message----- > > *From:* Dave Winchell [mailto:dwinchell@virtualiron.com] > > *Sent:* Sunday, January 27, 2008 2:22 PM > > *To:* dan.magenheimer@oracle.com; Keir Fraser > > *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; > > akira.ijuin@oracle.com; Dave Winchell > > *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that disables > > pending missed ticks > > > > Hi Dan, > > > > Hpet timer does have a fairly large error, as I was trying this > > one recently. > > I don't remember what I got for error, but 1% sounds = > about right. > > The problem is that hpet is not built on top of vpt.c, = > the module > > Keir and I did > > all the recent work in, for its periodic timer needs. Try > > specifying clock=3Dpit > > on the linux boot line. If it still picks the hpet, which it > > might, let me know > > and I'll tell you how to get around this. > > > > Regards, > > Dave > > > > > > > > > > = > -------------------------------------------------------------- > ---------- > > *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > *Sent:* Fri 1/25/2008 6:50 PM > > *To:* Dave Winchell; Keir Fraser > > *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; > > akira.ijuin@oracle.com > > *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode = > that disables > > pending missed ticks > > > > Sorry for the very late followup on this but we finally = > were able > > to get our testing set up again on stable 3.1 bits and have > > seen some very bad results on 3.1.3-rc1, on the order of 1%. > > > > Test enviroment was a 4-socket dual core machine with 24GB of > > memory running six two-vcpu 2GB domains, four hvm plus two pv. > > All six guests were running LTP simultaneously. The four hvm > > guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64. > > Timer_mode was set to 2 for 64-bit guests and 0 for = > 32-bit guests. > > All four hvm guests experienced skew around -1%, even the 32-bit > > guest. Less intensive testing didn't exhibit much skew at all. > > > > A representative graph is attached. > > > > Dave, I wonder if some portion of your patches didn't end up in > > the xen trees? > > > > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer > > 14.318MHz HPET.) > > > > Thanks, > > Dan > > > > P.S. Many thanks to Deepak and Akira for running tests. > > > > > -----Original Message----- > > > From: xen-devel-bounces@lists.xensource.com > > > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > > > Dave Winchell > > > Sent: Wednesday, January 09, 2008 9:53 AM > > > To: Keir Fraser > > > Cc: dan.magenheimer@oracle.com; = > xen-devel@lists.xensource.com; Dave > > > Winchell > > > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > > > disables pending > > > missed ticks > > > > > > > > > Hi Keir, > > > > > > The latest change, c/s 16690, looks fine. > > > I agree that the code in c/s 16690 is equivalent to > > > the code I submitted. Also, your version is more > > > concise. > > > > > > The error tests confirm the equivalence. With = > overnight cpu loads, > > > the checked in version was accurate to +.048% for sles > > > and +.038% for red hat. My version was +.046% and +.032% in a > > > 2 hour test. > > > I don't think the difference is significant. > > > > > > i/o loads produced errors of +.01%. > > > > > > Thanks for all your efforts on this issue. > > > > > > Regards, > > > Dave > > > > > > > > > > > > Keir Fraser wrote: > > > > > > >Applied as c/s 16690, although the checked-in patch is > > > smaller. I think the > > > >only important fix is to pt_intr_post() and the only bit of > > > the patch I > > > >totally omitted was the change to pt_process_missed_ticks(). > > > I don't think > > > >that change can be important, but let's see what = > happens to the > > error > > > >percentage... > > > > > > > > -- Keir > > > > > > > >On 4/1/08 23:24, "Dave Winchell" = > wrote: > > > > > > > > > > > > > > > >>Hi Dan and Keir, > > > >> > > > >>Attached is a patch that fixes some issues with the = > SYNC policy > > > >>(no_missed_ticks_pending). > > > >>I have not tried to make the change the minimal one, but, > > > rather, just > > > >>ported into > > > >>the new code what I know to work well. The error for > > > >>no_missed_ticks_pending goes from > > > >>over 3% to .03% with this change according to my testing. > > > >> > > > >>Regards, > > > >>Dave > > > >> > > > >>Dan Magenheimer wrote: > > > >> > > > >> > > > >> > > > >>>Hi Dave -- > > > >>> > > > >>>Did you get your correction ported? If so, it would be > > > nice to see this get > > > >>>into 3.1.3. > > > >>> > > > >>>Note that I just did some very limited testing with > > > timer_mode=3D2(=3DSYNC=3Dno > > > >>>missed ticks pending) > > > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the > > > worst error I've > > > >>>seen so far > > > >>>is 0.012%. But I haven't tried any exotic loads, just LTP. > > > >>> > > > >>>Thanks, > > > >>>Dan > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>>>-----Original Message----- > > > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] > > > >>>>Sent: Wednesday, December 19, 2007 12:33 PM > > > >>>>To: dan.magenheimer@oracle.com > > > >>>>Cc: Keir Fraser; Shan, Haitao; > > > xen-devel@lists.xensource.com; Dong, > > > >>>>Eddie; Jiang, Yunhong; Dave Winchell > > > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > > > >>>>disables pending > > > >>>>missed ticks > > > >>>> > > > >>>> > > > >>>>Dan, > > > >>>> > > > >>>>I did some testing with the constant tsc offset = > SYNC method > > > >>>>(now called > > > >>>>no_missed_ticks_pending) > > > >>>>and found the error to be very high, much larger = > than 1 %, as > > > >>>>I recall. > > > >>>>I have not had a chance to submit a correction. I = > will try to > > > >>>>do it later > > > >>>>this week or the first week in January. My version of > > constant tsc > > > >>>>offset SYNC method > > > >>>>produces .02 % error, so I just need to port that into the > > > >>>>current code. > > > >>>> > > > >>>>The error you got for both of those kernels is = > what I would > > expect > > > >>>>for the default mode, delay_for_missed_ticks. > > > >>>> > > > >>>>I'll let Keir answer on how to set the time mode. > > > >>>> > > > >>>>Regards, > > > >>>>Dave > > > >>>> > > > >>>>Dan Magenheimer wrote: > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>>>Anyone make measurements on the final patch? > > > >>>>> > > > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>about 0.2% with no load. This was xen-unstable tip today > > > >>>>with no options specified. 32-bit was about 0.01%. > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>>>I think I missed something... how do I run the various > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>accounting choices and which ones are known to be = > appropriate > > > >>>>for which kernels? > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>>>Thanks, > > > >>>>>Dan > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>>>-----Original Message----- > > > >>>>>>From: xen-devel-bounces@lists.xensource.com > > > = > >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>Keir Fraser > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM > > > >>>>>>To: Dave Winchell > > > >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, > > > Eddie; Jiang, > > > >>>>>>Yunhong > > > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > > > >>>>>>disables pending > > > >>>>>>missed ticks > > > >>>>>> > > > >>>>>> > > > >>>>>>Please take a look at xen-unstable changeset 16545. > > > >>>>>> > > > >>>>>>-- Keir > > > >>>>>> > > > >>>>>>On 26/11/07 20:57, "Dave Winchell" > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>> wrote: > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>Keir, > > > >>>>>>> > > > >>>>>>>The accuracy data I've collected for i/o loads for the > > > >>>>>>>various time protocols follows. In addition, the data > > > >>>>>>>for cpu loads is shown. > > > >>>>>>> > > > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 = > processor AMD > > box. > > > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. > > > >>>>>>>The cpu load is usex -e36 on each guest. > > > >>>>>>>(usex is available at > > http://people.redhat.com/anderson/usex.) > > > >>>>>>>i/o load is 8 instances of dd if=3D/dev/hda6 = > of=3D/dev/null. > > > >>>>>>> > > > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. > > > >>>>>>>Also, these are run on 4 cpu AMD box. > > > >>>>>>>In addition, there is an idle rh-32bit guest. > > > >>>>>>>All three guests are 8vcpu. > > > >>>>>>> > > > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 > > > >>>>>>>except that the redhat-64 guest has 4 instances of dd. > > > >>>>>>> > > > >>>>>>>Date Duration Protocol sles, rhat error load > > > >>>>>>> > > > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, > > > +.005% cpu > > > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, > > > +.012% cpu > > > >>>>>>> > > > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, > > -.004% cpu > > > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, = > -.005%, -.005% cpu > > > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, = > -.008%, -.003% cpu > > > >>>>>>> > > > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, = > -.040% cpu > > > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, > > > -.031% cpu > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, > > > -.09% i/o-8 > > > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% > > > -.14% i/o-8 > > > >>>>>>> > > > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, > > > -.022% i/o-8 > > > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, = > -.017%, -.018% i/o-8 > > > >>>>>>> > > > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, > > > -.029% i/o-8 > > > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, > > > -.031% i/o-8 > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, = > -.04% i/o-32 > > > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, > > > -.005% i/o-32 > > > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, = > -.11% i/o-32 > > > >>>>>>> > > > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, > > > .003% i/o-4/32 > > > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, > > > .01% i/o-4/32 > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>Overhead measurements: > > > >>>>>>> > > > >>>>>>>Progress in terms of number of passes through a fixed > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>system workload > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. > > > >>>>>>>The workload was usex -b48. > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>ASYNC 167 min 145 passes .868 passes/min > > > >>>>>>>SYNC 167 min 144 passes .862 passes/min > > > >>>>>>>SYNC 1065 min 919 passes .863 passes/min > > > >>>>>>>MIXED 221 min 196 passes .887 passes/min > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>Conclusions: > > > >>>>>>> > > > >>>>>>>The only protocol which meets the .05% accuracy > > > requirement for ntp > > > >>>>>>>tracking under the loads > > > >>>>>>>above is the SYNC protocol. The worst case = > accuracies for > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>SYNC, MIXED, > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>and ASYNC > > > >>>>>>>are .022%, .12%, and .14%, respectively. > > > >>>>>>> > > > >>>>>>>We could reduce the cost of the SYNC method by only > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>scheduling the extra > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>wakeups if a certain number > > > >>>>>>>of ticks are missed. > > > >>>>>>> > > > >>>>>>>Regards, > > > >>>>>>>Dave > > > >>>>>>> > > > >>>>>>>Keir Fraser wrote: > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>> wrote: > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>>>Since I had a high error (~.03%) for the = > ASYNC method a > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>couple of days ago, > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>I ran another ASYNC test. I think there may have > > > been something > > > >>>>>>>>>wrong with the code I used a couple of days ago for > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>ASYNC. It may have been > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>missing the immediate delivery of interrupt = > after context > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>switch in. > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>My results indicate that either SYNC or ASYNC give > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>acceptable accuracy, > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>each running consistently around or under = > .01%. MIXED has > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>a fairly high > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>error of > > > >>>>>>>>>greater than .03%. Probably too close to .05% ntp > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>threshold for comfort. > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>I don't have an overnight run with SYNC. I = > plan to leave > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>SYNC running > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>over the weekend. If you'd rather I can leave MIXED > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>running instead. > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>It may be too early to pick the protocol and = > I can run > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>more overnight tests > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>next week. > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>I'm a bit worried about any unwanted side = > effects of the > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>SYNC+run_timer > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>approach -- e.g., whether timer wakeups will = > cause higher > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>system-wide CPU > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>contention. I find it easier to think through the > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>implications of ASYNC. I'm > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>surprised that MIXED loses time, and is less = > accurate than > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>ASYNC. Perhaps it > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>delivers more timer interrupts than the other = > approaches, > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>and each interrupt > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>event causes a small accumulated error? > > > >>>>>>>> > > > >>>>>>>>Overall I would consider MIXED and ASYNC as = > favourites and > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>if the latter is > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>actually more accurate then I can simply revert the > > > changeset that > > > >>>>>>>>implemented MIXED. > > > >>>>>>>> > > > >>>>>>>>Perhaps rather than running more of the same = > workloads you > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>could try idle > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated = > large disc reads > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>to /dev/null)? We > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>don't have any data on workloads that aren't = > CPU bound, so > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>that's really an > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>obvious place to put any further effort imo. > > > >>>>>>>> > > > >>>>>>>>-- Keir > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>_______________________________________________ > > > >>>>>>Xen-devel mailing list > > > >>>>>>Xen-devel@lists.xensource.com > > > >>>>>>http://lists.xensource.com/xen-devel > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c > > > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000 > > > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500 > > > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru > > > >> > > > >> missed_ticks =3D missed_ticks / (s_time_t) = > pt->period + 1; > > > >> if ( mode_is(pt->vcpu->domain, = > no_missed_ticks_pending) ) > > > >>- pt->do_not_freeze =3D !pt->pending_intr_nr; > > > >>+ pt->do_not_freeze =3D 1; > > > >> else > > > >> pt->pending_intr_nr +=3D missed_ticks; > > > >> pt->scheduled +=3D missed_ticks * pt->period; > > > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) > > > >> > > > >> pt_lock(pt); > > > >> > > > >>- pt->pending_intr_nr++; > > > >>+ if ( mode_is(pt->vcpu->domain, = > no_missed_ticks_pending) ) { > > > >>+ pt->pending_intr_nr =3D 1; > > > >>+ pt->do_not_freeze =3D 0; > > > >>+ } > > > >>+ else > > > >>+ pt->pending_intr_nr++; > > > >> > > > >> if ( !pt->one_shot ) > > > >> { > > > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct > > > >> return; > > > >> } > > > >> > > > >>- pt->do_not_freeze =3D 0; > > > >>- > > > >> if ( pt->one_shot ) > > > >> { > > > >> pt->enabled =3D 0; > > > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu = > *v, struct > > > >> pt->last_plt_gtime =3D hvm_get_guest_time(v); > > > >> pt->pending_intr_nr =3D 0; /* 'collapse' all > > > missed ticks */ > > > >> } > > > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) { > > > >>+ pt->pending_intr_nr--; > > > >>+ pt->last_plt_gtime =3D hvm_get_guest_time(v); > > > >>+ } > > > >> else > > > >> { > > > >> pt->last_plt_gtime +=3D pt->period_cycles; > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xensource.com > > > http://lists.xensource.com/xen-devel > > > > > > = > ---------6184412361844123 Content-Type: application/octet-stream; name="el5u1-64-hvm-const-ltp.png" Content-Disposition: attachment; filename="el5u1-64-hvm-const-ltp.png" Content-Transfer-Encoding: base64 iVBORw0KGgoAAAANSUhEUgAAAZoAAAGkEAAAAAB9jWAhAAAACXBIWXMAAABIAAAASABGyWs+ AAAACXZwQWcAAAGaAAABpADE8ZBCAAAjOElEQVR42u2dvYrlRvqHlQ5/lkUYs0GDzSoweJlg 1wq2YcAOVkGzyyQLAm/WwSI63ElWDY42GBAdG4NiZ0r6AhTYFyDwFegWdAvz79c15SrpqKSq bn28Uv2eoE+fL52q0vtIpfpS8AEA4ESwdwIAOBqQBgBHIM2JyfNgcv8WRZ7vk7KmSZL1th6G T4EdrJc3VtJ0XZ5HURCEYZZ13eW7WRaGQRDHTTP+/SzTC6ooaFtBkKZta/5NCiyJ3JHqm2O/ JN6NorGd0rZpav7mZRrHPj+1ffGdwHKvSWmaJtCoa/m+qSTHSq6uk4ReG9sz8/uo6/r5yfOi sMuDbU5VrqoqTZU0+jtLwUoa2i2UybIMwzgevhvHYViWdU2PYxpQaKgdQwFD26oq+rx5R+d5 HNcfETu6KCg05DeHvyS3Kx7773UdpbuqzL/ZT+PY56e2L8gyV2nqOgiqSuZSposC2fStYcnR FtLUtGfm9xEdGvTcxPHUgcyd/p4vS/XKGmccRtJQQMnjj9jN+rtlGQQiqLvusiAo9Ok4qF6n s5X4j47M5uNaktBxSSeK5Ctj31TbpWNq/z3STQTD2Dcv0zj2+antU2rFOcCuRKU049W0shwP p7GSiyJ5Flb7YWyL4/tInLnU87adEu856FuPIhLdE2mqSq86DDObplMFTQHQPzar/8ckU8Tx 8L0koSPVeCpIZplG/X/5TVVT1/83pfHy89Pbp/DPczofj+WkbUXVSFWElDRjZSeqhGkqgkyE PP3eZcnR4ax/CBtnfB81TRj2z7BFIQ8osjqe5yINTSOqqyoPZZkkXUd5pk+J1+gcSJ+is7Qo J4HcgtpzqkpK25mvPNvBSBodKggVuoQosqZR1YuRzPRO0mFIISBq2ebKABVhf8dJ6JvDo2r/ mD1Ualg9nEvj5eentz+15bYNwyiiqhHlXJwr5Cfp/CTDRWpIVUP5SL9C35dXAcOSozNi94Ss wJoY20ddJw5Lel7kFY+sCBZFGFKok14kgqicqjzQYayuKQcUE7KqSJVAOlNTukSFWnxeCCl+ T75DV8tCRdIniqYiYh6W0lDhyePfbwkN6Dgmjhum80b/HXHh2r/0HUIVEFELp1DRzw3im5dV LK7SkOAyEGQ1SX5SVI7EVYosjbKUlTA6w1dVkqizxLDkRHCLV6LILM7YPpJnOfVK24prKb0K S1sXksq9Tq9QfuhRHD7leU/lv23zXORZbT2K+q+oBgH9DKqqoM+BnTRUyRjbNSqE6bg3ro3+ eprScY8CJUnEkXP812Sxi1BTlRA6klFK+uc7vtLQMbr+DVFWqnom80UX/yKIs0zllY7hVIUy lRxth7YurkbMzSqX+4iEFNtVeZHXUuL81c+DCmWplJ5bdfZI06rSvyu3rq6WxqRRZ5ex60V7 mElDJ+rLipKe+aks6ztGP79Q9YQKtB4w/JXLIB3+kr4LL6+VlpXGdC02vuVgQP+YfPltveWM rjj0C/1hydF3pGJTzSqX+4iqXVJjqiZR2MorisvU9fN7mQf5PuksznoyJfIddbU0Js1cGdrC ShrKiqlXRb9gnw/IsSLSe2TGq230fTr3qLPc8Jde0hAwTONzGgLMuVeXydOfFK9VlV49EW1y QouxkptuopnaR5cqq4busTON3hxvOtOofSGvctQ7qinbkzMNHcPMNU1qNZL/6/XvXmZ+K9T+ bjZ9vl+U8kqgH9bDbz6/yXmYxuc0Ocs0X76aZeoqsCioxUl+st+oIhrUs0y9Qldz1K4kql1j JScDmLjsDLDbRzLf6lpK72KgsldpkLnUr8vUNvJcRUlfDXm1dPnOaa9pRCVAVZ5EOCWJyCpl lE711N+rLlD7R/N+C42qmZt3M7UZxfFlmxN9UxzJ5PFXjRZQnY9yl6s00hWD6KyUITyVxrHP T21flZP+TGy/3/LUr5LoZSGuXVRwytAV5WsqOVkm1LlpHgAzto+G+aarEfkaXTPqWzW1nvW3 QYKp/SN+RTQj68OClDSixVDUM07XekZFOKyVU6blTpJt+Ekid8ewx6IfkKIPQG9mHeOyd0P2 Hujf1H9JDnNRZxKVRtUTIHfKVBrHPj+9fZE6fYtq+5d9HPKTaniS+B3Zl0HEsdy2aEUzldyw R2Wcy300zHd/FMJ8P81Y9UymT/2KUELvf9EPHKSWOHOJa6GT9tMAsCwvu/TvA2mAF0AasCtU zbnkZVWetYE0ADhCY8+W2hakAcARSAOAI5AGAEcgDQCOQBoAHIE0ADgCaQBwBNIA4AikAcAR SAOAIwtJI9cCkWu4zK3ECMBxWUCapqHBcFKaLItjWu4njl8yNw4AviwgDa0uoqSR/9G02L0z B8AaLBTYY6pMrTcGwHHZSJquu7/Pnnj4yONjDQAzHh9lfFKs3t8b13fbRpr7+1evrp64/sjd Xb4I19fLbKfP1dUaW725ub1dfqtv375+vfxWb29vbtYog3VKdqkouLuT8Umx+urV/f0m0piu abLs6mqZX+qzzq2BPvlkja2ucaeUDx8eHq6vl98qLUSxRhmsU7LrRMHVlakpa3FpxlvPIA2k ISCNvplgup9mnV27Fmve2m5p1grvdThSyV5fPzyMv7NRszCkWQtIsxaQxokj7VpIsxaQxokj 7VpIsxa7S/P4eHe3dyHYw3sFrz5dt+wtX9flSCV7d/f4OP7ORtIc63gIwFRrJ6QBYBRIA4Aj kAYARyANAI5AGgAcgTQAOAJpAHAE0gDgCKQBwBFIA4AjkAYARyANAI5AGgAcgTQAOAJpAHAE 0gDgCKQBwBFIA4AjkAYARxhIc3tLS0zvXRAA2ECxShE7/u5m0ogFtfcuDABsoFi9udldGggD jgWD6hmkAccC0gDgCKQBwBFIA4AjkAYARyANAI5AGgAcgTQAOAJpAHAE0gDgCKQBwBFIA4Aj kAYARyANAI5AGgAcgTQAOAJpAHAE0gDgCKQBwBFIA4AjkAYARyANAI5sKE3XZVnwRJZ1nXoV 0oCjsaE0WRbH3RNxnGXqVV2aritLXahx2jbPi+Ll6cGqnuB5bChNEIifqutA27aUpmlIqixL U/079G6aJon4ZtsWRRwnSVnGcVmKT5BoaRqG8rl6pWnUdpqmKJRo9FtRlKb9cx4Admwmja6K 1Ee8/p//lGXyRFWJBFEokyxJQhKVZdO0bZKQTnFcFG1Ln6LzVVXlefxEnpMeaUpnKdKFXmnb poki+mZZCkHoMU1pu7RV8VskH63NS7+ZJFFE35QatW1dFwWJTJ+hX6x/oyzzXEmqXm9b+tQx Rax7UNlKkt+IojCkvSJKz18YSPPq1R/+8Je/0Aq5d3cU7jc3b95QUL5/LwQpS3qsqv7zpvn6 6/fvKUDF8677/PPPPiOpxPMPH7777ve/T5I8f/dOfu/770kX+T49UnDc3Lx/T+kpy19+yfMw /L//e/OGQoTSQ6+/f//mDQkl0ifSSaJ9+im9TkEkXv/6a/G9L78Uj59//ssvKr30KHLt+vju 3bff0vn2yy+/+oq2++mnQUDBK55H0aefitflc/k5eozjIPj00y++CALxPfW5y8ebG/odyp94 fPfu4YEEevfu8VFPz/v333xD5/svv/z226qi9x8eyifevfvhB5v8yOXu5x9//JGU/eoJSvlX X/3jH7SNhydovz8+/vij/vnHx3//m9L97bfffEP5CYLf/Y7+fvGFyB9tx/RI3/vmm7HXxfbp kuDbb6lcrq//9Kc//nF3ab7++n//o93zww+Pj3ScbhpxrOb++NNPQgrT+9999+c/03/qdSGP 6+Pt7X//W9dNY/+9n3+m3fzc37N9/Omn77+nYLu9FQeN29u//51k+utfxUHjn/+8u6uqfrrp /Ny24vn8448/Zhl9Sz0vijy/u/vXv2j7b96Ig9abN3d3VIX/7DMqJ1JNyGT/O7/8Ir7388/D 36c4pedUg6DtkrL393/7G5NrmjNS10ny8sqauTLAG1HJo4q1fKVpgkAEpO0WbGJDVo23yxmr 1rPz0TQv1+ao0kjoelL8lz3hsre5xgb6aVamaehQ8ZItHF2arksSaqpp2yj68CGK7EuDa2xg RMAGOXyZNkeXhg4cUURVNTrj6O2Oc3CNDUizAdS0/Xxtji+N6CKg84w839jBNTYgzSZQL9Fz v3sGaUQ/l/gvSWy/wzU2IM1G0EXw8755DmkUkObFcC2Y5aHxCM/53vmk0Qc4TcE1NiDNZtDg m+d872zS2OeHa2xAms2o6+dV0CANNyDNhsTxc74FabgBaTbkedI891qIK3luO0qaa2xAmg1J U9tLYB05m+gs0OBOu09yjQ1IsyEuveGKs0ljv8e5xgak2ZDn5RXScAPSbAjNQHX/FqThBqTZ lOc0BSTJlnNF1qeu++tATH2SZ2xAmk15zuya55ydOGN/vuUaG5BmU57T53I2aexzxDU2IM2m 6AtJ2QJpuAFpNoVWXHP9DqThBqTZGPemgPNJE1jGFq09s3dax4A0G+PeFHA+aWxzZD92YFsg zca4NwVAGm5Amo1xn1UDabgBaTamaWy79iRnlMauuxbSGPBNGlr5y+3z55Mmy+yqqJDGQF1f X9PKvHsXxHa4NgWcr2xsr+s4SkOxen29uzT8CmZdXJsCfJaG5009GJxpfJPGtSnAZ2l4ju+G NJvj2hRwRmns9jmkMeCfNK5NAeeTxnafQxoDPkrj1hQAabgBaXbALRggDTcgzQ7QbSfsP31G aezGekMaAz5K07YuTQHnk8Z27iakMeCjNG5NAa4jCI4ApHkRfkrjsliG7eyTIwFpXoSf0rj0 dUMabkCaXXBpCjijNHZ5gjQG/JTGZdnAM0pjN+kb0hjwUxqXNrEw3Dute+Ue0hjwVRp7Fc7X 5AxpXoiv0tircE5pbFoPIY0Bf6WxvVfNGaWxm7sJaQz4Ko19QJxRGrvcH16a+lfcl++ew19p bG/wdEZp7JYBPKw0XVcUcRz8RhTl+ZK3fvBVGvuemjNKY7fXDypNnpMkVSXr321L2Y3jLFvq nOOrNPb5Pqc0NkNWDyqN6c22Xeps46s09t2bZ5TGLlcHlaZP0yx/Ty5fpbGX4ZzS2IzdPrQ0 ZUldcVVF1zSu9ydWjYtdl2W0hX7Vzl9pbLs3zymNzUCaQ0sTRaRKHFdVXbvM7miaPA8C+QNZ FsfdE3RFpD7jrzR+n2ls1kk4tDQ0ZLDrxMBBl+GDtMKXkkb+V9f6NnyWxq5785zS2HRvHlqa KKrroqD2jrZ1HXM7pooSyWdpbEPinNLY5D5NDyxNWdK1CE2biiLXNe8hjQnb7s1zSmPTvZkk B5aGrk5EVaIozDXRoId6dU6aq6tPnkg+Yjsi6/jYdm+eUxqbgyUfaZpGxifF6tXVymPPcE1j wjbnZ5Vmvs7CR5o+M2ea4ALXXYjWMxO23ZvnlMYmXweVRgzTzPMwLApqDAhD15sfKGnQTzPE b2nmOy8OKo3MnvyQWz+NDT5LY1eWZ5Vmvnvz0NIEhv+XwGdpbPoquu6s0sx3b7rfPn4brKQJ Q5xp1sCm/ayuzyrN/CGDa86tpKFrmrKsaxqDtnSI+ywNNYrMfea80swvmMg155b9NDSvRkxA WzoBPktjs8DEeaWZP89yzTnWCNiV+VEB55Vmvsmda86tpwZc9vUvg9/SNM3cnVrOK818+xnX nFs2OdNFm2TZBPgtzXyz85nLJ02nK6eHlmbN1YTPHBQ2zF3VnLl85poCDi1NFC0/zVly5qCw YW6I/JnLZy5vh5aGxnhSkzOqZ8sz1xRw5vKZ67g9tDRJMjbofxnOHBRL5P/c5TN9RXdoadbk 3EFhk//p9rNzl0+aTs2fOrQ0XUdjAoJg+fEAZw8KG6ZD49zlM105PbQ0tKamGEYTxxhGszTT VZRzl8907g4tTRjK1rO2Xfq+XOcOCht8PtOcWJogkEO05UJOy3HuoLBhuqfm3OVzYmnSNMto x7ZtlrmuRjPHuYPChumemnOXz/QgoUNLQ0PYRXMzzfJfNgHnDgobpi+Gz10+0z01h5aGaJq6 XmN5pXMHxctL4Ozlc2JpaBVnqp65Ln8+z9mDwqYEpnpqzl4+p5WGpqDR0Lq2RZPzGkwFx9nL 57TSqCbnrkOT8/JMlenZy2dqTs2hpUGT87r4fKaZiqdDS6M3OS+dkbq+vV1+7PSxmFrB+uzS HK16RrFKETv+bq/JWY5zXqPJ+eYmz88dGHNM9dRAGl5QrN7cWDY5032d0eS8DpDG/b09wdQA BkxJc/az8JQYSzc6LYX1umc0NWCNhUIhje/SmOsva65N8RKspwbQvZ1pKVGMPVueqQUmzi7N 1OK0h5ZG9NPot6tdDkgzLcbZpZk6yx5aGnVfZ0izBlMLtEIaflj209COoy5OVM/WYKoM5pcJ PzanlYb6aWgBdLp5IBoClmdamnN3/E4dFA4tDYF+mvXwWxpz3udvRLJXmh36aZpm+ZU2IY3f 0kzl/dCdm3TXALpcpQra0jNqIM30jBpIww/LtZxJFeqrwe0D12Bqpjyk4Yd1k7NsbEaT8xpA mjEOLQ3dEr0oqLG5bSHNGvgsjbkT49DSlCVdzVDTYBShn2YN/JVm6iaCh5aG2s1Ec3NRoJ9m DfyVZirvB5UmTce6nuhOkUv12UAaAtK4vbMvM9LQBOcwzLKiEDd0KgpamSZJltuRkIYwzxyB NPywqJ61bZ7L6c5JkufLjguANITP0pj7/Q8szbpAGsLn6pm5PRbSGIA0hM/S4EzjDKQhfJbm lNc06wJpCEjj9s6+rCoNtbTRnTqzjHp3aAobNSaIZxJIQ0Aat3f2xUqaLKsq905NGhst7jUQ xzSOIMtoqUG6140+phfSEOZGfB+kMU04Obg0dMZI06JwmU9DN7aV/1ELSRCInxLP1KcgzdSa LOeXJk1NOTy0NATdmyZNw/A5s+momqarIvUhIA1h3g3mkDoL5rwfXhoKburidJVGjCmg+6iZ pHn79uGJ+iNLj207BlOBA2n2p+tkfFKsvn1rIQ3JEkVzVzZBD/FTeU4X/lSpM0vz+vX1E/lH lp9QfQQgzXje906bhEbFCChWX7+2WiyQlClLt4CmS/40Vd/BNY0ZSDOe973T5priXvWMrmio OYDksd90vyqH1jMz5oWMfJDGFAEHl4ag8I4i+5Xc5SBPWV1DP42ZqcBZY+EsTpjXFz20NKIJ IIrWWO0R0hDHC5zlMEcA17xbSRPHRbHW8Q7STJcC18DxOe8Ye8aC4wXO+nlvGq55x02dWOC3 NOONS1Orwe0LburEAp+lMa1Hc3BpcFOntTEvTMs1cJbklNLgpk5rYw4QroGzJOPdGAeXBjd1 Wh+fpTnlmQY3dVofv6UZ69A4uDQEbuq0Lj5LMx6EfCPD8kyj/rMfe2YH36LZFr+lGbvrEd/I sBwRILSpqudNQpuCb9Fsi8/SjMcA38iwvGtAHDcNzdssiqUTwLdotsU0EBbS8MPymoZutmFe AOEl8C2abfFZGmpounyVb2RYNwTQ2WaNqch8i2ZbfK6ejeeSb2TMSBOMsGwC+BbNtvgtzdid XPlGxow09QjLJoBv0WyL39Kc6kxDNI2olsm7oS0L36LZFr+lSdPL2OIbGTPS0Ix+sSCGmKq8 /HUN36LZFr+lGQtDvpExIw2NNSNNmoZuVEtdm+jcXAfTAhpL94vxZKx7k29kzEgjJgWolWW6 zn5hDTv4Fs22mBamXbrhhSdjUcA3MmZbz8STOJYZQOvZOph2BKThh9WZhmbRiI+1Lc406+C3 NGPdm3wjY0YaqpbRrEKpCq5p1sJvacYaPPhGxmzrGbWahaG4TKOlApceSlPXNzfmpfL8AdIM X+EpDcXqzY3DEk5rrOlf17e3y3eZHg/TQox+NDmPHRx4SkOxShFryMVWieBYNNtTFOPl4Is0 lwNp+EYGFgtkgqkcfJHmKNUzAtIwAdIMX+EbGZCGCZBm+ArfyMAaAUwwLRcIafiBNQKYYFru 2x9phuOc+UYG1ghgg9/SXI694xsZWCOADX5LcxmIfCMDawSwAdL0X+EbGVgjgA1j8+R9lsbU 3bs/WCOADZBm+ArXyLCsnlUVfYxujL50AiCNxPfq2bCR6eDS0K3QaThh26rJaEsBaSR+S3NZ GTu4NGpCAKY7r4ff0lzGwcGlods5if9wJ7T1GL9LC6Thh+Wd0LKMzjVtm2VL70RIIxlfWgPS 8MP6TmiiuRnrnq3H+K7wWZrxaXn7gzuhscF3aYZ3czWH5t5YnmnyPAxprYDlAxzSSPyW5nLA 6sGloTVpyrKuaSgNmpzX4rKngvPNWpfnZNKoJmese7YeVTW2YB6k4QeanNkwvsokpOGHc5Pz 8HLtpUAaCaTpPz+4NOKGG+5NzmUZRUEQRWLEmlh4MAiyTN8GpJH4Ls1wwOrBpSGaxrXJuW3p 9hy044OAzlNZRsqRgPpseEgjGRPEp9IZXi0fVpqXTA3oOvFZkob+k4uo03P1KZ/CYppjLQK+ PKepnr10EhoJQhU0Ekh9U+ojPuFPWMwBaXQOK80StG2eU+XOJM3V1SdPJB9ZY9TBUYA0Opyk oa5XAcXq1ZX1umdtO72whvl8VFVRhDPNPJBGh5M0tikL5AfCUHwkz4NA3IHTdtNynmdVkTC4 ppnjKCvnr8NwtaPDSkPKlKUUpa5dhtFI3Wi+J/XuoPVsjsuFGH0qnTTth+JhpRHTnBV1Pb78 g2nj1E8ThqJnBv00c/hdPRuG4mGluawwYBjNekCaqed8mJWmfw2DsWdrAmmmnvNhRpo07e+0 PMfYs/VIkuOsMrk8p5GmaeiKRPSdNA3d5XnpfhSfwmKOIy3Nukbu+6vqHVYa0a0j+17W6Hr0 KSzm8F2aYa3msNIQNAhmjTs7Ez6FxRx+SzPM68GlWROfwmKOy/VXfCodSGONT2Exx+VKXz6V DqSxxqewmOOyLHwqHUhjjU9hMQek0Z9DGiM+hcUcl3d49ql0hjNXh2PR+ABpGHE5d9Ov0unn /rKrlwuQhhVDafjeQm/93EMaI5BG53JxCZ9KB9JYAml0Lmcv+lQ6kMYSSKOTJP1xF35J05+E B2mMQBqd4Y2d/JKmP+0E0hiBNDqXw+OXv582X4bVs73TYwLSsOI4w+PXANJYAml0hrfbgDQc gTSsOM5QkjXoL+IEaYxAGp3hnSf9kqbfDAJpjECaPv1Q8Uuafm4hjRFI06ffVwFpOMJAmutr Wlh674LgQn8xRkjDDYrV6+vdpcGZRsfv6pkeCzylEemENKzotyD5JU0/FiCNEUjTp79DIA1H IA0zII0E0hiBNH3yvCj0Z35Jo/dSQRojkKZPvzz8kqY/3RvSGIE0fXyWpi8KpDECafq0rR4s kIYjkIYd+pgA36TR8w5pjECaIfqwRd+k0RcWgTRGIM0QfU6Nb9KgemYFpBlC97+W/0MajkAa hsSxHEozXGjj7EAaKyDNJWqlAL4rsqyDnl9IYwTSXEK3bKx/JY79kkYPR0hjBNKMkST5r6xx l1POQBorIA1QQBorIA1Q6Ou+QRojkAYoylJFw/AOCnyANIARejQEG0WgO5AGMALSWAFpgALS WAFpgIJ6qOT/HkvTdWEofqLrsix4Isv0GxdBGqADaX79gSAQP5Flcdx1NBxRv/E3pAE6kOZD UdS1lEY+0ivqE5AG6ChpPO2naRpaXUTIoqsi9SEgDdBRUeKlNFQRoyHuc9K8ffvwRP2R/o1a gW+ouUScpOk6GZ8Uq2/fLiZN0EOtzDsnzevX10/kH9EXZQX+wbN61rYyPilWX79e7UyTJH2J cE0D5uEpTZ8N+mnQegbsUQvAQ5oP6KcBNqSpDEivpZkG0gAdFZCQxgikATqQxgJIA3QgjQWQ BujoK/HsnRZzGiENYERRyHiANEYgDdBR8QBpjEAaoANpLIA0QAfSWABpgA6ksQDSAB014RnS GIE0oA+kmQXSgD6QZhZIA/pAmlkgDegj525CGiOQBvSRUxQhjRFIA/qgejYLpAF9IM0skAb0 gTSzQBrQJ0nEdHhIYwTSgD7iDs/6UujcgDSAGSIk6xrSGIE0oA+kmQXSgD6QZhZIA/rkeVVB mknq+vaWFpbeuyAAF8Qa4FyloViliB1/dzNpbm7kUukAyLoHV2koVm9udpcGwgAdKQ3fuGBQ PeNbOGAPIM0snAsH7AGkmYVz4YA9EFcznOMC0gB2QJoZOBcO2AdIMwPnwgH7AGlm4Fw4YB8g zQycCwfsA60SwDkuIA1gB61HwzkuIA1gB6pnM3AuHLAPkGYGzoUD9gHSzMC5cMA+0CoBnOMC 0gB2UFByjgtIA9gBaWbgXDhgH2jCM+e4gDSAHVWVZZzjAtIAhkQR57iANIAhWVYUfONid2ke H+/u9i4Ee5pm7xTY03Vtu3ca7OmXbFXFMV9p7u4eH8ff2Uiah4fr670LwR6eK6SMc6xzeL9k uy4I+Kb++vrhYfwdSDMCpFmLYcmmKd/UrypNkgQfoQLouiyj/7NM3EpBAGnW4tjSlCXf1K8q TRDoF0xZFsfdE3GcZepVSLMWx5ambfmmfkVp2jbobUMqVNf661l2dbVGxtYJ708+WWOr5taY l7DO4WgtFS9LdokyWScKrq70w77Oi6UhOYgwpAqZrop+BoI0kIZYp2QPKE2eU6Nn21KFzCTN /f2rV1dPXH/k7i5fhOvrZbbT5+pqja3e3NzeLr/Vt29fv15+q7e3Yu3tpVmnZJeKgrs7GZ8U q69e3d8vJE3QQ3+nqkiTcWm67v4+e+LhI4+PNQDMeHyU8Umxen+vN2UteqYpS6lGVYWh6ZoG gPPw4sAuijAkTah6RvXg8dYzAM7DAmeDPI+iIIgicek43k8DwHlAFQoARyANAI5AGgAcgTQA OLKBNLybBmhOh566shTNGmUp3s/zMOSV9rYV7ZWXJTtM+76Ikg1DObagX5K8oqJp0pRSk6Zi xo9InUp7v2Q3kIZzIzSNnCsK0WCepqKviQKyrsOQ7nNPhdW2XZemfNJOu1NI0y/ZYdr3hVIh uiKiiEp4WJK8okJOhstzWmGaUpemNL1PaDIs2Q2k4dzdqUbZitQliTou0ogm+ZxP2mkEmyzR fskO074vw5kyw5LkFRVyMlye91PXjwL5fPUEmwbWcIN6m/QUinSr5zzSLuafiLQMS3aY9n2h qo2snlEFrF+S3KKCZFFzwlTqhlHw8fnayeFWPGNQDTYMqTbLW5qmoSrNMaShAJQVXyW6TCu3 qKgqUrvraC02SGMBXaBmmViegrc0caxXy7hLI6uINJCXtzR0ZSubJ4KgbXeXhlvtdQhdkKo1 Unhf0+jjy3+tWzO+poljJQ0N5OV8TUNpUNJQuna+puHWTtKHdqje5HmE1jO1Szm3npVlv3rG u/VMTKEU1XSRup1bz3i1yPdRF4BydhD/fholDe9+mv5A3qP109DzHftpADgXkAYARyANAI5A GgAcgTQAOAJpAHAE0gDgCKTZBLVIvEAf9OLG1Dqddr3rdT3fM5KmR7rrzdZAmg3Rg9omdC+h HvSpd21EtNG1rvcfiMMXSLMhLx9ntcTt9uzOcTToZatyORqQZkMux/Y2TZKUpRheIgZuiIEa w2EckjgWo8rkt5JEDEbRB0OqbYpXh8M65ef7w1xo2oEYSCLOgFnG9yYYewNpNuRSGnpFBLB6 pHfFuFo52FFBA9fFIwU3fZsexSracvtym/SqPkeoP76YfokElb8hBKURYULbokAFzQSk2ZBx afR3xCONrhWfEsPqL78/nPA8lEacLfrD3PvSRJEcXVxV9GtqTtFlWkEfFMyG2Eoj7/kzvDOD vTT6p8alGd79oW2zLIrCUI7zhTRmUDAbYitN06hJUfpF+8ukETMo1ZlGzrlpW9JEtObRhF9x boM0ZlAwG2IrDV3TyElRtLCUQlyluElDFS9x7SKlIV1o2Qshi2h8kI0OYoERXNNMAWk2xF4a 0Xp2OUVLtp65SCMnUFWV0IDaz2hOpan1TGiJ1jMzkOZQLNFPYwf6acxAmkMxPSJgOaqKw8x9 rkCagyFW5lobjD2bAtIA4AikAcCR/wcHSqeKqv8QEwAAADx0RVh0Q29tbWVudAAgSW1hZ2Ug Z2VuZXJhdGVkIGJ5IEVTUCBHaG9zdHNjcmlwdCAoZGV2aWNlPXBubXJhdykKldNUtQAAACJ0 RVh0cHM6SGlSZXNCb3VuZGluZ0JveAA0MTB4NDIwKzgxKzE4NBeLyiUAAAAcdEVYdHBzOkxl dmVsAEFkb2JlLTMuMCBFUFNGLTMuMAqbcLvjAAAAAElFTkSuQmCC ---------6184412361844123 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ---------6184412361844123--