From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Fri, 04 Jan 2008 18:24:02 -0500 Message-ID: <477EC012.6040206@virtualiron.com> References: <20080103155752531.00000000500@djm-pc> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------000708050109000100010403" Return-path: In-Reply-To: <20080103155752531.00000000500@djm-pc> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" , Keir Fraser Cc: Dave Winchell , "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --------------000708050109000100010403 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi Dan and Keir, Attached is a patch that fixes some issues with the SYNC policy (no_missed_ticks_pending). I have not tried to make the change the minimal one, but, rather, just ported into the new code what I know to work well. The error for no_missed_ticks_pending goes from over 3% to .03% with this change according to my testing. Regards, Dave Dan Magenheimer wrote: >Hi Dave -- > >Did you get your correction ported? If so, it would be nice to see this get into 3.1.3. > >Note that I just did some very limited testing with timer_mode=2(=SYNC=no missed ticks pending) >on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error I've seen so far >is 0.012%. But I haven't tried any exotic loads, just LTP. > >Thanks, >Dan > > > >>-----Original Message----- >>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>Sent: Wednesday, December 19, 2007 12:33 PM >>To: dan.magenheimer@oracle.com >>Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com; Dong, >>Eddie; Jiang, Yunhong; Dave Winchell >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>disables pending >>missed ticks >> >> >>Dan, >> >>I did some testing with the constant tsc offset SYNC method >>(now called >>no_missed_ticks_pending) >>and found the error to be very high, much larger than 1 %, as >>I recall. >>I have not had a chance to submit a correction. I will try to >>do it later >>this week or the first week in January. My version of constant tsc >>offset SYNC method >>produces .02 % error, so I just need to port that into the >>current code. >> >>The error you got for both of those kernels is what I would expect >>for the default mode, delay_for_missed_ticks. >> >>I'll let Keir answer on how to set the time mode. >> >>Regards, >>Dave >> >>Dan Magenheimer wrote: >> >> >> >>>Anyone make measurements on the final patch? >>> >>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of >>> >>> >>about 0.2% with no load. This was xen-unstable tip today >>with no options specified. 32-bit was about 0.01%. >> >> >>>I think I missed something... how do I run the various >>> >>> >>accounting choices and which ones are known to be appropriate >>for which kernels? >> >> >>>Thanks, >>>Dan >>> >>> >>> >>> >>> >>>>-----Original Message----- >>>>From: xen-devel-bounces@lists.xensource.com >>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>> >>>> >>Keir Fraser >> >> >>>>Sent: Thursday, December 06, 2007 4:57 AM >>>>To: Dave Winchell >>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang, >>>>Yunhong >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>disables pending >>>>missed ticks >>>> >>>> >>>>Please take a look at xen-unstable changeset 16545. >>>> >>>>-- Keir >>>> >>>>On 26/11/07 20:57, "Dave Winchell" >>>> >>>> >> wrote: >> >> >>>> >>>> >>>> >>>>>Keir, >>>>> >>>>>The accuracy data I've collected for i/o loads for the >>>>>various time protocols follows. In addition, the data >>>>>for cpu loads is shown. >>>>> >>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box. >>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. >>>>>The cpu load is usex -e36 on each guest. >>>>>(usex is available at http://people.redhat.com/anderson/usex.) >>>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null. >>>>> >>>>>The loads labeled i/o-32 are 32 instances of dd. >>>>>Also, these are run on 4 cpu AMD box. >>>>>In addition, there is an idle rh-32bit guest. >>>>>All three guests are 8vcpu. >>>>> >>>>>The loads labeled i/o-4/32 are the same as i/o-32 >>>>>except that the redhat-64 guest has 4 instances of dd. >>>>> >>>>>Date Duration Protocol sles, rhat error load >>>>> >>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu >>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu >>>>> >>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu >>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu >>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu >>>>> >>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu >>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu >>>>> >>>>> >>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8 >>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8 >>>>> >>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8 >>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 >>>>> >>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8 >>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8 >>>>> >>>>> >>>>> >>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 >>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32 >>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 >>>>> >>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32 >>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32 >>>>> >>>>> >>>>>Overhead measurements: >>>>> >>>>>Progress in terms of number of passes through a fixed >>>>> >>>>> >>>>> >>>>> >>>>system workload >>>> >>>> >>>> >>>> >>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. >>>>>The workload was usex -b48. >>>>> >>>>> >>>>>ASYNC 167 min 145 passes .868 passes/min >>>>>SYNC 167 min 144 passes .862 passes/min >>>>>SYNC 1065 min 919 passes .863 passes/min >>>>>MIXED 221 min 196 passes .887 passes/min >>>>> >>>>> >>>>>Conclusions: >>>>> >>>>>The only protocol which meets the .05% accuracy requirement for ntp >>>>>tracking under the loads >>>>>above is the SYNC protocol. The worst case accuracies for >>>>> >>>>> >>>>> >>>>> >>>>SYNC, MIXED, >>>> >>>> >>>> >>>> >>>>>and ASYNC >>>>>are .022%, .12%, and .14%, respectively. >>>>> >>>>>We could reduce the cost of the SYNC method by only >>>>> >>>>> >>>>> >>>>> >>>>scheduling the extra >>>> >>>> >>>> >>>> >>>>>wakeups if a certain number >>>>>of ticks are missed. >>>>> >>>>>Regards, >>>>>Dave >>>>> >>>>>Keir Fraser wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>>On 9/11/07 19:22, "Dave Winchell" >>>>>> >>>>>> >>>>>> >>>>>> >>>> wrote: >>>> >>>> >>>> >>>> >>>>>> >>>>>> >>>>>> >>>>>>>Since I had a high error (~.03%) for the ASYNC method a >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>couple of days ago, >>>> >>>> >>>> >>>> >>>>>>>I ran another ASYNC test. I think there may have been something >>>>>>>wrong with the code I used a couple of days ago for >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>ASYNC. It may have been >>>> >>>> >>>> >>>> >>>>>>>missing the immediate delivery of interrupt after context >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>switch in. >>>> >>>> >>>> >>>> >>>>>>>My results indicate that either SYNC or ASYNC give >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>acceptable accuracy, >>>> >>>> >>>> >>>> >>>>>>>each running consistently around or under .01%. MIXED has >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>a fairly high >>>> >>>> >>>> >>>> >>>>>>>error of >>>>>>>greater than .03%. Probably too close to .05% ntp >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>threshold for comfort. >>>> >>>> >>>> >>>> >>>>>>>I don't have an overnight run with SYNC. I plan to leave >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>SYNC running >>>> >>>> >>>> >>>> >>>>>>>over the weekend. If you'd rather I can leave MIXED >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>running instead. >>>> >>>> >>>> >>>> >>>>>>>It may be too early to pick the protocol and I can run >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>more overnight tests >>>> >>>> >>>> >>>> >>>>>>>next week. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>I'm a bit worried about any unwanted side effects of the >>>>>> >>>>>> >>>>>> >>>>>> >>>>SYNC+run_timer >>>> >>>> >>>> >>>> >>>>>>approach -- e.g., whether timer wakeups will cause higher >>>>>> >>>>>> >>>>>> >>>>>> >>>>system-wide CPU >>>> >>>> >>>> >>>> >>>>>>contention. I find it easier to think through the >>>>>> >>>>>> >>>>>> >>>>>> >>>>implications of ASYNC. I'm >>>> >>>> >>>> >>>> >>>>>>surprised that MIXED loses time, and is less accurate than >>>>>> >>>>>> >>>>>> >>>>>> >>>>ASYNC. Perhaps it >>>> >>>> >>>> >>>> >>>>>>delivers more timer interrupts than the other approaches, >>>>>> >>>>>> >>>>>> >>>>>> >>>>and each interrupt >>>> >>>> >>>> >>>> >>>>>>event causes a small accumulated error? >>>>>> >>>>>>Overall I would consider MIXED and ASYNC as favourites and >>>>>> >>>>>> >>>>>> >>>>>> >>>>if the latter is >>>> >>>> >>>> >>>> >>>>>>actually more accurate then I can simply revert the changeset that >>>>>>implemented MIXED. >>>>>> >>>>>>Perhaps rather than running more of the same workloads you >>>>>> >>>>>> >>>>>> >>>>>> >>>>could try idle >>>> >>>> >>>> >>>> >>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads >>>>>> >>>>>> >>>>>> >>>>>> >>>>to /dev/null)? We >>>> >>>> >>>> >>>> >>>>>>don't have any data on workloads that aren't CPU bound, so >>>>>> >>>>>> >>>>>> >>>>>> >>>>that's really an >>>> >>>> >>>> >>>> >>>>>>obvious place to put any further effort imo. >>>>>> >>>>>>-- Keir >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>_______________________________________________ >>>>Xen-devel mailing list >>>>Xen-devel@lists.xensource.com >>>>http://lists.xensource.com/xen-devel >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >>> >> >> > > > --------------000708050109000100010403 Content-Type: text/plain; name="p.1.04.sync.time" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="p.1.04.sync.time" diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c --- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000 +++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500 @@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru missed_ticks = missed_ticks / (s_time_t) pt->period + 1; if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) - pt->do_not_freeze = !pt->pending_intr_nr; + pt->do_not_freeze = 1; else pt->pending_intr_nr += missed_ticks; pt->scheduled += missed_ticks * pt->period; @@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) pt_lock(pt); - pt->pending_intr_nr++; + if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) { + pt->pending_intr_nr = 1; + pt->do_not_freeze = 0; + } + else + pt->pending_intr_nr++; if ( !pt->one_shot ) { @@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct return; } - pt->do_not_freeze = 0; - if ( pt->one_shot ) { pt->enabled = 0; @@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct pt->last_plt_gtime = hvm_get_guest_time(v); pt->pending_intr_nr = 0; /* 'collapse' all missed ticks */ } + else if ( mode_is(v->domain, no_missed_ticks_pending) ) { + pt->pending_intr_nr--; + pt->last_plt_gtime = hvm_get_guest_time(v); + } else { pt->last_plt_gtime += pt->period_cycles; --------------000708050109000100010403 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --------------000708050109000100010403--