From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 09 Jan 2008 11:53:07 -0500 Message-ID: <4784FBF3.3040503@virtualiron.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "dan.magenheimer@oracle.com" , "xen-devel@lists.xensource.com" , Dave Winchell List-Id: xen-devel@lists.xenproject.org Hi Keir, The latest change, c/s 16690, looks fine. I agree that the code in c/s 16690 is equivalent to the code I submitted. Also, your version is more concise. The error tests confirm the equivalence. With overnight cpu loads, the checked in version was accurate to +.048% for sles and +.038% for red hat. My version was +.046% and +.032% in a 2 hour test. I don't think the difference is significant. i/o loads produced errors of +.01%. Thanks for all your efforts on this issue. Regards, Dave Keir Fraser wrote: >Applied as c/s 16690, although the checked-in patch is smaller. I think the >only important fix is to pt_intr_post() and the only bit of the patch I >totally omitted was the change to pt_process_missed_ticks(). I don't think >that change can be important, but let's see what happens to the error >percentage... > > -- Keir > >On 4/1/08 23:24, "Dave Winchell" wrote: > > > >>Hi Dan and Keir, >> >>Attached is a patch that fixes some issues with the SYNC policy >>(no_missed_ticks_pending). >>I have not tried to make the change the minimal one, but, rather, just >>ported into >>the new code what I know to work well. The error for >>no_missed_ticks_pending goes from >>over 3% to .03% with this change according to my testing. >> >>Regards, >>Dave >> >>Dan Magenheimer wrote: >> >> >> >>>Hi Dave -- >>> >>>Did you get your correction ported? If so, it would be nice to see this get >>>into 3.1.3. >>> >>>Note that I just did some very limited testing with timer_mode=2(=SYNC=no >>>missed ticks pending) >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error I've >>>seen so far >>>is 0.012%. But I haven't tried any exotic loads, just LTP. >>> >>>Thanks, >>>Dan >>> >>> >>> >>> >>> >>>>-----Original Message----- >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>Sent: Wednesday, December 19, 2007 12:33 PM >>>>To: dan.magenheimer@oracle.com >>>>Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com; Dong, >>>>Eddie; Jiang, Yunhong; Dave Winchell >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>disables pending >>>>missed ticks >>>> >>>> >>>>Dan, >>>> >>>>I did some testing with the constant tsc offset SYNC method >>>>(now called >>>>no_missed_ticks_pending) >>>>and found the error to be very high, much larger than 1 %, as >>>>I recall. >>>>I have not had a chance to submit a correction. I will try to >>>>do it later >>>>this week or the first week in January. My version of constant tsc >>>>offset SYNC method >>>>produces .02 % error, so I just need to port that into the >>>>current code. >>>> >>>>The error you got for both of those kernels is what I would expect >>>>for the default mode, delay_for_missed_ticks. >>>> >>>>I'll let Keir answer on how to set the time mode. >>>> >>>>Regards, >>>>Dave >>>> >>>>Dan Magenheimer wrote: >>>> >>>> >>>> >>>> >>>> >>>>>Anyone make measurements on the final patch? >>>>> >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of >>>>> >>>>> >>>>> >>>>> >>>>about 0.2% with no load. This was xen-unstable tip today >>>>with no options specified. 32-bit was about 0.01%. >>>> >>>> >>>> >>>> >>>>>I think I missed something... how do I run the various >>>>> >>>>> >>>>> >>>>> >>>>accounting choices and which ones are known to be appropriate >>>>for which kernels? >>>> >>>> >>>> >>>> >>>>>Thanks, >>>>>Dan >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>>-----Original Message----- >>>>>>From: xen-devel-bounces@lists.xensource.com >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>>> >>>>>> >>>>>> >>>>>> >>>>Keir Fraser >>>> >>>> >>>> >>>> >>>>>>Sent: Thursday, December 06, 2007 4:57 AM >>>>>>To: Dave Winchell >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang, >>>>>>Yunhong >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>>>disables pending >>>>>>missed ticks >>>>>> >>>>>> >>>>>>Please take a look at xen-unstable changeset 16545. >>>>>> >>>>>>-- Keir >>>>>> >>>>>>On 26/11/07 20:57, "Dave Winchell" >>>>>> >>>>>> >>>>>> >>>>>> >>>> wrote: >>>> >>>> >>>> >>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>Keir, >>>>>>> >>>>>>>The accuracy data I've collected for i/o loads for the >>>>>>>various time protocols follows. In addition, the data >>>>>>>for cpu loads is shown. >>>>>>> >>>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box. >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. >>>>>>>The cpu load is usex -e36 on each guest. >>>>>>>(usex is available at http://people.redhat.com/anderson/usex.) >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null. >>>>>>> >>>>>>>The loads labeled i/o-32 are 32 instances of dd. >>>>>>>Also, these are run on 4 cpu AMD box. >>>>>>>In addition, there is an idle rh-32bit guest. >>>>>>>All three guests are 8vcpu. >>>>>>> >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 >>>>>>>except that the redhat-64 guest has 4 instances of dd. >>>>>>> >>>>>>>Date Duration Protocol sles, rhat error load >>>>>>> >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu >>>>>>> >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu >>>>>>> >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu >>>>>>> >>>>>>> >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8 >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8 >>>>>>> >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8 >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 >>>>>>> >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8 >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8 >>>>>>> >>>>>>> >>>>>>> >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32 >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 >>>>>>> >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32 >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32 >>>>>>> >>>>>>> >>>>>>>Overhead measurements: >>>>>>> >>>>>>>Progress in terms of number of passes through a fixed >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>system workload >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. >>>>>>>The workload was usex -b48. >>>>>>> >>>>>>> >>>>>>>ASYNC 167 min 145 passes .868 passes/min >>>>>>>SYNC 167 min 144 passes .862 passes/min >>>>>>>SYNC 1065 min 919 passes .863 passes/min >>>>>>>MIXED 221 min 196 passes .887 passes/min >>>>>>> >>>>>>> >>>>>>>Conclusions: >>>>>>> >>>>>>>The only protocol which meets the .05% accuracy requirement for ntp >>>>>>>tracking under the loads >>>>>>>above is the SYNC protocol. The worst case accuracies for >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>SYNC, MIXED, >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>and ASYNC >>>>>>>are .022%, .12%, and .14%, respectively. >>>>>>> >>>>>>>We could reduce the cost of the SYNC method by only >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>scheduling the extra >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>wakeups if a certain number >>>>>>>of ticks are missed. >>>>>>> >>>>>>>Regards, >>>>>>>Dave >>>>>>> >>>>>>>Keir Fraser wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>On 9/11/07 19:22, "Dave Winchell" >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>Since I had a high error (~.03%) for the ASYNC method a >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>couple of days ago, >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>I ran another ASYNC test. I think there may have been something >>>>>>>>>wrong with the code I used a couple of days ago for >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>ASYNC. It may have been >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>missing the immediate delivery of interrupt after context >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>switch in. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>My results indicate that either SYNC or ASYNC give >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>acceptable accuracy, >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>each running consistently around or under .01%. MIXED has >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>a fairly high >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>error of >>>>>>>>>greater than .03%. Probably too close to .05% ntp >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>threshold for comfort. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>I don't have an overnight run with SYNC. I plan to leave >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>SYNC running >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>over the weekend. If you'd rather I can leave MIXED >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>running instead. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>It may be too early to pick the protocol and I can run >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>more overnight tests >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>next week. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>I'm a bit worried about any unwanted side effects of the >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>SYNC+run_timer >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>approach -- e.g., whether timer wakeups will cause higher >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>system-wide CPU >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>contention. I find it easier to think through the >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>implications of ASYNC. I'm >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>surprised that MIXED loses time, and is less accurate than >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>ASYNC. Perhaps it >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>delivers more timer interrupts than the other approaches, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>and each interrupt >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>event causes a small accumulated error? >>>>>>>> >>>>>>>>Overall I would consider MIXED and ASYNC as favourites and >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>if the latter is >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>actually more accurate then I can simply revert the changeset that >>>>>>>>implemented MIXED. >>>>>>>> >>>>>>>>Perhaps rather than running more of the same workloads you >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>could try idle >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>to /dev/null)? We >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>don't have any data on workloads that aren't CPU bound, so >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>that's really an >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>obvious place to put any further effort imo. >>>>>>>> >>>>>>>>-- Keir >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>_______________________________________________ >>>>>>Xen-devel mailing list >>>>>>Xen-devel@lists.xensource.com >>>>>>http://lists.xensource.com/xen-devel >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>> >>> >>> >>> >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000 >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500 >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru >> >> missed_ticks = missed_ticks / (s_time_t) pt->period + 1; >> if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) >>- pt->do_not_freeze = !pt->pending_intr_nr; >>+ pt->do_not_freeze = 1; >> else >> pt->pending_intr_nr += missed_ticks; >> pt->scheduled += missed_ticks * pt->period; >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) >> >> pt_lock(pt); >> >>- pt->pending_intr_nr++; >>+ if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) { >>+ pt->pending_intr_nr = 1; >>+ pt->do_not_freeze = 0; >>+ } >>+ else >>+ pt->pending_intr_nr++; >> >> if ( !pt->one_shot ) >> { >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct >> return; >> } >> >>- pt->do_not_freeze = 0; >>- >> if ( pt->one_shot ) >> { >> pt->enabled = 0; >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct >> pt->last_plt_gtime = hvm_get_guest_time(v); >> pt->pending_intr_nr = 0; /* 'collapse' all missed ticks */ >> } >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) { >>+ pt->pending_intr_nr--; >>+ pt->last_plt_gtime = hvm_get_guest_time(v); >>+ } >> else >> { >> pt->last_plt_gtime += pt->period_cycles; >> >> > > > >