Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell <dwinchell@virtualiron.com>]

[-- Attachment #1: Type: text/plain, Size: 9851 bytes --]

Hi Dan and Keir,

Attached is a patch that fixes some issues with the SYNC policy 
(no_missed_ticks_pending).
I have not tried to make the change the minimal one, but, rather, just 
ported into
the new code what I know to work well. The error for 
no_missed_ticks_pending goes from
over 3% to .03% with this change according to my testing.

Regards,
Dave

Dan Magenheimer wrote:

>Hi Dave --
>
>Did you get your correction ported?  If so, it would be nice to see this get into 3.1.3.
>
>Note that I just did some very limited testing with timer_mode=2(=SYNC=no missed ticks pending)
>on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error I've seen so far
>is 0.012%.  But I haven't tried any exotic loads, just LTP.
>
>Thanks,
>Dan
>
>  
>
>>-----Original Message-----
>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>Sent: Wednesday, December 19, 2007 12:33 PM
>>To: dan.magenheimer@oracle.com
>>Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com; Dong,
>>Eddie; Jiang, Yunhong; Dave Winchell
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>Dan,
>>
>>I did some testing with the constant tsc offset SYNC method 
>>(now called
>>no_missed_ticks_pending)
>>and found the error to be very high, much larger than 1 %, as 
>>I recall.
>>I have not had a chance to submit a correction. I will try to 
>>do it later
>>this week or the first week in January. My version of constant tsc
>>offset SYNC method
>>produces .02 % error, so I just need to port that into the 
>>current code.
>>
>>The error you got for both of those kernels is what I would expect
>>for the default mode, delay_for_missed_ticks.
>>
>>I'll let Keir answer on how to set the time mode.
>>
>>Regards,
>>Dave
>>
>>Dan Magenheimer wrote:
>>
>>    
>>
>>>Anyone make measurements on the final patch?
>>>
>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of 
>>>      
>>>
>>about 0.2% with no load.  This was xen-unstable tip today 
>>with no options specified.  32-bit was about 0.01%.
>>    
>>
>>>I think I missed something... how do I run the various 
>>>      
>>>
>>accounting choices and which ones are known to be appropriate 
>>for which kernels?
>>    
>>
>>>Thanks,
>>>Dan
>>>
>>>
>>>
>>>      
>>>
>>>>-----Original Message-----
>>>>From: xen-devel-bounces@lists.xensource.com
>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of 
>>>>        
>>>>
>>Keir Fraser
>>    
>>
>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>To: Dave Winchell
>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang,
>>>>Yunhong
>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>disables pending
>>>>missed ticks
>>>>
>>>>
>>>>Please take a look at xen-unstable changeset 16545.
>>>>
>>>>-- Keir
>>>>
>>>>On 26/11/07 20:57, "Dave Winchell" 
>>>>        
>>>>
>><dwinchell@virtualiron.com> wrote:
>>    
>>
>>>>
>>>>        
>>>>
>>>>>Keir,
>>>>>
>>>>>The accuracy data I've collected for i/o loads for the
>>>>>various time protocols follows. In addition, the data
>>>>>for cpu loads is shown.
>>>>>
>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>>>The cpu load is usex -e36 on each guest.
>>>>>(usex is available at http://people.redhat.com/anderson/usex.)
>>>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
>>>>>
>>>>>The loads labeled i/o-32 are 32 instances of dd.
>>>>>Also, these are run on 4 cpu AMD box.
>>>>>In addition, there is an idle rh-32bit guest.
>>>>>All three guests are 8vcpu.
>>>>>
>>>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>>>except that the redhat-64 guest has 4 instances of dd.
>>>>>
>>>>>Date Duration Protocol sles, rhat error load
>>>>>
>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu
>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu
>>>>>
>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
>>>>>
>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu
>>>>>
>>>>>
>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8
>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8
>>>>>
>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8
>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
>>>>>
>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8
>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8
>>>>>
>>>>>
>>>>>
>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32
>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
>>>>>
>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32
>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32
>>>>>
>>>>>
>>>>>Overhead measurements:
>>>>>
>>>>>Progress in terms of number of passes through a fixed
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>system workload
>>>>
>>>>
>>>>        
>>>>
>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>The workload was usex -b48.
>>>>>
>>>>>
>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>
>>>>>
>>>>>Conclusions:
>>>>>
>>>>>The only protocol which meets the .05% accuracy requirement for ntp
>>>>>tracking under the loads
>>>>>above is the SYNC protocol. The worst case accuracies for
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>SYNC, MIXED,
>>>>
>>>>
>>>>        
>>>>
>>>>>and ASYNC
>>>>>are .022%, .12%, and .14%, respectively.
>>>>>
>>>>>We could reduce the cost of the SYNC method by only
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>scheduling the extra
>>>>
>>>>
>>>>        
>>>>
>>>>>wakeups if a certain number
>>>>>of ticks are missed.
>>>>>
>>>>>Regards,
>>>>>Dave
>>>>>
>>>>>Keir Fraser wrote:
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>><dwinchell@virtualiron.com> wrote:
>>>>
>>>>
>>>>        
>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>Since I had a high error (~.03%) for the ASYNC method a
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>couple of days ago,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>I ran another ASYNC test. I think there may have been something
>>>>>>>wrong with the code I used a couple of days ago for
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>ASYNC. It may have been
>>>>
>>>>
>>>>        
>>>>
>>>>>>>missing the immediate delivery of interrupt after context
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>switch in.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>My results indicate that either SYNC or ASYNC give
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>acceptable accuracy,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>each running consistently around or under .01%. MIXED has
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>a fairly high
>>>>
>>>>
>>>>        
>>>>
>>>>>>>error of
>>>>>>>greater than .03%. Probably too close to .05% ntp
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>threshold for comfort.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>I don't have an overnight run with SYNC. I plan to leave
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>SYNC running
>>>>
>>>>
>>>>        
>>>>
>>>>>>>over the weekend. If you'd rather I can leave MIXED
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>running instead.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>It may be too early to pick the protocol and I can run
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>more overnight tests
>>>>
>>>>
>>>>        
>>>>
>>>>>>>next week.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>I'm a bit worried about any unwanted side effects of the
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>SYNC+run_timer
>>>>
>>>>
>>>>        
>>>>
>>>>>>approach -- e.g., whether timer wakeups will cause higher
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>system-wide CPU
>>>>
>>>>
>>>>        
>>>>
>>>>>>contention. I find it easier to think through the
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>implications of ASYNC. I'm
>>>>
>>>>
>>>>        
>>>>
>>>>>>surprised that MIXED loses time, and is less accurate than
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>ASYNC. Perhaps it
>>>>
>>>>
>>>>        
>>>>
>>>>>>delivers more timer interrupts than the other approaches,
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>and each interrupt
>>>>
>>>>
>>>>        
>>>>
>>>>>>event causes a small accumulated error?
>>>>>>
>>>>>>Overall I would consider MIXED and ASYNC as favourites and
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>if the latter is
>>>>
>>>>
>>>>        
>>>>
>>>>>>actually more accurate then I can simply revert the changeset that
>>>>>>implemented MIXED.
>>>>>>
>>>>>>Perhaps rather than running more of the same workloads you
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>could try idle
>>>>
>>>>
>>>>        
>>>>
>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>to /dev/null)? We
>>>>
>>>>
>>>>        
>>>>
>>>>>>don't have any data on workloads that aren't CPU bound, so
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>that's really an
>>>>
>>>>
>>>>        
>>>>
>>>>>>obvious place to put any further effort imo.
>>>>>>
>>>>>>-- Keir
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>_______________________________________________
>>>>Xen-devel mailing list
>>>>Xen-devel@lists.xensource.com
>>>>http://lists.xensource.com/xen-devel
>>>>
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>
>>>      
>>>
>>    
>>
>
>  
>


[-- Attachment #2: p.1.04.sync.time --]
[-- Type: text/plain, Size: 1445 bytes --]

diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
--- a/xen/arch/x86/hvm/vpt.c	Thu Dec 06 15:36:07 2007 +0000
+++ b/xen/arch/x86/hvm/vpt.c	Fri Jan 04 17:58:16 2008 -0500
@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
 
     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
     if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) )
-        pt->do_not_freeze = !pt->pending_intr_nr;
+        pt->do_not_freeze = 1;
     else
         pt->pending_intr_nr += missed_ticks;
     pt->scheduled += missed_ticks * pt->period;
@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
 
     pt_lock(pt);
 
-    pt->pending_intr_nr++;
+    if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) {
+        pt->pending_intr_nr = 1;
+	pt->do_not_freeze = 0;
+    }
+    else
+	pt->pending_intr_nr++;
 
     if ( !pt->one_shot )
     {
@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
         return;
     }
 
-    pt->do_not_freeze = 0;
-
     if ( pt->one_shot )
     {
         pt->enabled = 0;
@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
             pt->last_plt_gtime = hvm_get_guest_time(v);
             pt->pending_intr_nr = 0; /* 'collapse' all missed ticks */
         }
+	else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
+	    pt->pending_intr_nr--;
+	    pt->last_plt_gtime = hvm_get_guest_time(v);
+	}
         else
         {
             pt->last_plt_gtime += pt->period_cycles;

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel